Apple AI Revolutionizes Photos With 3D Scenes
Apple Intelligence: A Look at its Current Progress
As WWDC 2025 approaches, it will signify one year since the introduction of Apple Intelligence for iPhone, iPad, and Mac. Currently, however, there isn't a wealth of exciting features, particularly when measured against the capabilities of AI tools such as ChatGPT and Gemini.
This situation was largely unforeseen a year ago. While some delays for Apple Intelligence were anticipated, the expectation was that it would eventually be released. Instead, Apple was compelled to withhold its primary feature: an enhanced Siri capable of learning from user data on iPhones and managing app functionalities.
Apple still has an opportunity to close the gap, and the advanced Siri experience envisioned previously is likely in development. WWDC 2025 is also anticipated to unveil new Apple Intelligence 2.0 features, expected to arrive with iOS 19, iPadOS 19, and macOS 16.
Introducing Matrix3D: Apple's Leap into AI-Powered 3D Scene Generation
Meanwhile, a promising advancement has emerged from Apple's AI research, although its integration timeline with Apple Intelligence remains uncertain. Researchers at Apple have unveiled a model named Matrix3D, capable of generating 3D scenes from merely three photographs of an object.
This represents a significant breakthrough with potential for immediate application in Apple's software. Nevertheless, it is still unclear if Apple intends to incorporate this technology into its applications in the near future.
Apple researchers, working with scientists from Nanjing University and the Hong Kong University of Science and Technology, have devised a method to create 3D scenes from still images using artificial intelligence. The outcomes, as demonstrated in examples available on GitHub, are remarkably impressive, resembling actual video footage captured by a moving camera.
Image source: GitHub
How Matrix3D Magically Creates 3D Worlds from Photos
These 3D scenes are produced by analyzing the actual photographs provided for each instance. Matrix3D determines the original camera positioning, predicts pixel depth, and subsequently generates new perspectives to create the desired 3D view.
An interesting aspect is Matrix3D's training process. Researchers provided the AI with images of objects, concealed some of the extractable data, and allowed the model to independently process the remaining information.
The research team utilized tens of millions of images and depth frames sourced from six public datasets to train the AI in constructing 3D scenes from conventional photographs.
The Big Question: Will Matrix3D Enhance Your Apple Experience?
How does this benefit the average iPhone user looking to leverage Apple Intelligence? This is a question Apple must address as it continues to publish its AI research. The value of impressive AI breakthroughs diminishes if they are not integrated into consumer products, similar to showcasing potential iPhone patents without confirming their inclusion in future devices.
Image source: GitHub
Exploring the Potential: How Matrix3D Could Reshape Apple's Ecosystem
Despite the uncertainties, it is easy to envision practical applications for Matrix3D within the Apple Intelligence framework. Users could potentially edit their photos using AI, creating alternative perspectives or repositioning elements within the scene.
Apple currently enables iPhone users to capture Spatial Photos on devices such as the iPhone 16 series, which are optimally viewed on the Vision Pro headset.
Matrix3D could also serve as a potent tool for developers creating augmented reality (AR) and virtual reality (VR) content for Vision Pro. Similarly, game developers for iPhone, iPad, and Mac could leverage Matrix3D to streamline the creation of 3D assets, moving away from more resource-intensive methods.
Furthermore, the extensive market for shopping applications on iPhone presents another opportunity. AI tools like Matrix3D could allow users to view 3D representations of products, like furniture, prior to making a purchase decision.
Given that Apple has made both the research paper and the code publicly available online, other AI companies may begin to develop their own iterations of Matrix3D.
For now, these possibilities remain speculative, but there is optimism that Apple will integrate some of its AI research into tangible products via Apple Intelligence.
In the interim, you can explore the Matrix3D page on GitHub and view the 3D scene samples shared by Apple, some of which are interactive.
