Photos become 3D environments: Try out Apple's new AI model with iPhone
Apple has released an AI model that transforms single photos into 3D environments. An app makes the technology accessible for free on the iPhone.
On the left, a conventional photo; on the right, a 3D environment generated by Apple's algorithm based on the photo. The face has been subsequently anonymized.
(Image: Tomislav Bezmalinović / heise medien)
The AI model is named SHARP (Single-image High-Accuracy Real-time Parallax) and was released in December along with a research paper. The source code is available on Github.
Until now, basic programming knowledge was required to use the model on a Mac or Windows PC. Now, SHARP can be tried out for free on the iPhone with the iOS app "Sapling". The computation is done locally on the device and takes about a minute per photo on newer iPhones.
In developing the AI model, Apple's researchers pursued several goals: Firstly, to ensure the fastest possible 3D reconstruction from single images, enabling immersive exploration of personal photo libraries in the future. Secondly, the 3D environments should be rendered in real-time and allow for perspective changes, similar to what occurs when looking around with VR goggles. The rendering is done using 3D Gaussian Splatting, a technique that Apple also uses for its Personas on the Apple Vision Pro.
Videos by heise
According to the researchers, SHARP delivers significantly higher image quality than other AI models while requiring only a fraction of the computing time. Whether and in what form the AI model will be integrated into Apple's products is currently unknown.
Apple SHARP tried out with iPhone
SHARP differs from existing immersive Apple features such as spatial photos and videos and spatial scenes. The former are stereoscopic recordings with two slightly offset images for the left and right eye, which only unfold their depth effect on devices like Apple Vision Pro and Meta Quest. Spatial scenes, in turn, were introduced with visionOS, iOS, and iPadOS 26 and go a step further by allowing slight perspective changes. This feature can be tried out on iPhones by tapping the hexagonal icon in the upper right corner of a photo. SHARP significantly expands this approach and creates a 3D environment that allows for more comprehensive reconstruction and greater perspective changes.
The 3D scanning app "Sapling" is among the first applications that have integrated Apple's AI model and enable free use via a graphical user interface. Users select the "Image to 3D Scene" function, choose a photo from the Photos app, and start the generation. On an iPhone 15 Pro, the 3D reconstruction takes about a minute. Subsequently, the scene can be rendered on the iPhone: either as an interactive representation against a black background or as an AR view directly in the room.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externes YouTube-Video (Google Ireland Limited) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (Google Ireland Limited) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
In terms of resolution and sharpness, the result is convincing. However, how impressive the 3D reconstruction is depends heavily on the subject: images with depth stratification and a lot of space work better than close-ups of people or objects. Like other AI-powered reconstruction methods, SHARP also has difficulties with non-visible areas, for example, behind photographed people. As soon as you move too far away from the original perspective, artifacts, and distortions become visible, destroying the illusion.
Users can export the resulting Gaussian splats for further processing in their original file format. In the future, developer Matthew Waller also wants to support the AI model on the Vision Pro version of his app. With "Splat Studio", there is already a third-party app specifically developed for this purpose. We have not yet been able to find any programs for macOS or Windows that integrate Apple SHARP into a graphical user interface.
(wpl)
