Portals to the past: a VR app makes memories accessible
Black Mirror sends its regards: the smartphone app "Wist" creates walk-in memories from videos. We have tried it out.
An immersive memory, seen through the lenses of a Quest 3.
(Image: tobe)
Wist transforms iPhone videos into room-filling scenes that Wist CEO Andrew McHugh calls “spatial memories”. The AR constructs can be faded into the real room and entered with mixed reality glasses such as Meta Quest or Apple Vision Pro. The technology is intended to keep memories alive and make them tangible in a way that was previously not possible.
Wist uses various data sources for the 3D reconstruction of the videos, including a depth image generated by the LiDAR sensor and the position and orientation of the iPhone in space, determined via the motion sensors. Since Wist stores the raw data, the spatial memories can be continuously improved by updating the reconstruction pipeline. The app supports all Pro and Pro Max models from iPhone 13 onwards.
No technical know-how required
We did the self-experiment and tried out Wist for an afternoon. Access to the early access version via TestFlight is currently required. It is simple and intuitive to use: the app starts directly in recording mode so that the desired moment can be captured immediately, just like with a normal camera app. The video is then processed, a process that can currently take several hours. As soon as the 3D reconstruction is complete, you will receive a notification. The maximum video length is currently 40 seconds.
Videos by heise
The “spatialized” videos can be projected directly into the room via AR view, either on the iPhone display or much more immersively with a headset such as Meta Quest 3 or Apple Vision Pro. A dedicated mixed reality app is available for the devices. As the saved reminders are uploaded from the smartphone to the cloud, no cumbersome data transfer is necessary: the scenes can be called up immediately when the application is started.
Trying out Wist with Quest 3: a bit of a holodeck
These scenes only really come into their own in mixed reality glasses. Thanks to the wide field of vision and stereoscopic depth, the subjects appear vivid and present: in our case, a dog that has made itself comfortable on a patch of grass in the summer heat. The fact that the pet and the surroundings appear in their original size directly in front of you further enhances the impression of reality.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externes YouTube-Video (Google Ireland Limited) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (Google Ireland Limited) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
In the evening, we returned to the same location and used the passthrough mode to place the recording exactly over the original scene to test the spatial accuracy of the 3D reconstruction. It blended amazingly seamlessly into the real geometry. We stood in the same place and looked at the mixed reality projection of a dog that had been lying there a few hours ago.
(Image:Â tobe)
The illusion breaks down as soon as you move too far away from the original camera position. Objects fray and cast translucent shadows: empty spaces that the camera could not capture and therefore cannot reconstruct. In the future, AI could fill these gaps. At present, these artifacts are still irritating or, depending on the angle of view, add to the dystopian charm of spatial memories.
Immersive memories: These solutions exist
Apple is one of the few major companies that sees potential in the idea of immersive memories. It promotes Apple Vision Pro as a device that allows memories to be captured and experienced more vividly.
The mixed reality headset and newer iPhones can be used to record “spatial videos” in rectangular format. This is based on a well-known technology: stereoscopic recordings that convey spatial depth when played back in the Vision Pro or other headsets. Special 3D cameras achieve the same effect, often in combination with a fisheye lens that enables a 180- or even 360-degree perspective. The standpoint is fixed with all these solutions, unlike Wist, which in principle allows a free change of perspective.
With visionOS 26, Apple introduced “Spatial Scenes,” a function that at least partially goes in this direction: it transforms 2D photos into multi-perspective backdrops using generative AI. This function is not available for videos.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externes YouTube-Video (Google Ireland Limited) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (Google Ireland Limited) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
Wist goes beyond Apple's solutions with its spatial memories and accepts that the result is imperfect. However, even Wist cannot offer an all-round view in which people or events are fully visible from every perspective.
A technique called Gaussian splatting, which is now also being used on Meta Quest, is suitable for snapshots of this kind. However, recording moving scenes remains a considerable technical hurdle.
Reality inspires sci-fi and vice versa
It is not yet possible to estimate what emotional impact the spatial memories we record will have one day. There is not enough time to do so. In a broader context and with a view to long-term developments, the question arises as to how immersive memory techniques will affect human identity and whether nostalgia could become an addiction once a certain degree of technical perfection is reached. This is a topic that science fiction has addressed many times.
The latest Black Mirror season provides a surprisingly optimistic answer to this question: in the episode “Eulogy,” a man gains a new, healing view of his past through technically reconstructed memories. The technology shown is strongly reminiscent of Wist and related techniques.
(dahe)
