Interface for the world: researchers present innovative AR interaction model
With "Reality Proxy", researchers have developed an AR operating concept that facilitates remote interaction with real objects.
Object interaction using digital substitutes ("proxies").
(Image: Liu et al. / NYU / University of Minnesota / Google)
AR glasses could one day provide a user interface for the world: for example, the light could be switched off from the sofa, a book on the shelf could be highlighted and displayed as digital text or the vacuum robot could be selected and sent to a dusty corner.
The question is how an interface of this kind could be designed to be user-friendly. After all, the world is three-dimensional and much more complex than a computer desktop or home screen. How do you effortlessly select objects that are far away and therefore appear tiny in the field of vision, are located between many others or are partially obscured by them? Whether by hand pointing or eye focus: these methods are prone to error and often exhausting.
Videos by heise
Researchers from New York University, the University of Minnesota, and Google are now proposing an unusual approach to interacting with such objects: Their interaction model, called “Reality Proxy,” fades in an interface within reach of the user, via which objects can be selected using digital substitutes (“proxies”) of the same. This abstracts from individual object properties such as position, size, and distance, which would make selection or manipulation more difficult. With the help of the displayed proxies, users can quickly jump from object to object, select several objects at the same time, group them, filter them by attributes, and more.
Tested with Apple Vision Pro
In a video, the researchers show some unusual application scenarios: from controlling several drones to spatial navigation aids for buildings.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externes YouTube-Video (Google Ireland Limited) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (Google Ireland Limited) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
The system requires the selectable objects to be correctly segmented and semantically identified by an AI in advance. The AI model must therefore be able to separate individual objects from one another and assign them to logical groups (light switch, book, vacuum robot). A task that can be very challenging in itself.
The researchers tested Reality Proxy with an Apple Vision Pro. In principle, however, the system could also be used on AR glasses and implemented directly in the operating system as a basic function.
The research work is freely accessible on the Internet.
(mki)
