Virtual Reality with ChatGPT: Voice-controlled hand replaces controller
A virtual hand is supposed to follow voice commands and make virtual reality usable without hands and controllers – thanks to ChatGPT.
The control of VR glasses could soon work without hand tracking or controllers.
(Image: Apple)
A research team from the University of Michigan has developed software called "HandProxy" that aims to make interacting with virtual reality and augmented reality more intuitive. Instead of controllers or hand tracking, voice commands will suffice in the future: a digital hand moves, grasps objects, moves windows, or even tidies up virtual workspaces at a spoken command. The researchers want to improve accessibility and open up new application possibilities in AR and VR environments.
Unlike many previous voice controls that only understand predefined commands like "Open menu" or "Change volume," "HandProxy" is designed to react flexibly to context. The software is based on Open AI's GPT-4o language model and, according to the developers, independently recognizes which action is intended next. For example, if someone says "Grab the peach" while the hand is already holding another object, the system automatically drops it. Shortened commands like "Do that again" or "Three times in a row" should also be interpreted correctly.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externes YouTube-Video (Google Ireland Limited) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (Google Ireland Limited) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
Voice replaces hand tracking
Doctoral student Chen Liang, the lead author of the study, describes HandProxy as a kind of "virtual proxy" that translates voice commands directly into hand movements without individual applications needing to be adapted. Technically, the app uses the same interfaces as hand tracking systems or VR controllers. This means it should also work seamlessly with existing VR apps, such as in games or construction environments.
Co-author and computer science professor Anhong Guo adds: "Mobile devices have long had assistance functions, but such alternatives are almost completely missing in VR." With HandProxy, on the other hand, you can seamlessly switch between voice, gestures, and controllers. People with motor impairments could particularly benefit from this. A voice-controlled hand would also be practical in everyday situations like cooking or crafting. AR glasses like the Snap Specs announced for next year offer, for example, assistive AR applications for the kitchen or repairing devices.
Videos by heise
Initial tests with 20 participants were promising, according to the researchers: most of the 781 tasks were successfully executed after a few attempts. Problems occurred with commands like "Like the photo" when the system could not clearly identify the correct button. In total, "HandProxy" failed on 64 tasks. In the future, the AI is intended to ask targeted follow-up questions in such cases to avoid misunderstandings.
Currently, "HandProxy" exists only as a demo app, but in the long term, the technology could also be incorporated into real robotic arms. A patent has already been applied for, and the team is looking for partners for a possible market launch. The project was funded by the University of Michigan.
(joe)
