With AI: Apple introduces new accessibility features
With the next versions of iOS, iPadOS, watchOS, and macOS, Apple plans new features not just for people with disabilities. They use Apple Intelligence.
Apple's Live Recognition: Helps not only visually impaired people.
(Image: Apple)
Apple has announced various improvements to its accessibility features, affecting VoiceOver, Voice Control, Accessibility Reader, and video captioning, among others. The company also demonstrated how a wheelchair can be controlled via eye tracking with the Vision Pro. The features are expected to be released in the coming months, likely alongside iOS 27, iPadOS 27, macOS 27, and watchOS 27, which are anticipated for autumn. Apple will present the new operating systems at its developer conference WWDC 2026 on June 8.
Image analysis, live recognition, and more thanks to AI
As is well known, VoiceOver allows users to have the interface of Apple devices explained to them and then operate them directly – for example, if they are visually impaired. However, the function can also describe the surroundings. With the so-called Image Explorer, VoiceOver will in the future allow all images appearing on the system to be analyzed. Thanks to Apple Intelligence, this will be possible for photos, documents, scanned invoices, and more – even if they are purely image files. The improved Magnifier also uses AI to describe the surroundings and then presents it in a way that people with low vision can see.
(Image:Â Apple)
The so-called Live Recognition via the iPhone's Action button recognizes what is in the field of view and allows questions to be asked – for example, what is written on an elevator. For the first time, it is possible to ask follow-up questions that relate to what has already been said. Voice Control, the system control via voice for people who cannot operate the Apple device directly or only with difficulty, now understands natural language better, which works on iPhone and iPad. Instead of having to name specific areas of the screen, users can, for example, say that the “yellow folder” should be opened. Buttons and controls can also be activated this way. AI-powered voice control is also helpful when app developers have set incorrect or only partially correct labels for control areas.
Videos by heise
Apple Intelligence makes texts more understandable
Further innovations concern the so-called accessibility reader. This offers special reading modes for existing documents, tailored, for example, to visual abilities or dyslexia. The system can process scientific articles or PDFs, handles images, tables, and multiple columns. Thanks to Apple Intelligence, analysis and presentation are said to have been improved, and integrated translation is also available. Finally, there are summaries and extensive configuration options regarding font, color, and layout.
(Image:Â Apple)
Initially, unfortunately, only intended for US English is a new AI captioning feature. This is available for all videos, including those you have recorded yourself, and can be turned on and off as desired. Apple did not state when other language versions will be available. Since the technology has been used multilingually for years, for example on YouTube, a prompt implementation is to be hoped for. Until then, the existing live captions can be used.
Vision Pro controls wheelchairs via eye tracking
Apple demonstrated an impressive new feature in connection with its headset Vision Pro: Together with TOLT Technologies and LUCI, there will be software that allows a wheelchair to be controlled via eye tracking. Users only need to look at specific virtual buttons in their field of view. The technology works both via Bluetooth and wired (although it remained unclear whether a Dev Strap is then necessary) and, thanks to an API, will also be made available to other providers of fully motorized wheelchairs in the future. Apple did not announce prices, but the Vision Pro alone costs at least 3700 euros.
(Image:Â Apple)
When asked, Apple did not want to disclose how much server-based AI is used for the new functions. As is usual with Apple Intelligence, usage is divided between local models and those in Apple's Private Cloud Compute environment (PCC). The question now is what happens when you are, for example, in a subway station with no mobile reception. What fallback mechanisms are planned here will only be revealed by testing. As usual, Apple did not announce when the new features will arrive and whether they will be part of the beta versions of iOS 27 and Co. Accessibility features are often also worth seeing and using for people without disabilities. They often also end up in a modified form in the “normal” operating system.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externer Preisvergleich (heise Preisvergleich) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (heise Preisvergleich) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
(bsc)
