App Intents API: How Siri will read the screen content in future
From iOS 18.2, Apple is preparing for the new "Onscreen Awareness" of the AI assistance system. In future, Siri will know what users see in order to help them.
(Image: Apple)
Apple has provided details on app usage with a new Siri function that will enable the voice assistant to read iPhone screen content. The so-called Onscreen Awareness is expected with an upcoming iOS 18 update as part of Apple Intelligence (probably by spring 2025) and complements, among other things, the new context function, which should make Siri much smarter because content contained on the device can be included.
Testing from iOS 18.2 – for developers
The feature requires a certain approach on the part of developers, as Apple states in a new developer document. It deals with how apps need to be prepared in order to prepare content that can be seen on the screen for Siri and Apple Intelligence. API features are already implemented in iOS 18.2 so that they can be tested.
Apple apparently wants to make Onscreen Awareness as privacy-friendly as possible. Only when a user asks a question about a screen content or wants to perform an action based on it can Siri and Apple Intelligence view the content and perform the action. Whether this only happens locally on the device (which Apple is actually aiming for) or whether Apple's Private Cloud Compute is also used remains unclear at present. The information is only forwarded to third-party services at the express request of the user. Apple cites the example of displaying a website and then having Siri summarize it. This is currently already possible with Apple Intelligence, but it requires a click.
Videos by heise
Forwarding to ChatGPT on request
When using ChatGPT, it should also be possible to send photos or documents that can be seen on the screen to OpenAI. This applies, for example, to the generation of image descriptions or the analysis of PDFs. Here too, a request is always made before data is sent. Siri offers ChatGPT, for example, when users request actions that the voice assistant is currently unable to perform. However, Apple will also soon be providing image descriptions itself as part of the "Visual Intelligence" function.
In its developer document, Apple also states that "current and future Personal Intelligence features" require explicit approval of the screen content via the App Intents framework. It is therefore conceivable that older applications will not support on-screen awareness, as only relatively few iPhone apps use App Intents to date. This could therefore quickly become frustrating.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externer Preisvergleich (heise Preisvergleich) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (heise Preisvergleich) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
(bsc)
