Apple Study: Users Want Transparent AI Agents Instead of Black-Box Systems

Apple researchers investigated in a two-phase study how users want to interact with AI agents. The result is surprising: people prefer less powerful, but rather transparent agents over powerful black-box systems. The study "Mapping the Design Space of User Experience for Computer Use Agents", published in February 2026, identifies four central categories for UX design and analyzes nine existing systems such as Claude Computer Use Tool, OpenAI Operator, and Google's Project Mariner.

In Phase 1 of their study, the researchers examined nine commercial AI agent systems and conducted interviews with eight UX and AI practitioners from major technology companies. In Phase 2, they tested their findings with 20 participants in a so-called Wizard-of-Oz experiment. This describes an experiment in which a human (participant) believes they are communicating with an autonomous (in the sense of artificial intelligence) system, but in reality is interacting with a human. The participants were asked to complete tasks such as booking vacation rentals or online shopping, while a researcher in the next room simulated the agent's actions. The participants could stop the supposed agent at any time with an interrupt button. The recorded videos and chat logs provided insights into actual user expectations.

Transparency More Important Than Automation

A key finding: users want insight into agent activities, but not micromanagement. Too much control would mean they could do the tasks themselves. Transparency is particularly important to participants when dealing with unfamiliar user interfaces. There, they desire more intermediate steps, explanations, and confirmation pauses – even in low-risk scenarios. For actions with real consequences, such as purchases, account changes, or contacting other people, users demand more control.

Trust in AI agents quickly erodes when the system makes silent assumptions or errors. In ambiguous choices, users prefer the agent to pause and ask for clarification rather than choose randomly. This is particularly evident in decisions that could lead to incorrect product selections.

Existing Systems Only Partially Meet Expectations

According to the researchers, the nine analyzed systems, including Anthropic's Claude Computer Use Tool, OpenAI Operator, and Google's Project Mariner, only partially meet user expectations. The study also shows context-dependent expectations: users want different agent behavior depending on whether they are exploring options or performing a familiar task. Expectations also change based on familiarity with an interface. The collected findings could have a direct impact on Apple's planned Siri overhaul. The iPhone manufacturer announced in the summer of 2024 that the voice assistant would be able to perform cross-app tasks in the future. However, the release was delayed. It is currently expected to be released in the coming months.

Apple is taking a significantly more conservative approach to AI agents than competitors like OpenAI, Google, and Meta. While these companies are investing billions in large, general-purpose language models, Apple is focusing on targeted, privacy-oriented features with an emphasis on on-device processing.

For computationally intensive tasks, Apple is evaluating external models, particularly Google's Gemini, but plans to operate an adapted version on its own servers. Personal data and device context remain with Apple's own in-house models. The current study plays into Apple's hands: users are more willing to accept systems that launch later but are better designed than quickly introduced black-box solutions.

(mki)