Why we don't trust the accessibility tree

When an agent is faced with the task of figuring out what's on a screen, there are two approaches:

1. Read the accessibility tree on mobile or the DOM on web, and get a structured list of elements with labels, roles and bounding boxes.
2. Look at the screen the way a person does and work it out from the pixels.

Almost everyone building in this space picks the first option. It's fast, cheap, precise, and the data is already there. We picked the second, with no fallback to the tree at all, and the question we get asked most is why we'd deliberately make our lives harder. Here's why.

The tree often isn't there

In 2021, Apple published Screen Recognition, a system that generates accessibility metadata from pixels precisely because developers so often don't provide it. They collected 77,637 screens from 4,068 iPhone apps, had humans annotate every visible element, and compared that with what each app exposed through the accessibility APIs.

On 59% of screens, at least one visible element had no match in the tree. 94% of apps had at least one such screen. On 4% of screens, the tree exposed nothing at all.

Two things make these numbers conservative. The dataset excluded games entirely, and Apple used deliberately lenient matching rules, noting themselves that they probably undercounted the missing elements.

Where it exists, it's the app's own account of itself

The tree isn't a description of what's on screen. It's what the app says is on screen, generated by the same code you're trying to test, and it's wrong in exactly the situations you most want to catch. An element can be in the tree and invisible: off-screen, under a modal, at zero opacity, behind a loading state that never resolves. A view can report itself as tappable and do nothing when tapped.

Large parts of the screen aren't in it by design

Many parts of a UI are opaque within the tree: game engines, canvas-drawn UI, maps, videos, ads, cameras. Each shows up as a single rectangle with nothing inside it, so a map is one element rather than a map with pins, and a game is one element rather than a menu with buttons. Apple's own analysis skipped ads and maps altogether because there was nothing in the tree to compare against.

Pixels are the only universal substrate

Every platform has a screen, so an agent that works from pixels works on iOS, Android, web and system UI with no per-platform adapter to build or maintain.

It also survives an underlying code change. A framework upgrade can rewrite the entire accessibility hierarchy while the app looks pixel-identical to a user. Selector-based automation falls over, but a pixel-based agent doesn't notice, because nothing has changed in the pixel layer.

It finds what the tree can't represent

Text truncated at large dynamic type. Overlapping layout on a smaller device. Dark mode that's technically valid and unreadable. An image that didn't load. A spinner that never stops. None of these exist in the tree, but all of them are things your customers complain about.

The honest trade-offs

Pixels are slower than reading a tree and cost more per step, and for the narrow set of things the tree does well, like exact text and element roles, they're less precise out of the box. We fine-tuned our way past most of that, which we cover below, but it was work the tree would have given us for free.

We also share a limit the Apple paper names: anything that's invisible on the screen is invisible to us too. Their example is a book app where turning the page meant swiping in from the screen edge, with nothing on screen to suggest it. If a user couldn't work it out by looking, neither can we.

We still concluded pixels should be the only source of truth rather than one of two. A second signal that's missing from most screens means maintaining two systems and leaning on the weaker one at exactly the moments it's least trustworthy. A fallback you can't rely on isn't a fallback.

The hard part is pointing, not seeing

Frontier models describe screens well. The hard part is grounding: turning "tap the add money button" into a coordinate that actually lands on the button. We wrote about taking our pointing accuracy from 74% to 98% in Finding the Nano Banana.

Apple hit the same wall and solved it much the same way, with a dedicated on-device detector plus OCR, icon recognition and grouping heuristics layered on top. Their detector alone reached 71.3% mean average precision before any of those layers were added, which gives you a sense of how much engineering sits between "the model can describe this screen" and "the agent can reliably use it".

Why this matters beyond testing

Read the tree and you get a log of what the app reported. Work from the screen and you get a record of what actually happened to a person on a real device. This matters the moment someone asks what your app really does to your customers, whether that's a regulator asking how a consent flow behaved, a support team disputing whether a payment screen ever loaded, or an App Store reviewer rejecting a build. In each case you need to show them rather than tell them.