Deciding what to do is the easy part

In the early days of Semaloop, our agents constantly got stuck on a deceptively simple problem: tapping small, specific things on a screen.

An agent might need to close a card by tapping the × in the corner, but instead tap a nearby +, or something else entirely. It understood what it needed to do. It just couldn't reliably translate that decision into the right tap.

That exposed a less glamorous problem beneath goal-based testing: once an agent decides what to do, how does that become an actual tap on an actual phone, on a screen it's seeing for the first time?

Seeing a button isn't the same as touching it

Ask a frontier vision model what's on a screen and it does a solid job, noting an edit icon, a paywall, and a pause button in the centre. Ask it for the coordinates of that pause button and it's a different story. Sometimes it's off by a few pixels; other times, it points at a completely different part of the screen while sounding just as sure of itself.

In testing, that gap erodes trust. A vaguely right chatbot answer gets shrugged off, but a vaguely right tap can hit the wrong button and cause the app to respond in an unexpected way. At that point, you've lost the confidence of whoever is relying on your test results.

We wrote about the fix to my opening problem in Finding the Nano Banana. In short, we borrow spatial precision from an image generation model and smuggle it back out through a colour channel, which took our internal pointing accuracy from 74% to 98%.

Why we didn't just read the accessibility tree

The obvious shortcut is to skip vision models entirely and read the accessibility tree instead. It's faster to build, and it works right up until an agent rewrites your app, an A/B test swaps out a layout, or a screen never exposes the information cleanly to begin with.

You could ask an agent to mark up the accessibility tree and generate all the tests for you. But then who ensures that the intent behind how you wanted the app to behave remains true? A test that depends on the app's internal description of itself risks verifying the implementation rather than the experience.

That is why the final action matters so much. If an agent is meant to test an app like a person, it has to interact with what a person can actually perceive and use.

A phone has more senses than a screen

Vision gets most of the attention because it's the most visible part of the problem—pun intended. But a real phone isn't just a rectangle of pixels. It also has a microphone, a network connection, and much more. A test that only looks at pixels is testing a fraction of what the app actually does.

Agents need to run on real devices, not just simulators. When a step calls for speaking into a microphone, the agent should actually speak into one, and what gets captured should be whatever the hardware produces—not a mocked endpoint standing in for it.

The same goes for network conditions. Real devices drop from Wi-Fi to cellular the way real users' phones do. An agent running on one can see whether the app recovers or simply hangs—something a simulator can't tell you if it was never on a real network to begin with.

The trade-off

None of this is free. Vision-based, audio-aware testing on real hardware is a harder engineering problem than reading a tree and calling a tap function. It's also slower to build and run.

But if you can stomach that performance hit, you can build a test suite that tests what you intended to build, not merely how you happened to implement it. And because it interacts with the app the way a user does, it is far more likely to withstand the test of time.