Loops need verifiers

Loops loops loops. That’s all we’ve been hearing this summer from AI labs and startups. The idea is that we need to stop thinking of agents as simple task rabbits, and instead start using them in a goal-oriented way.

But, I think we’ve jumped the gun. An agent is no good in a loop if it doesn’t have an accurate verifier.

The OG verifier

Cast your mind back to when you wrote code (remember that?), and try to recall your iteration loop. In my case, 14 year old me was sitting at home making Android apps, following this process:

1. Put together a plan
2. Research on StackOverflow and compare against my existing codebase
3. Make the changes
4. Wait about 20 minutes for Eclipse to build my app (put some coffee on, read a book)
5. Open the app on my phone and poke/prod my way around the feature until I found a problem/bug/crash
6. Go back to step 1 and repeat

That was my loop: I was both coder and verifier, writing the code, and controlling my phone to see whether I’d made things worse or not. I didn’t realise it at the time, but that verification process embodied knowledge of Android (Cupcake, if you’re curious), multi-gestures, touch-screens and adversarial testing methodologies.

A false start

About a year ago, coding models reached a level that meant they could complete steps 2-4 without any hand-holding. Give them a task, and they could make changes that probably helped. They just didn’t have the means to verify that the changes had actually made progress in the right direction, so that job was still left to the engineer.

Being a lazy lot, we engineers were unhappy being the bottleneck. We wanted a way to let coding agents understand the changes they’d made, without needing to call on us for help.

Well it turns out that on web, this was relatively straightforward. Every webpage has a DOM, which is really just a textual representation of what we’re seeing on the page. LLMs love text, so feeding these DOMs into agents provided them with an accurate enough way to understand what was on-screen. Partnered with Playwright to help them interact with the page, the agents were able to close the loop and mark their own homework.

The same couldn’t be said for mobile. Although iOS and Android did expose text representations similar to the web, they were often patchy, broken, or downright wrong. Those representations weren’t accurate reflections of what was actually on-screen, so agents went round in circles using faulty verifiers.

That’s exactly why Charlie and I set out to build Semaloop. We wanted a way to verify software the way a person actually does: by looking at and interacting with the screen.

Closing the loop

It turns out that it was a great problem to have spent the last year on, right as loops became the thing that everyone’s chasing.

The Semaloop agents that we’ve built interact the same way humans do: through the screen and over voice. Coding agents now have a comprehensive verifier for iOS and Android which can not only comprehend and interact like a human, but go beyond what a human could ever do, by analysing network requests, logs and performance telemetry to spot unexpected bugs and issues. I know for a fact that 14 year old Rory would have been in complete awe.

If you’re trying to run agent loops against a real mobile app and you’re missing a verifier, come see what a real device catches that an XML tree can’t: semaloop.com