Why Claude isn't enough to test your app

The question people actually ask is whether a coding agent that can drive a simulator means they need nothing else, and that depends on which of two jobs you thought you were getting.

A coding agent verifies the change it just made, once, now, on your machine, with you watching. A suite verifies everything that change did not touch, on every build, when nobody remembers the flow exists.

Nothing remembers what the app did last week

A fresh agent can tell you the app does not match your description of it, which is worth knowing but is not the same as telling you the app changed. It has no memory of the last build, so intent mismatch is the only kind of failure available to it. Regression needs something that outlives the session, holds the expected behaviour of every flow you have ever cared about, and can say which of them moved this morning.

The thing that wrote the code should not be the thing that grades it

If the agent that wrote the code also writes the test and then decides whether it passed, your suite inherits its blind spots exactly. A QA lead with thirty years behind him put this back to us as the reason you do not let engineers sign off their own work, and a green suite that is wrong is worse than no suite at all. Running a second agent helps a little, but it still inherits your test IDs and your definition of working, and it still remembers nothing about last week.

The flows most likely to break are the ones a simulator cannot run

One-time passcodes arriving over SMS, push notifications, the camera, a Safari redirect handing back to the app, and the way a phone behaves once it is warm. A simulator either handles these differently or does not handle them at all, and a good share of the bugs that reach your users live in exactly that gap. Anthropic's own documentation suggests not signing in to real accounts on a device Claude uses, which is the tidiest evidence available that the simulator is not where your logged-in flows get tested.

So, what next?

Some teams do not need any of this yet. One engineer, a handful of flows, no passcodes or camera or payments anywhere near them, shipping weekly with the app open in front of you: the agent's own verification pass is a reasonable place to stop. For everyone else, the three things above are not the edges of the problem: generation got cheap, but verification did not.