Flaky tests should embarrass us

Somewhere along the way, the industry decided that tests failing at random were normal. We gave it a friendly name, flakiness, and built a whole culture around managing it: retry the test three times, quarantine the flaky ones, add a wait, merge anyway.

Each of those rituals makes sense on its own. A retry gets the build green, a quarantine stops one bad test blocking everyone, and a wait papers over a race nobody has time to find. Put together, though, they amount to an agreement to stop believing what the suite tells you.

What "flaky" actually means

Think about what "flaky" actually means. It means your test suite tells you something is broken when it isn't, which is the one thing a test exists not to do. A smoke alarm that goes off when there's no fire is broken, and you'd replace it, yet we've trained an entire profession to shrug at exactly that behaviour.

The real cost

The cost of a flaky test isn't the few minutes spent re-running it. It's what happens to trust. Once a team learns that red sometimes means nothing, every red starts to mean "probably nothing". Real failures get waved through as probably flaky, retried until they pass, and found later by customers. The suite is still running, and still costing time and money, but it has stopped doing its job, which is to give you a signal you'd act on.

There's a quieter cost too. Every hour spent re-running, quarantining and tuning waits is an hour not spent fixing anything, and a quarantined test has stopped protecting whatever it was written to protect.

Where it comes from on mobile

Mobile makes all of this worse. A test has to drive a real device, with its own animations, network conditions and system prompts, and most mobile automation is built on ideas borrowed from web testing: find an element by its identifier, tap it, wait for the next one to appear. That couples the test to how the screen is built rather than to what the user is trying to do.

So a test fails when a label is renamed, when an element moves in the view hierarchy, when a screen takes half a second longer to load than the hard-coded wait allowed, or when a permission prompt appears that the script didn't expect. None of those are bugs a user would notice. The app works, and the test says it doesn't.

Testing on intent

Tests that run on intent don't break when a label moves or a screen loads half a second late, because they're checking whether the user's goal still works, not whether the screen's structure matches a snapshot from three sprints ago. Take away the brittleness and the flakiness goes with it.

That's the approach we've taken with Semaloop. You describe what should happen in natural language, and our agents work from what's on the screen, the way a person does, rather than from element identifiers or the accessibility tree. We've written about why we don't trust the accessibility tree. A renamed button or a redesigned screen doesn't change whether a user can still finish what they came to do, so it doesn't change the result. How you write the test matters too, and How to write a great test covers describing the goal rather than every tap.

None of this means every red result is a real bug. Devices misbehave, and no test is perfect. But when you're not sure, you shouldn't have to guess: Ask can tell you whether a test is actually flaky by reading every run of it on every build, so the answer comes from the history rather than a hunch.

And I think we can all agree that the only flakes we should have to deal with are the ones sticking out of an ice cream.