Semaloop now summarises every build

Semaloop now summarises every build. When your tests finish, you get a short account of what's changed since the last build: what's started failing and why, what's been fixed, and what's still broken and since when.

Where do I start?

A build lands, the tests run, and a handful come back red. The first few minutes after that usually go on working out which of them matter: opening each failed run, checking whether it failed last time too, and separating the new problem from the ones you already knew about. None of it is hard, but it's slow, and it happens on every build.

What a summary tells you

Once every run on a build has finished, Semaloop writes a few sentences comparing it with the build before. It leads with what changed: tests that passed last time and now fail, each with a short reason, and tests that were failing and now pass. Tests that were already failing get a brief mention of how long they've been failing, and the summary ends with how many of the rest still pass. If nothing changed, it says so in one sentence.

A run that couldn't finish because of a problem on our side, rather than in your app, is described as not completed rather than as a failure, so it doesn't send you chasing a bug that isn't there.

Instead of opening every failed run to work out where to start, you read one paragraph.

Where it shows up

The summary sits at the top of the build page, and in your Slack notification if you've connected a channel. The Slack message waits for the summary, so the first notification you get already includes it. If the results change later, because you rerun a test or edit a spec, the summary is rewritten and the message updates in place.

Every claim is linked

A summary is only useful if you can trust it, so every claim links to the run behind it. If the summary says a test started failing because an item went missing from a list, one click takes you to that run, with its video, system logs and network trace, and you can check for yourself. When you want to dig further, Ask a follow-up takes you straight into Ask, where you can question the build's results in natural language.

Built on Ask

Build summaries is one of the first things we've built on top of Ask, and it came together quickly because Ask already understands your app's whole testing history. A good summary is mostly a matter of knowing what changed, why, and since when, which is exactly the kind of question Ask was built to answer. There are plenty more features like it coming over the next few weeks.

Another great feature from Sam Starling.

Try it

If you already use Semaloop, your next build will come with a summary, and the build summaries docs cover what goes into one. If you don't yet, book a demo and we'll show you Semaloop running against your own app.