You can now ask Semaloop about your app

You can now ask Semaloop about your app. Type a question in your own words, like when a test started failing or whether it's actually flaky, and Semaloop reads every run on every build to answer it. The longer Semaloop has been testing your app, the more it can tell you.

The answers were already there

Every test run leaves evidence behind: which build it ran on, which device and OS version, whether it passed and, when it failed, why. Over a few months of builds, that adds up to a detailed record of how your app has behaved. The trouble is that it lives in run history, and getting a straight answer out of run history meant opening run after run and piecing it together yourself.

Is this test flaky, or did something break? When did it start failing? Does it only fail on one device? These are simple questions with tedious answers, so most of the time nobody asks them.

How Ask works

Open Ask from the sidebar and type a question. Semaloop looks across the app's builds, its tests and their current specs, every run including reruns, the device and OS version each one ran on, and the root cause analysis for each failure. You can watch the steps it takes while it works, and the answer comes back short, with links to the runs, builds and tests it mentions. Ask only links to things it actually looked up, so you can click through and check the evidence rather than take the answer on trust.

A few of the questions you can ask:

• When did this test start failing?
• Is this test actually flaky?
• What failed on the latest build?
• What changed between the last two builds?
• When did this test last pass?

You can ask follow-ups in the same conversation, and if a question could mean more than one thing, Ask checks with you before answering. When the results don't settle a question either way, it says so rather than guessing. The Ask docs cover exactly what it can and can't see.

It gets better the longer it runs

Ask is only as good as the history behind it. On a new app with a handful of builds, it can tell you what's failing right now. After months of builds, it can tell you that a test has failed on every run since a particular build and passed on every build before it, which is the difference between "this is flaky" and "something broke in that release".

What it makes possible

What I'm most excited about, though, is what Ask makes possible. It gives Semaloop a real understanding of your app's whole testing history, and a lot of what we're shipping next is built on top of that. The first of those is close behind.

Ask was built by Sam Starling. Brilliant job, Sam!

Try it

Ask is live for every Semaloop customer today. If you already use Semaloop, open Ask from the sidebar on any of your apps and start with whatever you'd normally go digging through runs to find out. If you don't yet, book a demo and we'll show you Semaloop running against your own app.