Android: How to stop instrumentation test after first failure

Question

We have a suite of Android instrumentation tests which we run from CL via ./gradlew app:connectedDebugAndroidTest If there is a failing test we basically have to wait for all of them to complete before the HTML report is generated. What we would like is for the testing to stop on the first failing test. Is there a gradle flag we can apply to make this happen? I haven't found anything in the official docs.

Answer 1

I don't know if you found a solution yet, but I was thinking about it and I came up with using a TestRunner or TestRule. I think the TestRunner approach is more appropriate though. I'm not the biggest fan of the companion objects / static fields used, but it works. I've tested it using a single emulator, but not with sharded tests across multiple emulators (but I think it won't propagate the any failure across all emulators).

Custom test runner

A custom TestRunner that is aware of test failures and ignores tests after the first test fails. I've created a cut down version of AndroidJUnit4 with the functionality here: https://gist.github.com/AniFichadia/0a0aa665dfc554e963d2ab3276247efb#file-skiponfirstfailuretestrunner-kt

AndroidJUnit4 essentially delegates to either androidx.test.internal.runner.junit4.AndroidJUnit4ClassRunner or org.robolectric.RobolectricTestRunner.

The above implementation has a subclass of AndroidJUnit4ClassRunner and prevents test methods from running (by treating them as ignored) if any test fails. If you need Robolectric, then the implementation for a custom RobolectricTestRunner is the same as the custom AndroidJUnit4ClassRunner in the gist, and you can use a similar check as what's in AndroidJUnit4.loadRunner(Class).

To use it, you just have to annotate your test classes with the runner: @RunWith(SkipOnFirstFailureTestRunner::class).

Just have to watch out if AndroidJUnit4 changes internally (eg. with Nitrogen).

Custom test rule

A test rule that will skip tests after the first test fails (by treating them as ignored): https://gist.github.com/AniFichadia/0a0aa665dfc554e963d2ab3276247efb#file-skiponfirstfailuretestrule-kt

You need to add the rule to your test classes and this can be applied to unit test classes too. It logs the first failed test, but I don't like this approach though.

Enabling the TestRunner / TestRule

Perhaps you could use a BuildConfig field to enable it. This can be set to true in a non-productionised variant if the tests are running on CICD.

For the TestRunner you need to set the enabled property in the class itself.

For the TestRule, you can set if it's enabled using the enabled field in the constructor wherever it's being used.

Let me know what you think.