Outside of Chromium, it is most common to run all tests of a test suite using a single adb shell am instrument command (a single execution / OS process). However, Chromium's test runner runs each test using a separate command, which means tests cannot interfere with one another, but also that tests take much longer to run. Test batching is a way to make our tests run faster by not restarting the process between every test.
All on-device tests would ideally be annotated with one of:
@Batch(Batch.UNIT_TESTS): For tests the do not rely on global application state.@Batch(Batch.PER_CLASS): For test classes where the process does not need to be restarted between @Tests within the class, but should be restarted before and after the suite runs.@DoNotBatch(reason = "...": For tests classes that require the process to be restarted for each test or are infeasible to batch.Tests that are not annotated are treated as @DoNotBatch and are assumed to have not yet been assessed.
Add the @Batch annotation to the test class, and ensure that each test within the chosen batch doesn't leave behind state that could cause other tests in the batch to fail.
For some tests, batching won’t be as useful (tests that test Activity startup, for example), and tests that test process startup shouldn’t be batched at all.
If a few tests within a larger batched suite cannot be batched (eg. it tests process initialization), you may add the @RequiresRestart annotation to test methods to exclude them from the batch.
Tests that belong in this category are tests that are effectively unit tests. They may be written as instrumentation tests rather than junit tests for a variety of reasons such as needing to use real Android APIs, or needing to use the native library.
Batching Unit Test style tests is usually fairly simple (example). It requires adding the @Batch(Batch.UNIT_TESTS) annotation, and ensuring no global state, like test overrides, persists across tests. Unit Tests should also not start the browser process, but may load the native library. Note that even with Batched tests, the test fixture (the class) is recreated for each test.
Note that since the browser isn't initialized for unit tests, if you would like to take advantage of feature annotations in your test you will have to use Features.JUnitProcessor instead of Features.InstrumentationProcessor.
This batching type is typically for larger and more complex test suites, and will run the suite in its own batch. This will limit side-effects and reduce the complexity of managing state from these tests as you only have to think about tests within the suite.
Tests with different @Features annotations (@EnableFeatures and @DisableFeatures) or @CommandLineFlags will be run in separate batches.
This batching type is best for smaller and less complex test suites, that require browser initialization, or something else that prevents them from being unit tests. Custom batches allow you to pay the process startup cost once per batch instead of once per test suite. To put multiple test suites into the same batch, you will have to use a shared custom batch name (example). When batching across suites you’ll want to use something like BlankCTATabInitialStateRule to persist static state (like the Activity) between test suites and perform any necessary state cleanup between tests.
Note that there is an inherent tradeoff here between batch size and debuggability - the larger your batch, the harder it will be to diagnose one test causing a different test to fail/flake. I would recommend grouping tests semantically to make it easier to understand relationships between the tests and which shared state is relevant.
Run all tests with @Batch=UnitTests:
out/<dir>/bin/run_chrome_public_unit_test_apk -A Batch=UnitTests out/<dir>/bin/run_chrome_public_test_apk -A Batch=UnitTests
Run all tests in a custom batch:
./tools/autotest.py -C out/Debug BluetoothChooserDialogTest \ --gtest_filter="*" -A Batch=device_dialog
./tools/autotest.py -C out/Debug ExternalNavigationHandlerTest \ --gtest_filter="*#testOrdinaryIncognitoUri:*#testChromeReferrer"