ResultDB is a LUCI service for storing and retrieving test results. See also its public source and Google internal documentation. As of Q4 2021, all tests on Chrome/Chromium builders have their test results uploaded to ResultDB, and nearly all pass-fail decisions on the builders are based on these results. Consequently, any JSON output from a test no longer has little to no impact on the results of the bots. (For example, if a test‘s JSON say that there was a failure but this isn’t similarly reflected in ResultDB, then the build will not fail.
All test harnesses are responsible for uploading their results to ResultDB. This is most easily done via ResultSink (see internal documentation). ResultSink is a local HTTP server that proxies requests to ResultDB's backend. It can be launched via the rdb stream ... command and listens on a port specified in a file pointed to by the LUCI_CONTEXT environment variable. This server currently runs in the background on all test bots.
On a machine with the server running, a test can report its results to ResultDB by making JSON-formatted RPCs to the local HTTP ResultSink server. See ResultSink's API for more details.
There are several libraries used within Chromium that have ResultSink integration:
Web Tests: Blink's web tests upload their results through ResultSink via test_result_sink.py.
typ: Typ is a testing library used by performance benchmarks and GPU tests. It integrates with ResultSink in result_sink.py.
//build/util/lib/results/: //build/util/lib/results/ contains a generic wrapper around ResultSink. This is used by both the Android and ChromeOS test runners to upload their results to ResultDB.
iOS test runner: The iOS test runner has its own ResultSink integration in result_sink_util.py.
result_adapter: Most remaining tests use the result_adapter tool. See below for more details.
result_adapter is a command-line tool that wraps a test invocation and will automatically convert the test's output JSON to the ResultSink format and uploads those results to ResultDB via ResultSink. Known JSON formats include:
gtest for the JSON generated by running GTests using the support code in //base/test/launcher/, specifically the SaveSummaryAsJSON function in test_results_tracker.h.
json for the JSON format described in json_test_results_format.md. Tests that can generate this type of JSON include Blink web tests and typ-supported tests.
Though it eased the migration of results into ResultDB, using result_adapter has a few drawbacks:
result_adapter uploads all results at once after the test invocation exits. If the test execution crashes partway through, ResultDB will not track any of the results that were successfully produced.
result_adapter is limited by the JSON format of the test. It would be unable to use any new or advanced feature in ResultDB.
Consequently, it‘s preferred when possible for new test types and new test harnesses to integrate with ResultSink directly rather than using result_adapter. But if circumstances make integration difficult (e.g. we don’t have access to the test harness implementation) result_adapter can be needed.
The *.pyl spec files in //testing/buildbot/ control what tests a given bot runs. These testing specs also control how result_adapter is used. By default, a isolated_scripts test will have result_adapter added using the json output format, and a gtest_tests test will have result_adapter added using the gtest output format. This can be overwritten using the has_native_resultdb_integration mixin which will disable result_adapter for that test. Additionally, a custom result_format can be specified for a test to overwrite the expected format of the JSON: example.