The JSON Test Results Format

The JSON Test Results Format is a generic file format we use to record the results of each individual test in test run (whether the test is run on a bot, or run locally).

Introduction

We use these files on the bots in order to determine whether a test step had any failing tests (using a separate file means that we don't need to parse the output of the test run, and hence the test can be tailored for human readability as a result). We also upload the test results to dashboards like the Flakiness Dashboard.

The test format originated with the Blink web tests, but has since been adopted by GTest-based tests and Python unittest-based tests, so we've standardized on it for anything related to tracking test flakiness.

Example

Here's a very simple example for one Python test:

% python mojo/tools/run_mojo_python_tests.py --write-full-results-to results.json mojom_tests.parse.ast_unittest.ASTTest.testNodeBase
Running Python unit tests under mojo/public/tools/bindings/pylib ...
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
% cat results.json
{
  "tests": {
    "mojom_tests": {
      "parse": {
        "ast_unittest": {
          "ASTTest": {
            "testNodeBase": {
              "expected": "PASS",
              "actual": "PASS",
              "artifacts": {
                "screenshot": ["screenshots/page.png"],
              }
            }
          }
        }
      }
    }
  },
  "interrupted": false,
  "path_delimiter": ".",
  "version": 3,
  "seconds_since_epoch": 1406662283.764424,
  "num_failures_by_type": {
    "FAIL": 0,
    "PASS": 1
  },
  "artifact_types": {
    "screenshot": "image/png"
  }
}

As you can see, the format consists of a one top level dictionary containing a set of metadata fields describing the test run, plus a single tests key that contains the results of every test run, structured in a hierarchical trie format to reduce duplication of test suite names (as you can see from the deeply hierarchical Python test name).

The file is strictly JSON-compliant. As a part of this, the fields in each object may appear in any order.

Top-level field names

Per-test fields

Each leaf of the tests trie contains a dict containing the results of a particular test name. If a test is run multiple times, the dict contains the results for each invocation in the actual field. Unless otherwise noted, if the test is run multiple times, all of the other fields represent the overall / final / last value. For example, if a test unexpectedly fails and then is retried and passes, both is_regression and is_unexpected will be false).

Test result types

Any test may fail in one of several different ways. There are a few generic types of failures, and the web tests contain a few additional specialized failure types.

Unexpected results, failures, and regressions are different things.

An unexpected result is simply a result that didn't appear in the expected field. It may be used for tests that pass unexpectedly, i.e. tests that were expected to fail but passed. Such results should not be considered failures.

Anything other than PASS, SKIP, SLOW, or one of the REBASELINE types is considered a failure.

A regression is a result that is both unexpected and a failure.

full_results.json and failing_results.json

The web tests produce two different variants of the above file. The full_results.json file matches the above definition and contains every test executed in the run. The failing_results.json file contains just the tests that produced unexpected results, so it is a subset of the full_results.json data. The failing_results.json file is also in the JSONP format, so it can be read via as a <script> tag from an html file run from the local filesystem without falling prey to the same-origin restrictions for local files. The failing_results.json file is converted into JSONP by containing the JSON data preceded by the string “ADD_RESULTS(” and followed by the string “);”, so you can extract the JSON data by stripping off that prefix and suffix.