The chrome.perf waterfall continuously builds and runs our performance tests on real Android, Windows, Mac, and Linux hardware; see list of platforms. Results are reported to the Performance Dashboard for analysis. The Perfbot Health Sheriffing Rotation ensures that the benchmarks stay green. The Perf Sheriff Rotation ensures that any regressions detected by those benchmarks are addressed quickly. Together, these rotations maintain Chrome's Core Principles of speed:
“If you make a change that regresses measured performance, you will be required to fix it or revert”.
The perf waterfall is split into three stages - builders, testers, and processors.
For each commit to the Chromium repo, a Builder is invoked. The builder builds Google Chrome, acquires test assets, and bundles everything up to be passed along to Testers. We execute a builder for each platform we support. To ensure we can keep up with the rapid flow of commits, we have a set of builders per platform, each building a different commit.
For each platform, we have a single tester job running continuously. Each run of a tester covers the set of commits from the end commit of the previous invocation to the latest built commit. Each tester has one or more subdevices (shards). The tester delegates a subset of tests to each shard and aggregates high-level results (pass/fail, runtimes).
Processors analyze the raw data generated by each Tester and convert it into a form that can be utilized by the Chrome Performance Dashboard. For some jobs, this work is executed by the tester instead.
