I had originally thought to use Push Health for this, but I find that this is different enough that using Push Health would require a lot of changes.

The use case here is when there are large changes made to infrastructure, we need to verify parity. This could be different machines, cloud providers, upgrades to OS, etc.

I view the workflow like this:

1. push to try with recent push for all jobs for given affected platform, --rebuild 5
2. apply change to use new version, push to try again like in #1
3. when all jobs are done, compare push 1 vs 2
4. compare average runtime of all passing jobs, report anything >3% difference
5. compare total failures, failures by suite, failures by job
6. compare failure types (infra, timeout, crashes, test fail, etc.)

In a perfect world everything is the same, some small variance <5% is usually ok.

This could either be part of push health, perfherder compare, or something new that is modeled after one of those.

If this existed, then release operations would be able to make changes without CI-A helping out all the time, likewise taskcluster could upgrade workers, wpt-sync could benefit as well.

Initial work has been carried out in here
https://github.com/mozilla/treeherder/commit/b24a8d0219d77ee51b1c5b23e8defea1eff6ea75
More features needed are

1. A column to show the percentage change in both the revisions
2. A row to show the total time of all suites and total failures of all suites

thanks for mentioning these. I would like to adjust the filter of time to be 5% difference. For adding column, it might get cluttered, play with it a bit.

the row at the top is great, that would be a good start.

Additional features added here: https://github.com/mozilla/treeherder/commit/bd9af49dd86b2db34eaf70cb7525b2f10aeafe2a

1. Add a custom range from 0-20% to show important filtered jobs
2. The chunk number was removed from the suite but it was displayed multiple times.

The first commit added containing the basic features were
https://github.com/mozilla/treeherder/pull/6328