misc(lantern-collect): drop sampling #15072
Conversation
| /** @typedef {Result & {metrics: LH.Artifacts.TimingSummary}} ResultWithMetrics */ | ||
| /** @typedef {{results: ResultsForUrl[], warnings: string[]}} Summary */ | ||
| /** @typedef {{results: ResultsForUrl[]}} Summary */ | ||
| /** @typedef {import('../run-on-all-assets.js').Golden} Golden */ |
There was a problem hiding this comment.
Reminder that the Summary and all types in collect/ can change without implication for the lantern baseline testing code - as long as we continue to target the Golden output format.
| * @param {string} message | ||
| */ | ||
| function warn(message) { | ||
| summary.warnings.push(message); |
There was a problem hiding this comment.
co-locating these with the url run result is way more useful.
| @@ -16,14 +17,13 @@ import fetch from 'node-fetch'; | |||
| import defaultTestUrls from './urls.js'; | |||
| import * as common from './common.js'; | |||
| import {LH_ROOT} from '../../../../root.js'; | |||
| import {makeGolden} from './golden.js'; | |||
There was a problem hiding this comment.
golden.js only has this function, could we move it in here?
Also, WDYT about renaming the function to makeUrlSummary or something since the "golden" part refers to it taking the median of the samples, and this no longer happens.
There was a problem hiding this comment.
Many things inside of scripts/lantern refer to this thing as golden, and I don't want to propagate that change everywhere for this PR.
When we next update the lantern test database, we don't want to do any sampling. Simpler to run ~10x the amount of URLs and just drop whatever runs happened to flake even once.
Much of the complexity of this collection script came from the sampling, so it's nice to be rid of that.
No longer need two archives.
Added a "retries" property - if something was retried, it won't show in the "golden" file. Can still be found in the "summary" file (also their traces are still in the zip) for purposes of future analysis (could be interesting to see how lantern does with URLs that LH tends to fail on?)