Closed Bug 1907637 Opened 1 month ago Closed 1 month ago

Extend timeout on WebGPU CTS tests to x5.0

Tracking

()

Status:

RESOLVED FIXED

People

(Reporter: ErichDonGubler, Assigned: teoxoy)

References

Details

Attachments

(5 files, 1 obsolete file)

Bug 1907637 - feat(wpt): expose `--timeout-multiplier` in `mozharness` r=#webgpu-reviewers 1 month ago Erich Gubler [:ErichDonGubler] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1907637 - test(webgpu): set `--timeout-multiplier=5.0` for CTS r=#webgpu-reviewers! 1 month ago Erich Gubler [:ErichDonGubler] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1907637 - test(webgpu): `update-expected --preset reset-all` for `try:46d3500fb3c3346d4fc7b66e24cfb1d17a4231ce` r=#webgpu-reviewers! 1 month ago Erich Gubler [:ErichDonGubler] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1907637 - test(webgpu): promote long tests to tier 2 r=#webgpu-reviewers! 1 month ago Erich Gubler [:ErichDonGubler] 48 bytes, text/x-phabricator-request		Details \| Review
before-and-after.png 1 month ago Teodor Tanasoaia [:teoxoy] 189.50 KB, image/png		Details
Bug 1907637 - style(webgpu): fix up `WebGPUChild.cpp` r=#webgpu-reviewers! 1 month ago Erich Gubler [:ErichDonGubler] 48 bytes, text/x-phabricator-request		Details \| Review

Erich Gubler [:ErichDonGubler]

Reporter

Description

•

1 month ago

:teoxoy and I were suspicious that WebGPU CTS tests would Just Run To Completion™ if we extended the WPT timeout windows by a sufficient amount. The Chromium team publishes (very) rough timing data for their WebGPU test runs

It turns out, if we match our current WPT long timeouts' to the worst case observed in Chromium by setting a --timeout-multipler=5.0 in invocations to wptrunner, the vast majority of tests Just Run Right™. So, we're filing a patch stack to expose and use the timeout multiplier properly, adjust expected WPT test outcomes for WebGPU CTS, and then promote perma-PASSing tests on all platforms to tier 2 (as we've been doing with other CTS tests).

Erich Gubler [:ErichDonGubler]

Reporter

Updated

•

1 month ago

Assignee: nobody → ttanasoaia

Status: NEW → ASSIGNED

Erich Gubler [:ErichDonGubler]

Reporter

Comment 1

•

1 month ago

Attached file Bug 1907637 - feat(wpt): expose `--timeout-multiplier` in `mozharness` r=#webgpu-reviewers — Details

Erich Gubler [:ErichDonGubler]

Reporter

Comment 2

•

1 month ago

Attached file Bug 1907637 - test(webgpu): set `--timeout-multiplier=5.0` for CTS r=#webgpu-reviewers! — Details

Phabricator Automation

Updated

•

1 month ago

Attachment #9412548 - Attachment description: WIP: Bug 1907637 - feat(wpt): expose `--timeout-multiplier` in `mozharness` → Bug 1907637 - feat(wpt): expose `--timeout-multiplier` in `mozharness` r=#webgpu-reviewers

Erich Gubler [:ErichDonGubler]

Reporter

Comment 3

•

1 month ago

Attached file Bug 1907637 - test(webgpu): `update-expected --preset reset-all` for `try:46d3500fb3c3346d4fc7b66e24cfb1d17a4231ce` r=#webgpu-reviewers! — Details

Erich Gubler [:ErichDonGubler]

Reporter

Comment 4

•

1 month ago

Attached file Bug 1907637 - test(webgpu): promote long tests to tier 2 r=#webgpu-reviewers! — Details

Teodor Tanasoaia [:teoxoy]

Assignee

Comment 5

•

1 month ago

Attached image before-and-after.png — Details

Erich Gubler [:ErichDonGubler]

Reporter

Comment 6

•

1 month ago

:teoxoy's prototypical Try push that proved that we got very interesting results, with ~400 intermittents on Windows[^1]: try:46d3500fb3c3

First Mega Serious™ Try push that actually tries to do this in a landable way: try:4ec08f6bd1a0

[^1]: Other platforms have similar results, but Windows is our team's current focus. 🙂

Updated

•

1 month ago

Attachment #9412562 - Attachment description: Bug 1907637 - test(webgpu): `update-backlog promote-perma-passing` r=#webgpu-reviewers! → Bug 1907637 - test(webgpu): promote long tests to tier 2 r=#webgpu-reviewers!

Comment hidden (obsolete)

Erich Gubler [:ErichDonGubler]

Reporter

Comment 10

•

1 month ago

Attached file Bug 1907637 - style(webgpu): fix up `WebGPUChild.cpp` r=#webgpu-reviewers! (obsolete) — Details

Erich Gubler [:ErichDonGubler]

Reporter

Updated

•

1 month ago

Keywords: leave-open

Phabricator Automation

Updated

•

1 month ago

Attachment #9412811 - Attachment is obsolete: true

Pulsebot

Comment 11

•

1 month ago

Pushed by egubler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6be3328e38a0
feat(wpt): expose `--timeout-multiplier` in `mozharness` r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/ec4906410b70
test(webgpu): set `--timeout-multiplier=5.0` for CTS r=webgpu-reviewers,teoxoy
https://hg.mozilla.org/integration/autoland/rev/58dc9170f7e5
test(webgpu): `update-expected --preset reset-all` for `try:46d3500fb3c3346d4fc7b66e24cfb1d17a4231ce` r=webgpu-reviewers,teoxoy

pstanciu

Comment 12

•

1 month ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/6be3328e38a0
https://hg.mozilla.org/mozilla-central/rev/ec4906410b70
https://hg.mozilla.org/mozilla-central/rev/58dc9170f7e5

Comment hidden (obsolete)

Oof, looks like Try push 4 is still failing, with the abort from timeout moving to a new chunk. Using ErichDonGubler/fx-scripts#1 to analyze its first webgpu-backlog5 log with an invocation like this:

use wpt.nu

http get 'https://treeherder.mozilla.org/jobs?repo=try&author=egubler%40mozilla.com&selectedTaskRun=MJNEIlPjT3-wvAmSmZUN4A.0'
  | wpt aggregate-timings webgpu 
  | explore`

…it can be observed that the webgpu:shader,execution,memory_model,weak test is getting cut off while still executing 15 minutes later. 😬 That's a lot of test time inside a single WPT file! Still figuring out how we can ultimately get that landed, but I think we don't have to resolve that to finish this particular bug.

Let's try abandon promotions in that particular file, for now, and create a separate follow-up bug to make it work. It's only 6 out of the ~88 tests we currently have slated for promotion in the latest revision I've got pushed here.

Erich Gubler [:ErichDonGubler]

Reporter

Comment 14

•

1 month ago

Try push 5: try:57ff5cc81725

Erich Gubler [:ErichDonGubler]

Reporter

Updated

•

1 month ago

Summary: Extend timeout on WebGPU CTS tests → Extend timeout on WebGPU CTS tests to x5.0

Erich Gubler [:ErichDonGubler]

Reporter

Updated

•

1 month ago

Comment 16

•

1 month ago

Pushed by egubler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ceb8667d6fc2
test(webgpu): promote long tests to tier 2 r=webgpu-reviewers,teoxoy

Serban Stanca [:SerbanS]

Comment 17

•

1 month ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/ceb8667d6fc2

Erich Gubler [:ErichDonGubler]

Reporter

Updated

•

1 month ago

Status: ASSIGNED → RESOLVED

Closed: 1 month ago

Keywords: leave-open

Resolution: --- → FIXED

You need to log in before you can comment on or make changes to this bug.