Interoperability between browsers is critical to Chromium's mission of improving the web. We believe that leveraging and contributing to a shared test suite is one of the most important tools in achieving interoperability between browsers. The web-platform-tests repository is the primary shared test suite where all browser engines are collaborating.
Chromium has a 2-way import/export process with the upstream web-platform-tests repository, where tests are imported into LayoutTests/external/wpt and any changes to the imported tests are also exported to web-platform-tests.
See http://web-platform-tests.org/ for general documentation on web-platform-tests, including tips for writing and reviewing tests.
Chromium has a mirror (web-platform-tests of the GitHub repos, and periodically imports a subset of the tests so that they are run as part of the regular Blink layout test testing process.
The goals of this process are to be able to run web-platform-tests unmodified locally just as easily as we can run the Blink tests, and ensure that we are tracking tip-of-tree in the web-platform-tests repository as closely as possible, and running as many of the tests as possible.
There is an automatic process for updating the Chromium copy of web-platform-tests. The import is done by the builder wpt-importer builder.
The easiest way to check the status of recent imports is to look at:
Automatic imports are intended to run at least once every 24 hours.
If a commit to Chromium master changes any files in the third_party/WebKit/LayoutTests/external/wpt directory, the WPT Exporter will create a Pull Request on GitHub for it. All PRs use the chromium-export label: see all of them here. The exporter runs continuously under the chromium.infra.cron master: see all recent builds. The source for the exporter lives in third_party/WebKit/Tools/Scripts/wpt-exporter.
In the unlikely event that the exporter starts misbehaving -- for example, creating the same PR over and over again -- all you need to do to disable the exporter is land this CL, which will put it in “dry run” mode.
We control which tests are imported via a file called W3CImportExpectations, which has a list of directories to skip while importing.
In addition to the directories and tests explicitly skipped there, tests may also be skipped for a couple other reasons, e.g. if the file path is too long for Windows. To check what files are skipped in import, check the recent logs for wpt-importer builder.
To pull the latest versions of the tests that are currently being imported, you can also directly invoke the wpt-import script.
That script will pull the latest version of the tests from our mirrors of the upstream repositories. If any new versions of tests are found, they will be committed locally to your local repository. You may then upload the changes.
If you wish to add more tests (by un-skipping some of the directories currently skipped in W3CImportExpectations), you can modify that file locally and commit it, and on the next auto-import, the new tests should be imported.
If you want to import immediately (in order to try the tests out locally, etc) you can also run wpt-import --allow-local-commits, but this is not required.
To contribute changes to web-platform-tests, just commit your changes directly to LayoutTests/external/wpt and the changes will be automatically upstreamed within 24 hours.
Changes involving adding, removing or modifying tests can all be upstreamed. Any changes outside of external/wpt will not be upstreamed, and any changes *-expected.txt, OWNERS, and MANIFEST.json, will also not be upstreamed.
Running the layout tests will automatically regenerate MANIFEST.json to pick up any local modifications.
Most tests are written using testharness.js, see Writing Layout Tests and Layout Tests Tips for general guidelines.
Tests in web-platform-tests are expected to match behavior defined by the relevant specification. In other words, all assertions that a test makes should be derived from a specification‘s normative requirements, and not go beyond them. It is often necessary to change the specification to clarify what is and isn’t required.
When the standards discussion is still ongoing or blocked on some implementation successfully shipping the hoped-for behavior, write the tests outside of web-platform-tests and upstream them when the specification is finally updated. Optionally, it may be possible to write deliberately failing tests against the current specification and later update them.
Tests that depend on internals.*, eventSender.* or other internal testing APIs cannot yet be written as part of web-platform-tests.
An alternative is to write manual tests that are automated with scripts from wpt_automation. Such tests still require case-by-case automation to run for other browser engines, but are more valuable than purely manual tests.
Entirely new top-level directories should generally be added upstream, since that's the only way to add an OWNERS file upstream. After adding a new top-level directory upstream, you should add a line for it in W3CImportExpectations.
Adding the new directory (and W3CImportExpectations entry) in Chromium and later adding an OWNERS file upstream also works.
The email you commit with in Chromium will be the author of the commit on GitHub. You can add it as a secondary address on your GitHub account to link your exported commits to your GitHub profile.
This cannot be avoided entirely as the two repositories are independent, but should be rare with frequent imports and exports. When it does happen, manual intervention will be needed and in non-trivial cases you may be asked to help resolve the conflict.
It's still possible to make direct pull requests to web-platform-tests, see http://web-platform-tests.org/appendix/github-intro.html.
Anyone who can review code and tests in Chromium can also review changes in external/wpt that will be automatically upstreamed. There will be no additional review in web-platform-tests as part of the export process.
If upstream reviewers have feedback on the changes, discuss on the pull request created during export, and if necessary work on a new pull request to iterate until everyone is satisfied.
When reviewing tests, check that they match the relevant specification, which may not fully match the implementation. See also Write tests against specifications.