Running web tests using the content shell

Compiling

If you want to run web tests, build the target blink_tests; this includes all the other binaries required to run the tests.

Running

Using run_web_tests.py

You can run web tests using run_web_tests.py (in src/third_party/blink/tools).

third_party/blink/tools/run_web_tests.py -t <build directory> storage/indexeddb

To see a complete list of arguments supported, run with --help.

Run Web Tests Directly with Content Shell

In some cases (e.g. for debugging), you can run web tests directly with Content Shell executable, with the --run-web-tests flag:

out/Default/content_shell --run-web-tests <url>|<full_test_source_path>|<relative_test_path>

<relative_test_path> is relative to the web_tests directory, regardless of the current directory.

For example:

out/Default/content_shell --run-web-tests fast/forms/001.html

or

out/Default/content_shell --run-web-tests \
    /home/user/chrome/src/third_party/blink/web_tests/fast/forms/001.html

or

out/Default/content_shell --run-web-tests ~/test/temp-test.html

By default, it dumps the text result only (as the dump of pixels and audio binary data is not human readable) and quits. This can meet the requirement of most debugging requirements. If you need to interactively debug the test page (e.g. using devtools), you'll need to run Content Shell as a simple browser.

In rare cases, to run Content Shell in the exact same way as run_web_tests.py runs it, you need to run it in the protocol mode.

out/Default/Content\ Shell.app/Contents/MacOS/Content\ Shell ...

Running HTTP Tests in Content Shell

HTTP tests reside under web_tests/http/tests. You need to start a web server first. By default it serves generated files from out/Release:

vpython3 third_party/blink/tools/run_blink_httpd.py -t <build directory>

Then run the test with a localhost URL:

out/Default/content_shell --run-web-tests http://localhost:8000/<test>

It may be necessary specify --enable-blink-features explicitly for some tests.

Running WPT Tests in Content Shell

Similar to HTTP tests, many WPT (a.k.a. web-platform-tests under web_tests/external/wpt directory) tests require some setup before running in Content Shell:

vpython3 third_party/blink/tools/run_blink_wptserve.py -t <build directory>

Then run the test:

out/Default/content_shell --run-web-tests http://localhost:8001/<test>

If the test requires HTTPS (e.g. the file name contains “.https.”), use the following command instead:

out/Default/content_shell --run-web-tests https://localhost:8444/<test>

As a simple browser

You can run the shell directly as a simple browser:

out/Default/content_shell

This allows you see how your changes look in Chromium. You can inspect the page by right clicking and selecting ‘Inspect Element’.

You can also use --remote-debugging-port

out/Default/content_shell --remote-debugging-port=9222

and open http://localhost:9222 from another browser to inspect the page. This is useful when you don‘t want DevTools to run in the same Content Shell, e.g. when you are logging a lot and don’t want the log from DevTools or when DevTools is unstable in the current revision due to some bugs.

Debug WPT

If you want to debug WPT with devtools in Content Shell, you will first need to start the server:

vpython3 third_party/blink/tools/run_blink_wptserve.py

Then start Content Shell with some additional flags:

out/Default/content_shell --enable-experimental-web-platform-features --ignore-certificate-errors --host-resolver-rules="MAP nonexistent.*.test ^NOTFOUND, MAP *.test. 127.0.0.1, MAP *.test 127.0.0.1"

You are also able to debug the inside of Chromium with a debugger for particular WPT tests. After starting the WPT server, run particular tests via Content Shell from the debugger with the following command. (Refer to your debugger's manual for how to start a program from your debugger.)

out/Default/content_shell --run-web-tests http://localhost:8001/<test>

Chromium adopts multi-process architecture. If you want to debug the child renderer processes, use --single-process Content Shell option, or --renderer-startup-dialog option and attach the debugger to the renderer processes after starting the tests. Refer to the Debugging section below for details.

Debugging

--single-process

The command line switch --single-process is useful for starting content_shell in gdb. In most cases, --single-process is good for debugging except when you want to debug the multiple process behavior or when we have some bug breaking --single-process in particular cases.

Web tests

See Run Web Tests Directly with Content Shell. In most cases you don't need --single-process because content_shell is in single process mode when running most web tests.

See DevTools frontend for the commands that are useful for debugging devtools web tests.

In The Default Multiple Process Mode

In rare cases, you need to debug Content Shell in multiple process mode. You can ask Content Shell to wait for you to attach a debugger once it spawns a renderer process by adding the --renderer-startup-dialog flag:

out/Default/content_shell --renderer-startup-dialog --no-sandbox

Debugging workers and other subprocesses is simpler with --wait-for-debugger-children, which can have one of two values: plugin or renderer.

Future Work

Reusing existing testing objects

To avoid writing (and maintaining!) yet another test controller, it is desirable to reuse an existing test controller. A possible solution would be to change DRT‘s test controller to not depend on DRT’s implementation of the Blink objects, but rather on the Blink interfaces. In addition, we would need to extract an interface from the test shell object that can be implemented by content shell. This would allow for directly using DRT's test controller in content shell.