On Linux, Chromium can use Breakpad to generate minidump files for crashes. It is possible to convert the minidump files to core files, and examine the core file in gdb, cgdb, or Qtcreator. In the examples below cgdb is assumed but any gdb based debugger can be used.
Use minidump-2-core
to convert the minidump file to a core file. On Linux, one can build the minidump-2-core target in a Chromium checkout, or alternatively, build it in a Google Breakpad checkout.
$ ninja -C out/Release minidump-2-core $ ./out/Release/minidump-2-core foo.dmp > foo.core
If the minidump is from a public build then Googlers can find Google Chrome Linux binaries and debugging symbols via https://goto.google.com/chromesymbols. Otherwise, use the locally built chrome files. Google Chrome uses the debug link method to specify the debugging file. Either way be sure to put chrome
and chrome.debug
(the stripped debug information) in the same directory as the core file so that the debuggers can find them.
For Chrome OS release binaries look for debug-*.tgz
files on GoldenEye.
The recommended syntax for loading a core file into gdb/cgdb is as follows, specifying both the executable and the core file:
$ cgdb chrome foo.core
If the executable is not available then the core file can be loaded on its own but debugging options will be limited:
$ cgdb -c foo.core
If symbols do not seem to work, see Advanced module loading below.
Qtcreator is a full GUI wrapper for gdb and it can also load Chrome's core files. From Qtcreator select the Debug menu, Start Debugging, Load Core File... and then enter the paths to the core file and executable. Qtcreator has windows to display the call stack, locals, registers, etc. For more information on debugging with Qtcreator see Getting Started Debugging on Linux.
If you have a Chromium repo that is synchronized to exactly (or even approximately) when the Chrome build was created then you can tell gdb/cgdb/Qtcreator
to load source code. Since all source paths in Chrome are relative to the out/Release directory you just need to add that directory to your debugger search path, by adding a line similar to this to ~/.gdbinit
:
(gdb) directory /usr/local/chromium/src/out/Release/
add-symbol-file
command to work, the file must have debugging symbols.If gdb doesn't find shared objects that are needed you can force it to load them. In gdb, the add-symbol-file
command takes a filename and an address. To figure out the address, look near the end of foo.dmp
, which contains a copy of /proc/pid/maps
from the process that crashed.
One quick way to do this is with grep
. For instance, if the executable is /path/to/chrome
, one can simply run:
$ grep -a /path/to/chrome$ foo.dmp 7fe749a90000-7fe74d28f000 r-xp 00000000 08:07 289158 /path/to/chrome 7fe74d290000-7fe74d4b7000 r--p 037ff000 08:07 289158 /path/to/chrome 7fe74d4b7000-7fe74d4e0000 rw-p 03a26000 08:07 289158 /path/to/chrome
In this case, 7fe749a90000
is the base address for /path/to/chrome
, but gdb takes the start address of the file's text section. To calculate this, one will need a copy of /path/to/chrome
, and run:
$ objdump -x /path/to/chrome | grep '\.text' | head -n 1 | tr -s ' ' | cut -d' ' -f 5 00000000005282c0
Now add the two addresses: 7fe749a90000 + 5282c0 = 7fe749fb82c0
and in gdb, run:
(gdb) add-symbol-file /path/to/chrome 0x7fe749fb82c0
Then use gdb as normal.
If all you need is a stack trace, you can use minidump_stackwalk instead of gdb as follows:
Compile minidump_stackwalk: autoninja -C out/Default minidump_stackwalk
Download breakpad_info.zip
for the correct version from go/chromesymbols (Googlers only).
You can instead generate the .sym files yourself, if you have a build with symbols: out/Default/dump_syms chrome > chrome.sym
minidump_stackwalk expects a directory structure like debugfile/BUILDID/debugfile.sym
. On Linux and macOS debugfile is just the name of the executable; on Windows, debugfile is that plus a .pdb
extension.
The buildid is in the first line of the breakpad info file:
$ head -1 chrome.breakpad.x64 MODULE Linux x86_64 0ACBB4D08FB145E1656ADD88DF70B1320 chrome.debug
So for this specific example, you'd do:
mkdir -p symbols/chrome/0ACBB4D08FB145E1656ADD88DF70B1320 mv chrome.breakpad.x64 symbols/chrome/0ACBB4D08FB145E1656ADD88DF70B1320
Now you can run minidump_stackwalk:
$ out/Default/minidump_stackwalk 5aeb7476-3200-41ed-8db8-b5d71bea28d3.dmp symbols/ [...] Thread 0 (crashed) 0 chrome!content::protocol::FedCmHandler::OnDialogShown() [vector : 1434 + 0x0] rax = 0x00003b2401be0720 rdx = 0x000000000000002e
For more discussion on this process see [Debugging a Minidump]. This page discusses the same process in the context of Chrome OS and many of the concepts and techniques overlap.