Chrome is client-side software, which means that sometimes there are bugs that can occur only on users' machines (“in production”) that cannot be reproduced by test or software engineering. When this happens, it's often helpful to gather bug- specific data from production to help pinpoint the cause of the crash. The crash key logging system is a generic method to help do that.
The core of the crash key logging system is in //components/crash/core/common/crash_key.h, which declares a crash_reporter::CrashKeyString class. Every crash key has an associated value maximum length and a string name to identify it. The maximum length is specified as a template parameter in order to allocate that amount of space for the value up-front. When a process is crashing, memory corruption can make it unsafe to call into the system allocator, so pre-allocating space for the value defends against that.
When a crash key is set, the specified value is copied to its internal storage. And if the process subsequently crashes, the name-value tuple is uploaded as POST form-multipart data when the crash report minidump is uploaded to the Google crash reporting system. (The data therefore are only accessible to those with access to crash reports internally at Google). For platforms that use Crashpad as the crash reporting platform, the crash keys are also stored in the minidump file itself. For platforms that use Breakpad, the keys are only available at upload.
The crash key system is used to report some common pieces of data, not just things that happen in exceptional cases: the URL of the webpage, command line switches, active extension IDs, GPU vendor information, experiment/variations information, etc.
Beware that certain on certain platforms (e.g. Android Webview) we sanitize the stack in the dump and only crash keys on an allowlist will be captured.
Imagine you are investigating a crash, and you want to know the value of some variable when the crash occurs; the crash key logging system enables you to do just that.
A crash key must be allocated using static storage duration, so that there is space for the value to be set. This can be done as a static variable in the global or function scope, or in an anonymous namespace:
static crash_reporter::CrashKeyString<32> crash_key_one("one");
namespace {
crash_reporter::CrashKeyString<64> crash_key_two("two");
}
void DoSomething(const std::string& arg) {
static crash_reporter::CrashKeyString<8> three("three");
crash_key_two.Set(arg);
three.Set("true");
}
The template argument specifies the maximum length a value can be, and it should include space for a trailing NUL byte. Values must be C-strings and cannot have embedded NULs. The constructor argument is the name of the crash key, and it is what you will use to identify your data in uploaded crash reports.
If you need to declare an array of crash keys (e.g., for recording N values of an array), you can use a constructor tag to avoid warnings about explicit:
static ArrayItemKey = crash_reporter::CrashKeyString<32>;
static ArrayItemKey crash_keys[] = {
{"array-item-1", ArrayItemKey::Tag::kArray},
{"array-item-2", ArrayItemKey::Tag::kArray},
{"array-item-3", ArrayItemKey::Tag::kArray},
{"array-item-4", ArrayItemKey::Tag::kArray},
};
The crash key system will require your target to have a dependency on //components/crash/core/common:crash_key. If you encounter link errors for unresolved symbols to crashpad::Annotation::SetSize(unsigned int), adding the dependency will resolve them.
After a key has been allocated, its Set(base::StringPiece) and Clear() methods can be used to record and clear a value. In addition, crash_key.h provides a ScopedCrashKeyString class to set the value for the duration of a scope and clear it upon exiting.
Using http://go/crash (internal only), find the crash report signature related to your bug, and click on the “N of M” reports link to drill down to report-specific information. From there, select a report and go to the “Product Data” section to view all the crash key-value pairs.
Not all targets in the Chromium source tree are permitted to depend on the //components/crash/core/common:crash_key target due to DEPS file include_rules.
If the crash key being added is only a temporary debugging aid to track down a crash, consider adding the dependency temporarily and removing it when done. A specific include rule can be added for crash_key.h:
# DEPS include_rules = [ '+components/crash/core/common/crash_key.h', ]
Then simply remove it (and the BUILD.gn dependency) once the crash is resolved and the crash key deleted.
If this crash key is more permanent, then there is an alternate API in //base that can be used. This API is used by the //content module to set its permanent crash key information. Note however that the base-level API is more limited in terms of features and flexibility. See the header documentation in //base/debug/crash_logging.h for usage examples.
Now imagine a scenario where you have a use-after-free. The crash reports coming in do not indicate where the object being used was initially freed, however, just where it is later being dereferenced. To make debugging easier, it would be nice to have the stack trace of the destructor, and the crash key system works for that, too.
Declaring the crash key is no different than written above, though special attention should be paid to the maximum size argument, which will affect the number of stack frames that are recorded. Typically a value of 1024 is recommended.
To set a stack trace to a crash key, use the SetCrashKeyStringToStackTrace() function in crash_logging.h:
Usemeafterfree::~Usemeafterfree() {
static crash_reporter::CrashKeyString<1024> trace_key("uaf-dtor-trace");
crash_reporter::SetCrashKeyStringToStackTrace(&trace_key,
base::debug::StackTrace());
}
Unlike with the previous example, a stack trace will just be a string of hexadecimal addresses. To turn the addresses back into symbols use, http://go/crsym (internal instance of https://github.com/chromium/crsym/). Using the Crash Key input type, give it a crash report ID and the name of your crash key. Crsym will then fetch the symbol data from the internal crash processing backends and return a formatted, symbolized stack trace.