Debugging and diagnosing cache misses
To make the most of task output caching, it is important that any necessary inputs to your tasks are specified correctly, while at the same time avoiding unneeded inputs. Failing to specify an input that affects the task’s outputs can result in incorrect builds, while needlessly specifying inputs that do not affect the task’s output can cause cache misses.
This chapter is about finding out why a cache miss happened. If you have a cache hit which you didn’t expect we suggest to declare whatever change you expected to trigger the cache miss as an input to the task.
Finding problems with task output caching
Below we describe a step-by-step process that should help shake out any problems with caching in your build.
Ensure incremental build works
First, make sure your build does the right thing without the cache. Run a build twice without enabling the Gradle build cache. The expected outcome is that all actionable tasks that produce file outputs are up-to-date. You should see something like this on the command-line:
$ ./gradlew clean --quiet (1) $ ./gradlew assemble (2) BUILD SUCCESSFUL 4 actionable tasks: 4 executed $ ./gradlew assemble (3) BUILD SUCCESSFUL 4 actionable tasks: 4 up-to-date
1 | Make sure we start without any leftover results by running clean first. |
2 | We are assuming your build is represented by running the assemble task in these examples, but you can substitute whatever tasks make sense for your build. |
3 | Run the build again without running clean . |
Tasks that have no outputs or no inputs will always be executed, but that shouldn’t be a problem. |
Use the methods as described below to diagnose and fix tasks that should be up-to-date but aren’t. If you find a task which is out of date, but no cacheable tasks depends on its outcome, then you don’t have to do anything about it. The goal is to achieve stable task inputs for cacheable tasks.
In-place caching with the local cache
When you are happy with the up-to-date performance then you can repeat the experiment above, but this time with a clean build, and the build cache turned on. The goal with clean builds and the build cache turned on is to retrieve all cacheable tasks from the cache.
When running this test make sure that you have no |
This would look something like this on the command-line:
$ rm -rf ~/.gradle/caches/build-cache-1 (1) $ ./gradlew clean --quiet (2) $ ./gradlew assemble --build-cache (3) BUILD SUCCESSFUL 4 actionable tasks: 4 executed $ ./gradlew clean --quiet (4) $ ./gradlew assemble --build-cache (5) BUILD SUCCESSFUL 4 actionable tasks: 1 executed, 3 from cache
1 | We want to start with an empty local cache. |
2 | Clean the project to remove any unwanted leftovers from previous builds. |
3 | Build it once to let it populate the cache. |
4 | Clean the project again. |
5 | Build it again: this time everything cacheable should load from the just populated cache. |
You should see all cacheable tasks loaded from cache, while non-cacheable tasks should be executed.
Testing cache relocatability
Once everything loads properly while building the same checkout with the local cache enabled, it’s time to see if there are any relocation problems. A task is considered relocatable if its output can be reused when the task is executed in a different location. (More on this in path sensitivity and relocatability.)
Tasks that should be relocatable but aren’t are usually a result of absolute paths being present among the task’s inputs. |
To discover these problems, first check out the same commit of your project in two different directories on your machine.
For the following example let’s assume we have a checkout in \~/checkout-1
and \~/checkout-2
.
Like with the previous test, you should have no |
$ rm -rf ~/.gradle/caches/build-cache-1 (1) $ cd ~/checkout-1 (2) $ ./gradlew clean --quiet (3) $ ./gradlew assemble --build-cache (4) BUILD SUCCESSFUL 4 actionable tasks: 4 executed $ cd ~/checkout-2 (5) $ ./gradlew clean --quiet (6) $ ./gradlew clean assemble --build-cache (7) BUILD SUCCESSFUL 4 actionable tasks: 1 executed, 3 from cache
1 | Remove all entries in the local cache first. |
2 | Go to the first checkout directory. |
3 | Clean the project to remove any unwanted leftovers from previous builds. |
4 | Run a build to populate the cache. |
5 | Go to the other checkout directory. |
6 | Clean the project again. |
7 | Run a build again. |
You should see the exact same results as you saw with the previous in place caching test step.
Cross-platform tests
If your build passes the relocation test, it is in good shape already. If your build requires support for multiple platforms, it is best to see if the required tasks get reused between platforms, too. A typical example of cross-platform builds is when CI runs on Linux VMs, while developers use macOS or Windows, or a different variety or version of Linux.
To test cross-platform cache reuse, set up a remote
cache (see share results between CI builds) and populate it from one platform and consume it from the other.
Incremental cache usage
After these experiments with fully cached builds, you can go on and try to make typical changes to your project and see if enough tasks are still cached. If the results are not satisfactory, you can think about restructuring your project to reduce dependencies between different tasks.
Evaluating cache performance over time
Consider recording execution times of your builds, generating graphs, and analyzing the results. Keep an eye out for certain patterns, like a build recompiling everything even though you expected compilation to be cached.
You can also make changes to your code base manually or automatically and check that the expected set of tasks is cached.
If you have tasks that are re-executing instead of loading their outputs from the cache, then it may point to a problem in your build. Techniques for debugging a cache miss are explained in the following section.
Helpful data for diagnosing a cache miss
A cache miss happens when Gradle calculates a build cache key for a task which is different from any existing build cache key in the cache. Only comparing the build cache key on its own does not give much information, so we need to look at some finer grained data to be able to diagnose the cache miss. A list of all inputs to the computed build cache key can be found in the section on cacheable tasks.
From most coarse grained to most fine grained, the items we will use to compare two tasks are:
-
Build cache keys
-
Task and Task action implementations
-
classloader hash
-
class name
-
-
Task output property names
-
Individual task property input hashes
-
Hashes of files which are part of task input properties
If you want information about the build cache key and individual input property hashes, use -Dorg.gradle.caching.debug=true
:
$ ./gradlew :compileJava --build-cache -Dorg.gradle.caching.debug=true . . . Appending implementation to build cache key: org.gradle.api.tasks.compile.JavaCompile_Decorated@470c67ec713775576db4e818e7a4c75d Appending additional implementation to build cache key: org.gradle.api.tasks.compile.JavaCompile_Decorated@470c67ec713775576db4e818e7a4c75d Appending input value fingerprint for 'options' to build cache key: e4eaee32137a6a587e57eea660d7f85d Appending input value fingerprint for 'options.compilerArgs' to build cache key: 8222d82255460164427051d7537fa305 Appending input value fingerprint for 'options.debug' to build cache key: f6d7ed39fe24031e22d54f3fe65b901c Appending input value fingerprint for 'options.debugOptions' to build cache key: a91a8430ae47b11a17f6318b53f5ce9c Appending input value fingerprint for 'options.debugOptions.debugLevel' to build cache key: f6bd6b3389b872033d462029172c8612 Appending input value fingerprint for 'options.encoding' to build cache key: f6bd6b3389b872033d462029172c8612 . . . Appending input file fingerprints for 'options.sourcepath' to build cache key: 5fd1e7396e8de4cb5c23dc6aadd7787a - RELATIVE_PATH{EMPTY} Appending input file fingerprints for 'stableSources' to build cache key: f305ada95aeae858c233f46fc1ec4d01 - RELATIVE_PATH{.../src/main/java=IGNORED / DIR, .../src/main/java/Hello.java='Hello.java' / 9c306ba203d618dfbe1be83354ec211d} Appending output property name to build cache key: destinationDir Appending output property name to build cache key: options.annotationProcessorGeneratedSourcesDirectory Build cache key for task ':compileJava' is 8ebf682168823f662b9be34d27afdf77
The log shows e.g. which source files constitute the stableSources
for the compileJava
task.
To find the actual differences between two builds you need to resort to matching up and comparing those hashes yourself.
Luckily, you do not have to capture this data yourself - the build scan plugin already takes care of this for you. Gradle Enterprise has the necessary data to diagnose the cache miss when comparing two build scans, and will show you the properties that caused a different cache key to be generated.
There are three ways to compare builds in Gradle Enterprise.
-
Use the scan list to find the scans to compare, selecting them by clicking the respective
or icon. -
From a scan that you wish to compare with another, click the
icon at the top of the scan, then select the second scan by clicking the icon from the list of scans. -
Click the
icon in the header of a build scan to show previous and subsequent build scans of the same workspace, then click the icon next to the build you want to compare.
It is also possible that task output caching for a cacheable task was disabled. When this happens the reason why caching was disabled for the task is reported on the info log level and in the build scan:
Diagnosing the reasons for a cache miss
Having the data from the last section at hand, you should be able to diagnose why the outputs of a certain task were not found in the build cache. Since you were expecting more tasks to be cached, you should be able to pinpoint a build which would have produced the artifact under question.
Before diving into how to find out why one task has not been loaded from the cache we should first look into which task caused the cache misses. There is a cascade effect which causes dependent tasks to be executed if one of the tasks earlier in the build is not loaded from the cache and has different outputs. Therefore, you should locate the first cacheable task which was executed and continue investigating from there. This can be done from the timeline view in a build scan or from the task input comparison directly:
At first, you should check if the implementation of the task changed. This would mean checking the class names and classloader hashes
for the task class itself and for each of its actions. If there is a change, this means that the build script, buildSrc
or the Gradle version has changed.
A change in the output of |
If the implementation is the same, then you need to start comparing inputs between the two builds. There should be at least one different input hash. If it is a simple value property, then the configuration of the task changed. This can happen for example by
-
changing the build script,
-
conditionally configuring the task differently for CI or the developer builds,
-
depending on a system property or an environment variable for the task configuration,
-
or having an absolute path which is part of the input.
If the changed property is a file property, then the reasons can be the same as for the change of a value property. Most probably though a file on the filesystem changed in a way that Gradle detects a difference for this input. The most common case will be that the source code was changed by a check in. It is also possible that a file generated by a task changed, e.g. since it includes a timestamp. As described in Java version tracking, the Java version can also influence the output of the Java compiler. If you did not expect the file to be an input to the task, then it is possible that you should alter the configuration of the task to not include it. For example, having your integration test configuration including all the unit test classes as a dependency has the effect that all integration tests are re-executed when a unit test changes. Another option is that the task tracks absolute paths instead of relative paths and the location of the project directory changed on disk.
Example
We will walk you through the process of diagnosing a cache miss.
Let’s say we have build A
and build B
and we expected all the test tasks for a sub-project sub1
to be cached in build B
since only a unit test for another sub-project sub2
changed.
Instead, all the tests for the sub-project have been executed.
Since we have the cascading effect when we have cache misses, we need to find the task which caused the caching chain to fail.
This can easily be done by filtering for all cacheable tasks which have been executed and then select the first one.
In our case, it turns out that the tests for the sub-project internal-testing
were executed even though there was no code change to this project.
We start the input property comparison in Gradle Enterprise and see that the property classpath
changed. This means that some file on the runtime classpath actually did change.
Looking deeper into this, we actually see that the inputs for the task processResources
changed in that project, too.
Finally, we find this in our build file:
def currentVersionInfo = tasks.register('currentVersionInfo', CurrentVersionInfo) {
version = project.version
versionInfoFile = layout.buildDirectory.file('generated-resources/currentVersion.properties')
}
sourceSets.main.output.dir(currentVersionInfo.map { it.versionInfoFile.get().asFile.parentFile })
abstract class CurrentVersionInfo extends DefaultTask {
@Input
abstract Property<String> getVersion()
@OutputFile
abstract RegularFileProperty getVersionInfoFile()
@TaskAction
void writeVersionInfo() {
def properties = new Properties()
properties.setProperty('latestMilestone', version.get())
versionInfoFile.get().asFile.withOutputStream { out ->
properties.store(out, null)
}
}
}
val currentVersionInfo = tasks.register<CurrentVersionInfo>("currentVersionInfo") {
version.set(project.version as String)
versionInfoFile.set(layout.buildDirectory.file("generated-resources/currentVersion.properties"))
}
sourceSets.main.get().output.dir(currentVersionInfo.map { it.versionInfoFile.get().asFile.parentFile })
abstract class CurrentVersionInfo : DefaultTask() {
@get:Input
abstract val version: Property<String>
@get:OutputFile
abstract val versionInfoFile: RegularFileProperty
@TaskAction
fun writeVersionInfo() {
val properties = Properties()
properties.setProperty("latestMilestone", version.get())
versionInfoFile.get().asFile.outputStream().use { out ->
properties.store(out, null)
}
}
}
Since properties files stored by Java’s Properties.store
method contain a timestamp, this will cause a change to the runtime classpath every time the build runs.
In order to solve this problem see non-repeatable task outputs or use input normalization.
The compile classpath is not affected since compile avoidance ignores non-class files on the classpath. |