-
Notifications
You must be signed in to change notification settings - Fork 305
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CodeQL Analysis times out while creating zip file of results #1541
Comments
Hi Thanks for the report. Does it reproduce if you re-run it with debug logs enabled? |
I have re-run the job numerous times today while debug logs are enabled without success in reproducing. I've enabled debug logs in the repo for now and will monitor for similar failures in the future. |
I ran into this again today, however it was while a java analysis was being done. Debug logging was enabled for this execution.
|
There's still not much to go on here. It looks like the results were uploaded successfully in this latest run, but zipping of the docker container failed due to the timeout. Is the problem repeatable? Do you know if this is a large database? If this is not happening on all runs, how long does it take to zip the database normally? It's likely that if you're running a docker container inside of a standard GitHub runner, there's not much resources left to actually do the work. Maybe it always takes a long time to zip, but most of the time you're just under the threshold for the timeout. |
The issue seems to be random when it occurs. The re-run of the job yesterday succeeded with the CodeQL analysis step taking 1m 2s. The overall job execution was 3m 9s. Is there a way to find the size of the database? All I can find in the logs is the size of the results upload (which is under 2 MB unzipped). Logs indicate that the database contains 24,707 lines of Java code.
It is running on a standard GitHub runner. |
If you run in debug mode, then the database will be uploaded as an artifact at the end of the analysis job (from the log messages, it looks like you're already doing that?). You can then download it later. However, 24,707 lines of Java is not large, so I'd be surprised if the database is large. The fact that you are running the analysis in a docker container does add a layer of complexity. Is this a requirement for your java code? It might help to build without docker if you can. |
Debug logging has been enabled in this repo since my initial report in hopes of catching more info around the timeout. This particular project contains both Java and C, which is why the build is being done inside of a docker container. The image is based on the deployment image with additional tooling added to support building the native code. While I've only run into this timeout with this particular project, other members of my team have reported codeql timeouts in other projects that don't build inside of a docker container. However, I just found out about this today so I haven't been able to compare those failures to this one to see if it looks like the same issue. |
You could try explicitly setting the timeout to a higher number. Based on the logs, however, it looks like zip time is either negligible or takes 5+ minutes, so perhaps something is getting stuck. |
I'm not inclined to increase the job timeout as it doesn't appear as though there's a reasonable expectation that will resolve the issue (other than chew up more GH runner minutes). Is there some additional logging that can be enabled to debug further? It looks like the action is executing a codeql binary at the time it gets hung up so I wasn't able to follow the code well enough to find this out on my own. |
Yes, the action is calling I've asked the rest of the team for ideas. It is odd that a call to write to a zip file is causing a process to hang. |
Some suggestions:
|
Thanks for sticking with me on this. 😃
|
No worries!
There is no functional impact. Database upload only happens to help you with some extra analysis later.
Thanks. I see it. I have to admit that I'm stumped about this. I am not sure why writing to a zip file may occasionally hang. There are some things I can think of:
Howver, I think your best bet for now is to avoid uploading the database. |
I'll try disabling the database upload and see how it goes. There are no large binaries involved. I'm not sure what you meant by soft/hard links in this context. Are we talking filesystem links? If so, none of that is applicable to this project. |
I have CodeQL analysis enabled on a project that performs scans of C code using the cpp configuration. Normally, the scans works just fine. However, we had a scheduled scan fail today due to a timeout while preparing the zip file of the analysis results. The scan was performed against a commit that was successfully scanned previously, so it shouldn't be caused by anything specific to changes in the source being scanned.
A log file of the scan is attached.
codeql-cpp-analysis-timeout.zip
Unfortunately, there isn't much to go on since debug logs aren't enabled. See line 1656 for the line that hangs before it hits the 1 hour timeout configured for the job. The previously successful scan of this commit completed in just under 10 minutes.
One thing of note is that this job executes within the context of a custom docker image configured on the workflow. However, I suspect this is fairly common when working with C code due to the platform dependencies when building and executing it.
The text was updated successfully, but these errors were encountered: