bug: LocalStack seems to break when downloading large files (~1.5GB, related to processing time?) #10988
Assignees
Labels
area: infrastructure
Installation and startup of LocalStack and components
aws:s3
Amazon Simple Storage Service
status: in progress
Currently being worked on
type: bug
Bug report
Comments
|
Hello @daniel-frak and thanks for your report! Could you try starting LocalStack with the following configuration variable: As a side note, this configuration variable value will be the default in our next minor release coming up this week, so when using our next version I hope this solves the issue, if not please feel free to ping me on this issue and we can investigate further. Thank you! |
|
Thanks for the fast response! I changed the docker-compose like this: Then I deleted the container and its volume, and created it again. I ran the import and, ~1.5min later, I unfortunately got the same error :( Is there some way to confirm that the Just in case, here's the setup log for LocalStack (there are some Docker SDK errors there): |
|
Ah, sorry to hear it's still an issue. You can also use With the above logs, we should see what server LocalStack uses as the logs are different, no mention to |
|
For whatever reason, using localstack:
image: localstack/localstack:latest
restart: unless-stopped
ports:
- "4566:4566"
environment:
- DEBUG=1
- LS_LOG=trace
- GATEWAY_SERVER=twisted
volumes:
- ./localstack/localstack-script.sh:/etc/localstack/init/ready.d/script.sh # Init script
- localstack:/var/lib/localstack"... Pulls What I apparently failed to notice before is that with I've attached logs with |
|
Hi! For the old image being used, this is because it is the one you have on your system. You can run The problem persisting with the old image makes sense. Looking at the logs, as they use
The LocalStack S3 server will send the response in chunks of I have a feeling maybe the In the meantime, if you'd like to repeat the experiment with the Sorry again for the inconvenience, we'll get to the bottom of this! |
|
Thanks for the great support! The file I was using I'll try it with a bigger file (>65536 bytes) tomorrow and I'll let you know how it went. |
|
Some new information:
Running the test twice more without The relevant code: try (var reader = new BufferedReader(new InputStreamReader(response, StandardCharsets.UTF_8))) {
var csvToBean = buildCsvToBeanReader(reader);
Thread.sleep(120_000);
csvToBean.parse();
}Is there any other way I can assist in finding the cause? Should I try with some other file sizes? |
|
Hello! Thanks for coming back to me. So there is a 120s sleep time before continuing processing and this fails. But does this 120s sleep time fails against AWS as well? Would there be any way to create a very small reproducer in a Git repo, so that we could reproduce the issue ourselves too? The sample could manually create the big CSV itself before uploading it. |
|
I asked about our production code and apparently it has already been tested with try (var reader = new BufferedReader(new InputStreamReader(response, StandardCharsets.UTF_8))) {
var csvToBean = buildCsvToBeanReader(reader);
var iterator = csvToBean.iterator();
count = 0;
while (iterator.hasNext()) {
iterator.next();
count++;
if (count % 500_000 == 0) {
Thread.sleep(60000);
}
}
}The whole import apparently took 20 minutes and was successful, so I guess that test's done. I'll try to create a minimal repro repository today! |
|
I created a minimal reproduction here: https://github.com/daniel-frak/localstack-s3-issue-reproduction I managed to make it fail when uploading a 10 megabyte file with a 90 second Anything much lower than that seemed to not reproduce the error. |
|
Thank you for taking the time to make the sample! I'll come back to you once I have more info to share, but will look into it shortly. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Current Behavior
Trying to download a large file (~1.5GB) from S3, where it takes some time to process the response, leads to a
Too little data for declared Content-Lengthexception.I personally witnessed the error within 3 minutes of starting the download, but reports from other teammates suggest that this can happen instantly (the delay before the error occurs seems random).
Expected Behavior
Downloading a large file should work, regardless of size and time it takes to process.
How are you starting LocalStack?
With a docker-compose file
Steps To Reproduce
How are you starting localstack (e.g.,
bin/localstackcommand, arguments, ordocker-compose.yml)Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
Upload a large file using
Test::thisWorks:Notice the download succeed.
Upload a large file using
Test::thisFails:Notice the download fail.
Environment
Anything else?
I've tried to find the fault in my code for a few days now but I can't find anything obvious. The logs are just cryptic enough for me to suggest that this might be a bug with LocalStack itself.
I included OpenCSV code in the example, as that is the most obvious way to make it fail for us.
Some additional information:
readAllBytesseems to work.Thread.sleep()(to simulate processing while the Response is open) and the error didn't occur.The log from my application is (shortened):
The log from LocalStack is:
The text was updated successfully, but these errors were encountered: