Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS S3 Sync function - missing files

I am transferring files from an S3 bucket to an EC2 instance.

Using the AWS CLI, I can see that there are 219 files of ~43GB total size in the S3 bucket in question

[IN]
    aws s3api list-objects --bucket resultsdata --output json --query "[sum(Contents[].Size), length(Contents[])]"

[OUT]

[
    43367899797,
    219
]

I have been using the AWS S3 sync command to transfer the files from the bucket to the EC2 machine.

When I run this I successfully see the downloading of the individual files within the bucket. However - it appears to end prematurly.

Completed 706 of 1550 part(s) with 36 file(s) remaining

When I go to the download location and carry out a count of files

ind /home/ec2-user -type f | wc -l

I get 198 files and not 219.

I have run the AWS sync command multiple times

aws s3 sync s3://MYBUCKETNAME .

What am I missing?

Edit

I have ruled out there being duplications in the S3 bucket

Edit 2

Deleting the files from ec2 and trying to use cp -

aws s3 cp s3://mybucket . --recursive

I get

Completed 4286 of 5084 part(s) with 34 file(s) remaining

But - when I count the files I get a different number again, 186.

So sync gives me 198, cp gives me 186. Both are short of the 219 files in the S3 bucket. Anyone?!

like image 890
LearningSlowly Avatar asked Sep 05 '25 03:09

LearningSlowly


1 Answers

I had no space left on the device!

I would have expected an error message in this case. A new mounted volume has solved the issue

like image 116
LearningSlowly Avatar answered Sep 09 '25 04:09

LearningSlowly