How to sync directory with AWS S3 using Java SDK?

Question

I came across this article https://aws.amazon.com/blogs/developer/syncing-data-with-amazon-s3/ which made me aware of the uploadDirectory() method. The blog states: "This small bit of code compares the contents of the local directory to the contents in the Amazon S3 bucket and only transfer files that have changed." This does not seem to be entirely correct since it appears to always transfer every file in a given directory as opposed to only the files that have changed.

I was able to do what I wanted using AWSCLI's s3 sync command, however the goal is to be able to do this syncing using the Java SDK. Is it possible to do this same type of sync using the Java SDK?

Answer 1

There is no SDK implementation of s3 sync command. You will have to implement it in Java if needed. According to the CLI doc https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html,

An s3 object will require downloading if one of the following conditions is true:

• The s3 object does not exist in the local directory.

• The size of the s3 object differs from the size of the local file.

• The last modified time of the s3 object is older than the last modified time of the local file.

Therefore essentially you will need to compare objects in target bucket with your local files based on above rules.

Also note that above checking will not handle --delete, so you might need to implement the logic for deleting remote objects when the local file does not exist if it is needed.