How to sync repo in bitbucket to S3 with jenkins

Question

Background: I have a repository say example.com along with master and develop branch. My master branch repo push the change to /efs/prod through jenkins build and develop branch repo push the change to /efs/qa through jenkins build. Now I have setup a folder say /s3 in my develop branch of repo to push the changes to amazon S3, for this I have used "Publish Artifacts to S3 Bucket" plugin in jenkins.

Requirement: I am able to move the bitbucket uploads to aws s3 but when someone delete the file in bitbucket repo then I am unable to sync it with AWS s3. I have followed below source to move the repo changes to s3

http://www.devops-share.com/upload-builds-from-jenkins-to-s3/

Could someone let me know how could I make jenkins to sync the bitbucket repo with S3.

Answer 1

A possible solution is to install aws-cli on the jenkins instance, set it up with aws configure, ideally granting it specific permissions to upload to a specific S3 bucket through IAM policies.

Next up is actually uploading your data, after your jenkins is done processing / building / compiling your code, run the following command from the directory you'd like to sync to S3:

aws s3 sync . s3://YOUR_BUCKET_NAME

If you'd like it to exclude certain directories, add --exclude "folder_name/**"

As it states in the aws-cli docs: "Recursively copies new and updated files from the source directory to the destination."

It will not delete files that are absent in the source directory but exist in the bucket.

If you really want this behaviour, you could blow away the bucket contents before each upload but this would result in downtime so perhaps consider a CloudFront setup that caches your bucket contents and you can clean this cache from Jenkins after your aws s3 sync has concluded successfully.

To invalidate a CloudFront cache, see the docs

Answer 2

I have resolved it by using below command at Build>Execute shell in jenkins

aws s3 sync --delete cdn/ s3://bucket-path