How to append data to an existing csv file in AWS S3 using python boto3

Question

I have a csv file in s3 but I have to append the data to that file whenever I call the function but i am not able to do that,

df = pd.DataFrame(data_list)
bytes_to_write = df.to_csv(None, header=None, index=False).encode()
file_name = "Words/word_dictionary.csv" # Not working the below line
s3_client.put_object(Body=bytes_to_write, Bucket='recengine', Key=file_name)

This code is directly replacing the data inside the file instead of appending, Any solution?

Answer 1

s3 has no append functionality. You need to read the file from s3, append the data in your code, then upload the complete file to the same key in s3.

Check this thread on the AWS forum for details

The code will probably look like:

df = pd.DataFrame(data_list)
bytes_to_write = df.to_csv(None, header=None, index=False).encode()
file_name = "Words/word_dictionary.csv"

# get the existing file
curent_data = s3_client.get_object(Bucket='recengine', Key=file_name)
# append
appended_data = current_data + bytes_to_write
# overwrite
s3_client.put_object(Body=appended_data, Bucket='recengine', Key=file_name)

Answer 2

You can try using aws data wrangler library from awslabs to append , overwrite csv dataset stored in s3. Check out their documentation and tutorials from here link