Why isn't boto returning the number of bytes written to S3?

Question

I'm using boto in a python script. After doing all the setup, I am trying to see if an upload to S3 succeeded by comparing the size of the file on disk with the number of bytes actually written to S3.

(Incidentally, if there's a better way to confirm a successful upload, I'd be happy to know.)

I dropped down to the Python command line and re-produced my problem. After connecting to S3, etc, I did this:

>>> bytes = os.path.getsize('file.zip')
>>> key = bucket.new_key('file.zip')
>>> bytesWritten = key.set_contents_from_filename('file.zip')
>>> bytesWritten is None
True

The docs say that the set_contents_from_filename() function should return the number of bytes written to S3. Why is it None?

(I'm using boto 2.8.0, and yes, the file exists and bytes is upward of 200k... I checked.)

Answer 1

I'm not sure at the moment why set_contents_from_filename is returning a value of None. A quick look at the code shows that it is at least trying to return the number of bytes written. I'll have to run some tests to see if I can reproduce that.

However, if the purpose of checking the bytes is to try to verify that the write operation to S3 completed successfully, there is already a much stronger integrity check built into boto. Prior to uploading the file, an MD5 checksum of the is calculated. This is sent as the Content-MD5 header in the request. In response, S3 sends an ETag header which contains the MD5 checksum that was calculated on the service side for the uploaded content. Boto compares these and raises an Exception if they do not match.