How to get multiple objects from S3 using boto3 get_object (Python 2.7)

Question

I've got 100s of thousands of objects saved in S3. My requirement entails me needing to load a subset of these objects (anywhere between 5 to ~3000) and read the binary content of every object. From reading through the boto3/AWS CLI docs it looks like it's not possible to get multiple objects in one request so currently I have implemented this as a loop that constructs the key of every object, requests for the object then reads the body of the object:

for column_key in outstanding_column_keys:
  try:
     s3_object_key = "%s%s-%s" % (path_prefix, key, column_key)
     data_object = self.s3_client.get_object(Bucket=bucket_key, Key=s3_object_key)
     metadata_dict = data_object["Metadata"]
     metadata_dict["key"] = column_key
     metadata_dict["version"] = float(metadata_dict["version"])
     metadata_dict["data"] = data_object["Body"].read()
     records.append(Record(metadata_dict))
   except Exception as exc:
     logger.info(exc)
if len(records) < len(column_keys):
  raise Exception("Some objects are missing!")

My issue is that when I attempt to get multiple objects (e.g 5 objects), I get back 3 and some aren't processed by the time I check if all objects have been loaded. I'm handling that in a custom exception. I'd come up with a solution to wrap the above code snippet in a while loop because I know the outstanding keys that I need:

while (len(outstanding_column_keys) > 0) and (load_attempts < 10):
 for column_key in outstanding_column_keys:
  try:
     s3_object_key = "%s%s-%s" % (path_prefix, key, column_key)
     data_object = self.s3_client.get_object(Bucket=bucket_key, Key=s3_object_key)
     metadata_dict = data_object["Metadata"]
     metadata_dict["key"] = column_key
     metadata_dict["version"] = float(metadata_dict["version"])
     metadata_dict["data"] = data_object["Body"].read()
     records.append(Record(metadata_dict))
   except Exception as exc:
     logger.info(exc)
if len(records) < len(column_keys):
  raise Exception("Some objects are missing!")

But I took this out suspecting that S3 is actually still processing the outstanding responses and the while loop would unnecessarily make additional requests for objects that S3 is already in the process of returning.

I did a separate investigation to verify that get_object requests are synchronous and it seems they are:

import boto3
import time
import os

s3_client = boto3.client('s3', aws_access_key_id=os.environ["S3_AWS_ACCESS_KEY_ID"], aws_secret_access_key=os.environ["S3_AWS_SECRET_ACCESS_KEY"])

print "Saving 3000 objects to S3..."
start = time.time()
for x in xrange(3000):
  key = "greeting_{}".format(x)
  s3_client.put_object(Body="HelloWorld!", Bucket='bucket_name', Key=key)
end = time.time()
print "Done saving 3000 objects to S3 in %s" % (end - start)

print "Sleeping for 20 seconds before trying to load the saved objects..."
time.sleep(20)

print "Loading the saved objects..."
arr = []
start_load = time.time()
for x in xrange(3000):
  key = "greeting_{}".format(x)
   try:
     obj = s3_client.get_object(Bucket='bucket_name', Key=key)
     arr.append(obj)
   except Exception as exc:
     print exc
end_load= time.time()
print "Done loading the saved objects. Found %s objects. Time taken - %s" % (len(arr), end_load - start_load)

My question and something I need confirmation is:

• Whether the get_object requests are indeed synchronous? If they are then I expect that when I check for loaded objects in the first code snippet then all of them should be returned.
• If the get_object requests are asynchronous then how do I handle the responses in a way that avoids making extra requests to S3 for objects that are still in the process of being returned?
• Further clarity/refuting of any of my assumptions about S3 would also be appreciated.

Thank you!

Answer 1

Unlike Javascript, Python processes requests synchronously unless you do some sort of multithreading (which you aren't doing in your snippet above). In your for loop, you issue a request to s3_client.get_object, and that call blocks until the data is returned. Since the records array is smaller than it should be, that must mean that some exception is being thrown, and it should be caught in the except block:

except Exception as exc:
    logger.info(exc)

If that isn't printing anything, it might be because logging is configured to ignore INFO level messages. If you aren't seeing any errors, you might try printing with logger.error.