Connect to S3 accelerate endpoint with boto3

Question

I want to download a file into a Python file object from an S3 bucket that has acceleration activated. I came across a few resources suggesting whether to overwrite the endpoint_url to "s3-accelerate.amazonaws.com" and/or to use the use_accelerate_endpoint attribute.

I have tried both, and several variations but the same error was returned everytime. One of the scripts I tried is:

    from botocore.config import Config
    import boto3
    from io import BytesIO
    session = boto3.session.Session()
    s3 = session.client(                                                                            
           service_name='s3',                                                                  
           aws_access_key_id=<MY_KEY_ID>,               
           aws_secret_access_key=<MY_KEY>,
           region_name="us-west-2",
       config=Config(s3={"use_accelerate_endpoint": True,
                         "addressing_style": "path"}))
   input = BytesIO() 
   s3.download_fileobj(<MY_BUCKET>,<MY_KEY>, input)

Returns the following error:

            ---------------------------------------------------------------------------
            ClientError                               Traceback (most recent call last)
            <ipython-input-61-92b89b45f215> in <module>()
                 11      "addressing_style": "path"}))
                 12 input = BytesIO()
            ---> 13 s3.download_fileobj(bucket, filename, input)
                 14 
                 15 

            ~/Project/venv/lib/python3.5/site-packages/boto3/s3/inject.py in download_fileobj(self, Bucket, Key, Fileobj, ExtraArgs, Callback, Config)
                568             bucket=Bucket, key=Key, fileobj=Fileobj,
                569             extra_args=ExtraArgs, subscribers=subscribers)
            --> 570         return future.result()
                571 
                572 

            ~/Project//venv/lib/python3.5/site-packages/s3transfer/futures.py in result(self)
                 71             # however if a KeyboardInterrupt is raised we want want to exit
                 72             # out of this and propogate the exception.
            ---> 73             return self._coordinator.result()
                 74         except KeyboardInterrupt as e:
                 75             self.cancel()

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/futures.py in result(self)
                231         # final result.
                232         if self._exception:
            --> 233             raise self._exception
                234         return self._result
                235 

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/tasks.py in _main(self, transfer_future, **kwargs)
                253             # Call the submit method to start submitting tasks to execute the
                254             # transfer.
            --> 255             self._submit(transfer_future=transfer_future, **kwargs)
                256         except BaseException as e:
                257             # If there was an exception raised during the submission of task

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/download.py in _submit(self, client, config, osutil, request_executor, io_executor, transfer_future)
                347                 Bucket=transfer_future.meta.call_args.bucket,
                348                 Key=transfer_future.meta.call_args.key,
            --> 349                 **transfer_future.meta.call_args.extra_args
                350             )
                351             transfer_future.meta.provide_transfer_size(

            ~/Project/venv/lib/python3.5/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
                310                     "%s() only accepts keyword arguments." % py_operation_name)
                311             # The "self" in this scope is referring to the BaseClient.
            --> 312             return self._make_api_call(operation_name, kwargs)
                313 
                314         _api_call.__name__ = str(py_operation_name)

            ~/Project/venv/lib/python3.5/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
                603             error_code = parsed_response.get("Error", {}).get("Code")
                604             error_class = self.exceptions.from_code(error_code)
            --> 605             raise error_class(parsed_response, operation_name)
                606         else:
                607             return parsed_response

            ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

When I run the same script with "use_accelerate_endpoint": False it works fine. However, it returned the same error when:

• I overwrite the endpoint_url with "s3-accelerate.amazonaws.com"
• I define "addressing_style": "virtual"

When running s3.get_bucket_accelerate_configuration(Bucket=<MY_BUCKET>)

I get {..., 'Status': 'Enabled'} as expected.

Any idea what is wrong with that code and what I should change to properly query the accelerate endpoint of that bucket?

Using python3.5 with boto3==1.4.7, botocore==1.7.43 on Ubuntu 17.04.

EDIT: I have also tried a similar script for uploads:

    from botocore.config import Config
    import boto3
    from io import BytesIO
    session = boto3.session.Session()
    s3 = session.client(                                                                            
           service_name='s3',                                                                  
           aws_access_key_id=<MY_KEY_ID>,               
           aws_secret_access_key=<MY_KEY>,
           region_name="us-west-2",
       config=Config(s3={"use_accelerate_endpoint": True,
                         "addressing_style": "virtual"}))
   output = BytesIO()
   output.seek(0)
   s3.upload_fileobj(output, <MY_BUCKET>,<MY_KEY>)

Which works without the use_accelerate_endpoint option (so my keys are fine), but returns this error when True:

   ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I have tried both addressing_style options here as well (virtual and path)

Answer 1

Using boto3==1.4.7 and botocore==1.7.43. Here is one way to retrieve an object from a bucket with transfer acceleration enabled.

import boto3
from botocore.config import Config
from io import BytesIO

config = Config(s3={"use_accelerate_endpoint": True})
s3_resource = boto3.resource("s3",
                             aws_access_key_id=<MY_KEY_ID>,               
                             aws_secret_access_key=<MY_KEY>,
                             region_name="us-west-2",
                             config=config)

s3_client = s3_resource.meta.client

file_object = BytesIO()
s3_client.download_fileobj(<MY_BUCKET>, <MY_KEY>, file_object)

Note that the client sends a HEAD request to the accelerated endpoint before a GET.

The canonical request of which looks somewhat like the following:

CanonicalRequest:
HEAD
/<MY_KEY>

host:<MY_BUCKET>.s3-accelerate.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200520T204128Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Some reasons why the HEAD request can fail include:

• Object with given key doesn't exist or has strict access control enabled
• Invalid credentials
• Transfer acceleration isn't enabled