AWS S3 AmazonServiceException A WebException with status SecureChannelFailure

Question

Have service that receives requests, generates data and saves this in files to AWS S3. If service receives many requests can try to save up to 20 files (2 servers x 10 workers each) in parallel. Data generated by requests and saved to S3 can be from few KB to around ~400MB

Problem is that sometimes (seems to be when service is busy/big files to save) S3 fails with the exception below: We discussed 2 solutions:

1) Implement a retry of S3.UploadAsync() if save fails. Not sure if will make any difference. Assume S3 already retries internally so maybe is no point to retry. If problem is that files are too big/takes to long to save this won't solve the issue maybe make it worst.

2) Increase the TransferUtilityConfig.DefaultTimeout to, lets say 10min (default is 5min). If problem is that saving takes more than 5 minutes, this will fix the issue but exception thrown by S3 doesn't indicate is a timeout exception so maybe this will solve anything.

3) Is this an intermittent issue in AWS infrastructure? Could retry help?

Does anyone have experience/solution when this exception happens? Any other ideas?

UPDATE: TransferUtilityConfig() does not contain DefaultTimeout if using NET 4.5. The functionality has been moved to AmazonS3Config. This offers more parameters to control the upload: Timeout, ReadWriteTimeout, MaxErrorRetry AmazonS3Config Class Settings are explained here AWS Retries and Timeouts

This is the code used by the service to save:

using (var amazonS3Client = new AmazonS3Client(RegionEndpoint.GetBySystemName(_iAwsS3Settings.RegionEndpoint)))
using (var fileTransferUtility = new TransferUtility(amazonS3Client))
using (var memoryStream = new MemoryStream(data))
{
    var fileTransferUtilityRequest = new TransferUtilityUploadRequest
    {
        BucketName = _iAwsS3Settings.BucketName,
        InputStream = memoryStream,
        StorageClass = S3StorageClass.ReducedRedundancy,
        PartSize = 6291456, // 6 MB.
        Key = fileLocation,
        CannedACL = S3CannedACL.BucketOwnerFullControl
    };

    await fileTransferUtility.UploadAsync(fileTransferUtilityRequest, ct);
}

This is the exception given when S3 save fails:

System.AggregateException: One or more errors occurred. ---> Amazon.Runtime.AmazonServiceException: A WebException with status SecureChannelFailure was thrown. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel. at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.HttpHandler1.<InvokeAsync>d__91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Amazon.Runtime.Internal.HttpHandler1.<InvokeAsync>d__91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.RedirectHandler.d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__31.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.S3.Internal.AmazonS3ResponseHandler.d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__51.MoveNext() --- End of inner exception stack trace --- at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext executionContext, WebException exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.d__51.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CredentialsRetriever.d__71.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__101.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Amazon.Runtime.Internal.RetryHandler.d__10`1.MoveNext()

Answer 1

This is kind of an old question but I just had a very similar issue and because I was unable to find an appropriate answer on Google, I wanted to contribute my solution.

I had started implementation of an older API which had the following line:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;

I was able to correct the issue by changing SecurityProtocolType.Ssl3 to SecurityProtocolType.Tls12

Further explanation:

SSL3 is not generally supported due to the Poodle vulnerability:

https://security.stackexchange.com/questions/70719/ssl3-poodle-vulnerability/70724#70724

I had not added this line of code myself and as a result, I was having great difficulty finding the source of the issue. However, while researching I had come across this post regarding a similar issue: https://github.com/aws/aws-sdk-net/issues/86

Near the bottom of this discussion, someone suggested adding the following line:

ServicePointManager.CheckCertificateRevocationList = true;

This fix was not able to correct my issue but it did lead me down the correct path. After searching for other references to the ServicePointManager, I was able to find the previously mentioned line regarding the SecurityProtocolType and correct it.