How to upload files to Amazon S3 (official SDK) that are larger than 5 MB (approx)?

Question

I am using the latest version of the official Amazon S3 SDK (1.0.14.1) to create a backup tool. So far everything works correctly if the size of the file I'm uploading is below 5 MB, but when any of the files is above 5 MB the upload fails with the following exception:

System.Net.WebException: The request was aborted: The request was canceled. ---> System.IO.IOException: Cannot close stream until all bytes are written. at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean aborting) --- End of inner exception stack trace --- at Amazon.S3.AmazonS3Client.ProcessRequestError(String actionName, HttpWebRequest request, WebException we, HttpWebResponse errorResponse, String requestAddr, WebHeaderCollection& respHdrs, Type t) at Amazon.S3.AmazonS3Client.Invoke[T](S3Request userRequest) at Amazon.S3.AmazonS3Client.PutObject(PutObjectRequest request) at BackupToolkit.S3Module.UploadFile(String sourceFileName, String destinationFileName) in W:\code\AutoBackupTool\BackupToolkit\S3Module.cs:line 88 at BackupToolkit.S3Module.UploadFiles(String sourceDirectory) in W:\code\AutoBackupTool\BackupToolkit\S3Module.cs:line 108

Note: 5 MB is roughly the boundary of failure, it can be slightly lower or anything higher

I am assuming that the connection is timing out and the stream is being automatically closed before the file upload completes.

I've tried to find a way to set a long timeout (but I can't find the option in either AmazonS3 or AmazonS3Config).

Any ideas on how to increase the time-out (like an application wide setting I can use) or is it unrelated to a timeout issue?

---

Code:

var s3Client = AWSClientFactory.CreateAmazonS3Client(AwsAccessKey, AwsSecretKey);  var putObjectRequest = new PutObjectRequest {      BucketName            = Bucket,     FilePath              = sourceFileName,     Key                   = destinationFileName,     MD5Digest             = md5Base64,     GenerateMD5Digest     = true };  using (var upload = s3Client.PutObject(putObjectRequest)) {  } 

Answer 1

Updated answer:

I recently updated one of my projects that uses the Amazon AWS .NET SDK (to version 1.4.1.0) and in this version there are two improvements that did not exist when I wrote the original answer here.

1. You can now set Timeout to -1 to have an infinite time limit for the put operation.
2. There is now an extra property on PutObjectRequest called ReadWriteTimeout which can be set (in milliseconds) to timeout on the stream read/write level opposed to the entire put operation level.

So my code now looks like this:

var putObjectRequest = new PutObjectRequest {      BucketName            = Bucket,     FilePath              = sourceFileName,     Key                   = destinationFileName,     MD5Digest             = md5Base64,     GenerateMD5Digest     = true,     Timeout               = -1,     ReadWriteTimeout      = 300000     // 5 minutes in milliseconds }; 

Original answer:

---

I managed to figure out the answer...

Before posting the question I had explored AmazonS3 and AmazonS3Config but not PutObjectRequest.

Inside PutObjectRequest there is a Timeout property (set in milliseconds). I have successfully used this to upload the larger files (note: setting it to 0 doesn't remove the timeout, you need to specify a positive number of miliseconds... I've gone for 1 hour).

This works fine:

var putObjectRequest = new PutObjectRequest {      BucketName            = Bucket,     FilePath              = sourceFileName,     Key                   = destinationFileName,     MD5Digest             = md5Base64,     GenerateMD5Digest     = true,     Timeout               = 3600000 }; 

Answer 2

I've been having similar problems to this and started to use the TransferUtility class to perform multi part uploads.

At the moment this code is working. I did have problems when the timeout was set too low though!

                var request = new TransferUtilityUploadRequest()                 .WithBucketName(BucketName)                 .WithFilePath(sourceFile.FullName)                 .WithKey(key)                 .WithTimeout(100 * 60 * 60 * 1000)                 .WithPartSize(10 * 1024 * 1024)                 .WithSubscriber((src, e) =>                 {                     Console.CursorLeft = 0;                     Console.Write("{0}: {1} of {2}    ", sourceFile.Name, e.TransferredBytes, e.TotalBytes);                 });             utility.Upload(request); 

As I'm typing this, I have a 4GB upload taking place and it's already got further through than ever before!