AppendBlockAsync with more than 4mb works locally but not on Azure app service

Question

I am trying to reproduce an issue I see on our production server when using AppendBlobs.

The docs state

Each block in an append blob can be a different size, up to a maximum of 4 MiB, and an append blob can include up to 50,000 blocks. The maximum size of an append blob is therefore slightly more than 195 GiB (4 MiB X 50,000 blocks).

Which rings true with what I'm seeing on our production app, and sure enough I see these exceptions:

The request body is too large and exceeds the maximum permissible limit.
RequestId:3cb3ffd7-001e-0087-5789-ae3e0c000000
Time:2023-07-04T15:10:01.2687679Z
Status: 413 (The request body is too large and exceeds the maximum permissible limit.) ErrorCode: RequestBodyTooLarge

The problem I am having is I cannot reproduce this issue in a test.

I've got a minimal reproducible example below, which essentially creates a memory stream to a specified size, by serializing a bunch of GUIDs to a string.

I then use AppendBlob to append the blob...

I can see the memoryStream.Length is indeed greater than 4mb.

However, the puzzling thing is, this works. The file is uploaded to Blob Storage correctly, without exception.

I have seen ways to 'fix' the exception (chunking the memory stream, for example) but I was trying to reproduce this in a test first, but I can't seem to reproduce the error anywhere.

Any ideas what is happening?

[Fact]
public async Task Can_append_blob_even_if_larger_than_4mb()
{
    var containerClient  = new BlobServiceClient(ConnectionString)
        .GetBlobContainerClient("test-123");
    
    await containerClient.CreateIfNotExistsAsync();

    var outputFilename = $"Test-{DateTime.UtcNow.Ticks}.txt";
    var appendBlobClient = containerClient.GetAppendBlobClient(outputFilename);
    await appendBlobClient.CreateIfNotExistsAsync();

    var json = JsonConvert
        .SerializeObject(CreateList(6));

    var memoryStream = new MemoryStream(Encoding.UTF8.GetBytes(json));

    await appendBlobClient
        .AppendBlockAsync(memoryStream);
}

private static List<object> CreateList(int sizeInMb)
{
    const int mbInBytes = 1024 * 1024;

    var maxSizeInBytes = sizeInMb * mbInBytes;

    var totalSize = 0;

    var list = new List<object>();

    while (totalSize < maxSizeInBytes)
    {
        var obj = Guid.NewGuid();
        var serializedObj = JsonConvert.SerializeObject(obj);
        
        var objectSize = Encoding.UTF8.GetBytes(serializedObj).Length;

        if (objectSize + totalSize > maxSizeInBytes)
        {
            break;
        }

        list.Add(obj);
        totalSize += objectSize;
    }

    return list;
}

Answer 1

Have a look at the source of int AppendBlobClient.AppendBlobMaxAppendBlockBytes

This is

public virtual int AppendBlobMaxAppendBlockBytes => 
    ClientConfiguration.Version < BlobClientOptions.ServiceVersion.V2022_11_02
        ? Constants.Blob.Append.Pre_2022_11_02_MaxAppendBlockBytes
        : Constants.Blob.Append.MaxAppendBlockBytes;

Where these constants are:

 public const int Pre_2022_11_02_MaxAppendBlockBytes = 4 * Constants.MB; // 4MB
 public const int MaxAppendBlockBytes = 100 * Constants.MB; // 100MB

This larger size is not (yet) documented. This is defined in the package Azure.Storage.Blobs Version 12.17.0, released 2023-07-11.

However, in the previous package version, 12.16.0 we see something different:

public virtual int AppendBlobMaxAppendBlockBytes => Constants.Blob.Append.MaxAppendBlockBytes;
        
const int MaxAppendBlockBytes = 4 * Constants.MB; // 4MB

Hypothesis:

The test code and container is using the new, larger 100Mb value. The failing code is using the smaller 4Mb value.

it looks like this check is carried out by Azure and not by the Azure client code, so updating the Azure client package does not in itself fix the issue; in fact can make it worse if it tells you that can write 100Mb, when you can not, or vice versa.

This could explain why a recently-created container works where an older one does not. Is this size limit in Azure a container or an account setting? Can it be changed non-destructively on existing containers? Unfortunately this is not yet documented.

You can control this setting with options, e.g.

var opts = new BlobClientOptions(BlobClientOptions.ServiceVersion.V2021_12_02);
var client = new AppendBlobClient(uri, creds, opts);

But while this sets int AppendBlobClient.AppendBlobMaxAppendBlockBytes, it does not in itself cause "chunking", you still have to append chunks of the max size like this.

The docs still mostly just talk about 4Mb, but see here in "Versioning for Azure Storage":

The max append block content length has been raised from 4 MiB to 100 MiB.