Get the correct content-length of remote files

Question

I'm attempting to get the content-length of a remote file so that I can limit the download size of remote images thus:

 WebResponse response = this.GetWebRequest().GetResponse();

 long contentLength = response.ContentLength;

 // WebResponse.ContentLength doesn't always know the value, 
 // it returns -1 in this case.
 if (contentLength == -1)
 {
     // Response headers may still have the Content-Length inside of it.
     string headerContentLength = response.Headers["Content-Length"];

     if (!String.IsNullOrWhiteSpace(headerContentLength))
     {
         contentLength = long.Parse(headerContentLength,
                                            CultureInfo.InvariantCulture);
     }
 }

My problem is that since the content-length header is optional, sometimes that value is indeterminable.

I have read in a similar questions here and here that seems to indicate that I could somehow use the Transfer-Encoding=chunked property to determine the length but I have found no example of how to do so.

Is this possible? Can anyone provide me with a code example?

Answer 1

No, this is impossible in the general case.

The only way to be sure is to download the whole resource, and see how big it was. Especially with dynamically generated and/or streamed resources, it may not be possible to tell what size the response will be (e.g. a live stream doesn't necessarily even have a well-defined beginning, or an end - it can just keep on sending the stream as one huge chunked response).

The chunked property is a workaround for an unknown size - the chunks are preceded with the size of the following chunk; however, there's no way to predict how many chunks there will be, nor what size.

Also: caveat Content-Length, there are (even in 2012!) buggy implementations which will misreport it.