Handle cancelled task and task exceptions in Task.WaitAll?

Question

I'm using TPL to crawl a set of Urls and then do some processing.

for (int i = 0; i < list.Count; i++)
{
    var tuple = list[i];
    string url = tuple.Item2;

    tasks[i] = httpClient.GetStringAsync(url).
        ContinueWith(task => {
        {
            ......

        });
}
Task.WaitAll(tasks);

The issue is that at the Task.WaitAll statement it seems it will often throw exceptions because a task has been cancelled. I understand that httpClient.GetStringAsync may not always ensure success so I want to add a retry logic in httpClient.GetStringAsync when exceptions occur. What will be the proper approach to do this?

Answer 1

You can easily wrap a retry around GetStringAsync with a for loop that tries until there's no exception or the retry limit is reached. I store the task and extract the result from it using await so if the retry limit was reached without success, the exception would be rethrown:

async Task<string> GetStringAsync(HttpClient client,string url, int retries)
{
    Task<string> task = null;
    for (int i = 0; i < retries; i++)
    {
        try
        {
            task = client.GetStringAsync(url);
            await task;
            break;
        }
        catch
        {
            // log
        }
    }

    return await task;
}

You could even have that as an extension method on HttpClient:

static async Task<string> GetStringAsync(this HttpClient client, string url, int retries);

Answer 2

If you don't want to use async/await you can use following extension method as a starting point.

static class HttpClientExtentions
{
    public static Task<string> GetStringWithRetryAsync(this HttpClient client, string url, int retries)
    {
        var completion = new TaskCompletionSource<string>();
        var ex = new List<Exception>();

        Task<string> getString = client.GetStringAsync(url);
        Action<Task<string>> continueAction = null;
        continueAction = (t) =>
        {
            if (t.IsFaulted)
            {
                ex.Add(t.Exception);

                if (retries-- > 0)
                {
                    getString = client.GetStringAsync(url);
                    getString.ContinueWith(continueAction);
                }
                else
                {
                    completion.SetException(new AggregateException(ex));
                }
            }
            else // assume success, you could also handle cancellation
            {
                completion.SetResult(t.Result);
            }

        };

        getString.ContinueWith(continueAction);

        return completion.Task;
    }
}

Use it this way:

for (int i = 0; i < list.Count; i++)
{
    var tuple = list[i];
    string url = tuple.Item2;

    int retryCount = 3;
    var httpClient = new HttpClient(); // should create new object for each req
    tasks[i] = httpClient.GetStringWithRetryAsync(url, retryCount).
        ContinueWith(task => {
        {
            //......

        });
}
Task.WaitAll(tasks);