On which errors should curl retry for web API requests?

Question

I am implementing a class which retrieves information from a web API. It returns XML to HTTPS GET requests. I am using curl for the request and want it to retry in case of errors to make it more robust.

I am currently retrying (up to 3 times) for all curl errors and HTTP error 500, but I am wondering if that is the best approach.

So on which curl and http errors should I retry for my API requests? what is best practice?

Answer 1

I disagree about 500 errors. The Mailgun API, for example, has returned intermittent 421 and 500 errors, and a retry might well succeed. Since you're only trying a few times and the errors should be rare, you might as well make more attempts when you get one.

I would retry on these HTTP errors:

408, 421, 429, 500, 502, 503, 504

There's a good explanation of what the codes mean HERE

Answer 2

I'd would say don't retry on 500 errors. That's some error on the API side and that's unlikely to change (at least, in a few seconds).

If you really want to retry on some responses, do it on timeouts, ie. 408, 504. It can be that the server is receiving more requests than it can handle, and waiting a few seconds to try again may be worthy.