How to make concurrent HTTP requests with basic authentication

Question

My goal is to import a client's history of orders from Shopify. Shopify only allows me to import 250 orders per request, but my client has thousands.

Here's (basically) my current working solution using httr which is very slow

fetchedList <- list()

# Loop through pages of orders and collect them
for(pg in 1:10){

  requestURL <- paste0("https://abc-store.myshopify.com/admin/orders.json?page=", p)

  fetched <- httr::GET(
    url = requestURL,
    httr::add_headers(Accept = "application/json"),
    httr::authenticate(user = "foo", password = "bar")
  )

  # Append the fetched response to fetchedList 
  fetchedList <- c(fetchedList, list(fetched))
}

# Process the results...

I would like to speed this up by making multiple concurrent requests. How can I achieve this? It seems curl and RCurl both have support for this but I'm fairly new to HTTP and couldn't get either solution working.

Answer 1

You should use the multi api to do concurrent requests. See the manual page for ?multi_run or the section about async requests in the vignette.

There are also packages that wrap the multi api to try and make it easier. The crul package (note crul is not a typo :) or more if you want to get real fancy the async package.