How to run multiple curl commands in parallel over https

Question

I'm trying to make multiple requests to a server running on localhost in parallel from the command line using curl, but I'm having difficulty getting them to run in parallel. One straightforward way to achieve this is something like the following:

for ((i = 0; i < 10; i++))
do
  time curl -k https://localhost:$PORT/fooBar &
done

However, the result of this was the following:

"my output"
real    0m7.555s
user    0m0.007s
sys     0m0.002s
"my output"
real    0m14.628s
user    0m0.007s
sys     0m0.002s
"my output"
real    0m21.705s
user    0m0.007s
sys     0m0.002s
"my output"
...

This despite the fact that the server is multithreaded and doesn't require locks (I know answerers will be suspicious of this, but consider the next option I tried). Because this appeared send the requests serially, I tried the following:

$ time curl -k https://localhost:$PORT/fooBar &
[1] 6780
$ time curl -k https://localhost:$PORT/fooBar &
[2] 6803
$ curl: (35) Unknown SSL protocol error in connection to localhost:<my port>

real    0m1.030s
user    0m0.003s
sys     0m0.002s
"my output"
real    0m7.128s
user    0m0.007s
sys     0m0.002s

Any ideas why these requests appear to be run serially when executed in a for loop with backgrounding and why there's an SSL error when I just use backgrounding directly from the command line?

EDIT: After further research, it appears that the lack of parallelism is because the server is using scala spray, and it appears that requests are being handled by actor in a single thread even though the handler for the endpoints is flagged by blocking. This observation splits this question in two. The one relevant to bash and curl is the fact that the for loop and multiple shell one-liners produces different results (one causing an SSL error). Resolving the lack of concurrency on the server side seems like a separate issue.

Unfortunately, I'm not sure how to reproduce the issue universally, since this is a private server, but I'm hoping someone may have an idea of why the for loop and one-liners might produce different results.

Answer 1

Just launching your processes in background sounds easy and is really tempting, but unless you are iterating on a small set, you would rather not do that. You can hog the CPU and/or network, since you do not have any control on the amount of concurrent download processes.

I would propose using xargs.

CONCURRENT_PROCESSES=10
seq 1 9 | 
     xargs -P ${CONCURRENT_PROCESSES}  \
         --replace time curl -k https://localhost:{}/fooBar 

xargs's --replace automatically sets the number of arguments per command invocation to 1 (-n 1) and replaces {} with each argument given in stdin.

If you want a more complex scenario (such as registering the timing of each individual request), you might want to invoke a wrapping script instead.