What causes a multithreading program hanging while switching threads?

Question

I have a multithreaded program whose thread number could be customized. The program is responsible for generating HTTP requests, sending the requests to a web service, receiving response and parsing the response to get some result value.

Since it takes almost 1 sec for each request to get the response, to make the program get as many responses as possible, multiple threads are started.

Following code is used to start multithreads:

    ...
    for (int i = 0; i < threadNum; i++) {
        threadArray[i] = new Thread(requestGeneratorArray[i]);
        threadArray[i].start();
    }

    for (int i = 0; i < threadNum; i++) {
        try {
            threadArray[i].join();
        } catch (InterruptedException e) {
            logger.error(e);
        }
    }

...

When the thread number is 50, and totally 10K requests are generated and sent, the program works well. When the thread number is still 50, and the total request number is 100K. The program was hanging when 95K+ requests were sent. No exception existed, and the program just hanging there.

And then I add a few JVM arguments like this: java -Xmx1024M -Xms512M -XX:MaxPermSize=256m ... With such arguments, 50 threads/ 100K request worked. However, 50 threads/ 1M requests was hanging again. I set the thread number to 20 and request number as 1M, it worked again.

I want to set the thread number to 50, since as tested with fewer requests number (10K), 50 threads makes the highest efficiency. The request number could be much larger as 10M, 100M, event 1B. In this cases, it would not be a good idea to increase the size of -Xmx -Xms or MaxPermSize. What should I do? What's the root cause of program hanging?

===========================================================

I used Executor Service instead of threads directly. But the problem occurred as well. I rechecked the code, and found there is a mistake : I instantiated a HttpClient object for each request. I changed the code to instantiated a HttpClient instance for each thread, and the program doesn't hang anymore.

I think the root cause of program hanging is that it set up too many HTTP connections to the web service, and used out all threads of the web service. That made the web service cannot answer to any new arrived requests. Is it correct?

Answer 1

It's hard to tell only from this information, but from the fact that your heap setting affects the result, my bet will be on poor scheduling between generation of content and parsing (storing) of content.

One frequent scenario in this kind of application is that the threads which generate contents generate at faster rate than the threads who take that content and store it away. This will gradually increase the amount of heap memory used to hold the content in-memory, and at some point throughput will start to plummet.

The first thing to do is to confirm this hypothesis by attaching a heap viewer like VisualVM. If you heap usage gradually increase and starts to peg at high levels and your throughput decreases, this is likely the culprit (You could also confirm that the stuff in your memory is indeed the contents generated).

Usually, the bottleneck is IO of the persistent layer used to store the content. You can have a CPU bottleneck in the parsing code (or elsewhere) depending on what your code is doing, but this is generally rare.

The most common remedy for this situation is to use a bounded queue to make the generation process wait for the parsing (storing) process to catch up. Have a look at this SO answer: How to make ThreadPoolExecutor's submit() method block if it is saturated?. You will have to learn about thread pools but it's really a vast improvement over raw threads, and it's the most clean way to deal with this kind of problems.