Java Garbage Collection- CPU Spikes - Longer connection establishment times

Question

We have a pool of server that sits behind the load balancer. The machines in this pool does garbage collection every 6 seconds on average. It takes almost half a second to garbage collect. We also see a CPU spike during garbage collection.

The client machines see a spike in average time to make a connection to the server almost 10% during a day.

Theory : CPU is busy doing GC and that's why it cannot allocate a connection faster.

Is it a valid theory?

JVM : IBM GC algorithm :gencon Nursery : 5 GB Heap Size : 18 GB

Answer 1

I'd say with that many allocations all bets are off--it could absolutely get worse over time, I mean if you are doing GC every 6 seconds all day long that seems problematic.

Do you have access to that code? Can it be re-written to reuse objects and be more intelligent about allocation? I've done a few embedded systems and the trick is to NEVER call new once the system is up and running (Quite doable if you have control over the entire system)

If you don't have access to the code, check into some of the GC tuning options available (including the selection of the garbage collector used)--both distributed with the JDK and 3rd party options. You may be able to improve performance with a few command-line modifications.

Answer 2

It's possible I guess.

Given garbage collection is such an intensive process, is there any reason for it to occur every 6 seconds? I'm not familiar with the IBM JVM or the particular collection algorithm you are using so I can't really comment on those. However, there are some good tuning documents provided by Sun (now offered by Oracle) that discuss the different types of collectors and when you would use them. See this link for some ideas.

One way to prove your theory could to be add some code that logs the time a connection was requested and the time when it was actually allocated. If the GC related CPU spikes seem to coincide with longer times in allocating connections, then that'd prove your theory. Your problem will then become how to get around it.