Jedis is running out of resource instances, web application blocks

Question

I'm running a Tomcat application which uses Jedis to access a Redis database. Form time to time the whole application blocks. By monitoring Tomcat using JavaMelody I found out that the problem seems be related to the JedisPool when a object requests a Jedis instance.

catalina-exec-74
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1104)
redis.clients.util.Pool.getResource(Pool.java:20)
....

This is the JedisPoolConfig I'm using

JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxActive(20);
poolConfig.setTestOnBorrow(true);
poolConfig.setTestOnReturn(true);
poolConfig.setMaxIdle(5);
poolConfig.setMinIdle(5);
poolConfig.setTestWhileIdle(true);
poolConfig.setNumTestsPerEvictionRun(10);
poolConfig.setTimeBetweenEvictionRunsMillis(10000);
jedisPool = new JedisPool(poolConfig, "localhost");

So obviously some threads try to get a Jedis instance but the pool is empty and cannot return an instance so the default pool behavior is wait.

I've already double checked my whole code and I'm pretty sure I return every Jedis instance to the pool that I used before. So I'm not sure why I'm running out of instance.

Is there a ways to check how many instances are left in the pool? I'm trying to find a sensible value for the maxActive parameter to prevent the application from blocking.

Are there any other ways to create memory holes other than not returning the Jedis instances to the pool?

Answer 1

Returning the resource to the Pool is important, so remember to do it. Otherwise when closing your app it'll wait for the resource to return.

https://groups.google.com/forum/?fromgroups=#!topic/jedis_redis/UeOhezUd1tQ

After each Jedis method call, return the resource pool. Your app has probably used all the threads and waits for some to be dropped. This may cause behavior you're explaining and the app is probably blocked.

Jedis jedis = JedisFactory.jedisPool.getResource();
try{
    jedis.set("key","val");
}finally{
    JedisFactory.jedisPool.returnResource(jedis);
}

Answer 2

Partial answer to hopefully be of some help to people in similar scenarios, though I'm unsure if my problem was the same as yours (if you've since figured it out, please let us know!).

I've already double checked my whole code and I'm pretty sure I return every Jedis instance to the pool that I used before. So I'm not sure why I'm running out of instance.

I thought I had to - I always put my code in try / finally blocks, but it turns out I did have a memory leak:

Jedis jedis = DBHelper.pool.getResource();
    try {
        // Next line causes the leak
        Jedis jedis = DBHelper.pool.getResource();
        ...
    } finally {
        DBHelper.pool.returnResource(jedis);
    }

No idea how that second call snuck in, but it did and caused both a leak and the web app to block.

Is there a ways to check how many instances are left in the pool? I'm trying to find a sensible value for the maxActive parameter to prevent the application from blocking.

In my case, I both found the bug and optimized based on the number of clients seen by the redis server. I set the loglevel (in redis.conf to verbose (default is notice), which will report about every 5-10 seconds the number of clients connected. Once I found my memory leak, I repeatedly sent requests to the page calling that method, and watched the redis clients reported by redis logs climb, but never drop.. I would think that would be a good start for optimizing?

Anyways, hope this is helpful to someone!