SolrCloud Error:No live SolrServers available to handle this request

Question

I get below exception at solrcloud client end during data ingestion:

ERROR com.aexp.ims.atworks.ingestion.service.impl.IngestionServiceImpl  - Solr Exception
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://tsnet1:8888/solr/TSEACH_shard1_replica2, http://tsnet2:8888/solr/TSEACH_shard2_replica2, http://tsnet3:8888/solr/TSEACH_shard1_replica1, http://tsnet4:8888/solr/TSEACH_shard2_replica1]
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://tsnet5:8888/solr/TSEACH_shard2_replica2
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://tsnet2:8888 refused
Caused by: java.net.ConnectException: Connection refused

Solr log error:

ERROR update.UpdateLog - Error inspecting tlog tlog{file=/data/tsearch/solr/TSEACH_shard2_replica1/data/tlog/tlog.0000000000000005405 refcount=2}
ERROR update.UpdateLog - Error inspecting tlog tlog{file=/data/tsearch/solr/TSEACH_shard2_replica1/data/tlog/tlog.0000000000000005406 refcount=2}
ERROR update.UpdateLog - Error inspecting tlog tlog{file=/data/tsearch/solr/TSEACH_shard4_replica1/data/tlog/tlog.0000000000000005433 refcount=2}
ERROR update.UpdateLog - Error inspecting tlog tlog{file=/data/tsearch/solr/TSEACH_shard4_replica1/data/tlog/tlog.0000000000000005434 refcount=2}

Any help to resolve above error would be greatly appreciated?

Answer 1

I had a similar issue.

There could be many reasons to cause this exception. This is a one general error message masking the inner details.

While I cant offer a concrete solution to this problem, here is how I fixed my issue.

1. Check logs of all nodes.

Note that in cloud mode server logs are spread across nodes. I was running cloud on single node. I could do this:

tail -100 example/cloud/node*/logs/solr.log

2. Find out inner cause and its fix

Do you see an exception? Great. Now google/lookup for the behind the scenes exception. In my case, the group by query with group.ngroups=0 was the issue. It was fixed in 6.3 https://issues.apache.org/jira/browse/SOLR-4164 So, I upgraded solr and it solved