Consistency level changes while reading from cassandra

Question

I have 10 nodes cassandra cluster with replication factor of {DC1:3,DC2:2}. DC1 has 6 nodes in rack1 and DC2 has 4 nodes in rack1. I am making a read request with consistency level of LocalQuorum. Even though I set consistency level at each query, some times it returns following error,

com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (5 responses were required but only 3 replica responded)

I am using cassandra 2.0.9 and datastax java driver version 2.0.5. In nodetool status, it shows all nodes are up & normal. How can i overcome this? How consistency level changed to ALL?

Cluster construction:

private static Cluster constructCluster(String hostName,String port) {
    String[] hostNames = hostName.split(",");
    SocketOptions sOptions = new SocketOptions();
    sOptions.setKeepAlive(true);
    QueryOptions qOptions = new QueryOptions().setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM)
            .setFetchSize(500);
    LatencyAwarePolicy loadBalancingPolicy = LatencyAwarePolicy.builder(new DCAwareRoundRobinPolicy(defaultDC))
            .build();
    Cluster cluster = Cluster.builder()
            .addContactPoints(hostNames)
            //.withCompression(Compression.SNAPPY)
            .withLoadBalancingPolicy(loadBalancingPolicy)
            .withPoolingOptions(new PoolingOptions())
            .withQueryOptions(qOptions)
            .withReconnectionPolicy(new ConstantReconnectionPolicy(TimeUnit.SECONDS.toMillis(5)))
            .withRetryPolicy(new LoggingRetryPolicy(DefaultRetryPolicy.INSTANCE))
            .withSocketOptions(sOptions)
            .build();
    return cluster;
}

Data retrieval:

Cluster cluster = constructCluster(hosts, null);
Session session = cluster.connect("mykeyspace");
Statement stmt = QueryBuilder.select().all().from(cf).where(QueryBuilder.eq("key", row_key)).setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM);
ResultSet resultSet = session.execute(stmt);

Answer 1

Your issue looks a lot like JIRA ISSUE 7868 which leads to JIRA ISSUE 7947, since looking at your code I do not see anything wrong.

As I see from second issue it has been fixed in 2.0.12 and 2.1.3 so you might want to update version.

We had same issue, basically error message was misleading, when quorum request has checksum mismatch, it issues read repair with CL ALL and if that times out it gives this error message.