Cassandra - Frequent cross-node timeouts

Question

I am observing timeouts in the Cassandra cluster with the following logs in debug.log:

time 1478 msec - slow timeout 500 msec/cross-node

Does this represent that the read request is spending 1378 ms for the other replicas to respond?

The NTP is in sync for this cluster with fewer data and good CPU and memory allocated.

Does setting cross_node_timeout: truegoing to help?

Cassandra version: 3.11.6

Thanks

Answer 1

The value 1478 msec reported in logs is the time recorderd for a particular query to execute. As it is cross-node which signifies that this query/operation was performed across nodes. This is just a warning that your queries are running slower. Default value of slow query timeout is 500ms and can be set in cassandra.yaml by slow_query_log_timeout_in_ms.

If this is one off log in your logs, then it could have been caused by GC. If it is consistently showing up, then something is wrong in your environment(network etc) or your query.

Regarding the property cross_node_timeout: true, it was introduced via CASSANDRA-4812. Purpose of this property is to avoid timeouts in case NTP server is not synced across nodes. Default value of this property is false. Since NTP is synced on your cluster, you can do it to true but it will not help in message you are getting.