Nodetool command from one node to another node is not working

Question

nodetool -h 10.16.252.129 -p 9042 -u cassandra -pw cassandra status

is giving error:

nodetool: Failed to connect to '10.16.252.129:9042' - 
ConnectIOException: 'non-JRMP server at remote endpoint'.

This is in cassandra.yaml file:

rpc_address: 10.16.252.129
rpc_port: 9160

Answer 1

You have to use 7199 port here for nodetool command. However you need to check whether your port is open or not if not then you should open/allow this port on firewall.

You can find the JMX port configuration on cassandra-env.sh.

Then you should try to run below command:-

nodetool -h Hostname/IP -p 7199 -u username -pw password status

You can find more details about nodetool syntax and usage on below link. http://cassandra.apache.org/doc/latest/tools/nodetool/compactionhistory.html

Answer 2

First of all, port 9042 is for the native binary protocol CQL client connections. Port 9160 is for legacy (deprecated) Thrift protocol client connections. Inter-node nodetool commands use the JMX (Java Management eXtensions) protocol over port 7199.

Do note that in order for remote JMX to work port 7199 will need to be open (firewall) and cassandra-env.sh has configuration lines for:

$JMX_PORT="7199"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=$HOST_IP"

You may also want to enable JMX password authentication:

JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"

Also, you shouldn't need to send the port or credentials. The cassandra/cassandra creds are the default for database auth, not JMX. If you enabled JMX password auth, then you'll need to send whatever username and password you defined in the password file. But otherwise, this should work (as long as both the current and target nodes have remote JMX enabled):

nodetool -h 10.16.252.148 status