Unable to add nodes to existing Cassandra Cluster

Question

We have cassandra cluster of 6 nodes on EC2,we have to double its capacity to 12 nodes. So to add 6 more nodes i followed the following steps.

1 Calculated the tokens for 12 nodes and configured the new nodes accordingly.

2 With proper configuration started the new nodes so that they new nodes will bisect the existing token ranges.

• In the beginning all the new nodes were showing the streaming in progress.
• In ring status all the node were in "Joining" state
• After 12 hours 2 nodes completed the streaming and came into the normal state.
• But on the remaining 4 nodes after streaming some amount of data they are not showing any progress , look like they are stuck

We have installed Cassandra-0.8.2 and have around 500 GB of data on each existing nodes and storing data on EBS volume.

How can i resolve this issue and get the balanced cluster of 12 nodes?

Can i restart the nodes?

If i cleaned the data directory of stuck Cassandra nodes and restarted with fresh installation, will it cause any data loss?

Answer 1

There will not be any data loss if you replication factor 2 or greater.

Version 0.8.2 of Cassandra has several known issues - please upgrade to 0.8.8 on all original nodes as well as the new the nodes that came up and then start the procedure over for the nodes that did not complete.

Also, be aware that storing data on EBS volumes is a bad idea :

http://www.mail-archive.com/[email protected]/msg11022.html