Unable to access HBase table

Question

I have a cluster of 5 nodes on which I created a few tables on HBase and populated it with data. One of the nodes failed and now I am able to list all of the tables on HBase but a scan or disable of two of the tables gives

NativeException: org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in .META. for region TableName,,1295871604968

I want to drop this table but cannot get past this problem. I'm using the hbase shell for this.

Answer 1

I resolved the problem by deleting the table entries in the .META. table for the tables that gave the error and then recreated the table. I think the .META. table saved the location to the region on the dead node and where unable to update the region location for some reason when the node died. After the entries was deleted I was able to recreate the table. I found a discussion about it here that help send me on the right path.

Answer 2

Is it possible to restart the node?

HBase should have redistributed the regions to other nodes. There are probably a few reasons that this wouldn't have happened, but the one I can think of off the top of my head is that you have a corrupted HDFS. This could happen if you don't have replication in your data nodes (or replication turned below the number of number of node failures). Check your file system (hadoop fsck /)

Also, it is helpful to list the version of hbase and hadoop you are using. If possible, move to HBase version 0.90 with the hadoop 0.20-append branch (included in CDH3)