Mongo DB Invariant failure

Question

Our DB of +- 400Gb is stopping on our one server.

From the logs:

2015-07-07T09:09:51.072+0200 I STORAGE  [conn10] _getOpenFile() invalid file index requested 8388701
2015-07-07T09:09:51.072+0200 I -        [conn10] Invariant failure false src/mongo/db/storage/mmap_v1/mmap_v1_extent_manager.cpp 201
2015-07-07T09:09:51.082+0200 I CONTROL  [conn10]

Any idea in what are I should start looking? Storage issue?

Answer 1

I am just answering this question in case some people make the same non-technical mistake again:

I tried to scp all the files in the /data/db directory to the server. As the files are many (dbname.1 to dbname.55, about 100GB), it was interrupted in the middle (last successful file dbname.22), and I restarted and uploaded dbname.23 to dbname.55. And when I run queries in mongo client, it worked for some cases, and failed for some others showing the error message the same as in the question. I thought it might be some file broken in the file transferring, but the md5 check was all right. Only after I spent a long time finishing all the md5 check I found the reason.

It turned out to be that scp uploads dbname.21 to dbname.29 after it uploads dbname.2, so dbname.3 to dbname.9 was never uploaded to the server. I am going to upload them, and this should solve the problem.