Mongo using (much) too much space

Question

UPDATE

I at first had some small databases, but I have since I dropped all databases in a (fruitless) effort to fix this.

show dbs shows

config  0.046875GB

But find /var/lib/mongo -type f -print0 | sort -z | xargs -0 du -b shows

134217728   /var/lib/mongo/a1/data/journal/j._3
88  /var/lib/mongo/a1/data/journal/lsn
134217728   /var/lib/mongo/a1/data/journal/prealloc.1
134217728   /var/lib/mongo/a1/data/journal/prealloc.2
536608768   /var/lib/mongo/a1/data/local.0
536608768   /var/lib/mongo/a1/data/local.1
536608768   /var/lib/mongo/a1/data/local.2
536608768   /var/lib/mongo/a1/data/local.3
536608768   /var/lib/mongo/a1/data/local.4
536608768   /var/lib/mongo/a1/data/local.5
536608768   /var/lib/mongo/a1/data/local.6
16777216    /var/lib/mongo/a1/data/local.ns
5   /var/lib/mongo/a1/data/mongod.lock
134217728   /var/lib/mongo/a2/data/journal/j._1
88  /var/lib/mongo/a2/data/journal/lsn
134217728   /var/lib/mongo/a2/data/journal/prealloc.1
134217728   /var/lib/mongo/a2/data/journal/prealloc.2
16777216    /var/lib/mongo/a2/data/local.0
33554432    /var/lib/mongo/a2/data/local.1
536608768   /var/lib/mongo/a2/data/local.2
536608768   /var/lib/mongo/a2/data/local.3
536608768   /var/lib/mongo/a2/data/local.4
536608768   /var/lib/mongo/a2/data/local.5
536608768   /var/lib/mongo/a2/data/local.6
536608768   /var/lib/mongo/a2/data/local.7
536608768   /var/lib/mongo/a2/data/local.8
16777216    /var/lib/mongo/a2/data/local.ns
5   /var/lib/mongo/a2/data/mongod.lock
134217728   /var/lib/mongo/b1/data/journal/j._4
88  /var/lib/mongo/b1/data/journal/lsn
134217728   /var/lib/mongo/b1/data/journal/prealloc.1
134217728   /var/lib/mongo/b1/data/journal/prealloc.2
536608768   /var/lib/mongo/b1/data/local.0
536608768   /var/lib/mongo/b1/data/local.1
536608768   /var/lib/mongo/b1/data/local.2
536608768   /var/lib/mongo/b1/data/local.3
536608768   /var/lib/mongo/b1/data/local.4
536608768   /var/lib/mongo/b1/data/local.5
536608768   /var/lib/mongo/b1/data/local.6
16777216    /var/lib/mongo/b1/data/local.ns
5   /var/lib/mongo/b1/data/mongod.lock
16384   /var/lib/mongo/b2/data/journal/j._1
88  /var/lib/mongo/b2/data/journal/lsn
16777216    /var/lib/mongo/b2/data/local.0
33554432    /var/lib/mongo/b2/data/local.1
536608768   /var/lib/mongo/b2/data/local.2
536608768   /var/lib/mongo/b2/data/local.3
536608768   /var/lib/mongo/b2/data/local.4
536608768   /var/lib/mongo/b2/data/local.5
536608768   /var/lib/mongo/b2/data/local.6
536608768   /var/lib/mongo/b2/data/local.7
536608768   /var/lib/mongo/b2/data/local.8
16777216    /var/lib/mongo/b2/data/local.ns
5   /var/lib/mongo/b2/data/mongod.lock
16777216    /var/lib/mongo/config/data/config.0
33554432    /var/lib/mongo/config/data/config.1
16777216    /var/lib/mongo/config/data/config.ns
134217728   /var/lib/mongo/config/data/journal/j._0
88  /var/lib/mongo/config/data/journal/lsn
134217728   /var/lib/mongo/config/data/journal/prealloc.1
134217728   /var/lib/mongo/config/data/journal/prealloc.2
5   /var/lib/mongo/config/data/mongod.lock

Thats over 17GB for no data! Sheesh.

Why is Mongo taking so much space, and how can I reduce it?

Answer 1

What are all these files?

It looks like whoever did the initial install in your environment created a sharded cluster, which was probably two shards each backed by two data-bearing nodes in a replica set. They chose to create subdirectories for each node under a common path of /var/lib/mongo/.

Based on the subdirectory names, I would guess that a1 and a2 were a replica set for shard a and b1 and b1 were part of a replica set for shard b. If you are curious, you could probably confirm the former configuration by inspecting the config database which still exists.

In a case, in a replica set each data-bearing node has a preallocated oplog (operation log) which is used for replication. The oplog is stored in the oplog.rs collection in the local database.

By default the oplog size is 5% of free disk space (at least on a 64-bit Linux system) which is why these are taking up a good chunk of disk despite having an "empty" database.

How can I safely clean up this data directory?

If you are only running a single MongoDB instance, by default it will ignore extra subdirectories and their contents. This is why you still have extra disk usage - you've dropped all the databases that your mongod knows about.

If you want to "reset" all the data and start fresh you should:

• stop your running mongod instance, eg: sudo service mongod stop
• Note: if you want to be extra certain there are no lingering mongod processes from the previous sharded installation, you could also run: killall mongod; killall mongos
• change to the data directory: cd /var/lib/mongo/
• make sure you are in the expected directory: pwd
• remove all files: rm -r *
• restart mongod, eg: sudo service mongod start