Mongo cannot find master during data lookups

Question

I am running a large data update using pymongo. To run the updates, individual records are found using collection.find_one(unique criteria), changes are made, the updates are batched, and finally sent in chunks using db.collection.save([long list of records to save])

On my local machine (running 1.6.3), the imports work fine.

On a remote server (running 1.6.0), which is much faster than my local machine, I can get through a portion of the inserts just fine, but then will suddenly get the following error when looking up original records:

connection = Connection(...)
...
raise AutoReconnect("could not find master/primary")
pymongo.errors.AutoReconnect: could not find master/primary

The number of records I can get through is varies somewhat, but is not random.

At first I thought I was running into the connection limit. I started closing connections manually after each record lookup:

collection.database.connection.disconnect()

Which didn't solve the problem. Am I on the right track?

Answer 1

So there are a couple of potential issues here:

raise AutoReconnect("could not find master/primary") 
pymongo.errors.AutoReconnect: could not find master/primary

That error indicates that the existing connection has somehow been invalidated. There are a number of reasons this could happen.

The most common reason this happens is that that the Primary of a Replica Set has stepped down or has failed. In this case your code needs to:

1. Catch (or trap) the error.
2. Decide on a retry strategy. (fail? retry once?...)

Are you doing this? Are you running Replica Sets or Master/Slave? Do you have any tracking for the performance of these servers? Are they having network issues? Are they switching roles?

collection.database.connection.disconnect()

Which didn't solve the problem. Am I on the right track?

Where is the exception "happening"? Is it coming from the connection itself or the save command?

On a remote server (running 1.6.0)

As of this writing, 1.6.0 is a very old version of MongoDB. There were multiple replication bugs fixed in the subsequent 1.6.x versions and 1.7.x versions. (we're already at 1.8.1rc-0)

I would start by looking at what's happening with your servers, but that may well lead you down the upgrade path.