Mongo DB Error "The server is in quiesce mode and will shut down"

Question

I'm currently getting a Mongo error - "NotPrimaryError: The server is in quiesce mode and will shut down".

I have a CRON job that runs every day after midnight that just retrieves some stuff from the DB and then resubmits it, and this error comes up every 2-3 days at around the same time. After it, the DB restarts, and everything continues normally. It doesn't seem that this is a RAM problem because there is a spike but there are more than 6 GB free. I've tried to "paginate" the query set because I thought that it couldn't be loaded in memory but this still happens. I've searched everywhere but I can't find a possible solution for this. Any help is greatly appreciated.

EDIT 1: The last time this happened was on the 21st at 01:09 AM and this happens before it:

{“t”:{"$date":“2022-01-21T01:08:32.753+00:00”},“s”:“I”, “c”:“COMMAND”, “id”:21581, “ctx”:“conn32”,“msg”:“Received replSetStepUp request”}
{“t”:{"$date":“2022-01-21T01:08:32.753+00:00”},“s”:“I”, “c”:“ELECTION”, “id”:4615661, “ctx”:“conn32”,“msg”:“Starting an election due to step up request”}
{“t”:{"$date":“2022-01-21T01:08:32.754+00:00”},“s”:“I”, “c”:“ELECTION”, “id”:21437, “ctx”:“conn32”,“msg”:“Skipping dry run and running for election”,“attr”:{“newTerm”:80}}
{“t”:{"$date":“2022-01-21T01:08:32.754+00:00”},“s”:“I”, “c”:“REPL”, “id”:6015303, “ctx”:“conn32”,“msg”:“Waiting for in-progress election to complete before finishing stepup”}

The number of connections at that time was 131 which shouldn't be a problem. There is a spike in CPU and memory usage but there's plenty left. There's also no significant message that happens before this that I could find. There are some Slow query warnings but they show up at different intervals and it doesn't seem to be connected with this issue.

Answer 1

I've experienced a similar issue with my MongoDB cluster on MongoDB Atlas.

"Quiesce" mode indicates that a node in the cluster is shutting down, usually as part of maintenance, updates, or optimizations by the service provider. This is a routine process where the node is replaced with a new one.

It's nothing major, but I recommend scheduling your cron jobs just before or after midnight, as most providers perform maintenance around that time.(Factor in the time zone of your cluster)

Also know that "Quiesce" will trigger connection to MongoDB exceptions too.

NB: you may still see this error but it shouldn't affect your job anymore.