Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

MongoDB - write concern majority - Attempted to read past the end of the stream

Tags:

mongodb

I'm connecting to version 3.4.4 of MongoDB using version 2.4.3 of the C# driver. I have a 3 member replica set (one primary, one secondary, one arbiter). I have set write concern to majority in my connection string. When both the primary and secondary nodes are online I can write to the database without any issues. But I'm having trouble when I take the primary offline. I am using this code:

    string connectionString = "mongodb://USERNAME:PASSWORD@PRIMARY,SECONDARY/?replicaSet=MyReplicaSet&w=majority";

    var client = new MongoClient(connectionString);
    IMongoDatabase database = client.GetDatabase("test");
    IMongoCollection<BsonDocument> collection = database.GetCollection<BsonDocument>("people");

    Console.WriteLine($"Number of documents: {collection.Count(FilterDefinition<BsonDocument>.Empty)}");

    collection.InsertOne(new BsonDocument("name", DateTime.Now.ToString("dd-MM-yyyy HH:mm:ss")));

    Console.WriteLine($"Number of documents: {collection.Count(FilterDefinition<BsonDocument>.Empty)}");

When I take the primary offline, the secondary gets elected and I can still write to the database (I can see the document in the database on the newly elected primary). But the call to collection.InsertOne ends up throwing a MongoDBConnectionException:

MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytes(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer(Int32 responseTo, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](DatabaseNamespace databaseNamespace, BsonDocument command, IElementNameValidator commandValidator, Func`1 responseHandling, Boolean slaveOk, IBsonSerializer`1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.ExecuteProtocol(IChannelHandle channel, BsonDocument command, Func`1 responseHandling, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.ExecuteBatch(IChannelHandle channel, BatchableSource`1 requestSource, Int32 originalIndex, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.ExecuteBatches(IChannelHandle channel, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.Execute(IChannelHandle channel, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteBatch(IChannelHandle channel, Run run, Boolean isLast, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperation[TResult](IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.InsertOne(TDocument document, InsertOneOptions options, CancellationToken cancellationToken)
   at ReplicaSetTest.Program.Main(String[] args) in C:\Code\TFS\Published Register\Main\ReplicaSetTest\Program.cs:line 57

If I don't set write concern to majority then I have no issues. I don't have much experience with MongoDB so I'm stumped as to what is going wrong. Can anyone suggest where I should begin to track down the cause of this issue?

Thanks,

David

like image 660
dlarkin77 Avatar asked Oct 18 '25 14:10

dlarkin77


1 Answers

But the call to collection.InsertOne ends up throwing a MongoDBConnectionException

Firstly, a write concern majority means that the client requests acknowledgement from the majority of the replica set (must includes the primary node). Hence, the client application will wait for a response from the primary and one of your secondary nodes. See Verify Write Operations to Replica Sets for a diagram of this.

If during the waiting period for an acknowledgement write the primary node is stepped down, the client connection to the primary would be lost. The secondary would also try to clear any established connections, because the node's state has changed from SECONDARY to PRIMARY ( See also Replica Set Member States ). This would result in a lost of connections (stream) for the client, as in the client is expecting a reply but it will never comes. Hence, you're getting:

System.IO.EndOfStreamException: Attempted to read past the end of the stream

You should try to catch this exception, and handle appropriately depending on the nature of the write.

You may also be interested to know that in MongoDB v3.6, there is Retryable Writes to provide handling of transient network errors/replica set elections.

In addition, I also noticed that your deployment consists of Primary with a Secondary and an Arbiter instead of Primary with Two Secondaries. If you have only 2 data bearing nodes and one of them is down, the write concern majority won't be able to be satisfied. Since it requires that the write operation is received by the majority of the replica set, but there's only one data bearing member active. Please consider converting the arbiter into a secondary member.

like image 84
Wan Bachtiar Avatar answered Oct 21 '25 02:10

Wan Bachtiar



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!