Multithread Batch Queue

Question

I am current writing a TCP listener that has many client applications sending short messages to. The TCP listener I have is a C# winform and what I need to do is to process these logs in batches to avoid hitting the database on every message I receive in the Queue. Currently, on every message I receive in the listener, I do Enqueue with the C# Queue Class.

A separate thread will execute every 5 minutes to check this Queue and start processing the Queue if there are any queued items. It seems that there is a concurrency/race condition issue with this design as when the 5 minute thread kicks off, the new messages being received can no longer access the Queue since I have a lock on it during DeQueue. Therefore, these new messages gets lost. It seems to be happening only when there are large amounts of messages being sent to the TCP listener.

Does anyone think this is a flawed design on my part or would there be a much better solution for this? I am not allowed to use MSMQ or WCF based on restrictions from the client application that are sending the messages.

Answer 1

So you have a producer-consumer scenario, with multiple producers and one (buffered) consumer. You may want to take a look at Reactive Extensions (they have a version for .NET 3.5). At the very least, you could leverage their backport of BlockingCollection<T>.