C#: Farm out jobs to worker processes on a multi-processor machine

Question

I have a generic check that needs to be run on ca. 1000 objects. The check takes about 3 seconds. We have a server with 4 processors (and we also have other multi-processor servers in our network) so we would like to create an exe / dll to do the checking and return the results to the "master".

Does anyone know of a framework for this, or how would one go about it in C#?

Specifically:

• What's the best way to transfer data between the master and the worker process?
• How would the master ensure that always 4 processes are running at any one time and as soon as a worker process is finished start a new one.
• How to register that the worker is finished and append it's results to a list?

Hope it's clear enough but happy to clarify.
A.

Some clarifications
* There is actually no interprocess communication outside the call and return of the process e.g.

     ResultObject = WorkerProcess(HeresYourDataSonDoSomethingWithIt);

* Initially one machine is a must, but thinking further down the line we will probably have some cases where we have 6000 objects to check and we would like to farm it out to multiple servers, so we would like to make the right design choice from the beginning, or at least not develop a solution for one server that would have to be completely rewritten for multiple. Thanks!

Answer 1

The best way to transfer data between processes in C# is using .NET remoting. As the machines are on the same network you can use binary serialisation and IPC channels which should be very fast. You don't need 4 processes on the server machine, just multiple threads. If using .NET 4.0 check out the parallel extensions and that can greatly simplify your code, if you are not familiar with multi-threading and the pitfalls that come with it.

Also, this is probably overkill for your needs but you could look at DryadLINQ from the Microsoft Research Labs.