In a Parallel.For, is it possible to synchronize each thread?

Question

In a Parallel.For, is it possible to synchronize each thread with a WaitAll?

  Parallel.For(0, maxIter, i =>
  {
       // Do stuffs

       // Synchronization : wait for all threads => ???

       // Do other stuffs
  });

Answer 1

Parallel.For, in the background, batches the iterations of the loop into one or more Tasks, which can executed in parallel. Unless you take ownership of the partitioning, the number of tasks (and threads) is (and should!) be abstracted away from you. Control will only exit the Parallel.For loop once all the tasks have completed (i.e. no need for WaitAll).

The idea of course is that each loop iteration is independent and doesn't require synchronization with other concurrent work.

If synchronization is required in the tight loop, then you haven't isolated the Tasks sufficiently, or it means that Amdahl's Law is in effect, and the problem can't be speeded up through parallelization.

However, for an aggregation type pattern, you may need to synchronize after completion of each Task - use the overload with the localInit / localFinally to do this, e.g.:

// allTheStrings is shared so can't be mutated in a thread safe manner
var allTheStrings = new List<string>();
Parallel.For(         // for (
  0,                  // var i = 0;
  numberOfIterations, // i < numberOfIterations;
  () => new List<string> (), // localInit - Setup each task. List<string> --> localStrings
  (i, parallelLoopState, localStrings) =>
  {
     // The "tight" loop. If you need to synchronize here, there is no point in using Parallel.For
     // using parallel at all
     localStrings.Add(i.ToString());
     return localStrings;
  },
  (localStrings) => // local Finally for each task.
  {
     // Synchronization needed here is needed - run once per task
     lock(allTheStrings)
     {
        allTheStrings.AddRange(localStrings);
     }
  });

In the above example, you could also have just declared allTheStrings as

var allTheStrings = new ConcurrentBag<string>();

In which case, we wouldn't have required the lock in the localFinally.