MPI allreduce with barrier

Question

There are few information about the internal structure of MPI allreduce operation and its relation to MPI barrier, and this few information is not consistent. So, I am trying to clarify the following two questions:

1. Should I use barrier with allreduce (my MPI version is 3.1.3)?
2. Are synchronization and blocking same things in the context of parallel programming, if not how they relate to allreduce and barrier operations?

Answer 1

No MPI collectives have barrier semantics except MPI_Barrier, although some other operations have barrier semantics due to data dependencies. MPI_All{reduce,gather,toall} have, at least for non-zero counts, such data dependencies and thus will impart a barrier.

Thus, no, you should never add a barrier to an allreduce except if you want to barrier when the count is zero.

I know of performance reasons to add barriers before collectives on some supercomputers but this is really an implementation problem.

Synchronization and blocking are not the same thing. Non-blocking collectives synchronize when completed. For example, MPI_Ibarrier is a nonblocking synchronization operation, as is MPI_Issend. See the MPI standard for more detailed definitions of the terms synchronization and blocking.