Understand multi-threading behavior in Julia

Question

I'm trying to understand multi-threading behavior in Julia and I've noticed the following two blocks of code behave differently in Julia v1.6.3 (I'm running Julia in Atom in some script.jl):

acc = 0
Threads.@threads for i in 1:1000
         global acc
         println(Threads.threadid(), ",", acc)
         acc += 1
      end
acc

and

acc = 0
Threads.@threads for i in 1:1000
         global acc
         acc += 1
      end
acc

Notice the only difference is I took away "println(Threads.threadid(), ",", acc)" in the later case. As a result, the first block would give me 1000 every time I run it and the second block would give me some number <1000 (due to race condition).

I'm entirely new to Julia's parallel computing (or parallel computing in general), so would appreciate any explanation on what is happening here and why a single print line changes the behavior of the code block.

Answer 1

You have several threads mutating the state acc at the same time and you end up with a race condition.

However, println takes relatively long compared to the addition operation and one println occurs in time and hence for small loops you have a good chance of observing a "correct" result. However, both your loops are incorrect.

When mutating a exactly the same shared state by many threads you need either to introduce locking or use an atomic variable.

1. For fast, short running loops use SpinLock as in:

julia> acc = 0;

julia> u = Threads.SpinLock();

julia> Threads.@threads for i in 1:1000
                global acc
                Threads.lock(u) do
                    acc += 1
                end
             end

julia> acc
1000

2. The second option is ReentrantLock which is generally better for longer running loops (it takes much longer to switch than a SpinLock) with heterogenous times within loop steps (it does not take CPU time "spinning" like the SpinLock):

julia> acc = 0
0

julia> u = ReentrantLock();

julia> Threads.@threads for i in 1:1000
                global acc
                Threads.lock(u) do
                    acc += 1
                end
             end

julia> acc
1000

3. If you are mutating a primitive value (like in your case) atomic operations will be the fastest (please note how I get the value from an Atomic):

julia> acc2 = Threads.Atomic{Int}(0)
Base.Threads.Atomic{Int64}(0)

julia> Threads.@threads for i in 1:1000
                global acc2
                Threads.atomic_add!(acc2, 1)
             end

julia> acc2[]
1000