PowerShell events received out of sequence

Question

It sounds like a reasonable expectation that events fired from one and the same thread should be received in the order in which they were fired. However, that doesn't seem to be the case. Is this a known/documented behavior, and is there any recourse to correct it?

Below are two ready-to-run code snippets that exhibit the issue, tested with PS v5.1 under both Win7 and Win10.

(a) Events fired from a thread in a separate job (i.e. a different process).

$events = 1000
$recvd = 0
$ooseq = 0

$job = Register-EngineEvent -SourceIdentifier 'Posted' -Action {
    $global:recvd++ 
    if($global:recvd -ne $event.messageData) {
        $global:ooseq++
        ("-?- event #{0} received as #{1}" -f $event.messageData, $global:recvd)
} }

$run = Start-Job -ScriptBlock {
    Register-EngineEvent -SourceIdentifier 'Posted' -Forward
    for($n = 1; $n -le $using:events; $n++) {
        [void] (New-Event -SourceIdentifier 'Posted' -MessageData $n)
} }
Receive-Job -Job $run -Wait -AutoRemoveJob

Unregister-Event -SourceIdentifier 'Posted'
Receive-Job -Job $job -Wait -AutoRemoveJob

if($events -eq $script:recvd) {
    ("total {0} events" -f $events)
} else {
    ("total {0} events events, {1} received" -f $events, $recvd)
}
if($ooseq -ne 0) {
    ("{0} out-of-sequence events" -f $ooseq)
}

Sample output from a failure case (out of a batch of 100 consecutive runs).

   -?- event #545 received as #543
   -?- event #543 received as #544
   -?- event #546 received as #545
   -?- event #544 received as #546
   total 1000 events
   4 out-of-sequence events

(b) Events fired from a separate runspace (i.e. a different thread).

$recvd = 0
$ooseq = 0

$job = Register-EngineEvent -SourceIdentifier 'Posted' -Action {
    $global:recvd++ 
    if($recvd -ne $event.messageData) {
        $global:ooseq++
        ("-?- event #{0} received as #{1}" -f $event.messageData, $recvd)
}}

$sync = [hashTable]::Synchronized(@{})
$sync.Host = $host
$sync.events = 1000
$sync.posted = 0

$rs = [runspaceFactory]::CreateRunspace()
$rs.ApartmentState = "STA"
$rs.ThreadOptions = "ReUseThread"
$rs.Open()
$rs.SessionStateProxy.SetVariable("sync",$sync)

$ps = [powerShell]::Create().AddScript({
    for($n = 1; $n -le $sync.events; $n++) {
        $sync.Host.Runspace.Events.GenerateEvent('Posted', $null, $null, $n)
        $sync.posted++
}})
$ps.runspace = $rs
$thd = $ps.BeginInvoke()
$ret = $ps.EndInvoke($thd)
$ps.Dispose()

Unregister-Event -SourceIdentifier 'Posted'
Receive-Job -Job $job -Wait -AutoRemoveJob

if($sync.events -eq $recvd) {
    ("total {0} events" -f $sync.events)
} else {
    ("total {0} events fired, {1} posted, {2} received" -f $sync.events, $sync.posted, $recvd)
}
if($ooseq -ne 0) {
    ("{0} out-of-sequence events" -f $ooseq)
}

Failure cases resemble the sample one posted under (a) above, except a few runs also had events dropped altogether. This, however, is more likely related to the other question Action-based object events sometimes lost.

total 1000 events fired, 1000 posted, 999 received
484 out-of-sequence events

---

[ EDIT ]   I ran some additional tests for case (b) specifically, and confirmed that:

• the receiving Action (where $global:recvd++) is always called on the same managed thread (this was confirmed by saving and comparing the [System.Threading.Thread]::CurrentThread.ManagedThreadId between calls);

• the receiving Action is not re-entered during execution (this was confirmed by adding a global "nesting" counter, wrapping the Action between [System.Threading.Interlocked]::Increment/Decrement calls and checking that the counter never takes any values other than 0 and 1).

These eliminate a couple of possible race conditions, but still do not explain why the observed behavior is happening or how to correct it, so the original question remains open.

Answer 1

Is this a known/documented behavior?

"Normally" event handling is asynchronous by design. And this is the case in PowerShell with cmdlets like Register-EngineEvent -Action. This is indeed known and intended behaviour. You can read more about PowerShell eventing here and here. Both Microsoft sources point out, that this way of event handling is asynchronous:

PowerShell Eventing lets you respond to the asynchronous notifications that many objects support.

and

NOTE These cmdlets can only be used for asynchronous .NET events. It’s not possible to set up event handlers for synchronous events using the PowerShell eventing cmdlets. This is because synchronous events all execute on the same thread and the cmdlets expect (require) that the events will happen on another thread. Without the second thread, the PowerShell engine will simply block the main thread and nothing will ever get executed.

So that's basically what you are doing. You forward events from your background job to the event subscriber that has an action defined and perform the action without blocking your background job. As far as I can tell, there is nothing more to expect. There is no requirement specified to process the forwarded events in any special order. Even the -Forward switch does not ensure anything more, except passing the events:

Indicates that the cmdlet sends events for this subscription to the session on the local computer. Use this parameter when you are registering for events on a remote computer or in a remote session.

It is hard and maybe impossible to find any documentation on the internals of the cmdlets. Keep in mind that Microsoft does not publish any documentation about internals afaik from the past, instead it is up to the MVPs to guess what happens inside and write books about it (drastically expressed).

So as there is no requirement to process the events in a certain order, and PowerShell just has the task to perfrom actions on an event queue, it is also allowed to perform those actions in parallel to accelerate the processing of the event queue.

Test your scripts on a VM with only one vCPU. The wrong order will still occur sometimes, but way more rarely. So less (real) parallelism, less possibilities to scramble the order. Of course, you cannot prevent the logical parallelism, implemented by different threads executed on one physical core. So some "errors" remain.

Is there any recourse to correct it?

I put "normally" into quotation marks, because there are ways to implement it synchronously. You will have to implement your own event handler of type System.EventHandler. I recommend reading this article to get an example for an implementation.

Another workaround is to store the events in an own event queue and sort them after collection (runs in ISE, not yet in PS):

$events = 10000
$recvd = 0
$ooseq = 0
$myEventQueue = @()

$job = Register-EngineEvent -SourceIdentifier 'Posted' -Action {$global:myEventQueue += $event}

$run = Start-Job -ScriptBlock {
    Register-EngineEvent -SourceIdentifier 'Posted' -Forward
    for($n = 1; $n -le $using:events; $n++) {
        [void] (New-Event -SourceIdentifier 'Posted' -MessageData $n)
    }
}
Receive-Job -Job $run -Wait -AutoRemoveJob
Unregister-Event -SourceIdentifier 'Posted'
Receive-Job -Job $job -Wait -AutoRemoveJob

Write-Host "Wrong event order in unsorted queue:"
$i = 1
foreach ($event in $myEventQueue) {
    if ($i -ne $event.messageData) {
        Write-Host "Event $($event.messageData) received as event $i"
    }
    $i++
}

$myEventQueue = $myEventQueue | Sort-Object -Property EventIdentifier

Write-Host "Wrong event order in sorted queue:"
$i = 1
foreach ($event in $myEventQueue) {
    if ($i -ne $event.messageData) {
        Write-Host "Event $($event.messageData) received as event $i"
    }
    $i++
}

---

Archived links:

• PowerShell eventing async 1
• PowerShell eventing async 2
• PowerShell eventing sync