AspNetCore app is leaking threads in production - how to debug?

Question

I've got an ASP.NET Core 2 app that is leaking threads. How can I determine why the threads never die?

If I run the app for 10 minutes in production, IIS starts spitting out 502.3 Bad Gateway errors. I loaded up Process Explorer and looked at the app->Properties->Threads, and I see an ever-increasing thread count:

Nearly all the threads are stuck with the following native stack trace:

1. ntoskrnl.exe!KeWaitForMutexObject+0x8d6
2. ntoskrnl.exe!KeDelayExecutionThread+0x9a9
3. ntoskrnl.exe!KeWaitForMutexObject+0x1cf
4. ntoskrnl.exe!_misaligned_access+0x80d
5. ntoskrnl.exe!SeAccessCheck+0x280
6. ntoskrnl.exe!KeDelayExecutionThread+0xb49
7. ntoskrnl.exe!KeWaitForMutexObject+0x1cf
8. ntoskrnl.exe!KeWaitForMultipleObjects+0x2ce
9. ntoskrnl.exe!ObWaitForMultipleObjects+0x29c
10. ntoskrnl.exe!ObWaitForMultipleObjects+0x713
11. ntoskrnl.exe!KeSaveStateForHibernate+0x2a33
12. ntdll.dll!ZwWaitForMultipleObjects+0xa
13. KERNELBASE.dll!GetProcessHeap+0x62 coreclr.dll+0x4d9c8
14. coreclr.dll+0x4db61 coreclr.dll!MetaDataGetDispenser+0x55008
15. System.Private.CoreLib.dll+0x58834c

These native stack traces don't tell me much, other than my web app is leaking threads. While Process Explorer can show me .NET stack traces for full .NET framework apps, it doesn't show me .NET stack traces for .NET Core apps.

So, how does one go about debugging such an issue?

I'm unable to repro this in my local dev environment. It appears to be related to high traffic or to my production server environment.

Things I've tried so far:

• Enabled stdout logs with my app: there's no clues about failure.
• Added a AppDomain.UnhandledException handler, writing to stdout. No clues.
• Looked at the Windows Event Logs. No clues.

Answer 1

I setup Remote Debugging on my server and was able to connect to the app in unhealthy state from my dev box.

Once connected to this app in Visual Studio, I was able to pause and inspect the threads. I see they are waiting on a database resource. That opens up another question, but it is off-topic for this particular StackOverflow question.

I'm going to mark this as resolved. Lesson here is you can use Remote Debugger + Visual Studio Threads window to shed light on the thread leak problems.