.NET 9 ThreadPool Starvation - High QueueSize with Many Idle Worker Threads

[vrecluse]

Problem Description
We've encountered a critical performance issue with the .NET 9.0 thread pool (using SDK 9.0.303, a stable release) in our production/high-load environment.

Under heavy load, the application becomes extremely slow or unresponsive. Upon diagnosis with dotnet-stack, we observed a pathological state:

The ThreadPool's global work item queue (QueueSize) was backlogged with thousands of items (in our log, the count was 6759).

Simultaneously, dozens of ThreadPool worker threads (PortableThreadPool+WorkerThread) were completely idle, waiting on LowLevelLifoSemaphore.Wait.

Expected Behavior:
When there are pending work items in the global queue, idle worker threads should be woken up immediately to process them.

Actual Behavior:
Work items are severely backlogged in the global queue while a large number of worker threads remain idle, leading to effective thread pool starvation.

Analysis
This issue appears to be related to the internal scheduling or signaling/wakeup mechanism of the new Portable Thread Pool introduced in .NET 9. The logs strongly suggest a failure in the mechanism responsible for waking up idle threads despite a massive number of pending tasks.

Our application is an Orleans-based service that uses ASP.NET Core Kestrel, MongoDB, Nacos, OpenTelemetry, and other libraries, involving a high degree of asynchronous I/O.

Evidence: dotnet-stack Trace
Below are the key parts of the dotnet-stack log we captured. The full log file can be provided if needed.

Log Summary:

QueueSize: 6759

Typical stack trace for the numerous idle threads:

Thread (0x3D6B):
CPU_TIME
System.Private.CoreLib!System.Threading.LowLevelLifoSemaphore.WaitNative(...)
System.Private.CoreLib!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int32)
System.Private.CoreLib!System.Threading.LowLevelLifoSemaphore.Wait(int32,bool)
System.Private.CoreLib!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
At the same time, at least one worker thread was active, processing an HTTP/2 frame write in Kestrel.

We believe this is a critical bug within the .NET 9.0 runtime. We hope this information is helpful for the investigation.

Panda.Silo.Server.stack_log_20250721_161744.txt

[mangod9]

Would you be able to provide a dump of when your service hits the issue?

[vrecluse]

Would you be able to provide a dump of when your service hits the issue?

My service is running in production environment, and this issue accurs in a short time, I'm running a background task that detecting threadpool queue length and auto call dotnet-stack when the problem happens. After a while, the queue length dropped to 0, so it's hard to intervent, but this problem shows up again and again.

[mangod9]

We will need some repro or dump to investigate why the threads might not be making progress.

[baal2000]

@vrecluse Thank you for submitting this issue.

We have observed the following symptom in both .net 8 and 9 where a containerized .net process gets to a state of very high CPU use: pinned at 100% in the VM and very high (but not pinned) for the .net process in question. This is always accompanied by the thread pool becoming quickly exhausted, slowly growing until the process recovers on its own or through a restart

Taking a trace or a process dump is rarely successful when the process is in this state, but when we get once most threads are in System.Private.CoreLib!System.Threading.LowLevelLifoSemaphore.WaitNative.

The container is executed in docker in Ubuntu 20.04 / 22.04.

We suspect that imperfect async implementation causes async threads to become blocked due to some external factors, which triggers this issue as a "fall of the cliff" scenario. Note that it is almost impossible to make a perfect async application: there are always some blocking calls for one reason or another whether the code is custom or in the framework. We are in fact losing hope we would ever be able to figure this out.

If this is not the same issue I will create a new. To determine that, could you answer a few questions:

• is this really .net 9 - only issue?
• how do you know it is "CPU_TIME" and not "SLEEP"?
• what is the overall CPU utilization when this happens?

Thank you!