Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stackSize override is not correct value format #60762

Open
1 task done
clintwar opened this issue Mar 5, 2025 · 8 comments · May be fixed by #60811
Open
1 task done

stackSize override is not correct value format #60762

clintwar opened this issue Mar 5, 2025 · 8 comments · May be fixed by #60811
Labels
area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions

Comments

@clintwar
Copy link

clintwar commented Mar 5, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Describe the bug

When aspnetcore for in process hosting overrides the default stack size for new threads on the threadpool (https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/advanced?view=aspnetcore-9.0) the value is incorrectly defaulted.

m_strStackSize = find_element(handlerSettings, CS_ASPNETCORE_HANDLER_STACK_SIZE).value_or(L"1048576");

3be11f6

The internals of coreclr is looking for the value as base16. The default value of 1048576 is treating as base16. This is not 1 MB. This is 17 MB.

If you take a memory dump of an inprocess hosted .net core 8 app and view !address -f:MEM_RESERVE,Stack you will notice reserved memory for stack around this 0x1048576 size. In 32bit world this is a bad situation as you will run out of memory if your process needs to create many threads.

In the learning document it indicates how to set it to 2MB however this actually would set it to 34~ MB.

Expected Behavior

Actually set stack to a reasonable size similar to what .net core does by default 1.5 MB for 32bit app and 4MB for x64.

Steps To Reproduce

If you take a memory dump of an inprocess hosted .net core 8 app and view !address -f:MEM_RESERVE,Stack you will notice reserved memory for stack around this 0x1048576 size. In 32bit world this is a bad situation as you will run out of memory if your process needs to create many threads.

Exceptions (if any)

No response

.NET Version

No response

Anything else?

No response

@dotnet-issue-labeler dotnet-issue-labeler bot added the area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions label Mar 5, 2025
@clintwar
Copy link
Author

clintwar commented Mar 5, 2025

Output from memory dump

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Stack 684 bf5b0000 ( 2.990 GB) 76.24% 74.75%
2263 29562000 ( 661.383 MB) 16.47% 16.15%
Image 3094 f46d000 ( 244.426 MB) 6.09% 5.97%
Free 548 500a000 ( 80.039 MB) 1.95%
Heap 126 2cb1000 ( 44.691 MB) 1.11% 1.09%
TEB 228 372000 ( 3.445 MB) 0.09% 0.08%
Other 29 41000 ( 260.000 kB) 0.01% 0.01%
PEB 1 3000 ( 12.000 kB) 0.00% 0.00%

!address -f:MEM_RESERVE,Stack
....
313d0000 32419000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~43; 1ff4.22e4]
32420000 33465000 1045000 MEM_PRIVATE MEM_RESERVE Stack [~44; 1ff4.2240]
33470000 344bc000 104c000 MEM_PRIVATE MEM_RESERVE Stack [~45; 1ff4.2170]
344c0000 35505000 1045000 MEM_PRIVATE MEM_RESERVE Stack [~46; 1ff4.193c]
35510000 36558000 1048000 MEM_PRIVATE MEM_RESERVE Stack [~47; 1ff4.1a68]
37340000 3838b000 104b000 MEM_PRIVATE MEM_RESERVE Stack [~48; 1ff4.1cbc]
38390000 393d7000 1047000 MEM_PRIVATE MEM_RESERVE Stack [~49; 1ff4.26cc]
393e0000 3a428000 1048000 MEM_PRIVATE MEM_RESERVE Stack [~50; 1ff4.2118]
3a430000 3b479000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~52; 1ff4.18cc]
3b480000 3c4c7000 1047000 MEM_PRIVATE MEM_RESERVE Stack [~53; 1ff4.2480]
3c4d0000 3d519000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~54; 1ff4.1854]
3d520000 3e569000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~55; 1ff4.1b98]
3e570000 3f5b1000 1041000 MEM_PRIVATE MEM_RESERVE Stack [~56; 1ff4.c8c]
3f5c0000 40607000 1047000 MEM_PRIVATE MEM_RESERVE Stack [~57; 1ff4.13c4]
40610000 41655000 1045000 MEM_PRIVATE MEM_RESERVE Stack [~58; 1ff4.21d0]
41660000 426a4000 1044000 MEM_PRIVATE MEM_RESERVE Stack [~59; 1ff4.1e3c]
426b0000 436f7000 1047000 MEM_PRIVATE MEM_RESERVE Stack [~60; 1ff4.144c]
43700000 44746000 1046000 MEM_PRIVATE MEM_RESERVE Stack [~61; 1ff4.202c]
44750000 45795000 1045000 MEM_PRIVATE MEM_RESERVE Stack [~62; 1ff4.23b0]
457a0000 467e9000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~63; 1ff4.2670]

....

This particular dump did result in oom in a bad place, starting new thread.

0:000> !pe 176c8030
Exception object: 176c8030
Exception type: System.OutOfMemoryException
Message:
InnerException:
StackTrace (generated):
SP IP Function
00000000 00000001 System_Private_CoreLib!System.Threading.Thread.StartInternal(System.Threading.ThreadHandle, Int32, Int32, Char*)+0x2
FE32EA38 D4925A2B System_Private_CoreLib!System.Threading.Thread.StartCore()+0xb3
FE32EA8C DCE05670 System_Private_CoreLib!System.Threading.PortableThreadPool+WorkerThread.CreateWorkerThread()+0x50
.....

0:000> !threadpool
Using the Portable thread pool.

CPU utilization: 9%
Workers Total: 178
Workers Running: 178
Workers Idle: 0
Worker Min Limit: 100
Worker Max Limit: 1023

@adityamandaleeka
Copy link
Member

Yup, we should fix this. This is the relevant code in runtime where you can see that it expects a hex value: https://github.com/dotnet/runtime/blob/3d275542626ac6420269168ecbebe2bcb8e6ae54/src/coreclr/vm/threads.cpp#L2051C8-L2051C42

Good catch!

@adityamandaleeka
Copy link
Member

Are you interested in opening a PR for this @clintwar?

@adityamandaleeka adityamandaleeka added this to the .NET 10 Planning milestone Mar 5, 2025
@clintwar
Copy link
Author

clintwar commented Mar 5, 2025

I can if no one else wants to :)

@adityamandaleeka
Copy link
Member

This should be a relatively straightforward fix (minor code change, and a doc update here: https://github.com/dotnet/AspNetCore.Docs/blob/main/aspnetcore/host-and-deploy/iis/advanced.md), so if you're interested, we'd love to have you take it on. Let me know if you need any pointers or guidance along the way. Otherwise, if you'd prefer not to, I can take care of it--just let me know what works best for you.

@BrennanConroy
Copy link
Member

BrennanConroy commented Mar 6, 2025

BrennanConroy added a commit to dotnet/runtime that referenced this issue Mar 6, 2025
[ASP.NET Core incorrectly assumed the `DefaultStackSize` value was parsed as an integer](dotnet/aspnetcore#60762) (most values are? But that's not a discussion for this PR). But the value is parsed as a base16 number. Just updating the comment here to explicitly mention that.
@apurvghai
Copy link

Here's a screen shot of we finally investigated, even though it's clear from Memory dump analysis

Image

@clintwar
Copy link
Author

clintwar commented Mar 7, 2025

PR for fix is here
PR for doc is Documentation Provides Incorrect Values For Default Stack Size

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants