Scheduler: Use a "scheduler" task for thread sleep #57544

kpamnany · 2025-02-26T22:46:00Z

A Julia thread runs Julia's scheduler in the context of the switching task. If no task is found to switch to, the thread will sleep while holding onto the (possibly completed) task, preventing the task from being garbage collected. This recent Discourse post illustrates precisely this problem.

A solution to this would be for an idle Julia thread to switch to a "scheduler" task, thereby freeing the old task.

This PR uses OncePerThread to create a "scheduler" task (that does nothing but run wait() in a loop) and switches to that task when the thread finds itself idle.

Other approaches considered and discarded in favor of this one: #57465 and #57543.

base/task.jl

kpamnany · 2025-03-05T15:14:28Z

This is currently blocked on what seems to be a bug in OncePerThread serialization/deserialization; found with @gbaraldi.

…simage (#57656) This is quite tricky to test unfortunately, but #57544 caught this and this fixes that --------- Co-authored-by: Jameson Nash <[email protected]>

…simage (#57656) This is quite tricky to test unfortunately, but #57544 caught this and this fixes that --------- Co-authored-by: Jameson Nash <[email protected]> (cherry picked from commit bf01638)

kpamnany · 2025-03-20T23:34:22Z

Unblocked... thanks @gbaraldi!

Now, some Channel tests and a Sockets test are failing. Looking into these failures.

vtjnash

I think those tests are specifically testing race conditions in the code, so you might need to adjust them slightly to account for the change in process_events ordering and count

base/task.jl

JamesWrigley · 2025-03-22T11:09:00Z

Is there any chance of getting this backported to 1.12? It's quite tricky to debug and would be nice if it was fixed in the next release.

This small group of tests is written with assumptions about when and how the libuv event loop is run. As this PR changes this behavior, the tests needed adjusting.

kpamnany · 2025-03-25T21:05:36Z

The channels tests are fixed, but I don't see a way to fix the Sockets test that's failing. Any ideas @gbaraldi or @vtjnash?

Previously, this test depended on scheduler behavior, which is slightly changed in this PR. Changed the test to connect to a non-routable IP address so that it no longer depends on task ordering.

kpamnany · 2025-03-26T17:01:25Z

Thanks @vtjnash for the idea on how to fix the Sockets test.

kpamnany · 2025-03-26T19:25:59Z

The channels tests that are failing on FreeBSD are a bit mystifying. How come Workqueue is empty on Linux but not on FreeBSD? Do we have an extra sticky task?

nsajko · 2025-04-14T18:53:43Z

base/task.jl

+    # We may have already switched tasks (via the scheduler task), so
+    # only switch if we haven't.
+    if !have_result
+        @assert task isa Task


Any specific reason this doesn't just typeassert? Just curious.

Suggested change

@assert task isa Task

task = task::Task

No specific reason. Seems like it should be a typeassert, actually.

A Julia thread runs Julia's scheduler in the context of the switching task. If no task is found to switch to, the thread will sleep while holding onto the (possibly completed) task, preventing the task from being garbage collected. This recent [Discourse post](https://discourse.julialang.org/t/weird-behaviour-of-gc-with-multithreaded-array-access/125433) illustrates precisely this problem. A solution to this would be for an idle Julia thread to switch to a "scheduler" task, thereby freeing the old task. This PR uses `OncePerThread` to create a "scheduler" task (that does nothing but run `wait()` in a loop) and switches to that task when the thread finds itself idle. Other approaches considered and discarded in favor of this one: #57465 and #57543. (cherry picked from commit 0d4d6d9)

A typeassert seems like better style. Given that `typeassert` is a builtin, why not put it to use. See: * JuliaLang#57544 (comment)

Follows up on this PR: * #57544 (comment)

Keno · 2025-06-16T21:35:16Z

This change interacts badly with ^C. Many interrupt exception will now get thrown to the scheduler task, which then dies and takes down the ability to ever schedule again in the future.

kpamnany · 2025-06-17T00:36:47Z

This change interacts badly with ^C. Many interrupt exception will now get thrown to the scheduler task, which then dies and takes down the ability to ever schedule again in the future.

Ugh.

^C behavior wasn't especially good before, but this is clearly worse. Suggestions?

Keno · 2025-06-17T00:39:57Z

I think we need to revert this for 1.12 and then work on implementing a proper behavior for 1.13 (#52291).

kpamnany · 2025-06-17T01:06:41Z

We could also simply never direct the interrupt at scheduler tasks?

Keno · 2025-06-17T01:09:23Z

That needs an extra feature to find the correct task to send it to then. This is too much for 1.12 at this point - we should reset to the old behavior for the release and we can work on something better for 1.13.

This reverts commit 640c4e1.

kpamnany mentioned this pull request Feb 26, 2025

Scheduler: Use the root task as a scheduler task #57465

Closed

vtjnash mentioned this pull request Feb 27, 2025

Scheduler: Use the root task as a scheduler task (alternative 1) #57543

Closed

vtjnash reviewed Feb 27, 2025

View reviewed changes

base/task.jl Show resolved Hide resolved

kpamnany force-pushed the kp-sched-task-alt2 branch 2 times, most recently from b345e87 to 4c08ecb Compare February 28, 2025 00:16

kpamnany commented Feb 28, 2025

View reviewed changes

base/task.jl Outdated Show resolved Hide resolved

kpamnany force-pushed the kp-sched-task-alt2 branch from 4c08ecb to aa69cb7 Compare February 28, 2025 15:32

vtjnash reviewed Feb 28, 2025

View reviewed changes

base/task.jl Outdated Show resolved Hide resolved

kpamnany marked this pull request as ready for review March 5, 2025 15:14

gbaraldi mentioned this pull request Mar 6, 2025

Fix OncePerThread issue when building a sysimage using the current sysimage #57656

Merged

kpamnany force-pushed the kp-sched-task-alt2 branch from f280243 to 684637a Compare March 20, 2025 18:56

vtjnash approved these changes Mar 21, 2025

View reviewed changes

vtjnash reviewed Mar 21, 2025

View reviewed changes

base/task.jl Outdated Show resolved Hide resolved

kpamnany added 3 commits March 24, 2025 19:15

Use a scheduler task for thread sleep

ba70cec

Rearrange the code to make things cleaner

8de1806

Drop extra process_events call

3bcde1b

kpamnany force-pushed the kp-sched-task-alt2 branch from 684637a to 3bcde1b Compare March 24, 2025 19:15

Fix channels tests

97d5a37

This small group of tests is written with assumptions about when and how the libuv event loop is run. As this PR changes this behavior, the tests needed adjusting.

Fix Sockets test

9c8c57e

Previously, this test depended on scheduler behavior, which is slightly changed in this PR. Changed the test to connect to a non-routable IP address so that it no longer depends on task ordering.

Remove bad tests for Workqueue contents

192f122

kpamnany merged commit 0d4d6d9 into master Mar 27, 2025
5 of 7 checks passed

kpamnany deleted the kp-sched-task-alt2 branch March 27, 2025 15:10

nsajko reviewed Apr 14, 2025

View reviewed changes

nsajko mentioned this pull request Apr 14, 2025

Ctrl-C on "Install package?" prompt now exits Julia #58105

Open

KristofferC added the backport 1.12 Change should be backported to release-1.12 label Apr 16, 2025

KristofferC removed the backport 1.12 Change should be backported to release-1.12 label Apr 25, 2025

This was referenced May 6, 2025

Thread-local current_task keeps garbage alive #40626

Closed

Memory not freed when using Multi-threading #52896

Closed

nsajko added a commit to nsajko/julia that referenced this pull request May 31, 2025

wait(): just typeassert instead of asserting isa

cfc7ed2

A typeassert seems like better style. Given that `typeassert` is a builtin, why not put it to use. See: * JuliaLang#57544 (comment)

nsajko mentioned this pull request May 31, 2025

simplify wait() #58595

Merged

nsajko added a commit to nsajko/julia that referenced this pull request May 31, 2025

wait(): just typeassert instead of asserting isa

9f0b963

A typeassert seems like better style. Given that `typeassert` is a builtin, why not put it to use. See: * JuliaLang#57544 (comment)

nsajko added a commit to nsajko/julia that referenced this pull request May 31, 2025

wait(): just typeassert instead of asserting isa

3846aac

A typeassert seems like better style. Given that `typeassert` is a builtin, why not put it to use. See: * JuliaLang#57544 (comment)

oscardssmith pushed a commit that referenced this pull request Jun 1, 2025

simplify wait() (#58595)

905a44b

Follows up on this PR: * #57544 (comment)

Keno mentioned this pull request Jun 16, 2025

CTRL+C is even worse on 1.13 #58689

Open

kpamnany added a commit that referenced this pull request Jun 17, 2025

Revert "Scheduler: Use a "scheduler" task for thread sleep (#57544)"

a7d630a

This reverts commit 640c4e1.

kpamnany mentioned this pull request Jun 17, 2025

[release-1.12] Revert "Scheduler: Use a "scheduler" task for thread sleep (#57544)" #58764

Closed

fonsp mentioned this pull request Jun 30, 2025

CI Testing for Ctrl+C interrupt #58849

Open

KristofferC pushed a commit that referenced this pull request Jun 30, 2025

Revert "Scheduler: Use a "scheduler" task for thread sleep (#57544)"

b608652

This reverts commit 640c4e1.

Uh oh!

Scheduler: Use a "scheduler" task for thread sleep #57544

Scheduler: Use a "scheduler" task for thread sleep #57544

Uh oh!

Conversation

kpamnany commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kpamnany commented Mar 5, 2025

Uh oh!

kpamnany commented Mar 20, 2025

Uh oh!

vtjnash left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JamesWrigley commented Mar 22, 2025

Uh oh!

kpamnany commented Mar 25, 2025

Uh oh!

kpamnany commented Mar 26, 2025

Uh oh!

kpamnany commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nsajko Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kpamnany May 6, 2025

Choose a reason for hiding this comment

Uh oh!

Keno commented Jun 16, 2025

Uh oh!

kpamnany commented Jun 17, 2025

Uh oh!

Keno commented Jun 17, 2025

Uh oh!

kpamnany commented Jun 17, 2025

Uh oh!

Keno commented Jun 17, 2025

Uh oh!

Uh oh!

kpamnany commented Feb 26, 2025 •

edited

Loading

kpamnany commented Mar 26, 2025 •

edited

Loading

nsajko Apr 14, 2025 •

edited

Loading