Skip to content

agent: improve log subscription lifecycle and event handling - #3259

Draft
thaJeztah wants to merge 11 commits into
moby:masterfrom
thaJeztah:better_watch
Draft

agent: improve log subscription lifecycle and event handling#3259
thaJeztah wants to merge 11 commits into
moby:masterfrom
thaJeztah:better_watch

Conversation

@thaJeztah

@thaJeztah thaJeztah commented Jul 26, 2026

Copy link
Copy Markdown
Member

agent: Agent.Publisher: scope variables

agent: worker.Subscribe: use waitgroup.Go

agent: worker.Subscribe: simplify subscription event loop

  • return early for closed event channels
  • use early "continue" to reduce nesting

agent: worker.Subscribe: use read-lock

Collecting the initial set of matching task managers only reads from
w.taskManagers; no shared state is modified. Use RLock instead of
Lock to allow concurrent readers while still protecting iteration over
the map.

agent: worker.Subscribe: avoid nil dereference

Avoid a panic if subscription.Options is nil; the function already
had guards in place for the "Follow" option, but lacked guards in
code before that.

While updating, also update the debug-logs to structured logs, and
include the Follow option.

agent: worker.Subscribe: wait for log streams to finish

The ControllerLogs contract requires Logs to return when its
context is cancelled, and taskManager.Logs passes the
subscription context through to the controller implementation.

Both the swarmd implementation and the dockerd implementation
honour this contract by propagating the context through their log
handling and cancellation paths.

Wait for the active log streams directly instead of starting a separate
goroutine and channel solely to make WaitGroup.Wait selectable.

Previously, Subscribe could return as soon as the context was
cancelled, closing the publisher before all active log streams had
finished handling cancellation. It could also leave behind the goroutine
waiting in WaitGroup.Wait, along with any Logs goroutines that had
not yet returned.

Waiting directly ensures Subscribe does not return until all log
goroutines it started have exited. If an implementation violates the
ControllerLogs contract and fails to return on cancellation, the
subscription now remains blocked instead of silently abandoning those
goroutines.

agent: worker.Subscribe: follow-mode: wait for log streams to finish

Use the same WaitGroup for log streams started while handling follow-mode
task events. This ensures Subscribe waits for all log streams it starts
before returning.

agent: worker.Subscribe:: reduce lock scope when starting log streams

Collect the initial set of matching task managers while holding the
worker's read lock, then release the lock before starting their log
streams.

This reduces the time the worker lock is held and avoids starting
goroutines while holding the lock.

agent: worker.Subscribe: register log watcher before taking task snapshot

Register the task event watcher before collecting the initial set of
matching task managers for follow-mode subscriptions.

This narrows the window in which newly created tasks could otherwise be
missed between taking the snapshot and starting the watcher.

Use CallbackWatchContext with a matcher to receive only matching task
events, allowing the event loop to rely on the watcher for filtering and
context cancellation.

agent: worker.Subscribe: start at most one log stream per task

Track task IDs for which a subscription has already started a log
stream.

A task may be present in the initial snapshot and also be delivered by
the watcher after it is registered. Use a shared helper for both paths
to ensure that only one log stream is started for each task.

- How to test it

- Description for the changelog

@codecov-commenter

codecov-commenter commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 14.74%. Comparing base (6e9e7b8) to head (f04f45c).
⚠️ Report is 46 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3259      +/-   ##
==========================================
+ Coverage   14.73%   14.74%   +0.01%     
==========================================
  Files         200      200              
  Lines       93077    93046      -31     
==========================================
+ Hits        13712    13721       +9     
+ Misses      78019    77977      -42     
- Partials     1346     1348       +2     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@thaJeztah

Copy link
Copy Markdown
Member Author

Oh, derp;

38.84 agent/worker.go:655:6: stdversion: sync.Go requires go1.25 or later (file is go1.24) (govet)
38.84 		wg.Go(func() {
38.84 		   ^

Will probably update it; it's cleaner, and we don't need older versions of go supported.

@thaJeztah
thaJeztah force-pushed the better_watch branch 3 times, most recently from 1f4521b to 57b9083 Compare July 28, 2026 16:40
thaJeztah added 11 commits July 28, 2026 20:56
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- return early for closed event channels
- use early "continue" to reduce nesting

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Collecting the initial set of matching task managers only reads from
w.taskManagers; no shared state is modified. Use RLock instead of
Lock to allow concurrent readers while still protecting iteration over
the map.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Avoid a panic if subscription.Options is nil; the function already
had guards in place for the "Follow" option, but lacked guards in
code before that.

While updating, also update the debug-logs to structured logs, and
include the Follow option.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The [ControllerLogs contract][1] requires `Logs` to return when its
context is cancelled, and [`taskManager.Logs`][2] passes the
subscription context through to the controller implementation.

Both the [swarmd implementation][3] and the [dockerd implementation][4]
honour this contract by propagating the context through their log
handling and cancellation paths.

Wait for the active log streams directly instead of starting a separate
goroutine and channel solely to make `WaitGroup.Wait` selectable.

Previously, `Subscribe` could return as soon as the context was
cancelled, closing the publisher before all active log streams had
finished handling cancellation. It could also leave behind the goroutine
waiting in `WaitGroup.Wait`, along with any `Logs` goroutines that had
not yet returned.

Waiting directly ensures `Subscribe` does not return until all log
goroutines it started have exited. If an implementation violates the
`ControllerLogs` contract and fails to return on cancellation, the
subscription now remains blocked instead of silently abandoning those
goroutines.

[1]: https://github.com/moby/swarmkit/blob/v2.1.2/agent/exec/controller.go#L47-L54
[2]: https://github.com/moby/swarmkit/blob/v2.1.2/agent/task.go#L65-L75
[3]: https://github.com/moby/swarmkit/blob/v2.1.2/swarmd/dockerexec/controller.go#L460-L537
[4]: https://github.com/moby/moby/blob/docker-v29.6.2/daemon/cluster/executor/container/controller.go#L505-L592

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use the same WaitGroup for log streams started while handling follow-mode
task events. This ensures Subscribe waits for all log streams it starts
before returning.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Collect the initial set of matching task managers while holding the
worker's read lock, then release the lock before starting their log
streams.

This reduces the time the worker lock is held and avoids starting
goroutines while holding the lock.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
…shot

Register the task event watcher before collecting the initial set of
matching task managers for follow-mode subscriptions.

This narrows the window in which newly created tasks could otherwise be
missed between taking the snapshot and starting the watcher.

Use CallbackWatchContext with a matcher to receive only matching task
events, allowing the event loop to rely on the watcher for filtering and
context cancellation.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Track task IDs for which a subscription has already started a log
stream.

A task may be present in the initial snapshot and also be delivered by
the watcher after it is registered. Use a shared helper for both paths
to ensure that only one log stream is started for each task.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants