[DO_NOT_MERGE] Dispatch activity cancellation to worker using Nexus by rkannan82 · Pull Request #9233 · temporalio/temporal

rkannan82 · 2026-02-05T21:20:53Z

What changed?

Sends a Nexus task to the control queue on which the worker is listening for cancellation.

Main changes:

cancel_activity.go: Handles dispatching activity cancellation requests to workers via Nexus control queue.
Called by transfer_queue_active_task_executor.go.

Controlled by dynamic config: EnableActivityCancellationNexusTask

Why?

To support activity cancellation without activity heartbeat.

Overall flow:

[https://github.com/[DO_NOT_MERGE] Store worker attributes needed by server to propagate nexus tasks to worker #9231] Store worker instance key in ActivityInfo as part of RecordActivityTaskStarted call.
[[DO_NOT_MERGE] Add new Nexus outbound task for cancelling activities #9232] When user cancels a workflow, create 1 or more transfer tasks. Group all activities belonging to a worker into a transfer task (for efficiency).
[This PR] Lookup the Nexus task queue for each worker, and send a Nexus operation for each transfer task.
[SDK] Worker will receive this cancel task and cancel the running activities.

How did you test it?

Potential risks

No. The task dispatch is disabled by default.

bergundy

Can you confirm whether you're covering cancel requests and pause requests?
Do we also care about canceling activities that we know timed out or are we letting the worker take care of that?
When will you be adding support for standalone activities too?

bergundy · 2026-02-12T20:08:40Z

service/history/cancel_activity.go

+		return nil
+	}
+
+	return t.dispatchCancelTaskToWorker(ctx, task.NamespaceID, task.WorkerControlTaskQueue, taskTokens)


You definitely want to release the lock before making this RPC.

Why store the worker control task queue on the task? That seems redundant if you're already reading from mutable state, which IMHO should serve as the source of truth.

bergundy · 2026-02-12T20:11:52Z

service/history/cancel_activity.go

+				Operation: nexus.CancelActivitiesOperation,
+				Payload: &commonpb.Payload{
+					Metadata: map[string][]byte{
+						"encoding": []byte("proto"),


Use the SDK's data converter here instead of inventing your own proto encoding scheme please.

bergundy · 2026-02-12T20:12:40Z

common/nexus/constants.go

+const (
+	// WorkerControlService is the Nexus service name for worker control operations.
+	WorkerControlService = "temporal.api.worker.v1.WorkerService"
+	// CancelActivitiesOperation is the operation to cancel running activities.
+	CancelActivitiesOperation = "cancel-activities"
+)


These aren't nexus properties IMHO. It's part of the worker service, not generic for Nexus.

bergundy · 2026-02-12T20:13:15Z

service/history/cancel_activity.go

+
+// buildActivityTaskTokens builds task tokens for activities that need cancellation.
+// Returns the serialized task tokens and any error.
+func buildActivityTaskTokens(


Share this function everywhere where you construct task tokens instead of reimplementing this.

bergundy · 2026-02-12T20:14:01Z

service/history/cancel_activity.go

+		return err
+	}
+
+	if resp.GetRequestTimeout() != nil {


What about async outcomes and operation failures?

bergundy · 2026-02-12T20:16:18Z

service/history/cancel_activity.go

+	"google.golang.org/protobuf/proto"
+)
+
+func (t *transferQueueActiveTaskExecutor) processCancelActivityTask(


I would put this on the outbound queue since it sync matches into workers and would have different latency characteristics than typical internal tasks.

That will an exception re. how we divided queues IMO.

Not sure I follow the concern re. fail to sync match, if it fails the task can retry using a user error, which can be handled in the same way as namespace rate limit error today. I think this task is best effort and will be discarded after 5mins.

Maybe I missed something on the nexus sync match dispatch flow?

IMHO this is still outbound because it blocks on a response from a worker. We would need to allow multiple seconds of timeout for this round trip.

ah...we wait for the response! that's the part I missed. I thought DispatchNexusTask only do the dispatch part.

yeah agree if we need to allow multiple seconds for a task, we can't use transfer queue. cc @rkannan82 sorry I missed this when reviewing the design and pointed you in the wrong direction.

bergundy · 2026-02-12T20:18:04Z

service/history/cancel_activity.go

+		return nil
+	}
+
+	ctx, cancel := context.WithTimeout(ctx, taskTimeout)


What's taskTimeout? You should be deliberate about setting a reasonable timeout for the whole dispatch to the worker. That's different from dispatching tasks internally within the cluster.

- Add activityCommandTaskDispatcher to dispatch tasks to workers - Use typed Nexus service definition for worker communication - Add proper error handling for DispatchNexusTask response - Release workflow lock before making RPC call - Add functional test for dispatch flow

rkannan82 requested review from a team as code owners February 5, 2026 21:20

This was referenced Feb 5, 2026

[DO_NOT_MERGE] Add new Nexus outbound task for cancelling activities #9232

Open

[DO_NOT_MERGE] Store worker attributes needed by server to propagate nexus tasks to worker #9231

Open

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from a91654f to 6a655f7 Compare February 5, 2026 22:02

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from d402868 to 54b6d1a Compare February 6, 2026 00:36

rkannan82 changed the title ~~Dispatch CancelActivityNexusTask to worker via Nexus control queue~~ Dispatch CancelActivityNexusTask to worker using Nexus operation Feb 6, 2026

rkannan82 changed the title ~~Dispatch CancelActivityNexusTask to worker using Nexus operation~~ Dispatch CancelActivityNexusTask to worker Feb 6, 2026

rkannan82 mentioned this pull request Feb 6, 2026

Add Nexus request and response payloads to cancel activities temporalio/api#708

Open

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch 2 times, most recently from 13d8513 to eb79c81 Compare February 11, 2026 19:39

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from d103589 to 417606c Compare February 11, 2026 20:10

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from eb79c81 to d1f2abf Compare February 11, 2026 20:11

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 417606c to 6246c4e Compare February 11, 2026 21:04

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from d1f2abf to 7489808 Compare February 11, 2026 21:04

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 6246c4e to fec3c41 Compare February 11, 2026 21:08

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from 37a51d2 to 11b1049 Compare February 11, 2026 21:08

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from fec3c41 to 1dd975d Compare February 11, 2026 21:22

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch 2 times, most recently from 45ac313 to e96dbe8 Compare February 11, 2026 21:30

rkannan82 requested a review from yycptt February 11, 2026 22:25

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 11c74b6 to 2cb3108 Compare February 12, 2026 00:43

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch 3 times, most recently from 72fe398 to 37c2a1c Compare February 12, 2026 07:09

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from ac7c57e to 89d2cf5 Compare February 12, 2026 07:11

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch 2 times, most recently from 5a9c85e to d1572d7 Compare February 12, 2026 07:18

bergundy reviewed Feb 12, 2026

View reviewed changes

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 89d2cf5 to 79e4e3e Compare February 15, 2026 03:33

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from 9ee5998 to 423bb54 Compare February 15, 2026 03:33

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 79e4e3e to 7910d4e Compare February 15, 2026 03:43

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch 3 times, most recently from 5b2a3b9 to a35281a Compare February 15, 2026 23:41

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 5b615d9 to 9231795 Compare February 15, 2026 23:44

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from a35281a to 3d1b6bb Compare February 15, 2026 23:45

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 9231795 to bc58c4f Compare February 16, 2026 02:44

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from 3d1b6bb to c868ae3 Compare February 16, 2026 02:46

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from bc58c4f to 3eb028c Compare February 16, 2026 02:48

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from c868ae3 to 4d2f4d5 Compare February 16, 2026 02:49

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 3eb028c to 92b1a81 Compare February 18, 2026 02:32

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from 4d2f4d5 to 597d390 Compare February 18, 2026 02:32

rkannan82 changed the title ~~Dispatch CancelActivityNexusTask to worker~~ [DO_NOT_MERGE] Dispatch CancelActivityNexusTask to worker Feb 18, 2026

rkannan82 changed the title ~~[DO_NOT_MERGE] Dispatch CancelActivityNexusTask to worker~~ [DO_NOT_MERGE] Dispatch activity cancellation to worker using Nexus Feb 18, 2026

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 92b1a81 to b775ca6 Compare February 18, 2026 03:34

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch 2 times, most recently from 34a1f8a to d8f33bf Compare February 18, 2026 03:44

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from b775ca6 to d6af322 Compare February 18, 2026 03:44

rkannan82 force-pushed the kannan/activity-cancel/dispatch-logic branch from d8f33bf to 3327132 Compare February 18, 2026 05:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[DO_NOT_MERGE] Dispatch activity cancellation to worker using Nexus#9233

[DO_NOT_MERGE] Dispatch activity cancellation to worker using Nexus#9233
rkannan82 wants to merge 1 commit intokannan/activity-cancel/task-definitionfrom
kannan/activity-cancel/dispatch-logic

rkannan82 commented Feb 5, 2026 •

edited

Loading

Uh oh!

bergundy left a comment

Uh oh!

bergundy Feb 12, 2026

Uh oh!

bergundy Feb 12, 2026

Uh oh!

bergundy Feb 12, 2026

Uh oh!

bergundy Feb 12, 2026

Uh oh!

bergundy Feb 12, 2026

Uh oh!

bergundy Feb 12, 2026

Uh oh!

bergundy Feb 12, 2026

Uh oh!

yycptt Feb 13, 2026

Uh oh!

bergundy Feb 13, 2026

Uh oh!

yycptt Feb 14, 2026

Uh oh!

bergundy Feb 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

rkannan82 commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Why?

How did you test it?

Potential risks

Uh oh!

bergundy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rkannan82 commented Feb 5, 2026 •

edited

Loading