roachtest: tpccbench/nodes=9/cpu=4/multi-region failed

[cockroach-teamcity]

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 3dc59b8f9bbac2f0b97eb530eba8bd588d3bc289:

Failed with:

(monitor.go:267).Wait: monitor failure: context canceled
unexpected node event: n6: cockroach process for system interface died (exit code 134)
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0001	34.26.87.220	10.142.2.4
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0002	34.73.239.58	10.142.2.0
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0003	34.26.177.200	10.142.2.5
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0004	35.227.36.92	10.142.2.1
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0005	34.158.247.146	10.138.0.76
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0006	8.229.23.24	10.138.0.61
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0007	34.105.60.96	10.138.0.62
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0008	8.231.225.48	10.138.0.89
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0009	34.39.94.163	10.154.0.73
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0010	34.89.112.23	10.154.0.72
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0011	34.105.143.177	10.154.0.71
teamcity-21340200-1778135524-242-n12cpu4hm-geo-0012	35.246.21.62	10.154.0.70

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-63716

[cpj2195]

/investigate

[github-actions]

Investigation: tpccbench/nodes=9/cpu=4/multi-region

Investigated failure: Issue body
Failure SHA: 3dc59b8f9bbac2f0b97eb530eba8bd588d3bc289
Confidence: high

What This Test Does

tpccbench/nodes=9/cpu=4/multi-region runs a TPC-C benchmark across a 9-node multi-region cluster (3 regions × 3 nodes, with 3 additional workload nodes) to measure throughput. It restores a TPC-C fixture, then runs the workload while monitoring for node crashes.

Where the Failure Occurs

Node n6 (CockroachDB nodeID=5, StoreID=5) crashed with exit code 134 (SIGABRT) approximately 26 minutes into the test due to an unrecovered panic in the MMA integration layer:

panic: no replica found for removal target=8 post-filter=[] pre-filter=r1020:/Table/114/1/27{83/3/-711/3-93/10/-690/7}
  [(n2,s2):8, (n9,s9):5, (n5,s5):3, next=9, gen=155, sticky=1778177949.708501831,0]

The panic occurs at mma_conversion.go:96 in convertReplicaChangeToMMA, called from allocator_sync.go:195 (NonMMAPreChangeReplicas), called from replicate_queue.go:1202 (changeReplicas).

Analysis

Likely cause: Stale descriptor/changes mismatch in the replicate queue (high confidence)

The panic is caused by a mismatch between the changes (planned replication operations) and the desc (range descriptor) passed to convertReplicaChangeToMMA. The changes include a REMOVE_VOTER targeting StoreID=8, but the current range descriptor no longer contains a replica on StoreID=8.

Root cause mechanism:

1. The replicate queue's process() function fetches desc once at replicate_queue.go:734 and reuses it across retry iterations.

2. PlanOneChange at line 974 plans a rebalance (REMOVE_VOTER s8, ADD_VOTER s2) using this potentially stale descriptor.

3. applyChange at line 917 fetches a fresh descriptor via replica.Desc() and passes it alongside the stale changes to changeReplicas.

4. changeReplicas passes the fresh descriptor + stale changes to NonMMAPreChangeReplicas, which calls convertReplicaChangeToMMA. The conversion tries to find StoreID=8 in the fresh descriptor, fails, and panics.

Timeline of r1020 on node 5 (goroutine 88995):

Time	Event
18:10:08.365	Goroutine 88995 plans rebalance s8→s2 for r1020 (desc gen=149, replicas: s8, s9, s5)
18:10:08.365	First NonMMAPreChangeReplicas call succeeds (desc still has s8)
18:10:08.365	changeReplicasImpl starts: adds (n2,s2):7LEARNER
18:10:10.192	Learner (n2,s2):7 removed (snapshot failure — remote raft group deleted)
18:10:11.330	Retry: adds (n2,s2):8LEARNER
18:10:27–29	Goroutine 90012 completes voter swap: (n8,s8):1→removed, (n2,s2):8→VOTER. Final desc gen=155: [(n2,s2):8, (n9,s9):5, (n5,s5):3]
~18:10:30+	Retry loop in process() calls processOneChange again with stale desc (gen=149, has s8)
~18:10:30+	PlanOneChange plans removal of s8 (from stale desc)
~18:10:30+	applyChange calls replica.Desc() → gets gen=155 (no s8)
~18:10:30+	changeReplicas → NonMMAPreChangeReplicas → convertReplicaChangeToMMA panics: s8 not in desc

The fix: The panic at mma_conversion.go:96 should be converted to an error return rather than crashing the node. The descriptor/changes mismatch is a transient condition — changeReplicasImpl would itself detect the stale descriptor and fail gracefully, but the convertReplicaChangeToMMA panic fires first. This is exactly the class of issue tracked in #167723 (MMA panic audit).

Additionally, the stale desc usage across retries in process() at line 734 could be addressed by re-fetching the descriptor on each retry iteration, though the more immediate fix is making the MMA conversion defensive.

Related Issues and PRs

• #167723 — "mmaprototype: audit panics/fatals" — directly tracks this class of MMA panics. The issue catalogues several panics/fatals in MMA code that can fire in production and need to be converted to error returns. The panic at mma_conversion.go:96 is in the same category but is not yet listed in that issue.
• #158005 and #156636 — prior tpccbench/multi-region failures on release-25.4, but with different failure mode (workload exit status 1, not node crash). Not the same bug.
• ff8481acffd — "mmaintegration: some assertion gardening" — converted some assertions in the same file to AssertionFailedf, but did not address the panic at line 96.
• 1725d0c5230 — original commit introducing convertReplicaChangeToMMA (Aug 2025).

Timeline

The convertReplicaChangeToMMA function was introduced in commit 1725d0c5230 (Aug 2025). The panic at line 96 has existed since the function was first written. The vulnerability to descriptor staleness was latent; it manifests when the replicate queue's retry loop processes a range whose descriptor changed between retries due to concurrent replication changes — an uncommon but realistic scenario in a multi-region cluster under load.

Detailed evidence and investigation notes

Log Excerpts

Panic stack trace (from logs/6.unredacted/cockroach-stderr.log):

panic: no replica found for removal target=8 post-filter=[] pre-filter=r1020:/Table/114/1/27{83/3/-711/3-93/10/-690/7} [(n2,s2):8, (n9,s9):5, (n5,s5):3, next=9, gen=155, sticky=1778177949.708501831,0]

goroutine 88995 gp=0x18d5ddf93680 m=11 mp=0x18d5dac46008 [running, "n":"5", "replicate":""]:
panic({0x6d176a0?, 0x18d5f0716390?})
	GOROOT/src/runtime/panic.go:879 +0x16f
convertReplicaChangeToMMA(...)
	pkg/kv/kvserver/mmaintegration/mma_conversion.go:96
(*AllocatorSync).NonMMAPreChangeReplicas(...)
	pkg/kv/kvserver/mmaintegration/allocator_sync.go:195
(*replicateQueue).changeReplicas(...)
	pkg/kv/kvserver/replicate_queue.go:1202
(*replicateQueue).applyChange(...)
	pkg/kv/kvserver/replicate_queue.go:913
(*replicateQueue).processOneChange(...)
	pkg/kv/kvserver/replicate_queue.go:1032
...

Concurrent replication activity on r1020 (from cockroach-kv-distribution.log):

18:10:08.365239 88995 — rebalancing voter n8,s8 to n2,s2 [planned]
18:10:08.365533 88995 — add [(n2,s2):7LEARNER], desc gen=149 [(n8,s8):1, (n9,s9):5, (n5,s5):3]
18:10:10.192299 88134 — remove (n2,s2):7LEARNER [snapshot failure]
18:10:11.330545 88948 — add (n2,s2):8LEARNER
18:10:27.733840 90012 — ENTER_JOINT: (n2,s2):8 VOTER_INCOMING, (n8,s8):1 VOTER_DEMOTING_LEARNER
18:10:28.466626 90012 — LEAVE_JOINT: (n8,s8):1LEARNER
18:10:29.199370 90012 — remove (n8,s8):1LEARNER → final: [(n2,s2):8, (n9,s9):5, (n5,s5):3] gen=155

Node 6 dmesg shows hung tasks (not OOM):

[Thu May  7 18:14:23 2026] INFO: task cockroach:cs_clock:3441 blocked for more than 123 seconds.
[Thu May  7 18:14:23 2026] INFO: task jemalloc_bg_thd:3375 blocked for more than 123 seconds.

These hung task warnings are a consequence of the panic — the process was in the middle of crashing (goroutine teardown after panic).

Code References

The panicking code (mma_conversion.go:91-99):

filteredSet := replicaSet.Filter(func(r roachpb.ReplicaDescriptor) bool {
    return r.StoreID == chg.Target.StoreID
})
replDescriptors := filteredSet.Descriptors()
if len(replDescriptors) != 1 {
    panic(fmt.Sprintf(
        "no replica found for removal target=%v post-filter=%v pre-filter=%v",
        chg.Target.StoreID, replDescriptors, desc))
}

The stale desc problem (replicate_queue.go:734-738):

desc := repl.Desc()  // fetched ONCE
for r := retry.StartWithCtx(ctx, retryOpts); r.Next(); {
    requeue, err := rq.processOneChangeWithTracing(ctx, repl, desc, &conf, priorityAtEnqueue)
    // retry loop reuses stale desc...

The fresh desc / stale changes mismatch (replicate_queue.go:912-917):

case plan.AllocationChangeReplicasOp:
    err = rq.changeReplicas(
        ctx,
        replica,
        op.Chgs,         // planned against stale desc
        replica.Desc(),   // FRESH desc — may not match op.Chgs

Things Ruled Out

• OOM kill: No OOM events in dmesg. Exit code 134 = SIGABRT (from Go panic), not 137 (SIGKILL from OOM).
• Infrastructure/VM issue: The crash is deterministically explained by the code path; no evidence of hardware or networking problems causing the panic.
• Same failure mode as roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #158005/roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #156636: Those were workload failures (exit status 1), not node crashes. Different root cause.
• debug.panic_on_failed_assertions: While this setting was enabled, it's irrelevant — the crash is from a raw panic() call, not from errors.AssertionFailedf.

Other Failure Occurrences

This is the first occurrence in this issue. Prior tpccbench/multi-region issues (#158005, #156636) had different failure signatures (workload exit status 1 vs. node crash with SIGABRT).

Tooling Feedback

No tooling limitations encountered during this investigation. All needed artifacts were accessible.

---

Workflow run

Was this investigation helpful? Leave a 👍 or 👎 on this comment.