v4.0.x: Reconciliation performance drop for concurrent workflows

[ramanNarasimhan77]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

#15079 flipped INFORMER_WRITE_BACK default from true to false. The false path calls time.Sleep(1 * time.Second) after every workflow update. With 32 default workers, this caps effective reconciliation throughput and causes workflows to remain in Pending far longer than on v3.7.x.

Submitting 101 workflows concurrently with parallelism=100:

v3.7.14: 886 reconciliations in 20s, all 101 workflows reach Running
v4.0.5: 161 reconciliations in 13s, only 66 workflows reach Running, 35 stuck in Pending
Expected: comparable throughput to v3.7.14.

Version(s)

v4.0.5, v4.0.6 (regression from v3.7.14).

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

https://github.com/argoproj/argo-workflows/blob/main/examples/coinflip.yaml


# Set parallelism=100 in workflow-controller-configmap, restart controller, then:
for i in $(seq 1 101); do argo submit workflow.yaml & done; wait
sleep 15
argo list --running | wc -l
# Expected: ~100. Actual on v4.0.x: ~60-70

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}


# Controller startup (v4.0.5, 32 workflow-workers)
time=2026-06-05T08:58:22.536Z level=INFO msg="Starting Workflow Controller" version=v4.0.5 defaultRequeueTime=10s
time=2026-06-05T08:58:22.536Z level=INFO msg="Current Worker Numbers" workflowWorkers=32 workflowTtlWorkers=4 podCleanup=4 cronWorkflowWorkers=8

# First batch processing starts at 08:58:24
time=2026-06-05T08:58:24.150Z level=INFO msg="Processing workflow" workflow=wf-coinflip-24h5g phase=Pending
time=2026-06-05T08:58:24.243Z level=INFO msg="Processing workflow" workflow=wf-coinflip-2d7hj phase=Running
time=2026-06-05T08:58:24.256Z level=INFO msg="Processing workflow" workflow=wf-coinflip-2dh5s phase=Pending
time=2026-06-05T08:58:24.258Z level=INFO msg="Processing workflow" workflow=wf-coinflip-2kzsl phase=""
time=2026-06-05T08:58:24.342Z level=INFO msg="Processing workflow" workflow=wf-coinflip-2lz4v phase=Running

# First updates complete ~1s later — the sleep gap is visible
time=2026-06-05T08:58:25.547Z level=INFO msg="Workflow update successful" workflow=wf-coinflip-2d7hj phase=Running
time=2026-06-05T08:58:25.553Z level=INFO msg="Workflow update successful" workflow=wf-coinflip-24h5g phase=Running
time=2026-06-05T08:58:25.555Z level=INFO msg="Workflow update successful" workflow=wf-coinflip-2lz4v phase=Running

# Parallelism correctly limits the 101st workflow
time=2026-06-05T08:58:31.450Z level=INFO msg="Workflow processing has been postponed due to max parallelism limit" workflow=wf-coinflip-zxbxz

# Updates per second (total 161 in 13s = ~12/s. v3.7.14 achieved 886 in 20s = ~44/s):
#   08:58:25 — 21    08:58:29 — 12    08:58:33 — 14
#   08:58:26 — 12    08:58:30 — 6     08:58:34 — 19
#   08:58:27 — 10    08:58:31 — 13    08:58:35 — 12
#   08:58:28 — 12    08:58:32 — 10    08:58:36 — 19

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

N/A — workflows are stuck in Pending phase. The bottleneck is controller reconciliation
throughput, not pod scheduling.

[coderabbitai]

🔗 Related PRs

#15090 - fix: workflow controller to detect stale workflows [merged]
#15124 - chore(ci): parallelize CLI builds in release workflow [merged]
#15158 - fix: test CheckWorkflowExistence check both mutex and sem [merged]
#15256 - feat: Add rate_limiter latency metrics [merged]
#15277 - fix: prevent int64 overflow in retry backoff calculation [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!