Race condition when set semaphore at both workflow and template level

[quantm0806]

Hi, I'm meeting race condition when using semaphore.
What I have setup

• A semaphore named total-pods=2
• A cron workflow acquiring semaphore at both workflow and template level. It looks like the image below. The format will be like:

"spec": {
    "schedule": "* * * * *",
    "concurrencyPolicy": "Replace",
    "workflowSpec": {
      "synchronization": {
        "semaphore": {
          "configMapKeyRef": {
            "name": "semaphore",
            "key": "total-pods"
          }
        }
      },
      "entrypoint": "main",
      "templates": [
        {
          "name": "main",
          "synchronization": {
            "semaphore": {
              "configMapKeyRef": {
                "name": "semaphore",
                "key": "total-pods"
              }
            }
          }

• A normal workflow also acquiring semaphore at both workflow and template level.

The cron workflow and normal workflow are attached here:
test_parallelism_cwf.json
test_semaphore_templates_with_priority_wf.json

How to reproduce

1. Submit the cron workflow and wait until the first cron workflow appears
2. When the first cron workflow runs, submit the normal workflow. The normal workflow doesn't run because the semaphore total-pods has been all used by the cron workflows (its workflow takes 1 and its template takes 1)
3. After one minute, when new cron workflow appears -> old cron workflow is replaced and releases semaphore -> new cron wf take 1 and the waiting normal workflow also take 1. Now, we have race condition. Two workflows acquire the semaphore but no template within them are running.

Note: I set priority: 1 in normal workflow at both workflow + template level but somehow it isn't prioritized to get 2 total-pods

Question

1. Why does my priority not work?
2. The docs say: You can set a [priority](https://argo-workflows.readthedocs.io/en/latest/parallelism/#priority) on Workflows. So does it mean that I cannot set a priority on template but workflow?

Thanks in advance.

[mg1986jp]

Setting a semaphore at both the workflow level and the template level can definitely cause issues. When you set it at the workflow level, the workflow itself acquires a slot before any templates run. Then each template tries to acquire its own slot too. So with total-pods=2, you could end up in a situation where two workflows each grab one workflow-level slot, and then neither can acquire a template-level slot because they're all used up.

If you're trying to limit the total number of running pods across all workflows, I'd pick one level and stick with it. Template-level semaphores give you more granular control (e.g., limiting specific expensive steps), while workflow-level limits how many workflows run concurrently.

The concurrencyPolicy: Replace on the CronWorkflow also adds complexity — it kills the existing workflow when a new one triggers, but the semaphore slot might not get released immediately if the workflow is still shutting down. That could look like a race condition.

What's the actual behavior you're seeing? Workflows stuck in pending, or something else?

[quantm0806]

Hello, the workflow stuck in pending with message of waiting for the total-pods lock.
However, after noticing race condition problem and I couldn't fix it, I chose to use concurrency control at only one level which is the workflow level. Thanks for your useful reply, i will close this discussion.