urunc has no `exec` OCI-runtime subcommand, causing Argo Workflows (and any exec-dependent controller) to hang forever

[Anand-240]

What happened

When you run an Argo Workflow step with runtimeClassName: urunc and the main container is a urunc unikernel image, the workflow never finishes. The unikernel container can fail, or even succeed and terminate cleanly, and it doesn't matter: the workflow stays stuck in Running forever. No timeout, no error, nothing.

The cause is that urunc's CLI has no exec subcommand, and Argo's default executor needs to exec into a sidecar container to know a step is done.

Root cause

Argo's default (emissary) executor runs each step as a pod with three containers: init, wait, and main. Once main exits, the workflow-controller tells the wait sidecar to wrap up by running:

kubectl exec <pod> -c wait -- /var/run/argo/argoexec kill 15 1

That call goes through kubelet, then CRI's ExecSync, then the containerd shim, then <runtime> exec. RuntimeClass is set at the pod level, not per container, so once one container in the pod uses runtimeClassName: urunc, every container in that pod, including the plain-Linux wait sidecar that has nothing to do with unikernels, has its exec calls routed through urunc's shim too.

I checked main and it's not implemented:

$ grep -n Commands cmd/urunc/main.go
111:  Commands: []*cli.Command{
112:    createCommand, deleteCommand, killCommand, runCommand, psCommand, startCommand,

No execCommand. So the exec call just fails, and containerd/kubelet report it as a generic internal error rather than something callers can treat as "not supported."

Reproduction

Setup: kind v0.32.0, Kubernetes v1.36.1, containerd 2.3.1, urunc 0.7.0-96f992d (installed with the official urunc-deploy DaemonSet, RBAC, and RuntimeClass manifests), Argo Workflows v3.6.5.

Test 1: plain container, no unikernel annotations, runtimeClassName: urunc

This works fine. Argo's sidecar model and shared emptyDir volumes are fine under urunc as long as nothing needs to boot a VM.

$ kubectl -n argo get wf urunc-argo-plain-pcnlx
NAME                     STATUS      AGE
urunc-argo-plain-pcnlx   Succeeded   32s

Test 2: real unikernel image (nginx-qemu-unikraft-initrd) as main, runtimeClassName: urunc

main fails here because this test environment has no KVM, which is expected:

main:
  State:          Terminated
    Reason:       StartError
    Message:      failed to create containerd task: ... could not get host device
                  /dev/kvm: no such file or directory

But the workflow never ends. wait keeps running, the pod sits at 1/2, and workflow-controller keeps retrying cleanup and failing:

level=info msg="signaled container" container=wait error="Internal error occurred: error executing
  command in container: failed to exec in container: failed to start exec \"...\":
  OCI runtime exec failed: urunc did not terminate successfully: exit status 1"
level=info msg="queueing pod for cleanup after" action=killContainers after=30s ...

This repeats on a loop (10s, 20s, 30s then SIGKILL) with no change. I left it running and polled for 4+ minutes, no progress. It doesn't resolve on its own.

This matches #135

Issue #135 ("argo workflow never ends with kubernetes, urunc and unikraft") describes the exact same thing on a real, KVM-enabled cluster where the unikernel actually boots. Their journalctl logs show the actual urunc error behind it:

Mar 14 08:38:39 workernode5 urunc[1804235]: {"level":"error","msg":"No help topic for 'exec'\n", ...}
Mar 14 08:39:09 workernode5 urunc[1804248]: {"level":"error","msg":"No help topic for 'exec'\n", ...}

repeated for hours. Same 1/2 NotReady pod, same permanently-running workflow, even though in their case the unikernel finished cleanly (Terminated: Reason: Completed, Exit Code: 0). That's a strong signal this is the missing exec verb, not a KVM or boot problem.

Suggested fix

Add urunc exec, or at least a stub that returns a clean "not supported" error instead of the CLI's usage-help failure. Right now containerd/kubelet can't tell the difference between "urunc doesn't support this" and "urunc is broken," and Argo just retries forever instead of failing the step.

Possible workaround to check: Argo's k8sapi executor mode doesn't rely on exec-based signaling, it polls container status through the Kubernetes API instead, so it might not hit this at all. I haven't verified this myself since I don't have a KVM-capable cluster to test against.

Environment

Reproduced on kind + Docker Desktop on macOS. No /dev/kvm in this setup, so the KVM failure in Test 2 is expected and already documented elsewhere, not a new finding. The actual bug here, the exec hang, doesn't depend on KVM at all and is confirmed independently by #135 on a real cluster.

Ran into this while working through a replication task for the "Integration of urunc's sandbox execution with Argo Workflows" LFX Mentorship proposal, related to #573.

[Anand-240]

@cmainas @gntouts @ananos, opened a fix for this as #883: a minimal exec subcommand that fails fast with a clear "not supported" error and a dedicated exit code (255, matching runc's own convention), instead of the current unknown-command failure that Argo (and any other exec-dependent controller) retries forever.

It's intentionally scoped as a fail-fast stub, not a real exec-into-unikernel implementation. Details on that scoping decision are in the PR description and the doc comment on execCommand.

Status: DCO passes and all checks on my side are green, but it's waiting on a maintainer to approve the CI workflow runs (and I've hit the open-PR limit on this repo, so it's still in Draft until that clears up). Happy to adjust the approach if you'd rather see this handled differently.

[cmainas]

This issue is related to the LFX mentorship program https://mentorship.lfx.linuxfoundation.org/project/3581574f-5f52-40c7-9a3b-b1dc019949d7 therefore we will not take into consideration before the mentorship program begins.