Add K8s failure classification, terminationReason, and retryOn directive

[rhassaine]

Summary

• K8s failure classification: The nf-k8s plugin now extracts container termination reasons from the K8s API and classifies failures as infrastructure (Evicted, Preempting, DeadlineExceeded, Shutdown → auto-retry) vs application (OOMKilled, Error → respect user's error strategy)
• terminationReason plumbing: New task.terminationReason field accessible in error strategy closures, plus native logging in retry messages ([reason: OOMKilled]) and hard failure error blocks
• retryOn directive: New declarative process directive to retry based on termination reason without Groovy closures (e.g. retryOn 'OOMKilled' or retryOn = ['OOMKilled', 'OOMKilled(exit137)'] in config)

Motivation

When running pipelines on Kubernetes, all task failures are treated identically — no distinction between OOMKilled, Evicted, or application errors. This makes it impossible to implement smart retry strategies like "retry OOM with more memory" or "fail fast on app errors" without fragile exit-code matching. Building on the work in #6436 and #6442 which introduced K8s exit code reading from the container terminated state.

Changes

nf-k8s plugin

• K8sTaskHandler: Extract termination reason from container state, classify infrastructure vs application failures, infer reason from exit codes (137→OOMKilled, 143→SignalTerm). Generic reasons (Error, Completed) are filtered out to avoid polluting logs.
• K8sClient: Fix exitcode→exitCode typo and 0 ?: readExitFile() Groovy truthiness bug in jobStateFallback0()

Core (nextflow module)

• TaskRun: Added volatile String terminationReason field
• TaskConfig: Added getRetryOn() getter for the new directive
• TaskProcessor: Plumbed terminationReason/aborted to TaskConfig, implemented retryOn override in checkErrorStrategy(), added native terminationReason logging in retry and error messages
• TaskErrorFormatter: Added "Termination reason" section to hard failure error blocks
• ProcessBuilder: Registered retryOn as valid process directive
• ProcessDsl.DirectiveDsl: Added retryOn methods for config validation

Tests

• 6 new + 1 updated test in K8sTaskHandlerTest (OOM, CPU, infrastructure classification, Evicted/aborted, OOMKilled/not-aborted, application failure)
• 1 new test in K8sClientTest (job fallback exit code)

Example usage

process ALIGN {
    retryOn 'OOMKilled'
    memory { 2.GB * task.attempt }
    maxRetries 3

    script:
    """
    bwa-mem2 mem ...
    """
}

Or in config:

process {
    retryOn = ['OOMKilled', 'OOMKilled(exit137)']
    memory = { 4.GB * task.attempt }
}

Test plan

• All 148 unit tests pass (make test)
• Tested on GKE with custom OOM pipeline: process allocates 150MB against a 100MB memory limit, confirmed [reason: OOMKilled] in native retry logs and successful retry with scaled memory (200MB on attempt 2)
• Tested on GKE with nf-core/oncoanalyser (on latest stable Nextflow): forced OOM on BWAMEM2_ALIGN at 1500MB, confirmed [reason: OOMKilled] in logs and successful retry with scaled memory
• Infrastructure failures (Evicted, Preempting) correctly set task.aborted = true
• Application failures (exit 1) do not trigger retryOn or set aborted
• Generic K8s reasons (Error, Completed) are filtered out — only meaningful reasons appear in logs

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	1d6afcb
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/69b416d19f94d00008d7ccb4