stageTimeout changes do not reach in-flight Jobs

[ayuskauskas]

Important

Base this work on feature/package-as-jobs, not main.

The Jobs migration (#223) is integrating on that branch. main does not yet execute
packages as Jobs, so a PR opened against main will be missing the code this depends
on. feature/package-as-jobs merges to main once #305 lands.

That branch squash-merges, so if your branch is stacked on another PR in this epic,
replay only your own commits when restacking:
git rebase --onto origin/feature/package-as-jobs <last-inherited-commit>

Part of #223. Refs #411 (item 4).

Problem

jobMatchesPackage (job_builder.go:163-169) compares only the pod-template subset via podMatchesPackage. ActiveDeadlineSeconds is set separately from effectiveStageTimeout(opts, _package) and is never compared.

Editing package.stageTimeout, or the operator-wide JOB_STAGE_TIMEOUT default, therefore leaves an existing Job looking current, so validation will not replace a Job that now carries the wrong deadline. The new CRD knob has no effect on in-flight work until the Job finishes or is deleted by hand.

Note on #402

PR #402 (#373) moves the per-attempt deadline onto the pod template and adds a derived Job-level ceiling, which changes where the field lives but does not add a comparison. Confirm against that PR's final shape before fixing — depending on what podMatchesPackage compares, part of this may resolve there.

Acceptance

• Changing package.stageTimeout or JOB_STAGE_TIMEOUT causes validation to replace the in-flight Job.
• Spec covers both the per-package field and the operator-wide default.

[ayuskauskas]

Resolved by decision, documented in #402 (4c11b177) — this issue was filed from a stale checklist and will close when that PR merges.

Making the knob reach in-flight Jobs means declaring the Job stale and recreating it — i.e. killing the running stage to grant it more time, which is perverse for the edit that actually motivates the change. And there is no restart-free path: the per-attempt bound lives on the pod template, which is immutable on a Job, and the running pod's own activeDeadlineSeconds is mutable but decrease-only. So on 2026-08-07 the call was to document the contract rather than implement propagation:

The value is fixed when a stage's Job is created. Editing it does not change a Job already running, because the bound lives on that Job's pod template and a Job's template is immutable. To apply a new value to work already under way, clear that Job — kubectl nodewright package rerun <package>, or deleting the Job — and the stage restarts under the new value. Otherwise it takes effect at the package's next stage.

Written into the CRD docstring (both CRD copies), chart/values.yaml, and the design doc's deadline section. The same contract now covers jobBackoffLimit, which #402 adds on the same footing.

[ayuskauskas]

Resolved as a documented contract in #402 (4c11b177), merged to feature/package-as-jobs.

Making the knob reach in-flight Jobs would mean declaring the Job stale and recreating it — killing a running stage to grant it more time — and there is no restart-free path: the per-attempt bound lives on the pod template, which is immutable on a Job, and the running pod's own activeDeadlineSeconds is decrease-only. So the contract is that the value is fixed when a stage's Job is created; kubectl nodewright package rerun applies a new one to work already under way. Written into the CRD docstring, both CRD copies, chart/values.yaml and the design doc.

Editing the package is different and does now clear a stage that has given up — see #402's spec-drift change.

Closing by hand: the Closes keyword does not fire for a merge into a feature branch.