Add an experimental prepared blue-green Agent rollout

[AliDatadog]

Why

A DaemonSet update can delete a working Agent before Kubernetes pulls and starts its replacement. A slow or failed pull can leave a node without an Agent.

Native surge cannot complete this handoff when the replacement must wait for the old Agent to release host resources. This experiment keeps rollout control in the Operator.

Design

The feature uses this opt-in annotation:

experimental.agent.datadoghq.com/node-agent-rollout-mode: prepared-blue-green-v1

The Operator maintains two OnDelete DaemonSets. One slot serves traffic. The other slot prepares the next revision. A one-time conventional rollout installs the lock wrapper before the Operator allows overlap.

An init container copies s6-setlock from the core Agent image into a shared Pod volume. Each Agent container then waits on two locks:

• a host lock prevents two generations from running the same component;
• a Pod-local lock tells the startup probe when that component starts.

The Operator observes a stable target, then removes the source on a bounded set of nodes. Each target component starts when its matching source component exits. The target must become Available before the Operator starts another batch.

The Operator uses the existing maxUnavailable value as the budget. Capacity fallback can use the same budget to remove a source that blocks target scheduling. This fallback is delete-first.

For hostNetwork: true, the Operator removes declared ports that would block scheduling. Processes bind the node ports after lock handoff. The two slots use different core command ports to prevent mixed-generation control traffic.

Failure handling and trade-offs

• Preparation proves that the image, init containers, and lock wrappers started. It does not prove Agent health.
• Readiness and liveness check the Agent after handoff. Startup probes report whether each component has acquired its lock.
• If a target fails before any target Pod becomes Ready, the Operator cancels the rollout and returns all in-flight nodes to the last source.
• If one target is Ready while another target fails, the Operator stops the rollout. This case needs a reviewed recovery plan.
• Capacity fallback has the same pull and startup downtime as the current rollout.
• Locks do not transfer queues, caches, assignments, or process state.
• Native OnDelete DaemonSet status does not describe a prepared pair correctly. The DatadogAgent can remain Updating while all Agent Pods are healthy.
• The earlier custom-gate PoC used an incompatible lock type. Schema 3 rejects that PoC instead of allowing unsafe overlap.

Validation

• go test ./... -count=1 passed.
• make lint passed with 0 issues.
• Focused tests cover migration, new nodes, capacity fallback, controller restart, concurrent updates, opt-out, and failed-target recovery.
• A two-worker Kind test recovered from a failed target and then completed a healthy rollout.
• An isolated Unix domain socket sender test measured maximum consecutive gaps of 3.896 seconds and 3.353 seconds on the two workers. The test used a fake API key. It did not validate intake, logs, or traces.

The experiment does not require an Agent code change.

Related PR

• Experimental deployment and recovery runbook: https://github.com/DataDog/k8s-datadog-agent-ops/pull/9477

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 80.89%
• Overall Coverage: 51.83% (+1.65%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: adb6696 | Docs | View more details | Give us feedback!}