Skip to content

Commit e68a396

Browse files
committed
fix(talosctl): reboot after a failed drain during upgrade
talosctl upgrade writes the new installer image to disk before it drains the Kubernetes node and reboots into it. When the drain step failed - for example because the apiserver is unreachable or the cluster is not yet bootstrapped - the command returned early, before the reboot. That left the upgrade half-applied: the new image was staged on disk while the node kept running the old version, with no reboot to activate it and no signal to the operator, since the earlier progress output already read "upgrade completed". The next unrelated reboot would then silently boot into the staged image. Once the image is committed to disk, aborting before the reboot is the worst outcome. The drain is a best-effort courtesy and must not strand the upgrade, so warn about the failed drain and continue to the reboot so the staged image is actually applied. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
1 parent ea95578 commit e68a396

1 file changed

Lines changed: 9 additions & 1 deletion

File tree

cmd/talosctl/cmd/talos/upgrade.go

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,15 @@ func upgradeViaLifecycleService(ctx context.Context, clientFactory *global.Clien
147147
if upgradeCmdFlags.drain {
148148
nodeNames, err = drainNodes(ctx, clientFactory, upgradeCmdFlags.drainTimeout, rep)
149149
if err != nil {
150-
return err
150+
// upgradeInternal above has already written the new installer image to disk;
151+
// the node only switches to it on the reboot below. Aborting here would leave
152+
// the upgrade half-applied: the new image is staged on disk, the node keeps
153+
// running the old version, and the next unrelated reboot silently boots into
154+
// the new one - all while the earlier progress output reads "upgrade completed".
155+
// A failed drain (e.g. the Kubernetes apiserver is unreachable, or the cluster
156+
// is not yet bootstrapped) is a best-effort courtesy and must not strand the
157+
// upgrade. Warn loudly and continue to the reboot so the staged image is applied.
158+
cli.Warning("failed to drain Kubernetes node(s) before reboot, continuing without drain: %s", err)
151159
}
152160
}
153161

0 commit comments

Comments
 (0)