The CloudStack Kubernetes Service documentation states:
"Using a pre-packaged ISO containing required binaries and docker images allows faster provisioning on the node Instances of a Kubernetes cluster. Complete offline provisioning of the Kubernetes cluster is not supported at present as the kubeadm init command needs active Internet access."
This guide documents that it is possible, with workarounds.
The CloudStack documentation explicitly states that complete offline provisioning of Kubernetes clusters is not supported — kubeadm init allegedly requires active internet access. This means if your management server or zone has no outbound connectivity to the public internet, cluster creation is expected to fail.
This is a real barrier for air-gapped deployments, government/military environments, and any setup where management servers are intentionally disconnected from the internet. The documentation's stance makes it seem like there's no way around it — but as this guide demonstrates, it is possible with some workarounds.
- All K8s images are already baked into the ISO — no local registry or pre-download step required
- The ISO is self-contained for cluster creation
- Pre-built Calico ISOs are available from download.cloudstack.org/cks/ (x86 and ARM variants)
Initial testing with the 1.32.5 Calico ISO shows that most CKS operations work without internet:
- ✅ Cluster creation — provisioning succeeds fully offline
- ✅ Day 2 scaling — adding/removing worker nodes works
- ✅ Upgrade to 1.33.1 — upgrading from 1.32.5 to the next version works
Upgrading beyond a certain point requires internet access:
- ❌ Upgrade to 1.34.x — upgrade fails when disconnected
- As soon as internet is restored, the same upgrade succeeds
Additionally, with the corrupted pause container in the 1.34.7 ISO:
- ❌ Creating a new cluster with 1.34.7 — also fails offline for the same reason (see Corrupted Pause Container below)
Additionally, with custom Cilium ISOs:
- ❌ Cilium CNI deployment fails offline — digest-pinned image references can't be verified without internet (see Section 4.2 below)
Two distinct root causes can prevent offline deployment:
- Pause container version mismatch (pre-built Calico ISOs): CKS upgrades only import new images onto the node currently being upgraded. When a health check pod or other intermediate Job is scheduled on a non-upgraded node, that node lacks newer image versions — including
pause:3.10.1. Without internet to pull them, pods can't start. - Image digest mismatch (custom Cilium ISOs): The Cilium manifest generated by Helm references images using digest pins (
@sha256:...). Kubernetes must verify these digests against an external registry — but the bundled image tarballs in the ISO don't carry digest metadata, so verification fails offline.
Both issues are silent when online: internet provides a fallback that masks the problem. The two scenarios are documented below.
During an offline upgrade from 1.33.x to 1.34.7, the process stalls while creating upgrade health check pods — a Job that verifies control plane upgrade completion before moving on to worker nodes.
On a failed (offline) upgrade, the health check pod gets stuck in a ContainerCreating → Terminated loop:
- The pod attempts to start but is immediately terminated
- Kubernetes retries, and the cycle repeats until the overall upgrade times out
On a successful (online) upgrade, the same pod reaches Completed status without issue.
The health check pod uses the pause container as its base image. Comparing the CKS ISOs:
| K8s Version | ISO Pause Image |
|---|---|
| 1.32.x | pause:3.10 |
| 1.33.x | pause:3.10 |
| 1.34.x | pause:3.10.1 |
The pause container version changed in the 1.34 ISO — from 3.10 to 3.10.1. This small change is the key.
In a cluster with 1 control plane and 1 worker node:
- The upgrade starts on the control plane first. During this phase, the management server imports all 1.34 images (including
pause:3.10.1) from the ISO onto that node. - Once the control plane is upgraded, the upgrade health check pod needs to run — and Kubernetes schedules it on a node that is not being actively upgraded (i.e., the worker node).
- The worker node still runs 1.33.x and has only
pause:3.10in its local image store (verified viacrictl images). - The health check pod requests
pause:3.10.1, but the worker node doesn't have it locally. - With internet: the worker pulls
pause:3.10.1from an external registry → pod completes → upgrade proceeds. - Without internet: the pull fails silently, the container is terminated, Kubernetes retries → infinite loop → upgrade stalls and eventually times out.
The root cause is that CKS upgrades only import new images onto the node currently being upgraded. When a health check pod (or any intermediate Job) gets scheduled on a different node that hasn't been upgraded yet, that node lacks the newer images — including pause:3.10.1. Without internet to pull them, the pod can't start.
This explains why upgrades up to 1.33.x work offline (same pause version as before) but 1.34+ fails (new pause version required on non-upgraded nodes).
A separate issue compounds the problem: the official 1.34.7 Calico x86_64 ISO contains a corrupted pause container image (pause:3.10.1).
This is extremely hard to detect — if you're online, the node silently pulls the correct pause:3.10.1 from an external registry instead of using the broken one from the ISO, and everything appears to work fine. Only in fully offline mode does this issue surface.
Affected operations: Not just upgrades — creating a new cluster with 1.34.7 also fails in an offline environment for the same reason. The pause container is required not only on worker nodes (for health check Jobs) but also on control plane nodes, where core pods like kube-apiserver, kube-scheduler, and kube-controller-manager depend on it as their base image. With a corrupted pause:3.10.1 in the ISO and no internet to fall back on, neither node type can start its pods.
Workaround: If the imported pause image from the 1.34.7 ISO is corrupted, you'll need to obtain a valid pause:3.10.1 tarball (e.g., export it from a node with internet access using ctr -n k8s.io images export) and import it manually before attempting the upgrade.
To avoid relying on official ISOs that may have corrupted images, you can build your own CKS ISO. This gives you full control over which K8s version and CNI are included, and ensures all container images are valid.
- See the CKS Custom ISO Build Guide for step-by-step instructions on building a custom ISO with
create-kubernetes-binaries-iso.shor the community script (for Cilium).
Testing your custom ISO offline: Before deploying to production, validate your ISO by creating a CKS cluster in an offline environment. This is the only way to catch issues like corrupted images that would be silently masked when internet connectivity provides a fallback. The same test methodology described in this guide (cluster creation → scaling → upgrade) will confirm the ISO works end-to-end without any network dependency.
When building a custom CKS ISO with Cilium CNI, there's another offline failure mode. The Cilium manifest is generated via Helm chart and references images using digest pins (e.g., quay.io/cilium/cilium@sha256:xxxxx). However, the bundled image tarballs in the ISO are stored without the digest reference.
When Kubernetes tries to start Cilium pods, it sees the digest-pinned image name and attempts to verify the digest against an external registry. In an offline environment, this verification fails — the pod can't be created even though the actual image tarball is already baked into the ISO.
The symptom: kube-proxy, coredns, kube-controller-manager, and kube-apiserver all start successfully (they use tag-based references), but Cilium CNI pods remain in ImagePullBackOff or ErrImagePull. Since Cilium is required for cluster networking, the overall CKS deployment is considered failed.
Running journalctl -u kubelet -f on the control node reveals the digest verification failure:
pod_workers.go:1324] "Error syncing pod, skipping" err=
"failed to \"StartContainer\" for \"cilium-envoy\" with ImagePullBackOff:
Back-off pulling image \"
quay.io/cilium/cilium-envoy:v1.36.6-1776000132-2437d2edeaf4d9b56ef279bd0d71127440c067aa
@sha256:ba0ab8adac082d50d525fd2c5ba096c8facea3a471561b7c61c7a5b9c2e0de0d\"
ErrImagePull: rpc error: code = DeadlineExceeded
desc = failed to pull and unpack image \"
quay.io/cilium/cilium-envoy
@sha256:ba0ab8adac082d50d525fd2c5ba096c8facea3a471561b7c61c7a5b9c2e0de0d\"
failed to resolve reference \"
quay.io/cilium/cilium-envoy
@sha256:ba0ab8adac082d50d525fd2c5ba096c8facea3a471561b7c61c7a5b9c2e0de0d\"
failed to do request:
Head \"https://quay.io/v2/cilium/cilium-envoy/manifests/
sha256:ba0ab8adac082d50d525fd2c5ba096c8facea3a471561b7c61c7a5b9c2e0de0d\"
dial tcp: lookup quay.io: i/o timeout"
pod="kube-system/cilium-envoy-6p6kf" podUID="90798992-b980-42a1-b952-66a66265e602"
The key detail is lookup quay.io: i/o timeout — Kubernetes attempts to verify the digest via a HEAD request to quay.io, which fails because there's no internet access. Note how the image reference includes @sha256:... digest pins (shown on their own lines above) — these are what trigger the external registry lookup.
This is a fundamental mismatch between how Helm generates manifests (digest-pinned) and how create-kubernetes-binaries-iso.sh packages images (tag-only).
The create-cilium-offline-kubernetes-binaries-iso.sh script solves this by stripping all @sha256:... digest pins from the generated YAML manifests before baking them into the ISO. This leaves tag-based image references that match what's already bundled in the containerd store, eliminating the need for external registry verification.
See Option C: Build Cilium Offline ISO in the custom ISO build guide.
Note: The pause container issue described in Section 4.1 also applies during upgrades of Cilium custom ISOs — if a new pause version is introduced, the same worker-node image-missing problem will occur.
This workaround addresses the pause container version mismatch described in Section 4.1.
Because CKS only imports new images onto the node currently being upgraded, intermediate Jobs (like health checks) scheduled on other nodes will fail if those nodes lack the newer image versions.
💡 Proactive Tip: If you know a CKS upgrade will introduce a new pause container version (or any new shared image), skip the reactive fix and pre-import all images to every node before triggering the upgrade. This guarantees all nodes have what they need immediately, preventing intermediate Jobs from failing in the first place.
If you know the target CKS version introduces new shared images (e.g., a new pause version), import them onto all nodes before triggering the upgrade. This prevents intermediate Jobs from failing entirely.
- SSH into each node (control plane and all workers):
sudo -i
- Attach the target CKS ISO as a secondary ISO to each VM via CloudStack UI or API.
- Mount the ISO:
mkdir -p /mnt/iso mount /dev/sr0 /mnt/iso
- Import all images from the ISO onto every node:
for f in /mnt/iso/docker/*.tar; do ctr -n k8s.io images import "$f"; done
- Repeat steps 1–4 on every remaining node.
- Trigger the CKS upgrade via cmk or UI as usual:
cmk upgrade kubernetescluster id=<cluster-id> kubernetesversionid=<new-version-id>
Because all nodes already have the new images locally, intermediate Jobs (health checks, etc.) can be scheduled on any node without failing.
If an offline upgrade has already stalled due to a missing pause container, follow these steps:
1. SSH into the worker node (non-upgraded node):
sudo -i2. Attach the 1.34 CKS ISO to the worker node:
From CloudStack UI or via API, attach the 1.34 CKS ISO as a secondary ISO to the worker VM.
3. Mount the ISO on the worker node:
The attached ISO will appear as /dev/sr0. Create a mount point and mount it:
mkdir -p /mnt/iso
mount /dev/sr0 /mnt/iso4. Import all new images from the ISO:
Instead of importing just the pause container, import everything in the docker/ directory — the node needs more than just that one image:
for f in /mnt/iso/docker/*.tar; do ctr -n k8s.io images import "$f"; doneNote:
ctr images importdoesn't accept multiple files at once, so a simple loop is needed.
5. Verify the images are imported:
crictl images | grep pauseYou should see pause with tag 3.10.1 along with other 1.34.x images in the output.
6. Restart the upgrade from CloudStack Management:
Trigger the upgrade again via UI or cmk:
cmk upgrade kubernetescluster id=<cluster-id> kubernetesversionid=<new-version-id>The upgrade health check pod should now reach Completed status, and the upgrade to 1.34 proceeds successfully.
By pre-loading all the new images from the ISO onto the worker node before any intermediate Jobs run, Kubernetes doesn't need to pull anything from an external registry — everything is already local in containerd's store.
- Offline upgrades work only up to 1.33.x; beyond that, the pause container version change breaks health check pods on non-upgraded nodes
- The root cause is that CKS imports images only onto the node currently being upgraded — intermediate Jobs scheduled on other nodes can't find their required images without internet
- This affects any upgrade where new image versions are introduced (pause container or otherwise) that must run on a node not yet upgraded
- Corrupted ISO: The official 1.34.7 Calico x86_64 ISO has a corrupted
pause:3.10.1image — goes unnoticed online (silent fallback to registry pull) but breaks offline deployment
The manual workarounds above solve immediate offline upgrades, but two architectural improvements would make air-gapped CKS fully robust long-term:
Instead of relying on bundled ISO tarballs or external registries, mirror all required container images to an internal registry (e.g., Harbor, Docker Registry) inside your air-gapped network.
How to update YAMLs to reference the local registry:
- On a build machine with internet access, pull upstream images and push them to
registry.local:5000. - Rewrite all
image:lines in the CKS manifests (network.yaml,dashboard.yaml, etc.) usingsed:# Example: replace k8s.gcr.io/pause with local mirror sed -i 's|k8s.gcr.io/\(.*:\)|registry.local:5000/k8s.gcr.io/\1|g' *.yaml
- Rebuild the custom ISO with these modified YAMLs.
- Configure kubelet/containerd on all CKS nodes to trust the local registry via
insecure_registries(or internal CA certs).
This decouples cluster operations from upstream registry availability and simplifies future upgrades — you only need to sync new images to your mirror, not rebuild ISOs every time.
The pause container issue occurs because CKS currently imports new images only onto the node being actively upgraded. When Kubernetes schedules a health check or intermediate Job on a different (non-upgraded) node, that node lacks the newer image versions and fails offline.
Proposed upstream fix: Modify the CKS upgrade orchestration logic to pre-import all required images from the target ISO onto every cluster node before upgrading any single node.
This guarantees that:
- Intermediate Jobs can be safely scheduled on any node without hitting
ImagePullBackOff. - Upgrades proceed seamlessly even when new shared base images (like
pause) are introduced in a newer K8s version.
Implementing this would require changes to the cloudstack-kubernetes-service plugin code. Contributing this fix upstream would make offline upgrades natively reliable across all future versions without manual workarounds.