KRD (Kubernetes Reference Deployment) is a framework for easily deploying Kubernetes clusters on bare-metal or virtual machines using Ansible. It provides a configurable and extensible platform for validating Cloud Native technologies across various Linux distributions.
A central concept in KRD is the use of the POD Descriptor File (PDF)—a YAML file that defines your cluster configuration. You can start with the default configuration or explore the sample setups for advanced scenarios.
| Distribution | Version |
|---|---|
| Ubuntu | 20.04/22.04 |
| openSUSE | Leap |
| Fedora | 37/38/39/40 |
KRD integrates a wide range of Kubernetes-related components for testing, benchmarking, or production experimentation:
| Component | Description | Source | Status |
|---|---|---|---|
| Kubernetes | Base Kubernetes deployment | kubespray | ✅ Done |
| Virtlet | Run VMs as Kubernetes pods | configure-virtlet.yml | ✅ Tested |
| Kubevirt | VM workloads on Kubernetes | ✅ Tested | |
| Virtink | VM workloads on Kubernetes | ✅ Tested | |
| Multus | Multi-network pod support | ✅ Tested | |
| NFD | Node Feature Discovery | nfd role | ✅ Tested |
| Istio | Service Mesh | ✅ Tested | |
| PMEM | Persistent Memory CSI | pmem role | 🚧 Implemented |
| QAT | Intel QuickAssist support | qat_plugin role | ✅ Tested |
| SR-IOV | SR-IOV Plugin | sriov_plugin role | 🚧 Implemented |
| Knative | Serverless platform | ✅ Tested | |
| Kong | Ingress Controller | ✅ Tested | |
| HAProxy | Ingress Controller | ✅ Tested | |
| MetalLB | Network load balancer | ✅ Tested | |
| Falco | Security runtime monitoring | ✅ Tested | |
| Rook | Storage Operator | ✅ Tested | |
| OPA Gatekeeper | Policy Enforcement | ✅ Tested | |
| Kyverno | Policy Enforcement | ✅ Tested | |
| Kubewarden | Policy Enforcement | ✅ Tested | |
| kube-monkey | Chaos engineering for Kubernetes | 🚧 Implemented | |
| Nephio | Telco-native orchestration | 🚧 Implemented | |
| LocalAI | Local LLM inference support | 🚧 Implemented | |
| K8SGPT | Kubernetes diagnostics | 🚧 Implemented | |
| ArgoCD | GitOps & CI/CD workflows | 🚧 Implemented | |
| Tekton | GitOps & CI/CD workflows | 🚧 Implemented | |
| Longhorn | Storage Operator | ✅ Tested | |
| TopoLVM | CSI storage with topology awareness | ✅ Tested | |
| Fluent | Logging agent | ✅ Tested | |
| CloudNativePG | PostgreSQL operator | ✅ Tested | |
| LiteLLM | Local LLM inference support | 🚧 Implemented | |
| kagent | KRM-native agentic AI platform | 🚧 Implemented |
Use the All-in-One script for a streamlined, unattended deployment:
curl -fsSL http://bit.ly/KRDaio | bash
This will clone the repository, install dependencies, and provision a default cluster setup.
KRD supports custom cluster definitions using the PDF (YAML) format. Vagrant uses the default configuration, but this can be overridden via a config/pdf.yml file.
You can also visualize your cluster topology using:
tox -e diagram
Use KRD command script to install optional components on an existing cluster.
./krd_command.sh -h
To install Kubernetes add-ons (e.g., Virtlet):
KRD_ADDONS_LIST=virtlet ./krd_command.sh -a install_k8s_addons
Note: Some add-ons include validation scripts in the tests directory.
KRD supports full lifecycle operations, including upgrades, node addition, and service removal.
Example: Upgrade your Kubernetes cluster to version v1.18.10 using Kubespray v2.14.2:
KRD_KUBE_VERSION=v1.18.10 KRD_KUBESPRAY_VERSION=v2.14.2 ./krd_command.sh -a upgrade_k8s
KRD relies on environment variables to control behavior and customization. Below are some of the most relevant variables.
| Name | Default | Description |
|---|---|---|
| KRD_DEBUG | false | Enables verbose execution |
| KRD_ANSIBLE_DEBUG | false | Enables ansible verbose execution |
| PKG_DEBUG | false | Enables cURL package installer verbose execution |
| KRD_KUBE_VERSION | v1.32.5 | Specifies the Kubernetes version to be upgraded |
| KRD_KUBESPRAY_VERSION | v2.28.0 | Specifies the Kubespray version to be used during the upgrade process |
| KRD_KUBESPRAY_REPO | https://github.com/kubernetes-sigs/kubespray |
Specifies the Git repository to fetch the Kubespray's source code |
| KRD_ENABLE_TESTS | false | Enables the functional tests during the deployment process |
| KRD_HELM_CHART | Specifies the Helm chart to be installed | |
| KRD_FOLDER | /opt/krd | KRD source code destination folder |
| KRD_ACTIONS_LIST | install_k8s | Specifies a list of KRD actions to be installed during the All-in-One execution |
| KRD_ADDONS_LIST | addons | Specifies the Kubernetes AddOns to be installed by install_k8s_addons |
| KRD_HUGEPAGES_ENABLED | true | Enable/Disable HugeTLB Pages on cluster nodes |
| KRD_QAT_PLUGIN_MODE | dpdk | Specifies the deployment mode for Intel QuickAssist Plugin |
| KRD_KUBESPHERE_DEVOPS_ENABLED | true | Enable/Disable KubeSphere DevOps System |
| KRD_KUBESPHERE_METRICS_SERVER_ENABLED | false | Enable/Disable Horizontal Pod Autoscaler on KubeSphere |
| KRD_KUBESPHERE_SERVICEMESH_ENABLED | false | Enable/Disable KubeSphere Service Mesh |
| KRD_ENABLE_ISTIO_ADDONS | false | Enable/Disable Istio AddOns(Grafana, Kiali and Prometheus) |
| KRD_METALLB_ADDRESS_POOLS_LIST | Specifies a list of L2 address pools for MetalLB configuration |
| Name | Default | Description |
|---|---|---|
| KRD_NETWORK_PLUGIN | flannel | Choose network plugin (calico, canal, cilium, contiv, flannel, weave) |
| KRD_FLANNEL_BACKEND_TYPE | host-gw | Type of flannel backend to use (vxlan, host-gw, udp) |
| KRD_CILIUM_TUNNEL_MODE | disabled | Encapsulation mode for communication between nodes (disabled, vxlan, geneve) |
| KRD_CILIUM_KUBE_PROXY_REPLACEMENT | probe | Specifies the type of kube-proxy replacement. |
| KRD_CILIUM_ENABLE_BPF_MASQUERADE | true | Enable/Disable native IP masquerade support in eBPF |
| KRD_CALICO_IPIP_MODE | Never | Configures Calico IP in IP encapsulation (Always, Never, CrossSubnet) |
| KRD_CALICO_VXLAN_MODE | Never | Configures Calico VXLAN encapsulation (Always, Never, CrossSubnet) |
| KRD_CALICO_NETWORK_BACKEND | bird | Configures Calico Network backend (bird, vxlan, none) |
| KRD_KUBE_PROXY_MODE | ipvs | Choose kube-proxy mode (iptables, ipvs) |
| KRD_DOWNLOAD_RUN_ONCE | true | Download images and binaries only once and then push them to the cluster nodes |
| KRD_DOWNLOAD_LOCALHOST | true | Make localhost the download delegate |
| KRD_MULTUS_ENABLED | false | Enable/Disable Kubernetes Multus CNI |
| KRD_CONTAINER_RUNTIME | containerd | Specifies the Container Runtime to be used for deploying kubernetes |
| KRD_DASHBOARD_ENABLED | false | Enable/Disable Kubernetes Dashboard |
| KRD_CERT_MANAGER_ENABLED | true | Enable/Disable Kubernetes Cert-Manager |
| KRD_INGRESS_NGINX_ENABLED | true | Enable/Disable NGINX Ingress Controller |
| KRD_KATA_CONTAINERS_ENABLED | false | Enable/Disable Kata Containers Runtime |
| KRD_CRUN_ENABLED | false | Enable/Disable crun |
| KRD_YOUKI_ENABLED | false | Enable/Disable youki |
| KRD_GVISOR_ENABLED | false | Enable/Disable gVisor |
| KRD_MANUAL_DNS_SERVER | Set to use a custom cluster DNS | |
| KRD_REGISTRY_MIRRORS_LIST | Specifies a list of additional registry mirrors | |
| KRD_INSECURE_REGISTRIES_LIST | Specifies a list of insecure-registries (IP address or domain name) | |
| KRD_LOCAL_VOLUME_PROVISIONER_ENABLED | true | Enable/Disable Local volume provisioner |
| KRD_DOCKER_VERSION | latest | Specifies the Docker version to be used for deploying Kubernetes |
| KRD_CONTAINERD_VERSION | 1.7.11 | Specifies the ContainerD version to be used for deploying Kubernetes |
| KRD_ENABLE_NODELOCALDNS | true | Enable/Disable NodeLocal DNSCache |
| KRD_NDOTS | 1 | Threshold for the number of dots which must appear in name resolution |
| KRD_RESOLVCONF_MODE | none | Specifies the DNS setup for non-k8s containers. |
| KRD_KUBE_PROXY_SCHEDULER | sh | Specifies the IPVS scheduling algorithm for allocating connections |
| KRD_METALLB_ENABLED | false | Enable/Disable MetalLB load-balancer |
| KRD_METALLB_ADDRESS_POOLS_LIST | Specifies a lists of Layer 2 Address pools for MetalLB | |
| KRD_KUBERNETES_AUDIT | false | Enable/Disable Auditing |
| KRD_KUBERNETES_AUDIT_WEBHOOK | false | Enable/Disable Audit Webhook |
| KRD_AUDIT_WEBHOOK_SERVER_URL | Audit Webhook server URL | |
| KRD_KUBELET_LOGFILES_MAX_NR | 5 | Maximum number of container log files that can be present for a container |
| KRD_KUBELET_LOGFILES_MAX_SIZE | 10Mi | Maximum size of the container log file before it is rotated |
| KRD_CONTAINER_CHECKPOINT_ENABLED | false | Enable/Disable Containers checkpoints creation |
KRD is an open-source initiative with contributions from developers, testers, and documenters across the community.
We welcome all forms of collaboration. Please see our CONTRIBUTING guide for details.
Thanks to everyone who has contributed so far!
