|
| 1 | +# Bottlerocket and Bottlerocket Update Operator |
| 2 | + |
| 3 | +[Bottlerocket](https://aws.amazon.com/bottlerocket/) is a Linux-based open-source operating system that focuses on security and maintainability, providing a reliable, consistent, and safe platform for container-based workloads. |
| 4 | + |
| 5 | +The [Bottlerocket Update Operator (BRUPOP)](https://github.com/bottlerocket-os/bottlerocket-update-operator/tree/develop) is a Kubernetes operator that coordinates Bottlerocket updates on hosts in a cluster. It relies on a controller deployment on one node to orchestrate updates across the cluster, an agent daemon set on every Bottlerocket node, which is responsible for periodically querying and performing updates rolled out in waves to reduce the impact of issues, and an API Server that performs additional authorization. |
| 6 | + |
| 7 | +[Cert-manager](https://cert-manager.io/) is required for the API server to use a CA certificate when communicating over SSL with the agents. |
| 8 | + |
| 9 | +- [Helm charts](https://github.com/bottlerocket-os/bottlerocket-update-operator/tree/develop/deploy/charts) |
| 10 | + |
| 11 | +## Requirements |
| 12 | + |
| 13 | +BRUPOP perform updates on Nodes running with Bottlerocket OS only. Here are some code snippets of how to setup up Bottlerocket OS Nodes using Managed Node Groups with [Terraform Amazon EKS module](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest) and [Karpenter Node Classes](https://karpenter.sh/docs/concepts/nodeclasses/). |
| 14 | + |
| 15 | +Notice the label `bottlerocket.aws/updater-interface-version=2.0.0` set in the `[settings.kubernetes.node-labels]` section. This label is required for the BRUPOP Agent to query and perform updates. Nodes not labeled will not be checked by the agent. |
| 16 | + |
| 17 | +### Managed Node Groups |
| 18 | + |
| 19 | +```hcl |
| 20 | +module "eks" { |
| 21 | + source = "terraform-aws-modules/eks/aws" |
| 22 | + version = "~> 19.21" |
| 23 | +... |
| 24 | + eks_managed_node_groups = { |
| 25 | + bottlerocket = { |
| 26 | + platform = "bottlerocket" |
| 27 | + ami_type = "BOTTLEROCKET_x86_64" |
| 28 | + instance_types = ["m5.large", "m5a.large"] |
| 29 | +
|
| 30 | + iam_role_attach_cni_policy = true |
| 31 | +
|
| 32 | + min_size = 1 |
| 33 | + max_size = 5 |
| 34 | + desired_size = 3 |
| 35 | +
|
| 36 | + enable_bootstrap_user_data = true |
| 37 | + bootstrap_extra_args = <<-EOT |
| 38 | + [settings.host-containers.admin] |
| 39 | + enabled = false |
| 40 | + [settings.host-containers.control] |
| 41 | + enabled = true |
| 42 | + [settings.kernel] |
| 43 | + lockdown = "integrity" |
| 44 | + [settings.kubernetes.node-labels] |
| 45 | + "bottlerocket.aws/updater-interface-version" = "2.0.0" |
| 46 | + [settings.kubernetes.node-taints] |
| 47 | + "CriticalAddonsOnly" = "true:NoSchedule" |
| 48 | + EOT |
| 49 | + } |
| 50 | + } |
| 51 | +} |
| 52 | +``` |
| 53 | + |
| 54 | +### Karpenter |
| 55 | + |
| 56 | +```yaml |
| 57 | +apiVersion: karpenter.k8s.aws/v1beta1 |
| 58 | +kind: EC2NodeClass |
| 59 | +metadata: |
| 60 | + name: bottlerocket-example |
| 61 | +spec: |
| 62 | +... |
| 63 | + amiFamily: Bottlerocket |
| 64 | + userData: | |
| 65 | + [settings.kubernetes] |
| 66 | + "kube-api-qps" = 30 |
| 67 | + "shutdown-grace-period" = "30s" |
| 68 | + "shutdown-grace-period-for-critical-pods" = "30s" |
| 69 | + [settings.kubernetes.eviction-hard] |
| 70 | + "memory.available" = "20%" |
| 71 | + [settings.kubernetes.node-labels] |
| 72 | + "bottlerocket.aws/updater-interface-version" = "2.0.0" |
| 73 | +``` |
| 74 | +
|
| 75 | +## Usage |
| 76 | +
|
| 77 | +[BRUPOP](https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/) can be deployed with the default configuration by enabling the add-on via the following. Notice the parameter `wait = true` set for Cert-Manager, this is needed since BRUPOP requires that Cert-Manager CRDs are already present in the cluster to be deployed. |
| 78 | + |
| 79 | +```hcl |
| 80 | +module "eks_blueprints_addons" { |
| 81 | + source = "aws-ia/eks-blueprints-addons/aws" |
| 82 | + version = "~> 1.13" |
| 83 | +
|
| 84 | + cluster_name = module.eks.cluster_name |
| 85 | + cluster_endpoint = module.eks.cluster_endpoint |
| 86 | + cluster_version = module.eks.cluster_version |
| 87 | + oidc_provider_arn = module.eks.oidc_provider_arn |
| 88 | +
|
| 89 | + enable_cert_manager = true |
| 90 | + cert_manager = { |
| 91 | + wait = true |
| 92 | + } |
| 93 | + enable_bottlerocket_update_operator = true |
| 94 | +} |
| 95 | +``` |
| 96 | + |
| 97 | +You can also customize the Helm charts that deploys `bottlerocket_update_operator` and the `bottlerocket_shadow` via the following configuration: |
| 98 | + |
| 99 | +```hcl |
| 100 | +enable_bottlerocket_update_operator = true |
| 101 | +
|
| 102 | +bottlerocket_update_operator = { |
| 103 | + name = "brupop-operator" |
| 104 | + description = "A Helm chart for BRUPOP" |
| 105 | + chart_version = "1.3.0" |
| 106 | + namespace = "brupop" |
| 107 | + set = [{ |
| 108 | + name = "scheduler_cron_expression" |
| 109 | + value = "0 * * * * * *" # Default Unix Cron syntax, set to check every hour. Example "0 0 23 * * Sat *" Perform update checks every Saturday at 23H / 11PM |
| 110 | + }] |
| 111 | +} |
| 112 | +
|
| 113 | +bottlerocket_shadow = { |
| 114 | + name = "brupop-crds" |
| 115 | + description = "A Helm chart for BRUPOP CRDs" |
| 116 | + chart_version = "1.0.0" |
| 117 | +} |
| 118 | +``` |
| 119 | + |
| 120 | +To see a complete working example, see the [`bottlerocket`](https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/patterns/bottlerocket) Blueprints Pattern. |
| 121 | + |
| 122 | +## Validate |
| 123 | + |
| 124 | +1. Run `update-kubeconfig` command: |
| 125 | + |
| 126 | +```bash |
| 127 | +aws eks --region <REGION> update-kubeconfig --name <CLUSTER_NAME> |
| 128 | +``` |
| 129 | + |
| 130 | +2. Test by listing velero resources provisioned: |
| 131 | + |
| 132 | +```bash |
| 133 | +$ kubectl -n brupop-bottlerocket-aws get all |
| 134 | +
|
| 135 | +NAME READY STATUS RESTARTS AGE |
| 136 | +pod/brupop-agent-5nv6m 1/1 Running 1 (33h ago) 33h |
| 137 | +pod/brupop-agent-h4vw9 1/1 Running 1 (33h ago) 33h |
| 138 | +pod/brupop-agent-sr9ms 1/1 Running 2 (33h ago) 33h |
| 139 | +pod/brupop-apiserver-6ccb74f599-4c9lv 1/1 Running 0 33h |
| 140 | +pod/brupop-apiserver-6ccb74f599-h6hg8 1/1 Running 0 33h |
| 141 | +pod/brupop-apiserver-6ccb74f599-svw8n 1/1 Running 0 33h |
| 142 | +pod/brupop-controller-deployment-58d46595cc-7vxnt 1/1 Running 0 33h |
| 143 | +
|
| 144 | +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE |
| 145 | +service/brupop-apiserver ClusterIP 172.20.153.72 <none> 443/TCP 33h |
| 146 | +service/brupop-controller-server ClusterIP 172.20.7.127 <none> 80/TCP 33h |
| 147 | +
|
| 148 | +NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE |
| 149 | +daemonset.apps/brupop-agent 3 3 3 3 3 <none> 33h |
| 150 | +
|
| 151 | +NAME READY UP-TO-DATE AVAILABLE AGE |
| 152 | +deployment.apps/brupop-apiserver 3/3 3 3 33h |
| 153 | +deployment.apps/brupop-controller-deployment 1/1 1 1 33h |
| 154 | +
|
| 155 | +NAME DESIRED CURRENT READY AGE |
| 156 | +replicaset.apps/brupop-apiserver-6ccb74f599 3 3 3 33h |
| 157 | +replicaset.apps/brupop-controller-deployment-58d46595cc 1 1 1 33h |
| 158 | +
|
| 159 | +$ kubectl describe apiservices.apiregistration.k8s.io v2.brupop.bottlerocket.aws |
| 160 | +Name: v2.brupop.bottlerocket.aws |
| 161 | +Namespace: |
| 162 | +Labels: kube-aggregator.kubernetes.io/automanaged=true |
| 163 | +Annotations: <none> |
| 164 | +API Version: apiregistration.k8s.io/v1 |
| 165 | +Kind: APIService |
| 166 | +Metadata: |
| 167 | + Creation Timestamp: 2024-01-30T16:27:15Z |
| 168 | + Resource Version: 8798 |
| 169 | + UID: 034abe22-7e5f-4040-9b64-8ca9d55a4af6 |
| 170 | +Spec: |
| 171 | + Group: brupop.bottlerocket.aws |
| 172 | + Group Priority Minimum: 1000 |
| 173 | + Version: v2 |
| 174 | + Version Priority: 100 |
| 175 | +Status: |
| 176 | + Conditions: |
| 177 | + Last Transition Time: 2024-01-30T16:27:15Z |
| 178 | + Message: Local APIServices are always available |
| 179 | + Reason: Local |
| 180 | + Status: True |
| 181 | + Type: Available |
| 182 | +Events: <none> |
| 183 | +``` |
| 184 | + |
| 185 | +1. If not set during the deployment, add the required label `bottlerocket.aws/updater-interface-version=2.0.0` as shown below to all the Nodes that you want to have updates handled by BRUPOP. |
| 186 | + |
| 187 | +```bash |
| 188 | +$ kubectl label node ip-10-0-34-87.us-west-2.compute.internal bottlerocket.aws/updater-interface-version=2.0.0 |
| 189 | +node/ip-10-0-34-87.us-west-2.compute.internal labeled |
| 190 | +
|
| 191 | +$ kubectl get nodes -L bottlerocket.aws/updater-interface-version |
| 192 | +NAME STATUS ROLES AGE VERSION UPDATER-INTERFACE-VERSION |
| 193 | +ip-10-0-34-87.us-west-2.compute.internal Ready <none> 34h v1.28.1-eks-d91a302 2.0.0 |
| 194 | +``` |
| 195 | + |
| 196 | +4. Because the default cron schedule for BRUPOP is set to check for updates every minute, you'll be able to see in a few minutes that the Node had it's version updated automatically with no downtime. |
| 197 | + |
| 198 | +```bash |
| 199 | +kubectl get nodes |
| 200 | +NAME STATUS ROLES AGE VERSION |
| 201 | +ip-10-0-34-87.us-west-2.compute.internal Ready <none> 34h v1.28.4-eks-d91a302 |
| 202 | +``` |
0 commit comments