Compared to the likes of Terraform and Pulumi, Crossplane is a relative newcomer to the IaC space. However, as a kubernetes based platform it is appealing by virtue of:
- leveraging our existing k8s skillset
- making use of a large and growing k8s ecosystem
The initial strategy is to create a configuration to deploy the base cluster for EKS then follow that with required services e.g. cert-manager, cluster-autoscaler, external-secrets-operator etc. This means a move to a different CSP will only require the cluster deployment to change.
The intial setup instructions describe how to use an IaC solution that runs in Kubernetes to deploy itself. Thankfully this does not have to be done very often.
The cluster deployments use kustomize to install the kubernetes cluster and components. To improve modularity in terms of deploying clusters that require different components, or in the case of deploying to a different CSP, the deployment is designed as follows:
- Nofrixion specific composite resource definitions (XRDs) and compositions have beend defined to deploy a VPC and kubernetes cluster to AWS. The configuration is essentially the same as those clusters initially deployed using
eksctl - Cluster components (e.g. nginx ingress controller, cert-manager, rabbitmq etc.) are deployed as seperate resources.
To deploy a cluster, create a kustomization.yaml (example) file to deploy the following resouces to a specific namespace:
- a cluster claim, which calls the xrds and compositions to create a specific cluster instance. For example, the it-ops-1 cluster
- crossplane objects and releases to deploy additional components.
Objectsuse the crossplane kubernetes provider to run the equivalent ofkubectl apply ...andReleasesuse the helm provider to deploy helm charts.
Crossplane did not get on with the cluster autoscaler, nodes were scaling up and down constantly. Given Azure AKS and Google GKE have proprietry (and at least in the case of Azure, simple) solutions to node autoprovisioning it is probably work looking at Karpenter to handle this.
However, to manually add a node to the crossplane cluster just change spec.parameters.node.count in the cluster manifest (see example) to the desired value.
This mostly happens during testing but, in case of emergency remove finalisers from the managed resource e.g.
TARGET="{YOUR STUCK RESOURCES NAME}"
kubectl patch $TARGET -p '{"metadata":{"finalizers": []}}' --type=merge- Crossplane docs:
- Anton Putra's tutorial for creating VPC and deploying EKS - basically what we want but with only two subnet pairs instead of three.
- GitOps model for provisioning and bootstrapping Amazon EKS clusters using Crossplane and Argo CD - see section,
Amazon EKS cluster provisioning using Crossplane - A video showing how to use a temporary local cluster to bootstrap Crossplane to manage Crossplane
- AWS Blueprints (community provider)
