Skip to content

Commit 727db02

Browse files
james-nofrixionjames-nofrixion
committed
added core services, removed cluster-autoscaler
1 parent 451a46e commit 727db02

9 files changed

+201
-225
lines changed

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
aws-credentials.txt
22
kubeconfig.txt
3+
*.log

‎README.md

+13-9
Original file line numberDiff line numberDiff line change
@@ -11,24 +11,28 @@ The initial strategy is to create a configuration to deploy the base cluster for
1111

1212
## Initial Setup ##
1313

14-
The [intial setup instructions](./docs/initial-setup.md) describe how to use and IaC solution that runs in Kubernetes to deploy itself. Thankfully this does not have to be done very often...
14+
The [intial setup instructions](./docs/initial-setup.md) describe how to use an IaC solution that runs in Kubernetes to deploy itself. Thankfully this does not have to be done very often.
1515

16-
# Cluster Deployment #
16+
## Cluster Deployment ##
1717

18-
The cluster deployments use kustomize to install the kubernetes cluster and components.
18+
The cluster deployments use kustomize to install the kubernetes cluster and components. To improve modularity in terms of deploying clusters that require different components, or in the case of deploying to a different CSP, the deployment is designed as follows:
1919

20-
- Nofrixion specific [composite resource definitions (XRDs) and compositions](../apis/aws/) have beend defined to deploy a VPC and kubernetes cluster to AWS. The configuration is the same as those cluster initially deployed using `eksctl`
21-
- Cluster components (e.g. cluster autoscaler, nginx ingress controller, cert-manager, rabbitmq etc.) are deployed as seperate resources.
20+
- Nofrixion specific [composite resource definitions (XRDs) and compositions](../apis/aws/) have beend defined to deploy a VPC and kubernetes cluster to AWS. The configuration is essentially the same as those clusters initially deployed using `eksctl`
21+
- Cluster components (e.g. nginx ingress controller, cert-manager, rabbitmq etc.) are deployed as seperate resources.
2222

23-
This approach improves modularity in terms of deploying clusters that require different components, or in the case of deploying to a different CSP, a different composition for the cluster can be created.
24-
25-
To deploy a cluster, create a kustomization.yaml file to deploy the following resouces to a specific namespace:
23+
To deploy a cluster, create a kustomization.yaml [(example)](./it-ops-cluster/kustomization.yaml) file to deploy the following resouces to a specific namespace:
2624

2725
* a cluster claim, which calls the xrds and compositions to create a specific cluster instance. For example, the [it-ops-1 cluster](./it-ops-cluster/it-ops-cluster.yaml)
2826
* crossplane objects and releases to deploy additional components. `Objects` use the crossplane kubernetes provider to run the equivalent of `kubectl apply ...` and `Releases` use the helm provider to deploy helm charts.
2927

28+
## Known Issues ##
29+
30+
### Cluster Autoscaler ###
31+
Crossplane did not get on with the cluster autoscaler, nodes were scaling up and down constantly. Given Azure AKS and Google GKE have proprietry (and at least in the case of Azure, simple) solutions to node autoprovisioning it is probably work looking at Karpenter to handle this.
32+
33+
However, to manually add a node to the crossplane cluster just change `spec.parameters.node.count` in the cluster manifest [(see example)](./it-ops-cluster/it-ops-cluster.yaml) to the desired value.
3034

31-
## Troubleshooting ##
35+
# Troubleshooting #
3236

3337
### Deleting 'stuck' resources ###
3438

‎apis/aws/eks-cluster/composition.yaml

+24-212
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
# Author: James Bramich ([email protected])
88
#
99
# TODO
10-
# - add cluster autoscaler
10+
# - fix cluster autoscaler (bouncing nodes every 10-15 min)
1111
#
1212
# History:
1313
# 25 March 2024 James Bramich Created, Northdown, Tasmania
@@ -793,6 +793,29 @@ spec:
793793
fmt: '%s-efs-csi-rolepolicyattachment'
794794
type: Format
795795

796+
- name: efsCsiPdb
797+
base:
798+
apiVersion: kubernetes.crossplane.io/v1alpha2
799+
kind: Object
800+
spec:
801+
forProvider:
802+
manifest:
803+
apiVersion: policy/v1
804+
kind: PodDisruptionBudget
805+
metadata:
806+
name: efs-csi-controller-temp-pdb
807+
namespace: kube-system
808+
spec:
809+
maxUnavailable: 1
810+
selector:
811+
matchLabels:
812+
app: efs-csi-controller
813+
app.kubernetes.io/instance: aws-efs-csi-driver
814+
app.kubernetes.io/name: aws-efs-csi-driver
815+
patches:
816+
- patchSetName: kubernetesProviderConfigRef
817+
type: PatchSet
818+
796819
- name: efsCsiAddon
797820
base:
798821
apiVersion: eks.aws.upbound.io/v1beta1
@@ -1031,217 +1054,6 @@ spec:
10311054
toFieldPath: spec.forProvider.values.clusterName
10321055
# end load balancer setup
10331056

1034-
# set up cluster autoscaler
1035-
- name: autoscalerIamRole
1036-
base:
1037-
apiVersion: iam.aws.upbound.io/v1beta1
1038-
kind: Role
1039-
metadata:
1040-
labels:
1041-
role: clusterAutoscaler
1042-
spec:
1043-
forProvider:
1044-
assumeRolePolicy: ""
1045-
patches:
1046-
- patchSetName: providerConfigRef
1047-
type: PatchSet
1048-
- patchSetName: deletionPolicy
1049-
type: PatchSet
1050-
- patchSetName: tags
1051-
type: PatchSet
1052-
- type: CombineFromComposite
1053-
toFieldPath: spec.forProvider.assumeRolePolicy
1054-
combine:
1055-
variables:
1056-
- fromFieldPath: status.eks.accountId
1057-
- fromFieldPath: status.eks.oidcUri
1058-
- fromFieldPath: status.eks.oidcUri
1059-
- fromFieldPath: status.eks.oidcUri
1060-
strategy: string
1061-
string:
1062-
fmt: |
1063-
{
1064-
"Version": "2012-10-17",
1065-
"Statement": [
1066-
{
1067-
"Effect": "Allow",
1068-
"Principal": {
1069-
"Federated": "arn:aws:iam::%s:oidc-provider/%s"
1070-
},
1071-
"Action": "sts:AssumeRoleWithWebIdentity",
1072-
"Condition": {
1073-
"StringEquals": {
1074-
"%s:aud": "sts.amazonaws.com",
1075-
"%s:sub": "system:serviceaccount:kube-system:cluster-autoscaler-sa"
1076-
}
1077-
}
1078-
}
1079-
]
1080-
}
1081-
- type: FromCompositeFieldPath
1082-
fromFieldPath: spec.parameters.id
1083-
toFieldPath: spec.forProvider.tags["Name"]
1084-
transforms:
1085-
- type: string
1086-
string:
1087-
fmt: '%s-cluster-autoscaler-role'
1088-
type: Format
1089-
- type: ToCompositeFieldPath
1090-
fromFieldPath: status.atProvider.arn
1091-
policy:
1092-
fromFieldPath: Optional
1093-
toFieldPath: status.eks.clusterAutoscalerRoleArn
1094-
- type: FromCompositeFieldPath
1095-
fromFieldPath: spec.parameters.id
1096-
toFieldPath: "metadata.annotations[crossplane.io/external-name]"
1097-
transforms:
1098-
- type: string
1099-
string:
1100-
fmt: '%s-cluster-autoscaler-role'
1101-
type: Format
1102-
1103-
- name: autoscalerIamPolicy
1104-
base:
1105-
apiVersion: iam.aws.upbound.io/v1beta1
1106-
kind: Policy
1107-
metadata:
1108-
spec:
1109-
forProvider:
1110-
policy: ""
1111-
patches:
1112-
- patchSetName: providerConfigRef
1113-
type: PatchSet
1114-
- patchSetName: deletionPolicy
1115-
type: PatchSet
1116-
- patchSetName: tags
1117-
type: PatchSet
1118-
- type: CombineFromComposite
1119-
toFieldPath: spec.forProvider.policy
1120-
combine:
1121-
variables:
1122-
- fromFieldPath: spec.parameters.region
1123-
- fromFieldPath: status.eks.accountId
1124-
- fromFieldPath: status.eks.nodeAutoscalingGroupName
1125-
strategy: string
1126-
string:
1127-
fmt: |
1128-
{
1129-
"Version": "2012-10-17",
1130-
"Statement": [
1131-
{
1132-
"Effect": "Allow",
1133-
"Action": [
1134-
"autoscaling:DescribeAutoScalingGroups",
1135-
"autoscaling:DescribeAutoScalingInstances",
1136-
"autoscaling:DescribeLaunchConfigurations",
1137-
"autoscaling:DescribeScalingActivities",
1138-
"autoscaling:DescribeTags",
1139-
"ec2:DescribeInstanceTypes",
1140-
"ec2:DescribeLaunchTemplateVersions"
1141-
],
1142-
"Resource": [
1143-
"*"
1144-
]
1145-
},
1146-
{
1147-
"Effect": "Allow",
1148-
"Action": [
1149-
"autoscaling:SetDesiredCapacity",
1150-
"autoscaling:TerminateInstanceInAutoScalingGroup",
1151-
"ec2:DescribeImages",
1152-
"ec2:GetInstanceTypesFromInstanceRequirements",
1153-
"eks:DescribeNodegroup"
1154-
],
1155-
"Resource": [
1156-
"arn:aws:autoscaling:%s:%s:autoScalingGroup:*:autoScalingGroupName/%s"
1157-
]
1158-
}
1159-
]
1160-
}
1161-
- type: FromCompositeFieldPath
1162-
fromFieldPath: spec.parameters.id
1163-
toFieldPath: spec.forProvider.tags["Name"]
1164-
transforms:
1165-
- type: string
1166-
string:
1167-
fmt: '%s-cluster-autoscaler-policy'
1168-
type: Format
1169-
- type: ToCompositeFieldPath
1170-
fromFieldPath: status.atProvider.arn
1171-
policy:
1172-
fromFieldPath: Optional
1173-
toFieldPath: status.eks.clusterAutoscalerPolicyArn
1174-
- type: FromCompositeFieldPath
1175-
fromFieldPath: spec.parameters.id
1176-
toFieldPath: "metadata.annotations[crossplane.io/external-name]"
1177-
transforms:
1178-
- type: string
1179-
string:
1180-
fmt: '%s-cluster-autoscaler-policy'
1181-
type: Format
1182-
1183-
- name: autoscalerRolePolicyAttachment
1184-
base:
1185-
apiVersion: iam.aws.upbound.io/v1beta1
1186-
kind: RolePolicyAttachment
1187-
spec:
1188-
forProvider:
1189-
policyArn: ""
1190-
roleSelector:
1191-
matchControllerRef: true
1192-
matchLabels:
1193-
role: clusterAutoscaler
1194-
patches:
1195-
- patchSetName: providerConfigRef
1196-
type: PatchSet
1197-
- patchSetName: deletionPolicy
1198-
type: PatchSet
1199-
- type: FromCompositeFieldPath
1200-
fromFieldPath: status.eks.clusterAutoscalerPolicyArn
1201-
toFieldPath: spec.forProvider.policyArn
1202-
- type: FromCompositeFieldPath
1203-
fromFieldPath: spec.parameters.id
1204-
toFieldPath: "metadata.annotations[crossplane.io/external-name]"
1205-
transforms:
1206-
- type: string
1207-
string:
1208-
fmt: '%s-autoscaler-rolepolicyattachment'
1209-
type: Format
1210-
1211-
- name: autoscalerHelmChart
1212-
base:
1213-
apiVersion: helm.crossplane.io/v1beta1
1214-
kind: Release
1215-
metadata:
1216-
name: autoscaler
1217-
spec:
1218-
forProvider:
1219-
chart:
1220-
name: cluster-autoscaler
1221-
repository: https://kubernetes.github.io/autoscaler
1222-
version: 9.36.0
1223-
namespace: kube-system
1224-
values:
1225-
rbac:
1226-
create: true
1227-
serviceAccount:
1228-
create: true
1229-
name: cluster-autoscaler-sa
1230-
patches:
1231-
- patchSetName: helmProviderConfigRef
1232-
type: PatchSet
1233-
- type: FromCompositeFieldPath
1234-
fromFieldPath: spec.parameters.id
1235-
toFieldPath: spec.forProvider.values.autoDiscovery.clusterName
1236-
- type: FromCompositeFieldPath
1237-
fromFieldPath: spec.parameters.region
1238-
toFieldPath: spec.forProvider.values.awsRegion
1239-
- type: FromCompositeFieldPath
1240-
fromFieldPath: status.eks.clusterAutoscalerRoleArn
1241-
toFieldPath: spec.forProvider.values.rbac.serviceAccount.annotations["eks.amazonaws.com/role-arn"]
1242-
1243-
# end cluster autoscaler setup
1244-
12451057
- name: providerConfigKubernetes
12461058
base:
12471059
apiVersion: kubernetes.crossplane.io/v1alpha1

‎it-ops-cluster/helm-azure-workload-identity.yaml

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#------------------------------------------------------------------------------
2+
#
3+
# Filename: helm-azure-workload-identity.yaml
4+
#
5+
# Description: Crossplane helm provider release for kubernetes external secrets operator.
6+
#
7+
# Usage: Add as resource to kustomization.yaml for deployment.
8+
#
9+
# Author: James Bramich ([email protected])
10+
#
11+
# History:
12+
# 21 May 2024 James Bramich Created, Northdown, Tasmania, Australia.
13+
#-----------------------------------------------------------------------------
14+
apiVersion: kubernetes.crossplane.io/v1alpha2
15+
kind: Object
16+
metadata:
17+
name: clusterName-azure-workload-identity-namespace
18+
spec:
19+
forProvider:
20+
manifest:
21+
apiVersion: v1
22+
kind: Namespace
23+
metadata:
24+
name: azure-workload-identity-system
25+
providerConfigRef:
26+
name: clusterName-k8s-provider
27+
---
28+
apiVersion: helm.crossplane.io/v1beta1
29+
kind: Release
30+
metadata:
31+
name: workload-identity-webhook
32+
spec:
33+
forProvider:
34+
chart:
35+
name: workload-identity-webhook
36+
repository: https://azure.github.io/azure-workload-identity/charts
37+
version: 1.2.2
38+
namespace: azure-workload-identity-system
39+
values:
40+
azureTenantID: 97c78528-728a-4eb7-b50e-ee7aced0a1ce
41+
providerConfigRef:
42+
name: clusterName-helm-provider

‎it-ops-cluster/helm-external-secrets.yaml

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#------------------------------------------------------------------------------
2+
#
3+
# Filename: helm-external-secrets.yaml
4+
#
5+
# Description: Crossplane helm provider release for kubernetes external secrets operator.
6+
#
7+
# Usage: Add as resource to kustomization.yaml for deployment.
8+
#
9+
# Author: James Bramich ([email protected])
10+
#
11+
# History:
12+
# 21 May 2024 James Bramich Created, Northdown, Tasmania, Australia.
13+
#-----------------------------------------------------------------------------
14+
apiVersion: kubernetes.crossplane.io/v1alpha2
15+
kind: Object
16+
metadata:
17+
name: clusterName-external-secrets-namespace
18+
spec:
19+
forProvider:
20+
manifest:
21+
apiVersion: v1
22+
kind: Namespace
23+
metadata:
24+
name: external-secrets
25+
providerConfigRef:
26+
name: clusterName-k8s-provider
27+
---
28+
apiVersion: helm.crossplane.io/v1beta1
29+
kind: Release
30+
metadata:
31+
name: external-secrets
32+
spec:
33+
forProvider:
34+
chart:
35+
name: external-secrets
36+
repository: https://charts.external-secrets.io
37+
version: 0.9.18
38+
namespace: external-secrets
39+
providerConfigRef:
40+
name: clusterName-helm-provider

0 commit comments

Comments
 (0)