Updated Reviewed Changes

Author: sudharshanibm

infra/ibmcloud/terraform/k8s-s390x-build-cluster/README.md

@@ -0,0 +1,127 @@
+# _TF: IBM K8s s390x Build Cluster_
+These terraform resources define a IBM Cloud project containing a s390xVS cluster intended to serve as a "build cluster" for prow.k8s.io.
+
+---
+## Initial Setup
+
+### Supporting infrastructure
+
+#### Deploy k8s-infra-setup resources
+
+- this covers things like Resource Group, s390x Virtual Server Instances, Virtual Private Cloud, IBM Cloud Secret Manager Secrets, etc.
+- Once the deployment successfully completes, the `secrets_manager_id` will be generated and should be used in the subsequent steps.
+
+---
+#### Deploy k8s-s390x-build-cluster resources
+
+**1. Navigate to the correct directory**
+<br> You need to be in the `k8s-s390x-build-cluster` directory to run the automation.
+
+**2. Export COS Secrets**
+<br> Export `access_key` and `secret_key` as environment variables.
+```
+export AWS_ACCESS_KEY_ID="<HMAC_ACCESS_KEY_ID>"
+export AWS_SECRET_ACCESS_KEY="<HMAC_SECRET_ACCESS_KEY>"
+```
+
+**3. Initialize Terraform**
+<br> Execute the following command to initialize Terraform in your project directory. This command will download the necessary provider plugins and prepare the working environment.
+```
+terraform init -reconfigure
+```
+
+**4. Check the `variables.tf` file**
+<br> Open the `variables.tf` file to review all the available variables. This file lists all customizable inputs for your Terraform configuration.
+
+`ibmcloud_api_key`, `secrets_manager_id` are the only required variables that you must set in order to proceed. You can set this key either by adding it to your `var.tfvars` file or by exporting it as an environment variable.
+
+**Option 1:** Set in `var.tfvars` file
+Create `var.tfvars` file and set the following variables in `var.tfvars` file:
+```
+ibmcloud_api_key    = "<YOUR_API_KEY>"
+secrets_manager_id  = "<SECRETS_MANAGER_ID>"
+```
+Tip: To get the secrets_manager_id (GUID) for IBM Cloud Secrets Manager instance:
+```
+ibmcloud resource service-instances --service-name secrets-manager --output JSON | \
+jq -r '.[] | select(.name | contains("k8s-s390x")) | .guid'
+```
+**Option 2:** Export as an environment variable
+Alternatively, you can export above as an environment variable before running Terraform:
+```
+export TF_VAR_ibmcloud_api_key="<YOUR_API_KEY>"
+export TF_VAR_secrets_manager_id=$(ibmcloud resource service-instances --service-name secrets-manager --output JSON | \
+jq -r '.[] | select(.name | contains("k8s-s390x")) | .guid')
+```
+
+**5. Run Terraform Apply**
+<br> After setting the necessary variables (particularly the API_KEY), execute the following command to apply the Terraform configuration and provision the infrastructure:
+```
+terraform apply -var-file var.tfvars
+```
+Terraform will display a plan of the actions it will take, and you'll be prompted to confirm the execution. Type `yes` to proceed.
+
+**6. Get Output Information**
+<br> Once the infrastructure has been provisioned, use the terraform output command to list details about the provisioned resources.
+```
+terraform output
+```
+
+**7. Set up the Kubernetes cluster using ansible**
+Clone the repository `https://github.com/kubernetes-sigs/provider-ibmcloud-test-infra` and change the directory to `kubetest2-tf/data/k8s-ansible`:
+```
+cd kubetest2-tf/data/k8s-ansible
+```
+
+**8. Install ansible on the deployer VM**
+```
+dnf install ansible -y
+```
+
+**9. Update the fields under `group_vars/all` to include the Kubernetes version to install**
+<br> The following lines will update the version to the latest stable release of Kubernetes. You can modify it accordingly to set up the CI (alpha) version.
+```
+K8S_VERSION=$(curl -Ls https://dl.k8s.io/release/stable.txt)
+LOADBALANCER_EP=<mention the loadbalancer endpoint obtained from terraform output>
+sed -i \
+-e "s/^directory: .*/directory: release/" \
+-e "s/build_version: .*/build_version: $K8S_VERSION/" \
+-e "s/release_marker: .*/release_marker: $K8S_VERSION/" \
+-e "s/loadbalancer: .*/loadbalancer: $LOADBALANCER_EP/" group_vars/all
+```
+
+**10. Update the fields under `examples/k8s-build-cluster/hosts.yml` to contain IP addresses of the VMs to set up Kubernetes**
+```
+For example:
+
+[bastion]
+56.77.34.6
+
+[masters]
+192.168.100.3
+192.168.100.4
+
+[workers]
+192.168.100.5
+192.168.100.6
+192.168.100.7
+
+[workers:vars]
+ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path/to/private-key> -q [email protected]" -i <path/to/private-key>'
+
+[masters:vars]
+ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path/to/private-key> -q [email protected]" -i <path/to/private-key>'
+```
+
+**11. Update the fields under `group_vars/bastion_configuration` to contain the information of the private network.**
+```
+For example:
+
+bastion_private_gateway: 192.168.100.1
+bastion_private_ip: 192.168.100.2
+```
+
+**12. Trigger the installation using ansible**
+```
+ansible-playbook -v -i examples/k8s-build-cluster/hosts.yml install-k8s-ha.yaml -e @group_vars/bastion_configuration --extra-vars @group_vars/all
+```

infra/ibmcloud/terraform/k8s-s390x-build-cluster/bastion.tf

@@ -13,24 +13,37 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 */
+
+locals {
+  bastion_nodes = {
+    "primary" = {
+      profile = var.bastion_profile
+      boot_volume = {
+        size = var.bastion_boot_volume_size
+      }
+    }
+  }
+}
+
 resource "ibm_is_instance" "bastion" {
-  count          = var.bastion.count
-  name           = "bastion-s390x-${count.index + 1}"
+  for_each       = local.bastion_nodes
+  name           = "bastion-s390x-${each.key}"
   vpc            = data.ibm_is_vpc.vpc.id
   zone           = var.zone
-  profile        = var.bastion.profile
+  profile        = each.value.profile
   image          = var.image_id
   keys           = [ibm_is_ssh_key.k8s_ssh_key.id]
   resource_group = data.ibm_resource_group.resource_group.id
+
   primary_network_interface {
-    name            = "public-nic"
+    name            = "public-nic-${each.key}"
     subnet          = data.ibm_is_subnet.subnet.id
-    security_groups = [data.ibm_is_security_group.bastion_sg.id]
+    security_groups = [data.ibm_is_security_group.bastion.id]
   }
 
   boot_volume {
-    name = "boot-vol-bastion-${count.index}"
-    size = var.bastion.boot_volume.size
+    name = "boot-vol-bastion-${each.key}"
+    size = each.value.boot_volume.size
   }
 
   user_data = <<-EOF
@@ -70,60 +83,16 @@ resource "ibm_is_instance" "bastion" {
                 - [netfilter-persistent, save]
                 - [systemctl, restart, systemd-networkd]
                 - [systemctl, restart, sshd]
-                - [hostnamectl, set-hostname, "bastion-s390x-${count.index + 1}.s390x-vpc.cloud.ibm.com"]
-                - [echo, "bastion-s390x-${count.index + 1}.s390x-vpc.cloud.ibm.com", ">", /etc/hostname]
-                - [sed, -i, "s/^127.0.1.1.*/127.0.1.1\tbastion-s390x-${count.index + 1}.s390x-vpc.cloud.ibm.com/", /etc/hosts]
+                - [hostnamectl, set-hostname, "bastion-s390x-${each.key}.s390x-vpc.cloud.ibm.com"]
+                - [echo, "bastion-s390x-${each.key}.s390x-vpc.cloud.ibm.com", ">", /etc/hostname]
+                - [sed, -i, "s/^127.0.1.1.*/127.0.1.1\tbastion-s390x-${each.key}.s390x-vpc.cloud.ibm.com/", /etc/hosts]
+                - [touch, /var/lib/cloud/instance/bastion-setup-success]
               EOF
 }
 
 resource "ibm_is_floating_ip" "bastion_fip" {
-  count          = var.bastion.count
-  name           = "bastion-fip-${count.index}"
-  target         = ibm_is_instance.bastion[count.index].primary_network_interface[0].id
+  for_each       = ibm_is_instance.bastion
+  name           = "bastion-fip-${each.key}"
+  target         = each.value.primary_network_interface[0].id
   resource_group = data.ibm_resource_group.resource_group.id
 }
-
-resource "time_sleep" "wait_for_bastion" {
-  count      = var.bastion.count
-  depends_on = [ibm_is_floating_ip.bastion_fip]
-
-  create_duration = "180s" # Wait 3 minutes for full initialization
-}
-
-resource "null_resource" "bastion_setup" {
-  count      = var.bastion.count
-  depends_on = [time_sleep.wait_for_bastion]
-
-  connection {
-    type        = "ssh"
-    user        = "root"
-    host        = ibm_is_floating_ip.bastion_fip[count.index].address
-    private_key = data.ibm_sm_arbitrary_secret.ssh_private_key.payload
-    timeout     = "5m"
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      # Network verification
-      "echo '=== Network Interfaces ==='",
-      "ip -4 addr show",
-      "echo '=== Routing Table ==='",
-      "ip route",
-      "echo '=== NAT Configuration ==='",
-      "iptables -t nat -L -n -v",
-      "echo '=== IP Forwarding ==='",
-      "sysctl net.ipv4.ip_forward",
-
-      # Hostname verification
-      "echo '=== Hostname ==='",
-      "hostname",
-      "hostnamectl",
-      "cat /etc/hostname",
-
-      # Final security updates
-      "command -v apt-get >/dev/null && apt-get update -y && apt-get upgrade -y --security || true",
-      "command -v yum >/dev/null && yum update -y --security || true",
-      "command -v dnf >/dev/null && dnf update -y --security || true"
-    ]
-  }
-}

infra/ibmcloud/terraform/k8s-s390x-build-cluster/data.tf

@@ -25,13 +25,13 @@ data "ibm_is_subnet" "subnet" {
   name = "k8s-s390x-subnet"
 }
 
-data "ibm_is_security_group" "bastion_sg" {
+data "ibm_is_security_group" "bastion" {
   name = "k8s-vpc-s390x-bastion-sg"
   vpc  = data.ibm_is_vpc.vpc.id
 }
 
-data "ibm_is_security_group" "master_sg" {
-  name = "k8s-vpc-s390x-master-sg"
+data "ibm_is_security_group" "control_plane_sg" {
+  name = "k8s-vpc-s390x-control-plane-sg"
   vpc  = data.ibm_is_vpc.vpc.id
 }
 

infra/ibmcloud/terraform/k8s-s390x-build-cluster/load_balancer.tf

@@ -13,38 +13,39 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 */
-resource "ibm_is_lb" "k8s_load_balancer" {
-  name            = "k8s-s390x-lb"
+resource "ibm_is_lb" "public" {
+  name            = "k8s-s390x-ci"
   type            = "public"
   subnets         = [data.ibm_is_subnet.subnet.id]
   resource_group  = data.ibm_resource_group.resource_group.id
-  security_groups = [data.ibm_is_security_group.master_sg.id]
+  security_groups = [data.ibm_is_security_group.control_plane_sg.id]
 }
 
 resource "ibm_is_lb_pool" "k8s_api_pool" {
   name                = "k8s-api-server-pool"
-  lb                  = ibm_is_lb.k8s_load_balancer.id
+  lb                  = ibm_is_lb.public.id
   protocol            = "tcp"
   algorithm           = "round_robin"
   health_delay        = 5
   health_retries      = 2
   health_timeout      = 2
   health_type         = "tcp"
   health_monitor_url  = "/"
-  health_monitor_port = 6443
+  health_monitor_port = var.api_server_port
 }
 
 resource "ibm_is_lb_listener" "k8s_api_listener" {
-  lb           = ibm_is_lb.k8s_load_balancer.id
+  lb           = ibm_is_lb.public.id
   protocol     = "tcp"
-  port         = 6443
+  port         = var.api_server_port
   default_pool = ibm_is_lb_pool.k8s_api_pool.pool_id
 }
 
 resource "ibm_is_lb_pool_member" "k8s_api_members" {
-  count          = var.control_plane.count
-  lb             = ibm_is_lb.k8s_load_balancer.id
+  for_each = ibm_is_instance.control_plane
+
+  lb             = ibm_is_lb.public.id
   pool           = ibm_is_lb_pool.k8s_api_pool.pool_id
-  port           = 6443
-  target_address = ibm_is_instance.control_plane[count.index].primary_network_interface[0].primary_ipv4_address
+  port           = var.api_server_port
+  target_address = each.value.primary_network_interface[0].primary_ipv4_address
 }