ClickyKeys Infrastructure

Production-grade, fully automated cloud-native infrastructure for the ClickyKeys web platform.

Provisioned with Terraform · Configured with Ansible · Orchestrated with Kubernetes (k3s) · Shipped with GitHub Actions

Note

This is the current production branch — a fully automated, IaC-driven Kubernetes (k3s) stack provisioned with Terraform and Ansible, with a complete GitHub Actions CI/CD pipeline and Grafana Cloud observability. The earlier Docker Compose deployment is preserved on the docker-legacy branch as a historical reference of the original setup that powered the ClickyKeys platform.

---

DevOps Portfolio — At a Glance

This repository is a working DevOps portfolio piece and the live infrastructure backing the ClickyKeys project. It demonstrates an end-to-end, declarative pipeline that takes an empty AWS account and produces a running, TLS-secured, Kubernetes-hosted web application running two isolated environments (production + staging) on a single node — with zero manual steps after the variables are filled in, and a push-to-deploy CI/CD pipeline that builds, tests, promotes through staging, and auto-rolls-back on failure.

Technology Stack

Cloud / IaC	AWS (EC2, Security Groups, Key Pairs) · Terraform (provider hashicorp/aws ~> 5.0)
Configuration Management	Ansible (roles, handlers, group_vars) · Ansible Vault for secrets
Container Orchestration	Kubernetes via k3s (lightweight, single-node, Traefik disabled)
Multi-environment	production and staging as separate namespaces on the same cluster, each with its own domain, database, credentials, and replica counts — driven by a single environments list
CI/CD	GitHub Actions — PHPUnit → build & push images → deploy to staging → smoke test → deploy to production → smoke test → auto-rollback on failure
Package Management	Helm 3 (ingress-nginx, cert-manager, grafana/k8s-monitoring)
Ingress & TLS	ingress-nginx (DaemonSet, hostNetwork) · cert-manager + Let's Encrypt (HTTP-01 ACME)
Observability	Grafana Cloud via the grafana/k8s-monitoring Helm chart (Grafana Alloy shipping cluster metrics + logs)
Containers	Docker images published to GitHub Container Registry (ghcr.io), tagged by short commit SHA · nginx:alpine · php-fpm
Database	MySQL (provisioned on the host by Ansible, exposed to the cluster via Service + Endpoints)
Analytics	First-party, cookieless visit analytics with GeoIP country lookup (DB-IP IP-to-Country Lite MMDB, refreshed monthly in CI) and IP anonymization
Templating	Jinja2 for parameterised Kubernetes manifests (cluster-wide and per-environment)
Secrets	Ansible Vault on disk · Kubernetes Secret resources at runtime

Highlights for Recruiters

• Infrastructure as Code, end to end. A single terraform apply followed by a single ansible-playbook produces a fully working, TLS-terminated stack hosting both production and staging — no clicks, no copy-paste.
• Push-to-deploy CI/CD. Merging to main triggers GitHub Actions to run PHPUnit, build and push both container images to GHCR, deploy to staging, smoke-test it, promote to production, smoke-test again, and automatically roll back the production deployment if the post-deploy health check fails.
• True multi-environment design. Production and staging are fully isolated (separate namespaces, domains, databases, credentials, replica counts) yet defined by one declarative environments list, rendered through the same Jinja2 manifests.
• Modular Ansible roles (swap → k3s → helm → mysql → k8s_cluster → monitoring, then a per-environment k8s_env loop) with idempotent tasks, handlers, and templated configuration.
• Cloud-native networking and security: ingress-nginx as a DaemonSet on host network, Let's Encrypt automation through cert-manager, namespace isolation, secrets injected from Kubernetes Secret objects.
• Observability out of the box: Grafana Alloy ships cluster metrics and logs to Grafana Cloud, installed declaratively by Ansible from Vault-encrypted credentials.
• Secrets hygiene: every sensitive value is stored encrypted with Ansible Vault and excluded from version control via .gitignore.
• Real running site. This infrastructure powers the public ClickyKeys website — it is not a toy demo.

---

Architecture

                              ┌─────────────────────────┐
                              │   Client (browser)      │
                              └────────────┬────────────┘
                                           │ HTTPS / 443
                                           ▼
                  ┌────────────────────────────────────────────────┐
                  │  AWS EC2 instance  (Ubuntu, t3.medium, gp3 20G) │
                  │                                                │
                  │   ┌────────────────────────────────────────┐   │
                  │   │ ingress-nginx (DaemonSet, hostNet)     │   │
                  │   │   TLS terminated by cert-manager       │   │
                  │   └──────────────┬─────────────────────────┘   │
                  │                  │                             │
                  │   ┌──────────────▼─────────────────────────┐   │
                  │   │ k3s cluster                            │   │
                  │   │  ├─ ns: production                     │   │
                  │   │  │    ├─ nginx-web  (Deployment + Svc) │   │
                  │   │  │    ├─ php-fpm    (Deployment + Svc) │   │
                  │   │  │    └─ mysql-svc  (Service + Endpts) │◄──┼── MySQL on host
                  │   │  ├─ ns: staging                        │   │   (Ansible-managed)
                  │   │  │    └─ (same set, scaled to 0 idle)  │   │
                  │   │  ├─ cert-manager / ingress-nginx       │   │
                  │   │  └─ ns: monitoring (Grafana Alloy) ────┼───┼──► Grafana Cloud
                  │   └────────────────────────────────────────┘   │   (metrics + logs)
                  └────────────────────────────────────────────────┘

Provisioning & configuration pipeline

terraform apply
   │  provisions EC2, Security Group, Key Pair (gp3 20 GB root volume)
   │  writes ansible/inventory.ini with the new public IP
   ▼
ansible-playbook playbook.yml --ask-vault-pass
   ├── swap          – RAM-aware swapfile
   ├── k3s           – installs k3s, sets up kubeconfig
   ├── helm          – installs ingress-nginx + cert-manager
   ├── mysql         – installs MySQL, creates per-env DBs & users, imports schema
   ├── k8s_cluster   – renders + applies cluster-wide manifests (ClusterIssuer)
   ├── monitoring    – installs grafana/k8s-monitoring (Alloy → Grafana Cloud)
   └── k8s_env (loop)– for each environment in `environments`:
                       namespace, mysql-credentials Secret, services,
                       nginx/php deployments, ingress

Continuous delivery pipeline (.github/workflows/build.yml)

push to main
   ├── phpunit_test          – PHPUnit on ./php-fpm (PHP 8.2)
   ├── images_build          – build & push web + php images to GHCR
   │                           (tags: <short-sha> and latest; refreshes GeoIP DB)
   ├── deploy-staging        – SSH: kubectl set image on staging namespace
   ├── smoke-test-staging    – GET /api/healthz.php must return 200 + healthy payload
   ├── deploy-production     – SSH: kubectl set image + change-cause annotation
   ├── smoke-test-production – GET /api/healthz.php; on failure → kubectl rollout undo
   └── turn-off-staging      – scale staging deployments back to 0 replicas

A manual-only test.yml workflow (workflow_dispatch) is also provided to build and push images on demand without deploying.

---

Repository Structure

.
├── terraform/                           # AWS provisioning (IaC)
│   ├── main.tf                          # EC2 + Security Group + Key Pair + inventory generation
│   ├── variables.tf
│   ├── outputs.tf
│   └── terraform.example.tfvars         # ← copy to terraform.tfvars and edit
│
├── ansible/                             # Server configuration
│   ├── playbook.yml                     # Host-wide roles + per-environment k8s_env loop
│   ├── group_vars/
│   │   └── all.example/                 # ← copy to all/ and edit, then encrypt vault.yml
│   │       ├── all.example.yml          # global + `environments` list + Grafana Cloud refs
│   │       └── vault.example.yml        # prod/staging secrets + Grafana Cloud credentials
│   └── roles/
│       ├── swap/                        # Swapfile (RAM-aware sizing)
│       ├── k3s/                         # k3s install + kubeconfig
│       ├── helm/                        # Helm + ingress-nginx + cert-manager
│       ├── mysql/                       # MySQL server, per-env users, schema, config
│       ├── k8s_cluster/                 # Cluster-wide manifests (ClusterIssuer)
│       ├── k8s_env/                     # Per-environment app resources (looped)
│       └── monitoring/                  # grafana/k8s-monitoring (Alloy → Grafana Cloud)
│
├── k8s/manifests/                       # Jinja2-templated Kubernetes manifests
│   ├── cluster/
│   │   └── clusterissuer.yml.j2         # Let's Encrypt staging + prod issuers
│   └── env/                             # Rendered once per environment
│       ├── namespace.yml.j2
│       ├── deployment-nginx.yml.j2      # nginx-web (static frontend)
│       ├── deployment-php.yml.j2        # php-fpm (backend)
│       ├── service-nginx.yml.j2
│       ├── service-php.yml.j2
│       ├── service-mysql.yml.j2         # ClusterIP + Endpoints pointing at host MySQL
│       └── ingress.yml.j2               # TLS-terminated routing
│
├── .github/workflows/
│   ├── build.yml                        # Full CI/CD: test → build → staging → prod → rollback
│   └── test.yml                         # Manual build & push to GHCR (workflow_dispatch)
│
├── scripts/
│   └── smoke-check.sh                   # Health check against /api/healthz.php
│
├── docs/
│   └── privacy-analytics.md             # Privacy-policy copy for the first-party analytics
│
├── website/                             # Frontend container source (nginx:alpine)
│   ├── Dockerfile
│   ├── nginx.conf
│   └── web/                             # HTML, CSS, JS, images
│
├── php-fpm/                             # Backend container source (php-fpm 8.2)
│   ├── Dockerfile
│   ├── geoip/                           # DB-IP IP-to-Country Lite MMDB (fetched in CI)
│   └── web/
│
└── README.md

---

Setup & Deployment

Prerequisites

• Terraform ≥ 1.0
• Ansible ≥ 2.12
• AWS CLI configured with credentials that can create EC2 / Security Groups / Key Pairs
• An SSH key pair on your local machine
• DNS A records for both the production and staging domains pointing at the EC2 public IP after provisioning
• A Grafana Cloud stack (free tier is enough) for metrics + logs
• docker only if you want to build and push the application images manually (CI does this for you)

1. Clone the repository

git clone https://github.com/Reksaku/clickykeys-infra.git
cd clickykeys-infra

2. Configure Terraform variables

All .example files in this repo are templates — copy them and fill in real values. The actual files (without .example) are gitignored.

cp terraform/terraform.example.tfvars terraform/terraform.tfvars

Edit terraform/terraform.tfvars:

aws_region           = "eu-central-1"
project_name         = "clickykeys"
ami_id               = "ami-xxxxxxxxxxxxxxxxx"   # Ubuntu 24.04 LTS in your region
instance_type        = "t3.medium"               # optional, this is the default
ssh_public_key_path  = "~/.ssh/id_rsa.pub"
ssh_private_key_path = "~/.ssh/id_rsa"

3. Configure Ansible variables (non-secret)

Copy the entire example group_vars directory, then edit the files in the new all/ folder:

cp -r ansible/group_vars/all.example ansible/group_vars/all

Edit ansible/group_vars/all/all.yml. Global settings, the GHCR image source, the per-environment list, and the Grafana Cloud references all live here:

project_name: clickykeys
letsencrypt_env: staging          # switch to "prod" once everything works
nginx_replicas: 1

# GHCR image source (CI pushes here as ghcr.io/<owner>/<image>:<short-sha>)
ghcr_owner: <your-github-user>
image_web_name: clickykeys-web
image_php_name: clickykeys-php

mysql_root_password: "{{ vault_mysql_root_password }}"

environments:
  - name: production
    namespace: "{{ vault_prod_namespace }}"
    domain:    "{{ vault_prod_domain }}"
    email:     "{{ vault_prod_email }}"
    mysql_database:      "{{ vault_prod_mysql_database }}"
    mysql_user_name:     "{{ vault_prod_mysql_user_name }}"
    mysql_user_password: "{{ vault_prod_mysql_user_password }}"
    web_image_tag: latest
    php_image_tag: latest
    php_replicas: 2

  - name: staging
    namespace: "{{ vault_staging_namespace }}"
    domain:    "{{ vault_staging_domain }}"
    email:     "{{ vault_staging_email }}"
    mysql_database:      "{{ vault_staging_mysql_database }}"
    mysql_user_name:     "{{ vault_staging_mysql_user_name }}"
    mysql_user_password: "{{ vault_staging_mysql_user_password }}"
    web_image_tag: latest
    php_image_tag: latest
    php_replicas: 1

# Grafana Cloud (values pulled from the encrypted vault)
grafana_cloud_metrics_url:      "{{ vault_grafana_cloud_metrics_url }}"
grafana_cloud_metrics_username: "{{ vault_grafana_cloud_metrics_username }}"
grafana_cloud_logs_url:         "{{ vault_grafana_cloud_logs_url }}"
grafana_cloud_logs_username:    "{{ vault_grafana_cloud_logs_username }}"
grafana_cloud_api_key:          "{{ vault_grafana_cloud_api_key }}"

4. Fill in and encrypt the Ansible Vault

The vault stores everything sensitive: per-environment domains, emails, MySQL credentials, and Grafana Cloud keys.

Edit ansible/group_vars/all/vault.yml with your real values:

vault_mysql_root_password: <strong-root-password>

# ── Production ──
vault_prod_namespace: clickykeys
vault_prod_domain:    clickykeys.example.com
vault_prod_email:     you@example.com
vault_prod_mysql_database:      clickykeys
vault_prod_mysql_user_name:     clickykeys_app
vault_prod_mysql_user_password: <strong-prod-password>

# ── Staging (must differ from production) ──
vault_staging_namespace: staging
vault_staging_domain:    staging.example.com
vault_staging_email:     you@example.com
vault_staging_mysql_database:      clickykeys_staging
vault_staging_mysql_user_name:     clickykeys_staging
vault_staging_mysql_user_password: <strong-staging-password>

# ── Grafana Cloud ──
vault_grafana_cloud_metrics_url:      <prometheus-remote-write-url>
vault_grafana_cloud_metrics_username: <metrics-user-id>
vault_grafana_cloud_logs_url:         <loki-push-url>
vault_grafana_cloud_logs_username:    <logs-user-id>
vault_grafana_cloud_api_key:          <grafana-cloud-api-key>

# Encrypt the file with Ansible Vault
ansible-vault encrypt ansible/group_vars/all/vault.yml

You will be prompted for a vault password — remember it; you will need it every time you run the playbook. To edit the encrypted file later use ansible-vault edit ansible/group_vars/all/vault.yml.

5. Provision the EC2 instance with Terraform

cd terraform
terraform init
terraform apply

Terraform creates the EC2 instance (20 GB gp3 root volume), security group, and SSH key pair, and writes ansible/inventory.ini with the new public IP. Point your production and staging DNS A records at this IP before switching to prod Let's Encrypt.

6. Run the Ansible playbook

cd ../ansible
ansible-playbook -i inventory.ini playbook.yml --ask-vault-pass

The host-wide roles run once (swap → k3s → helm → mysql → k8s_cluster → monitoring), then the k8s_env role is applied once per environment in the environments list. To limit the run to a subset of environments, override target_envs:

ansible-playbook -i inventory.ini playbook.yml --ask-vault-pass \
  -e 'target_envs=["staging"]'

7. Build & push the application images

In normal operation CI does this for you — every push to main builds and pushes both images to GHCR (see CI/CD below). To build them manually:

# Frontend (nginx:alpine + static site)
docker build -t ghcr.io/<your-user>/clickykeys-web:latest ./website
docker push  ghcr.io/<your-user>/clickykeys-web:latest

# Backend (php-fpm)
docker build -t ghcr.io/<your-user>/clickykeys-php:latest ./php-fpm
docker push  ghcr.io/<your-user>/clickykeys-php:latest

8. Verify the deployment

ssh ubuntu@<public_ip>

sudo kubectl get pods --all-namespaces
sudo kubectl get ingress  -A
sudo kubectl get certificate -A

Each environment's site goes live at its configured domain once cert-manager finishes the ACME HTTP-01 challenge (typically 1–2 minutes on prod). You can also run the smoke check locally:

BASE_URL=https://clickykeys.example.com ./scripts/smoke-check.sh

---

CI/CD

The pipeline lives in .github/workflows/build.yml and runs on every push to main (and via workflow_dispatch).

1. phpunit_test — runs PHPUnit against ./php-fpm on PHP 8.2 with cached Composer deps.
2. images_build — builds and pushes the web and php images to GHCR, tagged with both the short commit SHA and latest, using GitHub Actions layer caching. The php build step refreshes the DB-IP IP-to-Country Lite GeoIP database before building.
3. deploy-staging — SSHes to the host and kubectl set image on the staging namespace (scaling it up from 0 first).
4. smoke-test-staging — GET /api/healthz.php; must return HTTP 200 with {"status":"ok"}.
5. deploy-production — kubectl set image on the production namespace and records a kubernetes.io/change-cause annotation (tag, commit, run URL) for auditable rollout history.
6. smoke-test-production — same health check; on failure it automatically runs kubectl rollout undo on both deployments to restore the previous ReplicaSet.
7. turn-off-staging — scales the staging deployments back to 0 replicas to save resources.

Required GitHub Actions secrets: PROD_HOST, PROD_SSH_KEY, PROD_NAMESPACE, STAGING_URL, PROD_URL (plus the built-in GITHUB_TOKEN for GHCR).

---

Observability

The monitoring Ansible role installs the grafana/k8s-monitoring Helm chart (chart major ^4) into a dedicated monitoring namespace. It deploys Grafana Alloy to collect cluster metrics and pod logs and remote-writes them to Grafana Cloud. All endpoints, usernames, and the API key are templated from Ansible Vault (grafana_cloud_*), so no credentials are stored in plaintext. The role waits for the alloy-metrics StatefulSet to become ready before completing.

---

Configuration Reference

Terraform — terraform/terraform.tfvars

Variable	Description
aws_region	AWS region to deploy into
project_name	Name tag / prefix applied to AWS resources (SG, key pair)
ami_id	AMI ID (Ubuntu 24.04 LTS recommended)
instance_type	EC2 instance type (default t3.medium)
ssh_public_key_path	Path to your SSH public key
ssh_private_key_path	Path to your SSH private key (used by Ansible)

Ansible — ansible/group_vars/all/all.yml

Variable	Description
project_name	Used in resource naming and tags
letsencrypt_env	staging (default) or prod
nginx_replicas	Number of nginx-web pod replicas
ghcr_owner	GHCR owner/namespace that images are pushed under
image_web_name / image_php_name	Image names for the frontend / backend
mysql_root_password	Provisioning-only root password (from vault)
environments[]	List of environments; each entry defines name, namespace, domain, email, mysql_*, web_image_tag, php_image_tag, php_replicas (vault-backed)
grafana_cloud_*	Grafana Cloud metrics/logs endpoints, usernames, API key (from vault)

Ansible Vault — ansible/group_vars/all/vault.yml (encrypted)

Variable	Description
vault_mysql_root_password	Shared MySQL root password (provisioning only)
vault_prod_*	Production namespace, domain, email, and MySQL database/user/password
vault_staging_*	Staging namespace, domain, email, and MySQL database/user/password
vault_grafana_cloud_metrics_url / _username	Grafana Cloud Prometheus remote-write endpoint + user
vault_grafana_cloud_logs_url / _username	Grafana Cloud Loki push endpoint + user
vault_grafana_cloud_api_key	Grafana Cloud API key

---

Security

• TLS everywhere. Public traffic is terminated at ingress-nginx and certificates are issued and renewed automatically by cert-manager via Let's Encrypt.
• Locked-down security group. Only ports 22, 80, and 443 are exposed publicly.
• Key-based SSH only. No password authentication.
• Secrets are never committed. .gitignore covers *.tfvars, inventory.ini, the real all/ group_vars, vault.yml, *.pem, *.key, and .env. Sensitive Ansible variables live in an Ansible Vault encrypted file.
• Kubernetes Secrets are created at deploy time with kubectl create secret … --dry-run=client -o yaml | kubectl apply -f - (with no_log: true) so plaintext values are never written to disk.
• Environment isolation. Production and staging run in separate namespaces with distinct databases, users, and credentials.
• Auditable rollouts. Production deployments are annotated with the image tag, commit, and CI run URL, and an automatic rollback restores the previous ReplicaSet if the post-deploy smoke test fails.
• Privacy-respecting analytics. First-party, cookieless visit stats with IP anonymization and GeoIP country lookup (see docs/privacy-analytics.md).

---

Branch Status — main (current) vs docker-legacy

Aspect	main (k3s, current production — this branch)	docker-legacy (Docker Compose, legacy)
Orchestration	Kubernetes (k3s)	Docker Compose
Provisioning	Terraform (AWS EC2)	Manual
Configuration	Ansible (fully automated, idempotent roles)	.env file, manual docker compose up
Environments	Production + staging (isolated namespaces)	Single
CI/CD	GitHub Actions (test → build → staging → prod → auto-rollback)	None
Ingress / proxy	ingress-nginx via Helm	nginx-proxy container
TLS	cert-manager (automatic renewal)	Let's Encrypt via acme-companion
Scaling	kubectl scale / replica count	docker compose scale
Secrets	Ansible Vault + Kubernetes Secret resources	.env files
Observability	Grafana Cloud (Alloy: metrics + logs)	Prometheus + cAdvisor + Grafana (in-stack)
Portability	Any Kubernetes / k3s cluster	Any Docker host

The main branch (this one) is the active production deployment. The docker-legacy branch remains in the repository as a legacy reference of the original Docker Compose setup that powered ClickyKeys.

---

Related Projects

• ClickyKeys production page — clickykeys.fun
• ClickyKeys application — GitHub · Microsoft Store
• ClickyKeys infrastructure (legacy, Docker Compose) — docker-legacy branch

---

Author

Mateusz Wyrzykowski — github.com/Reksaku

---

License

This project is part of a personal portfolio.

This README was co-authored with Claude (Anthropic).