Kubernetes Off Hours Staging Resource Optimization POC

Project Overview

This POC implements an automated Kubernetes resource optimization system that reduces staging environment costs by 60-80% during off-hours while maintaining seamless developer experience through GitOps workflows and Slack integration.

Architecture

graph TB
    A[ArgoCD] --> B[kube-green Operator]
    B --> C[Staging Workloads]
    B --> D[Slack Webhook]
    D --> E[Slack Channel]
    E --> F[Manual Override Buttons]
    F --> G[GitLab Webhook]
    G --> A
    
    H[Prometheus] --> I[Grafana Dashboards]
    H --> J[AlertManager]
    B --> H
    
    style A fill:#326CE5
    style B fill:#00D4AA
    style D fill:#4A154B
    style H fill:#E6522C

Loading

Quick Start

Prerequisites

• Kubernetes cluster (1.20+)
• ArgoCD installed
• GitLab repository with CI/CD enabled
• Slack workspace with webhook permissions

Deployment

1. Clone the repository

   git clone <repository-url>
   cd k8s-resource-optimization

2. Configure environment variables

   cp .env.example .env
   # Edit .env with your configuration

3. Deploy using ArgoCD

   ./scripts/deploy.sh

4. Configure Slack integration

   ./scripts/setup-slack.sh

📁 Repository Structure

k8s-resource-optimization/
├── apps/                     # ArgoCD applications
│   ├── argocd/              # App-of-apps configuration
│   ├── kube-green/          # kube-green operator configs
│   ├── slack-integration/   # Slack webhook service
│   └── monitoring/          # Monitoring stack
├── charts/                  # Helm charts
├── docs/                    # Documentation
├── scripts/                 # Automation scripts
└── tests/                   # Test suites

Business Flow

1. Scheduled Automation: kube-green triggers based on CronJob schedules
2. Resource Sleep/Wake: Workloads scaled down/up automatically
3. Slack Notifications: Real-time status updates
4. Manual Override: Emergency sleep/wake via Slack buttons
5. GitOps Integration: All changes tracked in Git
6. Audit Trail: Complete history of all actions

Cost Optimization

• Weekday Off-hours: 6 PM - 8 AM (14 hours = 58% daily savings)
• Weekends: Complete shutdown (48 hours = 100% weekend savings)
• Holidays: Custom schedules
• Overall Projection: 60-80% cost reduction

Components

kube-green Operator

• Kubernetes-native resource management
• Custom Resource Definitions (CRDs)
• Time-based scheduling
• Namespace-aware operations

Slack Integration

• Interactive webhook endpoints
• Real-time notifications
• Manual override capabilities
• User permission management

Monitoring Stack

• Prometheus metrics collection
• Grafana visualization dashboards
• AlertManager for system health
• Custom cost tracking metrics

GitOps Workflow

• ArgoCD for continuous delivery
• Multi-environment management
• Automated sync policies
• Configuration drift detection

Documentation

• Architecture Documentation
• Deployment Guide
• Troubleshooting Guide
• API Documentation
• Slack Bot Usage

Testing

# Run integration tests
./scripts/test-integration.sh

# Run end-to-end tests
./scripts/test-e2e.sh

Configuration

Environment Variables

# Kubernetes Configuration
KUBECONFIG=/path/to/kubeconfig
NAMESPACE=kube-green-system

# Slack Configuration
SLACK_BOT_TOKEN=xoxb-your-bot-token
SLACK_SIGNING_SECRET=your-signing-secret
SLACK_CHANNEL=#k8s-optimization

# GitLab Configuration
GITLAB_TOKEN=your-gitlab-token
GITLAB_PROJECT_ID=12345

# ArgoCD Configuration
ARGOCD_SERVER=argocd.example.com
ARGOCD_TOKEN=your-argocd-token

Scheduling Configuration

# Default schedule: Off-hours weekdays + weekends
weekdaySchedule: "0 18 * * 1-5"    # Sleep at 6 PM
weekdayWakeup: "0 8 * * 1-5"       # Wake at 8 AM
weekendSchedule: "0 18 * * 5"       # Sleep Friday 6 PM
weekendWakeup: "0 8 * * 1"          # Wake Monday 8 AM

Monitoring & Alerts

Key Metrics

• Resource utilization before/after optimization
• Cost savings per environment
• Sleep/wake operation success rates
• Manual override frequency
• System component health

Alerting Rules

• Failed sleep/wake operations
• Slack integration connectivity issues
• Unexpected resource scaling events
• High manual override frequency

Security

• RBAC configurations for all components
• Secret management via external operators
• Network policies for service isolation
• Audit logging for compliance
• Encrypted communications

Performance

• Sub-minute response time for manual overrides
• 99.9% scheduling accuracy
• Minimal resource overhead (< 2% cluster resources)
• Graceful degradation during failures

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Update documentation
6. Submit a pull request

Support

• Slack Channel: #k8s-optimization
• GitLab Issues: Create an issue
• Documentation: Wiki
• Runbooks: Troubleshooting Guide

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• kube-green for the core optimization engine
• ArgoCD for GitOps workflows
• Prometheus for monitoring
• Grafana for visualization