Skip to content

Kubernetes Native Operator for NeuralBudget #4

Open
Open
Kubernetes Native Operator for NeuralBudget#4

Description

@pristley
pristley
opened on Jun 27, 2026

Feature Spec: Kubernetes Native Operator for NeuralBudget

1. Overview

Implement a Kubernetes Operator to manage NeuralBudget resources natively via CRDs, enabling SLO DAGs, aggregators, and policies to be deployed through kubectl apply instead of Python scripts.

2. Problem Statement

  • Manual Python scripts for deployments (error-prone, not auditable)
  • No native K8s integration (can't use standard tooling)
  • SLO DAGs treated as configuration, not resources
  • No automatic reconciliation on changes
  • Difficult to manage multi-cluster deployments
  • RBAC/audit trail limitations

3. Goals

  1. Native K8s Integration: Manage all NeuralBudget resources via kubectl
  2. GitOps Ready: Use standard tools (ArgoCD, Flux) for deployment
  3. Declarative: YAML-based SLO/DAG definitions
  4. Auto-Reconciliation: Detect drift, auto-heal
  5. Multi-Cluster: Federated deployments across clusters
  6. RBAC: Fine-grained access control
  7. Audit Trail: Full deployment history

4. Architecture

User/GitOps → kubectl apply ↓
┌────────────────────────────────────────────────┐
│ Kubernetes API Server                          │
│ ├─ SLODag CRD                                  │
│ ├─ MetricsAggregator CRD                       │
│ ├─ AlertingPolicy CRD                          │
│ └─ NotificationChannel CRD                     │
└────────────────────────────────────────────────┘
        ↓ Watch/List
┌────────────────────────────────────────────────┐
│ NeuralBudget Operator (Controller)             │
│ ├─ SLODag Reconciler                           │
│ ├─ MetricsAggregator Reconciler                │
│ ├─ AlertingPolicy Reconciler                   │
│ └─ Status Syncer                               │
└────────────────────────────────────────────────┘
        ↓ Create/Update
┌────────────────────────────────────────────────┐
│ NeuralBudget Control Plane                     │
│ (gRPC/REST APIs)                               │
└────────────────────────────────────────────────┘

5. Custom Resource Definitions (CRDs)

5.1 SLODag CRD

apiVersion: neuralbudget.io/v1
kind: SLODag
metadata:
  name: payment-service-slo
  namespace: observability
  labels:
    service: payment-service
spec:
  # DAG Definition
  nodes:
    - id: "latency_check"
      type: "metric_query"
      config:
        metric: "http_request_duration_seconds"
        selectors:
          service: "payment-service"
        percentile: 99
        
    - id: "error_rate_check"
      type: "metric_query"
      config:
        metric: "http_requests_total"
        selectors:
          service: "payment-service"
          status: "5xx"
        rate_window: "5m"
        
    - id: "availability_calc"
      type: "arithmetic"
      config:
        expression: "(1 - error_rate_check) * 100"
        
    - id: "slo_validator"
      type: "threshold"
      config:
        value_source: "availability_calc"
        operator: "gte"
        threshold: 99.5
        
    - id: "alert_on_breach"
      type: "alert"
      config:
        enabled: true
        severity: "warning"
        notification_channels:
          - "slack-platform"
          - "pagerduty-oncall"

  # DAG Edges (dependencies)
  edges:
    - from: "latency_check"
      to: "slo_validator"
    - from: "error_rate_check"
      to: "availability_calc"
    - from: "availability_calc"
      to: "slo_validator"
    - from: "slo_validator"
      to: "alert_on_breach"

  # Scheduling
  schedule:
    interval: "1m"
    timeout: "30s"
    
  # SLO Targets
  sloTargets:
    - metric: "availability"
      target: 99.5
      window: "30d"
    - metric: "latency_p99"
      target: 200
      window: "7d"
      
  # Notification Channels
  notificationChannels:
    - name: "slack-platform"
      type: "slack"
      config:
        webhookUrl: "secret:slack-webhook"
        channel: "#alerts"
        
    - name: "pagerduty-oncall"
      type: "pagerduty"
      config:
        integrationKey: "secret:pagerduty-key"
        severity: "critical"

  # Persistence
  persistence:
    enabled: true
    dataStore: "timescaledb"
    retentionDays: 30

  # Labels for query routing
  selector:
    service: "payment-service"
    cluster: "us-east-1"

status:
  phase: "Active"
  lastReconciliation: "2024-01-15T10:30:00Z"
  observedGeneration: 42
  conditions:
    - type: "Ready"
      status: "True"
      lastTransitionTime: "2024-01-15T10:30:00Z"
    - type: "Valid"
      status: "True"

5.2 MetricsAggregator CRD

apiVersion: neuralbudget.io/v1
kind: MetricsAggregator
metadata:
  name: central-aggregator
  namespace: observability
spec:
  # Cluster topology
  replicas: 4
  shards: 4
  
  # Pod spec
  template:
    spec:
      containers:
      - name: aggregator
        image: neuralbudget/aggregator:v1.0.0
        ports:
        - name: grpc
          containerPort: 4317
        - name: http
          containerPort: 8080
        resources:
          requests:
            cpu: 4
            memory: 8Gi
          limits:
            cpu: 8
            memory: 16Gi
        env:
        - name: AGGREGATION_WINDOW
          value: "3600"
        - name: RETENTION_DAYS
          value: "7"
          
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values: ["aggregator"]
              topologyKey: kubernetes.io/hostname
              
  # Networking
  service:
    type: ClusterIP
    ports:
    - name: grpc
      port: 4317
      protocol: TCP
    - name: http
      port: 8080
      protocol: TCP
      
  # Storage
  storage:
    rocksdb:
      size: 100Gi
      storageClassName: "fast-ssd"
    
  # Scaling policy
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 16
    targetMetrics:
      - type: "throughput"
        targetAverageValue: "250k"  # 250k metrics/sec per node
        
  # Monitoring
  monitoring:
    enabled: true
    serviceMonitor:
      enabled: true
      interval: "30s"
      
  # Security
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsReadOnlyRootFilesystem: true

status:
  ready: true
  replicas: 4
  throughputPerSec: 1000000
  healthyNodes: ["aggregator-0", "aggregator-1", "aggregator-2", "aggregator-3"]

5.3 AlertingPolicy CRD

apiVersion: neuralbudget.io/v1
kind: AlertingPolicy
metadata:
  name: payment-service-alerts
  namespace: observability
spec:
  # Disabled flag
  disabled: false
  
  # Rules
  rules:
    - name: "HighLatencyWarning"
      condition: "latency_p99 > 200"
      duration: "5m"
      severity: "warning"
      
    - name: "ErrorRateCritical"
      condition: "error_rate > 0.05"
      duration: "2m"
      severity: "critical"
      
    - name: "SLOBreach"
      condition: "availability < 99.5"
      duration: "1m"
      severity: "critical"
  
  # Notification routing
  routes:
    - match:
        severity: "critical"
      receivers: ["pagerduty-oncall", "slack-critical"]
      groupWait: "10s"
      groupInterval: "5m"
      repeatInterval: "1h"
      
    - match:
        severity: "warning"
      receivers: ["slack-platform"]
      groupWait: "30s"
      repeatInterval: "4h"
  
  # Receivers (notification channels)
  receivers:
    - name: "pagerduty-oncall"
      type: "pagerduty"
      config:
        integrationKeyRef: "pagerduty-key"
        
    - name: "slack-critical"
      type: "slack"
      config:
        webhookUrlRef: "slack-critical-webhook"
        
  # Silences
  silences:
    - matchers:
        - name: "alertname"
          value: "HighLatencyWarning"
      startsAt: "2024-01-15T00:00:00Z"
      endsAt: "2024-01-15T23:59:59Z"  # Maintenance window

status:
  rulesEvaluated: 3
  rulesActive: 3
  firedAlerts: 1
  lastEvaluationTime: "2024-01-15T10:30:00Z"

5.4 NotificationChannel CRD

apiVersion: neuralbudget.io/v1
kind: NotificationChannel
metadata:
  name: slack-platform
  namespace: observability
spec:
  type: "slack"
  
  # Credentials (referencing K8s secrets)
  credentials:
    webhookUrl:
      secretKeyRef:
        name: slack-credentials
        key: webhook-url
    botToken:
      secretKeyRef:
        name: slack-credentials
        key: bot-token
        
  config:
    channel: "#platform-alerts"
    username: "NeuralBudget"
    iconEmoji: ":bell:"
    
  # Rate limiting
  rateLimit:
    maxPerMinute: 10
    
  # Verification
  verification:
    enabled: true
    interval: "5m"

status:
  ready: true
  lastVerification: "2024-01-15T10:30:00Z"
  errorCount: 0

6. Operator Implementation

6.1 Controller Pattern

// Pseudocode for SLODag reconciler
pub struct SLODagReconciler {
    k8s_client: kube::Client,
    nb_client: neuralbudget::Client,
}

impl Reconciler for SLODagReconciler {
    async fn reconcile(&self, dag: SLODag) -> Result<Action> {
        // 1. Validate DAG spec
        self.validate_dag_spec(&dag)?;
        
        // 2. Compile to internal representation
        let compiled = compile_dag(&dag)?;
        
        // 3. Deploy to NeuralBudget control plane
        self.nb_client.deploy_dag(&compiled).await?;
        
        // 4. Create ConfigMap with compiled DAG
        self.k8s_client
            .create_configmap(&dag.name, &compiled)
            .await?;
        
        // 5. Update status
        self.update_status(&dag, Status::Active).await?;
        
        // Requeue after 5m for periodic validation
        Ok(Action::requeue(Duration::from_secs(300)))
    }
    
    async fn cleanup(&self, dag: SLODag) -> Result<()> {
        // Delete from NeuralBudget control plane
        self.nb_client.delete_dag(&dag.name).await?;
        
        // Delete ConfigMap
        self.k8s_client.delete_configmap(&dag.name).await?;
        
        Ok(())
    }
}

6.2 Reconciliation Loop

Watch SLODag Resources
        ↓
Detect Create/Update/Delete Events
        ↓
Validate YAML Spec (DAG syntax, references)
        ↓
Compile to Internal Representation
        ↓
Persist ConfigMap (for auditing)
        ↓
Deploy to NeuralBudget Control Plane (gRPC)
        ↓
Update Status (Ready/Error)
        ↓
Watch for Drift (compare deployed vs desired)
        ↓
Auto-Heal (redeploy if drift detected)

7. Installation

7.1 Install Operator

# Add Helm repository
helm repo add neuralbudget https://charts.neuralbudget.io
helm repo update

# Install operator
helm install neuralbudget-operator neuralbudget/operator \
  --namespace observability \
  --create-namespace \
  --values values.yaml

# Verify installation
kubectl get pods -n observability
kubectl get crd | grep neuralbudget

7.2 Helm Values

# values.yaml
operator:
  image:
    repository: neuralbudget/operator
    tag: v1.0.0
  
  replicas: 2
  
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi
      
  # RBAC
  serviceAccount:
    create: true
    name: neuralbudget-operator
    
  # Webhook (mutation/validation)
  webhook:
    enabled: true
    port: 9443
    
  # Logging
  logging:
    level: info
    
  # Control plane connection
  controlPlane:
    address: "neuralbudget-control-plane.observability.svc.cluster.local:6379"
    tls:
      enabled: true
      caSecret: "neuralbudget-ca"

8. Usage Examples

8.1 Deploy SLO DAG

# Create namespace
kubectl create namespace observability

# Apply SLO DAG
kubectl apply -f slo-dag.yaml

# Check status
kubectl get slodags -n observability
kubectl describe slodags payment-service-slo -n observability

# View logs
kubectl logs -n observability deployment/neuralbudget-operator -f

# Delete SLO DAG (triggers cleanup)
kubectl delete slodags payment-service-slo -n observability

8.2 GitOps Workflow (ArgoCD)

# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: neuralbudget-slos
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/company/observability-config
    targetRevision: main
    path: neuralbudget/slos
    
  destination:
    server: https://kubernetes.default.svc
    namespace: observability
    
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
# Deploy with ArgoCD
argocd app create neuralbudget-slos \
  --repo https://github.com/company/observability-config \
  --path neuralbudget/slos \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace observability

8.3 Multi-Cluster Deployment

# kustomization.yaml for multi-cluster
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - slo-dag-base.yaml

patches:
  - target:
      kind: SLODag
    patch: |-
      - op: replace
        path: /metadata/name
        value: payment-service-slo-${CLUSTER_NAME}
      - op: replace
        path: /spec/selector/cluster
        value: ${CLUSTER_NAME}

configMapGenerator:
  - name: cluster-config
    literals:
      - cluster=${CLUSTER_NAME}
      - region=${REGION}

9. RBAC & Access Control

9.1 ClusterRole for SLODag Admins

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: slodag-admin
rules:
- apiGroups: ["neuralbudget.io"]
  resources: ["slodags", "alertingpolicies"]
  verbs: ["create", "update", "patch", "delete", "get", "list", "watch"]
- apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: slodag-viewer
rules:
- apiGroups: ["neuralbudget.io"]
  resources: ["slodags", "alertingpolicies"]
  verbs: ["get", "list", "watch"]

9.2 Service Account Binding

apiVersion: v1
kind: ServiceAccount
metadata:
  name: platform-team
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: platform-team-slodag-admin
  namespace: observability
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: slodag-admin
subjects:
- kind: ServiceAccount
  name: platform-team
  namespace: observability

10. Validation & Webhooks

10.1 Validating Webhook

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: slodag-validator
webhooks:
- name: slodag.neuralbudget.io
  clientConfig:
    service:
      name: neuralbudget-operator
      namespace: observability
      path: "/validate-slodag"
    caBundle: <base64-ca>
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["neuralbudget.io"]
    apiVersions: ["v1"]
    resources: ["slodags"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail

10.2 Validation Logic

pub fn validate_slodag(dag: &SLODag) -> Result<()> {
    // Check DAG acyclicity
    if has_cycles(&dag.nodes, &dag.edges) {
        return Err("DAG contains cycles");
    }
    
    // Validate all metric queries
    for node in &dag.nodes {
        if node.type == "metric_query" {
            validate_metric_query(&node.config)?;
        }
    }
    
    // Validate notification channels exist
    for channel in &dag.notification_channels {
        validate_notification_channel(channel)?;
    }
    
    // Check SLO targets are reasonable
    for target in &dag.slo_targets {
        if target.target < 0.0 || target.target > 100.0 {
            return Err("SLO target must be 0-100");
        }
    }
    
    Ok(())
}

11. Status & Conditions

11.1 Status Fields

Phase: Pending → Active → Error → Terminating
Conditions:
  - Ready (all dependencies satisfied)
  - Valid (passed validation)
  - Deployed (successfully deployed to control plane)
  - Reconciling (currently processing)

11.2 Status Update

status:
  phase: "Active"
  observedGeneration: 42
  lastReconciliation: "2024-01-15T10:30:00Z"
  conditions:
  - type: "Ready"
    status: "True"
    lastTransitionTime: "2024-01-15T10:30:00Z"
    reason: "ReconciliationSucceeded"
  - type: "Valid"
    status: "True"
  - type: "Deployed"
    status: "True"
    message: "Successfully deployed to control plane"

12. Monitoring & Observability

12.1 Operator Metrics

neuralbudget_operator_reconciliations_total{resource_type, result}
neuralbudget_operator_reconciliation_duration_seconds{quantile, resource_type}
neuralbudget_operator_resources_created_total{resource_type}
neuralbudget_operator_resources_deleted_total{resource_type}
neuralbudget_operator_validation_failures_total{resource_type, reason}
neuralbudget_operator_webhooks_calls_total{type, result}

12.2 PrometheusRule

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: neuralbudget-operator
spec:
  groups:
  - name: neuralbudget.operator
    interval: 30s
    rules:
    - alert: OperatorReconciliationFailed
      expr: increase(neuralbudget_operator_reconciliations_total{result="error"}[5m]) > 0
      for: 5m
      annotations:
        summary: "NeuralBudget Operator reconciliation failed"
        
    - alert: OperatorWebhookDown
      expr: up{job="neuralbudget-operator-webhook"} == 0
      for: 1m
      annotations:
        summary: "NeuralBudget validation webhook is down"

13. Implementation Roadmap

Phase 1 (Weeks 1-3): Core operator scaffolding

  • Operator SDK setup (Rust/Go)
  • CRD definitions (SLODag, MetricsAggregator)
  • Basic reconciliation loop

Phase 2 (Weeks 4-6): Reconcilers

  • SLODag reconciler
  • AlertingPolicy reconciler
  • Status update logic

Phase 3 (Weeks 7-9): Webhooks & validation

  • Validating webhooks
  • Mutation webhooks
  • CRD schema validation

Phase 4 (Weeks 10-12): Advanced features

  • Multi-cluster support
  • Drift detection & healing
  • Garbage collection

Phase 5 (Weeks 13-16): Operations & tooling

  • Helm chart
  • Integration tests
  • Documentation

14. Configuration Management

14.1 ConfigMap for Static Config

apiVersion: v1
kind: ConfigMap
metadata:
  name: neuralbudget-config
  namespace: observability
data:
  control-plane-address: "neuralbudget-control-plane.observability.svc.cluster.local:6379"
  default-retention-days: "30"
  default-aggregation-window: "3600"
  metrics-export-interval: "60s"
  webhook-timeout: "10s"

14.2 Secret for Credentials

apiVersion: v1
kind: Secret
metadata:
  name: slack-credentials
  namespace: observability
type: Opaque
stringData:
  webhook-url: "https://hooks.slack.com/services/..."
  bot-token: "xoxb-..."

15. Failover & HA

15.1 Operator High Availability

# Multiple operator replicas with leader election
spec:
  replicas: 3
  
  # Leader election
  leaderElection:
    enabled: true
    namespace: observability
    name: neuralbudget-operator-leader

15.2 Resource Cleanup

If Operator Pod Dies:
  ↓
Leader Election triggers new leader
  ↓
Lease acquired by standby replica
  ↓
New leader takes over reconciliation
  ↓
Existing SLODags continue operating
  ↓
No data loss, minimal interruption

16. Migration from Python Scripts

16.1 Migration Steps

1. Generate CRDs from existing Python configs
   (automated conversion tool)

2. Test generated CRDs in non-prod cluster
   (validate logic matches)

3. Gradual rollout to production
   (run both in parallel for 1-2 weeks)

4. Switchover (point traffic to K8s Operator)

5. Decommission Python scripts

16.2 Conversion Tool

def python_config_to_slodag_crd(config: Dict) -> str:
    """Convert legacy Python SLO config to K8s CRD YAML"""
    
    dag_dict = {
        "apiVersion": "neuralbudget.io/v1",
        "kind": "SLODag",
        "metadata": {"name": config["name"]},
        "spec": {
            "nodes": convert_nodes(config["nodes"]),
            "edges": convert_edges(config["edges"]),
            "schedule": {"interval": config["interval"]},
        }
    }
    
    return yaml.dump(dag_dict)

17. Troubleshooting

17.1 Common Issues

Issue: SLODag stuck in "Pending" phase
Fix: Check operator logs for validation errors
     kubectl logs -n observability deployment/neuralbudget-operator

Issue: Webhook timeout on SLODag creation
Fix: Scale operator replicas
     kubectl scale deploy neuralbudget-operator --replicas=3

Issue: Status not updating
Fix: Check operator RBAC permissions
     kubectl auth can-i update slodags.status

17.2 Debug Commands

# Check all CRDs
kubectl api-resources | grep neuralbudget

# Inspect SLODag details
kubectl get slodags -o yaml payment-service-slo

# Check operator events
kubectl describe slodag payment-service-slo

# Stream operator logs
kubectl logs -f deploy/neuralbudget-operator -n observability

# Check webhook status
kubectl get validatingwebhookconfigurations slodag-validator

18. Security Best Practices

# Pod Security Policy
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: neuralbudget-operator
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  runAsNonRoot: true
  runAsUser:
    rule: 'MustRunAsNonRoot'
  seLinux:
    rule: 'MustRunAs'
  fsGroup:
    rule: 'MustRunAs'
  readOnlyRootFilesystem: true

19. Cost Analysis

Operator Overhead:
  2 replicas × 250m CPU = 0.5 cores
  2 replicas × 512Mi RAM = 1GB
  Monthly cost: ~$5 (negligible vs aggregators)

Benefit:
  Eliminates manual script deployment (~2 hours/week)
  Reduces errors by ~90%
  Enables GitOps (audit trail)
  
ROI: Positive within 2-3 months

20. Success Metrics

  • 100% of SLOs deployed via kubectl (no Python scripts)
  • <5 second SLODag reconciliation time
  • 99.9% webhook availability
  • Zero data loss during operator upgrades
  • Full audit trail of all config changes
  • <30 day operator adoption across all teams

21. Comparison: Before vs After

Aspect              Before (Python)    After (K8s Operator)
────────────────────────────────────────────────────────
Deployment method   Manual scripts     kubectl apply
Audit trail         Limited            Full K8s audit
Rollback            Manual             Instant via git
Multi-cluster       Complex tooling    Native support
RBAC                Custom             Built-in K8s RBAC
GitOps ready        No                 Yes (ArgoCD/Flux)
Learning curve      Python + custom    Standard K8s
Reconciliation      Manual trigger     Auto (drift detection)
Validation          Minimal            Comprehensive webhooks
Status tracking     Logs only          K8s resource status
HA & failover       Manual             Automatic

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions