NodeReaper

Automatically harvests empty and unhealthy nodes when autoscalers fail to scale down

---

NodeReaper acts as a cost-saving failsafe that detects and removes nodes running only system pods, helping reduce cloud costs when autoscalers miss cleanup opportunities. Like the Grim Reaper, it comes for nodes whose time has come - but only the ones that truly deserve it.

Sometimes autoscalers (Cluster Autoscaler, Karpenter, etc.) fail to scale down expensive nodes, leaving them running with only system pods. NodeReaper acts as a reliable failsafe that detects and removes these "zombie" nodes.

What It Does

1. Scans nodes every 10 minutes (configurable)
2. Skips protected nodes with protection annotations or labels
3. Waits for minimum age before considering nodes for deletion
4. Identifies empty nodes that only run DaemonSet pods
5. Uses Prometheus metrics (optional) to track how long nodes have been empty
6. Removes unhealthy nodes that are unreachable or not ready
7. Cleans stuck finalizers from nodes that won't terminate
8. Sends notifications when nodes are deleted

Quick Start

# Install with Helm
helm install nodereaper oci://ghcr.io/sdaberdaku/charts/nodereaper \
  --namespace nodereaper \
  --create-namespace

Prometheus Integration (Optional)

Enable Prometheus integration to track node empty duration more accurately:

# values.yaml - With authentication
prometheus:
  enabled: true
  url: "http://prometheus-server:9090"
  minEmptyDuration: "10m"  # Only delete nodes empty for 10+ minutes
  authentication:
    enabled: true
    existingSecret:
      name: "prometheus-credentials"
      usernameKey: "username"
      passwordKey: "password"

# values.yaml - Without authentication (for internal/development use)
prometheus:
  enabled: true
  url: "http://prometheus-server:9090"
  minEmptyDuration: "10m"
  auth:
    enabled: false  # No authentication required

When enabled, NodeReaper queries Prometheus to determine how long nodes have been empty, providing more precise deletion timing than age-based logic alone.

Safety Features

• Dry-run mode - Test without actually deleting nodes
• Age protection - Never deletes newly created nodes
• Annotation protection - Respects "do not delete" markers
• Label protection - Skips nodes with protection labels
• Finalizer whitelist - Only removes known-safe finalizers

Development

# Setup
git clone https://github.com/sdaberdaku/nodereaper.git
cd nodereaper
make install-dev

# Testing
make test              # Unit tests
make test-integration  # Integration tests with kind
make test-all          # All tests
make checks            # Quality checks

# Local testing
make setup-cluster     # Set up kind cluster
DRY_RUN=true LOG_LEVEL=DEBUG nodereaper

License

Apache License 2.0 - see LICENSE for details.