Summary
The controller exposes controller-runtime operational metrics, but no OpenSandbox business-capacity metrics. This makes reconcile health visible, while operators still cannot answer how many sandboxes are active, how much warm-pool capacity is allocated or available, or how requested capacity changes over time.
Checked against main@20dfde88.
Operational motivation
These signals are needed for pool sizing, cluster autoscaling and cost planning:
- current BatchSandbox and sandbox Pod counts;
- desired, current, allocated and ready replicas;
- Pool total, allocated, available and updated Pod counts;
- CPU and memory requests represented by total, allocated and available Pool Pods;
- peak, valley and percentile capacity over time.
Peak/valley values should be derived by the telemetry backend from current-state gauges. This proposal does not require a new Prometheus endpoint.
Existing authoritative state
The controller already computes the required business facts:
PoolStatus.Total, Allocated, Available, Updated;BatchSandboxStatus.Replicas, Allocated, Ready, Phase;- the Pool allocator's Pod-to-Sandbox allocation map.
The missing part is exporting those facts as low-cardinality telemetry.
Existing work and non-overlap
Draft PR #1651 follows that prior-art direction with a capacity-only implementation and leaves tracing and controller-runtime metrics unchanged.
Proposed first increment
The exact names are open to maintainer preference. The intended instruments are:
| Signal |
Type |
Attributes |
| Pool Pod count |
Observable Gauge |
namespace, pool_name, `state=total |
| BatchSandbox count |
Observable Gauge |
namespace, phase, `allocation_mode=pool |
| BatchSandbox Pod count |
Observable Gauge |
namespace, `state=desired |
| Pool requested CPU |
Observable Gauge, CPU cores |
namespace, pool_name, `state=total |
| Pool requested memory |
Observable Gauge, bytes |
namespace, pool_name, `state=total |
The resource gauges should use scheduler-equivalent requests from the actual Pod specs, not only the Pool template. Actual CPU and memory usage remains the responsibility of kubelet/cAdvisor and is out of scope.
Ratios such as Pool utilization should be derived from allocated/total instead of exported as another source of truth.
Correctness and cardinality constraints
- no
sandbox_id, Pod name or BatchSandbox name metric attributes;
- include
namespace on all multi-namespace signals;
- telemetry failures must never affect reconciliation;
- no export when OTLP is not configured;
- avoid duplicate cluster totals with multiple Controller replicas/leader election;
- deleted Pools and BatchSandboxes must not leave permanently stale values.
Implementation
Draft PR #1651 implements the first increment over OTLP/HTTP while leaving controller-runtime Prometheus metrics unchanged.
It exports Pool Pod counts, scheduler-equivalent Pool CPU/memory requests, and namespace-aggregated BatchSandbox counts. The exporter runs only on the elected Controller and remains disabled unless an explicit OTLP endpoint is configured. Focused unit tests, the full Controller test suite, lint, build, license verification, and the documentation build pass locally.
Summary
The controller exposes controller-runtime operational metrics, but no OpenSandbox business-capacity metrics. This makes reconcile health visible, while operators still cannot answer how many sandboxes are active, how much warm-pool capacity is allocated or available, or how requested capacity changes over time.
Checked against
main@20dfde88.Operational motivation
These signals are needed for pool sizing, cluster autoscaling and cost planning:
Peak/valley values should be derived by the telemetry backend from current-state gauges. This proposal does not require a new Prometheus endpoint.
Existing authoritative state
The controller already computes the required business facts:
PoolStatus.Total,Allocated,Available,Updated;BatchSandboxStatus.Replicas,Allocated,Ready,Phase;The missing part is exporting those facts as low-cardinality telemetry.
Existing work and non-overlap
sandbox_id, and Pool metrics need a namespace dimension.Draft PR #1651 follows that prior-art direction with a capacity-only implementation and leaves tracing and controller-runtime metrics unchanged.
Proposed first increment
The exact names are open to maintainer preference. The intended instruments are:
namespace,pool_name, `state=totalnamespace,phase, `allocation_mode=poolnamespace, `state=desirednamespace,pool_name, `state=totalnamespace,pool_name, `state=totalThe resource gauges should use scheduler-equivalent requests from the actual Pod specs, not only the Pool template. Actual CPU and memory usage remains the responsibility of kubelet/cAdvisor and is out of scope.
Ratios such as Pool utilization should be derived from allocated/total instead of exported as another source of truth.
Correctness and cardinality constraints
sandbox_id, Pod name or BatchSandbox name metric attributes;namespaceon all multi-namespace signals;Implementation
Draft PR #1651 implements the first increment over OTLP/HTTP while leaving controller-runtime Prometheus metrics unchanged.
It exports Pool Pod counts, scheduler-equivalent Pool CPU/memory requests, and namespace-aggregated BatchSandbox counts. The exporter runs only on the elected Controller and remains disabled unless an explicit OTLP endpoint is configured. Focused unit tests, the full Controller test suite, lint, build, license verification, and the documentation build pass locally.