fix: don't fabricate time_to_ready_ms when rediscovering an already-ready container

[dray92]

Why

Fixes the root cause investigated in devzero-inc/services#8891: container_startup_lifecycles rows in ClickHouse showed implausibly large time_to_ready_ms values (up to 58+ hours) for long-lived, stable pods, while time_to_running_ms for the same containers stayed correct. This corrupted avgTimeToReadyMs aggregates consumed by both the existing ContainerHealthService/GetWorkloadHealthSummaries frontend RPC and the new startup-timing fields streamed to the tunr WorkloadProfile CRD (devzero-inc/services#8886).

Root cause

trackStartupLifecycle creates a fresh tracker entry whenever it observes a (podUID, containerName, restartCount) key with no live entry, seeding pendingAt from the pod's PodScheduled condition — which can be days old for a long-lived pod. If the container is already Running and Ready in that same observation (no genuine watched transition through Pending -> ContainerCreating -> Running -> Ready), the code computed time_to_ready_ms = now.Sub(pendingAt) and emitted immediately, then deleted the entry. This guarantees the next unrelated status update for the same stable pod (a routine kubelet condition/heartbeat refresh, not necessarily a collector restart) repeats the exact cycle — recreating the entry, immediately matching Ready again, and computing an even larger, wrong duration each time.

Confirmed against live preprod ClickHouse data: of rows in the last 7 days, 10,953 had an implausible time_to_ready_ms (>10 min) — 100% of them also had container_creating_at IS NULL (never observed the ContainerCreating transition), and zero rows with container_creating_at populated showed the anomaly. That's the exact signature of this bug.

Fix

When creating a new tracker entry, skip it entirely if the container is already Running and Ready at first observation — there is no reliable data available to compute a real duration in that case. That container's genuine startup was either already captured correctly by snapshotStartupLifecycles (which uses real historical condition timestamps, not now(), specifically for containers observed as already-running during the initial informer cache sync), or was already correctly reported by an earlier real transition and this is a stale rediscovery.

Test plan

• New test TestTrackStartupLifecycle_DoesNotFabricateReadyDurationForStaleRediscovery reproduces the bug against the pre-fix code (confirmed: produced time_to_ready_ms=172800000 — exactly 48h — for a pod scheduled 48h earlier) and passes against the fix (no event emitted, no tracker entry left behind).
• New control test TestTrackStartupLifecycle_EmitsCorrectDurationForGenuineFreshStartup proves a genuine Pending -> ContainerCreating -> Running -> Ready transition (the normal, working path) still correctly emits a small, real duration — this fix doesn't affect legitimate fresh startups.
• go build ./..., go vet ./internal/collector/..., gofmt -l all clean.
• go test ./internal/... passes (full suite, excluding test/e2e which requires a live kind cluster + cert-manager/prometheus-operator not available in this environment — pre-existing, unrelated to this change).
• golangci-lint not run locally (repeatedly OOM-killed in the sandbox this was developed in, even scoped to a single package) — please confirm it passes in CI.

[gitar-bot]

Code Review ✅ Approved

Prevents the fabrication of implausible time_to_ready_ms values for long-lived containers by skipping tracking when a container is already running and ready upon observation.

_{Was this helpful? React with 👍 / 👎 | Gitar}