test: add end-to-end platform test for LGTM pack - #21
Open
tylerpotts wants to merge 25 commits into
Open
Conversation
…action, fresh span ts)
With nebariapp.auth.enabled=true, Grafana depends on the OAuth client Secret the NebariApp operator provisions async after Keycloak realm setup; it does not materialize within CI's window, leaving Grafana in CreateContainerConfigError and the Application Degraded. OAuth login is out of scope for this telemetry test, so disable auth and null grafana.envValueFrom/envFromConfigMaps (which reference the Secret unconditionally), mirroring the standalone test.yaml.
- verify-metrics: dump Mimir job/service_name/__name__ label values before asserting, so a run reveals whether metrics flow and under which labels (Mimir's OTLP ingest can rewrite Prometheus labels). - verify-traces: discover the collector OTLP/HTTP service by its 4318 port instead of hardcoding 'opentelemetry-collector' (wrong in daemonset mode). - workflow: dump monitoring/lgtm Services in the failure diagnostics.
The LGTM pack's contract is the collector override that ships otlp-received telemetry to Loki/Tempo/Mimir. Rather than asserting on organically-scraped kube metrics (which never reach Mimir on NIC latest: the collector SA lacks pod list/watch RBAC and only the kubernetes-pods job ships), push a synthetic OTLP log/metric/span through the collector and read each back from its backend. - lib.sh: pf() now takes a target ref (svc/… or pod/…); add otel_pod() to find a Running collector pod, since NIC's collector is a serviceless DaemonSet. - verify-logs/metrics/traces: push synthetic OTLP to a collector pod's 4318 receiver, read back from Loki/Mimir/Tempo. Deterministic, independent of NIC scraping. - README: document the approach and the upstream NIC scrape-RBAC gap (which leaves the k8s-views + gateway dashboards empty on a stock platform).
…eased The pack's collector override only applies on NIC with the software-pack extension point + scrape RBAC from nebari-dev/nebari-infrastructure-core#331 (merged 2026-06-17), which is not in the latest NIC release (v0.6.0). Pin nic-version to a main commit and build from source (adds Go setup) so the telemetry checks exercise the NIC the pack actually requires. TEMPORARY: must switch back to a released nic-version once a NIC release includes #331 (action item on the PR).
…ot a SHA)
The sandbox action builds source nic-version refs via `git clone --branch`,
which rejects a bare commit SHA ("Remote branch <sha> not found"). Use
nic-version: main so the source build works; still temporary, still must move
to a released tag once a NIC release includes #331.
…eling
metrics->Mimir and traces->Tempo pass on NIC main, proving the override export
pipeline works; only the logs->Loki leg fails to match {service_name=...}. Dump
Loki's label names + service_name values + recent streams on failure so we can
see whether OTLP logs land under a different label or not at all.
The OTLP log (service.name=lgtm-e2e-logs) never appears in Loki though the collector returns 200 and metrics/traces flow. Dump the collector pod's recent log lines (filtered for loki/export/error) on failure to see why the otlphttp/loki export isn't landing. Also silence SC2329 false positives on the retry-invoked helpers.
…label
The OTLP log reaches Loki (collector returns 200, no export error) but doesn't
match {service_name="lgtm-e2e-logs"} — Loki's OTLP ingestion likely labels it
unknown_service. Tag the log with a unique body token and query by content
filter across all streams, so the check passes if the log is ingested under
any label. Keep the introspection dump for the truly-absent case.
NIC v0.8.0 (2026-06-24) is the first release to include the software-pack collector extension point + scrape RBAC (#331), so the pack's override works against the released binary. Drop the temporary main-branch source build and the Go setup step; go back to nic-version: latest.
The collector DaemonSet pod is replaced on rollouts (the chart's post-install rollout fired just after install reported Healthy, replacing the pod mid-suite; metrics flaked because its port-forward was pinned to the now-dead pod while re-pushes failed silently). Add ensure_otel_pf() to lib.sh: it re-resolves a Running collector pod and re-establishes the 4318 port-forward whenever the prior one dies, and is called at the top of each push. A retry loop now recovers from a pod swap within one cycle instead of timing out.
…d resolve, doc note)
- verify-grafana: replace bash -c retry strings with functions passed to retry,
removing the ${AUTH[*]} flattening footgun and relying on the script's
pipefail for the curl|jq pipe.
- workflow: resolve the collector pod by name prefix in failure diagnostics
instead of hardcoding daemonset/opentelemetry-collector-agent, matching
lib.sh's otel_pod (collector name varies across NIC versions).
- design doc: note the logs/metrics checks were superseded by synthetic OTLP;
README is authoritative.
Kept nic-version: latest (integration test should track current NIC releases).
The authoritative description of the e2e suite lives in tests/e2e/README.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a black-box end-to-end test that installs the LGTM pack onto a real Nebari platform (NIC + foundational stack) via GitOps and verifies telemetry actually flows into Loki/Tempo/Mimir.
This complements — does not replace — the existing standalone
test.yamlsmoke test (which uses a stubbed OTel DaemonSet and never touches a real platform).How it works
New workflow
.github/workflows/e2e-platform.yaml:nebari-dev/action-nebari-sandbox@v2(profile: platform, NIClatest)../chartvia theadd-software-packsub-action (GitOps/ArgoCD path,nebariapp.enabled=true), waiting for the Application to reachHealthy.tests/e2e/verification suite.Verification scripts (
tests/e2e/)verify-grafana.sh/api/healthok; Loki/Tempo/Mimir datasources provisioned and pass datasource health checksverify-logs.shverify-metrics.shverify-traces.shotlp → otlp/tempoleg)Each is standalone and locally runnable; see
tests/e2e/README.md.Notes
envoy_*series don't flow on a stock platform and the chart'snebari-gateway-trafficdashboard has no data — a real product gap worth tracking separately. The metrics check targets the deterministic cAdvisor/kubelet/kube-state series instead.docs/superpowers/specs/anddocs/superpowers/plans/.This PR is opened to exercise the new workflow itself.
NIC version
The e2e uses
nic-version: latest. The pack's collector override + organicscraping require NIC PR #331 (software-pack extension point + collector scrape
RBAC), first released in NIC v0.8.0 (2026-06-24). Latest now satisfies this,
so no source build or pin is needed.
Findings surfaced by this e2e (for follow-up, not blocking this PR)
latest(v0.6.0) ignores the collector override and its collectorServiceAccount lacks pod
list/watchRBAC — both fixed by NIC #331,unreleased. Until a NIC release ships #331, the LGTM pack's telemetry
integration does not work on a released NIC.
nebari-gateway-traffic.jsondashboard relies on Envoy metricsthat NIC's collector does not scrape by default (no Envoy scrape job) — a
separate gap; this suite does not assert on it.