Flake audit: unit tests clean, functest has flake-prone patterns

[robot-primus]

Flake Audit Results

Ran all unit tests 20 times on main (commit 174df04) to identify flaky tests. Also performed static analysis of the functional test code for flake-prone patterns.

Unit Tests: 194 specs, 0 flakes (20/20 runs)

All unit test specs passed consistently across 20 consecutive runs. The recent fix in #464 (OVS interface race conditions) resolved the previously-reported unit test flakes.

Packages tested:

Package	Specs	Result
pkg/plugin	100	20/20 pass
pkg/mirror-consumer	40	20/20 pass
pkg/mirror-producer	48	20/20 pass
pkg/ovsdb	4	20/20 pass
pkg/utils	6	20/20 pass

Functional Tests: Cannot run locally, code analysis reveals flake-prone patterns

Functional tests require a Kubernetes cluster (--kubeconfig) and cannot be run in isolation. However, static analysis of the test code in tests/ and tests/cluster/ reveals several patterns likely to cause intermittent failures:

1. Silent error suppression (HIGH risk)

Multiple places discard errors from API calls, which can cause nil pointer panics instead of meaningful failures:

• tests/ovs_test.go:40 — pods, _ := clusterApi.Clientset.CoreV1().Pods("").List(...) discards error
• tests/mirror_test.go:80 — out, _ := clusterApi.ReadFileFromPod(...) discards error in Eventually loop
• tests/marker_test.go:48 — capacityInt, _ := capacity.AsInt64() discards error

2. Missing network readiness checks (HIGH risk)

Pod readiness polling in tests/cluster/cluster.go:113-130 only checks ContainerStatuses[].Ready, not whether the OVS secondary network is actually attached. Tests may proceed before OVS networking is functional, causing ping failures.

3. Unchecked array bounds (HIGH risk)

tests/marker_test.go:74 — pod.Status.ContainerStatuses[0].RestartCount accesses index 0 without bounds checking. Will panic if container statuses aren't populated yet during pod initialization.

4. tcpdump timing race in mirror tests (MEDIUM risk)

tests/mirror_test.go:103-109 — Only 30 seconds to capture mirrored ICMP traffic via tcpdump. Race between ping execution (line 100) and tcpdump file check. Flaky if tcpdump buffering hasn't flushed or if network setup is slow.

This is likely the root cause of the pattern seen in #455.

5. Unbounded exec context (MEDIUM risk)

tests/cluster/cluster.go:229-233 — exec.StreamWithContext(context.Background(), ...) has no timeout. If a remote pod hangs, the test suite will deadlock.

6. Long timeouts masking issues (LOW risk)

Pod deletion uses a 6-minute timeout with 1-second polling (360 API calls). Namespace termination uses 120 seconds. These mask underlying slowness rather than surfacing it.

Recommendations

1. Handle errors explicitly in Eventually loops — log or return errors instead of discarding with _
2. Add OVS network readiness check — after pod Ready, verify the OVS interface exists before proceeding
3. Add bounds checking for ContainerStatuses before indexing
4. Increase tcpdump capture timeout or add explicit flush/sync before checking capture results
5. Add timeout context to execOnPod to prevent deadlocks
6. Consider reducing cleanup timeouts and adding exponential backoff

Methodology

• Unit tests: go test -count=1 -ginkgo.json-report=<file> for each package, 20 iterations, with Ginkgo JSON reports parsed per-spec
• Functional tests: Static code analysis (requires Kubernetes cluster to run)
• Environment: Fedora 43, OVS 3.6.2, Go 1.26.2