[CI/CD Assessment] CI/CD Pipelines and Integration Tests Gap Assessment

[github-actions[bot]]

📊 Current CI/CD Pipeline Status

The repository has a mature and well-structured CI/CD system with 52 active workflows. The pipeline covers build verification, linting, type-checking, unit tests, integration tests, security scanning, performance monitoring, and AI-assisted agentic workflows. Recent workflow runs show primarily successful outcomes with failures concentrated in flaky integration tests (Docker/network-dependent) and the occasional skipped agentic workflow.

Workflow Inventory by Category:

Category	Count	Examples
Core build/test (on PR)	8	Build Verification, Lint, Type Check, Test Coverage, Integration Tests, CodeQL, Dependency Audit, PR Title Check
Integration/E2E tests	3	Integration Tests (test-integration-suite.yml), Chroot Integration Tests (test-chroot.yml), Examples Test
Security	4	CodeQL, Container Security Scan (Trivy), Dependency Audit (npm audit → SARIF), Security Guard (agentic)
AI Agentic (on PR)	4	Build Test Suite, Security Guard, Smoke Claude, Smoke Codex, Smoke Copilot
Documentation	3	Link Check, Deploy Docs, Docs Preview, Doc Maintainer
Scheduled maintenance	8+	Performance Monitor, Dependency Security Monitor, Secret Digger, CLI Flag Checker, etc.

---

✅ Existing Quality Gates

The following checks run on every pull request:

Check	Workflow	What It Tests
ESLint	lint.yml	TypeScript linting with security plugin, custom no-unsafe-execa rule
Markdown Lint	lint.yml	All .md files via markdownlint-cli2
TypeScript Type Check	test-integration.yml	tsc --noEmit strict type checking
Build Verification	build.yml	Compiles on Node 20 and 22; verifies dist/cli.js exists
API Proxy Unit Tests	build.yml	containers/api-proxy unit tests
Test Coverage	test-coverage.yml	Jest with coverage; comments diff on PR; fails on regression
Integration Tests (Domain/Network)	test-integration-suite.yml	blocked-domains, dns-servers, wildcard-patterns, ipv6, localhost-access, network-security
Integration Tests (Protocol/Security)	test-integration-suite.yml	protocol-support, credential-hiding, one-shot-tokens, token-unset, git-operations
Integration Tests (Container/Ops)	test-integration-suite.yml	container-workdir, environment-variables, error-handling, exit-code-propagation, log-commands, no-docker, volume-mounts
API Proxy Integration Tests	test-integration-suite.yml	api-proxy (basic), api-proxy-observability, api-proxy-rate-limit
Examples Test	test-examples.yml	Runs examples/*.sh end-to-end against local container builds
Setup Action Test	test-action.yml	Tests the action.yml setup action (latest and pinned versions)
CodeQL	codeql.yml	Static analysis for JS/TS and GitHub Actions YAML
Dependency Audit	dependency-audit.yml	npm audit --audit-level=high for main + docs-site packages
Container Security Scan	container-scan.yml (path-triggered)	Trivy CRITICAL/HIGH CVE scan on containers/** changes
PR Title Check	pr-title.yml	Enforces Conventional Commits format and allowed scopes
Link Check	link-check.yml (.md path-triggered)	Checks all markdown links with lychee
Security Guard	security-guard.lock.yml	AI review of security posture changes
Build Test Suite	build-test.lock.yml	AI-assisted build and test verification
Smoke Tests	smoke-claude/codex/copilot.lock.yml	End-to-end smoke tests with real AI agents through the firewall

---

🔍 Identified Gaps

🔴 High Priority

1. Eight Integration Test Files Not Wired into CI

Several test files exist in tests/integration/ but are not included in any CI workflow pattern match, meaning new PRs can break these features with no CI signal:

Test File	Feature Area
api-target-allowlist.test.ts	--copilot-api-target, --openai-api-target, --anthropic-api-target auto-allowlist
chroot-capsh-chain.test.ts	Chroot mode capsh privilege-drop chain
chroot-copilot-home.test.ts	Chroot mode Copilot home directory handling
ghes-auto-populate.test.ts	GitHub Enterprise Server URL auto-populate behavior
skip-pull.test.ts	--skip-pull / --image-tag flags
workdir-tmpfs-hiding.test.ts	Work directory tmpfs isolation/hiding

None of these appear in test-integration-suite.yml or test-chroot.yml. The api-target-allowlist feature in particular is a documented security-relevant feature with dedicated tests but no CI enforcement.

Recommendation: Add these six test patterns to the relevant jobs in test-integration-suite.yml and test-chroot.yml.

2. Coverage Thresholds Are Too Low (38%/31%/37%)

The current enforced thresholds are:

• Statements: 38% (threshold)
• Branches: 30% (threshold)
• Functions: 35% (threshold)

Two of the most critical files have near-zero coverage:

• cli.ts: 0% (0 of 69 statements)
• docker-manager.ts: 18% (45 of 250 statements, 1 of 25 functions)

These are the core orchestration files. A bug in CLI argument parsing or container lifecycle management will not be caught by unit tests.

Recommendation: Gradually ratchet thresholds upward (e.g., +2% per quarter) and add unit tests for cli.ts and docker-manager.ts.

3. No Required Status Checks Enforcement Visible in Workflow Config

It cannot be confirmed from workflow files alone whether branch protection rules mandate all PR checks to pass before merging. The presence of continue-on-error: true in the coverage comparison step means a coverage regression report can be ignored.

Recommendation: Verify that the Fail on coverage regression step actually blocks merges, and confirm branch protection requires all PR-triggered workflows to pass.

---

🟡 Medium Priority

4. Container Security Scan Is Path-Triggered Only

container-scan.yml only runs when containers/** files change. A change to src/docker-manager.ts that introduces a new base image reference or changes container configuration would not trigger a rescan.

Recommendation: Add src/docker-manager.ts to the path trigger, or run the scan on every PR to main.

5. Performance Benchmarks Are Scheduled-Only (Not PR-Gated)

performance-monitor.yml runs weekly on a schedule. There is no per-PR performance regression gate. A PR that dramatically increases container startup time or memory usage would merge without any signal.

Recommendation: Add a lightweight benchmark step to the build workflow (e.g., measure awf --version latency and container startup time) with a threshold check, running on PRs that touch src/**.

6. No Mutation Testing

Unit tests cover ~38% of statements, but even covered code may have tests that pass despite incorrect logic (tests that don't actually assert meaningful behavior). Without mutation testing, test quality is unknown.

Recommendation: Add [Stryker Mutator]((strykermutator.io/redacted) for the well-covered files (logger.ts, squid-config.ts) as a periodic scheduled check.

7. ESLint Runs Twice on Every PR

Both lint.yml and build.yml run npm run lint on PRs. This doubles lint job time without benefit.

Recommendation: Remove the Run linter step from build.yml since lint.yml already covers it as a required check.

8. Chroot Integration Tests Not Running on Regular PRs

test-chroot.yml runs only when src/**, containers/**, or specific workflow files change (path-filtered). Chroot mode is a significant feature, but PRs that only touch test files or config would skip these tests.

Recommendation: Ensure chroot tests run on all PRs touching src/** or add them as an always-on required check alongside the integration suite.

---

🟢 Low Priority

9. No Snapshot / Diff Testing for Generated Configs

generateSquidConfig() and generateDockerCompose() produce critical config files. While squid-config.ts has 100% unit test coverage, there are no snapshot tests that would catch unintended output format changes.

Recommendation: Add Jest snapshot tests for the generated squid.conf and docker-compose.yml content.

10. No SBOM (Software Bill of Materials) Generation

Container images are scanned with Trivy, but no SBOM is generated and published as an artifact. This is increasingly expected for security-conscious tooling.

Recommendation: Add trivy sbom output as a release artifact and as an optional PR artifact on containers/** changes.

11. docs-preview.yml / deploy-docs.yml Lack Broken Build Detection

Documentation builds (docs-site) succeed silently even if Astro emits warnings. There is no step that treats Astro build warnings as errors.

Recommendation: Add --logLevel warn or equivalent flag to fail the docs build on warnings; add a step to verify the built HTML output is non-empty.

12. No Conventional Commit Lint on Push to Main

pr-title.yml enforces Conventional Commits for PR titles. However, individual commit messages within a squash-merge or rebase-merge PR are not linted. Direct pushes to main (e.g., release commits) bypass title checking entirely.

Recommendation: The existing commitlint dev dependency and husky hook only run locally. Consider adding a push-to-main commitlint check in CI for the merge commit.

---

📋 Actionable Recommendations

Priority	Gap	Solution	Complexity	Impact
🔴 High	6 integration tests not in CI	Add to test-integration-suite.yml job patterns	Low	Prevents silent regressions in api-target, chroot, GHES, skip-pull features
🔴 High	Coverage thresholds too low	Incrementally raise thresholds; add tests for cli.ts + docker-manager.ts	Medium	Core orchestration logic gains regression protection
🔴 High	Coverage gate not enforced	Verify branch protection requires Fail on coverage regression step	Low	Ensures coverage gate actually blocks merges
🟡 Medium	Container scan path-scoped	Add src/docker-manager.ts to container-scan.yml paths	Low	Catches container config changes missed by scan
🟡 Medium	No PR performance gate	Add lightweight startup benchmark to build.yml	Medium	Catches startup regressions before they ship
🟡 Medium	No mutation testing	Add Stryker for logger.ts + squid-config.ts as weekly scheduled job	Medium	Validates test assertion quality, not just coverage
🟡 Medium	ESLint runs twice	Remove lint step from build.yml	Low	Saves ~1-2 min per PR
🟢 Low	No snapshot tests for generated configs	Add Jest snapshots for squid.conf / docker-compose.yml output	Low	Catches unintended format changes in critical configs
🟢 Low	No SBOM generation	Add trivy sbom to release workflow	Low	Improves supply chain security posture
🟢 Low	Docs build warnings not fatal	Add --logLevel warn to Astro build	Low	Prevents silent docs degradation

---

📈 Metrics Summary

Metric	Value
Total GitHub Actions workflows	52
Workflows triggered on PR	~18 (build, lint, type-check, coverage, 3× integration suites, examples, setup-action, CodeQL, dep-audit, container-scan*, pr-title, link-check*, security-guard, build-test-suite, 3× smoke tests)
Integration test files	32
Integration test files covered in CI	26 (81%)
Integration test files not in CI	6 (19%)
Unit test statement coverage	38.39%
cli.ts coverage	0% 🚨
docker-manager.ts coverage	18% 🚨
squid-config.ts coverage	100% ✅
logger.ts coverage	100% ✅
Security scan tools in use	CodeQL, Trivy, npm audit → SARIF, ESLint security plugin
Performance benchmarking	Weekly scheduled only (not PR-gated)

* path-triggered, may skip on some PRs

AI generated by CI/CD Pipelines and Integration Tests Gap Assessment

• expires on Mar 21, 2026, 10:20 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-21T22:20:07.084Z.

Closed by Workflow