PAYG B-3 / S-3: cucumber suite for shadow-mode flows + CI workflow#6522
Open
ConnorYoh wants to merge 6 commits into
Open
PAYG B-3 / S-3: cucumber suite for shadow-mode flows + CI workflow#6522ConnorYoh wants to merge 6 commits into
ConnorYoh wants to merge 6 commits into
Conversation
ConnorYoh
force-pushed
the
payg-shadow-filter
branch
from
0db4c18 to
e12c006
Compare
ConnorYoh
force-pushed
the
payg-s3-cucumber
branch
from
824b89f to
3a50fa0
Compare
ConnorYoh
force-pushed
the
payg-shadow-filter
branch
from
7267580 to
6895f57
Compare
ConnorYoh
force-pushed
the
payg-s3-cucumber
branch
from
3a50fa0 to
f4540f4
Compare
ConnorYoh
requested review from
EthanHealy01,
Frooodle,
Ludy87,
jbrunton96 and
reecebrowne
as code owners
ConnorYoh
changed the title
dosubot
Bot
added
size:XL
![aikido-pr-checks[bot]](https://avatars.githubusercontent.com/in/898896?s=60&v=4){kind=small}
dosubot
Bot
added
size:XXL
Lands the scaffolding for end-to-end coverage of the PAYG shadow charging engine (filter + interceptor stack from PR #6519). Marks the Gherkin contract for the 8 scenarios we want to cover before flipping the engine to real charging. This PR is intentionally a STARTING POINT — the Gherkin scenarios are the agreed test surface, but a few infrastructure pieces still need to slot together before the saas-cucumber CI job goes green. See testing/cucumber/features/payg/README.md "What still needs work" for the specifics (filter-toggle restart hook, user-seed timing, API-key auth under saas profile, CI job wiring). New files: testing/compose/docker-compose-saas.yml Stirling-PDF backend with STIRLING_FLAVOR=saas plus a Postgres holding the stirling_pdf schema. Disables the Supabase JWT auto- config so the API-key path the cucumber tests use is the live one. testing/compose/payg/saas-init.sql Container-init bootstrap (schema-only — actual table inserts wait for Flyway to run). testing/compose/payg/saas-seed.sql Piped through psql by the harness AFTER Flyway has applied V1-V13. Creates payg-cucumber-team + a test user with API key payg-cucumber-key + wallet_policy.engine = 'PAYG_SHADOW'. Idempotent. testing/cucumber/features/payg/shadow_charges.feature 8 scenarios covering: first-call CHARGED row; lineage join; 5xx first-step REFUNDED + CLOSED; 4xx leaves CHARGED; ZIP unpack per-PDF OUTPUT recording; multi-file group sizing; PIPELINE source via header; kill-switch produces zero rows. testing/cucumber/features/payg/README.md Operator's guide — how to run, what's covered, what still needs to land before CI green. testing/cucumber/features/steps/payg_step_definitions.py Step defs using requests (HTTP) + psycopg (direct DB inspection of payg_shadow_charge / processing_job / processing_job_step / job_artifact_hash). Direct DB reads are deliberate — we want to see the filter's side effects, not relay them through another API layer that itself might be wrong. testing/test-payg.sh Companion to testing/test.sh — boots the saas compose, waits for health, seeds the test data, runs behave against features/payg. Modified: testing/cucumber/behave.ini Default run excludes features/payg (saas-only). The saas-cucumber CI job will invoke behave with `features/payg` explicitly. testing/cucumber/requirements.in Adds psycopg[binary] — needed only for the saas-cucumber CI job. Why a separate harness from testing/test.sh: The existing test.sh covers the proprietary-flavor stack (no PAYG tables, no saas profile). Shoehorning PAYG into that flow would couple two CI matrices that fail and succeed independently. The saas-cucumber job is better off as its own focused CI workflow once the harness is reliable. Stacked on PR #6519 (the filter itself). When that merges to main, this branch needs to rebase but no code changes. Refs: PR-S3 in notes/PAYG_DESIGN.md §7.5.
End-to-end run against the saas docker-compose surfaced four real issues —
all fixed here. After the changes, 4 of 8 scenarios pass; the remaining 4
are tracked in the README as follow-ups (3 are realistic-test-endpoint
mismatches, 1 is the already-acknowledged kill-switch NotImplementedError).
docker-compose-saas.yml: disable Flyway, use ddl-auto=create-drop.
The saas Flyway migrations assume `users` and `teams` already exist
(V2 ALTERs `users`; V5 references `teams(id)`) because in production
Supabase provisions them. On a clean test postgres they don't exist
and Flyway crashes. Hibernate's create-drop builds the full schema
from the entity graph in one pass. V12's default-policy seed moves
into saas-seed.sql below.
saas-seed.sql: realign with what Hibernate actually creates.
- Seed the default pricing_policy + per-source step limits (was
previously handled by V12 in production).
- Explicit CURRENT_TIMESTAMP on every NOT NULL timestamp column.
Hibernate's @CreationTimestamp / @UpdateTimestamp are
application-side; direct INSERTs bypass them. Affects
pricing_policy.created_at, team_memberships.created_at +
invited_at + updated_at, wallet_policy.updated_at.
- Drop our own user INSERT — the backend auto-creates a
CUSTOM_API_USER from SECURITY_CUSTOMGLOBALAPIKEY, and inserting
a second one collides on the unique api_key index. Update that
user's team_id instead.
- Populate wallet_policy.degraded_feature_set, auto_group_strategy,
notification_emails (NOT NULL columns Hibernate creates without
Flyway-migration DEFAULTs).
shadow_charges.feature + payg_step_definitions.py: unambiguate step
pattern. behave's parser was matching the longer "with header" form
greedily against the shorter form. Reorder so "with header" precedes
the endpoint and the parser can distinguish them.
Local results after fixes (8 scenarios, default behave.ini exclude
temporarily removed for the run):
- 4 PASS: first-call CHARGED row; lineage join (2-step chain);
multi-file group sizing; PIPELINE source via header.
- 3 FAIL (assertion-level — wrong test endpoint for the case):
- 5xx: /add-password returns 400 for malformed PDF, not 5xx. Need
an endpoint that surfaces a server-side exception cleanly.
- 4xx: same endpoint accepts missing password (returns 200). Need
an endpoint that genuinely rejects invalid params.
- ZIP: /split-pdf-by-sections returned application/octet-stream
(single PDF) for a small fixture. Need /split-pages or a larger
multi-section fixture.
- 1 ERROR (acknowledged): kill-switch step is NotImplementedError
until a compose-override restart hook lands.
All 4 failures are test-data / endpoint-choice issues, not engine bugs.
The filter is verifiably writing shadow rows + tracking lineage end-to-end.
Local cucumber run: 6 passed, 0 failed, 2 skipped (both @Manual with documented reasons). Up from 4/3/1 in the previous iteration. PaygOutputExtractor: sniff PDF / ZIP magic bytes when Content-Type is missing or application/octet-stream. Stirling tool endpoints often emit generic octet-stream for streamed responses even when the body is a real PDF or ZIP — without sniffing the extractor was silently dropping every such response from lineage capture. The body's first 4-5 bytes tell us what it actually is; trust those over a generic Content-Type. Existing application/pdf and application/zip paths unchanged. PaygOutputExtractorTest gains 4 cases: octetStream + PDF magic → treated as PDF octetStream + ZIP magic → unpacked as ZIP null Content-Type + ZIP magic → unpacked as ZIP octetStream + no magic → empty shadow_charges.feature + payg_step_definitions.py: address the 3 prior scenario failures. Fixture constants were backwards — verified via pypdf: tables.pdf = 1 page (was labelled THREE_PAGE_PDF) ghost1.pdf = 3 pages (was labelled SINGLE_PAGE_PDF) Swap the constants so the right file gets sent to each endpoint. 4xx scenario: was using /add-password without password (returned 200 — endpoint tolerates empty). Switched to a chain: step 1: /add-password (encrypts) → CHARGED row step 2: /sanitize-pdf on encrypted output WITHOUT password → 400 The second call lineage-joins the first (step_count=2) and the 4xx appears as a FAILED step on the existing process. Exactly the contract the design demands. ZIP scenario: was using /split-pdf-by-sections (single-section output on small fixture → octet-stream single PDF). Switched to /split-pages with pageNumbers=1,2 on the 3-page fixture which reliably emits a ZIP. The magic-byte sniff above means application/octet-stream still parses. 5xx scenario tagged @Manual with explanation. Every "5xx" I found in Stirling turned out to be Spring 404-as-500 fallback for non-existent endpoints — those don't hit our interceptor (no HandlerMethod, no @AutoJobPostMapping). The refund-and-close engine path itself is unit-tested in PaygChargeInterceptorTest.afterCompletion_5xx_opened_*. Step definitions: new "I POST a {single|3}-page PDF to {endpoint} with form fields:" step accepts a Gherkin table of multipart form fields. _post_pdf() takes optional form_data override so scenarios can send arbitrary field combinations instead of the default password=... behave.ini: add `tags = ~@manual` so the default behave run skips scenarios that need infrastructure not yet built. Local results (testing/test-payg.sh-style run, temp behave.ini swap): 1 feature passed, 0 failed 6 scenarios passed, 0 failed, 2 skipped 57 steps passed, 0 failed, 20 skipped Passing scenarios verify end-to-end: - First-call CHARGED row - Lineage join (2-step chain joins single process) - 4xx leaves CHARGED + appends FAILED step on existing process - ZIP unpack via magic-byte sniff (application/octet-stream) - Multi-file group sizing (single row for merged inputs) - PIPELINE source via X-Stirling-Automation header Skipped (@Manual) with documented runbooks: - 5xx first-step REFUNDED + CLOSED - Filter kill-switch (needs harness restart hook)
Three changes that take this PR from draft-with-TODOs to merge-ready: 1. New `.github/workflows/docker-compose-tests-saas.yml` — self-contained workflow that triggers only on PAYG-relevant paths (app/saas/**, testing/cucumber/features/payg/**, testing/compose/docker-compose-saas.yml, testing/test-payg.sh, the workflow itself). Models the existing docker-compose-tests.yml structure: pick runner, Java 25, Docker Compose, Python 3.12, then runs ./testing/test-payg.sh. Uploads the behave HTML report + JUnit XML on every run; dumps container logs on failure for triage. Self-contained (not wired into build.yml's files-changed matrix) so the saas-cucumber job fails and succeeds independently. No JaCoCo coverage in v1 — the saas compose doesn't have the coverage override; can add later if useful. 2. Removed both @Manual scenarios from features/payg/shadow_charges.feature: the 5xx-refund and the kill-switch (PAYG_FILTER_ENABLED=false). Neither was running in CI; both required harness work that wasn't worth the complexity for what they catch. Replaced with a short pointer comment above the 4xx scenario. Also dropped the `step_restart_with_filter_toggle` step def that raised NotImplementedError — no longer referenced from any scenario. 3. Documented both manual scenarios in notes/PAYG_DESIGN.md §7.5.2 "PAYG cucumber: manual-only scenarios" with the full procedure for each: - 5xx-refund: add a temporary test-only throw endpoint, post, assert status=REFUNDED + refund_reason LIKE 'first-step-5xx:%', remove. - Kill-switch: tear down, flip PAYG_FILTER_ENABLED=false in compose, bring back up, assert zero shadow rows, revert. Engine paths for both are unit-tested in PaygChargeInterceptorTest.afterCompletion_5xx_opened_* — the manual procedures verify end-to-end behaviour only. After this commit: * 6/6 automated scenarios run in CI on every PAYG-touching PR * 0 @Manual / @Skip tags remain in the feature file * No NotImplementedError in step defs * Manual procedures captured in design doc, not in cucumber TODO comments
Two issues on the first real run of docker-compose-tests-saas.yml: 1. ModuleNotFoundError: psycopg — listed in requirements.in but requirements.txt was never regenerated, so pip-install at CI time skipped it. Ran `pip-compile --generate-hashes --strip-extras` to bring psycopg + transitive deps (tzdata, psycopg-binary) into the hashed requirements lock. Step defs in features/steps/payg_step_defs use psycopg for direct DB inspection. 2. Aikido flagged the `curl | sudo install /usr/local/bin/docker-compose` step as "binary pulled from remote source without integrity verification". The step turns out to be dead code — Ubuntu runners already ship `docker compose` v2 inside the Docker CLI, and our test-payg.sh uses the v2 form (`docker compose`, no hyphen) throughout. Removed the step entirely with a comment explaining why. That's why docker-compose-tests.yml has it (the legacy harness uses the v1 binary in some paths) — we don't.
ConnorYoh
force-pushed
the
payg-s3-cucumber
branch
from
0ce0963 to
0f7fb01
Compare
dosubot
Bot
added
size:XL
stirlingbot
Bot
added
Documentation
Second failure on the saas-cucumber CI run (also broke main
docker-compose-tests since both consume the same requirements.txt):
ERROR: In --require-hashes mode, all requirements must have their
versions pinned with ==. These do not:
typing-extensions>=4.6 from ... (from psycopg==3.3.4)
psycopg declares `typing-extensions>=4.6` under a Python-version marker
that pip-compile leaves unpinned in --strip-extras mode. Listing
typing-extensions in requirements.in explicitly forces pip-compile to
lock it to 4.15.0 with a hash.
Contributor
🚀 V2 Auto-Deployment Complete!Your V2 PR with embedded architecture has been deployed! 🔗 Direct Test URL (non-SSL) http://54.175.155.236:6522 🔐 Secure HTTPS URL: https://6522.ssl.stirlingpdf.cloud This deployment will be automatically cleaned up when the PR is closed. 🔄 Auto-deployed for approved V2 contributors. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR is
End-to-end cucumber coverage for the PAYG shadow charging engine (the filter + interceptor stack from #6519), wired into CI via a new
docker-compose-tests-saas.ymlworkflow that runs only on PAYG-touching PRs.Stacked on #6519.
Automated scenarios (run by
docker-compose-tests-saas.yml)See
testing/cucumber/features/payg/shadow_charges.feature:JobService.joinOrOpenmatching; no new shadow rowPaygOutputExtractorunpacks + records signaturesX-Stirling-Automationsets PIPELINE sourceJobSourcedetectionAll 6 run locally via
./testing/test-payg.shand will run on CI for any PR that touchesapp/saas/**, the PAYG cucumber features, the saas compose stack, or the workflow itself.Manual-only scenarios — documented in design doc, not in this suite
Two parts of the shadow engine are deliberately not automated; the engine paths are unit-tested in
PaygChargeInterceptorTest.afterCompletion_5xx_opened_*, and the manual procedures (which require a temporary throw endpoint or a container restart with a flag flipped) live innotes/PAYG_DESIGN.md§7.5.2 "PAYG cucumber: manual-only scenarios".PAYG_FILTER_ENABLED=false). Needs a container restart mid-suite; manual procedure tears down, flips env, brings up, asserts zero shadow rows.If either gets a hot-reload path (test-only throw endpoint shipped behind a profile gate, or admin endpoint for the kill switch), automate it in a follow-up and drop the manual procedure.
CI workflow
.github/workflows/docker-compose-tests-saas.yml(new) — self-contained, not wired intobuild.yml'sfiles-changedmatrix so the saas-cucumber job fails and succeeds independently. Triggers only on PAYG-relevant paths. No JaCoCo coverage in v1 (saas compose doesn't have the coverage override; can add later).Test infrastructure (recap)
testing/compose/docker-compose-saas.yml— Stirling-PDF backend withSTIRLING_FLAVOR=saas+ Postgres holding thestirling_pdfschema. Supabase JWT auto-config disabled; API-key auth viaSECURITY_CUSTOMGLOBALAPIKEYis the live path the cucumber tests exercise.testing/compose/payg/saas-init.sql+saas-seed.sql— schema bootstrap + idempotent seed (team / user / wallet_policy).testing/cucumber/features/payg/shadow_charges.feature— the 6 scenarios above.testing/cucumber/features/steps/payg_step_definitions.py— step defs usingrequests(HTTP) +psycopg(direct DB inspection). Direct DB reads are deliberate — we want to see the filter's side effects, not relay them through another API layer.testing/test-payg.sh— companion runner totesting/test.sh. Brings up the saas compose, waits for health, seeds, runs behave, tears down.behave.iniexcludesfeatures/paygfrom the default behave run (the saas-cucumber CI job invokes it explicitly).Why a separate harness from
testing/test.shThe existing
test.shcovers the proprietary-flavour stack (no PAYG tables, no saas profile). Coupling two CI matrices that fail and succeed independently into one script is asking for trouble. Keep the saas-cucumber job focused on its own concerns; once the harness is mature, the wider team can decide whether to merge them.Tracked in
notes/PAYG_DESIGN.md§7.5 (PR-S3) + §7.5.2 (manual scenarios).