Commit b63984a
Strengthen ACS-Core conformance tests with explicit contradictions
Self-audit caught six tests that were positive-only (asserted the
happy path but had no falsifier). A no-op validator or a non-chaining
chain would have passed several of them. Mutation-tested the
strengthened versions: each named bug is now caught with a clear
failure message citing the spec.
Strengthening per group:
Core02 Envelope shape
+ test_contradiction_validator_actually_works — a deliberately
broken envelope ({"jsonrpc": "2.0"} with no method/id/params)
MUST be rejected. Without this, a no-op validator passes every
drop-required-field test silently.
Core03 Hook taxonomy minimum
+ Reframed each of 6 hook tests as parametrized over a table of
(method, valid_payload, broken_payload, schema_file). Asserts
both that the valid case produces a KNOWN disposition (not
garbage) AND that the broken_payload is rejected by the
canonical hook payload schema. Falsifier-per-hook.
Core04 Dispositions
+ test_allow_response_without_required_envelope_fields_rejected —
synthesizes 5 broken allow responses (missing type, acs_version,
request_id, decision; bogus decision enum) and asserts each
fails response-envelope.json. Without this, the positive
"allow validates" test is tautological.
Core05 SessionContext + chain
* test_chain_externally_recomputable extended to THREE entries.
Old version only checked entry 1, which doesn't have a
previous_hash to fold in — a "chain that doesn't chain"
mutation produced the right value for the root and passed.
Now: entry 2's recomputed hash with previous_hash MUST match
what the Guardian published; AND must NOT match the computation
that ignores previous_hash. Entry 3 verifies transitive chaining.
Core08 Decision honoring
* Replaced wire-level "Guardian responds fast" check (wrong
property) with three adapter-side tests of the actual MUST:
- test_adapter_actually_applies_guardian_deny — Guardian
returns DENY, adapter MUST translate to deny, not allow.
- test_adapter_waits_for_a_slow_guardian — Guardian sleeps
1s, adapter MUST take at least 1s; an adapter that proceeds
without waiting is caught by elapsed-time check.
- test_fail_open_emits_audit_event — unchanged.
Core10 Wrapped MCP
* Strengthened the "no crash" test to also require the response
validates against response-envelope.json. A no-op Guardian
returning {} would no longer pass.
+ test_mcp_method_namespace_rejects_garbage_namespaces —
contradiction: methods outside the reserved namespaces (e.g.,
"arbitrary/method", "step/typo", "PROTOCOLS/upper") MUST be
rejected by the schema. Without this, "namespace pattern works"
is unverified.
Mutation-tested the strengthened suite. Four representative bugs were
injected one at a time; each was caught:
No-op _validate_request_envelope → 9 Core02 tests fail
compute_entry_hash ignores previous_hash → Core05 3-entry test fails at entry 2
Adapter silently fails open (no audit) → Core08 audit test fails
Adapter proceeds without waiting → Core08 deny + slow-guardian tests fail
Net: still 44 tests, but every one now has a verifiable falsifier.
Adopter who forks and breaks a Core MUST gets a precise failure with
spec citation, not a passing suite that papered over their bug.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>1 parent 9553897 commit b63984a
1 file changed
Lines changed: 308 additions & 103 deletions
0 commit comments