Skip to content

Commit b63984a

Browse files
bar-capsuleclaude
andcommitted
Strengthen ACS-Core conformance tests with explicit contradictions
Self-audit caught six tests that were positive-only (asserted the happy path but had no falsifier). A no-op validator or a non-chaining chain would have passed several of them. Mutation-tested the strengthened versions: each named bug is now caught with a clear failure message citing the spec. Strengthening per group: Core02 Envelope shape + test_contradiction_validator_actually_works — a deliberately broken envelope ({"jsonrpc": "2.0"} with no method/id/params) MUST be rejected. Without this, a no-op validator passes every drop-required-field test silently. Core03 Hook taxonomy minimum + Reframed each of 6 hook tests as parametrized over a table of (method, valid_payload, broken_payload, schema_file). Asserts both that the valid case produces a KNOWN disposition (not garbage) AND that the broken_payload is rejected by the canonical hook payload schema. Falsifier-per-hook. Core04 Dispositions + test_allow_response_without_required_envelope_fields_rejected — synthesizes 5 broken allow responses (missing type, acs_version, request_id, decision; bogus decision enum) and asserts each fails response-envelope.json. Without this, the positive "allow validates" test is tautological. Core05 SessionContext + chain * test_chain_externally_recomputable extended to THREE entries. Old version only checked entry 1, which doesn't have a previous_hash to fold in — a "chain that doesn't chain" mutation produced the right value for the root and passed. Now: entry 2's recomputed hash with previous_hash MUST match what the Guardian published; AND must NOT match the computation that ignores previous_hash. Entry 3 verifies transitive chaining. Core08 Decision honoring * Replaced wire-level "Guardian responds fast" check (wrong property) with three adapter-side tests of the actual MUST: - test_adapter_actually_applies_guardian_deny — Guardian returns DENY, adapter MUST translate to deny, not allow. - test_adapter_waits_for_a_slow_guardian — Guardian sleeps 1s, adapter MUST take at least 1s; an adapter that proceeds without waiting is caught by elapsed-time check. - test_fail_open_emits_audit_event — unchanged. Core10 Wrapped MCP * Strengthened the "no crash" test to also require the response validates against response-envelope.json. A no-op Guardian returning {} would no longer pass. + test_mcp_method_namespace_rejects_garbage_namespaces — contradiction: methods outside the reserved namespaces (e.g., "arbitrary/method", "step/typo", "PROTOCOLS/upper") MUST be rejected by the schema. Without this, "namespace pattern works" is unverified. Mutation-tested the strengthened suite. Four representative bugs were injected one at a time; each was caught: No-op _validate_request_envelope → 9 Core02 tests fail compute_entry_hash ignores previous_hash → Core05 3-entry test fails at entry 2 Adapter silently fails open (no audit) → Core08 audit test fails Adapter proceeds without waiting → Core08 deny + slow-guardian tests fail Net: still 44 tests, but every one now has a verifiable falsifier. Adopter who forks and breaks a Core MUST gets a precise failure with spec citation, not a passing suite that papered over their bug. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 9553897 commit b63984a

1 file changed

Lines changed: 308 additions & 103 deletions

File tree

0 commit comments

Comments
 (0)