Skip to content

[TAN-7901] Test suite for the mcp engine#14250

Open
adessy wants to merge 21 commits into
masterfrom
TAN-7901/mcp-engine-test-suite
Open

[TAN-7901] Test suite for the mcp engine#14250
adessy wants to merge 21 commits into
masterfrom
TAN-7901/mcp-engine-test-suite

Conversation

@adessy

@adessy adessy commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

The MCP engine was built velocity-first with minimal test coverage. This PR adds a systematic test suite, plus the production fixes the suite surfaced.

Production fixes (found while writing the suite)

  • Missing authorization guardsupdate_project, update_phase, update_resource and replace_form_fields called neither authorize_project! nor Pundit authorize, so the draft-only constraint was not enforced for updates: a super admin could mutate published projects through the MCP.
  • Broken get_form_fieldsreplace_form_fields round-tripSerializers::CustomField emitted a constraints key that is not a CustomField attribute; echoing fetched fields back crashed UpdateAllService. The serializer now drops the key (constraints are reported at the top level).
  • Silent empty list for a bad parent idlist_events and list_causes filtered without looking up the parent, so a nonexistent project_id/phase_id returned [] (an LLM reads that as "no events" rather than "wrong id"). They now return the standard not-found error like their siblings.
  • No way to remove a project header backgroundremote_header_bg_url was a non-nullable string; it now accepts null, which update_project translates into remove_header_bg! (CarrierWave ignores plain nil assignment), mirroring the web API's remove_image_if_requested!.
  • Group visibility was invisible and unreachable — with visible_to: 'groups', the project payload carried no group information, the schemas had no way to set the group list, and list_groups' project_id filter matched smart-group rules referencing the project rather than the groups that can see it. The project schemas now accept group_ids (replace-wholesale, like area_ids), the serializer surfaces group_ids, and the misleading filter is removed (read ids from the payload, resolve names via list_groups).
  • Opaque replace_form_fields validation errors — form-level error keys (stale_data, locked_deletion, …) were relayed as a raw JSON dump; they now map to actionable prose (re-fetch and retry, echo locked fields back, structural page rules), with the raw errors still in structuredContent.
  • Settable but not clearable fieldsend_at (the only way to make the last phase open-ended), access_denied_explanation_multiloc and uploader files (remote_<x>_url) now accept null to clear. Uploader clearing is a generic Runner helper driven by CarrierWave's mount registry (nil assignment is otherwise silently ignored); the permission multiloc also now merges per locale like every other multiloc field instead of replacing wholesale.
  • Unreachable CarrierWave rescues removed — download failures never raise out of save! (CarrierWave converts them into :carrierwave_download_error validation errors on assignment), so the dedicated rescues in the attach tools were dead code on every path.
  • Mass-assignment through undeclared schema keys — JSON Schema accepts undeclared keys by default and the MCP gem passes arguments to tools verbatim (verified at runtime), so create_phase (Phase.new(**params)) and replace_form_fields (assign_attributes via UpdateAllService — including resource_id, which re-parents a field to another form) were mass-assignable. Their schemas are now closed with additionalProperties: false. To keep the fetch → edit → replace round-trip working, the form-field schema is now shared between both tools with standard readOnly/writeOnly annotations on the directional keys; BaseTool.for strips readOnly keys from the params at dispatch (generic schema walker, unit-tested as a function), and a registry tripwire fails with the offending JSON-Pointer path if a readOnly ever lands where the walker can't reach it (only properties/items positions are supported — full coverage would take a JSON Schema evaluator).
  • Small cleanups — the BaseTool::Authorization concern is gone (its pundit_user was shadowed by Pundit::Authorization's identical default; the Runner now includes Pundit directly), and the draft-guard refusal no longer ends in a doubled period (reasons own their punctuation; the template only terminates its own fallback).

Refactors that rode along

  • Single serialization pathwaypaginated_response's as_json(only: …) fast path is gone; the three flat-payload tools got from-scratch serializers (no wraps), and Serializers::Base#to_h enforces symbols-as-top-level-keys so payload key types can't diverge.
  • Argument validation made explicit — the controller passes MCP::Configuration.new(validate_tool_call_arguments: true) instead of relying on the gem default; Runners trust schema-validated input, and a request-spec example pins the rejection end to end.

Coverage (lines, engines/commercial/mcp_server/app/**)

Baseline This PR
Total 740/887 = 83.4% 954/956 = 99.8%
Examples 76 285

The two uncovered lines are defensive branches (an unreachable RecordInvalid rescue in replace_form_fields, the non-redirectable branch of the OAuth authorize endpoint).

Notes / follow-ups (not addressed here)

  • Make additionalProperties: false structural instead of per-tool discipline.
  • List tools should go through Pundit policy scopes (policy_scope) instead of querying unscoped. Today McpPolicy restricts the endpoint to super admins so nothing leaks, but scoping would make the invariant local.

Changelog

Technical

  • [TAN-7901] Test suite for the MCP server engine (tools, serializers, base modules, HTTP dispatch): 76 → 285 examples.

@notion-workspace

Copy link
Copy Markdown

@adessy adessy changed the title Add missing authorization guards to MCP update tools [TAN-7901] Test suite for the mcp engine Jul 8, 2026
@cl-dev-bot

cl-dev-bot commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator
Messages
📖 Changelog provided 🎉
📖 Notion issue: TAN-7901
📖

Run the e2e tests

📖 Check translation progress

Generated by 🚫 dangerJS against 8cb2081

@adessy adessy force-pushed the TAN-7901/mcp-engine-test-suite branch 4 times, most recently from 6b11d5d to 3b5cf28 Compare July 10, 2026 09:57
adessy added 20 commits July 13, 2026 15:34
`update_project`, `update_phase`, `update_resource` and `replace_form_fields` called neither `authorize_project!` nor Pundit `authorize`, so the documented draft-only constraint was not enforced for updates: a super admin could mutate published projects through the MCP.
…form fields)

New specs for `update_phase`, `get_form_fields` and `replace_form_fields`; deepened `create_phase` (feature-flag-driven schema, participation-method variants) and `update_phase_permission` (demographic_questions semantics, rollback, verified expiry).

Also fixes a real round-trip bug the new specs caught: `Serializers::CustomField` emitted a `constraints` key that is not a `CustomField` attribute, so echoing `get_form_fields` output back into `replace_form_fields` crashed `UpdateAllService`. The MCP serializer now drops the key (constraints are reported at the top level of the response).
…tening and Serializers::Base

Direct unit coverage for Pagination (envelope, clamping, serialization paths), MultilocMerge, ResponseHelpers, the multiloc schema helper, JSONAPI flattening (id/ids suffixes, inlining) and the serializer base class (wraps, inline, from-scratch, collection semantics). base_tool_spec gains suite-wide definition invariants and `.unauthorized_message` coverage.
The request spec is the only place the HTTP gating is exercised: Doorkeeper `mcp:access` scope, `McpPolicy` (super admin + feature flag), the RFC 9728 WWW-Authenticate challenge, and end-to-end `tools/list` / `tools/call` through a real bearer token.
… Runner contract

Download failures surface as `carrierwave_download_error` model validation errors (not the `CarrierWave::DownloadError` rescue, which turns out to be defensive-only for the remote_url path). Adds `stub_failing_remote_download` next to the existing download stubs.
Tool runners trust their input, so the controller now enables `validate_tool_call_arguments` explicitly instead of relying on the gem default. A request-spec example pins end to end that non-conforming input is rejected.
…able messages

The tool used to relay `UpdateAllService` failures as a raw JSON dump (`Validation failed: {"form":[{"error":"stale_data"}]}`), which gives an MCP client no hint of what to do. Form-level error keys now map to prose that says how to recover. Unmapped keys fall back to the raw key, and the raw error hash rides along in `structuredContent[:errors]`.
`remote_header_bg_url` was typed as a non-nullable string, so an MCP client could set a header background but never remove one. The field is now nullable and `update_project` translates an explicit null into `remove_header_bg!`.
With `visible_to: 'groups'`, the group list was invisible and unreachable: the project payload carried no group information, the create/update schemas had no way to set it, and `list_groups`' `project_id` filter matched smart-group *rules* referencing the project — not the groups that can see it.

The project schemas now accept `group_ids` (replace-wholesale, like `area_ids`), the MCP project serializer surfaces `group_ids`, and the misleading `list_groups` filter is removed: the natural flow is reading `group_ids` from the project payload and resolving names via the unfiltered `list_groups`.
`paginated_response` had two serialization branches: MCP serializers, and an `as_json(only: ...)` fast path used by three flat-payload tools that also leaked string keys into `structuredContent`.
The mounts of all affected uploaders keep CarrierWave's default download/integrity validations, so download failures never raise out of `save!` — they are captured on assignment as `:carrierwave_download_error` validation errors and handled by the `RecordInvalid` rescue (as the download-failure specs assert). The dedicated `CarrierWave::DownloadError`/`IntegrityError` rescues were dead code on every path.
…clearable through the MCP

Three fields were settable but not clearable ("Pass null to …" now works for all of them):

- `end_at` is nullable in the phase schema; clearing it is the only way to make the last phase open-ended (the model already rejects it elsewhere).
- `access_denied_explanation_multiloc` accepts null and, while here, aligns with the other multiloc fields by merging per locale. An explicit null clears a multiloc to `{}`
- An explicit null on `remote_<uploader>_url` removes the file: CarrierWave ignores nil assignment, so a generic `Runner#clear_uploaders!` translates it to `remove_<uploader>!`, driven by the model's uploader registry.
JSON Schema accepts undeclared keys by default and the MCP gem passes arguments to tools verbatim (verified at runtime), so the two Runners that splat params into models were mass-assignable: `create_phase` (`Phase.new`) and `replace_form_fields`, whose field/option/matrix-statement params reach `assign_attributes` through `UpdateAllService` — an undeclared `resource_id` would re-parent a field to another form. These schemas now set `additionalProperties: false`.

To keep the `get_form_fields` → `replace_form_fields` round-trip working with closed schemas, the two tools now share a single form-field schema (the builder's input/output modes are gone) and directional keys carry the standard JSON Schema annotations: `readOnly: true` on the server-owned keys (`created_at`, `updated_at`, `resource_id`), `writeOnly: true` on `temp_id`. The annotations are validation-neutral.

`BaseTool.for` enforces the readOnly semantics by stripping annotated keys from the params at dispatch (`BaseTool::ReadonlyStrip`, unit-tested as a plain function). The strip walker deliberately supports only the schema subset the engine uses; a registry tripwire fails if a schema ever declares readOnly anywhere else.
@adessy adessy force-pushed the TAN-7901/mcp-engine-test-suite branch from 9d71797 to 373f5f3 Compare July 13, 2026 13:45
@adessy adessy requested a review from kogre July 13, 2026 14:01
@adessy adessy requested a review from amanda-anderson July 13, 2026 14:01
@adessy

adessy commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@amanda-anderson @kogre for whoever has the time... or the courage.

@amanda-anderson amanda-anderson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Didn't look as deeply into the individual specs themselves, but the ones I did look at made sense to me :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants