MCP engine test suite#14249
Open
adessy wants to merge 8 commits into
Open
Conversation
`update_project`, `update_phase`, `update_resource` and `replace_form_fields` called neither `authorize_project!` nor Pundit `authorize`, so the documented draft-only constraint was not enforced for updates: a super admin could mutate published projects through the MCP. Add both guards after the not-found check, per the convention every other mutating tool follows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pecs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New specs for `update_phase`, `get_form_fields` and `replace_form_fields`; deepened `create_phase` (feature-flag-driven schema, participation-method variants) and `update_phase_permission` (demographic_questions semantics, rollback, verified expiry). Also fixes a real round-trip bug the new specs caught: `Serializers::CustomField` emitted a `constraints` key that is not a `CustomField` attribute, so echoing `get_form_fields` output back into `replace_form_fields` crashed `UpdateAllService`. The MCP serializer now drops the key (constraints are reported at the top level of the response). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brings every existing mutating-tool spec up to the category template: `structured_content[:id]` assertions on happy paths, structured validation-error examples, parent not-found via the `be_not_found` matcher, published-project refusals for `update_project`/`update_resource`, dispatch-table completion for `update_resource` (event, poll_question, poll_option), option-transaction rollback for `create_poll_question`, and the file-pool survival assertion for `destroy_resource`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One spec per list tool following the list-tool canvas: scoped happy path, serialized shape assertion, shared pagination examples, parent not-found, and per-filter examples. `get_resource` covers all nine types of its dispatch table, doubling as execution coverage for the domain serializers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Serializers::Base Direct unit coverage for Pagination (envelope, clamping, serialization paths), MultilocMerge, ResponseHelpers, the multiloc schema helper, JSONAPI flattening (id/ids suffixes, inlining) and the serializer base class (wraps, inline, from-scratch, collection semantics). base_tool_spec gains suite-wide definition invariants and `.unauthorized_message` coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The request spec is the only place the HTTP gating is exercised: Doorkeeper `mcp:access` scope, `McpPolicy` (super admin + feature flag), the RFC 9728 WWW-Authenticate challenge, and end-to-end `tools/list` / `tools/call` through a real bearer token. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tract Download failures surface as `carrierwave_download_error` model validation errors (not the `CarrierWave::DownloadError` rescue, which turns out to be defensive-only for the remote_url path). Adds `stub_failing_remote_download` next to the existing download stubs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The MCP engine was built velocity-first with minimal test coverage. This PR adds a systematic test suite, plus fixes for two real bugs the new specs surfaced.
Production fixes (found while writing the suite)
update_project,update_phase,update_resourceandreplace_form_fieldscalled neitherauthorize_project!nor Punditauthorize, so the draft-only constraint was not enforced for updates: a super admin could mutate published projects through the MCP. Both guards are now in place, matching every other mutating tool.list_eventsandlist_causesfiltered without looking up the parent, so a nonexistentproject_id/phase_idreturned[](an LLM reads that as "no events" rather than "wrong id"). They now return the standard not-found error like their siblings.remote_header_bg_urlwas a non-nullable string; it now acceptsnull, whichupdate_projecttranslates intoremove_header_bg!(CarrierWave ignores plain nil assignment), mirroring the web API'sremove_image_if_requested!.replace_form_fieldsvalidation errors — form-level error keys (stale_data,locked_deletion, …) were relayed as a raw JSON dump; they now map to actionable prose (re-fetch and retry, echo locked fields back, structural page rules), with the raw errors still instructuredContent.visible_to: 'groups', the project payload carried no group information, the schemas had no way to set the group list, andlist_groups'project_idfilter matched smart-group rules referencing the project rather than the groups that can see it. The project schemas now acceptgroup_ids(replace-wholesale, likearea_ids), the serializer surfacesgroup_ids, and the misleading filter is removed (read ids from the payload, resolve names vialist_groups).get_form_fields→replace_form_fieldsround-trip —Serializers::CustomFieldemitted aconstraintskey (from the web serializer) that is not aCustomFieldattribute; echoing fetched fields back intoreplace_form_fieldscrashedUpdateAllServicewithActiveModel::UnknownAttributeError. The MCP serializer now drops the key (constraints are reported at the top level of theget_form_fieldsresponse). The round-trip is covered by a spec.Test strategy
Depth follows risk: complex tools (
create_phase's feature-flag-driven schema,update_phase_permission'sdemographic_questionssemantics, form-fields round-trip/guards,destroy_resourcecascades) get deep bespoke specs; thin list tools get short, deliberately uniform specs.Similar tools are tested with an identical structure ("canvas" per category: list / create / update / get / attach), documented in the engine
CLAUDE.md§12 (kept in the private claude repo since**/CLAUDE.mdis gitignored here). New helpers:be_not_found('Label')matcher (spec/support/matchers/be_not_found.rb)it_behaves_like 'a paginated list tool'shared examples (spec/support/mcp_server/shared_examples/paginated_list_tool.rb), used by all 13 paginated list specsstub_failing_remote_downloadnext to the existing download stubsShared examples deliberately not created (over-factorization would hurt more than help):
multiloc_merge_spec.rb; each update tool keeps one small integration example proving it routes through the merge.be_not_foundmatcher reduces it to a 3-line example.Serializers: no direct specs for domain serializers — they're exercised through
get_resource_spec(all 9 types), list-spec shape assertions and the form-fields specs. OnlySerializers::BaseandMcpServer::Jsonapi(where the logic lives) have unit specs.New layers beyond tools:
spec/requests/mcp_spec.rb— the only place the HTTP gating is exercised: Doorkeepermcp:accessscope,McpPolicy(super admin + feature flag), RFC 9728WWW-Authenticatechallenge, end-to-endtools/list/tools/callwith a real bearer token. Doubles as theMcpPolicyspec.BaseTool::Pagination,MultilocMerge,ResponseHelpers,Multiloc,Jsonapi,Serializers::Base; suite-wide definition invariants inbase_tool_spec.rb.Coverage (lines,
engines/commercial/mcp_server/app/**)Remaining uncovered lines are defensive: the
CarrierWave::DownloadErrorrescues in the attach tools (download failures actually surface as model validation errors — covered as such) andRunner#run'sNotImplementedError(covered) — plusBaseTool::Authorization#pundit_user(see below).Notes / follow-ups (not addressed here)
List tools should go through Pundit policy scopes (
policy_scope) instead of querying unscoped (Group.all,Volunteering::Cause.where(...), …). TodayMcpPolicyrestricts the endpoint to super admins so nothing leaks, but the tools silently rely on that outer gate; scoping them makes the invariant local and future-proofs any loosening of endpoint access.update_phase_permissionreplacesaccess_denied_explanation_multilocwholesale instead of merging like other update tools — the spec encodes current behavior with an explicit test name; consider aligning.BaseTool::Authorization#pundit_useris dead code: theincludedhook includesPundit::Authorizationafterwards, whose identicalpundit_usershadows it.Remaining "settable but not clearable" fields (same class as the header_bg fix):
update_phasecan't clearend_at(the only way to make the last phase open-ended);update_phase_permissioncan't clearaccess_denied_explanation_multiloc(schema rejects null and{}, and the Runner's.compactdrops nils);update_resourcesilently no-ops onremote_image_url: nullfor causes (CarrierWave ignores nil — needsremove_image!). The embed-URL fields were checked and are fine as is (presencevalidations make clearing invalid where they matter).Mass-assignment gaps (implemented, currently stashed —
git stash):create_phasesplats params intoPhase.newwithoutadditionalProperties: false;replace_form_fields' field/option/statement item schemas accept undeclared keys that reachCustomField.new/assign_attributes, includingresource_id(re-parenting a field to another form). The stash closes both schemas, declares the read-only keys for round-tripping and strips them in the tool before saving.Runner::NOT_DRAFT_MESSAGEends with a period, andunauthorized_messageappends another one → "…via MCP..".Feature-gated participation methods are enforced only by schema validation at MCP dispatch (the gem's
validate_tool_call_argumentsdefault; pinned by a request-spec example) — consider a runtime flag check inCreatePhase's Runner as defense-in-depth.🤖 Generated with Claude Code