fix(runner): send an MCP server the answer the person actually gave - #5957
Conversation
An MCP server asks a question with `elicitation/create` and declares the shape it wants back in `requestedSchema`. Omnigent's approval card already renders an enum schema as option buttons and POSTs the choice, and the server forwards it to the runner verbatim as an `approval` event carrying `content`. The runner then read only `action` and dropped the content. It had nowhere to put it: the verdict registry was `dict[str, Future[bool]]`. So the MCP caller was handed a bare "yes" and filled the answer in from the schema instead — first enum value, or `"allow"`, or `True`. Its own comment said as much. A person choosing "prod" from dev/staging/prod had "dev" sent on their behalf, and the deployment went to the wrong place with an audit trail saying they approved it. Carry a `Verdict` instead of a bool, forward the `content` the server already sends, and prefer the person's answer over the schema guess. Three things the answer has to clear before it travels. It is checked against the schema that asked for it — primitive values, known keys, enum members — because it arrives from a browser and a body outside the contract is one the server never agreed to parse. A refusal carries no answer, normalised at the registry so no consumer has to remember. And when a schema names fields that nothing collected and nothing can be guessed, the elicitation now declines rather than sending an accept with no content: a decline is a path the server already handles, a malformed accept is not. The schema fallback stays for consent-shaped prompts — a boolean, a lone enum, an explicit default — where a yes/no surface has only one sensible value and refusing would invert the person's actual answer. Signed-off-by: Paldom <3684864+Paldom@users.noreply.github.com>
Adds an end-to-end regression test driving the full journey: an MCP
server's deploy tool elicits a 3-option enum, the approval event carries
the user's third choice ("prod"), and the tool's output must contain the
chosen value — not the schema's first option that the auto-fill used to
invent. Includes the stdio MCP fixture server that returns the answer it
received so the test can observe what reached the server.
Independent review findings on the content-threading fix: - A supplied answer that fails schema validation now declines instead of falling back to the schema guess — substituting a value nobody chose is the very bug the fix removes. - The no-content decline gate consults the schema's `required` list, so an all-optional schema still accepts a bare consent instead of inverting it into a decline. - The proxy (MRTR) path validates browser-supplied content against the inputRequest's requestedSchema before forwarding, with the same fail-closed behavior; previously it forwarded content unvalidated. - Validation is shared in omnigent.tools._elicitation_schema and now checks declared property types (a bool is not a number), enum membership, oneOf consts, and required fields.
|
UI Preview for this PR has been removed. |
|
|
Triage of the Polly review notes (all non-blocking; none require a code change):
|
|
@bbqiu this fixes #5272 (OMNI-4778) — ready for your review.
|
|
…ields
Align proxy_mcp_manager's MRTR inputResponses handling with the inline
elicitation path: a bare accept (no content) against a requestedSchema that
marks fields `required` is malformed — the MCP server rejects it and the
retry loop spins ("Approval loop exceeded") — so decline instead, matching
mcp_manager._elicit's required-aware gate.
Move the required-fields predicate into the shared _elicitation_schema
module (schema_requires_fields) so the inline and proxy paths share one
implementation, and clarify that validate_content_against_schema returns
None both when there was no content and when the content did not conform
(callers disambiguate via the supplied content's own truthiness).
Addresses the two non-blocking notes from the automated (Polly) review.
|
Polly review triage (head
Also merged latest |
I, Daniel Lok <daniel.lok@databricks.com>, hereby add my Signed-off-by to this commit: 2ffde13 Signed-off-by: Daniel Lok <daniel.lok@databricks.com>
|
|
🤖 Otto merge started Otto is reviewing this approved PR, addressing actionable review feedback and CI failures, and resolving merge conflicts if needed. It will merge only after current approval, CI, Polly, review threads, and mergeability all pass. |
…ining A bare accept (the REPL's y/n prompt, the binary approve card) collects no content, and the proxy path's new required-aware gate declined it outright. The policy-ASK schema requires an 'approved' boolean, so every REPL tool approval through the MCP proxy became 'Tool call denied by user' (E2E shard 2: test_repl_tool_call_approval_allows_tool_to_run). Match the inline elicitation path's fallback order: when nobody supplied content, auto-fill from the schema first and decline only when the schema has required fields the auto-fill cannot answer (e.g. a free-form string). Also tighten validate_content_against_schema per review: anyOf/nullable unions, numeric and length bounds, property-level const, and array items enum/bounds are now enforced on the browser-to-server content path. Signed-off-by: omni-resolve-agent[bot] <omni-resolve-agent[bot]@users.noreply.github.com>
|
🏷️ Doc impact: Internal bugfix to MCP elicitation content plumbing (carrying the user's answer through the verdict registry and validating it against the requestedSchema); no user-facing surface, integration, or documented default changed. Auto-classified on merge. Set the label manually before merging to override. · run |
|
|
Otto post-merge note This PR was merged at head Polly's review of A fix with tests is ready on this branch as commit Please open a follow-up PR from that commit (or cherry-pick it) — this PR is already merged, so Otto is not opening a new PR on its own. |
Related issue
Closes #5272 — resolves OMNI-4778 (Linear).
Builds on #5273 by @Paldom — taken over because the fork branch was conflicting with
mainand needed review fixes that could not be pushed to the fork (App tokens can't push to fork branches). Commite14b10625is theirs, carried over with authorship preserved.Summary
When an MCP server elicits input (e.g.
ctx.elicitwith an enum schemadev/staging/prod), the runner told the server the schema's first option, not the one the person actually picked.runner/app.pydropped the verdict's content —pending_approvalswas adict[str, Future[bool]], so only the boolean survived, andmcp_manager._build_accept_contentauto-filled the first enum value.proxy_mcp_managerdroppedinputResponsescontent the same way.Verdict(approved + content) throughpending_approvals; prefer the person's validated answer over the schema guess; fail closed (decline) on a non-conforming supplied answer instead of substituting one; make the no-content decline gaterequired-aware so optional-field consents still accept; and validate proxy MRTRinputResponsescontent against therequestedSchemavia a shared validator (omnigent/tools/_elicitation_schema.py).sequenceDiagram participant U as User (approval card) participant R as runner/app.py participant P as pending_approvals participant M as mcp_manager participant S as MCP server U->>R: approve + content {answer: "prod"} R->>P: resolve(Verdict(approved, content)) %% was: Future[bool] — content dropped P->>M: verdict M->>M: validate content against requestedSchema M->>S: ElicitResult(accept, {answer: "prod"}) %% was: first enum value "dev"ELI5: the approval pipe only had room for "yes/no", so the actual answer fell on the floor and the code guessed the first option from the schema. The pipe now carries the answer too, checks it against the schema, and refuses (declines) rather than guessing when the answer doesn't fit.
Test Plan
Fail→pass proof (each facet fails on unfixed
main, passes on this branch):tests/e2e/test_mcp_elicitation_user_answer.py— user picks the 3rd enum option (prod); on unfixed main the tool output waselicit_answer:dev(FAIL), on this branch it iselicit_answer:prod(PASS). Re-verified green after merging latestmain.tests/runner/test_mcp_elicitation_content.py(17 tests) — chosen option reaches the waiting caller; non-conforming answer declines (fail closed) instead of substituting; required-aware decline gate (test_an_unanswerable_schema_declines_rather_than_inventing,test_optional_fields_still_accept_without_an_answer); proxy MRTRinputResponsescarries validated content and declines non-conforming browser content.tests/runner/test_pending_approvals.py,tests/runner/test_runner_idle_active_work.py(40 tests total).Independent review: a cross-vendor (codex) reviewer flagged 2 blocking issues (invalid supplied content fell back to a schema guess; decline gate ignored
required) and 1 security issue (proxy MRTR path forwarded browser content unvalidated) — all fixed at the root with tests for each.Validate the fix live
This fix runs in the runner/host process, so check out the PR — attaching a local runner to a UI preview would run an unfixed runner:
--server ''runs a local server from this same checkout, so the server and runner are the PR build.Demo
After-fix recording (send "Please deploy." → approval card renders dev/staging/prod → click prod → tool output shows
elicit_answer:prod, the user's actual choice):recordings/mcp-elicitation-answer/after-web.mp4, preserved in the resolve run's CI artifact bundle — this environment cannot attach media files directly to the PR.Type of change
Test coverage
Coverage notes
All facets are covered by automated tests that fail on unfixed
mainand pass here (see Test Plan).Changelog
MCP elicitation now sends the server the option you actually picked, and declines answers that don't match the requested schema instead of substituting the first option