Skip to content

Sui MCP: effectiveness eval (Promptfoo) — required acceptance #461

Description

@bidzyyys

Sub-issue of #376. The founding "how do we test its effectiveness?" acceptance. Owned by #376 (not shared with #377). Depends on #459 and #460.

Scope

  • Promptfoo harness with a corpus of natural-language asks (e.g. "access-controlled treasury with a timelock and a rate limiter") → expected recipe id(s).
  • Deterministic scoring vs examples/ as ground truth:
    • tool-call-f1 — correct recipe(s) selected;
    • sui move build on the scaffolded package;
    • r.mvr deps resolve;
    • sui move test passes;
    • zero anti-pattern hits.
  • Run pre-release / on schedule (needs Sui CLI + model calls).

Acceptance: harness runs against the live server and reports a pass/fail score per ask; seed corpus authored.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Ready for Dev

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions