Agent Evaluation framework

[nyrahul]

Equivalent to https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-create

[avp9-nexus]

Since this is scoped as "equivalent to Copilot Studio", one thing worth flagging before the shape is fixed: that framework measures answer quality. Its methods are general quality, compare meaning, tool use, keyword match, text similarity, exact match. Most of them are built around an expected answer and a pass score, and the closest thing to a safety check, tool use, verifies that the run called the resources it was supposed to call. None of them verifies that an agent refused something it should have refused.

For most agent platforms that is a reasonable place to start. For this one it leaves out the property the platform is actually selling. A default-deny sandbox, a credential the agent never holds, and secrets scoped to declared hosts are all claims about what does not happen. A test set made only of expected-behaviour cases cannot distinguish an enforcement layer that works from one that permits everything, because both produce the same green.

The shape that covers it is a negative suite: each vector is a frozen input, an expected verdict, and an expected first-failure code, replayable offline with no credentials. Three properties matter more than the vectors themselves. First, a positive control per category, an input near-identical to the negative one that must pass, otherwise a suite that refuses everything scores perfectly. Second, a denial has to name its reason, because a correct verdict reached for the wrong reason passes today and silently stops protecting the moment the cause changes. Third, the verdict vocabulary needs at least two denial states: evidence measured and found wrong is not the same as evidence that could not be measured, and collapsing them either fails open on the unmeasurable case or sends operators hunting for attacks that never happened.

We run this against our own enforcement layer and contributed it as an open PR to the OWASP GenAI Agent Control Standard, where it is 18 vectors across six categories, 6 positive controls, and 9 REJECT plus 3 UNMEASURABLE cases, with a runner and a reference adapter, Python and no third-party dependencies. Apache-2.0, same as here, so it can be lifted rather than reimplemented if the shape is useful. One caution from the literature while the categories are still open: in the Bounded Agents evaluation (arXiv 2608.15888), removing a single pair from the restriction set moved data-stealing success from zero to 39.9 percent, silently, because every individual call still passed its own check. Whatever a first evaluation set covers, saying what it does not cover is worth as much as the vectors.

Not asking for anything, and this is your scope call rather than mine. If a negative category is useful I am happy to draft one against the sandbox and secret-injection behaviour specifically. Related from the same angle: #31 attaches a budget quota to a workflow run or to an agent, and a quota is a refusal that can be tested the same way.