py: type evaluator callbacks as BaseEvalDataPoint

[jeffdh5]

Summary

Python's EvaluatorFn is typed as taking BaseDataPoint (test_case_id: str | None), but the evaluator stepper always assigns a UUID before calling the callback — same as JS, which types the callback as BaseEvalDataPoint (testCaseId required).

Current behavior

In py/packages/genkit/src/genkit/_ai/_evaluator.py:

if datapoint.test_case_id is None:
    datapoint.test_case_id = str(uuid.uuid4())
# ...
test_case_output = await fn(datapoint, req.options)

EvaluatorFn is still:

EvaluatorFn = Callable[[BaseDataPoint, T], Coroutine[Any, Any, EvalFnResponse]]

Both models already exist in genkit.evaluator / _typing.py:

Type	test_case_id
BaseDataPoint	optional (dataset input)
BaseEvalDataPoint	required

Expected

Mirror the JS split:

• Dataset / EvalRequest: BaseDataPoint (id optional)
• Evaluator callback: BaseEvalDataPoint (id guaranteed after the fill-in)

So typecheckers don't force datapoint.test_case_id or '' in custom evaluators.

Suggested fix

1. Change EvaluatorFn to Callable[[BaseEvalDataPoint, T], ...].
2. After the UUID fill-in, pass a BaseEvalDataPoint (validate/construct) into fn rather than a mutated BaseDataPoint.
3. Update first-party evaluators (genkit-evaluators, google-genai evaluators) and tests/docs that annotate the callback arg.

References

• JS: js/ai/src/evaluator.ts — testCaseId ?? randomUUID() then BaseEvalDataPoint runner type
• Python: py/packages/genkit/src/genkit/_ai/_evaluator.py (eval_stepper_fn)

[a2105z]

hi @jeffdh5 , I was able to reproduce the problem - if you don't mind, I will have a PR that will fix this in a jiff! Would love feedback from you with regards to this.

[a2105z]

@jeffdh5 this is ready for review.

Confirmed locally: evaluator callbacks now receive BaseEvalDataPoint with a guaranteed test_case_id (UUID fill-in, dataset BaseDataPoint left unmutated), and the related pytest suite is green (10/10).

PR: #5995