Skip to content

feat: add AI Config judge support#345

Merged
knfreemLD merged 9 commits intov7from
kfreeman/ai-judge-go-sdk
Feb 10, 2026
Merged

feat: add AI Config judge support#345
knfreemLD merged 9 commits intov7from
kfreeman/ai-judge-go-sdk

Conversation

@knfreemLD
Copy link
Contributor

@knfreemLD knfreemLD commented Feb 5, 2026

Requirements

  • I have added test coverage for new or changed functionality
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions

Related issues

See https://docs.google.com/document/d/1lzYwQqCcTzN_2zkxJZDfJtgUcEJ4jbpx0KSsJ2bRENw/edit?tab=t.0#heading=h.5d8l30brvyuw for context

For other SDK implementations, see:

Describe the solution you've provided

Extending the Go SDK to support AI Config evaluations. This includes custom evaluator support as well.

This SDK was written with hopes to be congruent with the python and node implementations. Changes were verified by a local app that was created; the resultant data can be observed in the evaluator metrics for this AI config.

Describe alternatives you've considered

Provide a clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context about the pull request here.


Note

Medium Risk
Adds new evaluation and metric-tracking paths (including dynamic metric keys and new event payload fields), which could affect analytics correctness and runtime behavior if misconfigured. Changes are well-covered by tests but touch core SDK tracking surfaces.

Overview
Adds judge-mode support to AI Configs by extending the config datamodel and builder with mode, evaluationMetricKey/evaluationMetricKeys, and judgeConfiguration (with defensive copying to keep configs immutable).

Introduces Client.JudgeConfig to fetch judge configs while preserving {{message_history}} / {{response_to_evaluate}} placeholders for a second Mustache interpolation pass during evaluation, and adds a new ldai/judge package that samples, interpolates, invokes a structured provider, and parses judge responses.

Extends Tracker with TrackJudgeResponse to emit evaluation scores as metrics (including optional judgeConfigKey in event data), and adds comprehensive tests covering parsing, placeholder preservation, schema generation, sampling, and response validation.

Written by Cursor Bugbot for commit 41141b9. This will update automatically on new commits. Configure here.

@knfreemLD knfreemLD requested a review from jsonbailey February 5, 2026 18:04
@knfreemLD knfreemLD requested a review from a team as a code owner February 5, 2026 18:04
knfreemLD and others added 3 commits February 5, 2026 13:20
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

@knfreemLD knfreemLD merged commit 4a9d03d into v7 Feb 10, 2026
24 checks passed
@knfreemLD knfreemLD deleted the kfreeman/ai-judge-go-sdk branch February 10, 2026 18:08
jsonbailey added a commit that referenced this pull request Feb 10, 2026
🤖 I have created a release *beep* *boop*
---


##
[0.8.0](ldai/v0.7.2...ldai/v0.8.0)
(2026-02-10)


### Features

* add AI Config judge support
([#345](#345))
([4a9d03d](4a9d03d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Release metadata and changelog-only changes; no functional code
changes beyond the version constant update.
> 
> **Overview**
> Publishes `ldai` v0.8.0 by updating the version in
`.release-please-manifest.json` and `ldai/package_info.go`.
> 
> Updates `ldai/CHANGELOG.md` with a new 0.8.0 entry noting the added
*AI Config judge support*.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
f1203ee. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jason Bailey <jbailey@launchdarkly.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments