Skip to content

Conversation

@epugh
Copy link
Collaborator

@epugh epugh commented Sep 16, 2025

In working with @frejonb on using the RAGElo project (https://github.com/zetaalphavector/RAGElo) we found that sometimes a the query and overall score level we want a string value, not a number.

For example, at the per query level, we wanted to know which agent was the winner, A or B. Yes, we could have modeled that as a 0 or a 1, but that is awkward.

We also discovered that the assumption that at the top level we would just roll up the average of all query level metrics was flawed. You can't average a string, so in our POC we just took the first value as the value to show at the Experiment level.

This PR doesn't touch on the idea that we need to store Experiment level metrics at the Experiment level instead of just averaging all the query level data.

Description

If a value is a String, treat it as String.

Issues Resolved

[List any issues this PR will resolve]

Check List

  • [ x] New functionality includes testing.
    • [ x] All tests pass, including unit test, integration test
  • New functionality has been documented.
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@epugh epugh added the 3.4.0 label Sep 17, 2025
@epugh epugh removed the 3.4.0 label Sep 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant