Support a metric being a string value as well as a numeric value #639
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In working with @frejonb on using the RAGElo project (https://github.com/zetaalphavector/RAGElo) we found that sometimes a the query and overall score level we want a string value, not a number.
For example, at the per query level, we wanted to know which agent was the winner, A or B. Yes, we could have modeled that as a 0 or a 1, but that is awkward.
We also discovered that the assumption that at the top level we would just roll up the average of all query level metrics was flawed. You can't average a string, so in our POC we just took the first value as the value to show at the Experiment level.
This PR doesn't touch on the idea that we need to store Experiment level metrics at the Experiment level instead of just averaging all the query level data.
Description
If a value is a String, treat it as String.
Issues Resolved
[List any issues this PR will resolve]
Check List
--signoffBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.