pairwise comparison data

[wwfnb]

A wonderful job！！！ I am using the benchmark provided by you to test my Reward model. But I have encountered some problems with the pairwise comparison data. Specifically，I used the test code provided in your github repository[scripts/gen_batch_api_judge.py]. But I found that this code only generates Judge responses for pair data. I'm not sure which answer is better so I have no way to calculate the score of the test result. I checked the entire code repository but didn't find the file storing this information. Could you help me? Thank you very much.

[URRealHero]

Hi there, what do you want for calculating the score? Currently, the code only support pair comparision, if you want to generate MLLM-score, you can refer to the prompt in "scripts/utils/ScoringPrompt"
If your question is about "what answer is better", sorry for the unclear instruction. But in our experiment setting:
0 means the first response is better, 1 means the two responses are equally good, 2 means the second response is better.

[wwfnb]

Hi there, what do you want for calculating the score? Currently, the code only support pair comparision, if you want to generate MLLM-score, you can refer to the prompt in "scripts/utils/ScoringPrompt"您好，请问您需要什么来计算分数？目前代码只支持成对比较，如果您想生成 MLLM 分数，可以参考“scripts/utils/ScoringPrompt”中的提示。 If your question is about "what answer is better", sorry for the unclear instruction. But in our experiment setting:如果你的问题是“哪个答案更好”，抱歉，我的说明不太清楚。但在我们的实验设置中： 0 means the first response is better, 1 means the two responses are equally good, 2 means the second response is better.0 表示第一个反应更好，1 表示两个反应同样好，2 表示第二个反应更好。

感谢您的回答。我的问题是， 我们没能找到存储了您所说的“0,1,2” 这样表示两个模型回答哪个更好的文件。在我们理解中， dataset/X2XRawBenchmark/X2XBenchmark.json 存储了benchmark中所有的Query，dataset/ResponseCollection/X2XBenchmarkResponse.json 存储了所有模型的回答，scripts/gen_batch_api_judge.py代码中成对比较的会遍历所有的Query并且找到对该Query做出响应的模型回答，拼接prompt，两两组成“models1 vs models2”的prompt，然后测试模型，让其判断那个Response更好。但是我们好像没有在仓库中找到这个golden 答案。

[URRealHero]

Hmmm, I still didn't quite understand what's the problem. You are right about prompt construction.
I think "0,1,2" is under scripts/utils/schemas.py? is that what you mean？

[wwfnb]

Hmmm, I still didn't quite understand what's the problem. You are right about prompt construction. I think "0,1,2" is under scripts/utils/schemas.py? is that what you mean？

感谢您的回复，我知道您在scripts/utils/schemas.py 定义了 Choice，0表示第一个回答更好。我们的问题是，我现在模型已经给出了判断，比如我的模型认为第一个回答更好，我怎么知道这个判断是正确的，也就是说我们好像并没有找到人工标注的golden数据。 很抱歉占用了您的时间。

[URRealHero]

oops sry, I will upload the golden truth this week

[wwfnb]

oops sry, I will upload the golden truth this week

我们还有一个问题， 我们在https://huggingface.co/datasets/pudashi/JudgeAnything 中下载您提供的数据集，我们在其中的 /Arena/arenaresult/Result/gang_Understanding_record.json 文件中找到了这样的记录 "34d8478e": [ "Unified-IO2 vs Next-GPT Unified-IO2 wins", "ModaVerse vs CoDi ModaVerse wins" ], 请问这表示对于unqd_id为”34d8478e“的任务， Unified-IO2的回答比Next-GPT的回答更好吗。我们正在尝试从这个文件中抽取可以用来评测的pair comparision 数据，但是我们比较疑惑的是，这个文件中的模型名称”Unified-IO2“ 和 github仓库dataset/ResponseCollection/X2XBenchmarkResponse.json 中提供的模型名称（salmonn， gemini， qwen， gama）好像是不一样的，我们不太清楚这个里面的区别。 十分感谢您的回复

[URRealHero]

Arena里面用的是可以进行所有Omni任务（MMU MMG）的模型，和前面用来测评的不一样