多模态reward模型是否支持正负样本为图片的数据格式？

[JoshonSmith]

我想利用rm.sh训练一个多模态模型，用于判断相同提示词生成的图片的好坏，
数据格式如下：

{"messages": 
    [
    {"role": "user", "content": "Please generate a video based on the text prompt: [a dog swimming.]"}, 
    {"role": "assistant", "content": "<image>",
    "images": ["/xxx/Good.jpg"]
    }
    ], 
    "rejected_response": "<image>",
    "rejected_images": ["/xxx/Bad.jpg"]
}

请问是否支持这种针对图像生成或视频生成任务的多模态奖励模型训练？
是否支持传入视频文件？

[JoshonSmith]

@Jintao-Huang 请问作者，或者下面这种是否可以：

{"messages": 
    [
    {"role": "user", "content": "Please generate a video based on the text prompt: [a dog swimming.] <image>"},
    "images": ["/xxx/Good.jpg"]
    }
    ], 
    "rejected_images": ["/xxx/Bad.jpg"]
}

[slin000111]

我想利用rm.sh训练一个多模态模型，用于判断相同提示词生成的图片的好坏， 数据格式如下：

{"messages": 
    [
    {"role": "user", "content": "Please generate a video based on the text prompt: [a dog swimming.]"}, 
    {"role": "assistant", "content": "<image>",
    "images": ["/xxx/Good.jpg"]
    }
    ], 
    "rejected_response": "<image>",
    "rejected_images": ["/xxx/Bad.jpg"]
}

请问是否支持这种针对图像生成或视频生成任务的多模态奖励模型训练？ 是否支持传入视频文件？

https://swift.readthedocs.io/zh-cn/latest/Customization/%E8%87%AA%E5%AE%9A%E4%B9%89%E6%95%B0%E6%8D%AE%E9%9B%86.html#dpo-orpo-cpo-simpo-rm