Skip to content

Issues: open-compass/opencompass

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

259 Open 364 Closed
259 Open 364 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[Bug] 示例代码 opencompass ./examples/eval_simpleqa.py 跑不通
#1885 opened Feb 20, 2025 by superzhangmch
2 tasks done
@acylam
[Feature] Can we support terminate dlc and volc tasks when oc evaluation task is terminated.
#1884 opened Feb 20, 2025 by zhulinJulia24
1 task
@acylam
[Bug] human-eval assertion failed while testing partial data
#1880 opened Feb 19, 2025 by dengbinbox
2 tasks done
[Bug] DeepSeek R1 32B 模型 测评 AIME2024 数据集 得分低
#1878 opened Feb 18, 2025 by carllisicau
2 tasks done
[Feature] 我需要对本地部署的Qwen-110B模型进行MMLU基准测试，请问该怎么操作呢？
#1877 opened Feb 18, 2025 by hi112233445566
1 task
[Feature] 目前是否有适配Codeforces、SWE Verified、Aider-Polyglot这些在R1中出现的数据集的计划呢？
#1875 opened Feb 17, 2025 by linbeyoung
1 task
@bittersweet1999
[Bug] Medbench dataset only provides test data, not the entire dataset
#1874 opened Feb 16, 2025 by ryan0980
2 tasks done
@tonysy
[Bug] The relative path in tools/list_configs.py
#1873 opened Feb 15, 2025 by Sibyl233
2 tasks done
@tonysy
2
[Feature] expose max_task_size in run.py, for quick debug
#1869 opened Feb 13, 2025 by yxdyc
1 task
@acylam
[Feature] 是否可以有输出结果后，单独去计算一下评分的功能
#1866 opened Feb 12, 2025 by Dagoli
1 task
@MaiziXiao
[Feature] 请问如何加载本地模型进行测评
#1865 opened Feb 11, 2025 by Castrol68
1 task
@liushz
[Bug] 使用提示词攻击，出现导入模块失败
#1860 opened Feb 8, 2025 by pomliuxj
2 tasks done
@tonysy
[Bug] Only Debug Mode can perform eval tasks correctly.
#1859 opened Feb 8, 2025 by GenerallyCovetous
2 tasks done
3
[Bug] chinese simpleQA dataset is not working
#1858 opened Feb 7, 2025 by hailsham
2 tasks done
@tonysy
[Bug] 在对DeepSeek-R1-Distill-Qwen-1.5B模型评测livecodebench数据集时，lcb_test_output为什么为0呢？
#1856 opened Feb 7, 2025 by guoguo1314
2 tasks done
@tonysy
1
[Bug] MBPP score significantly lower than official results
#1855 opened Feb 7, 2025 by GenerallyCovetous
2 tasks done
[Feature] humaneval_postprocess历史无效的单测清理或case更新
#1847 opened Jan 24, 2025 by anfreshman
1 task done
@tonysy
[Bug] 部分数据集中对question进行增广时使用浅拷贝会导致选项与答案错位
#1845 opened Jan 23, 2025 by QuanWanLear
2 tasks done
@acylam
1
Missing one required positional argument in tools/collect_code_preds.py#L190
#1844 opened Jan 23, 2025 by grassFlamingo
@acylam
Miss using GPT3 in GPT4.py?
#1836 opened Jan 20, 2025 by xcfcode
@bittersweet1999
[Bug] 无法使用多卡评测
#1828 opened Jan 16, 2025 by GenerallyCovetous
2 tasks done
@acylam
7
[Bug] 使用leval测评时报错，求助
#1825 opened Jan 15, 2025 by fxnie
2 tasks done
@MaiziXiao
8
[Bug] longbench报错
#1824 opened Jan 15, 2025 by fxnie
2 tasks done
@MaiziXiao
3
[Feature] 请问是否能有一个汇总，就是opencompass包含的所有评测数据集、这些数据集的metrics和链接(paper/github)？
#1820 opened Jan 14, 2025 by 141forever
1 task
@liushz
2
[Bug] 部分样本上opencompass的评测方法是否可能有点片面？
#1818 opened Jan 14, 2025 by laogonggong847
2 tasks done
@liushz
ProTip! Follow long discussions with comments:>50.