Skip to content

Latest commit

 

History

History
82 lines (74 loc) · 6.3 KB

README.md

File metadata and controls

82 lines (74 loc) · 6.3 KB

autohub-benchmark

This project designs common use scenarios for web-based code, model, and dataset hosting platforms, and provides corresponding prompts and ground truth. These resources can be used to evaluate the localization performance of visual language models (VLMs) in specialized scenarios.

Model Hosting Platform GUI Inference

Model Platform Accuracy (%) Error (%) Invalid (%) Completion Rate (%)
AriaUI Huggingface 70.8 12.5 6.7 100.0
ModelScope 57.6 14.2 28.2 100.0
OpenCSG 81.0 9.5 9.5 100.0
CogAgent Huggingface 73.3 26.7 0.0 100.0
ModelScope 57.9 29.1 13.0 96.3
OpenCSG 57.1 19.0 23.8 100.0
Qwen3B Huggingface 8.3 15.8 19.2 41.7
ModelScope 0.0 28.6 20.6 49.2
OpenCSG 4.8 4.8 9.5 19.0
Qwen7B Huggingface 73.3 11.7 10.8 95.8
ModelScope 55.5 30.2 8.5 95.2
OpenCSG 71.4 14.3 14.3 100.0
SeeClick Huggingface 39.2 36.7 24.2 100.0
ModelScope 52.4 29.0 18.6 100.0
OpenCSG 52.4 14.3 33.3 100.0
ShowUI Huggingface 30.0 45.0 11.7 86.7
ModelScope 43.3 26.7 14.3 88.9
OpenCSG 23.8 52.4 9.5 85.7

Summery:

Model Accuracy (%) Error (%) Invalid (%) Completion Rate (%)
AriaUI 67.7 19.3 11.4 100.0
CogAgent 63.3 34.8 3.0 98.7
Qwen3B 4.5 10.8 12.9 62.6
Qwen7B 66.9 18.8 10.1 100.0
SeeClick 45.6 26.2 26.8 97.9
ShowUI 32.1 42.8 11.6 85.0

Code Hosting Platform GUI Inference

Model Platform Accuracy (%) Error (%) Invalid (%) Completion Rate (%)
AriaUI GitCode 57.1 28.5 14.3 100.0
Gitea 71.4 28.5 0.0 100.0
Gitee 57.1 28.5 14.3 100.0
Github 71.4 14.3 14.3 100.0
GitLab 71.4 14.3 14.3 100.0
CogAgent GitCode 71.4 28.5 0.0 100.0
Gitea 71.4 28.5 0.0 100.0
Gitee 100.0 0.0 0.0 100.0
Github 57.1 42.8 0.0 100.0
GitLab 85.7 14.3 0.0 100.0
Qwen3B GitCode 14.2 28.5 42.8 85.7
Gitea 14.2 57.1 14.2 85.7
Gitee 14.2 42.8 28.5 100.0
Github 0.0 28.5 57.1 85.7
GitLab 14.2 28.5 28.5 71.4
Qwen7B GitCode 71.4 0.0 28.5 100.0
Gitea 57.1 28.5 14.2 100.0
Gitee 28.5 57.1 14.2 100.0
Github 0.0 14.2 85.7 100.0
GitLab 85.7 14.2 0.0 100.0
SeeClick GitCode 28.5 48.5 28.5 100.0
Gitea 28.5 28.5 48.5 100.0
Gitee 28.5 57.1 14.2 100.0
Github 14.2 57.1 28.5 100.0
GitLab 0.0 71.4 28.5 100.0
ShowUI GitCode 28.5 48.5 14.2 85.7
Gitea 57.1 48.5 0.0 100.0
Gitee 57.1 28.5 0.0 85.7
Github 48.5 14.2 28.5 85.7
GitLab 48.5 14.2 14.2 71.4

Summery:

Model Platform Accuracy (%) Error (%) Invalid (%) Completion Rate (%)
AriaUI 65.7 22.8 11.4 100.0
CogAgent 62.9 22.8 0.0 100.0
Qwen3B 11.4 37.1 37.1 85.7
Qwen7B 48.5 22.9 28.6 100.0
SeeClick 20.0 51.4 28.6 100.0
ShowUI 45.7 28.6 11.4 85.7