Skip to content

Add SycoBench-600 baseline task#3919

Open
debu-sinha wants to merge 1 commit into
EleutherAI:mainfrom
debu-sinha:add-sycobench-600
Open

Add SycoBench-600 baseline task#3919
debu-sinha wants to merge 1 commit into
EleutherAI:mainfrom
debu-sinha:add-sycobench-600

Conversation

@debu-sinha

Copy link
Copy Markdown

Summary

Adds sycobench_600_baseline, a zero-shot multiple-choice baseline task for SycoBench-600.

SycoBench-600 is an ACL Findings 2026 benchmark for measuring LLM sycophancy and correction selectivity. This lm-evaluation-harness task intentionally covers the single-turn MCQ baseline over the public Hugging Face dataset (dsinha/sycobench-600). The full multi-turn pressure/correction protocol remains in the SycoBench Inspect task, which is listed in the Inspect Evals Register.

Changes

  • Add lm_eval/tasks/sycobench/sycobench_600_baseline.yaml
  • Add a task README with the paper, DOI, dataset, reference implementation, and scope notes
  • Add SycoBench to the task-family README

Validation

  • python -m lm_eval validate --tasks sycobench_600_baseline -> All tasks found and valid
  • python -m lm_eval run --tasks sycobench_600_baseline --model dummy --limit 1 completed a one-sample smoke run and issued 4 multiple-choice loglikelihood requests
  • Live Hugging Face dataset sanity check: load_dataset(dsinha/sycobench-600, split=test, download_mode=force_redownload) loaded 600 rows

@debu-sinha debu-sinha requested review from 0xSMT and baberabb as code owners July 8, 2026 04:12
@CLAassistant

CLAassistant commented Jul 8, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants