🤹 Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

Yinghui He, Ling Yang, Jiarui Liu, Yongjin Yang, Lechen Zhang,
Yingcheng Wu, Zhenfei Yin, Mengdi Wang, Sanjeev Arora

---

Long-horizon tasks require models to switch between skills, not just execute a single skill well. This repository contains the full pipeline behind the paper:

• Skill²-Bench: a benchmark of multi-step tasks that force models to switch between skills — 558 skills across 9 domains (math, science, coding, logic, information extraction, planning, creative writing, context retrieval, instruction following). Also available on Hugging Face: Gen-Verse/Skill2-Bench.
• Skill entropy: a measure of how difficult a skill switch is: high when a reference model handles two skills well separately but fails when combining them. Model performance drops as skill entropy rises.
• Skill-Entropy RL: GRPO training whose reward scores the model's skill-switching plan, not just its final answers — the reward combines a final-answer reward with a skill-entropy reward on skill-annotated rollouts.

Get Started

Python ≥ 3.10, CUDA 12.x, PyTorch 2.8 recommended.

# core trainer (installs verl fork + dependencies)
cd skill_entropy_rl && pip install -e . && cd ..

# eval extras
pip install antlr4-python3-runtime==4.11.1 word2number timeout-decorator

Every script resolves the repo location from one environment variable:

export SKILL_ENTROPY_RL_ROOT=/path/to/this/repo
mkdir -p $SKILL_ENTROPY_RL_ROOT/job_log   # SLURM launchers write their logs here (gitignored)

Running the pipeline

1. Entropy calibration (needs an API gateway)

bash label_skills/label_skills.sh                # skill labeling (vLLM)
bash label_skills/deduplicate_skills.sh          # merge + dedupe labels → label_skills/merged_skill_labels/
cd calculate_skill_entropy && bash run.sh       # reference-model calibration → entropy map

2. Task generation

bash propose_tasks/propose_tasks_api.sh          # benchmark tasks (API proposer — produced skill2_bench/)
sbatch propose_tasks/propose_tasks.sh            # math RL task pool (vLLM proposer → propose_tasks/math/)

3. SFT warm-up

sbatch sft/sft.sh                                # cross-skill SFT (sft_math.sh for math-only)
# regenerate SFT traces from a task pool: sft/generate_skillrl_sft.sh / generate_skillrl_sft_math.sh

4. RL (GRPO / Skill-Entropy RL)

# builds train/val parquets, then launches training
sbatch skill_entropy_rl/examples/grpo_trainer/run_skill_entropy_rl.sh           # Skill-Entropy RL
sbatch skill_entropy_rl/examples/grpo_trainer/run_skill_entropy_rl_math.sh      # math RL (needs a stage-2 math pool in propose_tasks/math/)
# chained multi-day runs: skill_entropy_rl/examples/grpo_trainer/launch_chain.sh

Reward variants (e.g. gated / multi-gold r_ent) are selected with +env.reward_mode=<mode> and implemented in skill_entropy_rl/agent_system/environments/env_package/skill2_bench/reward_variants.py.

5. Evaluation

bash  scripts/merge_checkpoint.sh                  # first: merge a sharded RL checkpoint to HF format
sbatch evaluation/eval_checkpoint.sh               # Skill²-Bench eval of a checkpoint (vLLM)
bash  evaluation/eval_api_models.sh                # Skill²-Bench eval of API models (Claude/GPT/Gemini via gateway)

📝 Citation

@article{he2026skillnative,
  title={Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning},
  author={He, Yinghui and Yang, Ling and Liu, Jiarui and Yang, Yongjin and Zhang, Lechen and Wu, Yingcheng and Yin, Zhenfei and Wang, Mengdi and Arora, Sanjeev},
  journal={arXiv preprint arXiv:2608.05139},
  year={2026}
}

📄 License

MIT (see LICENSE).