Make AbstentionBench pip-installable

[jeqcho]

Fixes #12

Transform AbstentionBench into a Pip-Installable Python Package

Summary

Transform AbstentionBench into a pip-installable Python package by reorganizing the code structure, adding proper packaging configuration, and updating import paths throughout the codebase.

Background

AbstentionBench was previously structured as a standalone project without proper Python package organization. This transformation enables the project to be installed as a dependency via pip, making it easier to integrate into other projects and evaluation frameworks like Inspect Evals.

Changes Made

Package Structure Reorganization

• Created main package directory: abstentionbench/ with proper __init__.py files
• Moved all modules into package hierarchy:
  • recipe/ → abstentionbench/recipe/
  • configs/ → abstentionbench/configs/
  • analysis/ → abstentionbench/analysis/
  • data/ → abstentionbench/data/
• Added __init__.py files for all modules to make them proper Python packages

Modern Packaging Configuration

• Added pyproject.toml with:
  • Complete project metadata (name, version, description, authors)
  • Comprehensive dependency list covering all requirements
  • Optional dependencies for development and GPU support
  • CLI entry point: abstention-bench = "main:main"
  • Package discovery configuration
  • Proper file inclusion for YAML, JSON, and CSV files

Import System Updates

• Updated main.py imports:

  • recipe.abstention → abstentionbench.recipe.abstention
  • recipe.inference → abstentionbench.recipe.inference
  • recipe.models → abstentionbench.recipe.models

• Updated all Python test files to use new import paths:

  • tests/run_llama3_instruct_inference.py
  • tests/test_abstention_detector.py
  • tests/test_correctness_evaluator.py
  • tests/test_datasets.py
  • tests/test_inference.py
  • tests/test_job_manager.py
  • tests/test_models.py

YAML Configuration Files Update

Updated all 23 dataset configuration files in /abstentionbench/configs/dataset/ to use correct import paths:

Before:

datamodule:
  _target_: recipe.abstention_datasets.world_sense.${dataset_name}

After:

datamodule:
  _target_: abstentionbench.recipe.abstention_datasets.world_sense.${dataset_name}

Files updated:

• alcuna.yaml, bbq.yaml, big_bench_disambiguate.yaml, big_bench_known_unknowns.yaml
• coconot.yaml, dummy.yaml, falseqa.yaml, freshqa.yaml
• gpqa.yaml, gsm8k.yaml, kuq.yaml, mediq.yaml
• mmlu_history.yaml, mmlu_math.yaml, moralchoice.yaml, musique.yaml
• qaqa.yaml, qasper.yaml, self_aware.yaml, situated_qa.yaml
• squad2.yaml, umwp.yaml, worldsense.yaml

Git Configuration

• Updated .gitignore to include Python package-specific patterns and build artifacts

Installation Options

# From source (recommended)
pip install .

# Development mode
pip install -e .

# With GPU support
pip install .[gpu]

# From GitHub (when available)
pip install git+https://github.com/facebookresearch/AbstentionBench.git

Usage

# New CLI command
abstention-bench --config-path=abstentionbench/configs --config-name=default_pipeline

# Python module (also supported)
python -m abstentionbench.main --config-path=abstentionbench/configs --config-name=default_pipeline

# Legacy usage (still works)
python main.py --config-path=abstentionbench/configs --config-name=default_pipeline

Testing Completed

• ✅ Package structure validation (all files in correct locations)
• ✅ Python import system validation (all new import paths work correctly)
• ✅ Installation testing (pip install -e . succeeds)
• ✅ CLI command creation (entry point accessible)
• ✅ Dependencies loading (all required packages specified)
• ✅ YAML configuration files use correct import paths
• ✅ All test files updated with new import structure
• ✅ All old tests succeed (python pytest)

Backward Compatibility

• All existing functionality preserved
• Original usage patterns still supported through main.py
• No breaking changes to evaluation logic
• Same command-line arguments and behavior maintained
• Configuration files maintain same structure and options

This transformation enables AbstentionBench to be used as a standard Python package while maintaining full backward compatibility with existing workflows.