RAGs for Open Domain Complex QA

This is a fork from the BCQA Benchmarking Complex QA repo.

The scripts used for making the OpenAI API calls and the data processing and analysis can be found in the data_analysis folder.

BCQA (Benchmarking Complex QA)

BCQA is a benchmark for a wide range of complex Qa tasks. It also aims to provide a easy to use framework for evaluating retrieval and reasoning approaches for answering complex multi-hop questions.

Setup

1. Create a conda environment conda create -n bcqa python=3.10
   
2. pip install -e .
   
3. To be able to use GPU: pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
   
4. The data paths are absolute for my pc so you need to change it to fit yours.

Running Evaluation

The evaluation scripts for retreival and LLMs are in the evaluation folder

For instance to run dpr retreival for Wikimultihopqa run
python3 evaluation/wikimultihop/run_dpr_inference.py

Before running the above script make sure you have configured the correct paths for the data and corpus files in evaluation/config.ini

Example: wikimultihopqa = /home/bcqa/BCQA/2wikimultihopQA
wikimultihopqa-corpus = /home/bcqa/BCQA/wiki_musique_corpus.json

Coding Practices

Auto-formatting code

1. Install black: pip install black or conda install black
2. In your IDE: Enable formatting on save.
3. Install isort: pip install isort or conda install isort
4. In your IDE: Enable sorting import on save.

In VS Code, you can do this using the following config:

{
    "editor.formatOnSave": true,
    "editor.codeActionsOnSave": {
        "source.organizeImports": true
    }
}

Type hints

Use type hints for everything! No exceptions.

Docstrings

Write a docstring for every function (except the main function). We use the Google format. In VS Code, you can use autoDocstring.

Example

def sum(a: float, b: float) -> float:
    """Compute the sum of a and b.

    Args:
        a (float): First number.
        b (float): Second number.
    
    Returns:
        float: The sum of a and b.
    """

    return a + b