BCaLLM

BCaLLM: Call Graph-Guided Python Breaking Change Detection with Large Language Models, ISSTA 2026

Quickstart

Prepare Environment

Our BCaLLM is developed on Ubuntu 20.04.

conda create -n BCaLLM python=3.10
pip install -r requirements.txt

Uncompress the benchmark：

unzip PyBCEval.zip

Please set your API KEYs in bin/model_config.yaml.

Usage Example

Detect breaking changes for 2.0.1 and 2.1.0 of the Python package cachetools:

cd bin && python main.py --v1 2.0.1 --v2 2.1.0 --v1_path ../PyBCEval/repos/cachetools/2.0.1/cachetools-2.0.1 --v2_path ../PyBCEval/repos/cachetools/2.1.0/cachetools-2.1.0 --save_dir ../exp_results/results/cachetools --method ours --llm deepseek --force

Experiments

PyBCEval Dataset

It is saved in PyBCEval:

• repos: source code of Python packages;
• apis: specific APIs to be detected;
• metadata.json: metadata of PyBCEval;
• reference.json: pure file for ground truth;

Run

Replace method and llm to conduct experiments:

• method: cg-k, cg-inf, llm-k, ours, ours-k, ours-wosum (i.e., BCaLLM without change summaries);
• llm: gpt-4o, deepseek, qwencoder-32b, qwen3-8b.

cd experiments && python main.py --dataset_dir ../PyBCEval --out_dir ../exp_results/results --method ours --llm deepseek --only_targets --out_file ../exp_results/eval_results/ours#deepseek.json --force

Evaluate

All experiment results are saved in exp_results.

Compute precision, recall, F1-score, and exact match:

cd experiments && python evaluate.py --gt_file ../PyBCEval/reference.json --pred_file ../exp_results/eval_results/ours#deepseek.json

Citation

If you find the work useful, please kindly cite it as follows:

@inproceedings{BCaLLM,
  author    = {Wei Cheng and Chen Shen and Huan Zhang and Yuhan Wu and Jingyue Yang and Wei Hu},
  title     = {BCaLLM: Call Graph-Guided Python Breaking Change
Detection with Large Language Models},
  booktitle = {ISSTA},
  year      = {2026}
}