The code currently works (see test.ipynb), but may change in the future.
I intend to make it into a pypi package eventually. Requirements have not been listed, please look at the requirements of the dependencies for now. (Specifically the radgraph package's dependencies)
This repository reimplements the RadCliQ composite metric as defined in the paper "Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation".
This reimplementation allows for the calculation of RadCliQ via a function all in memory, while the original needed the ground truth and predicted reports to be saved to csvs in order to calculate RadCliQ. The original also extracted embeddings from Chexbert and then saved them into a file, while this reimplementation keeps them in memory.
Contains code for computing the individual metric scores that are used to compute RadCliQ. The individual metrics are:
- BLEU
- BERTscore
- CheXbert labeler vector similarity
- RadGraph entity and relation F1
Much of the code is liberally borrowed from rajpurkarlab/CXR-Report-Metric.
The code for computing the CheXbert metric score is adapted from stanfordmlgroup/CheXbert. Chexbert weights are not included in this repo. Download the CheXbert model checkpoint here and drag them into src/radcliq/CXR-Report-Metric (may be changed later)
Replaced the original RadGraph weights with the RadGraph-XL weights. Code to calculate RadGraph is from Stanford-AIMI/radgraph. This has the advantage of being more up to date and no longer requiring credentialed access, but scores may vary slightly from the original.
This repository is made publicly available under the MIT License.
If you are using this repo, please cite the original paper:
@article {Yu2022.08.30.22279318,
author = {Yu, Feiyang and Endo, Mark and Krishnan, Rayan and Pan, Ian and Tsai, Andy and Reis, Eduardo Pontes and Fonseca, Eduardo Kaiser Ururahy Nunes and Ho Lee, Henrique Min and Abad, Zahra Shakeri Hossein and Ng, Andrew Y. and Langlotz, Curtis P. and Venugopal, Vasantha Kumar and Rajpurkar, Pranav},
title = {Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation},
elocation-id = {2022.08.30.22279318},
year = {2022},
doi = {10.1101/2022.08.30.22279318},
publisher = {Cold Spring Harbor Laboratory Press},
URL = {https://www.medrxiv.org/content/early/2022/08/31/2022.08.30.22279318},
eprint = {https://www.medrxiv.org/content/early/2022/08/31/2022.08.30.22279318.full.pdf},
journal = {medRxiv}
}
Additionally, if you found this implementation useful, please mention:
@misc{SummitStudiosDev/RadCliQ, author = {Cody Chen}, title = {Remplementation of [Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation
]}, year = {2025}, url = {https://github.com/SummitStudiosDev/RadCliQ} }