KG-MCQA

Source codes for KR 2025 workshop paper
"Applying Relation Extraction and Graph Matching to Answering Multiple Choice Questions"
which will be presented at NeLaMKRR 2025, 12th November.

Original paper can be found here (pp. 57-62).

This repository contains:

1. Python scripts for the answering experiment
2. Original MCQ datasets
3. Streamlit app for visualizing the results

Setup

Create Environment

Run the following command to create a conda environment with the required libraries. It automatically installs the original package kgraph.

conda env create -f environment.yml
conda activate kg-mcqa

Activate Viewer

Activate Streamlit app for viewing datasets and experimental results. While running the command, you can open the app in a browser via localhost:8501.

streamlit run viewer/main.py

Experiments

Important

It is highly recommended to use GPUs to execute experiments.
The authors used an NVIDIA RTX 5000 Ada GPU with 32 GB RAM in the article.

Run all experiments

The following script runs all the MCQAs introduced in the study. Namely, MCQA on two crafted datasets (KR-200m and KR-200s) by using relation extraction (RE) methods, such as REBEL, mREBEL_400, mREBEL_32, and UniRel.

./scripts/mcqa_main.sh

Single experiment

To run a single MCQA using the specified RE method, use the next command.

python mcqa.py [dataset_path] [re_method]

dataset_path should be specified from any json files under ./dataset directory.
Choices of re_method are as follows.

re_method	Model name	Dataset	Relation types
rebel	REBEL	REBEL	220
mrebel	mREBEL_400	RED^{FM}	400
mrebel_32	mREBEL_32	SRED^{FM}	32
unirel	UniRel	NYT	24

Baseline result by LLM

For the baseline experiment using Google's FLAN-T5-xlarge (3B) model, run the following script.

python llm.py

Files & Directories

Files

• mcqa.py
• fever.py
• environment.yml

Directories

(ignored) shows the files not controlled by git.

• kgraph: Original package for handling the proposed method.
• scripts
• dataset: Original MCQ dataset.
• exp-mcqa: Result files of MCQA.
• KG_cache: Cache files of KG constructed from Wikipedia articles. (ignored)
• wikipedia: Downloaded Wikipedia articles. (ignored)

Citation

Please kindly cite the following paper if you refer to this work.

@inproceedings{NeLaMKRR-2025-07,
    title     = {{Applying Relation Extraction and Graph Matching to Answering Multiple Choice Questions}},
    author    = {Shimoda, Naoki and Yamamoto, Akihiro},
    booktitle = {{Proceedings of the Second International Workshop on Next-Generation Language Models for Knowledge Representation and Reasoning (NeLaMKRR 2025)}},
    pages     = {57--62},
    year      = {2025},
    month     = {11},
    url       = {https://jurisinformaticscenter.github.io/NeLaMKRR2025/proceedings.pdf},
  }