OpenBookQA (Inspect AI eval)

OpenBookQA is a multiple-choice question answering benchmark modeled after open-book exams. Each question targets elementary-level science and is paired with one "open book" core science fact plus broad common knowledge needed to answer it. The dataset contains roughly 6,000 questions (500 in the test split), each with four answer choices and a single correct answer.

Mihaylov, Clark, Khot, Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. EMNLP 2018.

This repository packages OpenBookQA as a standalone Inspect AI evaluation so it can be registered in the Inspect Evals Register.

What it measures

The model is shown a question stem and four lettered choices and must select the correct one. Accuracy is reported across the 500-question test split.

• Dataset: allenai/openbookqa, pinned to revision 388097ea7776314e93a529163e0fea805b8a6454 for reproducibility.
• Solver: multiple_choice() from inspect_ai.
• Scorer: choice() (exact-match on the selected letter), reported as accuracy with stderr.

Variants

• openbookqa (default): closed-book setting. The model sees only the question and choices. This is the standard reported configuration.
• openbookqa -T use_fact=true: open-book setting. The gold supporting science fact (fact1) is prepended to each question, reproducing the dataset's "open book" framing. This uses the additional dataset config, which carries the fact field.

Install

# with uv
uv venv
uv pip install -e .

# or with pip
python -m venv .venv
. .venv/Scripts/activate   # Windows; use .venv/bin/activate on POSIX
pip install -e .

Run

# full test split (500 questions), closed book
inspect eval src/openbookqa_eval/openbookqa.py@openbookqa --model <provider/model>

# open-book setting (prepends the gold fact)
inspect eval src/openbookqa_eval/openbookqa.py@openbookqa -T use_fact=true --model <provider/model>

# quick smoke test against the mock model (no API key needed)
inspect eval src/openbookqa_eval/openbookqa.py@openbookqa --model mockllm/model --limit 2

Task parameters:

• shuffle (bool, default True) — shuffle dataset order.
• use_fact (bool, default False) — prepend the gold supporting fact.

Test

# fast, offline unit tests only
pytest -m "not huggingface"

# include the dataset-download + mock-model end-to-end tests
pytest

License

MIT. See LICENSE.