MolDesignBench

A scenario-grounded benchmark for LLM-based agents in molecular design (COLM 2026). It has 2,000 generation and optimization instances with explicit and implied property / functional-group constraints, including infeasible ones, and a set of 17 chemistry tools. Models are evaluated without tools and with tools.

Installation

git clone https://github.com/KU-AGI/MolDesignBench.git
cd MolDesignBench

conda create -n moldesignbench python=3.11 -y
conda activate moldesignbench
conda install -c conda-forge openjdk=17 -y
pip install torch==2.7.1 --index-url https://download.pytorch.org/whl/cu126   # match your CUDA driver
pip install "Cython<3"
pip install -r requirements.txt
pip install -e .

Install PyTorch before requirements.txt. See docs/installation.md for details and Docker.

Data

The tool data is included in data/. Download the benchmark questions:

wget -P data/MolDesignBench https://huggingface.co/datasets/LG-AI-Research/MolDesignBench/resolve/main/data/moldesignbench.jsonl

Running the benchmark

Each model is run in two settings (Table 3 of the paper):

Setting	bench.py options
w/o tool	--prompt_type wotool
w/ tool	--prompt_type wtool --use_tools (17 tools, up to 20 tool-call rounds)

1. API keys. Fill in the keys you need in api_keys.py (Claude / OpenAI; NGC for MolMIM; Hugging Face for vLLM).

2. Services (w/ tool only).

docker build -t moldesignbench:latest .   # once (the MolMIM proxy runs in this image)
make start-molmim                         # MolMIM NIM, used by the two property-range design tools
make start-tool-servers                   # tool servers + Jupyter API
bash scripts/setup/check_tool_servers.sh  # all checks should pass

3. Run. The scripts run both settings for one model; uncomment its block first.

bash scripts/bench/run_bench_closed.sh   # Claude / GPT (API)
bash scripts/bench/run_bench_open.sh     # GPT-OSS, Qwen3, Intern-S1, Ether0 (vLLM; e.g. make start-vllm-oss)

Or run one setting directly:

export OPENAI_BASE_URLS="http://localhost:8080/v1" OPENAI_API_KEY="EMPTY"
python bench.py --model openai/gpt-oss-120b \
  --input data/MolDesignBench/moldesignbench.jsonl \
  --output results/gpt-oss-120b-w-tool/moldesignbench.json \
  --prompt_type wtool --use_tools --max_iterations 20 --reasoning_effort high

See docs/evaluation.md for all options and vLLM setup.

Evaluation

Scoring needs no LLM or MolMIM:

python evaluate/evaluate_benchmark.py --base_dir results --folder gpt-oss-120b-w-tool
bash scripts/evaluate/make_leaderboard.sh          # results/leaderboard_{generation,optimization}.txt

The leaderboard reports Success, CSR, MCD, InfAcc, F1, MCC, Validity and Tool Calls. The paper's results are in docs/leaderboard.md.

Licence

• Code: MIT, © 2026 Korea University and LG AI Research.
• Benchmark data (Hugging Face): CC BY-NC 4.0.
• Tool data in data/: MOSES molecules (MIT) with ADMET-AI predictions.
• third-party/: ADMET-AI and Chemprop, each under its own MIT licence.

Citation

@inproceedings{jeong2026moldesignbench,
  title     = {MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design},
  author    = {Jeong, Yongjun and Ko, Hanbum and Kim, Ye Rin and Lee, Chanhui and Hormazabal, Rodrigo and Lee, Jaewan and Han, Sehui and Lim, Sungbin and Kim, Sungwoong},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026}
}