ForecastBench-Sim

A multi-simulation forecasting benchmark that can host an arbitrary number of simulation "worlds." A world runs a simulation, and models read a world-state report and forecast future outcomes. The domain-agnostic machinery (question schema, resolver, generator, scoring, eval harness) lives in fbsim-core; each world is a plugin.

Two worlds ship today:

• FreeCiv (worlds/freeciv/) — strategy-game world on the CivRealm engine.
• Starsim (worlds/pandemic/) — SIR epidemic world with conditional vaccine forecasting.

Adding a third is the Adding a world flow below.

packages/fbsim-core/      # schema, resolver, registry, generator machinery,
                          # metrics, eval (models/sampling/difficulty)
worlds/freeciv/           # FreeCiv strategy-game world (CivRealm engine + benchmark)
worlds/pandemic/          # Starsim SIR epidemic world (conditional vaccine forecasting)

It's a uv workspace. Install with:

uv sync --all-packages

Local data and credentials

Every package writes its output to data/<package>/ (data/freeciv/, data/micropolis/, data/pandemic/; Micropolis runs with a config data_dir use sibling data/micropolis_* folders) and reads its keys from credentials/.env, one file with every variable any package needs (sourceable: set -a; . credentials/.env; set +a). Variables exported in the shell take precedence. Code finds both through fbsim_core.local (FBSIM_TEST_DATA_ROOT and FBSIM_TEST_ENV_FILE redirect them, for tests only).

Both are gitignored symlinks at the repository root, to directories outside the checkout so that every worktree shares them. In the main checkout they may also be real directories.

mkdir -p ../fbsim-local/data ../fbsim-local/credentials && chmod 700 ../fbsim-local/credentials   # once per machine
scripts/setup_worktree.sh             # links data/ and credentials/ (main checkout: to ../fbsim-local/)
git config core.hooksPath .githooks   # once per clone: every new worktree is then linked automatically

core.hooksPath replaces .git/hooks: git stops running hooks there. If you have any, skip the last command and run scripts/setup_worktree.sh in each new worktree instead.

Paper material

This repository holds reusable world code. Material specific to the ForecastBench-Sim paper lives in forecastbench-sim-paper: its analysis in src/, and the production configs, seeds, prompts, model panels and producer scripts in production/. Raw provider responses, simulator outputs, datasets, human-subject records and credentials stay in the private data archive, out of Git. Split contributor branches the same way when merging them.

Adding a world

A world provides these to fbsim-core (see worlds/pandemic/ for a compact example):

1. Serializer → game_data in the canonical turn-major layout time_series[metric][turn][entity_id] = value (see fbsim_core.questions.timeseries). Core never infers the layout — the world emits it.
2. Template registry — build a fbsim_core.questions.registry.TemplateRegistry from the world's QuestionTemplates and pass it to QuestionResolver(registry). (worlds/pandemic/pandemic_world/templates.py)
3. Report renderer — a function producing the model-facing world-state text for a snapshot. (pandemic_world/report.py)
4. Eval driver / adapter — turn questions + report into prompts, query models (fbsim_core.evaluation.models), score with fbsim_core.metrics. (pandemic_world/scale_eval.py)
5. (Optional) conditional executor — how an intervention is applied. FreeCiv forks a savegame; pandemic re-runs the sim with a vaccine. Core keeps the generic control-vs-intervention structure; the executor is world-specific.

Verify

# core (domain-agnostic) tests
cd packages/fbsim-core && uv run --project .. python -m pytest tests

# FreeCiv world: generate + resolve questions from a recorded game
uv run python worlds/freeciv/scripts/generate_questions.py \
    data/freeciv/games/seed1619_data.json --snapshot-turn 60 --summary

# Pandemic world: scaled conditional/unconditional benchmark + chart (cached)
uv run python -m pandemic_world.scale_eval --eval

Both worlds resolve through the same core resolver and registry — that's the point.

Offline cached-forecast interfaces

The additive fbsim-benchmark package provides installed-package examples and synthetic fixture tests for FreeCiv, Micropolis and Starsim without running engines or calling models. It depends on the existing core; world and human-subject workflows remain in place. See its workflow map for retained, migrated and legacy routes.

Micropolis world

Micropolis setup and workflows cover the external engine, question generation, cached/model gathering and raw scoring. Synthetic offline tests do not validate a live engine or reproduce paper results.