[fdrocha]

Brings Jaeho's FreeCiv v3 pipeline from the freeciv-v3 branch (at daca5a15) into main, split by the "Paper material" rule from #10 instead of merged. A merge would have brought freeciv-v3's raw provider responses (about 100 MB of results/) into main's history.

Companion PR in forecastbench-sim-paper: forecastingresearch/forecastbench-sim-paper#3. It adds the question sets, reports, model panels, launchers and run records to production/freeciv, plus the run-2 analysis scripts.

Commits

1. worlds/freeciv: the changes that ran on the replay pods. Applies fleet/pod_bundle/freeciv_world_as_run.patch unchanged: observer-death and server-wait handling in fork_manager.py, savegame parsing, report fixes, and a zstandard dependency (15 files). worlds/freeciv on main hasn't changed since freeciv-v3 branched, so the patch applies cleanly.
2. worlds/freeciv/fbsim_v3: 62 files copied byte for byte, with sources and hashes in its PROVENANCE.json:
   • draw/: corpus consolidation, candidate pools, the structural draw, criteria, natural conditionals, report regeneration, QC
   • run/: elicitation harnesses (one question per prompt, batched, natural-conditional, effort check), scorers v1 and v2 with the self-test, row rebuild, reparse, selection fractions
   • fleet/: RunPod scripts, on-pod scripts and the pod bundle's scripts
3. Run from this repository:
   • Question sets, reports and model panels are read from FBSIM_FREECIV_V3_DATA, which points at forecastbench-sim-paper's production/freeciv. A replay run's corpus is read from FBSIM_V3_CORPUS, and keys come from the environment.
   • Jaeho's local paths in draw/ and fleet/ are now arguments or environment variables.
   • finalize.sh takes the pod ids as arguments and drops the mine_v3.py step (that script was never committed).
   • New fleet/make_bundle.sh builds the pod bundle from packages/fbsim-core and worlds/freeciv, instead of the vendored copies on freeciv-v3.
   • uv.lock gains only the two zstandard lines.
   • A short README. PROVENANCE.json lists every adaptation.

Not brought over

These stay on freeciv-v3 only:

• results/: already in the private data archive.
• paper/: the generators are already in forecastbench-sim-paper.
• The dated manuscript-editing scripts and the October 2 logprob comparison.
• The smoke, cost, cache and saturation tools.
• draw/natcond_new_families.py.

Checks

• score_selftest.py passes.
• score_v2.py on the archived run-2b batched rows and run-2 natural-conditional rows reproduces run2_paper's score_items.csv, score_summary.csv and SCORES.md byte for byte.
• elicit_v2.py --dry-run builds 44 prompts per model, which is run 2b's design.
• Tests: worlds/freeciv 15 passed, worlds/micropolis 227, worlds/pandemic 25, packages/fbsim-core 16. In packages/fbsim-benchmark, 59 pass and 2 fail (test_single_core_owner, test_noneditable_installed_packages); the same two fail on main.

For review

• Commit 1 changes the shared freeciv_world package that the older CivBench scripts also use.

🤖 Generated with Claude Code

[elsehow]

This is great! Thanks for the PR. My biggest comment is inline on fork_manager.py.

Some minor notes:

• Pod IDs are hardcoded in finish_all.sh, rebalance.sh, fleet_ssh.sh, switch_anchor.sh and launch_pending.sh. Could we pull them into an env var or state file? That will help when we productize this more!
• bringup.sh:7 only builds the bundle if state/bundle is missing, so code changes don't reach the pods.
• The "No temperature: provider-default sampling" comment in parallel_evaluator.py (514/644/754) doesn't match main's fbsim-core, which still defaults to 0.0. Let's either port the None default or drop the comment.

[elsehow]

This makes every fork draw a fresh seed unless the caller pins one. That's right for the v3 replays, but conditional_runner.py (L4-15 on main) skips the control fork and compares the intervention fork against the recorded game. With this change, every fork rolls fresh dice.

I think we should either keep the saved dice for conditional_runner's forks, or run a no-intervention fork on the same new seed and compare against that instead of the recording

[fdrocha]

Great catch! I fixed this in 3300403, by adding a reseed bool parameter that conditional_runner.py uses.

[elsehow]

1/r per (band, T, family) is exact only if selection is uniform within each stratum. draw_v1 also caps 6 per world per cell and one item per subject, so inclusion probability still varies inside a stratum. And strata with n_selected = 0 are dropped, so the "whole pool" estimate covers 21,583 of 23,241 pool items (92.9%).

This script already reads the pool pickles. Could we get π_i by re-running draw_v1 over many seeds and weight by 1/π_i? At minimum, the paper should state the coverage.

[fdrocha]

I agree with all of that but I think that is out of scope for this PR. I am going to be making a PR soon with the scripts that do the 1/r weighting and I will address this there.

[fdrocha]

This is great! Thanks for the PR. My biggest comment is inline on fork_manager.py.

Thanks for the comments Nick, replies inline below.
My idea for this PR was to just bring in Jaeho's code that was missing with minimal modifications - bigger changes would wait for future PRs. That said, your notes are all reasonable and I will address all of those that don't require big changes.

Some minor notes:

* Pod IDs are hardcoded in finish_all.sh, rebalance.sh, fleet_ssh.sh, switch_anchor.sh and launch_pending.sh. Could we pull them into an env var or state file? That will help when we productize this more!

It looks like the code pulls most pod ids from a file called pods.tsv that Jaeho did not commit. Scripts do have some explicit pod ids in the code that are handled differently. I agree this needs to be cleaned up (also, Jaeho told me he used is personal Runpod account, so all the pod ids will change, even if we stick with Runpod).
But I would rather do all of that cleanup together at a later stage when I am trying to get this part of the code up and running, especially given that we might be switching to a different provider.

* `bringup.sh:7` only builds the bundle if state/bundle is missing, so code changes don't reach the pods.

Good catch, fixed in a2db70b

* The "No temperature: provider-default sampling" comment in `parallel_evaluator.py` (514/644/754) doesn't match main's fbsim-core, which still defaults to 0.0. Let's either port the `None` default or drop the comment.

Comments dropped in 66a00ac

This made me notice a small inconsistency between how prompting was done for Freeciv/Micropolis and Starsim.
Starsim uses fbsim-core defaults which sets T=0 for models that support it. Freeciv/Micropolis never specify temperature, just use provider defaults for every model. For paper updates we should stick with what was done until now, but for the benchmark we should be more consistent. I like the fbsim-core approach (I only switched to never setting temperature because I saw Jaeho did it and I didn't realize you hadn't), but let me know if you prefer the other one.