[fdrocha]

As I am preparing to do more runs of world generation and prompting I ran into inconsistencies about how the different world packages handle credentials and output data. I decided to re-organize it a bit before doing any of the runs and this PR does that.

Details from Claude:

Every package now writes its output to data/<package>/ and reads its keys from a single credentials/.env, both at the repository root and both normally symlinks to directories outside the checkout, so concurrent worktrees share one set of data and keys. New worktrees get the links automatically through a committed post-checkout hook. Also a FreeCiv v3 fleet bug fix found in the one-player pilot (400 replays on RunPod).

The commits are split by package, so each reviewer can read just theirs. The cross-package base is the second commit.

• 450dd944 FreeCiv v3 fleet: start the worker's archive sync through start_sync.sh. The inline pgrep guard ran inside the ssh command, whose own command line contains the sync_loop path, so it always matched and the sync loop never started (found in the pilot, where it had to be started by hand).
• 92e8d7f2 One data directory and one credentials directory, outside the checkout. fbsim_core.local (repo root, data_dir(package), load_env()); scripts/setup_worktree.sh and .githooks/post-checkout; root .gitignore, README section "Local data and credentials", CLAUDE.md. FBSIM_TEST_DATA_ROOT and FBSIM_TEST_ENV_FILE are test-only overrides. FBSIM_DATA_ROOT is deliberately not used, because in forecastbench-sim-paper it names the private data archive.
• a2dd75d4 Micropolis: data root and keys from fbsim_core.local. The data root is <repo>/data whatever the working directory (it was ./data). credentials/.env is read only when a runner needs the engine or keys, never at import or for --help. The tests keep the checkout's credentials out, subprocesses included.
• d4d44c8e Pandemic: cache and results under data/pandemic. scale_results.json, conditional_brier_gap.png and null_conditional_results.json leave git.
• 74cda6d9 FreeCiv: output under data/freeciv, keys from credentials/.env. The engine's logging root (logs, recordings) moves to data/freeciv/logs through logging_config.logging_dir; game scores move too. About 140 paths in scripts/ that were relative to the working directory become absolute under data/freeciv (mechanical). The scripts still fetch provider keys from GCP Secret Manager; only GOOGLE_CLOUD_PROJECT now comes from credentials/.env. The agentic shell scripts' dead src.civrealm import is fixed.
• cf9aed5c FreeCiv v3: fleet state and keys outside the checkout. fleet/paths.sh puts the run state in data/freeciv/fbsim_v3/fleet_state and the ssh keys in credentials/freeciv/. Paths are quoted and pods are reached through podsh, so checkout paths with spaces work. Pods record games under data/freeciv/logs. --env-file is removed from the run/ elicitation scripts. PROVENANCE.json records the adaptations.

Heads-up for coauthors

After this merges, once per clone:

1. Create credentials/.env at the repository root (gitignored), with every variable from your old worlds/*/.env files in one file: ANTHROPIC_API_KEY, OPENROUTER_API_KEY, GOOGLE_CLOUD_PROJECT (FreeCiv still fetches its provider keys from GCP Secret Manager with it), RUNPOD_API_KEY (fleet only), MICROPOLIS_CORE_PATH, and DEEPINFRA_API_KEY (Pandemic scale_eval.py --eval). Variables exported in the shell still win.

2. Move existing output:

   • worlds/freeciv/logs/ → data/freeciv/logs/;
   • FreeCiv's games/, questions/, conditional/, results/ and evaluations/ (from the root data/ or worlds/freeciv/data/) → data/freeciv/;
   • worlds/pandemic/.response_cache.json → data/pandemic/response_cache.json.

   data/micropolis* stays where it is.

3. Optional: git config core.hooksPath .githooks, so new worktrees get the links automatically. It replaces .git/hooks; if you have hooks there, run scripts/setup_worktree.sh by hand instead.

data/ and credentials/ can stay real directories in your main checkout; the symlinks are only needed to keep them outside it.

Testing

• Test suites: Micropolis 227, fbsim-core 16, Pandemic and FreeCiv 40, in the workspace venv. The same suites (227, and 77 for core and benchmark) also pass in the README's wheel-install venv, which exercises fbsim_core.local finding the checkout from the working directory. The two benchmark tests that assert a non-editable install fail in the workspace venv, as on main.
• Static checks: py_compile on all 60 changed Python files; bash -n on all 28 changed shell scripts; ruff F821/F401 on the changed Python shows no new findings compared with main.
• Fleet, local, read-only:
  • rp.sh list authenticates with the key from credentials/.env; rp.sh ledger, status.sh, launch_pending.sh and finish_all.sh --dry-run work against the moved state.
  • All of these were also run through a checkout path containing a space, with rsync -e and podsh checked to receive a spaced key path as one argument.
  • A built pod bundle contains no keys, .env files or symlinks.
• Worktree hook: a new worktree created with git worktree add got both links from the post-checkout hook.
• Paper build: forecastbench-sim-paper's build_paper.sh route (editable fbsim-core/fbsim-benchmark) was run against main and against this branch, into scratch copies of the manuscript. The outputs are byte-identical. The paper's default route uses the pinned 9e016b76, which this branch cannot affect.
• Not tested:
  • Anything on RunPod after these changes. The pod-side recordings path (/workspace/civbench/data/freeciv/logs) and the fleet scripts' changes will first run with the next anchor pod. The earlier fleet commits ran the 400-replay pilot.
  • The legacy FreeCiv scripts: their path rewrite was only compiled and checked statically, not run.
  • Live Micropolis, Pandemic or FreeCiv model calls.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Tze4Twn5EYv7cQxwxUcBpP