Repository navigation
Conversation
A run is one configuration (players, barbarians, fork turn, horizons, replays, anchor seeds, natcond, pod machines and network volume) in data/freeciv/fbsim_v3/runs/<run>/. fleet/1_new_run.sh creates it: run.env with the settings and the commit it was created at, shell.sh (source it to export FBSIM_RUN and set R, S, K, V3 for manual commands) and an empty fleet_state/. It refuses seed ranges that overlap another run's and a dirty checkout unless --force, since pods get the working tree; --replays N sets the rng range 5001-(5000+N). fleet/paths.sh requires FBSIM_RUN, sources run.env and keeps the ledger, queues and anchors in <run>/fleet_state. Pods are named fbsim3-<run>-..., archive to /vol/fbsim_v3/<run>/<pod>/ and receive run.env, from which anchor_one.sh, anchors_run.sh, worker.sh, rerun_list.sh and fork_one.sh take the player count, barbarians, fork turn and end turn. pull.sh, pull2.sh and finalize.sh read only the run's archive folder and horizon saves. runcfg.py is the Python reader of run.env; make_queue.py takes its --rng and --out defaults from it. rebalance.sh, anchors_run.sh and finish_pod.sh become executable like the other fleet scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2 Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
Turn 60, the 61..T windows, the 61-90 reveal window and the 90..210 horizons become the run's fork turn, fork turn + 1, first horizon and horizons, in the questions, criteria, reports and natural-conditional preamble. --run NAME or FBSIM_RUN defaults every folder to the run's; FBSIM_BARBARIANS=DISABLED implies --no-barbarians, FBSIM_NATCOND=0 --no-natcond, and consolidate_v4 --n-main takes the player count. A natural-conditional reveal must hold in a tenth of the replays (100 of 1,000, as before), like bank_v1's reveals. Without a run the paper's values apply: consolidation, sweep, bank, draws, criteria and reports are byte-identical to before on the 1-player pilot and on two of the paper's 5-player anchors. A turn-shifted copy of the pilot run as a T120 run gives the same sets with every turn moved by 60. draw_v1.py skips a draw cell with no candidates (a tails horizon could crash it), and stops with a clear message when the bank, tails or mirrors set has no candidates at all (too few replays per anchor). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2 Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
… band The bank's cells were filled one after another, low bands first, under a cap of one question per subject, so the low bands used up subjects the higher bands also had. With a run, the cells now take one question each in turn. With one player, a world has a single subject per family and horizon, so the subject cap applies per band. On p1_t60 the bank goes from 276 questions (149/91/15/12/9 per band) to 676 (149/140/130/123/134). Without a run the draw is byte-identical to before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
The background sentence ("observed at turn F", civilization count), the
no-news turns and the report file name come from the run in elicit_v1.py,
elicit_v2.py and elicit_natcond_v1.py; FBSIM_FREECIV_N_CIVS still overrides
the count. With a run, sets, reports and model panels default to the run
folder, --out to <run>/evals/<timestamp> and score_v2 --out to
<run>/scores/<name>. score_v2 also scores a draw without natural
conditionals (natcond_600.json and natcond_extra_turn1.json are optional,
as in criteria_v1.py), which a one-player run does not have. Without a run
every prompt and the scores on the paper's run-1 results are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
With a run, an unparsed binary answer is no longer scored as p = 0.5: it is left out of the scores, and its row stays in score_items.csv unscored so the parse rate still counts it. SCORES.md says so. Without a run the paper's rule (0.5) stays the default, and the outputs are byte-identical to before. `--impute-binary P` still forces a value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
score_v2.py normalizes CRPS by one constant per continuous family, which no script here produced. norm_constants_v1.py applies the paper's rule (median truth p05-p95 range, one significant figure) and reproduces the paper's file byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
Nothing reads sweep_v4.json: the bank is built from consol_v4/, and no script or paper code uses the sweep. finalize.sh no longer runs sweep_v4.py; draw/sweep_v4.py stays for manual use. finalize.sh now exits non-zero when the pull, the corpus QC or the consolidation fails, and removes consol_v4_SUMMARY.json with consol_v4/, so a failed rerun no longer leaves the old summary looking current. pull2.sh fails when the listing on the archive fails or is empty, or when any rsync chunk fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
Parallel launches raced on rp.sh's shared address-cache temp file, so launches failed with "no archive addr", and launch_pending.sh never retried a worker that had a launch log. Each cache write now uses its own temp file. Each launch log ends with launch_rc=; a launch that ends without lanes is kept as failed_launch_<id>_<time>.log, reported with an ERROR line and retried up to --max-retries times (default 5), after which launch_pending.sh exits 2. The line of a relaunch says which retry it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
bringup.sh ran pod_setup.sh in one long ssh session and took its exit code from that session's output, so a dropped connection failed a healthy pod (worker w01 of the pilot p1_pilot2). pod_setup.sh now runs detached and writes its exit code to logs/pod_setup.rc, which bringup.sh polls over fresh ssh sessions for up to 40 minutes; it still prints pod_setup_rc=<code>, or pod_setup_rc=timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
worker.sh N tops lanes up to N, counting a lane alive when kill -0 succeeds on its pid. A lane that exits after 15 idle minutes can stay a zombie, which kill -0 also accepts, so tasks moved to that pod later by rebalance.sh were never run (seven workers of run p1_t60 sat idle with about 220 tasks). A lane now counts only if its pid still runs "worker.sh __lane N". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
setup_worktree.sh (run by the post-checkout hook in every new worktree) links worlds/freeciv/fbsim_v3/fleet/runs to data/freeciv/fbsim_v3/runs, so `source runs/<run>/shell.sh` works from fleet/. The link is relative, so it resolves through each checkout's data/ link; it is gitignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
2_anchors.sh .. 9_score.sh run one step of a run each: play anchors (in batches of the pod's civservers), keep anchors, start the fleet (archive pod, workers, queue, parallel bringup, lanes), wait and finish (with an offer to rerun twice-failed tasks before any worker is deleted), pull and consolidate, draw sets and reports, elicit (dry run, then confirmation) and score. Each takes the run name first, sources the run's shell.sh, prints timestamped progress, waits through the long phases and can be rerun after an interruption, reusing pods, bringups and finished work. Pods are deleted only after a y/N prompt. steps_lib.sh holds the shared helpers; on_pod/anchor_status.sh reports each anchor's state, checking the process before the log so an anchor that just finished is not counted as failed. 4_start_fleet.sh passes --max-retries to launch_pending.sh and stops when a worker's launch gives up; 6_pull_corpus.sh stops when finalize.sh fails. Steps 2-7 were run against simulated pods, and steps 1-6 since on RunPod (pilot p1_pilot2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2 Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
RUNBOOK.md describes one run end to end: the run folder, keys, the nine step scripts with what each reads and writes (including the draw rules and scoring that differ from the paper's for a run), the existing runs, manual commands, and what to check before the first T120 run. README.md introduces runs, lists the step scripts and points to the runbook; AGENTS.md describes the per-run fleet state; PROVENANCE.json records the adaptations; CLAUDE.md names the run folders under data/freeciv. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2 Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
worker.sh gives each replay 3,600 s (was 1,800 s) before it is killed and retried. A replay from a turn-120 fork plays later, slower turns and is expected to take 12-21 minutes under a full lane load, too close to the old limit; a hung replay now costs one lane an hour instead of half. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
show_intelligence_report_hearsay (an opponent with no embassy) wrote the research target under `research`, a key the player observation space does not have, so gymnasium's env checker rejected the first reset. Turn-60 games never reached it; every turn-120 fork of a five-player game failed on it. Agents do not read this observation and results come from the saves, so game outcomes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
Step 4 gives a new worker 6 minutes to answer on ssh (bringup.sh's wait is now FBSIM_SSH_WAIT_S, default 15 minutes as before). A worker that does not answer is deleted and created again, up to three pods, and its queue slice, not yet shipped, moves to the new pod's ID, so the rerun no longer stops on a queue dealt to a deleted pod. On p5_t120_smoke one EU-SE-1 host took two workers in a row and 25 minutes each before giving up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
- 5_finish.sh runs finish_all.sh every interval, so each worker is finished and deleted as soon as its queue is
empty instead of after the whole fleet; tasks that failed every attempt are listed from failed_tasks.txt.
- Lanes give each task a third attempt (90-minute timeout) in place of the end-of-run rerun prompt; finish_all.sh
leaves workers running rerun_list.sh alone and records failed tasks before deleting a worker; rebalance.sh sends
no tasks to finishing workers.
- stall_watch.sh stops a replay whose game has gone 10 minutes without a savegame: the game client hangs after the
server rejects one of its commands ("Target not at target tile"), which on p5_t120 left replays waiting for the
one-hour timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
4_start_fleet.sh takes --anchors (deal only these kept anchors) and --name wX (workers fbsim3-<run>-wX01.., so their archive folders stay apart from the first fleet's). RUNBOOK: how to replace an anchor after the fleet is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Vag3wkZkNcQrvVuikmgyw
extract_ground_truth.py now writes yes_idx alongside counts: per question, the 0-based continuations resolving Yes, aligned with the values lists. Enables joint probabilities of binary events. Files lacking yes_idx are no longer treated as covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
draw_natcond.py pairs a question A with a revealed fact B at the horizon just before it (--max-gap widens this), in the same city and snapshot, and writes P(A) and P(A|B) from the yes_idx continuations. Truth requirements and caps follow FreeCiv's draw (0.05 < P(A) <= 0.95, B Yes in >= 1/10 of replays, 0.02 < P(A|B) < 0.98, <= 2 reveals per A, <= 8 cells per reveal). Cells are drawn by block (irrelevant 50, mechanical 200, inferential 150), template-balanced, never on effect size. FreeCiv's no-news and single-prompt control flags are not drawn: its no-news results cover the first and the paper never ran the second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tze4TySPdDQmYaFo9Za8fc
build_natcond_prompts.py turns a natcond draw into prompt jobs after FreeCiv's elicit_natcond_v1.py: turn 1 asks question A alone, turn 2 continues that conversation with fact B. A question in two cells is asked once at turn 1. The world report is the binary eval's cached one for the city and snapshot; the preamble is the binary one reworded for a single question. Turn 2 does not use FreeCiv's "One fact about turns X through Y has been revealed", which a smoke-test model read as a full account of those turns. It states the fact as the one extra thing learned, says nothing else is known, and restates question A with explicit turns and its full window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tze4TySPdDQmYaFo9Za8fc
elicit_natcond.py asks the natcond prompts through the binary eval's OpenRouter path (provider pin and effort from the model registry): turn 1 per question, then turn 2 per cell on top of the model's own turn-1 reply. One results.jsonl row per call with reply, parsed probability, usage and a hash of the messages sent; reruns resume, an unparsed reply is re-asked once, and a row only counts for the exact messages it was asked with, so a changed prompt is asked again and the closing summary counts answers to the current prompts only. Calls retry 8 times by default (backoff up to about 2 minutes), since providers' shared-pool 429s outlast the default 3. The default panel is configs/models-22.json5: the paper's 24 models without qwen/qwen3-235b-a22b and openai/o4-mini, which are no longer on OpenRouter; the same 22 answered FreeCiv run p5_t120. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tze4TySPdDQmYaFo9Za8fc
fdrocha
marked this pull request as ready for review
November 5, 2026 17:54
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A natural-conditional (natcond) study for Micropolis, after FreeCiv's: draw pairs of a question A and an earlier revealed fact B from the ground-truth replays, build the two-turn prompts, and ask models. The ground-truth files gain per-continuation Yes indices so that P(A|B) can be computed at all.
extract_ground_truth.pywritesyes_idxnext tocounts: per question, the 0-based continuations resolving Yes, aligned with thevalueslists. A file without it is no longer treated as covered, so a normal rerun regenerates old files.draw_natcond.pypairs A (at S + horizon_A) with B at the horizon just before it, same city and snapshot (S = 960, 1440). FreeCiv's truth requirements and caps: 0.05 < P(A) ≤ 0.95, B Yes in ≥ 100 of 1,000 replays, 0.02 < P(A|B) < 0.98, ≤ 2 reveals per A, ≤ 8 cells per revealed event. Three structural blocks (irrelevant 50, mechanical 200, inferential 150), template-balanced, never selected on effect size. No no-news or single-prompt controls: FreeCiv's no-news results cover the first, and the paper never ran the second.build_natcond_prompts.py: turn 1 asks A alone; turn 2 continues the conversation with B. The world report is the binary eval's cached one; the preamble is the binary one reworded for a single question (datafiles/preamble-natcond.txt). Turn 2 departs from FreeCiv's "One fact about turns X through Y has been revealed", which a smoke-test model read as a full account of those turns: it states B as the one extra thing learned, says nothing else is known, and restates A with explicit turns and its full window.elicit_natcond.pysends the prompts through the binary eval's OpenRouter path (provider pin and effort from the paper's model registry,--model-specs). Oneresults.jsonlrow per call, with a hash of the messages sent; reruns resume and re-ask changed prompts; unparsed replies are re-asked once; 8 retries per call. Default panelconfigs/models-22.json5: the paper's 24 without Qwen3 235B A22B and o4 Mini (gone from OpenRouter), the same 22 as FreeCiv run p5_t120.Testing
pytest worlds/micropolis/tests: 257 passed (17 new tests for the draw, prompts and elicitation, model calls stubbed; ground-truth tests extended). Each of the four commits passes on its own (240 / 240 / 246 / 257). Ruff is clean on the new files.fbsim_commitand the newyes_idxis byte-identical to the previous files; eachyes_idxlist matches its count, is sorted, unique and in range.draw_natcond.pyrun on those files with the defaults: 400 cells; every cell's P(A) and P(A|B) recomputed independently fromyes_idx, all requirements and caps checked, same seed reproduces the file byte for byte.build_natcond_prompts.py: 325 turn-1 and 400 turn-2 prompts; every world report is byte-identical to the binary eval's cached prompt.extract_ground_truth.pywas run only on the local configs above; no CI change (CI does not install the engine).🤖 Generated with Claude Code
https://claude.ai/code/session_01Tze4TySPdDQmYaFo9Za8fc