Skip to content

FreeCiv v3: run folders and numbered step scripts for the fleet - #28

Open
fdrocha wants to merge 18 commits into
forecastingresearch:mainfrom
fdrocha:freeciv-runs
Open

fdrocha wants to merge 18 commits into
forecastingresearch:mainfrom
fdrocha:freeciv-runs

Conversation

@fdrocha

@fdrocha fdrocha commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

FreeCiv v3 runs: each run is one configuration (players, barbarians, fork turn, horizons, replays, anchor seeds,
natcond, pod machines, network volume) in data/freeciv/fbsim_v3/runs/<run>/, driven end to end by nine numbered
step scripts. The draw, elicitation and scoring scripts take the fork turn, horizons and player count from the run;
without a run they keep the paper's settings, rules and outputs.

  • Run folders and run.env for the fleet: fleet/1_new_run.sh creates a run (run.env, shell.sh, empty
    fleet_state/), refusing overlapping seeds and a dirty checkout; paths.sh, the pod scripts and runcfg.py read
    run.env; pods are named fbsim3-<run>-… and archive to /vol/fbsim_v3/<run>/.
  • Draw settings from the run: questions, criteria, reports and the natural-conditional preamble use the run's
    fork turn and horizons. draw_v1.py skips empty draw cells and stops with a clear message when the bank, tails
    or mirrors set has no candidates (too few replays per anchor).
  • Bank cells take turns; one-player subjects recur per band: with a run, the bank's (q band × horizon) cells take
    one question each in turn instead of filling one after another, so low bands no longer use up the subjects the
    higher bands also have. With one player, a world has one subject per family and horizon, so the one-per-subject cap
    applies per band. On p1_t60 the bank goes from 276 questions (149/91/15/12/9 per band) to 676 (149/140/130/123/134).
  • Prompts and folders from the run: the background sentence, no-news turns and report names come from the run;
    sets, reports, panels, --out and scores default to the run folder. score_v2.py scores a draw without natural
    conditionals, as a one-player run has.
  • Unparsed answers left out of the scores for runs: an unparsed binary answer is no longer scored as 0.5; its row
    stays unscored so the parse rate still counts it. Without a run the paper's 0.5 stays the default.
  • norm_constants_v1.py: writes continuous_norm_constants.json by the paper's rule (reproduces the paper's file
    byte for byte).
  • finalize.sh stops at consolidation, and on errors: the unused sweep is no longer run; a failed pull, QC or
    consolidation now fails the step, and a stale consol_v4_SUMMARY.json is removed first.
  • Retry failed lane launches: fixes a race on rp.sh's address cache that failed parallel launches; a launch
    that ends without lanes is reported with an ERROR line and retried up to --max-retries times (default 5).
  • pod_setup survives a dropped ssh session: pod_setup.sh runs detached and bringup.sh polls its exit code,
    so a dropped connection no longer fails a healthy pod.
  • worker.sh restarts lanes left as zombies: a lane counts as alive only if its pid still runs that lane; kill -0
    also succeeded on exited lanes left as zombies, so a top-up after a rebalance started no new lanes and workers sat idle.
  • Runs shortcut in fleet/: fleet/runs links to the runs folder (made by setup_worktree.sh).
  • Numbered step scripts: 2_anchors.sh … 9_score.sh, one step each, rerunnable after an interruption; pods
    are deleted only after a y/N prompt.
  • Docs: RUNBOOK.md (one run end to end, the run-only draw and scoring rules, existing runs), README, AGENTS.md,
    PROVENANCE.json, CLAUDE.md.

Testing

  • Full run on RunPod: p1_t60, one player, fork at turn 60, anchors 9301–9308 × 1,000 replays, 16 workers
    (October 29–30). Steps 1–9 all ran: 8,000 replays with no failures or truncations, corpus QC and consolidation
    clean, about $77 of compute. Step 8 elicited 22 models for $16.78 (Qwen3 235B A22B and o4 Mini are no longer on
    OpenRouter); rerunning it with the same label resent only the calls whose stream had dropped. This run found the
    zombie-lane bug, the scorer's missing-natcond failure and the skewed one-player bank fixed here.
  • Smoke test: steps 1–6 for p1_pilot2 (3 anchors × 10 replays, 4 workers, October 28), which found the
    address-cache race and the dropped-ssh bringup failure. On it, step 7 stops with the new tails message.
  • Without a run, consolidation, bank, draws, criteria, reports and scores are byte-identical to before: on the
    1-player pilot, on two of the paper's 5-player anchors, and for draw_v1.py and score_v2.py on p1_t60's data
    after the draw and scoring changes. A T120 copy of the pilot gives the same sets with every turn moved by 60.
    norm_constants_v1.py reproduces the paper's file.
  • Locally, with stand-ins for ssh/rsync/setsid: bringup.sh's detached pod_setup (exit 0, exit 3, timeout),
    pull2.sh/finalize.sh failure paths, and launch_pending.sh retries up to the limit.
  • Not tested: any T120 run on RunPod; any run with two or more players (natural conditionals with a run); the
    zombie-lane fix as shipped by a normal bringup (on p1_t60 it was copied to the running pods by hand, and the lanes
    it restarted finished the run).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ

fdrocha and others added 13 commits October 30, 2026 14:36
A run is one configuration (players, barbarians, fork turn, horizons,
replays, anchor seeds, natcond, pod machines and network volume) in
data/freeciv/fbsim_v3/runs/<run>/. fleet/1_new_run.sh creates it: run.env
with the settings and the commit it was created at, shell.sh (source it to
export FBSIM_RUN and set R, S, K, V3 for manual commands) and an empty
fleet_state/. It refuses seed ranges that overlap another run's and a
dirty checkout unless --force, since pods get the working tree; --replays N
sets the rng range 5001-(5000+N).

fleet/paths.sh requires FBSIM_RUN, sources run.env and keeps the ledger,
queues and anchors in <run>/fleet_state. Pods are named fbsim3-<run>-...,
archive to /vol/fbsim_v3/<run>/<pod>/ and receive run.env, from which
anchor_one.sh, anchors_run.sh, worker.sh, rerun_list.sh and fork_one.sh
take the player count, barbarians, fork turn and end turn. pull.sh,
pull2.sh and finalize.sh read only the run's archive folder and horizon
saves. runcfg.py is the Python reader of run.env; make_queue.py takes its
--rng and --out defaults from it. rebalance.sh, anchors_run.sh and
finish_pod.sh become executable like the other fleet scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
Turn 60, the 61..T windows, the 61-90 reveal window and the 90..210
horizons become the run's fork turn, fork turn + 1, first horizon and
horizons, in the questions, criteria, reports and natural-conditional
preamble. --run NAME or FBSIM_RUN defaults every folder to the run's;
FBSIM_BARBARIANS=DISABLED implies --no-barbarians, FBSIM_NATCOND=0
--no-natcond, and consolidate_v4 --n-main takes the player count. A
natural-conditional reveal must hold in a tenth of the replays (100 of
1,000, as before), like bank_v1's reveals. Without a run the paper's values
apply: consolidation, sweep, bank, draws, criteria and reports are
byte-identical to before on the 1-player pilot and on two of the paper's
5-player anchors. A turn-shifted copy of the pilot run as a T120 run gives
the same sets with every turn moved by 60. draw_v1.py skips a draw cell
with no candidates (a tails horizon could crash it), and stops with a
clear message when the bank, tails or mirrors set has no candidates at
all (too few replays per anchor).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
… band

The bank's cells were filled one after another, low bands first, under a cap of one question per subject, so the low
bands used up subjects the higher bands also had. With a run, the cells now take one question each in turn. With one
player, a world has a single subject per family and horizon, so the subject cap applies per band. On p1_t60 the bank
goes from 276 questions (149/91/15/12/9 per band) to 676 (149/140/130/123/134). Without a run the draw is
byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
The background sentence ("observed at turn F", civilization count), the
no-news turns and the report file name come from the run in elicit_v1.py,
elicit_v2.py and elicit_natcond_v1.py; FBSIM_FREECIV_N_CIVS still overrides
the count. With a run, sets, reports and model panels default to the run
folder, --out to <run>/evals/<timestamp> and score_v2 --out to
<run>/scores/<name>. score_v2 also scores a draw without natural
conditionals (natcond_600.json and natcond_extra_turn1.json are optional,
as in criteria_v1.py), which a one-player run does not have. Without a run
every prompt and the scores on the paper's run-1 results are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
With a run, an unparsed binary answer is no longer scored as p = 0.5: it is left out of the scores, and its row stays in
score_items.csv unscored so the parse rate still counts it. SCORES.md says so. Without a run the paper's rule (0.5)
stays the default, and the outputs are byte-identical to before. `--impute-binary P` still forces a value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
score_v2.py normalizes CRPS by one constant per continuous family, which no
script here produced. norm_constants_v1.py applies the paper's rule (median
truth p05-p95 range, one significant figure) and reproduces the paper's
file byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
Nothing reads sweep_v4.json: the bank is built from consol_v4/, and no
script or paper code uses the sweep. finalize.sh no longer runs
sweep_v4.py; draw/sweep_v4.py stays for manual use. finalize.sh now exits
non-zero when the pull, the corpus QC or the consolidation fails, and
removes consol_v4_SUMMARY.json with consol_v4/, so a failed rerun no longer
leaves the old summary looking current. pull2.sh fails when the listing
on the archive fails or is empty, or when any rsync chunk fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
Parallel launches raced on rp.sh's shared address-cache temp file, so
launches failed with "no archive addr", and launch_pending.sh never
retried a worker that had a launch log. Each cache write now uses its
own temp file. Each launch log ends with launch_rc=; a launch that ends
without lanes is kept as failed_launch_<id>_<time>.log, reported with an
ERROR line and retried up to --max-retries times (default 5), after
which launch_pending.sh exits 2. The line of a relaunch says which retry
it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
bringup.sh ran pod_setup.sh in one long ssh session and took its exit
code from that session's output, so a dropped connection failed a healthy
pod (worker w01 of the pilot p1_pilot2). pod_setup.sh now runs detached
and writes its exit code to logs/pod_setup.rc, which bringup.sh polls over
fresh ssh sessions for up to 40 minutes; it still prints
pod_setup_rc=<code>, or pod_setup_rc=timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
worker.sh N tops lanes up to N, counting a lane alive when kill -0
succeeds on its pid. A lane that exits after 15 idle minutes can stay a
zombie, which kill -0 also accepts, so tasks moved to that pod later by
rebalance.sh were never run (seven workers of run p1_t60 sat idle with
about 220 tasks). A lane now counts only if its pid still runs
"worker.sh __lane N".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
setup_worktree.sh (run by the post-checkout hook in every new worktree)
links worlds/freeciv/fbsim_v3/fleet/runs to data/freeciv/fbsim_v3/runs, so
`source runs/<run>/shell.sh` works from fleet/. The link is relative, so it
resolves through each checkout's data/ link; it is gitignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
2_anchors.sh .. 9_score.sh run one step of a run each: play anchors (in
batches of the pod's civservers), keep anchors, start the fleet (archive
pod, workers, queue, parallel bringup, lanes), wait and finish (with an
offer to rerun twice-failed tasks before any worker is deleted), pull and
consolidate, draw sets and reports, elicit (dry run, then confirmation)
and score. Each takes the run name first, sources the run's shell.sh,
prints timestamped progress, waits through the long phases and can be
rerun after an interruption, reusing pods, bringups and finished work.
Pods are deleted only after a y/N prompt. steps_lib.sh holds the shared
helpers; on_pod/anchor_status.sh reports each anchor's state, checking the
process before the log so an anchor that just finished is not counted as
failed. 4_start_fleet.sh passes --max-retries to launch_pending.sh and
stops when a worker's launch gives up; 6_pull_corpus.sh stops when
finalize.sh fails. Steps 2-7 were run against simulated pods, and steps
1-6 since on RunPod (pilot p1_pilot2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
Claude-Session: https://claude.ai/code/session_015Vag3y25S6CPB5oXYT1UiB
RUNBOOK.md describes one run end to end: the run folder, keys, the nine
step scripts with what each reads and writes (including the draw rules and
scoring that differ from the paper's for a run), the existing runs, manual
commands, and what to check before the first T120 run. README.md introduces
runs, lists the step scripts and points to the runbook; AGENTS.md describes
the per-run fleet state; PROVENANCE.json records the adaptations; CLAUDE.md
names the run folders under data/freeciv.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3woZLamNbz7WR4bfQ2
Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
@fdrocha
fdrocha marked this pull request as ready for review October 30, 2026 14:48
fdrocha and others added 5 commits October 30, 2026 15:29
worker.sh gives each replay 3,600 s (was 1,800 s) before it is killed and
retried. A replay from a turn-120 fork plays later, slower turns and is
expected to take 12-21 minutes under a full lane load, too close to the
old limit; a hung replay now costs one lane an hour instead of half.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
show_intelligence_report_hearsay (an opponent with no embassy) wrote the
research target under `research`, a key the player observation space does
not have, so gymnasium's env checker rejected the first reset. Turn-60
games never reached it; every turn-120 fork of a five-player game failed
on it. Agents do not read this observation and results come from the
saves, so game outcomes are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
Step 4 gives a new worker 6 minutes to answer on ssh (bringup.sh's wait is
now FBSIM_SSH_WAIT_S, default 15 minutes as before). A worker that does
not answer is deleted and created again, up to three pods, and its queue
slice, not yet shipped, moves to the new pod's ID, so the rerun no longer
stops on a queue dealt to a deleted pod. On p5_t120_smoke one EU-SE-1
host took two workers in a row and 25 minutes each before giving up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
- 5_finish.sh runs finish_all.sh every interval, so each worker is finished and deleted as soon as its queue is
  empty instead of after the whole fleet; tasks that failed every attempt are listed from failed_tasks.txt.
- Lanes give each task a third attempt (90-minute timeout) in place of the end-of-run rerun prompt; finish_all.sh
  leaves workers running rerun_list.sh alone and records failed tasks before deleting a worker; rebalance.sh sends
  no tasks to finishing workers.
- stall_watch.sh stops a replay whose game has gone 10 minutes without a savegame: the game client hangs after the
  server rejects one of its commands ("Target not at target tile"), which on p5_t120 left replays waiting for the
  one-hour timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tze4TwpX5r8wVBbqZaQpZZ
4_start_fleet.sh takes --anchors (deal only these kept anchors) and --name wX (workers fbsim3-<run>-wX01.., so their
archive folders stay apart from the first fleet's). RUNBOOK: how to replace an anchor after the fleet is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Vag3wkZkNcQrvVuikmgyw

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant