Queue work for a dev stack: book its run, infer it from history, and never start a stack that does not fit

[SpoBo]

Why

Idle dev stacks were blocking the browser runs they exist for. On the MacBook (10 cores, 32 GB, mem_max 95), the caller queues a stack's run only once the stack is up, about 6 minutes later, so taskguard could not know the run was coming. Two things then went wrong:

• The progress rule ("nothing runs but steady stacks") started the next priority-10 stack even though it did not fit in memory.
• That rule was skipped altogether while macOS reported memory pressure at warning.

The first stack's run then waited until the agent's 10-minute command timeout gave up and took the stack down: 7 of 97 browser runs in three days, 6 of them at exactly 10.0 minutes. On Omarchy, one 9.1 GB stack that was 5.4 GB short held back 25 other jobs for 27 minutes.

The captain's ask: queue work for a stack, so the manager knows to prioritise it and knows the full load on the machine. Use context to tell whether work is coming and what it is. A stack with no work must stay possible.

What changed

1. Progress rule and head of line.
   • Progress rule. With nothing running, or only steady long-lived jobs, the first job in line that fits starts, up to 100% of RAM; CPU is ignored, as before. A long-lived job must fit under mem_max (+ noise_mem), or under the outside_admit ceiling when only programs outside taskguard keep it out. When nothing of taskguard runs, a lone job (stack or not) may go up to 100% of RAM, so outside programs can never keep it waiting forever. A job that does not fit holds nobody back. The rule also decides at macOS warning; critical still stops everything.
   • Head of line. A waiting job that would not fit even once every job that ends by itself has ended holds nobody back, whatever its priority or reservation. Live on Omarchy today, one 5.5 GB oxlint that only fit once a stack stopped was holding back 23 jobs, among them a 189 MB test.
   • NEXT line. A long-lived holder is named as "is stopped", not "finishes".
2. Declared work for a stack.
   • Declaring. --expect WORK (repeatable) or TASKGUARD_EXPECT=WORK,WORK on a long-lived job, where WORK is a pool name, job key or command pattern. DALP can pass the env var today through devenv up.
   • Booking. The stack books the declared work's need and starts only with room for both. For a pool, the booking is the pool's typical start need (ea52172).
   • Linking. A job in the same checkout that matches links (work_for) and goes to stack priority + 1. It takes over the booked room when it starts, and while its stack still waits it waits too, holding nobody back.
   • Strict hold. A stack with work queued for it, and work whose stack runs, hold their turn strictly instead of by backfill, as long as they could fit once the jobs that end by themselves have ended.
   • Status. (for stack KEY) on the work, (expects WORK, books SIZE) on the stack.
3. Work inferred from history (infer_work, on by default).
   • When it infers. A stack with no declaration infers work when at least infer_work_min (3) of its last infer_work_of (5) runs in the checkout had work of one kind (a pool, else a key) queue within infer_work_within (900 s) of their start.
   • What it books. That work's usual need, shown as (expects browser from history, books …).
   • Bounds:
     • The booking expires at 2× the usual delay (10 min at most), and is dropped once the work starts or the stack stops.
     • A guess that came true (its work queued and linked) does not expire while that work waits.
     • Other stacks respect an open guess. A stack booted with no work is therefore delayed at most until the guess expires; it starts at once if it fits alongside.
     • The progress rule ignores guesses for jobs that end by themselves, so the queue never stops for a guess.
     • Only declared work and open guesses link and lift priority.
     • A declaration wins, --expect none expects nothing, and infer_work = false turns inference off.
4. expect_wait (5 s; 0 turns it off). A stack that expects work, declared or inferred, waits up to that long before starting for the work to queue, and starts as soon as it has. A stack that expects nothing starts at once.
5. outside_admit under a high mem_max. The ceiling was pause_at (92), under mem_max 95, so outside_admit could never admit anything on the MacBook. It is now halfway from mem_max to 100% when mem_max >= pause_at (97.5 for 95).
6. Stale steady_mem_kb doc, test comment and README sentence fixed: a steady job books what it holds + 25%.
7. src/stall.rs: a test-only model driving the real decide(), Entry::follow() and link_work() second by second. It asserts the scenarios below.

Data formats (Mixed versions): all additive, with serde defaults

• queue::Entry: long_lived, expects: Vec<Expect> (work, mem_kb, from, started, implied, expire_after_s, expired), work_for: Option<WorkFor>, expect_until.
• queue::Nudge: work_started.
• Untouched: machine::MachineSample, the database schema (the history query reads existing runs columns), and job keys.
• New settings: expect_wait, infer_work, infer_work_min, infer_work_of, infer_work_within. Older versions warn about them and ignore them.
• Older versions (≤ 0.10.0):
  • They do not see bookings and may start into them; they honour the priority lift, which is an ordinary priority.
  • They may mark a stack waiting in expect_wait as stalled after 10 s, so keep expect_wait under 10 while they share a machine.

Evidence for "work is incoming"

• The declaration, which books the declared work.
• The job history for the same checkout (N of M within the window), which books the work's usual need as a bounded guess.
• Either one also triggers expect_wait. A stack with neither, or with --expect none, starts at once.

Model results (src/stall.rs)

Setup: 2 h, MacBook limits (mem_max 95, max_backfill 1800, warn_admit), outside load 11 / 12.8 / 14.5 GB. Two agents each bring up a stack, queue their browser run (3 cores, 4.3 GB) once it is up, and give the run up after 10 minutes. An older turbo run keeps queueing lints.

Columns: runs done · runs given up · worst run wait · stack idle while its run waited · max stacks at once · worst memory · seconds over the limit · lints done · worst stack wait.

Before (master, 0.10.0 rules):

outside 11 GB, outside cap 92%         4      8      600s    7047s      2  86.9%     0s    563      230s
outside 12.8 GB, outside cap 92%       5      7      600s    6350s      2  92.5%     0s    556      210s
outside 14.5 GB, outside cap 92%       5      7      600s    6370s      2  93.0%     0s    182      220s
Omarchy head of line: second stack started at once, worst memory 114.4%, small jobs waited up to 1000 s

After (head 965cdaa):

variant                             runs   gave worst run    stack stacks  worst   over  lints     worst
                                    done     up      wait     idle at once    mem  limit   done stack wait
outside 11 GB, inferred (default)     16      0      129s    1098s      2  84.1%     0s    207      233s
outside 11 GB, declared               13      0      120s     358s      2  84.7%     0s    381      600s
outside 11 GB, no declaration/history 15      0      300s    2394s      2  86.6%     0s    272      230s
outside 12.8 GB, inferred (default)   15      0      129s    1104s      2  89.4%     0s    141      235s
outside 12.8 GB, declared             11      0        0s      11s      1  86.8%     0s    462      720s
outside 12.8 GB, no decl./history     15      0      209s    1760s      2  89.9%     0s    272      230s
outside 14.5 GB, inferred (default)   15      0      149s    1094s      2  94.6%     0s    141      255s
outside 14.5 GB, declared             11      0        9s      20s      1  87.3%     0s    293      600s
outside 14.5 GB, no decl./history      6      6      600s    3980s      2  94.6%     0s    490      582s

Stack and run queued together (the run 2 s after the stack):

together 12.8 GB, nothing             47      0       41s      91s      1  84.8%     0s    140      270s  23 without stack
together 12.8 GB, history implies     28      0      210s       0s      2  79.4%     0s     86      212s   0 without stack
together 12.8 GB, declared            28      0      210s       0s      2  79.4%     0s     86      212s   0 without stack
together 14.5 GB, nothing             37      0      244s     996s      2  90.6%     0s    153      360s  18 without stack
together 14.5 GB, history implies     27      0      228s       0s      1  80.5%     0s    142      230s   0 without stack
together 14.5 GB, declared            27      0      228s       0s      1  80.5%     0s    142      230s   0 without stack

Bad guess (history infers a run after every stack, but the agents only boot stacks, 20 min each, and never queue a run):

variant                             lints worst lint  worst    stacks worst stack
                                     done       wait    mem   at once        wait
11 GB, no run follows, no inference    386       540s  86.3%         2          5s
11 GB, no run follows, inferred       444       327s  86.9%         2        243s
12.8 GB, no run follows, no inference  264       620s  88.7%         2         50s
12.8 GB, no run follows, inferred     302       385s  89.9%         2        235s
14.5 GB, no run follows, no inference  301       882s  89.6%         2        820s
14.5 GB, no run follows, inferred     221       880s  90.0%         2        825s
Omarchy head of line: second stack stays out, 24 of 25 jobs started, small jobs waited up to 234 s, worst memory 92.0%

Which scenarios clear:

• MacBook morning with the defaults (no declaration, history inference on): clears at 11, 12.8 and 14.5 GB. No run is given up and memory stays under 95%. This is the DALP flow as it is today, with no DALP change.
• Caller declares (TASKGUARD_EXPECT=browser): clears at all three loads. At 12.8 and 14.5 GB the run starts within 9 s of being queued. The cost: one stack at a time, so 11–13 runs in 2 h, and a second stack waits up to ~12 min.
• No declaration and no history (infer_work = false): the progress-rule fix alone clears 11 and 12.8 GB, but not 14.5 GB (6 given up).
• Stack and run queued together: with a declaration or history, no run is given up and none starts before its stack. With neither, 18–29 runs start without their stack.
• Bad guess: no starvation. Lints keep flowing (no lint waits longer than without inference), and a stack is delayed at most until the guess expires: +238 s at 11 GB, +185 s at 12.8 GB, +5 s at 14.5 GB. Memory stays ≤ 90%.
• Omarchy head of line: the second stack no longer starts to 114% of RAM. Small jobs start; worst wait 1000 s → 234 s.
• Scout's fallback (a steady stack books only what it holds, without growth room): not included. It gives up 5 runs at 12.8 GB.

Validation

• On head 965cdaa, with the CI commands: cargo fmt --check ok, cargo clippy --all-targets -- -D warnings ok, cargo test: 197 unit + 31 CLI tests passed.
• no-mistakes:
  • Review took 3 rounds; the 7 ask-user findings were decided by firstmate and fixed by the pipeline (744ec32, 460f763).
  • Test took 3 rounds and fixed two things: the declared 12.8 GB case still gave up one run, fixed by booking a pool's typical start need (ea52172); and a stack-hold test was aligned with the ruled lone-job ceiling (8d5e037). It then drove 7 live CLI scenarios against the real binary, all passing.
  • Document applied rustfmt and wording (c87d101).
  • The first CI run failed on macOS only: a_stack_waits_for_the_work_it_expects_when_both_queue_together waited 195 s against its 15 s bound. Its first-run bookings (1.5 GB each) did not fit in all of RAM on the 7 GB runner, which is the ruled hard ceiling, so the stack waited for memory. Fixed in the test, which now books 64 MB (50795e8).
  • A second run then simplified the guess lifecycle: "still a guess" is defined once, as an implied item no job is linked to, and that predicate drives booking, expiry, link lapse, the runner and the model (965cdaa).
  • The pipeline's PR and CI steps were skipped only because its gh session reported "not logged into any accounts on ssh.github.com". The pushed head is the validated one, and this PR was opened by hand with no new commits.
  • A third run validated the residual fix, the pre-release support and the version bump (review, test, document, lint, push) and pushed db82278. Its review fixed the explain memory line (3a73eed); its document step reflowed README and CHANGELOG (db82278).

Residual fixed, and pre-release 0.11.0-rc.1

• Second run on a came-true guess (fixed, 0e96cc0): still_expected is !x.implied || x.open() again. Once the run that took over a history guess has started, a second matching run in the same checkout no longer links to the stack, gets no priority lift and does not hold its turn; one that linked earlier lets go of its link. Declared work still links a second run. Test: a_second_run_does_not_ride_on_a_guess_the_first_run_made_come_true. The stall model's tables are unchanged by it.
• Label (fixed, 0e96cc0): taskguard status shows "from history" on a guess only until work queues for it (test a_guess_reads_from_history_until_work_queues_for_it).
• Explain's memory line (fixed by the review, 3a73eed): the memory check in explain/status leaves out a waiting undeclared stack's own guessed booking, as the scheduler does, so it no longer shows a failure for a stack that would start.
• Pre-release support (79beb9e): a tag with a suffix (v0.11.0-rc.1) is published as a pre-release and never as latest; the release skill notes this.
• Version (ff7e3b5): 0.11.0-rc.1, with its CHANGELOG.md section. This PR stays unmerged; the pre-release v0.11.0-rc.1 is tagged on its head.

Version in taskguard top, and pre-release 0.11.0-rc.2

• 2c5c3c6: taskguard top shows the binary's version (CARGO_PKG_VERSION; the build exposes no git describe and no build script was added) at the right of the key bar, in the room the keys leave: taskguard VERSION, vVERSION when narrow, nothing when the keys fill the line, so it never hides a key. Test: the_footer_shows_the_version_at_the_right_when_the_keys_leave_room.
• 4ae9d40: version 0.11.0-rc.2, with its CHANGELOG.md section; the pre-release v0.11.0-rc.2 is tagged on this head. Validated by a fourth no-mistakes run (review clean, test, document, lint, push) with no fixes.

Not in this PR

• No full release, no merge to master.
• No DALP pin change. Optionally, DALP can set TASKGUARD_EXPECT=browser where an agent brings a stack up for a browser run; with history inference this is no longer required.