fix(learn): a failed or one-off peak no longer pins the memory need

[SpoBo]

Why

Roderik saw a task that once took ~17 GB keep reserving that much on every later run. Vincent's DB shows the causes:

Key	Runs	Before	After
root:tsc_-p_tsconfig.json	0.2 ok, 0.3 exit 2, 14.5 exit 1 (starved=memory)	14.5 GB +25% boost = ~18 GB	0.2 GB, no boost
prd14394-gate	0.2 exit 1, 12.3 exit 1	12.3 GB	1.5 GB (new_job_mem)
host/dapi:…tsc…declarations.json	1–4.5 GB ok, then 5.9 GB exit 2 starved at need 4.5	7.4 GB for 5+ runs	4.5 GB, no boost
one-off 17 GB vs usual 9 GB (1.9x)		17 GB for 10 runs	9 GB
tsc-queue:packages_dapi_server (2 GB / 25 GB, 5/5)		25 GB	25 GB (unchanged, tested)

What changed (src/db.rs Db::learned)

1. A failed run (exit ≠ 0) never raises the memory need above the runs that passed. This holds with any number of runs, and also when the failed run is no ratio outlier. In code: the need is clamped to the max peak of the passing runs.
2. When every recent run failed, the need is failed_need: the second-highest failed peak, at least new_job_mem, at most the highest.
   • Why: one failed peak alone tells as little as no history, so the job reserves what a new job reserves.
   • A job that fails at the same peak again and again does get that peak.
   • It never reserves more than any failed run actually took.
3. Boost: a failed run that was starved of memory adds +25% only when its peak stayed within BOOST_FAILED_MAX (1.25) of its need. That means within what the boost would have given. OOM-killed at its need still boosts. A run that went 48x past its need and crashed does not.
4. outlier_ratio defaults to 1.5 (was 2). This holds back a one-off 1.9x peak.
   • The bimodal case stays safe: the rule that outliers must be fewer than the rest already leaves a 5/5 split alone.
   • On Vincent's real DB, 1.5 only adds 3 keys to taskguard outliers: one more vitest one-off, the dapi failed run, and web/dapp:bun_check (2 of 10 runs at ~3 GB vs 1.5 GB; it now reserves 2.3 GB).
5. Explain output. Learned.failed lists the failed runs whose peak is at least outlier_min above the need. Smaller ones are noise, and listing them flooded the output.
   • taskguard outliers shows them as ignored: the run failed (exit N).
   • doctor --explain shows a failed: line.
   • The job's end line warns when its run was one of them.

Example, on a copy of the real DB:

learned:    3 runs; 1.8 cores, 193 MB memory, usually 1s
failed:     1 run(s) up to 14.5 GB above the need; ignored: the run failed (exit 1), a run that goes wrong can take far more than the job needs (taskguard outliers)

Mixed versions: no change to the schema, queue entries or keys. Learn gets a new in-memory field, new_job_mem_kb.

Trade-off to know

The rule "failed never counts above passed" can under-reserve a job that only reaches its full size when it fails. The real DB has examples:

• e2e/agentic:bun_src/pr.ts: passes at 3.0 GB, failed 3 times at 4.1–4.4 GB. It now reserves 3.0 GB, before 4.4 GB.
• Many single-run e2e keys that failed at 5–7 GB now reserve 1.5 GB (new_job_mem) on their next run.

The README "Limits" section says this. A follow-up could let repeated failed peaks count by outlier_weights, the same way repeated outliers do.

Other paths that can pin a high need (not changed here)

• Long-lived jobs (dev stack): steady_mem_kb is the plain max of the last 10 steady peaks. It has no outlier or failed-run handling, so one steady spike holds for 10 runs. Vincent's dev-env:stack is 4.7–9.0 GB, with no one-off right now. See also A dev stack is admitted while the proof job that needs it cannot be, so the stack idles holding 7+ GB #27.
• New-job estimate: Db::estimate takes the max peak per similar key over 30 days and includes failed runs. One failed 14.5 GB tsc run can push the 90th percentile for new tsc keys.
• Imported tsc-queue history: a tsc/tsgo job with no runs of its own uses tsc-queue:<legacy key>, for example 25 GB for packages_dapi_server. This lasts only until its own first run. The new failed-run rule applies there too.
• --min-mem / config min_mem: prd14394-gate started with a need of 16 GB, more than it learned, so a floor is set somewhere. Roderik should check his DALP config and wrappers for a min_mem.
• I do not have Roderik's DB. taskguard outliers and taskguard doctor --explain "CMD" on the new build show which of these applies to him.

Tests

TDD in src/db.rs, with the real numbers:

• a_failed_run_never_raises_the_need_above_the_runs_that_passed (root:tsc and dapi)
• when_every_run_failed_one_high_peak_alone_does_not_set_the_need (prd14394-gate)
• a_failed_run_starved_at_its_need_still_raises_it
• a_one_off_peak_under_twice_the_usual_is_held_back
• a_job_with_two_sizes_reserves_the_big_one

cargo test: 167 + 28 pass. cargo clippy --all-targets and cargo fmt are clean. Not released.

🤖 Generated with Claude Code