fix(learn): a failed or one-off peak no longer pins the memory need

CHANGELOG.md

@@ -10,6 +10,24 @@ and the versions follow [semver](https://semver.org).
 
 ## [Unreleased]
 
+### Fixed
+
+- A run that failed no longer raises a job's memory need above what its
+  passing runs took, also with fewer than three runs and when it is no
+  outlier. One `tsc` run that failed at 14.5 GB made every later run reserve
+  18 GB. When every recent run failed, one high peak alone does not set the
+  need: the job reserves `new_job_mem` until a peak comes back.
+- A failed run starved of memory adds the 25% boost only when its peak stayed
+  within that 25% of its need.
+- `taskguard outliers`, `doctor --explain` and the job's end line show failed
+  runs whose peak is ignored, and why.
+
+### Changed
+
+- `outlier_ratio` defaults to 1.5 (was 2): a one-off 17 GB peak where a job
+  usually takes 9 GB no longer sets the need for its next 10 runs. A job with
+  two sizes, such as a cache hit and a full build, still reserves the big one.
+
 ## [0.9.0] - 2026-10-12
 
 ### Added

README.md

@@ -59,12 +59,17 @@ below).
   the need. A memory need never passes `mem_max`: a job that once took more
   is capped there, and says so.
 - **One wild run does not set the need.** A peak more than `outlier_ratio`
-  (2) times the next peak below it, and at least `outlier_min` (1 GB) above
+  (1.5) times the next peak below it, and at least `outlier_min` (1 GB) above
   it, is an outlier: a CI run that took 12 GB where it usually takes 4.5 GB.
   The first time it does not count, the second time it counts half, the third
-  time in full (`outlier_weights`). A run that failed never counts: a run that
-  goes wrong can take far more than the job needs, and on a real machine most
-  outliers were such runs. Jobs whose usual peak is under `outlier_min` have
+  time in full (`outlier_weights`). A run that failed (exit not 0) never
+  raises the need above the runs that passed, outlier or not, and with any
+  number of runs: a run that goes wrong can take far more than the job needs,
+  and on a real machine most outliers were such runs. When every recent run
+  failed, the need is the second-highest failed peak, at least `new_job_mem`
+  but never more than the highest: one failed peak alone does not set it.
+  A failed run that was starved of memory adds the 25% only when its peak
+  stayed within that 25% of its need. Jobs whose usual peak is under `outlier_min` have
   no outliers: there a jump is mostly a full build after cache hits, and
   holding it back would start the next full build into a full machine. The
   job's line says when its run was an outlier, `taskguard outliers` lists
@@ -531,7 +536,7 @@ most used ones:
 cpu_max = 100          # percent of all cores
 mem_max = 85           # percent of RAM
 hints = true           # agent hints on status lines
-outlier_ratio = 2      # a peak this many times the next one is an outlier; 0 = off
+outlier_ratio = 1.5    # a peak this many times the next one is an outlier; 0 = off
 outside_admit = true   # start jobs that only other programs keep out
 partial_fit = 0.5      # a job that cannot fit starts on half its CPU need; 0 = off
 max_backfill = 600     # small jobs pass a stuck job this long, then the machine drains
@@ -582,7 +587,9 @@ repo. Put a repo's own scripts in its `.taskguard.toml`:
   similar jobs can still take more than its estimate before it is learned.
 - An outlier that is real growth is learned late: the next run still
   reserves the usual peak, and only the third high run counts in full. A run
-  that failed is never learned, even when it really needed that memory. Set
+  that failed is not learned above the runs that passed, even when it really
+  needed that memory: a job that only reaches its full size when it fails,
+  such as a type check that passes on small changes, can be under-reserved. Set
   `outlier_weights = [0.5]` to learn sooner, or `outlier_ratio = 0` to learn
   every peak at once.
 - A job that starts early on `noise_mem` or `outside_admit` can push memory

src/commands.rs

@@ -316,7 +316,13 @@ pub fn outliers(args: &[String]) -> Result<i32> {
     let p = parse_prune(args)?;
     let rules = learn_rules(&p)?;
     let db = Db::open_dir(&config::state_dir())?;
-    let found = find_all_outliers(&db, p.pattern.as_deref(), &rules)?;
+    let mut found = Vec::new();
+    for k in db.keys(p.pattern.as_deref())? {
+        let l = db.learned(&k, &rules)?;
+        if l.outliers.is_some() || !l.failed.is_empty() {
+            found.push((k, l));
+        }
+    }
     println!(
         "a peak counts as an outlier when it is more than {}x and {} above the next peak below it (outlier_ratio, outlier_min)",
         rules.outlier_ratio,
@@ -328,19 +334,29 @@ pub fn outliers(args: &[String]) -> Result<i32> {
     }
     println!();
     println!("{:<52} {:>6} {:>9} {:>9} {:>9} {:>6} {:>9}  why", "key", "run", "ended", "peak", "usual", "x", "next run");
-    for (k, o) in &found {
-        for r in &o.runs {
+    for (k, l) in &found {
+        let next = l.mem_kb.unwrap_or(0);
+        let row = |r: &db::Outlier, usual: u64, why: String| {
             println!(
                 "{:<52} {:>6} {:>9} {:>9} {:>9} {:>6.1} {:>9}  {}",
                 k,
                 r.run_id,
                 ago(r.ended_at),
                 gb(r.peak_kb),
-                gb(o.normal_kb),
-                r.peak_kb as f64 / o.normal_kb.max(1) as f64,
-                gb(o.mem_kb),
-                o.why(r)
+                gb(usual),
+                r.peak_kb as f64 / usual.max(1) as f64,
+                gb(next),
+                why
             );
+        };
+        let outliers = l.outliers.iter().flat_map(|o| o.runs.iter().map(move |r| (o, r)));
+        for (o, r) in outliers.clone() {
+            row(r, o.normal_kb, o.why(r));
+        }
+        // Failed runs above the need that are no ratio outlier: usual is the
+        // need the runs that passed leave.
+        for r in l.failed.iter().filter(|f| !outliers.clone().any(|(_, r)| r.run_id == f.run_id)) {
+            row(r, next, r.why_ignored());
         }
     }
     println!();
@@ -560,6 +576,15 @@ fn explain(cmdline: &str, cwd: &Path, cfg: &Config, dir: &Path) -> Result<i32> {
                 if l.mem_boosted { " (+25% after a memory-starved run)" } else { "" },
                 l.dur_s.map(dur).unwrap_or("?".into())
             );
+            if let Some(f) = l.failed.first() {
+                let top = l.failed.iter().map(|r| r.peak_kb).max().unwrap_or(0);
+                println!(
+                    "failed:     {} run(s) up to {} above the need; {}, a run that goes wrong can take far more than the job needs (taskguard outliers)",
+                    l.failed.len(),
+                    gb(top),
+                    f.why_ignored()
+                );
+            }
             if let Some(o) = &l.outliers {
                 let top = o.runs.iter().map(|r| r.peak_kb).max().unwrap_or(0);
                 println!(

src/config.rs

@@ -200,7 +200,7 @@ impl Default for Config {
             max_bypass: 120,
             max_backfill: 600,
             boost_runs: 5,
-            outlier_ratio: 2.0,
+            outlier_ratio: 1.5,
             outlier_min_kb: 1024 * 1024,
             outlier_weights: vec![0.0, 0.5],
             noise_mem: 2.0,
@@ -370,6 +370,7 @@ impl Config {
             outlier_ratio: self.outlier_ratio,
             outlier_min_kb: self.outlier_min_kb,
             outlier_weights: self.outlier_weights.clone(),
+            new_job_mem_kb: self.new_job_mem_kb,
         }
     }