From 524b312b2bad080360910dda014d001076ecd1d8 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 15:08:56 +0100 Subject: [PATCH 01/16] M166 T1: final fit under a multi-element eval_time, invariant --- cairn/ROADMAP.md | 2 +- cairn/milestones/M166-finer-second-oracle.md | 7 ++-- tests/testthat/test-eval-time.R | 39 +++++++++++++++++++- 3 files changed, 43 insertions(+), 5 deletions(-) diff --git a/cairn/ROADMAP.md b/cairn/ROADMAP.md index de03b7c4..c7913b52 100644 --- a/cairn/ROADMAP.md +++ b/cairn/ROADMAP.md @@ -4,7 +4,7 @@ _Last hygiene check: 2026-10-31 M165 review. Graduated Guard the devel-vctrs fil ## Milestones | ID | Title | Status | Depends on | Priority | File/Archive | |---|---|---|---|---|---| -| M166 | Give finer results a second oracle | planned | — | normal | milestones/M166-finer-second-oracle.md | +| M166 | Give finer results a second oracle | in-progress | — | normal | milestones/M166-finer-second-oracle.md | | M165 | Keep the devel-vctrs test choice from drifting | done | — | normal | milestones/archive/M165-devel-vctrs-drift.md | | M164 | Give six one-type results a second oracle | done | — | normal | milestones/archive/M164-second-oracle-types.md | | M163 | Keep an oracle registry outside tests | done | — | normal | milestones/archive/M163-oracle-registry.md | diff --git a/cairn/milestones/M166-finer-second-oracle.md b/cairn/milestones/M166-finer-second-oracle.md index 85c20b0d..915c3552 100644 --- a/cairn/milestones/M166-finer-second-oracle.md +++ b/cairn/milestones/M166-finer-second-oracle.md @@ -1,13 +1,13 @@ # M166: Give finer results a second oracle -- **Status:** planned +- **Status:** in-progress - **Priority:** normal - **Depends on:** — - **Driving RR:** — - **Principles touched:** GP2 - **Resolves:** — - **Surface tier:** internal — tests and in-repo tracking records; no package code or user doc changes -- **Branch/PR:** — +- **Branch/PR:** m166-finer-second-oracle ## Goal @@ -41,7 +41,7 @@ Back each result that the candidate row "Give finer results a second oracle" nam ## Tasks -- [ ] T1: Final fit under `eval_time`, invariant. In `test-eval-time.R`, after the test at :491, run `nested_tune_grid()` through `memoised()` at `c(0.5, 10)`, `c(10, 0.5)`, `0.5` and `10`. Fit each run with `set.seed(11)` first, as that test does. Compare the selected `dist` and `predict()` on `srv_data()`. Plant: pair each order with the other element's run, and see the test fail. +- [x] T1: Final fit under `eval_time`, invariant. In `test-eval-time.R`, after the test at :491, run `nested_tune_grid()` through `memoised()` at `c(0.5, 10)`, `c(10, 0.5)`, `0.5` and `10`. Fit each run with `set.seed(11)` first, as that test does. Compare the selected `dist` and `predict()` on `srv_data()`. Plant: pair each order with the other element's run, and see the test fail. - [ ] T2: Resample weights, invariant. In `test-resample-weights.R`, beside the tests at :172 and :204, build the weighted design with a new weight on the failed fold (`break_fold()`, helper-orchestration.R:532) and on the NA fold (`plant_na`). Compare `collect_metrics()` rows. Plant a defect the fixture can show, such as matching weights to folds by position in `summarize_folds()` (R/nested-results.R:900), see it fail, and restore. - [ ] T3: Desirability, closed-form. Read desirability2 0.2.0's help pages for `select_best_desirability()` and its term functions. Write the source note with its `INDEX.md` line. In `test-nested-tune-grid-oracles.R`, beside the test at :432, reuse that run's rule and fixture. Compute each term's desirability over each fold's candidates, their geometric mean, and the top candidate. Compare with `.selected`. Fold 3 picks apart from the best `rmse`, so the rule shows. Plant: flip one term's direction, and see the test fail. - [ ] T4: Selection metric. Add the assertion to the test at `test-selection-metric.R:15`. Plant: compare with the set's last name, and see it fail. @@ -61,3 +61,4 @@ Back each result that the candidate row "Give finer results a second oracle" nam - 2026-10-31: plan gate chose a weight-change invariant for left-out folds over a comparison with `tune::fit_resamples()`, because tune ignores the weights or errors when a fold is missing (tune#1197); falsified by a tune release that drops a missing fold and keeps the weights. - 2026-10-31: plan gate chose a two-element-against-one-element invariant for the final fit under `eval_time` over the IPCW Brier score written out for the final fit's tuning run, because the invariant reuses fixtures and the closed form already backs the runs; falsified by the final fit reading `eval_time` by a route that the invariant cannot reach. - 2026-10-31: plan gate chose one milestone over two, because three tests, one assertion and the registry edit fit two sessions; falsified by the plan-owned body passing the cap or T1-T5 not fitting two sessions. +- 2026-10-31: T1: added "a final fit at a multi-element `eval_time` is the final fit at its first element" to `test-eval-time.R` (5 expectations, file green under `NESTEDTUNE_FULL_SUITE=true`). Plant: pairing each order with the other element's run failed 4 of 5 expectations (lognormal against weibull), reverted. Header comment names the test beside the run form. diff --git a/tests/testthat/test-eval-time.R b/tests/testthat/test-eval-time.R index 1e97f8ee..0790c44c 100644 --- a/tests/testthat/test-eval-time.R +++ b/tests/testthat/test-eval-time.R @@ -27,7 +27,9 @@ # at a multi-element `eval_time` and a run at that vector's first element have # to select the same candidate and report the same number at that element, # because `tune:::first_eval_time()` takes element one either way. Pinned by -# "a multi-element `eval_time` reports what its first element names". +# "a multi-element `eval_time` reports what its first element names", and in +# its final-fit form by "a final fit at a multi-element `eval_time` is the +# final fit at its first element". # The evaluation times a censored-regression metric is measured at. # @@ -545,6 +547,41 @@ test_that("nested_final_fit() tunes and selects under the caller's evaluation ti expect_false(identical(fits[[1L]]$selected$dist, fits[[2L]]$selected$dist)) }) +test_that("a final fit at a multi-element `eval_time` is the final fit at its first element", { + skip_if_no_censored() + + data <- srv_data() + nested <- srv_nested(data) + workflow <- srv_workflow(data) + times <- srv_eval_times() + + final_at <- function(eval_time) { + set.seed(9) + res <- suppressWarnings(memoised(nested_tune_grid( + workflow, + nested, + grid = srv_grid(), + metrics = srv_metrics(), + eval_time = eval_time + ))) + set.seed(11) + suppressWarnings(memoised(nested_final_fit(workflow, res))) + } + + orders <- list(times, rev(times)) + picked <- character(0) + for (k in seq_along(orders)) { + vector_fit <- final_at(orders[[k]]) + scalar_fit <- final_at(orders[[k]][[1L]]) + expect_identical(vector_fit$selected$dist, scalar_fit$selected$dist) + expect_identical(predict(vector_fit, data), predict(scalar_fit, data)) + picked[[k]] <- vector_fit$selected$dist + } + # The two orders pick apart, so a final fit that read the wrong element + # would fail one of the comparisons above. + expect_false(identical(picked[[1L]], picked[[2L]])) +}) + # Help pages ------------------------------------------------------------ # # Read from the generated `.Rd` rather than from the roxygen, because what a From c68fb36b9175a4694ab33df6320cc87a84b1572c Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 15:10:33 +0100 Subject: [PATCH 02/16] M166 T2: weights on a left-out fold, invariant --- cairn/milestones/M166-finer-second-oracle.md | 3 +- tests/testthat/test-resample-weights.R | 58 ++++++++++++++++++-- 2 files changed, 56 insertions(+), 5 deletions(-) diff --git a/cairn/milestones/M166-finer-second-oracle.md b/cairn/milestones/M166-finer-second-oracle.md index 915c3552..b1b45bb9 100644 --- a/cairn/milestones/M166-finer-second-oracle.md +++ b/cairn/milestones/M166-finer-second-oracle.md @@ -42,7 +42,7 @@ Back each result that the candidate row "Give finer results a second oracle" nam ## Tasks - [x] T1: Final fit under `eval_time`, invariant. In `test-eval-time.R`, after the test at :491, run `nested_tune_grid()` through `memoised()` at `c(0.5, 10)`, `c(10, 0.5)`, `0.5` and `10`. Fit each run with `set.seed(11)` first, as that test does. Compare the selected `dist` and `predict()` on `srv_data()`. Plant: pair each order with the other element's run, and see the test fail. -- [ ] T2: Resample weights, invariant. In `test-resample-weights.R`, beside the tests at :172 and :204, build the weighted design with a new weight on the failed fold (`break_fold()`, helper-orchestration.R:532) and on the NA fold (`plant_na`). Compare `collect_metrics()` rows. Plant a defect the fixture can show, such as matching weights to folds by position in `summarize_folds()` (R/nested-results.R:900), see it fail, and restore. +- [x] T2: Resample weights, invariant. In `test-resample-weights.R`, beside the tests at :172 and :204, build the weighted design with a new weight on the failed fold (`break_fold()`, helper-orchestration.R:532) and on the NA fold (`plant_na`). Compare `collect_metrics()` rows. Plant a defect the fixture can show, such as matching weights to folds by position in `summarize_folds()` (R/nested-results.R:900), see it fail, and restore. - [ ] T3: Desirability, closed-form. Read desirability2 0.2.0's help pages for `select_best_desirability()` and its term functions. Write the source note with its `INDEX.md` line. In `test-nested-tune-grid-oracles.R`, beside the test at :432, reuse that run's rule and fixture. Compute each term's desirability over each fold's candidates, their geometric mean, and the top candidate. Compare with `.selected`. Fold 3 picks apart from the best `rmse`, so the rule shows. Plant: flip one term's direction, and see the test fail. - [ ] T4: Selection metric. Add the assertion to the test at `test-selection-metric.R:15`. Plant: compare with the set's last name, and see it fail. - [ ] T5: Registry and decision. Add the AC5 rows and the AC6 lines under the file's quoting rule. Rewrite the selection group's paragraph and the header paragraph that point at the candidate row. Append the D-entry. Re-point `helper-time-budget.R` rows for the new blocks in one pass. @@ -62,3 +62,4 @@ Back each result that the candidate row "Give finer results a second oracle" nam - 2026-10-31: plan gate chose a two-element-against-one-element invariant for the final fit under `eval_time` over the IPCW Brier score written out for the final fit's tuning run, because the invariant reuses fixtures and the closed form already backs the runs; falsified by the final fit reading `eval_time` by a route that the invariant cannot reach. - 2026-10-31: plan gate chose one milestone over two, because three tests, one assertion and the registry edit fit two sessions; falsified by the plan-owned body passing the cap or T1-T5 not fitting two sessions. - 2026-10-31: T1: added "a final fit at a multi-element `eval_time` is the final fit at its first element" to `test-eval-time.R` (5 expectations, file green under `NESTEDTUNE_FULL_SUITE=true`). Plant: pairing each order with the other element's run failed 4 of 5 expectations (lognormal against weibull), reverted. Header comment names the test beside the run form. +- 2026-10-31: T2: added "a new weight on the fold a summary row leaves out leaves that row unchanged" to `test-resample-weights.R` (6 expectations, file green), and moved `plant_na()` to file level for both tests. A first draft set the left-out weight to 9 and differed in the 16th digit, because tune divides each weight by the total; the new weight now doubles the total, so the stored weights halve exactly. Plant: dividing the weighted sum by every fold's weight failed the NA case (0.196 against 0.392), reverted. The failed-fold case cannot show that plant, since a failed fold has no per-fold row. diff --git a/tests/testthat/test-resample-weights.R b/tests/testthat/test-resample-weights.R index 3a75075c..d05b4bd9 100644 --- a/tests/testthat/test-resample-weights.R +++ b/tests/testthat/test-resample-weights.R @@ -24,6 +24,9 @@ # Source: `tune::fit_resamples()` on the same outer `vfold_cv()` splits under # the same weights, whose `collect_metrics()` is tune's weighted branch itself. # Every-fold case only, where tune and this package agree. +# +# Source: two runs that differ only in the weight of a fold that a summary row +# leaves out must give that row the same mean, standard error and count. skip_heavy_on_cran() @@ -65,6 +68,12 @@ expect_weighted_rows <- function(summarized, per_fold, weights) { } } +# Fold 3's rsq estimate set to NA on the record. +plant_na <- function(x) { + x$.metrics[[3L]]$.estimate[x$.metrics[[3L]]$.metric == "rsq"] <- NA_real_ + x +} + # The normalized weights tune stores, named by fold id. stored_weights <- function(design) { w <- attr(design, ".resample_weights") @@ -211,10 +220,6 @@ test_that("with a fold scoring NA, the weights are renormalized over the folds t # record rather than the reader, so the summary's NA rule is what is under # test; the per-fold estimates the oracle averages come off the unweighted # run, as in the other tests. - plant_na <- function(x) { - x$.metrics[[3L]]$.estimate[x$.metrics[[3L]]$.metric == "rsq"] <- NA_real_ - x - } res <- plant_na(res) plain <- plant_na(plain) per_fold <- collect_metrics(plain, summarize = FALSE) @@ -230,6 +235,51 @@ test_that("with a fold scoring NA, the weights are renormalized over the folds t expect_identical(got$n[got$.metric == "rmse"], 3L) }) +test_that("a new weight on the fold a summary row leaves out leaves that row unchanged", { + skip_if_no_engines() + d <- make_reg_data() + summary_at <- function(weights, edit) { + design <- weighted_design(d, weights) + res <- suppressWarnings(weighted_grid_run(d, edit$design(design))) + got <- suppressWarnings(collect_metrics(edit$run(res))) + got[got$.metric %in% edit$rows, c(".metric", "mean", "std_err", "n")] + } + cases <- list( + failed = list( + design = function(x) break_fold(x, 2L, "inner tuning"), + run = identity, + rows = c("rmse", "rsq"), + left_out = 2L, + scored = 1L + ), + na = list( + design = identity, + run = plant_na, + rows = "rsq", + left_out = 3L, + scored = 1L + ) + ) + base <- c(1, 2, 3) + for (edit in cases) { + reference <- summary_at(base, edit) + expect_false(anyNA(reference$mean)) + + # The new weight doubles the total, so the weights tune stores (each over + # the total) halve exactly and the comparison can be exact. + moved <- base + moved[[edit$left_out]] <- base[[edit$left_out]] + sum(base) + expect_identical(summary_at(moved, edit), reference) + + # A new weight on a scored fold moves the mean, so the comparison above + # can fail. + moved <- base + moved[[edit$scored]] <- 9 + changed <- summary_at(moved, edit) + expect_true(all(changed$mean != reference$mean)) + } +}) + test_that("equal weights are no weights, and the numbers are the unweighted run's", { skip_if_no_engines() d <- make_reg_data() From c8d2a4a7400da8d12f3931532c6206d84bb96a6c Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 15:12:15 +0100 Subject: [PATCH 03/16] M166 T3: desirability rule by its definitions, closed-form --- cairn/milestones/M166-finer-second-oracle.md | 3 +- cairn/references/INDEX.md | 1 + cairn/references/desirability2-select-best.md | 27 ++++++++ .../testthat/test-nested-tune-grid-oracles.R | 68 +++++++++++++++++++ 4 files changed, 98 insertions(+), 1 deletion(-) create mode 100644 cairn/references/desirability2-select-best.md diff --git a/cairn/milestones/M166-finer-second-oracle.md b/cairn/milestones/M166-finer-second-oracle.md index b1b45bb9..310ad95d 100644 --- a/cairn/milestones/M166-finer-second-oracle.md +++ b/cairn/milestones/M166-finer-second-oracle.md @@ -43,7 +43,7 @@ Back each result that the candidate row "Give finer results a second oracle" nam - [x] T1: Final fit under `eval_time`, invariant. In `test-eval-time.R`, after the test at :491, run `nested_tune_grid()` through `memoised()` at `c(0.5, 10)`, `c(10, 0.5)`, `0.5` and `10`. Fit each run with `set.seed(11)` first, as that test does. Compare the selected `dist` and `predict()` on `srv_data()`. Plant: pair each order with the other element's run, and see the test fail. - [x] T2: Resample weights, invariant. In `test-resample-weights.R`, beside the tests at :172 and :204, build the weighted design with a new weight on the failed fold (`break_fold()`, helper-orchestration.R:532) and on the NA fold (`plant_na`). Compare `collect_metrics()` rows. Plant a defect the fixture can show, such as matching weights to folds by position in `summarize_folds()` (R/nested-results.R:900), see it fail, and restore. -- [ ] T3: Desirability, closed-form. Read desirability2 0.2.0's help pages for `select_best_desirability()` and its term functions. Write the source note with its `INDEX.md` line. In `test-nested-tune-grid-oracles.R`, beside the test at :432, reuse that run's rule and fixture. Compute each term's desirability over each fold's candidates, their geometric mean, and the top candidate. Compare with `.selected`. Fold 3 picks apart from the best `rmse`, so the rule shows. Plant: flip one term's direction, and see the test fail. +- [x] T3: Desirability, closed-form. Read desirability2 0.2.0's help pages for `select_best_desirability()` and its term functions. Write the source note with its `INDEX.md` line. In `test-nested-tune-grid-oracles.R`, beside the test at :432, reuse that run's rule and fixture. Compute each term's desirability over each fold's candidates, their geometric mean, and the top candidate. Compare with `.selected`. Fold 3 picks apart from the best `rmse`, so the rule shows. Plant: flip one term's direction, and see the test fail. - [ ] T4: Selection metric. Add the assertion to the test at `test-selection-metric.R:15`. Plant: compare with the set's last name, and see it fail. - [ ] T5: Registry and decision. Add the AC5 rows and the AC6 lines under the file's quoting rule. Rewrite the selection group's paragraph and the header paragraph that point at the candidate row. Append the D-entry. Re-point `helper-time-budget.R` rows for the new blocks in one pass. - [ ] T6: Run `devtools::test()` and `air format --check .` clean. @@ -63,3 +63,4 @@ Back each result that the candidate row "Give finer results a second oracle" nam - 2026-10-31: plan gate chose one milestone over two, because three tests, one assertion and the registry edit fit two sessions; falsified by the plan-owned body passing the cap or T1-T5 not fitting two sessions. - 2026-10-31: T1: added "a final fit at a multi-element `eval_time` is the final fit at its first element" to `test-eval-time.R` (5 expectations, file green under `NESTEDTUNE_FULL_SUITE=true`). Plant: pairing each order with the other element's run failed 4 of 5 expectations (lognormal against weibull), reverted. Header comment names the test beside the run form. - 2026-10-31: T2: added "a new weight on the fold a summary row leaves out leaves that row unchanged" to `test-resample-weights.R` (6 expectations, file green), and moved `plant_na()` to file level for both tests. A first draft set the left-out weight to 9 and differed in the 16th digit, because tune divides each weight by the total; the new weight now doubles the total, so the stored weights halve exactly. Plant: dividing the weighted sum by every fold's weight failed the NA case (0.196 against 0.392), reverted. The failed-fold case cannot show that plant, since a failed fold has no per-fold row. +- 2026-10-31: T3: added "the desirability rule picks the candidate its definition names in the fold's inner metrics" to `test-nested-tune-grid-oracles.R:493` (10 expectations, green), on the same rule, grid and seed 20 as the desirability2 test, memoised. Source note `cairn/references/desirability2-select-best.md` with its INDEX line. The page's minimization formula prints "data = low", a typo; the note takes `.comp_min`'s body. The test asserts one top candidate per fold, since the pages leave the tie order unstated. Plant: `df2` maximized in the expected pick failed folds 2 and 3, reverted. The lint hook's remaining hits on the note are cairn's provenance tokens and minus signs. diff --git a/cairn/references/INDEX.md b/cairn/references/INDEX.md index 78474a6e..63a754ce 100644 --- a/cairn/references/INDEX.md +++ b/cairn/references/INDEX.md @@ -15,6 +15,7 @@ _Entries are bullet lines — `- page.md — what it covers` — not table rows; - outer-loop-object-requirements.md — synthesis; what a nested-resampling object must carry for a driver: 19 reads (R1–R19), 7 reconstructions (C1–C7), 11 class-boundary workarounds (W1–W11), and the two size axes measured — the in-process ratio grows with v, and a `nested_cv()` fold's own materialized frame is wire cost no dispatch-side leaning can remove. - code-inventory.md — synthesis; every top-level function in `R/` sorted into core / glue / resampling-layer / furniture / ambiguous, 106 entries (F001–F106) plus the backtick-quoted `[.nested_results`. Names the tune fact that removes each `glue` entry and what rsample would have to accept for each `resampling-layer` one, at tune 2.1.0 and rsample 1.3.2. - tune-select-best.md — source; tune 2.1.0's help page for `select_best()`, `select_by_one_std_err()` and `select_by_pct_loss()`, and where its percent-loss example differs from the method body. +- desirability2-select-best.md — source; desirability2 0.2.0's help pages for `select_best_desirability()` and its term functions: each term over the observed range, their geometric mean, and the typo in the page's minimization formula. - bates2023.md — source; CV estimates average error not the model's own, and naive CV intervals under-cover; its "nested CV" is a variance-estimation device, not ours. - krstajic2014.md — source; four CV pitfalls, repeated nested CV for assessment, and the P-estimate — a nested estimate belongs to a (model, protocol) pair. Nearest external statement of IP4. - wainer2021.md — source; flat CV picks the same classifier as nested CV 62–71% of the time, so nesting is overzealous *for algorithm selection*; leaves the estimand argument intact. diff --git a/cairn/references/desirability2-select-best.md b/cairn/references/desirability2-select-best.md new file mode 100644 index 00000000..31395eb5 --- /dev/null +++ b/cairn/references/desirability2-select-best.md @@ -0,0 +1,27 @@ +# desirability2-select-best — what desirability2's selection picks, as its help pages define it + +**Provenance.** Ingested 2026-10-31 by M166 from desirability2 0.2.0's installed help pages `show_best_desirability.Rd` (`?desirability2::select_best_desirability`), `inline_desirability.Rd` (`?desirability2::d_max`) and `d_overall.Rd`. They were read with `tools::Rd2txt(tools::Rd_db("desirability2")[[...]])`. The bodies of `desirability2:::.comp_min` and `desirability2::d_overall` were printed in the same session. +Pagination: —. +Extraction: read directly 2026-10-31 from desirability2 0.2.0's installed help pages and function bodies — observed 2026-10-31. + +**Citation.** Kuhn, M. desirability2: Desirability Functions for Multiparameter Optimization. R package version 0.2.0. Help pages "Investigate best tuning parameters" (`show_best_desirability.Rd`), "Desirability functions for in-line computations" (`inline_desirability.Rd`) and "Determine overall desirability" (`d_overall.Rd`). The pages cite Derringer, G., and Suich, R. (1980), Simultaneous Optimization of Several Response Variables, *Journal of Quality Technology*, 12, 214-219. + +**Role.** The definitions that the closed-form desirability oracle recomputes by hand. Not a statistical source: it fixes what the `"desirability"` rule means in desirability2, which this package's `selection_rule()` hands to. + +## Extracted values + +- Each term maps a metric to [0, 1], "where 1.0 is most desirable and zero is unacceptable" (`show_best_desirability.Rd`, Details). The overall value is "the geometric mean of the individual desirabilities" (same paragraph). +- "If `low`, `high`, or `target` are not specified, the observed data are used to estimate their values" (same page, Details). The page's example sets `low` and `high` to the observed minimum and maximum of the column. +- Maximization: d = 1 above `high`, 0 below `low`, and ((data - low) / (high - low))^scale between them (`inline_desirability.Rd`, Equations). `scale` defaults to 1. +- Minimization: d = 0 above `high` and 1 below `low` (same page). Between them the page prints the fraction as "data = low" over "low - high", a typo. The body of `.comp_min` computes ((x - high) / (low - high))^scale. That is the one reading that gives 1 at `low` and 0 at `high`. +- `d_overall()` takes the geometric mean by default (`d_overall.Rd`, Usage, `geometric = TRUE`). +- `select_best_desirability()` "returns a single row" (`show_best_desirability.Rd`, Value). The page also says that the ranking can hold "more than one model configuration that has identical overall desirability" (Details). The pages do not say which tied row comes first. + +## Traces to + +- `tests/testthat/test-nested-tune-grid-oracles.R:493`: the test "the desirability rule picks the candidate its definition names in the fold's inner metrics". It writes the maximization and minimization terms with the observed range of each fold's candidates and takes their geometric mean. It asserts that one candidate holds the top value, so the pick does not rest on the unstated tie order. +- `cairn/ORACLES.md`, the selection group: the closed-form row for that test. + +## Open questions + +- None for the test's terms. `target()`, `constrain()` and `categories()` are not written out, since the test's rule uses none of them. diff --git a/tests/testthat/test-nested-tune-grid-oracles.R b/tests/testthat/test-nested-tune-grid-oracles.R index 01db4b77..6f716ce5 100644 --- a/tests/testthat/test-nested-tune-grid-oracles.R +++ b/tests/testthat/test-nested-tune-grid-oracles.R @@ -487,3 +487,71 @@ test_that("the desirability rule picks what desirability2 picks on the fold's in best <- lapply(ref_best, function(fold) fold$selected[cols]) expect_false(identical(lapply(res$.selected, `[`, cols), best)) }) + +# The desirability rule as desirability2 0.2.0's `?show_best_desirability` +# and `?d_max` pages define it (Derringer and Suich, 1980). +test_that("the desirability rule picks the candidate its definition names in the fold's inner metrics", { + skip_if_no_bayes_fixture() + skip_if_not_installed("desirability2", minimum_version = "0.2.0") + + d <- make_reg_data() + wf <- bayes_workflow(d) + folds <- det_nested(d) + p <- bayes_param_info(wf) + g <- expand.grid(df1 = c(2L, 5L, 8L), df2 = c(2L, 5L, 8L)) + rule <- selection_rule( + "desirability", + minimize(rmse), + maximize(rsq), + minimize(df1), + minimize(df2) + ) + set.seed(20) + res <- memoised(nested_tune_grid( + wf, + folds, + grid = g, + metrics = reg_metrics(), + param_info = p, + select = rule + )) + + # With no `low` or `high` given, each term's range is the observed one. + d_max <- function(x) (x - min(x)) / (max(x) - min(x)) + d_min <- function(x) (x - max(x)) / (min(x) - max(x)) + pick <- function(m) { + rmse <- m[m$.metric == "rmse", ] + rsq <- m[m$.metric == "rsq", ] + rsq <- rsq[match(rmse$.config, rsq$.config), ] + terms <- cbind( + d_min(rmse$mean), + d_max(rsq$mean), + d_min(rmse$df1), + d_min(rmse$df2) + ) + overall <- apply(terms, 1L, function(t) prod(t)^(1 / length(t))) + # One top candidate, so the pick does not rest on a tie-break. + expect_identical(sum(overall == max(overall)), 1L) + top <- which.max(overall) + c(df1 = rmse$df1[[top]], df2 = rmse$df2[[top]]) + } + + for (i in seq_len(nrow(res))) { + sel <- res$.selected[[i]] + expect_identical( + c(df1 = sel$df1, df2 = sel$df2), + pick(res$.inner_metrics[[i]]), + info = paste("fold", i) + ) + } + # A rule that picked by rmse alone would fail the comparison above in a + # fold (see the desirability note above). + best <- lapply(res$.inner_metrics, function(m) { + m <- m[m$.metric == "rmse", ] + c(df1 = m$df1[[which.min(m$mean)]], df2 = m$df2[[which.min(m$mean)]]) + }) + expect_false(identical( + lapply(seq_len(nrow(res)), function(i) pick(res$.inner_metrics[[i]])), + best + )) +}) From a9ee2998457f7c4bca9265dadf7dd26df5fb40b9 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 15:12:54 +0100 Subject: [PATCH 04/16] M166 T4: final fit's first metric against the set's first name --- cairn/milestones/M166-finer-second-oracle.md | 3 ++- tests/testthat/test-selection-metric.R | 7 ++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/cairn/milestones/M166-finer-second-oracle.md b/cairn/milestones/M166-finer-second-oracle.md index 310ad95d..26dd38e5 100644 --- a/cairn/milestones/M166-finer-second-oracle.md +++ b/cairn/milestones/M166-finer-second-oracle.md @@ -44,7 +44,7 @@ Back each result that the candidate row "Give finer results a second oracle" nam - [x] T1: Final fit under `eval_time`, invariant. In `test-eval-time.R`, after the test at :491, run `nested_tune_grid()` through `memoised()` at `c(0.5, 10)`, `c(10, 0.5)`, `0.5` and `10`. Fit each run with `set.seed(11)` first, as that test does. Compare the selected `dist` and `predict()` on `srv_data()`. Plant: pair each order with the other element's run, and see the test fail. - [x] T2: Resample weights, invariant. In `test-resample-weights.R`, beside the tests at :172 and :204, build the weighted design with a new weight on the failed fold (`break_fold()`, helper-orchestration.R:532) and on the NA fold (`plant_na`). Compare `collect_metrics()` rows. Plant a defect the fixture can show, such as matching weights to folds by position in `summarize_folds()` (R/nested-results.R:900), see it fail, and restore. - [x] T3: Desirability, closed-form. Read desirability2 0.2.0's help pages for `select_best_desirability()` and its term functions. Write the source note with its `INDEX.md` line. In `test-nested-tune-grid-oracles.R`, beside the test at :432, reuse that run's rule and fixture. Compute each term's desirability over each fold's candidates, their geometric mean, and the top candidate. Compare with `.selected`. Fold 3 picks apart from the best `rmse`, so the rule shows. Plant: flip one term's direction, and see the test fail. -- [ ] T4: Selection metric. Add the assertion to the test at `test-selection-metric.R:15`. Plant: compare with the set's last name, and see it fail. +- [x] T4: Selection metric. Add the assertion to the test at `test-selection-metric.R:15`. Plant: compare with the set's last name, and see it fail. - [ ] T5: Registry and decision. Add the AC5 rows and the AC6 lines under the file's quoting rule. Rewrite the selection group's paragraph and the header paragraph that point at the candidate row. Append the D-entry. Re-point `helper-time-budget.R` rows for the new blocks in one pass. - [ ] T6: Run `devtools::test()` and `air format --check .` clean. @@ -64,3 +64,4 @@ Back each result that the candidate row "Give finer results a second oracle" nam - 2026-10-31: T1: added "a final fit at a multi-element `eval_time` is the final fit at its first element" to `test-eval-time.R` (5 expectations, file green under `NESTEDTUNE_FULL_SUITE=true`). Plant: pairing each order with the other element's run failed 4 of 5 expectations (lognormal against weibull), reverted. Header comment names the test beside the run form. - 2026-10-31: T2: added "a new weight on the fold a summary row leaves out leaves that row unchanged" to `test-resample-weights.R` (6 expectations, file green), and moved `plant_na()` to file level for both tests. A first draft set the left-out weight to 9 and differed in the 16th digit, because tune divides each weight by the total; the new weight now doubles the total, so the stored weights halve exactly. Plant: dividing the weighted sum by every fold's weight failed the NA case (0.196 against 0.392), reverted. The failed-fold case cannot show that plant, since a failed fold has no per-fold row. - 2026-10-31: T3: added "the desirability rule picks the candidate its definition names in the fold's inner metrics" to `test-nested-tune-grid-oracles.R:493` (10 expectations, green), on the same rule, grid and seed 20 as the desirability2 test, memoised. Source note `cairn/references/desirability2-select-best.md` with its INDEX line. The page's minimization formula prints "data = low", a typo; the note takes `.comp_min`'s body. The test asserts one top candidate per fold, since the pages leave the tie order unstated. Plant: `df2` maximized in the expected pick failed folds 2 and 3, reverted. The lint hook's remaining hits on the note are cairn's provenance tokens and minus signs. +- 2026-10-31: T4: the grid test in `test-selection-metric.R` now also compares the final fit's `first_metric` with `names(attr(ms, "metrics"))[[1L]]`; file green (8 tests). Plant: the set's second name failed both metric sets ("mae" against "rmse"), reverted. Header comment names the added comparison. diff --git a/tests/testthat/test-selection-metric.R b/tests/testthat/test-selection-metric.R index e449da52..f3159ca6 100644 --- a/tests/testthat/test-selection-metric.R +++ b/tests/testthat/test-selection-metric.R @@ -2,7 +2,8 @@ # set, as tune resolves it for the workflow. The oracle is tune's own reading # of the final fit's stored tuning run, `tune::.get_tune_metric_names()`, the # call every fold and the final fit select on. It is compared with the entry -# on both the results object and the final fit built from it. +# on both the results object and the final fit built from it. The grid test +# also compares both entries with the first name of the set it passed. skip_heavy_on_cran() @@ -29,6 +30,10 @@ test_that("a grid run records the first metric of the set it was given, in eithe extract_procedure(res)$first_metric, names(attr(ms, "metrics"))[[1L]] ) + expect_identical( + extract_procedure(final)$first_metric, + names(attr(ms, "metrics"))[[1L]] + ) } }) From c45e54fef4ea527b172479c7408b813ae138a8a8 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 15:14:31 +0100 Subject: [PATCH 05/16] M166 T5: registry rows, D-123 lines, and D-123 --- cairn/DECISIONS.md | 6 +++++ cairn/ORACLES.md | 27 ++++++++++++++++---- cairn/milestones/M166-finer-second-oracle.md | 3 ++- 3 files changed, 30 insertions(+), 6 deletions(-) diff --git a/cairn/DECISIONS.md b/cairn/DECISIONS.md index 44ba994d..d6d2fc92 100644 --- a/cairn/DECISIONS.md +++ b/cairn/DECISIONS.md @@ -2141,6 +2141,12 @@ upstream, either of which would let the file return to a shared blob. **Decision:** on a push or pull request the test step runs the `tests/testthat/test-*.R` files that carry the line `# devel-vctrs.yaml runs this file on every push and pull request.` as a whole line, and stops with an error when none does. The step decides the whole-suite switch from `GITHUB_EVENT_NAME` and `inputs.full` alone, and the concurrency group is keyed on those two raw values with the workflow and ref, so two runs in one group make the same whole-suite choice. On a subset run each commit's markers decide the files. Rejected at the plan gate: a `-compat` file-name convention, which still drops a file renamed out of it. **Consequences:** a test file joins the push-time watch by taking the marker line, replacing D-119's "adds itself to the filter". A scheduled run and a full dispatch no longer share a group, nor do a `full` false dispatch on the default branch and a push to it, so each pair can run side by side. Falsified by a watched file losing its marker in an edit that no run flags, or by a subset run cancelling or replacing a whole-suite run. (Narrows D-119.) +### D-123 (2026-10-31): a finer result that runs the code its group's two oracle types run, on another input, owes no oracle of its own, and the Bayesian and annealing iteration stage keeps one correctness type. Extends D-121 + +**Context:** M163's review cut the `cairn/ORACLES.md` groups finer and found results that rest on one oracle type. M164 left nine of them on a candidate row. M166's plan gate asked what to do with a result where no second type can be built at low cost. The user chose to record it in the registry with its reason. +**Decision:** (1) A finer result that runs the same code as its group's oracles, on another input, owes no oracle of its own, because the testing rule forbids a check repeated across inputs that run one code path. Its group carries one line that names the rows backing it. This covers `compute_metrics()` on the run's own and on a new metric set, the averages that only `hand_average()` checks, and the racing fold record. (2) The Bayesian and annealing iteration stage keeps one correctness type, the reference loop, beside a serial-against-parallel check. No independent oracle exists for tune's Gaussian-process proposals or finetune's perturbations. +**Consequences:** a finer result that runs a branch that no listed oracle runs owes a second type of its own. Each group's line is the record a later audit reads. Falsified by a defect in one of the named results that every oracle its line lists passes. + - Tidy the interrupt block's leftovers: four items in `test-parallel-interrupt.R` from M159's review, none added by M159. The ledger budgets no wait for the 60 s collect when no signal is sent. A SIGINT sent after `dispatch_folds()` returns early lands outside the block's handler. The header's "Established by execution against mirai 2.7.2 before the fix" is narrative history. The `m15-fold-` marker names carry a milestone id. Promote on the next change to that file — added 2026-10-24 at M159's review - Give finer results a second oracle: one-type results in two-type `cairn/ORACLES.md` groups (M163 review): `compute_metrics()` own vs new metric set, final fit under `eval_time` and `event_level`, failed-fold weights, race fold record, Bayesian and annealing iteration stage, memory slope, `hand_average()`-only cases. Also the selection group's desirability rule and selection metric (M164 Out). Normal rung: shipped on one type — added 2026-10-31 at M164's plan gate — started M166 +- Add second oracles to one-type parts: three parts that `cairn/ORACLES.md` credits to one type (D-123). `compute_metrics()`'s survival branch on a new metric set rests on U1. The racing record's rows for eliminated candidates rest on two live rows. The final fit under `event_level` rests on `event-level/O2`, since its level exchange is yardstick's arithmetic. Normal rung: shipped on one type — added 2026-10-31 at M166's review - [low] Pin the byte figures in parallel.R: three `R/parallel.R` comments state wire sizes with no commit, date or procedure beside them: 5,141,166 B against 840,540 B for six split copies, 202,363 B against 524 B for the worker closure under `load_all()`, and 2,627 B for the 106-name manifest. The derived-figures rule asks for one. They predate M161, whose review found them — added 2026-10-24 at M161's review - [parked] Point oracle headers at the registry: 21 test-file headers say "DESIGN Conventions: oracles are recorded in the test file that asserts them", and `test-parallel-payload.R` and `test-nested-resamples-memory.R` say the records sit beside the test, but since M163 each oracle's id and type live in `cairn/ORACLES.md`. Promote on the next change to those headers — added 2026-10-31 at M163's review - [parked] Find why the shared pool loses a daemon: `test-parallel-identity.R`'s shared 2-daemon pool read 1 connection where 2 were started in a quiet full run at `9e45aaea` and a side-by-side run at `eb1ff5ac` (2026-10-24), and `shared_daemons(2)` then failed 10 expectations in four blocks (":241, :312, :357, :414"). A full run alone at `3488fffc` failed 11 blocks of that file with no messages kept. CI never showed it. M162 (dropped 2026-10-31) saw 0 of 40 trials lose a daemon at `dd6b5f7b`, on the file alone and the whole suite, beside 0 to 16 R processes spinning on arithmetic. Two suites running at once was not tried. Its probe, `benchmarks/probe-shared-pool-loss.R`, is on the unpushed local branch `m162-shared-pool-daemon-loss` at `2c0b02a7`. Promote when the loss recurs in CI, or in a local run whose failure output is kept — added 2026-10-24 at M159's T5, extended at M160's review, parked 2026-10-31 when M162 was dropped @@ -44,7 +45,7 @@ _Last hygiene check: 2026-10-31 M165 review. Graduated Guard the devel-vctrs fil - [parked] Inner designs M140 leaves out: added 2026-09-21 (M108), cut 2026-09-27 (M120, M121) and 2026-10-01 (M140). Every function but `nested_resamples()` and grid on the eight sliding pairs other than sliding-window / sliding-window. On that pair, `nested_final_fit()` for the four other tuners' results and `nested_workflow_map()` with a tuner `fn` (D-113). Under an outer `rolling_origin()` with an inner sliding design, the four other tuners, their final fits and the map with a tuner `fn`; and the grid final fit on the three sliding outer designs with an inner `rolling_origin()` (D-118, widened 2026-10-21 at M151). Any inner design other than `rolling_origin()` and the three sliding designs. Promote on a user running one - [parked] Lift the desirability selection limits: Desirability selection under the two racing tuners and simulated annealing, and on censored regression models, all of which refuse the rule after M109. Added 2026-09-21 at M109's plan gate. Racing drops candidates before the end, so the rule needs its own oracle there, and desirability2 ranks every evaluation time on a censored model. The row also holds M109's deferred review findings, which its archive lists. Promote when desirability2 documents its selector for racing results, or on a user asking - [parked] Nested tuning of tidyclust models: RR10 owns the analysis. `tune::last_fit()` refuses a cluster workflow, sum-of-squares metrics pick the grid's largest k, and only they have a closed-form oracle. Suggests takes a D-entry. Promote on a released `last_fit()` that scores a cluster workflow, plus a second oracle type for a metric whose selection is not degenerate — added 2026-09-21, M106-M109 plan gate, narrowed 2026-10-18 by a trivial edit -- [parked] M126 review leftovers on averaging: findings 3, 4 and 8, which `milestones/archive/M126-averaging-edge-cases.md` owns: gaps in what `benchmarks/averaging-speed.R` compares, a censored row of 0-row entries averaging to NULL where the help names only NULL entries, and the partial-run warning coming before a refusal in `collect_predictions(summarize = TRUE)` but after it in `augment()`. Promote on a user report of a 0-row entry, a median that disagrees with `stats::median()`, or the warning order misleading a reader — added 2026-09-28 at M126's review gate. Extended 2026-10-31 at M166's plan: the censored average tests hold each row out twice, so the median and the mean of `.pred_time` agree and neither oracle tells them apart +- [parked] M126 review leftovers on averaging: findings 3, 4 and 8, which `milestones/archive/M126-averaging-edge-cases.md` owns: gaps in what `benchmarks/averaging-speed.R` compares, a censored row of 0-row entries averaging to NULL where the help names only NULL entries, and the partial-run warning coming before a refusal in `collect_predictions(summarize = TRUE)` but after it in `augment()`. Promote on a user report of a 0-row entry, a median that disagrees with `stats::median()`, or the warning order misleading a reader — added 2026-09-28 at M126's review gate - [parked] DESCRIPTION quoting test word rules: `test-description.R` reads `tune_grid()` and `rsample::` as bare names, flags Suggests names that are common words (`survival`, `censored`), and checks only the next character on each side of a word. Added 2026-10-17 at M146's review (diff-bug #1, #2, #4). Promote when a Description edit trips one - [parked] CI job at the testthat floor: A CI job that runs the suite at the testthat floor in `DESCRIPTION`. Added 2026-09-29 at M130's plan gate, which chose one-time check runs at the floor over this job (3.3.0 since the re-plan, corrected 2026-09-29). The 3.0.0 floor went stale because tests adopted newer testthat functions and no job ran the floor. Promote on a test adopting a testthat function newer than the floor unnoticed - [parked] Move macOS CI off P3M binaries: the macOS leg of `R-CMD-check.yaml` takes P3M binaries, with `gower?source` for P3M's OpenMP-linked `gower`. zstd no longer blocks it (no zstd CRAN macOS binary in the closure on 2026-10-20; pkgdepends#485 is open). M150 tried CRAN binaries and was dropped: the check step took 29m50s and 26m29s against a 30-minute cap (`milestones/archive/M150-macos-cran-binaries.md`). Promote when a P3M macOS binary breaks as `gower` did, or on evidence the CRAN-binary leg runs well under its cap — added 2026-09-21, rewritten 2026-10-20 diff --git a/cairn/milestones/M166-finer-second-oracle.md b/cairn/milestones/M166-finer-second-oracle.md index 7651f427..42886f95 100644 --- a/cairn/milestones/M166-finer-second-oracle.md +++ b/cairn/milestones/M166-finer-second-oracle.md @@ -15,9 +15,9 @@ Back each result that the candidate row "Give finer results a second oracle" nam ## Scope -**In:** three new tests. One covers the final fit under a two-element `eval_time`, one covers resample weights with a left-out fold, and one covers the desirability rule by its definitions. One assertion joins the selection-metric test. A source note covers desirability2's help pages. `cairn/ORACLES.md` gains rows for the new oracles and for two assertions already in the suite. It gains one line for each result that owes no oracle of its own, and a D-entry records that rule. +**In:** five new tests. They cover the final fit under a two-element `eval_time`, resample weights with a left-out fold, the desirability rule by its definitions, renormalized probability averages, and the censored median over three repeats. One assertion joins the selection-metric test. A source note covers desirability2's help pages. `cairn/ORACLES.md` gains rows for the new oracles and for one assertion already in the suite. It gains one line for each of five results, naming the parts left on one type. A D-entry records the rule those lines apply. (Amended 2026-10-31 at review pass 1, the user's stop answer.) -**Out:** the averaging tests hold each censored row out twice, so the median and the mean of `.pred_time` agree there. That gap goes to the parked row "M126 review leftovers on averaging". Pointing the 21 test-file headers at the registry goes to its parked row. No change to package code. +**Out:** the parts left on one type (the survival branch on a new metric set, the eliminated candidates' rows, the final fit under `event_level`) → a new candidate row "Add second oracles to one-type parts". Pointing the 21 test-file headers at the registry goes to its parked row. No change to package code. ## Acceptance criteria @@ -25,9 +25,10 @@ Back each result that the candidate row "Give finer results a second oracle" nam - [ ] AC2: `test-resample-weights.R` gains a test on the weighted design with one failed fold, and on the same design with one fold scoring NA on one metric. In each case it asserts that a new weight on the left-out fold leaves each affected summary row's `mean`, `std_err` and `n` identical. It also asserts that a new weight on a scored fold changes that row's `mean`. - [ ] AC3: `test-nested-tune-grid-oracles.R` gains a test on one grid run under a desirability rule. Its expected selection for each fold is computed by explicit code from the definitions in desirability2's help pages, on that fold's inner metrics. The test asserts that each fold's recorded selection equals that candidate. A source note in `cairn/references/` records those definitions and desirability2's version. - [ ] AC4: the test `"a grid run records the first metric of the set it was given, in either order"` in `test-selection-metric.R` gains one assertion. For each of its two metric sets, the final fit's recorded first metric equals `names(attr(ms, "metrics"))[[1L]]`. -- [ ] AC5: `cairn/ORACLES.md` holds one oracle row for each test that AC1, AC2 and AC3 add. It holds one invariant row for the level exchange that `"nested_final_fit() tunes under the caller's event level"` in `test-event-level.R` asserts. It holds one closed-form row for the first-name assertions of the test that AC4 names. Each row gives id, type, source and its test quoted by name. The memory group counts `nested-resamples-memory/O2` for the size slope. -- [ ] AC6: `cairn/ORACLES.md` carries one line for each of four results. The first is `compute_metrics()` on the run's own and on a new metric set. The others are the averages that only `hand_average()` checks, the racing fold record, and the Bayesian and annealing iteration stage. Each line names the rows that back the result and why it owes no oracle of its own. A new `cairn/DECISIONS.md` entry records the rule that those lines apply. `grep -n 'Give finer results a second oracle' cairn/ORACLES.md` returns no line. -- [ ] AC7: `devtools::test()` reports no failures, and `air format --check .` is clean. +- [ ] AC5: `cairn/ORACLES.md` holds one oracle row for each test that AC1, AC2, AC3 and AC7 add. It holds one closed-form row for the first-name assertions of the test that AC4 names. Each row gives id, type, source and its test quoted by name. `grep -n 'event-level/O3' cairn/ORACLES.md` returns no line. The header of `test-event-level.R` says again that the file does not test the exchange of `sens` and `spec`. The memory group counts `nested-resamples-memory/O2` for the size slope. +- [ ] AC6: `cairn/ORACLES.md` carries one line for each of five results. The `compute_metrics()` line names the survival branch on a new metric set. The averages line names the renormalized probabilities and the median of `.pred_time`. The racing line names the rows of eliminated candidates. The line on the final fit under `event_level` names the level exchange. The iteration-stage line, in the Bayesian and in the annealing group, names the proposals and the perturbations. For each named part, the line names the registry rows it credits for that part. The lines for the survival branch, the eliminated candidates' rows, the level exchange, the proposals and the perturbations each name the candidate row that holds that part, or say why it owes no second type. A new `cairn/DECISIONS.md` entry records the rule that those lines apply. `grep -n 'Give finer results a second oracle' cairn/ORACLES.md` returns no line. +- [ ] AC7: `test-collect-predictions-summarize.R` gains two tests. The first uses the probability grid run with one `.pred_event` value planted as missing. It asserts that each row's averaged class probabilities sum to 1 within 1e-12. It also asserts that the planted row's probability means, before renormalization, do not sum to 1. The second runs the censored fixed workflow on a repeated v-fold outer design with three repeats. It asserts that `collect_predictions(summarize = TRUE)` matches tune's average on `tune::fit_resamples()` over the same outer splits, through `expect_matches_tune()`. It asserts that the same average matches `hand_average()` through `expect_matches_hand()`. It also asserts that for some row, the median of its three `.pred_time` values differs from their mean. +- [ ] AC8: `devtools::test()` reports no failures, and `air format --check .` is clean. ## Coverage @@ -35,9 +36,10 @@ Back each result that the candidate row "Give finer results a second oracle" nam - AC2 → T2 - AC3 → T3 - AC4 → T4 -- AC5 → T5 -- AC6 → T5 -- AC7 → T6 +- AC5 → T5, T8 +- AC6 → T5, T8 +- AC7 → T7 +- AC8 → T6, T10 ## Tasks @@ -47,6 +49,10 @@ Back each result that the candidate row "Give finer results a second oracle" nam - [x] T4: Selection metric. Add the assertion to the test at `test-selection-metric.R:15`. Plant: compare with the set's last name, and see it fail. - [x] T5: Registry and decision. Add the AC5 rows and the AC6 lines under the file's quoting rule. Rewrite the selection group's paragraph and the header paragraph that point at the candidate row. Append the D-entry. Re-point `helper-time-budget.R` rows for the new blocks in one pass. - [x] T6: Run `devtools::test()` and `air format --check .` clean. +- [x] T7: Averages. In `test-collect-predictions-summarize.R`, a sums-to-one invariant on `grid_prob_run()` with the planted missing value, and a three-repeat censored `fixed_pair()` checked with `expect_matches_tune()` and `expect_matches_hand()`. Plant: `mean_by` for `.pred_time`, and no renormalization. +- [x] T8: Registry rework. Drop `event-level/O3`, restore M152's sentence in `test-event-level.R`'s header, add the AC7 rows, and rewrite the five lines per AC6. Rewrite D-123 to the narrowed rule, naming the plugin's "What gets a test" rule where it is cited. Add the candidate row the Out item names. Remove the median gap from "M126 review leftovers on averaging", which AC7 closes. Carry the review return's fix-now findings. +- [x] T9: Weights failed case. Plant a missing renormalization over the stored weights in `summarize_folds()` and see the failed-fold case of the AC2 test fail; restore. +- [ ] T10: Run `devtools::test()` and `air format --check .` clean, alone. ## Work log @@ -80,6 +86,12 @@ Back each result that the candidate row "Give finer results a second oracle" nam - 2026-10-31: re-audit: AC6 (reduced) — "the rows that reach it" and "parts that rest on one type" are still decided by judgment; suggested crediting rows by the line's own naming and naming the one-type parts outright. Second line for AC6, so the wording goes to the user. - 2026-10-31: re-audit: AC7 (reduced) — nothing. - 2026-10-31: T7 (pending the amendment's write): `test-collect-predictions-summarize.R` gained "a renormalized probability average sums to one in every row" and "a censored average over three repeats takes the median time, as tune does"; `repeated_pair()` and `fixed_pair()` take `repeats` (default 2, injected so the stored call stays literal). File green, 34 tests. Plants in `R/nested-results-collect.R`: `mean_by` for `.pred_time` failed the three-repeat test (2 expectations) and the missing-median test; no renormalization failed the sums test and the hand renormalization test; both reverted, `git diff -- R/` empty. +- 2026-10-31: re-audit stop answer: the user chose to name the parts in AC6, crediting rows by each line's own naming. +- 2026-10-31: amendment return: AC5 — "It holds one closed-form row for the first-name assertions of the test that AC4 names. Each row gives id, type, source and its test quoted by name. `grep -n 'event-level/O3' cairn/ORACLES.md` returns no line. The header of `test-event-level.R` says again that the file does not test the exchange of `sens` and `spec`." +- 2026-10-31: amendment return: AC6 — "For each named part, the line names the registry rows it credits for that part. The lines for the survival branch, the eliminated candidates' rows, the level exchange, the proposals and the perturbations each name the candidate row that holds that part, or say why it owes no second type." +- 2026-10-31: substantive amendment: AC7 added (two averaging tests, the user's stop answer); the old AC7 is now AC8; Scope In and Out, Coverage and T7-T10 updated. +- 2026-10-31: T8: `cairn/ORACLES.md` dropped `event-level/O3` and gained `collect-predictions-summarize/O1` (invariant) and `/O2` (live), the three-repeat test listed under U2 as well. The five AC6 lines now credit rows per part: compute_metrics survival branch on U1, averages' renormalization on O1 and U2 and median on O2 and U2, racing eliminated rows on O1 and O3, the event_level final fit on O2, the iteration stage on O1 with no independent oracle. Headings for weights and averages read 3 types; memory slope is a group sentence; the 2026-09-01 cite gave way to D-123; the header counts six rows. D-123 rewritten on the branch to the narrowed rule, so the "testing rule" cite went. New candidate row "Add second oracles to one-type parts"; the median gap left "M126 review leftovers on averaging". `test-event-level.R`'s header already carried M152's sentence, unchanged on the branch. Selection-metric header names both oracles; the source note gained the `geomean()` sentence. +- 2026-10-31: T9: plant `sum(e$vals * e$w)` (no renormalization) in `summarize_folds()` failed the AC2 test's failed-fold case (rmse 0.525 against 1.051) and NA case; reverted. - 2026-10-31: review return 1: fix-now findings to carry in the implement pass: diff-bug #4 (show the failed-fold weights case failing), #8, #12; blame-history #6, #9; prior-review #2, #3, #5; ripple #6. ## Decisions diff --git a/cairn/references/desirability2-select-best.md b/cairn/references/desirability2-select-best.md index 31395eb5..38756d3b 100644 --- a/cairn/references/desirability2-select-best.md +++ b/cairn/references/desirability2-select-best.md @@ -14,7 +14,7 @@ Extraction: read directly 2026-10-31 from desirability2 0.2.0's installed help p - "If `low`, `high`, or `target` are not specified, the observed data are used to estimate their values" (same page, Details). The page's example sets `low` and `high` to the observed minimum and maximum of the column. - Maximization: d = 1 above `high`, 0 below `low`, and ((data - low) / (high - low))^scale between them (`inline_desirability.Rd`, Equations). `scale` defaults to 1. - Minimization: d = 0 above `high` and 1 below `low` (same page). Between them the page prints the fraction as "data = low" over "low - high", a typo. The body of `.comp_min` computes ((x - high) / (low - high))^scale. That is the one reading that gives 1 at `low` and 0 at `high`. -- `d_overall()` takes the geometric mean by default (`d_overall.Rd`, Usage, `geometric = TRUE`). +- `d_overall()` takes the geometric mean by default (`d_overall.Rd`, Usage, `geometric = TRUE`). If a term is 0, its internal `geomean()` returns 0. Otherwise it returns `exp(mean(log(x)))`, which can differ from `prod(x)^(1/k)` in the last bits. Only an exact tie at the top exposes that difference. - `select_best_desirability()` "returns a single row" (`show_best_desirability.Rd`, Value). The page also says that the ranking can hold "more than one model configuration that has identical overall desirability" (Details). The pages do not say which tied row comes first. ## Traces to diff --git a/tests/testthat/test-selection-metric.R b/tests/testthat/test-selection-metric.R index f3159ca6..ce1c18b2 100644 --- a/tests/testthat/test-selection-metric.R +++ b/tests/testthat/test-selection-metric.R @@ -1,9 +1,9 @@ # The record's `first_metric` entry: the name of the first metric in the -# set, as tune resolves it for the workflow. The oracle is tune's own reading +# set, as tune resolves it for the workflow. One oracle is tune's own reading # of the final fit's stored tuning run, `tune::.get_tune_metric_names()`, the # call every fold and the final fit select on. It is compared with the entry -# on both the results object and the final fit built from it. The grid test -# also compares both entries with the first name of the set it passed. +# on both the results object and the final fit built from it. The other, in +# the grid test, is the first name of the set the test passed. skip_heavy_on_cran() From 417a8d175e237ab2ded8d4944062da8dc48dee81 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 15:44:45 +0100 Subject: [PATCH 11/16] M166: implement pass 2 done, status review --- cairn/ROADMAP.md | 2 +- cairn/milestones/M166-finer-second-oracle.md | 7 +++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/cairn/ROADMAP.md b/cairn/ROADMAP.md index d06eb470..5822fa46 100644 --- a/cairn/ROADMAP.md +++ b/cairn/ROADMAP.md @@ -4,7 +4,7 @@ _Last hygiene check: 2026-10-31 M165 review. Graduated Guard the devel-vctrs fil ## Milestones | ID | Title | Status | Depends on | Priority | File/Archive | |---|---|---|---|---|---| -| M166 | Give finer results a second oracle | in-progress | — | normal | milestones/M166-finer-second-oracle.md | +| M166 | Give finer results a second oracle | review | — | normal | milestones/M166-finer-second-oracle.md | | M165 | Keep the devel-vctrs test choice from drifting | done | — | normal | milestones/archive/M165-devel-vctrs-drift.md | | M164 | Give six one-type results a second oracle | done | — | normal | milestones/archive/M164-second-oracle-types.md | | M163 | Keep an oracle registry outside tests | done | — | normal | milestones/archive/M163-oracle-registry.md | diff --git a/cairn/milestones/M166-finer-second-oracle.md b/cairn/milestones/M166-finer-second-oracle.md index 42886f95..be18b5c7 100644 --- a/cairn/milestones/M166-finer-second-oracle.md +++ b/cairn/milestones/M166-finer-second-oracle.md @@ -1,6 +1,6 @@ # M166: Give finer results a second oracle -- **Status:** in-progress +- **Status:** review - **Priority:** normal - **Depends on:** — - **Driving RR:** — @@ -52,7 +52,7 @@ Back each result that the candidate row "Give finer results a second oracle" nam - [x] T7: Averages. In `test-collect-predictions-summarize.R`, a sums-to-one invariant on `grid_prob_run()` with the planted missing value, and a three-repeat censored `fixed_pair()` checked with `expect_matches_tune()` and `expect_matches_hand()`. Plant: `mean_by` for `.pred_time`, and no renormalization. - [x] T8: Registry rework. Drop `event-level/O3`, restore M152's sentence in `test-event-level.R`'s header, add the AC7 rows, and rewrite the five lines per AC6. Rewrite D-123 to the narrowed rule, naming the plugin's "What gets a test" rule where it is cited. Add the candidate row the Out item names. Remove the median gap from "M126 review leftovers on averaging", which AC7 closes. Carry the review return's fix-now findings. - [x] T9: Weights failed case. Plant a missing renormalization over the stored weights in `summarize_folds()` and see the failed-fold case of the AC2 test fail; restore. -- [ ] T10: Run `devtools::test()` and `air format --check .` clean, alone. +- [x] T10: Run `devtools::test()` and `air format --check .` clean, alone. ## Work log @@ -92,6 +92,9 @@ Back each result that the candidate row "Give finer results a second oracle" nam - 2026-10-31: substantive amendment: AC7 added (two averaging tests, the user's stop answer); the old AC7 is now AC8; Scope In and Out, Coverage and T7-T10 updated. - 2026-10-31: T8: `cairn/ORACLES.md` dropped `event-level/O3` and gained `collect-predictions-summarize/O1` (invariant) and `/O2` (live), the three-repeat test listed under U2 as well. The five AC6 lines now credit rows per part: compute_metrics survival branch on U1, averages' renormalization on O1 and U2 and median on O2 and U2, racing eliminated rows on O1 and O3, the event_level final fit on O2, the iteration stage on O1 with no independent oracle. Headings for weights and averages read 3 types; memory slope is a group sentence; the 2026-09-01 cite gave way to D-123; the header counts six rows. D-123 rewritten on the branch to the narrowed rule, so the "testing rule" cite went. New candidate row "Add second oracles to one-type parts"; the median gap left "M126 review leftovers on averaging". `test-event-level.R`'s header already carried M152's sentence, unchanged on the branch. Selection-metric header names both oracles; the source note gained the `geomean()` sentence. - 2026-10-31: T9: plant `sum(e$vals * e$w)` (no renormalization) in `summarize_folds()` failed the AC2 test's failed-fold case (rmse 0.525 against 1.051) and NA case; reverted. +- 2026-10-31: T10: `air format --check .` clean; `devtools::test()` under `NESTEDTUNE_FULL_SUITE=true`, alone: 1164 tests, 0 failed, 0 errors, 0 skipped. +- 2026-10-31: claim audit: not owed — internal tier +- 2026-10-31: implement pass 2 done; status review. - 2026-10-31: review return 1: fix-now findings to carry in the implement pass: diff-bug #4 (show the failed-fold weights case failing), #8, #12; blame-history #6, #9; prior-review #2, #3, #5; ripple #6. ## Decisions From 6a50ea28c848df7822ceacc4636bb22e1a5c6f0f Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 15:52:06 +0100 Subject: [PATCH 12/16] M166 review pass 2: registry credits narrowed (checkpoint; test tweaks pending) --- cairn/DECISIONS.md | 2 +- cairn/ORACLES.md | 12 ++++++------ cairn/ROADMAP.md | 2 +- tests/testthat/test-collect-predictions-summarize.R | 3 ++- 4 files changed, 10 insertions(+), 9 deletions(-) diff --git a/cairn/DECISIONS.md b/cairn/DECISIONS.md index 44be957f..f38a443a 100644 --- a/cairn/DECISIONS.md +++ b/cairn/DECISIONS.md @@ -2141,7 +2141,7 @@ upstream, either of which would let the file return to a shared blob. **Decision:** on a push or pull request the test step runs the `tests/testthat/test-*.R` files that carry the line `# devel-vctrs.yaml runs this file on every push and pull request.` as a whole line, and stops with an error when none does. The step decides the whole-suite switch from `GITHUB_EVENT_NAME` and `inputs.full` alone, and the concurrency group is keyed on those two raw values with the workflow and ref, so two runs in one group make the same whole-suite choice. On a subset run each commit's markers decide the files. Rejected at the plan gate: a `-compat` file-name convention, which still drops a file renamed out of it. **Consequences:** a test file joins the push-time watch by taking the marker line, replacing D-119's "adds itself to the filter". A scheduled run and a full dispatch no longer share a group, nor do a `full` false dispatch on the default branch and a push to it, so each pair can run side by side. Falsified by a watched file losing its marker in an edit that no run flags, or by a subset run cancelling or replacing a whole-suite run. (Narrows D-119.) -### D-123 (2026-10-31): a part of a result is credited only to the oracle rows whose tests reach it with input that can show a defect there, a part left on one type is recorded in its group with the row that holds its second type, and the Bayesian and annealing iteration stage owes none. Extends D-121 +### D-123 (2026-10-31): a part of a result is credited only to the oracle rows whose tests reach it with input that can show a defect there, a part left on one type is recorded in its group with the candidate row that holds its second type, and the Bayesian and annealing iteration stage owes none. Extends D-121, and exempts the iteration stage from GP2's second type **Context:** M163's review cut the `cairn/ORACLES.md` groups finer and found results that rest on one oracle type. M164 left nine of them on a candidate row. M166's plan gate asked what to do with a result where no second type can be built at low cost, and the user chose to record it with its reason. M166's first review found that crediting a part to every row whose test runs the same code was false: a fixture can run a branch with input that cannot show a defect in it. Examples are a probability average whose rows already sum to one, and a median over two values. **Decision:** (1) A part of a result is credited only to the rows whose tests reach it with input that can show a defect there. (2) A part left on one type is named in its group's line, with the candidate row that holds its second type. (3) The Bayesian and annealing iteration stage keeps one correctness type, the reference loop, beside a serial-against-parallel check. No independent oracle exists for tune's Gaussian-process proposals or finetune's perturbations, so they owe no second type. The user chose this at M166's review stop over keeping the same-code reading. diff --git a/cairn/ORACLES.md b/cairn/ORACLES.md index a0bd0a6c..e23c5716 100644 --- a/cairn/ORACLES.md +++ b/cairn/ORACLES.md @@ -11,7 +11,7 @@ An id is the test file's name without `test-` and `.R`, then the record's own la Each test line gives a file under `tests/testthat/` and the first argument of a `test_that()` call in that file at the milestone's head. A literal name is in double quotes. For a name built in code, the line gives `code` and the one-line deparsed call. M163 mapped the names from `576c3d19` through `benchmarks/test-renames-576c3d19.csv`. Where a later commit merged tests, as six workflow-map tests became one `paste0()` loop, M163 read the file at the head. A test line ending "(found by reading)" is a test that the record does not name, found in the file's code. If a record's text quotes a test's name, the record names that test. In the quote, `...` or text in angle brackets matches any text. The full rule is M163's AC1, in the git history of `cairn/milestones/M163-oracle-registry.md`. A prose record names no test. Where a string quoted after "Pinned by" names no test, the row says so. -Each group collects rows that test one result, as this file cuts the results. The groups are one cut, and a finer cut can find more results backed by one type. M166 checked the nine that M163's and M164's reviews found. Each gained a row of a second type, or its group carries a line that names the part left on one type. The line says where that part's second type is held, or why it has none (D-123). Rows for time-series designs join their function's group, because the design is an input to the same result. D-079 gives that reason for the Bayesian, racing and annealing tuners: the outer design never reaches tuner code. For the other functions, and for inner designs, pairs and the splits group, the grouping is this file's choice. Each row's source names its design. A row that tests two results is written once, and the second group counts it by id. +Each group collects rows that test one result, as this file cuts the results. The groups are one cut, and a finer cut can find more results backed by one type. M166 checked the nine that M163's and M164's reviews found. Each gained a row of a second type or is counted on an existing row. Otherwise its group carries a line that names the part left on one type. The line says where that part's second type is held, or why it has none (D-123). Rows for time-series designs join their function's group, because the design is an input to the same result. D-079 gives that reason for the Bayesian, racing and annealing tuners: the outer design never reaches tuner code. For the other functions, and for inner designs, pairs and the splits group, the grouping is this file's choice. Each row's source names its design. A row that tests two results is written once, and the second group counts it by id. ## nested_tune_grid() estimate: live, invariant (2 types) @@ -67,7 +67,7 @@ M164 added the closed-form row, which covers tune's three rules. M166 added clos ## nested_tune_bayes() estimate: live, invariant (2 types) -The records split this result in two. At the initial stage, O1 and O2 are the two types. At the iteration stage only O1 and O3 apply. O3 is a serial-against-parallel identity, which test-parallel-identity.R calls a mode-independence check, not a correctness check. The proposals are credited to O1 alone. They are tune's own Gaussian-process search, and no independent oracle for them exists, so they owe no second type (D-123). +The records split this result in two. At the initial stage, O1 and O2 are the two types. At the iteration stage only O1 and O3 apply. O3 is a serial-against-parallel identity, which test-parallel-identity.R calls a mode-independence check, not a correctness check. The proposals are credited to no row. O1's reference loop runs tune's own Gaussian-process search, so it checks how the package drives the search and not the proposals. No independent oracle for them exists, so they owe no second type (D-123). - `nested-tune-bayes-oracles/O1` live. Source: `reference_nested_bayes_loop()` in helper-orchestration.R, written from the seed contract. - test-nested-tune-bayes-oracles.R :: "per-fold metrics and selections match a hand-rolled Bayesian reference loop" @@ -87,7 +87,7 @@ The records split this result in two. At the initial stage, O1 and O2 are the tw ## nested_tune_race_anova() and nested_tune_race_win_loss() estimate: live, invariant (2 types) -The fold record holds every candidate the race scored. O2 checks the survivors' rows against the grid path. The rows of eliminated candidates are credited to O1 and O3, both live, since O2 reaches no eliminated candidate. The candidate row "Add second oracles to one-type parts" holds their second type (D-123). +The fold record holds every candidate the race scored. O2 checks the survivors' rows against the grid path. The rows of eliminated candidates are credited to O3 alone, whose test asserts that some candidate was eliminated. O2 reaches no eliminated candidate. The candidate row "Add second oracles to one-type parts" holds their second type (D-123). - `nested-tune-race-oracles/O1` live. Source: `reference_nested_race_loop()` in helper-orchestration.R, written from the seed contract. The string its record quotes after "Pinned by" names no test: the test's name adds "(deterministic)". - test-nested-tune-race-oracles.R :: "per-fold metrics, selections and inner tables match a hand-rolled racing reference loop (deterministic)" (found by reading) @@ -104,7 +104,7 @@ The fold record holds every candidate the race scored. O2 checks the survivors' ## nested_tune_sim_anneal() estimate: live, invariant (2 types) -The records split this result in two. At the initial stage, O1 and O2 are the two types. At the iteration stage only O1 and O3 apply. O3 is a serial-against-parallel identity, which test-parallel-identity.R calls a mode-independence check, not a correctness check. The perturbations are credited to O1 alone. They are finetune's own search, and no independent oracle for them exists, so they owe no second type (D-123). +The records split this result in two. At the initial stage, O1 and O2 are the two types. At the iteration stage only O1 and O3 apply. O3 is a serial-against-parallel identity, which test-parallel-identity.R calls a mode-independence check, not a correctness check. The perturbations are credited to no row. O1's reference loop runs finetune's own search, so it checks how the package drives the search and not the perturbations. No independent oracle for them exists, so they owe no second type (D-123). - `nested-tune-sim-anneal-oracles/O1` live. Source: `reference_nested_anneal_loop()` in helper-orchestration.R, written from the seed contract. The string its record quotes after "Pinned by" names no test: the test's name adds "(deterministic)". - test-nested-tune-sim-anneal-oracles.R :: "per-fold metrics, selections and inner tables match a hand-rolled annealing reference loop (deterministic)" (found by reading) @@ -237,7 +237,7 @@ The records split this result in two. At the initial stage, O1 and O2 are the tw ## Event level on classification models: closed-form, live (2 types) -The final fit under `event_level` is credited to `event-level/O2` alone. The level exchange that its test also asserts is yardstick's arithmetic, so M152 removed its record and it backs no part here. The candidate row "Add second oracles to one-type parts" holds the final fit's second type (D-123). +The final fit under `event_level` is credited to `event-level/O2` alone. M152 removed the record of the level exchange as yardstick's arithmetic, and it backs no part here. The candidate row "Add second oracles to one-type parts" holds the final fit's second type (D-123). - `event-level/O1` closed-form. Source: the definition of sensitivity, recomputed from a refit of each fold's selected candidate. - test-event-level.R :: "the sensitivity each outer fold reports is the one yardstick computes at that event level" @@ -308,7 +308,7 @@ Two parts of the average need input that U1's fixtures lack. The renormalized pr ## compute_metrics() rescoring: invariant, live (2 types) -`compute_metrics()` runs the same scoring code, `score_fold()`, on the run's own metric set and on a new one. U1 and U2 both reach its numeric and class-probability branches. The survival branch on a new metric set is credited to U1 alone, since U2 rescores no censored run. The candidate row "Add second oracles to one-type parts" holds its second type (D-123). +`compute_metrics()` runs the same scoring code, `score_fold()`, on the run's own metric set and on a new one. U1 and U2 both reach its numeric and class-probability branches. In the survival branch, dynamic metrics are credited to U1 alone, since U2 rescores no censored run. Static and linear-predictor survival metrics are credited to no row, since U1's censored test scores `brier_survival` only. The candidate row "Add second oracles to one-type parts" holds the survival branch (D-123). - `compute-metrics/U1` invariant. Source: `collect_metrics()` on the same run, which reads the `.metrics` that `tune::last_fit()` wrote. - test-compute-metrics.R :: "with the run's metric set, compute_metrics() is collect_metrics() on a regression run" (found by reading) diff --git a/cairn/ROADMAP.md b/cairn/ROADMAP.md index 5822fa46..c411350e 100644 --- a/cairn/ROADMAP.md +++ b/cairn/ROADMAP.md @@ -14,7 +14,7 @@ _Last hygiene check: 2026-10-31 M165 review. Graduated Guard the devel-vctrs fil - Tidy the interrupt block's leftovers: four items in `test-parallel-interrupt.R` from M159's review, none added by M159. The ledger budgets no wait for the 60 s collect when no signal is sent. A SIGINT sent after `dispatch_folds()` returns early lands outside the block's handler. The header's "Established by execution against mirai 2.7.2 before the fix" is narrative history. The `m15-fold-` marker names carry a milestone id. Promote on the next change to that file — added 2026-10-24 at M159's review - Give finer results a second oracle: one-type results in two-type `cairn/ORACLES.md` groups (M163 review): `compute_metrics()` own vs new metric set, final fit under `eval_time` and `event_level`, failed-fold weights, race fold record, Bayesian and annealing iteration stage, memory slope, `hand_average()`-only cases. Also the selection group's desirability rule and selection metric (M164 Out). Normal rung: shipped on one type — added 2026-10-31 at M164's plan gate — started M166 -- Add second oracles to one-type parts: three parts that `cairn/ORACLES.md` credits to one type (D-123). `compute_metrics()`'s survival branch on a new metric set rests on U1. The racing record's rows for eliminated candidates rest on two live rows. The final fit under `event_level` rests on `event-level/O2`, since its level exchange is yardstick's arithmetic. Normal rung: shipped on one type — added 2026-10-31 at M166's review +- Add second oracles to one-type parts: three parts that `cairn/ORACLES.md` credits to one type or none (D-123). In `compute_metrics()`'s survival branch, dynamic metrics rest on U1, and static and linear-predictor metrics reach no test. The racing record's rows for eliminated candidates rest on one live row, O3. The final fit under `event_level` rests on `event-level/O2`, since its level exchange is yardstick's arithmetic. Normal rung: shipped on one type — added 2026-10-31 at M166's review - [low] Pin the byte figures in parallel.R: three `R/parallel.R` comments state wire sizes with no commit, date or procedure beside them: 5,141,166 B against 840,540 B for six split copies, 202,363 B against 524 B for the worker closure under `load_all()`, and 2,627 B for the 106-name manifest. The derived-figures rule asks for one. They predate M161, whose review found them — added 2026-10-24 at M161's review - [parked] Point oracle headers at the registry: 21 test-file headers say "DESIGN Conventions: oracles are recorded in the test file that asserts them", and `test-parallel-payload.R` and `test-nested-resamples-memory.R` say the records sit beside the test, but since M163 each oracle's id and type live in `cairn/ORACLES.md`. Promote on the next change to those headers — added 2026-10-31 at M163's review - [parked] Find why the shared pool loses a daemon: `test-parallel-identity.R`'s shared 2-daemon pool read 1 connection where 2 were started in a quiet full run at `9e45aaea` and a side-by-side run at `eb1ff5ac` (2026-10-24), and `shared_daemons(2)` then failed 10 expectations in four blocks (":241, :312, :357, :414"). A full run alone at `3488fffc` failed 11 blocks of that file with no messages kept. CI never showed it. M162 (dropped 2026-10-31) saw 0 of 40 trials lose a daemon at `dd6b5f7b`, on the file alone and the whole suite, beside 0 to 16 R processes spinning on arithmetic. Two suites running at once was not tried. Its probe, `benchmarks/probe-shared-pool-loss.R`, is on the unpushed local branch `m162-shared-pool-daemon-loss` at `2c0b02a7`. Promote when the loss recurs in CI, or in a local run whose failure output is kept — added 2026-10-24 at M159's T5, extended at M160's review, parked 2026-10-31 when M162 was dropped diff --git a/tests/testthat/test-collect-predictions-summarize.R b/tests/testthat/test-collect-predictions-summarize.R index cc8f1e43..e992f1db 100644 --- a/tests/testthat/test-collect-predictions-summarize.R +++ b/tests/testthat/test-collect-predictions-summarize.R @@ -7,7 +7,8 @@ # deterministic (lm, glm, survreg), because a stochastic engine's per-fold # predictions differ between the two runs. The second is `hand_average()` # below, base R written from the rules the help page states and read off -# `collect_predictions(res)`, never off the method's output. +# `collect_predictions(res)`, never off the method's output. The third is an +# invariant: averaged class probabilities sum to one in every row. skip_heavy_on_cran() From e1d3f35cf98a2a4ec3b162e921e735d6a4792c46 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 15:52:15 +0100 Subject: [PATCH 13/16] M166 review pass 2: drop the result count from the registry header --- cairn/ORACLES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cairn/ORACLES.md b/cairn/ORACLES.md index e23c5716..cf6d67ce 100644 --- a/cairn/ORACLES.md +++ b/cairn/ORACLES.md @@ -11,7 +11,7 @@ An id is the test file's name without `test-` and `.R`, then the record's own la Each test line gives a file under `tests/testthat/` and the first argument of a `test_that()` call in that file at the milestone's head. A literal name is in double quotes. For a name built in code, the line gives `code` and the one-line deparsed call. M163 mapped the names from `576c3d19` through `benchmarks/test-renames-576c3d19.csv`. Where a later commit merged tests, as six workflow-map tests became one `paste0()` loop, M163 read the file at the head. A test line ending "(found by reading)" is a test that the record does not name, found in the file's code. If a record's text quotes a test's name, the record names that test. In the quote, `...` or text in angle brackets matches any text. The full rule is M163's AC1, in the git history of `cairn/milestones/M163-oracle-registry.md`. A prose record names no test. Where a string quoted after "Pinned by" names no test, the row says so. -Each group collects rows that test one result, as this file cuts the results. The groups are one cut, and a finer cut can find more results backed by one type. M166 checked the nine that M163's and M164's reviews found. Each gained a row of a second type or is counted on an existing row. Otherwise its group carries a line that names the part left on one type. The line says where that part's second type is held, or why it has none (D-123). Rows for time-series designs join their function's group, because the design is an input to the same result. D-079 gives that reason for the Bayesian, racing and annealing tuners: the outer design never reaches tuner code. For the other functions, and for inner designs, pairs and the splits group, the grouping is this file's choice. Each row's source names its design. A row that tests two results is written once, and the second group counts it by id. +Each group collects rows that test one result, as this file cuts the results. The groups are one cut, and a finer cut can find more results backed by one type. M166 checked the finer results that M163's and M164's reviews found. Each gained a row of a second type or is counted on an existing row. Otherwise its group carries a line that names the part left on one type. The line says where that part's second type is held, or why it has none (D-123). Rows for time-series designs join their function's group, because the design is an input to the same result. D-079 gives that reason for the Bayesian, racing and annealing tuners: the outer design never reaches tuner code. For the other functions, and for inner designs, pairs and the splits group, the grouping is this file's choice. Each row's source names its design. A row that tests two results is written once, and the second group counts it by id. ## nested_tune_grid() estimate: live, invariant (2 types) From 22bc50c70a5e66c150faac27daac2ae05564e11b Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 16:02:49 +0100 Subject: [PATCH 14/16] M166 review pass 2: findings, gate evidence, two test tweaks --- cairn/milestones/M166-finer-second-oracle.md | 36 +++++++++++++++++++ .../test-collect-predictions-summarize.R | 15 ++++---- .../testthat/test-nested-tune-grid-oracles.R | 8 ++--- 3 files changed, 46 insertions(+), 13 deletions(-) diff --git a/cairn/milestones/M166-finer-second-oracle.md b/cairn/milestones/M166-finer-second-oracle.md index be18b5c7..28ff1e27 100644 --- a/cairn/milestones/M166-finer-second-oracle.md +++ b/cairn/milestones/M166-finer-second-oracle.md @@ -138,3 +138,39 @@ Back each result that the candidate row "Give finer results a second oracle" nam - ripple #4: the candidate row still lists the results — reject, planned change (graduated at step 9). - ripple #5: the new rows carry no "(found by reading)" mark — reject, false (rows added after `576c3d19` name their test in the row, as M164's do). - ripple #6: D-123 cites "the testing rule" without naming it — fix now (name the plugin's "What gets a test" rule, if the rule stays). +- Pass 1 fix-now lines settled in implement pass 2: diff-bug #4, fixed 7161e181 (T9 plant); diff-bug #8, #12, blame-history #6, #9, prior-review #2, #3, #5, fixed 7161e181; ripple #6, fixed 7161e181 (D-123 no longer cites the rule); the "at the stop" lines, fixed by the stop answer in a1af2737 and 7161e181. + +### Pass 2 + +- spawned: diff-bug, blame-history, prior-review, ripple +- cairn_validate: all checks passed. Gate at 417a8d17: `air format --check .` clean; `document()` changed no NAMESPACE or man file (it rewrites `Config/roxygen2/version` to the local 8.1.1, restored, not committed); `devtools::test()` under `NESTEDTUNE_FULL_SUITE=true`, alone: 1164 tests, 0 failed, 0 errors, 0 skipped; six gating prose sweeps exit 0; `devtools::check()` 0 errors, 0 warnings, 0 notes. After the pass-2 fixes, the two changed test files rerun green (34 and 10 tests, 0 failed) and `air` is clean. +- diff-bug #1: the compute_metrics survival branch credit to U1 is false for static and linear-predictor metrics, since U1's censored test scores `brier_survival` only — fix now, fixed 6a50ea28 (line and candidate row say they reach no test). +- diff-bug #2: O1's racing fixtures may eliminate no candidate, so crediting O1 counts one check twice — fix now, fixed 6a50ea28 (credited to O3 alone). +- diff-bug #3: the median guard's 1e-6 gap does not prove the comparisons can fail at `expect_equal`'s tolerance — fix now (guard on the column's relative gap above 1e-4, passing). +- diff-bug #4: the event-level line says the test "also asserts" the exchange while the header says the file does not test it — fix now, fixed 6a50ea28 (line cites M152's removal only). The header's mismatch with the assertions at :268 predates the diff — reject, unmodified lines. +- diff-bug #5: D-123 narrows D-121 and GP2 without saying so — fix now, fixed 6a50ea28 (heading names the GP2 exemption; D-121's owed second type stands, held by the candidate row). +- diff-bug #6: O1 runs tune's and finetune's own search, so it cannot show a defect in the proposals or perturbations — fix now, fixed 6a50ea28 (credited to no row; O1 checks the driving). +- diff-bug #7: the averaging header names no source for the sums-to-one invariant — fix now, fixed 6a50ea28. +- diff-bug #8: `collect-predictions-summarize/O2` shares U1's source on a new fixture — reject, planned change (AC5 asks one row per new test; the test's hand check is listed under U2, whose source it is). +- diff-bug #9: the AC4 assertion follows from three others — reject, planned change (as at pass 1). +- diff-bug #10: the three-repeat test makes the two-repeat censored tests repeats on one path — reject, unmodified lines (the older tests are outside the diff). +- diff-bug #11: the desirability tie check runs twice per fold — fix now (picks computed once). +- diff-bug #12: the eval_time `predict()` identity follows from the `dist` identity — reject, planned change (AC1 names it). +- diff-bug #13: the weights comment is near worked arithmetic — reject, false (as at pass 1). +- diff-bug #14: the registry header sentence does not describe the memory slope — fix now, fixed 6a50ea28 ("or is counted on an existing row"). +- diff-bug #15: D-123's heading says "row" where it means candidate row — fix now, fixed 6a50ea28. +- blame-history #1: the survival-branch credit on a new metric set rests on the same-code reading — fix now, fixed 6a50ea28 (same fix as diff-bug #1). +- blame-history #2: the memory slope is credited to O2 — reject, planned change (AC5 names it). +- blame-history #3: D-079's "stays two" count of selection oracles is stale — reject, history record. +- blame-history #4: the 2026-09-01 plan-gate cite is gone — reject, planned change (pass 1 prior-review #5 replaced it with D-123). +- blame-history #5: the racing rows credit two live rows — fix now, fixed 6a50ea28 (same fix as diff-bug #2). +- blame-history #6: the header's "nine" does not match the row's count of parts — fix now, fixed e1d3f35c (number dropped). +- blame-history #7: the tie check uses `prod()^(1/k)` where desirability2 uses `exp(mean(log))` — reject, false (the test asserts one top candidate, and the note records the difference). +- prior-review #1: the sums-to-one premise and the median guard add with `mean()`, against M126's double-arithmetic convention — fix now (the guard sums in double; the premise's 1e-6 margin is far above any long-double difference, reject for it, false). +- prior-review #2: the median guard uses `mean()` — fix now (same fix as #1). +- prior-review #3: no source comment for the sums-to-one invariant — fix now, fixed 6a50ea28 (same fix as diff-bug #7). +- prior-review #4: the O8 row does not name the page's typo — reject, false (as at pass 1; the row cites the note). +- ripple #1: U2 lists median tests that the averages line does not credit — reject, false (the line names U2's three-repeat test). +- ripple #2: the candidate row "Give finer results a second oracle" still lists the selection results — reject, planned change (graduated at step 9). +- ripple #3: the weights source sits away from its test — reject, false (it is in the header block at :28, the file's convention). +- ripple #4: "held out twice" at :793 is not true of the new three-repeat pair — reject, false (it describes the fixtures its helper reads). diff --git a/tests/testthat/test-collect-predictions-summarize.R b/tests/testthat/test-collect-predictions-summarize.R index e992f1db..96f595e2 100644 --- a/tests/testthat/test-collect-predictions-summarize.R +++ b/tests/testthat/test-collect-predictions-summarize.R @@ -341,14 +341,13 @@ test_that("a censored average over three repeats takes the median time, as tune hand_average(p, outcome), outcome ) - # Each row is held out three times, so a mean in place of the median - # would give another number on some row. - gap <- vapply( - split(p$.pred_time, p$.row), - function(t) stats::median(t) - mean(t), - numeric(1) - ) - expect_true(any(abs(gap) > 1e-6)) + # Each row is held out three times. A mean in place of the median would + # move the column by more than the comparisons above tolerate, which is + # a mean relative difference of about 1.5e-8. + by_row_time <- split(p$.pred_time, p$.row) + medians <- vapply(by_row_time, stats::median, numeric(1)) + means <- vapply(by_row_time, function(t) sum(t) / length(t), numeric(1)) + expect_gt(sum(abs(means - medians)) / sum(abs(medians)), 1e-4) }) # ---- across the candidates the folds selected ----------------------------- diff --git a/tests/testthat/test-nested-tune-grid-oracles.R b/tests/testthat/test-nested-tune-grid-oracles.R index 6f716ce5..ca2def15 100644 --- a/tests/testthat/test-nested-tune-grid-oracles.R +++ b/tests/testthat/test-nested-tune-grid-oracles.R @@ -536,11 +536,12 @@ test_that("the desirability rule picks the candidate its definition names in the c(df1 = rmse$df1[[top]], df2 = rmse$df2[[top]]) } + picks <- lapply(res$.inner_metrics, pick) for (i in seq_len(nrow(res))) { sel <- res$.selected[[i]] expect_identical( c(df1 = sel$df1, df2 = sel$df2), - pick(res$.inner_metrics[[i]]), + picks[[i]], info = paste("fold", i) ) } @@ -550,8 +551,5 @@ test_that("the desirability rule picks the candidate its definition names in the m <- m[m$.metric == "rmse", ] c(df1 = m$df1[[which.min(m$mean)]], df2 = m$df2[[which.min(m$mean)]]) }) - expect_false(identical( - lapply(seq_len(nrow(res)), function(i) pick(res$.inner_metrics[[i]])), - best - )) + expect_false(identical(picks, best)) }) From 1a06658b357b64c952fd4c61da82ccc56ee7c569 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 16:03:27 +0100 Subject: [PATCH 15/16] M166 review: criterion evidence (checkpoint) --- cairn/milestones/M166-finer-second-oracle.md | 25 +++++++++++++------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/cairn/milestones/M166-finer-second-oracle.md b/cairn/milestones/M166-finer-second-oracle.md index 28ff1e27..64bb7f2d 100644 --- a/cairn/milestones/M166-finer-second-oracle.md +++ b/cairn/milestones/M166-finer-second-oracle.md @@ -21,14 +21,14 @@ Back each result that the candidate row "Give finer results a second oracle" nam ## Acceptance criteria -- [ ] AC1: `test-eval-time.R` gains a test that runs `nested_tune_grid()` at each order of `srv_eval_times()` and at each order's first element alone. For each order, it asserts that `nested_final_fit()` on the two-element run selects the same candidate as on the one-element run. It also asserts that the two fits give the same predictions on the data. It also asserts that the two orders select different candidates. -- [ ] AC2: `test-resample-weights.R` gains a test on the weighted design with one failed fold, and on the same design with one fold scoring NA on one metric. In each case it asserts that a new weight on the left-out fold leaves each affected summary row's `mean`, `std_err` and `n` identical. It also asserts that a new weight on a scored fold changes that row's `mean`. -- [ ] AC3: `test-nested-tune-grid-oracles.R` gains a test on one grid run under a desirability rule. Its expected selection for each fold is computed by explicit code from the definitions in desirability2's help pages, on that fold's inner metrics. The test asserts that each fold's recorded selection equals that candidate. A source note in `cairn/references/` records those definitions and desirability2's version. -- [ ] AC4: the test `"a grid run records the first metric of the set it was given, in either order"` in `test-selection-metric.R` gains one assertion. For each of its two metric sets, the final fit's recorded first metric equals `names(attr(ms, "metrics"))[[1L]]`. -- [ ] AC5: `cairn/ORACLES.md` holds one oracle row for each test that AC1, AC2, AC3 and AC7 add. It holds one closed-form row for the first-name assertions of the test that AC4 names. Each row gives id, type, source and its test quoted by name. `grep -n 'event-level/O3' cairn/ORACLES.md` returns no line. The header of `test-event-level.R` says again that the file does not test the exchange of `sens` and `spec`. The memory group counts `nested-resamples-memory/O2` for the size slope. -- [ ] AC6: `cairn/ORACLES.md` carries one line for each of five results. The `compute_metrics()` line names the survival branch on a new metric set. The averages line names the renormalized probabilities and the median of `.pred_time`. The racing line names the rows of eliminated candidates. The line on the final fit under `event_level` names the level exchange. The iteration-stage line, in the Bayesian and in the annealing group, names the proposals and the perturbations. For each named part, the line names the registry rows it credits for that part. The lines for the survival branch, the eliminated candidates' rows, the level exchange, the proposals and the perturbations each name the candidate row that holds that part, or say why it owes no second type. A new `cairn/DECISIONS.md` entry records the rule that those lines apply. `grep -n 'Give finer results a second oracle' cairn/ORACLES.md` returns no line. -- [ ] AC7: `test-collect-predictions-summarize.R` gains two tests. The first uses the probability grid run with one `.pred_event` value planted as missing. It asserts that each row's averaged class probabilities sum to 1 within 1e-12. It also asserts that the planted row's probability means, before renormalization, do not sum to 1. The second runs the censored fixed workflow on a repeated v-fold outer design with three repeats. It asserts that `collect_predictions(summarize = TRUE)` matches tune's average on `tune::fit_resamples()` over the same outer splits, through `expect_matches_tune()`. It asserts that the same average matches `hand_average()` through `expect_matches_hand()`. It also asserts that for some row, the median of its three `.pred_time` values differs from their mean. -- [ ] AC8: `devtools::test()` reports no failures, and `air format --check .` is clean. +- [x] AC1: `test-eval-time.R` gains a test that runs `nested_tune_grid()` at each order of `srv_eval_times()` and at each order's first element alone. For each order, it asserts that `nested_final_fit()` on the two-element run selects the same candidate as on the one-element run. It also asserts that the two fits give the same predictions on the data. It also asserts that the two orders select different candidates. +- [x] AC2: `test-resample-weights.R` gains a test on the weighted design with one failed fold, and on the same design with one fold scoring NA on one metric. In each case it asserts that a new weight on the left-out fold leaves each affected summary row's `mean`, `std_err` and `n` identical. It also asserts that a new weight on a scored fold changes that row's `mean`. +- [x] AC3: `test-nested-tune-grid-oracles.R` gains a test on one grid run under a desirability rule. Its expected selection for each fold is computed by explicit code from the definitions in desirability2's help pages, on that fold's inner metrics. The test asserts that each fold's recorded selection equals that candidate. A source note in `cairn/references/` records those definitions and desirability2's version. +- [x] AC4: the test `"a grid run records the first metric of the set it was given, in either order"` in `test-selection-metric.R` gains one assertion. For each of its two metric sets, the final fit's recorded first metric equals `names(attr(ms, "metrics"))[[1L]]`. +- [x] AC5: `cairn/ORACLES.md` holds one oracle row for each test that AC1, AC2, AC3 and AC7 add. It holds one closed-form row for the first-name assertions of the test that AC4 names. Each row gives id, type, source and its test quoted by name. `grep -n 'event-level/O3' cairn/ORACLES.md` returns no line. The header of `test-event-level.R` says again that the file does not test the exchange of `sens` and `spec`. The memory group counts `nested-resamples-memory/O2` for the size slope. +- [x] AC6: `cairn/ORACLES.md` carries one line for each of five results. The `compute_metrics()` line names the survival branch on a new metric set. The averages line names the renormalized probabilities and the median of `.pred_time`. The racing line names the rows of eliminated candidates. The line on the final fit under `event_level` names the level exchange. The iteration-stage line, in the Bayesian and in the annealing group, names the proposals and the perturbations. For each named part, the line names the registry rows it credits for that part. The lines for the survival branch, the eliminated candidates' rows, the level exchange, the proposals and the perturbations each name the candidate row that holds that part, or say why it owes no second type. A new `cairn/DECISIONS.md` entry records the rule that those lines apply. `grep -n 'Give finer results a second oracle' cairn/ORACLES.md` returns no line. +- [x] AC7: `test-collect-predictions-summarize.R` gains two tests. The first uses the probability grid run with one `.pred_event` value planted as missing. It asserts that each row's averaged class probabilities sum to 1 within 1e-12. It also asserts that the planted row's probability means, before renormalization, do not sum to 1. The second runs the censored fixed workflow on a repeated v-fold outer design with three repeats. It asserts that `collect_predictions(summarize = TRUE)` matches tune's average on `tune::fit_resamples()` over the same outer splits, through `expect_matches_tune()`. It asserts that the same average matches `hand_average()` through `expect_matches_hand()`. It also asserts that for some row, the median of its three `.pred_time` values differs from their mean. +- [x] AC8: `devtools::test()` reports no failures, and `air format --check .` is clean. ## Coverage @@ -174,3 +174,12 @@ Back each result that the candidate row "Give finer results a second oracle" nam - ripple #2: the candidate row "Give finer results a second oracle" still lists the selection results — reject, planned change (graduated at step 9). - ripple #3: the weights source sits away from its test — reject, false (it is in the header block at :28, the file's convention). - ripple #4: "held out twice" at :793 is not true of the new three-repeat pair — reject, false (it describes the fixtures its helper reads). +- Pass 2 fix-now lines without a sha above (diff-bug #3, #11; prior-review #1, #2): fixed 22bc50c7. +- AC1 evidence: `test-eval-time.R` holds "a final fit at a multi-element `eval_time` is the final fit at its first element", which runs both orders of `srv_eval_times()` and each first element alone, and asserts the same `dist`, identical `predict()` on the data, and different picks across orders. Green in the 1164-test run at 417a8d17. T1's plant (pairing each order with the other element) failed 4 of 5 expectations. +- AC2 evidence: `test-resample-weights.R` holds "a new weight on the fold a summary row leaves out leaves that row unchanged", over a failed fold 2 (`break_fold()`) and an NA rsq on fold 3 (`plant_na()`). It asserts identical `mean`, `std_err` and `n` under a new left-out weight, and a changed `mean` under a new scored-fold weight. Green at 417a8d17. T9's plant (no renormalization) failed both cases. +- AC3 evidence: `test-nested-tune-grid-oracles.R:493` holds "the desirability rule picks the candidate its definition names in the fold's inner metrics" on one seed-20 grid run under the four-term rule. It writes `d_min`/`d_max` over each fold's observed range and their geometric mean, and asserts each fold's `.selected` equals the top candidate. The file reran green after pass 2 (10 tests). `cairn/references/desirability2-select-best.md` records the pages and desirability2 0.2.0. T3's plant (`df2` maximized) failed folds 2 and 3. +- AC4 evidence: the test "a grid run records the first metric of the set it was given, in either order" asserts `extract_procedure(final)$first_metric` equals `names(attr(ms, "metrics"))[[1L]]` for both metric sets (`test-selection-metric.R:34-37`). Green at 417a8d17. +- AC5 evidence: `cairn/ORACLES.md` holds `nested-tune-grid-oracles/O8` (:55), `eval-time/O4` (:235), `resample-weights/O3` (:286), `collect-predictions-summarize/O1` (:304) and `/O2` (:306) for the AC1-AC3 and AC7 tests, and closed-form `selection-metric/O1` (:65) for AC4's test, each with id, type, source and its test in quotes. `grep -c 'event-level/O3' cairn/ORACLES.md` prints 0. `test-event-level.R:18-19` reads "The exchange of sensitivity and specificity between the two levels is yardstick's arithmetic, so this file does not test it." The memory group's sentence at :262 counts O2 for the size slope. +- AC6 evidence: `cairn/ORACLES.md` lines: compute_metrics (:311) names the survival branch, crediting U1 for dynamic metrics and no row for static and linear-predictor ones, and the candidate row; averages (:291) names the renormalized probabilities (O1, U2) and the median of `.pred_time` (O2, U2's three-repeat test); racing (:90) names the eliminated candidates' rows (O3) and the candidate row; event_level (:240) names the level exchange, credits O2, and names the candidate row; Bayesian (:70) and annealing (:107) name the proposals and perturbations, credit no row, and say no independent oracle exists. `### D-123` sits at `cairn/DECISIONS.md:2144`. `grep -c 'Give finer results a second oracle' cairn/ORACLES.md` prints 0. +- AC7 evidence: `test-collect-predictions-summarize.R` holds "a renormalized probability average sums to one in every row" (sums within 1e-12; the planted row's raw means do not sum to 1) and "a censored average over three repeats takes the median time, as tune does" (`repeats = 3`; `expect_matches_tune()`, `expect_matches_hand()`, and a median-mean gap above 1e-4 of the column). The file reran green after pass 2 (34 tests). T7's plants (`mean_by` for `.pred_time`, no renormalization) failed both tests. +- AC8 evidence: `devtools::test()` at 417a8d17, alone: 1164 tests, 0 failed. The pass-2 edits touched two test files, which reran green; `air format --check .` clean at 22bc50c7. From 83bf2d6e00697d8ff2e5e5af620c5d92e263c782 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Sat, 31 Oct 2026 16:04:18 +0100 Subject: [PATCH 16/16] M166: step-7 approval --- cairn/milestones/M166-finer-second-oracle.md | 1 + 1 file changed, 1 insertion(+) diff --git a/cairn/milestones/M166-finer-second-oracle.md b/cairn/milestones/M166-finer-second-oracle.md index 64bb7f2d..664195d4 100644 --- a/cairn/milestones/M166-finer-second-oracle.md +++ b/cairn/milestones/M166-finer-second-oracle.md @@ -95,6 +95,7 @@ Back each result that the candidate row "Give finer results a second oracle" nam - 2026-10-31: T10: `air format --check .` clean; `devtools::test()` under `NESTEDTUNE_FULL_SUITE=true`, alone: 1164 tests, 0 failed, 0 errors, 0 skipped. - 2026-10-31: claim audit: not owed — internal tier - 2026-10-31: implement pass 2 done; status review. +- 2026-10-31: step-7 approval: m166-finer-second-oracle approved for merge - 2026-10-31: review return 1: fix-now findings to carry in the implement pass: diff-bug #4 (show the failed-fold weights case failing), #8, #12; blame-history #6, #9; prior-review #2, #3, #5; ripple #6. ## Decisions