Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions cairn/DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2141,6 +2141,12 @@ upstream, either of which would let the file return to a shared blob.
**Decision:** on a push or pull request the test step runs the `tests/testthat/test-*.R` files that carry the line `# devel-vctrs.yaml runs this file on every push and pull request.` as a whole line, and stops with an error when none does. The step decides the whole-suite switch from `GITHUB_EVENT_NAME` and `inputs.full` alone, and the concurrency group is keyed on those two raw values with the workflow and ref, so two runs in one group make the same whole-suite choice. On a subset run each commit's markers decide the files. Rejected at the plan gate: a `-compat` file-name convention, which still drops a file renamed out of it.
**Consequences:** a test file joins the push-time watch by taking the marker line, replacing D-119's "adds itself to the filter". A scheduled run and a full dispatch no longer share a group, nor do a `full` false dispatch on the default branch and a push to it, so each pair can run side by side. Falsified by a watched file losing its marker in an edit that no run flags, or by a subset run cancelling or replacing a whole-suite run. (Narrows D-119.)

### D-123 (2026-10-31): a part of a result is credited only to the oracle rows whose tests reach it with input that can show a defect there, a part left on one type is recorded in its group with the candidate row that holds its second type, and the Bayesian and annealing iteration stage owes none. Extends D-121, and exempts the iteration stage from GP2's second type

**Context:** M163's review cut the `cairn/ORACLES.md` groups finer and found results that rest on one oracle type. M164 left nine of them on a candidate row. M166's plan gate asked what to do with a result where no second type can be built at low cost, and the user chose to record it with its reason. M166's first review found that crediting a part to every row whose test runs the same code was false: a fixture can run a branch with input that cannot show a defect in it. Examples are a probability average whose rows already sum to one, and a median over two values.
**Decision:** (1) A part of a result is credited only to the rows whose tests reach it with input that can show a defect there. (2) A part left on one type is named in its group's line, with the candidate row that holds its second type. (3) The Bayesian and annealing iteration stage keeps one correctness type, the reference loop, beside a serial-against-parallel check. No independent oracle exists for tune's Gaussian-process proposals or finetune's perturbations, so they owe no second type. The user chose this at M166's review stop over keeping the same-code reading.
**Consequences:** each group's line is the record a later audit reads, and the candidate row "Add second oracles to one-type parts" holds the parts left on one type. Falsified by a defect in a credited part that every row its line credits passes.

<!-- Template:

### D-00N (YYYY-MM-DD): Title
Expand Down
37 changes: 30 additions & 7 deletions cairn/ORACLES.md

Large diffs are not rendered by default.

5 changes: 3 additions & 2 deletions cairn/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ _Last hygiene check: 2026-10-31 M165 review. Graduated Guard the devel-vctrs fil
## Milestones
| ID | Title | Status | Depends on | Priority | File/Archive |
|---|---|---|---|---|---|
| M166 | Give finer results a second oracle | planned | — | normal | milestones/M166-finer-second-oracle.md |
| M166 | Give finer results a second oracle | review | — | normal | milestones/M166-finer-second-oracle.md |
| M165 | Keep the devel-vctrs test choice from drifting | done | — | normal | milestones/archive/M165-devel-vctrs-drift.md |
| M164 | Give six one-type results a second oracle | done | — | normal | milestones/archive/M164-second-oracle-types.md |
| M163 | Keep an oracle registry outside tests | done | — | normal | milestones/archive/M163-oracle-registry.md |
Expand All @@ -14,6 +14,7 @@ _Last hygiene check: 2026-10-31 M165 review. Graduated Guard the devel-vctrs fil
<!-- unnumbered ideas; one line each: idea — added YYYY-MM-DD — links -->
- Tidy the interrupt block's leftovers: four items in `test-parallel-interrupt.R` from M159's review, none added by M159. The ledger budgets no wait for the 60 s collect when no signal is sent. A SIGINT sent after `dispatch_folds()` returns early lands outside the block's handler. The header's "Established by execution against mirai 2.7.2 before the fix" is narrative history. The `m15-fold-` marker names carry a milestone id. Promote on the next change to that file — added 2026-10-24 at M159's review
- Give finer results a second oracle: one-type results in two-type `cairn/ORACLES.md` groups (M163 review): `compute_metrics()` own vs new metric set, final fit under `eval_time` and `event_level`, failed-fold weights, race fold record, Bayesian and annealing iteration stage, memory slope, `hand_average()`-only cases. Also the selection group's desirability rule and selection metric (M164 Out). Normal rung: shipped on one type — added 2026-10-31 at M164's plan gate — started M166
- Add second oracles to one-type parts: three parts that `cairn/ORACLES.md` credits to one type or none (D-123). In `compute_metrics()`'s survival branch, dynamic metrics rest on U1, and static and linear-predictor metrics reach no test. The racing record's rows for eliminated candidates rest on one live row, O3. The final fit under `event_level` rests on `event-level/O2`, since its level exchange is yardstick's arithmetic. Normal rung: shipped on one type — added 2026-10-31 at M166's review
- [low] Pin the byte figures in parallel.R: three `R/parallel.R` comments state wire sizes with no commit, date or procedure beside them: 5,141,166 B against 840,540 B for six split copies, 202,363 B against 524 B for the worker closure under `load_all()`, and 2,627 B for the 106-name manifest. The derived-figures rule asks for one. They predate M161, whose review found them — added 2026-10-24 at M161's review
- [parked] Point oracle headers at the registry: 21 test-file headers say "DESIGN Conventions: oracles are recorded in the test file that asserts them", and `test-parallel-payload.R` and `test-nested-resamples-memory.R` say the records sit beside the test, but since M163 each oracle's id and type live in `cairn/ORACLES.md`. Promote on the next change to those headers — added 2026-10-31 at M163's review
- [parked] Find why the shared pool loses a daemon: `test-parallel-identity.R`'s shared 2-daemon pool read 1 connection where 2 were started in a quiet full run at `9e45aaea` and a side-by-side run at `eb1ff5ac` (2026-10-24), and `shared_daemons(2)` then failed 10 expectations in four blocks (":241, :312, :357, :414"). A full run alone at `3488fffc` failed 11 blocks of that file with no messages kept. CI never showed it. M162 (dropped 2026-10-31) saw 0 of 40 trials lose a daemon at `dd6b5f7b`, on the file alone and the whole suite, beside 0 to 16 R processes spinning on arithmetic. Two suites running at once was not tried. Its probe, `benchmarks/probe-shared-pool-loss.R`, is on the unpushed local branch `m162-shared-pool-daemon-loss` at `2c0b02a7`. Promote when the loss recurs in CI, or in a local run whose failure output is kept — added 2026-10-24 at M159's T5, extended at M160's review, parked 2026-10-31 when M162 was dropped
Expand Down Expand Up @@ -44,7 +45,7 @@ _Last hygiene check: 2026-10-31 M165 review. Graduated Guard the devel-vctrs fil
- [parked] Inner designs M140 leaves out: added 2026-09-21 (M108), cut 2026-09-27 (M120, M121) and 2026-10-01 (M140). Every function but `nested_resamples()` and grid on the eight sliding pairs other than sliding-window / sliding-window. On that pair, `nested_final_fit()` for the four other tuners' results and `nested_workflow_map()` with a tuner `fn` (D-113). Under an outer `rolling_origin()` with an inner sliding design, the four other tuners, their final fits and the map with a tuner `fn`; and the grid final fit on the three sliding outer designs with an inner `rolling_origin()` (D-118, widened 2026-10-21 at M151). Any inner design other than `rolling_origin()` and the three sliding designs. Promote on a user running one
- [parked] Lift the desirability selection limits: Desirability selection under the two racing tuners and simulated annealing, and on censored regression models, all of which refuse the rule after M109. Added 2026-09-21 at M109's plan gate. Racing drops candidates before the end, so the rule needs its own oracle there, and desirability2 ranks every evaluation time on a censored model. The row also holds M109's deferred review findings, which its archive lists. Promote when desirability2 documents its selector for racing results, or on a user asking
- [parked] Nested tuning of tidyclust models: RR10 owns the analysis. `tune::last_fit()` refuses a cluster workflow, sum-of-squares metrics pick the grid's largest k, and only they have a closed-form oracle. Suggests takes a D-entry. Promote on a released `last_fit()` that scores a cluster workflow, plus a second oracle type for a metric whose selection is not degenerate — added 2026-09-21, M106-M109 plan gate, narrowed 2026-10-18 by a trivial edit
- [parked] M126 review leftovers on averaging: findings 3, 4 and 8, which `milestones/archive/M126-averaging-edge-cases.md` owns: gaps in what `benchmarks/averaging-speed.R` compares, a censored row of 0-row entries averaging to NULL where the help names only NULL entries, and the partial-run warning coming before a refusal in `collect_predictions(summarize = TRUE)` but after it in `augment()`. Promote on a user report of a 0-row entry, a median that disagrees with `stats::median()`, or the warning order misleading a reader — added 2026-09-28 at M126's review gate. Extended 2026-10-31 at M166's plan: the censored average tests hold each row out twice, so the median and the mean of `.pred_time` agree and neither oracle tells them apart
- [parked] M126 review leftovers on averaging: findings 3, 4 and 8, which `milestones/archive/M126-averaging-edge-cases.md` owns: gaps in what `benchmarks/averaging-speed.R` compares, a censored row of 0-row entries averaging to NULL where the help names only NULL entries, and the partial-run warning coming before a refusal in `collect_predictions(summarize = TRUE)` but after it in `augment()`. Promote on a user report of a 0-row entry, a median that disagrees with `stats::median()`, or the warning order misleading a reader — added 2026-09-28 at M126's review gate
- [parked] DESCRIPTION quoting test word rules: `test-description.R` reads `tune_grid()` and `rsample::` as bare names, flags Suggests names that are common words (`survival`, `censored`), and checks only the next character on each side of a word. Added 2026-10-17 at M146's review (diff-bug #1, #2, #4). Promote when a Description edit trips one
- [parked] CI job at the testthat floor: A CI job that runs the suite at the testthat floor in `DESCRIPTION`. Added 2026-09-29 at M130's plan gate, which chose one-time check runs at the floor over this job (3.3.0 since the re-plan, corrected 2026-09-29). The 3.0.0 floor went stale because tests adopted newer testthat functions and no job ran the floor. Promote on a test adopting a testthat function newer than the floor unnoticed
- [parked] Move macOS CI off P3M binaries: the macOS leg of `R-CMD-check.yaml` takes P3M binaries, with `gower?source` for P3M's OpenMP-linked `gower`. zstd no longer blocks it (no zstd CRAN macOS binary in the closure on 2026-10-20; pkgdepends#485 is open). M150 tried CRAN binaries and was dropped: the check step took 29m50s and 26m29s against a 30-minute cap (`milestones/archive/M150-macos-cran-binaries.md`). Promote when a P3M macOS binary breaks as `gower` did, or on evidence the CRAN-binary leg runs well under its cap — added 2026-09-21, rewritten 2026-10-20
Expand Down
Loading
Loading