Repository navigation
fix(ci): unbreak main after the ubuntu-26.04 roll and the October advisories - #2768
Merged
Merged
Conversation
Three advisories published 2026-10-19/20 reddened the Advisory Audit job on last night's nightly, and with it the `Audit` job on main and on every PR, because the scan reads every lockfile in the tree regardless of the diff. - RUSTSEC-2026-0382 (rustls-webpki 0.103.15): path building could accept a certificate chain through a distrusted intermediate. 0.103.16 adds a fail-closed SPKI distrust check over the whole path -- leaf, intermediates and trust anchor -- surfaced as the new `Error::CertDistrusted`. Present in both `Cargo.lock` and `crates/bashkit/fuzz/Cargo.lock`, which resolve it independently, so the bump lands in both. - RUSTSEC-2026-0383 (webpki-root-certs 1.0.9): the bundled root store still carried "emSign ECC Root CA - G3", whose intermediates are the ones above. 1.0.10 drops that anchor; the delta is data-only. - GHSA-6q48-6c7w-9gw2 / PYSEC-2026-4320 (certifi 2026.7.22) in `examples/docs-grep-agent/uv.lock`: the same emSign distrust, one ecosystem over. `uv lock --upgrade-package certifi` moves it to 2026.10.19. The two Rust bumps are patch-level inside their current lines, so no other crate in any lockfile moves with them. `cargo vet` gains one delta audit per crate rather than an exemption. Both deltas were reviewed in full against the published sources: rustls-webpki adds the distrust list (eight DER SPKI blobs under `src/data/distrusted`), the new error variant and its ordering, with no unsafe code, no new dependencies and no change to signature verification or input bounds; webpki-root-certs removes one trust anchor and changes nothing else in `src/lib.rs`. No suppression was added: all three advisories have a fixed release, so `deny.toml` and the threat model's suppressed-advisory list are unchanged. Claude-Session: https://claude.ai/code/session_01HLxiDLRpTGkSoWxmLkxDeG
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
bashkit | 1cdf8f7 | Commit Preview URL Branch Preview URL |
Oct 21 2026, 09:46 AM |
`jq --indent 0` now pretty-prints with zero-width indentation -- newlines between tokens, no leading spaces -- instead of collapsing to compact output. `-c`/`--compact-output` is the only compact switch. jq 1.7 and earlier let `--indent 0` imply `-c`; jq 1.8 removed that implication (jqlang/jq#3232) and we follow current jq. The differential suite caught it the hard way: `ubuntu-latest` is rolling from 24.04 to 26.04, the two images ship jq 1.7 and jq 1.8, and `diff_indent_0` asserts byte-for-byte parity against whatever `jq` is on PATH. Nightly's `asan` job landed on 26.04 and went red with no change on our side -- and no implementation could satisfy both halves of the fleet. So the oracle is pinned the way ripgrep's already is: `scripts/install-jq-ci.sh` verifies a SHA-256 before the binary reaches PATH and is wired into the three jobs that run these tests (`ci.yml`'s `test` job, `coverage.yml`, nightly's `asan`), with `JQ_VERSION` beside `RG_VERSION`. A pin does not help a developer whose distro ships 1.7, so `assert_matches_from_jq((1, 8), ...)` skips just the affected case on an older host oracle and says why -- the bargain `minimum_bash_major` already strikes for the competitor corpus. Checking the flag combinations turned up a second, pre-existing parity bug: `-c` always won over `--indent N` and `--tab` always won over a later `--indent N`, but real jq -- 1.7 and 1.8 alike -- is strictly last-flag-wins. `jq -c --indent 4` was compact here and indented in real jq. The cause was redundant state: a `compact_output` bool beside an `indent: Indent` whose `Compact` variant already said the same thing, with the tie broken after parsing. One field now holds the output shape and each of the three flags overwrites it, so last-wins falls out of the parse order instead of being arbitrated later. Verified against a digest-checked jq 1.8.1 that `--indent 0` is the only behavior anywhere in the differential suite that 1.7 and 1.8 disagree on, so the precedence cases are asserted unconditionally and only `--indent 0` is gated. `yq` is deliberately untouched: its `-I 0` means compact, which is mikefarah/yq's own documented semantics, not jq's. Claude-Session: https://claude.ai/code/session_01HLxiDLRpTGkSoWxmLkxDeG
2 tasks done
`ubuntu-latest` is rolling from 24.04 onto 26.04, and three jobs went red on
the new image without a line of our code changing. Both fixes here are about
who chooses the oracle, not about the sanitizer or the coverage tool that
happened to report them.
Python 3.9 is not on the 26.04 hosted image and `setup-python` has no
fallback, so the 3.9 leg died outright ("The version '3.9' with architecture
'x64' was not found for Ubuntu 26.04"). The wheel is cp39-abi3 and the package
claims `>=3.9`, so the matrix needs an image that can host the whole declared
range -- shrinking it would stop testing a version the wheel still promises.
`python.yml`'s `test` and `publish-python.yml`'s `test-builds` both pin
`ubuntu-24.04`; `just check-workflow-parity` compares version coverage and
ignores runner labels, so the two stay in step.
The `*_match_gnu` differential tests (`dd`, `join`, `od`, `pr`) need real GNU
coreutils and fail closed on Linux CI rather than skipping, which is why all
four fell together: Ubuntu 26.04 installs uutils as the `coreutils` binary
package, so `dd --version` no longer says "GNU coreutils". The three jobs that
run them -- `ci.yml`'s `test`, `coverage.yml`'s `coverage`, nightly's `asan`
-- pin `ubuntu-24.04`, the last image shipping GNU as the default.
That GNU pin is explicitly a stopgap and is recorded as one: it keeps the
oracle at the 9.4 these expectations were written against, but 24.04 runners
retire and the pin stops those jobs exercising newer images. Installing the
GNU oracle explicitly, the way `rg` and `jq` already are, is the durable fix
and is left out of an unbreak-CI change on purpose -- moving the oracle's
version can move the expectations with it, and that wants its own differential
review. `join` also probes only `/opt/homebrew/bin/gjoin`, so it needs
normalising before a PATH-based GNU install would cover all four.
Pinning the image does not undo the jq pin in the same branch: that oracle
arrives through `GITHUB_PATH`, so `jq` is 1.8.1 on either image.
Claude-Session: https://claude.ai/code/session_01HLxiDLRpTGkSoWxmLkxDeG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Main CI and nightly are red in four independent ways, none caused by a code change of ours. Three commits, one per cause:
rustls-webpki0.103.15 (RUSTSEC-2026-0382),webpki-root-certs1.0.9 (RUSTSEC-2026-0383),certifi2026.7.22 (GHSA-6q48-6c7w-9gw2)Audit, on main and every PRcargo vetdelta audit per crate--indent 0imply-c(jqlang/jq#3232), and our jq oracle was unpinnedTest,Generate Rust Coverage, nightlyasanscripts/install-jq-ci.shsetup-pythonhas no fallbackTest (Python 3.9)ubuntu-24.04coreutilspackage, sodd/join/od/prstop reporting GNUGenerate Rust Coverage, andTestonce #2 is fixedubuntu-24.04Commit 2 also fixes a pre-existing parity bug found while checking the flag combinations:
-calways won over--indent N, and--tabalways won over a later--indent N, but real jq — 1.7 and 1.8 alike — is strictly last-flag-wins, sojq -c --indent 4was wrong here. The cause was redundant state: acompact_outputbool beside anindent: IndentwhoseCompactvariant already said the same thing, with the tie broken after parsing. One field now holds the output shape and each flag overwrites it, so last-wins falls out of parse order.yqis deliberately untouched — its-I 0means compact, which is mikefarah/yq's own documented semantics, not jq's.Why these are together
They block each other.
AuditandTestgate every PR, so while main is broken four ways no single-purpose PR can reach green — each fix is held red by the other three. Splitting them would mean porting fixes between branches to get any one of them verifiable. #2769 was the jq fix on its own; it is closed and absorbed here, with its diagnosis kept on that thread.Break #4 was also hidden behind #2:
cargo testaborts on the jq target before the integration binary runs, so fixing jq alone would only have moved theTestjob's red from one cause to another. It surfaced in the coverage job, which got further.Before / After
Advisory scan (
./scripts/audit-lockfiles.sh, the single source both the CIAuditjob and nightly'sAdvisory Auditcall) — before, from nightly run39541706404:After, across all four cargo lockfiles, five pnpm lockfiles and the uv lockfile: clean, with only the pre-existing allowlisted
atomic-polyfillwarning.cargo vet --locked→Vetting Succeeded (98 fully audited, 84 partially audited, 494 exempted).jq
--indent 0— oracle is jq 1.8.1, SHA-256020468de…c2a0d, matching the pin.echo '{"a":[1,2]}' | jq <flags> .:--indent 0{"a":[1,2]}-c --indent 0{"a":[1,2]}--indent 0 -c{"a":[1,2]}{"a":[1,2]}-c --indent 4{"a":[1,2]}--indent 4 -c{"a":[1,2]}{"a":[1,2]}-c --tab{"a":[1,2]}--tab --indent 4I replayed every case in the differential suite against both jq 1.7 and 1.8.1 to size the blast radius.
--indent 0is the only behavior the two versions disagree on anywhere, which is why the precedence cases assert unconditionally and only--indent 0is gated behindassert_matches_from_jq((1, 8), …):Local verification of the whole branch —
just checkwith the pinned jq 1.8.1 onPATH, exit 0, no failures. The tests that prove each fix:Plus
cargo vet --locked,just check-workflow-parity(version coverage matches; it ignores runner labels, so the Python pin keeps CI and release in step),just check-okf,just check-doc-links.Risk
--indent 0and the flag combinations change output shape, so a caller relying on--indent 0meaning compact now gets pretty output — exactly as from a real jq 1.8 upgrade.-calone,--indent Nalone and--tabalone are unchanged, and those are the only forms any jq spec case in the repo uses.knowledge/operations/testing.mdrather than left to rot. Pinning the image keeps the GNU 9.4 oracle these expectations were written against, but 24.04 runners retire and the pin stops those jobs exercising newer images. The durable fix is to install the GNU oracle explicitly, the wayrgandjqnow are — left out on purpose, because moving the oracle's version can move the expectations with it and that deserves its own differential review.joinalso probes only/opt/homebrew/bin/gjoin, so it needs normalising before a PATH-based GNU install would cover all four.GITHUB_PATH, sojqis 1.8.1 on either image.Checklist
--indent 0flat and nested,-cstill compact, the full last-wins matrix as unit and differential tests, a regression test for--tabfollowed by--indent N, and a version-gated oracle helperUnblocks #2765, whose only red check was the
Auditjob above.Generated by Claude Code