Skip to content

fix(ci): unbreak main after the ubuntu-26.04 roll and the October advisories - #2768

Merged
chaliy merged 3 commits into
mainfrom
eve/clever-mccarthy-2aza8q
Oct 21, 2026
Merged

chaliy merged 3 commits into
mainfrom
eve/clever-mccarthy-2aza8q

Conversation

@chaliy

@chaliy chaliy commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Main CI and nightly are red in four independent ways, none caused by a code change of ours. Three commits, one per cause:

# Break Jobs it reddens Fix
1 Advisories published 19–20 Oct: rustls-webpki 0.103.15 (RUSTSEC-2026-0382), webpki-root-certs 1.0.9 (RUSTSEC-2026-0383), certifi 2026.7.22 (GHSA-6q48-6c7w-9gw2) Audit, on main and every PR fixed releases + a reviewed cargo vet delta audit per crate
2 jq 1.8 stopped letting --indent 0 imply -c (jqlang/jq#3232), and our jq oracle was unpinned Test, Generate Rust Coverage, nightly asan follow current jq + digest-pinned scripts/install-jq-ci.sh
3 Python 3.9 is absent from the 26.04 hosted image; setup-python has no fallback Test (Python 3.9) pin the version matrix to ubuntu-24.04
4 Ubuntu 26.04 installs uutils as the coreutils package, so dd/join/od/pr stop reporting GNU Generate Rust Coverage, and Test once #2 is fixed pin the three jobs that need the GNU oracle to ubuntu-24.04

Commit 2 also fixes a pre-existing parity bug found while checking the flag combinations: -c always won over --indent N, and --tab always won over a later --indent N, but real jq — 1.7 and 1.8 alike — is strictly last-flag-wins, so jq -c --indent 4 was wrong here. The cause was redundant state: a compact_output bool beside an indent: Indent whose Compact variant already said the same thing, with the tie broken after parsing. One field now holds the output shape and each flag overwrites it, so last-wins falls out of parse order.

yq is deliberately untouched — its -I 0 means compact, which is mikefarah/yq's own documented semantics, not jq's.

Why these are together

They block each other. Audit and Test gate every PR, so while main is broken four ways no single-purpose PR can reach green — each fix is held red by the other three. Splitting them would mean porting fixes between branches to get any one of them verifiable. #2769 was the jq fix on its own; it is closed and absorbed here, with its diagnosis kept on that thread.

Break #4 was also hidden behind #2: cargo test aborts on the jq target before the integration binary runs, so fixing jq alone would only have moved the Test job's red from one cause to another. It surfaced in the coverage job, which got further.

Before / After

Advisory scan (./scripts/audit-lockfiles.sh, the single source both the CI Audit job and nightly's Advisory Audit call) — before, from nightly run 39541706404:

cargo audit ./Cargo.lock
  rustls-webpki 0.103.15     RUSTSEC-2026-0382
  webpki-root-certs 1.0.9    RUSTSEC-2026-0383
error: 2 vulnerabilities found!
cargo audit ./crates/bashkit/fuzz/Cargo.lock
  rustls-webpki 0.103.15     RUSTSEC-2026-0382
error: 1 vulnerability found!
osv audit ./examples/docs-grep-agent/uv.lock: 1 vulnerable packages
  certifi 2026.7.22   GHSA-6q48-6c7w-9gw2, PYSEC-2026-4320
Process completed with exit code 1

After, across all four cargo lockfiles, five pnpm lockfiles and the uv lockfile: clean, with only the pre-existing allowlisted atomic-polyfill warning. cargo vet --locked → Vetting Succeeded (98 fully audited, 84 partially audited, 494 exempted).

jq --indent 0 — oracle is jq 1.8.1, SHA-256 020468de…c2a0d, matching the pin. echo '{"a":[1,2]}' | jq <flags> .:

flags before after real jq 1.8.1
--indent 0 {"a":[1,2]} newlines, no indent same as after
-c --indent 0 {"a":[1,2]} newlines, no indent same as after
--indent 0 -c {"a":[1,2]} {"a":[1,2]} same
-c --indent 4 {"a":[1,2]} 4-space indented same as after
--indent 4 -c {"a":[1,2]} {"a":[1,2]} same
-c --tab {"a":[1,2]} tab indented same as after
--tab --indent 4 tab indented 4-space indented same as after

I replayed every case in the differential suite against both jq 1.7 and 1.8.1 to size the blast radius. --indent 0 is the only behavior the two versions disagree on anywhere, which is why the precedence cases assert unconditionally and only --indent 0 is gated behind assert_matches_from_jq((1, 8), …):

SAME -c --indent 4   SAME --indent 4 -c   SAME --indent 0 -c   SAME -c --tab
SAME --tab -c        SAME --tab --indent 4   SAME --indent 4 --tab
SAME --indent 0 --indent 4
DIFF -c --indent 0   1.7={"a":[1,2]}   1.8={|"a": [|1,|2|]|}

Local verification of the whole branch — just check with the pinned jq 1.8.1 on PATH, exit 0, no failures. The tests that prove each fix:

test builtins::jq::format::tests::indent_zero_renders_newlines_without_indent ... ok
test builtins::jq::format::tests::indent_zero_nests_without_indent ... ok
test builtins::jq::format::tests::compact_is_the_only_compact_switch ... ok
test builtins::jq::tests::output_shape_flags_are_last_wins ... ok
test builtins::jq::tests::later_indent_overrides_tab ... ok
test builtins::jq::tests::differential::diff_indent_0 ... ok
test builtins::jq::tests::differential::diff_indent_0_nested ... ok
test builtins::jq::tests::differential::diff_output_shape_flags_are_last_wins ... ok
test dd_tests::dd_normal_copies_match_gnu ... ok
test join_resource_tests::join_ordinary_options_match_gnu ... ok
test od_resource_tests::od_normal_layout_matches_gnu ... ok
test pr_resource_tests::pr_finite_pagination_matches_real_bash_and_gnu_pr ... ok

Plus cargo vet --locked, just check-workflow-parity (version coverage matches; it ignores runner labels, so the Python pin keeps CI and release in step), just check-okf, just check-doc-links.

Risk

Checklist

  • Tests added or updated — --indent 0 flat and nested, -c still compact, the full last-wins matrix as unit and differential tests, a regression test for --tab followed by --indent N, and a version-gated oracle helper
  • Backward compatibility considered — internal code, no shim; the jq behavior change is the point

Unblocks #2765, whose only red check was the Audit job above.


Generated by Claude Code

Three advisories published 2026-10-19/20 reddened the Advisory Audit job on
last night's nightly, and with it the `Audit` job on main and on every PR,
because the scan reads every lockfile in the tree regardless of the diff.

- RUSTSEC-2026-0382 (rustls-webpki 0.103.15): path building could accept a
  certificate chain through a distrusted intermediate. 0.103.16 adds a
  fail-closed SPKI distrust check over the whole path -- leaf, intermediates
  and trust anchor -- surfaced as the new `Error::CertDistrusted`. Present in
  both `Cargo.lock` and `crates/bashkit/fuzz/Cargo.lock`, which resolve it
  independently, so the bump lands in both.
- RUSTSEC-2026-0383 (webpki-root-certs 1.0.9): the bundled root store still
  carried "emSign ECC Root CA - G3", whose intermediates are the ones above.
  1.0.10 drops that anchor; the delta is data-only.
- GHSA-6q48-6c7w-9gw2 / PYSEC-2026-4320 (certifi 2026.7.22) in
  `examples/docs-grep-agent/uv.lock`: the same emSign distrust, one ecosystem
  over. `uv lock --upgrade-package certifi` moves it to 2026.10.19.

The two Rust bumps are patch-level inside their current lines, so no other
crate in any lockfile moves with them.

`cargo vet` gains one delta audit per crate rather than an exemption. Both
deltas were reviewed in full against the published sources: rustls-webpki adds
the distrust list (eight DER SPKI blobs under `src/data/distrusted`), the new
error variant and its ordering, with no unsafe code, no new dependencies and
no change to signature verification or input bounds; webpki-root-certs removes
one trust anchor and changes nothing else in `src/lib.rs`.

No suppression was added: all three advisories have a fixed release, so
`deny.toml` and the threat model's suppressed-advisory list are unchanged.

Claude-Session: https://claude.ai/code/session_01HLxiDLRpTGkSoWxmLkxDeG
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit 1cdf8f7 Commit Preview URL

Branch Preview URL
Oct 21 2026, 09:46 AM

`jq --indent 0` now pretty-prints with zero-width indentation -- newlines
between tokens, no leading spaces -- instead of collapsing to compact output.
`-c`/`--compact-output` is the only compact switch.

jq 1.7 and earlier let `--indent 0` imply `-c`; jq 1.8 removed that
implication (jqlang/jq#3232) and we follow current jq. The differential suite
caught it the hard way: `ubuntu-latest` is rolling from 24.04 to 26.04, the
two images ship jq 1.7 and jq 1.8, and `diff_indent_0` asserts byte-for-byte
parity against whatever `jq` is on PATH. Nightly's `asan` job landed on 26.04
and went red with no change on our side -- and no implementation could satisfy
both halves of the fleet.

So the oracle is pinned the way ripgrep's already is: `scripts/install-jq-ci.sh`
verifies a SHA-256 before the binary reaches PATH and is wired into the three
jobs that run these tests (`ci.yml`'s `test` job, `coverage.yml`, nightly's
`asan`), with `JQ_VERSION` beside `RG_VERSION`. A pin does not help a
developer whose distro ships 1.7, so `assert_matches_from_jq((1, 8), ...)`
skips just the affected case on an older host oracle and says why -- the
bargain `minimum_bash_major` already strikes for the competitor corpus.

Checking the flag combinations turned up a second, pre-existing parity bug:
`-c` always won over `--indent N` and `--tab` always won over a later
`--indent N`, but real jq -- 1.7 and 1.8 alike -- is strictly last-flag-wins.
`jq -c --indent 4` was compact here and indented in real jq. The cause was
redundant state: a `compact_output` bool beside an `indent: Indent` whose
`Compact` variant already said the same thing, with the tie broken after
parsing. One field now holds the output shape and each of the three flags
overwrites it, so last-wins falls out of the parse order instead of being
arbitrated later.

Verified against a digest-checked jq 1.8.1 that `--indent 0` is the only
behavior anywhere in the differential suite that 1.7 and 1.8 disagree on, so
the precedence cases are asserted unconditionally and only `--indent 0` is
gated.

`yq` is deliberately untouched: its `-I 0` means compact, which is
mikefarah/yq's own documented semantics, not jq's.

Claude-Session: https://claude.ai/code/session_01HLxiDLRpTGkSoWxmLkxDeG
`ubuntu-latest` is rolling from 24.04 onto 26.04, and three jobs went red on
the new image without a line of our code changing. Both fixes here are about
who chooses the oracle, not about the sanitizer or the coverage tool that
happened to report them.

Python 3.9 is not on the 26.04 hosted image and `setup-python` has no
fallback, so the 3.9 leg died outright ("The version '3.9' with architecture
'x64' was not found for Ubuntu 26.04"). The wheel is cp39-abi3 and the package
claims `>=3.9`, so the matrix needs an image that can host the whole declared
range -- shrinking it would stop testing a version the wheel still promises.
`python.yml`'s `test` and `publish-python.yml`'s `test-builds` both pin
`ubuntu-24.04`; `just check-workflow-parity` compares version coverage and
ignores runner labels, so the two stay in step.

The `*_match_gnu` differential tests (`dd`, `join`, `od`, `pr`) need real GNU
coreutils and fail closed on Linux CI rather than skipping, which is why all
four fell together: Ubuntu 26.04 installs uutils as the `coreutils` binary
package, so `dd --version` no longer says "GNU coreutils". The three jobs that
run them -- `ci.yml`'s `test`, `coverage.yml`'s `coverage`, nightly's `asan`
-- pin `ubuntu-24.04`, the last image shipping GNU as the default.

That GNU pin is explicitly a stopgap and is recorded as one: it keeps the
oracle at the 9.4 these expectations were written against, but 24.04 runners
retire and the pin stops those jobs exercising newer images. Installing the
GNU oracle explicitly, the way `rg` and `jq` already are, is the durable fix
and is left out of an unbreak-CI change on purpose -- moving the oracle's
version can move the expectations with it, and that wants its own differential
review. `join` also probes only `/opt/homebrew/bin/gjoin`, so it needs
normalising before a PATH-based GNU install would cover all four.

Pinning the image does not undo the jq pin in the same branch: that oracle
arrives through `GITHUB_PATH`, so `jq` is 1.8.1 on either image.

Claude-Session: https://claude.ai/code/session_01HLxiDLRpTGkSoWxmLkxDeG
@chaliy chaliy changed the title fix(deps): clear RUSTSEC-2026-0382, -0383 and the certifi advisory fix(ci): unbreak main after the ubuntu-26.04 roll and the October advisories Oct 21, 2026
@chaliy
chaliy merged commit c20672e into main Oct 21, 2026
53 checks passed
@chaliy
chaliy deleted the eve/clever-mccarthy-2aza8q branch October 21, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant