Skip to content

zccache: Add version 1.13.21 - #3229

Draft
luhenry wants to merge 2 commits into
mainfrom
zccache
Draft

luhenry wants to merge 2 commits into
mainfrom
zccache

Conversation

@luhenry

@luhenry luhenry commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Compiles the Rust zccache/zccache-fp binaries and three abi3 pyo3 extensions into one py3-none wheel. Upstream publishes no riscv64 wheel.

Mirrors the build/build-wheels/test-wheels jobs of upstream's release-auto.yml and the native-pytest job of python-tests.yml.

Differs from upstream

  • Native cargo build in the manylinux image, not soldr/zig cross-compilation - upstream has no riscv64 lane
  • dnf install protobuf-compiler - protoc-bin-vendored has no riscv64 binary
  • Calls upstream's build_wheel() for one platform - its build-wheels command requires all six

Matrix: tests on cp312/cp313/cp314; the abi3 extensions cannot load on free-threaded cp314t.

Testing

  • same as upstream, run against the installed wheel

License: Wheel ships zccache itself (AGPL-3.0-only); upstream's wheels carry no licence text, so 0003 adds it.

Patches

  • 0001-protocol-fall-back-to-prost-build-s-own-protoc-looku.patch - Inappropriate [protoc-bin-vendored has no riscv64 binary]. build.rs panics without it. riscv64-only.
  • 0002-deps-bump-crash-handler-to-0.8-for-riscv64-support.patch - To upstream. crash-context 0.6 has no riscv64 ucontext_t, so the build fails. Reproduces cross-compiling from x86.
  • 0003-release_workflow-ship-LICENSE-in-every-wheel-s-dist-.patch - To upstream. Wheels carry no LICENSE without it. Reproduces everywhere.
  • 0004-watcher-notify-main-thread-once-per-interrupt.patch - Backport [zackees/zccache@64b7e7c, released in 1.14.5]. A watcher predicate KeyboardInterrupt reached the main thread outside the tests' assertRaises and aborted the pytest session (exit 2). Reproduces on x86_64. Also applied in the test job, which checks the upstream tree out again.

luhenry commented Oct 14, 2026 •

Copy link
Copy Markdown
Member Author

The failed check_commit_messages run (job 115916157215) is the known pull_request "opened" webhook stale-body-snapshot case, same as #3221-#3228: it evaluated the PR body before create_pull_request's auto-injected footer was stripped via update_pull_request. The edited-triggered re-run already passed cleanly (all three checks green). No fix needed — riscv64 build now queued.

luhenry added a commit that referenced this pull request Oct 15, 2026
…ated

topk-sdk (#3230), zope-index (#3228), zccache (#3229), kfnetlist (#3231),
ligo-skymap (#3233), quantile-forest (#3232), fbuild (#3235),
py-ballisticcalc-exts (#3238), resdata (#3237), oxapy (#3239),
pyramids-gis (#3240), biscuit-python (#3223), ansys-mapdl-reader (#3227)
all had open PRs that queue.yml never reflected (status still queued,
pr still null). Verified each PR independently before updating.
@luhenry
luhenry marked this pull request as draft October 15, 2026 12:14
The watcher test suite aborted with exit code 2 and "!!! KeyboardInterrupt !!!"
after every reported test passed (cp313 and cp314 legs of run 38582612676). A
notification_predicate that raises KeyboardInterrupt made FileWatcher call the
real _thread.interrupt_main() twice, and the tests let it reach the main thread
while it was still inside file_path.write_text(), outside their assertRaises
block. Reproduces on x86_64 in 23 of 50 runs; upstream fixed it in 64b7e7c
(released in 1.14.5).

Backport that commit, and apply patches in the test job too, which checks the
upstream tree out again (gotcha 501).

luhenry commented Oct 15, 2026 •

Copy link
Copy Markdown
Member Author

Test zccache 1.13.21 cp313 / cp314 (run 38582612676, jobs 116324636431 / 116324636408): a real upstream test defect, not an infra flake. Fixed in fe036c4 by backporting upstream's fix.

What the full logs show. Both legs fail the same way, in crates/zccache-watcher/python/tests. The python/tests suite passes on both (64 passed, 1 skipped).

cp314: 1.02s call python/tests/test_compat.py::WatcherTests::test_context_manager_and_resume_reset_state_cleanly
       ...
       !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
       <frozen codecs>:189: KeyboardInterrupt
       11 passed in 8.52s
cp313: ...
       !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
       /opt/python/cp313-cp313/lib/python3.13/pathlib/_abc.py:651: KeyboardInterrupt
       12 passed in 9.91s
##[error]Process completed with exit code 2.

The banner comes before N passed. pytest prints it in pytest_sessionfinish only when the session's exit status is INTERRUPTED, so the KeyboardInterrupt was raised inside the run-test loop, not in teardown after the session. The suite has 13 tests, so cp314 was cut off during test #12 and cp313 during test #13. Collection is alphabetical, so those are test_keyboard_interrupt_stop_logs_once_per_watcher (#12) and test_poll_predicate_keyboard_interrupt_propagates (#13). Both crash sites are inside Path.write_text(): 3.13 pathlib/_abc.py:651 is with self.open(mode='w', ...), and 3.14 codecs:189 is IncrementalEncoder.__init__, which open(mode='w') reaches. The podman orphan podman pause reaped at cleanup is the normal rootless-podman pause process and has nothing to do with this. No signal is involved: the interrupt is a _thread.interrupt_main() call from inside the process.

Root cause (upstream 1.13.21). Both tests use a notification_predicate that raises KeyboardInterrupt on the watcher's dispatch thread:

  • FileWatcher._predicate_allows() catches it, calls handle_keyboard_interrupt(ke) (a real _thread.interrupt_main()), then re-raises. _dispatch_loop()'s except KeyboardInterrupt calls handle_keyboard_interrupt again, so one interrupt can reach the main thread twice.
  • The tests don't mock interrupt_main, and they call file_path.write_text("b\n") before entering with self.assertRaises(KeyboardInterrupt):. NativeWatcher.poll_batch (crates/zccache-watcher-py/src/lib.rs) blocks in poll_timeout for up to poll_interval while holding the GIL, so the dispatch thread can starve the main thread. The main thread is then still inside write_text when the interrupt lands. That is outside assertRaises, so pytest treats it as a user Ctrl-C and aborts the whole session with exit code 2.

It isn't specific to riscv64 or podman. With the published x86_64 zccache==1.13.21 wheel and the upstream 1.13.21 suite, 23 of 50 local runs fail with exactly these two signatures (11 passed interrupted at test_keyboard_interrupt_behavior.py:83, 12 passed interrupted at :57). Tracing interrupt_main confirmed both calls come from the same watcher's dispatch thread (_predicate_allows:202 and _dispatch_loop:231).

Fix. Upstream already fixed this in zackees/zccache@64b7e7c, "fix(watcher): notify main thread once per interrupt (#1616)", which fixes zackees/zccache#1615 and first shipped in 1.14.5. It removes the duplicate notification from _predicate_allows in both copies of the watcher module. It also keeps interrupt_main mocked until the watcher has stopped and asserts it was called exactly once. Commit fe036c4:

  • adds patches/zccache/1.13.21/0004-watcher-notify-main-thread-once-per-interrupt.patch (Upstream-Status: Backport, without upstream's tasks/todo.md hunk); 0001-0004 apply cleanly;
  • adds the same python-wheels checkout and git apply step to test_wheel that build_wheel has. The test job checks the upstream tree out again, so without this the test-side half of the patch never applies (gotcha 501).

With the patch, x86_64 passes 60 of 60 runs (30 on cp313, 30 on cp314; 14 tests, including the new test_poll_predicate_keyboard_interrupt_notifies_main_once).

Next steps. The push re-triggers the workflow through pull_request: paths, which rebuilds the wheel with the library fix and reruns cp312/cp313/cp314. No manual re-run is needed. The PR stays a draft and stays watched.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant