Repository navigation
Conversation
SQLite: Only apply the ZFS -shm workaround on the first open of a database
Previously, if a package was replaced by a different store path with the same version (e.g. a rebuild due to a dependency or source change), `nix profile history` reported "No changes" for that generation. Now it prints a `changed` line. Also add a `--show-source` flag that shows the (abbreviated) locked flake reference of each added, removed, upgraded or changed package. For upgraded/changed packages, the old and new references are shown, which explains where the change came from. Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
…ne-yield Fix GC stack scanning when a coroutine yields from another coroutine's stack
nix profile history: Show store path changes with the same version
The periodic purge of expired entries from the narinfo disk cache
does
delete from NARs where (present = 0 and timestamp < ?)
or (present = 1 and timestamp < ?)
which, with only the (cache, hashPart) primary key available, is a
full table scan. On busy builders the cache grows to millions of rows
and several GiB, and the scan was measured to hold the SQLite write
lock for over two minutes on a cold file cache (NIX-491). Every
concurrent `nix copy` blocks on that lock, and uploader timeouts that
kill the purging process just cause the next process to start the
scan over.
Add an index on NARs(present, timestamp). SQLite then executes the
purge as two covering-index range searches. On a synthetic 1.5M-row,
3.1 GiB cache with 5000 expired entries, the warm-cache purge drops
from ~9 s and ~3 GiB of page reads to ~1 s and <100 MiB of page
reads for the delete itself (~3.5 s including the commit), and the
`nix` process startup that triggers it goes from 9.8 s to 4.4 s.
Bump the database filename to v4 so the index is created as part of
a fresh, empty database. Building the index in place on an existing
multi-GiB cache would itself hold the write lock for a full table
read, which is exactly the stall this is meant to remove.
Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
The purge ran as a single DELETE in whichever process happened to open the cache, holding the SQLite write lock for its entire duration. Concurrent processes (e.g. parallel `nix copy` uploads) blocked on it for the whole time, and if the purging process was killed, e.g. by a caller's timeout, all work was rolled back and the next process started over from scratch. Delete in chunks of 1000 rows instead, each in its own autocommit transaction: * The write lock is released after every chunk, so concurrent writers wait at most for one chunk (a few hundred ms warm, under a second cold on NVMe) instead of the whole purge. * Committed chunks survive if the process is killed. LastPurge is only updated once a chunk comes back short, so the next process to open the cache resumes where the killed one left off, and concurrent processes that both find the purge due simply delete chunks cooperatively. The chunk subquery is served by the (present, timestamp) index, with the outer delete going by rowid. Total purge cost is unchanged: on tmpfs, deleting 100k expired rows from a 1.5M-row cache takes ~3 s either way. In a contention test with a concurrent writer, the longest wait for the write lock dropped from 33 s to 4.5 s (on ZFS, which inflates both numbers). Killing the purge after 6 s left the committed chunks in place with LastPurge unchanged, and the next run finished the job. Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
Sync with upstream 2.35-maintenance
Processes that open the cache at around the same time all see that the purge is due, since LastPurge is only updated when a purge completes. With chunked deletion they would interleave harmlessly, but they'd all contend for the SQLite write lock and do redundant work. Take a non-blocking exclusive lock on a lock file next to the database before purging. Processes that fail to get it skip the purge. The lock is released when its holder exits or is killed, and since LastPurge is still only updated on completion, the next process to open the cache then takes over. Tested by starting eight processes at once with the purge due: one of them did the entire purge, the other seven skipped it and started up in under a second. Killing the lock holder half-way left LastPurge unchanged and the next process picked up the remaining rows. Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
Make purging the narinfo disk cache cheap and non-blocking
…manual Add derivation metadata documentation to the Determinate Nix manual
Flake lock file updates:
• Updated input 'nix':
'path:../..'
→ 'path:../..'
• Updated input 'nix/nixpkgs':
'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-26.05/0.1.1015029%2Brev-bb04742a2b3f59073509b332de34a678bfdee9c4/01a0f2bf-d071-73b2-a585-3f332b21e4d9/source.tar.gz' (2026-09-30)
→ 'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-26.05/0.1.1015033%2Brev-ad9e1d72007a5b115c4ef59edaa50ac12f868d5d/01a0fda6-2748-7282-b564-aaa6761bf51a/source.tar.gz' (2026-10-02)
Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
This skill describes how to edit the auto-generated release notes (v<version>.md and changes.md) into shape during the release process. Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
…f1b-a099-48b4-8864-e0cd3ab2be0b Release v3.23.1
The GC roots server started a thread per client connection and only afterwards inserted it into the `connections` map keyed by the client fd. The client thread's cleanup handler erases its own entry from that map, so if a client disconnected immediately, the cleanup could run before the insert. The cleanup then found nothing to erase, the fd was closed, and the server inserted a finished-but-joinable thread under a now-closed fd number. When a later client got the same fd number from accept(), the server tried to insert under an existing key. std::map::insert doesn't overwrite, so the temporary pair holding the new (joinable) std::thread was destroyed, which calls std::terminate(). This showed up in Sentry as a SIGABRT in nix-daemon with no exception in flight, and another thread blocked on the `connections` lock in its cleanup handler. Fix this by holding the `connections` lock while starting the client thread and inserting it. The cleanup handler can't run until the entry exists, and the fd is only closed after the cleanup, so fd numbers can no longer collide. Also assert that the insert didn't hit an existing key. Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
Add make.nix files for libutil-c, libutil-test-support and libutil-tests, transcribed from their meson.build files. The `nix-util-tests` executable gets its main() from gtest_main, which is linked explicitly since no #include refers to it. deps.nix gains entries for gtest, gmock and rapidcheck (the latter with an explicit -lrapidcheck, as rapidcheck.pc lists no libraries). A new `runTest` helper in lib.nix (with run-test.py) produces a derivation that runs a test program. libutil-tests uses it for `tests.run` and `tests.run-without-new-syscalls`, the latter under `enosys` like package.nix. Every variant of the nix-make flake now has `nix-util-tests-run`, `nix-util-tests-run-without-new-syscalls` and `nix-all-tests`, an aggregate of all test runs (unit and functional); `checks` points at the release `nix-all-tests`, so `nix flake check` runs everything. Deviations from the Meson build: no static libnixutilc (the plugin C API), no precompiled headers, and `_NIX_TEST_ACCEPT` is not supported inside the derivation (same as package.nix). Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
Add make.nix files for libstore-c, libstore-test-support and libstore-tests, transcribed from their meson.build files. The benchmark sources are excluded (Meson's `benchmarks` option is off by default). The test data is rooted at the repository together with tests/functional/derivation, as in package.nix, and the test run has `openssl` on its PATH for the HTTPS binary cache tests. The store tests keep a store under $HOME, so run-test.py now sets HOME to a writable directory and expands `$HOME` in the `env` values of `runTest`. Every variant of the nix-make flake gains `nix-store-c`, `nix-store-test-support`, `nix-store-tests` and `nix-store-tests-run`, which is included in `nix-all-tests`. Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
Add make.nix files for libfetchers-c and libfetchers-tests, transcribed from their meson.build files. The libgit2 entry in deps.nix now matches `<git2.h>` as well as `<git2/...>`, which one of the tests includes. Every variant of the nix-make flake gains `nix-fetchers-c`, `nix-fetchers-tests` and `nix-fetchers-tests-run`, which is included in `nix-all-tests`. Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
…race Fix crash in the GC roots server on fd reuse
Add make.nix files for libexpr-c, libexpr-test-support and libexpr-tests, transcribed from their meson.build files. The benchmark sources are excluded (Meson's `benchmarks` option is off by default). Every variant of the nix-make flake gains `nix-expr-c`, `nix-expr-test-support`, `nix-expr-tests` and `nix-expr-tests-run`, which is included in `nix-all-tests`. Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
Add make.nix files for libflake-c and libflake-tests, transcribed from their meson.build files. Every variant of the nix-make flake gains `nix-flake-c`, `nix-flake-tests` and `nix-flake-tests-run`, which is included in `nix-all-tests`. With this, all the unit test suites that the Meson build runs by default are covered. Assisted-by: Claude Fable 5.1 <noreply@anthropic.com>
packaging/nix-make: Build and run the unit tests
Bash treats `<` and `>` as word breaks, so for e.g. nix nario list < /tmp/system-exp<TAB> `_get_comp_words_by_ref` returned `(nix nario list '<' /tmp/system-exp)`, and we ran `NIX_GET_COMPLETIONS=4 nix nario list '<' /tmp/system-exp`. Nix treats `<` as a regular argument and returns no completions, and since there is no fallback, nothing got completed. Use `_init_completion` instead. It does filename completion if the current word is the target of a redirection, and removes redirections from `words` otherwise (so `nix nario list < foo --j<TAB>` also works). Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
…ections bash completion: Handle redirections
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Not intended to be merged directly. This PR is a convenience to show the diff between upstream Nix and Determinate Nix (the
mainbranch).Continuation of #4.