diff --git a/.clang-format b/.clang-format index 1aadf2cadce3..44f493019a97 100644 --- a/.clang-format +++ b/.clang-format @@ -33,3 +33,4 @@ PPIndentWidth: 2 BinPackArguments: false BreakBeforeTernaryOperators: true SeparateDefinitionBlocks: Always +OneLineFormatOffRegex: "// NOFORMAT$" diff --git a/.claude/skills/release-notes/SKILL.md b/.claude/skills/release-notes/SKILL.md new file mode 100644 index 000000000000..2245a48986db --- /dev/null +++ b/.claude/skills/release-notes/SKILL.md @@ -0,0 +1,93 @@ +--- +name: release-notes +description: Edit the auto-generated Determinate Nix release notes for a release into shape. Use when asked to write, tweak, clean up or finish the release notes for a version (e.g. "write the release notes for 3.24.0"), or after the automated release process has committed "Generate release notes for ". +--- + +# Determinate Nix release notes + +The automated release process commits "Generate release notes for ", which creates or updates two files in `doc/manual/source/release-notes-determinate/`: + +* `v.md`: the release notes for this version, as a flat GitHub-style "What's Changed" list of PR titles. +* `changes.md`: the cumulative list of differences between upstream Nix and Determinate Nix, with the new PRs appended verbatim under ``. + +Both need manual editing. Do not change any other file. + +## Step 1: Gather information + +1. Determine the version from `.version-determinate` (or from the user). Read `v.md` and the tail of `changes.md`. +2. Read the two or three most recent `v*.md` files for style. `v3.23.0.md` (feature release) and `v3.23.1.md` (bug fix release) are good examples. +3. For every PR listed, read its description: + ``` + gh pr view --repo DeterminateSystems/nix-src --json title,body --jq '.title, .body' + ``` + The "Motivation" section usually says what the user-visible effect is. Ignore the auto-generated "Summary by CodeRabbit" section. If the description is empty, look at the commit messages (`git log --format='%h %s%n%b' ^1..^2`). +4. For "Sync with upstream" PRs, look at the merge commit's diff stat and the backported commits to summarize what user-visible fixes came in. + +## Step 2: Classify each PR + +* **Major**: new features, new commands/flags/settings, notable behavior changes, and significant performance improvements. Each gets its own `##` section. A bug fix only qualifies if the bug was a major issue for users in practice (widely reported, affecting common workflows); a fix for a crash that few users have hit yet is minor, even if the root cause was serious. +* **Minor**: bug fixes, test fixes, dependency/flake.lock updates, documentation, CI, refactoring. These are grouped into `##` sections like "Bug fixes", "Performance improvements", "Miscellaneous changes", with one concise line per PR. +* Closely related PRs (e.g. several PRs implementing one feature, or several GC crash fixes) can share one major section with `PRs: ...`. + +## Step 3: Edit `v.md` + +Keep the header, the "Based on upstream Nix ..." line, the "New Contributors" section (if present) and the "**Full Changelog**" line as generated. Replace the "What's Changed" list with: + +```markdown +## + + + +PR: [DeterminateSystems/nix-src#NNN](https://github.com/DeterminateSystems/nix-src/pull/NNN) + +## + +... + +PRs: [DeterminateSystems/nix-src#NNN](...), [DeterminateSystems/nix-src#MMM](...) + +## Bug fixes + +* by @author in [DeterminateSystems/nix-src#NNN](...) + +## Performance improvements + +* ... by @author in [DeterminateSystems/nix-src#NNN](...) + +## Miscellaneous changes + +* ... by @author in [DeterminateSystems/nix-src#NNN](...) +``` + +Guidelines: + +* Major sections come first, most important first. Then the grouped minor sections, then "New Contributors" and the "Full Changelog" line. +* Section titles are short and user-facing, e.g. "`nix flake bake`", "OpenTelemetry Support", "Garbage collector crash fixes". Backticks for command names. +* Write for Nix users, not Nix developers: say what the user experiences ("`nix flake check` no longer crashes when ...") rather than the implementation ("Don't force cached eval errors inside a catch handler"). Mention internal details only when they help the user understand the scope (e.g. "only affects setups without a Nix daemon"). +* The PR link lines are literally `PR: ` or `PRs: ` followed by comma-separated links. +* For a PR that cherry-picks an upstream PR, acknowledge the upstream author. Look up the author with `gh pr view --repo NixOS/nix --json author --jq .author.login` and append to the PR line: `, cherry-picked from [NixOS/nix#](https://github.com/NixOS/nix/pull/) by @`. +* In the minor-item lists, keep the generated `... by @author in [link]` format but reword cryptic PR titles into readable sentences. Drop internal prefixes like `libstore:` or function names unless they're the clearest description. +* Omit sections that would be empty. +* Link to manual pages where relevant using relative paths, e.g. `../language/derivation-metadata.md`. + +## Step 4: Edit `changes.md` + +`changes.md` lists only *differences relative to upstream Nix*: new features, new commands/flags/settings/builtins, and incompatible or deliberately different behavior. Under the new `` marker: + +* **Remove** bug fixes, performance improvements, crash fixes, test/CI/documentation/dependency changes, "Sync with upstream" PRs, and the "@user made their first contribution" line. +* **Remove** backports and cherry-picks from upstream Nix (e.g. PRs whose description says "Cherry-picked from https://github.com/NixOS/nix/pull/..."), even if they are new features, since they are not Determinate-specific. +* **Keep** the remaining items, rewritten as one-sentence present-tense feature descriptions followed by the PR link, matching the existing entries: + ```markdown + * Determinate Nix has a `nix foo` command for doing X. [DeterminateSystems/nix-src#NNN](https://github.com/DeterminateSystems/nix-src/pull/NNN) + + * `nix bar` has a flag `--baz` that does Y. [DeterminateSystems/nix-src#NNN](...) + ``` + Items are separated by blank lines and have no `by @author in` text. +* It is normal for a bug fix release to leave the marker with nothing under it. Keep the marker. + +Do not touch entries for earlier versions. The "This section lists the differences between upstream Nix ... and Determinate Nix ." line is updated by the release process, so leave it alone. + +## Step 5: Finish + +* Re-read both files once for consistency: every PR from the generated list appears exactly once in `v.md`; links are well-formed; no "What's Changed" header remains. +* Don't commit unless asked. If asked, commit both files on the release branch with the message "Tweak release notes". diff --git a/.coderabbit.yaml b/.coderabbit.yaml deleted file mode 100644 index 2220649caa28..000000000000 --- a/.coderabbit.yaml +++ /dev/null @@ -1,18 +0,0 @@ -# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json -# Disable CodeRabbit auto-review to prevent verbose comments on PRs. -# When enabled: false, CodeRabbit won't attempt reviews and won't post -# "Review skipped" or other automated comments. -reviews: - auto_review: - enabled: false - review_status: false - high_level_summary: false - poem: false - sequence_diagrams: false - changed_files_summary: false - tools: - github-checks: - enabled: false -chat: - art: false - auto_reply: false diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS deleted file mode 100644 index 763c5f27ee6b..000000000000 --- a/.github/CODEOWNERS +++ /dev/null @@ -1,17 +0,0 @@ -# Pull requests concerning the listed files will automatically invite the respective maintainers as reviewers. -# This file is not used for denoting any kind of ownership, but is merely a tool for handling notifications. -# -# Merge permissions are required for maintaining an entry in this file. -# For documentation on this mechanism, see https://help.github.com/articles/about-codeowners/ - -# Default reviewers if nothing else matches -* @edolstra - -# This file -.github/CODEOWNERS @edolstra - -# Documentation of built-in functions -src/libexpr/primops.cc @roberth - -# Libstore layer -/src/libstore @ericson2314 diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index af94c3e9e5bb..08a5851748d4 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,10 +1,9 @@ --- name: Bug report about: Report unexpected or incorrect behaviour -title: '' +title: "" labels: bug -assignees: '' - +assignees: "" --- ## Describe the bug @@ -32,7 +31,9 @@ assignees: '' ## Metadata - + + + ## Additional context @@ -42,13 +43,9 @@ assignees: '' -- [ ] checked [latest Nix manual] \([source]) +- [ ] checked [latest Determinate Nix manual] \([source]) - [ ] checked [open bug issues and pull requests] for possible duplicates -[latest Nix manual]: https://nix.dev/manual/nix/development/ -[source]: https://github.com/NixOS/nix/tree/master/doc/manual/source -[open bug issues and pull requests]: https://github.com/NixOS/nix/labels/bug - ---- - -Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc). +[latest Determinate Nix manual]: https://manual.determinate.systems/ +[source]: https://github.com/DeterminateSystems/nix-src/tree/main/doc/manual/source +[open bug issues and pull requests]: https://github.com/DeterminateSystems/nix-src/labels/bug diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index fe9f9dd209d4..b88e10937988 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -1,10 +1,9 @@ --- name: Feature request about: Suggest a new feature -title: '' +title: "" labels: feature -assignees: '' - +assignees: "" --- ## Is your feature request related to a problem? @@ -27,13 +26,9 @@ assignees: '' -- [ ] checked [latest Nix manual] \([source]) -- [ ] checked [open feature issues and pull requests] for possible duplicates - -[latest Nix manual]: https://nix.dev/manual/nix/development/ -[source]: https://github.com/NixOS/nix/tree/master/doc/manual/source -[open feature issues and pull requests]: https://github.com/NixOS/nix/labels/feature - ---- +- [ ] checked [latest Determinate Nix manual] \([source]) +- [ ] checked [open bug issues and pull requests] for possible duplicates -Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc). +[latest Determinate Nix manual]: https://manual.determinate.systems/ +[source]: https://github.com/DeterminateSystems/nix-src/tree/main/doc/manual/source +[open bug issues and pull requests]: https://github.com/DeterminateSystems/nix-src/labels/bug diff --git a/.github/ISSUE_TEMPLATE/installer.md b/.github/ISSUE_TEMPLATE/installer.md index 070e0bd9b25b..430bef971aac 100644 --- a/.github/ISSUE_TEMPLATE/installer.md +++ b/.github/ISSUE_TEMPLATE/installer.md @@ -1,18 +1,17 @@ --- name: Installer issue about: Report problems with installation -title: '' +title: "" labels: installer -assignees: '' - +assignees: "" --- ## Platform - + -- [ ] Linux: - [ ] macOS +- [ ] Linux: - [ ] WSL ## Additional information @@ -35,13 +34,9 @@ assignees: '' -- [ ] checked [latest Nix manual] \([source]) -- [ ] checked [open installer issues and pull requests] for possible duplicates - -[latest Nix manual]: https://nix.dev/manual/nix/development/ -[source]: https://github.com/NixOS/nix/tree/master/doc/manual/source -[open installer issues and pull requests]: https://github.com/NixOS/nix/labels/installer - ---- +- [ ] checked [latest Determinate Nix manual] \([source]) +- [ ] checked [open bug issues and pull requests] for possible duplicates -Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc). +[latest Determinate Nix manual]: https://manual.determinate.systems/ +[source]: https://github.com/DeterminateSystems/nix-src/tree/main/doc/manual/source +[open bug issues and pull requests]: https://github.com/DeterminateSystems/nix-src/labels/bug diff --git a/.github/ISSUE_TEMPLATE/missing_documentation.md b/.github/ISSUE_TEMPLATE/missing_documentation.md index 4e05b626d398..fcdd0d20135e 100644 --- a/.github/ISSUE_TEMPLATE/missing_documentation.md +++ b/.github/ISSUE_TEMPLATE/missing_documentation.md @@ -1,10 +1,9 @@ --- name: Missing or incorrect documentation about: Help us improve the reference manual -title: '' +title: "" labels: documentation -assignees: '' - +assignees: "" --- ## Problem @@ -19,13 +18,9 @@ assignees: '' -- [ ] checked [latest Nix manual] \([source]) -- [ ] checked [open documentation issues and pull requests] for possible duplicates - -[latest Nix manual]: https://nix.dev/manual/nix/development/ -[source]: https://github.com/NixOS/nix/tree/master/doc/manual/source -[open documentation issues and pull requests]: https://github.com/NixOS/nix/labels/documentation - ---- +- [ ] checked [latest Determinate Nix manual] \([source]) +- [ ] checked [open bug issues and pull requests] for possible duplicates -Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc). +[latest Determinate Nix manual]: https://manual.determinate.systems/ +[source]: https://github.com/DeterminateSystems/nix-src/tree/main/doc/manual/source +[open bug issues and pull requests]: https://github.com/DeterminateSystems/nix-src/labels/bug diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index c155bf8bfa4f..d3e1f8177364 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,26 +1,3 @@ - - ## Motivation @@ -34,9 +11,3 @@ PR stuck in review? We have two Nix team meetings per week online that are open - ---- - -Add :+1: to [pull requests you find important](https://github.com/NixOS/nix/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc). - -The Nix maintainer team uses a [GitHub project board](https://github.com/orgs/NixOS/projects/19) to [schedule and track reviews](https://github.com/NixOS/nix/tree/master/maintainers#project-board-protocol). diff --git a/.github/STALE-BOT.md b/.github/STALE-BOT.md index bc0005413f1a..281d0f79a8b7 100644 --- a/.github/STALE-BOT.md +++ b/.github/STALE-BOT.md @@ -2,34 +2,21 @@ - Thanks for your contribution! - To remove the stale label, just leave a new comment. -- _How to find the right people to ping?_ → [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.) -- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #users:nixos.org](https://matrix.to/#/#users:nixos.org). +- You can always ask for help on [Discord](https://determinate.systems/discord). ## Suggestions for PRs -1. GitHub sometimes doesn't notify people who commented / reviewed a PR previously, when you (force) push commits. If you have addressed the reviews you can [officially ask for a review](https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from those who commented to you or anyone else. -2. If it is unfinished but you plan to finish it, please mark it as a draft. -3. If you don't expect to work on it any time soon, closing it with a short comment may encourage someone else to pick up your work. -4. To get things rolling again, rebase the PR against the target branch and address valid comments. -5. If you need a review to move forward, ask in [the Discourse thread for PRs that need help](https://discourse.nixos.org/t/prs-in-distress/3604). -6. If all you need is a merge, check the git history to find and [request reviews](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from people who usually merge related contributions. +1. If it is unfinished but you plan to finish it, please mark it as a draft. +1. If you don't expect to work on it any time soon, closing it with a short comment may encourage someone else to pick up your work. +1. To get things rolling again, rebase the PR against the target branch and address valid comments. +1. If you need a review to move forward, ask in [Discord](https://determinate.systems/discord). ## Suggestions for issues 1. If it is resolved (either for you personally, or in general), please consider closing it. 2. If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough. -3. If you still have interest in resolving it, try to ping somebody who you believe might have an interest in the topic. Consider discussing the problem in [our Discourse Forum](https://discourse.nixos.org/). -4. As with all open source projects, your best option is to submit a Pull Request that addresses this issue. We :heart: this attitude! +3. If you still have interest in resolving it, try to ping somebody who you believe might have an interest in the topic. Consider discussing the problem in [Discord](https://determinate.systems/discord). **Memorandum on closing issues** Don't be afraid to close an issue that holds valuable information. Closed issues stay in the system for people to search, read, cross-reference, or even reopen--nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort. - -## Useful GitHub search queries - -- [Open PRs with any stale-bot interaction](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+) -- [Open PRs with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22) -- [Open PRs with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+) -- [Open Issues with any stale-bot interaction](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+) -- [Open Issues with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22+) -- [Open Issues with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+) diff --git a/.github/actions/install-nix-action/action.yaml b/.github/actions/install-nix-action/action.yaml deleted file mode 100644 index 2fb433b741c3..000000000000 --- a/.github/actions/install-nix-action/action.yaml +++ /dev/null @@ -1,54 +0,0 @@ -name: "Install Nix" -description: "Helper action for installing Nix with support for dogfooding from master" -inputs: - dogfood: - description: "Whether to use Nix installed from the latest artifact from master branch" - required: true # Be explicit about the fact that we are using unreleased artifacts - extra_nix_config: - description: "Gets appended to `/etc/nix/nix.conf` if passed." - install_url: - description: "URL of the Nix installer" - required: false - default: "https://releases.nixos.org/nix/nix-2.34.8/install" - github_token: - description: "Github token" - required: true - use_cache: - description: "Whether to setup github actions cache (not implemented currently)" - default: false - required: false -runs: - using: "composite" - steps: - - name: "Download nix install artifact from master" - shell: bash - id: download-nix-installer - if: inputs.dogfood == 'true' - run: | - RUN_ID=$(gh run list --repo "$DOGFOOD_REPO" --workflow ci.yml --branch master --status success --json databaseId --jq ".[0].databaseId") - - if [ "$RUNNER_OS" == "Linux" ]; then - INSTALLER_ARTIFACT="installer-linux" - elif [ "$RUNNER_OS" == "macOS" ]; then - INSTALLER_ARTIFACT="installer-darwin" - else - echo "::error ::Unsupported RUNNER_OS: $RUNNER_OS" - exit 1 - fi - - INSTALLER_DOWNLOAD_DIR="$GITHUB_WORKSPACE/$INSTALLER_ARTIFACT" - mkdir -p "$INSTALLER_DOWNLOAD_DIR" - - gh run download "$RUN_ID" --repo "$DOGFOOD_REPO" -n "$INSTALLER_ARTIFACT" -D "$INSTALLER_DOWNLOAD_DIR" - echo "installer-path=file://$INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT" - - echo "::notice ::Dogfooding Nix installer from master (https://github.com/$DOGFOOD_REPO/actions/runs/$RUN_ID)" - env: - GH_TOKEN: ${{ inputs.github_token }} - DOGFOOD_REPO: "NixOS/nix" - - uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1 - with: - # Ternary operator in GHA: https://www.github.com/actions/runner/issues/409#issuecomment-752775072 - install_url: ${{ inputs.dogfood == 'true' && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }} - install_options: ${{ inputs.dogfood == 'true' && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }} - extra_nix_config: ${{ inputs.extra_nix_config }} diff --git a/.github/release-notes.sh b/.github/release-notes.sh new file mode 100755 index 000000000000..f641e146d2e8 --- /dev/null +++ b/.github/release-notes.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash + +# SC2002 disables "useless cat" warnings. +# I prefer pipelines that start with an explicit input, and go from there. +# Overly fussy. +# shellcheck disable=SC2002 + +scratch=$(mktemp -d -t tmp.XXXXXXXXXX) +finish() { + rm -rf "$scratch" +} +trap finish EXIT + +DATE=$(date +%Y-%m-%d) +DETERMINATE_NIX_VERSION=$(cat .version-determinate) +TAG_NAME="v${DETERMINATE_NIX_VERSION}" +NIX_VERSION=$(cat .version) +NIX_VERSION_MAJOR_MINOR=$(echo "$NIX_VERSION" | cut -d. -f1,2) +GITHUB_REPOSITORY="${GITHUB_REPOSITORY:-DeterminateSystems/nix-src}" + +gh api "/repos/${GITHUB_REPOSITORY}/releases/generate-notes" \ + -f "tag_name=${TAG_NAME}" > "$scratch/notes.json" + +trim_trailing_newlines() { + local text + text="$(cat)" + echo -n "${text}" +} + +linkify_gh() { + sed \ + -e 's!\(https://github.com/DeterminateSystems/nix-src/\(pull\|issue\)/\([[:digit:]]\+\)\)![DeterminateSystems/nix-src#\3](\1)!' \ + -e 's#\(https://github.com/DeterminateSystems/nix-src/compare/\([^ ]\+\)\)#[\2](\1)#' +} + +( + cat doc/manual/source/release-notes-determinate/changes.md \ + | sed 's/^.*\(\)$/This section lists the differences between upstream Nix '"$NIX_VERSION_MAJOR_MINOR"' and Determinate Nix '"$DETERMINATE_NIX_VERSION"'.\1/' \ + + printf "\n\n" "$DETERMINATE_NIX_VERSION" + cat "$scratch/notes.json" \ + | jq -r .body \ + | grep -v '^#' \ + | grep -v "Full Changelog" \ + | trim_trailing_newlines \ + | sed -e 's/^\* /\n* /' \ + | linkify_gh + echo "" # final newline +) > "$scratch/changes.md" + +( + printf "# Release %s (%s)\n\n" \ + "$DETERMINATE_NIX_VERSION" \ + "$DATE" + printf "* Based on [upstream Nix %s](../release-notes/rl-%s.md).\n\n" \ + "$NIX_VERSION" \ + "$NIX_VERSION_MAJOR_MINOR" + + cat "$scratch/notes.json" | jq -r .body | linkify_gh +) > "$scratch/rl.md" + +( + cat doc/manual/source/SUMMARY.md.in \ + | sed 's/\(\)$/\1\n - [Release '"$DETERMINATE_NIX_VERSION"' ('"$DATE"')](release-notes-determinate\/'"$TAG_NAME"'.md)/' +) > "$scratch/summary.md" + +mv "$scratch/changes.md" doc/manual/source/release-notes-determinate/changes.md +mv "$scratch/rl.md" "doc/manual/source/release-notes-determinate/v${DETERMINATE_NIX_VERSION}.md" +mv "$scratch/summary.md" doc/manual/source/SUMMARY.md.in diff --git a/.github/workflows/backport.yml b/.github/workflows/backport.yml deleted file mode 100644 index b059853714e5..000000000000 --- a/.github/workflows/backport.yml +++ /dev/null @@ -1,37 +0,0 @@ -name: Backport -on: - pull_request_target: - types: [closed, labeled] -permissions: - contents: read -jobs: - backport: - name: Backport Pull Request - permissions: - # for korthout/backport-action - contents: write - pull-requests: write - if: github.repository_owner == 'NixOS' && github.event.pull_request.merged == true && (github.event_name != 'labeled' || startsWith('backport', github.event.label.name)) - runs-on: ubuntu-24.04-arm - steps: - - name: Generate GitHub App token - id: generate-token - uses: actions/create-github-app-token@v3 - with: - app-id: ${{ vars.CI_APP_ID }} - private-key: ${{ secrets.CI_APP_PRIVATE_KEY }} - - uses: actions/checkout@v6 - with: - ref: ${{ github.event.pull_request.head.sha }} - # required to find all branches - fetch-depth: 0 - - name: Create backport PRs - uses: korthout/backport-action@66065406958f46e82238fd59546f5a99e69e22aa # v4.5.2 - id: backport - with: - # Config README: https://github.com/korthout/backport-action#backport-action - github_token: ${{ steps.generate-token.outputs.token }} - github_workspace: ${{ github.workspace }} - auto_merge_enabled: true - pull_description: |- - Automatic backport to `${target_branch}`, triggered by a label in #${pull_number}. diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml new file mode 100644 index 000000000000..fd34f0edf2ed --- /dev/null +++ b/.github/workflows/build.yml @@ -0,0 +1,327 @@ +on: + workflow_call: + inputs: + flake: + required: false + default: "./packaging/secure-packages" + type: string + system: + required: true + type: string + runner: + required: true + type: string + runner_for_virt: + required: true + type: string + runner_small: + required: true + type: string + if: + required: false + default: true + type: boolean + run_tests: + required: false + default: true + type: boolean + run_vm_tests: + required: false + default: false + type: boolean + run_regression_tests: + required: false + default: false + type: boolean + publish_manual: + required: false + default: false + type: boolean + upload_artifacts: + required: false + default: true + type: boolean + secrets: + manual_netlify_auth_token: + required: false + manual_netlify_site_id: + required: false + sentry_auth_token: + required: false + sentry_org: + required: false + sentry_project: + required: false + +jobs: + build: + if: ${{ inputs.if }} + strategy: + fail-fast: false + runs-on: ${{ inputs.runner }} + timeout-minutes: 120 + steps: + - uses: actions/checkout@v4 + - uses: DeterminateSystems/determinate-nix-action@main + with: + extra-conf: | + extra-experimental-features = provenance + - uses: DeterminateSystems/flakehub-cache-action@main + - run: nix build ${{ inputs.flake }}#packages.${{ inputs.system }}.default .#packages.${{ inputs.system }}.binaryTarball --no-link -L + - run: nix build ${{ inputs.flake }}#packages.${{ inputs.system }}.binaryTarball --out-link tarball + - run: nix build ${{ inputs.flake }}#^debug,out + - name: Upload debug info to Sentry + run: ./maintainers/upload-debug-info-to-sentry.py --debug-dir ./result-debug ./result/bin/nix + if: env.SENTRY_AUTH_TOKEN != '' + env: + SENTRY_AUTH_TOKEN: ${{ secrets.sentry_auth_token }} + SENTRY_ORG: ${{ secrets.sentry_org }} + SENTRY_PROJECT: ${{ secrets.sentry_project }} + - uses: actions/upload-artifact@v6 + if: inputs.upload_artifacts + with: + name: ${{ inputs.system }} + path: ./tarball/*.xz + + build_static: + if: ${{ inputs.if }} + strategy: + fail-fast: false + runs-on: ${{ inputs.runner }} + timeout-minutes: 120 + steps: + - uses: actions/checkout@v4 + - uses: DeterminateSystems/determinate-nix-action@main + with: + extra-conf: | + extra-experimental-features = provenance + - uses: DeterminateSystems/flakehub-cache-action@main + - run: nix build ${{ inputs.flake }}#packages.${{ inputs.system }}.nix-cli-static --no-link -L + + test: + if: ${{ inputs.if && inputs.run_tests}} + needs: build + strategy: + fail-fast: false + runs-on: ${{ inputs.runner }} + timeout-minutes: 120 + steps: + - uses: actions/checkout@v4 + - uses: DeterminateSystems/determinate-nix-action@main + with: + extra-conf: | + extra-experimental-features = provenance + - uses: DeterminateSystems/flakehub-cache-action@main + - run: nix flake check --keep-going ${{ inputs.flake }} -L --system ${{ inputs.system }} + + # Run and test the new build system. + nix_make: + if: ${{ inputs.if && inputs.run_tests && inputs.system == 'x86_64-linux' }} + runs-on: ${{ inputs.runner }} + timeout-minutes: 120 + steps: + - uses: actions/checkout@v4 + - uses: DeterminateSystems/determinate-nix-action@main + with: + extra-conf: | + extra-experimental-features = wasm-builtin + - uses: DeterminateSystems/flakehub-cache-action@main + - run: nix build ./packaging/nix-make#release.functional-tests -L + + vm_tests_smoke: + if: inputs.run_vm_tests && github.event_name != 'merge_group' + needs: build + runs-on: ${{ inputs.runner_for_virt }} + steps: + - uses: actions/checkout@v4 + - uses: DeterminateSystems/determinate-nix-action@main + with: + extra-conf: | + extra-experimental-features = provenance + - uses: DeterminateSystems/flakehub-cache-action@main + - run: | + nix build -L \ + ${{ inputs.flake }}#hydraJobs.tests.functional_user \ + ${{ inputs.flake }}#hydraJobs.tests.githubFlakes \ + ${{ inputs.flake }}#hydraJobs.tests.nix-docker \ + ${{ inputs.flake }}#hydraJobs.tests.tarballFlakes \ + ; + + vm_tests_all: + if: inputs.run_vm_tests && github.event_name == 'merge_group' + needs: build + runs-on: ${{ inputs.runner_for_virt }} + steps: + - uses: actions/checkout@v4 + - uses: DeterminateSystems/determinate-nix-action@main + with: + extra-conf: | + extra-experimental-features = provenance + - uses: DeterminateSystems/flakehub-cache-action@main + - run: | + tests=$(nix flake show --json \ + | jq -r ' + .inventory.hydraJobs.output.children.tests.children + | with_entries(select(.value | has("derivation"))) + | keys[] + | "${{ inputs.flake }}#hydraJobs.tests." + .') + + if [ -z "$tests" ]; then + echo "error: no tests found in hydraJobs.tests" + exit 1 + fi + + cmd() { + nix build -L --keep-going --timeout 600 $tests + } + + if ! cmd; then + echo "failed, retrying once ..." + printf "\n\n\n\n\n\n\n\n" + cmd + fi + + flake_regressions: + if: | + (inputs.run_regression_tests && github.event_name == 'merge_group') + || ( + inputs.run_regression_tests + && github.event.pull_request.head.repo.full_name == 'DeterminateSystems/nix-src' + && ( + (github.event.action == 'labeled' && github.event.label.name == 'flake-regression-test') + || (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, 'flake-regression-test')) + ) + ) + needs: build + runs-on: ${{ inputs.runner }} + strategy: + matrix: + nix_config: + - "lazy-trees = true" + - "lazy-trees = false" + - "eval-cores = 24" + glob: + - "[0-9]*" + - "[a-b]*" + - "[c]*" + - "[d]*" + - "[e]*" + - "[f]*" + - "[g-h]*" + - "[i-k]*" + - "[l]*" + - "[m]*" + - "[n]*" + - "[o-s]*" + - "[t]*" + - "[u-z]*" + + steps: + - name: Checkout nix + uses: actions/checkout@v4 + - name: Checkout flake-regressions + uses: actions/checkout@v4 + with: + repository: NixOS/flake-regressions + path: flake-regressions + - name: Checkout flake-regressions-data + uses: actions/checkout@v4 + with: + repository: NixOS/flake-regressions-data + path: flake-regressions/tests + - uses: DeterminateSystems/determinate-nix-action@main + with: + extra-conf: | + extra-experimental-features = provenance + - uses: DeterminateSystems/flakehub-cache-action@main + - name: Run flake regression tests + env: + #PARALLEL: ${{ !contains(matrix.nix_config, 'eval-cores') && '-P 50%' || '-P 1' }} + PARALLEL: '-P 1' + FLAKE_REGRESSION_GLOB: ${{ matrix.glob }} + NIX_CONFIG: ${{ matrix.nix_config }} + PREFETCH: "1" + USE_NIX_FLAKE_SHOW: "1" + run: | + set -x + echo "PARALLEL: $PARALLEL" + echo "NIX_CONFIG: $NIX_CONFIG" + if [ ! -z "${NSC_CACHE_PATH:-}" ]; then + mkdir -p "${NSC_CACHE_PATH}/nix/xdg-cache" + export XDG_CACHE_HOME="${NSC_CACHE_PATH}/nix/xdg-cache" + fi + nix build ${{ inputs.flake }} -L --out-link ./new-nix + export PATH=$(pwd)/new-nix/bin:$PATH + [[ $(type -p nix) = $(pwd)/new-nix/bin/nix ]] + + nix config show lazy-trees + nix config show eval-cores + lscpu + nproc + + if ! flake-regressions/eval-all.sh; then + echo "Some failed, trying again" + printf "\n\n\n\n\n\n\n\n" + NIX_REMOTE=/tmp/nix flake-regressions/eval-all.sh + fi + + manual: + if: github.event_name != 'merge_group' + needs: build + runs-on: ${{ inputs.runner_small }} + permissions: + id-token: "write" + contents: "read" + pull-requests: "write" + statuses: "write" + deployments: "write" + steps: + - name: Checkout nix + uses: actions/checkout@v4 + - uses: DeterminateSystems/determinate-nix-action@main + with: + extra-conf: | + extra-experimental-features = provenance + - uses: DeterminateSystems/flakehub-cache-action@main + - name: Build manual + if: inputs.system == 'x86_64-linux' + run: nix build ${{ inputs.flake }}#hydraJobs.manual + - uses: nwtgck/actions-netlify@v3.0 + if: inputs.publish_manual && inputs.system == 'x86_64-linux' + with: + publish-dir: "./result/share/doc/nix/manual" + production-branch: main + github-token: ${{ secrets.GITHUB_TOKEN }} + deploy-message: "Deploy from GitHub Actions" + # NOTE(cole-h): We have a perpetual PR displaying our changes against upstream open, but + # its conversation is locked, so this PR comment can never be posted. + # https://github.com/DeterminateSystems/nix-src/pull/165 + enable-pull-request-comment: ${{ github.event.pull_request.number != 165 }} + enable-commit-comment: true + enable-commit-status: true + overwrites-pull-request-comment: true + env: + NETLIFY_AUTH_TOKEN: ${{ secrets.manual_netlify_auth_token }} + NETLIFY_SITE_ID: ${{ secrets.manual_netlify_site_id }} + + success: + needs: + - build + - test + - vm_tests_smoke + - vm_tests_all + - flake_regressions + - manual + if: ${{ always() }} + runs-on: ubuntu-latest + steps: + - run: "true" + - run: | + echo "A dependent in the build matrix failed:" + echo "$needs" + exit 1 + env: + needs: ${{ toJSON(needs) }} + if: | + contains(needs.*.result, 'failure') || + contains(needs.*.result, 'cancelled') diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 75637c096bd0..133d19c2173c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,286 +2,282 @@ name: "CI" on: pull_request: - merge_group: push: branches: - - master - workflow_dispatch: - inputs: - dogfood: - description: 'Use dogfood Nix build' - required: false - default: false - type: boolean - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true + # NOTE: make sure any branches here are also valid directory names, + # otherwise creating the directory and uploading to s3 will fail + - main + merge_group: + release: + types: + - published -permissions: read-all +permissions: + id-token: "write" + contents: "read" + pull-requests: "write" + statuses: "write" + deployments: "write" jobs: eval: - runs-on: ubuntu-24.04 + runs-on: UbuntuLatest32Cores128G steps: - - uses: actions/checkout@v6 - with: - fetch-depth: 0 - - uses: ./.github/actions/install-nix-action - with: - dogfood: false - extra_nix_config: - experimental-features = nix-command flakes - github_token: ${{ secrets.GITHUB_TOKEN }} - use_cache: false - - run: nix flake show --all-systems --json - - pre-commit-checks: - name: pre-commit checks - runs-on: ubuntu-24.04 - steps: - - uses: actions/checkout@v6 - - uses: ./.github/actions/install-nix-action + - uses: actions/checkout@v4 with: - dogfood: false - extra_nix_config: experimental-features = nix-command flakes - github_token: ${{ secrets.GITHUB_TOKEN }} - - run: ./ci/gha/tests/pre-commit-checks + fetch-depth: 0 + - uses: DeterminateSystems/determinate-nix-action@main + - run: nix flake show --all-systems --json + + build_x86_64-linux: + uses: ./.github/workflows/build.yml + # Determinate Secure Packages is a private flake dependency that external + # (fork) contributors can't access, so only build it for non-fork PRs. + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'DeterminateSystems/nix-src' + with: + system: x86_64-linux + runner: namespace-profile-linuxamd32c64g-cache + runner_for_virt: UbuntuLatest32Cores128G + runner_small: ubuntu-latest + run_tests: true + run_vm_tests: true + run_regression_tests: true + publish_manual: true + secrets: + manual_netlify_auth_token: ${{ secrets.NETLIFY_AUTH_TOKEN }} + manual_netlify_site_id: ${{ secrets.NETLIFY_SITE_ID }} + sentry_auth_token: ${{ secrets.SENTRY_AUTH_TOKEN }} + sentry_org: ${{ secrets.SENTRY_ORG }} + sentry_project: ${{ secrets.SENTRY_PROJECT }} + + build_x86_64-linux_no_dsp: + uses: ./.github/workflows/build.yml + # Build the non-DSP flake in the merge queue, on tags, and for external (fork) PRs, + # which can't access Determinate Secure Packages. + if: github.event_name == 'merge_group' || startsWith(github.ref, 'refs/tags/') || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != 'DeterminateSystems/nix-src') + with: + flake: . + system: x86_64-linux + runner: namespace-profile-linuxamd32c64g-cache + runner_for_virt: UbuntuLatest32Cores128G + runner_small: ubuntu-latest + run_tests: true + run_vm_tests: false + run_regression_tests: false + upload_artifacts: false + secrets: + sentry_auth_token: ${{ secrets.SENTRY_AUTH_TOKEN }} + sentry_org: ${{ secrets.SENTRY_ORG }} + sentry_project: ${{ secrets.SENTRY_PROJECT }} - basic-checks: - name: aggregate basic checks + build_aarch64-linux: + uses: ./.github/workflows/build.yml + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'DeterminateSystems/nix-src' + with: + system: aarch64-linux + runner: UbuntuLatest32Cores128GArm + runner_for_virt: UbuntuLatest32Cores128GArm + runner_small: UbuntuLatest32Cores128GArm + secrets: + sentry_auth_token: ${{ secrets.SENTRY_AUTH_TOKEN }} + sentry_org: ${{ secrets.SENTRY_ORG }} + sentry_project: ${{ secrets.SENTRY_PROJECT }} + + build_aarch64-linux_no_dsp: + uses: ./.github/workflows/build.yml + if: github.event_name == 'merge_group' || startsWith(github.ref, 'refs/tags/') || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != 'DeterminateSystems/nix-src') + with: + flake: . + system: aarch64-linux + runner: UbuntuLatest32Cores128GArm + runner_for_virt: UbuntuLatest32Cores128GArm + runner_small: UbuntuLatest32Cores128GArm + upload_artifacts: false + secrets: + sentry_auth_token: ${{ secrets.SENTRY_AUTH_TOKEN }} + sentry_org: ${{ secrets.SENTRY_ORG }} + sentry_project: ${{ secrets.SENTRY_PROJECT }} + + build_aarch64-darwin: + uses: ./.github/workflows/build.yml + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'DeterminateSystems/nix-src' + with: + system: aarch64-darwin + runner: namespace-profile-mac-m2-12c28g + runner_for_virt: namespace-profile-mac-m2-12c28g + runner_small: macos-latest-xlarge + secrets: + sentry_auth_token: ${{ secrets.SENTRY_AUTH_TOKEN }} + sentry_org: ${{ secrets.SENTRY_ORG }} + sentry_project: ${{ secrets.SENTRY_PROJECT }} + + build_aarch64-darwin_no_dsp: + uses: ./.github/workflows/build.yml + if: github.event_name == 'merge_group' || startsWith(github.ref, 'refs/tags/') || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != 'DeterminateSystems/nix-src') + with: + flake: . + system: aarch64-darwin + runner: namespace-profile-mac-m2-12c28g + runner_for_virt: namespace-profile-mac-m2-12c28g + runner_small: macos-latest-xlarge + upload_artifacts: false + secrets: + sentry_auth_token: ${{ secrets.SENTRY_AUTH_TOKEN }} + sentry_org: ${{ secrets.SENTRY_ORG }} + sentry_project: ${{ secrets.SENTRY_PROJECT }} + + success: + runs-on: ubuntu-latest + needs: + - eval + - build_x86_64-linux + - build_aarch64-linux + - build_aarch64-darwin + - build_x86_64-linux_no_dsp + - build_aarch64-linux_no_dsp + - build_aarch64-darwin_no_dsp if: ${{ always() }} - runs-on: ubuntu-24.04 - needs: [pre-commit-checks, eval] steps: - - name: Exit with any errors - if: ${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }} - run: | + - run: "true" + - run: | + echo "A dependent in the build matrix failed:" + echo "$needs" exit 1 + env: + needs: ${{ toJSON(needs) }} + if: | + contains(needs.*.result, 'failure') || + contains(needs.*.result, 'cancelled') - tests: - needs: basic-checks - strategy: - fail-fast: false - matrix: - include: - - scenario: on ubuntu - runs-on: ubuntu-24.04 - os: linux - instrumented: false - primary: true - stdenv: stdenv - - scenario: on macos - runs-on: macos-14 - os: darwin - instrumented: false - primary: true - stdenv: stdenv - - scenario: on ubuntu (with sanitizers / coverage) - runs-on: ubuntu-24.04 - os: linux - instrumented: true - primary: false - stdenv: clangStdenv - name: tests ${{ matrix.scenario }} - runs-on: ${{ matrix.runs-on }} - timeout-minutes: 60 - steps: - - uses: actions/checkout@v6 - with: - fetch-depth: 0 - - uses: ./.github/actions/install-nix-action - with: - github_token: ${{ secrets.GITHUB_TOKEN }} - dogfood: false - # The sandbox would otherwise be disabled by default on Darwin - extra_nix_config: "sandbox = true" - # Since ubuntu 22.30, unprivileged usernamespaces are no longer allowed to map to the root user: - # https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces - - run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 - if: matrix.os == 'linux' - - name: Run component tests - run: | - nix build --file ci/gha/tests/wrapper.nix componentTests -L \ - --arg withInstrumentation ${{ matrix.instrumented }} \ - --argstr stdenv "${{ matrix.stdenv }}" - - name: Run VM tests - run: | - nix build --file ci/gha/tests/wrapper.nix vmTests -L \ - --arg withInstrumentation ${{ matrix.instrumented }} \ - --argstr stdenv "${{ matrix.stdenv }}" - if: ${{ matrix.os == 'linux' }} - - name: Run flake checks and prepare the installer tarball - run: | - ci/gha/tests/build-checks - ci/gha/tests/prepare-installer-for-github-actions - if: ${{ matrix.primary }} - - name: Collect code coverage - run: | - nix build --file ci/gha/tests/wrapper.nix codeCoverage.coverageReports -L \ - --arg withInstrumentation ${{ matrix.instrumented }} \ - --argstr stdenv "${{ matrix.stdenv }}" \ - --out-link coverage-reports - cat coverage-reports/index.txt >> $GITHUB_STEP_SUMMARY - if: ${{ matrix.instrumented }} - - name: Upload coverage reports - uses: actions/upload-artifact@v7 - with: - name: coverage-reports - path: coverage-reports/ - if: ${{ matrix.instrumented }} - - name: Upload installer tarball - uses: actions/upload-artifact@v7 - with: - name: installer-${{matrix.os}} - path: out/* - if: ${{ matrix.primary }} + - uses: actions/checkout@v4 + - uses: DeterminateSystems/determinate-nix-action@main + - uses: DeterminateSystems/flakehub-cache-action@main - windows_tests: - needs: basic-checks - name: windows unit tests - runs-on: ubuntu-24.04 - continue-on-error: true - timeout-minutes: 60 - steps: - - uses: actions/checkout@v6 - with: - fetch-depth: 0 - - uses: ./.github/actions/install-nix-action - with: - github_token: ${{ secrets.GITHUB_TOKEN }} - dogfood: false - - name: Run Windows unit tests - run: | - nix build --file ci/gha/tests/windows.nix unitTests.nix-util-tests -L + - name: Create artifacts directory + run: mkdir -p ./artifacts - installer_test: - needs: [tests] - strategy: - fail-fast: false - matrix: - include: - - scenario: on ubuntu - runs-on: ubuntu-24.04 - os: linux - rust-installer: false - - scenario: on macos - runs-on: macos-14 - os: darwin - rust-installer: false - - scenario: on ubuntu (rust) - runs-on: ubuntu-24.04 - os: linux - rust-installer: true - - scenario: on macos (rust) - runs-on: macos-14 - os: darwin - rust-installer: true - name: installer test ${{ matrix.scenario }} - runs-on: ${{ matrix.runs-on }} - steps: - - uses: actions/checkout@v6 - - name: Download installer tarball - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - name: installer-${{matrix.os}} - path: out - - name: Looking up the installer tarball URL - id: installer-tarball-url - run: echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT" - - uses: cachix/install-nix-action@8aa03977d8d733052d78f4e008a241fd1dbf36b3 # v31.10.6 - if: ${{ !matrix.rust-installer }} - with: - install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }} - install_options: ${{ format('--tarball-url-prefix {0}', steps.installer-tarball-url.outputs.installer-url) }} - - name: Run rust installer - if: ${{ matrix.rust-installer }} - run: | - chmod +x out/nix-installer - ./out/nix-installer install --no-confirm - env: - RUST_BACKTRACE: full - - run: sudo apt install fish zsh - if: matrix.os == 'linux' - - run: brew install fish - if: matrix.os == 'darwin' - - run: exec bash -c "nix-instantiate -E 'builtins.currentTime' --eval" - - run: exec sh -c "nix-instantiate -E 'builtins.currentTime' --eval" - - run: exec zsh -c "nix-instantiate -E 'builtins.currentTime' --eval" - - run: exec fish -c "nix-instantiate -E 'builtins.currentTime' --eval" - - run: exec bash -c "nix-channel --add https://releases.nixos.org/nixos/unstable/nixos-23.05pre466020.60c1d71f2ba nixpkgs" - - run: exec bash -c "nix-channel --update && nix-env -iA nixpkgs.hello && hello" + - name: Fetch artifacts + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + path: downloaded + - name: Move downloaded artifacts to artifacts directory + run: | + shopt -s nullglob + for dir in ./downloaded/*; do + arch="$(basename "$dir")" + mv "$dir"/*.xz ./artifacts/"${arch}" + done - clang-tidy: - needs: basic-checks - runs-on: ubuntu-24.04 - name: clang-tidy - steps: - - uses: actions/checkout@v6 - with: - fetch-depth: 0 - - uses: ./.github/actions/install-nix-action - with: - github_token: ${{ secrets.GITHUB_TOKEN }} - dogfood: false - extra_nix_config: "sandbox = true" - - run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 - - name: Run clang-tidy - run: nix build .#hydraJobs.clangTidy.x86_64-linux -L + - name: Build fallback-paths.nix + id: build + if: ${{ + github.event_name != 'pull_request' + || ( + github.event.pull_request.head.repo.full_name == 'DeterminateSystems/nix-src' + && ( + (github.event.action == 'labeled' && github.event.label.name == 'upload to s3') + || (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, 'upload to s3')) + ) + ) + }} + env: + INCLUDE_NO_DSP: ${{ + needs.build_x86_64-linux_no_dsp.result == 'success' + && needs.build_aarch64-linux_no_dsp.result == 'success' + && needs.build_aarch64-darwin_no_dsp.result == 'success' + }} + run: | + for system in x86_64-linux aarch64-linux aarch64-darwin; do + nix build ./packaging/secure-packages#packages."$system".default^out,dev,doc,man --out-link /tmp/outputs/"$system"-dsp -j0 + [[ "$INCLUDE_NO_DSP" == "true" ]] && nix build .#packages."$system".default^out,dev,doc,man --out-link /tmp/outputs/"$system"-non-dsp -j0 + done + [[ "$INCLUDE_NO_DSP" == "true" ]] && nix build .#fallbackPathsNix --out-link /tmp/outputs/fallback-non-dsp + nix build ./packaging/secure-packages#fallbackPathsNix --out-link /tmp/outputs/fallback-dsp + cat /tmp/outputs/fallback-dsp > ./artifacts/fallback-paths.nix - flake_regressions: - needs: tests - runs-on: ubuntu-24.04 - steps: - - name: Checkout nix - uses: actions/checkout@v6 - - name: Checkout flake-regressions - uses: actions/checkout@v6 + outputs=() + for path in /tmp/outputs/*; do + outputs+=("$(readlink $path)") + done + echo "nix-outputs=${outputs[@]}" >> "$GITHUB_OUTPUT" + + - uses: DeterminateSystems/public-slice-mirror/.github/actions/upload-store-paths@main with: - repository: NixOS/flake-regressions - path: flake-regressions - - name: Checkout flake-regressions-data - uses: actions/checkout@v6 + store-paths: | + ${{ steps.build.outputs.nix-outputs }} + + - uses: DeterminateSystems/push-artifact-ids@main with: - repository: NixOS/flake-regressions-data - path: flake-regressions/tests - - name: Download installer tarball - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + s3_upload_role: ${{ secrets.AWS_S3_UPLOAD_ROLE_ARN }} + bucket: ${{ secrets.AWS_S3_UPLOAD_BUCKET_NAME }} + directory: ./artifacts + ids_project_name: determinate-nix + ids_binary_prefix: determinate-nix + skip_acl: true + allowed_branches: '["main"]' + outputs: + nix-outputs: ${{ steps.build.outputs.nix-outputs }} + + publish: + needs: + - success + # NOTE(cole-h): we need always() here because if any of the needs jobs are skipped (this is + # transitive as well, so if any of success's needs jobs are skipped, like the *_no_dsp ones + # usually are on release), this gets skipped too... + if: (always() && github.repository == 'DeterminateSystems/nix-src' && (github.ref == format('refs/heads/{0}', github.event.repository.default_branch) || startsWith(github.ref, 'refs/tags/'))) + environment: ${{ github.event_name == 'release' && 'production' || '' }} + runs-on: ubuntu-latest + permissions: + contents: write + id-token: write + steps: + - uses: actions/checkout@v4 + - uses: DeterminateSystems/determinate-nix-action@main + - uses: DeterminateSystems/flakehub-push@main with: - name: installer-linux - path: out - - name: Looking up the installer tarball URL - id: installer-tarball-url + rolling: ${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }} + visibility: "public" + tag: "${{ github.ref_name }}" + include-output-paths: true + - name: Ensure that we actually released + if: startsWith(github.ref, 'refs/tags/') + env: + TAG_NAME: ${{ github.ref_name }} run: | - echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT" - - uses: cachix/install-nix-action@8aa03977d8d733052d78f4e008a241fd1dbf36b3 # v31.10.6 - with: - install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }} - install_options: ${{ format('--tarball-url-prefix {0}', steps.installer-tarball-url.outputs.installer-url) }} - - name: Run flake regressions tests + nix flake metadata https://flakehub.com/f/DeterminateSystems/nix-src/"$TAG_NAME" + - name: Update the release notes + if: startsWith(github.ref, 'refs/tags/') + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + TAG_NAME: ${{ github.ref_name }} run: | - touch flake-regressions/tests/{0utkarsh/nixconfig,HariAmoor-professional/nixos-config}/disabled - MAX_FLAKES=25 flake-regressions/eval-all.sh + gh release edit "$TAG_NAME" --notes-file doc/manual/source/release-notes-determinate/"$TAG_NAME".md || true - profile_build: - needs: tests - runs-on: ubuntu-24.04 - timeout-minutes: 60 - if: >- - github.event_name == 'push' && - github.ref_name == 'master' - steps: - - uses: actions/checkout@v6 - with: - fetch-depth: 0 - - uses: ./.github/actions/install-nix-action - with: - github_token: ${{ secrets.GITHUB_TOKEN }} - dogfood: false - extra_nix_config: | - experimental-features = flakes nix-command ca-derivations impure-derivations - max-jobs = 1 - - run: | - nix build -L --file ./ci/gha/profile-build buildTimeReport --out-link build-time-report.md - cat build-time-report.md >> $GITHUB_STEP_SUMMARY + mirror: + needs: success + permissions: + id-token: write # required so the workflow can auth to FlakeHub + contents: read + if: ${{ + always() + && ( + github.ref == format('refs/heads/{0}', github.event.repository.default_branch) + || startsWith(github.ref, 'refs/tags/') + || ( + github.event_name == 'pull_request' + && github.event.pull_request.head.repo.full_name == 'DeterminateSystems/nix-src' + && ( + (github.event.action == 'labeled' && github.event.label.name == 'upload to s3') + || (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, 'upload to s3')) + ) + ) + ) + }} + uses: DeterminateSystems/public-slice-mirror/.github/workflows/subset-mirror.yml@main + with: + flake-name: DeterminateSystems/nix-artifacts + rolling: ${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }} + tag: "${{ startsWith(github.ref, 'refs/tags/') && github.ref_name || '' }}" + checkout-ref: "${{ github.ref }}" + flakehub-push: ${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) || startsWith(github.ref, 'refs/tags/') }} diff --git a/.github/workflows/labels.yml b/.github/workflows/labels.yml deleted file mode 100644 index 16038cb213ca..000000000000 --- a/.github/workflows/labels.yml +++ /dev/null @@ -1,24 +0,0 @@ -name: "Label PR" - -on: - pull_request_target: - types: [edited, opened, synchronize, reopened] - -# WARNING: -# When extending this action, be aware that $GITHUB_TOKEN allows some write -# access to the GitHub API. This means that it should not evaluate user input in -# a way that allows code injection. - -permissions: - contents: read - pull-requests: write - -jobs: - labels: - runs-on: ubuntu-24.04 - if: github.repository_owner == 'NixOS' - steps: - - uses: actions/labeler@v6 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - sync-labels: false diff --git a/.github/workflows/propose-release.yml b/.github/workflows/propose-release.yml new file mode 100644 index 000000000000..7593dc9e8510 --- /dev/null +++ b/.github/workflows/propose-release.yml @@ -0,0 +1,33 @@ +on: + workflow_dispatch: + inputs: + reference-id: + type: string + required: true + version: + type: string + required: true + +concurrency: + group: ${{ github.workflow }} + cancel-in-progress: true + +jobs: + propose-release: + uses: DeterminateSystems/propose-release/.github/workflows/workflow.yml@main + permissions: + id-token: write + contents: write + pull-requests: write + with: + update-flake: false + reference-id: ${{ inputs.reference-id }} + version: ${{ inputs.version }} + extra-commands-early: | + nix flake update --commit-lock-file --flake ./packaging/secure-packages + echo ${{ inputs.version }} > .version-determinate + git add .version-determinate + git commit -m "Set .version-determinate to ${{ inputs.version }}" || true + ./.github/release-notes.sh + git add doc + git commit -m "Generate release notes for ${{ inputs.version }}" || true diff --git a/.github/workflows/upload-release.yml b/.github/workflows/upload-release.yml deleted file mode 100644 index 9cf01e53c59f..000000000000 --- a/.github/workflows/upload-release.yml +++ /dev/null @@ -1,80 +0,0 @@ -name: Upload Release -on: - workflow_dispatch: - inputs: - eval_id: - description: "Hydra evaluation ID (from the maintenance-X.Y-release jobset)" - required: true - type: number - is_latest: - description: "Mark as latest release" - required: false - type: boolean - default: false -permissions: - contents: read - id-token: write - packages: write -jobs: - release: - runs-on: ubuntu-24.04 - environment: releases - steps: - - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - - uses: ./.github/actions/install-nix-action - with: - dogfood: false # Use stable version - use_cache: false # Don't want any cache injection shenanigans - extra_nix_config: | - experimental-features = nix-command flakes - - name: Set NIX_PATH from flake input - run: | - NIXPKGS_PATH=$(nix build --inputs-from .# nixpkgs#path --print-out-paths --no-link) - # Shebangs with perl have issues. Pin nixpkgs this way. nix shell should maybe - # get the same uberhack that nix-shell has to support it. - echo "NIX_PATH=nixpkgs=$NIXPKGS_PATH" >> "$GITHUB_ENV" - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e7f100cf4c008499ea8adda475de1042d6975c7b # v6.2.0 - with: - role-to-assume: "arn:aws:iam::080433136561:role/nix-release" - role-session-name: nix-release-oidc-${{ github.run_id }} - aws-region: eu-west-1 - - name: Disable containerd image store - run: | - # Docker 28+ defaults to the containerd image store, which - # pushes layers uncompressed instead of gzip. OCI clients - # that only support gzip (e.g. go-containerregistry) fail - # with "gzip: invalid header". Disabling the containerd - # snapshotter restores the classic storage driver, which - # preserves gzip-compressed layers through the - # `docker load` / `docker push` pipeline. - echo '{"features":{"containerd-snapshotter":false}}' | sudo tee /etc/docker/daemon.json > /dev/null - sudo systemctl restart docker - - name: Login to Docker Hub - uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v4.0.0 - with: - username: ${{ secrets.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} - - name: Login to GitHub Container Registry - uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v4.0.0 - with: - registry: ghcr.io - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - name: Upload release - run: | - ./maintainers/upload-release.pl \ - ${{ inputs.eval_id }} \ - --skip-git - env: - IS_LATEST: ${{ inputs.is_latest && '1' || '' }} - - name: Push to GHCR - run: | - DOCKER_OWNER="ghcr.io/$(echo '${{ github.repository_owner }}' | tr '[A-Z]' '[a-z]')/nix" - ./maintainers/upload-release.pl \ - ${{ inputs.eval_id }} \ - --skip-git \ - --skip-s3 \ - --docker-owner "$DOCKER_OWNER" - env: - IS_LATEST: ${{ inputs.is_latest && '1' || '' }} diff --git a/.version b/.version index c819688d6b53..c0ac27fe010c 100644 --- a/.version +++ b/.version @@ -1 +1 @@ -2.35.3 +2.35.2 diff --git a/.version-determinate b/.version-determinate new file mode 100644 index 000000000000..9b2f2a16885d --- /dev/null +++ b/.version-determinate @@ -0,0 +1 @@ +3.23.1 diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000000..a8a71cad047f --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,25 @@ +Observe the following rules when contributing to this repository: + +* Prefer using the Edit tool over external tools. + +* Before committing, run ./maintainers/format.sh to detect/fix any formatting issues. + +* Use "Assisted-by:" instead of "Co-Authored-By:" for the Claude trailer in commits. + +* Create branches for PRs using `git switch -c --no-track detsys/main`, and push them using `git push -u detsys :`. Never push to `main`. + +* Do not create PRs unless prompted. Create PRs using `gh pr create --repo DeterminateSystems/nix-src --base main`, observing `.github/PULL_REQUEST_TEMPLATE.md`. + +* Keep PR descriptions concise. Detailed information is better put in commit messages. + +* The code base uses C++23, so C++23 features (e.g. deducing-this lambdas) can be used freely. + +Building and testing Nix: + +* Build Nix using `nix build ./packaging/nix-make#release.nix`. The result is in `./result`. + +* Run all tests using `nix build -L ./packaging/nix-make#release.nix-all-tests`. This covers the unit tests (e.g. `release.nix-util-tests-run`, which runs `src/libutil-tests`) and the functional tests in `tests/functional` (`release.functional-tests`); each can also be built on its own. + +* Individual tests can be executed by doing `nix build ./packaging/nix-make#release.nix && nix run ./packaging/nix-make#test-runner -- --nix-bin-dir ./result/bin $TEST_NAMES`, where `TEST_NAMES` are the file names inside `tests/functional` (without the `.sh` suffix), e.g. `simple` or `flakes/relative-paths`. + +* If you don't need an optimized build (e.g. you're not benchmarking), then instead of `release` you can use `debug-fast` (clang with -O0), e.g. `nix build -L ./packaging/nix-make#debug-fast.nix`. diff --git a/README.md b/README.md index 02498944cdb7..c5cbcbed21bb 100644 --- a/README.md +++ b/README.md @@ -1,38 +1,111 @@ -# Nix +

+ +

+

+  Discord  +  Bluesky  +  Mastodon  +  Twitter  +  LinkedIn  +

-[![Open Collective supporters](https://opencollective.com/nixos/tiers/supporter/badge.svg?label=Supporters&color=brightgreen)](https://opencollective.com/nixos) -[![CI](https://github.com/NixOS/nix/workflows/CI/badge.svg)](https://github.com/NixOS/nix/actions/workflows/ci.yml) +# The Determinate Nix CLI -Nix is a powerful package manager for Linux and other Unix systems that makes package -management reliable and reproducible. Please refer to the [Nix manual](https://nix.dev/reference/nix-manual) -for more details. +[![CI](https://github.com/DeterminateSystems/nix-src/workflows/CI/badge.svg)](https://github.com/DeterminateSystems/nix-src/actions/workflows/ci.yml) -## Installation and first steps +**Nix** is a powerful [language], [package manager][package-management], and [build tool][cli] for [macOS](#macos), [Linux](#linux), and other Unix systems. +It enables you to create fully reproducible [development environments][envs], to build [packages] in sandboxed environments, to build entire Linux systems using [NixOS], and much more. -Visit [nix.dev](https://nix.dev) for [installation instructions](https://nix.dev/tutorials/install-nix) and [beginner tutorials](https://nix.dev/tutorials/first-steps). +[**Determinate Nix**][det-nix] is a downstream distribution of [Nix][upstream] created and maintained by [Determinate Systems][detsys]. +It has two components: -Full reference documentation can be found in the [Nix manual](https://nix.dev/reference/nix-manual). +- The Determinate Nix CLI, a distribution of the Nix CLI built from this repository. + It's based on the [upstream Nix CLI][upstream] and continuously rebased against it, but adds a wide variety of [features] and [improvements][changelog]. +- [Determinate Nixd][dnixd] is a useful daemon for Linux and macOS that handles vital tasks like configuration and enterprise certificate management. -## Building and developing +Determinate Nix is built on SOC-2-Type-II-compliant infrastructure using [Determinate Secure Packages][secure-packages], released via a carefully orchestrated process, and, for Determinate Systems customers, backed by formal security response SLAs that meet stringent compliance standards. -Follow instructions in the Nix reference manual to [set up a development environment and build Nix from source](https://nix.dev/manual/nix/development/development/building.html). +> [!NOTE] +> Determinate Nix, by definition, consists of _both_ the components listed above. +> While it's possible to use the code in this repository to run just our downstream Nix CLI, we do _not_ officially support this experience and provide none of the guarantees or SLAs that we provide for Determinate Nix proper. -## Contributing +Determinate Nix is part of the [Determinate platform][determinate], which also includes [FlakeHub], a secure flake repository with features like [FlakeHub Cache][cache], [private flakes][private-flakes], and [semantic versioning][semver] (SemVer) for [flakes]. + +## Installing Determinate Nix + +You can install Determinate Nix on [macOS](#macos), non-NixOS [Linux](#linux) and WSL, and [NixOS](#nixos). + +### macOS + +On macOS, we recommend using the graphical installer from Determinate Systems. +Click [here][gui] to download and run it. + +### Linux + +On Linux, including Windows Subsystem for Linux (WSL), we recommend installing Determinate Nix using [Determinate Nix Installer][installer]: + +```shell +curl -fsSL https://install.determinate.systems/nix | sh -s -- install +``` -Check the [contributing guide](./CONTRIBUTING.md) if you want to get involved with developing Nix. +### NixOS -## Additional resources +On [NixOS], we recommend following our [dedicated installation guide][nixos-install]. +We also provide both [Amazon Machine Images][amis] (AMIs) and [ISOs] for using Determinate on NixOS. -Nix was created by Eelco Dolstra and developed as the subject of his PhD thesis [The Purely Functional Software Deployment Model](https://edolstra.github.io/pubs/phd-thesis.pdf), published 2006. -Today, a world-wide developer community contributes to Nix and the ecosystem that has grown around it. +## Other resources -- [The Nix, Nixpkgs, NixOS Community on nixos.org](https://nixos.org/) -- [Official documentation on nix.dev](https://nix.dev) -- [Nixpkgs](https://github.com/NixOS/nixpkgs) is [the largest, most up-to-date free software repository in the world](https://repology.org/repositories/graphs) -- [NixOS](https://github.com/NixOS/nixpkgs/tree/master/nixos) is a Linux distribution that can be configured fully declaratively -- [Discourse](https://discourse.nixos.org/) -- Matrix: [#users:nixos.org](https://matrix.to/#/#users:nixos.org) for user support and [#nix-dev:nixos.org](https://matrix.to/#/#nix-dev:nixos.org) for development +Nix was created by [Eelco Dolstra][eelco] and developed as the subject of his 2006 PhD thesis, [The Purely Functional Software Deployment Model][thesis]. +Today, a worldwide developer community contributes to Nix and the ecosystem that has grown around it. + +- [Zero to Nix][z2n], Determinate Systems' guide to Nix and [flakes] for beginners +- [Nixpkgs], a collection of well over 100,000 software packages that you can build and manage using Nix +- [NixOS] is a Linux distribution that can be configured fully declaratively +- The Nix, Nixpkgs, and NixOS community on [nixos.org][website] + +## Reference + +The primary documentation for Determinate and Determinate Nix is available at [docs.determinate.systems][determinate]. +For deeply technical reference material, see the [Determinate Nix manual][manual] which is based on the upstream Nix manual. ## License -Nix is released under the [LGPL v2.1](./COPYING). +[Upstream Nix][upstream] is released under the [LGPL v2.1][license] license. +[Determinate Nix][det-nix] is also released under LGPL v2.1 in accordance with the terms of the upstream license. + +## Contributing + +Check the [contributing guide][contributing] if you want to get involved with developing Nix. + +[amis]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html +[cache]: https://docs.determinate.systems/flakehub/cache +[changelog]: https://determinate.systems/blog/categories/changelog +[cli]: https://manual.determinate.systems/command-ref/new-cli/nix.html +[contributing]: ./CONTRIBUTING.md +[det-nix]: https://docs.determinate.systems/determinate-nix +[determinate]: https://docs.determinate.systems +[detsys]: https://determinate.systems +[dnixd]: https://docs.determinate.systems/determinate-nix#determinate-nixd +[eelco]: https://determinate.systems/people/eelco-dolstra +[envs]: https://zero-to-nix.com/concepts/dev-env +[features]: https://docs.determinate.systems/determinate-nix/#special-features +[flakehub]: https://flakehub.com +[flakes]: https://zero-to-nix.com/concepts/flakes +[gui]: https://install.determinate.systems/determinate-pkg/stable/Universal +[installer]: https://github.com/DeterminateSystems/nix-installer +[isos]: https://github.com/DeterminateSystems/nixos-iso +[language]: https://zero-to-nix.com/concepts/nix-language +[license]: ./COPYING +[manual]: https://manual.determinate.systems +[nixpkgs]: https://github.com/NixOS/nixpkgs +[nixos]: https://github.com/NixOS/nixpkgs/tree/master/nixos +[nixos-install]: https://docs.determinate.systems/guides/advanced-installation#nixos +[packages]: https://zero-to-nix.com/concepts/packages +[package-management]: https://zero-to-nix.com/concepts/package-management +[private-flakes]: https://docs.determinate.systems/flakehub/private-flakes +[secure-packages]: https://determinate.systems/secure-packages +[semver]: https://docs.determinate.systems/flakehub/concepts/semver +[thesis]: https://edolstra.github.io/pubs/phd-thesis.pdf +[upstream]: https://github.com/NixOS/nix +[website]: https://nixos.org +[z2n]: https://zero-to-nix.com diff --git a/ci/gha/tests/default.nix b/ci/gha/tests/default.nix index 8d2383a92fb9..94327adc9f0d 100644 --- a/ci/gha/tests/default.nix +++ b/ci/gha/tests/default.nix @@ -74,14 +74,19 @@ rec { */ topLevel = { installerScriptForGHA = hydraJobs.installerScriptForGHA.${system}; - installTests = hydraJobs.installTests.${system}; nixpkgsLibTests = hydraJobs.tests.nixpkgsLibTests.${system}; + nixpkgsLibTestsLazy = hydraJobs.tests.nixpkgsLibTestsLazy.${system}; filetransfer-retry-backoff = hydraJobs.tests.filetransfer-retry-backoff.${system}; rl-next = pkgs.buildPackages.runCommand "test-rl-next-release-notes" { } '' LANG=C.UTF-8 ${pkgs.changelog-d}/bin/changelog-d ${../../../doc/manual/rl-next} >$out ''; repl-completion = pkgs.callPackage ../../../tests/repl-completion.nix { inherit (packages') nix; }; + lazyTrees = nixComponents.nix-functional-tests.override { + pname = "nix-lazy-trees-tests"; + lazyTrees = true; + }; + /** Checks for our packaging expressions. This shouldn't build anything significant; just check that things diff --git a/default.nix b/default.nix deleted file mode 100644 index 6466507b7140..000000000000 --- a/default.nix +++ /dev/null @@ -1,9 +0,0 @@ -(import ( - let - lock = builtins.fromJSON (builtins.readFile ./flake.lock); - in - fetchTarball { - url = "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz"; - sha256 = lock.nodes.flake-compat.locked.narHash; - } -) { src = ./.; }).defaultNix diff --git a/doc/manual/book.toml.in b/doc/manual/book.toml.in index c798afc4a8c0..11efca75f110 100644 --- a/doc/manual/book.toml.in +++ b/doc/manual/book.toml.in @@ -1,12 +1,12 @@ [book] -title = "Nix @version@ Reference Manual" +title = "Determinate Nix @version@ Reference Manual" src = "source" [output.html] additional-css = ["custom.css"] additional-js = ["redirects.js"] -edit-url-template = "https://github.com/NixOS/nix/tree/master/doc/manual/{path}" -git-repository-url = "https://github.com/NixOS/nix" +edit-url-template = "https://github.com/DeterminateSystems/nix-src/tree/master/doc/manual/{path}" +git-repository-url = "https://github.com/DeterminateSystems/nix-src" mathjax-support = true # Handles replacing @docroot@ with a path to ./source relative to that markdown file, diff --git a/doc/manual/custom.css b/doc/manual/custom.css index 7af150be391b..119c6d125430 100644 --- a/doc/manual/custom.css +++ b/doc/manual/custom.css @@ -1,5 +1,5 @@ :root { - --sidebar-width: 23em; + --sidebar-width: 23em; } h1.menu-title::before { @@ -7,11 +7,10 @@ h1.menu-title::before { background-image: url("./favicon.svg"); padding: 1.25em; background-position: center center; - background-size: 2em; + background-size: 1.5em; background-repeat: no-repeat; } - .menu-bar { padding: 0.5em 0em; } @@ -21,13 +20,13 @@ h1.menu-title::before { } h1:not(:first-of-type) { - margin-top: 1.3em; + margin-top: 1.3em; } h2 { - margin-top: 1em; + margin-top: 1em; } .hljs-meta { - user-select: none; + user-select: none; } diff --git a/doc/manual/generate-manpage.nix b/doc/manual/generate-manpage.nix index 31e74e17d264..7b2a02bd3f84 100644 --- a/doc/manual/generate-manpage.nix +++ b/doc/manual/generate-manpage.nix @@ -42,11 +42,6 @@ let let result = '' - > **Warning** \ - > This program is - > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command) - > and its interface is subject to change. - # Name `${command}` - ${details.description} @@ -81,7 +76,11 @@ let subcommands = if length categories > 1 then listCategories else listSubcommands details.commands; categories = sort (x: y: x.id < y.id) ( - unique (map (cmd: cmd.category) (attrValues details.commands)) + unique ( + map (cmd: { inherit (cmd.category) id description; }) ( + builtins.filter (cmd: cmd.category.id != 103) (attrValues details.commands) + ) + ) ); listCategories = concatStrings (map showCategory categories); @@ -89,7 +88,7 @@ let showCategory = cat: '' **${toString cat.description}:** - ${listSubcommands (filterAttrs (n: v: v.category == cat) details.commands)} + ${listSubcommands (filterAttrs (n: v: v.category.id == cat.id) details.commands)} ''; listSubcommands = cmds: concatStrings (attrValues (mapAttrs showSubcommand cmds)); diff --git a/doc/manual/meson.build b/doc/manual/meson.build index f8c37b75b301..1edf11afc1f4 100644 --- a/doc/manual/meson.build +++ b/doc/manual/meson.build @@ -5,19 +5,9 @@ project( license : 'LGPL-2.1-or-later', ) -# Compute documentation URL based on version and release type -version = meson.project_version() -official_release = get_option('official-release') +fs = import('fs') -if official_release - # For official releases, use versioned URL (dropping patch version) - version_parts = version.split('.') - major_minor = '@0@.@1@'.format(version_parts[0], version_parts[1]) - doc_url = 'https://nix.dev/manual/nix/@0@'.format(major_minor) -else - # For development builds, use /latest - doc_url = 'https://nix.dev/manual/nix/latest' -endif +doc_url = 'https://manual.determinate.systems/' nix = find_program('nix', native : true) @@ -39,7 +29,7 @@ nix_env_for_docs = { 'NIX_CONFIG' : 'cores = 0', } -nix_for_docs = [ nix, '--experimental-features', 'nix-command' ] +nix_for_docs = [ nix ] nix_eval_for_docs_common = nix_for_docs + [ 'eval', '-I', @@ -141,7 +131,7 @@ if get_option('html-manual') python.full_path(), mdbook.full_path(), meson.current_build_dir(), - meson.project_version(), + fs.read('../../.version-determinate').strip(), ), ], input : [ @@ -303,6 +293,8 @@ nix3_manpages = [ 'nix3-help', 'nix3-help-stores', 'nix3-key', + 'nix3-key-convert-public-to-pem', + 'nix3-key-convert-secret-to-pem', 'nix3-key-convert-secret-to-public', 'nix3-key-generate-secret', 'nix3-log', @@ -311,6 +303,10 @@ nix3_manpages = [ 'nix3-nar-dump-path', 'nix3-nar-ls', 'nix3-nar-pack', + 'nix3-nario', + 'nix3-nario-export', + 'nix3-nario-import', + 'nix3-nario-list', 'nix3-path-info', 'nix3-print-dev-env', 'nix3-profile', @@ -322,6 +318,10 @@ nix3_manpages = [ 'nix3-profile-rollback', 'nix3-profile-upgrade', 'nix3-profile-wipe-history', + 'nix3-provenance', + 'nix3-provenance-show', + 'nix3-provenance-verify', + 'nix3-ps', 'nix3-registry', 'nix3-registry-add', 'nix3-registry-list', @@ -331,6 +331,7 @@ nix3_manpages = [ 'nix3-repl', 'nix3-run', 'nix3-search', + 'nix3-serve', 'nix3-store', 'nix3-store-add', 'nix3-store-add-file', diff --git a/doc/manual/package.nix b/doc/manual/package.nix index 817b05dbc1c0..c66163e285e3 100644 --- a/doc/manual/package.nix +++ b/doc/manual/package.nix @@ -34,7 +34,7 @@ let in mkMesonDerivation (finalAttrs: { - pname = "nix-manual"; + pname = "determinate-nix-manual"; inherit version; workDir = ./.; @@ -42,6 +42,7 @@ mkMesonDerivation (finalAttrs: { fileset.difference (fileset.unions [ ../../.version + ../../.version-determinate # For example JSON ../../src/libutil-tests/data/memory-source-accessor ../../src/libutil-tests/data/hash diff --git a/doc/manual/redirects.json b/doc/manual/redirects.json index 04e14aacc3d5..76ce242178f3 100644 --- a/doc/manual/redirects.json +++ b/doc/manual/redirects.json @@ -243,29 +243,11 @@ "gloss-validity": "glossary.html#gloss-validity", "part-glossary": "glossary.html", "sec-building-source": "installation/building-source.html", - "ch-env-variables": "installation/env-variables.html", - "sec-installer-proxy-settings": "installation/env-variables.html#proxy-environment-variables", - "sec-nix-ssl-cert-file": "installation/env-variables.html#nix_ssl_cert_file", - "sec-nix-ssl-cert-file-with-nix-daemon-and-macos": "installation/env-variables.html#nix_ssl_cert_file", "chap-installation": "installation/index.html", - "ch-installing-binary": "installation/installing-binary.html", - "sect-macos-installation": "installation/installing-binary.html#macos-installation", - "sect-macos-installation-change-store-prefix": "installation/installing-binary.html#macos-installation", - "sect-macos-installation-encrypted-volume": "installation/installing-binary.html#macos-installation", - "sect-macos-installation-recommended-notes": "installation/installing-binary.html#macos-installation", - "sect-macos-installation-symlink": "installation/installing-binary.html#macos-installation", - "sect-multi-user-installation": "installation/installing-binary.html#multi-user-installation", - "sect-nix-install-binary-tarball": "installation/installing-binary.html#installing-from-a-binary-tarball", - "sect-nix-install-pinned-version-url": - "installation/installing-binary.html#installing-a-pinned-nix-version-from-a-url", - "sect-single-user-installation": "installation/installing-binary.html#single-user-installation", "ch-installing-source": "installation/installing-source.html", - "ssec-multi-user": "installation/multi-user.html", "ch-nix-security": "installation/nix-security.html", "sec-obtaining-source": "installation/obtaining-source.html", "sec-prerequisites-source": "installation/prerequisites-source.html", - "sec-single-user": "installation/single-user.html", - "ch-supported-platforms": "installation/supported-platforms.html", "ch-upgrading-nix": "installation/upgrading.html", "ch-about-nix": "introduction.html", "chap-introduction": "introduction.html", @@ -287,43 +269,7 @@ "sec-sharing-packages": "package-management/sharing-packages.html", "ssec-ssh-substituter": "package-management/ssh-substituter.html", "chap-quick-start": "quick-start.html", - "sec-relnotes": "release-notes/index.html", - "ch-relnotes-0.10.1": "release-notes/rl-0.10.1.html", - "ch-relnotes-0.10": "release-notes/rl-0.10.html", - "ssec-relnotes-0.11": "release-notes/rl-0.11.html", - "ssec-relnotes-0.12": "release-notes/rl-0.12.html", - "ssec-relnotes-0.13": "release-notes/rl-0.13.html", - "ssec-relnotes-0.14": "release-notes/rl-0.14.html", - "ssec-relnotes-0.15": "release-notes/rl-0.15.html", - "ssec-relnotes-0.16": "release-notes/rl-0.16.html", - "ch-relnotes-0.5": "release-notes/rl-0.5.html", - "ch-relnotes-0.6": "release-notes/rl-0.6.html", - "ch-relnotes-0.7": "release-notes/rl-0.7.html", - "ch-relnotes-0.8.1": "release-notes/rl-0.8.1.html", - "ch-relnotes-0.8": "release-notes/rl-0.8.html", - "ch-relnotes-0.9.1": "release-notes/rl-0.9.1.html", - "ch-relnotes-0.9.2": "release-notes/rl-0.9.2.html", - "ch-relnotes-0.9": "release-notes/rl-0.9.html", - "ssec-relnotes-1.0": "release-notes/rl-1.0.html", - "ssec-relnotes-1.1": "release-notes/rl-1.1.html", - "ssec-relnotes-1.10": "release-notes/rl-1.10.html", - "ssec-relnotes-1.11.10": "release-notes/rl-1.11.10.html", - "ssec-relnotes-1.11": "release-notes/rl-1.11.html", - "ssec-relnotes-1.2": "release-notes/rl-1.2.html", - "ssec-relnotes-1.3": "release-notes/rl-1.3.html", - "ssec-relnotes-1.4": "release-notes/rl-1.4.html", - "ssec-relnotes-1.5.1": "release-notes/rl-1.5.html", - "ssec-relnotes-1.5.2": "release-notes/rl-1.5.2.html", - "ssec-relnotes-1.5": "release-notes/rl-1.5.html", - "ssec-relnotes-1.6.1": "release-notes/rl-1.6.1.html", - "ssec-relnotes-1.6.0": "release-notes/rl-1.6.html", - "ssec-relnotes-1.7": "release-notes/rl-1.7.html", - "ssec-relnotes-1.8": "release-notes/rl-1.8.html", - "ssec-relnotes-1.9": "release-notes/rl-1.9.html", - "ssec-relnotes-2.0": "release-notes/rl-2.0.html", - "ssec-relnotes-2.1": "release-notes/rl-2.1.html", - "ssec-relnotes-2.2": "release-notes/rl-2.2.html", - "ssec-relnotes-2.3": "release-notes/rl-2.3.html" + "sec-relnotes": "release-notes/index.html" }, "language/types.html": { "simple-values": "#primitives", @@ -340,12 +286,10 @@ "builder-execution": "../store/building.html" }, "installation/installing-binary.html": { - "linux": "uninstall.html#linux", - "macos": "uninstall.html#macos", "uninstalling": "uninstall.html" }, "development/building.html": { - "nix-with-flakes": "#building-nix-with-flakes", + "nix-with-flakes": "#building-nix", "classic-nix": "#building-nix", "running-tests": "testing.html#running-tests", "unit-tests": "testing.html#unit-tests", diff --git a/doc/manual/rl-next/cgroup-delegate-controllers.md b/doc/manual/rl-next/cgroup-delegate-controllers.md new file mode 100644 index 000000000000..9dd82e6cb0c1 --- /dev/null +++ b/doc/manual/rl-next/cgroup-delegate-controllers.md @@ -0,0 +1,11 @@ +--- +synopsis: "Daemon enables cgroup controllers for build cgroups" +prs: [16024] +issues: [9675] +--- + +With `use-cgroups`, the daemon now enables the `cpu`, `memory`, `io` and `pids` +controllers in its cgroup's `cgroup.subtree_control`, so per-build cgroups +expose `memory.peak`, `io.stat` etc. instead of only `cpu.stat`. This requires +the controllers to be delegated to the daemon's cgroup (e.g. systemd +`Delegate=yes`). diff --git a/doc/manual/rl-next/shorter-build-dir-names.md b/doc/manual/rl-next/shorter-build-dir-names.md new file mode 100644 index 000000000000..e87fa5d04fb8 --- /dev/null +++ b/doc/manual/rl-next/shorter-build-dir-names.md @@ -0,0 +1,6 @@ +--- +synopsis: "Temporary build directories no longer include derivation names" +prs: [13839] +--- + +Temporary build directories created during derivation builds no longer include the derivation name in their path to avoid build failures when the derivation name is too long. This change ensures predictable prefix lengths for build directories under `/nix/var/nix/builds`. \ No newline at end of file diff --git a/doc/manual/source/SUMMARY.md.in b/doc/manual/source/SUMMARY.md.in index ae77759f5516..2e522d51d7f8 100644 --- a/doc/manual/source/SUMMARY.md.in +++ b/doc/manual/source/SUMMARY.md.in @@ -3,17 +3,12 @@ - [Introduction](introduction.md) - [Quick Start](quick-start.md) - [Installation](installation/index.md) - - [Supported Platforms](installation/supported-platforms.md) - - [Installing a Binary Distribution](installation/installing-binary.md) - [Installing Nix from Source](installation/installing-source.md) - [Prerequisites](installation/prerequisites-source.md) - [Obtaining a Source Distribution](installation/obtaining-source.md) - [Building Nix from Source](installation/building-source.md) - [Using Nix within Docker](installation/installing-docker.md) - [Security](installation/nix-security.md) - - [Single-User Mode](installation/single-user.md) - - [Multi-User Mode](installation/multi-user.md) - - [Environment Variables](installation/env-variables.md) - [Upgrading Nix](installation/upgrading.md) - [Uninstalling Nix](installation/uninstall.md) - [Nix Store](store/index.md) @@ -49,6 +44,7 @@ - [Built-ins](language/builtins.md) - [Derivations](language/derivations.md) - [Advanced Attributes](language/advanced-attributes.md) + - [Derivation metadata](language/derivation-metadata.md) - [Import From Derivation](language/import-from-derivation.md) - [Package Management](package-management/index.md) - [Profiles](package-management/profiles.md) @@ -66,8 +62,11 @@ - [Command Reference](command-ref/index.md) - [Common Options](command-ref/opt-common.md) - [Common Environment Variables](command-ref/env-common.md) - - [Main Commands](command-ref/main-commands.md) + - [Subcommands](command-ref/subcommands.md) +{{#include ./command-ref/new-cli/SUMMARY.md}} + - [Deprecated Commands](command-ref/main-commands.md) - [nix-build](command-ref/nix-build.md) + - [nix-channel](command-ref/nix-channel.md) - [nix-shell](command-ref/nix-shell.md) - [nix-store](command-ref/nix-store.md) - [nix-store --add-fixed](command-ref/nix-store/add-fixed.md) @@ -103,22 +102,17 @@ - [nix-env --uninstall](command-ref/nix-env/uninstall.md) - [nix-env --upgrade](command-ref/nix-env/upgrade.md) - [Utilities](command-ref/utilities.md) - - [nix-channel](command-ref/nix-channel.md) - [nix-collect-garbage](command-ref/nix-collect-garbage.md) - [nix-copy-closure](command-ref/nix-copy-closure.md) - [nix-daemon](command-ref/nix-daemon.md) - [nix-hash](command-ref/nix-hash.md) - [nix-instantiate](command-ref/nix-instantiate.md) - [nix-prefetch-url](command-ref/nix-prefetch-url.md) - - [Experimental Commands](command-ref/experimental-commands.md) -{{#include ./command-ref/new-cli/SUMMARY.md}} - [Files](command-ref/files.md) - [nix.conf](command-ref/conf-file.md) - [Profiles](command-ref/files/profiles.md) - [manifest.nix](command-ref/files/manifest.nix.md) - [manifest.json](command-ref/files/manifest.json.md) - - [Channels](command-ref/files/channels.md) - - [Default Nix expression](command-ref/files/default-nix-expression.md) - [Architecture and Design](architecture/architecture.md) - [Formats and Protocols](protocols/index.md) - [JSON Formats](protocols/json/index.md) @@ -143,6 +137,8 @@ - [`.narinfo` Format](protocols/binary-cache/narinfo.md) - [Derivation "ATerm" file format](protocols/derivation-aterm.md) - [Nix32 Encoding](protocols/nix32.md) + - [`builtins.wasm` Host Interface](protocols/wasm.md) + - [Flake Schemas](protocols/flake-schemas.md) - [C API](c-api.md) - [Glossary](glossary.md) - [Development](development/index.md) @@ -157,7 +153,79 @@ - [Static Analysis](development/static-analysis.md) - [Experimental Features](development/experimental-features.md) - [Contributing](development/contributing.md) -- [Releases](release-notes/index.md) +- [Determinate Nix Release Notes](release-notes-determinate/index.md) + - [Changes between Nix and Determinate Nix](release-notes-determinate/changes.md) + - [Release 3.23.1 (2026-10-05)](release-notes-determinate/v3.23.1.md) + - [Release 3.23.0 (2026-10-01)](release-notes-determinate/v3.23.0.md) + - [Release 3.22.5 (2026-09-17)](release-notes-determinate/v3.22.5.md) + - [Release 3.22.4 (2026-09-16)](release-notes-determinate/v3.22.4.md) + - [Release 3.22.3 (2026-09-02)](release-notes-determinate/v3.22.3.md) + - [Release 3.22.2 (2026-08-21)](release-notes-determinate/v3.22.2.md) + - [Release 3.22.1 (2026-08-18)](release-notes-determinate/v3.22.1.md) + - [Release 3.22.0 (2026-08-06)](release-notes-determinate/v3.22.0.md) + - [Release 3.21.9 (2026-07-30)](release-notes-determinate/v3.21.9.md) + - [Release 3.21.8 (2026-07-20)](release-notes-determinate/v3.21.8.md) + - [Release 3.21.7 (2026-07-15)](release-notes-determinate/v3.21.7.md) + - [Release 3.21.6 (2026-07-14)](release-notes-determinate/v3.21.6.md) + - [Release 3.21.5 (2026-07-08)](release-notes-determinate/v3.21.5.md) + - [Release 3.21.2 (2026-06-18)](release-notes-determinate/v3.21.2.md) + - [Release 3.21.1 (2026-06-08)](release-notes-determinate/v3.21.1.md) + - [Release 3.21.0 (2026-05-22)](release-notes-determinate/v3.21.0.md) + - [Release 3.20.0 (2026-05-07)](release-notes-determinate/v3.20.0.md) + - [Release 3.19.1 (2026-05-04)](release-notes-determinate/v3.19.1.md) + - [Release 3.19.0 (2026-04-28)](release-notes-determinate/v3.19.0.md) + - [Release 3.18.1 (2026-04-23)](release-notes-determinate/v3.18.1.md) + - [Release 3.18.0 (2026-04-20)](release-notes-determinate/v3.18.0.md) + - [Release 3.17.3 (2026-04-07)](release-notes-determinate/v3.17.3.md) + - [Release 3.17.2 (2026-03-27)](release-notes-determinate/v3.17.2.md) + - [Release 3.17.1 (2026-03-18)](release-notes-determinate/v3.17.1.md) + - [Release 3.17.0 (2026-03-04)](release-notes-determinate/v3.17.0.md) + - [Release 3.16.3 (2026-02-24)](release-notes-determinate/v3.16.3.md) + - [Release 3.16.2 (2026-02-23)](release-notes-determinate/v3.16.2.md) + - [Release 3.16.1 (2026-02-22)](release-notes-determinate/v3.16.1.md) + - [Release 3.16.0 (2026-02-12)](release-notes-determinate/v3.16.0.md) + - [Release 3.15.2 (2026-01-20)](release-notes-determinate/v3.15.2.md) + - [Release 3.15.1 (2025-12-24)](release-notes-determinate/v3.15.1.md) + - [Release 3.15.0 (2025-12-19)](release-notes-determinate/v3.15.0.md) + - [Release 3.14.0 (2025-12-08)](release-notes-determinate/v3.14.0.md) + - [Release 3.13.2 (2025-11-19)](release-notes-determinate/v3.13.2.md) + - [Release 3.13.1 (2025-11-12)](release-notes-determinate/v3.13.1.md) + - [Release 3.13.0 (2025-11-09)](release-notes-determinate/v3.13.0.md) + - [Release 3.12.2 (2025-11-05)](release-notes-determinate/v3.12.2.md) + - [Release 3.12.1 (2025-11-04)](release-notes-determinate/v3.12.1.md) + - [Release 3.12.0 (2025-10-23)](release-notes-determinate/v3.12.0.md) + - [Release 3.11.3 (2025-10-09)](release-notes-determinate/v3.11.3.md) + - [Release 3.11.2 (2025-09-12)](release-notes-determinate/v3.11.2.md) + - [Release 3.11.1 (2025-09-04)](release-notes-determinate/v3.11.1.md) + - [Release 3.11.0 (2025-09-03)](release-notes-determinate/v3.11.0.md) + - [Release 3.10.1 (2025-09-02)](release-notes-determinate/v3.10.1.md) + - [Release 3.10.0 (2025-09-02)](release-notes-determinate/v3.10.0.md) + - [Release 3.9.1 (2025-08-28)](release-notes-determinate/v3.9.1.md) + - [Release 3.9.0 (2025-08-26)](release-notes-determinate/v3.9.0.md) + - [Release 3.8.6 (2025-08-19)](release-notes-determinate/v3.8.6.md) + - [Release 3.8.5 (2025-08-04)](release-notes-determinate/rl-3.8.5.md) + - [Release 3.8.4 (2025-07-21)](release-notes-determinate/rl-3.8.4.md) + - [Release 3.8.3 (2025-07-18)](release-notes-determinate/rl-3.8.3.md) + - [Release 3.8.2 (2025-07-12)](release-notes-determinate/rl-3.8.2.md) + - [Release 3.8.1 (2025-07-11)](release-notes-determinate/rl-3.8.1.md) + - [Release 3.8.0 (2025-07-10)](release-notes-determinate/rl-3.8.0.md) + - [Release 3.7.0 (2025-07-03)](release-notes-determinate/rl-3.7.0.md) + - [Release 3.6.8 (2025-06-25)](release-notes-determinate/rl-3.6.8.md) + - [Release 3.6.7 (2025-06-24)](release-notes-determinate/rl-3.6.7.md) + - [Release 3.6.6 (2025-06-17)](release-notes-determinate/rl-3.6.6.md) + - [Release 3.6.5 (2025-06-16)](release-notes-determinate/rl-3.6.5.md) + - [Release 3.6.2 (2025-06-02)](release-notes-determinate/rl-3.6.2.md) + - [Release 3.6.1 (2025-05-24)](release-notes-determinate/rl-3.6.1.md) + - [Release 3.6.0 (2025-05-22)](release-notes-determinate/rl-3.6.0.md) + - [Release 3.5.2 (2025-05-12)](release-notes-determinate/rl-3.5.2.md) + - [Release 3.5.1 (2025-05-09)](release-notes-determinate/rl-3.5.1.md) + - [~~Release 3.5.0 (2025-05-09)~~](release-notes-determinate/rl-3.5.0.md) + - [Release 3.4.2 (2025-05-05)](release-notes-determinate/rl-3.4.2.md) + - [Release 3.4.0 (2025-04-25)](release-notes-determinate/rl-3.4.0.md) + - [Release 3.3.0 (2025-04-11)](release-notes-determinate/rl-3.3.0.md) + - [Release 3.1.0 (2025-03-27)](release-notes-determinate/rl-3.1.0.md) + - [Release 3.0.0 (2025-03-04)](release-notes-determinate/rl-3.0.0.md) +- [Nix Release Notes](release-notes/index.md) {{#include ./SUMMARY-rl-next.md}} - [Release 2.35 (2026-06-22)](release-notes/rl-2.35.md) - [Release 2.34 (2026-02-27)](release-notes/rl-2.34.md) @@ -169,60 +237,3 @@ - [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md) - [Release 2.27 (2025-03-03)](release-notes/rl-2.27.md) - [Release 2.26 (2025-01-22)](release-notes/rl-2.26.md) - - [Release 2.25 (2024-11-07)](release-notes/rl-2.25.md) - - [Release 2.24 (2024-07-31)](release-notes/rl-2.24.md) - - [Release 2.23 (2024-06-03)](release-notes/rl-2.23.md) - - [Release 2.22 (2024-04-23)](release-notes/rl-2.22.md) - - [Release 2.21 (2024-03-11)](release-notes/rl-2.21.md) - - [Release 2.20 (2024-01-29)](release-notes/rl-2.20.md) - - [Release 2.19 (2023-11-17)](release-notes/rl-2.19.md) - - [Release 2.18 (2023-09-20)](release-notes/rl-2.18.md) - - [Release 2.17 (2023-07-24)](release-notes/rl-2.17.md) - - [Release 2.16 (2023-05-31)](release-notes/rl-2.16.md) - - [Release 2.15 (2023-04-11)](release-notes/rl-2.15.md) - - [Release 2.14 (2023-02-28)](release-notes/rl-2.14.md) - - [Release 2.13 (2023-01-17)](release-notes/rl-2.13.md) - - [Release 2.12 (2022-12-06)](release-notes/rl-2.12.md) - - [Release 2.11 (2022-08-25)](release-notes/rl-2.11.md) - - [Release 2.10 (2022-07-11)](release-notes/rl-2.10.md) - - [Release 2.9 (2022-05-30)](release-notes/rl-2.9.md) - - [Release 2.8 (2022-04-19)](release-notes/rl-2.8.md) - - [Release 2.7 (2022-03-07)](release-notes/rl-2.7.md) - - [Release 2.6 (2022-01-24)](release-notes/rl-2.6.md) - - [Release 2.5 (2021-12-13)](release-notes/rl-2.5.md) - - [Release 2.4 (2021-11-01)](release-notes/rl-2.4.md) - - [Release 2.3 (2019-09-04)](release-notes/rl-2.3.md) - - [Release 2.2 (2019-01-11)](release-notes/rl-2.2.md) - - [Release 2.1 (2018-09-02)](release-notes/rl-2.1.md) - - [Release 2.0 (2018-02-22)](release-notes/rl-2.0.md) - - [Release 1.11.10 (2017-06-12)](release-notes/rl-1.11.10.md) - - [Release 1.11 (2016-01-19)](release-notes/rl-1.11.md) - - [Release 1.10 (2015-09-03)](release-notes/rl-1.10.md) - - [Release 1.9 (2015-06-12)](release-notes/rl-1.9.md) - - [Release 1.8 (2014-12-14)](release-notes/rl-1.8.md) - - [Release 1.7 (2014-04-11)](release-notes/rl-1.7.md) - - [Release 1.6.1 (2013-10-28)](release-notes/rl-1.6.1.md) - - [Release 1.6 (2013-09-10)](release-notes/rl-1.6.md) - - [Release 1.5.2 (2013-05-13)](release-notes/rl-1.5.2.md) - - [Release 1.5 (2013-02-27)](release-notes/rl-1.5.md) - - [Release 1.4 (2013-02-26)](release-notes/rl-1.4.md) - - [Release 1.3 (2013-01-04)](release-notes/rl-1.3.md) - - [Release 1.2 (2012-12-06)](release-notes/rl-1.2.md) - - [Release 1.1 (2012-07-18)](release-notes/rl-1.1.md) - - [Release 1.0 (2012-05-11)](release-notes/rl-1.0.md) - - [Release 0.16 (2010-08-17)](release-notes/rl-0.16.md) - - [Release 0.15 (2010-03-17)](release-notes/rl-0.15.md) - - [Release 0.14 (2010-02-04)](release-notes/rl-0.14.md) - - [Release 0.13 (2009-11-05)](release-notes/rl-0.13.md) - - [Release 0.12 (2008-11-20)](release-notes/rl-0.12.md) - - [Release 0.11 (2007-12-31)](release-notes/rl-0.11.md) - - [Release 0.10.1 (2006-10-11)](release-notes/rl-0.10.1.md) - - [Release 0.10 (2006-10-06)](release-notes/rl-0.10.md) - - [Release 0.9.2 (2005-09-21)](release-notes/rl-0.9.2.md) - - [Release 0.9.1 (2005-09-20)](release-notes/rl-0.9.1.md) - - [Release 0.9 (2005-09-16)](release-notes/rl-0.9.md) - - [Release 0.8.1 (2005-04-13)](release-notes/rl-0.8.1.md) - - [Release 0.8 (2005-04-11)](release-notes/rl-0.8.md) - - [Release 0.7 (2005-01-12)](release-notes/rl-0.7.md) - - [Release 0.6 (2004-11-14)](release-notes/rl-0.6.md) - - [Release 0.5 and earlier](release-notes/rl-0.5.md) diff --git a/doc/manual/source/advanced-topics/distributed-builds.md b/doc/manual/source/advanced-topics/distributed-builds.md index 08a980643e88..c39cf4500795 100644 --- a/doc/manual/source/advanced-topics/distributed-builds.md +++ b/doc/manual/source/advanced-topics/distributed-builds.md @@ -5,8 +5,8 @@ this allows multiple builds to be performed in parallel. Remote builds also allow Nix to perform multi-platform builds in a semi-transparent way. For example, if you perform a build for a -`x86_64-darwin` on an `i686-linux` machine, Nix can automatically -forward the build to a `x86_64-darwin` machine, if one is available. +`aarch64-darwin` on an `x86_64-linux` machine, Nix can automatically +forward the build to a `aarch64-darwin` machine, if one is available. ## Requirements @@ -59,7 +59,7 @@ then you need to ensure that the `PATH` of non-interactive login shells contains Nix. The [list of remote build machines](@docroot@/command-ref/conf-file.md#conf-builders) can be specified on the command line or in the Nix configuration file. -For example, the following command allows you to build a derivation for `x86_64-darwin` on a Linux machine: +For example, the following command allows you to build a derivation for `aarch64-darwin` on a Linux machine: ```console uname @@ -71,8 +71,8 @@ Linux ```console nix build --impure \ - --expr '(with import { system = "x86_64-darwin"; }; runCommand "foo" {} "uname > $out")' \ - --builders 'ssh://mac x86_64-darwin' + --expr '(with import { system = "aarch64-darwin"; }; runCommand "foo" {} "uname > $out")' \ + --builders 'ssh://mac aarch64-darwin' ``` ```console @@ -90,12 +90,12 @@ Darwin It is possible to specify multiple build machines separated by a semicolon or a newline, e.g. ```console - --builders 'ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd' + --builders 'ssh://mac aarch64-darwin ; ssh://beastie x86_64-freebsd' ``` Remote build machines can also be configured in [`nix.conf`](@docroot@/command-ref/conf-file.md), e.g. - builders = ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd + builders = ssh://mac aarch64-darwin ; ssh://beastie x86_64-freebsd After making changes to `nix.conf`, restart the Nix daemon for changes to take effect. @@ -107,4 +107,4 @@ file included in `builders` via the syntax `@/path/to/file`. For example, causes the list of machines in `/etc/nix/machines` to be included. (This is the default.) -[Nix instance]: @docroot@/glossary.md#gloss-nix-instance \ No newline at end of file +[Nix instance]: @docroot@/glossary.md#gloss-nix-instance diff --git a/doc/manual/source/command-ref/env-common.md b/doc/manual/source/command-ref/env-common.md index cc7fe77eae56..7f6293cba07d 100644 --- a/doc/manual/source/command-ref/env-common.md +++ b/doc/manual/source/command-ref/env-common.md @@ -167,6 +167,39 @@ Likewise for the state and cache directories. Indicator that tells if the current environment was set up by `nix-shell`. It can have the values `pure` or `impure`. +## OpenTelemetry tracing environment variables + +Nix can emit [OpenTelemetry](https://opentelemetry.io/) traces of its activities (such as evaluation, builds, substitutions and HTTP requests), exported using the OTLP/HTTP protocol. +Tracing is a no-op unless one of the endpoint variables below is set, or the [`otlp`](@docroot@/command-ref/conf-file.md#conf-otlp) and [`otlp-endpoint`](@docroot@/command-ref/conf-file.md#conf-otlp-endpoint) settings are configured. +The variables below take precedence over the corresponding settings; in particular, setting an endpoint variable enables tracing even if `otlp` is disabled. +When talking to the Nix daemon (including a remote one via `ssh-ng://`) or to a binary cache server, Nix propagates the [W3C trace context](https://www.w3.org/TR/trace-context/), so that work done on your behalf by other processes or servers shows up in the same trace. +Note that a daemon exports its own spans, so it needs its own OTLP configuration; in particular, environment variables aren't inherited over SSH, so a remote daemon has to be configured through the remote machine's `nix.conf`. + +- [`OTEL_EXPORTER_OTLP_ENDPOINT`](#env-OTEL_EXPORTER_OTLP_ENDPOINT) + + The base URL of the OpenTelemetry collector to which traces are sent, e.g. `https://otel.example.org`. + The path `/v1/traces` is appended automatically. + Note: the URL should *not* have a trailing slash, since the resulting double slash confuses some collectors. + Overrides the [`otlp-endpoint`](@docroot@/command-ref/conf-file.md#conf-otlp-endpoint) setting. + +- [`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`](#env-OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) + + The full URL to which traces are sent, used as-is (nothing is appended). + Takes precedence over `OTEL_EXPORTER_OTLP_ENDPOINT`. + +- Other standard [OTLP exporter variables](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/) are also honored, such as `OTEL_EXPORTER_OTLP_HEADERS` (e.g. to pass an authorization header) and `OTEL_EXPORTER_OTLP_COMPRESSION` (set to `gzip` to compress uploads). + These override the [`otlp-headers`](@docroot@/command-ref/conf-file.md#conf-otlp-headers) and [`otlp-compression`](@docroot@/command-ref/conf-file.md#conf-otlp-compression) settings respectively, and use the same syntax. + +- [`OTEL_TRACES_SAMPLER`](#env-OTEL_TRACES_SAMPLER) + + Selects the [sampler](https://opentelemetry.io/docs/languages/sdk-configuration/general/#otel_traces_sampler) used to decide which traces are recorded. + One of `always_on`, `always_off`, `traceidratio`, `parentbased_always_on` (the default), `parentbased_always_off` or `parentbased_traceidratio`. + The `parentbased_*` samplers follow the sampling decision of the parent span, which propagates in the W3C trace context — so the Nix daemon and binary cache servers follow the client's decision, keeping traces complete or absent as a whole. + +- [`OTEL_TRACES_SAMPLER_ARG`](#env-OTEL_TRACES_SAMPLER_ARG) + + For the `traceidratio` samplers: the fraction of traces to record, e.g. `0.01`. Defaults to `1.0`. + ## Appendix: OS-specific conventions This information is not Nix-specific, but is referenced above. diff --git a/doc/manual/source/command-ref/experimental-commands.md b/doc/manual/source/command-ref/experimental-commands.md deleted file mode 100644 index 1190729a2305..000000000000 --- a/doc/manual/source/command-ref/experimental-commands.md +++ /dev/null @@ -1,8 +0,0 @@ -# Experimental Commands - -This section lists [experimental commands](@docroot@/development/experimental-features.md#xp-feature-nix-command). - -> **Warning** -> -> These commands may be removed in the future, or their syntax may -> change in incompatible ways. diff --git a/doc/manual/source/command-ref/files/default-nix-expression.md b/doc/manual/source/command-ref/files/default-nix-expression.md index cd73b8e58ab7..66ce84b48f8b 100644 --- a/doc/manual/source/command-ref/files/default-nix-expression.md +++ b/doc/manual/source/command-ref/files/default-nix-expression.md @@ -31,12 +31,12 @@ Then, the resulting expression is interpreted like this: The file [`manifest.nix`](@docroot@/command-ref/files/manifest.nix.md) is always ignored. -The command [`nix-channel`] places a symlink to the current user's [channels] in this directory, the [user channel link](#user-channel-link). +The command [`nix-channel`] places a symlink to the current user's channels in this directory, the [user channel link](#user-channel-link). This makes all subscribed channels available as attributes in the default expression. ## User channel link -A symlink that ensures that [`nix-env`] can find the current user's [channels]: +A symlink that ensures that [`nix-env`] can find the current user's channels: - `~/.nix-defexpr/channels` - `$XDG_STATE_HOME/nix/defexpr/channels` if [`use-xdg-base-directories`] is set to `true`. @@ -51,4 +51,3 @@ In a multi-user installation, you may also have `~/.nix-defexpr/channels_root`, [`nix-channel`]: @docroot@/command-ref/nix-channel.md [`nix-env`]: @docroot@/command-ref/nix-env.md [`use-xdg-base-directories`]: @docroot@/command-ref/conf-file.md#conf-use-xdg-base-directories -[channels]: @docroot@/command-ref/files/channels.md diff --git a/doc/manual/source/command-ref/files/profiles.md b/doc/manual/source/command-ref/files/profiles.md index 4f022acceb3e..f137336747f1 100644 --- a/doc/manual/source/command-ref/files/profiles.md +++ b/doc/manual/source/command-ref/files/profiles.md @@ -67,7 +67,7 @@ By default, this symlink points to: - `$NIX_STATE_DIR/profiles/per-user/root/profile` for `root` The `PATH` environment variable should include `/bin` subdirectory of the profile link (e.g. `~/.nix-profile/bin`) for the user environment to be visible to the user. -The [installer](@docroot@/installation/installing-binary.md) sets this up by default, unless you enable [`use-xdg-base-directories`]. +The installer sets this up by default, unless you enable [`use-xdg-base-directories`]. [`nix-env`]: @docroot@/command-ref/nix-env.md [`nix profile`]: @docroot@/command-ref/new-cli/nix3-profile.md diff --git a/doc/manual/source/command-ref/nix-channel.md b/doc/manual/source/command-ref/nix-channel.md index 865f43ccce5b..59817be974b5 100644 --- a/doc/manual/source/command-ref/nix-channel.md +++ b/doc/manual/source/command-ref/nix-channel.md @@ -8,6 +8,12 @@ # Description +> **Warning** +> +> nix-channel is deprecated in favor of flakes in Determinate Nix. +> For a guide on Nix flakes, see: . +> For details and to offer feedback on the deprecation process, see: . + Channels are a mechanism for referencing remote Nix expressions and conveniently retrieving their latest version. The moving parts of channels are: diff --git a/doc/manual/source/command-ref/nix-env.md b/doc/manual/source/command-ref/nix-env.md index bda02149ed06..d01caaf7f787 100644 --- a/doc/manual/source/command-ref/nix-env.md +++ b/doc/manual/source/command-ref/nix-env.md @@ -52,7 +52,7 @@ These pages can be viewed offline: `nix-env` can obtain packages from multiple sources: - An attribute set of derivations from: - - The [default Nix expression](@docroot@/command-ref/files/default-nix-expression.md) (by default) + - The default Nix expression (by default) - A Nix file, specified via `--file` - A [profile](@docroot@/command-ref/files/profiles.md), specified via `--from-profile` - A Nix expression that is a function which takes default expression as argument, specified via `--from-expression` diff --git a/doc/manual/source/command-ref/nix-env/install.md b/doc/manual/source/command-ref/nix-env/install.md index 06aeb3925628..320fa530fdaa 100644 --- a/doc/manual/source/command-ref/nix-env/install.md +++ b/doc/manual/source/command-ref/nix-env/install.md @@ -22,12 +22,11 @@ It is based on the current generation of the active [profile](@docroot@/command- The arguments *args* map to store paths in a number of possible ways: -- By default, *args* is a set of names denoting derivations in the [default Nix expression]. +- By default, *args* is a set of names denoting derivations in the default Nix expression. These are [realised], and the resulting output paths are installed. Currently installed derivations with a name equal to the name of a derivation being added are removed unless the option `--preserve-installed` is specified. [derivation expression]: @docroot@/glossary.md#gloss-derivation-expression - [default Nix expression]: @docroot@/command-ref/files/default-nix-expression.md [realised]: @docroot@/glossary.md#gloss-realise If there are multiple derivations matching a name in *args* that @@ -45,7 +44,7 @@ The arguments *args* map to store paths in a number of possible ways: gcc-3.3.6 gcc-4.1.1` will install both version of GCC (and will probably cause a user environment conflict\!). -- If [`--attr`](#opt-attr) / `-A` is specified, the arguments are *attribute paths* that select attributes from the [default Nix expression]. +- If [`--attr`](#opt-attr) / `-A` is specified, the arguments are *attribute paths* that select attributes from the default Nix expression. This is faster than using derivation names and unambiguous. Show the attribute paths of available packages with [`nix-env --query`](./query.md): @@ -58,7 +57,7 @@ The arguments *args* map to store paths in a number of possible ways: easy way to copy user environment elements from one profile to another. -- If `--from-expression` is given, *args* are [Nix language functions](@docroot@/language/syntax.md#functions) that are called with the [default Nix expression] as their single argument. +- If `--from-expression` is given, *args* are [Nix language functions](@docroot@/language/syntax.md#functions) that are called with the default Nix expression as their single argument. The derivations returned by those function calls are installed. This allows derivations to be specified in an unambiguous way, which is necessary if there are multiple derivations with the same name. diff --git a/doc/manual/source/command-ref/nix-store/query.md b/doc/manual/source/command-ref/nix-store/query.md index aeb696535989..cc45eeb74cf1 100644 --- a/doc/manual/source/command-ref/nix-store/query.md +++ b/doc/manual/source/command-ref/nix-store/query.md @@ -103,6 +103,13 @@ symlink. example when *paths* were substituted from a binary cache. Use `--valid-derivers` instead to obtain valid paths only. + > **Note** + > + > `nix-store --query --deriver` is replaced with the following `nix` command: + > + > nix path-info --json ... | jq -r '.[].deriver' + + [deriver]: @docroot@/glossary.md#gloss-deriver - `--valid-derivers` diff --git a/doc/manual/source/command-ref/subcommands.md b/doc/manual/source/command-ref/subcommands.md new file mode 100644 index 000000000000..6a26732338d1 --- /dev/null +++ b/doc/manual/source/command-ref/subcommands.md @@ -0,0 +1,3 @@ +# Subcommands + +This section lists all the subcommands of the `nix` CLI. diff --git a/doc/manual/source/development/building.md b/doc/manual/source/development/building.md index 742170f76c6c..917e39e1ca5f 100644 --- a/doc/manual/source/development/building.md +++ b/doc/manual/source/development/building.md @@ -1,77 +1,9 @@ # Building Nix -This section provides some notes on how to start hacking on Nix. -To get the latest version of Nix from GitHub: - > **Note** > > When checking out the repo on Windows, make sure you have the git setting `core.symlinks` enabled, before cloning, as there are symlinks in the repo. -```console -$ git clone https://github.com/NixOS/nix.git -$ cd nix -``` - -> **Note** -> -> The following instructions assume you already have some version of Nix installed locally, so that you can use it to set up the development environment. -> If you don't have it installed, follow the [installation instructions](../installation/index.md). - - -To build all dependencies and start a shell in which all environment variables are set up so that those dependencies can be found: - -```console -$ nix-shell -``` - -To get a shell with one of the other [supported compilation environments](#compilation-environments): - -```console -$ nix-shell --attr devShells.x86_64-linux.native-clangStdenv -``` - -> **Note** -> -> You can use `native-ccacheStdenv` to drastically improve rebuild time. -> By default, [ccache](https://ccache.dev) keeps artifacts in `~/.cache/ccache/`. - -To build Nix itself in this shell: - -```console -[nix-shell]$ out="$(pwd)/outputs/out" dev=$out debug=$out mesonFlags+=" --prefix=${out}" -[nix-shell]$ dontAddPrefix=1 configurePhase -[nix-shell]$ buildPhase -``` - -To test it: - -```console -[nix-shell]$ checkPhase -``` - -To install it in `$(pwd)/outputs`: - -```console -[nix-shell]$ installPhase -[nix-shell]$ ./outputs/out/bin/nix --version -nix (Nix) 2.12 -``` - -To build a release version of Nix for the current operating system and CPU architecture: - -```console -$ nix-build -``` - -You can also build Nix for one of the [supported platforms](#platforms). - -## Building Nix with flakes - -This section assumes you are using Nix with the [`flakes`] and [`nix-command`] experimental features enabled. - -[`flakes`]: @docroot@/development/experimental-features.md#xp-feature-flakes -[`nix-command`]: @docroot@/development/experimental-features.md#xp-feature-nix-command - To build all dependencies and start a shell in which all environment variables are set up so that those dependencies can be found: ```console @@ -130,8 +62,6 @@ Nix can be built for various platforms, as specified in [`flake.nix`]: [`flake.nix`]: https://github.com/nixos/nix/blob/master/flake.nix - `x86_64-linux` -- `x86_64-darwin` -- `i686-linux` - `aarch64-linux` - `aarch64-darwin` - `armv6l-linux` @@ -149,12 +79,6 @@ platform. Common solutions include [remote build machines] and [binary format em Given such a setup, executing the build only requires selecting the respective attribute. For example, to compile for `aarch64-linux`: -```console -$ nix-build --attr packages.aarch64-linux.default -``` - -or for Nix with the [`flakes`] and [`nix-command`] experimental features enabled: - ```console $ nix build .#packages.aarch64-linux.default ``` @@ -223,6 +147,7 @@ For historic reasons and backward-compatibility, some CPU and OS identifiers are |-----------------------------|-------------------------|---------------------| | `x86` | | `i686` | | `arm` | | `host_machine.cpu()`| +| `arm64` | | `host_machine.cpu()`| | `ppc` | `little` | `powerpcle` | | `ppc64` | `little` | `powerpc64le` | | `ppc` | `big` | `powerpc` | @@ -247,20 +172,12 @@ To build with one of those environments, you can use $ nix build .#nix-cli-ccacheStdenv ``` -for flake-enabled Nix, or - -```console -$ nix-build --attr nix-cli-ccacheStdenv -``` - -for classic Nix. - You can use any of the other supported environments in place of `nix-cli-ccacheStdenv`. ## Editor integration The `clangd` LSP server is installed by default on the `clang`-based `devShell`s. -See [supported compilation environments](#compilation-environments) and instructions how to set up a shell [with flakes](#building-nix-with-flakes) or in [classic Nix](#building-nix). +See [supported compilation environments](#compilation-environments) and instructions how to [set up a shell](#building-nix). To use the LSP with your editor, you will want a `compile_commands.json` file telling `clangd` how we are compiling the code. Meson's configure always produces this inside the build directory. diff --git a/doc/manual/source/development/experimental-features.md b/doc/manual/source/development/experimental-features.md index ad5cffa91ee5..56a45b23890a 100644 --- a/doc/manual/source/development/experimental-features.md +++ b/doc/manual/source/development/experimental-features.md @@ -6,7 +6,7 @@ Experimental features are considered unstable, which means that they can be chan Users must explicitly enable them by toggling the associated [experimental feature flags](@docroot@/command-ref/conf-file.md#conf-experimental-features). This allows accessing unstable functionality without unwittingly relying on it. -Experimental feature flags were first introduced in [Nix 2.4](@docroot@/release-notes/rl-2.4.md). +Experimental feature flags were first introduced in [Nix 2.4](https://nix.dev/manual/nix/latest/release-notes/rl-2.4). Before that, Nix did have experimental features, but they were not guarded by flags and were merely documented as unstable. This was a source of confusion and controversy. diff --git a/doc/manual/source/favicon.png b/doc/manual/source/favicon.png deleted file mode 100644 index 1ed2b5fe0fdf..000000000000 Binary files a/doc/manual/source/favicon.png and /dev/null differ diff --git a/doc/manual/source/favicon.svg b/doc/manual/source/favicon.svg index 1d2a6e835d5f..55fb9479b06e 100644 --- a/doc/manual/source/favicon.svg +++ b/doc/manual/source/favicon.svg @@ -1 +1,29 @@ - \ No newline at end of file + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/doc/manual/source/glossary.md b/doc/manual/source/glossary.md index 112b0e3f8c0d..c7b8931509a8 100644 --- a/doc/manual/source/glossary.md +++ b/doc/manual/source/glossary.md @@ -1,13 +1,5 @@ # Glossary -- [`∅`]{#gloss-empty-set} - - The empty set symbol. In the context of profile history, this denotes a package is not present in a particular version of the profile. - -- [`ε`]{#gloss-epsilon} - - The epsilon symbol. In the context of a package, this means the version is empty. More precisely, the derivation does not have a version attribute. - - [base directory]{#gloss-base-directory} The location from which relative paths are resolved. diff --git a/doc/manual/source/installation/env-variables.md b/doc/manual/source/installation/env-variables.md deleted file mode 100644 index 0350904211ac..000000000000 --- a/doc/manual/source/installation/env-variables.md +++ /dev/null @@ -1,62 +0,0 @@ -# Environment Variables - -To use Nix, some environment variables should be set. In particular, -`PATH` should contain the directories `prefix/bin` and -`~/.nix-profile/bin`. The first directory contains the Nix tools -themselves, while `~/.nix-profile` is a symbolic link to the current -*user environment* (an automatically generated package consisting of -symlinks to installed packages). The simplest way to set the required -environment variables is to include the file -`prefix/etc/profile.d/nix.sh` in your `~/.profile` (or similar), like -this: - -```bash -source prefix/etc/profile.d/nix.sh -``` - -# `NIX_SSL_CERT_FILE` - -If you need to specify a custom certificate bundle to account for an -HTTPS-intercepting man in the middle proxy, you must specify the path to -the certificate bundle in the environment variable `NIX_SSL_CERT_FILE`. - -If you don't specify a `NIX_SSL_CERT_FILE` manually, Nix will install -and use its own certificate bundle. - -Set the environment variable and install Nix - -```console -$ export NIX_SSL_CERT_FILE=/etc/ssl/my-certificate-bundle.crt -$ curl -L https://nixos.org/nix/install | sh -``` - -In the shell profile and rc files (for example, `/etc/bashrc`, -`/etc/zshrc`), add the following line: - -```bash -export NIX_SSL_CERT_FILE=/etc/ssl/my-certificate-bundle.crt -``` - -> **Note** -> -> You must not add the export and then do the install, as the Nix -> installer will detect the presence of Nix configuration, and abort. - -If you use the Nix daemon, you should also add the following to -`/etc/nix/nix.conf`: - -``` -ssl-cert-file = /etc/ssl/my-certificate-bundle.crt -``` - -## Proxy Environment Variables - -The Nix installer has special handling for these proxy-related -environment variables: `http_proxy`, `https_proxy`, `ftp_proxy`, -`all_proxy`, `no_proxy`, `HTTP_PROXY`, `HTTPS_PROXY`, `FTP_PROXY`, -`ALL_PROXY`, `NO_PROXY`. - -If any of these variables are set when running the Nix installer, then -the installer will create an override file at -`/etc/systemd/system/nix-daemon.service.d/override.conf` so `nix-daemon` -will use them. diff --git a/doc/manual/source/installation/index.md b/doc/manual/source/installation/index.md index 3c09f103184a..aded684b0b59 100644 --- a/doc/manual/source/installation/index.md +++ b/doc/manual/source/installation/index.md @@ -1,44 +1,11 @@ # Installation -This section describes how to install and configure Nix for first-time use. - -The current recommended option on Linux and MacOS is [multi-user](#multi-user). - -## Multi-user - -This installation offers better sharing, improved isolation, and more security -over a single user installation. - -This option requires either: - -* Linux running systemd, with SELinux disabled -* MacOS - -> **Updating to macOS 15 Sequoia** -> -> If you recently updated to macOS 15 Sequoia and are getting -> ```console -> error: the user '_nixbld1' in the group 'nixbld' does not exist -> ``` -> when running Nix commands, refer to GitHub issue [NixOS/nix#10892](https://github.com/NixOS/nix/issues/10892) for instructions to fix your installation without reinstalling. +We recommend that macOS users install Determinate Nix using our graphical installer, [Determinate.pkg][pkg]. +For Linux and Windows Subsystem for Linux (WSL) users: ```console -$ curl -L https://nixos.org/nix/install | sh -s -- --daemon -``` - -## Single-user - -> Single-user is not supported on Mac. - -> `warning: installing Nix as root is not supported by this script!` - -This installation has less requirements than the multi-user install, however it -cannot offer equivalent sharing, isolation, or security. - -This option is suitable for systems without systemd. - -```console -$ curl -L https://nixos.org/nix/install | sh -s -- --no-daemon +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | \ + sh -s -- install ``` ## Distributions @@ -46,3 +13,5 @@ $ curl -L https://nixos.org/nix/install | sh -s -- --no-daemon The Nix community maintains installers for several distributions. They can be found in the [`nix-community/nix-installers`](https://github.com/nix-community/nix-installers) repository. + +[pkg]: https://install.determinate.systems/determinate-pkg/stable/Universal diff --git a/doc/manual/source/installation/installing-binary.md b/doc/manual/source/installation/installing-binary.md deleted file mode 100644 index 21c15637437d..000000000000 --- a/doc/manual/source/installation/installing-binary.md +++ /dev/null @@ -1,158 +0,0 @@ -# Installing a Binary Distribution - -> **Updating to macOS 15 Sequoia** -> -> If you recently updated to macOS 15 Sequoia and are getting -> ```console -> error: the user '_nixbld1' in the group 'nixbld' does not exist -> ``` -> when running Nix commands, refer to GitHub issue [NixOS/nix#10892](https://github.com/NixOS/nix/issues/10892) for instructions to fix your installation without reinstalling. - -To install the latest version Nix, run the following command: - -```console -$ curl -L https://nixos.org/nix/install | sh -``` - -This performs the default type of installation for your platform: - -- [Multi-user](#multi-user-installation): - - Linux with systemd and without SELinux - - macOS -- [Single-user](#single-user-installation): - - Linux without systemd - - Linux with SELinux - -We recommend the multi-user installation if it supports your platform and you can authenticate with `sudo`. - -The installer can be configured with various command line arguments and environment variables. -To show available command line flags: - -```console -$ curl -L https://nixos.org/nix/install | sh -s -- --help -``` - -To check what it does and how it can be customised further, [download and edit the second-stage installation script](#installing-from-a-binary-tarball). - -# Installing a pinned Nix version from a URL - -Version-specific installation URLs for all Nix versions since 1.11.16 can be found at [releases.nixos.org](https://releases.nixos.org/?prefix=nix/). -The directory for each version contains the corresponding SHA-256 hash. - -All installation scripts are invoked the same way: - -```console -$ export VERSION=2.19.2 -$ curl -L https://releases.nixos.org/nix/nix-$VERSION/install | sh -``` - -# Multi User Installation - -The multi-user Nix installation creates system users and a system service for the Nix daemon. - -Supported systems: - -- Linux running systemd, with SELinux disabled -- macOS - -To explicitly instruct the installer to perform a multi-user installation on your system: - -```console -$ bash <(curl -L https://nixos.org/nix/install) --daemon -``` - -You can run this under your usual user account or `root`. -The script will invoke `sudo` as needed. - -# Single User Installation - -To explicitly select a single-user installation on your system: - -```console -$ bash <(curl -L https://nixos.org/nix/install) --no-daemon -``` - -In a single-user installation, `/nix` is owned by the invoking user. -The script will invoke `sudo` to create `/nix` if it doesn’t already exist. -If you don’t have `sudo`, manually create `/nix` as `root`: - -```console -$ su root -# mkdir /nix -# chown alice /nix -``` - -# Installing from a binary tarball - -You can also download a binary tarball that contains Nix and all its dependencies: -- Choose a [version](https://releases.nixos.org/?prefix=nix/) and [system type](../development/building.md#platforms) -- Download and unpack the tarball -- Run the installer - -> **Example** -> -> ```console -> $ pushd $(mktemp -d) -> $ export VERSION=2.19.2 -> $ export SYSTEM=x86_64-linux -> $ curl -LO https://releases.nixos.org/nix/nix-$VERSION/nix-$VERSION-$SYSTEM.tar.xz -> $ tar xfj nix-$VERSION-$SYSTEM.tar.xz -> $ cd nix-$VERSION-$SYSTEM -> $ ./install -> $ popd -> ``` - -The installer can be customised with the environment variables declared in the file named `install-multi-user`. - -## Native packages for Linux distributions - -The Nix community maintains installers for some Linux distributions in their native packaging format(https://nix-community.github.io/nix-installers/). - -# macOS Installation - - -[]{#sect-macos-installation-change-store-prefix}[]{#sect-macos-installation-encrypted-volume}[]{#sect-macos-installation-symlink}[]{#sect-macos-installation-recommended-notes} - -We believe we have ironed out how to cleanly support the read-only root file system -on modern macOS. New installs will do this automatically. - -This section previously detailed the situation, options, and trade-offs, -but it now only outlines what the installer does. You don't need to know -this to run the installer, but it may help if you run into trouble: - -- create a new APFS volume for your Nix store -- update `/etc/synthetic.conf` to direct macOS to create a "synthetic" - empty root directory to mount your volume -- specify mount options for the volume in `/etc/fstab` - - `rw`: read-write - - `noauto`: prevent the system from auto-mounting the volume (so the - LaunchDaemon mentioned below can control mounting it, and to avoid - masking problems with that mounting service). - - `nobrowse`: prevent the Nix Store volume from showing up on your - desktop; also keeps Spotlight from spending resources to index - this volume - -- if you have FileVault enabled - - generate an encryption password - - put it in your system Keychain - - use it to encrypt the volume -- create a system LaunchDaemon to mount this volume early enough in the - boot process to avoid problems loading or restoring any programs that - need access to your Nix store - diff --git a/doc/manual/source/installation/nix-security.md b/doc/manual/source/installation/nix-security.md index 1e9036b68b21..61cad24c2b3b 100644 --- a/doc/manual/source/installation/nix-security.md +++ b/doc/manual/source/installation/nix-security.md @@ -1,15 +1,85 @@ # Security -Nix has two basic security models. First, it can be used in “single-user -mode”, which is similar to what most other package management tools do: -there is a single user (typically root) who performs all package -management operations. All other users can then use the installed -packages, but they cannot perform package management operations -themselves. - -Alternatively, you can configure Nix in “multi-user mode”. In this -model, all users can perform package management operations — for -instance, every user can install software without requiring root -privileges. Nix ensures that this is secure. For instance, it’s not -possible for one user to overwrite a package used by another user with a -Trojan horse. +Nix follows a [**multi-user**](#multi-user-model) security model in which all +users can perform package management operations. Every user can, for example, +install software without requiring root privileges, and Nix ensures that this +is secure. It's *not* possible for one user to, for example, overwrite a +package used by another user with a Trojan horse. + +## Multi-User model + +To allow a Nix store to be shared safely among multiple users, it is +important that users are not able to run builders that modify the Nix +store or database in arbitrary ways, or that interfere with builds +started by other users. If they could do so, they could install a Trojan +horse in some package and compromise the accounts of other users. + +To prevent this, the Nix store and database are owned by some privileged +user (usually `root`) and builders are executed under special user +accounts (usually named `nixbld1`, `nixbld2`, etc.). When a unprivileged +user runs a Nix command, actions that operate on the Nix store (such as +builds) are forwarded to a *Nix daemon* running under the owner of the +Nix store/database that performs the operation. + +> **Note** +> +> Multi-user mode has one important limitation: only root and a set of +> trusted users specified in `nix.conf` can specify arbitrary binary +> caches. So while unprivileged users may install packages from +> arbitrary Nix expressions, they may not get pre-built binaries. + +### Setting up the build users + +The *build users* are the special UIDs under which builds are performed. +They should all be members of the *build users group* `nixbld`. This +group should have no other members. The build users should not be +members of any other group. On Linux, you can create the group and users +as follows: + +```console +$ groupadd -r nixbld +$ for n in $(seq 1 10); do useradd -c "Nix build user $n" \ + -d /var/empty -g nixbld -G nixbld -M -N -r -s "$(which nologin)" \ + nixbld$n; done +``` + +This creates 10 build users. There can never be more concurrent builds +than the number of build users, so you may want to increase this if you +expect to do many builds at the same time. + +### Running the daemon + +The [Nix daemon](../command-ref/nix-daemon.md) should be started as +follows (as `root`): + +```console +$ nix-daemon +``` + +You’ll want to put that line somewhere in your system’s boot scripts. + +To let unprivileged users use the daemon, they should set the +[`NIX_REMOTE` environment variable](../command-ref/env-common.md) to +`daemon`. So you should put a line like + +```console +export NIX_REMOTE=daemon +``` + +into the users’ login scripts. + +### Restricting access + +To limit which users can perform Nix operations, you can use the +permissions on the directory `/nix/var/nix/daemon-socket`. For instance, +if you want to restrict the use of Nix to the members of a group called +`nix-users`, do + +```console +$ chgrp nix-users /nix/var/nix/daemon-socket +$ chmod ug=rwx,o= /nix/var/nix/daemon-socket +``` + +This way, users who are not in the `nix-users` group cannot connect to +the Unix domain socket `/nix/var/nix/daemon-socket/socket`, so they +cannot perform Nix operations. diff --git a/doc/manual/source/installation/single-user.md b/doc/manual/source/installation/single-user.md deleted file mode 100644 index f9a3b26edf41..000000000000 --- a/doc/manual/source/installation/single-user.md +++ /dev/null @@ -1,9 +0,0 @@ -# Single-User Mode - -In single-user mode, all Nix operations that access the database in -`prefix/var/nix/db` or modify the Nix store in `prefix/store` must be -performed under the user ID that owns those directories. This is -typically root. (If you install from RPM packages, that’s in fact the -default ownership.) However, on single-user machines, it is often -convenient to `chown` those directories to your normal user account so -that you don’t have to `su` to root all the time. diff --git a/doc/manual/source/installation/supported-platforms.md b/doc/manual/source/installation/supported-platforms.md deleted file mode 100644 index 8ca3ce8d445e..000000000000 --- a/doc/manual/source/installation/supported-platforms.md +++ /dev/null @@ -1,7 +0,0 @@ -# Supported Platforms - -Nix is currently supported on the following platforms: - - - Linux (i686, x86\_64, aarch64). - - - macOS (x86\_64, aarch64). diff --git a/doc/manual/source/installation/uninstall.md b/doc/manual/source/installation/uninstall.md index 4bb1c0e6ef48..e95634c213a1 100644 --- a/doc/manual/source/installation/uninstall.md +++ b/doc/manual/source/installation/uninstall.md @@ -1,196 +1,15 @@ # Uninstalling Nix -## Multi User +To uninstall Determinate Nix, use the uninstallation utility built into the [Determinate Nix Installer][installer]: -Removing a [multi-user installation](./installing-binary.md#multi-user-installation) depends on the operating system. - -### Linux - -If you are on Linux with systemd: - -1. Remove the Nix daemon service: - - ```console - sudo systemctl stop nix-daemon.service - sudo systemctl disable nix-daemon.socket nix-daemon.service - sudo systemctl daemon-reload - ``` - -2. Remove files created by Nix: - - ```console - sudo rm -rf /etc/nix /etc/profile.d/nix.sh /etc/tmpfiles.d/nix-daemon.conf /nix ~/.local/share/nix ~/.local/state/nix ~/.cache/nix ~/.nix-defexpr ~/.nix-profile ~/.nix-channels ~root/.nix-channels ~root/.nix-defexpr ~root/.nix-profile ~root/.cache/nix - ``` - -3. Remove build users and their group: - - ```console - for i in $(seq 1 32); do - sudo userdel nixbld$i - done - sudo groupdel nixbld - ``` - -4. There may also be references to Nix in - - `/etc/bash.bashrc` - - `/etc/bashrc` - - `/etc/profile` - - `/etc/zsh/zshrc` - - `/etc/zshrc` - - which you may remove. - -### FreeBSD - -1. Stop and remove the Nix daemon service: - - ```console - sudo service nix-daemon stop - sudo rm -f /usr/local/etc/rc.d/nix-daemon - sudo sysrc -x nix_daemon_enable - ``` - -2. Remove files created by Nix: - - ```console - sudo rm -rf /etc/nix /usr/local/etc/profile.d/nix.sh /nix ~/.local/share/nix ~/.local/state/nix ~/.cache/nix ~/.nix-defexpr ~/.nix-profile ~/.nix-channels ~root/.nix-channels ~root/.nix-defexpr ~root/.nix-profile ~root/.cache/nix - ``` - -3. Remove build users and their group: - - ```console - for i in $(seq 1 32); do - sudo pw userdel nixbld$i - done - sudo pw groupdel nixbld - ``` - -4. There may also be references to Nix in: - - `/usr/local/etc/bashrc` - - `/usr/local/etc/zshrc` - - Shell configuration files in users' home directories - - which you may remove. - -### macOS - -> **Updating to macOS 15 Sequoia** -> -> If you recently updated to macOS 15 Sequoia and are getting -> ```console -> error: the user '_nixbld1' in the group 'nixbld' does not exist -> ``` -> when running Nix commands, refer to GitHub issue [NixOS/nix#10892](https://github.com/NixOS/nix/issues/10892) for instructions to fix your installation without reinstalling. - -1. If system-wide shell initialisation files haven't been altered since installing Nix, use the backups made by the installer: - - ```console - sudo mv /etc/zshrc.backup-before-nix /etc/zshrc - sudo mv /etc/bashrc.backup-before-nix /etc/bashrc - sudo mv /etc/bash.bashrc.backup-before-nix /etc/bash.bashrc - ``` - - Otherwise, edit `/etc/zshrc`, `/etc/bashrc`, and `/etc/bash.bashrc` to remove the lines sourcing `nix-daemon.sh`, which should look like this: - - ```bash - # Nix - if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then - . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' - fi - # End Nix - ``` - -2. Stop and remove the Nix daemon services: - - ```console - sudo launchctl unload /Library/LaunchDaemons/org.nixos.nix-daemon.plist - sudo rm /Library/LaunchDaemons/org.nixos.nix-daemon.plist - sudo launchctl unload /Library/LaunchDaemons/org.nixos.darwin-store.plist - sudo rm /Library/LaunchDaemons/org.nixos.darwin-store.plist - ``` - - This stops the Nix daemon and prevents it from being started next time you boot the system. - -3. Remove the `nixbld` group and the `_nixbuildN` users: - - ```console - sudo dscl . -delete /Groups/nixbld - for u in $(sudo dscl . -list /Users | grep _nixbld); do sudo dscl . -delete /Users/$u; done - ``` - - This will remove all the build users that no longer serve a purpose. - -4. Edit fstab using `sudo vifs` to remove the line mounting the Nix Store volume on `/nix`, which looks like - - ``` - UUID= /nix apfs rw,noauto,nobrowse,suid,owners - ``` - or - - ``` - LABEL=Nix\040Store /nix apfs rw,nobrowse - ``` - - by setting the cursor on the respective line using the arrow keys, and pressing `dd`, and then `:wq` to save the file. - - This will prevent automatic mounting of the Nix Store volume. - -5. Edit `/etc/synthetic.conf` to remove the `nix` line. - If this is the only line in the file you can remove it entirely: - - ```bash - if [ -f /etc/synthetic.conf ]; then - if [ "$(cat /etc/synthetic.conf)" = "nix" ]; then - sudo rm /etc/synthetic.conf - else - sudo vi /etc/synthetic.conf - fi - fi - ``` - - This will prevent the creation of the empty `/nix` directory. - -6. Remove the files Nix added to your system, except for the store: - - ```console - sudo rm -rf /etc/nix /var/root/.nix-profile /var/root/.nix-defexpr /var/root/.nix-channels ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.local/share/nix ~/.local/state/nix ~/.cache/nix - ``` - - -7. Remove the Nix Store volume: - - ```console - sudo diskutil apfs deleteVolume /nix - ``` - - This will remove the Nix Store volume and everything that was added to the store. - - If the output indicates that the command couldn't remove the volume, you should make sure you don't have an _unmounted_ Nix Store volume. - Look for a "Nix Store" volume in the output of the following command: - - ```console - diskutil list - ``` - - If you _do_ find a "Nix Store" volume, delete it by running `diskutil apfs deleteVolume` with the store volume's `diskXsY` identifier. - - If you get an error that the volume is in use by the kernel, reboot and immediately delete the volume before starting any other process. - -> **Note** -> -> After you complete the steps here, you will still have an empty `/nix` directory. -> This is an expected sign of a successful uninstall. -> The empty `/nix` directory will disappear the next time you reboot. -> -> You do not have to reboot to finish uninstalling Nix. -> The uninstall is complete. -> macOS (Catalina+) directly controls root directories, and its read-only root will prevent you from manually deleting the empty `/nix` mountpoint. - -## Single User +```console +$ /nix/nix-installer uninstall +``` -To remove a [single-user installation](./installing-binary.md#single-user-installation) of Nix, run: +If you're certain that you want to uninstall, you can skip the confirmation step: ```console -rm -rf /nix ~/.nix-channels ~/.nix-defexpr ~/.nix-profile ~/.local/share/nix ~/.local/state/nix ~/.cache/nix +$ /nix/nix-installer uninstall --no-confirm ``` -You might also want to manually remove references to Nix from your `~/.profile`. + +[installer]: https://github.com/DeterminateSystems/nix-installer diff --git a/doc/manual/source/installation/upgrading.md b/doc/manual/source/installation/upgrading.md index a433f1d30e6c..8fe342b09b7c 100644 --- a/doc/manual/source/installation/upgrading.md +++ b/doc/manual/source/installation/upgrading.md @@ -1,40 +1,10 @@ # Upgrading Nix -> **Note** -> -> These upgrade instructions apply where Nix was installed following the [installation instructions in this manual](./index.md). - -Check which Nix version will be installed, for example from one of the [release channels](http://channels.nixos.org/) such as `nixpkgs-unstable`: - -```console -$ nix-shell -p nix -I nixpkgs=channel:nixpkgs-unstable --run "nix --version" -nix (Nix) 2.18.1 -``` - -> **Warning** -> -> Writing to the [local store](@docroot@/store/types/local-store.md) with a newer version of Nix, for example by building derivations with [`nix-build`](@docroot@/command-ref/nix-build.md) or [`nix-store --realise`](@docroot@/command-ref/nix-store/realise.md), may change the database schema! -> Reverting to an older version of Nix may therefore require purging the store database before it can be used. - -## Linux multi-user +You can upgrade Determinate Nix using Determinate Nixd: ```console -$ sudo su -# nix-env --install --file '' --attr nix cacert -I nixpkgs=channel:nixpkgs-unstable -# systemctl daemon-reload -# systemctl restart nix-daemon +sudo determinate-nixd upgrade ``` -## macOS multi-user +Note that the `sudo` is necessary here and upgrading fails without it. -```console -$ sudo nix-env --install --file '' --attr nix cacert -I nixpkgs=channel:nixpkgs-unstable -$ sudo launchctl remove org.nixos.nix-daemon -$ sudo launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist -``` - -## Single-user all platforms - -```console -$ nix-env --install --file '' --attr nix cacert -I nixpkgs=channel:nixpkgs-unstable -``` diff --git a/doc/manual/source/introduction.md b/doc/manual/source/introduction.md index 4df3a37e70b9..06518598aa10 100644 --- a/doc/manual/source/introduction.md +++ b/doc/manual/source/introduction.md @@ -1,4 +1,19 @@ -# Introduction +# Determinate Nix + +**Determinate Nix** is a downstream distribution of [Nix], a purely functional language, CLI tool, and package management system. +It's available on Linux, macOS, and Windows Subsystem for Linux (WSL). + +## Installing + +We recommend that macOS users install Determinate Nix using our graphical installer, [Determinate.pkg][pkg]. +For Linux and Windows Subsystem for Linux (WSL) users: + +```console +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | \ + sh -s -- install +``` + +## How Nix works Nix is a _purely functional package manager_. This means that it treats packages like values in a purely functional programming language @@ -184,10 +199,14 @@ to build configuration files in `/etc`). This means, among other things, that it is easy to roll back the entire configuration of the system to an earlier state. Also, users can install software without root privileges. For more information and downloads, see the [NixOS -homepage](https://nixos.org/). +homepage][nix]. ## License Nix is released under the terms of the [GNU LGPLv2.1 or (at your option) any later -version](http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html). +version][license]. + +[license]: http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html +[pkg]: https://install.determinate.systems/determinate-pkg/stable/Universal +[site]: https://nixos.org diff --git a/doc/manual/source/language/derivation-metadata.md b/doc/manual/source/language/derivation-metadata.md new file mode 100644 index 000000000000..221f1497de78 --- /dev/null +++ b/doc/manual/source/language/derivation-metadata.md @@ -0,0 +1,76 @@ +# Derivation metadata + +> **Note** +> +> This feature is only available if the [`provenance` experimental feature](@docroot@/development/experimental-features.md#xp-feature-provenance) is enabled. + +[`builtins.derivationWithMeta`](./builtins.md#builtins-derivationWithMeta) works like [`derivation`](./derivations.md) but with the important difference that it accepts an extra `__meta` attribute. +Nix stores the contents of `__meta` in the provenance of the derivation and of any outputs that Nix builds from it. +You can use it to record information such as a package's description, license, or maintainers alongside the store paths it produces, without changing things like the derivation's hash or the output store path. + +## Example + +```nix +builtins.derivationWithMeta { + name = "hello"; + system = builtins.currentSystem; + builder = "/bin/sh"; + args = [ "-c" "echo hello > $out" ]; + __meta = { + description = "Prints a greeting"; + license = "MIT"; + }; +} +``` + +When you've built this derivation, [`nix provenance show`](@docroot@/command-ref/new-cli/nix3-provenance-show.md) can display this metadata: + +```console +$ nix provenance show /nix/store/9388r86smd1v928zp59rwrn43mm2z4id-hello +/nix/store/9388r86smd1v928zp59rwrn43mm2z4id-hello +← built from derivation /nix/store/7j0yix78mp3cz0yxmkaki79f92x03v5a-hello.drv (output out) on my-machine for x86_64-linux +← with derivation metadata + { + "description": "Prints a greeting", + "license": "MIT" + } +``` + +The metadata also appears under the `provenance` field in the output of [`nix path-info --json`](@docroot@/command-ref/new-cli/nix3-path-info.md). + +## How `__meta` works + +- `__meta` must be an attribute set. + Nix converts it to JSON the same way that [`builtins.toJSON`](./builtins.md#builtins-toJSON) does. +- `__meta` isn't part of the derivation. + Nix doesn't pass it to the builder, and it has no effect on the derivation's store path, so changing the metadata doesn't cause a rebuild. +- Two derivations that differ only in `__meta` are considered the same derivation. + Nix keeps the metadata from whichever one it records first. +- Nix records metadata in an output's provenance only when it builds that output. + It doesn't update the provenance of outputs that already exist, such as outputs that were built earlier or substituted from a binary cache. +- Nix evaluates `__meta` in full when it instantiates the derivation. + Every value inside it must evaluate without errors, even ones nothing else uses. +- `__meta` can't contain [string context](./string-context.md). + That rules out paths and strings that refer to store paths, such as `"${pkgs.hello}"`. + +## Falling back to `derivation` + +If the `provenance` feature is disabled, `builtins.derivationWithMeta` doesn't exist. +Passing `__meta` to plain `derivation` doesn't work either, because Nix tries to turn it into an environment variable for the builder. +To support both cases, check for the builtin and drop `__meta` when it's missing: + +```nix +let + mkDerivation = + if builtins ? derivationWithMeta then + builtins.derivationWithMeta + else + attrs: derivation (removeAttrs attrs [ "__meta" ]); +in +mkDerivation { + # ... + __meta = { + description = "Prints a greeting"; + }; +} +``` diff --git a/doc/manual/source/language/derivations.md b/doc/manual/source/language/derivations.md index 50aa525acbf4..df771d21de80 100644 --- a/doc/manual/source/language/derivations.md +++ b/doc/manual/source/language/derivations.md @@ -7,6 +7,8 @@ this section just concerns how to create one from the Nix language. This builtin function takes as input an attribute set, the attributes of which specify the inputs to the process. It outputs an attribute set, and produces a [store derivation] as a side effect of evaluation. +To attach metadata to a derivation's provenance, use [`derivationWithMeta`](./derivation-metadata.md) instead. + [store derivation]: @docroot@/glossary.md#gloss-store-derivation ## Input attributes diff --git a/doc/manual/source/protocols/flake-schemas.md b/doc/manual/source/protocols/flake-schemas.md new file mode 100644 index 000000000000..f6a89f7a5d32 --- /dev/null +++ b/doc/manual/source/protocols/flake-schemas.md @@ -0,0 +1,87 @@ +# Flake Schemas + +Flake schemas are a mechanism to allow tools like `nix flake show` and `nix flake check` to enumerate and check the contents of a flake +in a generic way, without requiring built-in knowledge of specific flake output types like `packages` or `nixosConfigurations`. + +A flake can define schemas for its outputs by defining a `schemas` output. `schemas` should be an attribute set with an attribute for +every output type that you want to be supported. If a flake does not have a `schemas` attribute, Nix uses a built-in set of schemas (namely https://github.com/DeterminateSystems/flake-schemas). + +A schema is an attribute set with the following attributes: + +| Attribute | Description | Default | +| :---------------- | :----------------------------------------------------------------------------------------------------------------------------| :------ | +| `version` | Should be set to 1 | | +| `doc` | A string containing documentation about the flake output type in Markdown format. | | +| `allowIFD` | Whether the evaluation of the output attributes of this flake can read from derivation outputs. | `true` | +| `inventory` | A function that returns the contents of the flake output (described [below](#inventory)). | | +| `roles` | The roles supported by this flake output type (see [below](#roles)). | | +| `appendSystem` | Whether the current system type is appended to the flake output attribute path, as in outputs like `packages`. | | +| `defaultAttrPath` | A default flake output attribute path suffix. For example, `packages` will look for `default` if no attribute path is given. | | + +# Inventory + +The `inventory` function returns a _node_ describing the contents of the flake output. A node is either a _leaf node_ or a _non-leaf node_. This allows nested flake output attributes to be described (e.g. `x86_64-linux.hello` inside a `packages` output). + +Non-leaf nodes must have the following attribute: + +| Attribute | Description | +| :--------- | :------------------------------------------------------------------------------------- | +| `children` | An attribute set of nodes. If this attribute is missing, the attribute is a leaf node. | + +Leaf nodes can have the following attributes: + +| Attribute | Description | +| :------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `derivationAttrPath` | If not null, a list of strings denoting the attribute path of the "main" derivation of this node. | +| `evalChecks` | An attribute set of Boolean values, used by `nix flake check`. Each attribute must evaluate to `true`. | +| `isFlakeCheck` | Whether `nix flake check` should build the attribute denoted by `derivationAttrPath`. | +| `shortDescription` | A one-sentence description of the node (such as the `meta.description` attribute in Nixpkgs). | +| `what` | A brief human-readable string describing the type of the node, e.g. `"package"` or `"development environment"`. This is used by tools like `nix flake show` to describe the contents of a flake. | + +Both leaf and non-leaf nodes can have the following attributes: + +| Attribute | Description | +| :----------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `forSystems` | A list of Nix system types (e.g. `["x86_64-linux"]`) supported by this node. This is used by tools to skip nodes that cannot be built on the user's system. Setting this on a non-leaf node allows all the children to be skipped, regardless of the `forSystems` attributes of the children. If this attribute is not set, the node is never skipped. | +| `isLegacy` | If set to true, this node is skipped unless the `--legacy` CLI flag is set. | + +# Roles + +Roles allow schemas to declare what commands operate on them. For instance, to have the `nix build` command build a flake output, the schema should declare: +```nix +roles.nix-build = { }; +``` + +The following roles are used by various Nix commands: + +* `nix-build`: Used by `nix build`, `nix shell`, and `nix develop`. +* `nix-bundler`: Denotes bundler functions used by `nix bundle`. +* `nix-develop`: Used by `nix develop`. +* `nix-fmt`: Used by `nix formatter`. +* `nix-run`: Used by `nix run` and `nix bundle`. +* `nix-search`: Denotes an output that will be searched by `nix search`. +* `nix-template`: Used by `nix flake init` and `nix flake new`. + +Tools are free to define new roles. For instance, instead of hard-coding a flake output type like `nixosConfigurations`, `nixos-rebuild --flake` could use any flake output that implements a `nixos-configuration` role. + +# Example + +Here is a schema that checks that every element of the `nixosConfigurations` flake output evaluates and builds correctly (meaning that it has a `config.system.build.toplevel` attribute that yields a buildable derivation). + +```nix +outputs = { + schemas.nixosConfigurations = { + version = 1; + doc = '' + The `nixosConfigurations` flake output defines NixOS system configurations. + ''; + inventory = output: { + children = builtins.mapAttrs (configName: machine: + { + what = "NixOS configuration"; + derivationAttrPath = [ "config" "system" "build" "toplevel" ]; + }) output; + }; + }; +}; +``` diff --git a/doc/manual/source/protocols/json/schema/derivation-options-v1.yaml b/doc/manual/source/protocols/json/schema/derivation-options-v1.yaml index 58ff070882ff..d247802cd6c2 100644 --- a/doc/manual/source/protocols/json/schema/derivation-options-v1.yaml +++ b/doc/manual/source/protocols/json/schema/derivation-options-v1.yaml @@ -9,7 +9,7 @@ description: | > **Warning** > > This JSON format is currently - > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command) + > [**experimental**](@docroot@/development/experimental-features.md) > and subject to change. type: object diff --git a/doc/manual/source/protocols/json/schema/derivation-resolved-v4.yaml b/doc/manual/source/protocols/json/schema/derivation-resolved-v4.yaml index 781f07fb374f..705156eb8451 100644 --- a/doc/manual/source/protocols/json/schema/derivation-resolved-v4.yaml +++ b/doc/manual/source/protocols/json/schema/derivation-resolved-v4.yaml @@ -12,12 +12,6 @@ description: | [primary (not necessarily resolved) JSON schema](@docroot@/protocols/json/derivation/index.md), but actually it is the first version for this format. - > **Warning** - > - > This JSON format is currently - > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command) - > and subject to change. - type: object required: - name diff --git a/doc/manual/source/protocols/json/schema/derivation-v4.yaml b/doc/manual/source/protocols/json/schema/derivation-v4.yaml index 170c145c2389..fa443d982c25 100644 --- a/doc/manual/source/protocols/json/schema/derivation-v4.yaml +++ b/doc/manual/source/protocols/json/schema/derivation-v4.yaml @@ -9,7 +9,7 @@ description: | > **Warning** > > This JSON format is currently - > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command) + > [**experimental**](@docroot@/development/experimental-features.md) > and subject to change. type: object diff --git a/doc/manual/source/protocols/json/schema/signature-v2.yaml b/doc/manual/source/protocols/json/schema/signature-v2.yaml index e2b25dbc6fc0..d94798902f47 100644 --- a/doc/manual/source/protocols/json/schema/signature-v2.yaml +++ b/doc/manual/source/protocols/json/schema/signature-v2.yaml @@ -6,12 +6,6 @@ description: | This schema describes the JSON representation of signatures as used in various Nix JSON APIs. - > **Warning** - > - > This JSON format is currently - > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command) - > and subject to change. - ## Version History - Version 1: Colon-separated string in the format `:` diff --git a/doc/manual/source/protocols/json/schema/store-object-info-v3.yaml b/doc/manual/source/protocols/json/schema/store-object-info-v3.yaml index e0c1ffef134f..bf4e4dafe144 100644 --- a/doc/manual/source/protocols/json/schema/store-object-info-v3.yaml +++ b/doc/manual/source/protocols/json/schema/store-object-info-v3.yaml @@ -6,12 +6,6 @@ description: | This schema describes the JSON representation of store object metadata as returned by commands like [`nix path-info --json`](@docroot@/command-ref/new-cli/nix3-path-info.md). - > **Warning** - > - > This JSON format is currently - > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command) - > and subject to change. - ### Field Categories Store object information can come in a few different variations. @@ -187,6 +181,15 @@ $defs: The total size of this store object and every other object in its [closure](@docroot@/glossary.md#gloss-closure). > This field is not stored at all, but computed by traversing the other fields across all the store objects in a closure. + + provenance: + oneOf: + - type: "null" + - type: object # FIXME + title: Provenance + description: | + An arbitrary JSON object containing provenance information about the store object, or `null` if not available. + additionalProperties: false narInfo: @@ -270,4 +273,13 @@ $defs: > This is an impure "`.narinfo`" field that may not be included in certain contexts. > This field is not stored at all, but computed by traversing the other fields across all the store objects in a closure. + + provenance: + oneOf: + - type: "null" + - type: object # FIXME + title: Provenance + description: | + An arbitrary JSON object containing provenance information about the store object, or `null` if not available. + additionalProperties: false diff --git a/doc/manual/source/protocols/json/schema/store-path-v1.yaml b/doc/manual/source/protocols/json/schema/store-path-v1.yaml index b1251c7426e9..31760c64a617 100644 --- a/doc/manual/source/protocols/json/schema/store-path-v1.yaml +++ b/doc/manual/source/protocols/json/schema/store-path-v1.yaml @@ -6,12 +6,6 @@ description: | This schema describes the JSON representation of store paths as used in various Nix JSON APIs. - > **Warning** - > - > This JSON format is currently - > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command) - > and subject to change. - ## Format Store paths in JSON are represented as strings containing just the hash and name portion, without the store directory prefix. diff --git a/doc/manual/source/protocols/json/schema/store-v1.yaml b/doc/manual/source/protocols/json/schema/store-v1.yaml index b8ed3c8cc1c1..eaf5c624b0be 100644 --- a/doc/manual/source/protocols/json/schema/store-v1.yaml +++ b/doc/manual/source/protocols/json/schema/store-v1.yaml @@ -10,7 +10,7 @@ description: | > **Warning** > > This JSON format is currently - > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command) + > [**experimental**](@docroot@/development/experimental-features.md) > and subject to change. type: object diff --git a/doc/manual/source/protocols/wasm.md b/doc/manual/source/protocols/wasm.md new file mode 100644 index 000000000000..ad2ffe0630b7 --- /dev/null +++ b/doc/manual/source/protocols/wasm.md @@ -0,0 +1,414 @@ +# Wasm Host Interface + +Nix provides a builtin for calling WebAssembly modules: `builtins.wasm`. This allows extending Nix with custom functionality written in languages that compile to WebAssembly (such as Rust). + +## Overview + +WebAssembly modules can interact with Nix values through a host interface that provides functions for creating and inspecting Nix values. The WASM module receives Nix values as opaque `ValueId` handles and uses host functions to work with them. + +The `builtins.wasm` builtin takes two arguments: +1. A configuration attribute set with the following attributes: + - `path` - Path to the WebAssembly module (required) + - `function` - Name of the Wasm function to call (required for non-WASI modules, not allowed for WASI modules) +2. The argument value to pass to the function + +WASI mode is automatically detected by checking if the module imports from `wasi_snapshot_preview1`. There are two calling conventions: + +- **Non-WASI mode** (no WASI imports) calls the Wasm export specified by `function` directly. The function receives its input as a `ValueId` parameter and returns a `ValueId`. +- **WASI mode** (when the module imports from `wasi_snapshot_preview1`) runs the WASI module's `_start` entry point. The input `ValueId` is passed as a command-line argument (`argv[1]`), and the result is returned by calling the `return_to_nix` host function. + +## Value IDs + +Nix values are represented in Wasm code as a `u32` referred to below as a `ValueId`. These are opaque handles that reference values managed by the Nix evaluator. Value ID 0 is reserved to represent a missing attribute lookup result. + +## Entry Points + +### Non-WASI Mode + +Non-WASI mode is used when the module does **not** import from `wasi_snapshot_preview1`. + +Usage: +```nix +builtins.wasm { + path = ; + function = ; +} +``` + +Every Wasm module used in non-WASI mode must export: +- A `memory` object that the host can use to read/write data. +- `nix_wasm_init_v1()`, a function that is called once when the module is instantiated. +- The entry point function, whose name is specified by the `function` attribute. It takes a single `ValueId` and returns a single `ValueId` (i.e. it has type `fn(arg: u32) -> u32`). + +### Guest Allocator + +Some host functions return data in a buffer that the host allocates inside the Wasm memory. Modules (WASI or non-WASI) that call these functions must export: + +- `nix_wasm_alloc(size: u32, align: u32) -> u32`, a function that allocates `size` bytes with alignment `align` (a power of two) and returns a pointer to them. + +The module owns the returned buffer and is responsible for freeing it. + +### WASI Mode + +WASI mode is automatically used when the module imports a `wasi_snapshot_preview1` function. + +Usage: +```nix +builtins.wasm { + path = ; +} +``` + +Every WASI module must export: +- A `memory` object that the host can use to read/write data. +- `_start()`, the standard WASI entry point. This function takes no parameters. + +The input value is passed as a command-line argument: `argv[1]` is set to the decimal representation of the `ValueId` of the input value. + +To return a result to Nix, the module must call the `return_to_nix` host function (see below) with the `ValueId` of the result. If `_start` finishes without calling `return_to_nix`, an error is raised. + +Standard output and standard error from the WASI module are captured and emitted as Nix warnings (one warning per line). + +## Host Functions + +All host functions are imported from the `env` module. + +### Error Handling + +#### `panic(ptr: u32, len: u32)` + +Aborts execution with an error message. + +**Parameters:** +- `ptr` - Pointer to UTF-8 encoded error message in Wasm memory +- `len` - Length of the error message in bytes + +#### `warn(ptr: u32, len: u32)` + +Emits a warning message. + +**Parameters:** +- `ptr` - Pointer to UTF-8 encoded warning message in Wasm memory +- `len` - Length of the warning message in bytes + +### Type Inspection + +#### `get_type(value: ValueId) -> u32` + +Returns the type of a Nix value. + +**Parameters:** +- `value` - ID of a Nix value + +**Return values:** +- `1` - Integer +- `2` - Float +- `3` - Boolean +- `4` - String +- `5` - Path +- `6` - Null +- `7` - Attribute set +- `8` - List +- `9` - Function + +**Note:** Forces evaluation of the value. + +### Integer Operations + +#### `make_int(n: i64) -> ValueId` + +Creates a Nix integer value. + +**Parameters:** +- `n` - The integer value + +**Returns:** Value ID of the created integer + +#### `get_int(value: ValueId) -> i64` + +Extracts an integer from a Nix value. Throws an error if the value is not an integer. + +**Parameters:** +- `value` - ID of a Nix integer value + +**Returns:** The integer value + +### Float Operations + +#### `make_float(x: f64) -> ValueId` + +Creates a Nix float value. + +**Parameters:** +- `x` - The float value + +**Returns:** Value ID of the created float + +#### `get_float(value: ValueId) -> f64` + +Extracts a float from a Nix value. Throws an error if the value is not a float. + +**Parameters:** +- `value` - ID of a Nix float value + +**Returns:** The float value + +### Boolean Operations + +#### `make_bool(b: i32) -> ValueId` + +Creates a Nix Boolean value. + +**Parameters:** +- `b` - Boolean value (0 = false, non-zero = true) + +**Returns:** Value ID of the created Boolean + +#### `get_bool(value: ValueId) -> i32` + +Extracts a Boolean from a Nix value. Throws an error if the value is not a Boolean. + +**Parameters:** +- `value` - ID of a Nix Boolean value + +**Returns:** 0 for false, 1 for true + +### Null Operations + +#### `make_null() -> ValueId` + +Creates a Nix null value. + +**Returns:** Value ID of the null value + +### String Operations + +#### `make_string(ptr: u32, len: u32) -> ValueId` + +Creates a Nix string value from Wasm memory. + +**Parameters:** +- `ptr` - Pointer to a string in Wasm memory +- `len` - Length of the string in bytes + +**Note:** Strings do not require a null terminator. + +**Returns:** Value ID of the created string + +#### `copy_string(value: ValueId, ptr: u32, max_len: u32) -> u32` + +Copies a Nix string value into Wasm memory. + +**Parameters:** +- `value` - ID of a string value +- `ptr` - Pointer to buffer in Wasm memory +- `max_len` - Maximum number of bytes to copy + +**Returns:** The actual length of the string in bytes + +**Note:** If the returned length is greater than `max_len`, no data is copied. Call again with a larger buffer to get the full string. + +### Path Operations + +#### `make_path(base: ValueId, ptr: u32, len: u32) -> ValueId` + +Creates a Nix path value relative to a base path. + +**Parameters:** +- `base` - ID of a path value +- `ptr` - Pointer to a string in Wasm memory +- `len` - Length of the path string in bytes + +**Returns:** ID of a new path value + +**Note:** The path string is interpreted relative to the base path. The resulting path is in the same source tree ("source accessor") as the original path. + +#### `copy_path(value: ValueId, ptr: u32, max_len: u32) -> u32` + +Copies a Nix path value into Wasm memory as an absolute path string. + +**Parameters:** +- `value` - ID of a path value +- `ptr` - Pointer to buffer in Wasm memory +- `max_len` - Maximum number of bytes to copy + +**Returns:** The actual length of the path string in bytes + +**Note:** If the returned length is greater than `max_len`, no data is copied. + +### List Operations + +#### `make_list(ptr: u32, len: u32) -> ValueId` + +Creates a Nix list from an array of value IDs in Wasm memory. + +**Parameters:** +- `ptr` - Pointer to array of `ValueId` (u32) in Wasm memory +- `len` - Number of elements in the array + +**Returns:** Value ID of the created list + +**Note:** The array must contain `len * 4` bytes (each ValueId is 4 bytes). + +#### `copy_list(value: ValueId, ptr: u32, max_len: u32) -> u32` + +Copies a Nix list into Wasm memory as an array of value IDs. + +**Parameters:** +- `value` - ID of a list value +- `ptr` - Pointer to buffer in Wasm memory +- `max_len` - Maximum number of elements to copy + +**Returns:** The actual number of elements in the list + +**Note:** If the returned length is greater than `max_len`, no data is copied. Each element is written as a `ValueId` (4 bytes). The buffer must be `max_len * 4` bytes large. + +### Attribute Set Operations + +#### `make_attrset(ptr: u32, len: u32) -> ValueId` + +Creates a Nix attribute set from an array of attributes in Wasm memory. + +**Parameters:** +- `ptr` - Pointer to array of attribute structures in Wasm memory +- `len` - Number of attributes + +**Returns:** Value ID of the created attribute set + +**Attribute structure format:** +```c +struct Attr { + name_ptr: u32, // Pointer to attribute name + name_len: u32, // Length of attribute name in bytes + value_id: u32, // ID of the attribute value +} +``` + +Each `Attr` element is 12 bytes (3 × 4 bytes). + +#### `get_attrset(value: ValueId, ptr: u32, len: u32) -> u64` + +Copies the attribute names and value IDs of a Nix attribute set into Wasm memory in a single call. + +**Parameters:** +- `value` - ID of a Nix attribute set value +- `ptr` - Pointer to buffer in Wasm memory +- `len` - Size of the buffer in bytes + +**Returns:** The pointer to the buffer that was written in the low 32 bits, and the number of bytes written in the high 32 bits. + +**Note:** If the data fits in the `len` bytes at `ptr`, that buffer is used and `ptr` is returned. Otherwise, the host allocates a buffer of exactly the required size by calling `nix_wasm_alloc(size, 4)` (see [Guest Allocator](#guest-allocator)), and returns a pointer to that buffer instead. The module is responsible for freeing it. It is an error if the buffer is too small and the module does not export `nix_wasm_alloc`. + +**Output format:** + +- A `u32` specifying the number of attributes `n`. +- `n` `ValueId`s (4 bytes each) of the attribute values, in lexicographically sorted order of the attribute names. +- The `n` attribute names in the same order, each terminated by a null byte. + +The buffer is aligned to 4 bytes, so `ptr` must be a multiple of 4. + +#### `copy_attrset(value: ValueId, ptr: u32, max_len: u32) -> u32` + +> **Warning** +> +> This function is deprecated. Use `get_attrset` instead. + +Copies a Nix attribute set into Wasm memory as an array of attribute structures. + +**Parameters:** +- `value` - ID of a Nix attribute set value +- `ptr` - Pointer to buffer in Wasm memory +- `max_len` - Maximum number of attributes to copy + +**Returns:** The actual number of attributes in the set + +**Note:** If the returned length is greater than `max_len`, no data is copied. + +**Output structure format:** +```c +struct Attr { + value_id: u32, // ID of the attribute value + name_len: u32, // Length of attribute name in bytes +} +``` + +Each attribute is 8 bytes (2 × 4 bytes). Use `copy_attrname` to retrieve attribute names. + +#### `copy_attrname(value: ValueId, attr_idx: u32, ptr: u32, len: u32)` + +> **Warning** +> +> This function is deprecated. Use `get_attrset` instead. + +Copies an attribute name into Wasm memory. + +**Parameters:** +- `value` - ID of a Nix attribute set value +- `attr_idx` - Index of the attribute (from `copy_attrset`) +- `ptr` - Pointer to buffer in Wasm memory +- `len` - Length of the buffer (must exactly match the attribute name length) + +**Note:** Throws an error if `len` doesn't match the attribute name length or if `attr_idx` is out of bounds. + +#### `get_attr(value: ValueId, ptr: u32, len: u32) -> ValueId` + +Gets an attribute value from an attribute set by name. + +**Parameters:** +- `value` - ID of a Nix attribute set value +- `ptr` - Pointer to the attribute name in Wasm memory +- `len` - Length of the attribute name in bytes + +**Returns:** Value ID of the attribute value, or 0 if the attribute doesn't exist + +### Function Operations + +#### `call_function(fun: ValueId, ptr: u32, len: u32) -> ValueId` + +Calls a Nix function with arguments. + +**Parameters:** +- `fun` - ID of a Nix function value +- `ptr` - Pointer to array of `ValueId` arguments in Wasm memory +- `len` - Number of arguments + +**Returns:** Value ID of the function result + +#### `make_app(fun: ValueId, ptr: u32, len: u32) -> ValueId` + +Creates a lazy or partially applied function application. + +**Parameters:** +- `fun` - ID of a Nix function value +- `ptr` - Pointer to array of `ValueId` arguments in Wasm memory +- `len` - Number of arguments + +**Returns:** Value ID of the unevaluated application + +### Returning Results (WASI mode only) + +#### `return_to_nix(value: ValueId)` + +Returns a result value to the Nix evaluator from a WASI module. This function is only available in WASI mode. + +**Parameters:** +- `value` - ID of the Nix value to return as the result of the `builtins.wasm` call + +**Note:** Calling this function immediately terminates the WASI module's execution. The module must call `return_to_nix` before finishing; otherwise, an error is raised. + +### File I/O + +#### `read_file_v2(path: ValueId) -> u64` + +Reads a file into a buffer in Wasm memory allocated by the host. + +**Parameters:** +- `path` - Value ID of a Nix path value + +**Returns:** The pointer to the buffer in the low 32 bits, and the file size in bytes in the high 32 bits. + +**Note:** The buffer is allocated by calling `nix_wasm_alloc(n, 1)` (see [Guest Allocator](#guest-allocator)), and the module is responsible for freeing it. + +## Example Usage + +For Rust bindings to this interface and several examples, see https://github.com/DeterminateSystems/nix-wasm-rust/. diff --git a/doc/manual/source/quick-start.md b/doc/manual/source/quick-start.md index 9eb7a3265903..42e4e9c0c247 100644 --- a/doc/manual/source/quick-start.md +++ b/doc/manual/source/quick-start.md @@ -3,10 +3,13 @@ This chapter is for impatient people who don't like reading documentation. For more in-depth information you are kindly referred to subsequent chapters. -1. Install Nix: +1. Install Nix. + We recommend that macOS users install Determinate Nix using our graphical installer, [Determinate.pkg][pkg]. + For Linux and Windows Subsystem for Linux (WSL) users: ```console - $ curl -L https://nixos.org/nix/install | sh + $ curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | \ + sh -s -- install ``` The install script will use `sudo`, so make sure you have sufficient rights. @@ -41,3 +44,5 @@ For more in-depth information you are kindly referred to subsequent chapters. ```console $ nix-collect-garbage ``` + +[pkg]: https://install.determinate.systems/determinate-pkg/stable/Universal diff --git a/doc/manual/source/release-notes-determinate/changes.md b/doc/manual/source/release-notes-determinate/changes.md new file mode 100644 index 000000000000..c675c5fa9e4b --- /dev/null +++ b/doc/manual/source/release-notes-determinate/changes.md @@ -0,0 +1,252 @@ +# Changes between Nix and Determinate Nix + +This section lists the differences between upstream Nix 2.35 and Determinate Nix 3.23.1. + +* In Determinate Nix, flakes are stable. You no longer need to enable the `flakes` experimental feature. + +* In Determinate Nix, the new Nix CLI (i.e. the `nix` command) is stable. You no longer need to enable the `nix-command` experimental feature. + +* Determinate Nix has a setting [`json-log-path`](@docroot@/command-ref/conf-file.md#conf-json-log-path) to send a copy of all Nix log messages (in JSON format) to a file or Unix domain socket. + +* Determinate Nix has made `nix profile install` an alias to `nix profile add`, a more symmetrical antonym of `nix profile remove`. + +* `nix-channel` and `channel:` url syntax (like `channel:nixos-24.11`) is deprecated, see: https://github.com/DeterminateSystems/nix-src/issues/34 + +* Using indirect flake references and implicit inputs is deprecated, see: https://github.com/DeterminateSystems/nix-src/issues/37 + +* Warnings around "dirty trees" are updated to reduce "dirty" jargon, and now refers to "uncommitted changes". + + + + + + + +* `nix upgrade-nix` is now inert, and suggests using `determinate-nixd upgrade`. [DeterminateSystems/nix-src#55](https://github.com/DeterminateSystems/nix-src/pull/55) + +* Determinate Nix has Lazy Trees, avoiding expensive copying of flake inputs to the Nix store. ([DeterminateSystems/nix-src#27](https://github.com/DeterminateSystems/nix-src/pull/27), [DeterminateSystems/nix-src#56](https://github.com/DeterminateSystems/nix-src/pull/56)) + + + + + + + + + +* Documentation on how to replicate `nix-store --query --deriver` with the new `nix` cli. [DeterminateSystems/nix-src#82](https://github.com/DeterminateSystems/nix-src/pull/82) + +* In `nix profile`, the symbols `ε` and `∅` have been replaced with descriptive English words. [DeterminateSystems/nix-src#81](https://github.com/DeterminateSystems/nix-src/pull/81) + + + + + + + +* When remote building with `--keep-failed`, Determinate Nix shows "you can rerun" message if the derivation's platform is supported on this machine. [DeterminateSystems/nix-src#87](https://github.com/DeterminateSystems/nix-src/pull/87) + +* Improved error message when `sandbox-paths` specifies a missing file. [DeterminateSystems/nix-src#88](https://github.com/DeterminateSystems/nix-src/pull/88) + + + + + + + + + +* `nix store delete` now explains why deletion fails. [DeterminateSystems/nix-src#130](https://github.com/DeterminateSystems/nix-src/pull/130) + + + + + + + + + + + + + +* Tab completing arguments to Nix avoids network access. [DeterminateSystems/nix-src#161](https://github.com/DeterminateSystems/nix-src/pull/161) + +* Importing Nixpkgs and other tarballs to the cache is 2-4x faster. [DeterminateSystems/nix-src#149](https://github.com/DeterminateSystems/nix-src/pull/149) + +* Adding paths to the store is significantly faster. [DeterminateSystems/nix-src#162](https://github.com/DeterminateSystems/nix-src/pull/162) + + + + + +* Determinate Nix allows flake inputs to be fetched at build time. [DeterminateSystems/nix-src#49](https://github.com/DeterminateSystems/nix-src/pull/49) + + + +* The default `nix flake init` template is much more useful. [DeterminateSystems/nix-src#180](https://github.com/DeterminateSystems/nix-src/pull/180) + + + + + + + + +* Multithreaded evaluation support. [DeterminateSystems/nix-src#125](https://github.com/DeterminateSystems/nix-src/pull/125) + + + + + + +* Determinate Nix only tries to substitute inputs if fetching from its original location fails.[DeterminateSystems/nix-src#202](https://github.com/DeterminateSystems/nix-src/pull/202) + + + + + + +* A new command `nix nario` that replaces `nix-store --export|--export`. It also has a new file format (`--format 2`) that supports store path attributes such as signatures, and that can be imported more efficiently. [DeterminateSystems/nix-src#215](https://github.com/DeterminateSystems/nix-src/pull/215) + +* Determinate Nix prints the Nix version when using `-vv` or higher verbosity. [DeterminateSystems/nix-src#237](https://github.com/DeterminateSystems/nix-src/pull/237) + + + + +* During evaluation, you can read or import from the result of `builtins.fetchClosure`. [DeterminateSystems/nix-src#241](https://github.com/DeterminateSystems/nix-src/pull/241) + + + +* Flakerefs in error messages and lockfile diffs are abbreviated for readability. [DeterminateSystems/nix-src#243](https://github.com/DeterminateSystems/nix-src/pull/243), [DeterminateSystems/nix-src#264](https://github.com/DeterminateSystems/nix-src/pull/264) + + + + + + + + +* The Git fetcher doesn't compute `revCount` or `lastModified` if they're already specified [DeterminateSystems./nix-src#269](https://github.com/DeterminateSystems/nix-src/pull/269) + +* The Git fetcher avoids doing a shallow Git fetch if it previously did a non-shallow fetch of the same repository. [DeterminateSystems/nix-src#270](https://github.com/DeterminateSystems/nix-src/pull/270) + +* Determinate Nix has a builtin copy of the flake registry, making it more resilient to network outages. [DeterminateSystems/nix-src#271](https://github.com/DeterminateSystems/nix-src/pull/271) + + + +* `nix build` and `nix profile` report failing or succeeding installables. [DeterminateSystems/nix-src#281](https://github.com/DeterminateSystems/nix-src/pull/281) + +* `nix flake check` shows which outputs failed or succeeded. [DeterminateSystems/nix-src#285](https://github.com/DeterminateSystems/nix-src/pull/285) + +* Determinate Nix has a `nix ps` command to show active builds. [DeterminateSystems/nix-src#282](https://github.com/DeterminateSystems/nix-src/pull/282) + +* Determinate Nix has improved backward compatibility with lock files created by Nix < 2.20. [DeterminateSystems/nix-src#278](https://github.com/DeterminateSystems/nix-src/pull/278) + + + +* Determinate Nix has a builtin function `builtins.filterAttrs`. [DeterminateSystems/nix-src#291](https://github.com/DeterminateSystems/nix-src/pull/291) + +* `builtins.fetchTree` implicitly sets `__final = true` when a `narHash` is supplied. This allows the tree to be substituted. [DeterminateSystems/nix-src#297](https://github.com/DeterminateSystems/nix-src/pull/297) + + + + + + + +* Determinate Nix has an experimental builtin `builtins.wasm` that allows the Nix language to be extended using any language that compiles to Wasm. [DeterminateSystems/nix-src#309](https://github.com/DeterminateSystems/nix-src/pull/309) + +* `builtins.getFlake` supports path values. [DeterminateSystems/nix-src#338](https://github.com/DeterminateSystems/nix-src/pull/338) + +* Determinate Nix has support for keeping track of the provenance of store paths. [DeterminateSystems/nix-src#321](https://github.com/DeterminateSystems/nix-src/pull/321) + + + + + + + + + + + + + + + + + + + + +* Determinate Nix can upload crash info to Sentry. [DeterminateSystems/nix-src#418](https://github.com/DeterminateSystems/nix-src/pull/418) + +* Determinate Nix provides the pre-build hook with a JSON serialization of the derivation. [DeterminateSystems/nix-src#424](https://github.com/DeterminateSystems/nix-src/pull/424) + + + + + + + + + + +* Determinate Nix supports resuming from binary caches that don't support ranged requests. [DeterminateSystems/nix-src#445](https://github.com/DeterminateSystems/nix-src/pull/445) + +* JSON log messages have a configurable session ID field. [DeterminateSystems/nix-src#440](https://github.com/DeterminateSystems/nix-src/pull/440) + + + +* Determinate Nix has a command `nix serve` that allows any store to be served as a binary cache. [DeterminateSystems/nix-src#428](https://github.com/DeterminateSystems/nix-src/pull/428) + +* Determinate Nix has experimental support for signing store paths using CNSA algorithms. [DeterminateSystems/nix-src#449](https://github.com/DeterminateSystems/nix-src/pull/449) + + + +* Determinate Nix is built against Determinate Secure Packages. [DeterminateSystems/nix-src#288](https://github.com/DeterminateSystems/nix-src/pull/288) + + + +* `nix optimise store` is multi-threaded in Determinate Nix. [DeterminateSystems/nix-src#492](https://github.com/DeterminateSystems/nix-src/pull/492) + + + + + + + + + + + + + + + + + + + + + +* `nix eval --json` has a flag `--drv-link` to create symlinks to top-level derivations. This prevents them from being garbage-collected. [DeterminateSystems/nix-src#599](https://github.com/DeterminateSystems/nix-src/pull/599) + + + + + + + + + +* Determinate Nix has support for OpenTelemetry. [DeterminateSystems/nix-src#615](https://github.com/DeterminateSystems/nix-src/pull/615) + +* Determinate Nix has an experimental command `nix flake bake` for creating pre-evaluated flakes from existing flakes. [DeterminateSystems/nix-src#450](https://github.com/DeterminateSystems/nix-src/pull/450) + + + +* `nix profile history` reports packages whose store path changed but whose version stayed the same, and has a `--show-source` flag to show the flake reference of each package. [DeterminateSystems/nix-src#665](https://github.com/DeterminateSystems/nix-src/pull/665) + +* Running out of memory makes Nix exit with status 12 rather than abort. [DeterminateSystems/nix-src#653](https://github.com/DeterminateSystems/nix-src/pull/653) diff --git a/doc/manual/source/release-notes-determinate/index.md b/doc/manual/source/release-notes-determinate/index.md new file mode 100644 index 000000000000..bba33084424c --- /dev/null +++ b/doc/manual/source/release-notes-determinate/index.md @@ -0,0 +1,3 @@ +# Determinate Nix Release Notes + +This chapter lists the differences between Nix and Determinate Nix, as well as the release history of Determinate Nix. diff --git a/doc/manual/source/release-notes-determinate/rl-3.0.0.md b/doc/manual/source/release-notes-determinate/rl-3.0.0.md new file mode 100644 index 000000000000..d60786e9a72f --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.0.0.md @@ -0,0 +1,5 @@ +# Release 3.0.0 (2025-03-04) + +* Initial release of Determinate Nix. + +* Based on [upstream Nix 2.26.2](../release-notes/rl-2.26.md). diff --git a/doc/manual/source/release-notes-determinate/rl-3.1.0.md b/doc/manual/source/release-notes-determinate/rl-3.1.0.md new file mode 100644 index 000000000000..96b7819d08db --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.1.0.md @@ -0,0 +1,5 @@ +# Release 3.1.0 (2025-03-27) + +* Based on [upstream Nix 2.27.1](../release-notes/rl-2.27.md). + +* New setting `json-log-path` that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket. diff --git a/doc/manual/source/release-notes-determinate/rl-3.3.0.md b/doc/manual/source/release-notes-determinate/rl-3.3.0.md new file mode 100644 index 000000000000..badf96415df0 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.3.0.md @@ -0,0 +1,5 @@ +# Release 3.3.0 (2025-04-11) + +* Based on [upstream Nix 2.28.1](../release-notes/rl-2.28.md). + +* The `nix profile install` command is now an alias to `nix profile add`, a more symmetrical antonym of `nix profile remove`. diff --git a/doc/manual/source/release-notes-determinate/rl-3.4.0.md b/doc/manual/source/release-notes-determinate/rl-3.4.0.md new file mode 100644 index 000000000000..24ae03ca554f --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.4.0.md @@ -0,0 +1,50 @@ +# Release 3.4.0 (2025-04-25) + +* Based on [upstream Nix 2.28.2](../release-notes/rl-2.28.md). + +* **Warn users that `nix-channel` is deprecated.** + +This is the first change accomplishing our roadmap item of deprecating Nix channels: https://github.com/DeterminateSystems/nix-src/issues/34 + +This is due to user confusion and surprising behavior of channels, especially in the context of user vs. root channels. + +The goal of this change is to make the user experience of Nix more predictable. +In particular, these changes are to support users with lower levels of experience who are following guides that focus on channels as the mechanism of distribution. + +Users will now see this message: + +> nix-channel is deprecated in favor of flakes in Determinate Nix. For a guide on Nix flakes, see: https://zero-to-nix.com/. or details and to offer feedback on the deprecation process, see: https://github.com/DeterminateSystems/nix-src/issues/34. + + +* **Warn users that `channel:` URLs are deprecated.** + +This is the second change regarding our deprecation of Nix channels. +Using a `channel:` URL (like `channel:nixos-24.11`) will yield a warning like this: + +> Channels are deprecated in favor of flakes in Determinate Nix. Instead of 'channel:nixos-24.11', use 'https://nixos.org/channels/nixos-24.11/nixexprs.tar.xz'. For a guide on Nix flakes, see: https://zero-to-nix.com/. For details and to offer feedback on the deprecation process, see: https://github.com/DeterminateSystems/nix-src/issues/34. + +* **Warn users against indirect flake references in `flake.nix` inputs** + +This is the first change accomplishing our roadmap item of deprecating implicit and indirect flake inputs: https://github.com/DeterminateSystems/nix-src/issues/37 + +The flake registry provides an important UX affordance for using Nix flakes and remote sources in command line uses. +For that reason, the registry is not being deprecated entirely and will still be used for command-line incantations, like nix run. + +This move will eliminate user confusion and surprising behavior around global and local registries during flake input resolution. + +The goal of this change is to make the user experience of Nix more predictable. +We have seen a pattern of confusion when using automatic flake inputs and local registries. +Specifically, users' flake inputs resolving and locking inconsistently depending on the configuration of the host system. + +Users will now see the following warning if their flake.nix uses an implicit or indirect Flake reference input: + +> Flake input 'nixpkgs' uses the flake registry. Using the registry in flake inputs is deprecated in Determinate Nix. To make your flake future-proof, add the following to 'xxx/flake.nix': +> +> inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11"; +> +> For more information, see: https://github.com/DeterminateSystems/nix-src/issues/37 + + +### Other updates: +* Improve the "dirty tree" message. Determinate Nix will now say `Git tree '...' has uncommitted changes` instead of `Git tree '...' is dirty` +* Stop warning about uncommitted changes in a Git repository when using `nix develop` diff --git a/doc/manual/source/release-notes-determinate/rl-3.4.2.md b/doc/manual/source/release-notes-determinate/rl-3.4.2.md new file mode 100644 index 000000000000..8acabd4425fd --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.4.2.md @@ -0,0 +1,4 @@ +# Release 3.4.2 (2025-05-05) + +* Based on [upstream Nix 2.28.3](../release-notes/rl-2.28.md). + diff --git a/doc/manual/source/release-notes-determinate/rl-3.5.0.md b/doc/manual/source/release-notes-determinate/rl-3.5.0.md new file mode 100644 index 000000000000..d5b26b9419e7 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.5.0.md @@ -0,0 +1,4 @@ +# Release 3.5.0 (2025-05-09) + +* Based on [upstream Nix 2.28.3](../release-notes/rl-2.28.md). + diff --git a/doc/manual/source/release-notes-determinate/rl-3.5.1.md b/doc/manual/source/release-notes-determinate/rl-3.5.1.md new file mode 100644 index 000000000000..b0813ca59c90 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.5.1.md @@ -0,0 +1,57 @@ +# Release 3.5.1 (2025-05-09) + +* Based on [upstream Nix 2.28.3](../release-notes/rl-2.28.md). + +## What's Changed + +Most notably, Lazy Trees has merged in to Determinate Nix and is in Feature Preview status, but remains disabled by default. +Lazy trees massively improves performance in virtually all scenarios because it enables Nix to avoid making unnecessary copies of files into the Nix store. +In testing, we saw iteration times on Nixpkgs **drop from over 12 seconds to 3.5 seconds**. + +After upgrading to Determinate Nix 3.5.1 with `sudo determinate-nixd upgrade`, enable lazy trees by adding this to `/etc/nix/nix.custom.conf`: + +``` +lazy-trees = true +``` + +Please note that our full flake regression test suite passes with no changes with lazy trees, and please report compatibility issues. + +Read [this GitHub comment](https://github.com/DeterminateSystems/nix-src/pull/27#pullrequestreview-2822153088) for further details and next steps. +We'll be publishing an update on the [Determinate Systems blog](https://determinate.systems/posts/) in the next few days with more information as well. + +Relevant PRs: +* Lazy trees v2 by @edolstra in [DeterminateSystems/nix-src#27](https://github.com/DeterminateSystems/nix-src/pull/27) +* Improve lazy trees backward compatibility by @edolstra in [DeterminateSystems/nix-src#56](https://github.com/DeterminateSystems/nix-src/pull/56) + + +### Additional changes in this release: +* Bug fix: Flake input URLs are canonicalized before checking flake.lock file staleness, avoiding needlessly regenerating flake.lock files with `dir` in URL-style flakerefs by @edolstra in [DeterminateSystems/nix-src#57](https://github.com/DeterminateSystems/nix-src/pull/57) +* `nix upgrade-nix` is deprecated in favor of `determinate-nixd upgrade`, by @gustavderdrache in [DeterminateSystems/nix-src#55](https://github.com/DeterminateSystems/nix-src/pull/55) +* UX: Improved build failure and dependency failure error messages to include needed output paths by @edolstra in [DeterminateSystems/nix-src#58](https://github.com/DeterminateSystems/nix-src/pull/58). + +Previously: + +``` +error: builder for '/nix/store/[...]-nested-failure-bottom.drv' failed with exit code 1 +error: 1 dependencies of derivation '/nix/store/[...]-nested-failure-middle.drv' failed to build +error: 1 dependencies of derivation '/nix/store/[...]-nested-failure-top.drv' failed to build +``` + +Now: + +``` +error: Cannot build '/nix/store/w37gflm9wz9dcnsgy3sfrmnlvm8qigaj-nested-failure-bottom.drv'. + Reason: builder failed with exit code 1. + Output paths: + /nix/store/yzybs8kp35dfipbzdlqcc6lxz62hax04-nested-failure-bottom +error: Cannot build '/nix/store/00gr5hlxfc03x2675w6nn3pwfrz2fr62-nested-failure-middle.drv'. + Reason: 1 dependency failed. + Output paths: + /nix/store/h781j5h4bdchmb4c2lvy8qzh8733azhz-nested-failure-middle +error: Cannot build '/nix/store/8am0ng1gyx8sbzyr0yx6jd5ix3yy5szc-nested-failure-top.drv'. + Reason: 1 dependency failed. + Output paths: + /nix/store/fh12637kgvp906s9yhi9w2dc7ghfwxs1-nested-failure-top +``` + +**Full Changelog**: [v3.4.2...v3.5.1](https://github.com/DeterminateSystems/nix-src/compare/v3.4.2...v3.5.1) diff --git a/doc/manual/source/release-notes-determinate/rl-3.5.2.md b/doc/manual/source/release-notes-determinate/rl-3.5.2.md new file mode 100644 index 000000000000..bc5396c255b6 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.5.2.md @@ -0,0 +1,11 @@ +# Release 3.5.2 (2025-05-12) + +* Based on [upstream Nix 2.28.3](../release-notes/rl-2.28.md). + +## What's Changed +* Fix a regression where narHash was not added to lock files when lazy trees were disabled by @edolstra in [DeterminateSystems/nix-src#63](https://github.com/DeterminateSystems/nix-src/pull/63) + +* Tell users a source is corrupted ("cannot read file from tarball: Truncated tar archive detected while reading data"), improving over the previous 'cannot read file from tarball' error by @edolstra in [DeterminateSystems/nix-src#64](https://github.com/DeterminateSystems/nix-src/pull/64) + + +**Full Changelog**: [v3.5.1...v3.5.2](https://github.com/DeterminateSystems/nix-src/compare/v3.5.1...v3.5.2) diff --git a/doc/manual/source/release-notes-determinate/rl-3.6.0.md b/doc/manual/source/release-notes-determinate/rl-3.6.0.md new file mode 100644 index 000000000000..453ab6c301dc --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.6.0.md @@ -0,0 +1,11 @@ +# Release 3.6.0 (2025-05-22) + +* Based on [upstream Nix 2.29.0](../release-notes/rl-2.29.md). + +## What's Changed +* Install 'nix profile add' manpage by @edolstra in [DeterminateSystems/nix-src#69](https://github.com/DeterminateSystems/nix-src/pull/69) +* Sync with upstream 2.29.0 by @edolstra in [DeterminateSystems/nix-src#67](https://github.com/DeterminateSystems/nix-src/pull/67) +* Emit warnings when using import-from-derivation by setting the `trace-import-from-derivation` option to `true` by @gustavderdrache in [DeterminateSystems/nix-src#70](https://github.com/DeterminateSystems/nix-src/pull/70) + + +**Full Changelog**: [v3.5.2...v3.6.0](https://github.com/DeterminateSystems/nix-src/compare/v3.5.2...v3.6.0) diff --git a/doc/manual/source/release-notes-determinate/rl-3.6.1.md b/doc/manual/source/release-notes-determinate/rl-3.6.1.md new file mode 100644 index 000000000000..12505afee278 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.6.1.md @@ -0,0 +1,9 @@ +# Release 3.6.1 (2025-05-24) + +* Based on [upstream Nix 2.29.0](../release-notes/rl-2.29.md). + +## What's Changed +* Fix nlohmann error in fromStructuredAttrs() by @edolstra in [DeterminateSystems/nix-src#73](https://github.com/DeterminateSystems/nix-src/pull/73) + + +**Full Changelog**: [v3.6.0...v3.6.1](https://github.com/DeterminateSystems/nix-src/compare/v3.6.0...v3.6.1) diff --git a/doc/manual/source/release-notes-determinate/rl-3.6.2.md b/doc/manual/source/release-notes-determinate/rl-3.6.2.md new file mode 100644 index 000000000000..882c142f00c3 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.6.2.md @@ -0,0 +1,15 @@ +# Release 3.6.2 (2025-06-02) + +* Based on [upstream Nix 2.29.0](../release-notes/rl-2.29.md). + +## What's Changed +* Dramatically improve the performance of nix store copy-sigs: Use http-connections setting to control parallelism by @edolstra in [DeterminateSystems/nix-src#80](https://github.com/DeterminateSystems/nix-src/pull/80) +* Document how to replicate nix-store --query --deriver with the nix cli by @grahamc in [DeterminateSystems/nix-src#82](https://github.com/DeterminateSystems/nix-src/pull/82) +* The garbage collector no longer gives up if it encounters an undeletable file, by @edolstra in [DeterminateSystems/nix-src#83](https://github.com/DeterminateSystems/nix-src/pull/83) +* nix profile: Replace ε and ∅ with descriptive English words by @grahamc in [DeterminateSystems/nix-src#81](https://github.com/DeterminateSystems/nix-src/pull/81) +* Rework README to clarify that this distribution is our distribution, by @lucperkins in [DeterminateSystems/nix-src#84](https://github.com/DeterminateSystems/nix-src/pull/84) +* Include the source location when warning about inefficient double copies by @edolstra in [DeterminateSystems/nix-src#79](https://github.com/DeterminateSystems/nix-src/pull/79) +* Call out that `--keep-failed` with remote builders will keep the failed build directory on that builder by @cole-h in [DeterminateSystems/nix-src#85](https://github.com/DeterminateSystems/nix-src/pull/85) + + +**Full Changelog**: [v3.6.1...v3.6.2](https://github.com/DeterminateSystems/nix-src/compare/v3.6.1...v3.6.2) diff --git a/doc/manual/source/release-notes-determinate/rl-3.6.5.md b/doc/manual/source/release-notes-determinate/rl-3.6.5.md new file mode 100644 index 000000000000..8ef5be0fd0d3 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.6.5.md @@ -0,0 +1,19 @@ +# Release 3.6.5 (2025-06-12) + +* Based on [upstream Nix 2.29.0](../release-notes/rl-2.29.md). + +## What's Changed +* When remote building with --keep-failed, only show "you can rerun" message if the derivation's platform is supported on this machine by @cole-h in [DeterminateSystems/nix-src#87](https://github.com/DeterminateSystems/nix-src/pull/87) +* Indicate that sandbox-paths specifies a missing file in the corresponding error message. by @cole-h in [DeterminateSystems/nix-src#88](https://github.com/DeterminateSystems/nix-src/pull/88) +* Render lazy tree paths in messages withouth the/nix/store/hash... prefix in substituted source trees by @edolstra in [DeterminateSystems/nix-src#91](https://github.com/DeterminateSystems/nix-src/pull/91) +* Use FlakeHub inputs by @lucperkins in [DeterminateSystems/nix-src#89](https://github.com/DeterminateSystems/nix-src/pull/89) +* Proactively cache more flake inputs and fetches by @edolstra in [DeterminateSystems/nix-src#93](https://github.com/DeterminateSystems/nix-src/pull/93) +* Fix: register extra builtins just once by @edolstra in [DeterminateSystems/nix-src#97](https://github.com/DeterminateSystems/nix-src/pull/97) +* Fix the link to `builders-use-substitutes` documentation for `builders` by @lucperkins in [DeterminateSystems/nix-src#102](https://github.com/DeterminateSystems/nix-src/pull/102) +* Improve error messages that use the hypothetical future tense of "will" by @lucperkins in [DeterminateSystems/nix-src#92](https://github.com/DeterminateSystems/nix-src/pull/92) +* Make the `nix repl` test more stable by @edolstra in [DeterminateSystems/nix-src#103](https://github.com/DeterminateSystems/nix-src/pull/103) +* Run nixpkgsLibTests against lazy trees by @edolstra in [DeterminateSystems/nix-src#100](https://github.com/DeterminateSystems/nix-src/pull/100) +* Run the Nix test suite against lazy trees by @edolstra in [DeterminateSystems/nix-src#105](https://github.com/DeterminateSystems/nix-src/pull/105) +* Improve caching of inputs by @edolstra in [DeterminateSystems/nix-src#98](https://github.com/DeterminateSystems/nix-src/pull/98), [DeterminateSystems/nix-src#110](https://github.com/DeterminateSystems/nix-src/pull/110), and [DeterminateSystems/nix-src#115](https://github.com/DeterminateSystems/nix-src/pull/115) + +**Full Changelog**: [v3.6.2...v3.6.5](https://github.com/DeterminateSystems/nix-src/compare/v3.6.2...v3.6.4) diff --git a/doc/manual/source/release-notes-determinate/rl-3.6.6.md b/doc/manual/source/release-notes-determinate/rl-3.6.6.md new file mode 100644 index 000000000000..bf4e3690afa1 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.6.6.md @@ -0,0 +1,7 @@ +# Release 3.6.6 (2025-06-17) + +* Based on [upstream Nix 2.29.0](../release-notes/rl-2.29.md). + +## What's Changed + +* No-op release on the nix-src side, due to a regression on nix-darwin in determinate-nixd. diff --git a/doc/manual/source/release-notes-determinate/rl-3.6.7.md b/doc/manual/source/release-notes-determinate/rl-3.6.7.md new file mode 100644 index 000000000000..197587f1b3a9 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.6.7.md @@ -0,0 +1,17 @@ +# Release 3.6.7 (2025-06-24) + +* Based on [upstream Nix 2.29.1](../release-notes/rl-2.29.md). + +## What's Changed + +### Security contents + +* Patched against GHSA-g948-229j-48j3 + +### Lazy trees: + +* Lazy trees now produces `flake.lock` files with NAR hashes unless `lazy-locks` is set to `true` by @edolstra in [DeterminateSystems/nix-src#113](https://github.com/DeterminateSystems/nix-src/pull/113) +* Improved caching with lazy-trees when using --impure, with enhanced testing by @edolstra in [DeterminateSystems/nix-src#117](https://github.com/DeterminateSystems/nix-src/pull/117) + + +**Full Changelog**: [v3.6.6...v3.6.7](https://github.com/DeterminateSystems/nix-src/compare/v3.6.6...v3.6.7) diff --git a/doc/manual/source/release-notes-determinate/rl-3.6.8.md b/doc/manual/source/release-notes-determinate/rl-3.6.8.md new file mode 100644 index 000000000000..c4b4b96c9e73 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.6.8.md @@ -0,0 +1,12 @@ +# Release 3.6.8 (2025-06-25) + +* Based on [upstream Nix 2.29.1](../release-notes/rl-2.29.md). + +## What's Changed +* Fix fetchToStore() caching with --impure, improve testing by @edolstra in [DeterminateSystems/nix-src#117](https://github.com/DeterminateSystems/nix-src/pull/117) +* Add lazy-locks setting by @edolstra in [DeterminateSystems/nix-src#113](https://github.com/DeterminateSystems/nix-src/pull/113) +* Sync 2.29.1 by @edolstra in [DeterminateSystems/nix-src#124](https://github.com/DeterminateSystems/nix-src/pull/124) +* Release v3.6.7 by @github-actions in [DeterminateSystems/nix-src#126](https://github.com/DeterminateSystems/nix-src/pull/126) + + +**Full Changelog**: [v3.6.6...v3.6.8](https://github.com/DeterminateSystems/nix-src/compare/v3.6.6...v3.6.8) diff --git a/doc/manual/source/release-notes-determinate/rl-3.7.0.md b/doc/manual/source/release-notes-determinate/rl-3.7.0.md new file mode 100644 index 000000000000..615e858592e2 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.7.0.md @@ -0,0 +1,63 @@ +# Release 3.7.0 (2025-07-03) + +- Based on [upstream Nix 2.29.1](../release-notes/rl-2.29.md). + +## What's Changed + +### Prefetch flake inputs in parallel + +By @edolstra in [DeterminateSystems/nix-src#127](https://github.com/DeterminateSystems/nix-src/pull/127) + +This release brings the command `nix flake prefetch-inputs`. + +Flake inputs are typically fetched "just in time." +That means Nix fetches a flake input when the evaluator needs it, and not before. +When the evaluator needs an input, evaluation is paused until the source is available. + +This causes a significant slow-down on projects with lots of flake inputs. + +The new command `nix flake prefetch-inputs` fetches all flake inputs in parallel. +We expect running this new command before building will dramatically improve evaluation performance for most projects, especially in CI. +Note that projects which with many unused flake inputs may not benefit from this change, since the new command fetches every input whether they're used or not. + +### Deep flake input overrides now work as expected + +By @edolstra in [DeterminateSystems/nix-src#108](https://github.com/DeterminateSystems/nix-src/pull/108) + +An override like: + +``` +inputs.foo.inputs.bar.inputs.nixpkgs.follows = "nixpkgs"; +``` + +implicitly set `inputs.foo.inputs.bar` to `flake:bar`, which led to an unexpected error like: + +``` +error: cannot find flake 'flake:bar' in the flake registries +``` + +We now no longer create a parent override (like for `foo.bar` in the example above) if it doesn't set an explicit ref or follows attribute. +We only recursively apply its child overrides. + +### `nix store delete` now shows you why deletion was not possible + +By @edolstra in [DeterminateSystems/nix-src#130](https://github.com/DeterminateSystems/nix-src/pull/130) + +For example: + +``` +error: Cannot delete path '/nix/store/6fcrjgfjip2ww3sx51rrmmghfsf60jvi-patchelf-0.14.3' + because it's referenced by the GC root '/home/eelco/Dev/nix-master/build/result'. + +error: Cannot delete path '/nix/store/lf3lrf8bjfn8xvr0az9q96y989sxs5r9-cowsay-3.8.4' + because it's referenced by the GC root '/proc/3600568/environ'. + +error: Cannot delete path '/nix/store/klyng5rpdkwi5kbxkncy4gjwb490dlhb-foo.drv' + because it's in use by '{nix-process:3605324}'. +``` + +### Lazy-tree improvements + +- Improved lazy-tree evaluation caching for flakes accessed with a `path` flakeref by @edolstra in [DeterminateSystems/nix-src#131](https://github.com/DeterminateSystems/nix-src/pull/131) + +**Full Changelog**: [v3.6.8...v3.7.0](https://github.com/DeterminateSystems/nix-src/compare/v3.6.8...v3.7.0) diff --git a/doc/manual/source/release-notes-determinate/rl-3.8.0.md b/doc/manual/source/release-notes-determinate/rl-3.8.0.md new file mode 100644 index 000000000000..4103d6df94e0 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.8.0.md @@ -0,0 +1,29 @@ +# Release 3.8.0 (2025-07-10) + +* Based on [upstream Nix 2.30.0](../release-notes/rl-2.30.md). + +## What's Changed + +### Faster CI with `nix flake check` + +`nix flake check` no longer downloads flake outputs if no building is necessary. + +This command is intended to validate that a flake can fully evaluate and all outputs can build. +If the outputs are available in a binary cache then both properties are confirmed to be true. +Notably, downloading the output from the binary cache is not strictly necessary for the validation. + +Previously, `nix flake check` would download a flake output if the full build is available in a binary cache. + +Some users will find this change significantly reduces costly bandwidth and CI workflow time. + +PR: [DeterminateSystems/nix-src#134](https://github.com/DeterminateSystems/nix-src/pull/134) + +### Improved flake locking of transitive dependencies + +Determinate Nix now re-locks all transitive dependencies when changing a flake input's source URL. + +This fixes an issue where in some scenarios Nix would not re-lock those inputs and incorrectly use the old inputs' dependencies. + +PR: [DeterminateSystems/nix-src#137](https://github.com/DeterminateSystems/nix-src/pull/137) + +**Full Changelog**: [v3.7.0...v3.8.0](https://github.com/DeterminateSystems/nix-src/compare/v3.7.0...v3.8.0) diff --git a/doc/manual/source/release-notes-determinate/rl-3.8.1.md b/doc/manual/source/release-notes-determinate/rl-3.8.1.md new file mode 100644 index 000000000000..90dc328f6ec2 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.8.1.md @@ -0,0 +1,9 @@ +# Release 3.8.1 (2025-07-11) + +* Based on [upstream Nix 2.30.0](../release-notes/rl-2.30.md). + +## What's Changed +* Address ifdef problem with macOS/BSD sandboxing by @gustavderdrache in [DeterminateSystems/nix-src#142](https://github.com/DeterminateSystems/nix-src/pull/142) + + +**Full Changelog**: [v3.8.0...v3.8.1](https://github.com/DeterminateSystems/nix-src/compare/v3.8.0...v3.8.1) diff --git a/doc/manual/source/release-notes-determinate/rl-3.8.2.md b/doc/manual/source/release-notes-determinate/rl-3.8.2.md new file mode 100644 index 000000000000..638d90f6841b --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.8.2.md @@ -0,0 +1,10 @@ +# Release 3.8.2 (2025-07-12) + +* Based on [upstream Nix 2.30.0](../release-notes/rl-2.30.md). + +## What's Changed +* ci: don't run the full test suite for x86_64-darwin by @grahamc in [DeterminateSystems/nix-src#144](https://github.com/DeterminateSystems/nix-src/pull/144) +* Try publishing the manual again by @grahamc in [DeterminateSystems/nix-src#145](https://github.com/DeterminateSystems/nix-src/pull/145) + + +**Full Changelog**: [v3.8.1...v3.8.2](https://github.com/DeterminateSystems/nix-src/compare/v3.8.1...v3.8.2) diff --git a/doc/manual/source/release-notes-determinate/rl-3.8.3.md b/doc/manual/source/release-notes-determinate/rl-3.8.3.md new file mode 100644 index 000000000000..d3eb02bc7ea5 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.8.3.md @@ -0,0 +1,26 @@ +# Release 3.8.3 (2025-07-18) + +* Based on [upstream Nix 2.30.1](../release-notes/rl-2.30.md). + +## What's Changed + +### Non-blocking evaluation caching + +Users reported evaluation would occasionally block other evaluation processes. + +The evaluation cache database is now opened in write-ahead mode to prevent delaying evaluations. + +PR: [DeterminateSystems/nix-src#150](https://github.com/DeterminateSystems/nix-src/pull/150) + +### New experimental feature: `external-builders` + +This experimental feature allows Nix to call an external program for the build environment. + +The interface and behavior of this feature may change at any moment without a correspondingly major semver version change. + +PRs: +- [DeterminateSystems/nix-src#141](https://github.com/DeterminateSystems/nix-src/pull/141) +- [DeterminateSystems/nix-src#152](https://github.com/DeterminateSystems/nix-src/pull/152) +- [DeterminateSystems/nix-src#78](https://github.com/DeterminateSystems/nix-src/pull/78) + +**Full Changelog**: [v3.8.2...v3.8.3](https://github.com/DeterminateSystems/nix-src/compare/v3.8.2...v3.8.3) diff --git a/doc/manual/source/release-notes-determinate/rl-3.8.4.md b/doc/manual/source/release-notes-determinate/rl-3.8.4.md new file mode 100644 index 000000000000..7c73e75ca023 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.8.4.md @@ -0,0 +1,9 @@ +# Release 3.8.4 (2025-07-21) + +* Based on [upstream Nix 2.30.1](../release-notes/rl-2.30.md). + +## What's Changed +* Revert "Use WAL mode for SQLite cache databases" by @grahamc in [DeterminateSystems/nix-src#155](https://github.com/DeterminateSystems/nix-src/pull/155) + + +**Full Changelog**: [v3.8.3...v3.8.4](https://github.com/DeterminateSystems/nix-src/compare/v3.8.3...v3.8.4) diff --git a/doc/manual/source/release-notes-determinate/rl-3.8.5.md b/doc/manual/source/release-notes-determinate/rl-3.8.5.md new file mode 100644 index 000000000000..0f1bbe6f99d7 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/rl-3.8.5.md @@ -0,0 +1,58 @@ +## What's Changed + +### Less time "unpacking into the Git cache" + +Unpacking sources into the user's cache is now takes 1/2 to 1/4 of the time it used to. +Previously, Nix serially unpacked sources into the cache. +This change takes better advantage of our users' hardware by parallelizing the import. +Real life testing shows an initial Nixpkgs import takes 3.6s on Linux, when it used to take 11.7s. + +PR: [DeterminateSystems/nix-src#149](https://github.com/DeterminateSystems/nix-src/pull/149) + +### Copy paths to the daemon in parallel + +Determinate Nix's evaluator no longer blocks evaluation when copying paths to the store. +Previously, Nix would pause evaluation when it needed to add files to the store. +Now, the copying is performed in the background allowing evaluation to proceed. + +PR: [DeterminateSystems/nix-src#162](https://github.com/DeterminateSystems/nix-src/pull/162) + +### Faster Nix evaluation by reducing duplicate Nix daemon queries + +Determinate Nix more effectively caches store path validity data within a single evaluation. +Previously, the Nix client would perform many thousands of exra Nix daemon requests. +Each extra request takes real time, and this change reduced a sample evaluation by over 12,000 requests. + +PR: [DeterminateSystems/nix-src#157](https://github.com/DeterminateSystems/nix-src/pull/157) + +### More responsive tab completion + +Tab completion now implies the "--offline" flag, which disables most network requests. +Previously, tab completing Nix arguments would attempt to fetch sources and access binary caches. +Operating in offline mode improves the interactive experience of Nix when tab completing. + +PR: [DeterminateSystems/nix-src#161](https://github.com/DeterminateSystems/nix-src/pull/161) + +### ZFS users: we fixed the mysterious stall. + +Opening the Nix database is usually instantaneous but sometimes has a several second latency. +Determinate Nix works around this issue, eliminating the frustrating random stall when running Nix commands. + +PR: [DeterminateSystems/nix-src#158](https://github.com/DeterminateSystems/nix-src/pull/158) + +### Other changes + +* Determinate Nix is now fully formatted by clang-format, making it easier than ever to contribute to the project. + +PR: [DeterminateSystems/nix-src#159](https://github.com/DeterminateSystems/nix-src/pull/159) + +* Determinate Nix is now based on upstream Nix 2.30.2. + +PR: [DeterminateSystems/nix-src#160](https://github.com/DeterminateSystems/nix-src/pull/160) + +* Determinate Nix now uses `main` as our development branch, moving away from `detsys-main`. + +PRs: +* [DeterminateSystems/nix-src#164](https://github.com/DeterminateSystems/nix-src/pull/164) +* [DeterminateSystems/nix-src#166](https://github.com/DeterminateSystems/nix-src/pull/166) + diff --git a/doc/manual/source/release-notes-determinate/v3.10.0.md b/doc/manual/source/release-notes-determinate/v3.10.0.md new file mode 100644 index 000000000000..c644dd787446 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.10.0.md @@ -0,0 +1,10 @@ +# Release 3.10.0 (2025-09-02) + +* Based on [upstream Nix 2.31.0](../release-notes/rl-2.31.md). + +## What's Changed + +This release rebases Determinate Nix on upstream Nix 2.31.0. + + +**Full Changelog**: [v3.9.1...v3.10.0](https://github.com/DeterminateSystems/nix-src/compare/v3.9.1...v3.10.0) diff --git a/doc/manual/source/release-notes-determinate/v3.10.1.md b/doc/manual/source/release-notes-determinate/v3.10.1.md new file mode 100644 index 000000000000..08cbe4fd0583 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.10.1.md @@ -0,0 +1,9 @@ +# Release 3.10.1 (2025-09-02) + +* Based on [upstream Nix 2.31.1](../release-notes/rl-2.31.md). + +## What's Changed +This release rebases Determinate Nix on upstream Nix 2.31.1. + + +**Full Changelog**: [v3.10.0...v3.10.1](https://github.com/DeterminateSystems/nix-src/compare/v3.10.0...v3.10.1) diff --git a/doc/manual/source/release-notes-determinate/v3.11.0.md b/doc/manual/source/release-notes-determinate/v3.11.0.md new file mode 100644 index 000000000000..7abb665a5a9f --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.11.0.md @@ -0,0 +1,36 @@ +# Release 3.11.0 (2025-09-03) + +- Based on [upstream Nix 2.31.1](../release-notes/rl-2.31.md). + +## What's Changed + +### Parallel evaluation + +The following commands are now able to evaluate Nix expressions in parallel: + +- `nix search` +- `nix flake check` +- `nix flake show` +- `nix eval --json` + +This is currently in developer preview, and we'll be turning it on for more users in the coming weeks. +If you would like to try it right away, specify `eval-cores` in your `/etc/nix/nix.custom.conf`: + +```ini +eval-cores = 0 # Evaluate across all cores +``` + +Further, we introduced a new builtin: `builtins.parallel`. +This new builtin allows users to explicitly parallelize evaluation within a Nix expression. + +Using this new builtin requires turning on an additional experimental feature: + +```ini +extra-experimental-features = parallel-eval +``` + +Please note that this new builtin is subject to change semantics or even go away during the developer preview. + +PR: [DeterminateSystems/nix-src#125](https://github.com/DeterminateSystems/nix-src/pull/125) + +**Full Changelog**: [v3.10.1...v3.11.0](https://github.com/DeterminateSystems/nix-src/compare/v3.10.1...v3.11.0) diff --git a/doc/manual/source/release-notes-determinate/v3.11.1.md b/doc/manual/source/release-notes-determinate/v3.11.1.md new file mode 100644 index 000000000000..305971643330 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.11.1.md @@ -0,0 +1,9 @@ +# Release 3.11.1 (2025-09-04) + +* Based on [upstream Nix 2.31.1](../release-notes/rl-2.31.md). + +## What's Changed +* Fix race condition in Value::isTrivial() by @edolstra in [DeterminateSystems/nix-src#192](https://github.com/DeterminateSystems/nix-src/pull/192) + + +**Full Changelog**: [v3.11.0...v3.11.1](https://github.com/DeterminateSystems/nix-src/compare/v3.11.0...v3.11.1) diff --git a/doc/manual/source/release-notes-determinate/v3.11.2.md b/doc/manual/source/release-notes-determinate/v3.11.2.md new file mode 100644 index 000000000000..ac4fe569dffe --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.11.2.md @@ -0,0 +1,24 @@ +# Release 3.11.2 (2025-09-12) + +* Based on [upstream Nix 2.31.1](../release-notes/rl-2.31.md). + +## What's Changed + +### Fix some interactions with the registry and flakes that include a `?dir=` parameter + +Some users were experiencing issues when their flake registry contained a flake that included a `?dir=` parameter, causing commands like `nix eval registry-with-flake-in-subdir#output` and those that used --inputs-from` to fail or behave incorrectly. + +This is now fixed, so use your flakes inside subdirs without fear! + +PRs: [DeterminateSystems/nix-src#196](https://github.com/DeterminateSystems/nix-src/pull/196), [DeterminateSystems/nix-src#199](https://github.com/DeterminateSystems/nix-src/pull/199) + +### Only substitute inputs if they haven't already been fetched + +When using `lazy-trees`, you might have noticed Nix fetching some source inputs from a cache, even though you could have sworn it already fetched those inputs! + +This fixes that behavior such that Nix will try to fetch inputs from their original location, and only if that fails fall back to fetching from a substituter. + +PR: [DeterminateSystems/nix-src#202](https://github.com/DeterminateSystems/nix-src/pull/202) + + +**Full Changelog**: [v3.11.1...v3.11.2](https://github.com/DeterminateSystems/nix-src/compare/v3.11.1...v3.11.2) diff --git a/doc/manual/source/release-notes-determinate/v3.11.3.md b/doc/manual/source/release-notes-determinate/v3.11.3.md new file mode 100644 index 000000000000..fab5ed51a4b5 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.11.3.md @@ -0,0 +1,34 @@ +# Release 3.11.3 (2025-10-09) + +* Based on [upstream Nix 2.31.2](../release-notes/rl-2.31.md). + +## What's Changed + +### Fix some bugs and interactions with parallel eval + +We received some reports of parallel eval having issues, such as not being able to be interrupted, infinite recursion hanging forever, and segfaults when using the experimental `builtins.parallel`. + +Those have now been fixed. + +Additionally, the debugger now disables parallel eval, because the two features are incompatible. + +PRs: [DeterminateSystems/nix-src#206](https://github.com/DeterminateSystems/nix-src/pull/206), [DeterminateSystems/nix-src#213](https://github.com/DeterminateSystems/nix-src/pull/213), [DeterminateSystems/nix-src#218](https://github.com/DeterminateSystems/nix-src/pull/218), [DeterminateSystems/nix-src#205](https://github.com/DeterminateSystems/nix-src/pull/205) + +### `NIX_SSHOPTS` + `ssh-ng://root@localhost` fix + +We noticed that specifying `NIX_SSHOPTS=-p2222` when using a command that uses SSH (such as `nix copy --to ssh-ng://root@localhost`) stopped respecting the `NIX_SSHOPTS` setting because of an incorrect comparison. + +This has been fixed, so `NIX_SSHOPTS` and SSH stores that are accessed like `user@localhost` work again. + +PR: [DeterminateSystems/nix-src#219](https://github.com/DeterminateSystems/nix-src/pull/219) + +### Fix `error: [json.exception.type_error.302] type must be string, but is array` when using `exportReferencesGraph` + +We received a report of a `nix build` failing on a specific flake due to its expression using `exportReferencesGraph` with a heterogeneous array of dependencies, causing this inscrutable error. + +This specific case has been broken since Nix 2.29.0, and is now fixed. + +PRs: [DeterminateSystems/nix-src#221](https://github.com/DeterminateSystems/nix-src/pull/221), [DeterminateSystems/nix-src#225](https://github.com/DeterminateSystems/nix-src/pull/225) + + +**Full Changelog**: [v3.11.2...v3.11.3](https://github.com/DeterminateSystems/nix-src/compare/v3.11.2...v3.11.3) diff --git a/doc/manual/source/release-notes-determinate/v3.12.0.md b/doc/manual/source/release-notes-determinate/v3.12.0.md new file mode 100644 index 000000000000..55c1f10bf15f --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.12.0.md @@ -0,0 +1,17 @@ +# Release 3.12.0 (2025-10-23) + +* Based on [upstream Nix 2.32.1](../release-notes/rl-2.32.md). + +## What's Changed + +### `nix nario` + +Determinate Nix has a new command, `nix nario`, that replaces the commands `nix-store --export` and `nix-store --import` from the old CLI. `nix nario` allows you to serialize store paths to a file that can be imported into another Nix store. It is backwards compatible with the file format generated by `nix-store --export`. It also provides a new format (selected by passing `--format 2`) that supports store path attributes such as signatures, and allows store paths to be imported more efficiently. + +### Other changes + +`nix flake clone` now supports arbitrary input types. In particular, this allows you to clone tarball flakes, such as flakes on FlakeHub. + +When using `-vv`, Determinate Nix now prints the Nix version. This is useful when diagnosing Nix problems from the debug output of a Nix run. + +**Full Changelog**: [v3.11.3...v3.12.0](https://github.com/DeterminateSystems/nix-src/compare/v3.11.3...v3.12.0) diff --git a/doc/manual/source/release-notes-determinate/v3.12.1.md b/doc/manual/source/release-notes-determinate/v3.12.1.md new file mode 100644 index 000000000000..1be2b48e26d8 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.12.1.md @@ -0,0 +1,10 @@ +# Release 3.12.1 (2025-11-04) + +* Based on [upstream Nix 2.32.1](../release-notes/rl-2.32.md). + +## What's Changed +* Allow access to the result of fetchClosure by @edolstra in [DeterminateSystems/nix-src#241](https://github.com/DeterminateSystems/nix-src/pull/241) +* libstore/build: fixup JSON logger missing the resBuildResult result event by @cole-h in [DeterminateSystems/nix-src#246](https://github.com/DeterminateSystems/nix-src/pull/246) + + +**Full Changelog**: [v3.12.0...v3.12.1](https://github.com/DeterminateSystems/nix-src/compare/v3.12.0...v3.12.1) diff --git a/doc/manual/source/release-notes-determinate/v3.12.2.md b/doc/manual/source/release-notes-determinate/v3.12.2.md new file mode 100644 index 000000000000..4c8c3169aa72 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.12.2.md @@ -0,0 +1,42 @@ +# Release 3.12.2 (2025-11-05) + +* Based on [upstream Nix 2.32.2](../release-notes/rl-2.32.md). + +## What's Changed + +### Faster `revCount` computation + +When using Git repositories with a long history, calculating the `revCount` attribute can take a long time. Determinate Nix now computes `revCount` using multiple threads, making it much faster. + +Note that if you don't need `revCount`, you can disable it altogether by setting the flake input attribute `shallow = true`. + +PR: [DeterminateSystems/nix-src#245](https://github.com/DeterminateSystems/nix-src/pull/245) + +### More readable error messages + +Previously, Nix showed full flakerefs in error messages such as stack traces, e.g. +``` + … from call site + at «github:NixOS/nixpkgs/3bea86e918d8b54aa49780505d2d4cd9261413be?narHash=sha256-Ica%2B%2BSXFuLyxX9Q7YxhfZulUif6/gwM8AEQYlUxqSgE%3D»/lib/customisation.nix:69:16: + 68| let + 69| result = f origArgs; + | ^ + 70| +``` +It now abbreviates these by leaving out `narHash` and shortening Git revisions: +``` + … from call site + at «github:NixOS/nixpkgs/3bea86e»/lib/customisation.nix:69:16: + 68| let + 69| result = f origArgs; + | ^ + 70| +``` + +PR: [DeterminateSystems/nix-src#243](https://github.com/DeterminateSystems/nix-src/pull/243) + +### Other changes + +This release fixes an assertion failure in `nix flake check`. PR: [DeterminateSystems/nix-src#252](https://github.com/DeterminateSystems/nix-src/pull/252) + +**Full Changelog**: [v3.12.1...v3.12.2](https://github.com/DeterminateSystems/nix-src/compare/v3.12.1...v3.12.2) diff --git a/doc/manual/source/release-notes-determinate/v3.13.0.md b/doc/manual/source/release-notes-determinate/v3.13.0.md new file mode 100644 index 000000000000..09041c2acda0 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.13.0.md @@ -0,0 +1,45 @@ +# Release 3.13.0 (2025-11-09) + +* Based on [upstream Nix 2.32.3](../release-notes/rl-2.32.md). + +## What's Changed + + +### Git sources have a progress indicator again + +Nix used to feel "stuck" while it was cloning large repositories. +Determinate Nix now shows git's native progress indicator while fetching. + +PR: [DeterminateSystems/nix-src#250](https://github.com/DeterminateSystems/nix-src/pull/250) + +### C API improvements + +We've invested in the C API to support our work on closure analysis for SBOM generation, and made a couple of changes: + +* C API: add nix_locked_flake_read_path for flake file reading +* C API: make nix_store_get_fs_closure compatible with upstream + +PRs: +* [DeterminateSystems/nix-src#244](https://github.com/DeterminateSystems/nix-src/pull/244) +* [DeterminateSystems/nix-src#254](https://github.com/DeterminateSystems/nix-src/pull/254) + +### Dropping support for Intel Macs + +Determinate Nix no longer supports being installed on Intel Macs. +Determinate Nix will continue to support building for Intel macOS targets, but only from an Apple Silicon host. + +From our intent-to-ship: +> Over the past year, we’ve watched usage of Determinate on Intel macOS hosts dwindle to a minuscule fraction of total usage. +> It currently stands at approximately 0.02% of all installations. +> The vast majority are run in managed CI environments that, we anticipate, will be able to easily convert to using Apple Silicon runners. + +For more information: https://github.com/DeterminateSystems/nix-src/issues/224 + +PR: [DeterminateSystems/nix-src#257](https://github.com/DeterminateSystems/nix-src/pull/257) + +### Bugs fixed + +* IPv6 Store URLs now handles zone ID references like it did in previous releases [NixOS/nix#14434](https://github.com/NixOS/nix/pull/14434) + + +**Full Changelog**: [v3.12.2...v3.13.0](https://github.com/DeterminateSystems/nix-src/compare/v3.12.2...v3.13.0) diff --git a/doc/manual/source/release-notes-determinate/v3.13.1.md b/doc/manual/source/release-notes-determinate/v3.13.1.md new file mode 100644 index 000000000000..025a192c44ee --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.13.1.md @@ -0,0 +1,10 @@ +# Release 3.13.1 (2025-11-12) + +* Based on [upstream Nix 2.32.4](../release-notes/rl-2.32.md). + +## What's Changed +* nix bundle: Wait for async path writer by @edolstra in [DeterminateSystems/nix-src#260](https://github.com/DeterminateSystems/nix-src/pull/260) +* Sync with upstream 2.32.4 by @edolstra in [DeterminateSystems/nix-src#261](https://github.com/DeterminateSystems/nix-src/pull/261) + + +**Full Changelog**: [v3.13.0...v3.13.1](https://github.com/DeterminateSystems/nix-src/compare/v3.13.0...v3.13.1) diff --git a/doc/manual/source/release-notes-determinate/v3.13.2.md b/doc/manual/source/release-notes-determinate/v3.13.2.md new file mode 100644 index 000000000000..2490b865e6bc --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.13.2.md @@ -0,0 +1,68 @@ +# Release 3.13.2 (2025-11-19) + +* Based on [upstream Nix 2.32.4](../release-notes/rl-2.32.md). + +## What's Changed + +### Abbreviate flakerefs in lockfile diffs and `nix flake metadata` + +Flake refs are now abbreviated when possible, to reduce visual clutter. + +For example, this changes + +``` +• Updated input 'blender-bin': + 'https://api.flakehub.com/f/pinned/edolstra/blender-bin/1.0.19/01993ca7-2aa8-746f-96f5-ca8d2c2b962d/source.tar.gz?narHash=sha256-ZqVhVl9UYVErF8HW8lcvqss005VWYjuX//rZ%2BOmXyHg%3D' (2025-09-12) + → 'https://api.flakehub.com/f/pinned/edolstra/blender-bin/1.0.20/019a8772-b044-7738-8c03-109bdc9f0a01/source.tar.gz?narHash=sha256-sVj9Gmx0kwTDQPJ5kgQYszE3Hdjevu0zx0b/bL2fyUc%3D' (2025-11-15) +• Updated input 'nix': + 'github:DeterminateSystems/nix-src/236ebef6514f3a2a9765c8a1d80dd503b8e672be?narHash=sha256-s6/Err0yqOp5fM3OdCF1vhmEYpeElbPOWX88YrW2qj4%3D' (2025-10-23) + → 'github:DeterminateSystems/nix-src/ef054dc06e9701597bce0b0572af18cb4c7e7277?narHash=sha256-uqYmH0KA8caQqX5u4BMarZsuDlC%2B71HRsH3h4f3DPCA%3D' (2025-11-12) +``` + +to + +``` +• Updated input 'blender-bin': + 'https://api.flakehub.com/f/pinned/edolstra/blender-bin/1.0.19/01993ca7-2aa8-746f-96f5-ca8d2c2b962d/source.tar.gz' (2025-09-12) + → 'https://api.flakehub.com/f/pinned/edolstra/blender-bin/1.0.20/019a8772-b044-7738-8c03-109bdc9f0a01/source.tar.gz' (2025-11-15) +• Updated input 'nix': + 'github:DeterminateSystems/nix-src/236ebef' (2025-10-23) + → 'github:DeterminateSystems/nix-src/ef054dc' (2025-11-12) +``` + +PR: [DeterminateSystems/nix-src#264](https://github.com/DeterminateSystems/nix-src/pull/264) + +### `nix flake prefetch-inputs` now skips build-time inputs + +Build-time inputs can already be fetched in parallel, so prefetching them is usually not what you want. + +This can be especially noticeable in projects that make extensive use of build-time flake inputs. + +PR: [DeterminateSystems/nix-src#263](https://github.com/DeterminateSystems/nix-src/pull/263) + +### Don't compute `revCount`/`lastModified` if they're already specified + +We don't care if the user (or more likely the lock file) specifies an incorrect value for these attributes, since it doesn't matter for security (unlike content hashes like `narHash`). + +This can save time when operating on large repos -- having to recalculate these attributes could slow things down greatly. + +PR: [DeterminateSystems/nix-src#269](https://github.com/DeterminateSystems/nix-src/pull/269) + +### Avoid unnecessary Git refetches + +This fixes the issue where updating a Git input does a non-shallow fetch, and then a subsequent eval does a shallow refetch because the `revCount` is already known. + +Now the subsequent eval will reuse the repo used in the first fetch. + +PR: [DeterminateSystems/nix-src#270](https://github.com/DeterminateSystems/nix-src/pull/270) + +### Use our mirrored flake registry + +The flake registry is security-critical and thus should have high availability. + +By mirroring the upstream Nix flake registry, we can make it less likely that a GitHub outage affects being able to resolve from the registry. + +PR: [DeterminateSystems/nix-src#271](https://github.com/DeterminateSystems/nix-src/pull/271) + + +**Full Changelog**: [v3.13.1...v3.13.2](https://github.com/DeterminateSystems/nix-src/compare/v3.13.1...v3.13.2) diff --git a/doc/manual/source/release-notes-determinate/v3.14.0.md b/doc/manual/source/release-notes-determinate/v3.14.0.md new file mode 100644 index 000000000000..d72d5d21468c --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.14.0.md @@ -0,0 +1,159 @@ +# Release 3.14.0 (2025-12-08) + +* Based on [upstream Nix 2.32.4](../release-notes/rl-2.32.md). + +## What is going on?! `nix ps` to the rescue + +Determinate Nix now features a `nix ps` command to summarize all of the active builds and child processes: + +``` +$ nix ps +USER PID CPU DERIVATION/COMMAND +_nixbld1 30167 0.4s /nix/store/h431bcfml83czhpyzljhp9mw4yrq95vs-determinate-nix-manual-3.14.0.drv (wall=9s) +_nixbld1 30167 0.2s └───bash -e /nix/store/jwqf79v5p51x9mv8vx20fv9mzm2x7kig-source-stdenv.sh /nix/store/285whzixr5k1kfj6nidyj29mqqgv7n0b-default-builder.s +_nixbld1 30278 0.0s └───ninja -j14 +_nixbld1 30279 0.0s ├───/nix/store/p7rag2cw99d7alp6749rjqp71qc0mnzl-python3-3.12.11/bin/python3.12 /nix/store/8k5fancbc5fjmxq6izn0z4inwnmpj09y-mes +_nixbld1 30286 0.0s │ └───/nix/store/z59zm01pjwzil2qkvv0s4ibk54risy9a-determinate-nix-3.14.0/bin/nix config show --json +_nixbld1 30280 0.0s ├───/nix/store/p7rag2cw99d7alp6749rjqp71qc0mnzl-python3-3.12.11/bin/python3.12 /nix/store/8k5fancbc5fjmxq6izn0z4inwnmpj09y-mes +_nixbld1 30287 0.0s │ └───/nix/store/z59zm01pjwzil2qkvv0s4ibk54risy9a-determinate-nix-3.14.0/bin/nix __dump-language +_nixbld1 30281 0.0s ├───/nix/store/p7rag2cw99d7alp6749rjqp71qc0mnzl-python3-3.12.11/bin/python3.12 /nix/store/8k5fancbc5fjmxq6izn0z4inwnmpj09y-mes +_nixbld1 30288 0.0s │ └───/nix/store/z59zm01pjwzil2qkvv0s4ibk54risy9a-determinate-nix-3.14.0/bin/nix __dump-cli +_nixbld1 30282 0.0s ├───/nix/store/p7rag2cw99d7alp6749rjqp71qc0mnzl-python3-3.12.11/bin/python3.12 /nix/store/8k5fancbc5fjmxq6izn0z4inwnmpj09y-mes +_nixbld1 30284 0.0s │ └───/nix/store/z59zm01pjwzil2qkvv0s4ibk54risy9a-determinate-nix-3.14.0/bin/nix __dump-xp-features +_nixbld1 30283 0.0s └───/nix/store/p7rag2cw99d7alp6749rjqp71qc0mnzl-python3-3.12.11/bin/python3.12 /nix/store/8k5fancbc5fjmxq6izn0z4inwnmpj09y-mes +_nixbld1 30285 0.0s └───/nix/store/bs1pvy8margy5sj0jwahchxbjnqzi14i-bash-5.2p37/bin/bash -euo pipefail -c if type -p build-release-notes > /de +_nixbld1 30289 0.0s └───changelog-d ../source/release-notes/../../rl-next +``` + +For the integrators out there, it also has a `--json` flag with all the raw data. + +PRs: +* [DeterminateSystems/nix-src#282](https://github.com/DeterminateSystems/nix-src/pull/282) +* [DeterminateSystems/nix-src#287](https://github.com/DeterminateSystems/nix-src/pull/287) + + +## Nix `build`, `profile`, and `flake check` commands tell you what output failed + +These commands now tell you exactly what flake outputs failed to build. +Previously, the error would indicate only what derivation failed to build -- but not which output. + +Now, `nix build` and `nix profile` commands provide the specific output: + +``` +$ nix build .#oneFakeHash .#badSystem --keep-going +❌ git+file:///Users/grahamc/src/github.com/DeterminateSystems/samples#oneFakeHash +error: hash mismatch in fixed-output derivation '/nix/store/58pp1y74j4f5zxfq50xncv2wvnxf7w3y-one-fake-hash.drv': + specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= + got: sha256-i7j83d71sibS/ssSjLJ5PMKmbhjAM+BHW0aElvkgEwY= +❌ git+file:///Users/grahamc/src/github.com/DeterminateSystems/samples#badSystem +error: Cannot build '/nix/store/5vsaxi730yl2icngkyvn8wiflik5wfmq-bad-system.drv'. + Reason: required system or feature not available + Required system: 'bogus' with features {} + Current system: 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test} +``` + +And in a great change for CI, `nix flake check` users get improved summaries too: + +``` +$ nix flake check +❓ checks.aarch64-darwin.twoFakeHashes (cancelled) +❓ checks.aarch64-darwin.badSystemNested (cancelled) +❓ checks.aarch64-darwin.oneFakeHash (cancelled) +❓ checks.aarch64-darwin.failure (cancelled) +❓ checks.aarch64-darwin.badSystem (cancelled) +❓ checks.aarch64-darwin.weirdHash (cancelled) +❓ checks.aarch64-darwin.all (cancelled) +❓ checks.aarch64-darwin.fakeHashes (cancelled) +❓ checks.aarch64-darwin.incorrectHashes (cancelled) +❓ checks.aarch64-darwin.badFeaturesNested (cancelled) +❓ checks.aarch64-darwin.failureNested (cancelled) +❌ checks.aarch64-darwin.badFeatures +error: Cannot build '/nix/store/sc1cyhrpsm9yjx55cl2zzyr5lypwigi6-bad-feature.drv'. + Reason: required system or feature not available + Required system: 'aarch64-darwin' with features {bogus} + Current system: 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test} +``` + +PRs: +* [DeterminateSystems/nix-src#281](https://github.com/DeterminateSystems/nix-src/pull/281) +* [DeterminateSystems/nix-src#285](https://github.com/DeterminateSystems/nix-src/pull/285) + + +## More seamless upgrades from Nix 2.18 and Nix 2.19 + +We've heard from some users who are trying to upgrade from Nix 2.18. + +These users are primarily experiencing problems caused by Nix 2.20 switching from `git-archive` to `libgit2` for fetching repositories. +This change caused some `git-archive` filters to stop executing, like autocrlf. +Not running those filters is an improvement, and running those filters *can cause* instability in source hashes. +However, this switch *did* cause previously valid hashes to become invalid. + +Determinate Nix now retries fetching an old archive with `git-archive` as a fallback when libgit2 fails to provide the correct source. + +Further, to support a progressive migration Determinate Nix has a new option: `nix-219-compat`. +Set `nix-219-compat=true` to cause Nix to author new flake.nix files with a `git-archive` based source hash. + +Finally, a user identified `builtins.path` changed since 2.18 and stopped propagating references. +We have corrected this regression. + +PRs: +* [DeterminateSystems/nix-src#283](https://github.com/DeterminateSystems/nix-src/pull/283) +* [DeterminateSystems/nix-src#278](https://github.com/DeterminateSystems/nix-src/pull/278) + +## Flake registry mirroring + +Determinate Nix now includes a fallback copy of the Nix Registry. +This change builds on top of v3.13.2, where we changed from the upstream Nix registry to a mirrored copy hosted by `install.determinate.systems`. + +Combined, these changes increase the reliability of Nix in the face of network outages. + +> [!NOTE] +> Flake registry URLs for `flake.nix` inputs is deprecated. +> The flake registry should only be used for interactive use. +> See: https://github.com/DeterminateSystems/nix-src/issues/37 + +PR: [DeterminateSystems/nix-src#273](https://github.com/DeterminateSystems/nix-src/pull/273) + +## Flake registry resolution CLI + +We added the new command `nix registry resolve` to help debug issues with Flake registries. +This command looks up a flake registry input name and returns the flakeref it resolves to. + +For example, looking up Nixpkgs: + +``` +$ nix registry resolve nixpkgs +github:NixOS/nixpkgs/nixpkgs-unstable +``` + +Or looking up the 25.11 branch of Nixpkgs: +``` +$ nix registry resolve nixpkgs/release-25.11 +github:NixOS/nixpkgs/release-25.11 +``` + +> [!NOTE] +> Flake registry URLs for `flake.nix` inputs is deprecated. +> The flake registry should only be used for interactive use. +> See: https://github.com/DeterminateSystems/nix-src/issues/37 + +PR: [DeterminateSystems/nix-src#273](https://github.com/DeterminateSystems/nix-src/pull/273) + +## Improved Docker image packaging + +Thanks to `employee-64c7dcd530593118dcccc3fb`, the OCI / Docker images built by the Determinate Nix flake.nix can be further customized. + +Users can specify their own base image by specifying `fromImage`. + +Additionally, users can specify additional directories to include at the beginning or end of the PATH variable with `extraPrePaths` and `extraPostPaths`. + +PRs: +* [DeterminateSystems/nix-src#277](https://github.com/DeterminateSystems/nix-src/pull/277) +* [DeterminateSystems/nix-src#280](https://github.com/DeterminateSystems/nix-src/pull/280) + +## Bug fixes + +* Corrected an error with parallel evaluation which ([DeterminateSystems/nix-src#286](https://github.com/DeterminateSystems/nix-src/pull/286)) +* Fixed compatibility with updated Nixpkgs versions. Thank you SandaruKasa! ([DeterminateSystems/nix-src#284](https://github.com/DeterminateSystems/nix-src/pull/284)) + +**Full Changelog**: [v3.13.2...v3.14.0](https://github.com/DeterminateSystems/nix-src/compare/v3.13.2...v3.14.0) diff --git a/doc/manual/source/release-notes-determinate/v3.15.0.md b/doc/manual/source/release-notes-determinate/v3.15.0.md new file mode 100644 index 000000000000..fb568374c3f2 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.15.0.md @@ -0,0 +1,28 @@ +# Release 3.15.0 (2025-12-19) + +* Based on [upstream Nix 2.33.0](../release-notes/rl-2.33.md). + +## `fetchTree` improvement + +`builtins.fetchTree` now implicitly treats the fetched tree as "final" when a `narHash` is supplied, meaning that it will not return attributes like `lastModified` or `revCount` unless they were specified by the caller. This makes it possible to substitute the tree from a binary cache, which is often more efficient. Furthermore, for Git inputs, it allows Nix to perform a shallow fetch, which is much faster. + +This is primarily useful for users of `flake-compat`, since it uses `builtins.fetchTree` internally. + +PR: [DeterminateSystems/nix-src#297](https://github.com/DeterminateSystems/nix-src/pull/297) + +## New builtin function `builtins.filterAttrs` + +Nixpkgs heavily relies on this function to select attributes from an attribute set: + +```nix +filterAttrs = pred: set: removeAttrs set (filter (name: !pred name set.${name}) (attrNames set)); +``` + +Determinate Nix now has this function built-in, which makes it much faster. + +PR: [DeterminateSystems/nix-src#291](https://github.com/DeterminateSystems/nix-src/pull/291) + +## New Contributors +* @not-ronjinger made their first contribution in [DeterminateSystems/nix-src#291](https://github.com/DeterminateSystems/nix-src/pull/291) + +**Full Changelog**: [v3.14.0...v3.15.0](https://github.com/DeterminateSystems/nix-src/compare/v3.14.0...v3.15.0) diff --git a/doc/manual/source/release-notes-determinate/v3.15.1.md b/doc/manual/source/release-notes-determinate/v3.15.1.md new file mode 100644 index 000000000000..9243962cf4b5 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.15.1.md @@ -0,0 +1,15 @@ +# Release 3.15.1 (2025-12-24) + +* Based on [upstream Nix 2.33.0](../release-notes/rl-2.33.md). + +## What's Changed +Users reported the v3.15.0 tarball could not be fetched in a fixed-output derivation due to current stdenv paths present in the documentation. This release eliminated those paths. + +PR: [DeterminateSystems/nix-src#306](https://github.com/DeterminateSystems/nix-src/pull/306) + +Additionally, this change re-enables CodeRabbit's code review on our changes. CodeRabit was disabled by the upstream project, and we inadvertently included that change. + +PR: [DeterminateSystems/nix-src#305](https://github.com/DeterminateSystems/nix-src/pull/305) + + +**Full Changelog**: [v3.15.0...v3.15.1](https://github.com/DeterminateSystems/nix-src/compare/v3.15.0...v3.15.1) diff --git a/doc/manual/source/release-notes-determinate/v3.15.2.md b/doc/manual/source/release-notes-determinate/v3.15.2.md new file mode 100644 index 000000000000..c5e5339990b5 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.15.2.md @@ -0,0 +1,44 @@ +# Release 3.15.2 (2026-01-20) + +* Based on [upstream Nix 2.33.1](../release-notes/rl-2.33.md). + +## What's Changed + +### Improved performance for users with a lot of dependencies + +If you even had the occasion to query your binary cache for over 110,000 store path simultaneously you might have found it rather slow. +Previously, Nix would enqueue all the downloads at once. +This appears to trigger quadratic behavior in curl. + +Determinate Nix now enqueues a reasonable number of subtitutions once. +At the same time, we fixed a performance issue in the progress bar with so many dependencies. + +PR: [DeterminateSystems/nix-src#315](https://github.com/DeterminateSystems/nix-src/pull/315) + +### Lazy trees update: path inputs are now lazy + +Previously inputs like `path:///path/to/a/dependency` were eagerly fetched when lazy-trees is enabled. +In Determinate Nix 3.15.2, path input types are also fetched lazily. +This change saves time and improves performance for users with path inputs. + +PRs: +* [DeterminateSystems/nix-src#312](https://github.com/DeterminateSystems/nix-src/pull/312) +* [DeterminateSystems/nix-src#317](https://github.com/DeterminateSystems/nix-src/pull/317) + +### `nix repl` now reports the Determinate version + +A small change, but now `nix repl` correctly reports the Determinate Nix version: + +``` +$ nix repl +Nix (Determinate Nix 3.15.1) 2.33.0 +Type :? for help. +nix-repl> +``` + +PR: [DeterminateSystems/nix-src#316](https://github.com/DeterminateSystems/nix-src/pull/316) + +## New Contributors +* @dliberalesso made their first contribution in [DeterminateSystems/nix-src#313](https://github.com/DeterminateSystems/nix-src/pull/313) + +**Full Changelog**: [v3.15.1...v3.15.2](https://github.com/DeterminateSystems/nix-src/compare/v3.15.1...v3.15.2) diff --git a/doc/manual/source/release-notes-determinate/v3.16.0.md b/doc/manual/source/release-notes-determinate/v3.16.0.md new file mode 100644 index 000000000000..8e80ac68402a --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.16.0.md @@ -0,0 +1,53 @@ +# Release 3.16.0 (2026-02-12) + +* Based on [upstream Nix 2.33.3](../release-notes/rl-2.33.md). + +## Support `.gitattributes` in subdirectories + +For performance, the Git backwards compatibility hack was only applied to repositories that had a `.gitattributes` in the root directory. +However, it is possible to have a `.gitattributes` file in a subdirectory, and there are real-world repos that do this, so we have dropped that restriction. + +PR: [DeterminateSystems/nix-src#335](https://github.com/DeterminateSystems/nix-src/pull/335) + +## Fix hung downloads when `http-connections = 0` + +When we started limiting the number of active cURL handles in [DeterminateSystems/nix-src#315](https://github.com/DeterminateSystems/nix-src/pull/315), we did not take into account that `http-connections = 0` is a special value that means, roughly "as many connections as possible" (the exact behavior is up to cURL). + +This should now be fixed. + +PR: [DeterminateSystems/nix-src#327](https://github.com/DeterminateSystems/nix-src/pull/327) + +## `builtins.getFlake` now supports relative paths + +`builtins.getFlake` now supports using relative paths, like: + +```nix +builtins.getFlake ./.. +``` + +instead of the hacky + +```nix +builtins.getFlake (builtins.flakeRefToString { type = "path"; path = self.sourceInfo.outPath; narHash = self.narHash; }); +``` + +Note that allowing `builtins.getFlake` to fetch from store paths is probably a bad idea, since it's ambiguous when using chroot stores, so a warning will be printed when this is encountered. + +PRs: +* [DeterminateSystems/nix-src#337](https://github.com/DeterminateSystems/nix-src/pull/337) +* [DeterminateSystems/nix-src#338](https://github.com/DeterminateSystems/nix-src/pull/338) + +## Fixed a bug with too many open files + +Recently, some users have reported seeing errors like: + +``` +error: creating git packfile indexer: failed to create temporary file '/Users/anon/.cache/nix/tarball-cache-v2/objects/pack/pack_git2_56d617039ac17c2b': Too many open files +``` + +This should now be fixed. + +PR: [DeterminateSystems/nix-src#347](https://github.com/DeterminateSystems/nix-src/pull/347) + + +**Full Changelog**: [v3.15.2...v3.16.0](https://github.com/DeterminateSystems/nix-src/compare/v3.15.2...v3.16.0) diff --git a/doc/manual/source/release-notes-determinate/v3.16.1.md b/doc/manual/source/release-notes-determinate/v3.16.1.md new file mode 100644 index 000000000000..6ecd5262b7c9 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.16.1.md @@ -0,0 +1,24 @@ +# Release 3.16.1 (2026-02-22) + +* Based on [upstream Nix 2.33.3](../release-notes/rl-2.33.md). + +## What's Changed + +### `nix store info` now correctly support `--refresh` and `--offline` + +Previously, Nix had a hard-coded TTL of seven days. +Determinate Nix moved that TTL to a new setting `narinfo-cache-meta-ttl` and now `nix store info` respects the `--refresh` and `--offline` flags. + +This change makes it possible to freshly validate authenticating to a remote store. + +PR: [DeterminateSystems/nix-src#355](https://github.com/DeterminateSystems/nix-src/pull/355) + +### Corrected `builtins.hashString` behavior under lazy trees + +`builtins.hashString` now devirtualizes lazy paths, making the hash result stable. + +PR: [DeterminateSystems/nix-src#360](https://github.com/DeterminateSystems/nix-src/pull/360) + + + +**Full Changelog**: [v3.16.0...v3.16.1](https://github.com/DeterminateSystems/nix-src/compare/v3.16.0...v3.16.1) diff --git a/doc/manual/source/release-notes-determinate/v3.16.2.md b/doc/manual/source/release-notes-determinate/v3.16.2.md new file mode 100644 index 000000000000..73a1b25f21c8 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.16.2.md @@ -0,0 +1,8 @@ +# Release 3.16.2 (2026-02-23) + +* Based on [upstream Nix 2.33.3](../release-notes/rl-2.33.md). + +## What's Changed +This release is exclusively improvements to `determinate-nixd`. + + diff --git a/doc/manual/source/release-notes-determinate/v3.16.3.md b/doc/manual/source/release-notes-determinate/v3.16.3.md new file mode 100644 index 000000000000..fcc6fefa33c7 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.16.3.md @@ -0,0 +1,6 @@ +# Release 3.16.3 (2026-02-24) + +* Based on [upstream Nix 2.33.3](../release-notes/rl-2.33.md). + +## What's Changed +This release only includes changes in determinate-nixd. diff --git a/doc/manual/source/release-notes-determinate/v3.17.0.md b/doc/manual/source/release-notes-determinate/v3.17.0.md new file mode 100644 index 000000000000..e09938786e55 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.17.0.md @@ -0,0 +1,10 @@ +# Release 3.17.0 (2026-03-04) + +* Based on [upstream Nix 2.33.3](../release-notes/rl-2.33.md). + +## What's Changed +Determinate Nix 3.17.0 brings exciting improvements like Flake Schemas, provenance, and Wasm / WASI. +We'll be posting more details over the next week or so on our blog: https://determinate.systems/blog/. + + +**Full Changelog**: [v3.16.3...v3.17.0](https://github.com/DeterminateSystems/nix-src/compare/v3.16.3...v3.17.0) diff --git a/doc/manual/source/release-notes-determinate/v3.17.1.md b/doc/manual/source/release-notes-determinate/v3.17.1.md new file mode 100644 index 000000000000..19da0209931e --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.17.1.md @@ -0,0 +1,49 @@ +# Release 3.17.1 (2026-03-18) + +* Based on [upstream Nix 2.33.3](../release-notes/rl-2.33.md). + +## What's Changed + +### Provenance now supports "build-time" provenance tags + +Users can now specify key/value tags to attach to a build's provenance. +For example, the `nix-installer-action` now attaches the following properties to each build: + +- github_workflow_ref +- github_workflow_sha +- github_sha +- github_run_attempt +- github_run_id +- github_run_number +- github_job +- github_ref +- github_repository +- github_server_url + +PR: [DeterminateSystems/nix-src#374](https://github.com/DeterminateSystems/nix-src/pull/374) + +### Flake inputs are substituted when possible + +Locked flake inputs will be fetched from a binary cache when possible, instead of preferring the authoritative flake source. +This is intended to reduce load on code forges, and also improves the user experience on large flake inputs. + +PR: [DeterminateSystems/nix-src#380](https://github.com/DeterminateSystems/nix- + +### `nix profile upgrade` and `nix profile remove` now support tab completion + +PR: [DeterminateSystems/nix-src#382](https://github.com/DeterminateSystems/nix-src/pull/382) + +### Flake schemas can now define an output as "legacy" + +"Legacy" flakes are intended for legacyPackages on Nixpkgs. +The "legacy" mark is intended to reduce evaluation time due to the extreme size of legacyPackages. +Note: the name "legacy" is not intended as a value judgement, and at this point we're sort of stuck with the name. + +### Bug fixes + +* Fix crash in `nix repl` loading an invalid WASM file twice. [DeterminateSystems/nix-src#378](https://github.com/DeterminateSystems/nix-src/pull/378) +* Don't crash if SIGINT happens while printing an exception. [DeterminateSystems/nix-src#384](https://github.com/DeterminateSystems/nix-src/pull/384) +* `nix-env -i`: Wait for the async path writer. [DeterminateSystems/nix-src#385](https://github.com/DeterminateSystems/nix-src/pull/385) + + +**Full Changelog**: [v3.17.0...v3.17.1](https://github.com/DeterminateSystems/nix-src/compare/v3.17.0...v3.17.1) diff --git a/doc/manual/source/release-notes-determinate/v3.17.2.md b/doc/manual/source/release-notes-determinate/v3.17.2.md new file mode 100644 index 000000000000..ce45a8df7ec7 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.17.2.md @@ -0,0 +1,16 @@ +# Release 3.17.2 (2026-03-27) + +* Based on [upstream Nix 2.33.3](../release-notes/rl-2.33.md). + +## What's Changed + +### Bug fixes + +* Cache `getLegacyGitAccessor()`. [DeterminateSystems/nix-src#396](https://github.com/DeterminateSystems/nix-src/pull/396) +* Don't destroy `windowSize` mutex. [DeterminateSystems/nix-src#397](https://github.com/DeterminateSystems/nix-src/pull/397) +* When doing concurrent substitutions of the same path, download only once. [DeterminateSystems/nix-src#398](https://github.com/DeterminateSystems/nix-src/pull/398) +* `builtins.getFlake`: Handle `path:

` where *p* has a discarded string context. [DeterminateSystems/nix-src#402](https://github.com/DeterminateSystems/nix-src/pull/402) +* Ensure `_interruptCallbacks` is alive while `signalHandlerThread` is running. [DeterminateSystems/nix-src#403](https://github.com/DeterminateSystems/nix-src/pull/403) +* Fix assertion failure in `nix::BuiltPath::toRealisedPaths()`. [DeterminateSystems/nix-src#401](https://github.com/DeterminateSystems/nix-src/pull/401) + +**Full Changelog**: [v3.17.1...v3.17.2](https://github.com/DeterminateSystems/nix-src/compare/v3.17.1...v3.17.2) diff --git a/doc/manual/source/release-notes-determinate/v3.17.3.md b/doc/manual/source/release-notes-determinate/v3.17.3.md new file mode 100644 index 000000000000..24463e48485f --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.17.3.md @@ -0,0 +1,48 @@ +# Release 3.17.3 (2026-04-07) + +* Based on [upstream Nix 2.33.3](../release-notes/rl-2.33.md). + +## What's Changed + +### Fix for GHSA-g3g9-5vj6-r3gj: root privilege escalation via the Nix daemon + +This release contains a critical fix for a security vulnerability on Linux that allows any user that has access to the Nix daemon to obtain root privileges. +This vulnerability affects all versions of Determinate Nix prior to 3.17.3, and all versions of upstream Nix prior to 2.34.5, 2.33.4, 2.32.7, 2.31.4, 2.30.4, 2.29.3, and 2.28.6. +All Linux users are advised to upgrade immediately. + +For more details, see the upstream advisory [GHSA-g3g9-5vj6-r3gj](https://github.com/NixOS/nix/security/advisories/GHSA-g3g9-5vj6-r3gj). +Many thanks to edef for reporting this issue and to Sergei Zimmerman for implementing the fix. + +### Avoiding duplicate source tree downloads + +When multiple Nix processes (like `nix-eval-jobs` instances) fetch the same source tree at the same time, it was previously possible for each process to perform the download independently, resulting in wasteful multiple downloads of the same source tree. Nix now uses a per-source tree lock to ensure that only one process performs the download. + +PR: [DeterminateSystems/nix-src#410](https://github.com/DeterminateSystems/nix-src/pull/410) + +### WAT support in `builtins.wasm` + +`builtins.wasm` now supports WebAssembly Text Format (WAT) in addition to binary Wasm modules. +This is primarily useful for testing. + +PR: [DeterminateSystems/nix-src#405](https://github.com/DeterminateSystems/nix-src/pull/405) + +### Git shallow fetching + +Nix now removes Git's `shallow.lock` lock file before running `git fetch`. +This prevents fetches from hanging if Git was previously interrupted. + +PR: [DeterminateSystems/nix-src#414](https://github.com/DeterminateSystems/nix-src/pull/414) + +### Debugging improvements + +Certain C++ exceptions that should never happen (like `std::logic_error`) are now treated as aborts, providing stack traces and core dumps that are easier to debug. + +PR: [DeterminateSystems/nix-src#407](https://github.com/DeterminateSystems/nix-src/pull/407) + +### Bug fixes + +* `nix develop` respects `legacyPackages` again. [DeterminateSystems/nix-src#413](https://github.com/DeterminateSystems/nix-src/pull/413) + +* The `lookupPathResolved` data structure has been made thread-safe. [DeterminateSystems/nix-src#415](https://github.com/DeterminateSystems/nix-src/pull/415) + +**Full Changelog**: [v3.17.2...v3.17.3](https://github.com/DeterminateSystems/nix-src/compare/v3.17.2...v3.17.3) diff --git a/doc/manual/source/release-notes-determinate/v3.18.0.md b/doc/manual/source/release-notes-determinate/v3.18.0.md new file mode 100644 index 000000000000..aaf22deec8c8 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.18.0.md @@ -0,0 +1,42 @@ +# Release 3.18.0 (2026-04-20) + +* Based on [upstream Nix 2.33.3](../release-notes/rl-2.33.md). + +## What's Changed + +### Sentry integration + +In order to more proactively keep track of crashes, Sentry is now integrated into Determinate Nix. + +This allows us to more easily triage and remedy crashes that occur in the wild, without depending on manual use reports. + +It can be enabled by: + +* populating the file `/etc/nix/sentry-endpoint` with a Sentry DSN; or +* setting the `NIX_SENTRY_ENDPOINT` environment variable to a Sentry DSN + +and can be disabled by: + +* setting the environment variable `DETSYS_IDS_TELEMETRY` to the value `disabled`; or +* setting the environment variable `NIX_SENTRY_ENDPOINT` to an empty string + +PR: [DeterminateSystems/nix-src#418](https://github.com/DeterminateSystems/nix-src/pull/418) + +### Pre-build hook now receives the JSON serialization of the derivation + +The pre-build hook already received the path of the derivation as an argument, but that path doesn't typically exist when called as a remote build. + +Now, the pre-build hook is spawned with the environment variable `NIX_DERIVATION_V4` set to a file that contains the JSON representation of the derivation in v4 format, allowing instrospection of e.g. `requiredSystemFeatures` for scheduling decisions. + +PR: [DeterminateSystems/nix-src#424](https://github.com/DeterminateSystems/nix-src/pull/424) + +### Fix empty Git exports when using legacy Git compatibility + +A regression introduced in v3.16.0 made it possible for there to be empty Git exports in the Nix store when using legacy Git flakes (those depending on Nix 2.19 lockfile semantics). + +This is now fixed. + +PR: [DeterminateSystems/nix-src#425](https://github.com/DeterminateSystems/nix-src/pull/425) + + +**Full Changelog**: [v3.17.3...v3.18.0](https://github.com/DeterminateSystems/nix-src/compare/v3.17.3...v3.18.0) diff --git a/doc/manual/source/release-notes-determinate/v3.18.1.md b/doc/manual/source/release-notes-determinate/v3.18.1.md new file mode 100644 index 000000000000..26c9592561ad --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.18.1.md @@ -0,0 +1,23 @@ +# Release 3.18.1 (2026-04-23) + +* Based on [upstream Nix 2.33.4](../release-notes/rl-2.33.md). + +## What's Changed + +### Sentry integration improvements + +This release includes fixes for a couple of issues with the Sentry integration: + +* Ensures the chroot helper starts before the Sentry thread, allowing chroot stores to work again +* Resets Mach exception ports on macOS, so that exec'd programs do not communicate with Determinate Nix's `crashpad_handler` + +Additionally, Determinate Nix now includes the Nix command and subcommand to Sentry reports to make it easier to discern where an issue happened. +Not that this does _not_ include any command-line arguments other than the command (such as `nix-daemon` and `nix`) and subcommand (such as `flake show`). + +PRs: +* [DeterminateSystems/nix-src#433](https://github.com/DeterminateSystems/nix-src/pull/433) +* [DeterminateSystems/nix-src#432](https://github.com/DeterminateSystems/nix-src/pull/432) +* [DeterminateSystems/nix-src#436](https://github.com/DeterminateSystems/nix-src/pull/436) + + +**Full Changelog**: [v3.18.0...v3.18.1](https://github.com/DeterminateSystems/nix-src/compare/v3.18.0...v3.18.1) diff --git a/doc/manual/source/release-notes-determinate/v3.19.0.md b/doc/manual/source/release-notes-determinate/v3.19.0.md new file mode 100644 index 000000000000..9b0713fd3bbe --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.19.0.md @@ -0,0 +1,20 @@ +# Release 3.19.0 (2026-04-28) + +* Based on [upstream Nix 2.34.6](../release-notes/rl-2.34.md). + +## What's Changed + +### Update to upstream Nix 2.34 + +Determinate Nix is now based on [upstream Nix 2.34](../release-notes/rl-2.34.md), which includes a large number of bug fixes and improvements. + +### Provenance is preserved when copying paths to the daemon + +Previously, `nix copy` would not forward provenance information to the recipient. + +This has been fixed, such that building a derivation on machine A and `nix copy`ing it to machine B will show the same provenance. + +PR: [DeterminateSystems/nix-src#399](https://github.com/DeterminateSystems/nix-src/pull/399) + + +**Full Changelog**: [v3.18.1...v3.19.0](https://github.com/DeterminateSystems/nix-src/compare/v3.18.1...v3.19.0) diff --git a/doc/manual/source/release-notes-determinate/v3.19.1.md b/doc/manual/source/release-notes-determinate/v3.19.1.md new file mode 100644 index 000000000000..dcbd608b243c --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.19.1.md @@ -0,0 +1,35 @@ +# Release 3.19.1 (2026-05-04) + +* Based on [upstream Nix 2.34.6](../release-notes/rl-2.34.md). + +## What's Changed + +### Fix for GHSA-vh5x-56v6-4368: Coroutine stack-to-heap overflow via unbounded recursion in NAR directory parser + +Unbounded recursion in the NAR (Nix Archive) parser could lead to a stack-to-heap overflow when the parser is run on a coroutine stack. The stack was allocated without a guard page, which meant that a stack overflow could overwrite memory on the heap and could allow for arbitrary code execution as the Nix daemon (run as root in multi-user installations) if ASLR hardening is bypassed. This vulnerability can be exploited by all users able to connect to the Nix daemon (configurable via allowed-users setting - all users by default). + +The vulnerability is present since 2.24.4. + +For more details, see the upstream advisory [GHSA-vh5x-56v6-4368](https://github.com/NixOS/nix/security/advisories/GHSA-vh5x-56v6-4368). + +Many thanks to edef for reporting this issue and to Sergei Zimmerman and edef for implementing the fix and several hardening improvements. + +### Fix for GHSA-gr92-w2r5-qw5p: Absolute path traversal when unpacking archives to disk + +When an archive (e.g. a tarball) contains entries with absolute paths, `nix-prefetch-url --unpack`, `nix store prefetch-file --unpack` commands would write to a location outside the extraction root. This also affects `builtin:unpack-channel` builtin derivation builder, but it's sandboxed the same way as regular derivations builds, so in sandboxed builds it can't be used to achieve an arbitrary file write on the host filesystem. + +Other archive unpacking (`builtins.fetchTarball`, `builtins.fetchTree` and tarball-based flake inputs) is not affected, since Nix stores the unpacked tarball in a bare git repository (`.cache/nix/tarball-cache` or `.cache/nix/tarball-cache-v2`). + +This primarily affects packagers running `nix-prefetch-url --unpack` on untrusted archives. + +The vulnerability is present since 2.24.7. + +For more details, see the upstream advisory [GHSA-gr92-w2r5-qw5p](https://github.com/NixOS/nix/security/advisories/GHSA-gr92-w2r5-qw5p). + +Many thanks to edef for reporting this issue and to Sergei Zimmerman for implementing the fix. + +### Bug fixes + +* A fix for random crashes on exit in the `curlFileTransfer` destructor. [DeterminateSystems/nix-src#444](https://github.com/DeterminateSystems/nix-src/pull/444) + +**Full Changelog**: [v3.19.0...v3.19.1](https://github.com/DeterminateSystems/nix-src/compare/v3.19.0...v3.19.1) diff --git a/doc/manual/source/release-notes-determinate/v3.20.0.md b/doc/manual/source/release-notes-determinate/v3.20.0.md new file mode 100644 index 000000000000..b077f99dde1d --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.20.0.md @@ -0,0 +1,22 @@ +# Release 3.20.0 (2026-05-07) + +* Based on [upstream Nix 2.34.6](../release-notes/rl-2.34.md). + +## Support resuming from binary caches that don't support ranged requests + +Previously, Nix would just fail a download if something happened in the TCP or TLS layer, meaning a flaky proxy could fail to download a store path and fall back to building, when a retry might have not been flaky. + +Now, Determinate Nix will retry these failed substitutions, so you don't have to waste time building something that was cached but some proxy decided to cut you off early. + +PR: [DeterminateSystems/nix-src#445](https://github.com/DeterminateSystems/nix-src/pull/445) + +## Add a session ID to JSON log messages + +Determinate Nix now includes a freeform string session ID (`sid`) in its JSON log messages, which allows a anything that consumes these log messages to see which messages belong to the same session. + +By default, every Nix command invocation is its own session, but you can use the `session-id` setting to give a group of invocations the same session (otherwise, a random UUIDv7 like `019e037b-f603-7443-8071-cfd8585ec8c6` is set as the session ID). + +PR: [DeterminateSystems/nix-src#440](https://github.com/DeterminateSystems/nix-src/pull/440) + + +**Full Changelog**: [v3.19.1...v3.20.0](https://github.com/DeterminateSystems/nix-src/compare/v3.19.1...v3.20.0) diff --git a/doc/manual/source/release-notes-determinate/v3.21.0.md b/doc/manual/source/release-notes-determinate/v3.21.0.md new file mode 100644 index 000000000000..f06448792e91 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.21.0.md @@ -0,0 +1,44 @@ +# Release 3.21.0 (2026-05-22) + +* Based on [upstream Nix 2.34.6](../release-notes/rl-2.34.md). + +## CNSA 1.0 and 2.0 signing algorithm support + +Determinate Nix now supports a subset of CNSA 1.0 and 2.0 signature algorithms: `ecdsa-p384`, `ml-dsa-44`, `ml-dsa-65`, and `ml-dsa-87`. ML-DSA is a post-quantum cryptography signature scheme. +Support for these algorithms is gated behind the experimental feature `cnsa`. + +To use these signatures, enable the `cnsa` experimental feature and call `nix key generate-secret` with `--key-type ecdsa-p384|ml-dsa-{44,65,87}`. +Otherwise, everything works the same as the existing `ed25519` (libsodium) signatures, except that they produce larger keys/signatures (especially ML-DSA). + +We've also added commands `nix key convert-{public,secret}-to-pem`, which may be useful if you want to use the keys with the `openssl` CLI. + +PR: [DeterminateSystems/nix-src#449](https://github.com/DeterminateSystems/nix-src/pull/449) + +## `nix serve` command + +The new `nix serve` command provides a built-in binary cache server, similar to [`nix-serve`](https://github.com/edolstra/nix-serve/). +It's mostly intended for testing and to serve as a reference implementation of binary caches. + +PR: [DeterminateSystems/nix-src#428](https://github.com/DeterminateSystems/nix-src/pull/428) + +## Lazy `fetchTarball` + +`fetchTarball` is now lazy, which can prevent unnecessary copying of source trees into the Nix store. + +PRs: +* [DeterminateSystems/nix-src#468](https://github.com/DeterminateSystems/nix-src/pull/468) +* [DeterminateSystems/nix-src#469](https://github.com/DeterminateSystems/nix-src/pull/469) + +## Various bug fixes + +* Restore resBuildResult fields for failed builds [DeterminateSystems/nix-src#453](https://github.com/DeterminateSystems/nix-src/pull/453) +* parseString(): Fix out-of-bounds read [DeterminateSystems/nix-src#454](https://github.com/DeterminateSystems/nix-src/pull/454) +* filetransfer: handle exceptions thrown from enqueueItem [DeterminateSystems/nix-src#348](https://github.com/DeterminateSystems/nix-src/pull/348) +* libfetchers: verify git-lfs returns the same objects as git [DeterminateSystems/nix-src#456](https://github.com/DeterminateSystems/nix-src/pull/456) +* HttpBinaryCacheStore: Don't ignore 401/407 errors [DeterminateSystems/nix-src#461](https://github.com/DeterminateSystems/nix-src/pull/461) +* Remove non-atomically initialised variable vImportedDrvToDerivation [DeterminateSystems/nix-src#464](https://github.com/DeterminateSystems/nix-src/pull/464) +* Re-enable TCP keep-alive and handle S3's XML errors [DeterminateSystems/nix-src#463](https://github.com/DeterminateSystems/nix-src/pull/463) +* Fix logging segfault by leaking logger [DeterminateSystems/nix-src#470](https://github.com/DeterminateSystems/nix-src/pull/470) + + +**Full Changelog**: [v3.20.0...v3.21.0](https://github.com/DeterminateSystems/nix-src/compare/v3.20.0...v3.21.0) diff --git a/doc/manual/source/release-notes-determinate/v3.21.1.md b/doc/manual/source/release-notes-determinate/v3.21.1.md new file mode 100644 index 000000000000..2e9cb30b929f --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.21.1.md @@ -0,0 +1,58 @@ +# Release 3.21.1 (2026-06-08) + +* Based on [upstream Nix 2.34.7](../release-notes/rl-2.34.md). + +## Faster `nix copy` + +Previously, `nix copy` processed store paths in topological order, meaning that if store path A depends on B, then Nix wouldn't even start the fetch of store path A from the source store, or the writing of store path A to the destination store, until store path B had been added to the destination store. + +Now, Nix can start fetching and writing store paths before their dependencies have been added to the destination store. This can provide significant speedups. For instance, the command +```console +# nix copy \ + --from file:///tmp/binary-cache \ + --to file:///tmp/binary-cache-2?compression=xz \ + --no-check-sigs \ + /nix/store/8xyk1qxxjfb8cm62g61yc59phwha92w7-kdenlive-25.08.3 +``` +went from 117.7s to 47.2s on a machine with 24 cores. + +This optimisation affects copying to binary caches and local stores, but not when going through the Nix daemon. It also does not speed up substitution yet. + +PRs: [DeterminateSystems/nix-src#477](https://github.com/DeterminateSystems/nix-src/pull/477), [DeterminateSystems/nix-src#478](https://github.com/DeterminateSystems/nix-src/pull/478), [DeterminateSystems/nix-src#481](https://github.com/DeterminateSystems/nix-src/pull/481), [DeterminateSystems/nix-src#480](https://github.com/DeterminateSystems/nix-src/pull/480) + +## Determinate Secure Packages + +Determinate Nix is now built against [Determinate Secure Packages](https://determinate.systems/secure-packages/), a distribution of Nixpkgs that adds enterprise-grade security, compliance controls, and managed vulnerability remediation. + +Determinate Nix's [top-level flake](https://github.com/DeterminateSystems/nix-src/blob/main/flake.nix) still builds against upstream Nixpkgs. The flake `github:DeterminateSystems/nix-src?dir=packaging/secure-packages` uses Determinate Secure Packages. + +PR: [DeterminateSystems/nix-src#288](https://github.com/DeterminateSystems/nix-src/pull/288) + +## Flake schema changes + +Flakes can now distinguish between schemas for their own outputs (via the `schemas` output), and schemas that they make available to other flakes (via the `exportedSchemas` output). If you're using the `flake-schemas` flake, you should use its `exportedSchemas` output, e.g. +```nix +{ + inputs.flake-schemas.url = "https://flakehub.com/f/DeterminateSystems/flake-schemas/0.5"; + + outputs = { self, flake-schemas, ... }: { + schemas = flake-schemas.exportedSchemas // { ... my own schemas ... } + }; +} +``` + +PRs: [DeterminateSystems/nix-src#479](https://github.com/DeterminateSystems/nix-src/pull/479), [DeterminateSystems/nix-src#486](https://github.com/DeterminateSystems/nix-src/pull/486), +[DeterminateSystems/flake-schemas#63](https://github.com/DeterminateSystems/flake-schemas/pull/63) + +## Various bug fixes + +* Stop leaked logger in Finally block by @cole-h in [DeterminateSystems/nix-src#472](https://github.com/DeterminateSystems/nix-src/pull/472) +* Denote empty files in stack traces by @edolstra in [DeterminateSystems/nix-src#473](https://github.com/DeterminateSystems/nix-src/pull/473) +* Input::isRelative(): Fix assertion failure by @edolstra in [DeterminateSystems/nix-src#474](https://github.com/DeterminateSystems/nix-src/pull/474) +* Mark thread-local variables as initial-exec by @edolstra in [DeterminateSystems/nix-src#476](https://github.com/DeterminateSystems/nix-src/pull/476) +* Register temp roots in a few places by @edolstra in [DeterminateSystems/nix-src#482](https://github.com/DeterminateSystems/nix-src/pull/482) +* Sync with upstream 2.34.7 by @edolstra in [DeterminateSystems/nix-src#483](https://github.com/DeterminateSystems/nix-src/pull/483) +* Add a test for `nix ps` by @edolstra in [DeterminateSystems/nix-src#484](https://github.com/DeterminateSystems/nix-src/pull/484) +* nix develop: Restore hiding the "dirty" notice by @edolstra in [DeterminateSystems/nix-src#485](https://github.com/DeterminateSystems/nix-src/pull/485) + +**Full Changelog**: [v3.21.0...v3.21.1](https://github.com/DeterminateSystems/nix-src/compare/v3.21.0...v3.21.1) diff --git a/doc/manual/source/release-notes-determinate/v3.21.2.md b/doc/manual/source/release-notes-determinate/v3.21.2.md new file mode 100644 index 000000000000..caab2d1bd4d9 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.21.2.md @@ -0,0 +1,52 @@ +# Release 3.21.2 (2026-06-18) + +* Based on [upstream Nix 2.34.7](../release-notes/rl-2.34.md). + +## Performance improvements + +`nix store optimise` is now multi-threaded. This makes it much faster (e.g. from 301s to 111s on one particular store) since hashing can be done in parallel. PR: [DeterminateSystems/nix-src#492](https://github.com/DeterminateSystems/nix-src/pull/492) + +Copying of sources to the Nix store is now cached better in the evaluator. PR: [DeterminateSystems/nix-src#507](https://github.com/DeterminateSystems/nix-src/pull/507) + +## Improved error messages + +When a user isn't allowed to connect to the Nix daemon, previously they would get the unhelpful error message: +``` +error: cannot open connection to remote store 'daemon': error: read of 32768 bytes: Connection reset by peer +``` + +Now they get: +``` +error: cannot open connection to remote store 'daemon': access denied by the Nix daemon (check the `allowed-users` setting in the daemon) +``` + +PR: [DeterminateSystems/nix-src#498](https://github.com/DeterminateSystems/nix-src/pull/498) + +## Various bug fixes + +* Add missing manpages by @edolstra in [DeterminateSystems/nix-src#490](https://github.com/DeterminateSystems/nix-src/pull/490) +* SQLiteStmt::Use: Rename operator() -> apply() by @edolstra in [DeterminateSystems/nix-src#491](https://github.com/DeterminateSystems/nix-src/pull/491) +* Fix running hydraJobs in CI by @edolstra in [DeterminateSystems/nix-src#494](https://github.com/DeterminateSystems/nix-src/pull/494) +* ci: include output paths in FlakeHub upload by @cole-h in [DeterminateSystems/nix-src#495](https://github.com/DeterminateSystems/nix-src/pull/495) +* fixup: don't include linux-only drvs if not on linux by @cole-h in [DeterminateSystems/nix-src#496](https://github.com/DeterminateSystems/nix-src/pull/496) +* Daemon: Ignore interrupts between accept() and child start by @edolstra in [DeterminateSystems/nix-src#502](https://github.com/DeterminateSystems/nix-src/pull/502) +* nario tweaks by @edolstra in [DeterminateSystems/nix-src#499](https://github.com/DeterminateSystems/nix-src/pull/499) +* Fix crash in builtins.break by @edolstra in [DeterminateSystems/nix-src#505](https://github.com/DeterminateSystems/nix-src/pull/505) +* Fix SSH connection sharing by @pjungkamp in [DeterminateSystems/nix-src#506](https://github.com/DeterminateSystems/nix-src/pull/506) +* Replace select() by poll() by @edolstra in [DeterminateSystems/nix-src#504](https://github.com/DeterminateSystems/nix-src/pull/504) +* LocalStore::doAddToStore(): Check NAR size before NAR hash by @proski in [DeterminateSystems/nix-src#512](https://github.com/DeterminateSystems/nix-src/pull/512) +* Increase the sourceToSink/sinkToSource coroutine stack sizes by @xokdvium in [DeterminateSystems/nix-src#511](https://github.com/DeterminateSystems/nix-src/pull/511) +* Fix LocalStore::addMultipleToStore SinkToSource destructor unsoundness by @lilyinstarlight in [DeterminateSystems/nix-src#513](https://github.com/DeterminateSystems/nix-src/pull/513) +* Fix exceptions not being caught in nix::Pid::~Pid by @lilyinstarlight in [DeterminateSystems/nix-src#516](https://github.com/DeterminateSystems/nix-src/pull/516) + +## Refactorings + +* Binary cache info by @edolstra in [DeterminateSystems/nix-src#493](https://github.com/DeterminateSystems/nix-src/pull/493) + +## Release process improvements + +* Build all the archs without DSP by @grahamc in [DeterminateSystems/nix-src#503](https://github.com/DeterminateSystems/nix-src/pull/503) +* Mirror to nix-artifacts on release of tags by @grahamc in [DeterminateSystems/nix-src#501](https://github.com/DeterminateSystems/nix-src/pull/501) +* ci: fixup mirroring by @cole-h in [DeterminateSystems/nix-src#515](https://github.com/DeterminateSystems/nix-src/pull/515) + +**Full Changelog**: [v3.21.1...v3.21.2](https://github.com/DeterminateSystems/nix-src/compare/v3.21.1...v3.21.2) diff --git a/doc/manual/source/release-notes-determinate/v3.21.5.md b/doc/manual/source/release-notes-determinate/v3.21.5.md new file mode 100644 index 000000000000..68a58d8f8776 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.21.5.md @@ -0,0 +1,39 @@ +# Release 3.21.5 (2026-07-08) + +* Based on [upstream Nix 2.34.8](../release-notes/rl-2.34.md). + +## Binary cache performance improvements + +This release includes a number of async rewrites from upstream Nix that make querying missing paths much faster. PR: [DeterminateSystems/nix-src#514](https://github.com/DeterminateSystems/nix-src/pull/514) + +## Remote store performance improvements + +Registering temporary roots on remote stores (e.g. when doing `nix copy` to a remote store over `ssh-ng`) is now much faster. PR: [DeterminateSystems/nix-src#538](https://github.com/DeterminateSystems/nix-src/pull/538) + +Computing closures (e.g. `nix-store -qR`) is now much faster on high-latency `ssh-ng` stores. PR: [DeterminateSystems/nix-src#539](https://github.com/DeterminateSystems/nix-src/pull/539) + +## Parallel evaluation performance improvements + +This release delivers a number of performance improvements to parallel evaluation. + +PRs: [DeterminateSystems/nix-src#531](https://github.com/DeterminateSystems/nix-src/pull/531), [DeterminateSystems/nix-src#535](https://github.com/DeterminateSystems/nix-src/pull/535), [DeterminateSystems/nix-src#536](https://github.com/DeterminateSystems/nix-src/pull/536), [DeterminateSystems/nix-src#537](https://github.com/DeterminateSystems/nix-src/pull/537), [DeterminateSystems/nix-src#546](https://github.com/DeterminateSystems/nix-src/pull/546) + +## Miscellaneous improvements + +* `nix search` now prints the total number of derivations, in addition to the number of matching derivations. PR: [DeterminateSystems/nix-src#532](https://github.com/DeterminateSystems/nix-src/pull/532) + +## Bug fixes and other changes + +* Allow local networking in libmicrohttpd and functional tests by @lilyinstarlight in [DeterminateSystems/nix-src#518](https://github.com/DeterminateSystems/nix-src/pull/518) +* Don't ignore substitution failures for BasicDerivation inputSrcs by @edolstra in [DeterminateSystems/nix-src#520](https://github.com/DeterminateSystems/nix-src/pull/520) +* Sync with upstream 2.34-maintenance by @edolstra in [DeterminateSystems/nix-src#521](https://github.com/DeterminateSystems/nix-src/pull/521) +* fixup: make binary cache nuclear option more general by @cole-h in [DeterminateSystems/nix-src#524](https://github.com/DeterminateSystems/nix-src/pull/524) +* NarInfoDiskCache: Support arbitrary nix-cache-info fields by @edolstra in [DeterminateSystems/nix-src#525](https://github.com/DeterminateSystems/nix-src/pull/525) +* ci: update DSP on releases by @cole-h in [DeterminateSystems/nix-src#526](https://github.com/DeterminateSystems/nix-src/pull/526) +* Add some calls to addTempRoot() by @edolstra in [DeterminateSystems/nix-src#527](https://github.com/DeterminateSystems/nix-src/pull/527) +* flake.lock: Update by @edolstra in [DeterminateSystems/nix-src#529](https://github.com/DeterminateSystems/nix-src/pull/529) + +## New Contributors +* @lilyinstarlight made their first contribution in [DeterminateSystems/nix-src#518](https://github.com/DeterminateSystems/nix-src/pull/518) + +**Full Changelog**: [v3.21.2...v3.21.5](https://github.com/DeterminateSystems/nix-src/compare/v3.21.2...v3.21.5) diff --git a/doc/manual/source/release-notes-determinate/v3.21.6.md b/doc/manual/source/release-notes-determinate/v3.21.6.md new file mode 100644 index 000000000000..0c3b32c23dc4 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.21.6.md @@ -0,0 +1,33 @@ +# Release 3.21.6 (2026-07-14) + +* Based on [upstream Nix 2.34.8](../release-notes/rl-2.34.md). + +## Parallel evaluation improvements + +`NIX_COUNT_CALLS` is now compatible with parallel evaluation. In addition, we fixed a bug that prevented it from reporting attribute access statistics. PR: [DeterminateSystems/nix-src#562](https://github.com/DeterminateSystems/nix-src/pull/562) + +We improved parallel evaluation performance by allocating more blocks per acquisition of the global Boehm GC lock. PR: [DeterminateSystems/nix-src#547](https://github.com/DeterminateSystems/nix-src/pull/547) + +## Bug fixes + +We made several bug fixes to address Sentry crash reports. PRs: [DeterminateSystems/nix-src#558](https://github.com/DeterminateSystems/nix-src/pull/558), [DeterminateSystems/nix-src#560](https://github.com/DeterminateSystems/nix-src/pull/560), [DeterminateSystems/nix-src#561](https://github.com/DeterminateSystems/nix-src/pull/561), [DeterminateSystems/nix-src#563](https://github.com/DeterminateSystems/nix-src/pull/563) + +Other bug fixes: + +* Fix data race initializing AttrCursor::_value by @edolstra in [DeterminateSystems/nix-src#557](https://github.com/DeterminateSystems/nix-src/pull/557) +* On Linux, don't hang if the sandbox helper dies before opening the pty slave by @edolstra in [DeterminateSystems/nix-src#559](https://github.com/DeterminateSystems/nix-src/pull/559) + +## Release process improvements + +* libutil: Fix GCC build with -Wchanges-meaning by @edolstra in [DeterminateSystems/nix-src#549](https://github.com/DeterminateSystems/nix-src/pull/549) +* fixup: build non-dsp nix on tags as well by @cole-h in [DeterminateSystems/nix-src#550](https://github.com/DeterminateSystems/nix-src/pull/550) +* build.yml: Enable provenance by @edolstra in [DeterminateSystems/nix-src#552](https://github.com/DeterminateSystems/nix-src/pull/552) +* Drop soversion by @edolstra in [DeterminateSystems/nix-src#555](https://github.com/DeterminateSystems/nix-src/pull/555) +* Restore uploading of debug symbols for build_x86_64-linux_no_dsp by @edolstra in [DeterminateSystems/nix-src#556](https://github.com/DeterminateSystems/nix-src/pull/556) + +## Miscellaneous changes + +* Root values cleanup by @edolstra in [DeterminateSystems/nix-src#554](https://github.com/DeterminateSystems/nix-src/pull/554) +* Add initial CLAUDE.md by @edolstra in [DeterminateSystems/nix-src#551](https://github.com/DeterminateSystems/nix-src/pull/551) + +**Full Changelog**: [v3.21.5...v3.21.6](https://github.com/DeterminateSystems/nix-src/compare/v3.21.5...v3.21.6) diff --git a/doc/manual/source/release-notes-determinate/v3.21.7.md b/doc/manual/source/release-notes-determinate/v3.21.7.md new file mode 100644 index 000000000000..136a3e57c08a --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.21.7.md @@ -0,0 +1,13 @@ +# Release 3.21.7 (2026-07-15) + +* Based on [upstream Nix 2.34.8](../release-notes/rl-2.34.md). + +## Performance improvement + +`nix search` is slightly faster when using parallel evaluation. PR: [DeterminateSystems/nix-src#564](https://github.com/DeterminateSystems/nix-src/pull/564) + +## Bug fix + +We improved on a bug fix in the previous release to prevent crashes in OpenSSL when Nix exits. PR: [DeterminateSystems/nix-src#566](https://github.com/DeterminateSystems/nix-src/pull/566) + +**Full Changelog**: [v3.21.6...v3.21.7](https://github.com/DeterminateSystems/nix-src/compare/v3.21.6...v3.21.7) diff --git a/doc/manual/source/release-notes-determinate/v3.21.8.md b/doc/manual/source/release-notes-determinate/v3.21.8.md new file mode 100644 index 000000000000..5285f5898f1d --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.21.8.md @@ -0,0 +1,11 @@ +# Release 3.21.8 (2026-07-20) + +* Based on [upstream Nix 2.34.8](../release-notes/rl-2.34.md). + +## Bug fixes +* Fix libgit2 ll_find_delta leaking worker threads by @edolstra in [DeterminateSystems/nix-src#568](https://github.com/DeterminateSystems/nix-src/pull/568) +* Don't report SIGQUIT to Sentry by @edolstra in [DeterminateSystems/nix-src#570](https://github.com/DeterminateSystems/nix-src/pull/570) +* Don't abort if NIX_GET_COMPLETIONS is out of range by @edolstra in [DeterminateSystems/nix-src#571](https://github.com/DeterminateSystems/nix-src/pull/571) + + +**Full Changelog**: [v3.21.7...v3.21.8](https://github.com/DeterminateSystems/nix-src/compare/v3.21.7...v3.21.8) diff --git a/doc/manual/source/release-notes-determinate/v3.21.9.md b/doc/manual/source/release-notes-determinate/v3.21.9.md new file mode 100644 index 000000000000..91a479bf342e --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.21.9.md @@ -0,0 +1,9 @@ +# Release 3.21.9 (2026-07-30) + +* Based on [upstream Nix 2.34.8](../release-notes/rl-2.34.md). + +## Bug fixes +* Populate the srcToStore cache when we have a fetcher cache hit by @edolstra in [DeterminateSystems/nix-src#575](https://github.com/DeterminateSystems/nix-src/pull/575) + + +**Full Changelog**: [v3.21.8...v3.21.9](https://github.com/DeterminateSystems/nix-src/compare/v3.21.8...v3.21.9) diff --git a/doc/manual/source/release-notes-determinate/v3.22.1.md b/doc/manual/source/release-notes-determinate/v3.22.1.md new file mode 100644 index 000000000000..98c19440d69b --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.22.1.md @@ -0,0 +1,27 @@ +# Release 3.22.1 (2026-08-18) + +* Based on [upstream Nix 2.35.2](../release-notes/rl-2.35.md). + +## Upstream Nix 2.35 + +This release is based on upstream Nix 2.35.2, which includes numerous new features and performance improvements. See the [upstream release notes](../release-notes/rl-2.35.md) for more details. + +PRs: [DeterminateSystems/nix-src#530](https://github.com/DeterminateSystems/nix-src/pull/530), [DeterminateSystems/nix-src#584](https://github.com/DeterminateSystems/nix-src/pull/584) + +## Performance improvements + +* LocalStore: Reduce number of SQLite calls by @edolstra in [DeterminateSystems/nix-src#574](https://github.com/DeterminateSystems/nix-src/pull/574) +* Use std::move_only_function for parallel eval work items by @edolstra in [DeterminateSystems/nix-src#585](https://github.com/DeterminateSystems/nix-src/pull/585) + +## Bug fixes + +* Progress bar signal handling improvements by @edolstra in [DeterminateSystems/nix-src#586](https://github.com/DeterminateSystems/nix-src/pull/586) +* Prevent LocalCommand 'started' leak on stale SSH master socket by @DarthPJB in [DeterminateSystems/nix-src#569](https://github.com/DeterminateSystems/nix-src/pull/569) +* Fix randomly failing build.sh test by @edolstra in [DeterminateSystems/nix-src#587](https://github.com/DeterminateSystems/nix-src/pull/587) +* Don't let missing store paths abort 'nix store optimise' by @edolstra in [DeterminateSystems/nix-src#590](https://github.com/DeterminateSystems/nix-src/pull/590) +* Fix "waiting for exclusive access" hangs during schema migrations by @edolstra in [DeterminateSystems/nix-src#594](https://github.com/DeterminateSystems/nix-src/pull/594) + +## New Contributors +* @DarthPJB made their first contribution in [DeterminateSystems/nix-src#569](https://github.com/DeterminateSystems/nix-src/pull/569) + +**Full Changelog**: [v3.21.9...v3.22.1](https://github.com/DeterminateSystems/nix-src/compare/v3.21.9...v3.22.1) diff --git a/doc/manual/source/release-notes-determinate/v3.22.2.md b/doc/manual/source/release-notes-determinate/v3.22.2.md new file mode 100644 index 000000000000..2a8e5b030b82 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.22.2.md @@ -0,0 +1,22 @@ +# Release 3.22.2 (2026-08-21) + +* Based on [upstream Nix 2.35.2](../release-notes/rl-2.35.md). + +## `nix eval --json` has a new `--drv-link` flag + +When you're using `nix eval --json` to instantiate derivations, there is a possibility that a concurrent garbage collection run deletes the store derivations that were just created. Nix now has a new `--drv-link` flag that creates symlinks to the top-level derivations returned by the evaluations, avoiding this race condition. + +PR: [DeterminateSystems/nix-src#599](https://github.com/DeterminateSystems/nix-src/pull/599) + +## `nix copy` does not register remote store GC roots when talking to older daemons + +Older versions of Determinate Nix as well as upstream Nix lack the ability to create multiple GC roots in a single RPC call. This made `nix copy` over high-latency connections very slow. Therefore, we no longer create GC roots when talking to such daemons. Note that this means that a concurrent garbage collection run on the remote store may delete the store paths that were just copied. + +PR: [DeterminateSystems/nix-src#598](https://github.com/DeterminateSystems/nix-src/pull/598) + +## Bug fixes + +* upgradeDBSchema(): Don't show "executing migration" messages on a new store by @edolstra in [DeterminateSystems/nix-src#600](https://github.com/DeterminateSystems/nix-src/pull/600) +* CachingFilteringSourceAccessor: Make the isAllowed() cache thread-safe by @edolstra in [DeterminateSystems/nix-src#601](https://github.com/DeterminateSystems/nix-src/pull/601) + +**Full Changelog**: [v3.22.1...v3.22.2](https://github.com/DeterminateSystems/nix-src/compare/v3.22.1...v3.22.2) diff --git a/doc/manual/source/release-notes-determinate/v3.22.3.md b/doc/manual/source/release-notes-determinate/v3.22.3.md new file mode 100644 index 000000000000..db6cab11ffbe --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.22.3.md @@ -0,0 +1,29 @@ +# Release 3.22.3 (2026-09-02) + +* Based on [upstream Nix 2.35.2](../release-notes/rl-2.35.md). + +## Parallel evaluation performance improvements + +We're continuing to make performance improvements to parallel evaluation, in particular reducing contention on the Boehm-Demers-Weiser garbage collector's global lock by making more use of per-thread freelists and by removing the use of uncollectable allocations in `builtins.fromJSON`. + +PRs: [DeterminateSystems/nix-src#608](https://github.com/DeterminateSystems/nix-src/pull/608), [DeterminateSystems/nix-src#609](https://github.com/DeterminateSystems/nix-src/pull/609) + +## Miscellaneous changes + +* Update Nixpkgs by @edolstra in [DeterminateSystems/nix-src#607](https://github.com/DeterminateSystems/nix-src/pull/607) +* doc: mention that flake metadata's `Path` field is absent with lazy-trees by @edolstra in [DeterminateSystems/nix-src#620](https://github.com/DeterminateSystems/nix-src/pull/620) + +## Bug fixes + +* Fix `dir` attribute handling in nix-219-compat mode by @edolstra in [DeterminateSystems/nix-src#604](https://github.com/DeterminateSystems/nix-src/pull/604) +* Update the schema file atomically by @edolstra in [DeterminateSystems/nix-src#606](https://github.com/DeterminateSystems/nix-src/pull/606) +* nix copy: Don't register temp roots if the remote doesn't have AddTempRoots by @edolstra in [DeterminateSystems/nix-src#610](https://github.com/DeterminateSystems/nix-src/pull/610) +* Fix crash when GC is interrupted by @edolstra in [DeterminateSystems/nix-src#612](https://github.com/DeterminateSystems/nix-src/pull/612) +* Handle out-of-memory crashes better by @edolstra in [DeterminateSystems/nix-src#603](https://github.com/DeterminateSystems/nix-src/pull/603) +* Fix forwarding netrc-file to the daemon by @edolstra in [DeterminateSystems/nix-src#616](https://github.com/DeterminateSystems/nix-src/pull/616) +* Sync with upstream 2.35-maintenance by @edolstra in [DeterminateSystems/nix-src#617](https://github.com/DeterminateSystems/nix-src/pull/617) +* Fix Ctrl-C handling when Sentry is enabled by @edolstra in [DeterminateSystems/nix-src#618](https://github.com/DeterminateSystems/nix-src/pull/618) +* Eval cache: make the second evaluation hit the cache by @roberth in [DeterminateSystems/nix-src#619](https://github.com/DeterminateSystems/nix-src/pull/619) +* Deduplicate `nix build --print-out-paths` output by @edolstra in [DeterminateSystems/nix-src#622](https://github.com/DeterminateSystems/nix-src/pull/622) + +**Full Changelog**: [v3.22.2...v3.22.3](https://github.com/DeterminateSystems/nix-src/compare/v3.22.2...v3.22.3) diff --git a/doc/manual/source/release-notes-determinate/v3.22.4.md b/doc/manual/source/release-notes-determinate/v3.22.4.md new file mode 100644 index 000000000000..61fe55400add --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.22.4.md @@ -0,0 +1,18 @@ +# Release 3.22.4 (2026-09-16) + +* Based on [upstream Nix 2.35.2](../release-notes/rl-2.35.md). + +## Performance improvements + +* Tarball cache performance improvements by @xokdvium in [DeterminateSystems/nix-src#627](https://github.com/DeterminateSystems/nix-src/pull/627) + +## Bug fixes +* tests/functional: skip the sentry test when Yama restricts ptrace by @Reasonable-Solutions in [DeterminateSystems/nix-src#625](https://github.com/DeterminateSystems/nix-src/pull/625) +* fetchers/git: negotiate rev-pinned fetches from bucketed refs by @tomberek in [DeterminateSystems/nix-src#628](https://github.com/DeterminateSystems/nix-src/pull/628) +* Fix the Nix < 2.20 NAR hash fallback for inputs with submodules by @joshuaspence in [DeterminateSystems/nix-src#624](https://github.com/DeterminateSystems/nix-src/pull/624) + +## New Contributors +* @Reasonable-Solutions made their first contribution in [DeterminateSystems/nix-src#625](https://github.com/DeterminateSystems/nix-src/pull/625) +* @joshuaspence made their first contribution in [DeterminateSystems/nix-src#624](https://github.com/DeterminateSystems/nix-src/pull/624) + +**Full Changelog**: [v3.22.3...v3.22.4](https://github.com/DeterminateSystems/nix-src/compare/v3.22.3...v3.22.4) diff --git a/doc/manual/source/release-notes-determinate/v3.22.5.md b/doc/manual/source/release-notes-determinate/v3.22.5.md new file mode 100644 index 000000000000..c4b932d4197b --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.22.5.md @@ -0,0 +1,10 @@ +# Release 3.22.5 (2026-09-17) + +* Based on [upstream Nix 2.35.2](../release-notes/rl-2.35.md). + +## Performance improvements + +* libutil/serialise: Use zero-copy copy_file_range for copying PosixFileSourceAccessor -> FdSink by @xokdvium in [DeterminateSystems/nix-src#630](https://github.com/DeterminateSystems/nix-src/pull/630) + + +**Full Changelog**: [v3.22.4...v3.22.5](https://github.com/DeterminateSystems/nix-src/compare/v3.22.4...v3.22.5) diff --git a/doc/manual/source/release-notes-determinate/v3.23.0.md b/doc/manual/source/release-notes-determinate/v3.23.0.md new file mode 100644 index 000000000000..2e9e275e80f0 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.23.0.md @@ -0,0 +1,69 @@ +# Release 3.23.0 (2026-10-01) + +* Based on [upstream Nix 2.35.2](../release-notes/rl-2.35.md). + +## OpenTelemetry Support + +Determinate Nix now has support for sending telemetry to OpenTelemetry collectors. It emits traces for many Nix activities, such as building and substituting. It propagates trace context across Nix daemon calls, `ssh-ng://` remote store calls, remote builds, and binary cache requests. For more information, see [our documentation](https://docs.determinate.systems/determinate-nix/opentelemetry). + +PR: [DeterminateSystems/nix-src#615](https://github.com/DeterminateSystems/nix-src/pull/615) + +## `nix flake bake` + +Given a flake, the new experimental command `nix flake bake` generates a new flake in which all derivations have been pre-evaluated, i.e. replaced by pseudo-derivations consisting of pre-computed output paths. These output paths must be available in a binary cache. + +For example, the following command "bakes" the `patchelf` flake: +```console +# nix flake bake github:NixOS/patchelf --dest-dir /tmp/patchelf-baked +``` + +The resulting flake consists of a JSON file listing all the output paths and other metadata of flake outputs, along with a `flake.nix` shim: +```console +# ls -l /tmp/patchelf-baked/ +total 9 +-rw-r--r-- 1 eelco users 6097 Oct 1 18:58 flake.nix +-rw-r--r-- 1 eelco users 7532 Oct 1 18:58 outputs.json +``` + +This baked flake can be used normally by commands like `nix build` and `nix run`: +```console +# nix run /tmp/patchelf-baked -- --version +patchelf 0.19.2 +``` + +Note: this requires the `baked-derivations` experimental feature both in the client and the daemon. + +PR: [DeterminateSystems/nix-src#450](https://github.com/DeterminateSystems/nix-src/pull/450) + +## `builtins.wasm` Improvements + +We've made API improvements to `builtins.wasm` to allow Wasm modules to read files and access attribute sets more efficiently. We've also enabled the compilation cache, avoiding recompilation of the same Wasm module across multiple Nix invocations. + +PRs: [DeterminateSystems/nix-src#633](https://github.com/DeterminateSystems/nix-src/pull/633), [DeterminateSystems/nix-src#634](https://github.com/DeterminateSystems/nix-src/pull/634), [DeterminateSystems/nix-src#647](https://github.com/DeterminateSystems/nix-src/pull/647) + +## Parallel Evaluation Improvements + +Parallel evaluation now uses fibers instead of threads. This allows better CPU utilization: Nix only starts `eval-cores` threads, and whenever a fiber is blocked waiting for a value to be finished by another thread, its thread can switch to another fiber that is ready to run. + +PRs: [DeterminateSystems/nix-src#597](https://github.com/DeterminateSystems/nix-src/pull/597), [DeterminateSystems/nix-src#635](https://github.com/DeterminateSystems/nix-src/pull/635), [DeterminateSystems/nix-src#636](https://github.com/DeterminateSystems/nix-src/pull/636), [DeterminateSystems/nix-src#641](https://github.com/DeterminateSystems/nix-src/pull/641), [DeterminateSystems/nix-src#644](https://github.com/DeterminateSystems/nix-src/pull/644) + +## New Experimental Build System + +Nix can now be built with Nix as the C++ build system instead of Meson/Ninja, providing an incremental build, ccache-like caching, and substitution and remote building of individual compilation units. It does not require a development shell or `build` directory that can get out of sync. It relies on `builtins.wasm` to compute the C++ dependency graph at evaluation time. + +To try it out: +```console +# nix build ./packaging/nix-make +``` + +PRs: [DeterminateSystems/nix-src#632](https://github.com/DeterminateSystems/nix-src/pull/632), [DeterminateSystems/nix-src#640](https://github.com/DeterminateSystems/nix-src/pull/640), [DeterminateSystems/nix-src#642](https://github.com/DeterminateSystems/nix-src/pull/642) + +## Async Path Writer + +The asynchronous path writer now skips paths that are already valid ([DeterminateSystems/nix-src#637](https://github.com/DeterminateSystems/nix-src/pull/637)), and also handles paths written by `builtins.toFile` ([DeterminateSystems/nix-src#638](https://github.com/DeterminateSystems/nix-src/pull/638)). Together these improved the runtime of `nix flake show` on a flake consisting of three NixOS configurations from 12.8s to 8.5s. + +## Miscellaneous Changes +* Rename CLAUDE.md -> AGENTS.md by @edolstra in [DeterminateSystems/nix-src#639](https://github.com/DeterminateSystems/nix-src/pull/639) +* addErrorContext: Don't force the message inside the exception handler by @edolstra in [DeterminateSystems/nix-src#643](https://github.com/DeterminateSystems/nix-src/pull/643) + +**Full Changelog**: [v3.22.5...v3.23.0](https://github.com/DeterminateSystems/nix-src/compare/v3.22.5...v3.23.0) diff --git a/doc/manual/source/release-notes-determinate/v3.23.1.md b/doc/manual/source/release-notes-determinate/v3.23.1.md new file mode 100644 index 000000000000..2e259355daa8 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.23.1.md @@ -0,0 +1,58 @@ +# Release 3.23.1 (2026-10-05) + +* Based on [upstream Nix 2.35.2](../release-notes/rl-2.35.md). + +## Build cgroups expose memory and I/O statistics + +When the `use-cgroups` setting is enabled, the Nix daemon now enables the `cpu`, `memory`, `io` and `pids` controllers in its cgroup, so per-build cgroups expose `memory.peak`, `io.stat` and `memory.events` instead of only `cpu.stat`. This is useful for CI systems that want to measure the resource usage of builds. It requires the controllers to be delegated to the daemon's cgroup (e.g. via `Delegate=yes` in the systemd unit). This change was backported from upstream Nix. + +PR: [DeterminateSystems/nix-src#655](https://github.com/DeterminateSystems/nix-src/pull/655), cherry-picked from [NixOS/nix#16024](https://github.com/NixOS/nix/pull/16024) by @DerDennisOP + +## `nix profile history` improvements + +`nix profile history` previously reported "No changes" when a package was replaced by a different store path with the same version (e.g. after a rebuild due to a source or dependency change). It now prints a `changed` line for that case. + +It also has a new flag `--show-source` that shows the locked flake reference of each added, removed, upgraded or changed package: + +```console +# nix profile history --show-source +... +Version 59 (2026-10-05) <- 58: + github:NixOS/nixpkgs#legacyPackages.x86_64-linux.hello: 2.12.2 changed (github:NixOS/nixpkgs/6fd8c75 -> github:NixOS/nixpkgs/ffc5e35) +``` + +PR: [DeterminateSystems/nix-src#665](https://github.com/DeterminateSystems/nix-src/pull/665) + +## Cheap and non-blocking purging of the narinfo disk cache + +On busy builders, the narinfo disk cache (`~/.cache/nix/binary-cache-v*.sqlite`) can grow to millions of entries. The daily purge of expired entries was a full table scan that could hold the SQLite write lock for minutes, blocking every concurrent `nix copy`. The purge now uses an index, runs in small chunks so that other processes wait at most a few hundred milliseconds, resumes where it left off if it's interrupted, and is performed by only one process at a time. + +PR: [DeterminateSystems/nix-src#667](https://github.com/DeterminateSystems/nix-src/pull/667) + +## Bug fixes + +* Fix a garbage collector segfault when substituting store paths during evaluation (e.g. `nix flake check` with import-from-derivation and no daemon) by @edolstra in [DeterminateSystems/nix-src#662](https://github.com/DeterminateSystems/nix-src/pull/662) +* Fix a `Too many retries in GC_allocobj` abort in `nix flake show` on large flakes by disabling bdwgc black-listing, which also speeds up evaluation by @edolstra in [DeterminateSystems/nix-src#663](https://github.com/DeterminateSystems/nix-src/pull/663) +* Don't crash if printing completions fails during unwinding by @edolstra in [DeterminateSystems/nix-src#651](https://github.com/DeterminateSystems/nix-src/pull/651) +* Don't treat running out of memory as a crash (Nix now exits with status 12 instead of aborting) by @edolstra in [DeterminateSystems/nix-src#653](https://github.com/DeterminateSystems/nix-src/pull/653) +* Fix a stack overflow when parsing very long flake references by @edolstra in [DeterminateSystems/nix-src#652](https://github.com/DeterminateSystems/nix-src/pull/652) +* Fix a segfault when multiple evaluator threads call `builtins.getFlake` on the same flake at the same time by @edolstra in [DeterminateSystems/nix-src#657](https://github.com/DeterminateSystems/nix-src/pull/657) +* Fix a crash in the daemon when the build hook is killed while the worker is being interrupted by @edolstra in [DeterminateSystems/nix-src#658](https://github.com/DeterminateSystems/nix-src/pull/658) +* Fix a crash in `nix flake check --keep-going` when a check fails by @edolstra in [DeterminateSystems/nix-src#660](https://github.com/DeterminateSystems/nix-src/pull/660) +* Fix a possible `SIGBUS` in the daemon due to the ZFS `-shm` workaround dropping SQLite locks by @edolstra in [DeterminateSystems/nix-src#664](https://github.com/DeterminateSystems/nix-src/pull/664) +* Sync with upstream 2.35-maintenance (fixes the store mtime in installer tarballs, adds `x86_64-darwin` as an extra platform on Apple silicon systems with Rosetta 2, and makes `nix flake prefetch-inputs` skip relative path inputs) by @edolstra in [DeterminateSystems/nix-src#666](https://github.com/DeterminateSystems/nix-src/pull/666) + +## Performance improvements + +* Don't `fsync` the active build info file at the start of every build by @Jesssullivan in [DeterminateSystems/nix-src#650](https://github.com/DeterminateSystems/nix-src/pull/650) + +## Miscellaneous changes + +* Make the `nix ps` test more robust by @edolstra in [DeterminateSystems/nix-src#654](https://github.com/DeterminateSystems/nix-src/pull/654) +* packaging/nix-make/flake.lock: Update nix-wasm-module-make by @edolstra in [DeterminateSystems/nix-src#656](https://github.com/DeterminateSystems/nix-src/pull/656) +* Add [derivation metadata documentation](../language/derivation-metadata.md) to the Determinate Nix manual by @lucperkins in [DeterminateSystems/nix-src#668](https://github.com/DeterminateSystems/nix-src/pull/668) + +## New Contributors +* @Jesssullivan made their first contribution in [DeterminateSystems/nix-src#650](https://github.com/DeterminateSystems/nix-src/pull/650) + +**Full Changelog**: [v3.23.0...v3.23.1](https://github.com/DeterminateSystems/nix-src/compare/v3.23.0...v3.23.1) diff --git a/doc/manual/source/release-notes-determinate/v3.8.6.md b/doc/manual/source/release-notes-determinate/v3.8.6.md new file mode 100644 index 000000000000..8f917f2362ff --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.8.6.md @@ -0,0 +1,14 @@ +# Release 3.8.6 (2025-08-19) + +* Based on [upstream Nix 2.30.2](../release-notes/rl-2.30.md). + +## What's Changed +* Auto update release notes by @grahamc in [DeterminateSystems/nix-src#170](https://github.com/DeterminateSystems/nix-src/pull/170) +* Use WAL mode for SQLite cache databases (2nd attempt) by @edolstra in [DeterminateSystems/nix-src#167](https://github.com/DeterminateSystems/nix-src/pull/167) +* Enable parallel marking in boehm-gc by @edolstra in [DeterminateSystems/nix-src#168](https://github.com/DeterminateSystems/nix-src/pull/168) +* BasicClientConnection::queryPathInfo(): Don't throw exception for invalid paths by @edolstra in [DeterminateSystems/nix-src#172](https://github.com/DeterminateSystems/nix-src/pull/172) +* Fix queryPathInfo() negative caching by @edolstra in [DeterminateSystems/nix-src#173](https://github.com/DeterminateSystems/nix-src/pull/173) +* forceDerivation(): Wait for async path write after forcing value by @edolstra in [DeterminateSystems/nix-src#176](https://github.com/DeterminateSystems/nix-src/pull/176) + + +**Full Changelog**: [v3.8.5...v3.8.6](https://github.com/DeterminateSystems/nix-src/compare/v3.8.5...v3.8.6) diff --git a/doc/manual/source/release-notes-determinate/v3.9.0.md b/doc/manual/source/release-notes-determinate/v3.9.0.md new file mode 100644 index 000000000000..66deb69b6192 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.9.0.md @@ -0,0 +1,45 @@ +# Release 3.9.0 (2025-08-26) + +* Based on [upstream Nix 2.30.2](../release-notes/rl-2.30.md). + +## What's Changed + +### Build-time flake inputs + +Some of our users have hundreds or thousands of flake inputs. +In those cases, it is painfully slow for Nix to fetch all the inputs during evaluation of the flake. + +Determinate Nix has an experimental feature for deferring the fetching to build time of the dependent derivations. + +This is currently in developer preview. +If you would like to try it, add the experimental feature to your `/etc/nix/nix.custom.conf`: + +```ini +extra-experimental-features = build-time-fetch-tree +``` + +Then, mark an input to be fetched at build time: + +```nix +inputs.example = { + type = "github"; + owner = "DeterminateSystems"; + repo = "example"; + flake = false; # <-- currently required + buildTime = true; +}; +``` + +Let us know what you think! + +PR: [DeterminateSystems/nix-src#49](https://github.com/DeterminateSystems/nix-src/pull/49) + +### Corrected inconsistent behavior of `nix flake check` + +Users reported that `nix flake check` would not consistently validate the entire flake. + +We've fixed this issue and improved our testing around `nix flake check`. + +PR: [DeterminateSystems/nix-src#182](https://github.com/DeterminateSystems/nix-src/pull/182) + +**Full Changelog**: [v3.8.6...v3.9.0](https://github.com/DeterminateSystems/nix-src/compare/v3.8.6...v3.9.0) diff --git a/doc/manual/source/release-notes-determinate/v3.9.1.md b/doc/manual/source/release-notes-determinate/v3.9.1.md new file mode 100644 index 000000000000..38d17199c2c0 --- /dev/null +++ b/doc/manual/source/release-notes-determinate/v3.9.1.md @@ -0,0 +1,20 @@ +# Release 3.9.1 (2025-08-28) + +- Based on [upstream Nix 2.30.2](../release-notes/rl-2.30.md). + +### A useful `nix flake init` template default + +Nix's default flake template is [extremely bare bones](https://github.com/NixOS/templates/blob/ad0e221dda33c4b564fad976281130ce34a20cb9/trivial/flake.nix), and not a useful starting point. + +Deteminate Nix now uses [a more fleshed out default template](https://github.com/DeterminateSystems/flake-templates/blob/8af99b99627da41f16897f60eb226db30c775e76/default/flake.nix), including targeting multiple systems. + +PR: [DeterminateSystems/nix-src#180](https://github.com/DeterminateSystems/nix-src/pull/180) + +### Build cancellation is repaired on macOS + +A recent macOS update changed how signals are handled by Nix and broke using Ctrl-C to stop a build. +Determinate Nix on macOS correctly handles these signals and stops the build. + +PR: [DeterminateSystems/nix-src#184](https://github.com/DeterminateSystems/nix-src/pull/184) + +**Full Changelog**: [v3.9.0...v3.9.1](https://github.com/DeterminateSystems/nix-src/compare/v3.9.0...v3.9.1) diff --git a/doc/manual/source/release-notes/rl-2.19.md b/doc/manual/source/release-notes/rl-2.19.md index 813279c9f05f..0596ef909619 100644 --- a/doc/manual/source/release-notes/rl-2.19.md +++ b/doc/manual/source/release-notes/rl-2.19.md @@ -69,7 +69,7 @@ This makes it match `nix derivation show`, which also maps store paths to information. -- When Nix is installed using the [binary installer](@docroot@/installation/installing-binary.md), in supported shells (Bash, Zsh, Fish) +- When Nix is installed using the binary installer, in supported shells (Bash, Zsh, Fish) [`XDG_DATA_DIRS`](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html#variables) is now populated with the path to the `/share` subdirectory of the current profile. This means that command completion scripts, `.desktop` files, and similar artifacts installed via [`nix-env`](@docroot@/command-ref/nix-env.md) or [`nix profile`](@docroot@/command-ref/new-cli/nix3-profile.md) (experimental) can be found by any program that follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html). diff --git a/doc/manual/source/release-notes/rl-2.24.md b/doc/manual/source/release-notes/rl-2.24.md index e9b46bb22b70..f608fb54f7d9 100644 --- a/doc/manual/source/release-notes/rl-2.24.md +++ b/doc/manual/source/release-notes/rl-2.24.md @@ -268,6 +268,21 @@ be configured using the `warn-large-path-threshold` setting, e.g. `--warn-large-path-threshold 100M`. +- Wrap filesystem exceptions more correctly [#11378](https://github.com/NixOS/nix/pull/11378) + + With the switch to `std::filesystem` in different places, Nix started to throw `std::filesystem::filesystem_error` in many places instead of its own exceptions. + + This led to no longer generating error traces, for example when listing a non-existing directory. + + This version catches these types of exception correctly and wraps them into Nix's own exeception type. + + Author: [**@Mic92**](https://github.com/Mic92) + +- `` uses TLS verification [#11585](https://github.com/NixOS/nix/pull/11585) + + Previously `` did not do TLS verification. This was because the Nix sandbox in the past did not have access to TLS certificates, and Nix checks the hash of the fetched file anyway. However, this can expose authentication data from `netrc` and URLs to man-in-the-middle attackers. In addition, Nix now in some cases (such as when using impure derivations) does *not* check the hash. Therefore we have now enabled TLS verification. This means that downloads by `` will now fail if you're fetching from a HTTPS server that does not have a valid certificate. + + `` is also known as the builtin derivation builder `builtin:fetchurl`. It's not to be confused with the evaluation-time function `builtins.fetchurl`, which was not affected by this issue. ## Contributors diff --git a/doc/manual/source/release-notes/rl-2.33.md b/doc/manual/source/release-notes/rl-2.33.md index 1ad0cf3f0564..0849f097cd0e 100644 --- a/doc/manual/source/release-notes/rl-2.33.md +++ b/doc/manual/source/release-notes/rl-2.33.md @@ -279,3 +279,35 @@ This release was made possible by the following 33 contributors: - Henry [**(@cootshk)**](https://github.com/cootshk) - Martin Joerg [**(@mjoerg)**](https://github.com/mjoerg) - Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria) +# Release 2.33.3 (2026-02-13) + +- S3 binary caches now use virtual-hosted-style addressing by default [#15208](https://github.com/NixOS/nix/issues/15208) + + S3 binary caches now use virtual-hosted-style URLs + (`https://bucket.s3.region.amazonaws.com/key`) instead of path-style URLs + (`https://s3.region.amazonaws.com/bucket/key`) when connecting to standard AWS + S3 endpoints. This enables HTTP/2 multiplexing and fixes TCP connection + exhaustion (TIME_WAIT socket accumulation) under high-concurrency workloads. + + A new `addressing-style` store option controls this behavior: + + - `auto` (default): virtual-hosted-style for standard AWS endpoints, path-style + for custom endpoints. + - `path`: forces path-style addressing (deprecated by AWS). + - `virtual`: forces virtual-hosted-style addressing (bucket names must not + contain dots). + + Bucket names containing dots (e.g., `my.bucket.name`) automatically fall back + to path-style addressing in `auto` mode, because dotted names create + multi-level subdomains that break TLS wildcard certificate validation. + + Example using path-style for backwards compatibility: + + ``` + s3://my-bucket/key?region=us-east-1&addressing-style=path + ``` + + Additionally, TCP keep-alive is now enabled on all HTTP connections, preventing + idle connections from being silently dropped by intermediate network devices + (NATs, firewalls, load balancers). + diff --git a/doc/manual/source/release-notes/rl-2.34.md b/doc/manual/source/release-notes/rl-2.34.md index 473f351907c3..91ee4ad5e9af 100644 --- a/doc/manual/source/release-notes/rl-2.34.md +++ b/doc/manual/source/release-notes/rl-2.34.md @@ -350,3 +350,70 @@ This release was made possible by the following 43 contributors: - Michael Daniels [**(@mdaniels5757)**](https://github.com/mdaniels5757) - Matthew Kenigsberg [**(@mkenigs)**](https://github.com/mkenigs) - Shea Levy [**(@shlevy)**](https://github.com/shlevy) + +# Release 2.34.1 (2026-03-08) + +## Changes + +- C API: Fix `EvalState` pointer passed to primop callbacks [#15300](https://github.com/NixOS/nix/pull/15300) [#15383](https://github.com/NixOS/nix/pull/15383) + + The `EvalState *` passed to C API primop callbacks was incorrectly pointing to + the internal `nix::EvalState` rather than the C API wrapper struct. This caused + a segfault when the callback used the pointer with C API functions such as + `nix_alloc_value()`. The same issue affected `printValueAsJSON` and + `printValueAsXML` callbacks on external values. + +- Fix daemon not applying `FileTransferSettings` from `trusted-users` [#15408](https://github.com/NixOS/nix/pull/15408) + + Previously `nix-daemon` failed to apply settings for `libcurl` configuration configured by client connections from [`trusted-users`](@docroot@/command-ref/conf-file.md#conf-trusted-users). This was a pre-existing bug, which has been exacerbated by 2.34.0 moving more settings from the global `settings` into libcurl-specific `fileTransferSettings` (e.g. `netrc-file`, `http-connections` or `ssl-cert-file`). Note that the use of `trusted-users` is heavily discouraged unless you are fine with: + + > Adding a user to `trusted-users` is essentially equivalent to giving that user root access to the system. + > For example, the user can access or replace store path contents that are critical for system security. + +- Improve formatting of error messages and warnings [#15397](https://github.com/NixOS/nix/pull/15397) + +# Release 2.34.2 (2026-03-20) + +## Changes + +- Fixed `nix upgrade-nix` without an explicitly specified `--profile` argument [#15437](https://github.com/NixOS/nix/issues/15437) [#15438](https://github.com/NixOS/nix/pull/15438) + +- Erroneous `error (ignored): write of ... bytes: Bad file descriptor` warnings on interrupted store copy operations are now fixed [#15486](https://github.com/NixOS/nix/pull/15486) + +- Reverted changes enabling keep-alive in the HTTP client [#15522](https://github.com/NixOS/nix/pull/15522) + + Connection reuse for S3 stores has caused Hydra upload errors due to stale connections being closed by the remote servers. Nix currently lacks retry mechanisms for 400 `RequestTimeout` errors used by AWS S3. + +- Fixed evaluation that accesses the logical store directory using `ssh-ng://` `eval-store`s [#15417](https://github.com/NixOS/nix/pull/15417) + + Accessing the logical store directory (typically `/nix/store`) during evaluation would fail previously: + + ``` + nix build nixpkgs#hello --store ssh-ng://host + error: path '/nix/store/' is not in the Nix store + ``` + +- S3: restore STS WebIdentity and ECS container credential providers [#15507](https://github.com/NixOS/nix/pull/15507) + + Nix 2.33 replaced the S3 backend's `aws-sdk-cpp` credential chain with a + custom chain built on `aws-c-auth`. That chain omitted two providers, + breaking S3 binary cache access in container workloads: + + - **STS WebIdentity** (`AWS_WEB_IDENTITY_TOKEN_FILE`, `AWS_ROLE_ARN`, + `AWS_ROLE_SESSION_NAME`) — used by EKS IRSA, GitHub Actions OIDC, and + any `sts:AssumeRoleWithWebIdentity` federation. + - **ECS container metadata** (`AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`, + `AWS_CONTAINER_CREDENTIALS_FULL_URI`) — used by ECS tasks and EKS Pod + Identity. + + The typical symptom was a misleading IMDS error + (`Valid credentials could not be sourced by the IMDS provider`), because + IMDS is the last provider tried after the correct one was skipped. + + Both providers are now part of the chain, ordered to match the + pre-2.33 `DefaultAWSCredentialsProviderChain`: + `Environment → SSO → Profile → STS WebIdentity → (ECS | IMDS)`. + As in both the old and new AWS SDK default chains, ECS and IMDS are + mutually exclusive: when container credential environment variables are + set, IMDS is skipped. + diff --git a/docker.nix b/docker.nix index d85927a08020..2bd6751762fa 100644 --- a/docker.nix +++ b/docker.nix @@ -8,6 +8,7 @@ # Image configuration name ? "nix", tag ? "latest", + fromImage ? null, bundleNixpkgs ? true, channelName ? "nixpkgs", channelURL ? "https://channels.nixos.org/nixpkgs-unstable", @@ -27,6 +28,8 @@ "org.opencontainers.image.description" = "Nix container image"; }, Cmd ? [ (lib.getExe bashInteractive) ], + extraPrePaths ? [ ], + extraPostPaths ? [ ], # Default Packages nix ? pkgs.nix, bashInteractive ? pkgs.bashInteractive, @@ -336,7 +339,7 @@ let globalFlakeRegistryPath="$nixCacheDir/flake-registry.json" ln -s ${flake-registry-path} $out$globalFlakeRegistryPath mkdir -p $out/nix/var/nix/gcroots/auto - rootName=$(${lib.getExe' nix "nix"} --extra-experimental-features nix-command hash file --type sha1 --base32 <(echo -n $globalFlakeRegistryPath)) + rootName=$(${lib.getExe' nix "nix"} hash file --type sha1 --base32 <(echo -n $globalFlakeRegistryPath)) ln -s $globalFlakeRegistryPath $out/nix/var/nix/gcroots/auto/$rootName '') ); @@ -352,6 +355,7 @@ dockerTools.buildLayeredImageWithNixDb { gid uname gname + fromImage ; contents = [ baseSystem ]; @@ -372,11 +376,15 @@ dockerTools.buildLayeredImageWithNixDb { Env = [ "USER=${uname}" "PATH=${ - lib.concatStringsSep ":" [ - "${userHome}/.nix-profile/bin" - "/nix/var/nix/profiles/default/bin" - "/nix/var/nix/profiles/default/sbin" - ] + lib.concatStringsSep ":" ( + extraPrePaths + ++ [ + "${userHome}/.nix-profile/bin" + "/nix/var/nix/profiles/default/bin" + "/nix/var/nix/profiles/default/sbin" + ] + ++ extraPostPaths + ) }" "MANPATH=${ lib.concatStringsSep ":" [ diff --git a/flake.lock b/flake.lock index f1e2fad80273..f61a4a296a41 100644 --- a/flake.lock +++ b/flake.lock @@ -1,5 +1,22 @@ { "nodes": { + "bdwgc": { + "flake": false, + "locked": { + "lastModified": 1790015987, + "narHash": "sha256-bPVosUgucJR0hGSap3tJW84AR6eGYegVROV+O8VYX5A=", + "owner": "DeterminateSystems", + "repo": "bdwgc", + "rev": "373957929b9ca3b2ae69a3e16b830c44c15b42ca", + "type": "github" + }, + "original": { + "owner": "DeterminateSystems", + "ref": "user-defined-stacks", + "repo": "bdwgc", + "type": "github" + } + }, "flake-compat": { "flake": false, "locked": { @@ -23,52 +40,50 @@ ] }, "locked": { - "lastModified": 1778716662, - "narHash": "sha256-m1Yf0wZ8j1OHjTc2UwHwyQRSnNeSgLJOd7q5Y45hzi4=", - "owner": "hercules-ci", - "repo": "flake-parts", - "rev": "f7c1a2d347e4c52d5fb8d10cb4d94b5884e546fb", - "type": "github" + "lastModified": 1782949081, + "narHash": "sha256-vp6Y/Grm98ESt6ceOkWiHWyZRDV3J1RID4w+6NWK9yA=", + "rev": "17c9d6cdfc60c64f4ee8d306f9bc0b4ccb51481e", + "revCount": 480, + "type": "tarball", + "url": "https://api.flakehub.com/f/pinned/hercules-ci/flake-parts/0.1.480%2Brev-17c9d6cdfc60c64f4ee8d306f9bc0b4ccb51481e/019f2195-dee5-7233-9747-eca0c27f7406/source.tar.gz" }, "original": { - "owner": "hercules-ci", - "repo": "flake-parts", - "type": "github" + "type": "tarball", + "url": "https://flakehub.com/f/hercules-ci/flake-parts/0.1" } }, "git-hooks-nix": { "inputs": { - "flake-compat": [], - "gitignore": [], + "flake-compat": "flake-compat", "nixpkgs": [ "nixpkgs" ] }, "locked": { - "lastModified": 1778507602, - "narHash": "sha256-kTwur1wV+01SdqskVMSo6JMEpg71ps3HpbFY2GsflKs=", - "owner": "cachix", - "repo": "git-hooks.nix", - "rev": "61ab0e80d9c7ab14c256b5b453d8b3fb0189ba0a", - "type": "github" + "lastModified": 1784288435, + "narHash": "sha256-ReRHaLgr/uVqdD8afFSn+myXIfpHeOhP0yYe0TJqAA8=", + "rev": "43b3c1ab9d40fb1dbb008f451988a91e375825e9", + "revCount": 1231, + "type": "tarball", + "url": "https://api.flakehub.com/f/pinned/cachix/git-hooks.nix/0.1.1231%2Brev-43b3c1ab9d40fb1dbb008f451988a91e375825e9/019f7135-8fdf-76f0-b1a1-d2c67e91af8d/source.tar.gz" }, "original": { - "owner": "cachix", - "repo": "git-hooks.nix", - "type": "github" + "type": "tarball", + "url": "https://flakehub.com/f/cachix/git-hooks.nix/0.1.941" } }, "nixpkgs": { "locked": { - "lastModified": 1782535326, - "narHash": "sha256-r4TA57SL7nvj1R+GY/FCLwFU48w9IixTQlzBTIYkt8E=", - "rev": "714a5f8c4ead6b31148d829288440ed033ccc041", + "lastModified": 1787962033, + "narHash": "sha256-u6z9VTZA4Kf3RkHQo9sQI7NI4Ei/uiU9vrMqOiwWP1Y=", + "rev": "c5c4a43b0e8056328ec4529f735cabdb8f1942bb", + "revCount": 1012930, "type": "tarball", - "url": "https://releases.nixos.org/nixos/26.05/nixos-26.05.3494.714a5f8c4ead/nixexprs.tar.xz" + "url": "https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.2605.1012930%2Brev-c5c4a43b0e8056328ec4529f735cabdb8f1942bb/01a053c0-7061-7382-b001-102e9148a5b5/source.tar.gz" }, "original": { "type": "tarball", - "url": "https://channels.nixos.org/nixos-26.05/nixexprs.tar.xz" + "url": "https://flakehub.com/f/NixOS/nixpkgs/0.2605" } }, "nixpkgs-23-11": { @@ -105,7 +120,7 @@ }, "root": { "inputs": { - "flake-compat": "flake-compat", + "bdwgc": "bdwgc", "flake-parts": "flake-parts", "git-hooks-nix": "git-hooks-nix", "nixpkgs": "nixpkgs", diff --git a/flake.nix b/flake.nix index 67532424ee67..5d4dbfb17a86 100644 --- a/flake.nix +++ b/flake.nix @@ -1,24 +1,22 @@ { description = "The purely functional package manager"; - inputs.nixpkgs.url = "https://channels.nixos.org/nixos-26.05/nixexprs.tar.xz"; + inputs.nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/0.2605"; inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2"; inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446"; - inputs.flake-compat = { - url = "github:NixOS/flake-compat"; + + inputs.bdwgc = { + url = "github:DeterminateSystems/bdwgc/user-defined-stacks"; flake = false; }; # dev tooling - inputs.flake-parts.url = "github:hercules-ci/flake-parts"; - inputs.git-hooks-nix.url = "github:cachix/git-hooks.nix"; + inputs.flake-parts.url = "https://flakehub.com/f/hercules-ci/flake-parts/0.1"; + inputs.git-hooks-nix.url = "https://flakehub.com/f/cachix/git-hooks.nix/0.1.941"; # work around https://github.com/NixOS/nix/issues/7730 inputs.flake-parts.inputs.nixpkgs-lib.follows = "nixpkgs"; inputs.git-hooks-nix.inputs.nixpkgs.follows = "nixpkgs"; - # work around 7730 and https://github.com/NixOS/nix/issues/7807 - inputs.git-hooks-nix.inputs.flake-compat.follows = ""; - inputs.git-hooks-nix.inputs.gitignore.follows = ""; outputs = inputs@{ @@ -33,26 +31,24 @@ officialRelease = true; - linux32BitSystems = [ "i686-linux" ]; + linux32BitSystems = [ ]; linux64BitSystems = [ "x86_64-linux" "aarch64-linux" ]; linuxSystems = linux32BitSystems ++ linux64BitSystems; darwinSystems = [ - "x86_64-darwin" "aarch64-darwin" ]; systems = linuxSystems ++ darwinSystems; crossSystems = [ - "armv6l-unknown-linux-gnueabihf" - "armv7l-unknown-linux-gnueabihf" - "riscv64-unknown-linux-gnu" + #"armv6l-unknown-linux-gnueabihf" + #"armv7l-unknown-linux-gnueabihf" + #"riscv64-unknown-linux-gnu" # Disabled because of https://github.com/NixOS/nixpkgs/issues/344423 # "x86_64-unknown-netbsd" - "x86_64-unknown-freebsd" - "x86_64-w64-mingw32" + #"x86_64-unknown-freebsd" ]; stdenvs = [ @@ -365,6 +361,40 @@ nix-manual-manpages-only = nixpkgsFor.${system}.native.nixComponents2.nix-manual-manpages-only; nix-internal-api-docs = nixpkgsFor.${system}.native.nixComponents2.nix-internal-api-docs; nix-external-api-docs = nixpkgsFor.${system}.native.nixComponents2.nix-external-api-docs; + + fallbackPathsNix = + let + pkgs = nixpkgsFor.${system}.native; + + closures = forAllSystems (system: self.packages.${system}.default.outPath); + + closures_json = + pkgs.runCommand "versions.json" + { + buildInputs = [ pkgs.jq ]; + passAsFile = [ "json" ]; + json = builtins.toJSON closures; + } + '' + cat "$jsonPath" | jq . > $out + ''; + + closures_nix = + pkgs.runCommand "versions.nix" + { + buildInputs = [ pkgs.jq ]; + passAsFile = [ "template" ]; + jsonPath = closures_json; + template = '' + builtins.fromJSON('''@closures@''') + ''; + } + '' + export closures=$(cat "$jsonPath"); + substituteAll "$templatePath" "$out" + ''; + in + closures_nix; } # We need to flatten recursive attribute sets of derivations to pass `flake check`. // @@ -429,8 +459,6 @@ { # These attributes go right into `packages.`. "${pkgName}" = nixpkgsFor.${system}.native.nixComponents2.${pkgName}; - "${pkgName}-static" = nixpkgsFor.${system}.native.pkgsStatic.nixComponents2.${pkgName}; - "${pkgName}-llvm" = nixpkgsFor.${system}.native.pkgsLLVM.nixComponents2.${pkgName}; } // flatMapAttrs (lib.genAttrs stdenvs (_: { })) ( stdenvName: @@ -454,6 +482,15 @@ } ) ) + // lib.optionalAttrs (linuxOnly -> nixpkgsFor.${system}.native.stdenv.hostPlatform.isLinux) { + "${pkgName}-static" = + let + pkgs = nixpkgsFor.${system}; + in + ( + if pkgs.native.stdenv.hostPlatform.isDarwin then pkgs.nativeForStdenv.libcxxStdenv else pkgs.native + ).pkgsStatic.nixComponents2.${pkgName}; + } ) // lib.optionalAttrs (self.hydraJobs.rustInstaller ? ${system}) { rustInstaller = self.hydraJobs.rustInstaller.${system}; @@ -523,32 +560,6 @@ } ) ) - // lib.optionalAttrs (!nixpkgsFor.${system}.native.stdenv.isDarwin) ( - prefixAttrs "static" ( - forAllStdenvs ( - stdenvName: - makeShell' { - pkgs = nixpkgsFor.${system}.nativeForStdenv.${stdenvName}.pkgsStatic; - } - ) - ) - // prefixAttrs "llvm" ( - forAllStdenvs ( - stdenvName: - makeShell' { - pkgs = nixpkgsFor.${system}.nativeForStdenv.${stdenvName}.pkgsLLVM; - } - ) - ) - // prefixAttrs "cross" ( - forAllCrossSystems ( - crossSystem: - makeShell' { - pkgs = nixpkgsFor.${system}.cross.${crossSystem}; - } - ) - ) - ) // { native = self.devShells.${system}.native-stdenv; default = self.devShells.${system}.native; diff --git a/maintainers/flake-module.nix b/maintainers/flake-module.nix index 5978d997b6f8..84207a398a5d 100644 --- a/maintainers/flake-module.nix +++ b/maintainers/flake-module.nix @@ -16,6 +16,16 @@ # https://flake.parts/options/git-hooks-nix#options pre-commit.settings = { + # `self.outPath` from git-hooks-nix's flake-module is not path-normalized, + # so when this flake is evaluated via `packaging/secure-packages` (whose + # `inputs.nix.url = "../.."`), it expands to `.../source/packaging/secure-packages/../..`, + # which breaks the stdenv unpackPhase. Use a clean relative path instead. + rootSrc = lib.mkForce ( + builtins.path { + name = "source"; + path = ../.; + } + ); hooks = { # Conflicts are usually found by other checks, but not those in docs, # and potentially other places. @@ -101,6 +111,7 @@ # Don't format vendored code ''^doc/manual/redirects\.js$'' ''^doc/manual/theme/highlight\.js$'' + ''^src/libfetchers/builtin-flake-registry\.json$'' ]; }; shellcheck = { diff --git a/maintainers/link-headers b/maintainers/link-headers new file mode 100755 index 000000000000..2457a2dc8295 --- /dev/null +++ b/maintainers/link-headers @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 + +# This script must be run from the root of the Nix repository. +# +# For include path hygiene, we need to put headers in a separate +# directory than sources. But during development, it is nice to paths +# that are similar for headers and source files, e.g. +# `foo/bar/baz.{cc,hh}`, e.g. for less typing when opening one file, and +# then opening the other file. +# +# This script symlinks the headers next to the source files to +# facilitate such a development workflows. It also updates +# `.git/info/exclude` so that the symlinks are not accidentally committed +# by mistake. + +from pathlib import Path +import subprocess +import os + + +def main() -> None: + # Path to the source directory + GIT_TOPLEVEL = Path( + subprocess.run( + ["git", "rev-parse", "--show-toplevel"], + text=True, + stdout=subprocess.PIPE, + check=True, + ).stdout.strip() + ) + + # Get header files from git + result = subprocess.run( + ["git", "-C", str(GIT_TOPLEVEL), "ls-files", "*/include/nix/**.hh"], + text=True, + stdout=subprocess.PIPE, + check=True, + ) + header_files = result.stdout.strip().split("\n") + header_files.sort() + + links = [] + for file_str in header_files: + project_str, header_str = file_str.split("/include/nix/", 1) + project = Path(project_str) + header = Path(header_str) + + # Reconstruct the full path (relative to SRC_DIR) to the header file. + file = project / "include" / "nix" / header + + # The symlink should be created at "project/header", i.e. next to the project's sources. + link = project / header + + # Compute a relative path from the symlink's parent directory to the actual header file. + relative_source = os.path.relpath( + GIT_TOPLEVEL / file, GIT_TOPLEVEL / link.parent + ) + + # Create the symbolic link. + full_link_path = GIT_TOPLEVEL / link + full_link_path.parent.mkdir(parents=True, exist_ok=True) + if full_link_path.is_symlink(): + full_link_path.unlink() + full_link_path.symlink_to(relative_source) + links.append(link) + + # Generate .gitignore file + gitignore_path = GIT_TOPLEVEL / ".git" / "info" / "exclude" + gitignore_path.parent.mkdir(parents=True, exist_ok=True) + with gitignore_path.open("w") as gitignore: + gitignore.write("# DO NOT EDIT! Autogenerated\n") + gitignore.write( + "# Symlinks for headers to be next to sources for development\n" + ) + gitignore.write('# Run "maintainers/link-headers" to regenerate\n\n') + gitignore.write('# Run "maintainers/link-headers" to regenerate\n\n') + + for link in links: + gitignore.write(f"/{link}\n") + + +if __name__ == "__main__": + main() diff --git a/maintainers/upload-debug-info-to-sentry.py b/maintainers/upload-debug-info-to-sentry.py new file mode 100755 index 000000000000..d183311ea9f7 --- /dev/null +++ b/maintainers/upload-debug-info-to-sentry.py @@ -0,0 +1,170 @@ +#!/usr/bin/env nix +#!nix shell --inputs-from . nixpkgs#sentry-cli nixpkgs#python3 nixpkgs#binutils nixpkgs#xz nixpkgs#zstd --command python3 + +import argparse +import json +import os +import platform +import re +import subprocess +import sys +import urllib.error +import urllib.parse +import urllib.request + +NAR_DIR = "/tmp/nars" +DEBUG_INFO_DIR = "/tmp/debug-info" + + +def get_dynamic_libraries(executable: str) -> list[str]: + if platform.system() == "Darwin": + result = subprocess.run(["otool", "-L", executable], capture_output=True, text=True, check=True) + libs = [] + for line in result.stdout.splitlines()[1:]: # skip first line (the binary path itself) + # otool -L output lines look like: + # /nix/store/.../libfoo.dylib (compatibility version X.Y.Z, current version A.B.C) + m = re.match(r"\s+(\S+)\s+\(", line) + if m: + libs.append(m.group(1)) + return libs + else: + result = subprocess.run(["ldd", executable], capture_output=True, text=True, check=True) + libs = [] + for line in result.stdout.splitlines(): + # ldd output lines look like: + # libfoo.so.1 => /nix/store/.../libfoo.so.1 (0x...) + # /lib64/ld-linux-x86-64.so.2 (0x...) + m = re.search(r"=> (/\S+)", line) + if m: + libs.append(m.group(1)) + elif line.strip().startswith("/"): + path = line.strip().split()[0] + libs.append(path) + return libs + + +def get_build_id(path: str) -> str | None: + result = subprocess.run(["readelf", "-n", path], capture_output=True, text=True) + m = re.search(r"Build ID:\s+([0-9a-f]+)", result.stdout) + return m.group(1) if m else None + + +def download_nar(build_id: str, archive: str) -> str: + """Download a NAR to /tmp/nars and return the local path. Skips if already present.""" + base_url = f"https://cache.nixos.org/debuginfo/{build_id}" + nar_url = urllib.parse.urljoin(base_url, archive) + filename = nar_url.split("/")[-1] + local_path = os.path.join(NAR_DIR, filename) + if not os.path.exists(local_path): + os.makedirs(NAR_DIR, exist_ok=True) + print(f" downloading {nar_url} ...", file=sys.stderr) + urllib.request.urlretrieve(nar_url, local_path) + else: + print(f" already have {filename}", file=sys.stderr) + return local_path + + +def extract_debug_symbols(nar_path: str, member: str, build_id: str) -> str: + """Extract a member from a compressed NAR into /tmp/debug-info/.debug. Returns the output path.""" + out_path = os.path.join(DEBUG_INFO_DIR, f"{build_id}.debug") + if os.path.exists(out_path): + print(f" already extracted {out_path}", file=sys.stderr) + return out_path + os.makedirs(DEBUG_INFO_DIR, exist_ok=True) + print(f" extracting {member} -> {out_path} ...", file=sys.stderr) + if nar_path.endswith(".zst"): + decompress_cmd = ["zstd", "-d", "-c"] + elif nar_path.endswith(".xz"): + decompress_cmd = ["xz", "-d"] + else: + raise Exception(f"unknown NAR compression format: {nar_path}") + decompress = subprocess.Popen(decompress_cmd, stdin=open(nar_path, "rb"), stdout=subprocess.PIPE) + nar_cat = subprocess.run( + ["nix", "nar", "cat", "/dev/stdin", member], + stdin=decompress.stdout, + capture_output=True, + check=True, + ) + decompress.wait() + with open(out_path, "wb") as f: + f.write(nar_cat.stdout) + return out_path + + +def find_debug_file_in_dirs(build_id: str, debug_dirs: list[str]) -> str | None: + """Look for a .debug file by build ID under

/lib/debug/.build-id/NN/NNN.debug.""" + subpath = os.path.join("lib", "debug", ".build-id", build_id[:2], build_id[2:] + ".debug") + for d in debug_dirs: + candidate = os.path.join(d, subpath) + if os.path.exists(candidate): + return candidate + return None + + +def fetch_debuginfo(build_id: str) -> dict | None: + url = f"https://cache.nixos.org/debuginfo/{build_id}" + try: + with urllib.request.urlopen(url) as resp: + return json.loads(resp.read()) + except urllib.error.HTTPError as e: + if e.code == 404: + return None + raise + + +def main(): + parser = argparse.ArgumentParser( + description="Upload debug symbols to Sentry." + ) + parser.add_argument("executable", help="Path to the executable (e.g. ./result/bin/nix)") + parser.add_argument("--project", help="Sentry project ID") + parser.add_argument("--debug-dir", action="append", default=[], metavar="DIR", + help="Directory to search for debug files (may be repeated, Linux only)") + args = parser.parse_args() + + libs = [args.executable] + get_dynamic_libraries(args.executable) + + if platform.system() == "Darwin": + # On macOS there are no separate debug info files; upload the binaries directly. + print("Files to upload:", file=sys.stderr) + for lib in libs: + print(f" {lib}", file=sys.stderr) + files_to_upload = libs + else: + debug_files = [] + print("ELF files to process:", file=sys.stderr) + for lib in libs: + debug_files.append(lib) + + build_id = get_build_id(lib) + if build_id is None: + print(f" {lib} (no build ID, uploading binary)", file=sys.stderr) + continue + + local = find_debug_file_in_dirs(build_id, args.debug_dir) + if local: + print(f" {lib} ({build_id}): found locally at {local}", file=sys.stderr) + debug_files.append(local) + continue + + debuginfo = fetch_debuginfo(build_id) + if debuginfo is None: + print(f" {lib} ({build_id}): no separate debug info", file=sys.stderr) + continue + + print(f" {lib} ({build_id}): member={debuginfo['member']}", file=sys.stderr) + nar_path = download_nar(build_id, debuginfo["archive"]) + debug_file = extract_debug_symbols(nar_path, debuginfo["member"], build_id) + debug_files.append(debug_file) + files_to_upload = debug_files + + if files_to_upload: + print(f"Uploading {len(files_to_upload)} file(s) to Sentry...", file=sys.stderr) + cmd = ["sentry-cli", "debug-files", "upload"] + if args.project: + cmd += ["--project", args.project] + subprocess.run(cmd + files_to_upload, check=True) + + +if __name__ == "__main__": + main() diff --git a/misc/bash/completion.sh b/misc/bash/completion.sh index 96f98d6c13d8..384cd5a78991 100644 --- a/misc/bash/completion.sh +++ b/misc/bash/completion.sh @@ -1,8 +1,11 @@ # shellcheck shell=bash function _complete_nix { local -a words - local cword cur - _get_comp_words_by_ref -n ':=&' words cword cur + # shellcheck disable=SC2034 # prev is set by _init_completion + local cword cur prev + # This does filename completion after redirections (like `< foo`), + # and removes redirections from `words`. + _init_completion -n ':=&' || return local have_type while IFS= read -r line; do local completion=${line%% *} diff --git a/nix-meson-build-support/common/assert-fail/wrap-assert-fail.cc b/nix-meson-build-support/common/assert-fail/wrap-assert-fail.cc index d9e34168bc97..292c57e7e4c7 100644 --- a/nix-meson-build-support/common/assert-fail/wrap-assert-fail.cc +++ b/nix-meson-build-support/common/assert-fail/wrap-assert-fail.cc @@ -1,4 +1,5 @@ #include "nix/util/error.hh" +#include "nix/util/sentry.hh" #include #include @@ -13,5 +14,6 @@ __wrap___assert_fail(const char * assertion, const char * file, unsigned int lin snprintf(buf, sizeof(buf), "Assertion '%s' failed in %s at %s:%" PRIuLEAST32, assertion, function, file, line); if (n < 0) nix::panic("Assertion failed and could not format error message"); - nix::panic(std::string_view(buf, std::min(static_cast(sizeof(buf)), n))); + nix::setSentryTag("assertion", buf); + nix::panic(buf); } diff --git a/nix-meson-build-support/common/cxa-throw/interpose-cxa-throw.cc b/nix-meson-build-support/common/cxa-throw/interpose-cxa-throw.cc new file mode 100644 index 000000000000..7238800b1906 --- /dev/null +++ b/nix-meson-build-support/common/cxa-throw/interpose-cxa-throw.cc @@ -0,0 +1,21 @@ +#include +#include +#include + +#include "is-logic-error.hh" + +typedef void (*cxa_throw_type)(void *, std::type_info *, void (*)(void *)); + +extern "C" void __cxa_throw(void * exc, std::type_info * tinfo, void (*dest)(void *)) +{ + if (is_logic_error(tinfo)) + abort_on_exception(exc, tinfo); + + static auto * orig = (cxa_throw_type) dlsym(RTLD_NEXT, "__cxa_throw"); + if (!orig) + abort(); + + orig(exc, tinfo, dest); + + __builtin_unreachable(); +} diff --git a/nix-meson-build-support/common/cxa-throw/is-logic-error.hh b/nix-meson-build-support/common/cxa-throw/is-logic-error.hh new file mode 100644 index 000000000000..dc6285a51121 --- /dev/null +++ b/nix-meson-build-support/common/cxa-throw/is-logic-error.hh @@ -0,0 +1,62 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +#ifndef CXA_THROW_ON_LOGIC_ERROR +# define CXA_THROW_ON_LOGIC_ERROR() abort() +#endif + +#include + +#ifndef __GLIBCXX__ +/* libc++abi implements the Itanium ABI RTTI classes but, unlike + libstdc++, doesn't declare them in . Their layout is + fixed by the ABI, so declare what we need. */ +namespace __cxxabiv1 { +class __class_type_info : public std::type_info +{ +public: + ~__class_type_info() override; +}; + +class __si_class_type_info : public __class_type_info +{ +public: + const __class_type_info * __base_type; +}; +} // namespace __cxxabiv1 +#endif + +static bool is_logic_error(const std::type_info * tinfo) +{ + if (*tinfo == typeid(std::logic_error)) + return true; + + auto * si = dynamic_cast(tinfo); + if (si) + return is_logic_error(si->__base_type); + + return false; +} + +static void abort_on_exception(void * exc, const std::type_info * tinfo) +{ + if (!is_logic_error(tinfo)) + return; + + char buf[512]; + snprintf( + buf, + sizeof(buf), + "Aborting on unexpected exception of type '%s', error: %s\n", + tinfo->name(), + ((std::exception *) exc)->what()); + [[maybe_unused]] auto r = write(STDERR_FILENO, buf, strlen(buf)); + + CXA_THROW_ON_LOGIC_ERROR(); +} diff --git a/nix-meson-build-support/common/cxa-throw/meson.build b/nix-meson-build-support/common/cxa-throw/meson.build new file mode 100644 index 000000000000..e9947766cd40 --- /dev/null +++ b/nix-meson-build-support/common/cxa-throw/meson.build @@ -0,0 +1,30 @@ +have_cxa_throw = false + +if host_machine.system() == 'linux' + have_cxa_throw = true + + if get_option('default_library') == 'static' + wrap_cxa_throw_args = [ '-Wl,--wrap=__cxa_throw' ] + + wrap_cxa_throw_lib = static_library( + 'wrap-cxa-throw', + 'wrap-cxa-throw.cc', + ) + + cxa_throw_dep = declare_dependency( + link_whole : wrap_cxa_throw_lib, + link_args : wrap_cxa_throw_args + [ '-Wl,-u,__wrap___cxa_throw' ], + ) + + else + interpose_cxa_throw_lib = static_library( + 'interpose-cxa-throw', + 'interpose-cxa-throw.cc', + dependencies : cxx.find_library('dl', required : false), + ) + + cxa_throw_dep = declare_dependency( + link_whole : interpose_cxa_throw_lib, + ) + endif +endif diff --git a/nix-meson-build-support/common/cxa-throw/wrap-cxa-throw.cc b/nix-meson-build-support/common/cxa-throw/wrap-cxa-throw.cc new file mode 100644 index 000000000000..f9de15533de0 --- /dev/null +++ b/nix-meson-build-support/common/cxa-throw/wrap-cxa-throw.cc @@ -0,0 +1,16 @@ +#include +#include + +#include "is-logic-error.hh" + +extern "C" void __real___cxa_throw(void *, std::type_info *, void (*)(void *)); + +extern "C" void __wrap___cxa_throw(void * exc, std::type_info * tinfo, void (*dest)(void *)) +{ + if (is_logic_error(tinfo)) + abort_on_exception(exc, tinfo); + + __real___cxa_throw(exc, tinfo, dest); + + __builtin_unreachable(); +} diff --git a/nix-meson-build-support/common/meson.build b/nix-meson-build-support/common/meson.build index d4c89ce6f460..f7cca818057e 100644 --- a/nix-meson-build-support/common/meson.build +++ b/nix-meson-build-support/common/meson.build @@ -92,11 +92,18 @@ if is_using_libstdcxx add_project_arguments('-D_GLIBCXX_USE_TBB_PAR_BACKEND=0', language : 'cpp') endif -# Darwin ld doesn't like "X.Y.ZpreABCD+W" -nix_soversion = meson.project_version().split('+')[0].split('pre')[0] +cxx = meson.get_compiler('cpp') + +# Clang does not support prelinking on static builds +if cxx.get_id() == 'clang' and get_option('default_library') == 'static' + prelink = false +else + prelink = true +endif subdir('assert-fail') subdir('asan-options') +subdir('cxa-throw') if not meson.is_subproject() subdir('clang-tidy') endif diff --git a/nix-meson-build-support/default-system-cpu/meson.build b/nix-meson-build-support/default-system-cpu/meson.build index f63b07975b6e..3e872578efca 100644 --- a/nix-meson-build-support/default-system-cpu/meson.build +++ b/nix-meson-build-support/default-system-cpu/meson.build @@ -14,6 +14,6 @@ if (host_machine.cpu_family() in [ 'ppc64', 'ppc' ]) and host_machine.endian() = nix_system_cpu += 'le' elif host_machine.cpu_family() in [ 'mips64', 'mips' ] and host_machine.endian() == 'little' nix_system_cpu += 'el' -elif host_machine.cpu_family() == 'arm' +elif host_machine.cpu_family() in [ 'arm', 'arm64' ] nix_system_cpu = host_machine.cpu() endif diff --git a/packaging/binary-tarball.nix b/packaging/binary-tarball.nix index df407f8bf0c6..cc32268df6a7 100644 --- a/packaging/binary-tarball.nix +++ b/packaging/binary-tarball.nix @@ -79,10 +79,11 @@ runCommand "nix-binary-tarball-${version}" env '' fn=$out/$dir.tar.xz mkdir -p $out/nix-support echo "file binary-dist $fn" >> $out/nix-support/hydra-build-products + # Store mtime is 1 second into the epoch. tar cf - \ --sort=name \ --owner=0 --group=0 --mode=u+rw,uga+r \ - --mtime='1970-01-01' \ + --mtime='@1' \ --absolute-names \ --hard-dereference \ --transform "s,$TMPDIR/install,$dir/install," \ diff --git a/packaging/components.nix b/packaging/components.nix index 54d49bc60baf..d3f7e952a771 100644 --- a/packaging/components.nix +++ b/packaging/components.nix @@ -27,7 +27,7 @@ let pkg-config ; - baseVersion = lib.fileContents ../.version; + baseVersion = lib.fileContents ../.version-determinate; versionSuffix = lib.optionalString (!officialRelease) "pre"; @@ -54,15 +54,6 @@ let mkPackageBuilder = mkPackageBuilderFor stdenv; setVersionLayer = finalAttrs: prevAttrs: { - preConfigure = - prevAttrs.preConfigure or "" - + - # Update the repo-global .version file. - # Symlink ./.version points there, but by default only workDir is writable. - '' - chmod u+w ./.version - echo ${finalAttrs.version} > ./.version - ''; }; localSourceLayer = diff --git a/packaging/dependencies.nix b/packaging/dependencies.nix index 69c25b2b817a..169c68ffe266 100644 --- a/packaging/dependencies.nix +++ b/packaging/dependencies.nix @@ -17,31 +17,124 @@ scope: { inherit stdenv; mimalloc = - if lib.versionAtLeast pkgs.mimalloc.version "3.3.2" then - pkgs.mimalloc - else - pkgs.mimalloc.overrideAttrs rec { - version = "3.3.2"; - src = pkgs.fetchFromGitHub { - owner = "microsoft"; - repo = "mimalloc"; - tag = "v${version}"; - hash = "sha256-GZ37qQVDe9jgMb4Coe5oKvgaLTspZDlSkS5rdy1MfUU="; - }; - }; + ( + if lib.versionAtLeast pkgs.mimalloc.version "3.3.2" then + pkgs.mimalloc + else + pkgs.mimalloc.overrideAttrs rec { + version = "3.3.2"; + src = pkgs.fetchFromGitHub { + owner = "microsoft"; + repo = "mimalloc"; + tag = "v${version}"; + hash = "sha256-GZ37qQVDe9jgMb4Coe5oKvgaLTspZDlSkS5rdy1MfUU="; + }; + } + ).overrideAttrs + (attrs: { + cmakeFlags = (attrs.cmakeFlags or [ ]) ++ [ + # Don't `madvise(MADV_HUGEPAGE)` the 1 GiB arenas that + # mimalloc reserves. With the common kernel setting + # `transparent_hugepage/defrag=madvise`, that hint makes + # every first touch of a 2 MiB region in the arena attempt + # a huge page allocation with synchronous direct compaction. + # On a machine with fragmented physical memory this fails + # almost every time and costs several milliseconds per + # fault, which more than doubled the CPU time of evaluating + # large flakes. Defining this preprocessor macro only skips + # the `madvise()` call. (The cmake option `MI_NO_THP` and + # the runtime option `MIMALLOC_ALLOW_THP=0` are not + # equivalent: they also call `prctl(PR_SET_THP_DISABLE)`, + # which disables huge pages for the entire process and is + # inherited by the programs that `nix run` etc. execute.) + "-DMI_EXTRA_CPPDEFS=MI_NO_THP" + ]; + }); boehmgc = (pkgs.boehmgc.override { enableLargeConfig = true; + inherit stdenv; }).overrideAttrs (attrs: { - # Increase the initial mark stack size to avoid stack - # overflows, since these inhibit parallel marking (see - # GC_mark_some()). To check whether the mark stack is too - # small, run Nix with GC_PRINT_STATS=1 and look for messages - # such as `Mark stack overflow`, `No room to copy back mark - # stack`, and `Grew mark stack to ... frames`. - NIX_CFLAGS_COMPILE = "-DINITIAL_MARK_STACK_SIZE=1048576"; + src = inputs.bdwgc; + + nativeBuildInputs = (attrs.nativeBuildInputs or [ ]) ++ [ + pkgs.buildPackages.autoreconfHook + ]; + + env = (attrs.env or { }) // { + # Increase the initial mark stack size to avoid stack + # overflows, since these inhibit parallel marking (see + # GC_mark_some()). To check whether the mark stack is too + # small, run Nix with GC_PRINT_STATS=1 and look for messages + # such as `Mark stack overflow`, `No room to copy back mark + # stack`, and `Grew mark stack to ... frames`. + NIX_CFLAGS_COMPILE = toString ( + [ + "-DINITIAL_MARK_STACK_SIZE=1048576" + "-DGC_MANY_BLOCKS_DEFAULT=64" + # Disable black-listing (avoiding allocation on pages that + # false pointers refer to). The black lists are hash + # tables covering 8 GiB of address space; on larger heaps + # they alias and saturate, at which point the allocator + # cannot find any usable block for pointer-containing + # objects and aborts with "Too many retries in + # GC_allocobj" (bdwgc issues #691, #726). Nix doesn't + # benefit much from black-listing since interior pointers + # are disabled. + "-DNO_BLACK_LISTING" + # Serve allocations up to 1520 bytes (95 granules) from + # the per-thread freelists instead of taking the global + # allocation lock. The default (25, i.e. <= 384 bytes) is + # too small for parallel evaluation: e.g. a typical + # derivation attrset (~46 attrs) is a 752-byte Bindings, + # of which nixpkgs evaluation does hundreds of thousands, + # all serialized on GC_allocate_ml. + "-DGC_TINY_FREELISTS=96" + ] + # For some reason that is not clear, it is wanting to use libgcc_eh which is not available. + # Force this to be built with compiler-rt & libunwind over libgcc_eh works. + # Issue: https://github.com/NixOS/nixpkgs/issues/177129 + ++ + lib.optionals + ( + stdenv.cc.isClang + && stdenv.hostPlatform.isStatic + && stdenv.cc.libcxx != null + && stdenv.cc.libcxx.isLLVM + ) + [ + "-rtlib=compiler-rt" + "-unwindlib=libunwind" + ] + ); + }; + + buildInputs = + (attrs.buildInputs or [ ]) + ++ lib.optional ( + stdenv.cc.isClang + && stdenv.hostPlatform.isStatic + && stdenv.cc.libcxx != null + && stdenv.cc.libcxx.isLLVM + ) pkgs.llvmPackages.libunwind; + }); + + lowdown = + if lib.versionAtLeast pkgs.lowdown.version "2.0.2" then + pkgs.lowdown + else + pkgs.lowdown.overrideAttrs (prevAttrs: rec { + version = "2.0.2"; + src = pkgs.fetchurl { + url = "https://kristaps.bsd.lv/lowdown/snapshots/lowdown-${version}.tar.gz"; + hash = "sha512-cfzhuF4EnGmLJf5EGSIbWqJItY3npbRSALm+GarZ7SMU7Hr1xw0gtBFMpOdi5PBar4TgtvbnG4oRPh+COINGlA=="; + }; + nativeBuildInputs = prevAttrs.nativeBuildInputs ++ [ pkgs.buildPackages.bmake ]; + postInstall = + lib.replaceStrings [ "lowdown.so.1" "lowdown.1.dylib" ] [ "lowdown.so.2" "lowdown.2.dylib" ] + (prevAttrs.postInstall or ""); }); curl = pkgs.curl.override { @@ -50,17 +143,47 @@ scope: { zstdSupport = true; brotliSupport = true; zlibSupport = true; + # libpsl uses a data file needed at runtime, not useful for nix. + pslSupport = !stdenv.hostPlatform.isStatic; + idnSupport = !stdenv.hostPlatform.isStatic; }; - libblake3 = pkgs.libblake3.override { - useTBB = - !( - stdenv.hostPlatform.isWindows - || stdenv.hostPlatform.isStatic - # Some tbb tests fail with libc++. - || (stdenv.cc.libcxx != null && stdenv.cc.libcxx.isLLVM) + libblake3 = + (pkgs.libblake3.override { + inherit stdenv; + # Nixpkgs disables tbb on static + useTBB = + !( + stdenv.hostPlatform.isWindows + || stdenv.hostPlatform.isStatic + # Some tbb tests fail with libc++. + || (stdenv.cc.libcxx != null && stdenv.cc.libcxx.isLLVM) + ); + }) + # For some reason that is not clear, it is wanting to use libgcc_eh which is not available. + # Force this to be built with compiler-rt & libunwind over libgcc_eh works. + # Issue: https://github.com/NixOS/nixpkgs/issues/177129 + .overrideAttrs + ( + attrs: + lib.optionalAttrs + ( + stdenv.cc.isClang + && stdenv.hostPlatform.isStatic + && stdenv.cc.libcxx != null + && stdenv.cc.libcxx.isLLVM + ) + { + NIX_CFLAGS_COMPILE = [ + "-rtlib=compiler-rt" + "-unwindlib=libunwind" + ]; + + buildInputs = [ + pkgs.llvmPackages.libunwind + ]; + } ); - }; sqlite = if !stdenv.hostPlatform.isWindows then @@ -73,20 +196,35 @@ scope: { ]; }); - libgit2 = - if lib.versionAtLeast pkgs.libgit2.version "1.9.4" then - pkgs.libgit2 - else - # Grab newer libgit2. - pkgs.libgit2.overrideAttrs rec { - version = "1.9.4"; - src = pkgs.fetchFromGitHub { - owner = "libgit2"; - repo = "libgit2"; - tag = "v${version}"; - hash = "sha256-ZKUiz3pdFE2SKxh53X2oyr7hs32Njj5YVA0OXDXz7h0="; - }; + libgit2 = pkgs.libgit2.overrideAttrs ( + finalAttrs: prevAttrs: { + version = "2.0.0-rc.1"; + src = pkgs.fetchFromGitHub { + owner = "libgit2"; + repo = "libgit2"; + rev = "ae45d0d168f7e8dbfdb8c623589cb51caac96ab3"; + hash = "sha256-3sbqHm37SOwBeFgtjI2DLN6kx1F7G2N1m6rRIkqDXNI="; }; + patches = prevAttrs.patches or [ ] ++ [ + ./patches/0002-memory-config.patch + ./patches/0003-packbuilder-correct-config.patch + + # Fix a use-after-free crash when `git_thread_create` fails during + # pack building (e.g. with EAGAIN under thread pressure), leaving + # orphaned delta-search worker threads running while the + # packbuilder is freed. + # TODO: we can probably drop this patch since we're not finding deltas anymore. + ./patches/libgit2-packbuilder-dont-fail-on-thread-create-error.patch + ]; + separateDebugInfo = true; + # Nixpkgs derives `meta.changelog` from `src.tag`, which is null + # here since we fetch an untagged commit. This would be harmless + # except that nixpkgs variants with provenance support + # (`derivationWithMeta`) force `meta.changelog` at derivation + # instantiation time, causing an eval error. + meta = builtins.removeAttrs prevAttrs.meta [ "changelog" ]; + } + ); # TODO Hack until https://github.com/NixOS/nixpkgs/issues/45462 is fixed. boost = @@ -97,20 +235,35 @@ scope: { "--with-coroutine" "--with-iostreams" "--with-url" + "--with-thread" ]; - patches = lib.optional ( - lib.versionAtLeast pkgs.boost.version "1.88" - && lib.versionOlder pkgs.boost.version "1.92" - # may already be done in nixpkgs: https://github.com/NixOS/nixpkgs/pull/546405 - && !lib.any (patch: lib.hasInfix "5883212311535a0046031d74d1568ae173c1e35b" (baseNameOf patch)) ( - pkgs.boost.patches or [ ] - ) - ) ./patches/0001-Fix-uncaught_exceptions-not-accounting-for-forced_un.patch; enableIcu = false; + inherit stdenv; }).overrideAttrs (old: { # Need to remove `--with-*` to use `--with-libraries=...` buildPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.buildPhase; installPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.installPhase; }); + + # Build opentelemetry-cpp against the standard library, so that its + # API uses `std::string_view`, `std::shared_ptr` etc. instead of its + # own `nostd::` back-ports. This is an ABI switch, so it has to be + # done when building the library, not just on our side. + opentelemetry-cpp = pkgs.opentelemetry-cpp.override { cxxStandard = "20"; }; + + wasmtime = pkgs.callPackage ./wasmtime.nix { }; + + sentry-native = (pkgs.callPackage ./sentry-native.nix { }).override { + # Avoid having two curls in our closure. + inherit (scope) curl; + }; + + libmicrohttpd = pkgs.libmicrohttpd.overrideDerivation (old: { + # Don't pull in gnutls since it's pretty big and we don't need it. + configureFlags = old.configureFlags or [ ] ++ [ "--without-gnutls" ]; + + # Required for configuration detection for getsockname (for automatic port allocation for `nix serve`) + __darwinAllowLocalNetworking = true; + }); } diff --git a/packaging/dev-shell.nix b/packaging/dev-shell.nix index d18e77c52f12..ad634d475d19 100644 --- a/packaging/dev-shell.nix +++ b/packaging/dev-shell.nix @@ -210,7 +210,7 @@ nixComponents.callPackage ( }; # Remove the version suffix to avoid unnecessary attempts to substitute in nix develop - version = lib.fileContents ../.version; + version = lib.fileContents ../.version-determinate; name = finalAttrs.pname; installFlags = "sysconfdir=$(out)/etc"; diff --git a/packaging/everything.nix b/packaging/everything.nix index 152e633a11cc..c21c3cab681a 100644 --- a/packaging/everything.nix +++ b/packaging/everything.nix @@ -44,6 +44,10 @@ testers, patchedSrc ? null, + + curl, + boehmgc, + sentry-native, }: let @@ -66,7 +70,7 @@ let }; devdoc = buildEnv { - name = "nix-${nix-cli.version}-devdoc"; + name = "determinate-nix-${nix-cli.version}-devdoc"; paths = [ nix-internal-api-docs nix-external-api-docs @@ -75,7 +79,7 @@ let in stdenv.mkDerivation (finalAttrs: { - pname = "nix"; + pname = "determinate-nix"; version = nix-cli.version; /** @@ -93,6 +97,7 @@ stdenv.mkDerivation (finalAttrs: { "dev" "doc" "man" + "debug" ]; /** @@ -142,9 +147,18 @@ stdenv.mkDerivation (finalAttrs: { installPhase = let devPaths = lib.mapAttrsToList (_k: lib.getDev) finalAttrs.finalPackage.libs; + debugPaths = lib.map (lib.getOutput "debug") ( + lib.attrValues finalAttrs.finalPackage.libs + ++ [ + nix-cli + curl + boehmgc + ] + ++ lib.optional (stdenv.hostPlatform.isLinux && !stdenv.hostPlatform.isStatic) sentry-native + ); in '' - mkdir -p $out $dev/nix-support + mkdir -p $out $dev/nix-support $debug/lib/debug # Custom files echo $libs >> $dev/nix-support/propagated-build-inputs @@ -157,6 +171,12 @@ stdenv.mkDerivation (finalAttrs: { lndir $lib $dev done + for d in ${lib.escapeShellArgs debugPaths}; do + if [[ -d $d/lib/debug ]]; then + lndir $d/lib/debug $debug/lib/debug + fi + done + # Forwarded outputs ln -sT ${nix-manual} $doc ln -sT ${nix-manual.man} $man diff --git a/packaging/hydra.nix b/packaging/hydra.nix index e30b62eb578d..5600d394ebb5 100644 --- a/packaging/hydra.nix +++ b/packaging/hydra.nix @@ -122,85 +122,6 @@ rec { system: self.devShells.${system}.default.inputDerivation )) [ "i686-linux" ]; - buildStatic = forAllPackages ( - pkgName: - lib.genAttrs linux64BitSystems ( - system: nixpkgsFor.${system}.native.pkgsStatic.nixComponents2.${pkgName} - ) - ); - - buildCross = forAllPackages ( - pkgName: - # Hack to avoid non-evaling package - ( - if pkgName == "nix-functional-tests" then - lib.flip builtins.removeAttrs [ "x86_64-w64-mingw32" ] - else if pkgName == "nix-nswrapper" then - lib.filterAttrs ( - crossSystem: _do_not_touch: nixpkgsFor.x86_64-linux.cross.${crossSystem}.stdenv.hostPlatform.isLinux - ) - else - lib.id - ) - ( - forAllCrossSystems ( - crossSystem: - lib.genAttrs [ "x86_64-linux" ] ( - system: nixpkgsFor.${system}.cross.${crossSystem}.nixComponents2.${pkgName} - ) - ) - ) - ); - - # Builds with sanitizers already have GC disabled, so this buildNoGc can just - # point to buildWithSanitizers in order to reduce the load on hydra. - buildNoGc = buildWithSanitizers; - - buildWithSanitizers = - let - components = forAllSystems ( - system: - let - pkgs = nixpkgsFor.${system}.native; - in - pkgs.nixComponents2.overrideScope ( - self: super: { - # Boost coroutines fail with ASAN on darwin. - withASan = !pkgs.stdenv.buildPlatform.isDarwin; - withUBSan = true; - # Build without unity to catch include issues. - withUnityBuild = false; - nix-expr = super.nix-expr.override { enableGC = false; }; - } - ) - ); - in - forAllPackages ( - pkgName: - lib.filterAttrs ( - system: _do_not_touch: - pkgName == "nix-nswrapper" -> nixpkgsFor.${system}.native.stdenv.hostPlatform.isLinux - ) (forAllSystems (system: components.${system}.${pkgName})) - ); - - # Separate build because one cannot mix ASan + UBSan with TSan. - buildWithTSan = - let - components = - system: - let - pkgs = nixpkgsFor.${system}.native; - in - pkgs.nixComponents2.overrideScope ( - self: super: { - withTSan = true; - # TSan has issues with fork and threads. - nix-functional-tests = super.nix-functional-tests.overrideAttrs { doCheck = false; }; - } - ); - in - forAllPackages (pkgName: lib.genAttrs linux64BitSystems (system: (components system).${pkgName})); - # Static analysis with clang-tidy clangTidy = lib.genAttrs linux64BitSystems ( system: @@ -221,32 +142,6 @@ rec { tidyScope.nix-everything ); - buildNoTests = forAllSystems (system: nixpkgsFor.${system}.native.nixComponents2.nix-cli); - - # Toggles some settings for better coverage. Windows needs these - # library combinations, and Debian build Nix with GNU readline too. - buildReadlineNoMarkdown = - let - components = forAllSystems ( - system: - nixpkgsFor.${system}.native.nixComponents2.overrideScope ( - self: super: { - nix-cmd = super.nix-cmd.override { - enableMarkdown = false; - readlineFlavor = "readline"; - }; - } - ) - ); - in - forAllPackages ( - pkgName: - lib.filterAttrs ( - system: _do_not_touch: - pkgName == "nix-nswrapper" -> nixpkgsFor.${system}.native.stdenv.hostPlatform.isLinux - ) (forAllSystems (system: components.${system}.${pkgName})) - ); - # Binary tarball for various platforms, containing a Nix store # with the closure of 'nix' package, and the second half of # the installation script. @@ -254,31 +149,6 @@ rec { system: nixpkgsFor.${system}.native.callPackage ./binary-tarball.nix { } ); - binaryTarballCross = lib.genAttrs [ "x86_64-linux" ] ( - system: - forAllCrossSystems ( - crossSystem: nixpkgsFor.${system}.cross.${crossSystem}.callPackage ./binary-tarball.nix { } - ) - ); - - # The first half of the installation script. This is uploaded - # to https://nixos.org/nix/install. It downloads the binary - # tarball for the user's system and calls the second half of the - # installation script. - installerScript = installScriptFor [ - # Native - self.hydraJobs.binaryTarball."x86_64-linux" - self.hydraJobs.binaryTarball."i686-linux" - self.hydraJobs.binaryTarball."aarch64-linux" - self.hydraJobs.binaryTarball."x86_64-darwin" - self.hydraJobs.binaryTarball."aarch64-darwin" - # Cross - self.hydraJobs.binaryTarballCross."x86_64-linux"."armv6l-unknown-linux-gnueabihf" - self.hydraJobs.binaryTarballCross."x86_64-linux"."armv7l-unknown-linux-gnueabihf" - self.hydraJobs.binaryTarballCross."x86_64-linux"."riscv64-unknown-linux-gnu" - self.hydraJobs.binaryTarballCross."x86_64-linux"."x86_64-unknown-freebsd" - ]; - installerScriptForGHA = forAllSystems ( system: nixpkgsFor.${system}.native.callPackage ./installer { @@ -292,7 +162,6 @@ rec { ( linux64BitSystems ++ [ - "x86_64-darwin" "aarch64-darwin" ] ) @@ -373,6 +242,19 @@ rec { } ); + nixpkgsLibTestsLazy = forAllSystems ( + system: + lib.overrideDerivation + (import (nixpkgs + "/lib/tests/test-with-nix.nix") { + lib = nixpkgsFor.${system}.native.lib; + nix = self.packages.${system}.nix-cli; + pkgs = nixpkgsFor.${system}.native; + }) + (_: { + "NIX_CONFIG" = "lazy-trees = true"; + }) + ); + filetransfer-retry-backoff = forAllSystems ( system: nixpkgsFor.${system}.native.callPackage ../tests/filetransfer-retry-backoff { @@ -393,17 +275,12 @@ rec { in pkgs.runCommand "install-tests" { againstSelf = testNixVersions pkgs pkgs.nix; - againstCurrentLatest = - # FIXME: temporarily disable this on macOS because of #3605. - if system == "x86_64-linux" then testNixVersions pkgs pkgs.nixVersions.latest else null; + #againstCurrentLatest = + # # FIXME: temporarily disable this on macOS because of #3605. + # if system == "x86_64-linux" then testNixVersions pkgs pkgs.nixVersions.latest else null; # Disabled because the latest stable version doesn't handle # `NIX_DAEMON_SOCKET_PATH` which is required for the tests to work # againstLatestStable = testNixVersions pkgs pkgs.nixStable; } "touch $out" ); - - installerTests = import ../tests/installer { - binaryTarballs = self.hydraJobs.binaryTarball; - inherit nixpkgsFor; - }; } diff --git a/packaging/installer/default.nix b/packaging/installer/default.nix index e171f36f99f7..a8e344b496c8 100644 --- a/packaging/installer/default.nix +++ b/packaging/installer/default.nix @@ -32,7 +32,7 @@ runCommand "installer-script" in '' \ - --replace '@tarballHash_${system}@' $(nix --experimental-features nix-command hash-file --base16 --type sha256 ${tarball}/*.tar.xz) \ + --replace '@tarballHash_${system}@' $(nix hash-file --base16 --type sha256 ${tarball}/*.tar.xz) \ --replace '@tarballPath_${system}@' $(tarballPath ${tarball}/*.tar.xz) \ '' ) tarballs diff --git a/packaging/nix-make/compile.py b/packaging/nix-make/compile.py new file mode 100644 index 000000000000..b4aea25b08f4 --- /dev/null +++ b/packaging/nix-make/compile.py @@ -0,0 +1,77 @@ +# The builder for a compilation unit. It reads the derivation's structured +# attributes (see `compileUnit` in lib.nix) and runs the compiler once. +# +# This is Python rather than bash because the attributes carry typed JSON +# (e.g. `null` and booleans in `defines`), which bash cannot represent. The +# stdenv setup script has been sourced by the bash prelude in lib.nix, so +# the compiler (`$CXX`), the include paths of the build inputs and +# pkg-config are set up. + +import json +import os +import subprocess +import sys + + +def main(): + with open(os.environ["NIX_ATTRS_JSON_FILE"]) as f: + attrs = json.load(f) + + cxx = os.environ.get("CXX") + if not cxx: + sys.exit("CXX is not set; was the stdenv setup script sourced?") + + # Recreate the source layout: each header and the source file is a + # separate store path. + os.mkdir("tree") + os.chdir("tree") + for path, store in attrs["includes"].items(): + os.makedirs(os.path.dirname(path) or ".", exist_ok=True) + os.symlink(store, path) + os.makedirs(os.path.dirname(attrs["srcPath"]) or ".", exist_ok=True) + os.symlink(attrs["src"], attrs["srcPath"]) + + include_flags = ["-I" + (d or ".") for d in attrs["includeDirs"]] + + # The configuration macros this unit is sensitive to: `null` means + # undefined; strings become C string literals. + define_flags = [] + for name, value in attrs["defines"].items(): + if value is None: + define_flags.append(f"-U{name}") + elif isinstance(value, bool): + define_flags.append(f"-D{name}={int(value)}") + elif isinstance(value, int): + define_flags.append(f"-D{name}={value}") + elif isinstance(value, str): + define_flags.append(f"-D{name}={json.dumps(value)}") + else: + sys.exit(f"unsupported value for macro {name}: {value!r}") + + pkg_config_flags = [] + if attrs["pkgConfigDeps"]: + pkg_config_flags = subprocess.run( + ["pkg-config", "--cflags", *attrs["pkgConfigDeps"]], + check=True, + capture_output=True, + text=True, + ).stdout.split() + + # Replace this process by the compiler rather than waiting for it. + os.execvp( + cxx, + [ + cxx, + *attrs["cxxFlags"], + *include_flags, + *define_flags, + *pkg_config_flags, + "-c", + attrs["srcPath"], + "-o", + os.environ["out"], + ], + ) + + +main() diff --git a/packaging/nix-make/config.nix b/packaging/nix-make/config.nix new file mode 100644 index 000000000000..ca4a33332b07 --- /dev/null +++ b/packaging/nix-make/config.nix @@ -0,0 +1,59 @@ +# The build configuration options, as a NixOS-style module. A build +# variant is defined by evaluating this module together with a module +# setting the options, see `makeNixVariant` in flake.nix. +{ lib, ... }: + +{ + options = { + compiler = lib.mkOption { + type = lib.types.enum [ + "gcc" + "clang" + ]; + default = "gcc"; + description = "The C++ compiler to build Nix with. Dependencies are used as built by Nixpkgs either way."; + }; + + optimize = lib.mkOption { + type = lib.types.bool; + default = true; + description = "Whether to compile with optimization."; + }; + + debug = lib.mkOption { + type = lib.types.bool; + default = false; + description = "Whether to include debug information."; + }; + + sentry = lib.mkOption { + type = lib.types.bool; + default = true; + description = "Whether to enable Sentry crash reporting in the `nix` executable."; + }; + + otel = lib.mkOption { + type = lib.types.bool; + default = true; + description = "Whether to enable distributed tracing via OpenTelemetry in the `nix` executable."; + }; + + awsAuth = lib.mkOption { + type = lib.types.bool; + default = true; + description = "Whether to support AWS authentication for S3 binary caches."; + }; + + boehmgc = lib.mkOption { + type = lib.types.bool; + default = true; + description = "Whether to use the Boehm garbage collector in the Nix language evaluator."; + }; + + wasm = lib.mkOption { + type = lib.types.bool; + default = true; + description = "Whether to support WebAssembly: `builtins.wasm` in the evaluator and building WASI derivations."; + }; + }; +} diff --git a/packaging/nix-make/deps.nix b/packaging/nix-make/deps.nix new file mode 100644 index 000000000000..50a2454bb568 --- /dev/null +++ b/packaging/nix-make/deps.nix @@ -0,0 +1,247 @@ +# Table mapping external (angle-bracket) `#include`s to packages, by +# prefix. More specific prefixes must come first. Each entry has: +# +# - `prefix`: matched against the start of the included path. +# - `pkg`: the package(s) providing the headers and libraries. +# - `pkgconfig`: pkg-config names used for compile and link flags. +# - `libs`: extra linker flags for libraries without pkg-config files. +# - `cflags`: extra compiler flags for libraries without pkg-config files. +{ pkgs }: + +let + # The overridden dependencies used by the Meson build, layered over Nixpkgs. + deps = pkgs // pkgs.nixDependencies2; +in +[ + { + prefix = "boost/context"; + pkg = deps.boost; + libs = [ "-lboost_context" ]; + } + { + # Also matches coroutine2, which is header-only on top of libboost_context. + prefix = "boost/coroutine"; + pkg = deps.boost; + libs = [ + "-lboost_coroutine" + "-lboost_context" + ]; + } + { + prefix = "boost/thread"; + pkg = deps.boost; + libs = [ "-lboost_thread" ]; + } + { + prefix = "boost/iostreams"; + pkg = deps.boost; + libs = [ "-lboost_iostreams" ]; + } + { + prefix = "boost/url"; + pkg = deps.boost; + libs = [ "-lboost_url" ]; + } + { + prefix = "boost/container"; + pkg = deps.boost; + libs = [ "-lboost_container" ]; + } + { + prefix = "boost/"; + pkg = deps.boost; + } + { + prefix = "nlohmann/"; + pkg = deps.nlohmann_json; + pkgconfig = [ "nlohmann_json" ]; + } + { + prefix = "archive"; + pkg = deps.libarchive; + pkgconfig = [ "libarchive" ]; + } + { + prefix = "openssl/"; + pkg = deps.openssl; + pkgconfig = [ "libcrypto" ]; + } + { + prefix = "sodium.h"; + pkg = deps.libsodium; + pkgconfig = [ "libsodium" ]; + } + { + prefix = "blake3.h"; + pkg = deps.libblake3; + pkgconfig = [ "libblake3" ]; + } + { + prefix = "brotli/"; + pkg = deps.brotli; + pkgconfig = [ + "libbrotlicommon" + "libbrotlidec" + "libbrotlienc" + ]; + } + { + prefix = "zstd.h"; + pkg = deps.zstd; + pkgconfig = [ "libzstd" ]; + } + { + prefix = "libcpuid/"; + pkg = deps.libcpuid; + pkgconfig = [ "libcpuid" ]; + } + { + prefix = "curl/"; + pkg = deps.curl; + pkgconfig = [ "libcurl" ]; + } + { + prefix = "seccomp.h"; + pkg = deps.libseccomp; + pkgconfig = [ "libseccomp" ]; + } + { + prefix = "sqlite3.h"; + pkg = deps.sqlite; + pkgconfig = [ "sqlite3" ]; + } + { + prefix = "gc.h"; + pkg = deps.boehmgc; + pkgconfig = [ "bdw-gc" ]; + } + { + prefix = "gc/"; + pkg = deps.boehmgc; + pkgconfig = [ "bdw-gc" ]; + } + { + # Header-only. + prefix = "toml.hpp"; + pkg = deps.toml11; + } + { + prefix = "toml11/"; + pkg = deps.toml11; + } + { + # The AWS CRT and its C libraries have no pkg-config files; the + # defines are what their CMake targets would set. + prefix = "aws/"; + pkg = with deps; [ + aws-crt-cpp + aws-c-auth + aws-c-cal + aws-c-common + aws-c-compression + aws-c-event-stream + aws-checksums + aws-c-http + aws-c-io + aws-c-mqtt + aws-c-s3 + aws-c-sdkutils + s2n-tls + ]; + cflags = [ + "-DAWS_ENABLE_EPOLL" + "-DAWS_AUTH_USE_IMPORT_EXPORT" + "-DAWS_CAL_USE_IMPORT_EXPORT" + "-DAWS_CHECKSUMS_USE_IMPORT_EXPORT" + "-DAWS_COMMON_USE_IMPORT_EXPORT" + "-DAWS_COMPRESSION_USE_IMPORT_EXPORT" + "-DAWS_CRT_CPP_USE_IMPORT_EXPORT" + "-DAWS_EVENT_STREAM_USE_IMPORT_EXPORT" + "-DAWS_HTTP_USE_IMPORT_EXPORT" + "-DAWS_IO_USE_IMPORT_EXPORT" + "-DAWS_MQTT_USE_IMPORT_EXPORT" + "-DAWS_S3_USE_IMPORT_EXPORT" + "-DAWS_SDKUTILS_USE_IMPORT_EXPORT" + ]; + libs = [ + "-laws-crt-cpp" + "-laws-c-auth" + "-laws-c-cal" + "-laws-c-common" + "-laws-c-compression" + "-laws-c-event-stream" + "-laws-checksums" + "-laws-c-http" + "-laws-c-io" + "-laws-c-mqtt" + "-laws-c-s3" + "-laws-c-sdkutils" + "-ls2n" + ]; + } + { + prefix = "opentelemetry/"; + pkg = deps.opentelemetry-cpp; + pkgconfig = [ "opentelemetry_trace" ]; + } + { + # No pkg-config file. + prefix = "sentry.h"; + pkg = deps.sentry-native; + libs = [ "-lsentry" ]; + } + { + prefix = "microhttpd.h"; + pkg = deps.libmicrohttpd; + pkgconfig = [ "libmicrohttpd" ]; + } + { + # ``, ``, ... + prefix = "mimalloc"; + pkg = deps.mimalloc; + pkgconfig = [ "mimalloc" ]; + } + { + prefix = "lowdown.h"; + pkg = deps.lowdown; + pkgconfig = [ "lowdown" ]; + } + { + prefix = "editline.h"; + pkg = deps.editline; + pkgconfig = [ "libeditline" ]; + } + { + # Matches both and . + prefix = "git2"; + pkg = deps.libgit2; + pkgconfig = [ "libgit2" ]; + } + { + prefix = "wasmtime"; + pkg = [ + deps.wasmtime + deps.wasmtime.lib + ]; + libs = [ "-lwasmtime" ]; + } + # Unit test frameworks. + { + prefix = "gtest/"; + pkg = deps.gtest; + pkgconfig = [ "gtest" ]; + } + { + prefix = "gmock/"; + pkg = deps.gtest; + pkgconfig = [ "gmock" ]; + } + { + # Matches both and . rapidcheck.pc has + # an empty `Libs:`, so the library is named explicitly. + prefix = "rapidcheck"; + pkg = deps.rapidcheck; + pkgconfig = [ "rapidcheck" ]; + libs = [ "-lrapidcheck" ]; + } +] diff --git a/packaging/nix-make/flake.lock b/packaging/nix-make/flake.lock new file mode 100644 index 000000000000..1a988eba3a86 --- /dev/null +++ b/packaging/nix-make/flake.lock @@ -0,0 +1,181 @@ +{ + "nodes": { + "bdwgc": { + "flake": false, + "locked": { + "lastModified": 1790015987, + "narHash": "sha256-bPVosUgucJR0hGSap3tJW84AR6eGYegVROV+O8VYX5A=", + "owner": "DeterminateSystems", + "repo": "bdwgc", + "rev": "373957929b9ca3b2ae69a3e16b830c44c15b42ca", + "type": "github" + }, + "original": { + "owner": "DeterminateSystems", + "ref": "user-defined-stacks", + "repo": "bdwgc", + "type": "github" + } + }, + "flake-compat": { + "flake": false, + "locked": { + "lastModified": 1767039857, + "narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=", + "owner": "NixOS", + "repo": "flake-compat", + "rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab", + "type": "github" + }, + "original": { + "owner": "NixOS", + "repo": "flake-compat", + "type": "github" + } + }, + "flake-parts": { + "inputs": { + "nixpkgs-lib": [ + "nix", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1782949081, + "narHash": "sha256-vp6Y/Grm98ESt6ceOkWiHWyZRDV3J1RID4w+6NWK9yA=", + "rev": "17c9d6cdfc60c64f4ee8d306f9bc0b4ccb51481e", + "revCount": 480, + "type": "tarball", + "url": "https://api.flakehub.com/f/pinned/hercules-ci/flake-parts/0.1.480%2Brev-17c9d6cdfc60c64f4ee8d306f9bc0b4ccb51481e/019f2195-dee5-7233-9747-eca0c27f7406/source.tar.gz" + }, + "original": { + "type": "tarball", + "url": "https://flakehub.com/f/hercules-ci/flake-parts/0.1" + } + }, + "flake-schemas": { + "locked": { + "path": "../../src/libcmd/builtin-flake-schemas", + "type": "path" + }, + "original": { + "path": "../../src/libcmd/builtin-flake-schemas", + "type": "path" + }, + "parent": [] + }, + "git-hooks-nix": { + "inputs": { + "flake-compat": "flake-compat", + "nixpkgs": [ + "nix", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1784288435, + "narHash": "sha256-ReRHaLgr/uVqdD8afFSn+myXIfpHeOhP0yYe0TJqAA8=", + "rev": "43b3c1ab9d40fb1dbb008f451988a91e375825e9", + "revCount": 1231, + "type": "tarball", + "url": "https://api.flakehub.com/f/pinned/cachix/git-hooks.nix/0.1.1231%2Brev-43b3c1ab9d40fb1dbb008f451988a91e375825e9/019f7135-8fdf-76f0-b1a1-d2c67e91af8d/source.tar.gz" + }, + "original": { + "type": "tarball", + "url": "https://flakehub.com/f/cachix/git-hooks.nix/0.1.941" + } + }, + "nix": { + "inputs": { + "bdwgc": "bdwgc", + "flake-parts": "flake-parts", + "git-hooks-nix": "git-hooks-nix", + "nixpkgs": "nixpkgs", + "nixpkgs-23-11": "nixpkgs-23-11", + "nixpkgs-regression": "nixpkgs-regression" + }, + "locked": { + "path": "../..", + "type": "path" + }, + "original": { + "path": "../..", + "type": "path" + }, + "parent": [] + }, + "nix-wasm-module-make": { + "locked": { + "lastModified": 1790950568, + "narHash": "sha256-C4T1Q2D9c3JUisNhJCMHdCT75fPCpPak39N6P/brG7E=", + "rev": "b165eee8be4c5805a73e55a8541d27fce6a6df00", + "revCount": 123, + "type": "tarball", + "url": "https://api.flakehub.com/f/pinned/DeterminateSystems/nix-wasm-module-make/0.1.123%2Brev-b165eee8be4c5805a73e55a8541d27fce6a6df00/01a0fcf9-d934-7880-9901-53b2e75ee25b/source.tar.gz" + }, + "original": { + "type": "tarball", + "url": "https://flakehub.com/f/DeterminateSystems/nix-wasm-module-make/0" + } + }, + "nixpkgs": { + "locked": { + "lastModified": 1787962033, + "narHash": "sha256-u6z9VTZA4Kf3RkHQo9sQI7NI4Ei/uiU9vrMqOiwWP1Y=", + "rev": "c5c4a43b0e8056328ec4529f735cabdb8f1942bb", + "revCount": 1012930, + "type": "tarball", + "url": "https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.2605.1012930%2Brev-c5c4a43b0e8056328ec4529f735cabdb8f1942bb/01a053c0-7061-7382-b001-102e9148a5b5/source.tar.gz" + }, + "original": { + "type": "tarball", + "url": "https://flakehub.com/f/NixOS/nixpkgs/0.2605" + } + }, + "nixpkgs-23-11": { + "locked": { + "lastModified": 1717159533, + "narHash": "sha256-oamiKNfr2MS6yH64rUn99mIZjc45nGJlj9eGth/3Xuw=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "a62e6edd6d5e1fa0329b8653c801147986f8d446", + "type": "github" + }, + "original": { + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "a62e6edd6d5e1fa0329b8653c801147986f8d446", + "type": "github" + } + }, + "nixpkgs-regression": { + "locked": { + "lastModified": 1643052045, + "narHash": "sha256-uGJ0VXIhWKGXxkeNnq4TvV3CIOkUJ3PAoLZ3HMzNVMw=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2", + "type": "github" + }, + "original": { + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2", + "type": "github" + } + }, + "root": { + "inputs": { + "flake-schemas": "flake-schemas", + "nix": "nix", + "nix-wasm-module-make": "nix-wasm-module-make", + "nixpkgs": [ + "nix", + "nixpkgs" + ] + } + } + }, + "root": "root", + "version": 7 +} diff --git a/packaging/nix-make/flake.nix b/packaging/nix-make/flake.nix new file mode 100644 index 000000000000..290b83c48cd6 --- /dev/null +++ b/packaging/nix-make/flake.nix @@ -0,0 +1,178 @@ +# Proof of concept: building Nix with Nix as the C++ build system, with one +# derivation per compilation unit. The `#include` closure of each unit is +# computed at evaluation time by a scanner running under `builtins.wasm`. +{ + inputs.nix.url = "../.."; + inputs.nixpkgs.follows = "nix/nixpkgs"; + # The built-in flake schemas that ship with Nix. + inputs.flake-schemas.url = "../../src/libcmd/builtin-flake-schemas"; + inputs.nix-wasm-module-make.url = "https://flakehub.com/f/DeterminateSystems/nix-wasm-module-make/0"; + + outputs = + { + self, + nix, + nixpkgs, + nix-wasm-module-make, + flake-schemas, + }: + let + system = "x86_64-linux"; + + pkgs = import nixpkgs { + inherit system; + # Provides `nixDependencies2`, the same dependency overrides used by the Meson build. + overlays = [ nix.overlays.internal ]; + }; + + # Build all components with the configuration given by `cfg`, a module + # setting the options declared in config.nix. The result is a package + # scope (like `nixComponents` in packaging/components.nix), so it can be + # extended with `overrideScope`. + makeNixVariant = + cfg: + let + config = + (nixpkgs.lib.evalModules { + modules = [ + ./config.nix + cfg + ]; + }).config; + in + pkgs.lib.makeScope pkgs.newScope (self: { + nixMake = import ./lib.nix { inherit pkgs config nix-wasm-module-make; }; + + nix-util = self.callPackage ../../src/libutil/make.nix { }; + nix-store = self.callPackage ../../src/libstore/make.nix { }; + nix-fetchers = self.callPackage ../../src/libfetchers/make.nix { }; + nix-expr = self.callPackage ../../src/libexpr/make.nix { }; + nix-flake = self.callPackage ../../src/libflake/make.nix { }; + nix-main = self.callPackage ../../src/libmain/make.nix { }; + nix-cmd = self.callPackage ../../src/libcmd/make.nix { }; + nix = self.callPackage ../../src/nix/make.nix { }; + + # The unit tests: the test executables and derivations running them. + nix-util-c = self.callPackage ../../src/libutil-c/make.nix { }; + nix-util-test-support = self.callPackage ../../src/libutil-test-support/make.nix { }; + nix-util-tests = self.callPackage ../../src/libutil-tests/make.nix { }; + nix-util-tests-run = self.nix-util-tests.tests.run; + nix-util-tests-run-without-new-syscalls = self.nix-util-tests.tests.run-without-new-syscalls; + nix-store-c = self.callPackage ../../src/libstore-c/make.nix { }; + nix-store-test-support = self.callPackage ../../src/libstore-test-support/make.nix { }; + nix-store-tests = self.callPackage ../../src/libstore-tests/make.nix { }; + nix-store-tests-run = self.nix-store-tests.tests.run; + nix-fetchers-c = self.callPackage ../../src/libfetchers-c/make.nix { }; + nix-fetchers-tests = self.callPackage ../../src/libfetchers-tests/make.nix { }; + nix-fetchers-tests-run = self.nix-fetchers-tests.tests.run; + nix-expr-c = self.callPackage ../../src/libexpr-c/make.nix { }; + nix-expr-test-support = self.callPackage ../../src/libexpr-test-support/make.nix { }; + nix-expr-tests = self.callPackage ../../src/libexpr-tests/make.nix { }; + nix-expr-tests-run = self.nix-expr-tests.tests.run; + nix-flake-c = self.callPackage ../../src/libflake-c/make.nix { }; + nix-flake-tests = self.callPackage ../../src/libflake-tests/make.nix { }; + nix-flake-tests-run = self.nix-flake-tests.tests.run; + + test-runner = self.callPackage ../../tests/functional/test-runner.nix { }; + functional-tests = self.callPackage ../../tests/functional/make.nix { }; + + # Everything that runs tests; building it runs them all. + nix-all-tests = pkgs.linkFarmFromDrvs "nix-all-tests" [ + self.nix-util-tests-run + self.nix-util-tests-run-without-new-syscalls + self.nix-store-tests-run + self.nix-fetchers-tests-run + self.nix-expr-tests-run + self.nix-flake-tests-run + self.functional-tests + ]; + }); + in + rec { + packages.${system} = { + inherit (make.${system}.release) test-runner; + }; + + # `nix flake check` runs all tests against the release variant. + checks.${system} = { + inherit (make.${system}.release) nix-all-tests; + }; + + devShells.${system}.default = pkgs.mkShell { + packages = [ make.${system}.release.test-runner ]; + }; + + # The build variants, named after Meson's build types. + make.${system} = { + release = makeNixVariant { + optimize = true; + debug = false; + }; + debugoptimized = makeNixVariant { + optimize = true; + debug = true; + }; + debug = makeNixVariant { + optimize = false; + debug = true; + }; + # Like `release`, but without the Boehm garbage collector. + nogc = makeNixVariant { + optimize = true; + debug = false; + boehmgc = false; + }; + # Like `debug`, but built with clang, which compiles faster. + debug-fast = makeNixVariant { + optimize = false; + debug = true; + compiler = "clang"; + }; + }; + + # Describe the `make` output to `nix flake show` and `nix flake check`. + # Defining `schemas` replaces the built-in ones, so re-export the ones + # for the other outputs of this flake. + schemas = { + inherit (flake-schemas.schemas) + schemas + packages + checks + devShells + ; + + make = { + version = 1; + doc = '' + The `make` output provides the Nix components (libraries and the + `nix` executable) built with Nix as the build system, per system + and build variant (`release`, `debugoptimized`, `debug`, `nogc`, + `debug-fast`). + ''; + roles.nix-build = { }; + appendSystem = true; + defaultAttrPath = [ + "release" + "nix" + ]; + inventory = + output: + flake-schemas.lib.mkChildren ( + builtins.mapAttrs (system: variants: { + forSystems = [ system ]; + children = builtins.mapAttrs (variant: components: { + forSystems = [ system ]; + shortDescription = "The `${variant}` build variant"; + # The components; a scope also has helper attributes. + children = builtins.mapAttrs (name: package: { + what = "package"; + forSystems = [ system ]; + derivationAttrPath = [ ]; + }) (nixpkgs.lib.filterAttrs (_: nixpkgs.lib.isDerivation) components); + }) variants; + }) output + ); + }; + }; + }; +} diff --git a/packaging/nix-make/lib.nix b/packaging/nix-make/lib.nix new file mode 100644 index 000000000000..36928eba2377 --- /dev/null +++ b/packaging/nix-make/lib.nix @@ -0,0 +1,519 @@ +# Generic machinery for building C++ components with one derivation per +# compilation unit. +{ + pkgs, + config, + nix-wasm-module-make, +}: + +let + inherit (pkgs) lib; + + getDeps = builtins.wasm { + path = nix-wasm-module-make.nixWasmModules.nix_make; + function = "getDeps"; + }; + + externalDeps = import ./deps.nix { inherit pkgs; }; + + # The version, as in the package.nix files. + defaultVersion = lib.fileContents ../../.version-determinate; + + # The stdenv providing the compiler. `clangStdenv` uses libstdc++, so the + # dependencies built by Nixpkgs with GCC can be used as is. + stdenv = if config.compiler == "clang" then pkgs.clangStdenv else pkgs.stdenv; + + # `builtins.parallel xs x` starts evaluating the values `xs` on other + # threads and returns `x`. It requires the `parallel-eval` experimental + # feature (and `eval-cores`); without it, evaluation is sequential. + parallel = builtins.parallel or (xs: x: x); + + # Optimization and debug flags, following Meson's build types: `release` + # is `-O3`, `debugoptimized` is `-O2 -g`, and `debug` is `-O0 -g`. + optimizationFlags = + if config.optimize then [ (if config.debug then "-O2" else "-O3") ] else [ "-O0" ]; + debugFlags = lib.optional config.debug "-g"; + + # From nix-meson-build-support/common/meson.build, where Meson keeps the + # ones the compiler supports; the last few are compiler-specific. + warningFlags = [ + "-Wall" + "-Wdeprecated-copy" + "-Werror=suggest-override" + "-Werror=switch" + "-Werror=switch-enum" + "-Werror=undef" + "-Werror=unused-result" + "-Werror=sign-compare" + "-Werror=return-type" + "-Werror=non-virtual-dtor" + "-Wignored-qualifiers" + "-Wimplicit-fallthrough" + "-Wno-deprecated-declarations" + ] + ++ ( + if config.compiler == "clang" then + [ "-Werror=c99-designator" ] + else + [ + "-Wno-interference-size" + "-Wno-subobject-linkage" + ] + ); + + # Meson enables this for libutil, libstore and libexpr only: all vtables + # must have a "key" function so they are emitted as strong symbols, which + # matters for dynamic linking on Darwin. Clang only. + weakVtablesFlags = lib.optional (config.compiler == "clang") "-Werror=weak-vtables"; + + commonCxxFlags = [ + "-std=c++23" + "-fPIC" + "-pthread" + "-D_FILE_OFFSET_BITS=64" + "-D_GLIBCXX_USE_TBB_PAR_BACKEND=0" + "-fno-semantic-interposition" + ] + ++ optimizationFlags + ++ debugFlags + ++ warningFlags; + + commonLinkLibs = [ + "-pthread" + "-ldl" + "-lrt" + ]; + + # A stand-in for a configuration header (like Meson's `configure_file()` + # output). It is empty: the macros are passed to the compiler as `-D` + # flags instead, and only to the units that use them, so that changing a + # macro does not rebuild every unit that includes the header. + mkStubHeader = + name: + builtins.toFile name "#pragma once\n// The macros of this configuration header are passed to the compiler as -D flags.\n"; + + # Wrap a file in a C++ raw string literal, like the `gen_header` + # generator in nix-meson-build-support/generate-header. + mkStringHeader = + path: + builtins.toFile "${baseNameOf path}.gen.hh" "R\"__NIX_STR(\n${builtins.readFile path})__NIX_STR\"\n"; + + # The `dev` outputs, as `stdenv.mkDerivation` would pick for `buildInputs`. + depPackages = deps: map lib.getDev (lib.concatMap (d: lib.toList d.pkg) deps); + + # A minimal derivation running a Python build script, which (unlike + # bash) can read the typed JSON of the structured attributes + # (`$NIX_ATTRS_JSON_FILE`). A bash prelude sources the stdenv setup + # script first, which provides the C++ compiler and handles + # `buildInputs` etc. This avoids the evaluation cost of + # `stdenv.mkDerivation`, which is significant for hundreds of + # derivations, and unlike `runCommand` it does not set + # `preferLocalBuild`, so units can be built remotely. + mkPythonDerivation = + attrs: script: + derivation ( + { + system = stdenv.hostPlatform.system; + builder = stdenv.shell; + args = [ + "-e" + (builtins.toFile "builder.sh" '' + # With structured attrs, attributes are not in the environment. + if [ -e "$NIX_ATTRS_SH_FILE" ]; then . "$NIX_ATTRS_SH_FILE"; fi + # The setup script prints "structuredAttrs is enabled" on stdout. + source $stdenv/setup >/dev/null + exec python3 ${script} + '') + ]; + inherit stdenv; + __structuredAttrs = true; + } + // attrs + // { + nativeBuildInputs = attrs.nativeBuildInputs or [ ] ++ [ pkgs.python3 ]; + } + ); + + # A derivation that runs a test program (e.g. a gtest executable) with + # `env` added to the environment, and succeeds if it does. Its output is + # empty; the program's output is in the build log. `HOME` is a writable + # directory, which the values in `env` may refer to as `$HOME`. + runTest = + { + name, + command, + env ? { }, + nativeBuildInputs ? [ ], + }: + mkPythonDerivation { + inherit + name + command + env + nativeBuildInputs + ; + } ./run-test.py; + + # The stdenv's default hardening flags, except that `_FORTIFY_SOURCE` + # warns on every unit when not optimizing, so it is disabled then (like + # `hardeningDisable = [ "fortify" ]` in `stdenv.mkDerivation`). + hardeningFlags = lib.concatStringsSep " " ( + lib.subtractLists (lib.optionals (!config.optimize) [ + "fortify" + "fortify3" + ]) stdenv.cc.defaultHardeningFlags + ); + + # Deduplicate strings in O(n log n) rather than `lib.unique`'s O(n^2). + uniqueStrings = xs: lib.attrNames (lib.listToAttrs (map (x: lib.nameValuePair x null) xs)); + + # Deduplicate a list by a key, keeping the first occurrence and the order. + # Note: `lib.unique` on derivations would compare them by output path, + # which instantiates them and all their inputs. + uniqueBy = + key: xs: + map (x: x.value) ( + builtins.genericClosure { + startSet = map (x: { + key = key x; + value = x; + }) xs; + operator = _: [ ]; + } + ); + + compileUnit = + component: unit: + let + deps = component.externalDepsFor unit.externalIncludes; + in + mkPythonDerivation { + name = "${baseNameOf unit.path}.o"; + # The config macros this unit is sensitive to (see `configHeaders`), + # as reported by the scanner; `null` means undefined. The builder + # turns them into -D/-U flags. + defines = unit.usedDefines; + # Generated files come from the output of their generator derivation. + src = if unit.src == null then component.allGenerated.${unit.path}.path else unit.src; + includes = + unit.includes + // lib.listToAttrs ( + map (g: lib.nameValuePair g component.allGenerated.${g}.path) unit.generatedIncludes + ); + srcPath = unit.path; + includeDirs = component.includeDirs ++ component.depIncludeDirs; + cxxFlags = + commonCxxFlags + ++ component.extraCxxFlags + ++ (component.unitCxxFlags.${unit.path} or [ ]) + ++ lib.concatMap (d: d.cflags or [ ]) deps; + pkgConfigDeps = lib.concatMap (d: d.pkgconfig or [ ]) deps; + buildInputs = depPackages deps; + nativeBuildInputs = [ pkgs.pkg-config ]; + NIX_HARDENING_ENABLE = hardeningFlags; + } ./compile.py; + + # Link the objects into a shared library or an executable. + link = + component: objects: + let + # Link against every external dependency used by any unit. + deps = component.externalDepsFor component.allExternalIncludes; + in + mkPythonDerivation { + name = "${component.name}-${component.version}"; + # Instantiate the objects and the dependencies in parallel. + objects = parallel (map (d: d.drvPath) component.allDeps ++ map (o: o.drvPath) objects) objects; + inherit (component) + type + libName + exeName + binSymlinks + linkFlags + postInstall + ; + linkPkgConfig = lib.concatMap (d: d.pkgconfig or [ ]) deps; + linkLibs = + map (d: "-l${d.libName}") component.allDeps + ++ lib.concatMap (d: d.libs or [ ]) deps + ++ component.extraLinkLibs + ++ commonLinkLibs; + buildInputs = component.allDeps ++ depPackages deps; + nativeBuildInputs = [ pkgs.pkg-config ]; + } ./link.py + // { + inherit objects; + inherit (component) units; + inherit component; + }; + + /** + Build a component: a shared library (the default) or an executable. + + - `name`: component name (e.g. `determinate-nix-util`); the derivation is + named `-`. + - `version`: defaults to the contents of `.version-determinate`. + - `type`: `"library"` or `"executable"`. + - `libName`: library name without `lib` prefix (e.g. `nixutil`). + - `exeName`: executable name; defaults to `name`. + - `binSymlinks`: names of symlinks to the executable to create in `bin/`. + - `postInstall`: Python snippet run after linking, with `out` (and `os`) bound. + - `deps`: other components this one depends on. Their public headers + are made available under `_deps//` and their libraries are linked. + Dependencies are transitive. + - `root`: the directory scanned for sources and headers; shorthand for + a `roots` entry with an empty prefix. + - `roots`: further `{ root; prefix; }` directories scanned for sources + and headers, known by `prefix/`. + - `includeDirs`: include search path, relative to the root namespace. + - `publicIncludeDirs`: the subset of `includeDirs` exported to + dependent components. Defaults to all but the root. + - `sourceExtensions`: suffixes identifying compilation units. + - `sources`: explicit list of compilation units, as paths in the root + namespace. By default, every file in the roots or `files` with a + source extension is a compilation unit. + - `excludeSources`: paths (files, or directories with everything below + them) to leave out of the compilation units. + - `files`: extra files (e.g. generated headers) by path in the root namespace. + - `configHeaders`: generated config headers, as an attribute set from + path in the root namespace to the `#define`s. The header itself is an + empty stub; each unit gets the macros it uses, and the builder + (compile.py) turns them into `-D` flags. Their macros are also used + to evaluate preprocessor conditionals. + - `defines`, `undefines`: further macros known to be defined (with + value) or undefined when evaluating preprocessor conditionals, on + top of the platform defaults and the config headers. + - `generated`: files generated at build time (e.g. by bison), as an + attribute set from path in the root namespace to `{ from; path; }`: + `from` is the real file whose `#include`s it inherits (so that it + can be scanned without building it), and `path` is the generated + file in the output of a derivation. + - `extraCxxFlags`, `linkFlags`, `extraLinkLibs`: what they say. + - `unitCxxFlags`: extra compiler flags for specific units, as an + attribute set from path in the root namespace to a list of flags. + + External dependencies (compile flags and libraries) are derived from + the `#include`s of the units, via `deps.nix`. + */ + mkComponent = + { + name, + version ? defaultVersion, + type ? "library", + libName ? null, + exeName ? name, + binSymlinks ? [ ], + postInstall ? "", + deps ? [ ], + root ? null, + roots ? [ ], + includeDirs, + publicIncludeDirs ? lib.filter (d: d != "") includeDirs, + sourceExtensions ? [ + ".cc" + ".cpp" + ".cxx" + ".c" + ], + sources ? null, + excludeSources ? [ ], + files ? { }, + configHeaders ? { }, + generated ? { }, + defines ? { }, + undefines ? [ ], + extraCxxFlags ? [ ], + unitCxxFlags ? { }, + linkFlags ? [ ], + extraLinkLibs ? [ ], + }@args: + let + allRoots = + lib.optional (root != null) { + inherit root; + prefix = ""; + } + ++ roots; + + # The transitive closure of the dependencies, direct ones first. + allDeps = + let + item = d: { + key = d.component.name; + value = d; + }; + in + map (x: x.value) ( + builtins.genericClosure { + startSet = map item deps; + operator = x: map item x.value.component.deps; + } + ); + + # Dependencies contribute their roots, public include directories and + # extra files under `_deps//` (a prefix that cannot collide with + # the component's own directories), but no compilation units. + depPrefix = d: "_deps/${d.component.name}"; + prefixed = d: p: if p == "" then depPrefix d else "${depPrefix d}/${p}"; + depRoots = lib.concatMap ( + d: + map (r: { + inherit (r) root; + prefix = prefixed d r.prefix; + }) d.component.roots + ) allDeps; + depIncludeDirs = lib.concatMap (d: map (prefixed d) d.component.publicIncludeDirs) allDeps; + depFiles = lib.foldl' ( + acc: d: acc // lib.mapAttrs' (k: v: lib.nameValuePair (prefixed d k) v) d.component.files + ) { } allDeps; + depExcludes = map depPrefix allDeps; + depGenerated = lib.foldl' ( + acc: d: + acc + // lib.mapAttrs' ( + k: g: lib.nameValuePair (prefixed d k) (g // { from = prefixed d g.from; }) + ) d.component.generated + ) { } allDeps; + allGenerated = generated // depGenerated; + + # Generated config headers, whose macros are also known to the scanner. + configFiles = lib.mapAttrs (path: _: mkStubHeader (baseNameOf path)) configHeaders; + configDefines = lib.foldl' (acc: attrs: acc // attrs) { } (lib.attrValues configHeaders); + allFiles = files // configFiles; + + # The macros a unit may be sensitive to: the dependencies' config + # macros and our own. Each unit is passed only the ones it uses. + sensitiveDefines = + lib.foldl' (acc: d: acc // d.component.sensitiveDefines) { } allDeps // configDefines; + + # Macros for evaluating conditionals: platform defaults, then the + # dependencies' config macros, then our own. `null` means undefined. + allDefines = + lib.foldl' (acc: d: acc // d.component.allDefines) platformDefines allDeps + // configDefines + // defines; + knownDefines = lib.mapAttrs ( + _: v: if builtins.isBool v then (if v then "1" else "0") else toString v + ) (lib.filterAttrs (_: v: v != null) allDefines); + knownUndefines = lib.unique ( + platformUndefines ++ undefines ++ lib.attrNames (lib.filterAttrs (_: v: v == null) allDefines) + ); + + # Scan this component in parallel with its dependencies. + units = parallel (map (d: d.units) deps) ( + getDeps ( + { + inherit builtins; + inherit sourceExtensions; + roots = allRoots ++ depRoots; + includeDirs = includeDirs ++ depIncludeDirs; + excludeSources = excludeSources ++ depExcludes; + # Work around a crash in `builtins.wasm` (Nix <= 3.22.5) when + # copying "layered" attribute sets (the result of `//`) into Wasm: + # `mapAttrs` produces a fresh, non-layered attribute set. + files = lib.mapAttrs (_: v: v) (allFiles // depFiles); + defines = lib.mapAttrs (_: v: v) knownDefines; + undefines = knownUndefines; + # The scanner reports, per unit, which of these occur in its + # include closure, with their values. + trackedDefines = lib.mapAttrs (_: v: v) sensitiveDefines; + generated = lib.mapAttrs (_: g: g.from) allGenerated; + } + // lib.optionalAttrs (sources != null) { inherit sources; } + ) + ); + + # Every external include of any unit, and a map from external include to + # its `externalDeps` entry, computed once per component. `externalDepsFor` + # then maps a unit's external includes to its dependencies. + allExternalIncludes = uniqueStrings (lib.concatMap (u: u.externalIncludes) units); + externalDepByInclude = lib.listToAttrs ( + map ( + inc: lib.nameValuePair inc (lib.findFirst (d: lib.hasPrefix d.prefix inc) null externalDeps) + ) allExternalIncludes + ); + externalDepsFor = + includes: + uniqueBy (d: d.prefix) ( + lib.filter (d: d != null) (map (inc: externalDepByInclude.${inc}) includes) + ); + + component = args // { + inherit allExternalIncludes externalDepsFor; + inherit + version + type + libName + exeName + binSymlinks + postInstall + ; + roots = allRoots; + # Only our own files and generated files are exported to dependents; + # their own dependencies are resolved transitively. + files = allFiles; + inherit generated allGenerated; + inherit + deps + units + allDeps + allDefines + sensitiveDefines + depIncludeDirs + publicIncludeDirs + extraCxxFlags + unitCxxFlags + linkFlags + extraLinkLibs + ; + }; + in + link component (map (compileUnit component) units); + + # Macros the scanner can rely on when evaluating preprocessor conditionals + # (Linux, x86_64, GCC). Only what is certain: an unknown macro keeps both + # branches of a conditional, but a wrong one would drop includes. + platformDefines = { + __linux__ = 1; + __gnu_linux__ = 1; + __unix__ = 1; + __unix = 1; + __x86_64__ = 1; + __LP64__ = 1; + }; + platformUndefines = [ + "_WIN32" + "_WIN64" + "__CYGWIN__" + "__MINGW32__" + "__APPLE__" + "__MACH__" + "__FreeBSD__" + "__NetBSD__" + "__OpenBSD__" + "__DragonFly__" + "__sun" + "__clang__" + ]; + + # Sources from nix-meson-build-support that Meson links into every component. + commonSupportFiles = { + "nix-meson-build-support/common/assert-fail/wrap-assert-fail.cc" = + ../../nix-meson-build-support/common/assert-fail/wrap-assert-fail.cc; + }; + +in +{ + # The build configuration (see config.nix), for component files that + # need to vary with it. + inherit config; + + inherit + getDeps + weakVtablesFlags + mkStringHeader + mkComponent + runTest + commonSupportFiles + ; +} diff --git a/packaging/nix-make/link.py b/packaging/nix-make/link.py new file mode 100644 index 000000000000..6ce46d696b2a --- /dev/null +++ b/packaging/nix-make/link.py @@ -0,0 +1,55 @@ +# The builder for linking a component's objects into a shared library or +# an executable. It reads the derivation's structured attributes (see +# `link` in lib.nix). The stdenv setup script has been sourced by the bash +# prelude in lib.nix, so the compiler driver (`$CXX`), the library paths +# of the build inputs and pkg-config are set up. + +import json +import os +import subprocess +import sys + + +def main(): + with open(os.environ["NIX_ATTRS_JSON_FILE"]) as f: + attrs = json.load(f) + + cxx = os.environ.get("CXX") + if not cxx: + sys.exit("CXX is not set; was the stdenv setup script sourced?") + + out = os.environ["out"] + + pkg_config_libs = [] + if attrs["linkPkgConfig"]: + pkg_config_libs = subprocess.run( + ["pkg-config", "--libs", *attrs["linkPkgConfig"]], + check=True, + capture_output=True, + text=True, + ).stdout.split() + + common = ["-Wl,--as-needed", "-Wl,--no-undefined", *attrs["linkFlags"]] + inputs = [*attrs["objects"], *pkg_config_libs, *attrs["linkLibs"]] + + if attrs["type"] == "library": + lib = f"lib{attrs['libName']}.so" + os.makedirs(f"{out}/lib") + subprocess.run( + [cxx, "-shared", "-fPIC", f"-Wl,-soname,{lib}", *common, "-o", f"{out}/lib/{lib}", *inputs], + check=True, + ) + elif attrs["type"] == "executable": + os.makedirs(f"{out}/bin") + subprocess.run([cxx, *common, "-o", f"{out}/bin/{attrs['exeName']}", *inputs], check=True) + for name in attrs["binSymlinks"]: + os.symlink(attrs["exeName"], f"{out}/bin/{name}") + else: + sys.exit(f"unknown component type: {attrs['type']}") + + # A Python snippet from the component description, with `out` bound. + if attrs["postInstall"]: + exec(attrs["postInstall"], {"out": out, "os": os}) + + +main() diff --git a/packaging/nix-make/run-test.py b/packaging/nix-make/run-test.py new file mode 100644 index 000000000000..0f65f5aa919f --- /dev/null +++ b/packaging/nix-make/run-test.py @@ -0,0 +1,26 @@ +# The builder for running a test program (see `runTest` in lib.nix). It +# runs `command` with `env` added to the environment; the output of the +# program goes to the build log. + +import json +import os +import subprocess + + +def main(): + with open(os.environ["NIX_ATTRS_JSON_FILE"]) as f: + attrs = json.load(f) + + # A writable home directory, for tests that keep state (e.g. a store) + # there. The values in `env` may refer to it as `$HOME`. + os.environ["HOME"] = os.path.join(os.environ["TMPDIR"], "home") + os.mkdir(os.environ["HOME"]) + env = dict(os.environ) + for name, value in attrs["env"].items(): + env[name] = os.path.expandvars(value) + + subprocess.run(attrs["command"], env=env, check=True) + os.mkdir(os.environ["out"]) + + +main() diff --git a/packaging/patches/0001-Fix-uncaught_exceptions-not-accounting-for-forced_un.patch b/packaging/patches/0001-Fix-uncaught_exceptions-not-accounting-for-forced_un.patch deleted file mode 100644 index 7ec13724c40b..000000000000 --- a/packaging/patches/0001-Fix-uncaught_exceptions-not-accounting-for-forced_un.patch +++ /dev/null @@ -1,102 +0,0 @@ -From 5883212311535a0046031d74d1568ae173c1e35b Mon Sep 17 00:00:00 2001 -From: Sergei Zimmerman -Date: Tue, 21 Jul 2026 21:15:51 +0000 -Subject: [PATCH] Fix uncaught_exceptions() not accounting for forced_unwind - -Unwound fibers would see std::uncaught_exceptions() == 0, while a -forced_unwind exception is in "flight". This goes against the contract -of std::uncaught_exceptions() that scope guards rely upon. Failing -to report the correct number of uncaught exceptions (especially -misreporting zero) will lead to scope guards to misbehave badly and skip -running cleanup code which branches on whether the destructor is called -during stack unwinding or not. - -This is because the "throw" would happen before the destructor is run on -the fiber stack being switched to, but the increment would be clobbered -by the destructor of manage_exception_state. - -I'm not sure what the contract of run ontop_fcontext is wrt to whether -the the caller provided function can throw or not, but in my best -understanding the forced_unwind mechanism is mostly internal and so is -throwing from ontop_fcontext in the switched-to fiber. Thus, I've kept -the catch block scoped to detail::forced_unwind. ---- - include/boost/context/fiber_fcontext.hpp | 37 +++++++++++++++++------- - test/test_fiber.cpp | 24 +++++++++++++++ - 2 files changed, 51 insertions(+), 10 deletions(-) - -diff --git a/include/boost/context/fiber_fcontext.hpp b/include/boost/context/fiber_fcontext.hpp -index 543ba6c..38476c9 100644 ---- a/boost/context/fiber_fcontext.hpp -+++ b/boost/context/fiber_fcontext.hpp -@@ -70,7 +70,9 @@ namespace context { - namespace detail { - - // manage_exception_state is a dummy struct unless we have specific support --struct manage_exception_state {}; -+struct manage_exception_state { -+ void from_forced_unwind() noexcept {} -+}; - - } // namespace detail - } // namespace context -@@ -90,6 +92,11 @@ public: - manage_exception_state() { - exception_state_ = *__cxa_get_globals(); - } -+ // Hack to account for the forced_unwind exception thrown in fiber_unwind -+ // that's run ontop before the destructor. -+ void from_forced_unwind() noexcept { -+ exception_state_.uncaughtExceptions += 1; -+ } - ~manage_exception_state() { - *__cxa_get_globals() = exception_state_; - } -@@ -376,13 +383,18 @@ public: - BOOST_ASSERT( nullptr != fctx_); - detail::manage_exception_state exstate; - boost::ignore_unused(exstate); -- return { detail::jump_fcontext( -+ try { -+ return { detail::jump_fcontext( - #if defined(BOOST_NO_CXX14_STD_EXCHANGE) -- detail::exchange( fctx_, nullptr), -+ detail::exchange( fctx_, nullptr), - #else -- std::exchange( fctx_, nullptr), -+ std::exchange( fctx_, nullptr), - #endif -- nullptr).fctx }; -+ nullptr).fctx }; -+ } catch ( detail::forced_unwind const& ) { -+ exstate.from_forced_unwind(); -+ throw; -+ } - } - - template< typename Fn > -@@ -391,14 +403,19 @@ public: - detail::manage_exception_state exstate; - boost::ignore_unused(exstate); - auto p = std::forward< Fn >( fn); -- return { detail::ontop_fcontext( -+ try { -+ return { detail::ontop_fcontext( - #if defined(BOOST_NO_CXX14_STD_EXCHANGE) -- detail::exchange( fctx_, nullptr), -+ detail::exchange( fctx_, nullptr), - #else -- std::exchange( fctx_, nullptr), -+ std::exchange( fctx_, nullptr), - #endif -- & p, -- detail::fiber_ontop< fiber, decltype(p) >).fctx }; -+ & p, -+ detail::fiber_ontop< fiber, decltype(p) >).fctx }; -+ } catch ( detail::forced_unwind const& ) { -+ exstate.from_forced_unwind(); -+ throw; -+ } - } - - explicit operator bool() const noexcept { diff --git a/packaging/patches/0002-memory-config.patch b/packaging/patches/0002-memory-config.patch new file mode 100644 index 000000000000..ce1ff61acab3 --- /dev/null +++ b/packaging/patches/0002-memory-config.patch @@ -0,0 +1,22 @@ +diff --git a/include/git2/sys/config.h b/include/git2/sys/config.h +index dcce18e6f..8945a3285 100644 +--- a/include/git2/sys/config.h ++++ b/include/git2/sys/config.h +@@ -200,7 +200,7 @@ GIT_EXTERN(int) git_config_backend_memory_options_init( + * @param opts the options to initialize this backend with, or NULL + * @return 0 on success or an error code + */ +-extern int git_config_backend_from_string( ++GIT_EXTERN(int) git_config_backend_from_string( + git_config_backend **out, + const char *cfg, + size_t len, +@@ -216,7 +216,7 @@ extern int git_config_backend_from_string( + * @param opts the options to initialize this backend with, or NULL + * @return 0 on success or an error code + */ +-extern int git_config_backend_from_values( ++GIT_EXTERN(int) git_config_backend_from_values( + git_config_backend **out, + const char **values, + size_t len, diff --git a/packaging/patches/0003-packbuilder-correct-config.patch b/packaging/patches/0003-packbuilder-correct-config.patch new file mode 100644 index 000000000000..8d1ab47920f9 --- /dev/null +++ b/packaging/patches/0003-packbuilder-correct-config.patch @@ -0,0 +1,45 @@ +diff --git a/src/libgit2/pack-objects.c b/src/libgit2/pack-objects.c +index 0da84b657..d540795f9 100644 +--- a/src/libgit2/pack-objects.c ++++ b/src/libgit2/pack-objects.c +@@ -115,8 +115,9 @@ static int packbuilder_config(git_packbuilder *pb) + GIT_PACK_DELTA_CACHE_SIZE); + config_get("pack.deltaCacheLimit", pb->cache_max_small_delta_size, + GIT_PACK_DELTA_CACHE_LIMIT); +- config_get("pack.deltaCacheSize", pb->big_file_threshold, ++ config_get("core.bigFileThreshold", pb->big_file_threshold, + GIT_PACK_BIG_FILE_THRESHOLD); ++ config_get("pack.window", pb->window_size, GIT_PACK_WINDOW); + config_get("pack.windowMemory", pb->window_memory_limit, 0); + + #undef config_get +@@ -1341,7 +1342,7 @@ int git_packbuilder__prepare(git_packbuilder *pb) + size_t i, n = 0; + int error; + +- if (pb->nr_objects == 0 || pb->done) ++ if (pb->nr_objects == 0 || pb->done || pb->window_size == 0) + return 0; /* nothing to do */ + + /* +@@ -1369,7 +1370,7 @@ int git_packbuilder__prepare(git_packbuilder *pb) + if (n > 1) { + git__tsort((void **)delta_list, n, type_size_sort); + if ((error = ll_find_deltas(pb, delta_list, n, +- GIT_PACK_WINDOW + 1, ++ pb->window_size + 1, + GIT_PACK_DEPTH)) < 0) { + git__free(delta_list); + return error; +diff --git a/src/libgit2/pack-objects.h b/src/libgit2/pack-objects.h +index ad04fb0ab..d9e03df60 100644 +--- a/src/libgit2/pack-objects.h ++++ b/src/libgit2/pack-objects.h +@@ -94,6 +94,7 @@ struct git_packbuilder { + size_t cache_max_small_delta_size; + size_t big_file_threshold; + size_t window_memory_limit; ++ size_t window_size; + + unsigned int nr_threads; /* nr of threads to use */ + diff --git a/packaging/patches/libgit2-packbuilder-dont-fail-on-thread-create-error.patch b/packaging/patches/libgit2-packbuilder-dont-fail-on-thread-create-error.patch new file mode 100644 index 000000000000..8cef53ce8dbb --- /dev/null +++ b/packaging/patches/libgit2-packbuilder-dont-fail-on-thread-create-error.patch @@ -0,0 +1,133 @@ +Don't abort pack building when worker thread creation fails. + +Previously, if `git_thread_create` failed in `ll_find_deltas` (e.g. with +EAGAIN due to transient resource exhaustion), it returned an error while +the already-created worker threads were still running. The caller +(`git_packbuilder_write_buf`) would then propagate the error and the +packbuilder would typically be freed, causing the orphaned workers to +crash on freed memory (observed as SIGSEGV in `git_odb_read` → +`git_cache_get_raw` → `pthread_rwlock_rdlock`). + +Instead, continue with however many threads were successfully created. +The work-stealing loop redistributes the work of partitions that have no +thread (`started == false`) to the running threads; such partitions are +stolen from regardless of size, and in their entirety, so every object +is still searched for deltas. If no threads could be created at all, +fall back to the single-threaded delta search. + +Also turn the per-thread mutex lock failure in the work-stealing loop +into a GIT_ASSERT, consistent with the neighboring progress lock/unlock +checks. In particular it must not free the `thread_params` array, since +running worker threads still reference it. + +Upstream: https://github.com/libgit2/libgit2 (present on main as of 2026-07-16) +--- a/src/libgit2/pack-objects.c ++++ b/src/libgit2/pack-objects.c +@@ -1137,6 +1137,9 @@ + /* A pb->progress_cb can stop the packing process by returning an error. + When that happens, all threads observe the error and stop voluntarily. */ + bool stopped; ++ ++ /* Whether the worker thread for this partition was created. */ ++ bool started; + }; + + static void *threaded_find_deltas(void *arg) +@@ -1186,6 +1189,8 @@ + size_t list_size, size_t window, size_t depth) + { + struct thread_params *p; ++ git_pobject **whole_list = list; ++ size_t whole_list_size = list_size; + size_t i; + int ret, active_threads = 0; + +@@ -1213,6 +1218,7 @@ + p[i].working = 1; + p[i].data_ready = 0; + p[i].stopped = 0; ++ p[i].started = 0; + + /* try to split chunks on "path" boundaries */ + while (sub_size && sub_size < list_size && +@@ -1239,13 +1245,37 @@ + ret = git_thread_create(&p[i].thread, + threaded_find_deltas, &p[i]); + if (ret) { +- git_error_set(GIT_ERROR_THREAD, "unable to create thread"); +- return -1; ++ /* ++ * Thread creation may fail due to transient resource ++ * exhaustion (e.g. EAGAIN). Proceed with the threads ++ * created so far: this partition keeps `working` set, ++ * so it is never picked as a work-stealing target, and ++ * the loop below hands all of its work to the running ++ * threads (`started` remains false). ++ * ++ * Returning an error here instead would leave the ++ * already-created threads running while the caller ++ * frees the packbuilder, resulting in a use-after-free. ++ */ ++ git_cond_free(&p[i].cond); ++ git_mutex_free(&p[i].mutex); ++ continue; + } ++ p[i].started = 1; + active_threads++; + } + + /* ++ * If no worker threads could be created, fall back to doing the ++ * delta search inline. No thread has consumed any work at this ++ * point, so the whole object list is still intact. ++ */ ++ if (!active_threads) { ++ git__free(p); ++ return find_deltas(pb, whole_list, &whole_list_size, window, depth); ++ } ++ ++ /* + * Now let's wait for work completion. Each time a thread is done + * with its work, we steal half of the remaining work from the + * thread with the largest number of unprocessed objects and give +@@ -1274,16 +1304,23 @@ + + /* At this point we hold the progress lock and have located + * a thread to receive more work. We still need to locate a +- * thread from which to steal work (the victim). */ ++ * thread from which to steal work (the victim). A partition ++ * whose thread could not be created has nothing consuming ++ * it, so it may be stolen from regardless of its size. */ + for (i = 0; i < pb->nr_threads; i++) +- if (p[i].remaining > 2*window && ++ if (p[i].remaining > (p[i].started ? 2*window : 0) && + (!victim || victim->remaining < p[i].remaining)) + victim = &p[i]; + + if (victim && !target->stopped) { +- sub_size = victim->remaining / 2; ++ /* Steal half of the victim's work, or all of it if ++ * the victim has no thread consuming it. In the ++ * latter case there is also no point in splitting on ++ * a "path" boundary, since the partition is not ++ * shared with another thread. */ ++ sub_size = victim->remaining / (victim->started ? 2 : 1); + list = victim->list + victim->list_size - sub_size; +- while (sub_size && list[0]->hash && ++ while (victim->started && sub_size && list[0]->hash && + list[0]->hash == list[-1]->hash) { + list++; + sub_size--; +@@ -1307,11 +1344,7 @@ + target->working = 1; /* even when target->stopped, so that we don't process this thread again */ + GIT_ASSERT(git_packbuilder__progress_unlock(pb) == 0); + +- if (git_mutex_lock(&target->mutex)) { +- git_error_set(GIT_ERROR_THREAD, "unable to lock packfile condition mutex"); +- git__free(p); +- return -1; +- } ++ GIT_ASSERT(git_mutex_lock(&target->mutex) == 0); + + target->data_ready = 1; + git_cond_signal(&target->cond); diff --git a/packaging/release-jobs.nix b/packaging/release-jobs.nix index f82c23c9c1ac..0d89b33714f2 100644 --- a/packaging/release-jobs.nix +++ b/packaging/release-jobs.nix @@ -41,7 +41,6 @@ let # Only the cross targets that end up in `fallback-paths.nix`. inherit (hydraJobs.buildCross.nix-everything) riscv64-unknown-linux-gnu - x86_64-unknown-freebsd ; }; diff --git a/packaging/rust-installer/tarball.nix b/packaging/rust-installer/tarball.nix index 4f236e0215bb..ba103c77695e 100644 --- a/packaging/rust-installer/tarball.nix +++ b/packaging/rust-installer/tarball.nix @@ -36,10 +36,11 @@ runCommand "nix-installer-tarball-${nix.version}" cp ${installerClosureInfo}/registration $TMPDIR/reginfo + # Store mtime is 1 second into the epoch. tar cf - \ --sort=name \ --owner=0 --group=0 --mode=u+rw,uga+r \ - --mtime='1970-01-01' \ + --mtime='@1' \ --absolute-names \ --hard-dereference \ --transform "s,$TMPDIR/reginfo,$dir/.reginfo," \ diff --git a/packaging/secure-packages/flake.lock b/packaging/secure-packages/flake.lock new file mode 100644 index 000000000000..75f43e134736 --- /dev/null +++ b/packaging/secure-packages/flake.lock @@ -0,0 +1,150 @@ +{ + "nodes": { + "bdwgc": { + "flake": false, + "locked": { + "lastModified": 1790015987, + "narHash": "sha256-bPVosUgucJR0hGSap3tJW84AR6eGYegVROV+O8VYX5A=", + "owner": "DeterminateSystems", + "repo": "bdwgc", + "rev": "373957929b9ca3b2ae69a3e16b830c44c15b42ca", + "type": "github" + }, + "original": { + "owner": "DeterminateSystems", + "ref": "user-defined-stacks", + "repo": "bdwgc", + "type": "github" + } + }, + "flake-compat": { + "flake": false, + "locked": { + "lastModified": 1767039857, + "narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=", + "owner": "NixOS", + "repo": "flake-compat", + "rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab", + "type": "github" + }, + "original": { + "owner": "NixOS", + "repo": "flake-compat", + "type": "github" + } + }, + "flake-parts": { + "inputs": { + "nixpkgs-lib": [ + "nix", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1782949081, + "narHash": "sha256-vp6Y/Grm98ESt6ceOkWiHWyZRDV3J1RID4w+6NWK9yA=", + "rev": "17c9d6cdfc60c64f4ee8d306f9bc0b4ccb51481e", + "revCount": 480, + "type": "tarball", + "url": "https://api.flakehub.com/f/pinned/hercules-ci/flake-parts/0.1.480%2Brev-17c9d6cdfc60c64f4ee8d306f9bc0b4ccb51481e/019f2195-dee5-7233-9747-eca0c27f7406/source.tar.gz" + }, + "original": { + "type": "tarball", + "url": "https://flakehub.com/f/hercules-ci/flake-parts/0.1" + } + }, + "git-hooks-nix": { + "inputs": { + "flake-compat": "flake-compat", + "nixpkgs": [ + "nix", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1784288435, + "narHash": "sha256-ReRHaLgr/uVqdD8afFSn+myXIfpHeOhP0yYe0TJqAA8=", + "rev": "43b3c1ab9d40fb1dbb008f451988a91e375825e9", + "revCount": 1231, + "type": "tarball", + "url": "https://api.flakehub.com/f/pinned/cachix/git-hooks.nix/0.1.1231%2Brev-43b3c1ab9d40fb1dbb008f451988a91e375825e9/019f7135-8fdf-76f0-b1a1-d2c67e91af8d/source.tar.gz" + }, + "original": { + "type": "tarball", + "url": "https://flakehub.com/f/cachix/git-hooks.nix/0.1.941" + } + }, + "nix": { + "inputs": { + "bdwgc": "bdwgc", + "flake-parts": "flake-parts", + "git-hooks-nix": "git-hooks-nix", + "nixpkgs": "nixpkgs", + "nixpkgs-23-11": "nixpkgs-23-11", + "nixpkgs-regression": "nixpkgs-regression" + }, + "locked": { + "path": "../..", + "type": "path" + }, + "original": { + "path": "../..", + "type": "path" + }, + "parent": [] + }, + "nixpkgs": { + "locked": { + "lastModified": 1790960035, + "narHash": "sha256-OFXiop0EX5YtaUckoKMJEu+Pq3Blv19dY76bHl5nH2g=", + "rev": "ad9e1d72007a5b115c4ef59edaa50ac12f868d5d", + "revCount": 1015033, + "type": "tarball", + "url": "https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-26.05/0.1.1015033%2Brev-ad9e1d72007a5b115c4ef59edaa50ac12f868d5d/01a0fda6-2748-7282-b564-aaa6761bf51a/source.tar.gz" + }, + "original": { + "type": "tarball", + "url": "https://flakehub.com/f/DeterminateSystems/secure-packages-26.05/0" + } + }, + "nixpkgs-23-11": { + "locked": { + "lastModified": 1717159533, + "narHash": "sha256-oamiKNfr2MS6yH64rUn99mIZjc45nGJlj9eGth/3Xuw=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "a62e6edd6d5e1fa0329b8653c801147986f8d446", + "type": "github" + }, + "original": { + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "a62e6edd6d5e1fa0329b8653c801147986f8d446", + "type": "github" + } + }, + "nixpkgs-regression": { + "locked": { + "lastModified": 1643052045, + "narHash": "sha256-uGJ0VXIhWKGXxkeNnq4TvV3CIOkUJ3PAoLZ3HMzNVMw=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2", + "type": "github" + }, + "original": { + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2", + "type": "github" + } + }, + "root": { + "inputs": { + "nix": "nix" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/packaging/secure-packages/flake.nix b/packaging/secure-packages/flake.nix new file mode 100644 index 000000000000..723eb14fe414 --- /dev/null +++ b/packaging/secure-packages/flake.nix @@ -0,0 +1,6 @@ +{ + inputs.nix.url = "../.."; + inputs.nix.inputs.nixpkgs.url = "https://flakehub.com/f/DeterminateSystems/secure-packages-26.05/0"; + + outputs = { self, nix }: nix.outputs; +} diff --git a/packaging/sentry-native.nix b/packaging/sentry-native.nix new file mode 100644 index 000000000000..0e21e9be55c2 --- /dev/null +++ b/packaging/sentry-native.nix @@ -0,0 +1,54 @@ +{ + lib, + stdenv, + fetchgit, + cmake, + curl, + pkg-config, + python3, + darwin, +}: + +stdenv.mkDerivation rec { + pname = "sentry-native"; + version = "0.13.5"; + + src = fetchgit { + url = "https://github.com/getsentry/sentry-native"; + tag = version; + hash = "sha256-vDBI6lB1DMLleAgRCfsHvTSdtmXOzvJSaNAt+NwOd3c="; + fetchSubmodules = true; + }; + + dontFixCmake = true; + + nativeBuildInputs = [ + cmake + pkg-config + ] + ++ lib.optionals stdenv.hostPlatform.isDarwin [ + python3 + darwin.bootstrap_cmds + ]; + + postPatch = '' + # Borrowed from psutil: stick to the old SDK name for now. + substituteInPlace external/crashpad/util/mac/mac_util.cc \ + --replace-fail kIOMainPortDefault kIOMasterPortDefault + ''; + + buildInputs = [ + curl + ]; + + cmakeBuildType = "RelWithDebInfo"; + + cmakeFlags = [ ]; + + outputs = [ + "out" + "dev" + ]; + + separateDebugInfo = true; +} diff --git a/packaging/wasmtime.nix b/packaging/wasmtime.nix new file mode 100644 index 000000000000..ca2c758ae0dd --- /dev/null +++ b/packaging/wasmtime.nix @@ -0,0 +1,74 @@ +# Stripped-down version of https://github.com/NixOS/nixpkgs/blob/master/pkgs/by-name/wa/wasmtime/package.nix, +# license: https://github.com/NixOS/nixpkgs/blob/master/COPYING +{ + lib, + stdenv, + rust, + fetchFromGitHub, + cmake, + enableShared ? !stdenv.hostPlatform.isStatic, + enableStatic ? stdenv.hostPlatform.isStatic, +}: +rust.packages.stable.rustPlatform.buildRustPackage (finalAttrs: { + pname = "wasmtime"; + version = "48.0.2"; + + src = fetchFromGitHub { + owner = "bytecodealliance"; + repo = "wasmtime"; + tag = "v${finalAttrs.version}"; + hash = "sha256-XMCHmvPVmiWpMsICNOGhV5GN+yyPTVFfnUgOC68qyo0="; + fetchSubmodules = true; + }; + + # Disable cargo-auditable until https://github.com/rust-secure-code/cargo-auditable/issues/124 is solved. + auditable = false; + + cargoHash = "sha256-KNCqgHMP/bCUAmDwTY+90g7bI69ud0rckj1iKlxadfM="; + cargoBuildFlags = [ + "--package" + "wasmtime-c-api" + "--no-default-features" + "--features cache,cranelift,wasi,pooling-allocator,wat,demangle,gc,gc-null" + ]; + + outputs = [ + "out" + "lib" + ]; + + nativeBuildInputs = [ + cmake + ]; + + doCheck = + with stdenv.buildPlatform; + # SIMD tests are only executed on platforms that support all + # required processor features (e.g. SSE3, SSSE3 and SSE4.1 on x86_64): + # https://github.com/bytecodealliance/wasmtime/blob/v9.0.0/cranelift/codegen/src/isa/x64/mod.rs#L220 + (isx86_64 -> sse3Support && ssse3Support && sse4_1Support) + && + # The dependency `wasi-preview1-component-adapter` fails to build because of: + # error: linker `rust-lld` not found + !isAarch64; + + postInstall = + let + inherit (stdenv.hostPlatform.rust) cargoShortTarget; + in + '' + moveToOutput lib $lib + ${lib.optionalString (!enableShared) "rm -f $lib/lib/*.so{,.*}"} + ${lib.optionalString (!enableStatic) "rm -f $lib/lib/*.a"} + + # copy the build.rs generated c-api headers + # https://github.com/rust-lang/cargo/issues/9661 + mkdir -p $out + cp -r target/${cargoShortTarget}/release/build/wasmtime-c-api-impl-*/out/include $out/include + '' + + lib.optionalString stdenv.hostPlatform.isDarwin '' + install_name_tool -id \ + $lib/lib/libwasmtime.dylib \ + $lib/lib/libwasmtime.dylib + ''; +}) diff --git a/scripts/install-nix-from-tarball.sh b/scripts/install-nix-from-tarball.sh index 73f389b6ff16..dbfb8fac1d30 100644 --- a/scripts/install-nix-from-tarball.sh +++ b/scripts/install-nix-from-tarball.sh @@ -187,6 +187,10 @@ for i in $(cd "$self/store" >/dev/null && echo ./*); do mv "$i_tmp" "$dest/store/$i" chmod -w "$dest/store/$i" fi + # BusyBox tar is bad at preserving mtime because it doesn't seem to do the correct + # thing of doing a final fixup pass for fixing parent directory permissions and doesn't + # handle symlinks at all. So we can set the correct mtime here. + find "$dest/store/$i" -depth -exec touch -hmd "@1" {} + done echo "" >&2 diff --git a/shell.nix b/shell.nix deleted file mode 100644 index 918f4bbd9e9e..000000000000 --- a/shell.nix +++ /dev/null @@ -1,3 +0,0 @@ -(import (fetchTarball "https://github.com/edolstra/flake-compat/archive/master.tar.gz") { - src = ./.; -}).shellNix diff --git a/src/external-api-docs/package.nix b/src/external-api-docs/package.nix index df6aa9838d6a..79fe692fa572 100644 --- a/src/external-api-docs/package.nix +++ b/src/external-api-docs/package.nix @@ -14,7 +14,7 @@ let in mkMesonDerivation (finalAttrs: { - pname = "nix-external-api-docs"; + pname = "determinate-nix-external-api-docs"; inherit version; workDir = ./.; diff --git a/src/internal-api-docs/package.nix b/src/internal-api-docs/package.nix index 6c4f354aee5c..636c19653eab 100644 --- a/src/internal-api-docs/package.nix +++ b/src/internal-api-docs/package.nix @@ -14,7 +14,7 @@ let in mkMesonDerivation (finalAttrs: { - pname = "nix-internal-api-docs"; + pname = "determinate-nix-internal-api-docs"; inherit version; workDir = ./.; diff --git a/src/libcmd/builtin-flake-schemas/flake.nix b/src/libcmd/builtin-flake-schemas/flake.nix new file mode 100644 index 000000000000..8d6cbccbd5f3 --- /dev/null +++ b/src/libcmd/builtin-flake-schemas/flake.nix @@ -0,0 +1,495 @@ +{ + description = "Schemas for well-known Nix flake output types"; + + outputs = + { self }: + let + mapAttrsToList = f: attrs: map (name: f name attrs.${name}) (builtins.attrNames attrs); + + checkModule = + module_: + let + module = if builtins.isPath module_ then import module_ else module_; + in + builtins.isAttrs module || builtins.isFunction module; + + mkApp = system: app: { + forSystems = [ system ]; + evalChecks.isValidApp = + app ? type + && app.type == "app" + && app ? program + && builtins.isString app.program + && + builtins.removeAttrs app [ + "type" + "program" + "meta" + ] == { }; + what = "app"; + shortDescription = app.meta.description or ""; + }; + + mkPackage = isFlakeCheck: what: system: package: { + forSystems = [ system ]; + shortDescription = package.meta.description or ""; + derivationAttrPath = [ ]; + inherit what isFlakeCheck; + }; + + singleDerivationInventory = + what: isFlakeCheck: output: + self.lib.mkChildren (builtins.mapAttrs (mkPackage isFlakeCheck what) output); + + schemasSchema = doc: { + version = 1; + inherit doc; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (schemaName: schemaDef: { + shortDescription = "A schema checker for the `${schemaName}` flake output"; + evalChecks.isValidSchema = + schemaDef.version or 0 == 1 + && schemaDef ? doc + && builtins.isString (schemaDef.doc) + && schemaDef ? inventory + && builtins.isFunction (schemaDef.inventory); + what = "flake schema"; + }) output + ); + }; + + appsSchema = { + version = 1; + doc = '' + The `apps` output provides commands available via `nix run`. + ''; + roles.nix-run = { }; + appendSystem = true; + defaultAttrPath = [ "default" ]; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (system: apps: { + forSystems = [ system ]; + children = builtins.mapAttrs (appName: app: mkApp system app) apps; + }) output + ); + }; + + defaultAppSchema = { + version = 1; + doc = '' + **DEPRECATED**. Use `apps..default` instead. + ''; + roles.nix-run = { }; + appendSystem = true; + defaultAttrPath = [ ]; + inventory = output: self.lib.mkChildren (builtins.mapAttrs mkApp output); + }; + + packagesSchema = { + version = 1; + doc = '' + The `packages` flake output contains packages that can be added to a shell using `nix shell`. + ''; + roles.nix-build = { }; + roles.nix-run = { }; + roles.nix-develop = { }; + roles.nix-search = { }; + appendSystem = true; + defaultAttrPath = [ "default" ]; + inventory = self.lib.derivationsInventory "package" false; + }; + + defaultPackageSchema = { + version = 1; + doc = '' + **DEPRECATED**. Use `packages..default` instead. + ''; + roles.nix-build = { }; + roles.nix-run = { }; + roles.nix-develop = { }; + roles.nix-search = { }; + appendSystem = true; + defaultAttrPath = [ ]; + inventory = singleDerivationInventory "package" false; + }; + + ociImagesSchema = { + version = 1; + doc = '' + The `ociImages` flake output contains derivations that build valid Open Container Initiative images. + ''; + inventory = self.lib.derivationsInventory "OCI image" false; + }; + + legacyPackagesSchema = { + version = 1; + doc = '' + The `legacyPackages` flake output is similar to `packages` but different in that it can be nested and thus contain attribute sets that contain more packages. + Since enumerating packages in nested attribute sets can be inefficient, you should favor `packages` over `legacyPackages`. + ''; + roles.nix-build = { }; + roles.nix-run = { }; + roles.nix-search = { }; + roles.nix-develop = { }; + appendSystem = true; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (systemType: packagesForSystem: { + forSystems = [ systemType ]; + isLegacy = true; + children = + let + recurse = + prefix: attrs: + builtins.mapAttrs ( + attrName: attrs: + # Necessary to deal with `AAAAAASomeThingsFailToEvaluate` etc. in Nixpkgs. + self.lib.try ( + if attrs.type or null == "derivation" then + { + forSystems = [ attrs.system ]; + shortDescription = attrs.meta.description or ""; + derivationAttrPath = [ ]; + what = "package"; + } + else + # Recurse at the first and second levels, or if the + # recurseForDerivations attribute if set. + if attrs.recurseForDerivations or false then + { + children = recurse (prefix + attrName + ".") attrs; + } + else + { + what = "unknown"; + } + ) (throw "failed") + ) attrs; + in + # The top-level cannot be a derivation. + assert packagesForSystem.type or null != "derivation"; + recurse (systemType + ".") packagesForSystem; + }) output + ); + }; + + checksSchema = { + version = 1; + doc = '' + The `checks` flake output contains derivations that will be built by `nix flake check`. + ''; + # FIXME: add role + inventory = self.lib.derivationsInventory "CI test" true; + }; + + devShellsSchema = { + version = 1; + doc = '' + The `devShells` flake output contains derivations that provide a development environment for `nix develop`. + ''; + roles.nix-develop = { }; + appendSystem = true; + defaultAttrPath = [ "default" ]; + inventory = self.lib.derivationsInventory "development environment" false; + }; + + devShellSchema = { + version = 1; + doc = '' + **DEPRECATED**. Use `devShells..default` instead. + ''; + roles.nix-develop = { }; + appendSystem = true; + defaultAttrPath = [ ]; + inventory = singleDerivationInventory "development environment" false; + }; + + formatterSchema = { + version = 1; + doc = '' + The `formatter` output specifies the package to use to format the project. + ''; + roles.nix-fmt = { }; + appendSystem = true; + defaultAttrPath = [ ]; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (system: formatter: { + forSystems = [ system ]; + shortDescription = formatter.meta.description or ""; + derivationAttrPath = [ ]; + what = "formatter"; + isFlakeCheck = false; + }) output + ); + }; + + templatesSchema = { + version = 1; + doc = '' + The `templates` output provides project templates. + ''; + roles.nix-template = { }; + defaultAttrPath = [ "default" ]; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (templateName: template: { + shortDescription = template.description or ""; + evalChecks.isValidTemplate = + template ? path + && builtins.isPath template.path + && template ? description + && builtins.isString template.description; + what = "template"; + }) output + ); + }; + + hydraJobsSchema = { + version = 1; + doc = '' + The `hydraJobs` flake output defines derivations to be built by the Hydra continuous integration system. + ''; + allowIFD = false; + inventory = + output: + let + recurse = + prefix: attrs: + self.lib.mkChildren ( + builtins.mapAttrs ( + attrName: attrs: + if attrs.type or null == "derivation" then + { + forSystems = [ attrs.system ]; + shortDescription = attrs.meta.description or ""; + derivationAttrPath = [ ]; + what = "Hydra CI test"; + } + else + recurse (prefix + attrName + ".") attrs + ) attrs + ); + in + # The top-level cannot be a derivation. + assert output.type or null != "derivation"; + recurse "" output; + }; + + overlaysSchema = { + version = 1; + doc = '' + The `overlays` flake output defines ["overlays"](https://nixos.org/manual/nixpkgs/stable/#chap-overlays) that can be plugged into Nixpkgs. + Overlays add additional packages or modify or replace existing packages. + ''; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (overlayName: overlay: { + what = "Nixpkgs overlay"; + evalChecks.isOverlay = + # FIXME: should try to apply the overlay to an actual + # Nixpkgs. But we don't have access to a nixpkgs + # flake here. Maybe this schema should be moved to the + # nixpkgs flake, where it does have access. + if !builtins.isFunction overlay then + throw "Overlay is not a function. It should be structured like: `final: previous: { /* ... */ }`." + else + true; + }) output + ); + }; + + nixosConfigurationsSchema = { + version = 1; + doc = '' + The `nixosConfigurations` flake output defines [NixOS system configurations](https://nixos.org/manual/nixos/stable/#ch-configuration). + ''; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (configName: machine: { + what = "NixOS configuration"; + derivationAttrPath = [ + "config" + "system" + "build" + "toplevel" + ]; + forSystems = [ machine.pkgs.stdenv.system ]; + }) output + ); + }; + + nixosModulesSchema = { + version = 1; + doc = '' + The `nixosModules` flake output defines importable [NixOS modules](https://nixos.org/manual/nixos/stable/#sec-writing-modules). + ''; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (moduleName: module: { + what = "NixOS module"; + evalChecks.isFunctionOrAttrs = checkModule module; + }) output + ); + }; + + homeConfigurationsSchema = { + version = 1; + doc = '' + The `homeConfigurations` flake output defines [Home Manager configurations](https://github.com/nix-community/home-manager). + ''; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (configName: this: { + what = "Home Manager configuration"; + derivationAttrPath = [ "activationPackage" ]; + forSystems = [ this.activationPackage.system ]; + }) output + ); + }; + + homeModulesSchema = { + version = 1; + doc = '' + The `homeModules` flake output defines importable [Home Manager](https://github.com/nix-community/home-manager) modules. + ''; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (moduleName: module: { + what = "Home Manager module"; + evalChecks.isFunctionOrAttrs = checkModule module; + }) output + ); + }; + + darwinConfigurationsSchema = { + version = 1; + doc = '' + The `darwinConfigurations` flake output defines [nix-darwin configurations](https://github.com/nix-darwin/nix-darwin). + ''; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (configName: this: { + what = "nix-darwin configuration"; + derivationAttrPath = [ "system" ]; + forSystems = [ this.system.system ]; + }) output + ); + }; + + darwinModulesSchema = { + version = 1; + doc = '' + The `darwinModules` flake output defines importable [nix-darwin modules](https://github.com/nix-darwin/nix-darwin). + ''; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs (moduleName: module: { + what = "nix-darwin module"; + evalChecks.isFunctionOrAttrs = checkModule module; + }) output + ); + }; + + bundlersSchema = { + version = 1; + doc = '' + The `bundlers` flake output defines ["bundlers"](https://nix.dev/manual/nix/latest/command-ref/new-cli/nix3-bundle) that transform derivation outputs into other formats, typically self-extracting executables or container images. + ''; + roles.nix-bundler = { }; + appendSystem = true; + defaultAttrPath = [ "default" ]; + inventory = + output: + self.lib.mkChildren ( + builtins.mapAttrs ( + system: bundlers: + let + forSystems = [ system ]; + in + { + inherit forSystems; + children = builtins.mapAttrs (bundlerName: bundler: { + inherit forSystems; + evalChecks.isValidBundler = builtins.isFunction bundler; + what = "bundler"; + }) bundlers; + } + ) output + ); + }; + + in + + { + # Helper functions + lib = { + try = + e: default: + let + res = builtins.tryEval e; + in + if res.success then res.value else default; + + mkChildren = children: { inherit children; }; + + derivationsInventory = + what: isFlakeCheck: output: + self.lib.mkChildren ( + builtins.mapAttrs (systemType: packagesForSystem: { + forSystems = [ systemType ]; + children = builtins.mapAttrs ( + packageName: mkPackage isFlakeCheck what systemType + ) packagesForSystem; + }) output + ); + }; + + exportedSchemas = { + schemas = schemasSchema '' + The `schemas` flake output is used to define and document flake outputs. + For the expected format, consult the Nix manual. + ''; + exportedSchemas = schemasSchema '' + The `exportedSchemas` flake output is used to define flake schemas that you + intend for other flakes to use. + ''; + apps = appsSchema; + defaultApp = defaultAppSchema; + packages = packagesSchema; + defaultPackage = defaultPackageSchema; + legacyPackages = legacyPackagesSchema; + checks = checksSchema; + devShells = devShellsSchema; + devShell = devShellSchema; + formatter = formatterSchema; + templates = templatesSchema; + hydraJobs = hydraJobsSchema; + overlays = overlaysSchema; + nixosConfigurations = nixosConfigurationsSchema; + nixosModules = nixosModulesSchema; + homeConfigurations = homeConfigurationsSchema; + homeModules = homeModulesSchema; + darwinConfigurations = darwinConfigurationsSchema; + darwinModules = darwinModulesSchema; + ociImages = ociImagesSchema; + bundlers = bundlersSchema; + }; + + schemas = self.exportedSchemas; + }; +} diff --git a/src/libcmd/call-flake-schemas.nix b/src/libcmd/call-flake-schemas.nix new file mode 100644 index 000000000000..1060ad56c292 --- /dev/null +++ b/src/libcmd/call-flake-schemas.nix @@ -0,0 +1,38 @@ +# The flake providing default schemas. +defaultSchemasFlake: + +# The flake whose contents we want to extract. +flake: + +let + + # Helper functions. + + mapAttrsToList = f: attrs: map (name: f name attrs.${name}) (builtins.attrNames attrs); + + outputNames = builtins.attrNames flake.outputs; + + schemas = flake.outputs.schemas or defaultSchemasFlake.exportedSchemas; + +in + +{ + outputs = flake.outputs; + + inventory = builtins.mapAttrs ( + outputName: _: + if schemas ? ${outputName} && schemas.${outputName}.version == 1 then + schemas.${outputName} + // ( + if flake.outputs ? ${outputName} then + { + output = schemas.${outputName}.inventory flake.outputs.${outputName}; + } + else + { + } + ) + else + { unknown = true; } + ) (schemas // flake.outputs); +} diff --git a/src/libcmd/command.cc b/src/libcmd/command.cc index 821c9e0565cd..045665ebf2fa 100644 --- a/src/libcmd/command.cc +++ b/src/libcmd/command.cc @@ -166,6 +166,13 @@ ref EvalCommand::getEvalStore() ref EvalCommand::getEvalState() { if (!evalState) { + if (startReplOnEvalErrors && evalSettings.evalCores != 1U) { + // Disable parallel eval if the debugger is enabled, since + // they're incompatible at the moment. + warn("using the debugger disables multi-threaded evaluation"); + evalSettings.evalCores = 1; + } + evalState = std::allocate_shared( traceable_allocator(), lookupPath, getEvalStore(), fetchSettings, evalSettings, getStore()); diff --git a/src/libcmd/common-eval-args.cc b/src/libcmd/common-eval-args.cc index 984bed34882e..a396a172816e 100644 --- a/src/libcmd/common-eval-args.cc +++ b/src/libcmd/common-eval-args.cc @@ -17,17 +17,12 @@ namespace nix { -fetchers::Settings fetchSettings; - -static GlobalConfig::Register rFetchSettings(&fetchSettings); - EvalSettings evalSettings{ settings.readOnlyMode, { { "flake", [](EvalState & state, std::string_view rest) { - experimentalFeatureSettings.require(Xp::Flakes); // FIXME `parseFlakeRef` should take a `std::string_view`. auto flakeRef = parseFlakeRef(fetchSettings, std::string{rest}, {}, true, false); debug("fetching flake search path element '%s''", rest); @@ -184,7 +179,6 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const std::files } else if (hasPrefix(s, "flake:")) { - experimentalFeatureSettings.require(Xp::Flakes); auto flakeRef = parseFlakeRef(fetchSettings, std::string(s.substr(6)), {}, true, false); auto [accessor, lockedRef] = flakeRef.resolve(fetchSettings, *state.store).lazyFetch(fetchSettings, *state.store); diff --git a/src/libcmd/flake-schemas.cc b/src/libcmd/flake-schemas.cc new file mode 100644 index 000000000000..6725274137d1 --- /dev/null +++ b/src/libcmd/flake-schemas.cc @@ -0,0 +1,633 @@ +#include "nix/cmd/flake-schemas.hh" +#include "nix/expr/eval-settings.hh" +#include "nix/fetchers/fetch-to-store.hh" +#include "nix/util/memory-source-accessor.hh" +#include "nix/util/mounted-source-accessor.hh" +#include "nix/flake/provenance.hh" +#include "nix/expr/parallel-eval.hh" +#include "nix/store/globals.hh" + +#include + +namespace nix::flake_schemas { + +using namespace eval_cache; +using namespace flake; + +static LockedFlake getBuiltinDefaultSchemasFlake(EvalState & state) +{ + auto accessor = make_ref(); + + accessor->setPathDisplay("«builtin-flake-schemas»"); + + accessor->addFile( + CanonPath("flake.nix"), +#include "builtin-flake-schemas/flake.nix.gen.hh" + ); + + auto [storePath, narHash] = state.store->computeStorePath("source", {accessor}); + + state.allowPath(storePath); // FIXME: should just whitelist the entire virtual store + + state.storeFS->mount(CanonPath(state.store->printStorePath(storePath)), accessor); + + // Construct a dummy flakeref. + auto flakeRef = parseFlakeRef( + fetchSettings, + fmt("tarball+https://builtin-flake-schemas?narHash=%s", narHash.to_string(HashFormat::SRI, true))); + + auto flake = readFlake(state, flakeRef, flakeRef, flakeRef, state.storePath(storePath), {}); + + return lockFlake(flakeSettings, state, flakeRef, {}, flake); +} + +ref call( + EvalState & state, + std::shared_ptr lockedFlake, + std::optional defaultSchemasFlake, + bool allowEvalCache) +{ + auto fingerprint = lockedFlake->getFingerprint(*state.store, state.fetchSettings); + + std::string callFlakeSchemasNix = +#include "call-flake-schemas.nix.gen.hh" + ; + + auto lockedDefaultSchemasFlake = defaultSchemasFlake + ? flake::lockFlake(flakeSettings, state, *defaultSchemasFlake, {}) + : getBuiltinDefaultSchemasFlake(state); + auto lockedDefaultSchemasFlakeFingerprint = + lockedDefaultSchemasFlake.getFingerprint(*state.store, state.fetchSettings); + + std::optional fingerprint2; + if (allowEvalCache && evalSettings.useEvalCache && evalSettings.pureEval && fingerprint + && lockedDefaultSchemasFlakeFingerprint) + fingerprint2 = hashString( + HashAlgorithm::SHA256, + fmt("app:%s:%s:%s", + hashString(HashAlgorithm::SHA256, callFlakeSchemasNix).to_string(HashFormat::Base16, false), + fingerprint->to_string(HashFormat::Base16, false), + lockedDefaultSchemasFlakeFingerprint->to_string(HashFormat::Base16, false))); + + if (fingerprint2) { + auto i = state.evalCaches.find(*fingerprint2); + if (i != state.evalCaches.end()) + return i->second; + } + + auto cache = make_ref( + fingerprint2, state, [&state, lockedFlake, callFlakeSchemasNix, lockedDefaultSchemasFlake]() { + auto vCallFlakeSchemas = state.allocValue(); + state.eval( + state.parseExprFromString(callFlakeSchemasNix, state.rootPath(CanonPath::root)), *vCallFlakeSchemas); + + auto vFlake = state.allocValue(); + flake::callFlake(state, *lockedFlake, *vFlake); + + auto vDefaultSchemasFlake = state.allocValue(); + if (vFlake->type() == nAttrs && vFlake->attrs()->get(state.symbols.create("schemas"))) + vDefaultSchemasFlake->mkNull(); + else + flake::callFlake(state, lockedDefaultSchemasFlake, *vDefaultSchemasFlake); + + auto vRes = state.allocValue(); + Value * args[] = {vDefaultSchemasFlake, vFlake}; + state.callFunction(*vCallFlakeSchemas, args, *vRes, noPos); + + return vRes; + }); + + /* Derive the flake output attribute path from the cursor used to + traverse the inventory. We do this so we don't have to maintain + a separate attrpath for that. */ + cache->cleanupAttrPath = [&](AttrPath && attrPath) { + AttrPath res; + auto i = attrPath.begin(); + if (i == attrPath.end()) + return attrPath; + + if (state.symbols[*i] == "inventory") { + ++i; + if (i != attrPath.end()) { + res.push_back(*i++); // copy output name + if (i != attrPath.end()) + ++i; // skip "outputs" + while (i != attrPath.end()) { + ++i; // skip "children" + if (i != attrPath.end()) + res.push_back(*i++); + } + } + } + + else if (state.symbols[*i] == "outputs") { + res.insert(res.begin(), ++i, attrPath.end()); + } + + else + abort(); + + return res; + }; + + if (fingerprint2) + state.evalCaches.emplace(*fingerprint2, cache); + + return cache; +} + +void forEachOutput( + ref inventory, + std::function output, const std::string & doc, bool isLast)> f) +{ + auto outputNames = inventory->getAttrs(); + + auto doOutputs = [&](bool allowIFD) { + evalSettings.enableImportFromDerivation.setDefault(allowIFD); + for (const auto & [i, outputName] : enumerate(outputNames)) { + auto outputInfo = inventory->getAttr(outputName); + try { + auto allowIFDAttr = outputInfo->maybeGetAttr("allowIFD"); + if (allowIFD != (!allowIFDAttr || allowIFDAttr->getBool())) + continue; + auto isUnknown = (bool) outputInfo->maybeGetAttr("unknown"); + auto output = outputInfo->maybeGetAttr("output"); + if (!output && !isUnknown) + // We have a schema but no corresponding output, so skip this. + continue; + Activity act(*logger, lvlInfo, actUnknown, fmt("evaluating '%s'", outputInfo->getAttrPathStr())); + f(outputName, + isUnknown ? std::shared_ptr() : output, + isUnknown ? "" : outputInfo->getAttr("doc")->getString(), + i + 1 == outputNames.size()); + } catch (Error & e) { + e.addTrace(nullptr, "while evaluating the flake output '%s':", outputInfo->getAttrPathStr()); + throw; + } + } + }; + + // Do outputs that disallow import-from-derivation first. That way, they can't depend on outputs that do allow it. + doOutputs(false); + doOutputs(true); +} + +void visit( + std::optional system, + bool includeLegacy, + ref node, + std::shared_ptr provenance, + std::function visitLeaf, + std::function)> visitNonLeaf, + std::function node, const std::vector & systems)> visitFiltered, + std::function node)> visitLegacy) +{ + Activity act(*logger, lvlInfo, actUnknown, fmt("evaluating '%s'", node->getAttrPathStr())); + + PushProvenance pushedProvenance( + node->root->state, + provenance ? std::make_shared(provenance, node->getAttrPathStr(), evalSettings.pureEval) + : nullptr); + + /* Filter out legacy outputs, unless --legacy is enabled. */ + if (!includeLegacy) { + if (auto b = node->maybeGetAttr("isLegacy"); b && b->getBool()) { + visitLegacy(node); + return; + } + } + + /* Apply the system type filter. */ + if (system) { + if (auto forSystems = Node(node).forSystems()) { + if (std::find(forSystems->begin(), forSystems->end(), *system) == forSystems->end()) { + visitFiltered(node, *forSystems); + return; + } + } + } + + if (auto children = node->maybeGetAttr("children")) { + visitNonLeaf([&](ForEachChild f) { + auto attrNames = children->getAttrs(); + for (const auto & [i, attrName] : enumerate(attrNames)) { + try { + f(attrName, children->getAttr(attrName), i + 1 == attrNames.size()); + } catch (Error & e) { + // FIXME: use the `isLegacy` attribute. + if (node->root->state.symbols[node->getAttrPath()[0]] != "legacyPackages") { + e.addTrace( + nullptr, "while evaluating the flake output attribute '%s':", node->getAttrPathStr()); + throw; + } + } + } + }); + } + + else + visitLeaf(Leaf(node)); +} + +std::optional> Node::forSystems() const +{ + if (auto forSystems = node->maybeGetAttr("forSystems")) + return forSystems->getListOfStrings(); + else + return std::nullopt; +} + +ref Node::getOutput(const ref & outputs) const +{ + auto res = outputs->findAlongAttrPath(node->getAttrPath()); + if (!res) + throw Error("flake output '%s' should exist according to its schema, but it doesn't", node->getAttrPathStr()); + return *res; +} + +std::optional Leaf::what() const +{ + if (auto what = node->maybeGetAttr("what")) + return what->getString(); + else + return std::nullopt; +} + +std::optional Leaf::shortDescription() const +{ + if (auto what = node->maybeGetAttr("shortDescription")) + return what->getString(); + return std::nullopt; +} + +std::optional Leaf::derivationAttrPath() const +{ + auto n = node->maybeGetAttr("derivationAttrPath"); + if (!n) + return std::nullopt; + return AttrPath::fromStrings(node->root->state, n->getListOfStrings()); +} + +std::shared_ptr Leaf::derivation(const ref & outputs) const +{ + auto path = derivationAttrPath(); + if (!path) { + auto n = node->maybeGetAttr("derivation"); + if (n) + warn( + "Flake output '%s' has a schema that uses the deprecated 'derivation' attribute instead of 'derivationAttrPath'. " + "Please update the schema to use 'derivationAttrPath' instead. " + "You may want to upgrade to version 0.3.0 or higher of https://github.com/DeterminateSystems/flake-schemas.", + node->getAttrPathStr()); + return n; + } + auto drv = getOutput(outputs)->findAlongAttrPath(*path); + if (!drv) + throw Error( + "flake output '%s' does not have a derivation attribute '%s'", + node->getAttrPathStr(), + path->to_string(node->root->state)); + return *drv; +} + +bool Leaf::isFlakeCheck() const +{ + auto isFlakeCheck = node->maybeGetAttr("isFlakeCheck"); + return isFlakeCheck && isFlakeCheck->getBool(); +} + +std::optional getOutputInfo(ref inventory, AttrPath attrPath) +{ + if (attrPath.empty()) + return std::nullopt; + + auto outputName = attrPath.front(); + + auto schemaInfo = inventory->maybeGetAttr(outputName); + if (!schemaInfo) + return std::nullopt; + + auto node = schemaInfo->maybeGetAttr("output"); + if (!node) + return std::nullopt; + + auto pathLeft = std::span(attrPath).subspan(1); + + while (!pathLeft.empty()) { + auto children = node->maybeGetAttr("children"); + if (!children) + break; + auto attr = pathLeft.front(); + node = children->maybeGetAttr(attr); + if (!node) + return std::nullopt; + pathLeft = pathLeft.subspan(1); + } + + return OutputInfo{ + .schemaInfo = ref(schemaInfo), + .nodeInfo = ref(node), + .leafAttrPath = AttrPath(pathLeft.begin(), pathLeft.end()), + }; +} + +Schemas getSchemas(ref inventory) +{ + auto & state(inventory->root->state); + + Schemas schemas; + + for (auto & schemaName : inventory->getAttrs()) { + auto schema = inventory->getAttr(schemaName); + + SchemaInfo schemaInfo; + + if (auto roles = schema->maybeGetAttr("roles")) { + for (auto & roleName : roles->getAttrs()) { + schemaInfo.roles.insert(std::string(state.symbols[roleName])); + } + } + + if (auto appendSystem = schema->maybeGetAttr("appendSystem")) + schemaInfo.appendSystem = appendSystem->getBool(); + + if (auto defaultAttrPath = schema->maybeGetAttr("defaultAttrPath")) { + AttrPath attrPath; + for (auto & s : defaultAttrPath->getListOfStrings()) + attrPath.push_back(state.symbols.create(s)); + schemaInfo.defaultAttrPath = std::move(attrPath); + } + + schemas.insert_or_assign(std::string(state.symbols[schemaName]), std::move(schemaInfo)); + } + + return schemas; +} + +nlohmann::json getFlakeInventory( + EvalState & state, + Store & evalStore, + LockedFlake & flake, + ref cache, + const FlakeInventoryOptions & options) +{ + auto inventory = cache->getRoot()->getAttr("inventory"); + auto outputs = cache->getRoot()->getAttr("outputs"); + + auto localSystem = std::string(settings.thisSystem.get()); + + FutureVector futures(*state.executor); + + std::function node, nlohmann::json & obj)> visit; + + visit = [&](ref node, nlohmann::json & obj) { + /* Record the schema attributes that apply to both leaf and non-leaf nodes. Non-leaf `forSystems` and + `isLegacy` are only needed by `nix flake bake` to reproduce the schema. */ + auto addNodeInfo = [&](bool isLeaf) { + if (auto forSystems = flake_schemas::Node(node).forSystems(); forSystems && (isLeaf || options.bake)) + obj.emplace("forSystems", *forSystems); + if (options.bake) + if (auto b = node->maybeGetAttr("isLegacy"); b && b->getBool()) + obj.emplace("isLegacy", true); + }; + + flake_schemas::visit( + options.showAllSystems ? std::optional() : localSystem, + options.showLegacy, + node, + flake.flake.provenance, + + [&](const flake_schemas::Leaf & leaf) { + if (auto what = leaf.what()) + obj.emplace("what", *what); + + if (auto shortDescription = leaf.shortDescription()) + obj.emplace("shortDescription", *shortDescription); + + /* Record the attribute path of the derivation relative to the output attribute (e.g. + `config.system.build.toplevel` for `nixosConfigurations`), so that the baked flake can put the + derivation in the right place. Omitted if the output attribute is the derivation itself. */ + if (options.bake) { + if (auto path = leaf.derivationAttrPath(); path && !path->empty()) { + auto attrPath = nlohmann::json::array(); + for (auto & attr : *path) + attrPath.push_back(std::string(state.symbols[attr])); + obj.emplace("derivationAttrPath", std::move(attrPath)); + } + } + + if (auto drv = leaf.derivation(outputs)) { + auto drvObj = nlohmann::json::object(); + + if (options.showDrvNames) + drvObj.emplace("name", drv->getAttr(state.s.name)->getString()); + + if (options.showDrvPaths) { + auto drvPath = drv->forceDerivation(); + drvObj.emplace("path", state.store->printStorePath(drvPath)); + } + + /* Derivations whose output paths are not known in advance (e.g. content-addressed + derivations) cannot be baked, since baking requires the output paths to be known at + evaluation time. */ + bool bakeable = true; + + if (options.bake) + try { + /* Record `meta.mainProgram` so that the baked flake can preserve it for `nix run`. */ + if (auto aMeta = drv->maybeGetAttr(state.s.meta)) + if (auto aMainProgram = aMeta->maybeGetAttr("mainProgram")) + drvObj.emplace("mainProgram", aMainProgram->getString()); + + /* Record which output this attribute refers to, since it's not necessarily `out`. */ + if (auto aOutputName = drv->maybeGetAttr(state.s.outputName)) + drvObj.emplace("outputName", aOutputName->getString()); + + /* Record the system of the derivation. */ + if (auto aSystem = drv->maybeGetAttr(state.s.system)) + drvObj.emplace("system", aSystem->getString()); + + // FIXME: remove this once we have on-demand writing of .drvs. + /* Get the output paths from the derivation attribute set rather than from the `.drv` file + in the store. This avoids waiting for the derivation to be written to the store (which + would serialise parallel evaluation on the store writer) and works in read-only mode. + Outputs whose path is not known at evaluation time (e.g. content-addressed outputs) + have a placeholder rather than a store path. */ + auto outputs = nlohmann::json::object(); + auto getOutPath = [&](ref out) { + /* `derivation` produces an attribute set per output, but `import`ing a `.drv` + produces plain strings. */ + return out->forceValue().type() == nAttrs ? out->getAttr(state.s.outPath)->getString() + : out->getString(); + }; + auto addOutput = [&](const std::string & outputName, const std::string & outPath) { + if (state.store->isStorePath(outPath)) + outputs.emplace(outputName, outPath); + else { + outputs.emplace(outputName, nullptr); + bakeable = false; + } + }; + if (auto aOutputs = drv->maybeGetAttr(state.s.outputs)) + for (auto & outputName : aOutputs->getListOfStrings()) + addOutput(outputName, getOutPath(drv->getAttr(outputName))); + else { + auto aOutputName = drv->maybeGetAttr(state.s.outputName); + addOutput( + aOutputName ? aOutputName->getString() : "out", + drv->getAttr(state.s.outPath)->getString()); + } + drvObj.emplace("outputs", std::move(outputs)); + + if (!bakeable) { + warn( + "cannot bake '%s' because its output paths are not known in advance (e.g. it is a " + "content-addressed derivation)", + leaf.node->getAttrPathStr()); + drvObj.emplace("failed", true); + } + } catch (EvalError & e) { + /* The derivation cannot be evaluated (e.g. a package marked as broken). Record it anyway, + so that the baked flake can still list it (like the original flake would), but fail + when it is built. */ + drvObj.erase("outputs"); + drvObj.emplace("failed", true); + } + + else if (options.showOutputPaths) { + auto outputs = nlohmann::json::object(); + auto drvPath = drv->forceDerivation(); + auto drv = evalStore.derivationFromPath(drvPath); + for (auto & i : drv.outputsAndOptPaths(*state.store)) { + if (auto outPath = i.second.second) + outputs.emplace(i.first, state.store->printStorePath(*outPath)); + else + outputs.emplace(i.first, nullptr); + } + drvObj.emplace("outputs", std::move(outputs)); + } + + obj.emplace("derivation", std::move(drvObj)); + } + + if (options.bake && leaf.isFlakeCheck()) + obj.emplace("isFlakeCheck", true); + + addNodeInfo(true); + }, + + [&](std::function forEachChild) { + auto children = nlohmann::json::object(); + forEachChild([&](Symbol attrName, ref node, bool isLast) { + auto & j = children.emplace(state.symbols[attrName], nlohmann::json::object()).first.value(); + state.spawn(futures, 1, [&visit, &j, node]() { + try { + visit(node, j); + } catch (EvalError & e) { + // FIXME: make it a flake schema attribute whether to ignore evaluation errors. + if (node->root->state.symbols[node->getAttrPath()[0]] == "legacyPackages") + j.emplace("failed", true); + else + throw; + } + }); + }); + obj.emplace("children", std::move(children)); + + addNodeInfo(false); + }, + + [&](ref node, const std::vector & systems) { + obj.emplace("filtered", true); + }, + + [&](ref node) { obj.emplace("isLegacy", true); }); + }; + + auto inv = nlohmann::json::object(); + + auto schemas = getSchemas(inventory); + + flake_schemas::forEachOutput( + inventory, + [&](Symbol outputName, std::shared_ptr output, const std::string & doc, bool isLast) { + auto & j = inv.emplace(state.symbols[outputName], nlohmann::json::object()).first.value(); + + if (output) { + j.emplace("doc", doc); + + /* Record the schema-level attributes, so that `nix flake bake` can reproduce the schema. */ + if (auto schema = get(schemas, std::string(state.symbols[outputName])); schema && options.bake) { + if (!schema->roles.empty()) + j.emplace("roles", schema->roles); + if (schema->appendSystem) + j.emplace("appendSystem", true); + if (schema->defaultAttrPath) { + auto attrPath = nlohmann::json::array(); + for (auto & attr : *schema->defaultAttrPath) + attrPath.push_back(std::string(state.symbols[attr])); + j.emplace("defaultAttrPath", std::move(attrPath)); + } + } + auto & j2 = j.emplace("output", nlohmann::json::object()).first.value(); + state.spawn(futures, 1, [&visit, output, &j2]() { visit(ref(output), j2); }); + } else + j.emplace("unknown", true); + }); + + futures.finishAll(); + + if (options.bake) { + /* Prune the inventory to the nodes that can be baked: derivation leaves, and non-leaf nodes that contain at + least one of them. This drops non-derivation leaves (e.g. overlays), filtered/failed nodes, derivations + that cannot be baked, and unknown outputs. */ + auto prune = [](this const auto & prune, nlohmann::json & node) -> bool { + if (auto children = node.find("children"); children != node.end()) { + for (auto i = children->begin(); i != children->end();) { + if (prune(i.value())) + ++i; + else + i = children->erase(i); + } + return !children->empty(); + } + return node.contains("derivation"); + }; + + for (auto i = inv.begin(); i != inv.end();) { + auto output = i.value().find("output"); + if (output != i.value().end() && prune(*output)) + ++i; + else + i = inv.erase(i); + } + } + + return inv; +} + +} // namespace nix::flake_schemas + +namespace nix { + +MixFlakeSchemas::MixFlakeSchemas() +{ + addFlag( + {.longName = "default-flake-schemas", + .description = "The URL of the flake providing default flake schema definitions.", + .labels = {"flake-ref"}, + .handler = {&defaultFlakeSchemas}, + .completer = {[&](AddCompletions & completions, size_t, std::string_view prefix) { + completeFlakeRef(completions, getStore(), prefix); + }}}); +} + +std::optional MixFlakeSchemas::getDefaultFlakeSchemas() +{ + if (!defaultFlakeSchemas) + return std::nullopt; + else + return parseFlakeRef(fetchSettings, *defaultFlakeSchemas, absPath(getCommandBaseDir())); +} + +} // namespace nix diff --git a/src/libcmd/include/nix/cmd/command.hh b/src/libcmd/include/nix/cmd/command.hh index 92fd032de0cd..c43b4f87ed43 100644 --- a/src/libcmd/include/nix/cmd/command.hh +++ b/src/libcmd/include/nix/cmd/command.hh @@ -148,7 +148,16 @@ struct MixFlakeOptions : virtual Args, EvalCommand } }; -struct SourceExprCommand : virtual Args, MixFlakeOptions +struct MixFlakeSchemas : virtual Args, virtual StoreCommand +{ + std::optional defaultFlakeSchemas; + + MixFlakeSchemas(); + + std::optional getDefaultFlakeSchemas(); +}; + +struct SourceExprCommand : virtual Args, MixFlakeOptions, MixFlakeSchemas { std::optional file; std::optional expr; @@ -159,9 +168,13 @@ struct SourceExprCommand : virtual Args, MixFlakeOptions ref parseInstallable(ref store, const std::string & installable); - virtual Strings getDefaultFlakeAttrPaths(); - - virtual Strings getDefaultFlakeAttrPathPrefixes(); + /** + * Return a set of "roles" that this command implements + * (e.g. `nix-build` or `nix-develop`). This is used by flake + * schemas to determine which flake outputs are used as default + * attrpath prefixes. + */ + virtual StringSet getRoles(); /** * Complete an installable from the given prefix. @@ -230,6 +243,8 @@ struct InstallableCommand : virtual Args, SourceExprCommand { InstallableCommand(); + virtual void preRun(ref store); + virtual void run(ref store, ref installable) = 0; void run(ref store) override; @@ -388,8 +403,7 @@ void completeFlakeRefWithFragment( AddCompletions & completions, ref evalState, flake::LockFlags lockFlags, - Strings attrPathPrefixes, - const Strings & defaultFlakeAttrPaths, + const StringSet & roles, std::string_view prefix); std::string showVersions(const StringSet & versions); diff --git a/src/libcmd/include/nix/cmd/common-eval-args.hh b/src/libcmd/include/nix/cmd/common-eval-args.hh index 14897158ae6d..93ffbd435ef9 100644 --- a/src/libcmd/include/nix/cmd/common-eval-args.hh +++ b/src/libcmd/include/nix/cmd/common-eval-args.hh @@ -26,9 +26,6 @@ namespace flake { struct Settings; } -/** - * @todo Get rid of global settings variables - */ extern fetchers::Settings fetchSettings; /** diff --git a/src/libcmd/include/nix/cmd/flake-schemas.hh b/src/libcmd/include/nix/cmd/flake-schemas.hh new file mode 100644 index 000000000000..0db8f96d549c --- /dev/null +++ b/src/libcmd/include/nix/cmd/flake-schemas.hh @@ -0,0 +1,121 @@ +#pragma once + +#include "nix/expr/eval-cache.hh" +#include "nix/flake/flake.hh" +#include "nix/cmd/command.hh" + +namespace nix::flake { +struct LockedFlake; +} + +namespace nix::flake_schemas { + +using namespace eval_cache; + +ref call( + EvalState & state, + std::shared_ptr lockedFlake, + std::optional defaultSchemasFlake, + bool allowEvalCache = true); + +void forEachOutput( + ref inventory, + std::function output, const std::string & doc, bool isLast)> f); + +/** + * A convenience wrapper around `AttrCursor` for nodes in the `inventory` tree returned by call-flake-schemas.nix. + */ +struct Node +{ + const ref node; + + Node(const ref & node) + : node(node) + { + } + + /** + * Return the `forSystems` attribute. This can be null, which + * means "all systems". + */ + std::optional> forSystems() const; + + /** + * Return the actual output corresponding to this info node. + */ + ref getOutput(const ref & outputs) const; +}; + +struct Leaf : Node +{ + using Node::Node; + + std::optional what() const; + + std::optional shortDescription() const; + + std::optional derivationAttrPath() const; + + /** + * Return the attribute corresponding to `derivationAttrPath`, if set. + */ + std::shared_ptr derivation(const ref & outputs) const; + + bool isFlakeCheck() const; +}; + +typedef std::function attr, bool isLast)> ForEachChild; + +void visit( + std::optional system, + bool includeLegacy, + ref node, + std::shared_ptr provenance, + std::function visitLeaf, + std::function)> visitNonLeaf, + std::function node, const std::vector & systems)> visitFiltered, + std::function node)> visitLegacy); + +struct OutputInfo +{ + ref schemaInfo; + ref nodeInfo; + AttrPath leafAttrPath; +}; + +std::optional getOutputInfo(ref inventory, AttrPath attrPath); + +struct SchemaInfo +{ + std::string doc; + StringSet roles; + bool appendSystem = false; + std::optional defaultAttrPath; +}; + +using Schemas = std::map; + +Schemas getSchemas(ref root); + +struct FlakeInventoryOptions +{ + bool showLegacy = false; + bool showAllSystems = false; + bool showOutputPaths = false; + bool showDrvPaths = false; + bool showDrvNames = false; + /** + * Include additional information needed by `nix flake bake` + * (e.g. `derivationAttrPath` and `mainProgram`). + */ + bool bake = false; +}; + +nlohmann::json getFlakeInventory( + EvalState & state, + Store & evalStore, + flake::LockedFlake & flake, + ref cache, + const FlakeInventoryOptions & options); + +} // namespace nix::flake_schemas diff --git a/src/libcmd/include/nix/cmd/installable-attr-path.hh b/src/libcmd/include/nix/cmd/installable-attr-path.hh index ef9dac813346..c33b119a5fa8 100644 --- a/src/libcmd/include/nix/cmd/installable-attr-path.hh +++ b/src/libcmd/include/nix/cmd/installable-attr-path.hh @@ -17,6 +17,7 @@ #include "nix/util/url.hh" #include "nix/fetchers/registry.hh" #include "nix/store/build-result.hh" +#include "nix/expr/root-value.hh" #include #include diff --git a/src/libcmd/include/nix/cmd/installable-flake.hh b/src/libcmd/include/nix/cmd/installable-flake.hh index 9f449ad48f2e..3acce913dcb7 100644 --- a/src/libcmd/include/nix/cmd/installable-flake.hh +++ b/src/libcmd/include/nix/cmd/installable-flake.hh @@ -36,11 +36,14 @@ struct ExtraPathInfoFlake : ExtraPathInfoValue struct InstallableFlake : InstallableValue { FlakeRef flakeRef; - Strings attrPaths; - Strings prefixes; + std::string fragment; + AttrPath parsedFragment; + StringSet roles; ExtendedOutputsSpec extendedOutputsSpec; const flake::LockFlags & lockFlags; mutable std::shared_ptr _lockedFlake; + bool useEvalCache = true; + std::optional defaultFlakeSchemas; InstallableFlake( SourceExprCommand * cmd, @@ -48,17 +51,15 @@ struct InstallableFlake : InstallableValue FlakeRef && flakeRef, std::string_view fragment, ExtendedOutputsSpec extendedOutputsSpec, - Strings attrPaths, - Strings prefixes, - const flake::LockFlags & lockFlags); + StringSet roles, + const flake::LockFlags & lockFlags, + std::optional defaultFlakeSchemas); std::string what() const override { - return flakeRef.to_string() + "#" + *attrPaths.begin(); + return flakeRef.to_string() + "#" + fragment; } - std::vector getActualAttrPaths(); - DerivedPathsWithInfo toDerivedPaths() override; std::pair toValue(EvalState & state) override; @@ -67,11 +68,23 @@ struct InstallableFlake : InstallableValue * Get a cursor to every attrpath in getActualAttrPaths() that * exists. However if none exists, throw an exception. */ - std::vector> getCursors(EvalState & state) override; + std::vector> getCursors(EvalState & state, bool useDefaultAttrPath) override; + + void getCompletions(const std::string & flakeRefS, AddCompletions & completions); ref getLockedFlake() const; FlakeRef nixpkgsFlakeRef() const; + + std::shared_ptr makeProvenance(std::string_view attrPath) const; + + ref openEvalCache() const; + +private: + + mutable std::shared_ptr _evalCache; + + std::vector getAttrPaths(bool useDefaultAttrPath, ref inventory); }; /** diff --git a/src/libcmd/include/nix/cmd/installable-value.hh b/src/libcmd/include/nix/cmd/installable-value.hh index 27a1fb9815d4..09178c96c972 100644 --- a/src/libcmd/include/nix/cmd/installable-value.hh +++ b/src/libcmd/include/nix/cmd/installable-value.hh @@ -93,7 +93,7 @@ struct InstallableValue : Installable * However if none exists, throw exception instead of returning * empty vector. */ - virtual std::vector> getCursors(EvalState & state); + virtual std::vector> getCursors(EvalState & state, bool useDefaultAttrPath = true); /** * Get the first and most preferred cursor this Installable could diff --git a/src/libcmd/include/nix/cmd/installables.hh b/src/libcmd/include/nix/cmd/installables.hh index 530334e037b7..2ea35261c7fa 100644 --- a/src/libcmd/include/nix/cmd/installables.hh +++ b/src/libcmd/include/nix/cmd/installables.hh @@ -96,6 +96,22 @@ typedef std::vector DerivedPathsWithInfo; struct Installable; +struct InstallableWithBuildResult +{ + ref installable; + + using Success = BuiltPathWithResult; + + using Failure = BuildResult; // must be a `BuildResult::Failure` + + std::variant result; + + /** + * Throw an exception if this represents a failure, otherwise returns a `BuiltPathWithResult`. + */ + const BuiltPathWithResult & getSuccess() const; +}; + /** * Shorthand, for less typing and helping us keep the choice of * collection in sync. @@ -160,13 +176,15 @@ struct Installable const Installables & installables, BuildMode bMode = bmNormal); - static std::vector, BuiltPathWithResult>> build2( + static std::vector build2( ref evalStore, ref store, Realise mode, const Installables & installables, BuildMode bMode = bmNormal); + static void throwBuildErrors(std::vector & buildResults, const Store & store); + static std::set toStorePathSet( ref evalStore, ref store, Realise mode, OperateOn operateOn, const Installables & installables); diff --git a/src/libcmd/include/nix/cmd/meson.build b/src/libcmd/include/nix/cmd/meson.build index ac074b06ff3c..6dea1ea5be8c 100644 --- a/src/libcmd/include/nix/cmd/meson.build +++ b/src/libcmd/include/nix/cmd/meson.build @@ -9,6 +9,7 @@ headers = files( 'common-eval-args.hh', 'compatibility-settings.hh', 'editor-for.hh', + 'flake-schemas.hh', 'get-build-log.hh', 'installable-attr-path.hh', 'installable-derived-path.hh', diff --git a/src/libcmd/installable-attr-path.cc b/src/libcmd/installable-attr-path.cc index aa5da2e56646..524dc799f02f 100644 --- a/src/libcmd/installable-attr-path.cc +++ b/src/libcmd/installable-attr-path.cc @@ -21,7 +21,7 @@ InstallableAttrPath::InstallableAttrPath( ExtendedOutputsSpec extendedOutputsSpec) : InstallableValue(state) , cmd(cmd) - , v(allocRootValue(v)) + , v(RootValue(v)) , attrPath(attrPath) , extendedOutputsSpec(std::move(extendedOutputsSpec)) { @@ -78,7 +78,8 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths() } DerivedPathsWithInfo res; - for (auto & [drvPath, outputs] : byDrvPath) + for (auto & [drvPath, outputs] : byDrvPath) { + state->waitForPath(drvPath); res.push_back({ .path = DerivedPath::Built{ @@ -91,6 +92,7 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths() so we can fill in this info. */ }), }); + } return res; } diff --git a/src/libcmd/installable-flake.cc b/src/libcmd/installable-flake.cc index 5517f2d9741e..6c65fad10fd0 100644 --- a/src/libcmd/installable-flake.cc +++ b/src/libcmd/installable-flake.cc @@ -1,3 +1,4 @@ +#include "nix/store/globals.hh" #include "nix/cmd/installable-flake.hh" #include "nix/store/outputs-spec.hh" #include "nix/util/util.hh" @@ -9,37 +10,21 @@ #include "nix/expr/eval-error.hh" #include "nix/flake/flake.hh" #include "nix/expr/eval-cache.hh" +#include "nix/flake/provenance.hh" +#include "nix/cmd/flake-schemas.hh" #include namespace nix { -std::vector InstallableFlake::getActualAttrPaths() -{ - std::vector res; - if (attrPaths.size() == 1 && attrPaths.front().starts_with(".")) { - attrPaths.front().erase(0, 1); - res.push_back(attrPaths.front()); - return res; - } - - for (auto & prefix : prefixes) - res.push_back(prefix + *attrPaths.begin()); - - for (auto & s : attrPaths) - res.push_back(s); - - return res; -} - -static std::string showAttrPaths(const std::vector & paths) +static std::string showAttrPaths(EvalState & state, const std::vector & paths) { std::string s; for (const auto & [n, i] : enumerate(paths)) { if (n > 0) s += n + 1 == paths.size() ? " or " : ", "; s += '\''; - s += i; + s += i.to_string(state); s += '\''; } return s; @@ -51,15 +36,17 @@ InstallableFlake::InstallableFlake( FlakeRef && flakeRef, std::string_view fragment, ExtendedOutputsSpec extendedOutputsSpec, - Strings attrPaths, - Strings prefixes, - const flake::LockFlags & lockFlags) + StringSet roles, + const flake::LockFlags & lockFlags, + std::optional defaultFlakeSchemas) : InstallableValue(state) , flakeRef(flakeRef) - , attrPaths(fragment == "" ? attrPaths : Strings{(std::string) fragment}) - , prefixes(fragment == "" ? Strings{} : prefixes) + , fragment(fragment) + , parsedFragment(AttrPath::parse(*state, fragment)) + , roles(roles) , extendedOutputsSpec(std::move(extendedOutputsSpec)) , lockFlags(lockFlags) + , defaultFlakeSchemas(defaultFlakeSchemas) { if (cmd && cmd->getAutoArgs(*state)->size()) throw UsageError("'--arg' and '--argstr' are incompatible with flakes"); @@ -67,12 +54,22 @@ InstallableFlake::InstallableFlake( DerivedPathsWithInfo InstallableFlake::toDerivedPaths() { - Activity act(*logger, lvlTalkative, actUnknown, fmt("evaluating derivation '%s'", what())); + Activity act( + *logger, + lvlTalkative, + "EvaluateFlakeDerivationOutput", + std::to_array>({ + {"nix.installable", what()}, + }), + fmt("evaluating derivation '%s'", what())); + PushActivity pact(act.id); auto attr = getCursor(*state); auto attrPath = attr->getAttrPathStr(); + PushProvenance pushedProvenance(*state, makeProvenance(attrPath)); + if (!attr->isDerivation()) { // FIXME: use eval cache? @@ -91,6 +88,7 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths() } auto drvPath = attr->forceDerivation(); + state->waitForPath(drvPath); std::optional priority; @@ -146,39 +144,174 @@ std::pair InstallableFlake::toValue(EvalState & state) return {&getCursor(state)->forceValue(), noPos}; } -std::vector> InstallableFlake::getCursors(EvalState & state) +std::vector InstallableFlake::getAttrPaths(bool useDefaultAttrPath, ref inventory) +{ + if (fragment.starts_with(".")) + return {AttrPath::parse(*state, fragment.substr(1))}; + + std::vector attrPaths; + + auto schemas = flake_schemas::getSchemas(inventory); + + // FIXME: Ugly hack to preserve the historical precedence + // between outputs. We should add a way for schemas to declare + // priorities. + std::vector schemasSorted; + std::set schemasSeen; + auto doSchema = [&](const std::string & schema) { + if (schemas.contains(schema)) { + schemasSorted.push_back(schema); + schemasSeen.insert(schema); + } + }; + doSchema("apps"); + doSchema("defaultApp"); + doSchema("devShells"); + doSchema("devShell"); + doSchema("packages"); + doSchema("defaultPackage"); + doSchema("legacyPackages"); + for (auto & schema : schemas) + if (!schemasSeen.contains(schema.first)) + schemasSorted.push_back(schema.first); + + for (auto & role : roles) { + for (auto & schemaName : schemasSorted) { + auto & schema = schemas.find(schemaName)->second; + if (schema.roles.contains(role)) { + AttrPath attrPath{state->symbols.create(schemaName)}; + if (schema.appendSystem) + attrPath.push_back(state->symbols.create(settings.thisSystem.get())); + + if (useDefaultAttrPath && parsedFragment.empty()) { + if (schema.defaultAttrPath) { + auto attrPath2{attrPath}; + for (auto & x : *schema.defaultAttrPath) + attrPath2.push_back(x); + attrPaths.push_back(attrPath2); + } + } else { + auto attrPath2{attrPath}; + for (auto & x : parsedFragment) + attrPath2.push_back(x); + attrPaths.push_back(attrPath2); + } + } + } + } + + if (!parsedFragment.empty()) + attrPaths.push_back(parsedFragment); + + // FIXME: compatibility hack to get `nix repl` to return all + // outputs by default. + if (parsedFragment.empty() && roles.contains("nix-repl")) + attrPaths.push_back({}); + + return attrPaths; +} + +std::vector> InstallableFlake::getCursors(EvalState & state, bool useDefaultAttrPath) { - auto evalCache = openEvalCache(state, getLockedFlake()); + auto cache = openEvalCache(); + + auto inventory = cache->getRoot()->getAttr("inventory"); + auto outputs = cache->getRoot()->getAttr("outputs"); - auto root = evalCache->getRoot(); + auto attrPaths = getAttrPaths(useDefaultAttrPath, inventory); + + if (attrPaths.empty()) + throw Error( + "Flake '%s' does not have any schema that provides a default output for the role(s) %s.", + flakeRef, + concatStringsSep(", ", roles)); std::vector> res; Suggestions suggestions; - auto attrPaths = getActualAttrPaths(); for (auto & attrPath : attrPaths) { - debug("trying flake output attribute '%s'", attrPath); + debug("trying flake output attribute '%s'", attrPath.to_string(state)); + + PushProvenance pushedProvenance(state, makeProvenance(attrPath.to_string(state))); + +#if 0 + auto outputInfo = flake_schemas::getOutputInfo(inventory, attrPath); + + if (outputInfo && outputInfo->leafAttrPath.empty()) { + if (auto drv = outputInfo->nodeInfo->maybeGetAttr("derivation")) { + res.push_back(ref(drv)); + continue; + } + } +#endif try { - auto attr = root->findAlongAttrPath(AttrPath::parse(state, attrPath)); - if (attr) { + auto attr = outputs->findAlongAttrPath(attrPath); + if (attr) res.push_back(ref(*attr)); - } else { + else suggestions += attr.getSuggestions(); - } } catch (TypeError & e) { - debug("error resolving attribute '%s': %s", attrPath, e.msg()); + debug("error resolving attribute '%s': %s", attrPath.to_string(state), e.msg()); // Continue to next attribute path } } if (res.size() == 0) - throw Error(suggestions, "flake '%s' does not provide attribute %s", flakeRef, showAttrPaths(attrPaths)); + throw Error(suggestions, "flake '%s' does not provide attribute %s", flakeRef, showAttrPaths(state, attrPaths)); return res; } +void InstallableFlake::getCompletions(const std::string & flakeRefS, AddCompletions & completions) +{ + auto cache = openEvalCache(); + + auto inventory = cache->getRoot()->getAttr("inventory"); + auto outputs = cache->getRoot()->getAttr("outputs"); + + if (fragment.ends_with(".") || fragment.empty()) + // Represent that we're looking for attributes starting with the empty prefix (i.e. all attributes inside the + // parent. + parsedFragment.push_back(state->symbols.create("")); + + auto attrPaths = getAttrPaths(true, inventory); + + if (fragment.empty()) + // Return all top-level flake outputs. + attrPaths.push_back(AttrPath{state->symbols.create("")}); + + auto lastAttr = fragment.ends_with(".") || parsedFragment.empty() ? std::string_view("") + : state->symbols[parsedFragment.back()]; + std::string prefix; + if (auto dot = fragment.rfind('.'); dot != std::string::npos) + prefix = fragment.substr(0, dot); + if (fragment.starts_with(".") && !prefix.starts_with(".")) + prefix = "." + prefix; + + for (auto attrPath : attrPaths) { + if (attrPath.empty()) + attrPath.push_back(state->symbols.create("")); + + auto attrPathParent{attrPath}; + attrPathParent.pop_back(); + + auto attr = outputs->findAlongAttrPath(attrPathParent); + if (!attr) + continue; + + for (auto & childName : (*attr)->getAttrs()) { + if (hasPrefix(state->symbols[childName], lastAttr)) { + auto attrPathChild = (*attr)->getAttrPath(childName); + completions.add( + flakeRefS + "#" + prefix + (prefix.empty() || prefix.ends_with(".") ? "" : ".") + + state->symbols[childName]); + } + } + } +} + ref InstallableFlake::getLockedFlake() const { if (!_lockedFlake) { @@ -191,6 +324,14 @@ ref InstallableFlake::getLockedFlake() const return ref(_lockedFlake); } +ref InstallableFlake::openEvalCache() const +{ + if (!_evalCache) { + _evalCache = flake_schemas::call(*state, getLockedFlake(), defaultFlakeSchemas, useEvalCache); + } + return ref(_evalCache); +} + FlakeRef InstallableFlake::nixpkgsFlakeRef() const { auto lockedFlake = getLockedFlake(); @@ -207,4 +348,12 @@ FlakeRef InstallableFlake::nixpkgsFlakeRef() const return defaultNixpkgsFlakeRef(); } +std::shared_ptr InstallableFlake::makeProvenance(std::string_view attrPath) const +{ + auto provenance = getLockedFlake()->flake.provenance; + if (!provenance) + return nullptr; + return std::make_shared(provenance, std::string(attrPath), evalSettings.pureEval); +} + } // namespace nix diff --git a/src/libcmd/installable-value.cc b/src/libcmd/installable-value.cc index 92811c1d01da..8e19594c7bb4 100644 --- a/src/libcmd/installable-value.cc +++ b/src/libcmd/installable-value.cc @@ -4,7 +4,7 @@ namespace nix { -std::vector> InstallableValue::getCursors(EvalState & state) +std::vector> InstallableValue::getCursors(EvalState & state, bool useDefaultAttrPath) { auto evalCache = std::make_shared(std::nullopt, state, [&]() { return toValue(state).first; }); @@ -15,7 +15,7 @@ ref InstallableValue::getCursor(EvalState & state) { /* Although getCursors should return at least one element, in case it doesn't, bound check to avoid an undefined behavior for vector[0] */ - return getCursors(state).at(0); + return getCursors(state, true).at(0); } static UsageError nonValueInstallable(Installable & installable) @@ -54,11 +54,12 @@ InstallableValue::trySinglePathToDerivedPaths(Value & v, const PosIdx pos, std:: } else if (v.type() == nString) { - auto path = state->coerceToSingleDerivedPath(pos, v, errorCtx); - if (auto o = std::get_if(&path.raw())) - state->ensureLazyPathCopied(o->path); + auto path = state->devirtualize(state->coerceToSingleDerivedPath(pos, v, errorCtx)); + /* The path may still be being written asynchronously (e.g. by + `builtins.toFile`). */ + state->waitForPath(path); return {{ - .path = DerivedPath::fromSingle(path), + .path = DerivedPath::fromSingle(std::move(path)), .info = make_ref(), }}; } diff --git a/src/libcmd/installables.cc b/src/libcmd/installables.cc index 82edbfca0ee2..f2edad3a277c 100644 --- a/src/libcmd/installables.cc +++ b/src/libcmd/installables.cc @@ -20,6 +20,7 @@ #include "nix/expr/eval-cache.hh" #include "nix/fetchers/registry.hh" #include "nix/store/build-result.hh" +#include "nix/util/exit.hh" #include @@ -237,19 +238,9 @@ MixReadOnlyOption::MixReadOnlyOption() }); } -Strings SourceExprCommand::getDefaultFlakeAttrPaths() +StringSet SourceExprCommand::getRoles() { - return {"packages." + settings.thisSystem.get() + ".default", "defaultPackage." + settings.thisSystem.get()}; -} - -Strings SourceExprCommand::getDefaultFlakeAttrPathPrefixes() -{ - return {// As a convenience, look for the attribute in - // 'outputs.packages'. - "packages." + settings.thisSystem.get() + ".", - // As a temporary hack until Nixpkgs is properly converted - // to provide a clean 'packages' set, look in 'legacyPackages'. - "legacyPackages." + settings.thisSystem.get() + "."}; + return {"nix-build"}; } Args::CompleterClosure SourceExprCommand::getCompleteInstallable() @@ -303,13 +294,7 @@ void SourceExprCommand::completeInstallable(AddCompletions & completions, std::s } } } else { - completeFlakeRefWithFragment( - completions, - getEvalState(), - lockFlags, - getDefaultFlakeAttrPathPrefixes(), - getDefaultFlakeAttrPaths(), - prefix); + completeFlakeRefWithFragment(completions, getEvalState(), lockFlags, getRoles(), prefix); } } catch (EvalError &) { // Don't want eval errors to mess-up with the completion engine, so let's just swallow them @@ -320,91 +305,37 @@ void completeFlakeRefWithFragment( AddCompletions & completions, ref evalState, flake::LockFlags lockFlags, - Strings attrPathPrefixes, - const Strings & defaultFlakeAttrPaths, + const StringSet & roles, std::string_view prefix) -{ - /* Look for flake output attributes that match the - prefix. */ - try { - auto hash = prefix.find('#'); - if (hash == std::string::npos) { - completeFlakeRef(completions, evalState->store, prefix); - } else { - completions.setType(AddCompletions::Type::Attrs); - - auto fragment = prefix.substr(hash + 1); - std::string prefixRoot = ""; - if (fragment.starts_with(".")) { - fragment = fragment.substr(1); - prefixRoot = "."; - } - auto flakeRefS = std::string(prefix.substr(0, hash)); - - // TODO: ideally this would use the command base directory instead of assuming ".". - auto flakeRef = - parseFlakeRef(fetchSettings, expandTilde(flakeRefS), std::filesystem::current_path().string()); - - auto evalCache = openEvalCache( - *evalState, make_ref(lockFlake(flakeSettings, *evalState, flakeRef, lockFlags))); - - auto root = evalCache->getRoot(); - - if (prefixRoot == ".") { - attrPathPrefixes.clear(); - } - /* Complete 'fragment' relative to all the - attrpath prefixes as well as the root of the - flake. */ - attrPathPrefixes.push_back(""); - - for (auto & attrPathPrefixS : attrPathPrefixes) { - auto attrPathPrefix = AttrPath::parse(*evalState, attrPathPrefixS); - auto attrPathS = attrPathPrefixS + std::string(fragment); - auto attrPath = AttrPath::parse(*evalState, attrPathS); - - std::string lastAttr; - if (!attrPath.empty() && !hasSuffix(attrPathS, ".")) { - lastAttr = evalState->symbols[attrPath.back()]; - attrPath.pop_back(); - } +try { + auto hash = prefix.find('#'); + if (hash == std::string::npos) { + completeFlakeRef(completions, evalState->store, prefix); + return; + } - auto attr = root->findAlongAttrPath(attrPath); - if (!attr) - continue; + completions.setType(AddCompletions::Type::Attrs); - for (auto & attr2 : (*attr)->getAttrs()) { - if (hasPrefix(evalState->symbols[attr2], lastAttr)) { - auto attrPath2 = (*attr)->getAttrPath(attr2); - /* Strip the attrpath prefix. */ - attrPath2.erase(attrPath2.begin(), attrPath2.begin() + attrPathPrefix.size()); - // FIXME: handle names with dots - completions.add(flakeRefS + "#" + prefixRoot + attrPath2.to_string(*evalState)); - } - } - } + auto fragment = prefix.substr(hash + 1); + auto flakeRefS = std::string(prefix.substr(0, hash)); - /* And add an empty completion for the default - attrpaths. */ - if (fragment.empty()) { - for (auto & attrPath : defaultFlakeAttrPaths) { - auto attr = root->findAlongAttrPath(AttrPath::parse(*evalState, attrPath)); - if (!attr) - continue; - completions.add(flakeRefS + "#" + prefixRoot); - } - } - } - } catch (Error & e) { - logWarning(e.info()); - } + InstallableFlake{ + nullptr, + evalState, + // TODO: ideally this would use the command base directory instead of assuming ".". + parseFlakeRef(fetchSettings, expandTilde(flakeRefS), std::filesystem::current_path().string()), + fragment, + ExtendedOutputsSpec::Default{}, // FIXME: could be that we're completing the outputs spec... + roles, + lockFlags, + {}} + .getCompletions(flakeRefS, completions); +} catch (Error & e) { + logWarning(e.info()); } void completeFlakeRef(AddCompletions & completions, ref store, std::string_view prefix) { - if (!experimentalFeatureSettings.isEnabled(Xp::Flakes)) - return; - if (prefix == "") completions.add("."); @@ -514,9 +445,9 @@ Installables SourceExprCommand::parseInstallables(ref store, std::vector< std::move(flakeRef), fragment, std::move(extendedOutputsSpec), - getDefaultFlakeAttrPaths(), - getDefaultFlakeAttrPathPrefixes(), - lockFlags)); + getRoles(), + lockFlags, + getDefaultFlakeSchemas())); continue; } catch (...) { ex = std::current_exception(); @@ -536,64 +467,99 @@ ref SourceExprCommand::parseInstallable(ref store, const std return installables.front(); } -std::vector Installable::build( - ref evalStore, ref store, Realise mode, const Installables & installables, BuildMode bMode) +const BuiltPathWithResult & InstallableWithBuildResult::getSuccess() const { - std::vector res; - for (auto & [_, builtPathWithResult] : build2(evalStore, store, mode, installables, bMode)) - res.push_back(builtPathWithResult); - return res; + if (auto * failure = std::get_if(&result)) { + auto failure2 = failure->tryGetFailure(); + assert(failure2); + throw *failure2; + } else + return *std::get_if(&result); } -static void throwBuildErrors(std::vector & buildResults, const Store & store) +void Installable::throwBuildErrors(std::vector & buildResults, const Store & store) { - std::vector> failed; for (auto & buildResult : buildResults) { - if (auto * failure = buildResult.tryGetFailure()) { - failed.push_back({&buildResult, failure}); - } - } + if (std::get_if(&buildResult.result)) { + // Report success first. + for (auto & buildResult : buildResults) { + if (std::get_if(&buildResult.result)) + notice("✅ " ANSI_BOLD "%s" ANSI_NORMAL, buildResult.installable->what()); + } - auto failedResult = failed.begin(); - if (failedResult != failed.end()) { - if (failed.size() == 1) { - throw *failedResult->second; - } else { - StringSet failedPaths; - for (; failedResult != failed.end(); failedResult++) { - if (!failedResult->second->message().empty()) { - logError(failedResult->second->info()); + // Then cancelled builds. + for (auto & buildResult : buildResults) { + if (auto failure = std::get_if(&buildResult.result)) { + if (failure->isCancelled()) + notice( + "❓ " ANSI_BOLD "%s" ANSI_NORMAL ANSI_FAINT " (cancelled)", + buildResult.installable->what()); + } + } + + // Then failures. + for (auto & buildResult : buildResults) { + if (auto failure = std::get_if(&buildResult.result)) { + if (failure->isCancelled()) + continue; + auto failure2 = failure->tryGetFailure(); + assert(failure2); + printError("❌ " ANSI_RED "%s" ANSI_NORMAL, buildResult.installable->what()); + try { + throw *failure2; + } catch (Error & e) { + logError(e.info()); + } } - failedPaths.insert(failedResult->first->path.to_string(store)); } - throw Error("build of %s failed", concatStringsSep(", ", quoteStrings(failedPaths))); + + throw Exit(1); } } } -std::vector, BuiltPathWithResult>> Installable::build2( +std::vector Installable::build( + ref evalStore, ref store, Realise mode, const Installables & installables, BuildMode bMode) +{ + auto results = build2(evalStore, store, mode, installables, bMode); + throwBuildErrors(results, *store); + std::vector res; + for (auto & b : results) + res.push_back(b.getSuccess()); + return res; +} + +std::vector Installable::build2( ref evalStore, ref store, Realise mode, const Installables & installables, BuildMode bMode) { if (mode == Realise::Nothing) settings.readOnlyMode = true; - struct Aux + struct Request { + DerivedPath path; ref info; ref installable; }; + /* All requests in command-line order, which the results must + preserve; pathsToBuild holds each distinct derived path once. */ + std::vector requests; std::vector pathsToBuild; - std::map> backmap; + std::set seen; for (auto & i : installables) { for (auto b : i->toDerivedPaths()) { - pathsToBuild.push_back(b.path); - backmap[b.path].push_back({.info = b.info, .installable = i}); + if (seen.insert(b.path).second) + pathsToBuild.push_back(b.path); + requests.push_back({.path = b.path, .info = b.info, .installable = i}); } } - std::vector, BuiltPathWithResult>> res; + Activity act(*logger, lvlTalkative, "BuildInstallables", {}); + PushActivity pact(act.id); + + std::vector res; switch (mode) { @@ -601,28 +567,30 @@ std::vector, BuiltPathWithResult>> Installable::build case Realise::Derivation: printMissing(store, pathsToBuild, lvlError); - for (auto & path : pathsToBuild) { - for (auto & aux : backmap[path]) { - std::visit( - overloaded{ - [&](const DerivedPath::Built & bfd) { - auto outputs = resolveDerivedPath(*store, bfd, &*evalStore); - res.push_back( - {aux.installable, - {.path = - BuiltPath::Built{ - .drvPath = - make_ref(getBuiltPath(evalStore, store, *bfd.drvPath)), - .outputs = outputs, - }, - .info = aux.info}}); - }, - [&](const DerivedPath::Opaque & bo) { - res.push_back({aux.installable, {.path = BuiltPath::Opaque{bo.path}, .info = aux.info}}); - }, + for (auto & req : requests) { + std::visit( + overloaded{ + [&](const DerivedPath::Built & bfd) { + auto outputs = resolveDerivedPath(*store, bfd, &*evalStore); + res.push_back( + {.installable = req.installable, + .result = InstallableWithBuildResult::Success{ + .path = + BuiltPath::Built{ + .drvPath = + make_ref(getBuiltPath(evalStore, store, *bfd.drvPath)), + .outputs = outputs, + }, + .info = req.info}}); }, - path.raw()); - } + [&](const DerivedPath::Opaque & bo) { + res.push_back( + {.installable = req.installable, + .result = InstallableWithBuildResult::Success{ + .path = BuiltPath::Opaque{bo.path}, .info = req.info}}); + }, + }, + req.path.raw()); } break; @@ -632,18 +600,25 @@ std::vector, BuiltPathWithResult>> Installable::build printMissing(store, pathsToBuild, lvlInfo); auto buildResults = store->buildPathsWithResults(pathsToBuild, bMode, evalStore); - throwBuildErrors(buildResults, *store); - for (auto & buildResult : buildResults) { - for (auto & aux : backmap[buildResult.path]) { - res.push_back({ - aux.installable, - BuiltPathWithResult{ - .path = toBuiltPath(buildResult, evalStore, store), - .info = aux.info, - .result = buildResult, - }, - }); + + std::map resultsByPath; + for (auto & buildResult : buildResults) + resultsByPath.emplace(buildResult.path, &buildResult); + + for (auto & req : requests) { + auto & buildResult = *resultsByPath.at(req.path); + if (buildResult.tryGetFailure()) { + res.push_back({.installable = req.installable, .result = buildResult}); + continue; } + res.push_back({ + req.installable, + BuiltPathWithResult{ + .path = toBuiltPath(buildResult, evalStore, store), + .info = req.info, + .result = buildResult, + }, + }); } break; @@ -800,8 +775,11 @@ InstallableCommand::InstallableCommand() }); } +void InstallableCommand::preRun(ref store) {} + void InstallableCommand::run(ref store) { + preRun(store); auto installable = parseInstallable(store, _installable); run(store, std::move(installable)); } diff --git a/src/libcmd/make.nix b/src/libcmd/make.nix new file mode 100644 index 000000000000..2b36eb79532c --- /dev/null +++ b/src/libcmd/make.nix @@ -0,0 +1,49 @@ +# libnixcmd, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-store, + nix-fetchers, + nix-expr, + nix-flake, + nix-main, +}: + +nixMake.mkComponent { + name = "determinate-nix-cmd"; + libName = "nixcmd"; + + deps = [ + nix-util + nix-store + nix-fetchers + nix-expr + nix-flake + nix-main + ]; + + root = ./.; + + # unix/unix-socket-server.cc is a plain source file; there is no windows/ directory. + includeDirs = [ + "" + "include" + ]; + + files = nixMake.commonSupportFiles // { + "call-flake-schemas.nix.gen.hh" = nixMake.mkStringHeader ./call-flake-schemas.nix; + "builtin-flake-schemas/flake.nix.gen.hh" = nixMake.mkStringHeader ./builtin-flake-schemas/flake.nix; + }; + + configHeaders = { + "cmd-config-private.hh" = { + HAVE_LOWDOWN = 1; + HAVE_LOWDOWN_1_4 = 1; + HAVE_LOWDOWN_3 = 1; + # Use editline rather than readline. + USE_READLINE = 0; + }; + }; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libcmd/meson.build b/src/libcmd/meson.build index 9638b491f2c4..15567ae77c58 100644 --- a/src/libcmd/meson.build +++ b/src/libcmd/meson.build @@ -71,6 +71,7 @@ config_priv_h = configure_file( ) subdir('nix-meson-build-support/common') +subdir('nix-meson-build-support/generate-header') sources = files( 'built-path.cc', @@ -78,6 +79,7 @@ sources = files( 'command.cc', 'common-eval-args.cc', 'editor-for.cc', + 'flake-schemas.cc', 'get-build-log.cc', 'installable-attr-path.cc', 'installable-derived-path.cc', @@ -97,6 +99,14 @@ if host_machine.system() != 'windows' ) endif +sources += [ + gen_header.process('call-flake-schemas.nix'), + gen_header.process( + 'builtin-flake-schemas/flake.nix', + preserve_path_from : meson.current_source_dir(), + ), +] + subdir('include/nix/cmd') subdir('nix-meson-build-support/export-all-symbols') @@ -106,11 +116,10 @@ this_library = library( 'nixcmd', sources, config_priv_h, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, link_args : linker_export_flags, - prelink : true, # For C++ static initializers + prelink : prelink, # For C++ static initializers install : true, cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [], ) diff --git a/src/libcmd/package.nix b/src/libcmd/package.nix index c382f0e5760d..ccd0ca61e9e0 100644 --- a/src/libcmd/package.nix +++ b/src/libcmd/package.nix @@ -35,7 +35,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-cmd"; + pname = "determinate-nix-cmd"; inherit version; workDir = ./.; @@ -49,6 +49,8 @@ mkMesonLibrary (finalAttrs: { ./include/nix/cmd/meson.build (fileset.fileFilter (file: file.hasExt "cc") ./.) (fileset.fileFilter (file: file.hasExt "hh") ./.) + ./call-flake-schemas.nix + ./builtin-flake-schemas/flake.nix ]; buildInputs = [ diff --git a/src/libcmd/repl.cc b/src/libcmd/repl.cc index c6523fe57389..6eaab847a0a6 100644 --- a/src/libcmd/repl.cc +++ b/src/libcmd/repl.cc @@ -189,7 +189,7 @@ ReplExitStatus NixRepl::mainLoop() if (state->debugRepl) { debuggerNotice = " debugger"; } - notice("Nix %1%%2%\nType :? for help.", nixVersion, debuggerNotice); + notice("Nix %1%%2%\nType :? for help.", version(), debuggerNotice); } isFirstRepl = false; @@ -329,6 +329,7 @@ StorePath NixRepl::getDerivationPath(Value & v) auto drvPath = packageInfo->queryDrvPath(); if (!drvPath) throw Error("expression did not evaluate to a valid derivation (no 'drvPath' attribute)"); + state->waitForPath(*drvPath); if (!state->store->isValidPath(*drvPath)) throw Error("expression evaluated to invalid derivation '%s'", state->store->printStorePath(*drvPath)); return *drvPath; @@ -344,7 +345,7 @@ void NixRepl::loadDebugTraceEnv(DebugTrace & dt) // add staticenv vars. for (auto & [name, value] : *(vm.get())) - addVarToScope(state->symbols.create(name), *value); + addVarToScope(state->symbols.create(name), **value); } } @@ -954,7 +955,7 @@ ReplExitStatus AbstractNixRepl::runSimple(ref evalState, const ValMap // add 'extra' vars. for (auto & [name, value] : extraEnv) - repl->addVarToScope(repl->state->symbols.create(name), *value); + repl->addVarToScope(repl->state->symbols.create(name), **value); return repl->mainLoop(); } diff --git a/src/libcmd/unix/unix-socket-server.cc b/src/libcmd/unix/unix-socket-server.cc index d5fbe0cc70aa..84a72b597c5a 100644 --- a/src/libcmd/unix/unix-socket-server.cc +++ b/src/libcmd/unix/unix-socket-server.cc @@ -122,7 +122,6 @@ PeerInfo getPeerInfo(Descriptor remote) socklen_t remoteAddrLen = sizeof(remoteAddr); AutoCloseFD remote = accept(fd.fd, (struct sockaddr *) &remoteAddr, &remoteAddrLen); - checkInterrupt(); if (!remote) { if (errno == EINTR) continue; diff --git a/src/libexpr-c/make.nix b/src/libexpr-c/make.nix new file mode 100644 index 000000000000..29758f322ad4 --- /dev/null +++ b/src/libexpr-c/make.nix @@ -0,0 +1,34 @@ +# libnixexprc, the C API of libnixexpr, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-util-c, + nix-store, + nix-store-c, + nix-expr, +}: + +nixMake.mkComponent { + name = "determinate-nix-expr-c"; + libName = "nixexprc"; + + deps = [ + nix-util + nix-util-c + nix-store + nix-store-c + nix-expr + ]; + + root = ./.; + + includeDirs = [ "" ]; + # The headers live in the root, which the default would not export. + publicIncludeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + extraCxxFlags = nixMake.weakVtablesFlags; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libexpr-c/meson.build b/src/libexpr-c/meson.build index 5c6dc6d666ad..90435b4095bb 100644 --- a/src/libexpr-c/meson.build +++ b/src/libexpr-c/meson.build @@ -51,7 +51,6 @@ subdir('nix-meson-build-support/windows-version') build_both_libraries = get_option('plugin-c-api') library_kwargs = { - 'soversion' : nix_soversion, 'dependencies' : deps_public + deps_private + deps_other, 'include_directories' : include_dirs, 'link_args' : linker_export_flags, diff --git a/src/libexpr-c/nix_api_expr.cc b/src/libexpr-c/nix_api_expr.cc index 2387ee8c38c2..d78c504bd8c7 100644 --- a/src/libexpr-c/nix_api_expr.cc +++ b/src/libexpr-c/nix_api_expr.cc @@ -50,6 +50,7 @@ nix_err nix_expr_eval_from_string( nix::Expr * parsedExpr = state->state.parseExprFromString(expr, state->state.rootPath(nix::CanonPath(path))); state->state.eval(parsedExpr, *value->value); state->state.forceValue(*value->value, nix::noPos); + state->state.waitForAllPaths(); } NIXC_CATCH_ERRS } @@ -61,6 +62,7 @@ nix_err nix_value_call(nix_c_context * context, EvalState * state, Value * fn, n try { state->state.callFunction(*fn->value, *arg->value, *value->value, nix::noPos); state->state.forceValue(*value->value, nix::noPos); + state->state.waitForAllPaths(); } NIXC_CATCH_ERRS } @@ -79,6 +81,7 @@ nix_err nix_value_call_multi( try { state->state.callFunction(*fn->value, {internal_args.data(), nargs}, *value->value, nix::noPos); state->state.forceValue(*value->value, nix::noPos); + state->state.waitForAllPaths(); } NIXC_CATCH_ERRS } @@ -89,6 +92,7 @@ nix_err nix_value_force(nix_c_context * context, EvalState * state, nix_value * context->last_err_code = NIX_OK; try { state->state.forceValue(*value->value, nix::noPos); + state->state.waitForAllPaths(); } NIXC_CATCH_ERRS } @@ -99,6 +103,7 @@ nix_err nix_value_force_deep(nix_c_context * context, EvalState * state, nix_val context->last_err_code = NIX_OK; try { state->state.forceValueDeep(*value->value); + state->state.waitForAllPaths(); } NIXC_CATCH_ERRS } diff --git a/src/libexpr-c/nix_api_value.cc b/src/libexpr-c/nix_api_value.cc index b52dab702449..e2e3e9697f4b 100644 --- a/src/libexpr-c/nix_api_value.cc +++ b/src/libexpr-c/nix_api_value.cc @@ -392,6 +392,7 @@ nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value auto attr = v.attrs()->get(s); if (attr) { state->state.forceValue(*attr->value, nix::noPos); + state->state.waitForAllPaths(); return new_nix_value(attr->value, state->state.mem); } nix_set_err_msg(context, NIX_ERR_KEY, "missing attribute"); diff --git a/src/libexpr-c/nix_api_value.h b/src/libexpr-c/nix_api_value.h index 0220bf68e2df..a01bfb280599 100644 --- a/src/libexpr-c/nix_api_value.h +++ b/src/libexpr-c/nix_api_value.h @@ -101,6 +101,8 @@ typedef enum { * @see Externals */ NIX_TYPE_EXTERNAL, + /** @brief Failed value. Contains an exception that can be rethrown. + */ NIX_TYPE_FAILED, } ValueType; diff --git a/src/libexpr-c/package.nix b/src/libexpr-c/package.nix index 9a7a46c1663d..937d7d3911c4 100644 --- a/src/libexpr-c/package.nix +++ b/src/libexpr-c/package.nix @@ -16,7 +16,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-expr-c"; + pname = "determinate-nix-expr-c"; inherit version; workDir = ./.; diff --git a/src/libexpr-test-support/include/nix/expr/tests/value/context.hh b/src/libexpr-test-support/include/nix/expr/tests/value/context.hh index b8f4b4f67acf..ed0eba94c30d 100644 --- a/src/libexpr-test-support/include/nix/expr/tests/value/context.hh +++ b/src/libexpr-test-support/include/nix/expr/tests/value/context.hh @@ -14,6 +14,12 @@ struct Arbitrary static Gen arbitrary(); }; +template<> +struct Arbitrary +{ + static Gen arbitrary(); +}; + template<> struct Arbitrary { diff --git a/src/libexpr-test-support/make.nix b/src/libexpr-test-support/make.nix new file mode 100644 index 000000000000..1ecbb726f00c --- /dev/null +++ b/src/libexpr-test-support/make.nix @@ -0,0 +1,33 @@ +# libnix-expr-test-support, helpers for the unit tests of libnixexpr and +# the components built on it, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-util-test-support, + nix-store, + nix-store-test-support, + nix-expr, + nix-expr-c, +}: + +nixMake.mkComponent { + name = "determinate-nix-expr-test-support"; + libName = "nix-expr-test-support"; + + deps = [ + nix-util + nix-util-test-support + nix-store + nix-store-test-support + nix-expr + nix-expr-c + ]; + + root = ./.; + + includeDirs = [ "include" ]; + + files = nixMake.commonSupportFiles; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libexpr-test-support/meson.build b/src/libexpr-test-support/meson.build index 4a87bb4545fd..7d6f871d7acd 100644 --- a/src/libexpr-test-support/meson.build +++ b/src/libexpr-test-support/meson.build @@ -44,13 +44,12 @@ subdir('nix-meson-build-support/windows-version') this_library = library( 'nix-expr-test-support', sources, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, # TODO: Remove `-lrapidcheck` when https://github.com/emil-e/rapidcheck/pull/326 # is available. See also ../libutil/build.meson link_args : linker_export_flags + [ '-lrapidcheck' ], - prelink : true, # For C++ static initializers + prelink : prelink, # For C++ static initializers install : true, ) diff --git a/src/libexpr-test-support/package.nix b/src/libexpr-test-support/package.nix index 5cb4adaa8c46..1879a5716082 100644 --- a/src/libexpr-test-support/package.nix +++ b/src/libexpr-test-support/package.nix @@ -18,7 +18,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-util-test-support"; + pname = "determinate-nix-util-test-support"; inherit version; workDir = ./.; diff --git a/src/libexpr-test-support/tests/value/context.cc b/src/libexpr-test-support/tests/value/context.cc index 22f8aa7cf0ff..f82bd1c552e8 100644 --- a/src/libexpr-test-support/tests/value/context.cc +++ b/src/libexpr-test-support/tests/value/context.cc @@ -15,6 +15,15 @@ Gen Arbitrary::arb }); } +Gen Arbitrary::arbitrary() +{ + return gen::map(gen::arbitrary(), [](StorePath storePath) { + return NixStringContextElem::Path{ + .storePath = storePath, + }; + }); +} + Gen Arbitrary::arbitrary() { return gen::mapcat( @@ -30,6 +39,8 @@ Gen Arbitrary::arbitrary() case 2: return gen::map( gen::arbitrary(), [](NixStringContextElem a) { return a; }); + case 3: + return gen::map(gen::arbitrary(), [](NixStringContextElem a) { return a; }); default: assert(false); } diff --git a/src/libexpr-tests/coroutine-gc.cc b/src/libexpr-tests/coroutine-gc.cc new file mode 100644 index 000000000000..413f7555bfc6 --- /dev/null +++ b/src/libexpr-tests/coroutine-gc.cc @@ -0,0 +1,61 @@ +#include "nix/expr/tests/libexpr.hh" +#include "nix/expr/eval-gc.hh" +#include "nix/util/serialise.hh" + +namespace nix { + +class CoroutineGCTest : public LibExprTest +{}; + +/* Regression test for a crash in the garbage collector: when the + body of a `sinkToSource` coroutine streams its output through a + `sourceToSink` coroutine (as `copyStorePath()` does with a + decompression sink), the outer coroutine yields from the *inner* + coroutine's stack. The GC hooks used to record such a yield as the + saved stack pointer of the outer coroutine's own stack, so a + collection would scan from the inner stack up to the base of the + outer stack, i.e. across a guard page, and crash with SIGSEGV. */ +TEST_F(CoroutineGCTest, nestedCoroutineYield) +{ +#if NIX_USE_BOEHMGC + std::string data; + for (size_t i = 0; i < 1024 * 1024; ++i) + data.push_back('a' + i % 26); + + auto source = sinkToSource([&](Sink & outerSink) { + /* Like a decompression sink: the inner coroutine's body + writes to the outer coroutine's sink, so the outer + coroutine yields on the inner coroutine's stack. */ + auto inner = sourceToSink([&](Source & src) { src.drainInto(outerSink); }); + std::string_view rest = data; + while (!rest.empty()) { + auto chunk = rest.substr(0, 100000); + (*inner)(chunk); + rest.remove_prefix(chunk.size()); + } + inner->finish(); + }); + + std::string result; + char buf[65536]; + while (true) { + size_t n; + try { + n = source->read(buf, sizeof(buf)); + } catch (EndOfFile &) { + break; + } + result.append(buf, n); + /* The outer coroutine is now suspended from the inner + coroutine's stack. Run a collection, which scans every + suspended stack from its saved stack pointer to its base. */ + GC_gcollect(); + } + + ASSERT_EQ(result, data); +#else + GTEST_SKIP() << "requires the Boehm GC"; +#endif +} + +} // namespace nix diff --git a/src/libexpr-tests/make.nix b/src/libexpr-tests/make.nix new file mode 100644 index 000000000000..9f3637748120 --- /dev/null +++ b/src/libexpr-tests/make.nix @@ -0,0 +1,60 @@ +# The unit tests of libnixexpr, transcribed from meson.build and +# package.nix: the `nix-expr-tests` executable, with the derivation that +# runs it as `tests.run`. +{ + pkgs, + nixMake, + nix-expr, + nix-expr-c, + nix-expr-test-support, +}: + +let + inherit (pkgs) lib; + + tests = nixMake.mkComponent { + name = "determinate-nix-expr-tests"; + type = "executable"; + exeName = "nix-expr-tests"; + + deps = [ + nix-expr + nix-expr-c + nix-expr-test-support + ]; + + root = ./.; + + # The benchmarks (Meson option `benchmarks`, off by default). + excludeSources = [ + "bench-main.cc" + "dynamic-attrs-bench.cc" + "get-drvs-bench.cc" + "regex-cache-bench.cc" + ]; + + includeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + configHeaders = { + "expr-tests-config.hh" = { + PACKAGE_VERSION = lib.fileContents ../../.version; + }; + }; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; + }; +in + +tests +// { + tests.run = nixMake.runTest { + name = "${tests.name}-run"; + command = [ "${tests}/bin/${tests.exeName}" ]; + env = { + _NIX_TEST_UNIT_DATA = ./data; + NIX_STORE = ""; + }; + }; +} diff --git a/src/libexpr-tests/meson.build b/src/libexpr-tests/meson.build index 50d158209ba7..cd80c1da1afd 100644 --- a/src/libexpr-tests/meson.build +++ b/src/libexpr-tests/meson.build @@ -47,6 +47,7 @@ config_priv_h = configure_file( subdir('nix-meson-build-support/common') sources = files( + 'coroutine-gc.cc', 'derived-path.cc', 'error_traces.cc', 'eval.cc', diff --git a/src/libexpr-tests/nix_api_expr.cc b/src/libexpr-tests/nix_api_expr.cc index 3ceb0a24c0df..366aca89eb42 100644 --- a/src/libexpr-tests/nix_api_expr.cc +++ b/src/libexpr-tests/nix_api_expr.cc @@ -587,6 +587,7 @@ struct DeploymentResourceState bool vm_created = false; }; +#if 0 static void primop_load_resource_input( void * user_data, nix_c_context * context, EvalState * state, nix_value ** args, nix_value * ret) { @@ -674,5 +675,6 @@ TEST_F(nix_api_expr_test, nix_expr_thunk_re_evaluation_after_deployment) assert_ctx_ok(); ASSERT_STREQ("vm-12345", result.c_str()); } +#endif } // namespace nixC diff --git a/src/libexpr-tests/primops.cc b/src/libexpr-tests/primops.cc index 08f610ced0f4..03014903fe0b 100644 --- a/src/libexpr-tests/primops.cc +++ b/src/libexpr-tests/primops.cc @@ -19,12 +19,12 @@ class CaptureLogger : public Logger return oss.str(); } - void log(Verbosity lvl, std::string_view s) override + void log(Verbosity lvl, std::string_view s) noexcept override { oss << s << std::endl; } - void logEI(const ErrorInfo & ei) override + void logEI(const ErrorInfo & ei) noexcept override { showErrorInfo(oss, ei, loggerSettings.showTrace.get()); } diff --git a/src/libexpr-tests/value/print.cc b/src/libexpr-tests/value/print.cc index 654a50b0ae0b..c9cabf3fa421 100644 --- a/src/libexpr-tests/value/print.cc +++ b/src/libexpr-tests/value/print.cc @@ -194,14 +194,14 @@ TEST_F(ValuePrintingTests, vFailed) try { throw Error("nope"); } catch (...) { - v.mkFailed(std::current_exception(), nullptr); + v.mkFailed(); } // Historically, a tried and then ignored value (e.g. through tryEval) was // reverted to the original thunk. - test(v, "«thunk»"); - test(v, ANSI_MAGENTA "«thunk»" ANSI_NORMAL, PrintOptions{.ansiColors = true}); + test(v, "«failed»"); + test(v, ANSI_MAGENTA "«failed»" ANSI_NORMAL, PrintOptions{.ansiColors = true}); } TEST_F(ValuePrintingTests, depthAttrs) diff --git a/src/libexpr-tests/value/value.cc b/src/libexpr-tests/value/value.cc index 285b586c0dfc..e22e70f6e336 100644 --- a/src/libexpr-tests/value/value.cc +++ b/src/libexpr-tests/value/value.cc @@ -13,7 +13,6 @@ TEST_F(ValueTest, unsetValue) { Value unsetValue; ASSERT_EQ(false, unsetValue.isValid()); - ASSERT_EQ(nThunk, unsetValue.type()); } TEST_F(ValueTest, vInt) diff --git a/src/libexpr/attr-path.cc b/src/libexpr/attr-path.cc index 8c2fcd327e77..5d412282c2e3 100644 --- a/src/libexpr/attr-path.cc +++ b/src/libexpr/attr-path.cc @@ -44,9 +44,23 @@ AttrPath AttrPath::parse(EvalState & state, std::string_view s) return res; } +AttrPath AttrPath::fromStrings(EvalState & state, const std::vector & attrNames) +{ + AttrPath res; + for (auto & attrName : attrNames) + res.push_back(state.symbols.create(attrName)); + return res; +} + std::string AttrPath::to_string(EvalState & state) const { - return dropEmptyInitThenConcatStringsSep(".", state.symbols.resolve({*this})); + std::ostringstream ss; + for (const auto & [idx, attr] : enumerate(state.symbols.resolve({*this}))) { + if (idx) + ss << '.'; + printAttributeName(ss, attr); + } + return ss.str(); } std::vector AttrPath::resolve(EvalState & state) const diff --git a/src/libexpr/eval-cache.cc b/src/libexpr/eval-cache.cc index 7422cd095bc6..f70594b407ec 100644 --- a/src/libexpr/eval-cache.cc +++ b/src/libexpr/eval-cache.cc @@ -58,6 +58,9 @@ struct AttrDb SQLiteStmt insertAttributeWithContext; SQLiteStmt queryAttribute; SQLiteStmt queryAttributes; + SQLiteStmt upsertAttribute; + SQLiteStmt insertAttributeIfNotExists; + SQLiteStmt deleteMissingChildren; std::unique_ptr txn; }; @@ -92,6 +95,17 @@ struct AttrDb state->queryAttributes.create(state->db, "select name from Attributes where parent = ?"); + state->upsertAttribute.create( + state->db, + "insert into Attributes(parent, name, type, value) values (?, ?, ?, ?) " + "on conflict(parent, name) do update set type = excluded.type, value = excluded.value " + "returning rowid"); + + state->insertAttributeIfNotExists.create( + state->db, "insert or ignore into Attributes(parent, name, type, value) values (?, ?, ?, ?)"); + + state->deleteMissingChildren.create(state->db, "delete from Attributes where parent = ? and type = 3"); + state->txn = std::make_unique(state->db); } @@ -126,18 +140,20 @@ struct AttrDb return doSQLite([&]() { auto state(_state->lock()); - state->insertAttribute.use() - .apply(key.first) - .apply(symbols[key.second]) - .apply(AttrType::FullAttrs) - .apply(0, false) - .exec(); - - AttrId rowId = state->db.getLastInsertedRowId(); + // Seal the attribute names: we now know the complete set. + auto upsertAttribute(state->upsertAttribute.use() + .apply(key.first) + .apply(symbols[key.second]) + .apply(AttrType::FullAttrs) + .apply(0, false)); + upsertAttribute.next(); + auto rowId = (AttrId) upsertAttribute.getInt(0); assert(rowId); + state->deleteMissingChildren.use().apply(rowId).exec(); + // Insert children as placeholders, but don't replace existing entries for (auto & attr : attrs) - state->insertAttribute.use() + state->insertAttributeIfNotExists.use() .apply(rowId) .apply(symbols[attr]) .apply(AttrType::Placeholder) @@ -351,21 +367,49 @@ static std::shared_ptr makeAttrDb(const StoreDirConfig & cfg, const Hash } } +/** + * An expression that evaluates to the root value of an `EvalCache` by + * calling its root loader. See `EvalCache::rootLoaderExpr`. + */ +struct ExprRootLoader : Expr +{ + EvalCache::RootLoader rootLoader; + + ExprRootLoader(EvalCache::RootLoader rootLoader) + : rootLoader(std::move(rootLoader)) + { + } + + void eval(EvalState & state, Env & env, Value & v) override; + + void show(const SymbolTable & symbols, std::ostream & str) const override + { + str << ""; + } +}; + +void ExprRootLoader::eval(EvalState & state, Env & env, Value & v) +{ + debug("getting root value"); + auto res = rootLoader(); + state.forceValue(*res, noPos); + v = *res; +} + EvalCache::EvalCache( std::optional> useCache, EvalState & state, RootLoader rootLoader) : db(useCache ? makeAttrDb(*state.store, *useCache, state.symbols) : nullptr) , state(state) - , rootLoader(rootLoader) + , rootLoaderExpr(std::make_unique(std::move(rootLoader))) + , rootValue(state.allocValue()) { + (*rootValue)->mkThunk(&state.baseEnv, rootLoaderExpr.get()); } Value * EvalCache::getRootValue() { - if (!value) { - debug("getting root value"); - value = allocRootValue(rootLoader()); - } - return *value; + state.forceValue(**rootValue, noPos); + return *rootValue; } ref EvalCache::getRoot() @@ -380,7 +424,7 @@ AttrCursor::AttrCursor( , cachedValue(std::move(cachedValue)) { if (value) - _value = allocRootValue(value); + *_value.lock() = RootValue(value); } AttrKey AttrCursor::getKey() @@ -396,18 +440,35 @@ AttrKey AttrCursor::getKey() Value & AttrCursor::getValue() { - if (!_value) { - if (parent) { - auto & vParent = parent->first->getValue(); - root->state.forceAttrs(vParent, noPos, "while searching for an attribute"); - auto attr = vParent.attrs()->get(parent->second); - if (!attr) - throw Error("attribute '%s' is unexpectedly missing", getAttrPathStr()); - _value = allocRootValue(attr->value); - } else - _value = allocRootValue(root->getRootValue()); + { + auto value(_value.lock()); + if (*value) + return ***value; } - return **_value; + + /* Note: the lock must not be held while the value is being + evaluated: the evaluation may suspend the current fiber, and + another fiber on the same thread blocking on the lock would + then deadlock the fiber scheduler (fibers are pinned to their + thread, see `Executor::Worker`). Concurrent calls may therefore + both force the parent, which the thunk machinery deduplicates + (by suspending the fiber rather than blocking the thread), and + both look up the attribute, which is idempotent. */ + Value * v; + if (parent) { + auto & vParent = parent->first->getValue(); + root->state.forceAttrs(vParent, noPos, "while searching for an attribute"); + auto attr = vParent.attrs()->get(parent->second); + if (!attr) + throw Error("attribute '%s' is unexpectedly missing", getAttrPathStr()); + v = attr->value; + } else + v = root->getRootValue(); + + auto value(_value.lock()); + if (!*value) + *value = RootValue(v); + return ***value; } void AttrCursor::fetchCachedValue() @@ -418,23 +479,33 @@ void AttrCursor::fetchCachedValue() throw CachedEvalError(parent->first, parent->second); } -AttrPath AttrCursor::getAttrPath() const +AttrPath AttrCursor::getAttrPathRaw() const { if (parent) { - auto attrPath = parent->first->getAttrPath(); + auto attrPath = parent->first->getAttrPathRaw(); attrPath.push_back(parent->second); return attrPath; } else return {}; } -AttrPath AttrCursor::getAttrPath(Symbol name) const +AttrPath AttrCursor::getAttrPath() const +{ + return root->cleanupAttrPath(getAttrPathRaw()); +} + +AttrPath AttrCursor::getAttrPathRaw(Symbol name) const { - auto attrPath = getAttrPath(); + auto attrPath = getAttrPathRaw(); attrPath.push_back(name); return attrPath; } +AttrPath AttrCursor::getAttrPath(Symbol name) const +{ + return root->cleanupAttrPath(getAttrPathRaw(name)); +} + std::string AttrCursor::getAttrPathStr() const { return getAttrPath().to_string(root->state); @@ -461,9 +532,13 @@ Value & AttrCursor::forceValue() } if (root->db && (!cachedValue || std::get_if(&cachedValue->second))) { - if (v.type() == nString) - cachedValue = {root->db->setString(getKey(), v.string_view(), v.context()), string_t{v.string_view(), {}}}; - else if (v.type() == nPath) { + if (v.type() == nString) { + NixStringContext context; + copyContext(v, context); + cachedValue = { + root->db->setString(getKey(), v.string_view(), v.context()), + string_t{v.string_view(), std::move(context)}}; + } else if (v.type() == nPath) { auto path = v.path().path; cachedValue = {root->db->setString(getKey(), path.abs()), string_t{path.abs(), {}}}; } else if (v.type() == nBool) @@ -608,19 +683,22 @@ string_t AttrCursor::getStringWithContext() if (auto s = std::get_if(&cachedValue->second)) { bool valid = true; for (auto & c : s->second) { - const StorePath & path = std::visit( + const StorePath * path = std::visit( overloaded{ - [&](const NixStringContextElem::DrvDeep & d) -> const StorePath & { return d.drvPath; }, - [&](const NixStringContextElem::Built & b) -> const StorePath & { - return b.drvPath->getBaseStorePath(); + [&](const NixStringContextElem::DrvDeep & d) -> const StorePath * { return &d.drvPath; }, + [&](const NixStringContextElem::Built & b) -> const StorePath * { + return &b.drvPath->getBaseStorePath(); }, - [&](const NixStringContextElem::Opaque & o) -> const StorePath & { return o.path; }, + [&](const NixStringContextElem::Opaque & o) -> const StorePath * { return &o.path; }, + [&](const NixStringContextElem::Path & p) -> const StorePath * { return nullptr; }, }, c.raw); - root->state.store->addTempRoot(path); - if (!root->state.store->isValidPath(path)) { - valid = false; - break; + if (path) { + root->state.store->addTempRoot(*path); + if (!root->state.store->isValidPath(*path)) { + valid = false; + break; + } } } if (valid) { @@ -768,6 +846,7 @@ StorePath AttrCursor::forceDerivation() /* The eval cache contains 'drvPath', but the actual path has been garbage-collected. So force it to be regenerated. */ aDrvPath->forceValue(); + root->state.waitForPath(drvPath); if (!root->state.store->isValidPath(drvPath)) throw Error( "don't know how to recreate store derivation '%s'!", root->state.store->printStorePath(drvPath)); diff --git a/src/libexpr/eval-gc.cc b/src/libexpr/eval-gc.cc index 5bdcd7984644..402983094c86 100644 --- a/src/libexpr/eval-gc.cc +++ b/src/libexpr/eval-gc.cc @@ -1,4 +1,5 @@ #include "nix/util/environment-variables.hh" +#include "nix/util/processes.hh" #include "nix/expr/eval-settings.hh" #include "nix/util/config-global.hh" #include "nix/expr/eval-gc.hh" @@ -8,17 +9,10 @@ #if NIX_USE_BOEHMGC -# include -# ifdef __FreeBSD__ -# include -# endif - # include # include // For GC_GRANULE_BYTES -# include -# include -# include +# include "nix/util/coroutine-gc.hh" #endif @@ -40,10 +34,164 @@ static_assert(sizeof(void *) * 2 == GC_GRANULE_BYTES, "Boehm GC must use GC_GRAN /* Called when the Boehm GC runs out of memory. */ static void * oomHandler(size_t requested) { - /* Convert this to a proper C++ exception. */ - throw std::bad_alloc(); + outOfMemory(); +} + +static size_t getFreeMem() +{ + /* On Linux, use the `MemAvailable` or `MemFree` fields from + /proc/cpuinfo. */ +# ifdef __linux__ + { + std::unordered_map fields; + for (auto & line : + tokenizeString>(readFile(std::filesystem::path("/proc/meminfo")), "\n")) { + auto colon = line.find(':'); + if (colon == line.npos) + continue; + fields.emplace(line.substr(0, colon), trim(line.substr(colon + 1))); + } + + auto i = fields.find("MemAvailable"); + if (i == fields.end()) + i = fields.find("MemFree"); + if (i != fields.end()) { + auto kb = tokenizeString>(i->second, " "); + if (kb.size() == 2 && kb[1] == "kB") + return string2Int(kb[0]).value_or(0) * 1024; + } + } +# endif + + /* On non-Linux systems, conservatively assume that 25% of memory is free. */ + long pageSize = sysconf(_SC_PAGESIZE); + long pages = sysconf(_SC_PHYS_PAGES); + if (pageSize > 0 && pages > 0) + return (static_cast(pageSize) * static_cast(pages)) / 4; + return 0; } +/** + * Implementations of the libutil coroutine GC hooks (see + * `coroutine-gc.hh`) in terms of bdwgc's registered stacks. Together + * with the fiber stack registration in `parallel-eval.cc`, this makes + * every stack that can hold GC roots — thread stacks (registered + * automatically), fiber stacks and coroutine stacks — scannable by + * the collector, including the frames of a fiber that has switched + * onto a coroutine stack. + * + * The invariant maintained here is that `GC_current_stack` is the + * registered stack the thread is executing on, and that every other + * stack holding live frames has a `saved_sp` that lies *within* that + * stack. So the stack pointer is always recorded into + * `GC_current_stack` (the stack actually being left), never into a + * stack named by the caller: as explained in `coroutine-gc.hh`, a + * coroutine may yield from another coroutine's stack, and recording + * such a yield into the coroutine's own stack would make the + * collector scan from one stack up to the base of another, running + * into guard pages and unrelated mappings along the way. + */ + +void gcSaveStackPointer(struct GC_stack * stk) +{ + auto sp = (char *) GC_get_approx_sp(); + /* Check the stack pointer before lowering it by the slack, so that + a stack that is nearly exhausted (but whose guard page is not + included in `limit`, as for thread stacks) doesn't trip the + check. */ + if (!(sp < (char *) stk->base && (!stk->limit || sp >= (char *) stk->limit))) + panic( + fmt("stack pointer %p is not within the stack [%p, %p) that is being switched away from", + (void *) sp, + stk->limit, + stk->base)); + stk->saved_sp = sp - gcStackSwitchSlack; +} + +static void * coroStackRegisterImpl(void * base, size_t size) +{ + auto stk = new GC_stack{}; + stk->base = base; + stk->limit = (char *) base - size; + GC_register_stack(stk); + return stk; +} + +static void coroStackUnregisterImpl(void * cookie) +{ + auto stk = (struct GC_stack *) cookie; + GC_unregister_stack(stk); + delete stk; +} + +static void * coroSwitchToImpl(void * cookie) +{ + auto prev = GC_current_stack; + /* `prev` is null on threads not registered with the GC; such + threads hold no GC roots and need no scanning. */ + if (prev) + gcSaveStackPointer(prev); + /* Provisional: the resumed coroutine may actually continue on + another stack, in which case `coroResumeImpl()` corrects this + right after the switch. The body start (`coroEnterImpl()`) + corrects the null cookie of a coroutine that is being started. */ + GC_current_stack = (struct GC_stack *) cookie; + return prev; +} + +static void coroSwitchBackImpl(void * prevHandle) +{ + auto prev = (struct GC_stack *) prevHandle; + GC_current_stack = prev; + if (prev) + prev->saved_sp = nullptr; +} + +static void coroEnterImpl(void * cookie) +{ + GC_current_stack = (struct GC_stack *) cookie; +} + +static void * coroYieldImpl() +{ + auto cur = GC_current_stack; + if (cur) + gcSaveStackPointer(cur); + return cur; +} + +static void coroResumeImpl(void * handle) +{ + auto cur = (struct GC_stack *) handle; + GC_current_stack = cur; + if (cur) + cur->saved_sp = nullptr; +} + +/** + * A forked child process (e.g. a builtin builder, or an in-process + * build sandbox running on a `clone()` stack) inherits the parent + * thread's `GC_current_stack` and registered stacks, but it is not + * executing on the stack that `GC_current_stack` refers to (if it + * inherited the main thread's, it may be running on an entirely + * different one) and it never runs the collector, so the stack + * bookkeeping must not be done there: at best it is useless, at worst + * it trips the consistency check in `gcSaveStackPointer()` or takes + * the collector's lock, which may have been held by another thread of + * the parent at the time of the fork. So disable the hooks in the + * child. + */ +static RegisterForkCallback disableCoroutineGCHooks([]() { + coroStackRegister = nullptr; + coroStackUnregister = nullptr; + coroSwitchTo = nullptr; + coroSwitchBack = nullptr; + coroEnter = nullptr; + coroYield = nullptr; + coroResume = nullptr; + GC_current_stack = nullptr; +}); + static inline void initGCReal() { /* Initialise the Boehm garbage collector. */ @@ -74,8 +222,19 @@ static inline void initGCReal() GC_set_oom_fn(oomHandler); + /* Make the coroutine stacks of libutil scannable by the GC (fiber + stacks are registered in `parallel-eval.cc`; thread stacks are + registered automatically by bdwgc). */ + coroStackRegister = coroStackRegisterImpl; + coroStackUnregister = coroStackUnregisterImpl; + coroSwitchTo = coroSwitchToImpl; + coroSwitchBack = coroSwitchBackImpl; + coroEnter = coroEnterImpl; + coroYield = coroYieldImpl; + coroResume = coroResumeImpl; + /* Funnel boehm warnings into debug logs. */ - GC_set_warn_proc([](char * msg, GC_word word) noexcept { + GC_set_warn_proc([](const char * msg, GC_word word) noexcept { std::array buffer{}; auto res = snprintf(buffer.data(), buffer.size(), msg, word); /* Ignore garbage. */ @@ -89,8 +248,8 @@ static inline void initGCReal() } }); - /* Set the initial heap size to something fairly big (25% of - physical RAM, up to a maximum of 384 MiB) so that in most cases + /* Set the initial heap size to something fairly big (80% of + free RAM, up to a maximum of 4 GiB) so that in most cases we don't need to garbage collect at all. (Collection has a fairly significant overhead.) The heap size can be overridden through libgc's GC_INITIAL_HEAP_SIZE environment variable. We @@ -101,15 +260,10 @@ static inline void initGCReal() if (!getEnv("GC_INITIAL_HEAP_SIZE")) { size_t size = 32 * 1024 * 1024; # if HAVE_SYSCONF && defined(_SC_PAGESIZE) && defined(_SC_PHYS_PAGES) - size_t maxSize = 384 * 1024 * 1024; - long pageSize = sysconf(_SC_PAGESIZE); - long pages = sysconf(_SC_PHYS_PAGES); - if (pageSize != -1) - size = (pageSize * pages) / 4; // 25% of RAM - if (size > maxSize) - size = maxSize; + size_t maxSize = 4ULL * 1024 * 1024 * 1024; + auto free = getFreeMem(); + size = std::max(size, std::min((size_t) (free * 0.5), maxSize)); # endif - debug("setting initial heap size to %1% bytes", size); GC_expand_hp(size); } } diff --git a/src/libexpr/eval-profiler.cc b/src/libexpr/eval-profiler.cc index 12c2f239df6e..9c7448d59076 100644 --- a/src/libexpr/eval-profiler.cc +++ b/src/libexpr/eval-profiler.cc @@ -171,6 +171,7 @@ class SampleStack : public EvalProfiler EvalState & state; std::chrono::nanoseconds sampleInterval; AutoCloseFD profileFd; + // FIXME: this needs to become per-thread to support multi-threaded evaluation. FrameStack stack; std::map callCount; std::chrono::time_point lastStackSample = diff --git a/src/libexpr/eval-settings.cc b/src/libexpr/eval-settings.cc index c438ff6c9fb9..630f7b67067f 100644 --- a/src/libexpr/eval-settings.cc +++ b/src/libexpr/eval-settings.cc @@ -115,9 +115,19 @@ bool EvalSettings::isPseudoUrl(std::string_view s) std::string EvalSettings::resolvePseudoUrl(std::string_view url) { - if (hasPrefix(url, "channel:")) - return "https://channels.nixos.org/" + std::string(url.substr(8)) + "/nixexprs.tar.xz"; - else + if (hasPrefix(url, "channel:")) { + auto realUrl = "https://channels.nixos.org/" + std::string(url.substr(8)) + "/nixexprs.tar.xz"; + static bool haveWarned = false; + warnOnce( + haveWarned, + "Channels are deprecated in favor of flakes in Determinate Nix. " + "Instead of '%s', use '%s'. " + "See https://zero-to-nix.com for a guide to Nix flakes. " + "For details and to offer feedback on the deprecation process, see: https://github.com/DeterminateSystems/nix-src/issues/34.", + url, + realUrl); + return realUrl; + } else return std::string(url); } diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 29ecec69ee64..67fd035d0074 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -27,6 +27,8 @@ #include "nix/fetchers/tarball.hh" #include "nix/fetchers/input-cache.hh" #include "nix/util/current-process.hh" +#include "nix/store/async-path-writer.hh" +#include "nix/expr/parallel-eval.hh" #include "parser-tab.hh" @@ -48,6 +50,11 @@ #include #include #include +#include + +#ifndef _WIN32 // TODO use portable implementation +# include +#endif #include "nix/util/strings-inline.hh" @@ -63,7 +70,7 @@ static char * allocString(size_t size) char * t; t = (char *) GC_MALLOC_ATOMIC(size); if (!t) - throw std::bad_alloc(); + outOfMemory(); return t; } @@ -89,7 +96,7 @@ StringData & StringData::alloc(EvalMemory & mem, size_t size) { void * t = mem.allocBytes(sizeof(StringData) + size + 1); if (!t) - throw std::bad_alloc(); + outOfMemory(); auto res = new (t) StringData(size); return *res; } @@ -104,11 +111,6 @@ const StringData & StringData::make(EvalMemory & mem, std::string_view s) return res; } -RootValue allocRootValue(Value * v) -{ - return std::allocate_shared(traceable_allocator(), v); -} - // Pretty print types for assertion errors std::ostream & operator<<(std::ostream & os, const ValueType t) { @@ -199,20 +201,48 @@ PosIdx Value::determinePos(const PosIdx pos) const return attrs()->pos; case tLambda: return lambda().fun->pos; +#if 0 + // FIXME: disabled because reading from an app is racy. case tApp: return app().left->determinePos(pos); +#endif default: return pos; } #pragma GCC diagnostic pop } -bool Value::isTrivial() const +template<> +bool ValueStorage::isTrivial() const { - return !isa() - && (!isa() - || (dynamic_cast(thunk().expr) && ((ExprAttrs *) thunk().expr)->dynamicAttrs->empty()) - || dynamic_cast(thunk().expr) || dynamic_cast(thunk().expr)); + auto p0_ = p0.load(std::memory_order_acquire); + + auto pd = static_cast(p0_ & discriminatorMask); + + if (pd == pdThunk) { + auto p1_ = p1; + + /* `p1` is only valid as a thunk payload if no other thread + has started forcing the thunk in the meantime, since + `finish()` overwrites `p1` before it updates `p0`. So check + that the value is still a thunk. */ + if (p0.load(std::memory_order_acquire) != p0_) + /* The thunk is now pending, awaited or finished. */ + return true; + + bool isApp = p1_ & discriminatorMask; + if (isApp) + return false; + auto expr = untagPointer(p1_); + return (dynamic_cast(expr) && ((ExprAttrs *) expr)->dynamicAttrs->empty()) + || dynamic_cast(expr) || dynamic_cast(expr); + } + + else + /* Note: if the value is pending or awaited, we can't safely + inspect its expression, so treat it as trivial (i.e. the + caller will wait for the thread that is forcing it). */ + return true; } static Symbol getName(const AttrName & name, EvalState & state, Env & env) @@ -234,6 +264,8 @@ EvalMemory::EvalMemory() assertGCInitialized(); } +[[gnu::tls_model("initial-exec")]] thread_local EvalState::EvalContext EvalState::evalContext; + EvalState::EvalState( const LookupPath & lookupPathFromArguments, ref store, @@ -276,8 +308,10 @@ EvalState::EvalState( instance if we're evaluating a file from the physical /nix/store while using a chroot store, and also for lazy mounted fetchTree. */ - auto accessor = settings.pureEval ? storeFS.cast() - : makeUnionSourceAccessor({getFSSourceAccessor(), storeFS}); + auto accessor = settings.pureEval + ? storeFS.cast() + : makeUnionSourceAccessor({getFSSourceAccessor(), storeFS}, storeFS.cast()); + /* Cache positive lstat/readlink results to speed up resolveSymlinks. */ accessor = makeCachingSourceAccessor(accessor); @@ -308,6 +342,7 @@ EvalState::EvalState( , debugRepl(nullptr) , debugStop(false) , trylevel(0) + , asyncPathWriter(AsyncPathWriter::make(store)) , importResolutionCache(make_ref()) , fileEvalCache(make_ref()) , positionToDocComment(make_ref()) @@ -320,6 +355,11 @@ EvalState::EvalState( , baseEnv(mem.allocEnv(BASE_ENV_SIZE)) #endif , staticBaseEnv{std::make_shared(nullptr, nullptr)} + , countCalls(getEnv("NIX_COUNT_CALLS").value_or("0") != "0") + , primOpCalls(make_ref()) + , functionCalls(make_ref()) + , attrSelects(make_ref()) + , executor{make_ref(settings)} { #ifndef _WIN32 static std::once_flag stackSizeBumped; @@ -335,8 +375,6 @@ EvalState::EvalState( corepkgsFS->setPathDisplay(""); internalFS->setPathDisplay("«nix-internal»", ""); - countCalls = getEnv("NIX_COUNT_CALLS").value_or("0") != "0"; - static_assert(sizeof(Env) <= 16, "environment must be <= 16 bytes"); /* Construct the Nix expression search path. */ @@ -477,7 +515,8 @@ void EvalState::checkURI(const std::string & uri0) Value * EvalState::addConstant(const std::string & name, Value & v, Constant info) { Value * v2 = allocValue(); - *v2 = v; + // Do a raw copy since `operator =` barfs on thunks. + memcpy((char *) v2, (char *) &v, sizeof(Value)); addConstant(name, v2, info); return v2; } @@ -493,8 +532,10 @@ void EvalState::addConstant(const std::string & name, Value * v, Constant info) We might know the type of a thunk in advance, so be allowed to just write it down in that case. */ - if (auto gotType = v->type(); gotType != nThunk) - assert(info.type == gotType); + if (v->isFinished()) { + if (auto gotType = v->type(); gotType != nThunk) + assert(info.type == gotType); + } /* Install value the base environment. */ staticBaseEnv->vars.emplace_back(symbols.create(name), baseEnvDispl); @@ -564,7 +605,7 @@ Value * EvalState::addPrimOp(PrimOp && primOp) v->mkPrimOp(new PrimOp(primOp)); if (primOp.internal) - internalPrimOps.emplace(primOp.name, v); + internalPrimOps.emplace(primOp.name, RootValue(v)); else { staticBaseEnv->vars.emplace_back(envName, baseEnvDispl); baseEnv.values[baseEnvDispl++] = v; @@ -688,7 +729,7 @@ static void printStaticEnvBindings(const SymbolTable & st, const StaticEnv & se) // just for the current level of Env, not the whole chain. static void printWithBindings(const SymbolTable & st, const Env & env) { - if (!env.values[0]->isThunk()) { + if (env.values[0]->isFinished()) { std::cout << "with: "; std::cout << ANSI_MAGENTA; auto * bindings = env.values[0]->attrs(); @@ -742,14 +783,14 @@ void mapStaticEnvBindings(const SymbolTable & st, const StaticEnv & se, const En if (env.up && se.up) { mapStaticEnvBindings(st, *se.up, *env.up, vm); - if (se.isWith && !env.values[0]->isThunk()) { + if (se.isWith && env.values[0]->isFinished()) { // add 'with' bindings. for (auto & j : *env.values[0]->attrs()) - vm.insert_or_assign(std::string(st[j.name]), j.value); + vm.insert_or_assign(std::string(st[j.name]), RootValue(j.value)); } else { // iterate through staticenv bindings and add them. for (auto & i : se.vars) - vm.insert_or_assign(std::string(st[i.first]), env.values[i.second]); + vm.insert_or_assign(std::string(st[i.first]), RootValue(env.values[i.second])); } } } @@ -938,7 +979,7 @@ void Value::mkPath(const SourcePath & path, EvalMemory & mem) forceAttrs(*env->values[0], fromWith->pos, "while evaluating the first subexpression of a with expression"); if (auto j = env->values[0]->attrs()->get(var.name)) { if (countCalls) [[unlikely]] - attrSelects[j->pos]++; + attrSelects->try_emplace_or_visit(j->pos, 1, [](auto & i) { i.second++; }); return j->value; } if (!fromWith->parentWith) [[unlikely]] @@ -981,7 +1022,14 @@ void EvalState::mkPos(Value & v, PosIdx p) auto origin = positions.originOf(p); if (auto path = std::get_if(&origin)) { auto attrs = buildBindings(3); - attrs.alloc(s.file).mkString(path->path.abs(), mem); + if (path->accessor == rootFS && store->isInStore(path->path.abs())) + // FIXME: only do this for virtual store paths? + attrs.alloc(s.file).mkString( + path->path.abs(), + {NixStringContextElem::Path{.storePath = store->toStorePath(path->path.abs()).first}}, + mem); + else + attrs.alloc(s.file).mkString(path->path.abs(), mem); makePositionThunks(*this, p, attrs.alloc(s.line), attrs.alloc(s.column)); v.mkAttrs(attrs); } else @@ -1029,6 +1077,7 @@ std::string EvalState::mkSingleDerivedPathStringRaw(const SingleDerivedPath & p) auto optStaticOutputPath = std::visit( overloaded{ [&](const SingleDerivedPath::Opaque & o) { + waitForPath(o.path); auto drv = store->readDerivation(o.path); auto i = drv.outputs.find(b.output); if (i == drv.outputs.end()) @@ -1072,7 +1121,7 @@ Value * ExprVar::maybeThunk(EvalState & state, Env & env) state.nrAvoided++; return v; } - return Expr::maybeThunk(state, env); + [[gnu::musttail]] return Expr::maybeThunk(state, env); } Value * ExprString::maybeThunk(EvalState & state, Env & env) @@ -1106,10 +1155,9 @@ namespace { * from a thunk, ensuring that every file is parsed/evaluated only * once (via the thunk stored in `EvalState::fileEvalCache`). */ -struct ExprParseFile : Expr, gc +struct ExprParseFile : Expr { - // FIXME: make this a reference (see below). - SourcePath path; + SourcePath & path; bool mustBeTrivial; ExprParseFile(SourcePath & path, bool mustBeTrivial) @@ -1155,28 +1203,28 @@ void EvalState::evalFile(const SourcePath & path, Value & v, bool mustBeTrivial) importResolutionCache->emplace(path, *resolvedPath); } - if (auto v2 = getConcurrent(*fileEvalCache, *resolvedPath)) { - forceValue(**v2, noPos); - v = **v2; - return; + { + Value * v2 = nullptr; + fileEvalCache->cvisit(*resolvedPath, [&](auto & i) { v2 = *i.second; }); + if (v2) { + forceValue(*v2, noPos); + v = *v2; + return; + } } Value * vExpr; - // FIXME: put ExprParseFile on the stack instead of the heap once - // https://github.com/NixOS/nix/pull/13930 is merged. That will ensure - // the post-condition that `expr` is unreachable after - // `forceValue()` returns. - auto expr = new ExprParseFile{*resolvedPath, mustBeTrivial}; + ExprParseFile expr{*resolvedPath, mustBeTrivial}; fileEvalCache->try_emplace_and_cvisit( *resolvedPath, - nullptr, + RootValue(nullptr), [&](auto & i) { vExpr = allocValue(); - vExpr->mkThunk(&baseEnv, expr); - i.second = vExpr; + vExpr->mkThunk(&baseEnv, &expr); + *i.second = vExpr; }, - [&](auto & i) { vExpr = i.second; }); + [&](auto & i) { vExpr = *i.second; }); forceValue(*vExpr, noPos); @@ -1359,7 +1407,12 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v) sort = true; } - bindings.bindings->pos = pos; + /* Empty attrsets share the static Bindings::emptyBindings, which we + must not write to: apart from being a data race, it causes false + sharing on emptyBindings' cache line (which may also hold other hot + globals such as Counter::enabled) between all evaluator threads. */ + if (bindings.bindings != &Bindings::emptyBindings) + bindings.bindings->pos = pos; v.mkAttrs(sort ? bindings.finish() : bindings.alreadySorted()); } @@ -1381,11 +1434,12 @@ void ExprLet::eval(EvalState & state, Env & env, Value & v) env2.values[displ++] = i.second.e->maybeThunk(state, *i.second.chooseByKind(&env2, &env, inheritEnv)); } - auto dts = state.debugRepl - ? makeDebugTraceStacker(state, *this, env2, getPos(), "while evaluating a '%1%' expression", "let") - : nullptr; - - body->eval(state, env2, v); + if (state.debugRepl) { + auto dts = makeDebugTraceStacker(state, *this, env2, getPos(), "while evaluating a '%1%' expression", "let"); + return body->eval(state, env2, v); + } else { + [[gnu::musttail]] return body->eval(state, env2, v); + } } void ExprList::eval(EvalState & state, Env & env, Value & v) @@ -1477,10 +1531,10 @@ void ExprSelect::eval(EvalState & state, Env & env, Value & v) vAttrs = j->value; pos2 = j->pos; if (state.countCalls) - state.attrSelects[pos2]++; + state.attrSelects->try_emplace_or_visit(pos2, 1, [](auto & i) { i.second++; }); } - state.forceValue(*vAttrs, (pos2 ? pos2 : this->pos)); + state.forceValue(*vAttrs, pos2 ? pos2 : this->pos); } catch (Error & e) { if (pos2) { @@ -1539,6 +1593,8 @@ void ExprLambda::eval(EvalState & state, Env & env, Value & v) v.mkLambda(&env, this); } +[[gnu::tls_model("initial-exec")]] thread_local size_t CallDepth::callDepth = 0; + void EvalState::callFunction(Value & fun, std::span args, Value & vRes, const PosIdx pos) { auto _level = addCallDepth(pos); @@ -1554,12 +1610,13 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, forceValue(fun, pos); - Value vCur(fun); + Value vCur = fun; auto makeAppChain = [&]() { for (auto arg : args) { auto fun2 = allocValue(); *fun2 = vCur; + vCur.reset(); vCur.mkPrimOpApp(fun2, arg); } vRes = vCur; @@ -1658,6 +1715,7 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, lambda.name ? concatStrings("'", symbols[lambda.name], "'") : "anonymous lambda") : nullptr; + vCur.reset(); lambda.body->eval(*this, env2, vCur); } catch (Error & e) { if (loggerSettings.showTrace.get()) { @@ -1689,10 +1747,12 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, nrPrimOpCalls++; if (countCalls) - primOpCalls[fn->name]++; + primOpCalls->try_emplace_or_visit(fn->name, 1, [](auto & i) { i.second++; }); try { - fn->impl(*this, vCur.determinePos(noPos), args.data(), vCur); + auto pos = vCur.determinePos(noPos); + vCur.reset(); + fn->impl(*this, pos, args.data(), vCur); } catch (Error & e) { if (fn->addTrace) addErrorTrace(e, pos, "while calling the '%1%' builtin", fn->name); @@ -1714,6 +1774,7 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, assert(primOp->isPrimOp()); auto arity = primOp->primOp()->arity; auto argsLeft = arity - argsDone; + assert(argsLeft); if (args.size() < argsLeft) { /* We still don't have enough arguments, so extend the tPrimOpApp chain. */ @@ -1734,7 +1795,7 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, auto fn = primOp->primOp(); nrPrimOpCalls++; if (countCalls) - primOpCalls[fn->name]++; + primOpCalls->try_emplace_or_visit(fn->name, 1, [](auto & i) { i.second++; }); try { // TODO: @@ -1742,7 +1803,9 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, // 2. Create a fake env (arg1, arg2, etc.) and a fake expr (arg1: arg2: etc: builtins.name arg1 arg2 // etc) // so the debugger allows to inspect the wrong parameters passed to the builtin. - fn->impl(*this, vCur.determinePos(noPos), vArgs, vCur); + auto pos = vCur.determinePos(noPos); + vCur.reset(); + fn->impl(*this, pos, vArgs, vCur); } catch (Error & e) { if (fn->addTrace) addErrorTrace(e, pos, "while calling the '%1%' builtin", fn->name); @@ -1759,6 +1822,7 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, heap-allocate a copy and use that instead. */ Value * args2[] = {allocValue(), args[0]}; *args2[0] = vCur; + vCur.reset(); try { callFunction(*functor->value, args2, vCur, functor->pos); } catch (Error & e) { @@ -1805,7 +1869,7 @@ void ExprCall::eval(EvalState & state, Env & env, Value & v) // prevents tail-call optimisation. void EvalState::incrFunctionCall(ExprLambda * fun) { - functionCalls[fun]++; + functionCalls->try_emplace_or_visit(fun, 1, [](auto & i) { i.second++; }); } void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res) @@ -1820,7 +1884,7 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res Value * v = allocValue(); callFunction(*found->value, fun, *v, pos); forceValue(*v, pos); - return autoCallFunction(args, *v, res); + [[gnu::musttail]] return autoCallFunction(args, *v, res); } } @@ -1859,7 +1923,7 @@ values, or passed explicitly with '--arg' or '--argstr'. See } } - callFunction(fun, allocValue()->mkAttrs(attrs), res, pos); + [[gnu::musttail]] return callFunction(fun, allocValue()->mkAttrs(attrs), res, pos); } void ExprWith::eval(EvalState & state, Env & env, Value & v) @@ -1868,13 +1932,14 @@ void ExprWith::eval(EvalState & state, Env & env, Value & v) env2.up = &env; env2.values[0] = attrs->maybeThunk(state, env); - body->eval(state, env2, v); + [[gnu::musttail]] return body->eval(state, env2, v); } void ExprIf::eval(EvalState & state, Env & env, Value & v) { // We cheat in the parser, and pass the position of the condition as the position of the if itself. - (state.evalBool(env, cond, pos, "while evaluating a branch condition") ? then : else_)->eval(state, env, v); + [[gnu::musttail]] return (state.evalBool(env, cond, pos, "while evaluating a branch condition") ? then : else_) + ->eval(state, env, v); } void ExprAssert::eval(EvalState & state, Env & env, Value & v) @@ -1899,7 +1964,7 @@ void ExprAssert::eval(EvalState & state, Env & env, Value & v) state.error("assertion '%1%' failed", exprStr).atPos(pos).withFrame(env, *this).debugThrow(); } - body->eval(state, env, v); + [[gnu::musttail]] return body->eval(state, env, v); } void ExprOpNot::eval(EvalState & state, Env & env, Value & v) @@ -1946,8 +2011,12 @@ void ExprOpImpl::eval(EvalState & state, Env & env, Value & v) || state.evalBool(env, e2, pos, "in the right operand of the IMPL (->) operator")); } -void ExprOpUpdate::eval(EvalState & state, Value & v, Value & v1, Value & v2) +void ExprOpUpdate::eval(EvalState & state, Env & env, Value & v) { + Value v1, v2; + state.evalAttrs(env, e1, v1, pos, "in the left operand of the update (//) operator"); + state.evalAttrs(env, e2, v2, pos, "in the right operand of the update (//) operator"); + state.nrOpUpdates++; const Bindings & bindings1 = *v1.attrs(); @@ -2021,42 +2090,6 @@ void ExprOpUpdate::eval(EvalState & state, Value & v, Value & v1, Value & v2) state.nrOpUpdateValuesCopied += v.attrs()->size(); } -void ExprOpUpdate::eval(EvalState & state, Env & env, Value & v) -{ - UpdateQueue q; - evalForUpdate(state, env, q); - - Value vTmp; - vTmp.mkAttrs(&Bindings::emptyBindings); - - for (auto & rhs : std::views::reverse(q)) { - /* Remember that queue is sorted rightmost attrset first. */ - eval(state, /*v=*/vTmp, /*v1=*/vTmp, /*v2=*/rhs); - } - - v = vTmp; -} - -void Expr::evalForUpdate(EvalState & state, Env & env, UpdateQueue & q, std::string_view errorCtx) -{ - Value v; - state.evalAttrs(env, this, v, getPos(), errorCtx); - q.push_back(v); -} - -void ExprOpUpdate::evalForUpdate(EvalState & state, Env & env, UpdateQueue & q) -{ - /* Output rightmost attrset first to the merge queue as the one - with the most priority. */ - e2->evalForUpdate(state, env, q, "in the right operand of the update (//) operator"); - e1->evalForUpdate(state, env, q, "in the left operand of the update (//) operator"); -} - -void ExprOpUpdate::evalForUpdate(EvalState & state, Env & env, UpdateQueue & q, std::string_view errorCtx) -{ - evalForUpdate(state, env, q); -} - void ExprOpConcatLists::eval(EvalState & state, Env & env, Value & v) { Value v1; @@ -2176,7 +2209,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v) } else if (firstType == nFloat) { v.mkFloat(nf); } else if (firstType == nPath) { - if (!context.empty()) + if (hasContext(context)) state.error("a string that refers to a store path cannot be appended to a path") .atPos(pos) .withFrame(env, *this) @@ -2204,71 +2237,6 @@ void ExprPos::eval(EvalState & state, Env & env, Value & v) state.mkPos(v, pos); } -void ExprBlackHole::eval(EvalState & state, [[maybe_unused]] Env & env, Value & v) -{ - throwInfiniteRecursionError(state, v); -} - -[[gnu::noinline]] [[noreturn]] void ExprBlackHole::throwInfiniteRecursionError(EvalState & state, Value & v) -{ - state.error("infinite recursion encountered").atPos(v.determinePos(noPos)).debugThrow(); -} - -// always force this to be separate, otherwise forceValue may inline it and take -// a massive perf hit -[[gnu::noinline]] -void EvalState::handleEvalExceptionForThunk(Env * env, Expr * expr, Value & v, const PosIdx pos) -{ - if (!env) - tryFixupBlackHolePos(v, pos); - - auto e = std::current_exception(); - Value * recovery = nullptr; - try { - std::rethrow_exception(e); - } catch (const Interrupted & e) { - recovery = allocValue(); - } catch (const RecoverableEvalError & e) { - recovery = allocValue(); - } catch (...) { - } - if (recovery) { - recovery->mkThunk(env, expr); - } - v.mkFailed(e, recovery); -} - -[[gnu::noinline]] -void EvalState::handleEvalExceptionForApp(Value & v, const Value & savedApp) -{ - auto e = std::current_exception(); - Value * recovery = nullptr; - try { - std::rethrow_exception(e); - } catch (const Interrupted & e) { - recovery = allocValue(); - } catch (const RecoverableEvalError & e) { - recovery = allocValue(); - } catch (...) { - } - if (recovery) { - *recovery = savedApp; - } - v.mkFailed(e, recovery); -} - -[[gnu::noinline]] -void EvalState::handleEvalFailed(Value & v, const PosIdx pos) -{ - assert(v.isFailed()); - if (auto recoveryValue = v.failed().recoveryValue) { - v = *recoveryValue; - forceValue(v, pos); - } else { - v.failed().rethrow(); - } -} - void EvalState::tryFixupBlackHolePos(Value & v, PosIdx pos) { if (!v.isBlackhole()) @@ -2299,6 +2267,7 @@ void EvalState::forceValueDeep(Value & v) for (auto & i : *v.attrs()) try { // If the value is a thunk, we're evaling. Otherwise no trace necessary. + // FIXME: race, thunk might be updated by another thread auto dts = state.debugRepl && i.value->isThunk() ? makeDebugTraceStacker( state, *i.value->thunk().expr, @@ -2451,13 +2420,17 @@ std::string_view EvalState::forceStringNoCtx(Value & v, const PosIdx pos, std::s { auto s = forceString(v, pos, errorCtx); if (v.context()) { - auto ctxElem = NixStringContextElem::parse((*v.context()->begin())->view()); - error( - "the string '%1%' is not allowed to refer to a store path (such as '%2%')", - v.string_view(), - ctxElem.display(*store)) - .withTrace(pos, errorCtx) - .debugThrow(); + NixStringContext context; + copyContext(v, context); + if (hasContext(context)) { + auto ctxElem = NixStringContextElem::parse((*v.context()->begin())->view()); + error( + "the string '%1%' is not allowed to refer to a store path (such as '%2%')", + v.string_view(), + ctxElem.display(*store)) + .withTrace(pos, errorCtx) + .debugThrow(); + } } return s; } @@ -2514,14 +2487,24 @@ BackedStringView EvalState::coerceToString( } if (v.type() == nPath) { + // FIXME: instead of copying the path to the store, we could + // return a virtual store path that lazily copies the path to + // the store in devirtualize(). if (!canonicalizePath && !copyToStore) { // FIXME: hack to preserve path literals that end in a // slash, as in /foo/${x}. return v.pathStrView(); } else if (copyToStore) { - return store->printStorePath(copyPathToStore(context, v.path())); + return store->printStorePath(copyPathToStore(context, v.path(), v.determinePos(pos))); } else { - return std::string{v.path().path.abs()}; + auto path = v.path(); + if (path.accessor == rootFS && store->isInStore(path.path.abs())) { + try { + context.insert(NixStringContextElem::Path{.storePath = store->toStorePath(path.path.abs()).first}); + } catch (Error &) { + } + } + return std::string(path.path.abs()); } } @@ -2593,7 +2576,7 @@ BackedStringView EvalState::coerceToString( .debugThrow(); } -StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePath & path) +StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePath & path, PosIdx pos) { if (nix::isDerivation(path.path.abs())) error("file names are not allowed to end in '%1%'", drvExtension).debugThrow(); @@ -2603,7 +2586,7 @@ StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePat *store, path.resolveSymlinks(SymlinkResolution::Ancestors), settings.isReadOnly() ? FetchMode::DryRun : FetchMode::Copy, - path.baseName(), + computeBaseName(path, pos), ContentAddressMethod::Raw::NixArchive, nullptr, repair); @@ -2651,7 +2634,9 @@ EvalState::coerceToStorePath(const PosIdx pos, Value & v, NixStringContext & con auto path = coerceToString(pos, v, context, errorCtx, false, false, true).toOwned(); if (auto storePath = store->maybeParseStorePath(path)) return *storePath; - error("path '%1%' is not in the Nix store", path).withTrace(pos, errorCtx).debugThrow(); + error("cannot coerce '%s' to a store path because it is not a subpath of the Nix store", path) + .withTrace(pos, errorCtx) + .debugThrow(); } std::pair EvalState::coerceToSingleDerivedPathUnchecked( @@ -2675,6 +2660,9 @@ std::pair EvalState::coerceToSingleDerivedP .debugThrow(); }, [&](NixStringContextElem::Built && b) -> SingleDerivedPath { return std::move(b); }, + [&](NixStringContextElem::Path && p) -> SingleDerivedPath { + error("string '%s' has no context", s).withTrace(pos, errorCtx).debugThrow(); + }, }, ((NixStringContextElem &&) *context.begin()).raw); return { @@ -3119,10 +3107,22 @@ void EvalState::printStatistics() topObj["nrOpUpdates"] = nrOpUpdates.load(); topObj["nrOpUpdateValuesCopied"] = nrOpUpdateValuesCopied.load(); topObj["nrThunks"] = nrThunks.load(); + topObj["nrThunksAwaited"] = nrThunksAwaited.load(); + topObj["nrThunksAwaitedSlow"] = nrThunksAwaitedSlow.load(); + topObj["nrSpuriousWakeups"] = nrSpuriousWakeups.load(); + topObj["maxWaiting"] = maxWaiting.load(); + topObj["waitingTime"] = microsecondsWaiting / (double) 1000000; + topObj["nrFibersSpawned"] = executor->nrFibersSpawned.load(); + topObj["nrFiberWakeups"] = executor->nrFiberWakeups.load(); + topObj["maxSuspendedFibers"] = executor->maxSuspendedFibers.load(); + topObj["maxLiveFibers"] = executor->maxLiveFibers.load(); + topObj["nrFiberStacksAllocated"] = executor->nrFiberStacksAllocated.load(); topObj["nrAvoided"] = nrAvoided.load(); topObj["nrLookups"] = nrLookups.load(); topObj["nrPrimOpCalls"] = nrPrimOpCalls.load(); topObj["nrFunctionCalls"] = nrFunctionCalls.load(); + topObj["nrWasmGuestAllocs"] = nrWasmGuestAllocs.load(); + topObj["wasmGuestAllocBytes"] = wasmGuestAllocBytes.load(); #if NIX_USE_BOEHMGC topObj["gc"] = { {"heapSize", heapSize}, @@ -3132,11 +3132,16 @@ void EvalState::printStatistics() #endif if (countCalls) { - topObj["primops"] = primOpCalls; + { + auto & obj = topObj["primops"]; + obj = json::object(); + primOpCalls->visit_all([&](auto & i) { obj[i.first] = i.second; }); + } { auto & list = topObj["functions"]; list = json::array(); - for (auto & [fun, count] : functionCalls) { + functionCalls->visit_all([&](auto & i) { + auto & [fun, count] = i; json obj = json::object(); if (fun->name) obj["name"] = (std::string_view) symbols[fun->name]; @@ -3150,12 +3155,12 @@ void EvalState::printStatistics() } obj["count"] = count; list.push_back(obj); - } + }); } { - auto list = topObj["attributes"]; + auto & list = topObj["attributes"]; list = json::array(); - for (auto & i : attrSelects) { + attrSelects->visit_all([&](auto & i) { json obj = json::object(); if (auto pos = positions[i.first]) { if (auto path = std::get_if(&pos.origin)) @@ -3165,15 +3170,15 @@ void EvalState::printStatistics() } obj["count"] = i.second; list.push_back(obj); - } + }); } } if (getEnv("NIX_SHOW_SYMBOLS").value_or("0") != "0") { + auto list = json::array(); + symbols.dump([&](std::string_view s) { list.emplace_back(std::string(s)); }); // XXX: overrides earlier assignment - topObj["symbols"] = json::array(); - auto & list = topObj["symbols"]; - symbols.dump([&](std::string_view s) { list.emplace_back(s); }); + topObj["symbols"] = std::move(list); } if (outPath == "-") { std::cerr << topObj.dump(2) << std::endl; @@ -3480,4 +3485,24 @@ void forceNoNullByte(std::string_view s, std::function pos) } } +void EvalState::waitForPath(const StorePath & path) +{ + asyncPathWriter->waitForPath(path); +} + +void EvalState::waitForPath(const SingleDerivedPath & path) +{ + std::visit( + overloaded{ + [&](const DerivedPathOpaque & p) { waitForPath(p.path); }, + [&](const SingleDerivedPathBuilt & p) { waitForPath(*p.drvPath); }, + }, + path.raw()); +} + +void EvalState::waitForAllPaths() +{ + asyncPathWriter->waitForAllPaths(); +} + } // namespace nix diff --git a/src/libexpr/include/nix/expr/attr-path.hh b/src/libexpr/include/nix/expr/attr-path.hh index fd48705b8b7b..25384f5c4c87 100644 --- a/src/libexpr/include/nix/expr/attr-path.hh +++ b/src/libexpr/include/nix/expr/attr-path.hh @@ -25,6 +25,8 @@ struct AttrPath : std::vector static AttrPath parse(EvalState & state, std::string_view s); + static AttrPath fromStrings(EvalState & state, const std::vector & attrNames); + std::string to_string(EvalState & state) const; std::vector resolve(EvalState & state) const; diff --git a/src/libexpr/include/nix/expr/eval-cache.hh b/src/libexpr/include/nix/expr/eval-cache.hh index 0feb7d5649ff..2280bde4c933 100644 --- a/src/libexpr/include/nix/expr/eval-cache.hh +++ b/src/libexpr/include/nix/expr/eval-cache.hh @@ -8,6 +8,7 @@ #include #include +#include "nix/expr/root-value.hh" namespace nix::eval_cache { @@ -38,10 +39,28 @@ class EvalCache : public std::enable_shared_from_this friend struct CachedEvalError; std::shared_ptr db; + +public: EvalState & state; + + std::function cleanupAttrPath = [](AttrPath && attrPath) { return std::move(attrPath); }; + +public: typedef fun RootLoader; - RootLoader rootLoader; - RootValue value; + +private: + + /** + * The expression that calls the root loader (see + * `ExprRootLoader`), and the thunk that evaluates it. Forcing + * the thunk goes through the evaluator's regular thunk machinery, + * so concurrent requests for the root value are deduplicated + * without holding a mutex across the evaluation, which could + * deadlock the fiber scheduler (fibers are pinned to their + * thread, see `Executor::Worker`). + */ + std::unique_ptr rootLoaderExpr; + RootValue rootValue; Value * getRootValue(); @@ -102,10 +121,13 @@ class AttrCursor : public std::enable_shared_from_this friend class EvalCache; friend struct CachedEvalError; - ref root; +public: + const ref root; + +private: using Parent = std::optional, Symbol>>; - Parent parent; - RootValue _value; + const Parent parent; + Sync _value; std::optional> cachedValue; AttrKey getKey(); @@ -130,8 +152,12 @@ public: AttrPath getAttrPath() const; + AttrPath getAttrPathRaw() const; + AttrPath getAttrPath(Symbol name) const; + AttrPath getAttrPathRaw(Symbol name) const; + std::string getAttrPathStr() const; std::string getAttrPathStr(Symbol name) const; diff --git a/src/libexpr/include/nix/expr/eval-gc.hh b/src/libexpr/include/nix/expr/eval-gc.hh index 1b1eb7e627f0..4865ce3dbd8f 100644 --- a/src/libexpr/include/nix/expr/eval-gc.hh +++ b/src/libexpr/include/nix/expr/eval-gc.hh @@ -55,6 +55,36 @@ void assertGCInitialized(); * The number of GC cycles since initGC(). */ size_t getGCCycles(); + +/** + * Slack to subtract from `GC_get_approx_sp()` when recording the stack + * pointer of a stack that is about to be switched away from (see + * `GC_stack::saved_sp`). The approximation is taken in a callee of the + * function performing the switch, so it already lies below that + * function's own frame; the slack additionally covers what the switch + * itself pushes below the call site: the frames of the Boost.Context + * resume path and the block of callee-saved registers stored by its + * trampoline (72 bytes on x86-64, 176 bytes on aarch64). The latter is + * essential, since those spilled registers may hold GC pointers of the + * callers. Erring on the low side merely makes the GC scan a few bytes + * of (mapped) stack below the true stack pointer. + */ +constexpr size_t gcStackSwitchSlack = 512; + +/** + * Record the current stack pointer (lowered by `gcStackSwitchSlack`) + * as the saved stack pointer of `stk`, which must be the registered + * stack that the current thread is executing on and about to switch + * away from. Call this from the function performing the switch. + * + * Panics if the current stack pointer does not lie within `stk`: + * recording it anyway would make the collector scan from one stack up + * to the base of another (crashing on the guard page in between, or + * silently dropping roots), so it's better to fail right here, where + * the backtrace shows which stack switch went wrong. See + * `coroutine-gc.hh` for how such a mismatch can come about. + */ +void gcSaveStackPointer(struct GC_stack * stk); #endif } // namespace nix diff --git a/src/libexpr/include/nix/expr/eval-inline.hh b/src/libexpr/include/nix/expr/eval-inline.hh index 9163238d4f6e..b0aed3f1d5ac 100644 --- a/src/libexpr/include/nix/expr/eval-inline.hh +++ b/src/libexpr/include/nix/expr/eval-inline.hh @@ -22,7 +22,7 @@ inline void * EvalMemory::allocBytes(size_t n) p = calloc(n, 1); #endif if (!p) - throw std::bad_alloc(); + outOfMemory(); return p; } @@ -32,7 +32,7 @@ Value * EvalMemory::allocValue() #if NIX_USE_BOEHMGC /* Allocation cache for GC'd Value objects. Boehm GC is already a global resource, so thread_local is a natural solution. Multiple EvalState instances on the same thread will reuse the same cache. */ - static thread_local std::shared_ptr valueAllocCache{ + [[gnu::tls_model("initial-exec")]] static thread_local std::shared_ptr valueAllocCache{ std::allocate_shared(traceable_allocator(), nullptr)}; /* We use the boehm batch allocator to speed up allocations of Values (of which there are many). @@ -42,7 +42,7 @@ Value * EvalMemory::allocValue() if (!*valueAllocCache) { *valueAllocCache = GC_malloc_many(sizeof(Value)); if (!*valueAllocCache) - throw std::bad_alloc(); + outOfMemory(); } /* GC_NEXT is a convenience macro for accessing the first word of an object. @@ -70,13 +70,13 @@ Env & EvalMemory::allocEnv(size_t size) if (size == 1) { /* Allocation cache for size-1 Env objects. Boehm GC is already a global resource, so thread_local is a natural solution. Multiple EvalState instances on the same thread will reuse the same cache. */ - static thread_local std::shared_ptr env1AllocCache{ + [[gnu::tls_model("initial-exec")]] static thread_local std::shared_ptr env1AllocCache{ std::allocate_shared(traceable_allocator(), nullptr)}; /* see allocValue for explanations. */ if (!*env1AllocCache) { *env1AllocCache = GC_malloc_many(sizeof(Env) + sizeof(Value *)); if (!*env1AllocCache) - throw std::bad_alloc(); + outOfMemory(); } void * p = *env1AllocCache; @@ -92,34 +92,68 @@ Env & EvalMemory::allocEnv(size_t size) return *env; } -[[gnu::always_inline]] -void EvalState::forceValue(Value & v, const PosIdx pos) +/** + * An identifier of the current thread for deadlock detection, stored + * in p0 of pending/awaited thunks. We're not using std::thread::id + * because it's not guaranteed to fit. + */ +[[gnu::tls_model("initial-exec")]] extern thread_local uint32_t myEvalThreadId; + +template +void ValueStorage>>::force( + EvalState & state, PosIdx pos) { - if (v.isThunk()) { - Env * env = v.thunk().env; - assert(env || v.isBlackhole()); - Expr * expr = v.thunk().expr; - try { - v.mkBlackhole(); - if (env) [[likely]] - expr->eval(*this, *env, v); - else - ExprBlackHole::throwInfiniteRecursionError(*this, v); - } catch (...) { - handleEvalExceptionForThunk(env, expr, v, pos); - throw; - } - } else if (v.isApp()) { - Value savedApp = v; + auto p0_ = p0.load(std::memory_order_acquire); + + auto pd = static_cast(p0_ & discriminatorMask); + + if (pd == pdThunk) { try { - callFunction(*v.app().left, *v.app().right, v, pos); + // The value we get here is only valid if we can set the + // thunk to pending. + auto p1_ = p1; + + // Atomically set the thunk to "pending". + if (!p0.compare_exchange_strong( + p0_, + pdPending | (myEvalThreadId << discriminatorBits), + std::memory_order_acquire, + std::memory_order_acquire)) { + pd = static_cast(p0_ & discriminatorMask); + if (pd == pdPending || pd == pdAwaited) { + // The thunk is already "pending" or "awaited", so + // we need to wait for it. + p0_ = waitOnThunk(state, p0_); + goto done; + } + assert(pd != pdThunk); + // Another thread finished this thunk, no need to wait. + goto done; + } + + bool isApp = p1_ & discriminatorMask; + if (isApp) { + auto left = untagPointer(p0_); + auto right = untagPointer(p1_); + state.callFunction(*left, *right, (Value &) *this, pos); + } else { + auto env = untagPointer(p0_); + auto expr = untagPointer(p1_); + expr->eval(state, *env, (Value &) *this); + } } catch (...) { - handleEvalExceptionForApp(v, savedApp); + state.tryFixupBlackHolePos((Value &) *this, pos); + setStorage(new Value::Failed{.ex = std::current_exception()}); throw; } - } else if (v.isFailed()) { - handleEvalFailed(v, pos); } + + else if (pd == pdPending || pd == pdAwaited) + p0_ = waitOnThunk(state, p0_); + +done: + if (InternalType(p0_ & 0xff) == tFailed) + std::rethrow_exception((std::bit_cast(p1))->ex); } [[gnu::always_inline]] @@ -155,10 +189,10 @@ inline void EvalState::forceList(Value & v, const PosIdx pos, std::string_view e [[gnu::always_inline]] inline CallDepth EvalState::addCallDepth(const PosIdx pos) { - if (callDepth > settings.maxCallDepth) + if (CallDepth::callDepth > settings.maxCallDepth) error().atPos(pos).debugThrow(); - return CallDepth(callDepth); + return CallDepth(); }; } // namespace nix diff --git a/src/libexpr/include/nix/expr/eval-settings.hh b/src/libexpr/include/nix/expr/eval-settings.hh index d8a1fb97362f..9d077613937a 100644 --- a/src/libexpr/include/nix/expr/eval-settings.hh +++ b/src/libexpr/include/nix/expr/eval-settings.hh @@ -133,7 +133,7 @@ public: - `$HOME/.nix-defexpr/channels` - The [user channel link](@docroot@/command-ref/files/default-nix-expression.md#user-channel-link), pointing to the current state of [channels](@docroot@/command-ref/files/channels.md) for the current user. + The user channel link pointing to the current state of channels for the current user. - `nixpkgs=$NIX_STATE_DIR/profiles/per-user/root/channels/nixpkgs` @@ -143,7 +143,7 @@ public: The current state of all channels for the `root` user. - These files are set up by the [Nix installer](@docroot@/installation/installing-binary.md). + These files are set up by the Nix installer. See [`NIX_STATE_DIR`](@docroot@/command-ref/env-common.md#env-NIX_STATE_DIR) for details on the environment variable. > **Note** @@ -184,7 +184,7 @@ public: R"( If set to `true`, the Nix evaluator doesn't allow access to any files outside of - [`builtins.nixPath`](@docroot@/language/builtins.md#builtins-nixPath), + [`builtins.nixPath`](@docroot@/language/builtins.md#builtins-nixPath) or to URIs outside of [`allowed-uris`](@docroot@/command-ref/conf-file.md#conf-allowed-uris). )"}; @@ -313,7 +313,7 @@ public: "ignore-try", R"( If set to true, ignore exceptions inside 'tryEval' calls when evaluating Nix expressions in - debug mode (using the --debugger flag). By default the debugger pauses on all exceptions. + debug mode (using the --debugger flag). By default, the debugger pauses on all exceptions. )"}; Setting traceVerbose{ @@ -331,7 +331,7 @@ public: "debugger-on-trace", R"( If set to true and the `--debugger` flag is given, the following functions - enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break): + enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break). * [`builtins.trace`](@docroot@/language/builtins.md#builtins-trace) * [`builtins.traceVerbose`](@docroot@/language/builtins.md#builtins-traceVerbose) @@ -347,7 +347,7 @@ public: "debugger-on-warn", R"( If set to true and the `--debugger` flag is given, [`builtins.warn`](@docroot@/language/builtins.md#builtins-warn) - will enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break). + enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break). This is useful for debugging warnings in third-party Nix code. @@ -361,7 +361,7 @@ public: R"( If set to true, [`builtins.warn`](@docroot@/language/builtins.md#builtins-warn) throws an error when logging a warning. - This will give you a stack trace that leads to the location of the warning. + This gives you a stack trace that leads to the location of the warning. This is useful for finding information about warnings in third-party Nix code when you can not start the interactive debugger, such as when Nix is called from a non-interactive script. See [`debugger-on-warn`](#conf-debugger-on-warn). @@ -477,6 +477,59 @@ public: The default value is chosen to balance performance and memory usage. On 32 bit systems where memory is scarce, the default is a large value to reduce the amount of allocations. )"}; + + Setting lazyTrees{ + this, + false, + "lazy-trees", + R"( + If set to true, flakes and trees fetched by [`builtins.fetchTree`](@docroot@/language/builtins.md#builtins-fetchTree) are only copied to the Nix store when they're used as a dependency of a derivation. This avoids copying (potentially large) source trees unnecessarily. + )"}; + + // FIXME: this setting should really be in libflake, but it's + // currently needed in mountInput(). + Setting lazyLocks{ + this, + false, + "lazy-locks", + R"( + If enabled, Nix only includes NAR hashes in lock file entries if they're necessary to lock the input (i.e. when there is no other attribute that allows the content to be verified, like a Git revision). + This is not backward compatible with older versions of Nix. + If disabled, lock file entries always contain a NAR hash. + )"}; + + Setting evalCores{ + this, + 1, + "eval-cores", + R"( + The number of threads used to evaluate Nix expressions. This currently affects the following commands: + + * `nix search` + * `nix flake check` + * `nix flake show` + * `nix eval --json` + * Any evaluation that uses `builtins.parallel` + + The value `0` causes Nix to use all available CPU cores in the system. + + Note that enabling the debugger (`--debugger`) disables multi-threaded evaluation. + )"}; + + Setting evalMaxFibers{ + this, + 0, + "eval-max-fibers", + R"( + The maximum number of fibers that the multi-threaded evaluator keeps in existence at the same time. + + Each work item (e.g. an attribute evaluated by `nix search` or `nix flake show`) runs on its own fiber, which has its own stack. + When a fiber blocks on a value that is being evaluated by another thread, it is suspended and its worker thread starts the next work item on a new fiber. + A suspended fiber is always resumed by the thread that started it, so this limit is applied per evaluation thread (see `eval-cores`): each thread may keep at most `eval-max-fibers / eval-cores` fibers (but at least one) in existence, and when it reaches that limit, it only resumes its suspended fibers instead of starting new work items. + Lower values reduce memory usage and page faults; higher values allow more work to be in progress at the same time. + + The value `0` means four fibers per evaluation thread. + )"}; }; /** @@ -484,4 +537,10 @@ public: */ std::filesystem::path getNixDefExpr(); +/** + * Stack size for evaluator threads. This used to be 64 MiB, but macOS as deployed on GitHub Actions has a + * hard limit slightly under that, so we round it down a bit. + */ +constexpr size_t evalStackSize = 60 * 1024 * 1024; + } // namespace nix diff --git a/src/libexpr/include/nix/expr/eval.hh b/src/libexpr/include/nix/expr/eval.hh index f0a89343d856..e9a535cb79f7 100644 --- a/src/libexpr/include/nix/expr/eval.hh +++ b/src/libexpr/include/nix/expr/eval.hh @@ -6,6 +6,7 @@ #include "nix/expr/eval-profiler.hh" #include "nix/util/types.hh" #include "nix/expr/value.hh" +#include "nix/expr/root-value.hh" #include "nix/expr/nixexpr.hh" #include "nix/expr/symbol-table.hh" #include "nix/util/configuration.hh" @@ -52,6 +53,9 @@ struct SingleDerivedPath; enum RepairFlag : bool; struct MemorySourceAccessor; struct MountedSourceAccessor; +struct AsyncPathWriter; +struct Provenance; +struct Executor; namespace eval_cache { class EvalCache; @@ -62,18 +66,24 @@ class EvalCache; */ class CallDepth { - size_t & count; - public: - CallDepth(size_t & count) - : count(count) + /** + * Current Nix call stack depth, used with `max-call-depth` + * setting to throw stack overflow hopefully before we run out of + * system stack. The fiber scheduler saves/restores this on fiber + * switches, since several fibers, each with their own call chain, + * are interleaved on the same thread. + */ + [[gnu::tls_model("initial-exec")]] thread_local static size_t callDepth; + + CallDepth() { - ++count; + ++callDepth; } ~CallDepth() { - --count; + --callDepth; } }; @@ -163,11 +173,16 @@ struct Constant * Whether the constant is impure, and not available in pure mode. */ bool impureOnly = false; + + /** + * Experimental feature required to use this constant, if any. + * Only used for documentation; the caller is responsible for + * not adding the constant when the feature is disabled. + */ + std::optional experimentalFeature; }; -typedef std:: - map, traceable_allocator>> - ValMap; +typedef std::map ValMap; typedef boost::unordered_flat_map> DocCommentMap; @@ -226,7 +241,7 @@ struct StaticEvalSymbols line, column, functor, toString, right, wrong, structuredAttrs, json, allowedReferences, allowedRequisites, disallowedReferences, disallowedRequisites, maxSize, maxClosureSize, builder, args, contentAddressed, impure, outputHash, outputHashAlgo, outputHashMode, recurseForDerivations, description, self, epsilon, startSet, - operator_, key, path, prefix, outputSpecified; + operator_, key, path, prefix, outputSpecified, __meta; Expr::AstSymbols exprSymbols; @@ -279,6 +294,7 @@ struct StaticEvalSymbols .path = alloc.create("path"), .prefix = alloc.create("prefix"), .outputSpecified = alloc.create("outputSpecified"), + .__meta = alloc.create("__meta"), .exprSymbols = { .sub = alloc.create("__sub"), .lessThan = alloc.create("__lessThan"), @@ -420,6 +436,8 @@ public: std::list debugTraces; boost::unordered_flat_map> exprEnvs; + ref asyncPathWriter; + const std::shared_ptr getStaticEnv(const Expr & expr) const { auto i = exprEnvs.find(&expr); @@ -468,13 +486,7 @@ private: /** * A cache from resolved paths to values. */ - const ref, - std::equal_to, - traceable_allocator>>> - fileEvalCache; + const ref> fileEvalCache; /** * Associate source positions of certain AST nodes with their preceding doc comment, if they have one. @@ -580,7 +592,12 @@ public: /** * Mount an input on the Nix store. */ - StorePath mountInput(fetchers::Input & input, const fetchers::Input & originalInput, ref accessor); + StorePath mountInput( + fetchers::Input & input, + const fetchers::Input & originalInput, + ref accessor, + bool requireLockable, + bool forceNarHash = false); /** * Parse a Nix expression from the specified file. @@ -655,25 +672,10 @@ public: * application, call the function and overwrite `v` with the * result. Otherwise, this is a no-op. */ - inline void forceValue(Value & v, const PosIdx pos); - -private: - - /** - * Internal support function for forceValue - * - * This code is factored out so that it's not in the heavily inlined hot path. - */ - void handleEvalExceptionForThunk(Env * env, Expr * expr, Value & v, const PosIdx pos); - - /** - * Internal support function for forceValue - * - * This code is factored out so that it's not in the heavily inlined hot path. - */ - void handleEvalExceptionForApp(Value & v, const Value & savedApp); - - void handleEvalFailed(Value & v, PosIdx pos); + inline void forceValue(Value & v, const PosIdx pos) + { + v.force(*this, pos); + } void tryFixupBlackHolePos(Value & v, PosIdx pos); @@ -733,25 +735,11 @@ public: std::optional tryAttrsToString( const PosIdx pos, Value & v, NixStringContext & context, bool coerceMore = false, bool copyToStore = true); - enum class CopyLazyPaths : bool { - PreserveLazy = false, - Copy = true, - }; + StorePath devirtualize(const StorePath & path, StringMap * rewrites = nullptr); - /** - * For efficiency reasons, some store paths (as seen by the evaluator) in - * the storeFS at their content-addressed locations don't get copied to the - * store eagerly. This saves on needless I/O and possibly IPC if all the - * evaluator does is just evaluate nix expressions from those locations. - * This function copies such store objects to the store if they aren't already valid. - */ - void ensureLazyPathCopied(const StorePath & path); + SingleDerivedPath devirtualize(const SingleDerivedPath & path, StringMap * rewrites = nullptr); - /** - * Ensure that all NixStringContextElem::Opaque context elements get fetched - * to the store. - */ - void ensureLazyPathsCopied(const NixStringContext & context); + std::string devirtualize(std::string_view s, const NixStringContext & context); /** * String coercion. @@ -770,7 +758,19 @@ public: bool copyToStore = true, bool canonicalizePath = true); - StorePath copyPathToStore(NixStringContext & context, const SourcePath & path); + StorePath copyPathToStore(NixStringContext & context, const SourcePath & path, PosIdx pos); + + /** + * Compute the base name for a `SourcePath`. For non-store paths, + * this is just `SourcePath::baseName()`. But for store paths, for + * backwards compatibility, it needs to be `-source`, + * i.e. as if the path were copied to the Nix store. This results + * in a "double-copied" store path like + * `/nix/store/--source`. We don't need to + * materialize /nix/store/-source though. Still, this + * requires reading/hashing the path twice. + */ + std::string computeBaseName(const SourcePath & path, PosIdx pos); /** * Path coercion. @@ -833,13 +833,7 @@ public: /** * Internal primops not exposed to the user. */ - boost::unordered_flat_map< - std::string, - Value *, - StringViewHash, - std::equal_to<>, - traceable_allocator>> - internalPrimOps; + boost::unordered_flat_map internalPrimOps; /** * Name and documentation about every constant. @@ -919,12 +913,6 @@ private: const SourcePath & basePath, const std::shared_ptr & staticEnv); - /** - * Current Nix call stack depth, used with `max-call-depth` setting to throw stack overflow hopefully before we run - * out of system stack. - */ - size_t callDepth = 0; - public: /** @@ -1059,14 +1047,9 @@ public: /** * Coerce `v` to a path and realise it, i.e. build anything in the value's string context using `realiseContext()`. - * @param copyLazyPaths When encountering a lazy path (i.e. a string with Opaque context that's also "mounted" on - * the storeFS), fetch the store path to the store. */ SourcePath realisePath( - const PosIdx pos, - Value & v, - std::optional resolveSymlinks = SymlinkResolution::Full, - CopyLazyPaths copyLazyPaths = CopyLazyPaths::PreserveLazy); + const PosIdx pos, Value & v, std::optional resolveSymlinks = SymlinkResolution::Full); /** * Realise the given string with context, and return the string with outputs instead of downstream output @@ -1084,6 +1067,10 @@ public: DocComment getDocCommentForPos(PosIdx pos); + void waitForPath(const StorePath & path); + void waitForPath(const SingleDerivedPath & path); + void waitForAllPaths(); + private: /** @@ -1109,21 +1096,34 @@ private: Counter nrPrimOpCalls; Counter nrFunctionCalls; - bool countCalls; +public: + // Buffers allocated in Wasm guests on behalf of host functions (e.g. `read_file_v2`). + Counter nrWasmGuestAllocs; + Counter wasmGuestAllocBytes; + + Counter nrThunksAwaited; + Counter nrThunksAwaitedSlow; + Counter microsecondsWaiting; + Counter currentlyWaiting; + Counter maxWaiting; + Counter nrSpuriousWakeups; - typedef boost::unordered_flat_map> PrimOpCalls; - PrimOpCalls primOpCalls; +private: + const bool countCalls; + + typedef boost::concurrent_flat_map> PrimOpCalls; + const ref primOpCalls; - typedef boost::unordered_flat_map FunctionCalls; - FunctionCalls functionCalls; + typedef boost::concurrent_flat_map FunctionCalls; + const ref functionCalls; /** Evaluation/call profiler. */ MultiEvalProfiler profiler; void incrFunctionCall(ExprLambda * fun); - typedef boost::unordered_flat_map> AttrSelects; - AttrSelects attrSelects; + typedef boost::concurrent_flat_map> AttrSelects; + const ref attrSelects; friend struct ExprOpUpdate; friend struct ExprOpConcatLists; @@ -1139,6 +1139,64 @@ private: friend struct Value; friend class ListBuilder; + +public: + + /** + * Evaluation context. This context is propagated to worker threads when a value is evaluated + * asynchronously. + */ + struct EvalContext + { + std::shared_ptr provenance; + }; + + /** + * The evaluation context of the current execution context: a + * fiber's own context while a fiber is running (`Executor::runFiber()` + * swaps it with the context stored in the fiber record on every + * switch-in/out, which is just a pointer exchange, i.e. doesn't + * touch the `provenance` shared_ptr's atomic reference count), or + * the thread's own context otherwise. + */ + [[gnu::tls_model("initial-exec")]] thread_local static EvalContext evalContext; + + /** + * Create a work item that propagates the current evaluation context. + */ + template + auto makeWork(T && t) + { + return [this, t{std::move(t)}, evalContext(evalContext)]() mutable { + this->evalContext = std::move(evalContext); + t(); + }; + } + + /** + * Add a work item to the given work vector that propagates the current evaluation context. + */ + template + void addWork(WorkItems & work, uint8_t priority, T && t) + { + work.emplace_back(makeWork(std::move(t)), priority); + } + + template + void spawn(FuturesVector & futures, uint8_t priority, T && t) + { + futures.spawn(priority, makeWork(std::move(t))); + } + + /** + * Worker threads manager. + * + * Note: keep this last to ensure that it's destroyed first, so we + * don't have any background work items (e.g. from + * `builtins.parallel`) referring to a partially destroyed + * `EvalState`. + */ + ref executor; }; struct DebugTraceStacker @@ -1175,6 +1233,24 @@ SourcePath resolveExprPath(SourcePath path, bool addDefaultNix = true); */ bool isAllowedURI(std::string_view uri, const Strings & allowedPaths); +struct PushProvenance +{ + EvalState & state; + std::shared_ptr prev; + + PushProvenance(EvalState & state, std::shared_ptr prov) + : state(state) + { + state.evalContext.provenance.swap(prev); + state.evalContext.provenance.swap(prov); + } + + ~PushProvenance() + { + state.evalContext.provenance.swap(prev); + } +}; + } // namespace nix #include "nix/expr/eval-inline.hh" diff --git a/src/libexpr/include/nix/expr/get-drvs.hh b/src/libexpr/include/nix/expr/get-drvs.hh index 4beccabe2ad3..4f655d514ed9 100644 --- a/src/libexpr/include/nix/expr/get-drvs.hh +++ b/src/libexpr/include/nix/expr/get-drvs.hh @@ -32,6 +32,7 @@ private: */ bool failed = false; + // FIXME: make this a RootValue. const Bindings *attrs = nullptr, *meta = nullptr; const Bindings * getMeta(); diff --git a/src/libexpr/include/nix/expr/meson.build b/src/libexpr/include/nix/expr/meson.build index 7334b42f8f17..83661f1c7165 100644 --- a/src/libexpr/include/nix/expr/meson.build +++ b/src/libexpr/include/nix/expr/meson.build @@ -26,12 +26,15 @@ headers = [ config_pub_h ] + files( 'get-drvs.hh', 'json-to-value.hh', 'nixexpr.hh', + 'parallel-eval.hh', 'parser-state.hh', 'primops.hh', 'print-ambiguous.hh', 'print-options.hh', 'print.hh', + 'provenance.hh', 'repl-exit-status.hh', + 'root-value.hh', 'search-path.hh', 'static-string-data.hh', 'symbol-table.hh', diff --git a/src/libexpr/include/nix/expr/nixexpr.hh b/src/libexpr/include/nix/expr/nixexpr.hh index b13c00ad541e..85a6c0c3e417 100644 --- a/src/libexpr/include/nix/expr/nixexpr.hh +++ b/src/libexpr/include/nix/expr/nixexpr.hh @@ -8,7 +8,6 @@ #include #include -#include "nix/expr/gc-small-vector.hh" #include "nix/expr/value.hh" #include "nix/expr/symbol-table.hh" #include "nix/expr/eval-error.hh" @@ -92,8 +91,6 @@ typedef std::vector AttrSelectionPath; std::string showAttrSelectionPath(const SymbolTable & symbols, std::span attrPath); -using UpdateQueue = SmallTemporaryValueVector; - /* Abstract syntax of Nix expressions. */ struct Expr @@ -124,14 +121,6 @@ struct Expr * of thunks allocated. */ virtual Value * maybeThunk(EvalState & state, Env & env); - - /** - * Only called when performing an attrset update: `//` or similar. - * Instead of writing to a Value &, this function writes to an UpdateQueue. - * This allows the expression to perform multiple updates in a delayed manner, gathering up all the updates before - * applying them. - */ - virtual void evalForUpdate(EvalState & state, Env & env, UpdateQueue & q, std::string_view errorCtx); virtual void setName(Symbol name); virtual void setDocComment(DocComment docComment) {}; @@ -739,7 +728,7 @@ struct ExprOpNot : Expr struct name : Expr \ { \ MakeBinOpMembers(name, s) \ - } + }; MakeBinOp(ExprOpEq, "=="); MakeBinOp(ExprOpNEq, "!="); @@ -750,14 +739,7 @@ MakeBinOp(ExprOpConcatLists, "++"); struct ExprOpUpdate : Expr { -private: - /** Special case for merging of two attrsets. */ - void eval(EvalState & state, Value & v, Value & v1, Value & v2); - void evalForUpdate(EvalState & state, Env & env, UpdateQueue & q); - -public: - MakeBinOpMembers(ExprOpUpdate, "//"); - virtual void evalForUpdate(EvalState & state, Env & env, UpdateQueue & q, std::string_view errorCtx) override; + MakeBinOpMembers(ExprOpUpdate, "//") }; struct ExprConcatStrings : Expr @@ -812,20 +794,6 @@ struct ExprPos : Expr COMMON_METHODS }; -/* only used to mark thunks as black holes. */ -struct ExprBlackHole : Expr -{ - void show(const SymbolTable & symbols, std::ostream & str) const override {} - - void eval(EvalState & state, Env & env, Value & v) override; - - void bindVars(EvalState & es, const std::shared_ptr & env) override {} - - [[noreturn]] static void throwInfiniteRecursionError(EvalState & state, Value & v); -}; - -extern ExprBlackHole eBlackHole; - class Exprs { /* Thread-safe fallback resource, which might be a bit slower. */ diff --git a/src/libexpr/include/nix/expr/parallel-eval.hh b/src/libexpr/include/nix/expr/parallel-eval.hh new file mode 100644 index 000000000000..34a9b980ad00 --- /dev/null +++ b/src/libexpr/include/nix/expr/parallel-eval.hh @@ -0,0 +1,269 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include "nix/util/move-only-function.hh" +#include "nix/util/sync.hh" +#include "nix/util/logging.hh" +#include "nix/util/environment-variables.hh" +#include "nix/util/util.hh" +#include "nix/util/signals.hh" + +namespace nix { + +struct Executor +{ + using work_t = MoveOnlyFunction; + + struct Item + { + std::promise promise; + work_t work; + }; + + /** + * A work item running on its own stack. Defined in + * `parallel-eval.cc`; opaque here to keep the Boost.Context + * dependency out of this header. + */ + struct Fiber; + + using FiberPtr = std::unique_ptr; + + /** + * A pool of reusable fiber stacks, to avoid the cost of + * allocating and faulting in a fresh stack for every work + * item. Defined in `parallel-eval.cc`. + */ + struct StackPool; + + const std::unique_ptr stackPool; + + struct State + { + /** + * Fresh work items, ordered by priority. Each gets a new fiber + * when it's picked up by a worker. + */ + std::multimap queue; + + /** + * The number of fibers in existence, i.e. running, suspended + * or ready to be resumed, across all workers. Each has its own + * stack. Only used for statistics; the fiber limit is + * per-worker (see `Worker::nrLiveFibers`). + */ + size_t nrLiveFibers = 0; + }; + + /** + * A worker thread. Fibers are pinned to the worker that created + * them: a suspended fiber is only ever resumed by its owner. This + * is essential, since compilers assume that the address of a + * thread-local variable is invariant for the duration of a + * function (e.g. on AArch64 the thread pointer is read once in the + * prologue and kept in a callee-saved register), so a fiber + * resumed on another thread would keep using the thread-locals of + * its previous thread. Fresh work items, on the other hand, are + * taken from the shared `State::queue` by any worker. + * + * All fields except `thread` are protected by `Executor::state_`. + */ + struct Worker + { + std::thread thread; + + /** + * Suspended fibers owned by this worker whose awaited value + * has been finished, in FIFO order. The worker resumes these + * before starting fresh work items, so that existing work is + * drained first. + */ + std::deque readyFibers; + + /** + * Signalled when a fiber is pushed onto `readyFibers`, when a + * fresh work item is queued, or on shutdown. + */ + std::condition_variable wakeup; + + /** + * Whether the worker is blocked on `wakeup`. Producers only + * notify sleeping workers, to avoid pointless futex traffic. + */ + bool sleeping = false; + + /** + * The number of fibers owned by this worker, i.e. running, + * suspended or ready to be resumed. The worker doesn't exit + * while this is non-zero. + */ + size_t nrLiveFibers = 0; + }; + + std::atomic_bool quit{false}; + + const unsigned int evalCores; + + /** + * The maximum number of live fibers per worker (derived from + * `eval-max-fibers`). See `canStartFiber()`. + */ + const unsigned int maxFibersPerWorker; + + const bool enabled; + + Sync state_; + + /** + * The worker threads. Only mutated in the constructor; the + * `Worker` objects have stable addresses (fibers refer to their + * owner by reference). + */ + std::vector> workers; + + /** + * Note: declared last so that it is destroyed first, i.e. an + * interrupt arriving during destruction cannot touch the members + * above after they have been destroyed. + */ + const std::unique_ptr interruptCallback; + + std::atomic nrFibersSpawned{0}; + std::atomic nrFiberWakeups{0}; + std::atomic currentSuspendedFibers{0}; + std::atomic maxSuspendedFibers{0}; + std::atomic maxLiveFibers{0}; + std::atomic nrFiberStacksAllocated{0}; + + static unsigned int getEvalCores(const EvalSettings & evalSettings); + + static unsigned int getMaxFibersPerWorker(const EvalSettings & evalSettings, unsigned int evalCores); + + Executor(const EvalSettings & evalSettings); + + ~Executor(); + + void createWorker(); + + void worker(Worker & self); + + /** + * Wake up all sleeping workers, e.g. so that they can observe + * `quit` or an interrupt. + */ + void wakeupAllWorkers(); + + /** + * Create a fiber for a fresh work item, owned by `owner`. If fiber + * creation fails (e.g. stack allocation failure), the item's + * promise receives the exception and a null pointer is returned. + */ + FiberPtr makeFiber(Worker & owner, Item && item); + + /** + * Start or resume a fiber on the current thread, which must be + * its owner. On return, the fiber has either finished (its + * promise is fulfilled and the fiber is destroyed) or suspended + * itself waiting on a thunk (in which case it has been registered + * with the thunk's waiter domain). Returns whether the fiber + * finished. + */ + bool runFiber(FiberPtr fiber); + + /** + * Whether a worker may start a fresh work item on a new fiber, + * i.e. its fiber limit has not been reached. + * + * Waiting for a fiber slot cannot deadlock: the limit only applies + * to fresh work items, never to resuming ready fibers. A suspended + * fiber is waiting on a value that is being evaluated by some + * fiber or thread, so whenever fibers are suspended, either some + * fiber is runnable (and its owner, which doesn't exit while it + * owns fibers, will resume it), or a non-fiber thread (e.g. the + * main thread) is evaluating and will eventually finish the + * value. Fresh work items that haven't started hold no values + * pending, so they are never what a suspended fiber is waiting + * for. (This relies on non-fiber threads never blocking on the + * futures of queued work items while they have a value pending.) + */ + bool canStartFiber(const Worker & worker) const + { + return worker.nrLiveFibers < maxFibersPerWorker; + } + + /** + * Put a previously suspended fiber back onto its owner's ready + * queue, and wake up the owner if necessary. Called when the thunk + * it was waiting on has been finished, or on shutdown/interrupt. + */ + void enqueueFiber(FiberPtr fiber); + + /** + * Fail all queued work items that haven't started with an + * `Interrupted` exception on their promise. Called on + * shutdown/interrupt. + */ + void failQueuedItems(); + + using WorkItems = std::vector>; + + std::vector> spawn(WorkItems && items); + + [[gnu::tls_model("initial-exec")]] static thread_local bool amWorkerThread; +}; + +/** + * RAII guard that prevents the current fiber (if any) from being + * suspended while the guard is alive: if it has to wait for a value + * that is being evaluated by another thread, it blocks its worker + * thread instead of yielding it to other fibers. This is needed around + * code that keeps per-thread state which must not be interleaved with + * or migrated to other threads, such as wasmtime's stack of active + * Wasm calls. Nesting is allowed. + */ +struct FiberNoSuspend +{ + FiberNoSuspend(); + ~FiberNoSuspend(); + FiberNoSuspend(const FiberNoSuspend &) = delete; + FiberNoSuspend & operator=(const FiberNoSuspend &) = delete; + +private: + Executor::Fiber * fiber; +}; + +struct FutureVector +{ + Executor & executor; + + struct State + { + std::vector> futures; + }; + + Sync state_; + + ~FutureVector(); + + // FIXME: add a destructor that cancels/waits for all futures. + + void spawn(Executor::WorkItems && work); + + void spawn(uint8_t prioPrefix, Executor::work_t && work) + { + Executor::WorkItems items; + items.emplace_back(std::move(work), prioPrefix); + spawn(std::move(items)); + } + + void finishAll(); +}; + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/print-ambiguous.hh b/src/libexpr/include/nix/expr/print-ambiguous.hh index 07fcc337e355..7e44a6b66ebc 100644 --- a/src/libexpr/include/nix/expr/print-ambiguous.hh +++ b/src/libexpr/include/nix/expr/print-ambiguous.hh @@ -17,12 +17,6 @@ class EvalState; * * See: https://github.com/NixOS/nix/issues/9730 */ -void printAmbiguous( - EvalState & state, - Value & v, - std::ostream & str, - std::set * seen, - NixStringContext * context = nullptr, - size_t depth = 0); +void printAmbiguous(EvalState & state, Value & v, std::ostream & str, std::set * seen, size_t depth = 0); } // namespace nix diff --git a/src/libexpr/include/nix/expr/print.hh b/src/libexpr/include/nix/expr/print.hh index 8e6d0f9bf09a..229f7159d15a 100644 --- a/src/libexpr/include/nix/expr/print.hh +++ b/src/libexpr/include/nix/expr/print.hh @@ -10,7 +10,6 @@ #include #include "nix/util/fmt.hh" -#include "nix/expr/value/context.hh" #include "nix/expr/print-options.hh" namespace nix { @@ -65,12 +64,7 @@ bool isReservedKeyword(const std::string_view str); */ std::ostream & printIdentifier(std::ostream & o, std::string_view s); -void printValue( - EvalState & state, - std::ostream & str, - Value & v, - PrintOptions options = PrintOptions{}, - NixStringContext * context = nullptr); +void printValue(EvalState & state, std::ostream & str, Value & v, PrintOptions options = PrintOptions{}); /** * A partially-applied form of `printValue` which can be formatted using `<<` @@ -83,15 +77,12 @@ private: EvalState & state; Value & value; PrintOptions options; - NixStringContext * context; public: - ValuePrinter( - EvalState & state, Value & value, PrintOptions options = PrintOptions{}, NixStringContext * context = nullptr) + ValuePrinter(EvalState & state, Value & value, PrintOptions options = PrintOptions{}) : state(state) , value(value) , options(options) - , context(context) { } }; diff --git a/src/libexpr/include/nix/expr/provenance.hh b/src/libexpr/include/nix/expr/provenance.hh new file mode 100644 index 000000000000..f4cc887a6b2a --- /dev/null +++ b/src/libexpr/include/nix/expr/provenance.hh @@ -0,0 +1,23 @@ +#pragma once + +#include "nix/util/provenance.hh" + +namespace nix { + +/** + * Provenance indicating that this store path was instantiated by the `derivation` builtin function. Its main purpose is + * to record `meta` fields. + */ +struct DerivationProvenance : Provenance +{ + std::shared_ptr next; + ref meta; + + DerivationProvenance(std::shared_ptr next, ref meta) + : next(std::move(next)) + , meta(std::move(meta)) {}; + + nlohmann::json to_json() const override; +}; + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/root-value.hh b/src/libexpr/include/nix/expr/root-value.hh new file mode 100644 index 000000000000..7d5fa8767240 --- /dev/null +++ b/src/libexpr/include/nix/expr/root-value.hh @@ -0,0 +1,80 @@ +#pragma once +///@file + +#include +#include + +namespace nix { + +struct Value; + +/** + * A move-only handle rooting a Value, i.e. keeping it and everything + * reachable from it alive across garbage collections. Prefer this + * over `RootValue` unless the handle must be copyable (e.g. when it's + * captured in a `std::function`-backed lambda). + */ +class RootValue +{ + Value ** slot = nullptr; + + /** + * Clear the given slot and return it to the root value pool. + */ + void freeRootValueSlot(); + +public: + RootValue() = default; + + /** + * Allocate a slot from the root value pool, i.e. a GC-visible + * `Value *` cell that keeps the value it points to alive across + * garbage collections. Use `RootValue`/`RootValue` rather than + * calling this directly. + */ + explicit RootValue(Value * v); + + RootValue(const RootValue &) = delete; + RootValue & operator=(const RootValue &) = delete; + + RootValue(RootValue && other) noexcept + : slot(std::exchange(other.slot, nullptr)) + { + } + + RootValue & operator=(RootValue && other) noexcept + { + if (this == &other) + return *this; + if (slot) + freeRootValueSlot(); + slot = std::exchange(other.slot, nullptr); + return *this; + } + + ~RootValue() + { + reset(); + } + + /** + * Release the slot, i.e. stop rooting the value. + */ + void reset() + { + if (slot) + freeRootValueSlot(); + } + + Value *& operator*() const + { + return *slot; + } + + explicit operator bool() const + { + return slot != nullptr; + } +}; + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/symbol-table.hh b/src/libexpr/include/nix/expr/symbol-table.hh index 134cffd9e12e..231510829331 100644 --- a/src/libexpr/include/nix/expr/symbol-table.hh +++ b/src/libexpr/include/nix/expr/symbol-table.hh @@ -2,11 +2,11 @@ ///@file #include + #include "nix/expr/value.hh" -#include "nix/expr/static-string-data.hh" -#include "nix/util/chunked-vector.hh" -#include "nix/util/bump-memory-resource.hh" #include "nix/util/error.hh" +#include "nix/util/sync.hh" +#include "nix/util/alignment.hh" #include #include @@ -18,17 +18,27 @@ class SymbolValue : protected Value friend class SymbolStr; friend class SymbolTable; - uint32_t idx; - - SymbolValue() = default; - -public: operator std::string_view() const noexcept { return string_view(); } }; +struct ContiguousArena +{ + const char * data; + const size_t maxSize; + + // Put this in a separate cache line to ensure that a thread + // adding a symbol doesn't slow down threads dereferencing symbols + // by invalidating the read-only `data` field. + alignas(64) std::atomic size{0}; + + ContiguousArena(size_t maxSize); + + size_t allocate(size_t bytes); +}; + class StaticSymbolTable; /** @@ -43,6 +53,7 @@ class Symbol friend class StaticSymbolTable; private: + /// The offset of the symbol in `SymbolTable::arena`. uint32_t id; explicit constexpr Symbol(uint32_t id) noexcept @@ -74,6 +85,8 @@ public: constexpr auto operator<=>(const Symbol & other) const noexcept = default; friend class std::hash; + + constexpr static size_t alignment = alignof(SymbolValue); }; /** @@ -85,28 +98,20 @@ class SymbolStr { friend class SymbolTable; - constexpr static size_t chunkSize{65536}; - using SymbolValueStore = ChunkedVector< - SymbolValue, - chunkSize, - /*MaxChunks=*/std::numeric_limits::max() / chunkSize>; - const SymbolValue * s; struct Key { using HashType = boost::hash; - SymbolValueStore & store; std::string_view s; std::size_t hash; - std::pmr::memory_resource & resource; + ContiguousArena & arena; - Key(SymbolValueStore & store, std::string_view s, std::pmr::memory_resource & stringMemory) - : store(store) - , s(s) + Key(std::string_view s, ContiguousArena & arena) + : s(s) , hash(HashType{}(s)) - , resource(stringMemory) + , arena(arena) { } }; @@ -117,22 +122,7 @@ public: { } - SymbolStr(const Key & key) - { - auto size = key.s.size(); - if (size >= std::numeric_limits::max()) { - throw Error("Size of symbol exceeds 4GiB and cannot be stored"); - } - // for multi-threaded implementations: lock store and allocator here - const auto & [v, idx] = key.store.add(SymbolValue{}); - if (size == 0) { - v.mkStringNoCopy(""_sds, nullptr); - } else { - v.mkStringNoCopy(StringData::make(key.resource, key.s)); - } - v.idx = idx; - this->s = &v; - } + SymbolStr(const Key & key); bool operator==(std::string_view s2) const noexcept { @@ -161,11 +151,7 @@ public: [[gnu::always_inline]] bool empty() const noexcept { - auto * p = &s->string_data(); - // Save a dereference in the sentinel value case - if (p == &""_sds) - return true; - return p->size() == 0; + return !s->string_data().size(); } [[gnu::always_inline]] @@ -180,11 +166,6 @@ public: return s; } - explicit operator Symbol() const noexcept - { - return Symbol{s->idx + 1}; - } - struct Hash { using is_transparent = void; @@ -222,6 +203,11 @@ public: return operator()(b, a); } }; + + constexpr static size_t computeSize(std::string_view s) + { + return alignUp(sizeof(Value) + sizeof(StringData) + s.size() + 1, Symbol::alignment); + } }; class SymbolTable; @@ -241,6 +227,7 @@ class StaticSymbolTable std::array symbols; std::size_t size = 0; + std::size_t nextId = alignof(SymbolValue); public: constexpr StaticSymbolTable() = default; @@ -249,8 +236,9 @@ public: { /* No need to check bounds because out of bounds access is a compilation error. */ - auto sym = Symbol(size + 1); //< +1 because Symbol with id = 0 is reserved + auto sym = Symbol(nextId); symbols[size++] = {str, sym}; + nextId += SymbolStr::computeSize(str); return sym; } @@ -264,79 +252,71 @@ public: class SymbolTable { private: - /* Thread-safe fallback resource, which might be a bit slower. */ - std::pmr::synchronized_pool_resource fallbackResource{}; /** * SymbolTable is an append only data structure. * During its lifetime the monotonic buffer holds all strings and nodes, if the symbol set is node based. */ - BumpMemoryResource buffer{BumpMemoryResource::defaultReserveSize, &fallbackResource}; - SymbolStr::SymbolValueStore store; + ContiguousArena arena; /** - * Transparent lookup of string view for a pointer to a ChunkedVector entry -> return offset into the store. - * ChunkedVector references are never invalidated. + * Transparent lookup of string view for a pointer to a + * SymbolValue in the arena. */ - boost::concurrent_flat_set symbols{SymbolStr::chunkSize}; + boost::concurrent_flat_set symbols; public: SymbolTable(const StaticSymbolTable & staticSymtab) + : arena(1 << 30) { + // Reserve symbol ID 0 and ensure alignment of the first allocation. + arena.allocate(Symbol::alignment); + staticSymtab.copyIntoSymbolTable(*this); } /** * Converts a string into a symbol. */ - Symbol create(std::string_view s) - { - Symbol res; - symbols.insert_and_cvisit( - SymbolStr::Key{store, s, buffer}, - [&](const auto & key) { res = Symbol(key); }, - [&](const auto & key) { res = Symbol(key); }); - return res; - } + Symbol create(std::string_view s); std::vector resolve(const std::span & symbols) const { std::vector result; result.reserve(symbols.size()); - for (auto sym : symbols) + for (auto & sym : symbols) result.push_back((*this)[sym]); return result; } SymbolStr operator[](Symbol s) const { - uint32_t idx = s.id - uint32_t(1); - if (idx >= store.size()) - unreachable(); - - /* Since ChunkedVector itself doesn't provide happens-before guarantees - about the object construction, we rely on other synchronisation - primitives to ensure happens-before relation between insertion - into the ChunkedVector and the load here. If another thread interns - the symbol via the symbols map, then the happens-before guarantee - should be provided by the concurrent_map itself. If thread B gets - Symbol from thread A through other means then it should also ensure - happens-before relation. */ - - return store[idx]; + assert(s.id); + // Note: we don't check arena.size here to avoid a dependency + // on other threads creating new symbols. + return SymbolStr(*reinterpret_cast(arena.data + s.id)); } - [[gnu::always_inline]] size_t size() const noexcept { - return store.size(); + return symbols.size(); } - size_t totalSize() const; + size_t totalSize() const + { + return arena.size; + } template void dump(T callback) const { - store.forEach(callback); + std::string_view left{arena.data, arena.size}; + left = left.substr(Symbol::alignment); + while (!left.empty()) { + auto v = reinterpret_cast(left.data()); + callback(v->string_view()); + left = left.substr( + alignUp(sizeof(SymbolValue) + sizeof(StringData) + v->string_view().size() + 1, Symbol::alignment)); + } } }; diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 913a44bb418b..aadb05f6200d 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -1,6 +1,7 @@ #pragma once ///@file +#include #include #include #include @@ -9,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -22,15 +24,24 @@ #include #include -#if defined(__x86_64__) && defined(__SSE2__) -# include -#endif - namespace nix { struct Value; class BindingsBuilder; +static constexpr int discriminatorBits = 3; + +enum PrimaryDiscriminator : int { + pdSingleDWord = 0, + pdThunk = 1, + pdPending = 2, + pdAwaited = 3, + pdPairOfPointers = 4, + pdListN = 5, // FIXME: get rid of this by putting the size in the first word + pdString = 6, + pdPath = 7, // FIXME: get rid of this by ditching the `accessor` field +}; + /** * Internal type discriminator, which is more detailed than `ValueType`, as * it specifies the exact representation used (for types that have multiple @@ -40,32 +51,50 @@ class BindingsBuilder; * about how this is mapped into the alignment bits to save significant memory. * This also restricts the number of internal types represented with distinct memory layouts. */ -enum InternalType { - tUninitialized = 0, - /* layout: Single/zero field payload */ - tInt = 1, - tBool, - tNull, - tFloat, - tFailed, - tExternal, - tPrimOp, - tAttrs, - /* layout: Pair of pointers payload */ - tFirstPairOfPointers, - tListSmall = tFirstPairOfPointers, - tPrimOpApp, - tApp, - tThunk, - tLambda, - tLastPairOfPointers = tLambda, - /* layout: Single untaggable field */ - tFirstSingleUntaggable, - tListN = tFirstSingleUntaggable, - tString, - tPath, - tNumberOfInternalTypes, // Must be last -}; +typedef enum { + /* Values that have more type bits in the first word, and the + payload (a single word) in the second word. */ + tUninitialized = PrimaryDiscriminator::pdSingleDWord | (0 << discriminatorBits), + tInt = PrimaryDiscriminator::pdSingleDWord | (1 << discriminatorBits), + tFloat = PrimaryDiscriminator::pdSingleDWord | (2 << discriminatorBits), + tBool = PrimaryDiscriminator::pdSingleDWord | (3 << discriminatorBits), + tNull = PrimaryDiscriminator::pdSingleDWord | (4 << discriminatorBits), + tAttrs = PrimaryDiscriminator::pdSingleDWord | (5 << discriminatorBits), + tPrimOp = PrimaryDiscriminator::pdSingleDWord | (6 << discriminatorBits), + tFailed = PrimaryDiscriminator::pdSingleDWord | (7 << discriminatorBits), + tExternal = PrimaryDiscriminator::pdSingleDWord | (8 << discriminatorBits), + + /* Thunks. */ + tThunk = PrimaryDiscriminator::pdThunk | (0 << discriminatorBits), + tApp = PrimaryDiscriminator::pdThunk | (1 << discriminatorBits), + + tPending = PrimaryDiscriminator::pdPending, + tAwaited = PrimaryDiscriminator::pdAwaited, + + /* Values that consist of two pointers. The second word contains + more type bits in its alignment niche. */ + tListSmall = PrimaryDiscriminator::pdPairOfPointers | (0 << discriminatorBits), + tPrimOpApp = PrimaryDiscriminator::pdPairOfPointers | (1 << discriminatorBits), + tLambda = PrimaryDiscriminator::pdPairOfPointers | (2 << discriminatorBits), + + /* Special values. */ + tListN = PrimaryDiscriminator::pdListN, + tString = PrimaryDiscriminator::pdString, + tPath = PrimaryDiscriminator::pdPath, +} InternalType; + +/** + * Return true if `type` denotes a "finished" value, i.e. a weak-head + * normal form. + * + * Note that tPrimOpApp is considered "finished" because it represents + * a primop call with an incomplete number of arguments, and therefore + * cannot be evaluated further. + */ +inline bool isFinished(InternalType t) +{ + return t != tUninitialized && t != tThunk && t != tApp && t != tPending && t != tAwaited; +} /** * This type abstracts over all actual value types in the language, @@ -91,7 +120,6 @@ class Bindings; struct Env; struct Expr; struct ExprLambda; -struct ExprBlackHole; struct PrimOp; class Symbol; class SymbolStr; @@ -296,7 +324,7 @@ namespace detail { /** * Implementation mixin class for defining the public types - * In can be inherited from by the actual ValueStorage implementations + * In can be inherited by the actual ValueStorage implementations * for free due to Empty Base Class Optimization (EBCO). */ struct ValueBase @@ -432,61 +460,9 @@ struct ValueBase Value * const * elems; }; - /** - * Wrapper that stores a std::exception_ptr on the GC heap with a finaliser - * that runs the exception_ptr destructor (which is refcounted internally). - * This is not a part of the Failed structure to avoid cycles with finalisers, - * which Boehm warns about. - */ - struct ExceptionRef : gc_cleanup - { - ExceptionRef(std::exception_ptr ex) - : ex(std::move(ex)) - { - assert(this->ex); - } - - ExceptionRef(ExceptionRef &&) = delete; - ExceptionRef(const ExceptionRef &) = delete; - ExceptionRef & operator=(ExceptionRef &&) = delete; - ExceptionRef & operator=(const ExceptionRef &) = delete; - - /* To appease -Wweak-vtables. */ - virtual ~ExceptionRef(); - - std::exception_ptr ex; - }; - struct Failed : gc { - ExceptionRef * exRef; - - /** - * Optional value for recovering `RecoverableEvalError` - * Must be set iff `ex` is an instance of `RecoverableEvalError`. - */ - Value * recoveryValue; - - Failed(std::exception_ptr ex, Value * recoveryValue) - : exRef(new /* ExceptionRef : gc_cleanup */ ExceptionRef(ex)) - , recoveryValue(recoveryValue) - { - } - - [[noreturn]] void rethrow() const - { - try { - std::rethrow_exception(exRef->ex); - } catch (BaseError & e) { - /* Rethrow the copy of the exception - not the original one. - Stack tracing mechanisms rely on being able to modify the exceptions - they catch by reference. */ - e.throwClone(); - } catch (...) { - throw; - } - unreachable(); - } + std::exception_ptr ex; }; }; @@ -586,11 +562,6 @@ protected: { return internalType; } - - static bool isAtomic() - { - return false; - } }; namespace detail { @@ -623,16 +594,44 @@ class alignas(16) }; using PackedPointer = typename PackedPointerTypeStruct::type; - using Payload = std::array; -#if defined(__x86_64__) && defined(__SSE2__) - __m128i payloadWords; -#else - Payload payloadWords = {}; -#endif - - static constexpr int discriminatorBits = 3; + + /** + * For multithreaded evaluation, we have to make sure that thunks/apps + * (the only mutable types of values) are updated in a safe way. A + * value can have the following states (see `force()`): + * + * * "thunk"/"app". When forced, this value transitions to + * "pending". The current thread will evaluate the + * thunk/app. When done, it will override the value with the + * result. If the value is at that point in the "awaited" state, + * the thread will wake up any waiting threads. + * + * * "pending". This means it's currently being evaluated. If + * another thread forces this value, it transitions to "awaited" + * and the thread will wait for the value to be updated (see + * `waitOnThunk()`). + * + * * "awaited". Like pending, only it means that there already are + * one or more threads waiting for this thunk. + * + * To ensure race-free access, the non-atomic word `p1` must + * always be updated before `p0`. Writes to `p0` should use + * *release* semantics (so that `p1` and any referenced values become + * visible to threads that read `p0`), and reads from `p0` should + * use `*acquire* semantics. + * + * Note: at some point, we may want to switch to 128-bit atomics + * so that `p0` and `p1` can be updated together + * atomically. However, 128-bit atomics are a bit problematic at + * present on x86_64 (see + * e.g. https://ibraheem.ca/posts/128-bit-atomics/). + */ + std::atomic p0{0}; + PackedPointer p1{0}; + static constexpr PackedPointer discriminatorMask = (PackedPointer(1) << discriminatorBits) - 1; + // FIXME: move/update /** * The value is stored as a pair of 8-byte double words. All pointers are assumed * to be 8-byte aligned. This gives us at most 6 bits of discriminator bits @@ -662,63 +661,6 @@ class alignas(16) * The primary discriminator with value 0 is reserved for uninitialized Values, * which are useful for diagnostics in C bindings. */ - enum PrimaryDiscriminator : int { - pdUninitialized = 0, - pdSingleDWord, //< layout: Single/zero field payload - /* The order of these enumerations must be the same as in InternalType. */ - pdListN, //< layout: Single untaggable field. - pdString, - pdPath, - pdPairOfPointers, //< layout: Pair of pointers payload - }; - -#if defined(__x86_64__) && defined(__SSE2__) - /* Why do we even bother with hand-rolling these arch-specific intrinsics - * and don't use libatomic directly for 16 byte atomics? Here's why: - * - https://gcc.gnu.org/legacy-ml/gcc/2018-02/msg00224.html - * - https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84563 - * - * Basically, we don't really ever want to go through libatomic. As is - * so happens on x86_64 with AVX, MOVDQA/MOVAPS instructions (16-byte aligned - * 128-bit loads and stores) are atomic [^]. Note that - * these instructions are not part of AVX but rather SSE2, which is x86_64-v1. - * They are just not guaranteed to be atomic without AVX. - * - * For more details see: - * - [^] Intel® 64 and IA-32 Architectures Software Developer’s Manual (10.1.1 Guaranteed Atomic Operations). - * - https://patchwork.sourceware.org/project/gcc/patch/YhxkfzGEEQ9KHbBC@tucnak/ - * - https://ibraheem.ca/posts/128-bit-atomics/ - * - https://rigtorp.se/isatomic/ - */ - [[gnu::always_inline]] - void updatePayload(Payload payload) noexcept - { - /* This intrinsic corresponds to MOVAPS. Note that Value (and thus the first member - payloadWords) is 16 bytes aligned. */ - _mm_store_si128(&payloadWords, std::bit_cast<__m128i>(payload)); - } - - [[gnu::always_inline]] - Payload loadPayload() const noexcept - { - /* This intrinsic corresponds to MOVDQA. Note that Value (and thus the first member - payloadWords) is 16 bytes aligned. */ - __m128i res = _mm_load_si128(&payloadWords); - return std::bit_cast(res); - } -#else - [[gnu::always_inline]] - void updatePayload(Payload payload) noexcept - { - payloadWords = payload; - } - - [[gnu::always_inline]] - Payload loadPayload() const noexcept - { - return payloadWords; - } -#endif template requires std::is_pointer_v @@ -727,9 +669,9 @@ class alignas(16) return std::bit_cast(val & ~discriminatorMask); } - PrimaryDiscriminator getPrimaryDiscriminator(PackedPointer firstDWord) const noexcept + PrimaryDiscriminator getPrimaryDiscriminator() const noexcept { - return static_cast(firstDWord & discriminatorMask); + return static_cast(p0 & discriminatorMask); } static void assertAligned(PackedPointer val) noexcept @@ -737,200 +679,178 @@ class alignas(16) assert((val & discriminatorMask) == 0 && "Pointer is not 8 bytes aligned"); } + void finish(PackedPointer p0_, PackedPointer p1_) + { + // Note: p1 *must* be updated before p0. + p1 = p1_; + p0_ = p0.exchange(p0_, std::memory_order_release); + + auto pd = static_cast(p0_ & discriminatorMask); + if (pd == pdPending) + // Nothing to do; no thread is waiting on this thunk. + ; + else if (pd == pdAwaited) + // Slow path: wake up the threads that are waiting on this + // thunk. + notifyWaiters(); + else if (pd == pdThunk) + unreachable(); + } + template void setSingleDWordPayload(PackedPointer untaggedVal) noexcept { - Payload payload; - /* There's plenty of free upper bits in the first dword, which is - used only for the discriminator. */ - payload[0] = static_cast(pdSingleDWord) | (static_cast(type) << discriminatorBits); - payload[1] = untaggedVal; - updatePayload(payload); + /* There's plenty of free upper bits in the first byte, which + is used only for the discriminator. */ + finish(static_cast(type), untaggedVal); } template void setUntaggablePayload(T * firstPtrField, U untaggableField) noexcept { - Payload payload; static_assert(discriminator >= pdListN && discriminator <= pdPath); auto firstFieldPayload = std::bit_cast(firstPtrField); assertAligned(firstFieldPayload); - payload[0] = static_cast(discriminator) | firstFieldPayload; - payload[1] = std::bit_cast(untaggableField); - updatePayload(payload); + finish(static_cast(discriminator) | firstFieldPayload, std::bit_cast(untaggableField)); } template void setPairOfPointersPayload(T * firstPtrField, U * secondPtrField) noexcept { - Payload payload; - static_assert(type >= tFirstPairOfPointers && type <= tLastPairOfPointers); - { - auto firstFieldPayload = std::bit_cast(firstPtrField); - assertAligned(firstFieldPayload); - payload[0] = static_cast(pdPairOfPointers) | firstFieldPayload; - } - { - auto secondFieldPayload = std::bit_cast(secondPtrField); - assertAligned(secondFieldPayload); - payload[1] = (type - tFirstPairOfPointers) | secondFieldPayload; - } - updatePayload(payload); - } - - template - requires std::is_pointer_v && std::is_pointer_v - void getPairOfPointersPayload(T & firstPtrField, U & secondPtrField) const noexcept - { - Payload payload = loadPayload(); - firstPtrField = untagPointer(payload[0]); - secondPtrField = untagPointer(payload[1]); - } - -public: - ValueStorage() - { - updatePayload({}); - } - - ValueStorage(const ValueStorage & other) - { - updatePayload(other.loadPayload()); - } - - ValueStorage & operator=(const ValueStorage & other) - { - updatePayload(other.loadPayload()); - return *this; + static_assert(type >= tListSmall && type <= tLambda); + auto firstFieldPayload = std::bit_cast(firstPtrField); + assertAligned(firstFieldPayload); + auto secondFieldPayload = std::bit_cast(secondPtrField); + assertAligned(secondFieldPayload); + finish( + static_cast(pdPairOfPointers) | firstFieldPayload, + ((type - tListSmall) >> discriminatorBits) | secondFieldPayload); } - ValueStorage(ValueStorage && other) noexcept + template + void setThunkPayload(T * firstPtrField, U * secondPtrField) noexcept { - updatePayload(other.loadPayload()); - other.updatePayload({}); // Zero out rhs + static_assert(type >= tThunk && type <= tApp); + auto secondFieldPayload = std::bit_cast(secondPtrField); + assertAligned(secondFieldPayload); + p1 = ((type - tThunk) >> discriminatorBits) | secondFieldPayload; + auto firstFieldPayload = std::bit_cast(firstPtrField); + assertAligned(firstFieldPayload); + // Note: awaited values can never become a thunk, so no need + // to check for waiters. + p0.store(static_cast(pdThunk) | firstFieldPayload, std::memory_order_release); } - ValueStorage & operator=(ValueStorage && other) noexcept + template + requires std::is_pointer_v && std::is_pointer_v + void getPairOfPointersPayload(T & firstPtrField, U & secondPtrField) const noexcept { - updatePayload(other.loadPayload()); - other.updatePayload({}); // Zero out rhs - return *this; + firstPtrField = untagPointer(p0); + secondPtrField = untagPointer(p1); } - ~ValueStorage() noexcept {} - protected: - static bool isAtomic(); - /** Get internal type currently occupying the storage. */ InternalType getInternalType() const noexcept { - Payload payload = loadPayload(); - switch (auto pd = getPrimaryDiscriminator(payload[0])) { - case pdUninitialized: - /* Discriminator value of zero is used to distinguish uninitialized values. */ - return tUninitialized; + switch (auto pd = getPrimaryDiscriminator()) { case pdSingleDWord: - /* Payloads that only use up a single double word store the InternalType - in the upper bits of the first double word. */ - return InternalType(payload[0] >> discriminatorBits); + /* Payloads that only use up a single double word store + the full InternalType in the first byte. */ + return InternalType(p0 & 0xff); + case pdThunk: + return static_cast(tThunk + ((p1 & discriminatorMask) << discriminatorBits)); + case pdPending: + return tPending; + case pdAwaited: + return tAwaited; + case pdPairOfPointers: + return static_cast(tListSmall + ((p1 & discriminatorMask) << discriminatorBits)); /* The order must match that of the enumerations defined in InternalType. */ case pdListN: case pdString: case pdPath: - return static_cast(tFirstSingleUntaggable + (pd - pdListN)); - case pdPairOfPointers: - return static_cast(tFirstPairOfPointers + (payload[1] & discriminatorMask)); + return static_cast(tListN + (pd - pdListN)); [[unlikely]] default: nixUnreachableWhenHardened(); } } -#define NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(TYPE, MEMBER_A, MEMBER_B) \ - \ - void getStorage(TYPE & val) const noexcept \ - { \ - getPairOfPointersPayload(val MEMBER_A, val MEMBER_B); \ - } \ - \ - void setStorage(TYPE val) noexcept \ - { \ - setPairOfPointersPayload>(val MEMBER_A, val MEMBER_B); \ +#define NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(TYPE, SET, MEMBER_A, MEMBER_B) \ + \ + void getStorage(TYPE & val) const noexcept \ + { \ + getPairOfPointersPayload(val MEMBER_A, val MEMBER_B); \ + } \ + \ + void setStorage(TYPE val) noexcept \ + { \ + SET>(val MEMBER_A, val MEMBER_B); \ } - NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(SmallList, [0], [1]) - NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(PrimOpApplicationThunk, .left, .right) - NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(FunctionApplicationThunk, .left, .right) - NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(ClosureThunk, .env, .expr) - NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(Lambda, .env, .fun) + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(SmallList, setPairOfPointersPayload, [0], [1]) + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(PrimOpApplicationThunk, setPairOfPointersPayload, .left, .right) + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(Lambda, setPairOfPointersPayload, .env, .fun) + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(FunctionApplicationThunk, setThunkPayload, .left, .right) + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(ClosureThunk, setThunkPayload, .env, .expr) #undef NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS void getStorage(NixInt & integer) const noexcept { - Payload payload = loadPayload(); /* PackedPointerType -> int64_t here is well-formed, since the standard requires this conversion to follow 2's complement rules. This is just a no-op. */ - integer = NixInt(payload[1]); + integer = NixInt(p1); } void getStorage(bool & boolean) const noexcept { - Payload payload = loadPayload(); - boolean = payload[1]; + boolean = p1; } void getStorage(Null & null) const noexcept {} void getStorage(NixFloat & fpoint) const noexcept { - Payload payload = loadPayload(); - fpoint = std::bit_cast(payload[1]); + fpoint = std::bit_cast(p1); } void getStorage(ExternalValueBase *& external) const noexcept { - Payload payload = loadPayload(); - external = std::bit_cast(payload[1]); + external = std::bit_cast(p1); } void getStorage(PrimOp *& primOp) const noexcept { - Payload payload = loadPayload(); - primOp = std::bit_cast(payload[1]); + primOp = std::bit_cast(p1); } void getStorage(Bindings *& attrs) const noexcept { - Payload payload = loadPayload(); - attrs = std::bit_cast(payload[1]); + attrs = std::bit_cast(p1); } void getStorage(List & list) const noexcept { - Payload payload = loadPayload(); - list.elems = untagPointer(payload[0]); - list.size = payload[1]; + list.elems = untagPointer(p0); + list.size = p1; } void getStorage(StringWithContext & string) const noexcept { - Payload payload = loadPayload(); - string.context = untagPointer(payload[0]); - string.str = std::bit_cast(payload[1]); + string.context = untagPointer(p0); + string.str = std::bit_cast(p1); } void getStorage(Path & path) const noexcept { - Payload payload = loadPayload(); - path.accessor = untagPointer(payload[0]); - path.path = std::bit_cast(payload[1]); + path.accessor = untagPointer(p0); + path.path = std::bit_cast(p1); } void getStorage(Failed *& failed) const noexcept { - Payload payload = loadPayload(); - failed = std::bit_cast(payload[1]); + failed = std::bit_cast(p1); } void setStorage(NixInt integer) noexcept @@ -987,8 +907,80 @@ protected: { setSingleDWordPayload(std::bit_cast(failed)); } + + ValueStorage() {} + + ValueStorage(const ValueStorage & v) + { + *this = v; + } + + /** + * Copy a value. This is not allowed to be a thunk to avoid + * accidental work duplication. + */ + ValueStorage & operator=(const ValueStorage & v) + { + auto p0_ = v.p0.load(std::memory_order_acquire); + auto p1_ = v.p1; // must be loaded after p0 + auto pd = static_cast(p0_ & discriminatorMask); + if (pd == pdThunk || pd == pdPending || pd == pdAwaited) + unreachable(); + finish(p0_, p1_); + return *this; + } + +public: + + /** + * Check whether forcing this value requires a trivial amount of + * computation. A value is trivial if it's finished or if it's a + * thunk whose expression is an attrset with no dynamic + * attributes, a lambda or a list. Note that it's up to the caller + * to check whether the members of those attrsets or lists must be + * trivial. + */ + bool isTrivial() const; + + inline void reset() + { + p1 = 0; + p0.store(0, std::memory_order_relaxed); + } + + /// Only used for testing. + inline void mkBlackhole() + { + p0.store(pdPending, std::memory_order_relaxed); + } + + void force(EvalState & state, PosIdx pos); + +private: + + /** + * Given a thunk that was observed to be in the pending or awaited + * state, wait for it to finish. Returns the first word of the + * value. + */ + PackedPointer waitOnThunk(EvalState & state, PackedPointer p0); + + /** + * Wake up any threads that are waiting on this value. + */ + void notifyWaiters(); }; +template<> +void ValueStorage::notifyWaiters(); + +template<> +ValueStorage::PackedPointer +ValueStorage::waitOnThunk(EvalState & state, PackedPointer p0); + +template<> +bool ValueStorage::isTrivial() const; + /** * View into a list of Value * that is itself immutable. * @@ -1230,11 +1222,16 @@ public: void print(EvalState & state, std::ostream & str, PrintOptions options = PrintOptions{}); + // FIXME: optimize, only look at first word + inline bool isFinished() const + { + return nix::isFinished(getInternalType()); + } + // Functions needed to distinguish the type // These should be removed eventually, by putting the functionality that's // needed by callers into methods of this type - // type() == nThunk inline bool isThunk() const { return isa(); @@ -1245,7 +1242,11 @@ public: return isa(); } - inline bool isBlackhole() const; + inline bool isBlackhole() const + { + auto t = getInternalType(); + return t == tPending || t == tAwaited; + } // type() == nFunction inline bool isLambda() const @@ -1271,17 +1272,13 @@ public: /** * Returns the normal type of a Value. This only returns nThunk if * the Value hasn't been forceValue'd - * - * @param invalidIsThunk Instead of UB an an invalid (probably - * 0, so uninitialized) internal type, return `nThunk`. */ - template inline ValueType type() const { /* Explicit lookup table. switch() might compile down (and it does at least with GCC 14) to a jump table. Let's help the compiler a bit here. */ static constexpr auto table = [] { - std::array t{}; + std::array t{}; t[tUninitialized] = nThunk; t[tInt] = nInt; t[tBool] = nBool; @@ -1302,15 +1299,7 @@ public: return t; }(); - auto it = getInternalType(); - if (it == tUninitialized || it >= tNumberOfInternalTypes) [[unlikely]] { - if constexpr (invalidIsThunk) - return nThunk; - else - nixUnreachableWhenHardened(); - } - - return table[it]; + return table[getInternalType()]; } /** @@ -1401,8 +1390,6 @@ public: setStorage(Lambda{.env = e, .fun = f}); } - inline void mkBlackhole(); - void mkPrimOp(PrimOp * p); inline void mkPrimOpApp(Value * l, Value * r) noexcept @@ -1425,9 +1412,9 @@ public: setStorage(n); } - inline void mkFailed(std::exception_ptr e, Value * recovery) noexcept + inline void mkFailed() noexcept { - setStorage(new Value::Failed(e, recovery)); + setStorage(new Value::Failed{.ex = std::current_exception()}); } bool isList() const noexcept @@ -1447,13 +1434,6 @@ public: PosIdx determinePos(const PosIdx pos) const; - /** - * Check whether forcing this value requires a trivial amount of - * computation. In particular, function applications are - * non-trivial. - */ - bool isTrivial() const; - SourcePath path() const { return SourcePath( @@ -1515,6 +1495,7 @@ public: return getStorage(); } + // FIXME: remove this since reading it is racy. ClosureThunk thunk() const noexcept { return getStorage(); @@ -1525,6 +1506,7 @@ public: return getStorage(); } + // FIXME: remove this since reading it is racy. FunctionApplicationThunk app() const noexcept { return getStorage(); @@ -1545,43 +1527,13 @@ public: return getStorage().accessor; } - Failed & failed() const noexcept + Failed * failed() const noexcept { - auto p = getStorage(); - assert(p); - return *p; + return getStorage(); } }; -extern ExprBlackHole eBlackHole; - -bool Value::isBlackhole() const -{ - return isThunk() && thunk().expr == (Expr *) &eBlackHole; -} - -void Value::mkBlackhole() -{ - mkThunk(nullptr, (Expr *) &eBlackHole); -} - typedef std::vector> ValueVector; -typedef boost::unordered_flat_map< - Symbol, - Value *, - std::hash, - std::equal_to, - traceable_allocator>> - ValueMap; -typedef std::map, traceable_allocator>> - ValueVectorMap; - -/** - * A value allocated in traceable memory. - */ -typedef std::shared_ptr RootValue; - -RootValue allocRootValue(Value * v); void forceNoNullByte(std::string_view s, std::function = nullptr); } // namespace nix diff --git a/src/libexpr/include/nix/expr/value/context.hh b/src/libexpr/include/nix/expr/value/context.hh index 8fc948622768..29cf9ca0b841 100644 --- a/src/libexpr/include/nix/expr/value/context.hh +++ b/src/libexpr/include/nix/expr/value/context.hh @@ -66,7 +66,31 @@ struct NixStringContextElem */ using Built = SingleDerivedPath::Built; - using Raw = std::variant; + /** + * A store path that will not result in a store reference when + * used in a derivation or toFile. + * + * When you apply `builtins.toString` to a path value representing + * a path in the Nix store (as is the case with flake inputs), + * historically you got a string without context + * (e.g. `/nix/store/...-source`). This is broken, since it allows + * you to pass a store path to a derivation/toFile without a + * proper store reference. This is especially a problem with lazy + * trees, since the store path is a virtual path that doesn't + * exist. + * + * For backwards compatibility, and to warn users about this + * unsafe use of `toString`, we keep track of such strings as a + * special type of context. + */ + struct Path + { + StorePath storePath; + + GENERATE_CMP(Path, me->storePath); + }; + + using Raw = std::variant; Raw raw; @@ -102,4 +126,10 @@ struct NixStringContextElem */ typedef std::set NixStringContext; +/** + * Returns false if `context` has no elements other than + * `NixStringContextElem::Path`. + */ +bool hasContext(const NixStringContext & context); + } // namespace nix diff --git a/src/libexpr/json-to-value.cc b/src/libexpr/json-to-value.cc index d218a0beaefc..cee95813fbee 100644 --- a/src/libexpr/json-to-value.cc +++ b/src/libexpr/json-to-value.cc @@ -3,6 +3,7 @@ #include "nix/expr/eval.hh" #include +#include #include using json = nlohmann::json; @@ -19,7 +20,18 @@ class JSONSax : nlohmann::json_sax { protected: std::unique_ptr parent; - RootValue v; + + /** + * The value being built by this state, or null. Kept alive by + * `JSONSax::rootValues`. + */ + Value * v = nullptr; + + /** + * Reference to `JSONSax::rootValues`. + */ + ValueVector & rootValues; + public: virtual std::unique_ptr resolve(EvalState &) { @@ -28,11 +40,13 @@ class JSONSax : nlohmann::json_sax explicit JSONState(std::unique_ptr && p) : parent(std::move(p)) + , rootValues(parent->rootValues) { } - explicit JSONState(Value * v) - : v(allocRootValue(v)) + JSONState(Value * v, ValueVector & rootValues) + : v(v) + , rootValues(rootValues) { } @@ -40,9 +54,13 @@ class JSONSax : nlohmann::json_sax Value & value(EvalState & state) { - if (!v) - v = allocRootValue(state.allocValue()); - return **v; + if (!v) { + v = state.allocValue(); + /* Root the value for the duration of the parse (see + `JSONSax::rootValues`). */ + rootValues.push_back(v); + } + return *v; } virtual ~JSONState() {} @@ -53,14 +71,29 @@ class JSONSax : nlohmann::json_sax class JSONObjectState : public JSONState { using JSONState::JSONState; - ValueMap attrs; + + /** + * Note: deliberately not `ValueMap`: the values are rooted via + * `JSONSax::rootValues`, so there is no need for a GC-visible + * (uncollectable) container, whose allocations would take the + * global GC allocation lock. The inline capacity avoids a heap + * allocation for small objects. + */ + boost::container::small_vector, 16> attrs; std::unique_ptr resolve(EvalState & state) override { + /* JSON allows duplicate keys, and the last occurrence + wins. `Bindings` requires unique keys, so drop all but + the last entry of every run of equal keys. The stable + sort keeps equal keys in insertion order. */ + std::stable_sort( + attrs.begin(), attrs.end(), [](const auto & a, const auto & b) { return a.first < b.first; }); auto attrs2 = state.buildBindings(attrs.size()); - for (auto & i : attrs) - attrs2.insert(i.first, i.second); - parent->value(state).mkAttrs(attrs2); + for (auto i = attrs.begin(); i != attrs.end(); ++i) + if (std::next(i) == attrs.end() || std::next(i)->first != i->first) + attrs2.insert(i->first, i->second); + parent->value(state).mkAttrs(attrs2.alreadySorted()); return std::move(parent); } @@ -72,13 +105,19 @@ class JSONSax : nlohmann::json_sax void key(string_t & name, EvalState & state) { forceNoNullByte(name); - attrs.insert_or_assign(state.symbols.create(name), &value(state)); + attrs.emplace_back(state.symbols.create(name), &value(state)); } }; class JSONListState : public JSONState { - ValueVector values; + using JSONState::JSONState; + + /** + * Note: deliberately not `ValueVector` (see + * `JSONObjectState::attrs`). + */ + boost::container::small_vector values; std::unique_ptr resolve(EvalState & state) override { @@ -91,24 +130,30 @@ class JSONSax : nlohmann::json_sax void add() override { - values.push_back(*v); + values.push_back(v); v = nullptr; } - public: - JSONListState(std::unique_ptr && p, std::size_t reserve) - : JSONState(std::move(p)) - { - values.reserve(reserve); - } }; EvalState & state; + + /** + * Keeps all values allocated during the parse alive. This is the + * only GC-visible container of the parser: the per-node containers + * in `JSONObjectState`/`JSONListState` use ordinary allocators, + * since GC-visible (uncollectable) allocations take the global GC + * allocation lock, and a big JSON document would otherwise do one + * or more of them per object/array node — a significant source of + * lock contention during parallel evaluation. + */ + ValueVector rootValues; + std::unique_ptr rs; public: JSONSax(EvalState & state, Value & v) : state(state) - , rs(new JSONState(&v)) {}; + , rs(new JSONState(&v, rootValues)) {}; bool null() override { @@ -192,7 +237,7 @@ class JSONSax : nlohmann::json_sax bool start_array(size_t len) override { - rs = std::make_unique(std::move(rs), len != std::numeric_limits::max() ? len : 128); + rs = std::make_unique(std::move(rs)); return true; } diff --git a/src/libexpr/make.nix b/src/libexpr/make.nix new file mode 100644 index 000000000000..e601e368bca7 --- /dev/null +++ b/src/libexpr/make.nix @@ -0,0 +1,103 @@ +# libnixexpr, transcribed from meson.build and its subdirectories. +{ + pkgs, + nixMake, + nix-util, + nix-store, + nix-fetchers, +}: + +let + # The parser and lexer are generated by bison and flex. The scanner is + # told (via `generated`) that their outputs inherit the #includes of + # parser.y and lexer.l, so it does not need to build them at evaluation + # time. + parserTab = pkgs.runCommand "parser-tab" { nativeBuildInputs = [ pkgs.bison ]; } '' + mkdir $out + bison -v -o $out/parser-tab.cc -d ${./parser.y} + ''; + + lexerTab = pkgs.runCommand "lexer-tab" { nativeBuildInputs = [ pkgs.flex ]; } '' + mkdir $out + flex -Cf --outfile $out/lexer-tab.cc --header-file=$out/lexer-tab.hh ${./lexer.l} + ''; +in + +nixMake.mkComponent { + name = "determinate-nix-expr"; + libName = "nixexpr"; + + deps = [ + nix-util + nix-store + nix-fetchers + ]; + + root = ./.; + + # `builtins.wasm` is only built with WebAssembly support. + excludeSources = pkgs.lib.optional (!nixMake.config.wasm) "primops/wasm.cc"; + + includeDirs = [ + "" + "include" + ]; + + files = nixMake.commonSupportFiles // { + "imported-drv-to-derivation.nix.gen.hh" = nixMake.mkStringHeader ./imported-drv-to-derivation.nix; + "fetchurl.nix.gen.hh" = nixMake.mkStringHeader ./fetchurl.nix; + "primops/derivation.nix.gen.hh" = nixMake.mkStringHeader ./primops/derivation.nix; + }; + + generated = { + "parser-tab.hh" = { + from = "parser.y"; + path = "${parserTab}/parser-tab.hh"; + }; + "parser-tab.cc" = { + from = "parser.y"; + path = "${parserTab}/parser-tab.cc"; + }; + "lexer-tab.hh" = { + from = "lexer.l"; + path = "${lexerTab}/lexer-tab.hh"; + }; + "lexer-tab.cc" = { + from = "lexer.l"; + path = "${lexerTab}/lexer-tab.cc"; + }; + }; + + configHeaders = { + "include/nix/expr/config.hh" = { + NIX_USE_BOEHMGC = nixMake.config.boehmgc; + }; + "expr-config-private.hh" = { + HAVE_PTHREAD_GETATTR_NP = 1; + HAVE_SYSCONF = 1; + HAVE_TOML11_4 = 1; + }; + }; + + # Build the parser without assertions and with a higher inlining + # threshold in non-debug builds: Bison/Flex are particularly sensitive + # to the compiler failing to inline. + unitCxxFlags = + let + parserFlags = pkgs.lib.optionals (!nixMake.config.debug) [ + "-DNDEBUG" + "--param=max-inline-insns-single=1000" + "--param=max-inline-insns-auto=1000" + "--param=inline-unit-growth=400" + ]; + in + { + "parser-tab.cc" = parserFlags; + "lexer-tab.cc" = parserFlags; + "lexer-helpers.cc" = parserFlags; + }; + + extraCxxFlags = nixMake.weakVtablesFlags; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libexpr/meson.build b/src/libexpr/meson.build index d44a0965d96b..ac89b000b379 100644 --- a/src/libexpr/meson.build +++ b/src/libexpr/meson.build @@ -53,6 +53,10 @@ deps_other += boost nlohmann_json = dependency('nlohmann_json', version : '>= 3.9') deps_public += nlohmann_json +# eval-cache uses `INSERT ... RETURNING` which requires >= 3.35 +sqlite = dependency('sqlite3', 'sqlite', version : '>= 3.35') +deps_private += sqlite + bdw_gc_required = get_option('gc').disable_if( 'address' in get_option('b_sanitize'), error_message : 'Building with Boehm GC and ASAN is not supported', @@ -62,7 +66,6 @@ bdw_gc = dependency('bdw-gc', required : bdw_gc_required) if bdw_gc.found() deps_public += bdw_gc foreach funcspec : [ - 'pthread_attr_get_np', 'pthread_getattr_np', ] define_name = 'HAVE_' + funcspec.underscorify().to_upper() @@ -77,6 +80,17 @@ endif # Used in public header. Affects ABI! configdata_pub.set('NIX_USE_BOEHMGC', bdw_gc.found().to_int()) +link_args = [] + +wasmtime_required = get_option('wasm').disable_if( + get_option('default_library') == 'static', + error_message : 'Building with wasmtime and static linking is not supported', +) + +if wasmtime_required.enabled() + link_args += '-lwasmtime' +endif + toml11 = dependency( 'toml11', version : '>=3.7.0', @@ -91,10 +105,6 @@ configdata_priv.set( deps_other += toml11 -cpuid = dependency('libcpuid', 'cpuid', required : false) -configdata_priv.set('HAVE_LIBCPUID', cpuid.found().to_int()) -deps_private += cpuid - config_priv_h = configure_file( configuration : configdata_priv, output : 'expr-config-private.hh', @@ -169,11 +179,15 @@ sources = files( 'get-drvs.cc', 'json-to-value.cc', 'nixexpr.cc', + 'parallel-eval.cc', 'paths.cc', 'primops.cc', 'print-ambiguous.cc', 'print.cc', + 'provenance.cc', + 'root-value.cc', 'search-path.cc', + 'symbol-table.cc', 'value-to-json.cc', 'value-to-xml.cc', 'value.cc', @@ -241,12 +255,11 @@ this_library = library( parser_tab[1], lexer_tab[1], generated_headers, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, - link_args : linker_export_flags, + link_args : linker_export_flags + link_args, link_whole : [ parser_library ], - prelink : true, # For C++ static initializers + prelink : prelink, # For C++ static initializers install : true, cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [], ) diff --git a/src/libexpr/meson.options b/src/libexpr/meson.options index 847bb211d302..2defbf77aaa0 100644 --- a/src/libexpr/meson.options +++ b/src/libexpr/meson.options @@ -3,3 +3,9 @@ option( type : 'feature', description : 'enable garbage collection in the Nix expression evaluator (requires Boehm GC)', ) + +option( + 'wasm', + type : 'feature', + description : 'enable wasmtime integration into the Nix expression evaluator', +) diff --git a/src/libexpr/nixexpr.cc b/src/libexpr/nixexpr.cc index 4a2f71a11b8f..b52370816f5a 100644 --- a/src/libexpr/nixexpr.cc +++ b/src/libexpr/nixexpr.cc @@ -13,8 +13,6 @@ namespace nix { Counter Expr::nrExprs; -ExprBlackHole eBlackHole; - // FIXME: remove, because *symbols* are abstract and do not have a single // textual representation; see printIdentifier() std::ostream & operator<<(std::ostream & str, const SymbolStr & symbol) @@ -626,15 +624,6 @@ void ExprLambda::setDocComment(DocComment docComment) // belongs in the same conditional. body->setDocComment(docComment); } -}; - -/* Symbol table. */ - -size_t SymbolTable::totalSize() const -{ - size_t n = 0; - dump([&](SymbolStr s) { n += s.size(); }); - return n; } std::string DocComment::getInnerText(const PosTable & positions) const diff --git a/src/libexpr/package.nix b/src/libexpr/package.nix index e9d12035cde1..b5bccc90791f 100644 --- a/src/libexpr/package.nix +++ b/src/libexpr/package.nix @@ -13,8 +13,9 @@ boost, boehmgc, nlohmann_json, + sqlite, toml11, - libcpuid, + wasmtime, # Configuration Options @@ -30,6 +31,11 @@ # Temporarily disabled on Windows because the `GC_throw_bad_alloc` # symbol is missing during linking. enableGC ? !stdenv.hostPlatform.isWindows, + + # Whether to use wasmtime for wasm integration in the Nix language evaluator + # + # Temporarily disabled when static linking due to Rust not compiling + enableWasm ? !stdenv.hostPlatform.isStatic, }: let @@ -37,7 +43,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-expr"; + pname = "determinate-nix-expr"; inherit version; workDir = ./.; @@ -64,9 +70,10 @@ mkMesonLibrary (finalAttrs: { ]; buildInputs = [ + sqlite toml11 ] - ++ lib.optional stdenv.hostPlatform.isx86_64 libcpuid; + ++ lib.optional enableWasm wasmtime; propagatedBuildInputs = [ nix-util @@ -79,8 +86,20 @@ mkMesonLibrary (finalAttrs: { mesonFlags = [ (lib.mesonEnable "gc" enableGC) + (lib.mesonEnable "wasm" enableWasm) ]; + # Fixes a problem with the "nix-expr-libcxxStdenv-static" package output. + # For some reason that is not clear, it is wanting to use libgcc_eh which is not available. + # Force this to be built with compiler-rt over libgcc_eh works. + # Issue: https://github.com/NixOS/nixpkgs/issues/177129 + NIX_CFLAGS_COMPILE = lib.optionalString ( + stdenv.cc.isClang + && stdenv.hostPlatform.isStatic + && stdenv.cc.libcxx != null + && stdenv.cc.libcxx.isLLVM + ) "-rtlib=compiler-rt"; + meta = { platforms = lib.platforms.unix ++ lib.platforms.windows; }; diff --git a/src/libexpr/parallel-eval.cc b/src/libexpr/parallel-eval.cc new file mode 100644 index 000000000000..95fc0fd4a3d1 --- /dev/null +++ b/src/libexpr/parallel-eval.cc @@ -0,0 +1,945 @@ +#include "nix/expr/eval.hh" +#include "nix/expr/parallel-eval.hh" +#include "nix/store/globals.hh" +#include "nix/expr/primops.hh" + +#include +#include + +#include +#include + +#if NIX_USE_BOEHMGC +# include +#endif + +namespace nix { + +struct WaiterDomain; + +[[gnu::tls_model("initial-exec")]] thread_local bool Executor::amWorkerThread{false}; + +static std::atomic nextEvalThreadId{1}; +[[gnu::tls_model("initial-exec")]] thread_local uint32_t myEvalThreadId(nextEvalThreadId++); + +/** + * The fiber currently executing on this thread, or null if we're not + * inside a fiber (e.g. on the main thread or in a worker's scheduler + * loop). + */ +[[gnu::tls_model("initial-exec")]] static thread_local Executor::Fiber * currentFiber{nullptr}; + +/** + * The worker whose scheduler loop is running on this thread, or null + * on non-worker threads. Only used to assert that fibers are resumed + * by their owner. + */ +[[gnu::tls_model("initial-exec")]] static thread_local Executor::Worker * currentWorker{nullptr}; + +struct Executor::Fiber +{ + Executor & executor; + + /** + * The worker that created this fiber, which is the only thread + * that ever runs it (see `Executor::Worker`). + */ + Worker & owner; + + /** + * The value of `myEvalThreadId` while this fiber is running. Each + * fiber needs its own id (rather than a per-thread id) since the + * self-wait check in `waitOnThunk()` would otherwise produce false + * "infinite recursion" errors when two fibers running on the same + * thread touch the same thunk. + */ + const uint32_t evalThreadId; + + /** + * This fiber's Nix call stack depth while it's not running. + * `runFiber()` swaps it with the thread-local counter (see + * `CallDepth`) on every switch-in/out, since several fibers, each + * with their own call chain, are interleaved on the same thread. + */ + size_t callDepth = 0; + + /** + * This fiber's evaluation context while it's not running. + * `runFiber()` swaps it with the thread-local + * `EvalState::evalContext` on every switch-in/out (a mere pointer + * exchange), while this slot holds the thread's own context in the + * meantime. + */ + EvalState::EvalContext evalContext; + + std::promise promise; + + work_t work; + + /** + * The number of active `FiberNoSuspend` guards on this fiber. + * While non-zero, `waitOnThunk()` blocks the thread instead of + * suspending the fiber. + */ + unsigned int noSuspendDepth = 0; + + /** + * The fiber's continuation. Valid while the fiber is queued or + * suspended; invalid while it's running or after it has finished. + */ + boost::context::fiber ctx; + + /** + * The scheduler's continuation. Valid while the fiber is running. + */ + boost::context::fiber schedCtx; + + /** + * Side channel for the suspension handshake, filled in by + * `suspendFiber()` immediately before switching back to the + * scheduler, and consumed by `runFiber()`. + */ + detail::ValueBase * waitingOn = nullptr; + WaiterDomain * suspendDomain = nullptr; + +#if NIX_USE_BOEHMGC + /** + * The registered GC descriptor of the stack that this fiber's + * continuation lives on: initially its own stack; after a + * suspension, the stack it suspended on (see `suspendFiber()`). + */ + struct GC_stack * gcStack = nullptr; +#endif + + Fiber(Executor & executor, Worker & owner, Item && item) + : executor(executor) + , owner(owner) + , evalThreadId(nextEvalThreadId++) + , promise(std::move(item.promise)) + , work(std::move(item.work)) + { + } + + ~Fiber() + { + /* A fiber must never be destroyed while suspended, since that + would forcibly unwind its stack. Suspended fibers are always + resumed (with `quit` or an interrupt flag set) so that they + exit normally. */ + assert(!ctx); + } +}; + +struct Executor::StackPool +{ + struct State + { + std::vector stacks; +#if NIX_USE_BOEHMGC + /** + * The registered GC descriptor of each stack ever allocated, + * keyed by the stack's hi end (`stack_context::sp`). Pooled + * stacks keep their registration; their descriptors have a + * null `saved_sp`, so the GC never scans them. + */ + std::unordered_map gcInfo; +#endif + }; + + Sync state_; + + ~StackPool() + { + auto state(state_.lock()); +#if NIX_USE_BOEHMGC + for (auto & [_, gs] : state->gcInfo) { + GC_unregister_stack(gs); + delete gs; + } +#endif + for (auto & sctx : state->stacks) + boost::context::protected_fixedsize_stack(evalStackSize).deallocate(sctx); + } +}; + +#if NIX_USE_BOEHMGC +/** + * Side channel from `PooledStackAllocator::allocate()` (called from + * inside the `boost::context::fiber` constructor) to `makeFiber()`: + * the GC descriptor of the most recently allocated stack. + */ +[[gnu::tls_model("initial-exec")]] static thread_local struct GC_stack * lastFiberGCStack{nullptr}; +#endif + +/** + * A Boost.Context stack allocator that reuses stacks from the + * executor's pool. Reused stacks also come with their previously + * faulted-in pages, avoiding both the mmap/munmap system calls and + * the page faults of a fresh stack for every work item. + */ +struct PooledStackAllocator +{ + Executor & executor; + + boost::context::stack_context allocate() + { + { + auto state(executor.stackPool->state_.lock()); + if (!state->stacks.empty()) { + auto sctx = state->stacks.back(); + state->stacks.pop_back(); +#if NIX_USE_BOEHMGC + lastFiberGCStack = state->gcInfo.at(sctx.sp); +#endif + return sctx; + } + } + executor.nrFiberStacksAllocated++; + auto sctx = boost::context::protected_fixedsize_stack(evalStackSize).allocate(); +#if NIX_USE_BOEHMGC + auto gs = new GC_stack{}; + gs->base = sctx.sp; + gs->limit = (char *) sctx.sp - sctx.size; + GC_register_stack(gs); + executor.stackPool->state_.lock()->gcInfo.emplace(sctx.sp, gs); + lastFiberGCStack = gs; +#endif + return sctx; + } + + void deallocate(boost::context::stack_context & sctx) + { + executor.stackPool->state_.lock()->stacks.push_back(sctx); + } +}; + +// cache line alignment to prevent false sharing +struct alignas(64) WaiterDomain +{ + /* Note: not using `Sync` because the suspension handshake requires + locking the mutex on the fiber's stack and unlocking it from the + scheduler (see `Executor::runFiber()`), which `Sync::Lock` can't + express. */ + std::mutex mutex; + + /** + * Wakes up non-fiber threads (e.g. the main thread) blocked on a + * thunk in this domain. + */ + std::condition_variable cv; + + /** + * Fibers suspended waiting for a specific value to be finished. + * Unlike the condition variable, this is keyed on the exact value, + * so fiber wakeups are never spurious. + */ + std::unordered_map> waiters; +}; + +static std::array waiterDomains; + +/** + * Move all suspended fibers out of the wait lists and back onto their + * owner's ready queue, and wake up all non-fiber waiters. Called on + * interrupt and shutdown so that waiting fibers/threads can observe + * the interrupt/`quit` flag and unwind. + */ +static void flushWaiters() +{ + std::vector woken; + for (auto & domain : waiterDomains) { + std::unique_lock lk(domain.mutex); + for (auto & [_, fibers] : domain.waiters) + for (auto & fiber : fibers) + woken.push_back(std::move(fiber)); + domain.waiters.clear(); + domain.cv.notify_all(); + } + for (auto & fiber : woken) { + auto & executor = fiber->executor; + executor.enqueueFiber(std::move(fiber)); + } +} + +unsigned int Executor::getEvalCores(const EvalSettings & evalSettings) +{ + /* Note: the default number of cores is currently limited to 32 + due to scalability bottlenecks. */ + return evalSettings.evalProfilerMode != EvalProfilerMode::disabled ? 1 + : evalSettings.evalCores == 0UL ? std::min(32U, Settings::getDefaultCores()) + : evalSettings.evalCores; +} + +unsigned int Executor::getMaxFibersPerWorker(const EvalSettings & evalSettings, unsigned int evalCores) +{ + /* By default, allow three suspended fibers per thread on top of + the running one. This bounds the number of stacks (and thus + memory and page faults) without leaving threads idle in typical + workloads: e.g. `nix search nixpkgs` on 24 threads peaks at + 120-220 live fibers when unlimited, and is not measurably slower + with 96 (4 per thread), while a limit of 48 costs about 5% + elapsed time. */ + return evalSettings.evalMaxFibers == 0U ? 4 : std::max(1U, evalSettings.evalMaxFibers / evalCores); +} + +Executor::Executor(const EvalSettings & evalSettings) + : stackPool(std::make_unique()) + , evalCores(getEvalCores(evalSettings)) + , maxFibersPerWorker(getMaxFibersPerWorker(evalSettings, evalCores)) + , enabled(evalCores > 1) + , interruptCallback(createInterruptCallback([&]() { + /* Wake up all waiting fibers and threads so they can observe + the interrupt and unwind. Note: `_isInterrupted` has already + been set at this point. */ + flushWaiters(); + wakeupAllWorkers(); + })) +{ + debug("executor using %d threads and at most %d fibers per thread", evalCores, maxFibersPerWorker); + // FIXME: create worker threads on demand? + for (size_t n = 0; n < evalCores; ++n) + try { + createWorker(); + } catch (std::system_error & e) { + if (n == 0) + throw Error("could not create any evaluator worker threads: %s", e.what()); + warn("could only create %d evaluator worker threads: %s", n, e.what()); + break; + } +} + +Executor::~Executor() +{ + { + auto state(state_.lock()); + quit = true; + debug("executor shutting down with %d items left", state->queue.size()); + } + + /* Hand suspended fibers back to their owners and wake up idle + workers so they can wind down. Note: the shutdown guard in + `waitOnThunk()` ensures that no fiber registers in a wait list + after this flush. */ + flushWaiters(); + wakeupAllWorkers(); + + for (auto & worker : workers) + worker->thread.join(); + + /* A worker doesn't exit while it owns fibers, so no fiber stack + can outlive this destructor. */ + for (auto & worker : workers) { + assert(worker->readyFibers.empty()); + assert(worker->nrLiveFibers == 0); + } + + /* Fail any work items that were queued after the workers exited. */ + failQueuedItems(); +} + +/** + * Wake up `worker` if it's asleep. Must be called with the state lock + * held. Clears `sleeping` right away, so that producers running before + * the worker has reacquired the lock don't notify it again (and, in + * `spawn()`, don't spend their wake budget on a worker that is already + * being woken). The worker re-checks all its conditions after waking + * up, so one pending notification is enough. + */ +static bool wakeWorker(Executor::Worker & worker) +{ + if (!worker.sleeping) + return false; + worker.sleeping = false; + worker.wakeup.notify_one(); + return true; +} + +void Executor::wakeupAllWorkers() +{ + auto state(state_.lock()); + for (auto & worker : workers) + wakeWorker(*worker); +} + +void Executor::createWorker() +{ + auto worker = std::make_unique(); + auto w = worker.get(); + /* Note: worker threads can have a small (default-sized) stack, + since they only run the scheduler loop; the actual evaluation + happens on fibers, which have their own stacks. */ + worker->thread = std::thread([this, w]() { +#if NIX_USE_BOEHMGC + /* Register the worker thread with the garbage collector. This + is not for the sake of the worker stack (which holds no GC + roots), but because fibers running on this thread allocate + from it: without registration, Boehm has no thread-local + allocation freelists for this thread and every allocation + takes the global allocation lock, which is several times + slower. */ + GC_stack_base sb; + GC_get_stack_base(&sb); + GC_register_my_thread(&sb); +#endif + this->worker(*w); +#if NIX_USE_BOEHMGC + GC_unregister_my_thread(); +#endif + }); + /* Only register the worker once its thread exists, so that a + failure to create the thread doesn't leave a thread-less worker + behind. Note: `workers` is read under the state lock by + `spawn()` and `wakeupAllWorkers()`, which the interrupt + callback can invoke concurrently. */ + auto state(state_.lock()); + workers.push_back(std::move(worker)); +} + +Executor::FiberPtr Executor::makeFiber(Worker & owner, Item && item) +{ + auto fiber = std::make_unique(*this, owner, std::move(item)); + auto fib = fiber.get(); + try { + fiber->ctx = boost::context::fiber( + std::allocator_arg, + PooledStackAllocator{*this}, + [fib](boost::context::fiber && sched) -> boost::context::fiber { + fib->schedCtx = std::move(sched); + try { + fib->work(); + fib->promise.set_value(); + } catch (const Interrupted &) { + fib->executor.quit = true; + fib->promise.set_exception(std::current_exception()); + } catch (...) { + fib->promise.set_exception(std::current_exception()); + } + // Terminate the fiber, freeing its stack, and return + // to the scheduler. + return std::move(fib->schedCtx); + }); + } catch (...) { + // Stack allocation failure. Report it to whoever is waiting + // on this work item. + fiber->promise.set_exception(std::current_exception()); + return nullptr; + } +#if NIX_USE_BOEHMGC + fiber->gcStack = lastFiberGCStack; + assert(fiber->gcStack); +#endif + nrFibersSpawned++; + return fiber; +} + +bool Executor::runFiber(FiberPtr fiber) +{ + auto fib = fiber.get(); + assert(fib->ctx); + assert(!currentFiber); + /* Fibers must only ever run on their owner thread, see + `Executor::Worker`. */ + assert(currentWorker == &fib->owner); + + auto savedThreadId = myEvalThreadId; + auto savedCallDepth = CallDepth::callDepth; + currentFiber = fib; + myEvalThreadId = fib->evalThreadId; + CallDepth::callDepth = fib->callDepth; + std::swap(EvalState::evalContext, fib->evalContext); + +#if NIX_USE_BOEHMGC + /* Make this thread's stack scannable by the GC while the fiber + runs, and make the stack the fiber continues on the current + one. The fiber clears that stack's `saved_sp` after it has been + resumed (see `suspendFiber()`). */ + auto prevStack = GC_current_stack; + if (prevStack) + gcSaveStackPointer(prevStack); + GC_current_stack = fib->gcStack; +#endif + + fib->ctx = std::move(fib->ctx).resume(); + +#if NIX_USE_BOEHMGC + GC_current_stack = prevStack; + if (prevStack) + prevStack->saved_sp = nullptr; +#endif + + currentFiber = nullptr; + myEvalThreadId = savedThreadId; + fib->callDepth = CallDepth::callDepth; + CallDepth::callDepth = savedCallDepth; + std::swap(EvalState::evalContext, fib->evalContext); + + if (fib->ctx) { + /* The fiber suspended itself in `waitOnThunk()`. We are still + holding the waiter domain mutex, which the fiber locked on + this thread before switching back to us (so `std::mutex` + thread ownership is respected). Register the fiber in the + wait list, then release the mutex. Only after the unlock can + `notifyWaiters()` extract and resume the fiber — at which + point `fib->ctx` is fully formed. */ + auto domain = fib->suspendDomain; + assert(domain && fib->waitingOn); + auto n = ++currentSuspendedFibers; + if (n > maxSuspendedFibers) + maxSuspendedFibers = n; + domain->waiters[fib->waitingOn].push_back(std::move(fiber)); + domain->mutex.unlock(); + /* Ownership of `fib` has passed to the wait list; it may be + moved back onto our ready queue by another thread from this + point on, so don't touch it anymore. */ + return false; + } else { + /* The fiber has finished; its promise has been fulfilled + inside the fiber. Destroying `fiber` frees the record (the + stack was already freed on fiber termination). */ + return true; + } +} + +void Executor::enqueueFiber(FiberPtr fiber) +{ + nrFiberWakeups++; + currentSuspendedFibers--; + auto & owner = fiber->owner; + auto state(state_.lock()); + owner.readyFibers.push_back(std::move(fiber)); + /* Note: `sleeping` is maintained under the state lock, so the + owner either sees our push in its queue check, or is already + blocked on `wakeup` and gets notified — no lost wakeups. */ + wakeWorker(owner); +} + +/** + * Fail work items that haven't started with an `Interrupted` + * exception, so we get a nicer error than "std::future_error: Broken + * promise". Note: a fresh exception per item, not a shared one. + */ +static void failItems(std::multimap && items) +{ + for (auto & [_, item] : items) + item.promise.set_exception(std::make_exception_ptr(Interrupted("interrupted by the user"))); +} + +void Executor::failQueuedItems() +{ + failItems(std::exchange(state_.lock()->queue, {})); +} + +void Executor::worker(Worker & self) +{ + ReceiveInterrupts receiveInterrupts; + + unix::interruptCheck = [&]() { return (bool) quit; }; + + amWorkerThread = true; + currentWorker = &self; + + /* Whether the fiber we ran in the previous iteration finished (as + opposed to suspending itself), if any. The fiber accounting for + it is done under the state lock at the start of the next + iteration. */ + std::optional finished; + + while (true) { + FiberPtr fiber; + std::optional item; + + while (true) { + auto state(state_.lock()); + + if (finished) { + if (*finished) { + self.nrLiveFibers--; + state->nrLiveFibers--; + } + finished.reset(); + } + + /* Resume ready fibers before anything else, so that + existing work is drained first. This includes shutdown: + a suspended fiber can only be resumed by its owner + (i.e. us), and it has to be resumed so that it can + observe `quit` and unwind its stack. */ + if (!self.readyFibers.empty()) { + fiber = std::move(self.readyFibers.front()); + self.readyFibers.pop_front(); + break; + } + + if (quit) { + /* Fail queued work items that haven't started, so that + threads blocked on their futures are unblocked + promptly. */ + failItems(std::exchange(state->queue, {})); + /* Don't exit while we still own fibers: they're + suspended in a wait list and will be handed back to + us by `flushWaiters()`. */ + if (self.nrLiveFibers == 0) + return; + } else if (!state->queue.empty() && canStartFiber(self)) { + item = std::move(state->queue.begin()->second); + state->queue.erase(state->queue.begin()); + self.nrLiveFibers++; + state->nrLiveFibers++; + if (state->nrLiveFibers > maxLiveFibers) + maxLiveFibers = state->nrLiveFibers; + break; + } + + /* Nothing to do (or our fiber limit has been reached); + sleep until a producer wakes us up. Note: `sleeping` is + maintained under the state lock, so a producer either + sees it (and wakes us), or its insertion happened before + our checks above — no lost wakeups. */ + self.sleeping = true; + state.wait(self.wakeup); + self.sleeping = false; + } + + if (item) { + fiber = makeFiber(self, std::move(*item)); + if (!fiber) { + /* Stack allocation failure; the item's promise has + received the exception. Release the fiber slot. */ + finished = true; + continue; + } + } + + finished = runFiber(std::move(fiber)); + } +} + +std::vector> Executor::spawn(WorkItems && items) +{ + if (items.empty()) + return {}; + + std::vector> futures; + + auto state(state_.lock()); + + if (quit) { + /* The workers may have exited already, so nobody would ever + pick up these items. */ + for (auto & _ : items) { + std::promise promise; + futures.push_back(promise.get_future()); + promise.set_exception(std::make_exception_ptr(Interrupted("interrupted by the user"))); + } + return futures; + } + + for (auto & item : items) { + std::promise promise; + futures.push_back(promise.get_future()); + static thread_local uint32_t local = 0; + auto key = (uint64_t(item.second) << 48) | local++; + state->queue.emplace(key, Item{.promise = std::move(promise), .work = std::move(item.first)}); + } + + /* Wake up one worker per item, but only workers that are asleep + and can actually start a fresh item (a worker at its fiber + limit would just go back to sleep). In the steady state (all + workers busy), this does no futex calls at all. */ + auto toWake = items.size(); + for (auto & worker : workers) { + if (toWake == 0) + break; + if (canStartFiber(*worker) && wakeWorker(*worker)) + toWake--; + } + + return futures; +} + +FiberNoSuspend::FiberNoSuspend() + : fiber(currentFiber) +{ + if (fiber) + fiber->noSuspendDepth++; +} + +FiberNoSuspend::~FiberNoSuspend() +{ + /* Note: `fiber` is still the current fiber, since it cannot have + been suspended (and thus migrated) while this guard was alive. */ + if (fiber) { + assert(fiber == currentFiber); + fiber->noSuspendDepth--; + } +} + +FutureVector::~FutureVector() +{ + try { + finishAll(); + } catch (...) { + ignoreExceptionInDestructor(); + } +} + +void FutureVector::spawn(Executor::WorkItems && work) +{ + auto futures = executor.spawn(std::move(work)); + auto state(state_.lock()); + for (auto & future : futures) + state->futures.push_back(std::move(future)); +} + +void FutureVector::finishAll() +{ + std::exception_ptr ex; + while (true) { + std::vector> futures; + { + auto state(state_.lock()); + std::swap(futures, state->futures); + } + debug("got %d futures", futures.size()); + if (futures.empty()) + break; + for (auto & future : futures) + try { + future.get(); + } catch (...) { + if (ex) { + if (!getInterrupted()) + logExceptionExceptInterrupt(); + } else + ex = std::current_exception(); + } + } + if (ex) + std::rethrow_exception(ex); +} + +static WaiterDomain & getWaiterDomain(detail::ValueBase & v) +{ + auto domain = (((size_t) &v) >> 5) % waiterDomains.size(); + return waiterDomains[domain]; +} + +/** + * Suspend the current fiber until `v` is finished. Must be called with + * `lk` holding the domain's mutex and `v` in the "awaited" state. On + * return, the fiber has been resumed (by its owner thread, after other + * fibers may have run on it) and the lock has been released. + */ +[[gnu::noinline]] static void +suspendFiber(WaiterDomain & domain, std::unique_lock & lk, detail::ValueBase & v) +{ + auto fib = currentFiber; + assert(fib); + /* A fiber must not suspend while an exception is being handled or + unwound: the C++ exception state lives in thread-local storage, + so it would get mixed up with that of the other fibers that run + on this thread in the meantime. */ + assert(!std::current_exception() && !std::uncaught_exceptions()); + fib->waitingOn = &v; + fib->suspendDomain = &domain; + /* Hand ownership of the domain mutex over to the scheduler, which + will unlock it (via `domain.mutex`) after registering us in the + wait list. `lk` lives on this stack, so it must not be + considered owning anymore once we've switched away. */ + lk.release(); +#if NIX_USE_BOEHMGC + /* Publish the used portion of the stack we're running on, so that + the garbage collector will scan it while we're suspended (see + `gcStackSwitchSlack` for why the approximation is lowered). + This is the fiber's own stack unless we're suspending from + inside a coroutine body, in which case it's the coroutine's + stack (the fiber's own stack then already has a `saved_sp`, + recorded when it switched onto the coroutine). Either way, + `runFiber()` makes it the current stack again when we're + resumed. Note: from this point until we clear `saved_sp` after + being resumed, the GC may see the stack both as some thread's + active stack and as a suspended one; it scans it only once + (from the lower stack pointer). */ + auto stk = GC_current_stack; + assert(stk); + gcSaveStackPointer(stk); + fib->gcStack = stk; +#endif + /* Switch back to the scheduler (`Executor::runFiber()`), which + will register us in the domain's wait list and then release the + lock. We can't do that here: the fiber's continuation only + materializes on the scheduler side, and publishing it before the + switch-out completes would allow another thread to resume a + half-suspended fiber. */ + fib->schedCtx = std::move(fib->schedCtx).resume(); +#if NIX_USE_BOEHMGC + /* We're running again, so our stack is scanned as the thread's + active stack from here on. */ + stk->saved_sp = nullptr; +#endif + /* We've been resumed because the value was finished (or because + we're shutting down); the scheduler released the lock long + ago. */ + assert(!lk.owns_lock()); +} + +template<> +ValueStorage::PackedPointer +ValueStorage::waitOnThunk(EvalState & state, PackedPointer expectedP0) +{ + state.nrThunksAwaited++; + + auto & domain = getWaiterDomain(*this); + std::unique_lock lk(domain.mutex); + + auto threadId = expectedP0 >> discriminatorBits; + + if (static_cast(expectedP0 & discriminatorMask) == pdAwaited) { + /* Make sure that the value is still awaited, now that we're + holding the domain lock. */ + auto p0_ = p0.load(std::memory_order_acquire); + auto pd = static_cast(p0_ & discriminatorMask); + + /* If the value has been finalized in the meantime (i.e. is no + longer pending), we're done. */ + if (pd != pdAwaited) { + assert(pd != pdThunk && pd != pdPending); + return p0_; + } + } else { + /* Mark this value as being waited on. */ + PackedPointer p0_ = expectedP0; + if (!p0.compare_exchange_strong( + p0_, + pdAwaited | (threadId << discriminatorBits), + std::memory_order_acquire, + std::memory_order_acquire)) { + /* If the value has been finalized in the meantime (i.e. is + no longer pending), we're done. */ + auto pd = static_cast(p0_ & discriminatorMask); + if (pd != pdAwaited) { + assert(pd != pdThunk && pd != pdPending); + return p0_; + } + /* The value was already in the "waited on" state, so we're + not the only thread waiting on it. */ + } + } + + /* Wait for another thread to finish this value. */ + if (threadId == myEvalThreadId) + state.error("infinite recursion encountered") + .atPos(((Value &) *this).determinePos(noPos)) + .debugThrow(); + + state.nrThunksAwaitedSlow++; + state.currentlyWaiting++; + state.maxWaiting = std::max(state.maxWaiting, state.currentlyWaiting); + + auto now1 = std::chrono::steady_clock::now(); + + /* Note: a fiber inside a `FiberNoSuspend` region uses the blocking + wait below, like a non-fiber thread. */ + if (auto fib = currentFiber; fib && fib->noSuspendDepth == 0) { + /* Shutdown guard: `quit`/`_isInterrupted` are always set + *before* the wait lists are flushed, and flushing takes the + domain lock that we're currently holding. So either we see + the flag here and throw, or our registration completes + before the flush and we get woken by it — we can't be + stranded in the wait list. */ + if (fib->executor.quit) + throw Interrupted("interrupted by the user"); + checkInterrupt(); + + /* We're running on a fiber, so suspend it and let this thread + run other work. `notifyWaiters()` will re-enqueue the fiber + when the value is finished. */ + suspendFiber(domain, lk, *this); + + /* Note: `currentFiber` has been restored to `fib` by + `runFiber()`, but the captured `fib` is cheaper. */ + if (fib->executor.quit) + throw Interrupted("interrupted by the user"); + checkInterrupt(); + + auto p0_ = p0.load(std::memory_order_acquire); + auto pd = static_cast(p0_ & discriminatorMask); + /* Unlike the condition variable path below, fiber wakeups + cannot be spurious: the wait list is keyed on the exact + value and we're only woken after it has been finished (or on + shutdown/interrupt, handled above). */ + assert(pd != pdThunk && pd != pdPending && pd != pdAwaited); + auto now2 = std::chrono::steady_clock::now(); + state.microsecondsWaiting += std::chrono::duration_cast(now2 - now1).count(); + state.currentlyWaiting--; + return p0_; + } + + while (true) { + domain.cv.wait(lk); + auto p0_ = p0.load(std::memory_order_acquire); + auto pd = static_cast(p0_ & discriminatorMask); + if (pd != pdAwaited) { + assert(pd != pdThunk && pd != pdPending); + auto now2 = std::chrono::steady_clock::now(); + state.microsecondsWaiting += std::chrono::duration_cast(now2 - now1).count(); + state.currentlyWaiting--; + return p0_; + } + state.nrSpuriousWakeups++; + checkInterrupt(); + } +} + +template<> +void ValueStorage::notifyWaiters() +{ + auto & domain = getWaiterDomain(*this); + + /* Extract the fibers waiting on this value, then re-enqueue them + after releasing the domain lock (to keep a trivial lock order + between domain mutexes and the executor's state lock). */ + std::vector woken; + { + std::unique_lock lk(domain.mutex); + if (auto nh = domain.waiters.extract(this)) + woken = std::move(nh.mapped()); + /* Wake up any non-fiber waiters (e.g. the main thread). */ + domain.cv.notify_all(); + } + + for (auto & fiber : woken) { + auto & executor = fiber->executor; + executor.enqueueFiber(std::move(fiber)); + } +} + +static void prim_parallel(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.parallel"); + + if (state.executor->enabled) { + Executor::WorkItems work; + for (auto value : args[0]->listView()) + if (!value->isFinished()) + state.addWork(work, 0, [value(RootValue(value)), &state, pos]() { state.forceValue(**value, pos); }); + state.executor->spawn(std::move(work)); + } + + state.forceValue(*args[1], pos); + v = *args[1]; +} + +// FIXME: gate this behind an experimental feature. +static RegisterPrimOp r_parallel({ + .name = "__parallel", + .args = {"xs", "x"}, + .arity = 2, + .doc = R"( + Start evaluation of the values `xs` in the background and return `x`. + )", + .impl = prim_parallel, + .experimentalFeature = Xp::ParallelEval, +}); + +} // namespace nix diff --git a/src/libexpr/paths.cc b/src/libexpr/paths.cc index 4ceab2b4dbfe..6d644fe961e8 100644 --- a/src/libexpr/paths.cc +++ b/src/libexpr/paths.cc @@ -22,69 +22,97 @@ SourcePath EvalState::storePath(const StorePath & path) return {rootFS, CanonPath{store->printStorePath(path)}}; } -void EvalState::ensureLazyPathCopied(const StorePath & path) +StorePath EvalState::devirtualize(const StorePath & path, StringMap * rewrites) { - if (settings.isReadOnly()) - return; - - auto mount = storeFS->getMount(CanonPath(store->printStorePath(path))); - if (!mount) - return; - - /* TODO: We could memoise this in-memory if necessary. */ - auto storePath = fetchToStore( - fetchSettings, - *store, - SourcePath{ref(mount)}, - /* Force a copy. mountInput does a dryRun to just calculate the storePath and narHash. */ - FetchMode::Copy, - path.name()); - - /* This can happen if the source gets modified by another process while we are evaluaing - from it. Alternatively, the caching might be unsound and fetcher cache is poisoned somehow. - See https://github.com/NixOS/nix/issues/14317. */ - if (storePath != path) { - throw Error( - (unsigned int) 102, - "store path ('%1%') was hashed to avoid a full copy at first, but upon reading it again, the contents have changed ('%2%'), so we can not proceed. Make sure files do not change during evaluation", - store->printStorePath(path), - store->printStorePath(storePath)); - } + if (auto mount = storeFS->getMount(CanonPath(store->printStorePath(path)))) { + auto storePath = fetchToStore( + fetchSettings, + *store, + SourcePath{ref(mount)}, + settings.isReadOnly() ? FetchMode::DryRun : FetchMode::Copy, + path.name()); + assert(storePath.name() == path.name()); + if (rewrites) + rewrites->emplace(path.hashPart(), storePath.hashPart()); + return storePath; + } else + return path; +} + +SingleDerivedPath EvalState::devirtualize(const SingleDerivedPath & path, StringMap * rewrites) +{ + if (auto o = std::get_if(&path.raw())) + return SingleDerivedPath::Opaque{devirtualize(o->path, rewrites)}; + else + return path; +} + +std::string EvalState::devirtualize(std::string_view s, const NixStringContext & context) +{ + StringMap rewrites; + + for (auto & c : context) + if (auto o = std::get_if(&c.raw)) + devirtualize(o->path, &rewrites); + + return rewriteStrings(std::string(s), rewrites); } -void EvalState::ensureLazyPathsCopied(const NixStringContext & context) +std::string EvalState::computeBaseName(const SourcePath & path, PosIdx pos) { - for (const auto & c : context) - if (auto * o = std::get_if(&c.raw)) - /* TODO: This could be done in parallel. */ - ensureLazyPathCopied(o->path); + if (path.accessor == rootFS) { + if (auto storePath = store->maybeParseStorePath(path.path.abs())) { + debug( + "Copying '%s' to the store again.\n" + "You can make Nix evaluate faster and copy fewer files by replacing `./.` with the `self` flake input, " + "or `builtins.path { path = ./.; name = \"source\"; }`.\n", + path); + return std::string( + fetchToStore(fetchSettings, *store, path, FetchMode::DryRun, storePath->name()).to_string()); + } + } + return std::string(path.baseName()); } -StorePath -EvalState::mountInput(fetchers::Input & input, const fetchers::Input & originalInput, ref accessor) +StorePath EvalState::mountInput( + fetchers::Input & input, + const fetchers::Input & originalInput, + ref accessor, + bool requireLockable, + bool forceNarHash) { - /* To mount the input, dryRun is sufficient. We still compute the narHash (to check for mismatches) and the store - path to figure out where to mount it. TODO: This could be relaxed in the future by making outPath and narHash - lazier. Good code that doesn't do `toString ./.` or otherwise inspects the outPath string and only uses it for - doing relative imports does not even require computing the store path. That is a big invasive change though and - would require having a special "LazyStorePathString" thunk. narHash also doesn't need to be computed eagerly in - case it's not actually specified (like during local development with a dirty tree) - in that case narHash could - also become a lazy app/thunk that shares the state with the storePath delayed computation. */ - auto [storePath, narHash] = fetchToStore2(fetchSettings, *store, accessor, FetchMode::DryRun, input.getName()); + auto storePath = settings.lazyTrees + ? StorePath::random(input.getName()) + : fetchToStore(fetchSettings, *store, accessor, FetchMode::Copy, input.getName()); allowPath(storePath); // FIXME: should just whitelist the entire virtual store + std::optional _narHash; + + auto getNarHash = [&]() { + if (!_narHash) { + if (store->isValidPath(storePath)) + _narHash = store->queryPathInfo(storePath)->narHash; + else + _narHash = fetchToStore2(fetchSettings, *store, accessor, FetchMode::DryRun, input.getName()).second; + } + return _narHash; + }; + storeFS->mount(CanonPath(store->printStorePath(storePath)), accessor); - input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true)); + if (forceNarHash + || (requireLockable && (!settings.lazyTrees || !settings.lazyLocks || !input.isLocked(fetchSettings)) + && !input.getNarHash())) + input.attrs.insert_or_assign("narHash", getNarHash()->to_string(HashFormat::SRI, true)); - if (originalInput.getNarHash() && narHash != *originalInput.getNarHash()) + if (originalInput.getNarHash() && *getNarHash() != *originalInput.getNarHash()) throw Error( (unsigned int) 102, "NAR hash mismatch in input '%s', expected '%s' but got '%s'", originalInput.to_string(), - narHash.to_string(HashFormat::SRI, true), - originalInput.getNarHash()->to_string(HashFormat::SRI, true)); + originalInput.getNarHash()->to_string(HashFormat::SRI, true), + getNarHash()->to_string(HashFormat::SRI, true)); return storePath; } diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 46ef49a36759..ced25790ad58 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -1,3 +1,4 @@ +#include "nix/store/async-path-writer.hh" #include "nix/store/derivations.hh" #include "nix/store/downstream-placeholder.hh" #include "nix/expr/eval-inline.hh" @@ -10,7 +11,6 @@ #include "nix/store/names.hh" #include "nix/store/path-references.hh" #include "nix/store/store-api.hh" -#include "nix/util/mounted-source-accessor.hh" #include "nix/util/util.hh" #include "nix/util/os-string.hh" #include "nix/util/processes.hh" @@ -19,6 +19,9 @@ #include "nix/expr/primops.hh" #include "nix/fetchers/fetch-to-store.hh" #include "nix/util/sort.hh" +#include "nix/util/mounted-source-accessor.hh" +#include "nix/expr/provenance.hh" +#include "nix/util/override-provenance-source-accessor.hh" #include #include @@ -64,7 +67,7 @@ std::string EvalState::realiseString(Value & s, StorePathSet * storePathsOutMayb nix::NixStringContext stringContext; auto rawStr = coerceToString(pos, s, stringContext, "while realising a string").toOwned(); auto rewrites = realiseContext(stringContext, storePathsOutMaybe, isIFD); - ensureLazyPathsCopied(stringContext); + return nix::rewriteStrings(rawStr, rewrites); } @@ -75,6 +78,7 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS for (auto & c : context) { auto ensureValid = [&](const StorePath & p) { + waitForPath(p); if (!store->isValidPath(p)) error(p).debugThrow(); }; @@ -89,9 +93,8 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS ensureValid(b.drvPath->getBaseStorePath()); }, [&](const NixStringContextElem::Opaque & o) { - /* If the path happens to be mounted on the storeFS, that means it's lazy path string and would get - copied to the store on-demand (when referenced in a derivation). The string is equal to final - store path where the store object would end up (the path is hashed before mounting). */ + // We consider virtual store paths valid here. They'll + // be devirtualized if needed elsewhere. if (!storeFS->getMount(CanonPath(store->printStorePath(o.path)))) ensureValid(o.path); if (maybePathsOut) @@ -103,6 +106,9 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS if (maybePathsOut) maybePathsOut->emplace(d.drvPath); }, + [&](const NixStringContextElem::Path & p) { + // FIXME: do something? + }, }, c.raw); } @@ -163,8 +169,7 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS return res; } -SourcePath EvalState::realisePath( - const PosIdx pos, Value & v, std::optional resolveSymlinks, CopyLazyPaths copyLazyPaths) +SourcePath EvalState::realisePath(const PosIdx pos, Value & v, std::optional resolveSymlinks) { NixStringContext context; @@ -173,8 +178,6 @@ SourcePath EvalState::realisePath( try { if (!context.empty() && path.accessor == rootFS) { auto rewrites = realiseContext(context); - if (copyLazyPaths == CopyLazyPaths::Copy) - ensureLazyPathsCopied(context); path = {path.accessor, CanonPath(rewriteStrings(path.path.abs(), rewrites))}; } return resolveSymlinks ? path.resolveSymlinks(*resolveSymlinks) : path; @@ -299,6 +302,7 @@ static void import(EvalState & state, const PosIdx pos, Value & vPath, Value * v if (!state.store->isStorePath(path2)) return std::nullopt; auto storePath = state.store->parseStorePath(path2); + state.waitForPath(storePath); if (!(state.store->isValidPath(storePath) && isDerivation(path2))) return std::nullopt; return storePath; @@ -441,11 +445,10 @@ static RegisterPrimOp primop_import( /* !!! Should we pass the Pos or the file name too? */ extern "C" typedef void (*ValueInitializer)(EvalState & state, Value & v); -/* Load a ValueInitializer from a DSO and return whatever it initializes. FIXME: This doesn't - work with chroot stores. */ +/* Load a ValueInitializer from a DSO and return whatever it initializes */ void prim_importNative(EvalState & state, const PosIdx pos, Value ** args, Value & v) { - auto path = state.realisePath(pos, *args[0], SymlinkResolution::Full, EvalState::CopyLazyPaths::Copy); + auto path = state.realisePath(pos, *args[0]); std::string sym( state.forceStringNoCtx(*args[1], pos, "while evaluating the second argument passed to builtins.importNative")); @@ -472,7 +475,7 @@ void prim_importNative(EvalState & state, const PosIdx pos, Value ** args, Value /* We don't dlclose because v may be a primop referencing a function in the shared object file */ } -/* Execute a program and parse its output. FIXME: This doesn't work with chroot stores. */ +/* Execute a program and parse its output */ void prim_exec(EvalState & state, const PosIdx pos, Value ** args, Value & v) { state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.exec"); @@ -510,7 +513,6 @@ void prim_exec(EvalState & state, const PosIdx pos, Value ** args, Value & v) .debugThrow(); } - state.ensureLazyPathsCopied(context); auto output = runProgram(program, true, toOsStrings(std::move(commandArgs))); Expr * parsed; try { @@ -783,10 +785,12 @@ struct CompareValues } }; -typedef std::list> ValueList; - static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** args, Value & v) { + /* The values are rooted via RootValue, so the lists themselves + don't need to be visible to the GC. */ + using ValueList = std::list; + state.forceAttrs(*args[0], noPos, "while evaluating the first argument passed to builtins.genericClosure"); /* Get the start set. */ @@ -800,7 +804,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar ValueList workSet; for (auto elem : startSet->value->listView()) - workSet.push_back(elem); + workSet.push_back(RootValue(elem)); if (startSet->value->listSize() == 0) { v = *startSet->value; @@ -821,7 +825,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar auto cmp = CompareValues(state, noPos, ""); std::map keyToElem(cmp); while (!workSet.empty()) { - Value * e = *(workSet.begin()); + Value * e = **workSet.begin(); workSet.pop_front(); try { @@ -866,7 +870,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar } throw; } - res.push_back(e); + res.push_back(RootValue(e)); /* Call the `operator' function with `e' as argument. */ Value newElements; @@ -881,7 +885,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar for (auto elem : newElements.listView()) { state.forceValue(*elem, noPos); // "while evaluating one one of the elements returned by the `operator` // passed to builtins.genericClosure"); - workSet.push_back(elem); + workSet.push_back(RootValue(elem)); } } catch (Error & err) { err.addTrace( @@ -896,7 +900,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar /* Create the result list. */ auto list = state.buildList(res.size()); for (const auto & [n, i] : enumerate(res)) - list[n] = i; + list[n] = *i; v.mkList(list); } @@ -1018,20 +1022,32 @@ static RegisterPrimOp primop_throw( static void prim_addErrorContext(EvalState & state, const PosIdx pos, Value ** args, Value & v) { + std::exception_ptr error; try { state.forceValue(*args[1], pos); v = *args[1]; + return; + } catch (Error &) { + error = std::current_exception(); + } + + /* Evaluate the message outside of the catch handler: it may force + thunks and thereby suspend the current fiber, which is not + allowed while an exception is being handled. */ + NixStringContext context; + auto message = state + .coerceToString( + pos, + *args[0], + context, + "while evaluating the error message passed to builtins.addErrorContext", + false, + false) + .toOwned(); + + try { + std::rethrow_exception(error); } catch (Error & e) { - NixStringContext context; - auto message = state - .coerceToString( - pos, - *args[0], - context, - "while evaluating the error message passed to builtins.addErrorContext", - false, - false) - .toOwned(); e.addTrace(nullptr, HintFmt(message), TracePrint::Always); throw; } @@ -1179,7 +1195,7 @@ static RegisterPrimOp primop_floor({ a NixInt and if `*number* < -9007199254740992` or `*number* > 9007199254740992`. If the datatype of *number* is neither a NixInt (signed 64-bit integer) nor a NixFloat - (IEEE-754 double-precision floating-point number), an evaluation error will be thrown. + (IEEE-754 double-precision floating-point number), an evaluation error is thrown. )", .impl = prim_floor, }); @@ -1226,7 +1242,7 @@ static RegisterPrimOp primop_tryEval({ `false` if an error was thrown) and `value`, equalling *e* if successful and `false` otherwise. `tryEval` only prevents errors created by `throw` or `assert` from being thrown. - Errors `tryEval` doesn't catch are, for example, those created + Errors that `tryEval` doesn't catch are, for example, those created by `abort` and type errors generated by builtins. Also note that this doesn't evaluate *e* deeply, so `let e = { x = throw ""; }; in (builtins.tryEval e).success` is `true`. Using @@ -1379,7 +1395,7 @@ static RegisterPrimOp primop_warn({ [`debugger-on-trace`](@docroot@/command-ref/conf-file.md#conf-debugger-on-trace) or [`debugger-on-warn`](@docroot@/command-ref/conf-file.md#conf-debugger-on-warn) option is set to `true` and the `--debugger` flag is given, the - interactive debugger will be started when `warn` is called (like + interactive debugger is started when `warn` is called (like [`break`](@docroot@/language/builtins.md#builtins-break)). If the @@ -1403,16 +1419,15 @@ static void prim_second(EvalState & state, const PosIdx pos, Value ** args, Valu * Derivations *************************************************************/ -static void derivationStrictInternal(EvalState & state, std::string_view name, const Bindings * attrs, Value & v); +static void derivationStrictInternal( + EvalState & state, + std::string_view name, + const Bindings * attrs, + Value & v, + std::shared_ptr provenance, + bool acceptMeta); -/* Construct (as a unobservable side effect) a Nix derivation - expression that performs the derivation described by the argument - set. Returns the original set extended with the following - attributes: `outPath' containing the primary output path of the - derivation; `drvPath' containing the path of the Nix expression; - and `type' set to `derivation' to indicate that this is a - derivation. */ -static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value ** args, Value & v) +static void prim_derivationStrictGeneric(EvalState & state, const PosIdx pos, Value ** args, Value & v, bool acceptMeta) { state.forceAttrs(*args[0], pos, "while evaluating the argument passed to builtins.derivationStrict"); @@ -1432,7 +1447,7 @@ static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value ** } try { - derivationStrictInternal(state, drvName, attrs, v); + derivationStrictInternal(state, drvName, attrs, v, state.evalContext.provenance, acceptMeta); } catch (Error & e) { Pos pos = state.positions[nameAttr->pos]; /* @@ -1463,6 +1478,18 @@ static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value ** } } +/* Construct a Nix derivation with metadata provenance */ +static RegisterPrimOp primop_derivationStrictWithMeta( + PrimOp{ + .name = "derivationStrictWithMeta", + .arity = 1, + .impl = + [](EvalState & state, const PosIdx pos, Value ** args, Value & v) { + prim_derivationStrictGeneric(state, pos, args, v, /*acceptMeta=*/true); + }, + .internal = true, + }); + /** * Early validation for the derivation name, for better error message. * It is checked again when constructing store paths. @@ -1486,7 +1513,13 @@ static void checkDerivationName(EvalState & state, std::string_view drvName) } } -static void derivationStrictInternal(EvalState & state, std::string_view drvName, const Bindings * attrs, Value & v) +static void derivationStrictInternal( + EvalState & state, + std::string_view drvName, + const Bindings * attrs, + Value & v, + std::shared_ptr provenance, + bool acceptMeta) { checkDerivationName(state, drvName); @@ -1611,7 +1644,19 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName the environment. */ default: - if (jsonObject) { + if (acceptMeta && i->name == EvalState::s.__meta) { + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) { + state.forceAttrs(*i->value, pos, "while evaluating __meta"); + NixStringContext ctx; + auto obj = printValueAsJSON(state, true, *i->value, pos, ctx); + + if (!ctx.empty()) + throw Error("Derivation __meta provenance can't contain string context like store paths."); + + provenance = + std::make_shared(provenance, make_ref(obj)); + } + } else if (jsonObject) { if (i->name == state.s.structuredAttrs) continue; @@ -1735,6 +1780,10 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName /* Everything in the context of the strings in the derivation attributes should be added as dependencies of the resulting derivation. */ + StringMap rewrites; + + std::optional drvS; + for (auto & c : context) { std::visit( overloaded{ @@ -1746,6 +1795,8 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName [&](const NixStringContextElem::DrvDeep & d) { /* !!! This doesn't work if readOnlyMode is set. */ StorePathSet refs; + // FIXME: don't need to wait, we only need the references. + state.waitForPath(d.drvPath); state.store->computeFSClosure(d.drvPath, refs); for (auto & j : refs) { drv.inputSrcs.insert(j); @@ -1758,13 +1809,26 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName drv.inputDrvs.ensureSlot(*b.drvPath).value.insert(b.output); }, [&](const NixStringContextElem::Opaque & o) { - state.ensureLazyPathCopied(o.path); - drv.inputSrcs.insert(o.path); + drv.inputSrcs.insert(state.devirtualize(o.path, &rewrites)); + }, + [&](const NixStringContextElem::Path & p) { + if (!drvS) + drvS = drv.unparse(*state.store, true); + if (drvS->find(p.storePath.to_string()) != drvS->npos) { + auto devirtualized = state.devirtualize(p.storePath, &rewrites); + warn( + "Using 'builtins.derivation' to create a derivation named '%s' that references the store path '%s' without a proper context. " + "The resulting derivation will not have a correct store reference, so this is unreliable and may stop working in the future.", + drvName, + state.store->printStorePath(devirtualized)); + } }, }, c.raw); } + drv.applyRewrites(rewrites); + /* Do we have all required attributes? */ if (drv.builder == "") state.error("required attribute 'builder' missing").atPos(v).debugThrow(); @@ -1851,13 +1915,11 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName drv.fillInOutputPaths(*state.store); } - /* Write the resulting term into the Nix store directory. - - Unless we are in read-only mode, that is, in which case we do not - write anything. Users commonly do this to speed up evaluation in - contexts where they don't actually want to build anything. */ - auto drvPath = - settings.readOnlyMode ? computeStorePath(*state.store, drv) : state.store->writeDerivation(drv, state.repair); + /* Write the resulting term into the Nix store directory. The + writer handles read-only mode (in which case nothing is written, + which users commonly rely on to speed up evaluation in contexts + where they don't actually want to build anything). */ + auto drvPath = state.store->writeDerivation(*state.asyncPathWriter, drv, state.repair, provenance); auto drvPathS = state.store->printStorePath(drvPath); printMsg(lvlChatty, "instantiated '%1%' -> '%2%'", drvName, drvPathS); @@ -1884,11 +1946,21 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName v.mkAttrs(result); } +/* Construct (as a unobservable side effect) a Nix derivation + expression that performs the derivation described by the argument + set. Returns the original set extended with the following + attributes: `outPath' containing the primary output path of the + derivation; `drvPath' containing the path of the Nix expression; + and `type' set to `derivation' to indicate that this is a + derivation. */ static RegisterPrimOp primop_derivationStrict( PrimOp{ .name = "derivationStrict", .arity = 1, - .impl = prim_derivationStrict, + .impl = + [](EvalState & state, const PosIdx pos, Value ** args, Value & v) { + prim_derivationStrictGeneric(state, pos, args, v, /*acceptMeta=*/false); + }, }); /* Return a placeholder string for the specified output that will be @@ -1972,8 +2044,12 @@ static void prim_storePath(EvalState & state, const PosIdx pos, Value ** args, V if (!state.store->isInStore(sourcePath.path.abs())) state.error("path '%1%' is not in the Nix store", sourcePath).atPos(pos).debugThrow(); auto storePath = state.store->toStorePath(sourcePath.path.abs()).first; - if (!state.storeFS->getMount(CanonPath(state.store->printStorePath(storePath))) && !settings.readOnlyMode) + if (!state.storeFS->getMount(CanonPath(state.store->printStorePath(storePath))) && !settings.readOnlyMode) { + /* The path may still be being written asynchronously (e.g. by + `builtins.toFile`). */ + state.waitForPath(storePath); state.store->ensurePath(storePath); + } context.insert(NixStringContextElem::Opaque{.path = storePath}); v.mkString(sourcePath.path.abs(), context, state.mem); } @@ -2124,14 +2200,17 @@ static void prim_readFile(EvalState & state, const PosIdx pos, Value ** args, Va .debugThrow(); StorePathSet refs; if (state.store->isInStore(path.path.abs())) { - try { - refs = state.store->queryPathInfo(state.store->toStorePath(path.path.abs()).first)->references; - } catch (Error &) { // FIXME: should be InvalidPathError + auto storePath = state.store->toStorePath(path.path.abs()).first; + // Skip virtual paths since they don't have references and + // don't exist anyway. + if (!state.storeFS->getMount(CanonPath(state.store->printStorePath(storePath)))) { + if (auto info = state.store->maybeQueryPathInfo(state.store->toStorePath(path.path.abs()).first)) { + // Re-scan references to filter down to just the ones that actually occur in the file. + auto refsSink = PathRefScanSink::fromPaths(info->references); + refsSink << s; + refs = refsSink.getResultPaths(); + } } - // Re-scan references to filter down to just the ones that actually occur in the file. - auto refsSink = PathRefScanSink::fromPaths(refs); - refsSink << s; - refs = refsSink.getResultPaths(); } NixStringContext context; for (auto && p : std::move(refs)) { @@ -2367,6 +2446,23 @@ static RegisterPrimOp primop_hashFile({ .impl = prim_hashFile, }); +static RegisterPrimOp primop_narHash({ + .name = "__narHash", + .args = {"p"}, + .doc = R"( + Return an SRI representation of the SHA-256 hash of the NAR serialisation of the path *p*. + )", + .impl = + [](EvalState & state, const PosIdx pos, Value ** args, Value & v) { + auto path = state.realisePath(pos, *args[0]); + auto hash = + fetchToStore2(state.fetchSettings, *state.store, path.resolveSymlinks(), FetchMode::DryRun).second; + v.mkString(hash.to_string(HashFormat::SRI, true), state.mem); + }, + // FIXME: may be useful to expose to the user. + .internal = true, +}); + static const Value & fileTypeToString(EvalState & state, SourceAccessor::Type type) { struct Constants @@ -2694,15 +2790,24 @@ static void prim_toFile(EvalState & state, const PosIdx pos, Value ** args, Valu { NixStringContext context; auto name = state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.toFile"); - auto contents = - state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile"); + std::string contents( + state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile")); StorePathSet refs; + StringMap rewrites; for (auto c : context) { - if (auto p = std::get_if(&c.raw)) { - state.ensureLazyPathCopied(p->path); + if (auto p = std::get_if(&c.raw)) refs.insert(p->path); + else if (auto p = std::get_if(&c.raw)) { + if (contents.find(p->storePath.to_string()) != contents.npos) { + auto devirtualized = state.devirtualize(p->storePath, &rewrites); + warn( + "Using 'builtins.toFile' to create a file named '%s' that references the store path '%s' without a proper context. " + "The resulting file will not have a correct store reference, so this is unreliable and may stop working in the future.", + name, + state.store->printStorePath(devirtualized)); + } } else state .error( @@ -2714,23 +2819,14 @@ static void prim_toFile(EvalState & state, const PosIdx pos, Value ** args, Valu .debugThrow(); } - auto storePath = settings.readOnlyMode ? state.store->makeFixedOutputPathFromCA( - name, - TextInfo{ - .hash = hashString(HashAlgorithm::SHA256, contents), - .references = std::move(refs), - }) - : ({ - StringSource s{contents}; - state.store->addToStoreFromDump( - s, - name, - FileSerialisationMethod::Flat, - ContentAddressMethod::Raw::Text, - HashAlgorithm::SHA256, - refs, - state.repair); - }); + contents = rewriteStrings(contents, rewrites); + + /* Write the file asynchronously. The writer skips paths it has + already written during this evaluation and handles read-only + mode. Any consumer of the resulting path waits for the write + to complete (see `EvalState::waitForPath()`). */ + auto storePath = state.asyncPathWriter->addPath( + std::move(contents), std::string(name), std::move(refs), state.repair, state.evalContext.provenance); /* Note: we don't need to add `context' to the context of the result, since `storePath' itself has references to the paths @@ -2873,11 +2969,23 @@ static void addPath( name, ContentAddressWithReferences::fromParts(method, *expectedHash, {refs})); if (!expectedHash || !state.store->isValidPath(*expectedStorePath)) { + // FIXME: make this lazy? // FIXME: support refs in fetchToStore()? + auto path2 = path.resolveSymlinks(); + // Don't use source path provenance if we have a filter applied, since we can't accurately + // record that. Instead, use the current global provenance, since it's better than nothing. + auto path3 = filter + ? SourcePath{ + make_ref( + path2.accessor, state.evalContext.provenance), + path2.path + } + : path2; + auto dstPath = refs.empty() ? fetchToStore( state.fetchSettings, *state.store, - path.resolveSymlinks(), + path3, settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy, name, method, @@ -2885,7 +2993,7 @@ static void addPath( state.repair) : state.store->addToStore( name, - path.resolveSymlinks(), + path3, method, HashAlgorithm::SHA256, refs, @@ -2915,7 +3023,15 @@ static void prim_filterSource(EvalState & state, const PosIdx pos, Value ** args state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.filterSource"); addPath( - state, pos, path.baseName(), path, args[0], ContentAddressMethod::Raw::NixArchive, std::nullopt, v, context); + state, + pos, + state.computeBaseName(path, pos), + path, + args[0], + ContentAddressMethod::Raw::NixArchive, + std::nullopt, + v, + context); } static RegisterPrimOp primop_filterSource({ @@ -3133,7 +3249,7 @@ void prim_getAttr(EvalState & state, const PosIdx pos, Value ** args, Value & v) auto i = state.getAttr(state.symbols.create(attr), args[1]->attrs(), "in the attribute set under consideration"); // !!! add to stack trace? if (state.countCalls && i->pos) - state.attrSelects[i->pos]++; + state.attrSelects->try_emplace_or_visit(i->pos, 1, [](auto & j) { j.second++; }); state.forceValue(*i->value, pos); v = *i->value; } @@ -3591,57 +3707,139 @@ static RegisterPrimOp primop_mapAttrs({ .impl = prim_mapAttrs, }); -static void prim_zipAttrsWith(EvalState & state, const PosIdx pos, Value ** args, Value & v) +static void prim_filterAttrs(EvalState & state, const PosIdx pos, Value ** args, Value & v) { - // we will first count how many values are present for each given key. - // we then allocate a single attrset and pre-populate it with lists of - // appropriate sizes, stash the pointers to the list elements of each, - // and populate the lists. after that we replace the list in the every - // attribute with the merge function application. this way we need not - // use (slightly slower) temporary storage the GC does not know about. - - struct Item - { - size_t size = 0; - size_t pos = 0; - std::optional list; - }; + state.forceAttrs(*args[1], pos, "while evaluating the second argument passed to builtins.filterAttrs"); + + if (args[1]->attrs()->empty()) { + v = *args[1]; + return; + } + + state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.filterAttrs"); + + auto attrs = state.buildBindings(args[1]->attrs()->size()); + + for (auto & i : *args[1]->attrs()) { + Value * vName = Value::toPtr(state.symbols[i.name]); + Value * callArgs[] = {vName, i.value}; + Value res; + state.callFunction(*args[0], callArgs, res, noPos); + if (state.forceBool( + res, pos, "while evaluating the return value of the filtering function passed to builtins.filterAttrs")) + attrs.insert(i.name, i.value); + } + + v.mkAttrs(attrs.alreadySorted()); +} + +static RegisterPrimOp primop_filterAttrs({ + .name = "__filterAttrs", + .args = {"f", "attrset"}, + .doc = R"( + Return an attribute set consisting of the attributes in *attrset* for which + the function *f* returns `true`. The function *f* is called with two arguments: + the name of the attribute and the value of the attribute. For example, + + ```nix + builtins.filterAttrs (name: value: name == "foo") { foo = 1; bar = 2; } + ``` + + evaluates to `{ foo = 1; }`. + )", + .impl = prim_filterAttrs, +}); + +/** + * A (name, value) pair collected by primops that group values by name + * (`zipAttrsWith`, `groupBy`). + * + * Note that vectors of these can be invisible to the GC (provided that + * the values are reachable in some other way for the duration of the + * primop, e.g. from the primop's arguments), which avoids allocating + * GC-visible (uncollectable) storage for temporaries. The latter is + * expensive and a source of GC allocation lock contention during + * parallel evaluation. + */ +struct NameValue +{ + Symbol name; + Value * value; +}; - std::map, traceable_allocator>> attrsSeen; +/** + * Stably sort `items` by name, so that the values of equal names form + * contiguous runs in their original relative order, and return the + * number of distinct names. The resulting order is the same as that of + * a std::map. + */ +static size_t sortByName(std::vector & items) +{ + std::stable_sort( + items.begin(), items.end(), [](const NameValue & a, const NameValue & b) { return a.name < b.name; }); + + size_t nrNames = 0; + for (size_t i = 0; i < items.size(); ++i) + if (i == 0 || items[i - 1].name != items[i].name) + nrNames++; + return nrNames; +} +/** + * Call `f(name, list)` for every distinct name in `items` (which must + * have been sorted with `sortByName()`), where `list` is a ListBuilder + * containing the values associated with that name. + */ +template +static void forEachByName(EvalState & state, const std::vector & items, F f) +{ + for (size_t i = 0; i < items.size();) { + auto sym = items[i].name; + size_t j = i; + while (j < items.size() && items[j].name == sym) + j++; + auto list = state.buildList(j - i); + for (size_t k = i; k < j; ++k) + list[k - i] = items[k].value; + f(sym, list); + i = j; + } +} + +static void prim_zipAttrsWith(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.zipAttrsWith"); state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.zipAttrsWith"); const auto listItems = args[1]->listView(); + size_t nrAttrs = 0; for (auto & vElem : listItems) { state.forceAttrs( *vElem, noPos, "while evaluating a value of the list passed as second argument to builtins.zipAttrsWith"); - for (auto & attr : *vElem->attrs()) - attrsSeen.try_emplace(attr.name).first->second.size++; + nrAttrs += vElem->attrs()->size(); } - for (auto & [sym, elem] : attrsSeen) - elem.list.emplace(state.buildList(elem.size)); + /* The vector is invisible to the GC, but that's fine: the values are + kept alive by the attrsets in *args[1]. */ + std::vector attrsSeen; + attrsSeen.reserve(nrAttrs); - for (auto & vElem : listItems) { - for (auto & attr : *vElem->attrs()) { - auto & item = attrsSeen.at(attr.name); - (*item.list)[item.pos++] = attr.value; - } - } + for (auto & vElem : listItems) + for (auto & attr : *vElem->attrs()) + attrsSeen.push_back({attr.name, attr.value}); - auto attrs = state.buildBindings(attrsSeen.size()); + auto attrs = state.buildBindings(sortByName(attrsSeen)); - for (auto & [sym, elem] : attrsSeen) { + forEachByName(state, attrsSeen, [&](Symbol sym, ListBuilder & list) { auto name = Value::toPtr(state.symbols[sym]); auto call1 = state.allocValue(); call1->mkApp(args[0], name); auto call2 = state.allocValue(); auto arg = state.allocValue(); - arg->mkList(*elem.list); + arg->mkList(list); call2->mkApp(call1, arg); attrs.insert(sym, call2); - } + }); v.mkAttrs(attrs.alreadySorted()); } @@ -4000,8 +4198,8 @@ static void anyOrAll(bool any, EvalState & state, const PosIdx pos, Value ** arg std::string_view errorCtx = any ? "while evaluating the return value of the function passed to builtins.any" : "while evaluating the return value of the function passed to builtins.all"; - Value vTmp; for (auto elem : args[1]->listView()) { + Value vTmp; state.callFunction(*args[0], *elem, vTmp, pos); bool res = state.forceBool(vTmp, pos, errorCtx); if (res == any) { @@ -4266,26 +4464,24 @@ static void prim_groupBy(EvalState & state, const PosIdx pos, Value ** args, Val state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.groupBy"); state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.groupBy"); - ValueVectorMap attrs; + const auto listItems = args[1]->listView(); + + /* The vector is invisible to the GC, but that's fine: the values are + kept alive by the list in *args[1]. */ + std::vector items; + items.reserve(listItems.size()); - for (auto vElem : args[1]->listView()) { + for (auto vElem : listItems) { Value res; state.callFunction(*args[0], *vElem, res, pos); auto name = state.forceStringNoCtx( res, pos, "while evaluating the return value of the grouping function passed to builtins.groupBy"); - auto sym = state.symbols.create(name); - auto vector = attrs.try_emplace(sym, {}).first; - vector->second.push_back(vElem); + items.push_back({state.symbols.create(name), vElem}); } - auto attrs2 = state.buildBindings(attrs.size()); + auto attrs2 = state.buildBindings(sortByName(items)); - for (auto & i : attrs) { - auto size = i.second.size(); - auto list = state.buildList(size); - memcpy(list.elems, i.second.data(), sizeof(Value *) * size); - attrs2.alloc(i.first).mkList(list); - } + forEachByName(state, items, [&](Symbol sym, ListBuilder & list) { attrs2.alloc(sym).mkList(list); }); v.mkAttrs(attrs2.alreadySorted()); } @@ -4694,8 +4890,9 @@ static void prim_hashString(EvalState & state, const PosIdx pos, Value ** args, state.error("unknown hash algorithm '%1%'", algo).atPos(pos).debugThrow(); NixStringContext context; // discarded - auto s = - state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.hashString"); + auto s = state.devirtualize( + state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.hashString"), + context); v.mkString(hashString(*ha, s).to_string(HashFormat::Base16, false), state.mem); } @@ -5326,9 +5523,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings) )", }); - if (!settings.pureEval) { - v.mkInt(time(nullptr)); - } + v.mkInt(time(nullptr)); addConstant( "__currentTime", v, @@ -5356,8 +5551,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings) .impureOnly = true, }); - if (!settings.pureEval) - v.mkString(settings.getCurrentSystem(), mem); + v.mkString(settings.getCurrentSystem(), mem); addConstant( "__currentSystem", v, @@ -5543,6 +5737,40 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings) .type = nFunction, }); + auto vDerivationWithMeta = allocValue(); + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) { + addConstant( + "derivationWithMeta", + vDerivationWithMeta, + { + .type = nFunction, + .doc = R"( + Create a derivation with metadata. + `derivationWithMeta` takes the same attributes as [`derivation`](@docroot@/language/derivations.md) plus an optional `__meta` attribute set. + + Nix records `__meta` as JSON in the [provenance](@docroot@/development/experimental-features.md#xp-feature-provenance) of the derivation and of any outputs that Nix builds from it, while [`nix provenance show`](@docroot@/command-ref/new-cli/nix3-provenance-show.md) displays it. + `__meta` doesn't affect the derivation's store path and Nix doesn't pass it to the builder. + Nix doesn't update the provenance of outputs that already exist, such as outputs that were built earlier or substituted. + + ```nix + builtins.derivationWithMeta { + name = "hello"; + system = builtins.currentSystem; + builder = "/bin/sh"; + args = [ "-c" "echo hello > $out" ]; + __meta = { + description = "Prints a greeting"; + license = "MIT"; + }; + } + ``` + + See [Derivation metadata](@docroot@/language/derivation-metadata.md) for details. + )", + .experimentalFeature = Xp::Provenance, + }); + } + /* Now that we've added all primops, sort the `builtins' set, because attribute lookups expect it to be sorted. */ const_cast(getBuiltins().attrs())->sort(); @@ -5551,7 +5779,14 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings) /* Note: we have to initialize the 'derivation' constant *after* building baseEnv/staticBaseEnv because it uses 'builtins'. */ - evalFile(derivationInternal, *vDerivation); + auto vDerivationValue = allocValue(); + evalFile(derivationInternal, *vDerivationValue); + + callFunction(*vDerivationValue, getBuiltin("derivationStrict"), *vDerivation, PosIdx()); + + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) + callFunction( + *vDerivationValue, ***get(internalPrimOps, "derivationStrictWithMeta"), *vDerivationWithMeta, PosIdx()); } } // namespace nix diff --git a/src/libexpr/primops/bakedDerivation.cc b/src/libexpr/primops/bakedDerivation.cc new file mode 100644 index 000000000000..74f72ad2b2db --- /dev/null +++ b/src/libexpr/primops/bakedDerivation.cc @@ -0,0 +1,181 @@ +#include "nix/expr/primops.hh" +#include "nix/expr/eval-inline.hh" +#include "nix/store/derivations.hh" +#include "nix/store/derived-path.hh" +#include "nix/store/store-api.hh" +#include "nix/store/globals.hh" + +#include + +namespace nix { + +static void prim_bakedDerivation(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + state.forceAttrs(*args[0], pos, "while evaluating the argument passed to builtins.bakedDerivation"); + + std::optional name; + + /* TODO: add a `narHash` attribute to assert the known contents of a pre-built output. */ + std::map outputs; + + for (auto & attr : *args[0]->attrs()) { + std::string_view attrName = state.symbols[attr.name]; + auto attrHint = fmt("while evaluating the attribute '%s' passed to builtins.bakedDerivation", attrName); + + if (attrName == "name") { + name = state.forceStringNoCtx(*attr.value, attr.pos, attrHint); + } + + else if (attrName == "outputs") { + state.forceAttrs(*attr.value, attr.pos, attrHint); + for (auto & outAttr : *attr.value->attrs()) { + std::string_view outName = state.symbols[outAttr.name]; + + state.forceAttrs( + *outAttr.value, + outAttr.pos, + fmt("while evaluating the output '%s' passed to builtins.bakedDerivation", outName)); + + std::optional path; + + for (auto & outField : *outAttr.value->attrs()) { + std::string_view fieldName = state.symbols[outField.name]; + auto fieldHint = + fmt("while evaluating the attribute '%s' of output '%s' passed to builtins.bakedDerivation", + fieldName, + outName); + + if (fieldName == "path") { + auto s = state.forceStringNoCtx(*outField.value, outField.pos, fieldHint); + path = state.store->parseStorePath(s); + } + + else + state + .error( + "attribute '%s' isn't supported in an output passed to 'builtins.bakedDerivation'", + fieldName) + .atPos(outField.pos) + .debugThrow(); + } + + if (!path) + state + .error( + "attribute 'path' is missing in output '%s' passed to 'builtins.bakedDerivation'", outName) + .atPos(outAttr.pos) + .debugThrow(); + + outputs.emplace(std::string(outName), std::move(*path)); + } + } + + else + state.error("attribute '%s' isn't supported in call to 'builtins.bakedDerivation'", attrName) + .atPos(attr.pos) + .debugThrow(); + } + + if (!name) + state.error("attribute 'name' is missing in call to 'builtins.bakedDerivation'") + .atPos(pos) + .debugThrow(); + + if (outputs.empty()) + state.error("attribute 'outputs' is missing or empty in call to 'builtins.bakedDerivation'") + .atPos(pos) + .debugThrow(); + + Derivation drv; + drv.name = *name; + drv.platform = "builtin"; + drv.builder = "builtin:substitute"; + + for (auto & [outName, outPath] : outputs) + drv.outputs.insert_or_assign(outName, DerivationOutput::InputAddressed{.path = outPath}); + + /* As in `derivationStrict`, don't write the derivation in read-only mode. */ + auto drvPath = settings.readOnlyMode ? computeStorePath(*state.store, drv) + : state.store->writeDerivation(*state.asyncPathWriter, drv, state.repair); + + /* As in `derivationStrict`, cache the derivation hash so that derivations depending on this one don't need to + read it back from the store. */ + drvHashes.insert_or_assign(drvPath, hashDerivationModulo(*state.store, drv, false)); + + /* Return an attribute set of the same shape as `derivationStrict`: the derivation path and one string per + output. The output strings carry string context, so that they can be used as inputs of other derivations. */ + auto result = state.buildBindings(1 + drv.outputs.size()); + result.alloc(state.s.drvPath) + .mkString( + state.store->printStorePath(drvPath), + { + NixStringContextElem::DrvDeep{.drvPath = drvPath}, + }, + state.mem); + for (auto & [outName, outPath] : outputs) + state.mkOutputString( + result.alloc(outName), + SingleDerivedPath::Built{ + .drvPath = makeConstantStorePathRef(drvPath), + .output = outName, + }, + outPath); + + v.mkAttrs(result); +} + +static RegisterPrimOp primop_bakedDerivation({ + .name = "__bakedDerivation", + .args = {"attrs"}, + .doc = R"( + Create a *baked derivation*: a derivation that is never built, but + whose outputs are obtained by substitution from a binary cache. + This is what [`nix flake bake`](@docroot@/command-ref/new-cli/nix3-flake-bake.md) + uses to represent the pre-evaluated outputs of a flake. + + The argument is an attribute set with the following attributes: + + - `name`: The name of the derivation. + + - `outputs`: An attribute set mapping output names to attribute + sets with a single attribute `path`, the store path of that + output. + + The result has the same shape as the result of + `builtins.derivationStrict`: an attribute set containing `drvPath` + (the store path of the derivation) and one attribute per output + containing that output's store path. These strings carry + [string context](@docroot@/language/string-context.md), so the + outputs of a baked derivation can be used as inputs of other + derivations. + + The resulting derivation uses the builder `builtin:substitute` + and has no inputs. Realising it does not run anything; instead, + Nix substitutes the given output paths. If they cannot be + substituted, the build fails. + + Example: + + ```nix + builtins.bakedDerivation { + name = "hello-2.12.1"; + outputs = { + out.path = "/nix/store/1q8w6grh…-hello-2.12.1"; + }; + } + ``` + + evaluates to + + ```nix + { + drvPath = "/nix/store/1jczli5n…-hello-2.12.1.drv"; + out = "/nix/store/1q8w6grh…-hello-2.12.1"; + } + ``` + )", + .impl = prim_bakedDerivation, + .experimentalFeature = Xp::BakedDerivations, +}); + +} // namespace nix diff --git a/src/libexpr/primops/context.cc b/src/libexpr/primops/context.cc index d5d5de0b9aaf..3410a4b4a453 100644 --- a/src/libexpr/primops/context.cc +++ b/src/libexpr/primops/context.cc @@ -8,10 +8,16 @@ namespace nix { static void prim_unsafeDiscardStringContext(EvalState & state, const PosIdx pos, Value ** args, Value & v) { - NixStringContext context; + NixStringContext context, filtered; + auto s = state.coerceToString( pos, *args[0], context, "while evaluating the argument passed to builtins.unsafeDiscardStringContext"); - v.mkString(*s, state.mem); + + for (auto & c : context) + if (auto * p = std::get_if(&c.raw)) + filtered.insert(*p); + + v.mkString(*s, filtered, state.mem); } static RegisterPrimOp primop_unsafeDiscardStringContext({ @@ -23,11 +29,19 @@ static RegisterPrimOp primop_unsafeDiscardStringContext({ .impl = prim_unsafeDiscardStringContext, }); +bool hasContext(const NixStringContext & context) +{ + for (auto & c : context) + if (!std::get_if(&c.raw)) + return true; + return false; +} + static void prim_hasContext(EvalState & state, const PosIdx pos, Value ** args, Value & v) { NixStringContext context; state.forceString(*args[0], context, pos, "while evaluating the argument passed to builtins.hasContext"); - v.mkBool(!context.empty()); + v.mkBool(hasContext(context)); } static RegisterPrimOp primop_hasContext( @@ -62,6 +76,7 @@ static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx p NixStringContext context2; for (auto && c : context) { if (auto * ptr = std::get_if(&c.raw)) { + state.waitForPath(ptr->drvPath); // FIXME: why? context2.emplace(NixStringContextElem::Opaque{.path = ptr->drvPath}); } else { /* Can reuse original item */ @@ -133,6 +148,11 @@ static void prim_addDrvOutputDependencies(EvalState & state, const PosIdx pos, V above does not make much sense. */ return std::move(c); }, + [&](const NixStringContextElem::Path & p) -> NixStringContextElem::DrvDeep { + state.error("`addDrvOutputDependencies` does not work on a string without context") + .atPos(pos) + .debugThrow(); + }, }, context.begin()->raw)}), }; @@ -201,6 +221,7 @@ static void prim_getContext(EvalState & state, const PosIdx pos, Value ** args, contextInfos[std::move(drvPath)].outputs.emplace_back(std::move(b.output)); }, [&](NixStringContextElem::Opaque && o) { contextInfos[std::move(o.path)].path = true; }, + [&](NixStringContextElem::Path && p) {}, }, ((NixStringContextElem &&) i).raw); } @@ -271,8 +292,11 @@ static void prim_appendContext(EvalState & state, const PosIdx pos, Value ** arg if (!state.store->isStorePath(name)) state.error("context key '%s' is not a store path", name).atPos(i.pos).debugThrow(); auto namePath = state.store->parseStorePath(name); - if (!settings.readOnlyMode) + if (!settings.readOnlyMode) { + /* The path may be a derivation that is still being written asynchronously. */ + state.waitForPath(namePath); state.store->ensurePath(namePath); + } state.forceAttrs(*i.value, i.pos, "while evaluating the value of a string context"); if (auto attr = i.value->attrs()->get(sPath)) { diff --git a/src/libexpr/primops/derivation.nix b/src/libexpr/primops/derivation.nix index 1044fbd26db2..06c7eb8a67e0 100644 --- a/src/libexpr/primops/derivation.nix +++ b/src/libexpr/primops/derivation.nix @@ -26,6 +26,7 @@ Note that `derivation` is very bare-bones, and provides almost no commands during the build. Most likely, you'll want to use functions like `stdenv.mkDerivation` in Nixpkgs to set up a basic environment. */ +drvFunc: drvAttrs@{ outputs ? [ "out" ], ... @@ -33,7 +34,7 @@ drvAttrs@{ let - strict = derivationStrict drvAttrs; + strict = drvFunc drvAttrs; commonAttrs = drvAttrs diff --git a/src/libexpr/primops/fetchClosure.cc b/src/libexpr/primops/fetchClosure.cc index fe3873c101b4..aebafb35b0d4 100644 --- a/src/libexpr/primops/fetchClosure.cc +++ b/src/libexpr/primops/fetchClosure.cc @@ -139,7 +139,7 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value ** args std::optional inputAddressedMaybe; for (auto & attr : *args[0]->attrs()) { - const auto & attrName = state.symbols[attr.name]; + std::string_view attrName = state.symbols[attr.name]; auto attrHint = [&]() -> std::string { return fmt("while evaluating the attribute '%s' passed to builtins.fetchClosure", attrName); }; diff --git a/src/libexpr/primops/fetchMercurial.cc b/src/libexpr/primops/fetchMercurial.cc index 97aeb0a03574..172c37948c93 100644 --- a/src/libexpr/primops/fetchMercurial.cc +++ b/src/libexpr/primops/fetchMercurial.cc @@ -79,7 +79,7 @@ static void prim_fetchMercurial(EvalState & state, const PosIdx pos, Value ** ar attrs.insert_or_assign("rev", rev->gitRev()); auto input = fetchers::Input::fromAttrs(state.fetchSettings, std::move(attrs)); - auto [storePath, input2] = input.fetchToStore(state.fetchSettings, *state.store); + auto [storePath, accessor, input2] = input.fetchToStore(state.fetchSettings, *state.store); auto attrs2 = state.buildBindings(8); state.mkStorePathString(storePath, attrs2.alloc(state.s.outPath)); diff --git a/src/libexpr/primops/fetchTree.cc b/src/libexpr/primops/fetchTree.cc index 0e092927b39d..392294abc7e0 100644 --- a/src/libexpr/primops/fetchTree.cc +++ b/src/libexpr/primops/fetchTree.cc @@ -126,12 +126,16 @@ void emitTreeAttrs( { auto attrs = state.buildBindings(100); - state.mkStorePathString(storePath, attrs.alloc(state.s.outPath)); + auto & vStorePath = attrs.alloc(state.s.outPath); + state.mkStorePathString(storePath, vStorePath); // FIXME: support arbitrary input attributes. if (auto narHash = input.getNarHash()) attrs.alloc("narHash").mkString(narHash->to_string(HashFormat::SRI, true), state.mem); + else + // Lazily compute the NAR hash for backward compatibility. + attrs.alloc("narHash").mkApp(**get(state.internalPrimOps, "narHash"), &vStorePath); if (input.getType() == "git") attrs.alloc("submodules").mkBool(fetchers::maybeGetBoolAttr(input.attrs, "submodules").value_or(false)); @@ -175,7 +179,6 @@ struct FetchTreeParams bool emptyRevFallback = false; bool allowNameArgument = false; bool isFetchGit = false; - bool isFinal = false; }; static void fetchTree( @@ -249,11 +252,6 @@ static void fetchTree( attrs.emplace("exportIgnore", Explicit{true}); } - // fetchTree should fetch git repos with shallow = true by default - if (type == "git" && !params.isFetchGit && !attrs.contains("shallow")) { - attrs.emplace("shallow", Explicit{true}); - } - if (!params.allowNameArgument) if (auto nameIter = attrs.find("name"); nameIter != attrs.end()) state.error("argument 'name' isn’t supported in call to '%s'", fetcher) @@ -282,17 +280,11 @@ static void fetchTree( } input = fetchers::Input::fromAttrs(state.fetchSettings, std::move(attrs)); } else { - if (!experimentalFeatureSettings.isEnabled(Xp::Flakes)) - state - .error( - "passing a string argument to '%s' requires the 'flakes' experimental feature", fetcher) - .atPos(pos) - .debugThrow(); input = fetchers::Input::fromURL(state.fetchSettings, url); } } - if (!state.settings.pureEval && !input.isDirect() && experimentalFeatureSettings.isEnabled(Xp::Flakes)) + if (!state.settings.pureEval && !input.isDirect()) input = lookupInRegistries(state.fetchSettings, *state.store, input, fetchers::UseRegistries::Limited).first; if (state.settings.pureEval && !input.isLocked(state.fetchSettings)) { @@ -311,17 +303,13 @@ static void fetchTree( state.checkURI(input.toURLString()); - if (params.isFinal) { + if (input.getNarHash()) input.attrs.insert_or_assign("__final", Explicit(true)); - } else { - if (input.isFinal()) - throw Error("input '%s' is not allowed to use the '__final' attribute", input.to_string()); - } auto cachedInput = state.inputCache->getAccessor(state.fetchSettings, *state.store, input, fetchers::UseRegistries::No); - auto storePath = state.mountInput(cachedInput.lockedInput, input, cachedInput.accessor); + auto storePath = state.mountInput(cachedInput.lockedInput, input, cachedInput.accessor, true); emitTreeAttrs(state, storePath, cachedInput.lockedInput, v, params.emptyRevFallback, false); } @@ -416,7 +404,6 @@ static RegisterPrimOp primop_fetchTree({ - `"mercurial"` *input* can also be a [URL-like reference](@docroot@/command-ref/new-cli/nix3-flake.md#flake-references). - The additional input types and the URL-like syntax requires the [`flakes` experimental feature](@docroot@/development/experimental-features.md#xp-feature-flakes) to be enabled. > **Example** > @@ -456,19 +443,6 @@ static RegisterPrimOp primop_fetchTree({ return doc; }(), .impl = prim_fetchTree, - .experimentalFeature = Xp::FetchTree, -}); - -void prim_fetchFinalTree(EvalState & state, const PosIdx pos, Value ** args, Value & v) -{ - fetchTree(state, pos, args, v, {.isFinal = true}); -} - -static RegisterPrimOp primop_fetchFinalTree({ - .name = "fetchFinalTree", - .args = {"input"}, - .impl = prim_fetchFinalTree, - .internal = true, }); static void fetch( @@ -586,7 +560,7 @@ static void fetch( auto input = fetchers::Input::fromAttrs(state.fetchSettings, std::move(attrs)); auto cachedInput = state.inputCache->getAccessor(state.fetchSettings, *state.store, input, fetchers::UseRegistries::No); - auto storePath = state.mountInput(cachedInput.lockedInput, input, cachedInput.accessor); + auto storePath = state.mountInput(cachedInput.lockedInput, input, cachedInput.accessor, false); state.mkStorePathString(storePath, v); } else { auto storePath = fetchers::downloadFile(*state.store, state.fetchSettings, *url, name).storePath; @@ -826,7 +800,7 @@ static RegisterPrimOp primop_fetchGit({ name in the `ref` attribute. However, if the revision you're looking for is in a future - branch for the non-default branch you will need to specify the + branch for the non-default branch you need to specify the the `ref` attribute as well. ```nix diff --git a/src/libexpr/primops/meson.build b/src/libexpr/primops/meson.build index b8abc6409af9..9d15b4003fcf 100644 --- a/src/libexpr/primops/meson.build +++ b/src/libexpr/primops/meson.build @@ -4,9 +4,14 @@ generated_headers += gen_header.process( ) sources += files( + 'bakedDerivation.cc', 'context.cc', 'fetchClosure.cc', 'fetchMercurial.cc', 'fetchTree.cc', 'fromTOML.cc', ) + +if wasmtime_required.enabled() + sources += files('wasm.cc') +endif diff --git a/src/libexpr/primops/wasm.cc b/src/libexpr/primops/wasm.cc new file mode 100644 index 000000000000..f13a6955b10f --- /dev/null +++ b/src/libexpr/primops/wasm.cc @@ -0,0 +1,920 @@ +#include "nix/expr/primops.hh" +#include "nix/expr/eval-inline.hh" +#include "nix/expr/parallel-eval.hh" +#include "nix/util/users.hh" +#include "nix/util/file-system.hh" + +#include +#include + +using namespace wasmtime; + +namespace nix { + +using ValueId = uint32_t; + +template +T unwrap(Result && res) +{ + if (res) + return res.ok(); + throw Error(res.err().message()); +} + +/** + * Enable wasmtime's compilation cache, so that a module is only compiled + * once per machine rather than once per process. The cache lives in Nix's + * cache directory; wasmtime is configured through a TOML file, which we + * write there as well. + */ +static void enableCompilationCache(wasmtime::Config & config) +{ + try { + auto cacheDir = getCacheDir() / "wasmtime"; + createDirs(cacheDir); + + // TOML literal strings cannot contain single quotes. + auto dir = cacheDir.string(); + if (dir.find('\'') != std::string::npos) + throw Error("cache directory '%s' contains a single quote", dir); + + auto configFile = cacheDir / "config.toml"; + auto contents = fmt("[cache]\ndirectory = '%s'\n", dir); + writeFile(configFile, contents); + + unwrap(config.cache_load(configFile.string())); + } catch (Error & e) { + // Not being able to cache is not fatal. + warn("unable to enable the Wasm compilation cache: %s", e.msg()); + } +} + +static Engine & getEngine() +{ + static Engine engine = []() { + wasmtime::Config config; + config.pooling_allocation_strategy(PoolAllocationConfig()); + config.memory_init_cow(true); + enableCompilationCache(config); + return Engine(std::move(config)); + }(); + return engine; +} + +static std::span string2span(std::string_view s) +{ + return std::span((uint8_t *) s.data(), s.size()); +} + +static std::string_view span2string(std::span s) +{ + return std::string_view((char *) s.data(), s.size()); +} + +// FIXME: move to wasmtime C++ wrapper. +class InstancePre +{ + WASMTIME_OWN_WRAPPER(InstancePre, wasmtime_instance_pre); + +public: + TrapResult instantiate(wasmtime::Store::Context cx) + { + wasmtime_instance_t instance; + wasm_trap_t * trap = nullptr; + auto * error = wasmtime_instance_pre_instantiate(ptr.get(), cx.capi(), &instance, &trap); + if (error != nullptr) { + return TrapError(wasmtime::Error(error)); + } + if (trap != nullptr) { + return TrapError(Trap(trap)); + } + return Instance(instance); + } +}; + +TrapResult instantiate_pre(Linker & linker, const Module & m) +{ + wasmtime_instance_pre_t * instance_pre; + auto * error = wasmtime_linker_instantiate_pre(linker.capi(), m.capi(), &instance_pre); + if (error != nullptr) { + return TrapError(wasmtime::Error(error)); + } + return InstancePre(instance_pre); +} + +static void regFuns(Linker & linker, bool useWasi); + +struct NixWasmInstancePre +{ + Engine & engine = getEngine(); + std::string name; + bool useWasi = false; + InstancePre instancePre; + + InstancePre compile(std::span bytes) + { + // Compile the module + auto module = unwrap(Module::compile(engine, bytes)); + + // Auto-detect WASI by checking for wasi_snapshot_preview1 imports. + for (const auto & ref : module.imports()) + if (const_cast &>(ref).module() == "wasi_snapshot_preview1") { + useWasi = true; + break; + } + + // Create linker with appropriate WASI support + Linker linker(engine); + if (useWasi) + unwrap(linker.define_wasi()); + regFuns(linker, useWasi); + + return unwrap(instantiate_pre(linker, module)); + } + + NixWasmInstancePre(SourcePath wasmPath) + : name(wasmPath.baseName()) + , instancePre(compile(string2span(wasmPath.readFile()))) + { + } + + NixWasmInstancePre(std::string_view wat) + : name("") + , instancePre([&] { + auto wasm = unwrap(wat2wasm(wat)); + return compile(std::span(wasm)); + }()) + { + } +}; + +struct NixWasmInstance +{ + EvalState & state; + ref pre; + wasmtime::Store wasmStore; + wasmtime::Store::Context wasmCtx; + Instance instance; + Memory memory_; + // The guest's allocator, if it exports one (see `allocInGuest`). + std::optional allocFn; + + ValueVector values; + std::exception_ptr ex; + + std::optional functionName; + + ValueId resultId = 0; + + std::string logPrefix; + + NixWasmInstance(EvalState & _state, ref _pre) + : state(_state) + , pre(_pre) + , wasmStore(pre->engine) + , wasmCtx(wasmStore) + , instance(unwrap(pre->instancePre.instantiate(wasmCtx))) + , memory_(getExport("memory")) + , logPrefix(pre->name) + { + wasmCtx.set_data(this); + + if (auto ext = instance.get(wasmCtx, "nix_wasm_alloc")) { + auto fun = std::get_if(&*ext); + if (!fun) + throw Error("export 'nix_wasm_alloc' of Wasm module '%s' is not a function", pre->name); + auto type = fun->type(wasmCtx); + auto params = type->params().begin(); + auto results = type->results().begin(); + auto i32Type = ValType::i32(); + ValType::Ref i32(i32Type); + if (type->params().size() != 2 || type->results().size() != 1 || params[0] != i32 || params[1] != i32 + || results[0] != i32) + throw Error( + "export 'nix_wasm_alloc' of Wasm module '%s' does not have type '(size: i32, align: i32) -> i32'", + pre->name); + allocFn = *fun; + } + + /* Reserve value ID 0 so it can be used in functions like get_attr() to denote a missing attribute. */ + values.push_back(nullptr); + } + + ValueId addValue(Value * v) + { + auto id = values.size(); + values.emplace_back(v); + return id; + } + + std::pair allocValue() + { + auto v = state.allocValue(); + auto id = addValue(v); + return {id, *v}; + } + + Value & getValue(ValueId id) + { + if (id >= values.size() || id == 0) + throw Error("invalid ValueId %d", id); + return *values[id]; + } + + template + T getExport(std::string_view name) + { + auto ext = instance.get(wasmCtx, name); + if (!ext) + throw Error("Wasm module '%s' does not export '%s'", pre->name, name); + auto res = std::get_if(&*ext); + if (!res) + throw Error("export '%s' of Wasm module '%s' does not have the right type", name, pre->name); + return *res; + } + + std::vector runFunction(std::string_view name, const std::vector & args) + { + functionName = name; + return unwrap(getExport(name).call(wasmCtx, args)); + } + + auto memory() + { + return memory_.data(wasmCtx); + } + + /** + * A view of `len` bytes of guest memory at `ptr`, checked against the current size of the memory. All accesses + * to guest memory with guest-supplied offsets must go through this, since `std::span::subspan` does not check + * bounds. The memory is fetched afresh each time, since it may have grown (e.g. by `allocInGuest`). + */ + std::span guestSpan(uint32_t ptr, size_t len) + { + auto mem = memory(); + if (ptr > mem.size() || len > mem.size() - ptr) + throw Error( + "Wasm memory access out of bounds (offset %d, length %d, memory size %d)", ptr, len, mem.size()); + return mem.subspan(ptr, len); + } + + /** + * Like `guestSpan`, for an array of `count` values of type `T`. + */ + template + std::span guestSpan(uint32_t ptr, size_t count) + { + if (count > std::numeric_limits::max() / sizeof(T)) + throw Error("Wasm memory access out of bounds (offset %d, count %d)", ptr, count); + auto s = guestSpan(ptr, count * sizeof(T)); + return std::span((T *) s.data(), count); + } + + std::monostate panic(uint32_t ptr, uint32_t len) + { + throw Error("Wasm panic: %s", Uncolored(span2string(guestSpan(ptr, len)))); + } + + std::monostate warn(uint32_t ptr, uint32_t len) + { + doWarn(span2string(guestSpan(ptr, len))); + return {}; + } + + void doWarn(std::string_view s) + { + if (functionName) + nix::warn("'%s' function '%s': %s", logPrefix, functionName.value_or(""), s); + else + nix::warn("'%s': %s", logPrefix, s); + } + + uint32_t get_type(ValueId valueId) + { + auto & value = getValue(valueId); + state.forceValue(value, noPos); + auto t = value.type(); + return t == nInt ? 1 + : t == nFloat ? 2 + : t == nBool ? 3 + : t == nString ? 4 + : t == nPath ? 5 + : t == nNull ? 6 + : t == nAttrs ? 7 + : t == nList ? 8 + : t == nFunction ? 9 + : []() -> int { throw Error("unsupported type"); }(); + } + + ValueId make_int(int64_t n) + { + auto [valueId, value] = allocValue(); + value.mkInt(n); + return valueId; + } + + int64_t get_int(ValueId valueId) + { + return state.forceInt(getValue(valueId), noPos, "while evaluating a value from Wasm").value; + } + + ValueId make_float(double x) + { + auto [valueId, value] = allocValue(); + value.mkFloat(x); + return valueId; + } + + double get_float(ValueId valueId) + { + return state.forceFloat(getValue(valueId), noPos, "while evaluating a value from Wasm"); + } + + ValueId make_string(uint32_t ptr, uint32_t len) + { + auto [valueId, value] = allocValue(); + value.mkString(span2string(guestSpan(ptr, len)), state.mem); + return valueId; + } + + uint32_t copy_string(ValueId valueId, uint32_t ptr, uint32_t maxLen) + { + auto s = state.forceString(getValue(valueId), noPos, "while evaluating a value from Wasm"); + if (s.size() <= maxLen) { + auto buf = guestSpan(ptr, maxLen); + memcpy(buf.data(), s.data(), s.size()); + } + return s.size(); + } + + ValueId make_path(ValueId baseId, uint32_t ptr, uint32_t len) + { + auto & baseValue = getValue(baseId); + state.forceValue(baseValue, noPos); + if (baseValue.type() != nPath) + throw Error("make_path expects a path value"); + auto base = baseValue.path(); + + auto [valueId, value] = allocValue(); + value.mkPath({base.accessor, CanonPath(span2string(guestSpan(ptr, len)), base.path)}, state.mem); + return valueId; + } + + uint32_t copy_path(ValueId valueId, uint32_t ptr, uint32_t maxLen) + { + auto & v = getValue(valueId); + state.forceValue(v, noPos); + if (v.type() != nPath) + throw Error("copy_path expects a path value"); + auto path = v.path().path; + auto s = path.abs(); + if (s.size() <= maxLen) { + auto buf = guestSpan(ptr, maxLen); + memcpy(buf.data(), s.data(), s.size()); + } + return s.size(); + } + + ValueId make_bool(int32_t b) + { + return addValue(state.getBool(b)); + } + + int32_t get_bool(ValueId valueId) + { + return state.forceBool(getValue(valueId), noPos, "while evaluating a value from Wasm"); + } + + ValueId make_null() + { + return addValue(&Value::vNull); + } + + ValueId make_list(uint32_t ptr, uint32_t len) + { + auto vs = guestSpan(ptr, len); + + auto [valueId, value] = allocValue(); + + auto list = state.buildList(len); + for (const auto & [n, v] : enumerate(list)) + v = &getValue(vs[n]); // FIXME: endianness + value.mkList(list); + + return valueId; + } + + uint32_t copy_list(ValueId valueId, uint32_t ptr, uint32_t maxLen) + { + auto & value = getValue(valueId); + state.forceList(value, noPos, "while getting a list from Wasm"); + + if (value.listSize() <= maxLen) { + auto out = guestSpan(ptr, value.listSize()); + + for (const auto & [n, elem] : enumerate(value.listView())) + out[n] = addValue(elem); + } + + return value.listSize(); + } + + ValueId make_attrset(uint32_t ptr, uint32_t len) + { + struct Attr + { + // FIXME: endianness + uint32_t attrNamePtr; + uint32_t attrNameLen; + ValueId value; + }; + + auto attrs = guestSpan(ptr, len); + + auto [valueId, value] = allocValue(); + auto builder = state.buildBindings(len); + for (auto & attr : attrs) + builder.insert( + state.symbols.create(span2string(guestSpan(attr.attrNamePtr, attr.attrNameLen))), + &getValue(attr.value)); + value.mkAttrs(builder); + + return valueId; + } + + /** + * Deprecated: requires a `copy_attrname` call per attribute and does not return the attributes in + * lexicographically sorted order. Use `get_attrset`. + */ + uint32_t copy_attrset(ValueId valueId, uint32_t ptr, uint32_t maxLen) + { + auto & value = getValue(valueId); + state.forceAttrs(value, noPos, "while copying an attrset into Wasm"); + + if (value.attrs()->size() <= maxLen) { + // FIXME: endianness. + struct Attr + { + ValueId value; + uint32_t nameLen; + }; + + auto buf = guestSpan(ptr, maxLen); + + for (const auto & [n, attr] : enumerate(*value.attrs())) { + buf[n].value = addValue(attr.value); + buf[n].nameLen = state.symbols[attr.name].size(); + } + } + + return value.attrs()->size(); + } + + std::monostate copy_attrname(ValueId valueId, uint32_t attrIdx, uint32_t ptr, uint32_t len) + { + auto & value = getValue(valueId); + state.forceAttrs(value, noPos, "while copying an attr name into Wasm"); + + auto & attrs = *value.attrs(); + + if ((size_t) attrIdx >= attrs.size()) + throw Error("copy_attrname: attribute index out of bounds"); + + /* Note: `Bindings::operator[]` is not supported for layered + bindings (e.g. the result of `//`), so iterate instead. This + has to match the iteration order used by `copy_attrset`. + + Deprecated: since this function is called once per attribute, + copying an attrset is O(n^2) (and n host calls). Use + `get_attrset`. */ + std::string_view name = state.symbols[std::next(attrs.begin(), attrIdx)->name]; + + if ((size_t) len != name.size()) + throw Error("copy_attrname: buffer length does not match attribute name length"); + + memcpy(guestSpan(ptr, len).data(), name.data(), name.size()); + + return {}; + } + + /** + * Copy an attrset into Wasm memory in one go. The buffer consists of the number of attributes `n` (a `u32`), the + * `n` `ValueId`s of the attributes in lexicographically sorted order of their names, and the `n` attribute names + * in the same order, each terminated by a null byte. The buffer is aligned to 4 bytes. + * + * If this fits in the `len` bytes at `ptr`, that buffer is used; otherwise a buffer is allocated in the guest via + * `nix_wasm_alloc`. Returns the pointer of the buffer that was used in the low 32 bits and the number of bytes + * written in the high 32 bits (see `read_file_v2`). + */ + uint64_t get_attrset(ValueId valueId, uint32_t ptr, uint32_t len) + { + if (ptr % alignof(uint32_t)) + throw Error("get_attrset: buffer is not aligned to %d bytes", alignof(uint32_t)); + + auto & value = getValue(valueId); + state.forceAttrs(value, noPos, "while copying an attrset into Wasm"); + + auto attrs = value.attrs()->lexicographicOrder(state.symbols); + + size_t namesOffset = (1 + attrs.size()) * sizeof(uint32_t); + size_t size = namesOffset; + for (auto attr : attrs) + size += state.symbols[attr->name].size() + 1; + + if (size > std::numeric_limits::max()) + throw Error("attrset is too large to process in Wasm (size: %d)", size); + + auto bufPtr = size <= len ? ptr : allocInGuest(size, alignof(uint32_t)); + + // Note: the allocation may have grown the memory; `guestSpan` fetches it afresh. + auto buf = guestSpan(bufPtr, size); + + // FIXME: endianness. + auto ints = guestSpan(bufPtr, 1 + attrs.size()); + ints[0] = attrs.size(); + for (const auto & [n, attr] : enumerate(attrs)) + ints[n + 1] = addValue(attr->value); + + auto out = buf.data() + namesOffset; + for (auto attr : attrs) { + std::string_view name = state.symbols[attr->name]; + memcpy(out, name.data(), name.size()); + out += name.size(); + *out++ = 0; + } + + return ((uint64_t) size << 32) | bufPtr; + } + + ValueId get_attr(ValueId valueId, uint32_t ptr, uint32_t len) + { + auto attrName = span2string(guestSpan(ptr, len)); + + auto & value = getValue(valueId); + state.forceAttrs(value, noPos, "while getting an attribute from Wasm"); + + auto attr = value.attrs()->get(state.symbols.create(attrName)); + + return attr ? addValue(attr->value) : 0; + } + + ValueId call_function(ValueId funId, uint32_t ptr, uint32_t len) + { + auto & fun = getValue(funId); + state.forceFunction(fun, noPos, "while calling a function from Wasm"); + + ValueVector args; + for (auto argId : guestSpan(ptr, len)) + args.push_back(&getValue(argId)); + + auto [valueId, value] = allocValue(); + + state.callFunction(fun, args, value, noPos); + + return valueId; + } + + ValueId make_app(ValueId funId, uint32_t ptr, uint32_t len) + { + if (!len) + return funId; + + auto args = guestSpan(ptr, len); + + auto res = &getValue(funId); + + while (!args.empty()) { + auto arg = &getValue(args[0]); + auto tmp = state.allocValue(); + tmp->mkApp(res, arg); + res = tmp; + args = args.subspan(1); + } + + return addValue(res); + } + + /** + * Allocate `size` bytes with alignment `align` (a power of two) in the guest by calling its `nix_wasm_alloc` + * export. The guest is responsible for freeing the buffer (e.g. by reconstructing a `Vec` with that size and + * alignment from it). + */ + uint32_t allocInGuest(uint32_t size, uint32_t align) + { + if (!allocFn) + throw Error("Wasm module '%s' does not export 'nix_wasm_alloc'", pre->name); + auto res = unwrap(allocFn->call(wasmCtx, {(int32_t) size, (int32_t) align})); + if (res.size() != 1 || res[0].kind() != ValKind::I32) + throw Error("'nix_wasm_alloc' of Wasm module '%s' did not return an i32", pre->name); + auto ptr = (uint32_t) res[0].i32(); + if (ptr % align) + throw Error("'nix_wasm_alloc' of Wasm module '%s' returned a misaligned pointer", pre->name); + state.nrWasmGuestAllocs++; + state.wasmGuestAllocBytes += size; + return ptr; + } + + /** + * Read the contents of a file into Wasm memory. This is like calling `builtins.readFile`, except that it can handle + * binary files that cannot be represented as Nix strings. + * + * Deprecated: the guest has to guess the buffer size, so the file may be read twice. Use `read_file_v2`. + */ + uint32_t read_file(ValueId pathId, uint32_t ptr, uint32_t len) + { + auto & pathValue = getValue(pathId); + auto path = state.realisePath(noPos, pathValue); + + auto contents = path.readFile(); + + if (contents.size() > std::numeric_limits::max()) + throw Error("file '%s' is too large to process in Wasm (size: %d)", path, contents.size()); + + if (contents.size() <= len) { + auto buf = guestSpan(ptr, len); + memcpy(buf.data(), contents.data(), contents.size()); + } + + return contents.size(); + } + + /** + * Read the contents of a file into a buffer allocated in the guest via `nix_wasm_alloc`, so that the file is read + * and copied only once. Returns the buffer pointer in the low 32 bits and its length in the high 32 bits (Rust's + * C ABI cannot express Wasm multi-value returns, so pack both into one `u64`). + */ + uint64_t read_file_v2(ValueId pathId) + { + auto & pathValue = getValue(pathId); + auto path = state.realisePath(noPos, pathValue); + + auto contents = path.readFile(); + + if (contents.size() > std::numeric_limits::max()) + throw Error("file '%s' is too large to process in Wasm (size: %d)", path, contents.size()); + + auto ptr = allocInGuest(contents.size(), 1); + + // Note: the allocation may have grown the memory; `guestSpan` fetches it afresh. + memcpy(guestSpan(ptr, contents.size()).data(), contents.data(), contents.size()); + + return ((uint64_t) contents.size() << 32) | ptr; + } +}; + +template +static void regFun(Linker & linker, std::string_view name, R (NixWasmInstance::*f)(Args...)) +{ + unwrap(linker.func_wrap("env", name, [f](Caller caller, Args... args) -> Result { + try { + auto instance = std::any_cast(caller.context().get_data()); + return (*instance.*f)(args...); + } catch (std::exception & e) { + return Trap(e.what()); + } catch (...) { + return Trap("unknown exception"); + } + })); +} + +static void regFuns(Linker & linker, bool useWasi) +{ + regFun(linker, "panic", &NixWasmInstance::panic); + regFun(linker, "warn", &NixWasmInstance::warn); + regFun(linker, "get_type", &NixWasmInstance::get_type); + regFun(linker, "make_int", &NixWasmInstance::make_int); + regFun(linker, "get_int", &NixWasmInstance::get_int); + regFun(linker, "make_float", &NixWasmInstance::make_float); + regFun(linker, "get_float", &NixWasmInstance::get_float); + regFun(linker, "make_string", &NixWasmInstance::make_string); + regFun(linker, "copy_string", &NixWasmInstance::copy_string); + regFun(linker, "make_path", &NixWasmInstance::make_path); + regFun(linker, "copy_path", &NixWasmInstance::copy_path); + regFun(linker, "make_bool", &NixWasmInstance::make_bool); + regFun(linker, "get_bool", &NixWasmInstance::get_bool); + regFun(linker, "make_null", &NixWasmInstance::make_null); + regFun(linker, "make_list", &NixWasmInstance::make_list); + regFun(linker, "copy_list", &NixWasmInstance::copy_list); + regFun(linker, "make_attrset", &NixWasmInstance::make_attrset); + regFun(linker, "copy_attrset", &NixWasmInstance::copy_attrset); + regFun(linker, "copy_attrname", &NixWasmInstance::copy_attrname); + regFun(linker, "get_attrset", &NixWasmInstance::get_attrset); + regFun(linker, "get_attr", &NixWasmInstance::get_attr); + regFun(linker, "call_function", &NixWasmInstance::call_function); + regFun(linker, "make_app", &NixWasmInstance::make_app); + regFun(linker, "read_file", &NixWasmInstance::read_file); + regFun(linker, "read_file_v2", &NixWasmInstance::read_file_v2); + + if (useWasi) { + unwrap(linker.func_wrap( + "env", "return_to_nix", [](Caller caller, ValueId resultId) -> Result { + auto instance = std::any_cast(caller.context().get_data()); + instance->resultId = resultId; + return Trap("return_to_nix"); + })); + } +} + +template +struct LazyMakeRef +{ + ref p; + + template + LazyMakeRef(Args &&... args) + : p(make_ref(std::move(args...))) + { + } +}; + +static NixWasmInstance instantiateWasm(EvalState & state, const SourcePath & wasmPath) +{ + // FIXME: make this a weak Boehm GC pointer so that it can be freed during GC. + // FIXME: move to EvalState? + // Note: InstancePre in Rust is Send+Sync so it should be safe to share between threads. + static boost::concurrent_flat_map> instancesPre; + + std::shared_ptr instancePre; + + instancesPre.try_emplace_and_cvisit( + wasmPath, wasmPath, [&](auto & i) { instancePre = i.second.p; }, [&](auto & i) { instancePre = i.second.p; }); + + return NixWasmInstance{state, ref(instancePre)}; +} + +/** + * Callback for WASI stdout/stderr writes. It splits the output into lines and logs each line separately. + */ +struct WasiLogger +{ + NixWasmInstance & instance; + + std::string data; + + ~WasiLogger() + { + if (!data.empty()) + instance.doWarn(data); + } + + void operator()(std::string_view s) + { + data.append(s); + + while (true) { + auto pos = data.find('\n'); + if (pos == std::string_view::npos) + break; + instance.doWarn(data.substr(0, pos)); + data.erase(0, pos + 1); + } + } +}; + +static void prim_wasm(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + state.forceAttrs(*args[0], pos, "while evaluating the first argument to `builtins.wasm`"); + + // Check for unknown attributes + for (auto & attr : *args[0]->attrs()) { + auto name = state.symbols[attr.name]; + if (name != "path" && name != "wat" && name != "function") + throw Error("unknown attribute '%s' in first argument to `builtins.wasm`", name); + } + + auto pathAttr = args[0]->attrs()->get(state.symbols.create("path")); + auto watAttr = args[0]->attrs()->get(state.symbols.create("wat")); + + if (pathAttr && watAttr) + throw Error("'path' and 'wat' are mutually exclusive in first argument to `builtins.wasm`"); + if (!pathAttr && !watAttr) + throw Error("missing required 'path' or 'wat' attribute in first argument to `builtins.wasm`"); + + // Second argument is the value to pass to the function + auto argValue = args[1]; + + /* wasmtime keeps a per-thread stack of active Wasm calls, so the + host functions below (which may have to wait for values being + evaluated by other threads) must not suspend this fiber: another + fiber's Wasm call could then be interleaved on this thread, or + we could be resumed on another thread. */ + FiberNoSuspend noSuspend; + + try { + auto instance = pathAttr ? instantiateWasm(state, state.realisePath(pos, *pathAttr->value)) + : NixWasmInstance{ + state, + make_ref(state.forceStringNoCtx( + *watAttr->value, pos, "while evaluating the 'wat' attribute"))}; + + // Extract 'function' attribute (optional for wasi, required for non-wasi) + std::string functionName; + auto functionAttr = args[0]->attrs()->get(state.symbols.create("function")); + if (instance.pre->useWasi) { + functionName = "_start"; + if (functionAttr) + throw Error("'function' attribute is not allowed for WASI modules"); + } else { + if (!functionAttr) + throw Error( + "missing required 'function' attribute in first argument to `builtins.wasm` for non-WASI modules"); + functionName = std::string( + state.forceStringNoCtx(*functionAttr->value, pos, "while evaluating the 'function' attribute")); + } + + debug("calling wasm module"); + + auto argId = instance.addValue(argValue); + + if (instance.pre->useWasi) { + WasiLogger logger{instance}; + + auto loggerTrampoline = [](void * data, const unsigned char * buf, size_t len) -> ptrdiff_t { + auto logger = static_cast(data); + (*logger)(std::string_view((const char *) buf, len)); + return len; + }; + + WasiConfig wasiConfig; + wasi_config_set_stdout_custom(wasiConfig.capi(), loggerTrampoline, &logger, nullptr); + wasi_config_set_stderr_custom(wasiConfig.capi(), loggerTrampoline, &logger, nullptr); + wasiConfig.argv({"wasi", std::to_string(argId)}); + unwrap(instance.wasmStore.context().set_wasi(std::move(wasiConfig))); + + auto res = instance.getExport(functionName).call(instance.wasmCtx, {}); + if (!instance.resultId) { + unwrap(std::move(res)); + throw Error( + "Wasm function '%s' from '%s' finished without returning a value", + functionName, + instance.pre->name); + } + + auto & vRes = instance.getValue(instance.resultId); + state.forceValue(vRes, pos); + v = vRes; + } else { + // FIXME: use the "start" function if present. + instance.runFunction("nix_wasm_init_v1", {}); + + auto res = instance.runFunction(functionName, {(int32_t) argId}); + if (res.size() != 1) + throw Error( + "Wasm function '%s' from '%s' did not return exactly one value", functionName, instance.pre->name); + if (res[0].kind() != ValKind::I32) + throw Error( + "Wasm function '%s' from '%s' did not return an i32 value", functionName, instance.pre->name); + auto & vRes = instance.getValue(res[0].i32()); + state.forceValue(vRes, pos); + v = vRes; + } + } catch (Error & e) { + e.addTrace(state.positions[pos], "while executing a Wasm module"); + throw; + } +} + +static RegisterPrimOp primop_wasm( + {.name = "__wasm", + .args = {"config", "arg"}, + .doc = R"( + Call a Wasm function with the specified argument. + + The first argument must be an attribute set with the following attributes: + - `path`: Path to the Wasm module (mutually exclusive with `wat`) + - `wat`: WebAssembly Text format source as a string (mutually exclusive with `path`) + - `function`: Function name to call (required for non-WASI modules, not allowed for WASI modules) + + Exactly one of `path` or `wat` must be specified. + + The second argument is the value to pass to the function. + + WASI mode is automatically enabled if the module imports from `wasi_snapshot_preview1`. + + Example (non-WASI): + ```nix + builtins.wasm { + path = ./foo.wasm; + function = "fib"; + } 33 + ``` + + Example (reading from a WAT file): + ```nix + builtins.wasm { + wat = builtins.readFile ./fib.wat; + function = "fib"; + } 10 + ``` + + Example (WASI): + ```nix + builtins.wasm { + path = ./bar.wasm; + } { x = 42; } + ``` + )", + .impl = prim_wasm, + .experimentalFeature = Xp::WasmBuiltin}); + +} // namespace nix diff --git a/src/libexpr/print-ambiguous.cc b/src/libexpr/print-ambiguous.cc index b0a5224f26e6..1912262ee576 100644 --- a/src/libexpr/print-ambiguous.cc +++ b/src/libexpr/print-ambiguous.cc @@ -7,13 +7,7 @@ namespace nix { // See: https://github.com/NixOS/nix/issues/9730 -void printAmbiguous( - EvalState & state, - Value & v, - std::ostream & str, - std::set * seen, - NixStringContext * context, - size_t depth) +void printAmbiguous(EvalState & state, Value & v, std::ostream & str, std::set * seen, size_t depth) { checkInterrupt(); @@ -26,11 +20,13 @@ void printAmbiguous( case nBool: printLiteralBool(str, v.boolean()); break; - case nString: - printLiteralString(str, v.string_view()); - if (context) - copyContext(v, *context); + case nString: { + NixStringContext context; + copyContext(v, context); + // FIXME: make devirtualization configurable? + printLiteralString(str, state.devirtualize(v.string_view(), context)); break; + } case nPath: str << v.path().to_string(); // !!! escaping? break; @@ -44,7 +40,7 @@ void printAmbiguous( str << "{ "; for (auto & i : v.attrs()->lexicographicOrder(state.symbols)) { str << state.symbols[i->name] << " = "; - printAmbiguous(state, *i->value, str, seen, context, depth + 1); + printAmbiguous(state, *i->value, str, seen, depth + 1); str << "; "; } str << "}"; @@ -60,7 +56,7 @@ void printAmbiguous( str << "[ "; for (auto v2 : v.listView()) { if (v2) - printAmbiguous(state, *v2, str, seen, context, depth + 1); + printAmbiguous(state, *v2, str, seen, depth + 1); else str << "(nullptr)"; str << " "; @@ -84,7 +80,7 @@ void printAmbiguous( case nFailed: // Historically, a tried and then ignored value (e.g. through tryEval) was // reverted to the original thunk. - str << ""; + str << "«failed»"; break; case nFunction: if (v.isLambda()) { diff --git a/src/libexpr/print.cc b/src/libexpr/print.cc index 0c95ae60c5b6..57441fa379a7 100644 --- a/src/libexpr/print.cc +++ b/src/libexpr/print.cc @@ -162,7 +162,6 @@ class Printer std::ostream & output; EvalState & state; PrintOptions options; - NixStringContext * context; std::optional seen; size_t totalAttrsPrinted = 0; size_t totalListItemsPrinted = 0; @@ -250,7 +249,11 @@ class Printer void printString(Value & v) { - printLiteralString(output, v.string_view(), options.maxStringLength, options.ansiColors); + NixStringContext context; + copyContext(v, context); + std::ostringstream s; + printLiteralString(s, v.string_view(), options.maxStringLength, options.ansiColors); + output << state.devirtualize(s.str(), context); } void printPath(Value & v) @@ -499,7 +502,7 @@ class Printer output << "«potential infinite recursion»"; if (options.ansiColors) output << ANSI_NORMAL; - } else if (v.isThunk() || v.isApp()) { + } else if (!v.isFinished()) { if (options.ansiColors) output << ANSI_MAGENTA; output << "«thunk»"; @@ -516,7 +519,7 @@ class Printer output << ANSI_MAGENTA; // Historically, a tried and then ignored value (e.g. through tryEval) was // reverted to the original thunk. - output << "«thunk»"; + output << "«failed»"; if (options.ansiColors) output << ANSI_NORMAL; } @@ -578,12 +581,9 @@ class Printer printBool(v); break; - case nString: { + case nString: printString(v); - if (context) - copyContext(v, *context); break; - } case nPath: printPath(v); @@ -636,11 +636,10 @@ class Printer } public: - Printer(std::ostream & output, EvalState & state, PrintOptions options, NixStringContext * context) + Printer(std::ostream & output, EvalState & state, PrintOptions options) : output(output) , state(state) , options(options) - , context(context) { } @@ -661,14 +660,14 @@ class Printer } }; -void printValue(EvalState & state, std::ostream & output, Value & v, PrintOptions options, NixStringContext * context) +void printValue(EvalState & state, std::ostream & output, Value & v, PrintOptions options) { - Printer(output, state, options, context).print(v); + Printer(output, state, options).print(v); } std::ostream & operator<<(std::ostream & output, const ValuePrinter & printer) { - printValue(printer.state, output, printer.value, printer.options, printer.context); + printValue(printer.state, output, printer.value, printer.options); return output; } diff --git a/src/libexpr/provenance.cc b/src/libexpr/provenance.cc new file mode 100644 index 000000000000..8bce4f120763 --- /dev/null +++ b/src/libexpr/provenance.cc @@ -0,0 +1,25 @@ +#include "nix/expr/provenance.hh" +#include "nix/util/json-utils.hh" + +#include + +namespace nix { + +nlohmann::json DerivationProvenance::to_json() const +{ + return nlohmann::json{ + {"type", "derivation"}, + {"meta", *meta}, + {"next", next ? next->to_json() : nlohmann::json(nullptr)}, + }; +} + +Provenance::Register registerDerivationProvenance("derivation", [](nlohmann::json json) { + auto & obj = getObject(json); + std::shared_ptr next; + if (auto p = optionalValueAt(obj, "next"); p && !p->is_null()) + next = Provenance::from_json(*p); + return make_ref(next, make_ref(valueAt(obj, "meta"))); +}); + +} // namespace nix diff --git a/src/libexpr/root-value.cc b/src/libexpr/root-value.cc new file mode 100644 index 000000000000..d1ba0e3d5f2c --- /dev/null +++ b/src/libexpr/root-value.cc @@ -0,0 +1,82 @@ +#include "nix/expr/root-value.hh" +#include "nix/expr/eval-gc.hh" +#include "nix/util/sync.hh" + +namespace nix { + +#if NIX_USE_BOEHMGC + +namespace { +/** + * A root value slot: either in use (rooting a value) or on the + * freelist. Slots are carved out of uncollectable slabs, which are + * permanently part of the GC root set. This avoids doing a + * GC_MALLOC_UNCOLLECTABLE() / GC_FREE() pair per root value, which + * requires taking the global GC allocation lock — a significant + * source of contention during parallel evaluation. + * + * GC safety: the slabs are conservatively scanned. In-use slots + * contain a `Value *`, which roots the value. Free slots contain a + * pointer to the next free slot (or null), i.e. a pointer into a + * slab, which is scanned harmlessly since slabs are never freed + * anyway. + */ +union Slot +{ + Value * value; + Slot * nextFree; +}; + +static_assert(sizeof(Slot) == sizeof(Value *)); +} // namespace + +/* Head of the freelist of slots. Never destroyed since root values + held by statics may be released after us during shutdown. */ +static auto & freeSlots = *new Sync{nullptr}; + +#endif + +RootValue::RootValue(Value * v) +{ +#if NIX_USE_BOEHMGC + Slot * slot; + { + auto head(freeSlots.lock()); + if (!*head) { + constexpr size_t slabSize = 4096; + auto slab = (Slot *) GC_MALLOC_UNCOLLECTABLE(slabSize * sizeof(Slot)); + if (!slab) + outOfMemory(); + for (size_t i = 0; i + 1 < slabSize; ++i) + slab[i].nextFree = &slab[i + 1]; + slab[slabSize - 1].nextFree = nullptr; + *head = slab; + } + slot = *head; + *head = slot->nextFree; + } + + slot->value = v; + + this->slot = &slot->value; +#else + this->slot = new Value *(v); +#endif +} + +void RootValue::freeRootValueSlot() +{ +#if NIX_USE_BOEHMGC + /* Note: writing `nextFree` overwrites the `Value *`, so this also + stops the slot from keeping the value alive. */ + auto s = reinterpret_cast(slot); + auto head(freeSlots.lock()); + s->nextFree = *head; + *head = s; +#else + delete slot; +#endif + slot = nullptr; +} + +} // namespace nix diff --git a/src/libexpr/symbol-table.cc b/src/libexpr/symbol-table.cc new file mode 100644 index 000000000000..052c72570371 --- /dev/null +++ b/src/libexpr/symbol-table.cc @@ -0,0 +1,63 @@ +#include "nix/expr/symbol-table.hh" +#include "nix/util/logging.hh" + +#include + +namespace nix { + +#ifndef MAP_NORESERVE +# define MAP_NORESERVE 0 +#endif + +static void * allocateLazyMemory(size_t maxSize) +{ + auto p = mmap(nullptr, maxSize, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + if (p == MAP_FAILED) + throw SysError("allocating arena using mmap"); + return p; +} + +ContiguousArena::ContiguousArena(size_t maxSize) + : data((char *) allocateLazyMemory(maxSize)) + , maxSize(maxSize) +{ +} + +size_t ContiguousArena::allocate(size_t bytes) +{ + auto offset = size.fetch_add(bytes); + if (offset + bytes > maxSize) + throw Error("arena ran out of space"); + return offset; +} + +Symbol SymbolTable::create(std::string_view s) +{ + uint32_t idx; + + auto visit = [&](const SymbolStr & sym) { idx = ((const char *) sym.s) - arena.data; }; + + symbols.insert_and_visit(SymbolStr::Key{s, arena}, visit, visit); + + return Symbol(idx); +} + +SymbolStr::SymbolStr(const SymbolStr::Key & key) +{ + auto size = SymbolStr::computeSize(key.s); + + auto id = key.arena.allocate(size); + + auto v = (SymbolValue *) (const_cast(key.arena.data) + id); + + auto s = (StringData *) (v + 1); + s->size_ = key.s.size(); + std::memcpy(s->data_, key.s.data(), key.s.size()); + s->data_[key.s.size()] = '\0'; + + v->mkStringNoCopy(*s); + + this->s = v; +} + +} // namespace nix diff --git a/src/libexpr/value-to-json.cc b/src/libexpr/value-to-json.cc index c844cd143032..134417e165d6 100644 --- a/src/libexpr/value-to-json.cc +++ b/src/libexpr/value-to-json.cc @@ -2,106 +2,146 @@ #include "nix/expr/eval-inline.hh" #include "nix/store/store-api.hh" #include "nix/util/signals.hh" +#include "nix/expr/parallel-eval.hh" #include #include namespace nix { + using json = nlohmann::json; +#pragma GCC diagnostic ignored "-Wswitch-enum" + +static void parallelForceDeep(EvalState & state, Value & v, PosIdx pos) +{ + state.forceValue(v, pos); + + Executor::WorkItems work; + + switch (v.type()) { + + case nAttrs: { + NixStringContext context; + if (state.tryAttrsToString(pos, v, context, false, false)) + return; + if (v.attrs()->get(state.s.outPath)) + return; + for (auto & a : *v.attrs()) + state.addWork( + work, 0, [value(RootValue(a.value)), pos(a.pos), &state]() { parallelForceDeep(state, **value, pos); }); + break; + } + + default: + break; + } + + state.executor->spawn(std::move(work)); +} + // TODO: rename. It doesn't print. json printValueAsJSON( EvalState & state, bool strict, Value & v, const PosIdx pos, NixStringContext & context, bool copyToStore) { - checkInterrupt(); + if (strict && state.executor->enabled && !Executor::amWorkerThread) + parallelForceDeep(state, v, pos); - auto _level = state.addCallDepth(pos); + auto recurse = [&](this const auto & recurse, json & res, Value & v, PosIdx pos) -> void { + checkInterrupt(); - if (strict) - state.forceValue(v, pos); + auto _level = state.addCallDepth(pos); - json out; + if (strict) + state.forceValue(v, pos); - switch (v.type()) { + switch (v.type()) { - case nInt: - out = v.integer().value; - break; + case nInt: + res = v.integer().value; + break; - case nBool: - out = v.boolean(); - break; + case nBool: + res = v.boolean(); + break; - case nString: - copyContext(v, context); - out = v.string_view(); - break; + case nString: { + copyContext(v, context); + res = v.string_view(); + break; + } - case nPath: - if (copyToStore) - out = state.store->printStorePath(state.copyPathToStore(context, v.path())); - else - out = v.path().path.abs(); - break; + case nPath: + if (copyToStore) + res = state.store->printStorePath(state.copyPathToStore(context, v.path(), v.determinePos(pos))); + else + res = v.path().path.abs(); + break; - case nNull: - // already initialized as null - break; + case nNull: + // already initialized as null + break; - case nAttrs: { - auto maybeString = state.tryAttrsToString(pos, v, context, false, false); - if (maybeString) { - out = *maybeString; + case nAttrs: { + auto maybeString = state.tryAttrsToString(pos, v, context, false, false); + if (maybeString) { + res = *maybeString; + break; + } + if (auto i = v.attrs()->get(state.s.outPath)) + return recurse(res, *i->value, i->pos); + else { + res = json::object(); + for (auto & a : v.attrs()->lexicographicOrder(state.symbols)) { + json & j = res.emplace(state.symbols[a->name], json()).first.value(); + try { + recurse(j, *a->value, a->pos); + } catch (Error & e) { + e.addTrace( + state.positions[a->pos], + HintFmt("while evaluating attribute '%1%'", state.symbols[a->name])); + throw; + } + } + } break; } - if (auto i = v.attrs()->get(state.s.outPath)) - return printValueAsJSON(state, strict, *i->value, i->pos, context, copyToStore); - else { - out = json::object(); - for (auto & a : v.attrs()->lexicographicOrder(state.symbols)) { + + case nList: { + res = json::array(); + for (const auto & [i, elem] : enumerate(v.listView())) { try { - out.emplace( - state.symbols[a->name], - printValueAsJSON(state, strict, *a->value, a->pos, context, copyToStore)); + res.push_back(json()); + recurse(res.back(), *elem, pos); } catch (Error & e) { - e.addTrace( - state.positions[a->pos], HintFmt("while evaluating attribute '%1%'", state.symbols[a->name])); + e.addTrace(state.positions[pos], HintFmt("while evaluating list element at index %1%", i)); throw; } } + break; } - break; - } - case nList: { - out = json::array(); - int i = 0; - for (auto elem : v.listView()) { - try { - out.push_back(printValueAsJSON(state, strict, *elem, pos, context, copyToStore)); - } catch (Error & e) { - e.addTrace(state.positions[pos], HintFmt("while evaluating list element at index %1%", i)); - throw; - } - i++; + case nExternal: { + res = v.external()->printValueAsJSON(state, strict, context, copyToStore); + break; } - break; - } - case nExternal: - return v.external()->printValueAsJSON(state, strict, context, copyToStore); - break; + case nFloat: + res = v.fpoint(); + break; - case nFloat: - out = v.fpoint(); - break; + case nThunk: + case nFailed: + case nFunction: + state.error("cannot convert %1% to JSON", showType(v)).atPos(v.determinePos(pos)).debugThrow(); + } + }; - case nThunk: - case nFailed: - case nFunction: - state.error("cannot convert %1% to JSON", showType(v)).atPos(v.determinePos(pos)).debugThrow(); - } - return out; + json res; + + recurse(res, v, pos); + + return res; } void JSONSerializationError::anchor() {} diff --git a/src/libexpr/value.cc b/src/libexpr/value.cc index bfdee3cded9a..07d036b0dd4c 100644 --- a/src/libexpr/value.cc +++ b/src/libexpr/value.cc @@ -1,15 +1,7 @@ #include "nix/expr/value.hh" -#include "expr-config-private.hh" - -#if HAVE_LIBCPUID -# include -#endif - namespace nix { -Value::ExceptionRef::~ExceptionRef() {} - Value Value::vEmptyList = []() { Value res; res.setStorage(List{.size = 0, .elems = nullptr}); @@ -34,19 +26,4 @@ Value Value::vFalse = []() { return res; }(); -template<> -bool ValueStorage<8>::isAtomic() -{ -#if HAVE_LIBCPUID - struct cpu_id_t data; - - if (cpu_identify(NULL, &data) < 0) - return false; - - return data.flags[CPU_FEATURE_AVX]; -#else - return false; // Can't tell -#endif -} - } // namespace nix diff --git a/src/libexpr/value/context.cc b/src/libexpr/value/context.cc index c61e1681c473..13b346af11fc 100644 --- a/src/libexpr/value/context.cc +++ b/src/libexpr/value/context.cc @@ -51,6 +51,11 @@ NixStringContextElem NixStringContextElem::parse(std::string_view s0, const Expe .drvPath = StorePath{s.substr(1)}, }; } + case '@': { + return NixStringContextElem::Path{ + .storePath = StorePath{s.substr(1)}, + }; + } default: { // Ensure no '!' if (s.find("!") != std::string_view::npos) { @@ -91,6 +96,10 @@ std::string NixStringContextElem::to_string() const res += '='; res += d.drvPath.to_string(); }, + [&](const NixStringContextElem::Path & p) { + res += '@'; + res += p.storePath.to_string(); + }, }, raw); @@ -108,6 +117,9 @@ std::string NixStringContextElem::display(const StoreDirConfig & store) const return store.printStorePath(d.drvPath) + " (deep)"; }, [&](const NixStringContextElem::Built & b) -> std::string { return SingleDerivedPath{b}.to_string(store); }, + [&](const NixStringContextElem::Path & p) -> std::string { + return store.printStorePath(p.storePath) + " (untracked)"; + }, }, raw); } diff --git a/src/libfetchers-c/make.nix b/src/libfetchers-c/make.nix new file mode 100644 index 000000000000..a3b84b324070 --- /dev/null +++ b/src/libfetchers-c/make.nix @@ -0,0 +1,32 @@ +# libnixfetchersc, the C API of libnixfetchers, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-util-c, + nix-store, + nix-store-c, + nix-fetchers, +}: + +nixMake.mkComponent { + name = "determinate-nix-fetchers-c"; + libName = "nixfetchersc"; + + deps = [ + nix-util + nix-util-c + nix-store + nix-store-c + nix-fetchers + ]; + + root = ./.; + + includeDirs = [ "" ]; + # The headers live in the root, which the default would not export. + publicIncludeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libfetchers-c/meson.build b/src/libfetchers-c/meson.build index c55f4c8b1bda..44d0e7c31876 100644 --- a/src/libfetchers-c/meson.build +++ b/src/libfetchers-c/meson.build @@ -54,7 +54,6 @@ subdir('nix-meson-build-support/windows-version') build_both_libraries = get_option('plugin-c-api') library_kwargs = { - 'soversion' : nix_soversion, 'dependencies' : deps_public + deps_private + deps_other, 'include_directories' : include_dirs, 'link_args' : linker_export_flags, diff --git a/src/libfetchers-c/package.nix b/src/libfetchers-c/package.nix index f490c91e7cb2..39147228f2c0 100644 --- a/src/libfetchers-c/package.nix +++ b/src/libfetchers-c/package.nix @@ -18,7 +18,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-fetchers-c"; + pname = "determinate-nix-fetchers-c"; inherit version; workDir = ./.; diff --git a/src/libfetchers-tests/access-tokens.cc b/src/libfetchers-tests/access-tokens.cc index 5e92a9bc7be7..a482b70dcecf 100644 --- a/src/libfetchers-tests/access-tokens.cc +++ b/src/libfetchers-tests/access-tokens.cc @@ -13,10 +13,7 @@ class AccessKeysTest : public ::testing::Test protected: public: - void SetUp() override - { - experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes); - } + void SetUp() override {} void TearDown() override {} }; diff --git a/src/libfetchers-tests/make.nix b/src/libfetchers-tests/make.nix new file mode 100644 index 000000000000..6ae60646e15c --- /dev/null +++ b/src/libfetchers-tests/make.nix @@ -0,0 +1,47 @@ +# The unit tests of libnixfetchers, transcribed from meson.build and +# package.nix: the `nix-fetchers-tests` executable, with the derivation +# that runs it as `tests.run`. +{ + nixMake, + nix-store-test-support, + nix-fetchers, + nix-fetchers-c, +}: + +let + tests = nixMake.mkComponent { + name = "determinate-nix-fetchers-tests"; + type = "executable"; + exeName = "nix-fetchers-tests"; + + deps = [ + nix-store-test-support + nix-fetchers + nix-fetchers-c + ]; + + root = ./.; + + includeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + # The tests have no main() of their own; it comes from gtest_main + # (see src/libutil-tests/make.nix). + extraLinkLibs = [ "-lgtest_main" ]; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; + }; +in + +tests +// { + tests.run = nixMake.runTest { + name = "${tests.name}-run"; + command = [ "${tests}/bin/${tests.exeName}" ]; + env = { + _NIX_TEST_UNIT_DATA = ./data; + NIX_STORE = ""; + }; + }; +} diff --git a/src/libfetchers/attrs.cc b/src/libfetchers/attrs.cc index f3ecc3a87bc0..84c74df275df 100644 --- a/src/libfetchers/attrs.cc +++ b/src/libfetchers/attrs.cc @@ -38,6 +38,9 @@ nlohmann::json attrsToJSON(const Attrs & attrs) { nlohmann::json json; for (auto & attr : attrs) { + /* The __final attribute is purely internal, so never serialize it. */ + if (attr.first == "__final") + continue; auto resolved = forceAttr(attr.second); if (auto v = std::get_if(&resolved)) { json[attr.first] = *v; diff --git a/src/libfetchers/builtin-flake-registry.json b/src/libfetchers/builtin-flake-registry.json new file mode 100644 index 000000000000..8071e8ae7719 --- /dev/null +++ b/src/libfetchers/builtin-flake-registry.json @@ -0,0 +1,436 @@ +{ + "flakes": [ + { + "from": { + "id": "agda", + "type": "indirect" + }, + "to": { + "owner": "agda", + "repo": "agda", + "type": "github" + } + }, + { + "from": { + "id": "agenix", + "type": "indirect" + }, + "to": { + "owner": "ryantm", + "repo": "agenix", + "type": "github" + } + }, + { + "from": { + "id": "arion", + "type": "indirect" + }, + "to": { + "owner": "hercules-ci", + "repo": "arion", + "type": "github" + } + }, + { + "from": { + "id": "blender-bin", + "type": "indirect" + }, + "to": { + "dir": "blender", + "owner": "edolstra", + "repo": "nix-warez", + "type": "github" + } + }, + { + "from": { + "id": "bundlers", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "repo": "bundlers", + "type": "github" + } + }, + { + "from": { + "id": "cachix", + "type": "indirect" + }, + "to": { + "owner": "cachix", + "repo": "cachix", + "type": "github" + } + }, + { + "from": { + "id": "composable", + "type": "indirect" + }, + "to": { + "owner": "ComposableFi", + "repo": "composable", + "type": "github" + } + }, + { + "from": { + "id": "disko", + "type": "indirect" + }, + "to": { + "owner": "nix-community", + "repo": "disko", + "type": "github" + } + }, + { + "from": { + "id": "dreampkgs", + "type": "indirect" + }, + "to": { + "owner": "nix-community", + "repo": "dreampkgs", + "type": "github" + } + }, + { + "from": { + "id": "dwarffs", + "type": "indirect" + }, + "to": { + "owner": "edolstra", + "repo": "dwarffs", + "type": "github" + } + }, + { + "from": { + "id": "emacs-overlay", + "type": "indirect" + }, + "to": { + "owner": "nix-community", + "repo": "emacs-overlay", + "type": "github" + } + }, + { + "from": { + "id": "fenix", + "type": "indirect" + }, + "to": { + "owner": "nix-community", + "repo": "fenix", + "type": "github" + } + }, + { + "from": { + "id": "flake-parts", + "type": "indirect" + }, + "to": { + "owner": "hercules-ci", + "repo": "flake-parts", + "type": "github" + } + }, + { + "from": { + "id": "flake-utils", + "type": "indirect" + }, + "to": { + "owner": "numtide", + "repo": "flake-utils", + "type": "github" + } + }, + { + "from": { + "id": "helix", + "type": "indirect" + }, + "to": { + "owner": "helix-editor", + "repo": "helix", + "type": "github" + } + }, + { + "from": { + "id": "hercules-ci-agent", + "type": "indirect" + }, + "to": { + "owner": "hercules-ci", + "repo": "hercules-ci-agent", + "type": "github" + } + }, + { + "from": { + "id": "hercules-ci-effects", + "type": "indirect" + }, + "to": { + "owner": "hercules-ci", + "repo": "hercules-ci-effects", + "type": "github" + } + }, + { + "from": { + "id": "home-manager", + "type": "indirect" + }, + "to": { + "owner": "nix-community", + "repo": "home-manager", + "type": "github" + } + }, + { + "from": { + "id": "hydra", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "repo": "hydra", + "type": "github" + } + }, + { + "from": { + "id": "mach-nix", + "type": "indirect" + }, + "to": { + "owner": "DavHau", + "repo": "mach-nix", + "type": "github" + } + }, + { + "from": { + "id": "ngipkgs", + "type": "indirect" + }, + "to": { + "owner": "ngi-nix", + "repo": "ngipkgs", + "type": "github" + } + }, + { + "from": { + "id": "nickel", + "type": "indirect" + }, + "to": { + "owner": "tweag", + "repo": "nickel", + "type": "github" + } + }, + { + "from": { + "id": "nix", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "repo": "nix", + "type": "github" + } + }, + { + "from": { + "id": "nix-darwin", + "type": "indirect" + }, + "to": { + "owner": "nix-darwin", + "repo": "nix-darwin", + "type": "github" + } + }, + { + "from": { + "id": "nix-serve", + "type": "indirect" + }, + "to": { + "owner": "edolstra", + "repo": "nix-serve", + "type": "github" + } + }, + { + "from": { + "id": "nixops", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "repo": "nixops", + "type": "github" + } + }, + { + "from": { + "id": "nixos-anywhere", + "type": "indirect" + }, + "to": { + "owner": "nix-community", + "repo": "nixos-anywhere", + "type": "github" + } + }, + { + "from": { + "id": "nixos-hardware", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "repo": "nixos-hardware", + "type": "github" + } + }, + { + "from": { + "id": "nixos-homepage", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "repo": "nixos-homepage", + "type": "github" + } + }, + { + "from": { + "id": "nixos-search", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "repo": "nixos-search", + "type": "github" + } + }, + { + "exact": true, + "from": { + "id": "nixpkgs", + "type": "indirect" + }, + "to": { + "type": "tarball", + "url": "https://flakehub.com/f/DeterminateSystems/nixpkgs-weekly/0.1" + } + }, + { + "from": { + "id": "nixpkgs", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "ref": "nixpkgs-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + { + "from": { + "id": "nur", + "type": "indirect" + }, + "to": { + "owner": "nix-community", + "repo": "NUR", + "type": "github" + } + }, + { + "from": { + "id": "patchelf", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "repo": "patchelf", + "type": "github" + } + }, + { + "from": { + "id": "poetry2nix", + "type": "indirect" + }, + "to": { + "owner": "nix-community", + "repo": "poetry2nix", + "type": "github" + } + }, + { + "from": { + "id": "pridefetch", + "type": "indirect" + }, + "to": { + "owner": "SpyHoodle", + "repo": "pridefetch", + "type": "github" + } + }, + { + "from": { + "id": "sops-nix", + "type": "indirect" + }, + "to": { + "owner": "Mic92", + "repo": "sops-nix", + "type": "github" + } + }, + { + "from": { + "id": "systems", + "type": "indirect" + }, + "to": { + "owner": "nix-systems", + "repo": "default", + "type": "github" + } + }, + { + "from": { + "id": "templates", + "type": "indirect" + }, + "to": { + "owner": "NixOS", + "repo": "templates", + "type": "github" + } + } + ], + "version": 2 +} diff --git a/src/libfetchers/builtin.cc b/src/libfetchers/builtin.cc new file mode 100644 index 000000000000..ccd7cc845abf --- /dev/null +++ b/src/libfetchers/builtin.cc @@ -0,0 +1,53 @@ +#include "nix/store/builtins.hh" +#include "nix/store/parsed-derivations.hh" +#include "nix/fetchers/fetchers.hh" +#include "nix/fetchers/fetch-settings.hh" +#include "nix/util/archive.hh" +#include "nix/store/store-open.hh" + +#include + +namespace nix { + +static void builtinFetchTree(const BuiltinBuilderContext & ctx) +{ + experimentalFeatureSettings.require(Xp::BuildTimeFetchTree); + + auto out = get(ctx.drv.outputs, "out"); + if (!out) + throw Error("'builtin:fetch-tree' requires an 'out' output"); + + if (!(ctx.drv.type().isFixed() || ctx.drv.type().isImpure())) + throw Error("'builtin:fetch-tree' must be a fixed-output or impure derivation"); + + if (!ctx.drv.structuredAttrs) + throw Error("'builtin:fetch-tree' must have '__structuredAttrs = true'"); + + setenv("NIX_CACHE_HOME", ctx.tmpDirInSandbox.c_str(), 1); + + using namespace fetchers; + + fetchers::Settings myFetchSettings; + myFetchSettings.accessTokens = fetchSettings.accessTokens.get(); + + // FIXME: disable use of the git/tarball cache + + auto input = Input::fromAttrs(myFetchSettings, jsonToAttrs(ctx.drv.structuredAttrs->structuredAttrs.at("input"))); + + std::cerr << fmt("fetching '%s'...\n", input.to_string()); + + /* Functions like downloadFile() expect a store. We can't use the + real one since we're in a forked process. FIXME: use recursive + Nix's daemon so we can use the real store? */ + auto tmpStore = openStore(ctx.tmpDirInSandbox + "/nix"); + + auto [accessor, lockedInput] = input.getAccessor(myFetchSettings, *tmpStore); + + auto source = sinkToSource([&](Sink & sink) { accessor->dumpPath(CanonPath::root, sink); }); + + restorePath(ctx.outputs.at("out"), *source); +} + +static RegisterBuiltinBuilder registerUnpackChannel("fetch-tree", builtinFetchTree); + +} // namespace nix diff --git a/src/libfetchers/cache.cc b/src/libfetchers/cache.cc index 390180080145..0c24b1072a4d 100644 --- a/src/libfetchers/cache.cc +++ b/src/libfetchers/cache.cc @@ -121,7 +121,7 @@ struct CacheImpl : Cache upsert(key, value); } - std::optional lookupStorePath(Key key, Store & store) override + std::optional lookupStorePath(Key key, Store & store, bool allowInvalid) override { key.second.insert_or_assign("store", store.storeDir); @@ -135,7 +135,7 @@ struct CacheImpl : Cache ResultWithStorePath res2(*res, StorePath(storePathS)); store.addTempRoot(res2.storePath); - if (!store.isValidPath(res2.storePath)) { + if (!allowInvalid && !store.isValidPath(res2.storePath)) { // FIXME: we could try to substitute 'storePath'. debug( "ignoring disappeared cache entry '%s:%s' -> '%s'", @@ -157,7 +157,7 @@ struct CacheImpl : Cache std::optional lookupStorePathWithTTL(Key key, Store & store) override { - auto res = lookupStorePath(std::move(key), store); + auto res = lookupStorePath(std::move(key), store, false); return res && !res->expired ? res : std::nullopt; } }; diff --git a/src/libfetchers/fetch-settings.cc b/src/libfetchers/fetch-settings.cc index 8839a0fde4f0..568e805a0b57 100644 --- a/src/libfetchers/fetch-settings.cc +++ b/src/libfetchers/fetch-settings.cc @@ -1,4 +1,5 @@ #include "nix/fetchers/fetch-settings.hh" +#include "nix/util/config-global.hh" namespace nix::fetchers { @@ -7,3 +8,11 @@ Settings::Settings() {} void Settings::anchor() {} } // namespace nix::fetchers + +namespace nix { + +fetchers::Settings fetchSettings; + +static GlobalConfig::Register rFetchSettings(&fetchSettings); + +} // namespace nix diff --git a/src/libfetchers/fetch-to-store.cc b/src/libfetchers/fetch-to-store.cc index 085c97da7d01..c1f136880a78 100644 --- a/src/libfetchers/fetch-to-store.cc +++ b/src/libfetchers/fetch-to-store.cc @@ -76,7 +76,7 @@ std::pair fetchToStore2( if (mode != FetchMode::DryRun) store.addTempRoot(storePath); - if (mode == FetchMode::DryRun || store.isValidPath(storePath)) { + if (mode == FetchMode::DryRun || store.maybeQueryPathInfo(storePath)) { debug( "source path '%s' cache hit in '%s' (hash '%s')", path, @@ -89,7 +89,7 @@ std::pair fetchToStore2( } } else { static auto barf = getEnv("_NIX_TEST_BARF_ON_UNCACHEABLE").value_or("") == "1"; - if (barf && !filter) + if (barf && !filter && !(path.to_string().starts_with("/") || path.to_string().starts_with("«path:/"))) throw Error("source path '%s' is uncacheable (filter=%d)", path, (bool) filter); debug("source path '%s' is uncacheable", path); } @@ -97,7 +97,10 @@ std::pair fetchToStore2( Activity act( *logger, lvlChatty, - actUnknown, + mode == FetchMode::DryRun ? "HashSourcePath" : "CopySourcePath", + std::to_array>({ + {"nix.source.path", path.to_string()}, + }), fmt(mode == FetchMode::DryRun ? "hashing '%s'" : "copying '%s' to the store", path)); auto filter2 = filter ? *filter : defaultPathFilter; diff --git a/src/libfetchers/fetchers.cc b/src/libfetchers/fetchers.cc index 6d7266e09dbb..dde72d79e28a 100644 --- a/src/libfetchers/fetchers.cc +++ b/src/libfetchers/fetchers.cc @@ -5,7 +5,7 @@ #include "nix/fetchers/fetch-to-store.hh" #include "nix/util/json-utils.hh" #include "nix/fetchers/fetch-settings.hh" -#include "nix/fetchers/fetch-to-store.hh" +#include "nix/fetchers/provenance.hh" #include "nix/util/url.hh" #include "nix/util/users.hh" #include "nix/store/pathlocks.hh" @@ -13,6 +13,7 @@ #include #include +#include namespace nix::fetchers { @@ -104,7 +105,9 @@ Input Input::fromAttrs(const Settings & settings, Attrs && attrs) auto allowedAttrs = inputScheme->allowedAttrs(); for (auto & [name, _] : attrs) - if (name != "type" && name != "__final" && allowedAttrs.count(name) == 0) + /* `__final` and `__legacyExport` are purely internal attributes, so they're not listed in + `allowedAttrs()`. */ + if (name != "type" && name != "__final" && name != "__legacyExport" && allowedAttrs.count(name) == 0) throw Error("input attribute '%s' not supported by scheme '%s'", name, schemeName); auto res = inputScheme->inputFromAttrs(settings, attrs); @@ -130,24 +133,30 @@ std::optional Input::getFingerprint(Store & store) const return fingerprint; } -ParsedURL Input::toURL() const +ParsedURL Input::toURL(bool abbreviate) const { if (!scheme) throw Error("cannot show unsupported input '%s'", attrsToJSON(attrs)); - return scheme->toURL(*this); + + auto url = scheme->toURL(*this, abbreviate); + + if (abbreviate) + url.query.erase("narHash"); + + return url; } -std::string Input::toURLString(const StringMap & extraQuery) const +std::string Input::toURLString(const StringMap & extraQuery, bool abbreviate) const { - auto url = toURL(); + auto url = toURL(abbreviate); for (auto & attr : extraQuery) url.query.insert(attr); return url.to_string(); } -std::string Input::to_string() const +std::string Input::to_string(bool abbreviate) const { - return toURL().to_string(); + return toURL(abbreviate).to_string(); } bool Input::isDirect() const @@ -192,36 +201,30 @@ bool Input::contains(const Input & other) const return false; } -// FIXME: remove -std::pair Input::fetchToStore(const Settings & settings, Store & store) const +std::tuple, Input> Input::fetchToStore(const Settings & settings, Store & store) const { if (!scheme) throw Error("cannot fetch unsupported input '%s'", attrsToJSON(toAttrs())); - auto [storePath, input] = [&]() -> std::pair { - try { - auto [accessor, result] = getAccessorUnchecked(settings, store); - - auto storePath = - nix::fetchToStore(settings, store, SourcePath(accessor), FetchMode::Copy, result.getName()); + try { + auto [accessor, result] = getAccessorUnchecked(settings, store); - auto narHash = store.queryPathInfo(storePath)->narHash; - result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true)); + auto storePath = nix::fetchToStore(settings, store, SourcePath(accessor), FetchMode::Copy, result.getName()); - result.attrs.insert_or_assign("__final", Explicit(true)); + auto narHash = store.queryPathInfo(storePath)->narHash; + result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true)); - assert(result.isFinal()); + result.attrs.insert_or_assign("__final", Explicit(true)); - checkLocks(*this, result); + assert(result.isFinal()); - return {storePath, result}; - } catch (Error & e) { - e.addTrace({}, "while fetching the input '%s'", to_string()); - throw; - } - }(); + checkLocks(*this, result); - return {std::move(storePath), input}; + return {std::move(storePath), accessor, result}; + } catch (Error & e) { + e.addTrace({}, "while fetching the input '%s'", to_string()); + throw; + } } void Input::checkLocks(Input specified, Input & result) @@ -304,72 +307,96 @@ std::pair, Input> Input::getAccessorUnchecked(const Settings if (!scheme) throw Error("cannot fetch unsupported input '%s'", attrsToJSON(toAttrs())); - /* The tree may already be in the Nix store, or it could be - substituted (which is often faster than fetching from the - original source). So check that. We only do this for final - inputs, otherwise there is a risk that we don't return the - same attributes (like `lastModified`) that the "real" fetcher - would return. - - FIXME: add a setting to disable this. - FIXME: substituting may be slower than fetching normally, - e.g. for fetchers like Git that are incremental! - */ - if (isFinal() && getNarHash()) { - try { - auto storePath = computeStorePath(store); - - store.addTempRoot(storePath); + std::optional storePath; + if (isFinal() && getNarHash()) + storePath = computeStorePath(store); + + auto makeStoreAccessor = [&]() -> std::pair, Input> { + auto accessor = store.requireStoreObjectAccessor(*storePath); + + // FIXME: use the NAR hash for fingerprinting Git trees since it may have a .gitattributes file and we don't + // know if we used `git archive` or libgit2 to fetch it. + accessor->fingerprint = getType() == "git" ? std::optional(storePath->hashPart()) : getFingerprint(store); + cachedFingerprint = accessor->fingerprint; + + // Store a cache entry for the substituted tree so later fetches + // can reuse the existing nar instead of copying the unpacked + // input back into the store on every evaluation. + if (accessor->fingerprint) { + settings.getCache()->upsert( + makeSourcePathToHashCacheKey( + *accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, CanonPath::root), + {{"hash", store.queryPathInfo(*storePath)->narHash.to_string(HashFormat::SRI, true)}}); + } - store.ensurePath(storePath); + accessor->provenance = std::make_shared(*this); - debug("using substituted/cached input '%s' in '%s'", to_string(), store.printStorePath(storePath)); + // FIXME: ideally we would use the `showPath()` of the + // "real" accessor for this fetcher type. + accessor->setPathDisplay("«" + to_string(true) + "»"); - auto accessor = store.requireStoreObjectAccessor(storePath); + return {accessor, *this}; + }; - accessor->fingerprint = getFingerprint(store); + /* If a tree with the expected hash is already in the Nix store, + reuse it. We only do this for final inputs, since otherwise + there is a risk that we don't return the same attributes (like + `lastModified`) that the "real" fetcher would return. */ + if (storePath) { + store.addTempRoot(*storePath); + if (store.isValidPath(*storePath)) { + debug("using input '%s' in '%s'", to_string(), store.printStorePath(*storePath)); + return makeStoreAccessor(); + } + } - // Store a cache entry for the substituted tree so later fetches - // can reuse the existing nar instead of copying the unpacked - // input back into the store on every evaluation. - if (accessor->fingerprint) { - settings.getCache()->upsert( - makeSourcePathToHashCacheKey( - *accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, CanonPath::root), - {{"hash", store.queryPathInfo(storePath)->narHash.to_string(HashFormat::SRI, true)}}); - } + auto fixupAccessor = [&](ref accessor, Input result) -> std::pair, Input> { + if (auto fp = accessor->getFingerprint(CanonPath::root).second) + result.cachedFingerprint = *fp; + else + accessor->fingerprint = result.getFingerprint(store); - accessor->setPathDisplay("«" + to_string() + "»"); + accessor->provenance = std::make_shared(result); - return {accessor, *this}; - } catch (Error & e) { - debug("substitution of input '%s' failed: %s", to_string(), e.what()); - } - } + return {accessor, result}; + }; /* Acquire a path lock on this input. Note that fetching the same input in parallel is supposed to be safe (it's up - * to the fetchers to guarantee this), so this is merely intended to avoid work duplication. Note that we don't need - * this when substituting the input. */ + * to the fetchers to guarantee this), so this is merely intended to avoid work duplication. */ auto lockFilePath = getCacheDir() / "fetcher-locks" / hashString(HashAlgorithm::SHA256, attrsToJSON(toAttrs()).dump()).to_string(HashFormat::Base16, false); - createDirs(lockFilePath.parent_path()); + std::filesystem::create_directories(lockFilePath.parent_path()); PathLocks lock( {lockFilePath.string()}, fmt("waiting for another Nix process to finish fetching input '%s'...", to_string())); - lock.setDeletion(true); static auto inTest = getEnv("_NIX_TEST_CONCURRENT_FETCHES") == "1"; if (inTest) std::this_thread::sleep_for(std::chrono::seconds(1)); - auto [accessor, result] = scheme->getAccessor(settings, store, *this); + /* See if the input is in the cache of the fetcher. */ + try { + if (auto res = scheme->getAccessor(settings, store, *this, true)) + return fixupAccessor(res->first, std::move(res->second)); + } catch (...) { + } - if (!accessor->fingerprint) - accessor->fingerprint = result.getFingerprint(store); - else - result.cachedFingerprint = accessor->fingerprint; + /* If not, try to substitute the input. */ + if (storePath) { + try { + store.ensurePath(*storePath); + return makeStoreAccessor(); + } + // Ignore any substitution error. + catch (Error & e2) { + debug("substitution of input '%s' failed: %s", to_string(), e2.info().msg); + } catch (...) { + } + } - return {accessor, std::move(result)}; + /* If we can't substitute, then fetch normally. */ + auto [accessor, result] = scheme->getAccessor(settings, store, *this); + return fixupAccessor(accessor, result); } Input Input::applyOverrides(std::optional ref, std::optional rev) const @@ -471,7 +498,7 @@ std::optional Input::getLastModified() const return {}; } -ParsedURL InputScheme::toURL(const Input & input) const +ParsedURL InputScheme::toURL(const Input & input, bool abbreviate) const { throw Error("don't know how to convert input '%s' to a URL", attrsToJSON(input.attrs)); } diff --git a/src/libfetchers/filtering-source-accessor.cc b/src/libfetchers/filtering-source-accessor.cc index ceb6457d473d..0b809e7824cb 100644 --- a/src/libfetchers/filtering-source-accessor.cc +++ b/src/libfetchers/filtering-source-accessor.cc @@ -1,6 +1,8 @@ #include "nix/fetchers/filtering-source-accessor.hh" #include "nix/util/sync.hh" +#include "nix/util/util.hh" +#include #include namespace nix { @@ -62,6 +64,13 @@ std::pair> FilteringSourceAccessor::getFin return next->getFingerprint(prefix / path); } +std::shared_ptr FilteringSourceAccessor::getProvenance(const CanonPath & path) +{ + if (provenance) + return SourceAccessor::getProvenance(path); + return next->getProvenance(prefix / path); +} + void FilteringSourceAccessor::checkAccess(const CanonPath & path) { if (!isAllowed(path)) @@ -108,13 +117,19 @@ ref AllowListSourceAccessor::create( return make_ref(next, allowedPrefixes, allowedPaths, std::move(makeNotAllowedError)); } +CachingFilteringSourceAccessor::CachingFilteringSourceAccessor( + const SourcePath & src, MakeNotAllowedError && makeNotAllowedError) + : FilteringSourceAccessor(src, std::move(makeNotAllowedError)) + , cache(make_ref>()) +{ +} + bool CachingFilteringSourceAccessor::isAllowed(const CanonPath & path) { - auto i = cache.find(path); - if (i != cache.end()) - return i->second; + if (auto allowed = getConcurrent(*cache, path)) + return *allowed; auto res = isAllowedUncached(path); - cache.emplace(path, res); + cache->emplace(path, res); return res; } diff --git a/src/libfetchers/git-utils.cc b/src/libfetchers/git-utils.cc index bf5f1b24bcd5..8fb02c79fcec 100644 --- a/src/libfetchers/git-utils.cc +++ b/src/libfetchers/git-utils.cc @@ -13,13 +13,16 @@ #include "nix/util/util.hh" #include "nix/util/thread-pool.hh" #include "nix/util/pool.hh" +#include "nix/util/executable-path.hh" #include "nix/util/deleter.hh" +#include #include #include #include #include #include +#include #include #include #include @@ -114,6 +117,11 @@ typedef std::unique_ptr> typedef std::unique_ptr> StatusList; typedef std::unique_ptr> Remote; typedef std::unique_ptr> GitConfig; +typedef std::unique_ptrfree(backend); + })> + GitConfigBackend; typedef std::unique_ptr> ConfigIterator; typedef std::unique_ptr> ObjectDb; typedef std::unique_ptr> PackBuilder; @@ -122,10 +130,22 @@ typedef std::unique_ptr> Index; static Hash toHash(const git_oid & oid) { -#ifdef GIT_EXPERIMENTAL_SHA256 - assert(oid.type == GIT_OID_SHA1); + HashAlgorithm algo; +#if LIBGIT2_VERSION_CHECK(2, 0, 0) + switch (oid.type) { + case GIT_OID_SHA1: + algo = HashAlgorithm::SHA1; + break; + case GIT_OID_SHA256: + algo = HashAlgorithm::SHA256; + break; + default: + unreachable(); + } +#else + algo = HashAlgorithm::SHA1; #endif - Hash hash(HashAlgorithm::SHA1); + Hash hash(algo); memcpy(hash.hash, oid.id, hash.hashSize); return hash; } @@ -136,14 +156,41 @@ static void initLibGit2() std::call_once(initialized, []() { if (git_libgit2_init() < 0) throw GitError("initialising libgit2"); + + /* Nuke the "hashing on all reads" behavior, since that can lead to bad + performance https://github.com/libgit2/libgit2/issues/4951. It's a + compromise of course, but one that is mostly in line with git cli and + like how we don't recalculate narHash when reading from a store. */ + git_libgit2_opts(GIT_OPT_ENABLE_STRICT_HASH_VERIFICATION, 0); }); } static git_oid hashToOID(const Hash & hash) { git_oid oid; +#if LIBGIT2_VERSION_CHECK(2, 0, 0) + git_oid_t t; +# pragma GCC diagnostic push +# pragma GCC diagnostic ignored "-Wswitch-enum" + switch (hash.algo) { + case HashAlgorithm::SHA1: + t = GIT_OID_SHA1; + break; + case HashAlgorithm::SHA256: + t = GIT_OID_SHA256; + break; + default: + throw Error("unsupported hash algorithm for Git: %s", printHashAlgo(hash.algo)); + } +# pragma GCC diagnostic pop + if (git_oid_from_raw(&oid, hash.hash, t)) + /* This can really never happen, since libgit2 just reads out our raw bytes. + The only failure mode is us specifying an invalid `type` parameter. */ + unreachable(); +#else if (git_oid_fromstr(&oid, hash.gitRev().c_str())) throw GitError("cannot convert '%s' to a Git OID", hash.gitRev()); +#endif return oid; } @@ -296,6 +343,28 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this if (git_repository_open(Setter(repo), path.string().c_str())) throw GitError("opening Git repository %s", PathFmt(path)); + GitConfig config; + if (git_repository_config(Setter(config), *this)) + throw GitError("getting Git repository config"); + + /* Create an in-memory configuration so that we can set config options without modifying the + config file on-disk. */ + git_config_backend_memory_options configOpts = GIT_CONFIG_BACKEND_MEMORY_OPTIONS_INIT; + configOpts.backend_type = "nix"; + + std::vector configValues; + if (options.dontFindDeltas) + configValues.push_back("pack.window=0"); + + GitConfigBackend memBackend; + if (git_config_backend_from_values(Setter(memBackend), configValues.data(), configValues.size(), &configOpts)) + throw GitError("creating an in-memory Git config"); + + if (git_config_add_backend(config.get(), memBackend.get(), GIT_CONFIG_LEVEL_APP, *this, /*force=*/false)) + throw GitError("adding the in-memory Git configuration backend"); + + memBackend.release(); + ObjectDb odb; if (options.packfilesOnly) { /* Create a fresh object database because by default the repo also @@ -309,7 +378,16 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this if (git_odb_new(Setter(odb))) throw GitError("creating Git object database"); - if (git_odb_backend_pack(&packBackend, (path / "objects").string().c_str())) +#if LIBGIT2_VERSION_CHECK(2, 0, 0) + git_odb_backend_pack_options packOpts = GIT_ODB_OPTIONS_INIT; +#endif + if (git_odb_backend_pack( + &packBackend, + (path / "objects").string().c_str() +#if LIBGIT2_VERSION_CHECK(2, 0, 0) + , &packOpts // NOFORMAT +#endif + )) throw GitError("creating pack backend"); if (git_odb_add_backend(odb.get(), packBackend, 1)) @@ -367,7 +445,20 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this // (synchronously on the git_packbuilder_write_buf thread) Indexer indexer; git_indexer_progress stats; - if (git_indexer_new(Setter(indexer), pack_dir_path.c_str(), 0, nullptr, nullptr)) +#if LIBGIT2_VERSION_CHECK(2, 0, 0) + git_indexer_options indexerOpts = GIT_INDEXER_OPTIONS_INIT; +#endif + if (git_indexer_new( + Setter(indexer), + pack_dir_path.c_str(), +#if LIBGIT2_VERSION_CHECK(2, 0, 0) + &indexerOpts +#else + 0, + nullptr, + nullptr +#endif + )) throw GitError("creating git packfile indexer"); // TODO: provide index callback for checkInterrupt() termination @@ -432,7 +523,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this ThreadPool pool; - auto process = [&done, &pool, &repoPool](this const auto & process, const git_oid & oid) -> void { + auto process = [&done, &pool, &repoPool](this auto const & process, const git_oid & oid) -> void { auto repo(repoPool.get()); auto _commit = lookupObject(*repo, oid, GIT_OBJECT_COMMIT); @@ -659,33 +750,46 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this // that) // then use code that was removed in this commit (see blame) - auto dir = this->path; + if (ExecutablePath::load().findName("git")) { + auto dir = this->path; - // Remove shallow.lock left behind by a previously interrupted `git fetch`, as it would prevent `git fetch` - // from running. Note that we already have a repository-wide `PathLock` (see git.cc), so this is safe. - tryUnlink(dir / "shallow.lock"); + // Remove shallow.lock left behind by a previously interrupted `git fetch`, as it would prevent `git fetch` + // from running. Note that we already have a repository-wide `PathLock` (see git.cc), so this is safe. + tryUnlink(dir / "shallow.lock"); - OsStrings gitArgs = { - OS_STR("-C"), - dir.native(), - OS_STR("--git-dir"), - OS_STR("."), - OS_STR("fetch"), - OS_STR("--progress"), - OS_STR("--force"), - }; - if (shallow) { - gitArgs.push_back(OS_STR("--depth")); - gitArgs.push_back(OS_STR("1")); - } - gitArgs.push_back(OS_STR("--")); - gitArgs.push_back(string_to_os_string(url)); - gitArgs.push_back(string_to_os_string(refspec)); + OsStrings gitArgs{"-C", dir.native(), "--git-dir", ".", "fetch", "--progress", "--force"}; + if (shallow) { + gitArgs.push_back(OS_STR("--depth")); + gitArgs.push_back(OS_STR("1")); + } + gitArgs.push_back(OS_STR("--")); + gitArgs.push_back(string_to_os_string(url)); + gitArgs.push_back(string_to_os_string(refspec)); + + auto status = runProgram(RunOptions{.program = "git", .args = gitArgs, .isInteractive = true}).first; + + if (status > 0) + throw Error("Failed to fetch git repository '%s'", url); + } else { + // Fall back to using libgit2 for fetching. This does not + // support SSH very well. + Remote remote; + + if (git_remote_create_anonymous(Setter(remote), *this, url.c_str())) + throw Error("cannot create Git remote '%s': %s", url, git_error_last()->message); - auto status = runProgram({.program = "git", .args = gitArgs, .isInteractive = true}).first; + char * refspecs[] = {(char *) refspec.c_str()}; + git_strarray refspecs2{.strings = refspecs, .count = 1}; - if (status > 0) - throw Error("Failed to fetch git repository '%s'", url); + git_fetch_options opts = GIT_FETCH_OPTIONS_INIT; + // FIXME: for some reason, shallow fetching over ssh barfs + // with "could not read from remote repository". + opts.depth = shallow && parseURL(url).scheme != "ssh" ? 1 : GIT_FETCH_DEPTH_FULL; + opts.callbacks.payload = &act; + + if (git_remote_fetch(remote.get(), &refspecs2, &opts, nullptr)) + throw Error("fetching '%s' from '%s': %s", refspec, url, git_error_last()->message); + } } void verifyCommit(const Hash & rev, const std::vector & publicKeys) override @@ -801,10 +905,14 @@ ref GitRepo::openRepo(const std::filesystem::path & path, GitRepo::Opti return make_ref(path, options); } +std::string GitAccessorOptions::makeFingerprint(const Hash & rev) const +{ + return "git:" + rev.gitRev() + (exportIgnore ? ";e" : "") + (smudgeLfs ? ";l" : "") + (legacy ? ";legacy" : ""); +} + /** * Raw git tree input accessor. */ - struct GitSourceAccessor : SourceAccessor { private: @@ -828,6 +936,7 @@ struct GitSourceAccessor : SourceAccessor .options = options, }} { + fingerprint = options.makeFingerprint(rev); } void readBlob(const CanonPath & path, bool symlink, Sink & sink, std::function sizeCallback) @@ -987,7 +1096,7 @@ struct GitSourceAccessor : SourceAccessor auto path2 = *parent; path2.push(entryName); - lookupCache.emplace(path2, std::move(copy)).first->second.get(); + lookupCache.emplace(path2, std::move(copy)); } return res; @@ -1017,7 +1126,7 @@ struct GitSourceAccessor : SourceAccessor { auto entry = lookup(state, path); if (!entry) - throw Error("'%s' does not exist", showPath(path)); + throw FileNotFound("path '%s' does not exist", showPath(path)); return entry; } @@ -1532,7 +1641,19 @@ ref Settings::getTarballCache() const * for optimal packfiles. */ static auto repoDir = std::filesystem::path(getCacheDir()) / "tarball-cache-v2"; - return GitRepo::openRepo(repoDir, {.create = true, .bare = true, .packfilesOnly = true}); + auto tarballCache(_tarballCache.lock()); + if (!*tarballCache) + *tarballCache = GitRepo::openRepo( + repoDir, + { + .create = true, + .bare = true, + .packfilesOnly = true, + /* Tarball unpacking is not expected to benefit from deltas much, + compared to how much CPU times it takes to find. */ + .dontFindDeltas = true, + }); + return ref(*tarballCache); } } // namespace fetchers diff --git a/src/libfetchers/git.cc b/src/libfetchers/git.cc index cab5bfae9219..297959fea113 100644 --- a/src/libfetchers/git.cc +++ b/src/libfetchers/git.cc @@ -3,6 +3,7 @@ #include "nix/fetchers/fetchers.hh" #include "nix/util/users.hh" #include "nix/fetchers/cache.hh" +#include "nix/util/tarfile.hh" #include "nix/store/store-api.hh" #include "nix/store/pathlocks.hh" #include "nix/util/os-string.hh" @@ -15,6 +16,7 @@ #include "nix/util/archive.hh" #include "nix/util/memo.hh" #include "nix/util/mounted-source-accessor.hh" +#include "nix/fetchers/fetch-to-store.hh" #include @@ -404,15 +406,17 @@ struct GitInputScheme : InputScheme return input; } - ParsedURL toURL(const Input & input) const override + ParsedURL toURL(const Input & input, bool abbreviate) const override { auto url = parseURL(getStrAttr(input.attrs, "url")); if (url.scheme != "git") url.scheme = "git+" + url.scheme; if (auto rev = input.getRev()) url.query.insert_or_assign("rev", rev->gitRev()); - if (auto ref = input.getRef()) - url.query.insert_or_assign("ref", *ref); + if (auto ref = input.getRef()) { + if (!abbreviate || (*ref != "master" && *ref != "main")) + url.query.insert_or_assign("ref", *ref); + } if (getShallowAttr(input)) url.query.insert_or_assign("shallow", "1"); if (getLfsAttr(input)) @@ -579,10 +583,10 @@ struct GitInputScheme : InputScheme { if (workdirInfo.isDirty) { if (!settings.allowDirty) - throw Error("Git tree '%s' is dirty", locationToArg()); + throw Error("Git tree '%s' has uncommitted changes", locationToArg()); if (settings.warnDirty) - warn("Git tree '%s' is dirty", locationToArg()); + warn("Git tree '%s' has uncommitted changes", locationToArg()); } } @@ -614,6 +618,19 @@ struct GitInputScheme : InputScheme return maybeGetBoolAttr(input.attrs, "allRefs").value_or(false); } + /** + * Whether to export this input using Nix < 2.20 semantics. This is a purely internal + * attribute: it is set only on the submodule inputs synthesized by + * `getAccessorFromCommit()`, so that the export of a repository composes the export of the + * top-level repo with exports of its submodules using the *same* semantics. If it's absent, + * we're a top-level input and the `nix-219-compat` setting decides. It is deliberately not + * part of `allowedAttrs()`. + */ + std::optional getLegacyExportAttr(const Input & input) const + { + return maybeGetBoolAttr(input.attrs, "__legacyExport"); + } + RepoInfo getRepoInfo(const Input & input) const { auto checkHashAlgorithm = [&](const std::optional & hash) { @@ -812,15 +829,126 @@ struct GitInputScheme : InputScheme } } - std::pair, Input> - getAccessorFromCommit(const Settings & settings, Store & store, RepoInfo & repoInfo, Input && input) const + /** + * Decide whether we can do a shallow clone, which is faster. This is possible if the user explicitly specified + * `shallow = true`, or if we already have a `revCount`. + */ + bool canDoShallow(const Input & input) const + { + bool shallow = getShallowAttr(input); + return shallow || input.getRevCount().has_value(); + } + + GitAccessorOptions getGitAccessorOptions(const Input & input) const + { + return GitAccessorOptions{ + .exportIgnore = getExportIgnoreAttr(input), + .smudgeLfs = getLfsAttr(input), + .submodules = getSubmodulesAttr(input), + .legacy = getLegacyExportAttr(input).value_or(false), + }; + } + + /** + * Get a `SourceAccessor` for the given Git revision using Nix < 2.20 semantics, i.e. using `git archive` or `git + * checkout`. + */ + ref getLegacyGitAccessor( + const Settings & settings, + Store & store, + RepoInfo & repoInfo, + const std::filesystem::path & repoDir, + const Hash & rev, + GitAccessorOptions & options) const + { + if (!options.submodules) + options.exportIgnore = true; + + options.legacy = true; + + auto fingerprint = options.makeFingerprint(rev); + + auto cacheKey = + makeSourcePathToHashCacheKey(fingerprint, ContentAddressMethod::Raw::NixArchive, CanonPath::root); + + auto makeAccessor = [&](const auto & storePath) -> ref { + auto accessor = store.getFSAccessor(storePath); + accessor->fingerprint = fingerprint; + return ref{accessor}; + }; + + if (auto res = settings.getCache()->lookup(cacheKey)) { + auto hash = Hash::parseSRI(fetchers::getStrAttr(*res, "hash")); + auto storePath = store.makeFixedOutputPathFromCA( + "source", ContentAddressWithReferences::fromParts(ContentAddressMethod::Raw::NixArchive, hash, {})); + store.addTempRoot(storePath); + if (store.maybeQueryPathInfo(storePath)) { + debug("using cached legacy export of revision '%s'", rev.gitRev()); + return makeAccessor(storePath); + } + } + + debug("doing legacy export of revision '%s'", rev.gitRev()); + + auto tmpDir = createTempDir(); + AutoDelete delTmpDir(tmpDir, true); + + auto storePath = + options.submodules + ? [&]() { + // Nix < 2.20 used `git checkout` for repos with submodules. + StringSink sink; // don't pollute stdout + runProgram2({.program = "git", .args = {"init", tmpDir, "-b", "master"}, .standardOut = &sink}); + runProgram2( + {.program = "git", + .args = {"-C", tmpDir, "remote", "add", "origin", repoDir}, + .standardOut = &sink}); + runProgram2( + {.program = "git", + .args = {"-C", tmpDir, "fetch", "--quiet", "origin", rev.gitRev()}, + .standardOut = &sink}); + runProgram2( + {.program = "git", + .args = {"-C", tmpDir, "checkout", "--quiet", rev.gitRev()}, + .standardOut = &sink}); + PathFilter filter = [&](const std::string & path) { return baseNameOf(path) != ".git"; }; + return store.addToStore( + "source", + {getFSSourceAccessor(), CanonPath(tmpDir.string())}, + ContentAddressMethod::Raw::NixArchive, + HashAlgorithm::SHA256, + {}, + filter); + }() + : [&]() { + // Nix < 2.20 used `git archive` for repos without submodules. + auto source = sinkToSource([&](Sink & sink) { + runProgram2( + {.program = "git", + .args = {"-C", repoDir, "--git-dir", repoInfo.gitDir, "archive", rev.gitRev()}, + .standardOut = &sink}); + }); + + unpackTarfile(*source, tmpDir); + + return store.addToStore("source", {getFSSourceAccessor(), CanonPath(tmpDir.string())}); + }(); + + settings.getCache()->upsert( + cacheKey, {{"hash", store.queryPathInfo(storePath)->narHash.to_string(HashFormat::SRI, true)}}); + + return makeAccessor(storePath); + } + + std::optional, Input>> getAccessorFromCommit( + const Settings & settings, Store & store, RepoInfo & repoInfo, Input && input, bool fastOnly) const { assert(!repoInfo.workdirInfo.isDirty); auto origRev = input.getRev(); auto originalRef = input.getRef(); - bool shallow = getShallowAttr(input); + bool shallow = canDoShallow(input); auto ref = originalRef ? *originalRef : getDefaultRef(settings, repoInfo, shallow); input.attrs.insert_or_assign("ref", ref); @@ -831,11 +959,27 @@ struct GitInputScheme : InputScheme if (!input.getRev()) input.attrs.insert_or_assign("rev", GitRepo::openRepo(repoDir, {})->resolveRef(ref).gitRev()); } else { + auto rev = input.getRev(); auto repoUrl = std::get(repoInfo.location); std::filesystem::path cacheDir = getCachePath(repoUrl.to_string(), shallow); repoDir = cacheDir; repoInfo.gitDir = "."; + /* If shallow = false, but we have a non-shallow repo that already contains the desired rev, then use that + * repo instead. */ + std::filesystem::path cacheDirNonShallow = getCachePath(repoUrl.to_string(), false); + if (rev && shallow && pathExists(cacheDirNonShallow)) { + auto nonShallowRepo = GitRepo::openRepo(cacheDirNonShallow, {.create = true, .bare = true}); + if (nonShallowRepo->hasObject(*rev)) { + debug( + "using non-shallow cached repo for '%s' since it contains rev '%s'", + repoUrl.to_string(), + rev->gitRev()); + repoDir = cacheDirNonShallow; + goto have_rev; + } + } + createDirs(cacheDir.parent_path()); PathLocks cacheDirLock({cacheDir.string()}); @@ -851,7 +995,7 @@ struct GitInputScheme : InputScheme /* If a rev was specified, we need to fetch if it's not in the repo. */ - if (auto rev = input.getRev()) { + if (rev) { doFetch = !repo->hasObject(*rev); } else { if (getAllRefsAttr(input)) { @@ -865,10 +1009,19 @@ struct GitInputScheme : InputScheme } if (doFetch) { - bool shallow = getShallowAttr(input); + if (fastOnly) + return std::nullopt; + try { - auto fetchRef = getAllRefsAttr(input) ? "refs/*:refs/*" - : input.getRev() ? input.getRev()->gitRev() + static constexpr std::string_view reservedRefNamespace = + "__nix_internal_fetchers_48ae34d6b380c5ec__"; + /* Fetch into a bucketed ref instead of leaving the rev + un-refed, so a later fetch of a nearby rev can + negotiate against it instead of re-downloading. */ + auto rev = input.getRev(); + auto revStr = rev ? rev->gitRev() : ""; + auto fetchRef = getAllRefsAttr(input) ? "refs/*:refs/*" + : rev ? fmt("%s:refs/%s/tip-%s", revStr, reservedRefNamespace, revStr.substr(0, 2)) : ref.compare(0, 5, "refs/") == 0 ? fmt("%1%:%1%", ref) : ref == "HEAD" ? "HEAD:HEAD" : fmt("%1%:%1%", "refs/heads/" + ref); @@ -893,7 +1046,7 @@ struct GitInputScheme : InputScheme warn("could not update cached head '%s' for '%s'", ref, repoInfo.locationToArg()); } - if (auto rev = input.getRev()) { + if (rev) { if (!repo->hasObject(*rev)) throw Error( "Cannot find Git revision '%s' in ref '%s' of repository '%s'! " @@ -910,6 +1063,7 @@ struct GitInputScheme : InputScheme // the remainder } + have_rev: auto repo = GitRepo::openRepo(repoDir, {}); // FIXME: check whether rev is an ancestor of ref? @@ -922,69 +1076,131 @@ struct GitInputScheme : InputScheme if (!input.attrs.contains("lastModified")) input.attrs.insert_or_assign("lastModified", getLastModified(settings, repoInfo, repoDir, rev)); - if (!getShallowAttr(input)) { - /* Like lastModified, skip revCount if supplied by the caller. */ - if (!input.attrs.contains("revCount")) - input.attrs.insert_or_assign("revCount", lazyRevCount(settings, repoInfo, repoDir, rev)); + /* Like lastModified, skip revCount if supplied by the caller. */ + if (!shallow && !input.attrs.contains("revCount")) { + auto isShallow = repo->isShallow(); + + if (isShallow && !shallow) + throw Error( + "'%s' is a shallow Git repository, but shallow repositories are only allowed when `shallow = true;` is specified", + repoInfo.locationToArg()); + + input.attrs.insert_or_assign("revCount", lazyRevCount(settings, repoInfo, repoDir, rev)); } printTalkative("using revision %s of repo '%s'", rev.gitRev(), repoInfo.locationToArg()); verifyCommit(input, repo); - bool exportIgnore = getExportIgnoreAttr(input); - bool smudgeLfs = getLfsAttr(input); - auto accessor = repo->getAccessor( - rev, {.exportIgnore = exportIgnore, .smudgeLfs = smudgeLfs}, "«" + input.to_string() + "»"); - - /* If the repo has submodules, fetch them and return a mounted - input accessor consisting of the accessor for the top-level - repo and the accessors for the submodules. */ - if (getSubmodulesAttr(input)) { - std::map> mounts; - - for (auto & [submodule, submoduleRev] : repo->getSubmodules(rev, exportIgnore)) { - auto resolved = repo->resolveSubmoduleUrl(submodule.url); - debug( - "Git submodule %s: %s %s %s -> %s", - submodule.path, - submodule.url, - submodule.branch, - submoduleRev.gitRev(), - resolved); - fetchers::Attrs attrs; - attrs.insert_or_assign("type", "git"); - attrs.insert_or_assign("url", resolved); - if (submodule.branch != "") { - // A special value of . is used to indicate that the name of the branch in the submodule - // should be the same name as the current branch in the current repository. - // https://git-scm.com/docs/gitmodules - if (submodule.branch == ".") { - attrs.insert_or_assign("ref", ref); - } else { - attrs.insert_or_assign("ref", submodule.branch); + auto options = getGitAccessorOptions(input); + + auto expectedNarHash = input.getNarHash(); + + /* Return an accessor for the complete tree denoted by `rev`, that is, the top-level repo + with any submodules mounted into it. If `legacy` is set, Nix < 2.20 semantics are used + (i.e. `git archive` / `git checkout`, which apply Git filters, `export-ignore` and + `export-subst`); the submodules are then exported using those semantics as well. + + Note that submodules must be mounted before the NAR hash of the tree can be compared + against `expectedNarHash`, since that hash covers the submodule contents as well. */ + auto getTree = [&](bool legacy) -> nix::ref { + auto options2 = options; + options2.legacy = legacy; + + auto accessor = legacy ? getLegacyGitAccessor(settings, store, repoInfo, repoDir, rev, options2) + : repo->getAccessor(rev, options2, "«" + input.to_string(true) + "»"); + + /* If the repo has submodules, fetch them and return a mounted + input accessor consisting of the accessor for the top-level + repo and the accessors for the submodules. */ + if (options2.submodules) { + std::map> mounts; + + for (auto & [submodule, submoduleRev] : repo->getSubmodules(rev, options2.exportIgnore)) { + auto resolved = repo->resolveSubmoduleUrl(submodule.url); + debug( + "Git submodule %s: %s %s %s -> %s", + submodule.path, + submodule.url, + submodule.branch, + submoduleRev.gitRev(), + resolved); + fetchers::Attrs attrs; + attrs.insert_or_assign("type", "git"); + attrs.insert_or_assign("url", resolved); + if (submodule.branch != "") { + // A special value of . is used to indicate that the name of the branch in the submodule + // should be the same name as the current branch in the current repository. + // https://git-scm.com/docs/gitmodules + if (submodule.branch == ".") { + attrs.insert_or_assign("ref", ref); + } else { + attrs.insert_or_assign("ref", submodule.branch); + } } + attrs.insert_or_assign("rev", submoduleRev.gitRev()); + attrs.insert_or_assign("exportIgnore", Explicit{options2.exportIgnore}); + attrs.insert_or_assign("submodules", Explicit{true}); + attrs.insert_or_assign("lfs", Explicit{options2.smudgeLfs}); + attrs.insert_or_assign("allRefs", Explicit{true}); + /* Export the submodule using the same semantics as the top-level repo, + regardless of the `nix-219-compat` setting. */ + attrs.insert_or_assign("__legacyExport", Explicit{options2.legacy}); + auto submoduleInput = fetchers::Input::fromAttrs(settings, std::move(attrs)); + auto [submoduleAccessor, submoduleInput2] = submoduleInput.getAccessor(settings, store); + submoduleAccessor->setPathDisplay("«" + submoduleInput.to_string(true) + "»"); + mounts.insert_or_assign(submodule.path, submoduleAccessor); + } + + if (!mounts.empty()) { + auto newFingerprint = accessor->getFingerprint(CanonPath::root).second->append(";s"); + mounts.insert_or_assign(CanonPath::root, accessor); + auto mounted = makeMountedSourceAccessor(std::move(mounts)); + mounted->fingerprint = newFingerprint; + return mounted; } - attrs.insert_or_assign("rev", submoduleRev.gitRev()); - attrs.insert_or_assign("exportIgnore", Explicit{exportIgnore}); - attrs.insert_or_assign("submodules", Explicit{true}); - attrs.insert_or_assign("lfs", Explicit{smudgeLfs}); - attrs.insert_or_assign("allRefs", Explicit{true}); - auto submoduleInput = fetchers::Input::fromAttrs(settings, std::move(attrs)); - auto [submoduleAccessor, submoduleInput2] = submoduleInput.getAccessor(settings, store); - submoduleAccessor->setPathDisplay("«" + submoduleInput.to_string() + "»"); - mounts.insert_or_assign(submodule.path, submoduleAccessor); } - if (!mounts.empty()) { - mounts.insert_or_assign(CanonPath::root, accessor); - accessor = makeMountedSourceAccessor(std::move(mounts)); + return accessor; + }; + + /* Use Nix 2.19 semantics if requested. If we're a submodule, follow whatever the top-level + repo is being exported with instead of the setting. */ + auto legacyExport = getLegacyExportAttr(input); + bool useLegacy = legacyExport.value_or(settings.nix219Compat) && !options.smudgeLfs; + + if (useLegacy && !legacyExport) + warn("Using Nix 2.19 semantics to export Git repository '%s'.", input.to_string()); + + auto accessor = getTree(useLegacy); + + /* Backward compatibility hack: a lock may have been produced by a Nix version that used the + other export semantics. In particular, locks produced by Nix < 2.20 depend on Nix applying + Git filters, `export-ignore` or `export-subst`, while Nix >= 2.20 doesn't do those. So if + we get a NAR hash mismatch, try again using the other semantics. */ + if (expectedNarHash) { + auto narHash = fetchToStore2(settings, store, {accessor}, FetchMode::DryRun, input.getName()).second; + if (*expectedNarHash != narHash) { + auto accessorOther = getTree(!useLegacy); + auto narHashOther = + fetchToStore2(settings, store, {accessorOther}, FetchMode::DryRun, input.getName()).second; + if (*expectedNarHash == narHashOther) { + if (!useLegacy) + warn( + "Git input '%s' specifies a NAR hash '%s' that was created by Nix < 2.20.\n" + "Nix >= 2.20 does not apply Git filters, `export-ignore` and `export-subst` by default, which changes the NAR hash.\n" + "Please update the NAR hash to '%s'.", + input.to_string(), + expectedNarHash->to_string(HashFormat::SRI, true), + narHash.to_string(HashFormat::SRI, true)); + accessor = accessorOther; + } } } assert(!origRev || origRev == rev); - return {accessor, std::move(input)}; + return {{accessor, std::move(input)}}; } std::pair, Input> @@ -1028,7 +1244,7 @@ struct GitInputScheme : InputScheme auto submoduleInput = fetchers::Input::fromAttrs(settings, std::move(attrs)); auto [submoduleAccessor, submoduleInput2] = submoduleInput.getAccessor(settings, store); - submoduleAccessor->setPathDisplay("«" + submoduleInput.to_string() + "»"); + submoduleAccessor->setPathDisplay("«" + submoduleInput.to_string(true) + "»"); /* If the submodule is dirty, mark this repo dirty as well. */ @@ -1080,8 +1296,8 @@ struct GitInputScheme : InputScheme return {accessor, std::move(input)}; } - std::pair, Input> - getAccessor(const Settings & settings, Store & store, const Input & _input) const override + std::optional, Input>> + getAccessor(const Settings & settings, Store & store, const Input & _input, bool fastOnly) const override { Input input(_input); @@ -1096,22 +1312,19 @@ struct GitInputScheme : InputScheme throw UnimplementedError("exportIgnore and submodules are not supported together yet"); } - auto [accessor, final] = input.getRef() || input.getRev() || !repoInfo.getPath() - ? getAccessorFromCommit(settings, store, repoInfo, std::move(input)) - : getAccessorFromWorkdir(settings, store, repoInfo, std::move(input)); - - return {accessor, std::move(final)}; + return input.getRef() || input.getRev() || !repoInfo.getPath() + ? getAccessorFromCommit(settings, store, repoInfo, std::move(input), fastOnly) + : std::optional{getAccessorFromWorkdir(settings, store, repoInfo, std::move(input))}; } std::optional getFingerprint(Store & store, const Input & input) const override { - auto makeFingerprint = [&](const Hash & rev) { - return rev.gitRev() + (getSubmodulesAttr(input) ? ";s" : "") + (getExportIgnoreAttr(input) ? ";e" : "") - + (getLfsAttr(input) ? ";l" : ""); - }; + auto options = getGitAccessorOptions(input); if (auto rev = input.getRev()) - return makeFingerprint(*rev); + // FIXME: this can return a wrong fingerprint for the legacy (`git archive`) case, since we don't know here + // whether to append the `;legacy` suffix or not. + return options.makeFingerprint(*rev); else { auto repoInfo = getRepoInfo(input); if (auto repoPath = repoInfo.getPath(); repoPath && repoInfo.workdirInfo.submodules.empty()) { @@ -1127,7 +1340,7 @@ struct GitInputScheme : InputScheme writeString("deleted:", hashSink); writeString(file.abs(), hashSink); } - return makeFingerprint(repoInfo.workdirInfo.headRev.value_or(nullRev)) + return options.makeFingerprint(repoInfo.workdirInfo.headRev.value_or(nullRev)) + ";d=" + hashSink.finish().hash.to_string(HashFormat::Base16, false); } return std::nullopt; diff --git a/src/libfetchers/github.cc b/src/libfetchers/github.cc index 855475d7fcb0..930aeb050b5e 100644 --- a/src/libfetchers/github.cc +++ b/src/libfetchers/github.cc @@ -150,7 +150,7 @@ struct GitArchiveInputScheme : InputScheme return input; } - ParsedURL toURL(const Input & input) const override + ParsedURL toURL(const Input & input, bool abbreviate) const override { auto owner = getStrAttr(input.attrs, "owner"); auto repo = getStrAttr(input.attrs, "repo"); @@ -161,7 +161,7 @@ struct GitArchiveInputScheme : InputScheme if (ref) path.push_back(*ref); if (rev) - path.push_back(rev->to_string(HashFormat::Base16, false)); + path.push_back(abbreviate ? rev->gitShortRev() : rev->gitRev()); auto url = ParsedURL{ .scheme = std::string{schemeName()}, .path = path, @@ -259,7 +259,8 @@ struct GitArchiveInputScheme : InputScheme time_t lastModified; }; - std::pair downloadArchive(const Settings & settings, Store & store, Input input) const + std::optional> + downloadArchive(const Settings & settings, Store & store, Input input, bool fastOnly) const { if (!maybeGetStrAttr(input.attrs, "ref")) input.attrs.insert_or_assign("ref", "HEAD"); @@ -287,12 +288,16 @@ struct GitArchiveInputScheme : InputScheme auto treeHash = getRevAttr(*treeHashAttrs, "treeHash"); auto lastModified = getIntAttr(*lastModifiedAttrs, "lastModified"); if (settings.getTarballCache()->hasObject(treeHash)) - return {std::move(input), TarballInfo{.treeHash = treeHash, .lastModified = (time_t) lastModified}}; + return { + {std::move(input), TarballInfo{.treeHash = treeHash, .lastModified = (time_t) lastModified}}}; else debug("Git tree with hash '%s' has disappeared from the cache, refetching...", treeHash.gitRev()); } } + if (fastOnly) + return std::nullopt; + /* Stream the tarball into the tarball cache. */ auto url = getDownloadUrl(settings, input); @@ -328,13 +333,17 @@ struct GitArchiveInputScheme : InputScheme rev->gitRev(), input.to_string(), upstreamTreeHash->gitRev(), tarballInfo.treeHash.gitRev()); #endif - return {std::move(input), tarballInfo}; + return {{std::move(input), tarballInfo}}; } - std::pair, Input> - getAccessor(const Settings & settings, Store & store, const Input & _input) const override + std::optional, Input>> + getAccessor(const Settings & settings, Store & store, const Input & _input, bool fastOnly) const override { - auto [input, tarballInfo] = downloadArchive(settings, store, _input); + auto res = downloadArchive(settings, store, _input, fastOnly); + if (fastOnly && !res) + return std::nullopt; + assert(res); + auto [input, tarballInfo] = *res; #if 0 input.attrs.insert_or_assign("treeHash", tarballInfo.treeHash.gitRev()); @@ -342,9 +351,16 @@ struct GitArchiveInputScheme : InputScheme input.attrs.insert_or_assign("lastModified", uint64_t(tarballInfo.lastModified)); auto accessor = - settings.getTarballCache()->getAccessor(tarballInfo.treeHash, {}, "«" + input.to_string() + "»"); + settings.getTarballCache()->getAccessor(tarballInfo.treeHash, {}, "«" + input.to_string(true) + "»"); + + if (!settings.trustTarballsFromGitForges) + // FIXME: computing the NAR hash here is wasteful if + // copyInputToStore() is just going to hash/copy it as + // well. + input.attrs.insert_or_assign( + "narHash", accessor->hashPath(CanonPath::root).to_string(HashFormat::SRI, true)); - return {accessor, input}; + return {{accessor, input}}; } bool isLocked(const Settings & settings, const Input & input) const override @@ -356,15 +372,10 @@ struct GitArchiveInputScheme : InputScheme return input.getRev().has_value() && (settings.trustTarballsFromGitForges || input.getNarHash().has_value()); } - std::optional experimentalFeature() const override - { - return Xp::Flakes; - } - std::optional getFingerprint(Store & store, const Input & input) const override { if (auto rev = input.getRev()) - return rev->gitRev(); + return "github:" + rev->gitRev(); else return std::nullopt; } @@ -442,8 +453,7 @@ struct GitHubInputScheme : GitArchiveInputScheme : headers.empty() ? "https://%s/%s/%s/archive/%s.tar.gz" : "https://api.%s/repos/%s/%s/tarball/%s"; - const auto url = - fmt(urlFmt, host, getOwner(input), getRepo(input), input.getRev()->to_string(HashFormat::Base16, false)); + const auto url = fmt(urlFmt, host, getOwner(input), getRepo(input), input.getRev()->gitRev()); return DownloadUrl{parseURL(url), headers}; } @@ -530,7 +540,7 @@ struct GitLabInputScheme : GitArchiveInputScheme host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo"), - input.getRev()->to_string(HashFormat::Base16, false)); + input.getRev()->gitRev()); Headers headers = makeHeadersWithAuthTokens(settings, host, input); return DownloadUrl{parseURL(url), headers}; @@ -626,7 +636,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo"), - input.getRev()->to_string(HashFormat::Base16, false)); + input.getRev()->gitRev()); Headers headers = makeHeadersWithAuthTokens(settings, host, input); return DownloadUrl{parseURL(url), headers}; diff --git a/src/libfetchers/include/nix/fetchers/cache.hh b/src/libfetchers/include/nix/fetchers/cache.hh index df2aa8b7de4d..fd21d7f384d3 100644 --- a/src/libfetchers/include/nix/fetchers/cache.hh +++ b/src/libfetchers/include/nix/fetchers/cache.hh @@ -72,9 +72,9 @@ public: /** * Look up a store path in the cache. The returned store path will - * be valid, but it may be expired. + * be valid (unless `allowInvalid` is true), but it may be expired. */ - virtual std::optional lookupStorePath(Key key, Store & store) = 0; + virtual std::optional lookupStorePath(Key key, Store & store, bool allowInvalid = false) = 0; /** * Look up a store path in the cache. Return nothing if its TTL diff --git a/src/libfetchers/include/nix/fetchers/fetch-settings.hh b/src/libfetchers/include/nix/fetchers/fetch-settings.hh index bb9a67d068d4..b68e2c0316f6 100644 --- a/src/libfetchers/include/nix/fetchers/fetch-settings.hh +++ b/src/libfetchers/include/nix/fetchers/fetch-settings.hh @@ -95,10 +95,7 @@ struct Settings : public Config are subsequently modified. Therefore lock files with dirty locks should generally only be used for local testing, and should not be pushed to other users. - )", - {}, - true, - Xp::Flakes}; + )"}; Setting trustTarballsFromGitForges{ this, @@ -119,16 +116,23 @@ struct Settings : public Config Setting flakeRegistry{ this, - "https://channels.nixos.org/flake-registry.json", + "https://install.determinate.systems/flake-registry/stable/flake-registry.json", "flake-registry", R"( Path or URI of the global flake registry. When empty, disables the global flake registry. - )", - {}, - true, - Xp::Flakes}; + )"}; + + Setting nix219Compat{ + this, + false, + "nix-219-compat", + R"( + If enabled, Nix will generate lock files that are compatible with Nix 2.19. + In particular, Nix will use `git archive` rather than `libgit2` to copy Git inputs. + The resulting locks may not be compatible with Nix >= 2.20. + )"}; Setting tarballTtl{ this, @@ -166,6 +170,17 @@ private: void anchor() override; mutable Sync> _cache; + + mutable Sync> _tarballCache; }; } // namespace nix::fetchers + +namespace nix { + +/** + * @todo Get rid of global setttings variables + */ +extern fetchers::Settings fetchSettings; + +} // namespace nix diff --git a/src/libfetchers/include/nix/fetchers/fetchers.hh b/src/libfetchers/include/nix/fetchers/fetchers.hh index 180d10e9dbbf..d830d83c840a 100644 --- a/src/libfetchers/include/nix/fetchers/fetchers.hh +++ b/src/libfetchers/include/nix/fetchers/fetchers.hh @@ -61,11 +61,11 @@ public: */ static Input fromAttrs(const Settings & settings, Attrs && attrs); - ParsedURL toURL() const; + ParsedURL toURL(bool abbreviate = false) const; - std::string toURLString(const StringMap & extraQuery = {}) const; + std::string toURLString(const StringMap & extraQuery = {}, bool abbreviate = false) const; - std::string to_string() const; + std::string to_string(bool abbreviate = false) const; Attrs toAttrs() const; @@ -113,7 +113,7 @@ public: * Fetch the entire input into the Nix store, returning the * location in the Nix store and the locked input. */ - std::pair fetchToStore(const Settings & settings, Store & store) const; + std::tuple, Input> fetchToStore(const Settings & settings, Store & store) const; /** * Check the locking attributes in `result` against @@ -225,7 +225,7 @@ struct InputScheme */ virtual const std::map & allowedAttrs() const = 0; - virtual ParsedURL toURL(const Input & input) const; + virtual ParsedURL toURL(const Input & input, bool abbreviate = false) const; virtual Input applyOverrides(const Input & input, std::optional ref, std::optional rev) const; @@ -240,8 +240,19 @@ struct InputScheme std::string_view contents, std::optional commitMsg) const; + virtual std::optional, Input>> + getAccessor(const Settings & settings, Store & store, const Input & input, bool fastOnly) const + { + if (fastOnly) + return std::nullopt; + return getAccessor(settings, store, input); + } + virtual std::pair, Input> - getAccessor(const Settings & settings, Store & store, const Input & input) const = 0; + getAccessor(const Settings & settings, Store & store, const Input & input) const + { + return getAccessor(settings, store, input, false).value(); + } /** * Is this `InputScheme` part of an experimental feature? diff --git a/src/libfetchers/include/nix/fetchers/filtering-source-accessor.hh b/src/libfetchers/include/nix/fetchers/filtering-source-accessor.hh index c5c1ce282b30..e77ae0dfed67 100644 --- a/src/libfetchers/include/nix/fetchers/filtering-source-accessor.hh +++ b/src/libfetchers/include/nix/fetchers/filtering-source-accessor.hh @@ -5,6 +5,8 @@ #include #include +#include + namespace nix { /** @@ -53,6 +55,8 @@ struct FilteringSourceAccessor : SourceAccessor std::pair> getFingerprint(const CanonPath & path) override; + std::shared_ptr getProvenance(const CanonPath & path) override; + void invalidateCache() override { next->invalidateCache(); @@ -95,9 +99,9 @@ struct AllowListSourceAccessor : public FilteringSourceAccessor */ struct CachingFilteringSourceAccessor : FilteringSourceAccessor { - std::map cache; + const ref> cache; - using FilteringSourceAccessor::FilteringSourceAccessor; + CachingFilteringSourceAccessor(const SourcePath & src, MakeNotAllowedError && makeNotAllowedError); bool isAllowed(const CanonPath & path) override; diff --git a/src/libfetchers/include/nix/fetchers/git-utils.hh b/src/libfetchers/include/nix/fetchers/git-utils.hh index f34f3bfde1b9..dc97cb6f9d15 100644 --- a/src/libfetchers/include/nix/fetchers/git-utils.hh +++ b/src/libfetchers/include/nix/fetchers/git-utils.hh @@ -30,6 +30,16 @@ struct GitAccessorOptions { bool exportIgnore = false; bool smudgeLfs = false; + bool submodules = false; // Currently implemented in GitInputScheme rather than GitAccessor + + /** + * Whether to export the repository using Nix < 2.20 semantics, i.e. using `git archive` or + * `git checkout` (which apply Git filters, `export-ignore` and `export-subst`) rather than + * libgit2. Currently implemented in GitInputScheme rather than GitAccessor. + */ + bool legacy = false; + + std::string makeFingerprint(const Hash & rev) const; }; struct GitRepo @@ -41,6 +51,12 @@ struct GitRepo bool create = false; bool bare = false; bool packfilesOnly = false; + /** + * Whether to avoid finding deltas when writing packfiles. It's an + * expensive operation, which should be avoided if no benefit is + * expected from possible deduplication in the same packfile. + */ + bool dontFindDeltas = false; }; static ref openRepo(const std::filesystem::path & path, Options options); diff --git a/src/libfetchers/include/nix/fetchers/meson.build b/src/libfetchers/include/nix/fetchers/meson.build index a313b1e0bc0c..f3bb80942a28 100644 --- a/src/libfetchers/include/nix/fetchers/meson.build +++ b/src/libfetchers/include/nix/fetchers/meson.build @@ -10,6 +10,7 @@ headers = files( 'git-lfs-fetch.hh', 'git-utils.hh', 'input-cache.hh', + 'provenance.hh', 'registry.hh', 'tarball.hh', ) diff --git a/src/libfetchers/include/nix/fetchers/provenance.hh b/src/libfetchers/include/nix/fetchers/provenance.hh new file mode 100644 index 000000000000..82dc1b3b8109 --- /dev/null +++ b/src/libfetchers/include/nix/fetchers/provenance.hh @@ -0,0 +1,31 @@ +#pragma once + +#include "nix/util/provenance.hh" +#include "nix/fetchers/fetchers.hh" + +namespace nix { + +struct TreeProvenance : Provenance +{ + ref attrs; + + TreeProvenance(const fetchers::Input & input); + + TreeProvenance(ref attrs) + : attrs(std::move(attrs)) + { + } + + nlohmann::json to_json() const override; +}; + +struct FetchurlProvenance : Provenance +{ + std::string url; + + FetchurlProvenance(std::string url, bool sanitize = true); + + nlohmann::json to_json() const override; +}; + +} // namespace nix diff --git a/src/libfetchers/include/nix/fetchers/registry.hh b/src/libfetchers/include/nix/fetchers/registry.hh index dc7e3edb590e..ca38dd805d6d 100644 --- a/src/libfetchers/include/nix/fetchers/registry.hh +++ b/src/libfetchers/include/nix/fetchers/registry.hh @@ -39,6 +39,9 @@ struct Registry static std::shared_ptr read(const Settings & settings, const SourcePath & path, RegistryType type); + static std::shared_ptr + read(const Settings & settings, std::string_view whence, std::string_view jsonStr, RegistryType type); + void write(const std::filesystem::path & path); void add(const Input & from, const Input & to, const Attrs & extraAttrs); diff --git a/src/libfetchers/indirect.cc b/src/libfetchers/indirect.cc index b2a41a7421fd..4205f7ce19fe 100644 --- a/src/libfetchers/indirect.cc +++ b/src/libfetchers/indirect.cc @@ -3,9 +3,11 @@ #include "nix/util/url-parts.hh" #include "nix/store/path.hh" +#include + namespace nix::fetchers { -std::regex flakeRegex("[a-zA-Z][a-zA-Z0-9_-]*", std::regex::ECMAScript); +boost::regex flakeRegex("[a-zA-Z][a-zA-Z0-9_-]*", boost::regex::ECMAScript); struct IndirectInputScheme : InputScheme { @@ -39,7 +41,7 @@ struct IndirectInputScheme : InputScheme throw BadURL("GitHub URL '%s' is invalid", url); std::string id = path[0]; - if (!std::regex_match(id, flakeRegex)) + if (!boost::regex_match(id, flakeRegex)) throw BadURL("'%s' is not a valid flake ID", id); // FIXME: forbid query params? @@ -92,7 +94,7 @@ struct IndirectInputScheme : InputScheme std::optional inputFromAttrs(const Settings & settings, const Attrs & attrs) const override { auto id = getStrAttr(attrs, "id"); - if (!std::regex_match(id, flakeRegex)) + if (!boost::regex_match(id, flakeRegex)) throw BadURL("'%s' is not a valid flake ID", id); Input input{}; @@ -100,7 +102,7 @@ struct IndirectInputScheme : InputScheme return input; } - ParsedURL toURL(const Input & input) const override + ParsedURL toURL(const Input & input, bool abbreviate) const override { ParsedURL url{ .scheme = "flake", @@ -131,11 +133,6 @@ struct IndirectInputScheme : InputScheme throw Error("indirect input '%s' cannot be fetched directly", input.to_string()); } - std::optional experimentalFeature() const override - { - return Xp::Flakes; - } - bool isDirect(const Input & input) const override { return false; diff --git a/src/libfetchers/make.nix b/src/libfetchers/make.nix new file mode 100644 index 000000000000..af4c15723873 --- /dev/null +++ b/src/libfetchers/make.nix @@ -0,0 +1,29 @@ +# libnixfetchers, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-store, +}: + +nixMake.mkComponent { + name = "determinate-nix-fetchers"; + libName = "nixfetchers"; + + deps = [ + nix-util + nix-store + ]; + + root = ./.; + + includeDirs = [ + "" + "include" + ]; + + files = nixMake.commonSupportFiles // { + "builtin-flake-registry.json.gen.hh" = nixMake.mkStringHeader ./builtin-flake-registry.json; + }; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libfetchers/mercurial.cc b/src/libfetchers/mercurial.cc index ab1d31ed330f..b7023efab609 100644 --- a/src/libfetchers/mercurial.cc +++ b/src/libfetchers/mercurial.cc @@ -122,7 +122,7 @@ struct MercurialInputScheme : InputScheme return input; } - ParsedURL toURL(const Input & input) const override + ParsedURL toURL(const Input & input, bool abbreviate) const override { auto url = parseURL(getStrAttr(input.attrs, "url")); url.scheme = "hg+" + url.scheme; @@ -278,9 +278,7 @@ struct MercurialInputScheme : InputScheme auto revInfoKey = [&](const Hash & rev) { if (rev.algo != HashAlgorithm::SHA1) - throw Error( - "Hash '%s' is not supported by Mercurial. Only sha1 is supported.", - rev.to_string(HashFormat::Base16, true)); + throw Error("Hash '%s' is not supported by Mercurial. Only sha1 is supported.", rev.gitRev()); return Cache::Key{"hgRev", {{"store", store.storeDir}, {"name", name}, {"rev", input.getRev()->gitRev()}}}; }; @@ -402,7 +400,7 @@ struct MercurialInputScheme : InputScheme auto storePath = fetchToStore(settings, store, input); auto accessor = store.requireStoreObjectAccessor(storePath); - accessor->setPathDisplay("«" + input.to_string() + "»"); + accessor->setPathDisplay("«" + input.to_string(true) + "»"); return {accessor, input}; } @@ -415,7 +413,7 @@ struct MercurialInputScheme : InputScheme std::optional getFingerprint(Store & store, const Input & input) const override { if (auto rev = input.getRev()) - return rev->gitRev(); + return "hg:" + rev->gitRev(); else return std::nullopt; } diff --git a/src/libfetchers/meson.build b/src/libfetchers/meson.build index ed52e565279d..134fd496a7a1 100644 --- a/src/libfetchers/meson.build +++ b/src/libfetchers/meson.build @@ -35,6 +35,7 @@ subdir('nix-meson-build-support/common') sources = files( 'attrs.cc', + 'builtin.cc', 'cache.cc', 'fetch-settings.cc', 'fetch-to-store.cc', @@ -48,23 +49,30 @@ sources = files( 'input-cache.cc', 'mercurial.cc', 'path.cc', + 'provenance.cc', 'registry.cc', 'tarball.cc', ) subdir('include/nix/fetchers') +# Generate builtin-flake-registry.json.gen.hh +subdir('nix-meson-build-support/generate-header') + +sources += gen_header.process( + 'builtin-flake-registry.json', +) + subdir('nix-meson-build-support/export-all-symbols') subdir('nix-meson-build-support/windows-version') this_library = library( 'nixfetchers', sources, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, link_args : linker_export_flags, - prelink : true, # For C++ static initializers + prelink : prelink, # For C++ static initializers install : true, cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [], ) diff --git a/src/libfetchers/package.nix b/src/libfetchers/package.nix index 14592087999c..1a30ac293018 100644 --- a/src/libfetchers/package.nix +++ b/src/libfetchers/package.nix @@ -17,7 +17,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-fetchers"; + pname = "determinate-nix-fetchers"; inherit version; workDir = ./.; @@ -28,6 +28,7 @@ mkMesonLibrary (finalAttrs: { ./.version ./meson.build ./include/nix/fetchers/meson.build + ./builtin-flake-registry.json (fileset.fileFilter (file: file.hasExt "cc") ./.) (fileset.fileFilter (file: file.hasExt "hh") ./.) ]; diff --git a/src/libfetchers/path.cc b/src/libfetchers/path.cc index 9d846c675367..9844c7a84b31 100644 --- a/src/libfetchers/path.cc +++ b/src/libfetchers/path.cc @@ -87,7 +87,7 @@ struct PathInputScheme : InputScheme return input; } - ParsedURL toURL(const Input & input) const override + ParsedURL toURL(const Input & input, bool abbreviate) const override { auto query = attrsToQuery(input.attrs); query.erase("path"); @@ -138,51 +138,37 @@ struct PathInputScheme : InputScheme throw Error("cannot fetch input '%s' because it uses a relative path", input.to_string()); } - std::pair, Input> - getAccessor(const Settings & settings, Store & store, const Input & _input) const override + std::optional, Input>> + getAccessor(const Settings & settings, Store & store, const Input & _input, bool fastOnly) const override { + // Note: fastOnly is ignored because the path fetcher is always fast. + Input input(_input); - auto path = getStrAttr(input.attrs, "path"); auto absPath = getAbsPath(input); // FIXME: check whether access to 'path' is allowed. + + auto accessor = makeFSSourceAccessor(absPath); + auto storePath = store.maybeParseStorePath(absPath.string()); - if (storePath) + if (storePath) { store.addTempRoot(*storePath); - time_t mtime = 0; - if (!storePath || storePath->name() != "source" || !store.isValidPath(*storePath)) { - Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", PathFmt(absPath))); - // FIXME: try to substitute storePath. - auto src = sinkToSource( - [&](Sink & sink) { mtime = dumpPathAndGetMtime(absPath.string(), sink, defaultPathFilter); }); - storePath = store.addToStoreFromDump(*src, "source"); + // To prevent `fetchToStore()` copying the path again to Nix + // store, pre-create an entry in the fetcher cache. + auto info = store.maybeQueryPathInfo(*storePath); + if (info) { + accessor->fingerprint = fmt("path:%s", info->narHash.to_string(HashFormat::SRI, true)); + settings.getCache()->upsert( + makeSourcePathToHashCacheKey( + *accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, CanonPath::root), + {{"hash", info->narHash.to_string(HashFormat::SRI, true)}}); + } } - auto accessor = store.requireStoreObjectAccessor(*storePath); - - // To prevent `fetchToStore()` copying the path again to Nix - // store, pre-create an entry in the fetcher cache. - auto narHash = store.queryPathInfo(*storePath)->narHash.to_string(HashFormat::SRI, true); - accessor->fingerprint = fmt("path:%s", narHash); - settings.getCache()->upsert( - makeSourcePathToHashCacheKey( - *accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, CanonPath::root), - {{"hash", narHash}}); - - /* Trust the lastModified value supplied by the user, if - any. It's not a "secure" attribute so we don't care. */ - if (!input.getLastModified()) - input.attrs.insert_or_assign("lastModified", uint64_t(mtime)); - - return {accessor, std::move(input)}; - } - - std::optional experimentalFeature() const override - { - return Xp::Flakes; + return {{accessor, std::move(input)}}; } }; diff --git a/src/libfetchers/provenance.cc b/src/libfetchers/provenance.cc new file mode 100644 index 000000000000..e984b2271014 --- /dev/null +++ b/src/libfetchers/provenance.cc @@ -0,0 +1,58 @@ +#include "nix/fetchers/provenance.hh" +#include "nix/fetchers/attrs.hh" +#include "nix/util/json-utils.hh" + +#include + +namespace nix { + +TreeProvenance::TreeProvenance(const fetchers::Input & input) + : attrs(make_ref([&]() { + // Remove the narHash attribute from the provenance info, as it's redundant (it's already recorded in the store + // path info). + auto attrs2 = input.attrs; + attrs2.erase("narHash"); + return fetchers::attrsToJSON(attrs2); + }())) +{ +} + +nlohmann::json TreeProvenance::to_json() const +{ + return nlohmann::json{ + {"type", "tree"}, + {"attrs", *attrs}, + }; +} + +Provenance::Register registerTreeProvenance("tree", [](nlohmann::json json) { + auto & obj = getObject(json); + auto & attrsJson = valueAt(obj, "attrs"); + return make_ref(make_ref(attrsJson)); +}); + +FetchurlProvenance::FetchurlProvenance(std::string _url, bool sanitize) + : url(std::move(_url)) +{ + if (sanitize) { + try { + url = parseURL(url, true).renderSanitized(); + } catch (BadURL &) { + } + } +} + +nlohmann::json FetchurlProvenance::to_json() const +{ + return nlohmann::json{ + {"type", "fetchurl"}, + {"url", url}, + }; +} + +Provenance::Register registerFetchurlProvenance("fetchurl", [](nlohmann::json json) { + auto & obj = getObject(json); + return make_ref(getString(valueAt(obj, "url")), false); +}); + +} // namespace nix diff --git a/src/libfetchers/registry.cc b/src/libfetchers/registry.cc index 0aad8ef48acb..82a65061deae 100644 --- a/src/libfetchers/registry.cc +++ b/src/libfetchers/registry.cc @@ -14,14 +14,24 @@ std::shared_ptr Registry::read(const Settings & settings, const Source { debug("reading registry '%s'", path); - auto registry = std::make_shared(type); - if (!path.pathExists()) return std::make_shared(type); try { + return read(settings, path.to_string(), path.readFile(), type); + } catch (Error & e) { + warn("cannot read flake registry '%s': %s", path, e.what()); + return std::make_shared(type); + } +} - auto json = nlohmann::json::parse(path.readFile()); +std::shared_ptr +Registry::read(const Settings & settings, std::string_view whence, std::string_view jsonStr, RegistryType type) +{ + auto registry = std::make_shared(type); + + try { + auto json = nlohmann::json::parse(jsonStr); auto version = json.value("version", 0); @@ -45,12 +55,10 @@ std::shared_ptr Registry::read(const Settings & settings, const Source } else - throw Error("flake registry '%s' has unsupported version %d", path, version); + warn("flake registry '%s' has unsupported version %d", whence, version); } catch (nlohmann::json::exception & e) { - warn("cannot parse flake registry '%s': %s", path, e.what()); - } catch (Error & e) { - warn("cannot read flake registry '%s': %s", path, e.what()); + warn("cannot parse flake registry '%s': %s", whence, e.what()); } return registry; @@ -139,25 +147,39 @@ void overrideRegistry(const Input & from, const Input & to, const Attrs & extraA static std::shared_ptr getGlobalRegistry(const Settings & settings, Store & store) { static auto reg = [&]() { - auto path = settings.flakeRegistry.get(); - if (path == "") { - return std::make_shared(Registry::Global); // empty registry - } + try { + auto path = settings.flakeRegistry.get(); + if (path == "") { + return std::make_shared(Registry::Global); // empty registry + } - return Registry::read( - settings, - [&] -> SourcePath { - std::filesystem::path fsPath{path}; - if (!fsPath.is_absolute()) { - auto storePath = downloadFile(store, settings, path, "flake-registry.json").storePath; - if (auto store2 = dynamic_cast(&store)) - store2->addPermRoot(storePath, (getCacheDir() / "flake-registry.json").string()); - return {store.requireStoreObjectAccessor(storePath)}; - } else { - return SourcePath{getFSSourceAccessor(), CanonPath{fsPath.string()}}.resolveSymlinks(); - } - }(), - Registry::Global); + return Registry::read( + settings, + [&] -> SourcePath { + std::filesystem::path fsPath{path}; + if (!fsPath.is_absolute()) { + auto storePath = downloadFile(store, settings, path, "flake-registry.json").storePath; + if (auto store2 = dynamic_cast(&store)) + store2->addPermRoot(storePath, (getCacheDir() / "flake-registry.json").string()); + return {store.requireStoreObjectAccessor(storePath)}; + } else { + return SourcePath{getFSSourceAccessor(), CanonPath{fsPath.string()}}.resolveSymlinks(); + } + }(), + Registry::Global); + } catch (Error & e) { + warn( + "cannot fetch global flake registry '%s', will use builtin fallback registry: %s", + settings.flakeRegistry.get(), + e.info().msg); + // Use builtin registry as fallback + return Registry::read( + settings, + "builtin flake registry", +#include "builtin-flake-registry.json.gen.hh" + , + Registry::Global); + } }(); return reg; diff --git a/src/libfetchers/tarball.cc b/src/libfetchers/tarball.cc index 065174433110..5586229e56cf 100644 --- a/src/libfetchers/tarball.cc +++ b/src/libfetchers/tarball.cc @@ -9,6 +9,9 @@ #include "nix/store/store-api.hh" #include "nix/fetchers/git-utils.hh" #include "nix/fetchers/fetch-settings.hh" +#include "nix/fetchers/provenance.hh" + +#include namespace nix::fetchers { @@ -83,6 +86,8 @@ DownloadFileResult downloadFile( }, hashString(HashAlgorithm::SHA256, sink.s)); info.narSize = sink.s.size(); + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) + info.provenance = std::make_shared(request.uri.to_string()); auto source = StringSource{sink.s}; store.addToStore(info, source, NoRepair, NoCheckSigs); storePath = std::move(info.path); @@ -104,8 +109,12 @@ DownloadFileResult downloadFile( }; } -static DownloadTarballResult downloadTarball_( - const Settings & settings, const std::string & urlS, const Headers & headers, const std::string & displayPrefix) +static std::optional downloadTarball_( + const Settings & settings, + const std::string & urlS, + const Headers & headers, + const std::string & displayPrefix, + bool fastOnly) { ParsedURL url = parseURL(urlS); @@ -155,6 +164,9 @@ static DownloadTarballResult downloadTarball_( `tarballTtl`, so no need to check the server. */ return attrsToResult(cached->value); + if (fastOnly) + return std::nullopt; + auto _res = std::make_shared>(); auto source = sinkToSource([&](Sink & sink) { @@ -382,7 +394,7 @@ struct CurlInputScheme : InputScheme return input; } - ParsedURL toURL(const Input & input) const override + ParsedURL toURL(const Input & input, bool abbreviate) const override { auto url = parseURL(getStrAttr(input.attrs, "url")); // NAR hashes are preferred over file hashes since tar/zip @@ -437,7 +449,7 @@ struct FileInputScheme : CurlInputScheme auto accessor = ref{store.getFSAccessor(file.storePath)}; - accessor->setPathDisplay("«" + input.to_string() + "»"); + accessor->setPathDisplay("«" + input.to_string(true) + "»"); return {accessor, input}; } @@ -487,12 +499,16 @@ struct TarballInputScheme : CurlInputScheme : (requireTree || hasTarballExtension(url))); } - std::pair, Input> - getAccessor(const Settings & settings, Store & store, const Input & _input) const override + std::optional, Input>> + getAccessor(const Settings & settings, Store & store, const Input & _input, bool fastOnly) const override { auto input(_input); - auto result = downloadTarball_(settings, getStrAttr(input.attrs, "url"), {}, "«" + input.to_string() + "»"); + auto res = + downloadTarball_(settings, getStrAttr(input.attrs, "url"), {}, "«" + input.to_string(true) + "»", fastOnly); + if (!res) + return std::nullopt; + auto & result = *res; if (result.immutableUrl) { auto immutableInput = Input::fromURL(settings, *result.immutableUrl); @@ -510,15 +526,15 @@ struct TarballInputScheme : CurlInputScheme "narHash", settings.getTarballCache()->treeHashToNarHash(settings, result.treeHash).to_string(HashFormat::SRI, true)); - return {result.accessor, input}; + return {{result.accessor, input}}; } std::optional getFingerprint(Store & store, const Input & input) const override { if (auto narHash = input.getNarHash()) - return narHash->to_string(HashFormat::SRI, true); + return "tarball:" + narHash->to_string(HashFormat::SRI, true); else if (auto rev = input.getRev()) - return rev->gitRev(); + return "tarball:" + rev->gitRev(); else return std::nullopt; } diff --git a/src/libflake-c/make.nix b/src/libflake-c/make.nix new file mode 100644 index 000000000000..fcc65d40e745 --- /dev/null +++ b/src/libflake-c/make.nix @@ -0,0 +1,40 @@ +# libnixflakec, the C API of libnixflake, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-util-c, + nix-store, + nix-store-c, + nix-fetchers, + nix-fetchers-c, + nix-expr, + nix-expr-c, + nix-flake, +}: + +nixMake.mkComponent { + name = "determinate-nix-flake-c"; + libName = "nixflakec"; + + deps = [ + nix-util + nix-util-c + nix-store + nix-store-c + nix-fetchers + nix-fetchers-c + nix-expr + nix-expr-c + nix-flake + ]; + + root = ./.; + + includeDirs = [ "" ]; + # The headers live in the root, which the default would not export. + publicIncludeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libflake-c/meson.build b/src/libflake-c/meson.build index 4fef3ea0018a..b1067a5d054e 100644 --- a/src/libflake-c/meson.build +++ b/src/libflake-c/meson.build @@ -54,7 +54,6 @@ subdir('nix-meson-build-support/windows-version') build_both_libraries = get_option('plugin-c-api') library_kwargs = { - 'soversion' : nix_soversion, 'dependencies' : deps_public + deps_private + deps_other, 'include_directories' : include_dirs, 'link_args' : linker_export_flags, diff --git a/src/libflake-c/nix_api_flake.cc b/src/libflake-c/nix_api_flake.cc index cbf5430f54a9..2558236a7e5a 100644 --- a/src/libflake-c/nix_api_flake.cc +++ b/src/libflake-c/nix_api_flake.cc @@ -209,4 +209,20 @@ nix_value * nix_locked_flake_get_output_attrs( NIXC_CATCH_ERRS_NULL } +nix_err nix_locked_flake_read_path( + nix_c_context * context, + nix_locked_flake * lockedFlake, + const char * path, + nix_get_string_callback callback, + void * user_data) +{ + nix_clear_err(context); + try { + auto source_path = lockedFlake->lockedFlake->flake.path.parent() / nix::CanonPath(path); + auto v = source_path.readFile(); + return call_nix_get_string_callback(v, callback, user_data); + } + NIXC_CATCH_ERRS +} + } // extern "C" diff --git a/src/libflake-c/nix_api_flake.h b/src/libflake-c/nix_api_flake.h index a3221e676bec..9884a3d39a49 100644 --- a/src/libflake-c/nix_api_flake.h +++ b/src/libflake-c/nix_api_flake.h @@ -239,6 +239,23 @@ void nix_flake_reference_free(nix_flake_reference * store); nix_value * nix_locked_flake_get_output_attrs( nix_c_context * context, nix_flake_settings * settings, EvalState * evalState, nix_locked_flake * lockedFlake); +/** + * @brief Reads a file within the flake. + * @note The callback borrows the string only for the duration of the call. + * + * @param[out] context Optional, stores error information + * @param[in] locked_flake the flake to get the path for + * @param[in] path The path within the flake. + * @param[in] callback The callback to call with the string + * @param[in] user_data Additional data to pass for the callback + */ +nix_err nix_locked_flake_read_path( + nix_c_context * context, + nix_locked_flake * lockedFlake, + const char * path, + nix_get_string_callback callback, + void * user_data); + #ifdef __cplusplus } // extern "C" #endif diff --git a/src/libflake-c/package.nix b/src/libflake-c/package.nix index e64179c72d1e..344412b47fc0 100644 --- a/src/libflake-c/package.nix +++ b/src/libflake-c/package.nix @@ -18,7 +18,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-flake-c"; + pname = "determinate-nix-flake-c"; inherit version; workDir = ./.; diff --git a/src/libflake-tests/flakeref.cc b/src/libflake-tests/flakeref.cc index 5fcbbc9c30bf..8ef0b996ecf2 100644 --- a/src/libflake-tests/flakeref.cc +++ b/src/libflake-tests/flakeref.cc @@ -10,6 +10,7 @@ #include "nix/util/configuration.hh" #include "nix/util/error.hh" #include "nix/util/experimental-features.hh" +#include "nix/util/terminal.hh" namespace nix { @@ -17,8 +18,6 @@ namespace nix { TEST(parseFlakeRef, path) { - experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes); - fetchers::Settings fetchSettings; { @@ -67,8 +66,6 @@ TEST(parseFlakeRef, path) TEST(parseFlakeRef, GitArchiveInput) { - experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes); - fetchers::Settings fetchSettings; { @@ -111,7 +108,6 @@ class InputFromURLTest : public ::testing::WithParamInterface & info) { return info.param.description; }); +TEST(parseFlakeRef, nix219CompatRetainsDirInUrlAttr) +{ + fetchers::Settings fetchSettings; + fetchSettings.nix219Compat = true; + + /* Nix < 2.20 retained the `dir` query parameter in the `url` + attribute of input types that have one. */ + { + auto flakeref = parseFlakeRef(fetchSettings, "git+https://example.org/repo?dir=sub"); + ASSERT_EQ(flakeref.subdir, "sub"); + ASSERT_EQ(flakeref.toAttrs().at("url"), Attr("https://example.org/repo?dir=sub")); + } + + { + auto flakeref = parseFlakeRef(fetchSettings, "https://example.org/foo.tar.gz?dir=sub"); + ASSERT_EQ(flakeref.subdir, "sub"); + ASSERT_EQ(flakeref.toAttrs().at("url"), Attr("https://example.org/foo.tar.gz?dir=sub")); + } + + /* But for input types that don't have a `url` attribute (such as + `github`), the `dir` parameter was never part of the input + attributes, so it should not cause a parse failure. */ + { + auto flakeref = parseFlakeRef(fetchSettings, "github:NixOS/nix?dir=perl"); + ASSERT_EQ(flakeref.subdir, "perl"); + ASSERT_EQ( + flakeref.toAttrs(), + (fetchers::Attrs{ + {"dir", Attr("perl")}, + {"owner", Attr("NixOS")}, + {"repo", Attr("nix")}, + {"type", Attr("github")}, + })); + } +} + TEST(to_string, doesntReencodeUrl) { fetchers::Settings fetchSettings; @@ -283,10 +315,26 @@ TEST(to_string, doesntReencodeUrl) ASSERT_EQ(unparsed, expected); } -TEST(parseFlakeRef, malformedGithubUrlDoesNotCrash) +TEST(parseFlakeRef, urlInterpretationErrorsAreNotMasked) { - experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes); + fetchers::Settings fetchSettings; + // Errors that occur while interpreting a syntactically valid URL + // (such as an unsupported query parameter) should be shown to the + // user, rather than causing the flake ref to be reinterpreted as a + // path, leading to a confusing "not an absolute path" error. + try { + parseFlakeRef(fetchSettings, "github:foo/bar?xyzzy=1"); + FAIL() << "expected parseFlakeRef to throw"; + } catch (BadURL & e) { + auto msg = filterANSIEscapes(e.msg()); + EXPECT_NE(msg.find("xyzzy"), std::string::npos) << msg; + EXPECT_EQ(msg.find("not an absolute path"), std::string::npos) << msg; + } +} + +TEST(parseFlakeRef, malformedGithubUrlDoesNotCrash) +{ fetchers::Settings fetchSettings; // Using ref= instead of rev= with a github: URL should produce an diff --git a/src/libflake-tests/make.nix b/src/libflake-tests/make.nix new file mode 100644 index 000000000000..c075190595c9 --- /dev/null +++ b/src/libflake-tests/make.nix @@ -0,0 +1,47 @@ +# The unit tests of libnixflake, transcribed from meson.build and +# package.nix: the `nix-flake-tests` executable, with the derivation that +# runs it as `tests.run`. +{ + nixMake, + nix-expr-test-support, + nix-flake, + nix-flake-c, +}: + +let + tests = nixMake.mkComponent { + name = "determinate-nix-flake-tests"; + type = "executable"; + exeName = "nix-flake-tests"; + + deps = [ + nix-expr-test-support + nix-flake + nix-flake-c + ]; + + root = ./.; + + includeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + # The tests have no main() of their own; it comes from gtest_main + # (see src/libutil-tests/make.nix). + extraLinkLibs = [ "-lgtest_main" ]; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; + }; +in + +tests +// { + tests.run = nixMake.runTest { + name = "${tests.name}-run"; + command = [ "${tests}/bin/${tests.exeName}" ]; + env = { + _NIX_TEST_UNIT_DATA = ./data; + NIX_STORE = ""; + }; + }; +} diff --git a/src/libflake-tests/meson.build b/src/libflake-tests/meson.build index 00b592195c2c..394380f80668 100644 --- a/src/libflake-tests/meson.build +++ b/src/libflake-tests/meson.build @@ -60,7 +60,6 @@ test( this_exe, env : { '_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data', - 'NIX_CONFIG' : 'extra-experimental-features = flakes', 'HOME' : meson.current_build_dir() / 'test-home', 'NIX_STORE' : '', }, diff --git a/src/libflake-tests/package.nix b/src/libflake-tests/package.nix index 397ef4192447..2a359e1c2b47 100644 --- a/src/libflake-tests/package.nix +++ b/src/libflake-tests/package.nix @@ -60,7 +60,6 @@ mkMesonExecutable (finalAttrs: { } ('' export _NIX_TEST_UNIT_DATA=${resolvePath ./data} - export NIX_CONFIG="extra-experimental-features = flakes" ${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage} touch $out ''); diff --git a/src/libflake/call-flake.nix b/src/libflake/call-flake.nix index ed7947e06012..d1037efcd23b 100644 --- a/src/libflake/call-flake.nix +++ b/src/libflake/call-flake.nix @@ -10,9 +10,6 @@ lockFileStr: # unlocked trees. overrides: -# This is `prim_fetchFinalTree`. -fetchTreeFinal: - let inherit (builtins) mapAttrs; @@ -45,14 +42,24 @@ let parentNode = allNodes.${getInputByPath lockFile.root node.parent}; sourceInfo = - if hasOverride then + if node.buildTime or false then + derivation { + name = "source"; + builder = "builtin:fetch-tree"; + system = "builtin"; + __structuredAttrs = true; + input = node.locked; + outputHashMode = "recursive"; + outputHash = node.locked.narHash; + } + else if hasOverride then overrides.${key}.sourceInfo else if isRelative then parentNode.sourceInfo else # FIXME: remove obsolete node.info. # Note: lock file entries are always final. - fetchTreeFinal (node.info or { } // removeAttrs node.locked [ "dir" ]); + builtins.fetchTree (node.info or { } // removeAttrs node.locked [ "dir" ]); subdir = overrides.${key}.dir or node.locked.dir or ""; @@ -93,6 +100,7 @@ let result = if node.flake or true then assert builtins.isFunction flake.outputs; + assert !(node.buildTime or false); result else sourceInfo // { inherit sourceInfo outPath; }; diff --git a/src/libflake/flake-primops.cc b/src/libflake/flake-primops.cc index 5ee587c3e051..b8838ab18e52 100644 --- a/src/libflake/flake-primops.cc +++ b/src/libflake/flake-primops.cc @@ -5,6 +5,7 @@ #include #include "nix/flake/flake-primops.hh" +#include "nix/store/store-api.hh" #include "nix/expr/eval.hh" #include "nix/flake/flake.hh" #include "nix/flake/flakeref.hh" @@ -19,12 +20,11 @@ #include "nix/fetchers/fetchers.hh" #include "nix/util/error.hh" #include "nix/util/experimental-features.hh" -#include "nix/util/mounted-source-accessor.hh" #include "nix/util/pos-idx.hh" #include "nix/util/pos-table.hh" #include "nix/util/types.hh" #include "nix/util/util.hh" -#include "nix/store/store-api.hh" +#include "nix/util/mounted-source-accessor.hh" namespace nix::flake::primops { @@ -44,8 +44,16 @@ PrimOp getFlake(const Settings & settings) auto path = state.realisePath(pos, *args[0]); callFlake(state, lockFlake(settings, state, path, lockFlags), v); } else { + NixStringContext context; std::string flakeRefS( - state.forceStringNoCtx(*args[0], pos, "while evaluating the argument passed to builtins.getFlake")); + state.forceString(*args[0], context, pos, "while evaluating the argument passed to builtins.getFlake")); + auto rewrites = state.realiseContext(context); + flakeRefS = state.devirtualize(rewriteStrings(flakeRefS, rewrites), context); + if (hasContext(context)) + // FIXME: this should really be an error. + warn( + "In 'builtins.getFlake', the flakeref '%s' has string context, but that's not allowed. This may become a fatal error in the future.", + flakeRefS); auto flakeRef = nix::parseFlakeRef(state.fetchSettings, flakeRefS, {}, true); if (state.settings.pureEval && !flakeRef.input.isLocked(state.fetchSettings)) @@ -54,18 +62,19 @@ PrimOp getFlake(const Settings & settings) flakeRefS, state.positions[pos]); - /* Backwards compatibility: since flakes used to be copied to the store eagerly, some users - relied on being able to do builtins.getFlake on a flakeref with discarded string context. - So if a flake input has a physical source path that is inside the store, first try to look it up in the - storeFS. */ - if (auto sourcePath = flakeRef.input.getSourcePath(); - flakeRef.input.getType() == "path" && sourcePath && state.store->isInStore(sourcePath->string())) { - auto [storePath, subPath] = state.store->toStorePath(sourcePath->string()); - if (auto mount = state.storeFS->getMount(CanonPath(state.store->printStorePath(storePath)))) { - auto path = state.storePath(storePath) / CanonPath(subPath); - if (!flakeRef.subdir.empty()) - path = path / flakeRef.subdir; - return callFlake(state, lockFlake(settings, state, path, lockFlags), v); + /* Backward compatibility hack: If this is a `path` flake and it's a virtual path that had + * `unsafeDiscardStringContext` applied to it, then treat it like the `nPath` case, i.e. call lockFlake() on + * the virtual path directly. This is necessary because the `path` fetcher doesn't see virtual paths. */ + if (flakeRef.input.getType() == "path") { + if (auto sourcePath = flakeRef.input.getSourcePath(); + sourcePath && state.store->isInStore(sourcePath->string())) { + auto [storePath, subPath] = state.store->toStorePath(sourcePath->string()); + if (auto mount = state.storeFS->getMount(CanonPath(state.store->printStorePath(storePath)))) { + auto path = state.storePath(storePath) / CanonPath(subPath); + if (!flakeRef.subdir.empty()) + path = path / flakeRef.subdir; + return callFlake(state, lockFlake(settings, state, path, lockFlags), v); + } } } @@ -92,7 +101,6 @@ PrimOp getFlake(const Settings & settings) ``` )", .impl = prim_getFlake, - .experimentalFeature = Xp::Flakes, }; } @@ -135,13 +143,13 @@ nix::PrimOp parseFlakeRef({ ``` )", .impl = prim_parseFlakeRef, - .experimentalFeature = Xp::Flakes, }); static void prim_flakeRefToString(EvalState & state, const PosIdx pos, Value ** args, Value & v) { state.forceAttrs(*args[0], noPos, "while evaluating the argument passed to builtins.flakeRefToString"); fetchers::Attrs attrs; + NixStringContext context; for (const auto & attr : *args[0]->attrs()) { state.forceValue(*attr.value, attr.pos); auto t = attr.value->type(); @@ -160,7 +168,9 @@ static void prim_flakeRefToString(EvalState & state, const PosIdx pos, Value ** } else if (t == nBool) { attrs.emplace(state.symbols[attr.name], Explicit{attr.value->boolean()}); } else if (t == nString) { - attrs.emplace(state.symbols[attr.name], std::string(attr.value->string_view())); + auto s = state.forceString( + *attr.value, context, attr.pos, "while evaluating an attribute in 'builtins.flakeRefToString'"); + attrs.emplace(state.symbols[attr.name], std::string(s)); } else { state .error( @@ -172,7 +182,7 @@ static void prim_flakeRefToString(EvalState & state, const PosIdx pos, Value ** } } auto flakeRef = FlakeRef::fromAttrs(state.fetchSettings, attrs); - v.mkString(flakeRef.to_string(), state.mem); + v.mkString(flakeRef.to_string(), context, state.mem); } nix::PrimOp flakeRefToString({ @@ -196,7 +206,6 @@ nix::PrimOp flakeRefToString({ ``` )", .impl = prim_flakeRefToString, - .experimentalFeature = Xp::Flakes, }); } // namespace nix::flake::primops diff --git a/src/libflake/flake.cc b/src/libflake/flake.cc index bd3ea4acbc65..0ba33ac326f0 100644 --- a/src/libflake/flake.cc +++ b/src/libflake/flake.cc @@ -35,6 +35,7 @@ #include "nix/expr/value-to-json.hh" #include "nix/fetchers/fetch-to-store.hh" #include "nix/util/memory-source-accessor.hh" +#include "nix/util/mounted-source-accessor.hh" #include "nix/fetchers/input-cache.hh" #include "nix/expr/attr-set.hh" #include "nix/expr/eval-error.hh" @@ -64,19 +65,22 @@ namespace nix { struct SourceAccessor; +using namespace fetchers; + namespace flake { static void forceTrivialValue(EvalState & state, Value & value, const PosIdx pos) { - if (value.isThunk() && value.isTrivial()) + if (value.isTrivial()) state.forceValue(value, pos); } static void expectType(EvalState & state, ValueType type, Value & value, const PosIdx pos) { forceTrivialValue(state, value, pos); - if (value.type() != type) - throw Error("expected %s but got %s at %s", showType(type), showType(value.type()), state.positions[pos]); + auto t = value.type(); + if (t != type) + throw Error("expected %s but got %s at %s", showType(type), showType(t), state.positions[pos]); } static std::pair, fetchers::Attrs> parseFlakeInputs( @@ -87,7 +91,7 @@ static std::pair, fetchers::Attrs> parseFlakeInput const SourcePath & flakeDir, bool allowSelf); -static void parseFlakeInputAttr(EvalState & state, const Attr & attr, fetchers::Attrs & attrs) +static void parseFlakeInputAttr(EvalState & state, const nix::Attr & attr, fetchers::Attrs & attrs) { // Allow selecting a subset of enum values #pragma GCC diagnostic push @@ -141,6 +145,7 @@ static FlakeInput parseFlakeInput( auto sUrl = state.symbols.create("url"); auto sFlake = state.symbols.create("flake"); auto sFollows = state.symbols.create("follows"); + auto sBuildTime = state.symbols.create("buildTime"); fetchers::Attrs attrs; std::optional url; @@ -169,6 +174,11 @@ static FlakeInput parseFlakeInput( } else if (attr.name == sFlake) { expectType(state, nBool, *attr.value, attr.pos); input.isFlake = attr.value->boolean(); + } else if (attr.name == sBuildTime) { + expectType(state, nBool, *attr.value, attr.pos); + input.buildTime = attr.value->boolean(); + if (input.buildTime) + experimentalFeatureSettings.require(Xp::BuildTimeFetchTree); } else if (attr.name == sInputs) { input.overrides = parseFlakeInputs(state, attr.value, attr.pos, lockRootAttrPath, flakeDir, false).first; @@ -237,7 +247,7 @@ static std::pair, fetchers::Attrs> parseFlakeInput return {inputs, selfAttrs}; } -static Flake readFlake( +Flake readFlake( EvalState & state, const FlakeRef & originalRef, const FlakeRef & resolvedRef, @@ -257,6 +267,7 @@ static Flake readFlake( .resolvedRef = resolvedRef, .lockedRef = lockedRef, .path = flakePath, + .provenance = flakePath.getProvenance(), }; if (auto description = vInfo.attrs()->get(state.s.description)) { @@ -366,7 +377,8 @@ static Flake getFlake( EvalState & state, const FlakeRef & originalRef, fetchers::UseRegistries useRegistries, - const InputAttrPath & lockRootAttrPath) + const InputAttrPath & lockRootAttrPath, + bool requireLockable) { // Fetch a lazy tree first. auto cachedInput = @@ -392,14 +404,20 @@ static Flake getFlake( lockedRef = FlakeRef(std::move(cachedInput2.lockedInput), newLockedRef.subdir); } - auto rootDir = state.storePath(state.mountInput(lockedRef.input, originalRef.input, cachedInput.accessor)); // Re-parse flake.nix from the store. - return readFlake(state, originalRef, resolvedRef, lockedRef, rootDir, lockRootAttrPath); + return readFlake( + state, + originalRef, + resolvedRef, + lockedRef, + state.storePath(state.mountInput(lockedRef.input, originalRef.input, cachedInput.accessor, requireLockable)), + lockRootAttrPath); } -Flake getFlake(EvalState & state, const FlakeRef & originalRef, fetchers::UseRegistries useRegistries) +Flake getFlake( + EvalState & state, const FlakeRef & originalRef, fetchers::UseRegistries useRegistries, bool requireLockable) { - return getFlake(state, originalRef, useRegistries, {}); + return getFlake(state, originalRef, useRegistries, {}, requireLockable); } static LockFile readLockFile(const fetchers::Settings & fetchSettings, const SourcePath & lockFilePath) @@ -411,8 +429,6 @@ static LockFile readLockFile(const fetchers::Settings & fetchSettings, const Sou LockedFlake lockFlake( const Settings & settings, EvalState & state, const FlakeRef & topRef, const LockFlags & lockFlags, Flake flake) { - experimentalFeatureSettings.require(Xp::Flakes); - auto useRegistries = lockFlags.useRegistries.value_or(settings.useRegistries); auto useRegistriesTop = useRegistries ? fetchers::UseRegistries::All : fetchers::UseRegistries::No; auto useRegistriesInputs = useRegistries ? fetchers::UseRegistries::Limited : fetchers::UseRegistries::No; @@ -595,7 +611,7 @@ LockedFlake lockFlake( if (auto resolvedPath = resolveRelativePath()) { return readFlake(state, ref, ref, ref, *resolvedPath, inputAttrPath); } else { - return getFlake(state, ref, useRegistries, inputAttrPath); + return getFlake(state, ref, useRegistriesInputs, inputAttrPath, true); } }; @@ -618,7 +634,11 @@ LockedFlake lockFlake( didn't change and there is no override from a higher level flake. */ auto childNode = make_ref( - oldLock->lockedRef, oldLock->originalRef, oldLock->isFlake, oldLock->parentInputAttrPath); + oldLock->lockedRef, + oldLock->originalRef, + oldLock->isFlake, + oldLock->buildTime, + oldLock->parentInputAttrPath); node->inputs.insert_or_assign(id, childNode); @@ -707,12 +727,34 @@ LockedFlake lockFlake( auto inputIsOverride = explicitCliOverrides.contains(nonEmptyInputAttrPath); auto ref = (input2.ref && inputIsOverride) ? *input2.ref : *input.ref; + /* Warn against the use of indirect flakerefs + (but only at top-level since we don't want + to annoy users about flakes that are not + under their control). */ + auto warnRegistry = [&](const FlakeRef & resolvedRef) { + if (inputAttrPath.size() == 1 && !input.ref->input.isDirect()) { + std::ostringstream s; + printLiteralString(s, resolvedRef.to_string()); + warn( + "Flake input '%1%' uses the flake registry. " + "Using the registry in flake inputs is deprecated in Determinate Nix. " + "To make your flake future-proof, add the following to '%2%':\n" + "\n" + " inputs.%1%.url = %3%;\n" + "\n" + "For more information, see: https://github.com/DeterminateSystems/nix-src/issues/37", + inputAttrPathS, + flake.path, + s.str()); + } + }; + if (input.isFlake) { auto inputFlake = getInputFlake( *input.ref, inputIsOverride ? fetchers::UseRegistries::All : useRegistriesInputs); - auto childNode = - make_ref(inputFlake.lockedRef, ref, true, overriddenParentPath); + auto childNode = make_ref( + inputFlake.lockedRef, ref, true, input.buildTime, overriddenParentPath); node->inputs.insert_or_assign(id, childNode); @@ -734,6 +776,8 @@ LockedFlake lockFlake( inputAttrPath, inputFlake.path, false); + + warnRegistry(inputFlake.resolvedRef); } else { @@ -745,16 +789,21 @@ LockedFlake lockFlake( auto cachedInput = state.inputCache->getAccessor( state.fetchSettings, *state.store, input.ref->input, useRegistriesInputs); + auto resolvedRef = + FlakeRef(std::move(cachedInput.resolvedInput), input.ref->subdir); auto lockedRef = FlakeRef(std::move(cachedInput.lockedInput), input.ref->subdir); + warnRegistry(resolvedRef); + return { - state.storePath( - state.mountInput(lockedRef.input, input.ref->input, cachedInput.accessor)), + state.storePath(state.mountInput( + lockedRef.input, input.ref->input, cachedInput.accessor, true, true)), lockedRef}; } }(); - auto childNode = make_ref(lockedRef, ref, false, overriddenParentPath); + auto childNode = + make_ref(lockedRef, ref, false, input.buildTime, overriddenParentPath); nodePaths.emplace(childNode, path); @@ -870,7 +919,7 @@ LockedFlake lockFlake( repo, so we should re-read it. FIXME: we could also just clear the 'rev' field... */ auto prevLockedRef = flake.lockedRef; - flake = getFlake(state, topRef, useRegistriesTop); + flake = getFlake(state, topRef, useRegistriesTop, lockFlags.requireLockable); if (lockFlags.commitLockFile && flake.lockedRef.input.getRev() && prevLockedRef.input.getRev() != flake.lockedRef.input.getRev()) @@ -899,7 +948,8 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & topRef, { auto useRegistries = lockFlags.useRegistries.value_or(settings.useRegistries); auto useRegistriesTop = useRegistries ? fetchers::UseRegistries::All : fetchers::UseRegistries::No; - return lockFlake(settings, state, topRef, lockFlags, getFlake(state, topRef, useRegistriesTop, {})); + + return lockFlake(settings, state, topRef, lockFlags, getFlake(state, topRef, useRegistriesTop, {}, false)); } LockedFlake @@ -933,8 +983,6 @@ static Value * requireInternalFile(EvalState & state, CanonPath path) void callFlake(EvalState & state, const LockedFlake & lockedFlake, Value & vRes) { - experimentalFeatureSettings.require(Xp::Flakes); - auto [lockFileStr, keyMap] = lockedFlake.lockFile.to_string(); auto overrides = state.buildBindings(lockedFlake.nodePaths.size()); @@ -971,10 +1019,7 @@ void callFlake(EvalState & state, const LockedFlake & lockedFlake, Value & vRes) auto vLocks = state.allocValue(); vLocks->mkString(lockFileStr, state.mem); - auto vFetchFinalTree = get(state.internalPrimOps, "fetchFinalTree"); - assert(vFetchFinalTree); - - Value * args[] = {vLocks, &vOverrides, *vFetchFinalTree}; + Value * args[] = {vLocks, &vOverrides}; state.callFunction(*vCallFlake, args, vRes, noPos); } @@ -1018,41 +1063,6 @@ std::optional LockedFlake::getFingerprint(Store & store, const fetc Flake::~Flake() {} -ref openEvalCache(EvalState & state, ref lockedFlake) -{ - auto fingerprint = state.settings.useEvalCache && state.settings.pureEval - ? lockedFlake->getFingerprint(*state.store, state.fetchSettings) - : std::nullopt; - auto rootLoader = [&state, lockedFlake]() { - /* For testing whether the evaluation cache is - complete. */ - if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0") - throw Error("not everything is cached, but evaluation is not allowed"); - - auto vFlake = state.allocValue(); - callFlake(state, *lockedFlake, *vFlake); - - state.forceAttrs(*vFlake, noPos, "while parsing cached flake data"); - - auto aOutputs = vFlake->attrs()->get(state.symbols.create("outputs")); - assert(aOutputs); - - return aOutputs->value; - }; - - if (fingerprint) { - auto search = state.evalCaches.find(fingerprint.value()); - if (search == state.evalCaches.end()) { - search = state.evalCaches - .emplace(fingerprint.value(), make_ref(fingerprint, state, rootLoader)) - .first; - } - return search->second; - } else { - return make_ref(std::nullopt, state, rootLoader); - } -} - } // namespace flake } // namespace nix diff --git a/src/libflake/flakeref.cc b/src/libflake/flakeref.cc index 01b5be1a82d2..b68f5b7606f3 100644 --- a/src/libflake/flakeref.cc +++ b/src/libflake/flakeref.cc @@ -11,6 +11,8 @@ #include #include +#include + #include "nix/flake/flakeref.hh" #include "nix/util/url.hh" #include "nix/util/url-parts.hh" @@ -26,6 +28,7 @@ #include "nix/store/outputs-spec.hh" #include "nix/util/ref.hh" #include "nix/util/types.hh" +#include "nix/fetchers/fetch-settings.hh" namespace nix { class Store; @@ -42,12 +45,12 @@ const static std::string subDirElemRegex = "(?:[a-zA-Z0-9_-]+[a-zA-Z0-9._-]*)"; const static std::string subDirRegex = subDirElemRegex + "(?:/" + subDirElemRegex + ")*"; #endif -std::string FlakeRef::to_string() const +std::string FlakeRef::to_string(bool abbreviate) const { StringMap extraQuery; if (subdir != "") extraQuery.insert_or_assign("dir", subdir); - return input.toURLString(extraQuery); + return input.toURLString(extraQuery, abbreviate); } fetchers::Attrs FlakeRef::toAttrs() const @@ -95,7 +98,33 @@ fromParsedURL(const fetchers::Settings & fetchSettings, ParsedURL && parsedURL, std::string fragment; std::swap(fragment, parsedURL.fragment); - return {FlakeRef(fetchers::Input::fromURL(fetchSettings, parsedURL, isFlake), dir), fragment}; + auto input = fetchers::Input::fromURL(fetchSettings, parsedURL, isFlake); + + /* Backward compatibility hack: Nix < 2.20 retained the `dir` + query parameter in the `url` attribute of input types that have + one (i.e. `git`, `hg`, `tarball` and `file`), resulting in lock + file entries like + + "original": { + "dir": "subdir", + "type": "git", + "url": "https://foo/bar?dir=subdir" + } + + In compat mode, reproduce that behavior so that we write lock + files that Nix < 2.20 considers up to date. Input types that + don't have a `url` attribute (such as `github`) never included + the `dir` parameter in their attributes, so they're left + alone. */ + if (fetchSettings.nix219Compat && dir != "") { + if (auto url = fetchers::maybeGetStrAttr(input.attrs, "url")) { + auto parsedUrlAttr = parseURL(*url, /*lenient=*/true); + parsedUrlAttr.query.insert_or_assign("dir", dir); + input.attrs.insert_or_assign("url", parsedUrlAttr.to_string()); + } + } + + return {FlakeRef(std::move(input), dir), fragment}; } std::pair parsePathFlakeRefWithFragment( @@ -106,10 +135,10 @@ std::pair parsePathFlakeRefWithFragment( bool isFlake, bool preserveRelativePaths) { - static std::regex pathFlakeRegex(R"(([^?#]*)(\?([^#]*))?(#(.*))?)", std::regex::ECMAScript); + static boost::regex pathFlakeRegex(R"(([^?#]*)(\?([^#]*))?(#(.*))?)", boost::regex::ECMAScript); - std::smatch match; - auto succeeds = std::regex_match(url, match, pathFlakeRegex); + boost::smatch match; + auto succeeds = boost::regex_match(url, match, pathFlakeRegex, boost::match_not_dot_newline); if (!succeeds) throw Error("invalid flakeref '%s'", url); std::filesystem::path path = match[1].str(); @@ -222,13 +251,13 @@ std::pair parsePathFlakeRefWithFragment( static std::optional> parseFlakeIdRef(const fetchers::Settings & fetchSettings, const std::string & url, bool isFlake) { - std::smatch match; + boost::smatch match; - static std::regex flakeRegex( + static boost::regex flakeRegex( "((" + flakeIdRegexS + ")(?:/(?:" + refAndOrRevRegex + "))?)" + "(?:#(" + fragmentRegex + "))?", - std::regex::ECMAScript); + boost::regex::ECMAScript); - if (std::regex_match(url, match, flakeRegex)) { + if (boost::regex_match(url, match, flakeRegex)) { auto parsedURL = ParsedURL{ .scheme = "flake", .authority = std::nullopt, @@ -248,18 +277,27 @@ std::optional> parseURLFlakeRef( const std::optional & baseDir, bool isFlake) { + std::optional parsed; try { - auto parsed = parseURL(url, /*lenient=*/true); - if (baseDir && (parsed.scheme == "path" || parsed.scheme == "git+file")) { - /* Here we know that the path must not contain encoded '/' or NUL bytes. */ - auto path = urlPathToPath(parsed.path); - if (!path.is_absolute()) - parsed.path = pathToUrlPath(absPath(path, get(baseDir))); - } - return fromParsedURL(fetchSettings, std::move(parsed), isFlake); + parsed = parseURL(url, /*lenient=*/true); } catch (BadURL &) { + /* The string is not a URL, so try to interpret it as a path + instead (in the caller). Note that errors that occur while + *interpreting* a syntactically valid URL (such as an + unsupported query parameter) are not caught here; they + should be shown to the user rather than masked by a + confusing error about the flake ref not being a valid + path. */ return std::nullopt; } + + if (baseDir && (parsed->scheme == "path" || parsed->scheme == "git+file")) { + /* Here we know that the path must not contain encoded '/' or NUL bytes. */ + auto path = urlPathToPath(parsed->path); + if (!path.is_absolute()) + parsed->path = pathToUrlPath(absPath(path, get(baseDir))); + } + return fromParsedURL(fetchSettings, std::move(*parsed), isFlake); } std::pair parseFlakeRefWithFragment( diff --git a/src/libflake/include/nix/flake/flake.hh b/src/libflake/include/nix/flake/flake.hh index aa063a08e396..99720c9403a0 100644 --- a/src/libflake/include/nix/flake/flake.hh +++ b/src/libflake/include/nix/flake/flake.hh @@ -10,6 +10,7 @@ namespace nix { class EvalState; +struct Provenance; namespace flake { @@ -44,12 +45,18 @@ typedef std::map FlakeInputs; struct FlakeInput { std::optional ref; + /** - * true = process flake to get outputs - * - * false = (fetched) static source path + * Whether to call the `flake.nix` file in this input to get its outputs. */ bool isFlake = true; + + /** + * Whether to fetch this input at evaluation time or at build + * time. + */ + bool buildTime = false; + std::optional follows; FlakeInputs overrides; }; @@ -88,6 +95,11 @@ struct Flake */ SourcePath path; + /** + * Cached provenance of `flake.nix` (equivalent to `path.getProvenance()`). + */ + std::shared_ptr provenance; + /** * Pretend that `lockedRef` is dirty. */ @@ -116,7 +128,8 @@ struct Flake } }; -Flake getFlake(EvalState & state, const FlakeRef & flakeRef, fetchers::UseRegistries useRegistries); +Flake getFlake( + EvalState & state, const FlakeRef & flakeRef, fetchers::UseRegistries useRegistries, bool requireLockable = true); /** * Fingerprint of a locked flake; used as a cache key. @@ -212,8 +225,25 @@ struct LockFlags * for those inputs will be ignored. */ std::set inputUpdates; + + /** + * Whether to require a locked input. + */ + bool requireLockable = true; }; +/** + * Return a `Flake` object representing the flake read from the + * `flake.nix` file in `rootDir`. + */ +Flake readFlake( + EvalState & state, + const FlakeRef & originalRef, + const FlakeRef & resolvedRef, + const FlakeRef & lockedRef, + const SourcePath & rootDir, + const InputAttrPath & lockRootPath); + /* * Compute an in-memory lock file for the specified top-level flake, and optionally write it to file, if the flake is * writable. @@ -221,23 +251,14 @@ struct LockFlags LockedFlake lockFlake(const Settings & settings, EvalState & state, const FlakeRef & flakeRef, const LockFlags & lockFlags); +LockedFlake lockFlake( + const Settings & settings, EvalState & state, const FlakeRef & topRef, const LockFlags & lockFlags, Flake flake); + LockedFlake lockFlake(const Settings & settings, EvalState & state, const SourcePath & flakeDir, const LockFlags & lockFlags); void callFlake(EvalState & state, const LockedFlake & lockedFlake, Value & v); -/** - * Open an evaluation cache for a flake. - */ -ref openEvalCache(EvalState & state, ref lockedFlake); - } // namespace flake -/** - * An internal builtin similar to `fetchTree`, except that it - * always treats the input as final (i.e. no attributes can be - * added/removed/changed). - */ -void prim_fetchFinalTree(EvalState & state, const PosIdx pos, Value ** args, Value & v); - } // namespace nix diff --git a/src/libflake/include/nix/flake/flakeref.hh b/src/libflake/include/nix/flake/flakeref.hh index b557433a9fb8..1f39d62ebbf9 100644 --- a/src/libflake/include/nix/flake/flakeref.hh +++ b/src/libflake/include/nix/flake/flakeref.hh @@ -68,8 +68,7 @@ struct FlakeRef { } - // FIXME: change to operator <<. - std::string to_string() const; + std::string to_string(bool abbreviate = false) const; fetchers::Attrs toAttrs() const; diff --git a/src/libflake/include/nix/flake/lockfile.hh b/src/libflake/include/nix/flake/lockfile.hh index 89029aec491c..27232b20a669 100644 --- a/src/libflake/include/nix/flake/lockfile.hh +++ b/src/libflake/include/nix/flake/lockfile.hh @@ -111,6 +111,7 @@ struct LockedNode : Node { FlakeRef lockedRef, originalRef; bool isFlake = true; + bool buildTime = false; /* The node relative to which relative source paths (e.g. 'path:../foo') are interpreted. */ @@ -120,10 +121,12 @@ struct LockedNode : Node const FlakeRef & lockedRef, const FlakeRef & originalRef, bool isFlake = true, + bool buildTime = false, std::optional parentInputAttrPath = {}) : lockedRef(std::move(lockedRef)) , originalRef(std::move(originalRef)) , isFlake(isFlake) + , buildTime(buildTime) , parentInputAttrPath(std::move(parentInputAttrPath)) { } diff --git a/src/libflake/include/nix/flake/meson.build b/src/libflake/include/nix/flake/meson.build index fc580164eaec..fbe54f41208b 100644 --- a/src/libflake/include/nix/flake/meson.build +++ b/src/libflake/include/nix/flake/meson.build @@ -6,6 +6,7 @@ headers = files( 'flake.hh', 'flakeref.hh', 'lockfile.hh', + 'provenance.hh', 'settings.hh', 'url-name.hh', ) diff --git a/src/libflake/include/nix/flake/provenance.hh b/src/libflake/include/nix/flake/provenance.hh new file mode 100644 index 000000000000..011744f5e65d --- /dev/null +++ b/src/libflake/include/nix/flake/provenance.hh @@ -0,0 +1,21 @@ +#pragma once + +#include "nix/util/provenance.hh" + +namespace nix { + +struct FlakeProvenance : Provenance +{ + std::shared_ptr next; + std::string flakeOutput; + bool pure = true; + + FlakeProvenance(std::shared_ptr next, std::string flakeOutput, bool pure) + : next(std::move(next)) + , flakeOutput(std::move(flakeOutput)) + , pure(pure) {}; + + nlohmann::json to_json() const override; +}; + +} // namespace nix diff --git a/src/libflake/include/nix/flake/settings.hh b/src/libflake/include/nix/flake/settings.hh index 7187a3294a34..05b36f5b779c 100644 --- a/src/libflake/include/nix/flake/settings.hh +++ b/src/libflake/include/nix/flake/settings.hh @@ -21,13 +21,7 @@ struct Settings : public Config void configureEvalSettings(nix::EvalSettings & evalSettings) const; Setting useRegistries{ - this, - true, - "use-registries", - "Whether to use flake registries to resolve flake references.", - {}, - true, - Xp::Flakes}; + this, true, "use-registries", "Whether to use flake registries to resolve flake references.", {}, true}; Setting acceptFlakeConfig{ this, @@ -35,8 +29,7 @@ struct Settings : public Config "accept-flake-config", "Whether to accept Nix configuration settings from a flake without prompting.", {}, - true, - Xp::Flakes}; + true}; Setting commitLockFileSummary{ this, @@ -47,8 +40,7 @@ struct Settings : public Config empty, the summary is generated based on the action performed. )", {"commit-lockfile-summary"}, - true, - Xp::Flakes}; + true}; }; } // namespace nix::flake diff --git a/src/libflake/lockfile.cc b/src/libflake/lockfile.cc index 75127521b942..c44ae77dff91 100644 --- a/src/libflake/lockfile.cc +++ b/src/libflake/lockfile.cc @@ -31,6 +31,7 @@ #include "nix/util/ansicolor.hh" #include "nix/util/error.hh" #include "nix/util/fmt.hh" +#include "nix/util/json-utils.hh" #include "nix/util/logging.hh" #include "nix/util/ref.hh" #include "nix/util/types.hh" @@ -66,6 +67,7 @@ LockedNode::LockedNode(const fetchers::Settings & fetchSettings, const nlohmann: : lockedRef(getFlakeRef(fetchSettings, json, "locked", "info")) // FIXME: remove "info" , originalRef(getFlakeRef(fetchSettings, json, "original", nullptr)) , isFlake(json.find("flake") != json.end() ? (bool) json["flake"] : true) + , buildTime(json.find("buildTime") != json.end() ? (bool) json["buildTime"] : false) , parentInputAttrPath( json.find("parent") != json.end() ? (std::optional) json["parent"] : std::nullopt) { @@ -224,13 +226,11 @@ std::pair LockFile::toJSON() const if (auto lockedNode = node.dynamic_pointer_cast()) { n["original"] = fetchers::attrsToJSON(lockedNode->originalRef.toAttrs()); n["locked"] = fetchers::attrsToJSON(lockedNode->lockedRef.toAttrs()); - /* For backward compatibility, omit the "__final" - attribute. We never allow non-final inputs in lock files - anyway. */ assert(lockedNode->lockedRef.input.isFinal() || lockedNode->lockedRef.input.isRelative()); - n["locked"].erase("__final"); if (!lockedNode->isFlake) n["flake"] = false; + if (lockedNode->buildTime) + n["buildTime"] = true; if (lockedNode->parentInputAttrPath) n["parent"] = *lockedNode->parentInputAttrPath; } @@ -347,7 +347,7 @@ std::map LockFile::getAllInputs() const static std::string describe(const FlakeRef & flakeRef) { - auto s = fmt("'%s'", flakeRef.to_string()); + auto s = fmt("'%s'", flakeRef.to_string(true)); if (auto lastModified = flakeRef.input.getLastModified()) s += fmt(" (%s)", std::put_time(std::gmtime(&*lastModified), "%Y-%m-%d")); diff --git a/src/libflake/make.nix b/src/libflake/make.nix new file mode 100644 index 000000000000..f35007865d4e --- /dev/null +++ b/src/libflake/make.nix @@ -0,0 +1,33 @@ +# libnixflake, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-store, + nix-fetchers, + nix-expr, +}: + +nixMake.mkComponent { + name = "determinate-nix-flake"; + libName = "nixflake"; + + deps = [ + nix-util + nix-store + nix-fetchers + nix-expr + ]; + + root = ./.; + + includeDirs = [ + "" + "include" + ]; + + files = nixMake.commonSupportFiles // { + "call-flake.nix.gen.hh" = nixMake.mkStringHeader ./call-flake.nix; + }; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libflake/meson.build b/src/libflake/meson.build index c06bf6ba5450..1bf6f7e5fba6 100644 --- a/src/libflake/meson.build +++ b/src/libflake/meson.build @@ -45,6 +45,7 @@ sources = files( 'flake.cc', 'flakeref.cc', 'lockfile.cc', + 'provenance.cc', 'settings.cc', 'url-name.cc', ) @@ -58,11 +59,10 @@ this_library = library( 'nixflake', sources, generated_headers, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, link_args : linker_export_flags, - prelink : true, # For C++ static initializers + prelink : prelink, # For C++ static initializers install : true, ) diff --git a/src/libflake/package.nix b/src/libflake/package.nix index dd442a44ec9a..2b0c827a09ce 100644 --- a/src/libflake/package.nix +++ b/src/libflake/package.nix @@ -18,7 +18,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-flake"; + pname = "determinate-nix-flake"; inherit version; workDir = ./.; diff --git a/src/libflake/provenance.cc b/src/libflake/provenance.cc new file mode 100644 index 000000000000..c80c4154561a --- /dev/null +++ b/src/libflake/provenance.cc @@ -0,0 +1,28 @@ +#include "nix/flake/provenance.hh" +#include "nix/util/json-utils.hh" + +#include + +namespace nix { + +nlohmann::json FlakeProvenance::to_json() const +{ + return nlohmann::json{ + {"type", "flake"}, + {"next", next ? next->to_json() : nlohmann::json(nullptr)}, + {"flakeOutput", flakeOutput}, + {"pure", pure}}; +} + +Provenance::Register registerFlakeProvenance("flake", [](nlohmann::json json) { + auto & obj = getObject(json); + std::shared_ptr next; + if (auto p = optionalValueAt(obj, "next"); p && !p->is_null()) + next = Provenance::from_json(*p); + bool pure = true; + if (auto p = optionalValueAt(obj, "pure")) + pure = getBoolean(*p); + return make_ref(next, getString(valueAt(obj, "flakeOutput")), pure); +}); + +} // namespace nix diff --git a/src/libmain-c/meson.build b/src/libmain-c/meson.build index 998151142fda..5b9106f455b9 100644 --- a/src/libmain-c/meson.build +++ b/src/libmain-c/meson.build @@ -46,7 +46,6 @@ subdir('nix-meson-build-support/windows-version') build_both_libraries = get_option('plugin-c-api') library_kwargs = { - 'soversion' : nix_soversion, 'dependencies' : deps_public + deps_private + deps_other, 'include_directories' : include_dirs, 'link_args' : linker_export_flags, diff --git a/src/libmain-c/package.nix b/src/libmain-c/package.nix index 5f86df5ea577..dd2faa913300 100644 --- a/src/libmain-c/package.nix +++ b/src/libmain-c/package.nix @@ -18,7 +18,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-main-c"; + pname = "determinate-nix-main-c"; inherit version; workDir = ./.; diff --git a/src/libmain/include/nix/main/shared.hh b/src/libmain/include/nix/main/shared.hh index 7b0e7a54d4e0..0416fc080343 100644 --- a/src/libmain/include/nix/main/shared.hh +++ b/src/libmain/include/nix/main/shared.hh @@ -26,6 +26,8 @@ void parseCmdLine( const Strings & args, fun parseArg); +std::string version(); + [[noreturn]] void printVersion(const std::string & programName); /** diff --git a/src/libmain/make.nix b/src/libmain/make.nix new file mode 100644 index 000000000000..dee739ddd4b0 --- /dev/null +++ b/src/libmain/make.nix @@ -0,0 +1,37 @@ +# libnixmain, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-store, + nix-expr, +}: + +nixMake.mkComponent { + name = "determinate-nix-main"; + libName = "nixmain"; + + deps = [ + nix-util + nix-store + # Only for the NIX_USE_BOEHMGC macro, as in the Meson build. + nix-expr + ]; + + root = ./.; + + # unix/stack.cc is a plain source file; there is no windows/ directory. + includeDirs = [ + "" + "include" + ]; + + files = nixMake.commonSupportFiles; + + configHeaders = { + "main-config-private.hh" = { + HAVE_PUBSETBUF = 1; + }; + }; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libmain/meson.build b/src/libmain/meson.build index 0084643bdaad..97b89b9fa79a 100644 --- a/src/libmain/meson.build +++ b/src/libmain/meson.build @@ -77,11 +77,10 @@ this_library = library( 'nixmain', sources, config_priv_h, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, link_args : linker_export_flags, - prelink : true, # For C++ static initializers + prelink : prelink, # For C++ static initializers install : true, ) diff --git a/src/libmain/package.nix b/src/libmain/package.nix index 7b0a4dee7dad..119e1f1aca59 100644 --- a/src/libmain/package.nix +++ b/src/libmain/package.nix @@ -18,7 +18,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-main"; + pname = "determinate-nix-main"; inherit version; workDir = ./.; diff --git a/src/libmain/progress-bar.cc b/src/libmain/progress-bar.cc index 512ba4faad67..4d2a36672afc 100644 --- a/src/libmain/progress-bar.cc +++ b/src/libmain/progress-bar.cc @@ -19,16 +19,20 @@ namespace { static std::string_view getS(const std::vector & fields, size_t n) { - if (n >= fields.size() || fields[n].type != Logger::Field::tString) - throw Error("could not get expected log field of type 'string' at index %d", n); - return fields[n].s; + if (n < fields.size()) { + if (auto p = std::get_if(&fields[n].raw)) + return *p; + } + throw Error("could not get expected log field of type 'string' at index %d", n); } static uint64_t getI(const std::vector & fields, size_t n) { - if (n >= fields.size() || fields[n].type != Logger::Field::tInt) - throw Error("could not get expected log field of type 'int' at index %d", n); - return fields[n].i; + if (n < fields.size()) { + if (auto p = std::get_if(&fields[n].raw)) + return *p; + } + throw Error("could not get expected log field of type 'int' at index %d", n); } static std::string_view storePathToName(std::string_view path) @@ -62,6 +66,7 @@ class ProgressBar final : public Logger ActivityId parent; std::optional name; std::chrono::time_point startTime; + bool logged = false; }; struct ActivitiesByType @@ -105,7 +110,7 @@ class ProgressBar final : public Logger bool printBuildLogs = false; bool isTTY; - std::unique_ptr interruptCallback; + std::unique_ptr interruptCallback, stopCallback, contCallback, winchCallback; void hideCursorIfNeeded() const { @@ -130,6 +135,30 @@ class ProgressBar final : public Logger { hideCursorIfNeeded(); state_.lock()->active = isTTY; + + /* On Ctrl-Z, unhide the cursor before the process is + stopped. */ + stopCallback = createSignalCallback(SignalType::Stop, [&]() { + auto state(state_.lock()); + if (state->active && !state->isPaused()) + unhideCursorIfNeeded(); + }); + + /* Redraw the progress bar when the process is resumed or the + terminal size changes. */ + auto redrawCallback = [&]() { + auto state(state_.lock()); + if (state->active && !state->isPaused()) { + /* Force a redraw, since the new output may be + identical to the one cached by redraw(). */ + invalidateRedrawCache(); + hideCursorIfNeeded(); + update(*state); + } + }; + contCallback = createSignalCallback(SignalType::Cont, redrawCallback); + winchCallback = createSignalCallback(SignalType::Winch, redrawCallback); + updateThread = std::thread([&]() { auto state(state_.lock()); auto nextWakeup = std::chrono::milliseconds::max(); @@ -175,6 +204,11 @@ class ProgressBar final : public Logger if (state->active) { clearProgressDisplay(); + /* Show activities that were previously only shown on the + progress bar. Otherwise the user won't know what's + happening. */ + for (auto & act : state->activities) + logActivity(*state, lvlNotice, act); unhideCursorIfNeeded(); } } @@ -203,7 +237,7 @@ class ProgressBar final : public Logger return printBuildLogs; } - void log(Verbosity lvl, std::string_view s) override + void log(Verbosity lvl, std::string_view s) noexcept override { if (lvl > verbosity) return; @@ -211,7 +245,7 @@ class ProgressBar final : public Logger log(*state, lvl, s); } - void logEI(const ErrorInfo & ei) override + void logEI(const ErrorInfo & ei) noexcept override { auto state(state_.lock()); @@ -221,7 +255,7 @@ class ProgressBar final : public Logger log(*state, ei.level, oss.view()); } - void log(State & state, Verbosity lvl, std::string_view s) + void log(State & state, Verbosity lvl, std::string_view s) noexcept { if (state.active) { invalidateRedrawCache(); @@ -232,37 +266,38 @@ class ProgressBar final : public Logger } } + void logActivity(State & state, Verbosity lvl, ActInfo & act) noexcept + { + if (!act.logged && lvl <= verbosity && !act.s.empty() && act.type != actBuildWaiting) { + log(state, lvl, act.s + "..."); + act.logged = true; + } + } + void startActivity( ActivityId act, Verbosity lvl, ActivityType type, const std::string & s, const Fields & fields, - ActivityId parent) override + ActivityId parent) noexcept override { auto state(state_.lock()); - if (lvl <= verbosity && !s.empty() && type != actBuildWaiting) - log(*state, lvl, s + "..."); - state->activities.emplace_back( ActInfo{.s = s, .type = type, .parent = parent, .startTime = std::chrono::steady_clock::now()}); auto i = std::prev(state->activities.end()); state->its.emplace(act, i); state->activitiesByType[type].its.emplace(act, i); + logActivity(*state, lvl, *i); + if (type == actBuild) { auto name = storePathToNameWithoutDrvSuffix(getS(fields, 0)); i->s = fmt("building " ANSI_BOLD "%s" ANSI_NORMAL, name); auto machineName = getS(fields, 1); if (machineName != "") i->s += fmt(" on " ANSI_BOLD "%s" ANSI_NORMAL, machineName); - - // Used to be curRound and nrRounds, but the - // implementation was broken for a long time. - if (getI(fields, 2) != 1 || getI(fields, 3) != 1) { - throw Error("log message indicated repeating builds, but this is not currently implemented"); - } i->name = DrvName(name).name; } @@ -310,7 +345,7 @@ class ProgressBar final : public Logger return false; } - void stopActivity(ActivityId act) override + void stopActivity(ActivityId act) noexcept override { auto state(state_.lock()); @@ -332,7 +367,7 @@ class ProgressBar final : public Logger update(*state); } - void result(ActivityId act, ResultType type, const std::vector & fields) override + void result(ActivityId act, ResultType type, const std::vector & fields) noexcept override { auto state(state_.lock()); @@ -411,7 +446,7 @@ class ProgressBar final : public Logger } } - void update(State & state) + void update(State & state) noexcept { state.haveUpdate = true; updateCV.notify_one(); @@ -424,7 +459,7 @@ class ProgressBar final : public Logger * with text selection in some terminals, including libvte-based terminal * emulators. */ - void redraw(std::string newOutput) + void redraw(std::string newOutput) noexcept { auto lastOutput(lastOutput_.lock()); if (newOutput != *lastOutput) { @@ -444,7 +479,7 @@ class ProgressBar final : public Logger writeToStderr("\r\e[K"); } - std::chrono::milliseconds draw(State & state) + std::chrono::milliseconds draw(State & state) noexcept { auto nextWakeup = std::chrono::milliseconds::max(); @@ -499,16 +534,12 @@ class ProgressBar final : public Logger } } - auto width = getWindowSize().second; - if (width <= 0) - width = std::numeric_limits::max(); - - redraw("\r" + filterANSIEscapes(line, false, width) + ANSI_NORMAL + "\e[K"); + redraw("\r" + filterANSIEscapes(line, false, getWindowWidth()) + ANSI_NORMAL + "\e[K"); return nextWakeup; } - std::string getStatus(State & state) + std::string getStatus(State & state) noexcept { std::string res; diff --git a/src/libmain/shared.cc b/src/libmain/shared.cc index f1d419af5ed1..8cf14890036c 100644 --- a/src/libmain/shared.cc +++ b/src/libmain/shared.cc @@ -196,6 +196,13 @@ void initNix(bool loadConfig) if (sigaction(SIGWINCH, &act, 0)) throw SysError("handling SIGWINCH"); + /* Same for SIGCONT and SIGTSTP, which are also handled by + * signalHandlerThread. */ + if (sigaction(SIGCONT, &act, 0)) + throw SysError("handling SIGCONT"); + if (sigaction(SIGTSTP, &act, 0)) + throw SysError("handling SIGTSTP"); + /* Disable SA_RESTART for interrupts, so that system calls on this thread * error with EINTR like they do on Linux. * Most signals on BSD systems default to SA_RESTART on, but Nix @@ -209,8 +216,6 @@ void initNix(bool loadConfig) throw SysError("handling SIGHUP"); if (sigaction(SIGPIPE, &act, 0)) throw SysError("handling SIGPIPE"); - if (sigaction(SIGQUIT, &act, 0)) - throw SysError("handling SIGQUIT"); if (sigaction(SIGTRAP, &act, 0)) throw SysError("handling SIGTRAP"); #endif @@ -336,9 +341,14 @@ void parseCmdLine( LegacyArgs(programName, parseArg).parseCmdline(args); } +std::string version() +{ + return fmt("(Determinate Nix %s) %s", determinateNixVersion, nixVersion); +} + void printVersion(const std::string & programName) { - std::cout << fmt("%1% (Nix) %2%", programName, nixVersion) << std::endl; + std::cout << fmt("%s %s", programName, version()) << std::endl; if (verbosity > lvlInfo) { Strings cfg; #if NIX_USE_BOEHMGC diff --git a/src/libmain/unix/stack.cc b/src/libmain/unix/stack.cc index bec0d389f5ed..e049bae50de1 100644 --- a/src/libmain/unix/stack.cc +++ b/src/libmain/unix/stack.cc @@ -10,6 +10,8 @@ namespace nix { +static struct sigaction savedSigsegvAction; + static void sigsegvHandler(int signo, siginfo_t * info, void * ctx) { /* Detect stack overflows by comparing the faulting address with @@ -34,12 +36,8 @@ static void sigsegvHandler(int signo, siginfo_t * info, void * ctx) } } - /* Restore default behaviour (i.e. segfault and dump core). */ - struct sigaction act; - sigfillset(&act.sa_mask); - act.sa_handler = SIG_DFL; - act.sa_flags = 0; - if (sigaction(SIGSEGV, &act, 0)) + /* Restore the original SIGSEGV handler. */ + if (sigaction(SIGSEGV, &savedSigsegvAction, 0)) abort(); } @@ -63,7 +61,7 @@ void detectStackOverflow() sigfillset(&act.sa_mask); act.sa_sigaction = sigsegvHandler; act.sa_flags = SA_SIGINFO | SA_ONSTACK; - if (sigaction(SIGSEGV, &act, 0)) + if (sigaction(SIGSEGV, &act, &savedSigsegvAction)) throw SysError("resetting SIGSEGV"); #endif } diff --git a/src/libstore-c/make.nix b/src/libstore-c/make.nix new file mode 100644 index 000000000000..1a6baf4bc8cc --- /dev/null +++ b/src/libstore-c/make.nix @@ -0,0 +1,30 @@ +# libnixstorec, the C API of libnixstore, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-util-c, + nix-store, +}: + +nixMake.mkComponent { + name = "determinate-nix-store-c"; + libName = "nixstorec"; + + deps = [ + nix-util + nix-util-c + nix-store + ]; + + root = ./.; + + includeDirs = [ "" ]; + # The headers live in the root, which the default would not export. + publicIncludeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + extraCxxFlags = nixMake.weakVtablesFlags; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libstore-c/meson.build b/src/libstore-c/meson.build index 600de4d2ea61..4099e6931316 100644 --- a/src/libstore-c/meson.build +++ b/src/libstore-c/meson.build @@ -49,7 +49,6 @@ subdir('nix-meson-build-support/windows-version') build_both_libraries = get_option('plugin-c-api') library_kwargs = { - 'soversion' : nix_soversion, 'dependencies' : deps_public + deps_private + deps_other, 'include_directories' : include_dirs, 'link_args' : linker_export_flags, diff --git a/src/libstore-c/nix_api_store.cc b/src/libstore-c/nix_api_store.cc index fbb3c418566c..f0d8aeab3aa1 100644 --- a/src/libstore-c/nix_api_store.cc +++ b/src/libstore-c/nix_api_store.cc @@ -289,6 +289,29 @@ nix_derivation * nix_derivation_from_json(nix_c_context * context, Store * store NIXC_CATCH_ERRS_NULL } +nix_err nix_derivation_make_outputs( + nix_c_context * context, + Store * store, + const char * json, + void (*callback)(void * userdata, const char * output_name, const char * path), + void * userdata) +{ + if (context) + context->last_err_code = NIX_OK; + try { + auto drv = nix::Derivation::parseJsonAndValidate(*store->ptr, nlohmann::json::parse(json)); + + for (auto & output : drv.outputs) { + auto outPath = output.second.path(*store->ptr, drv.name, output.first); + + if (callback && outPath) { + callback(userdata, output.first.c_str(), store->ptr->printStorePath(*outPath).c_str()); + } + } + } + NIXC_CATCH_ERRS +} + nix_err nix_derivation_to_json( nix_c_context * context, const nix_derivation * drv, nix_get_string_callback callback, void * userdata) { @@ -342,6 +365,96 @@ nix_derivation * nix_store_drv_from_store_path(nix_c_context * context, Store * NIXC_CATCH_ERRS_NULL } +nix_err nix_store_drv_from_path( + nix_c_context * context, + Store * store, + const StorePath * path, + void (*callback)(void * userdata, const nix_derivation * drv), + void * userdata) +{ + if (context) + context->last_err_code = NIX_OK; + try { + nix::Derivation drv = store->ptr->derivationFromPath(path->path); + if (callback) { + const nix_derivation tmp{drv, store}; + callback(userdata, &tmp); + } + } + NIXC_CATCH_ERRS +} + +nix_err nix_store_query_path_info( + nix_c_context * context, + Store * store, + const StorePath * store_path, + void * userdata, + nix_get_string_callback callback) +{ + if (context) + context->last_err_code = NIX_OK; + try { + auto info = store->ptr->queryPathInfo(store_path->path); + if (callback) { + auto result = info->toJSON(&store->ptr->config, true, nix::PathInfoJsonFormat::V1).dump(); + callback(result.data(), result.size(), userdata); + } + } + NIXC_CATCH_ERRS +} + +nix_err nix_store_build_paths( + nix_c_context * context, + Store * store, + const StorePath ** store_paths, + unsigned int num_store_paths, + void (*callback)(void * userdata, const char * path, const char * result), + void * userdata) +{ + if (context) + context->last_err_code = NIX_OK; + try { + std::vector derived_paths; + for (size_t i = 0; i < num_store_paths; i++) { + const StorePath * store_path = store_paths[i]; + derived_paths.push_back(nix::SingleDerivedPath::Opaque{store_path->path}); + } + + auto results = store->ptr->buildPathsWithResults(derived_paths); + for (auto & result : results) { + if (callback) + callback( + userdata, result.path.to_string(store->ptr->config).c_str(), nlohmann::json(result).dump().c_str()); + } + } + NIXC_CATCH_ERRS +} + +nix_err nix_derivation_get_outputs_and_optpaths( + nix_c_context * context, + const nix_derivation * drv, + const Store * store, + void (*callback)(void * userdata, const char * name, const StorePath * path), + void * userdata) +{ + if (context) + context->last_err_code = NIX_OK; + try { + auto value = drv->drv.outputsAndOptPaths(store->ptr->config); + if (callback) { + for (const auto & [name, result] : value) { + if (auto store_path = result.second) { + const StorePath tmp_path{*store_path}; + callback(userdata, name.c_str(), &tmp_path); + } else { + callback(userdata, name.c_str(), nullptr); + } + } + } + } + NIXC_CATCH_ERRS +} + StorePath * nix_store_query_path_from_hash_part(nix_c_context * context, Store * store, const char * hash) { if (context) diff --git a/src/libstore-c/nix_api_store.h b/src/libstore-c/nix_api_store.h index e60a7d6c0f7c..964c4066154b 100644 --- a/src/libstore-c/nix_api_store.h +++ b/src/libstore-c/nix_api_store.h @@ -201,6 +201,22 @@ nix_store_get_version(nix_c_context * context, Store * store, nix_get_string_cal */ nix_derivation * nix_derivation_from_json(nix_c_context * context, Store * store, const char * json); +/** + * @brief Hashes the derivation and gives the output paths + * + * @param[in] context Optional, stores error information. + * @param[in] store nix store reference. + * @param[in] json JSON of the derivation as a string. + * @param[in] callback Called for every output to provide the output path. + * @param[in] userdata User data to pass to the callback. + */ +nix_err nix_derivation_make_outputs( + nix_c_context * context, + Store * store, + const char * json, + void (*callback)(void * userdata, const char * output_name, const char * path), + void * userdata); + /** * @brief Add the given `nix_derivation` to the given store * @@ -259,6 +275,88 @@ nix_err nix_store_get_fs_closure( */ nix_derivation * nix_store_drv_from_store_path(nix_c_context * context, Store * store, const StorePath * path); +/** + * @note The callback borrows the Derivation only for the duration of the call. + * + * @param[out] context Optional, stores error information + * @param[in] store The nix store + * @param[in] path The nix store path + * @param[in] callback The callback to call + * @param[in] userdata The userdata to pass to the callback + */ +nix_err nix_store_drv_from_path( + nix_c_context * context, + Store * store, + const StorePath * path, + void (*callback)(void * userdata, const nix_derivation * drv), + void * userdata); + +/** + * @brief Queries for the nix store path info. + * + * @param[out] context Optional, stores error information + * @param[in] store nix store reference + * @param[in] store_path A store path + * @param[in] userdata The data to pass to the callback + * @param[in] callback Called for when the path info is resolved + */ +nix_err nix_store_query_path_info( + nix_c_context * context, + Store * store, + const StorePath * store_path, + void * userdata, + nix_get_string_callback callback); + +/** + * @brief Builds the paths, if they are a derivation then they get built. + * + * @note Path and result for the callback only exist for the lifetime of + * the call. Result is a string containing the build result in JSON. + * + * @param[out] context Optional, stores error information + * @param[in] store nix store reference + * @param[in] store_paths Pointer to list of nix store paths + * @param[in] num_store_paths Number of nix store paths + * @param[in] callback The callback to trigger for build results + * @param[in] userdata User data to pass to the callback + */ +nix_err nix_store_build_paths( + nix_c_context * context, + Store * store, + const StorePath ** store_paths, + unsigned int num_store_paths, + void (*callback)(void * userdata, const char * path, const char * result), + void * userdata); + +/** + * @brief Iterate and get all of the store paths for each output. + * + * @note The callback borrows the StorePath only for the duration of the call. + * + * @param[out] context Optional, stores error information + * @param[in] drv The derivation + * @param[in] store The nix store + * @param[in] callback The function to call on every output and store path + * @param[in] userdata The userdata to pass to the callback + */ +nix_err nix_derivation_get_outputs_and_optpaths( + nix_c_context * context, + const nix_derivation * drv, + const Store * store, + void (*callback)(void * userdata, const char * name, const StorePath * path), + void * userdata); + +/** + * @brief Gets the derivation as a JSON string + * + * @param[out] context Optional, stores error information + * @param[in] drv The derivation + * @param[in] callback Called with the JSON string + * @param[in] userdata Arbitrary data passed to the callback + */ +nix_err nix_derivation_to_json( + nix_c_context * context, const nix_derivation * drv, nix_get_string_callback callback, void * userdata); + /** * @brief Query the full store path given the hash part of a valid store * path, or empty if no matching path is found. diff --git a/src/libstore-c/nix_api_store_internal.h b/src/libstore-c/nix_api_store_internal.h index 712d96488a57..0199628da8a9 100644 --- a/src/libstore-c/nix_api_store_internal.h +++ b/src/libstore-c/nix_api_store_internal.h @@ -18,6 +18,7 @@ struct StorePath struct nix_derivation { nix::Derivation drv; + Store * store; }; } // extern "C" diff --git a/src/libstore-c/package.nix b/src/libstore-c/package.nix index 822acb00fc70..ecfc278cf660 100644 --- a/src/libstore-c/package.nix +++ b/src/libstore-c/package.nix @@ -16,7 +16,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-store-c"; + pname = "determinate-nix-store-c"; inherit version; workDir = ./.; diff --git a/src/libstore-test-support/make.nix b/src/libstore-test-support/make.nix new file mode 100644 index 000000000000..9c14f7917e68 --- /dev/null +++ b/src/libstore-test-support/make.nix @@ -0,0 +1,29 @@ +# libnix-store-test-support, helpers for the unit tests of libnixstore and +# the components built on it, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-util-test-support, + nix-store, + nix-store-c, +}: + +nixMake.mkComponent { + name = "determinate-nix-store-test-support"; + libName = "nix-store-test-support"; + + deps = [ + nix-util + nix-util-test-support + nix-store + nix-store-c + ]; + + root = ./.; + + includeDirs = [ "include" ]; + + files = nixMake.commonSupportFiles; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libstore-test-support/meson.build b/src/libstore-test-support/meson.build index ca451db4abce..79f3c8235d2e 100644 --- a/src/libstore-test-support/meson.build +++ b/src/libstore-test-support/meson.build @@ -50,13 +50,12 @@ subdir('nix-meson-build-support/windows-version') this_library = library( 'nix-store-test-support', sources, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, # TODO: Remove `-lrapidcheck` when https://github.com/emil-e/rapidcheck/pull/326 # is available. See also ../libutil/build.meson link_args : linker_export_flags + [ '-lrapidcheck' ], - prelink : true, # For C++ static initializers + prelink : prelink, # For C++ static initializers install : true, ) diff --git a/src/libstore-test-support/package.nix b/src/libstore-test-support/package.nix index 4f20f2cbe716..bd5a4c0aa7a0 100644 --- a/src/libstore-test-support/package.nix +++ b/src/libstore-test-support/package.nix @@ -19,7 +19,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-store-test-support"; + pname = "determinate-nix-store-test-support"; inherit version; workDir = ./.; diff --git a/src/libstore-tests/make.nix b/src/libstore-tests/make.nix new file mode 100644 index 000000000000..c1ab2f7f31b3 --- /dev/null +++ b/src/libstore-tests/make.nix @@ -0,0 +1,73 @@ +# The unit tests of libnixstore, transcribed from meson.build and +# package.nix: the `nix-store-tests` executable, with the derivation that +# runs it as `tests.run`. +{ + pkgs, + nixMake, + nix-store, + nix-store-c, + nix-store-test-support, +}: + +let + inherit (pkgs) lib; + + tests = nixMake.mkComponent { + name = "determinate-nix-store-tests"; + type = "executable"; + exeName = "nix-store-tests"; + + deps = [ + nix-store + nix-store-c + nix-store-test-support + ]; + + root = ./.; + + # The benchmarks (Meson option `benchmarks`, off by default). + excludeSources = [ + "bench-main.cc" + "derivation-parser-bench.cc" + "ref-scan-bench.cc" + "register-valid-paths-bench.cc" + ]; + + includeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + configHeaders = { + "store-tests-config.hh" = { + PACKAGE_VERSION = lib.fileContents ../../.version; + NIX_STORE_DIR = "/nix/store"; + }; + }; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; + }; + + # Some data is shared with the functional tests: they create it, we + # consume it. Rooted at the repository so that both are present. + data = lib.fileset.toSource { + root = ../..; + fileset = lib.fileset.unions [ + ./data + ../../tests/functional/derivation + ]; + }; +in + +tests +// { + tests.run = nixMake.runTest { + name = "${tests.name}-run"; + command = [ "${tests}/bin/${tests.exeName}" ]; + env = { + _NIX_TEST_UNIT_DATA = "${data}/src/libstore-tests/data"; + NIX_REMOTE = "$HOME/store"; + }; + # The HTTPS binary cache tests generate certificates with `openssl`. + nativeBuildInputs = [ pkgs.openssl ]; + }; +} diff --git a/src/libstore-tests/nar-info-disk-cache.cc b/src/libstore-tests/nar-info-disk-cache.cc index 7612250c661d..b54c98fe5e7d 100644 --- a/src/libstore-tests/nar-info-disk-cache.cc +++ b/src/libstore-tests/nar-info-disk-cache.cc @@ -15,6 +15,13 @@ TEST(NarInfoDiskCacheImpl, create_and_read) int prio = 12345; bool wantMassQuery = true; + auto mkFields = [](bool wantMassQuery, int prio) { + return std::map{ + {"WantMassQuery", wantMassQuery ? "1" : "0"}, + {"Priority", std::to_string(prio)}, + }; + }; + auto tmpDir = createTempDir(); AutoDelete delTmpDir(tmpDir); auto dbPath(tmpDir / "test-narinfo-disk-cache.sqlite"); @@ -30,22 +37,19 @@ TEST(NarInfoDiskCacheImpl, create_and_read) // Set up "background noise" and check that different caches receive different ids { - auto bc1 = - cache->createCache("https://bar", "/nix/storedir", {.wantMassQuery = wantMassQuery, .priority = prio}); - auto bc2 = cache->createCache("https://xyz", "/nix/storedir", {.priority = 12}); + auto bc1 = cache->createCache("https://bar", "/nix/storedir", {.fields = mkFields(wantMassQuery, prio)}); + auto bc2 = cache->createCache("https://xyz", "/nix/storedir", {.fields = mkFields(false, 12)}); ASSERT_NE(bc1, bc2); barId = bc1; } // Check that the fields are saved and returned correctly. This does not test // the select statement yet, because of in-memory caching. - savedId = cache->createCache("http://foo", "/nix/storedir", {.wantMassQuery = wantMassQuery, .priority = prio}); - ; + savedId = cache->createCache("http://foo", "/nix/storedir", {.fields = mkFields(wantMassQuery, prio)}); { auto r = cache->upToDateCacheExists("http://foo"); ASSERT_TRUE(r); - ASSERT_EQ(r->priority, prio); - ASSERT_EQ(r->wantMassQuery, wantMassQuery); + ASSERT_EQ(r->fields, mkFields(wantMassQuery, prio)); ASSERT_EQ(savedId, r->id); } @@ -68,8 +72,7 @@ TEST(NarInfoDiskCacheImpl, create_and_read) { auto r = cache->upToDateCacheExists("http://foo"); ASSERT_TRUE(r); - ASSERT_EQ(r->priority, prio); - ASSERT_EQ(r->wantMassQuery, wantMassQuery); + ASSERT_EQ(r->fields, mkFields(wantMassQuery, prio)); } } @@ -85,13 +88,12 @@ TEST(NarInfoDiskCacheImpl, create_and_read) } // "Update", same data, check that the id number is reused - cache2->createCache("http://foo", "/nix/storedir", {.wantMassQuery = wantMassQuery, .priority = prio}); + cache2->createCache("http://foo", "/nix/storedir", {.fields = mkFields(wantMassQuery, prio)}); { auto r = cache2->upToDateCacheExists("http://foo"); ASSERT_TRUE(r); - ASSERT_EQ(r->priority, prio); - ASSERT_EQ(r->wantMassQuery, wantMassQuery); + ASSERT_EQ(r->fields, mkFields(wantMassQuery, prio)); ASSERT_EQ(r->id, savedId); } @@ -108,11 +110,9 @@ TEST(NarInfoDiskCacheImpl, create_and_read) auto r0 = cache2->upToDateCacheExists("https://bar"); ASSERT_FALSE(r0); - cache2->createCache( - "https://bar", "/nix/storedir", {.wantMassQuery = !wantMassQuery, .priority = prio + 10}); + cache2->createCache("https://bar", "/nix/storedir", {.fields = mkFields(!wantMassQuery, prio + 10)}); auto r = cache2->upToDateCacheExists("https://bar"); - ASSERT_EQ(r->wantMassQuery, !wantMassQuery); - ASSERT_EQ(r->priority, prio + 10); + ASSERT_EQ(r->fields, mkFields(!wantMassQuery, prio + 10)); ASSERT_EQ(r->id, barId); } diff --git a/src/libstore-tests/nix_api_store.cc b/src/libstore-tests/nix_api_store.cc index 0162684daf4b..0c4739bbc91f 100644 --- a/src/libstore-tests/nix_api_store.cc +++ b/src/libstore-tests/nix_api_store.cc @@ -977,4 +977,55 @@ TEST_F(nix_api_store_test, nix_derivation_clone) nix_derivation_free(drv2); } +TEST_F(nix_api_store_test, nix_store_build_paths) +{ + nix::experimentalFeatureSettings.set("extra-experimental-features", "ca-derivations"); + nix::settings.getWorkerSettings().substituters = {}; + + auto * store = open_local_store(); + + std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")}; + std::ifstream t{unitTestData / "derivation/ca/self-contained.json"}; + std::stringstream buffer; + buffer << t.rdbuf(); + + // Replace the hardcoded system with the current system + std::string jsonStr = nix::replaceStrings(buffer.str(), "x86_64-linux", nix::settings.thisSystem.get()); + + auto * drv = nix_derivation_from_json(ctx, store, jsonStr.c_str()); + assert_ctx_ok(); + ASSERT_NE(drv, nullptr); + + auto * drvPath = nix_add_derivation(ctx, store, drv); + assert_ctx_ok(); + ASSERT_NE(drv, nullptr); + + // Realise the derivation - capture the order outputs are returned + std::map outputs; + std::vector output_order; + auto cb = LambdaAdapter{.fun = [&](const char * path, const char * result) { + ASSERT_NE(path, nullptr); + ASSERT_NE(result, nullptr); + output_order.push_back(path); + outputs.emplace(path, result); + }}; + + std::vector paths = {drvPath}; + + auto ret = nix_store_build_paths( + ctx, + store, + const_cast(paths.data()), + paths.size(), + decltype(cb)::call_void, + static_cast(&cb)); + assert_ctx_ok(); + ASSERT_EQ(ret, NIX_OK); + ASSERT_EQ(outputs.size(), 1); + + nix_store_path_free(drvPath); + nix_derivation_free(drv); + nix_store_free(store); +} + } // namespace nixC diff --git a/src/libstore-tests/realisation.cc b/src/libstore-tests/realisation.cc index 814b6713664a..cec9db9087ec 100644 --- a/src/libstore-tests/realisation.cc +++ b/src/libstore-tests/realisation.cc @@ -35,8 +35,8 @@ DrvOutput testDrvOutput{ .outputName = "foo", }; -const SecretKey testSecretKey{ - "test-key:tU7tTvLcScf8pmz/eTV0BEtLmRsPpZfKaRcd0nCN+pysBZPHSeg61/u2oc7mIOewfuAY1V1BiX32homTaDJ2Jw=="}; +auto testSecretKey = + "test-key:tU7tTvLcScf8pmz/eTV0BEtLmRsPpZfKaRcd0nCN+pysBZPHSeg61/u2oc7mIOewfuAY1V1BiX32homTaDJ2Jw=="; Realisation simple{ unkeyedSimple, @@ -164,7 +164,7 @@ TEST_P(RealisationSigningTest, sign) { const auto & [name, realisation] = GetParam(); - LocalSigner signer(SecretKey{testSecretKey}); + LocalSigner signer(SecretKey::parse(testSecretKey)); auto sig = realisation.sign(realisation.id, signer); @@ -175,9 +175,9 @@ TEST_P(RealisationSigningTest, verify) { const auto & [name, realisation] = GetParam(); - auto publicKey = testSecretKey.toPublicKey(); + auto publicKey = SecretKey::parse(testSecretKey)->toPublicKey(); PublicKeys publicKeys; - publicKeys.insert_or_assign(publicKey.name, publicKey); + publicKeys.insert_or_assign(publicKey->name, std::move(publicKey)); readTest(std::string{name} + "-sig.json", [&](const auto & encoded) { Signature sig = json::parse(encoded); @@ -189,13 +189,13 @@ TEST_P(RealisationSigningTest, verify_rejects_wrong_key) { const auto & [name, realisation] = GetParam(); - auto wrongKey = SecretKey::generate("wrong-key"); - auto wrongPublicKey = wrongKey.toPublicKey(); + auto wrongKey = SecretKey::generate("wrong-key", KeyType::Ed25519); + auto wrongPublicKey = wrongKey->toPublicKey(); PublicKeys publicKeys; - publicKeys.insert_or_assign(wrongPublicKey.name, wrongPublicKey); + publicKeys.insert_or_assign(wrongPublicKey->name, std::move(wrongPublicKey)); auto r = static_cast(realisation); - LocalSigner signer(SecretKey{testSecretKey}); + LocalSigner signer(SecretKey::parse(testSecretKey)); r.sign(realisation.id, signer); ASSERT_EQ(r.checkSignatures(realisation.id, publicKeys), 0); @@ -205,12 +205,12 @@ TEST_P(RealisationSigningTest, verify_rejects_tampered_outpath) { const auto & [name, realisation] = GetParam(); - auto publicKey = testSecretKey.toPublicKey(); + auto publicKey = SecretKey::parse(testSecretKey)->toPublicKey(); PublicKeys publicKeys; - publicKeys.insert_or_assign(publicKey.name, publicKey); + publicKeys.insert_or_assign(publicKey->name, std::move(publicKey)); auto r = static_cast(realisation); - LocalSigner signer(SecretKey{testSecretKey}); + LocalSigner signer(SecretKey::parse(testSecretKey)); r.sign(realisation.id, signer); // Tamper with the output path after signing. diff --git a/src/libstore/active-builds.cc b/src/libstore/active-builds.cc new file mode 100644 index 000000000000..4b2688a9dd25 --- /dev/null +++ b/src/libstore/active-builds.cc @@ -0,0 +1,156 @@ +#include "nix/store/active-builds.hh" +#include "nix/util/json-utils.hh" + +#include + +#ifndef _WIN32 +# include +#endif + +namespace nix { + +void QueryActiveBuildsStore::anchor() {} + +void TrackActiveBuildsStore::anchor() {} + +UserInfo UserInfo::fromUid(uid_t uid) +{ + UserInfo info; + info.uid = uid; + +#ifndef _WIN32 + // Look up the user name for the UID (thread-safe) + struct passwd pwd; + struct passwd * result; + std::vector buf(16384); + if (getpwuid_r(uid, &pwd, buf.data(), buf.size(), &result) == 0 && result) + info.name = result->pw_name; +#endif + + return info; +} + +} // namespace nix + +namespace nlohmann { + +using namespace nix; + +UserInfo adl_serializer::from_json(const json & j) +{ + return UserInfo{ + .uid = j.at("uid").get(), + .name = j.contains("name") && !j.at("name").is_null() + ? std::optional(j.at("name").get()) + : std::nullopt, + }; +} + +void adl_serializer::to_json(json & j, const UserInfo & info) +{ + j = nlohmann::json{ + {"uid", info.uid}, + {"name", info.name}, + }; +} + +// Durations are serialized as floats representing seconds. +static std::optional parseDuration(const json & j, const char * key) +{ + if (j.contains(key) && !j.at(key).is_null()) + return std::chrono::duration_cast( + std::chrono::duration(j.at(key).get())); + else + return std::nullopt; +} + +static nlohmann::json printDuration(const std::optional & duration) +{ + return duration + ? nlohmann::json( + std::chrono::duration_cast>(*duration) + .count()) + : nullptr; +} + +ActiveBuildInfo::ProcessInfo adl_serializer::from_json(const json & j) +{ + return ActiveBuildInfo::ProcessInfo{ + .pid = j.at("pid").get(), + .parentPid = j.at("parentPid").get(), + .user = j.at("user").get(), + .argv = j.at("argv").get>(), + .utime = parseDuration(j, "utime"), + .stime = parseDuration(j, "stime"), + .cutime = parseDuration(j, "cutime"), + .cstime = parseDuration(j, "cstime"), + }; +} + +void adl_serializer::to_json(json & j, const ActiveBuildInfo::ProcessInfo & process) +{ + j = nlohmann::json{ + {"pid", process.pid}, + {"parentPid", process.parentPid}, + {"user", process.user}, + {"argv", process.argv}, + {"utime", printDuration(process.utime)}, + {"stime", printDuration(process.stime)}, + {"cutime", printDuration(process.cutime)}, + {"cstime", printDuration(process.cstime)}, + }; +} + +ActiveBuild adl_serializer::from_json(const json & j) +{ + auto type = j.at("type").get(); + if (type != "build") + throw Error("invalid active build JSON: expected type 'build' but got '%s'", type); + std::optional cgroup; + if (!j.at("cgroup").is_null()) + cgroup = j.at("cgroup").get(); + return ActiveBuild{ + .nixPid = j.at("nixPid").get(), + .clientPid = j.at("clientPid").get>(), + .clientUid = j.at("clientUid").get>(), + .mainPid = j.at("mainPid").get(), + .mainUser = j.at("mainUser").get(), + .cgroup = std::move(cgroup), + .startTime = (time_t) j.at("startTime").get(), + .derivation = StorePath{getString(j.at("derivation"))}, + }; +} + +void adl_serializer::to_json(json & j, const ActiveBuild & build) +{ + j = nlohmann::json{ + {"type", "build"}, + {"nixPid", build.nixPid}, + {"clientPid", build.clientPid}, + {"clientUid", build.clientUid}, + {"mainPid", build.mainPid}, + {"mainUser", build.mainUser}, + {"cgroup", build.cgroup ? nlohmann::json(*build.cgroup) : nlohmann::json(nullptr)}, + {"startTime", (double) build.startTime}, + {"derivation", build.derivation.to_string()}, + }; +} + +ActiveBuildInfo adl_serializer::from_json(const json & j) +{ + ActiveBuildInfo info(adl_serializer::from_json(j)); + info.processes = j.at("processes").get>(); + info.utime = parseDuration(j, "utime"); + info.stime = parseDuration(j, "stime"); + return info; +} + +void adl_serializer::to_json(json & j, const ActiveBuildInfo & build) +{ + adl_serializer::to_json(j, build); + j["processes"] = build.processes; + j["utime"] = printDuration(build.utime); + j["stime"] = printDuration(build.stime); +} + +} // namespace nlohmann diff --git a/src/libstore/async-path-writer.cc b/src/libstore/async-path-writer.cc new file mode 100644 index 000000000000..80f29dbc967a --- /dev/null +++ b/src/libstore/async-path-writer.cc @@ -0,0 +1,234 @@ +#include "nix/store/async-path-writer.hh" +#include "nix/store/globals.hh" +#include "nix/util/archive.hh" +#include "nix/util/provenance.hh" + +#include +#include + +namespace nix { + +struct AsyncPathWriterImpl : AsyncPathWriter +{ + ref store; + + struct Item + { + StorePath storePath; + std::string contents; + std::string name; + Hash hash; + StorePathSet references; + RepairFlag repair; + std::shared_ptr provenance; + std::promise promise; + }; + + struct State + { + std::vector items; + std::unordered_map> futures; + bool quit = false; + }; + + Sync state_; + + std::thread workerThread; + + std::condition_variable wakeupCV; + + AsyncPathWriterImpl(ref store) + : store(store) + { + workerThread = std::thread([&]() { + while (true) { + std::vector items; + + { + auto state(state_.lock()); + while (!state->quit && state->items.empty()) + state.wait(wakeupCV); + if (state->items.empty() && state->quit) + return; + std::swap(items, state->items); + } + + try { + writePaths(items); + for (auto & item : items) + item.promise.set_value(); + } catch (...) { + for (auto & item : items) + item.promise.set_exception(std::current_exception()); + } + } + }); + } + + virtual ~AsyncPathWriterImpl() + { + state_.lock()->quit = true; + wakeupCV.notify_all(); + workerThread.join(); + } + + StorePath addPath( + std::string contents, + std::string name, + StorePathSet references, + RepairFlag repair, + std::shared_ptr provenance) override + { + auto hash = hashString(HashAlgorithm::SHA256, contents); + + auto storePath = store->makeFixedOutputPathFromCA( + name, + TextInfo{ + .hash = hash, + .references = references, + }); + + /* In read-only mode, only compute the store path; don't + write anything. */ + if (settings.readOnlyMode) + return storePath; + + auto state(state_.lock()); + + /* If we've already written or queued this path, there is + nothing to do. This also applies when `repair` is set, + since the first request already repaired it. */ + if (state->futures.contains(storePath)) + return storePath; + + std::promise promise; + state->futures.emplace(storePath, promise.get_future()); + state->items.push_back( + Item{ + .storePath = storePath, + .contents = std::move(contents), + .name = std::move(name), + .hash = hash, + .references = std::move(references), + .repair = repair, + .provenance = provenance, + .promise = std::move(promise), + }); + wakeupCV.notify_all(); + + return storePath; + } + + void waitForPath(const StorePath & path) override + { + auto future = ({ + auto state = state_.lock(); + auto i = state->futures.find(path); + if (i == state->futures.end()) + return; + i->second; + }); + future.get(); + } + + void waitForAllPaths() override + { + /* Copy rather than move the futures, since `futures` also + serves as the record of paths already written (see + `addPath()`). */ + auto futures = ({ + auto state(state_.lock()); + state->futures; + }); + for (auto & future : futures) + future.second.get(); + } + + void writePaths(const std::vector & items) + { +// FIXME: addMultipeToStore() shouldn't require a NAR hash. +#if 0 + Store::PathsSource sources; + RepairFlag repair = NoRepair; + + for (auto & item : items) { + ValidPathInfo info{item.storePath, Hash(HashAlgorithm::SHA256)}; + info.references = item.references; + info.ca = ContentAddress { + .method = ContentAddressMethod::Raw::Text, + .hash = item.hash, + }; + if (item.repair) repair = item.repair; + auto source = sinkToSource([&](Sink & sink) + { + dumpString(item.contents, sink); + }); + sources.push_back({std::move(info), std::move(source)}); + } + + Activity act(*logger, lvlDebug, actUnknown, fmt("adding %d paths to the store", items.size())); + + store->addMultipleToStore(std::move(sources), act, repair); +#endif + + /* Filter out the paths that the store already has. Add temp + roots first so that a path found to be valid cannot be + garbage-collected before we return. */ + StorePathSet allPaths; + for (auto & item : items) + allPaths.insert(item.storePath); + + store->addTempRoots(allPaths); + + StorePathSet valid; + if (!allPaths.empty()) { + asio::io_context ctx; + std::exception_ptr ex; + asio::co_spawn( + ctx, + store->queryPathInfos( + allPaths, + [&](std::vector>> infos) { + for (auto & [path, info] : infos) + if (info) + valid.insert(path); + }), + [&](std::exception_ptr e) { + if (e) + ex = e; + }); + ctx.run(); + if (ex) + std::rethrow_exception(ex); + } + + for (auto & item : items) { + if (!item.repair && valid.contains(item.storePath)) + continue; + StringSource source(item.contents); + auto storePath = store->addToStoreFromDump( + source, + item.storePath.name(), + FileSerialisationMethod::Flat, + ContentAddressMethod::Raw::Text, + HashAlgorithm::SHA256, + item.references, + item.repair, + item.provenance); + assert(storePath == item.storePath); + } + } + + void anchor() override; +}; + +void AsyncPathWriter::anchor() {} + +void AsyncPathWriterImpl::anchor() {} + +ref AsyncPathWriter::make(ref store) +{ + return make_ref(store); +} + +} // namespace nix diff --git a/src/libstore/binary-cache-store.cc b/src/libstore/binary-cache-store.cc index 5294ee7a0330..14bf76b87e5e 100644 --- a/src/libstore/binary-cache-store.cc +++ b/src/libstore/binary-cache-store.cc @@ -34,13 +34,13 @@ BinaryCacheStore::BinaryCacheStore(Config & config) : config{config} { if (auto & skf = config.secretKeyFile.get()) - signers.push_back(std::make_unique(SecretKey{readFile(*skf)})); + signers.push_back(std::make_unique(SecretKey::parse(readFile(*skf)))); if (config.secretKeyFiles != "") { std::stringstream ss(config.secretKeyFiles); std::string keyPath; while (std::getline(ss, keyPath, ',')) { - signers.push_back(std::make_unique(SecretKey{readFile(keyPath)})); + signers.push_back(std::make_unique(SecretKey::parse(readFile(keyPath)))); } } @@ -49,8 +49,10 @@ BinaryCacheStore::BinaryCacheStore(Config & config) narMagic = sink.s; } -void BinaryCacheStore::init() +std::map BinaryCacheStore::parseNixCacheInfo() { + std::map fields; + auto cacheInfo = getNixCacheInfo(); if (!cacheInfo) { upsertFile(cacheInfoFile, "StoreDir: " + storeDir + "\n", "text/x-nix-cache-info"); @@ -68,13 +70,31 @@ void BinaryCacheStore::init() config.getHumanReadableURI(), value, storeDir); - } else if (name == "WantMassQuery") { - config.wantMassQuery.setDefault(value == "1"); - } else if (name == "Priority") { - config.priority.setDefault(std::stoi(value)); + } else { + /* Keep every other field verbatim, including ones we + don't (yet) understand. The known ones are applied + by applyCacheInfoFields(). */ + fields.insert_or_assign(name, value); } } } + + return fields; +} + +void BinaryCacheStore::applyCacheInfoFields(const std::map & fields) +{ + if (auto * value = get(fields, "WantMassQuery")) + config.wantMassQuery.setDefault(*value == "1"); + if (auto * value = get(fields, "Priority")) { + if (auto priority = string2Int(*value)) + config.priority.setDefault(*priority); + } +} + +void BinaryCacheStore::init() +{ + applyCacheInfoFields(parseNixCacheInfo()); } std::optional BinaryCacheStore::getNixCacheInfo() @@ -461,7 +481,8 @@ StorePath BinaryCacheStore::addToStoreFromDump( ContentAddressMethod hashMethod, HashAlgorithm hashAlgo, const StorePathSet & references, - RepairFlag repair) + RepairFlag repair, + std::shared_ptr provenance) { std::optional caHash; std::string nar; @@ -524,6 +545,7 @@ StorePath BinaryCacheStore::addToStoreFromDump( }), nar.hash); info.narSize = nar.numBytesDigested; + info.provenance = provenance; return info; }) ->path; @@ -659,6 +681,7 @@ StorePath BinaryCacheStore::addToStore( }), nar.hash); info.narSize = nar.numBytesDigested; + info.provenance = path.getProvenance(); return info; }) ->path; diff --git a/src/libstore/build-result.cc b/src/libstore/build-result.cc index c9d78e5ccf30..dbbfa4af7995 100644 --- a/src/libstore/build-result.cc +++ b/src/libstore/build-result.cc @@ -1,5 +1,7 @@ #include "nix/store/build-result.hh" #include "nix/util/json-utils.hh" +#include "nix/util/provenance.hh" + #include namespace nix { @@ -92,7 +94,7 @@ static BuildResult::Success::Status successStatusFromString(std::string_view str throw Error("unknown built result success status '%s'", str); } -static constexpr std::array, 12> failureStatusStrings{{ +static constexpr std::array, 13> failureStatusStrings{{ #define ENUM_ENTRY(e) {BuildResult::Failure::e, #e} ENUM_ENTRY(PermanentFailure), ENUM_ENTRY(InputRejected), @@ -106,6 +108,7 @@ static constexpr std::array::to_json(json & res, const BuildResult & br) res["success"] = true; res["status"] = successStatusToString(success.status); res["builtOutputs"] = success.builtOutputs; + if (success.provenance) + res["provenance"] = success.provenance->to_json(); }, [&](const BuildResult::Failure & failure) { res["success"] = false; res["status"] = failureStatusToString(failure.status); res["errorMsg"] = failure.message(); res["isNonDeterministic"] = failure.isNonDeterministic; + if (failure.provenance) + res["provenance"] = failure.provenance->to_json(); }, }, br.inner); @@ -203,17 +210,24 @@ BuildResult adl_serializer::from_json(const json & _json) bool success = getBoolean(valueAt(json, "success")); std::string statusStr = getString(valueAt(json, "status")); + auto provenanceFromJson = [](const nlohmann::json * j) -> std::shared_ptr { + if (j && !j->is_null()) + return Provenance::from_json(*j); + return nullptr; + }; + if (success) { BuildResult::Success s; s.status = successStatusFromString(statusStr); s.builtOutputs = valueAt(json, "builtOutputs"); + s.provenance = provenanceFromJson(optionalValueAt(json, "provenance")); br.inner = std::move(s); } else { - br.inner = BuildResult::Failure{{ - .status = failureStatusFromString(statusStr), - .msg = HintFmt(getString(valueAt(json, "errorMsg"))), - .isNonDeterministic = getBoolean(valueAt(json, "isNonDeterministic")), - }}; + br.inner = BuildResult::Failure{ + {.status = failureStatusFromString(statusStr), + .msg = HintFmt(getString(valueAt(json, "errorMsg"))), + .isNonDeterministic = getBoolean(valueAt(json, "isNonDeterministic")), + .provenance = provenanceFromJson(optionalValueAt(json, "provenance"))}}; } return br; diff --git a/src/libstore/build/build-log.cc b/src/libstore/build/build-log.cc index a8fb64fc68bf..85b8877d6590 100644 --- a/src/libstore/build/build-log.cc +++ b/src/libstore/build/build-log.cc @@ -2,7 +2,7 @@ namespace nix { -BuildLog::BuildLog(size_t maxTailLines, std::unique_ptr act) +BuildLog::BuildLog(size_t maxTailLines, ref act) : maxTailLines(maxTailLines) , act(std::move(act)) { diff --git a/src/libstore/build/derivation-building-goal.cc b/src/libstore/build/derivation-building-goal.cc index 81612b4dbc39..49c0fc40bc71 100644 --- a/src/libstore/build/derivation-building-goal.cc +++ b/src/libstore/build/derivation-building-goal.cc @@ -169,7 +169,7 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution(bool storeDerivation) "dependency '%s' of '%s' does not exist, and substitution is disabled", worker.store.printStorePath(i), worker.store.printStorePath(drvPath)); - waitees.insert(upcast_goal(worker.makePathSubstitutionGoal(i))); + waitees.insert(upcast_goal(worker.makePathSubstitutionGoal(i, true))); } co_await await(std::move(waitees)); @@ -417,8 +417,8 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths) WrongLocalStore wrongStore; - if (drv->platform != settings.thisSystem.get() && !settings.extraPlatforms.get().count(drv->platform) - && !drv->isBuiltin()) + if (drv->platform != settings.thisSystem.get() && drv->platform != "wasm32-wasip1" + && !settings.extraPlatforms.get().count(drv->platform) && !drv->isBuiltin()) wrongStore.badPlatform = WrongLocalStore::Pair{drv->platform, settings.thisSystem.get()}; { @@ -703,12 +703,13 @@ Goal::Co DerivationBuildingGoal::buildWithHook( std::unique_ptr buildLog = std::make_unique( worker.settings.logLines, - std::make_unique( + make_ref( *logger, lvlInfo, actBuild, msg, - Logger::Fields{worker.store.printStorePath(drvPath), hook->machineName, 1, 1})); + Logger::Fields{worker.store.printStorePath(drvPath), hook->machineName, 1, 1}, + worker.actDerivations.id)); mcRunningBuilds = std::make_unique>(worker.runningBuilds); worker.updateProgress(); @@ -734,6 +735,8 @@ Goal::Co DerivationBuildingGoal::buildWithHook( if (c == '\n') { auto json = parseJSONMessage(currentHookLine, "the derivation builder"); if (json) { + /* The hook exports its own telemetry. */ + RemoteLogSource remoteLogSource; auto s = handleJSONLogMessage( *json, worker.act, hook->activities, "the derivation builder", true); // ensure that logs from a builder using `ssh-ng://` as protocol @@ -770,7 +773,7 @@ Goal::Co DerivationBuildingGoal::buildWithHook( break; } else if (auto * timeout = std::get_if>(&event)) { hook.reset(); - co_return doneFailure(std::move(**timeout)); + co_return doneFailure(std::move(**timeout), buildLog->act->id); } } @@ -805,7 +808,7 @@ Goal::Co DerivationBuildingGoal::buildWithHook( /* TODO (once again) support fine-grained error codes, see issue #12641. */ - co_return doneFailure(std::move(e)); + co_return doneFailure(std::move(e), buildLog->act->id); } /* Compute the FS closure of the outputs and register them as @@ -854,7 +857,7 @@ Goal::Co DerivationBuildingGoal::buildWithHook( outputLocks.setDeletion(true); outputLocks.unlock(); - co_return doneSuccess(BuildResult::Success::Built, std::move(builtOutputs)); + co_return doneSuccess(BuildResult::Success::Built, std::move(builtOutputs), buildLog->act->id); #endif } @@ -870,6 +873,18 @@ Goal::Co DerivationBuildingGoal::buildLocally( #ifdef _WIN32 // TODO enable `DerivationBuilder` on Windows throw UnimplementedError("building derivations is not yet implemented on Windows"); #else + auto msg = + fmt(buildMode == bmRepair ? "repairing outputs of '%s'" + : buildMode == bmCheck ? "checking outputs of '%s'" + : "building '%s'", + worker.store.printStorePath(drvPath)); + auto act = make_ref( + *logger, + lvlInfo, + actBuild, + msg, + Logger::Fields{worker.store.printStorePath(drvPath), "", 1, 1}, + worker.actDerivations.id); std::unique_ptr buildLog; std::unique_ptr logFile; @@ -880,15 +895,7 @@ Goal::Co DerivationBuildingGoal::buildLocally( auto closeLogFile = [&]() { logFile.reset(); }; auto started = [&]() { - auto msg = - fmt(buildMode == bmRepair ? "repairing outputs of '%s'" - : buildMode == bmCheck ? "checking outputs of '%s'" - : "building '%s'", - worker.store.printStorePath(drvPath)); - buildLog = std::make_unique( - worker.settings.logLines, - std::make_unique( - *logger, lvlInfo, actBuild, msg, Logger::Fields{worker.store.printStorePath(drvPath), "", 1, 1})); + buildLog = std::make_unique(worker.settings.logLines, act); mcRunningBuilds = std::make_unique>(worker.runningBuilds); worker.updateProgress(); }; @@ -897,6 +904,11 @@ Goal::Co DerivationBuildingGoal::buildLocally( DerivationBuilderUnique builder; Descriptor builderOut; + /* Get the provenance of the derivation, if available. */ + std::shared_ptr provenance; + if (auto info = worker.evalStore.maybeQueryPathInfo(drvPath)) + provenance = info->provenance; + // Will continue here while waiting for a build user below while (true) { @@ -969,11 +981,12 @@ Goal::Co DerivationBuildingGoal::buildLocally( desugaredEnv = DesugaredEnv::create(worker.store, *drv, drvOptions, inputPaths); } catch (BuildError & e) { outputLocks.unlock(); - co_return doneFailure(std::move(e)); + co_return doneFailure(std::move(e), act->id); } DerivationBuilderParams params{ .drvPath = drvPath, + .drvProvenance = provenance, .buildResult = buildResult, .drv = *drv, .drvOptions = drvOptions, @@ -983,6 +996,7 @@ Goal::Co DerivationBuildingGoal::buildLocally( .defaultPathsInChroot = std::move(defaultPathsInChroot), .systemFeatures = worker.store.config.systemFeatures.get(), .desugaredEnv = std::move(desugaredEnv), + .act = act, }; /* If we have to wait and retry (see below), then `builder` will @@ -1041,7 +1055,7 @@ Goal::Co DerivationBuildingGoal::buildLocally( break; } else if (auto * timeout = std::get_if>(&event)) { builder->killChild(); - co_return doneFailure(std::move(**timeout)); + co_return doneFailure(std::move(**timeout), act->id); } } @@ -1053,11 +1067,11 @@ Goal::Co DerivationBuildingGoal::buildLocally( } catch (BuilderFailureError & e) { builder.reset(); outputLocks.unlock(); - co_return doneFailure(fixupBuilderFailureErrorMessage(std::move(e), *buildLog)); + co_return doneFailure(fixupBuilderFailureErrorMessage(std::move(e), *buildLog), act->id); } catch (BuildError & e) { builder.reset(); outputLocks.unlock(); - co_return doneFailure(std::move(e)); + co_return doneFailure(std::move(e), act->id); } { builder.reset(); @@ -1097,7 +1111,7 @@ Goal::Co DerivationBuildingGoal::buildLocally( (unlinked) lock files. */ outputLocks.setDeletion(true); outputLocks.unlock(); - co_return doneSuccess(BuildResult::Success::Built, std::move(builtOutputs)); + co_return doneSuccess(BuildResult::Success::Built, std::move(builtOutputs), act->id, provenance); } #endif } @@ -1172,7 +1186,7 @@ BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailur msg += line; msg += "\n"; } - auto nixLogCommand = experimentalFeatureSettings.isEnabled(Xp::NixCommand) ? "nix log" : "nix-store -l"; + auto nixLogCommand = "nix log"; // The command is on a separate line for easy copying, such as with triple click. // This message will be indented elsewhere, so removing the indentation before the // command will not put it at the start of the line unfortunately. @@ -1220,6 +1234,8 @@ HookReply DerivationBuildingGoal::tryBuildHook(const DerivationOptionsactivities, "the build hook", true)) ; else if (s.substr(0, 2) == "# ") { @@ -1302,11 +1318,13 @@ LogFile::~LogFile() Goal::Done DerivationBuildingGoal::doneFailureLogTooLong(BuildLog & buildLog) { - return doneFailure(BuildError( - BuildResult::Failure::LogLimitExceeded, - "%s killed after writing more than %d bytes of log output", - getName(), - worker.settings.maxLogSize)); + return doneFailure( + BuildError( + BuildResult::Failure::LogLimitExceeded, + "%s killed after writing more than %d bytes of log output", + getName(), + worker.settings.maxLogSize), + buildLog.act->id); } std::map> DerivationBuildingGoal::queryPartialDerivationOutputMap() @@ -1392,7 +1410,11 @@ DerivationBuildingGoal::checkPathValidity(std::map & return {allValid, validOutputs}; } -Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Success::Status status, SingleDrvOutputs builtOutputs) +Goal::Done DerivationBuildingGoal::doneSuccess( + BuildResult::Success::Status status, + SingleDrvOutputs builtOutputs, + ActivityId act, + std::shared_ptr provenance) { mcRunningBuilds.reset(); @@ -1401,14 +1423,24 @@ Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Success::Status stat worker.updateProgress(); - return Goal::doneSuccess( + auto res = Goal::doneSuccess( BuildResult::Success{ .status = status, .builtOutputs = std::move(builtOutputs), + .provenance = provenance, }); + + logger->result( + act, + resBuildResult, + nlohmann::json(KeyedBuildResult( + buildResult, + DerivedPath::Built{.drvPath = makeConstantStorePathRef(drvPath), .outputs = OutputsSpec::All{}}))); + + return res; } -Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex) +Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex, ActivityId act) { mcRunningBuilds.reset(); @@ -1418,7 +1450,16 @@ Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex) worker.updateProgress(); - return Goal::doneFailure(ecFailed, std::move(ex)); + auto res = Goal::doneFailure(ecFailed, std::move(ex)); + + logger->result( + act, + resBuildResult, + nlohmann::json(KeyedBuildResult( + buildResult, + DerivedPath::Built{.drvPath = makeConstantStorePathRef(drvPath), .outputs = OutputsSpec::All{}}))); + + return res; } } // namespace nix diff --git a/src/libstore/build/derivation-check.cc b/src/libstore/build/derivation-check.cc index d0d9a8da76d3..8421642042b3 100644 --- a/src/libstore/build/derivation-check.cc +++ b/src/libstore/build/derivation-check.cc @@ -8,7 +8,11 @@ namespace nix { void checkCAFixedOutput( - StoreDirConfig & store, const StorePath & drvPath, const DerivationOutput & outputSpec, const ValidPathInfo & info) + StoreDirConfig & store, + const StorePath & drvPath, + const DerivationOutput & outputSpec, + const ValidPathInfo & info, + Activity & act) { if (const auto * dof = std::get_if(&outputSpec.raw)) { auto & wanted = dof->ca.hash; @@ -17,6 +21,15 @@ void checkCAFixedOutput( assert(info.ca); auto & got = info.ca->hash; if (wanted != got) { + /* Throw an error after registering the path as + valid. */ + act.result( + resHashMismatch, + { + {"storePath", store.printStorePath(drvPath)}, + {"wanted", wanted}, + {"got", got}, + }); throw BuildError( BuildResult::Failure::HashMismatch, "hash mismatch in fixed-output derivation '%s':\n specified: %s\n got: %s", @@ -41,7 +54,8 @@ void checkOutputs( const StorePath & drvPath, const decltype(Derivation::outputs) & drvOutputs, const decltype(DerivationOptions::outputChecks) & outputChecks, - const std::map & outputs) + const std::map & outputs, + Activity & act) { std::map outputsByPath; for (auto & output : outputs) @@ -56,7 +70,7 @@ void checkOutputs( auto * outputSpec = get(drvOutputs, outputName); assert(outputSpec); - checkCAFixedOutput(store, drvPath, *outputSpec, info); + checkCAFixedOutput(store, drvPath, *outputSpec, info, act); /* Compute the closure and closure size of some output. This is slightly tricky because some of its references (namely diff --git a/src/libstore/build/derivation-check.hh b/src/libstore/build/derivation-check.hh index f55f9aac2055..1832264ea1a9 100644 --- a/src/libstore/build/derivation-check.hh +++ b/src/libstore/build/derivation-check.hh @@ -13,7 +13,11 @@ namespace nix { * not a CAFixed output. */ void checkCAFixedOutput( - StoreDirConfig & store, const StorePath & drvPath, const DerivationOutput & outputSpec, const ValidPathInfo & info); + StoreDirConfig & store, + const StorePath & drvPath, + const DerivationOutput & outputSpec, + const ValidPathInfo & info, + Activity & act); /** * Check that outputs meets the requirements specified by the @@ -30,6 +34,7 @@ void checkOutputs( const StorePath & drvPath, const decltype(Derivation::outputs) & drvOutputs, const decltype(DerivationOptions::outputChecks) & drvOptions, - const std::map & outputs); + const std::map & outputs, + Activity & act); } // namespace nix diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 24f111c8b5d1..55bf3923903b 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -62,9 +62,12 @@ Goal::Co DerivationGoal::haveDerivation(bool storeDerivation) if (!drv->type().hasKnownOutputPaths()) experimentalFeatureSettings.require(Xp::CaDerivations); + StorePathSet outputPaths; for (auto & i : drv->outputsAndOptPaths(worker.store)) if (i.second.second) - worker.store.addTempRoot(*i.second.second); + outputPaths.insert(*i.second.second); + + worker.store.addTempRoots(outputPaths); /* We don't yet have any safe way to cache an impure derivation at this step. */ @@ -108,6 +111,7 @@ Goal::Co DerivationGoal::haveDerivation(bool storeDerivation) auto * cap = getDerivationCA(*drv); waitees.insert(upcast_goal(worker.makePathSubstitutionGoal( checkResult->first.outPath, + false, buildMode == bmRepair ? Repair : NoRepair, cap ? std::optional{*cap} : std::nullopt))); } @@ -335,7 +339,7 @@ Goal::Co DerivationGoal::repairClosure() worker.store.printStorePath(drvPath)); auto drvPath2 = outputsToDrv.find(i); if (drvPath2 == outputsToDrv.end()) - waitees.insert(upcast_goal(worker.makePathSubstitutionGoal(i, Repair))); + waitees.insert(upcast_goal(worker.makePathSubstitutionGoal(i, false, Repair))); else waitees.insert(worker.makeGoal( DerivedPath::Built{ @@ -431,11 +435,20 @@ Goal::Done DerivationGoal::doneSuccess(BuildResult::Success::Status status, Unke worker.updateProgress(); - return Goal::doneSuccess( + auto res = Goal::doneSuccess( BuildResult::Success{ .status = status, .builtOutputs = {{wantedOutput, std::move(builtOutput)}}, }); + + logger->result( + getCurActivity(), + resBuildResult, + nlohmann::json(KeyedBuildResult( + buildResult, + DerivedPath::Built{.drvPath = makeConstantStorePathRef(drvPath), .outputs = OutputsSpec::All{}}))); + + return res; } Goal::Done DerivationGoal::doneFailure(BuildError ex) @@ -448,6 +461,12 @@ Goal::Done DerivationGoal::doneFailure(BuildError ex) worker.updateProgress(); + logger->result( + getCurActivity(), + resBuildResult, + nlohmann::json(KeyedBuildResult( + {ex}, DerivedPath::Built{.drvPath = makeConstantStorePathRef(drvPath), .outputs = OutputsSpec::All{}}))); + return Goal::doneFailure(ecFailed, std::move(ex)); } diff --git a/src/libstore/build/derivation-trampoline-goal.cc b/src/libstore/build/derivation-trampoline-goal.cc index 14864ed052f8..f676afcad43a 100644 --- a/src/libstore/build/derivation-trampoline-goal.cc +++ b/src/libstore/build/derivation-trampoline-goal.cc @@ -150,6 +150,65 @@ Goal::Co DerivationTrampolineGoal::haveDerivation(StorePath drvPath, Derivation /* Must have at least one wanted output. This is assumed below. */ assert(!resolvedWantedOutputs.empty()); + /* Short-circuit `builtin:substitute`. Since these are never actually built, just create substitution goals for the + * outputs. */ + if (drv.type() == DerivationType::Substituted{}) { + struct OutputGoal + { + OutputName outputName; + StorePath outPath; + GoalPtr goal; + }; + + std::vector outputGoals; + Goals waitees; + for (auto & outputName : resolvedWantedOutputs) { + auto i = drv.outputs.find(outputName); + if (i == drv.outputs.end()) + throw Error( + "derivation '%s' does not have wanted output '%s'", + worker.store.printStorePath(drvPath), + outputName); + auto outPath = i->second.path(worker.store, drv.name, outputName); + if (!outPath) + throw Error( + "output '%s' of derivation '%s' has no store path", + outputName, + worker.store.printStorePath(drvPath)); + auto g = upcast_goal(worker.makePathSubstitutionGoal(*outPath)); + outputGoals.push_back({outputName, *outPath, g}); + waitees.insert(g); + } + + co_await await(std::move(waitees)); + + if (nrFailed != 0) { + StringSet failedPaths; + for (auto & og : outputGoals) + if (og.goal->exitCode != ecSuccess) + failedPaths.insert(worker.store.printStorePath(og.outPath)); + co_return doneFailure( + ecFailed, + BuildResult::Failure{{ + .status = BuildResult::Failure::DependencyFailed, + .msg = HintFmt( + "failed to substitute outputs of '%s': %s", + worker.store.printStorePath(drvPath), + concatStringsSep(", ", quoteStrings(failedPaths))), + }}); + } + + SingleDrvOutputs builtOutputs; + for (auto & og : outputGoals) + builtOutputs.emplace(og.outputName, UnkeyedRealisation{.outPath = og.outPath}); + + co_return doneSuccess( + BuildResult::Success{ + .status = BuildResult::Success::Substituted, + .builtOutputs = std::move(builtOutputs), + }); + } + Goals concreteDrvGoals; /* Build this step! */ diff --git a/src/libstore/build/entry-points.cc b/src/libstore/build/entry-points.cc index 5b97966847b8..dbc688cb033b 100644 --- a/src/libstore/build/entry-points.cc +++ b/src/libstore/build/entry-points.cc @@ -64,11 +64,6 @@ std::vector Store::buildPathsWithResults( results.reserve(state.size()); for (auto & [req, goalPtr] : state) { - /* Goals that were never started or were cancelled have exitCode - ecBusy and a default buildResult with empty errorMsg. Skip them - to avoid reporting spurious failures with empty messages. */ - if (goalPtr->exitCode == Goal::ecBusy) - continue; results.emplace_back( KeyedBuildResult{ goalPtr->buildResult, @@ -103,7 +98,7 @@ void Store::ensurePath(const StorePath & path) return; Worker worker(*this, *this); - GoalPtr goal = worker.makePathSubstitutionGoal(path); + GoalPtr goal = worker.makePathSubstitutionGoal(path, true); Goals goals = {goal}; worker.run(goals); @@ -118,7 +113,7 @@ void Store::ensurePath(const StorePath & path) void Store::repairPath(const StorePath & path) { Worker worker(*this, *this); - GoalPtr goal = worker.makePathSubstitutionGoal(path, Repair); + GoalPtr goal = worker.makePathSubstitutionGoal(path, true, Repair); Goals goals = {goal}; worker.run(goals); diff --git a/src/libstore/build/substitution-goal.cc b/src/libstore/build/substitution-goal.cc index 90273493e288..45556a5699a7 100644 --- a/src/libstore/build/substitution-goal.cc +++ b/src/libstore/build/substitution-goal.cc @@ -7,12 +7,19 @@ #include "nix/util/signals.hh" #include "nix/util/callback.hh" +#include + namespace nix { PathSubstitutionGoal::PathSubstitutionGoal( - const StorePath & storePath, Worker & worker, RepairFlag repair, std::optional ca) + const StorePath & storePath, + Worker & worker, + bool pathRequired, + RepairFlag repair, + std::optional ca) : Goal(worker, init()) , storePath(storePath) + , pathRequired(pathRequired) , repair(repair) , ca(ca) { @@ -26,6 +33,15 @@ PathSubstitutionGoal::~PathSubstitutionGoal() cleanup(); } +Goal::Done PathSubstitutionGoal::doneFailure(ExitCode result, BuildResult::Failure failure, ActivityId act) +{ + auto res = Goal::doneFailure(result, std::move(failure)); + + logger->result(act, resBuildResult, nlohmann::json(KeyedBuildResult(buildResult, DerivedPath::Opaque{storePath}))); + + return res; +} + Goal::Co PathSubstitutionGoal::init() { trace("init"); @@ -41,6 +57,20 @@ Goal::Co PathSubstitutionGoal::init() throw Error( "cannot substitute path '%s' - no write access to the Nix store", worker.store.printStorePath(storePath)); + /* Covers the lifetime of this goal, including querying + substituters and the actual substitution. Note that the + progress bar only shows the `actSubstitute` activity, which + denotes that the actual substitution is happening. */ + Activity act( + *logger, + lvlDebug, + "SubstitutionGoal", + std::to_array>({ + {"nix.store.path", worker.store.printStorePath(storePath)}, + }), + "", + worker.actSubstitutions.id); + auto subs = worker.getSubstituters(); bool substituterFailed = false; @@ -73,7 +103,12 @@ Goal::Co PathSubstitutionGoal::init() try { info = co_await AsyncCallback>( - [sub, path = subPath.value_or(storePath)](auto cb) { sub->queryPathInfo(path, std::move(cb)); }); + [&act, sub, path = subPath.value_or(storePath)](auto cb) { + /* Note: narrowly scoped so that the `PushActivity` + does not extend across a suspension point. */ + PushActivity pact(act.id); + sub->queryPathInfo(path, std::move(cb)); + }); } catch (InvalidPath &) { continue; } catch (SubstituterDisabled & e) { @@ -127,13 +162,13 @@ Goal::Co PathSubstitutionGoal::init() paths referenced by this one. */ for (auto & i : info->references) if (i != storePath) /* ignore self-references */ - waitees.insert(worker.makePathSubstitutionGoal(i)); + waitees.insert(worker.makePathSubstitutionGoal(i, pathRequired)); co_await await(std::move(waitees)); // FIXME: consider returning boolean instead of passing in reference bool out = false; // is mutated by tryToRun - co_await tryToRun(subPath ? *subPath : storePath, sub, info, out); + co_await tryToRun(subPath ? *subPath : storePath, sub, info, out, act.id); substituterFailed = substituterFailed || out; } @@ -155,17 +190,22 @@ Goal::Co PathSubstitutionGoal::init() In that case the calling derivation should just do a build. */ co_return doneFailure( - substituterFailed ? ecFailed : ecNoSubstituters, + substituterFailed || pathRequired ? ecFailed : ecNoSubstituters, BuildResult::Failure{{ .status = BuildResult::Failure::NoSubstituters, .msg = HintFmt( "path '%s' is required, but there is no substituter that can build it", worker.store.printStorePath(storePath)), - }}); + }}, + act.id); } Goal::Co PathSubstitutionGoal::tryToRun( - StorePath subPath, nix::ref sub, std::shared_ptr info, bool & substituterFailed) + StorePath subPath, + nix::ref sub, + std::shared_ptr info, + bool & substituterFailed, + ActivityId parentAct) { trace("all references realised"); @@ -176,7 +216,8 @@ Goal::Co PathSubstitutionGoal::tryToRun( .status = BuildResult::Failure::DependencyFailed, .msg = HintFmt( "some references of path '%s' could not be realised", worker.store.printStorePath(storePath)), - }}); + }}, + parentAct); } for (auto & i : info->references) @@ -204,7 +245,7 @@ Goal::Co PathSubstitutionGoal::tryToRun( auto maintainRunningSubstitutions = std::make_unique>(worker.runningSubstitutions); worker.updateProgress(); - auto promise = std::promise(); + auto promise = std::promise>(); auto future = promise.get_future(); /* Be careful with ownership. cleanup() doesn't signal the worker thread @@ -218,7 +259,8 @@ Goal::Co PathSubstitutionGoal::tryToRun( repair = repair, sub, maybeWaker = worker.getCrossThreadWaker(), - maybeWorkerStore = worker.store.weak_from_this()]() mutable { + maybeWorkerStore = worker.store.weak_from_this(), + parentAct]() mutable { try { ReceiveInterrupts receiveInterrupts; @@ -230,12 +272,12 @@ Goal::Co PathSubstitutionGoal::tryToRun( Activity act( *logger, actSubstitute, - Logger::Fields{workerStore->printStorePath(storePath), sub->config.getHumanReadableURI()}); + Logger::Fields{workerStore->printStorePath(storePath), sub->config.getHumanReadableURI()}, + parentAct); PushActivity pact(act.id); - copyStorePath(*sub, *workerStore, subPath, repair, sub->config.isTrusted ? NoCheckSigs : CheckSigs); - - promise.set_value(); + promise.set_value( + copyStorePath(*sub, *workerStore, subPath, repair, sub->config.isTrusted ? NoCheckSigs : CheckSigs)); } catch (...) { promise.set_exception(std::current_exception()); } @@ -258,8 +300,12 @@ Goal::Co PathSubstitutionGoal::tryToRun( thr.join(); worker.childTerminated(this); + std::shared_ptr provenance; + try { - future.get(); + auto info = future.get(); + if (info) + provenance = info->provenance; } catch (std::exception & e) { /* Cause the parent build to fail unless --fallback is given, or the substitute has disappeared. The latter case behaves @@ -300,7 +346,12 @@ Goal::Co PathSubstitutionGoal::tryToRun( worker.updateProgress(); - co_return doneSuccess(BuildResult::Success{.status = BuildResult::Success::Substituted}); + auto success = BuildResult::Success{.status = BuildResult::Success::Substituted, .provenance = provenance}; + + logger->result( + parentAct, resBuildResult, nlohmann::json(KeyedBuildResult({success}, DerivedPath::Opaque{storePath}))); + + co_return doneSuccess(std::move(success)); } void PathSubstitutionGoal::cleanup() diff --git a/src/libstore/build/worker.cc b/src/libstore/build/worker.cc index 38b12e7107b1..b812d5c34b96 100644 --- a/src/libstore/build/worker.cc +++ b/src/libstore/build/worker.cc @@ -20,8 +20,8 @@ Worker::Worker(Store & store, Store & evalStore) /* Can't use make_ref, because the constructor is private. */ : wakerState(ref(new Waker{})) , act(*logger, actRealise) - , actDerivations(*logger, actBuilds) - , actSubstitutions(*logger, actCopyPaths) + , actDerivations(*logger, actBuilds, {}, act.id) + , actSubstitutions(*logger, actCopyPaths, {}, act.id) #ifdef _WIN32 , ioport{CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0)} #endif @@ -116,10 +116,10 @@ std::shared_ptr Worker::makeDerivationBuildingGoal( derivationBuildingGoals[drvPath], drvPath, std::move(drv), *this, buildMode, storeDerivation); } -std::shared_ptr -Worker::makePathSubstitutionGoal(const StorePath & path, RepairFlag repair, std::optional ca) +std::shared_ptr Worker::makePathSubstitutionGoal( + const StorePath & path, bool pathRequired, RepairFlag repair, std::optional ca) { - return initGoalIfNeeded(substitutionGoals[path], path, *this, repair, ca); + return initGoalIfNeeded(substitutionGoals[path], path, *this, pathRequired, repair, ca); } std::shared_ptr Worker::makeDrvOutputSubstitutionGoal(const DrvOutput & id) @@ -135,7 +135,7 @@ GoalPtr Worker::makeGoal(const DerivedPath & req, BuildMode buildMode) return makeDerivationTrampolineGoal(bfd.drvPath, bfd.outputs, buildMode); }, [&](const DerivedPath::Opaque & bo) -> GoalPtr { - return makePathSubstitutionGoal(bo.path, buildMode == bmRepair ? Repair : NoRepair); + return makePathSubstitutionGoal(bo.path, false, buildMode == bmRepair ? Repair : NoRepair); }, }, req.raw()); diff --git a/src/libstore/common-protocol.cc b/src/libstore/common-protocol.cc index dfddd042df81..e9bd5fa56459 100644 --- a/src/libstore/common-protocol.cc +++ b/src/libstore/common-protocol.cc @@ -26,13 +26,13 @@ void CommonProto::Serialise::write( StorePath CommonProto::Serialise::read(const StoreDirConfig & store, CommonProto::ReadConn conn) { - return store.parseStorePath(readString(conn.from)); + return conn.shortStorePaths ? StorePath(readString(conn.from)) : store.parseStorePath(readString(conn.from)); } void CommonProto::Serialise::write( const StoreDirConfig & store, CommonProto::WriteConn conn, const StorePath & storePath) { - conn.to << store.printStorePath(storePath); + conn.to << (conn.shortStorePaths ? storePath.to_string() : store.printStorePath(storePath)); } ContentAddress CommonProto::Serialise::read(const StoreDirConfig & store, CommonProto::ReadConn conn) @@ -50,13 +50,15 @@ std::optional CommonProto::Serialise>::read(const StoreDirConfig & store, CommonProto::ReadConn conn) { auto s = readString(conn.from); - return s == "" ? std::optional{} : store.parseStorePath(s); + return s == "" ? std::optional{} : conn.shortStorePaths ? StorePath(s) : store.parseStorePath(s); } void CommonProto::Serialise>::write( const StoreDirConfig & store, CommonProto::WriteConn conn, const std::optional & storePathOpt) { - conn.to << (storePathOpt ? store.printStorePath(*storePathOpt) : ""); + conn.to + << (storePathOpt ? (conn.shortStorePaths ? storePathOpt->to_string() : store.printStorePath(*storePathOpt)) + : ""); } std::optional @@ -105,6 +107,8 @@ constexpr static BuildResultStatus buildResultStatusTable[] = { BuildResultFailureStatus::NotDeterministic, // 12 BuildResultSuccessStatus::ResolvesToAlreadyValid, // 13 BuildResultFailureStatus::NoSubstituters, // 14 + BuildResultFailureStatus::HashMismatch, // 15 + BuildResultFailureStatus::Cancelled, // 16 }; BuildResultStatus diff --git a/src/libstore/daemon.cc b/src/libstore/daemon.cc index 563a5bafd030..9e0b54ebdfe1 100644 --- a/src/libstore/daemon.cc +++ b/src/libstore/daemon.cc @@ -17,6 +17,10 @@ #include "nix/util/args.hh" #include "nix/util/logging.hh" #include "nix/store/globals.hh" +#include "nix/store/active-builds.hh" +#include "nix/util/provenance.hh" +#include "nix/util/async.hh" + #include #ifndef _WIN32 // TODO need graceful async exit support on Windows? @@ -31,12 +35,13 @@ Sink & operator<<(Sink & sink, const Logger::Fields & fields) { sink << fields.size(); for (auto & f : fields) { - sink << f.type; - if (f.type == Logger::Field::tInt) - sink << f.i; - else if (f.type == Logger::Field::tString) - sink << f.s; - else + if (auto p = std::get_if(&f.raw)) { + sink << 0; + sink << *p; + } else if (auto p = std::get_if(&f.raw)) { + sink << 1; + sink << *p; + } else unreachable(); } return sink; @@ -67,7 +72,7 @@ struct TunnelLogger : public Logger { } - void enqueueMsg(const std::string & s) + void enqueueMsg(const std::string & s) noexcept { auto state(state_.lock()); @@ -78,15 +83,18 @@ struct TunnelLogger : public Logger to.flush(); } catch (...) { /* Write failed; that means that the other side is - gone. */ + gone, so stop sending it messages. Note that we + don't propagate the error, since logging must not + throw. The client's death will be detected + elsewhere (e.g. by `MonitorFdHup` or by the next + protocol read/write). */ state->canSendStderr = false; - throw; } } else state->pendingMsgs.push_back(s); } - void log(Verbosity lvl, std::string_view s) override + void log(Verbosity lvl, std::string_view s) noexcept override { if (lvl > verbosity) return; @@ -96,7 +104,7 @@ struct TunnelLogger : public Logger enqueueMsg(buf.s); } - void logEI(const ErrorInfo & ei) override + void logEI(const ErrorInfo & ei) noexcept override { if (ei.level > verbosity) return; @@ -149,7 +157,7 @@ struct TunnelLogger : public Logger ActivityType type, const std::string & s, const Fields & fields, - ActivityId parent) override + ActivityId parent) noexcept override { if (clientVersion.number < WorkerProto::Version::Number{1, 20}) { if (!s.empty()) @@ -162,7 +170,7 @@ struct TunnelLogger : public Logger enqueueMsg(buf.s); } - void stopActivity(ActivityId act) override + void stopActivity(ActivityId act) noexcept override { if (clientVersion.number < WorkerProto::Version::Number{1, 20}) return; @@ -171,7 +179,7 @@ struct TunnelLogger : public Logger enqueueMsg(buf.s); } - void result(ActivityId act, ResultType type, const Fields & fields) override + void result(ActivityId act, ResultType type, const Fields & fields) noexcept override { if (clientVersion.number < WorkerProto::Version::Number{1, 20}) return; @@ -284,7 +292,7 @@ struct ClientSettings trusted || name == settings.getWorkerSettings().buildTimeout.name || name == settings.getWorkerSettings().maxSilentTime.name || name == settings.getWorkerSettings().pollInterval.name || name == "connect-timeout" - || (name == "builders" && value == "")) { + || name == loggerSettings.sessionId.name || (name == "builders" && value == "")) { settings.set(name, value); fileTransferSettings.set(name, value); } else if (setSubstituters(settings.getWorkerSettings().substituters)) @@ -410,6 +418,9 @@ static void performOp( bool repairBool; conn.from >> repairBool; auto repair = RepairFlag{repairBool}; + auto provenance = conn.protoVersion.features.contains(WorkerProto::featureProvenance) + ? Provenance::from_json_str_optional(readString(conn.from)) + : nullptr; logger->startWork(); auto pathInfo = [&]() { @@ -435,8 +446,8 @@ static void performOp( assert(false); } // TODO these two steps are essentially RemoteStore::addCAToStore. Move it up to Store. - auto path = - store->addToStoreFromDump(source, name, dumpMethod, contentAddressMethod, hashAlgo, refs, repair); + auto path = store->addToStoreFromDump( + source, name, dumpMethod, contentAddressMethod, hashAlgo, refs, repair, provenance); return store->queryPathInfo(path); }(); logger->stopWork(); @@ -503,7 +514,9 @@ static void performOp( *store, WorkerProto::ReadConn{ .from = source, - .version = {.number = {.major = 1, .minor = 16}}, + .version = conn.protoVersion.features.contains(WorkerProto::featureVersionedAddToStoreMultiple) + ? conn.protoVersion + : WorkerProto::Version{.number = {.major = 1, .minor = 16}}, }); info.ultimate = false; EnsureRead wrapper{source, info.narSize}; @@ -675,6 +688,15 @@ static void performOp( break; } + case WorkerProto::Op::AddTempRoots: { + auto paths = WorkerProto::Serialise::read(*store, rconn); + logger->startWork(); + store->addTempRoots(paths); + logger->stopWork(); + conn.to << 1; + break; + } + case WorkerProto::Op::AddPermRoot: { if (!trusted) throw Error( @@ -747,6 +769,7 @@ static void performOp( }; } conn.from >> options.ignoreLiveness >> options.maxFreed; + options.censor = !trusted; // obsolete fields readInt(conn.from); readInt(conn.from); @@ -762,7 +785,7 @@ static void performOp( GCResults results; logger->startWork(); - if (options.ignoreLiveness) + if (options.ignoreLiveness && !getEnv("_NIX_IN_TEST").has_value()) throw Error("you are not allowed to ignore liveness"); auto & gcStore = require(*store); gcStore.collectGarbage(options, results); @@ -874,6 +897,37 @@ static void performOp( break; } + case WorkerProto::Op::QueryPathInfos: { + auto paths = WorkerProto::Serialise::read(*store, rconn); + logger->startWork(); + std::vector infos; + { + asio::io_context ctx; + std::exception_ptr ex; + asio::co_spawn( + ctx, + [&]() -> asio::awaitable { + co_await store->queryPathInfos( + paths, [&](std::vector>> results) { + for (auto & [path, info] : results) + if (info) + infos.push_back(*info); + }); + }, + [&](std::exception_ptr e) { ex = e; }); + ctx.run(); + if (ex) + std::rethrow_exception(ex); + } + logger->stopWork(); + /* Write the infos for the valid paths. Paths not reported are + invalid. */ + conn.to << infos.size(); + for (auto & info : infos) + WorkerProto::write(*store, wconn, info); + break; + } + case WorkerProto::Op::OptimiseStore: logger->startWork(); store->optimiseStore(); @@ -922,6 +976,9 @@ static void performOp( conn.from >> info.registrationTime >> info.narSize >> info.ultimate; info.sigs = WorkerProto::Serialise>::read(*store, rconn); info.ca = ContentAddress::parseOpt(readString(conn.from)); + info.provenance = conn.protoVersion.features.contains(WorkerProto::featureProvenance) + ? Provenance::from_json_str_optional(readString(conn.from)) + : nullptr; conn.from >> repair >> dontCheckSigs; if (!trusted && dontCheckSigs) dontCheckSigs = false; @@ -1014,14 +1071,38 @@ static void performOp( break; } + case WorkerProto::Op::QueryActiveBuilds: { + logger->startWork(); + auto & activeBuildsStore = require(*store); + auto activeBuilds = activeBuildsStore.queryActiveBuilds(); + logger->stopWork(); + conn.to << nlohmann::json(activeBuilds).dump(); + break; + } + default: throw Error("invalid operation %1%", op); } } -void processConnection(ref store, FdSource && from, FdSink && to, TrustedFlag trusted, RecursiveFlag recursive) +void processConnection( + ref store, + FdSource && from, + FdSink && to, + TrustedFlag trusted, + RecursiveFlag recursive, + std::function setupTelemetry) { #ifndef _WIN32 // TODO need graceful async exit support on Windows? + /* The client hanging up triggers an interrupt (via `MonitorFdHup` + below), which is how we abort whatever we were doing for it. + Once the connection is over, that interrupt has served its + purpose, so clear it, e.g. so that we can still export our + telemetry. Note: this has to run *after* the monitor has been + destroyed, i.e. its thread joined, since it might otherwise + still trigger the interrupt after we've cleared it. */ + Finally clearInterrupt([]() { setInterrupted(false); }); + auto monitor = !recursive ? std::make_unique(from.fd) : nullptr; (void) monitor; // suppress warning ReceiveInterrupts receiveInterrupts; @@ -1041,6 +1122,8 @@ void processConnection(ref store, FdSource && from, FdSink && to, Trusted auto localVersion = WorkerProto::latest; if (recursive) localVersion.features.insert(std::string{WorkerProto::featureDisableSetOptions}); + if (!experimentalFeatureSettings.isEnabled(Xp::Provenance)) + localVersion.features.erase(std::string(WorkerProto::featureProvenance)); WorkerProto::BasicServerConnection conn; conn.protoVersion = WorkerProto::BasicServerConnection::handshake(to, from, localVersion); @@ -1048,6 +1131,10 @@ void processConnection(ref store, FdSource && from, FdSink && to, Trusted if (conn.protoVersion.number < WorkerProto::minimum.number) throw Error("the Nix client version is too old"); + std::string traceparent; + if (conn.protoVersion.features.contains(WorkerProto::featureOpenTelemetry)) + traceparent = readString(from); + conn.to = std::move(to); conn.from = std::move(from); @@ -1059,12 +1146,12 @@ void processConnection(ref store, FdSource && from, FdSink && to, Trusted applyJSONLogger(); } + if (setupTelemetry) + setupTelemetry(traceparent); + unsigned int opCount = 0; - Finally finally([&]() { - setInterrupted(false); - printMsgUsing(prevLogger, lvlDebug, "%d operations", opCount); - }); + Finally finally([&]() { printMsgUsing(prevLogger, lvlDebug, "%d operations", opCount); }); conn.postHandshake( *store, @@ -1094,12 +1181,32 @@ void processConnection(ref store, FdSource && from, FdSink && to, Trusted break; } + std::string traceparent; + if (conn.protoVersion.features.contains(WorkerProto::featureOpenTelemetry)) + traceparent = readString(conn.from); + printMsgUsing(prevLogger, lvlDebug, "received daemon op %d", op); opCount++; debug("performing daemon worker op: %d", op); + /* Parent our work under the client activity that + initiated this operation. */ + std::optional act; + std::optional pact; + if (!traceparent.empty()) { + act.emplace( + *logger, + lvlDebug, + "daemon operation", + std::to_array>({ + {"nix.daemon.op", (uint64_t) op}, + {"traceparent", traceparent}, + })); + pact.emplace(act->id); + } + try { performOp(tunnelLogger, store, trusted, recursive, conn, op); } catch (Error & e) { @@ -1112,10 +1219,6 @@ void processConnection(ref store, FdSource && from, FdSink && to, Trusted tunnelLogger->stopWork(&e); if (!errorAllowed) throw; - } catch (std::bad_alloc & e) { - auto ex = Error("Nix daemon out of memory"); - tunnelLogger->stopWork(&ex); - throw; } conn.to.flush(); diff --git a/src/libstore/derivations.cc b/src/libstore/derivations.cc index 1dfb507adad1..7cd9457aca9b 100644 --- a/src/libstore/derivations.cc +++ b/src/libstore/derivations.cc @@ -7,6 +7,7 @@ #include "nix/store/common-protocol-impl.hh" #include "nix/util/strings-inline.hh" #include "nix/util/json-utils.hh" +#include "nix/store/async-path-writer.hh" #include #include @@ -55,6 +56,7 @@ bool DerivationType::isCA() const [](const InputAddressed & ia) { return false; }, [](const ContentAddressed & ca) { return true; }, [](const Impure &) { return true; }, + [](const Substituted &) { return false; }, }, raw); } @@ -66,6 +68,7 @@ bool DerivationType::isFixed() const [](const InputAddressed & ia) { return false; }, [](const ContentAddressed & ca) { return ca.fixed; }, [](const Impure &) { return false; }, + [](const Substituted &) { return false; }, }, raw); } @@ -77,6 +80,7 @@ bool DerivationType::hasKnownOutputPaths() const [](const InputAddressed & ia) { return !ia.deferred; }, [](const ContentAddressed & ca) { return ca.fixed; }, [](const Impure &) { return false; }, + [](const Substituted &) { return true; }, }, raw); } @@ -88,6 +92,7 @@ bool DerivationType::isSandboxed() const [](const InputAddressed & ia) { return true; }, [](const ContentAddressed & ca) { return ca.sandboxed; }, [](const Impure &) { return false; }, + [](const Substituted &) { return true; }, }, raw); } @@ -99,6 +104,7 @@ bool DerivationType::isImpure() const [](const InputAddressed & ia) { return false; }, [](const ContentAddressed & ca) { return false; }, [](const Impure &) { return true; }, + [](const Substituted &) { return false; }, }, raw); } @@ -134,7 +140,8 @@ StorePath computeStorePath(const StoreDirConfig & store, const Derivation & drv) return path; } -StorePath Store::writeDerivation(const Derivation & drv, RepairFlag repair) +StorePath +Store::writeDerivation(const Derivation & drv, RepairFlag repair, std::shared_ptr provenance) { auto [suffix, contents, references, path] = infoForDerivation(*this, drv); @@ -155,12 +162,26 @@ StorePath Store::writeDerivation(const Derivation & drv, RepairFlag repair) ContentAddressMethod::Raw::Text, HashAlgorithm::SHA256, references, - repair); + repair, + provenance); assert(path2 == path); return path; } +StorePath Store::writeDerivation( + AsyncPathWriter & asyncPathWriter, + const Derivation & drv, + RepairFlag repair, + std::shared_ptr provenance) +{ + auto references = drv.inputSrcs; + for (auto & i : drv.inputDrvs.map) + references.insert(i.first); + return asyncPathWriter.addPath( + drv.unparse(*this, false), std::string(drv.name) + drvExtension, references, repair, provenance); +} + namespace { /** * This mimics std::istream to some extent. We use this much smaller implementation @@ -856,6 +877,14 @@ DerivationType BasicDerivation::type() const if (!ty) throw Error("must have at least one output"); + if (builder == "builtin:substitute") { + experimentalFeatureSettings.require(Xp::BakedDerivations, "'builtin:substitute' derivation"); + auto * ia = std::get_if(&ty.value().raw); + if (!ia || ia->deferred) + throw Error("'builtin:substitute' derivation must have input-addressed outputs"); + return DerivationType::Substituted{}; + } + return ty.value(); } @@ -925,7 +954,8 @@ DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool m assert(!ca.fixed); return true; }, - [](const DerivationType::Impure &) { return true; }}, + [](const DerivationType::Impure &) { return true; }, + [](const DerivationType::Substituted &) { return false; }}, drv.type().raw)) { return DrvHashModulo::DeferredDrv{}; } @@ -1139,6 +1169,7 @@ bool Derivation::shouldResolve() const : true; }, [&](const DerivationType::Impure &) { return true; }, + [&](const DerivationType::Substituted &) { return false; }, }, drvType.raw); @@ -1260,6 +1291,20 @@ std::optional Derivation::tryResolve( template static void processDerivationOutputPaths(Store & store, auto && drv, std::string_view drvName) { + if (drv.builder == "builtin:substitute") { + if (drv.platform != "builtin") + throw Error("'builtin:substitute' derivation must be a builtin"); + if (!drv.args.empty()) + throw Error("'builtin:substitute' derivation must have no arguments"); + if (!drv.env.empty()) + throw Error("'builtin:substitute' derivation must have no environment variables"); + if (!drv.inputSrcs.empty()) + throw Error("'builtin:substitute' derivation must have no inputs"); + /* Check the output types (and the experimental feature). */ + drv.type(); + return; + } + std::optional hashModulo_; auto hashModulo = [&]() -> const auto & { diff --git a/src/libstore/dummy-store.cc b/src/libstore/dummy-store.cc index 6238c8890499..3c4de14edecd 100644 --- a/src/libstore/dummy-store.cc +++ b/src/libstore/dummy-store.cc @@ -228,7 +228,9 @@ struct DummyStoreImpl : DummyStore if (info.path.isDerivation()) { warn("back compat supporting `addToStore` for inserting derivations in dummy store"); writeDerivation( - parseDerivation(*this, accessor->readFile(CanonPath::root), Derivation::nameFromPath(info.path))); + parseDerivation(*this, accessor->readFile(CanonPath::root), Derivation::nameFromPath(info.path)), + repair, + info.provenance); return; } @@ -245,11 +247,12 @@ struct DummyStoreImpl : DummyStore StorePath addToStoreFromDump( Source & source, std::string_view name, - FileSerialisationMethod dumpMethod = FileSerialisationMethod::NixArchive, - ContentAddressMethod hashMethod = FileIngestionMethod::NixArchive, - HashAlgorithm hashAlgo = HashAlgorithm::SHA256, - const StorePathSet & references = StorePathSet(), - RepairFlag repair = NoRepair) override + FileSerialisationMethod dumpMethod, + ContentAddressMethod hashMethod, + HashAlgorithm hashAlgo, + const StorePathSet & references, + RepairFlag repair, + std::shared_ptr provenance) override { if (isDerivation(name)) throw Error("Do not insert derivation into dummy store with `addToStoreFromDump`"); @@ -297,6 +300,7 @@ struct DummyStoreImpl : DummyStore std::move(narHash.first)); info.narSize = narHash.second.value(); + info.provenance = provenance; auto path = info.path; auto accessor = make_ref(std::move(*temp)); @@ -312,7 +316,8 @@ struct DummyStoreImpl : DummyStore return path; } - StorePath writeDerivation(const Derivation & drv, RepairFlag repair = NoRepair) override + StorePath + writeDerivation(const Derivation & drv, RepairFlag repair, std::shared_ptr provenance) override { auto drvPath = nix::computeStorePath(*this, drv); @@ -320,6 +325,7 @@ struct DummyStoreImpl : DummyStore if (config->readOnly) unsupported("writeDerivation"); derivations.insert({drvPath, drv}); + // FIXME: record provenance } return drvPath; diff --git a/src/libstore/export-import.cc b/src/libstore/export-import.cc index c0b76158a822..8dec3fcd5080 100644 --- a/src/libstore/export-import.cc +++ b/src/libstore/export-import.cc @@ -4,91 +4,173 @@ #include "nix/util/archive.hh" #include "nix/store/common-protocol.hh" #include "nix/store/common-protocol-impl.hh" - -#include +#include "nix/store/worker-protocol.hh" namespace nix { -static void exportPath(Store & store, const StorePath & path, Sink & sink) -{ - auto info = store.queryPathInfo(path); - - HashSink hashSink(HashAlgorithm::SHA256); - TeeSink teeSink(sink, hashSink); - - store.narFromPath(path, teeSink); - - /* Refuse to export paths that have changed. This prevents - filesystem corruption from spreading to other machines. - Don't complain if the stored hash is zero (unknown). */ - Hash hash = hashSink.currentHash().hash; - if (hash != info->narHash && info->narHash != Hash(info->narHash.algo)) - throw Error( - "hash of path '%s' has changed from '%s' to '%s'!", - store.printStorePath(path), - info->narHash.to_string(HashFormat::Nix32, true), - hash.to_string(HashFormat::Nix32, true)); - - teeSink << exportMagic << store.printStorePath(path); - CommonProto::write(store, CommonProto::WriteConn{.to = teeSink}, info->references); - teeSink << (info->deriver ? store.printStorePath(*info->deriver) : "") << 0; -} +static const uint32_t exportMagicV1 = 0x4558494e; +static const uint64_t exportMagicV2 = 0x324f4952414e; // = 'NARIO2' -void exportPaths(Store & store, const StorePathSet & paths, Sink & sink) +static WorkerProto::Version exportProtoVersion{ + .number = + { + .major = 1, + .minor = 16, + }, +}; + +void exportPaths(Store & store, const StorePathSet & paths, Sink & sink, unsigned int version) { auto sorted = store.topoSortPaths(paths); - - for (auto & path : sorted | std::views::reverse) { - sink << 1; - exportPath(store, path, sink); + std::reverse(sorted.begin(), sorted.end()); + + auto dumpNar = [&](const ValidPathInfo & info) { + HashSink hashSink(HashAlgorithm::SHA256); + TeeSink teeSink(sink, hashSink); + + store.narFromPath(info.path, teeSink); + + /* Refuse to export paths that have changed. This prevents + filesystem corruption from spreading to other machines. + Don't complain if the stored hash is zero (unknown). */ + Hash hash = hashSink.currentHash().hash; + if (hash != info.narHash && info.narHash != Hash(info.narHash.algo)) + throw Error( + "hash of path '%s' has changed from '%s' to '%s'!", + store.printStorePath(info.path), + info.narHash.to_string(HashFormat::Nix32, true), + hash.to_string(HashFormat::Nix32, true)); + }; + + switch (version) { + + case 1: + for (auto & path : sorted) { + sink << 1; + auto info = store.queryPathInfo(path); + dumpNar(*info); + sink << exportMagicV1 << store.printStorePath(path); + CommonProto::write(store, CommonProto::WriteConn{.to = sink}, info->references); + sink << (info->deriver ? store.printStorePath(*info->deriver) : "") << 0; + } + sink << 0; + break; + + case 2: + sink << exportMagicV2; + + for (auto & path : sorted) { + Activity act(*logger, lvlTalkative, actUnknown, fmt("exporting path '%s'", store.printStorePath(path))); + sink << 1; + auto info = store.queryPathInfo(path); + // FIXME: move to CommonProto? + WorkerProto::Serialise::write( + store, + WorkerProto::WriteConn{.to = sink, .version = exportProtoVersion, .shortStorePaths = true}, + *info); + dumpNar(*info); + } + + sink << 0; + break; + + default: + throw Error("unsupported nario version %d", version); } - - sink << 0; } StorePaths importPaths(Store & store, Source & source, CheckSigsFlag checkSigs) { StorePaths res; - while (true) { - auto n = readNum(source); - if (n == 0) - break; - if (n != 1) - throw Error("input doesn't look like something created by 'nix-store --export'"); - - /* Extract the NAR from the source. */ + + auto version = readNum(source); + + /* Note: nario version 1 lacks an explicit header. The first + integer denotes whether a store path follows or not. So look + for 0 or 1. */ + switch (version) { + + case 0: + /* Empty version 1 nario, nothing to do. */ + break; + + case 1: { + /* Reuse a string buffer to avoid kernel overhead allocating + memory for large strings. */ StringSink saved; - TeeSource tee{source, saved}; - NullFileSystemObjectSink ether; - parseDump(ether, tee); - uint32_t magic = readInt(source); - if (magic != exportMagic) - throw Error("Nix archive cannot be imported; wrong format"); + /* Non-empty version 1 nario. */ + while (true) { + /* Extract the NAR from the source. */ + saved.s.clear(); + TeeSource tee{source, saved}; + NullFileSystemObjectSink ether; + parseDump(ether, tee); + + uint32_t magic = readInt(source); + if (magic != exportMagicV1) + throw Error("nario cannot be imported; wrong format"); + + auto path = store.parseStorePath(readString(source)); + + Activity act(*logger, lvlTalkative, actUnknown, fmt("importing path '%s'", store.printStorePath(path))); + + auto references = CommonProto::Serialise::read(store, CommonProto::ReadConn{.from = source}); + auto deriver = readString(source); + + // Ignore optional legacy signature. + if (readInt(source) == 1) + readString(source); + + if (!store.isValidPath(path)) { + auto narHash = hashString(HashAlgorithm::SHA256, saved.s); + + ValidPathInfo info{path, {store, narHash}}; + if (deriver != "") + info.deriver = store.parseStorePath(deriver); + info.references = references; + info.narSize = saved.s.size(); + + // Can't use underlying source, which would have been exhausted. + auto source2 = StringSource(saved.s); + store.addToStore(info, source2, NoRepair, checkSigs); + } + + res.push_back(path); + + auto n = readNum(source); + if (n == 0) + break; + if (n != 1) + throw Error("input doesn't look like a nario"); + } + break; + } - auto path = store.parseStorePath(readString(source)); + case exportMagicV2: + while (true) { + auto n = readNum(source); + if (n == 0) + break; + if (n != 1) + throw Error("input doesn't look like a nario"); - // Activity act(*logger, lvlInfo, "importing path '%s'", info.path); + auto info = WorkerProto::Serialise::read( + store, WorkerProto::ReadConn{.from = source, .version = exportProtoVersion, .shortStorePaths = true}); - auto references = CommonProto::Serialise::read(store, CommonProto::ReadConn{.from = source}); - auto deriver = readString(source); - auto narHash = hashString(HashAlgorithm::SHA256, saved.s); + Activity act( + *logger, lvlTalkative, actUnknown, fmt("importing path '%s'", store.printStorePath(info.path))); - ValidPathInfo info{path, {store, narHash}}; - if (deriver != "") - info.deriver = store.parseStorePath(deriver); - info.references = references; - info.narSize = saved.s.size(); + EnsureRead wrapper{source, info.narSize}; + store.addToStore(info, wrapper, NoRepair, checkSigs); - // Ignore optional legacy signature. - if (readInt(source) == 1) - readString(source); + res.push_back(info.path); + } - // Can't use underlying source, which would have been exhausted - auto source = StringSource(saved.s); - store.addToStore(info, source, NoRepair, checkSigs); + break; - res.push_back(info.path); + default: + throw Error("input doesn't look like a nario"); } return res; diff --git a/src/libstore/filetransfer.cc b/src/libstore/filetransfer.cc index 6e9f2256eaa0..24714d2623d8 100644 --- a/src/libstore/filetransfer.cc +++ b/src/libstore/filetransfer.cc @@ -4,6 +4,7 @@ #include "nix/util/config-global.hh" #include "nix/util/finally.hh" #include "nix/util/callback.hh" +#include "nix/util/processes.hh" #include "nix/util/signals.hh" #include "nix/util/util.hh" @@ -22,6 +23,9 @@ #include +#include + +#include #include #include #include @@ -59,6 +63,23 @@ constexpr bool operator==(long lhs, HttpStatus rhs) noexcept return lhs == static_cast(rhs); } +const char * httpMethodName(HttpMethod method) +{ + switch (method) { + case HttpMethod::Get: + return "GET"; + case HttpMethod::Put: + return "PUT"; + case HttpMethod::Head: + return "HEAD"; + case HttpMethod::Post: + return "POST"; + case HttpMethod::Delete: + return "DELETE"; + } + unreachable(); +} + } // namespace std::chrono::milliseconds computeRetryDelayMs(const RetryDelayParams & p, std::mt19937 & rng) @@ -160,6 +181,12 @@ struct curlFileTransfer : public FileTransfer FileTransferRequest request; FileTransferResult result; std::unique_ptr _act; + + /** + * Activity covering the current HTTP request. Recreated on + * every attempt, as a child of the transfer activity. + */ + std::unique_ptr attemptAct; Callback callback; CURL * req = 0; // buffer to accompany the `req` above @@ -199,6 +226,11 @@ struct curlFileTransfer : public FileTransfer */ bool acceptRanges:1 = false; + /** + * When retrying, whether to request a range. + */ + bool requestRange:1 = false; + /** * Whether the response has a non-trivial (not "identity") Content-Encoding. */ @@ -211,6 +243,8 @@ struct curlFileTransfer : public FileTransfer */ std::optional retryAfterMs; + curl_off_t bytesReceived = 0; + curl_off_t writtenToSink = 0; std::chrono::steady_clock::time_point startTime = std::chrono::steady_clock::now(); @@ -238,7 +272,7 @@ struct curlFileTransfer : public FileTransfer { curlSList tmpSList = curlSList(::curl_slist_append(requestHeaders.get(), requireCString(header))); if (!tmpSList) - throw std::bad_alloc(); + outOfMemory(); requestHeaders.release(); requestHeaders = std::move(tmpSList); } @@ -261,6 +295,18 @@ struct curlFileTransfer : public FileTransfer /* Only write data to the sink if this is a successful response. */ if (successfulStatuses.count(httpStatus)) { + + auto prevReceived = bytesReceived; + bytesReceived += data.size(); + + /* Discard data that we've already received and sent to the sink in a previous try. */ + if (httpStatus != 206 && prevReceived < writtenToSink) { + if (writtenToSink - prevReceived >= (curl_off_t) data.size()) { + return; + } + data = data.substr(writtenToSink - prevReceived); + } + writtenToSink += data.size(); PauseTransfer needsPause = this->request.dataCallback(data); if (needsPause == PauseTransfer::Yes) { @@ -275,14 +321,6 @@ struct curlFileTransfer : public FileTransfer }) { result.urls.push_back(request.uri.to_string()); - - if (!request.expectedETag.empty()) - appendHeaders("If-None-Match: " + request.expectedETag); - if (!request.mimeType.empty()) - appendHeaders("Content-Type: " + request.mimeType); - for (auto it = request.headers.begin(); it != request.headers.end(); ++it) { - appendHeaders(fmt("%s: %s", it->first, it->second)); - } } ~TransferItem() @@ -292,12 +330,8 @@ struct curlFileTransfer : public FileTransfer curl_multi_remove_handle(fileTransfer.curlm.get(), req); curl_easy_cleanup(req); } - try { - if (!done && enqueued) - failInterruptedOrCancelled(); - } catch (...) { - ignoreExceptionInDestructor(); - } + if (!done && enqueued) + failInterruptedOrCancelled(); } void failEx(std::exception_ptr ex) noexcept @@ -397,6 +431,7 @@ struct curlFileTransfer : public FileTransfer acceptRanges = false; hasContentEncoding = false; retryAfterMs = std::nullopt; + bytesReceived = 0; appendCurrentUrl(); } else { @@ -588,6 +623,37 @@ struct curlFileTransfer : public FileTransfer curl_easy_reset(req); + bytesReceived = 0; + + /* Start an activity for this HTTP request. Note that + act() is called here (rather than lazily from a libcurl + callback) so that the activity exists before the + request headers are built. */ + attemptAct = std::make_unique( + *logger, + lvlDebug, + "FileTransferAttempt", + std::to_array>({ + {"url.full", request.displayUri()}, + {"http.request.method", httpMethodName(request.method)}, + {"http.request.resend_count", (uint64_t) attempt}, + }), + "", + act().id); + + /* (Re)build the request headers, since the traceparent + header differs per attempt. */ + requestHeaders = curlSList{}; + if (!request.expectedETag.empty()) + appendHeaders("If-None-Match: " + request.expectedETag); + if (!request.mimeType.empty()) + appendHeaders("Content-Type: " + request.mimeType); + for (auto it = request.headers.begin(); it != request.headers.end(); ++it) { + appendHeaders(fmt("%s: %s", it->first, it->second)); + } + for (auto & [name, value] : logger->getTraceContext(attemptAct->id)) + appendHeaders(fmt("%s: %s", name, value)); + if (verbosity >= lvlVomit) { curl_easy_setopt(req, CURLOPT_VERBOSE, 1); curl_easy_setopt(req, CURLOPT_DEBUGFUNCTION, TransferItem::debugCallback); @@ -599,7 +665,7 @@ struct curlFileTransfer : public FileTransfer Skip for uploads (Accept-Encoding is meaningless when sending data) and when resuming from an offset (byte ranges don't work with compressed content). */ - if (writtenToSink == 0 && !request.data) + if (!requestRange && !request.data) /* Empty string means to enable all supported (that libcurl has been linked to support) encodings. */ curl_easy_setopt(req, CURLOPT_ACCEPT_ENCODING, ""); @@ -610,7 +676,7 @@ struct curlFileTransfer : public FileTransfer curl_easy_setopt( req, CURLOPT_USERAGENT, - ("curl/" LIBCURL_VERSION " Nix/" + nixVersion + ("curl/" LIBCURL_VERSION " Nix/" + nixVersion + " DeterminateNix/" + determinateNixVersion + (fileTransfer.settings.userAgentSuffix != "" ? " " + fileTransfer.settings.userAgentSuffix.get() : "")) .c_str()); @@ -695,7 +761,7 @@ struct curlFileTransfer : public FileTransfer curl_easy_setopt(req, CURLOPT_NETRC_FILE, fileTransfer.settings.netrcFile.get().string().c_str()); curl_easy_setopt(req, CURLOPT_NETRC, CURL_NETRC_OPTIONAL); - if (writtenToSink) + if (requestRange) curl_easy_setopt(req, CURLOPT_RESUME_FROM_LARGE, writtenToSink); /* Note that the underlying strings get copied by libcurl, so the path -> string conversion is ok: @@ -765,6 +831,15 @@ struct curlFileTransfer : public FileTransfer httpStatus = std::to_underlying(HttpStatus::NotModified); } + if (attemptAct) { + nlohmann::json json; + if (httpStatus) + json["httpStatus"] = httpStatus; + json["bodySize"] = result.bodySize; + logger->result(attemptAct->id, resHttpStatus, json); + attemptAct.reset(); + } + if (callbackException) failEx(callbackException); @@ -933,12 +1008,14 @@ struct curlFileTransfer : public FileTransfer return false; if (attempt >= effAttempts) return false; - // If we've already streamed bytes to the callback, we can only - // resume via a Range request. That requires the server to accept - // byte ranges AND the response to be uncompressed (the Range - // applies to the encoded stream, but the sink saw decoded bytes). + // If we've already streamed bytes to the callback, we can + // resume via a Range request (if the server accepts byte + // ranges), or start over and discard the data we've already + // received. Neither works if the response is compressed (the + // Range applies to the encoded stream, but the sink saw + // decoded bytes). if (request.dataCallback && writtenToSink != 0) - return acceptRanges && !hasContentEncoding; + return !hasContentEncoding; return true; }(); @@ -958,6 +1035,8 @@ struct curlFileTransfer : public FileTransfer fileTransfer.mt19937); if (writtenToSink) { + if (acceptRanges) + requestRange = true; warn( "%s; retrying from offset %d in %d ms (attempt %d/%d)", exc.message(), @@ -1060,8 +1139,13 @@ struct curlFileTransfer : public FileTransfer void workerThreadMain() { + /* NOTE(cole-h): the maxQueueSize needs to be >0 or else things will hang */ + assert(maxQueueSize > 0); + /* Cause this thread to be notified on SIGINT. */ -#ifndef _WIN32 // TODO need graceful async exit support on Windows? +#if !defined(_WIN32) && !defined(IS_STATIC) // TODO need graceful async exit support on Windows? + // FIXME(RossComputerGuy): this causes issues on static builds. + // In particular, it causes a segfault to happen at the end of the program running. auto callback = createInterruptCallback([&]() { stopWorkerThread(); }); #endif @@ -1218,16 +1302,33 @@ struct curlFileTransfer : public FileTransfer return ItemHandle(item.get_ptr()); } - ItemHandle enqueueFileTransfer(const FileTransferRequest & request, Callback callback) override + inline ref + makeTransferItem(const FileTransferRequest & request, Callback callback) { /* Handle s3:// URIs by converting to HTTPS and optionally adding auth */ if (request.uri.scheme() == "s3") { auto modifiedRequest = request; modifiedRequest.setupForS3(); - return enqueueItem(make_ref(*this, std::move(modifiedRequest), std::move(callback))); + return make_ref(*this, std::move(modifiedRequest), std::move(callback)); + } else { + return make_ref(*this, request, std::move(callback)); } + } - return enqueueItem(make_ref(*this, request, std::move(callback))); + ItemHandle + enqueueFileTransfer(const FileTransferRequest & request, Callback callback) noexcept override + { + const auto item = makeTransferItem(request, std::move(callback)); + + try { + return enqueueItem(item); + } catch (const nix::BaseError &) { + // NOTE(cole-h): catches both nix::Error and nix::Interrupted -- enqueueItem calls + // writeFull which may throw nix::Interrupted, and the rest of enqueueItem may throw + // nix::Error + item->failEx(std::current_exception()); + return ItemHandle(item.get_ptr()); + } } void unpauseTransfer(std::weak_ptr item) @@ -1272,6 +1373,17 @@ ref getFileTransfer() return ref(*fileTransfer); } +/* The curl worker thread doesn't exist in a forked child, so discard + the singleton there. Note that it looks healthy otherwise, so + `getFileTransfer()` wouldn't replace it by itself. */ +static RegisterForkCallback resetFileTransfer([]() { + auto fileTransfer(_fileTransfer->lock()); + /* Deliberately leak the previous object: destroying it would join + its worker thread, which doesn't exist in this process. */ + new std::shared_ptr(std::move(*fileTransfer)); + fileTransfer->reset(); +}); + ref makeFileTransfer(const FileTransferSettings & settings) { return makeCurlFileTransfer(settings); @@ -1321,7 +1433,7 @@ void FileTransferRequest::setupForS3() #endif } -std::future FileTransfer::enqueueFileTransfer(const FileTransferRequest & request) +std::future FileTransfer::enqueueFileTransfer(const FileTransferRequest & request) noexcept { auto promise = std::make_shared>(); enqueueFileTransfer(request, {[promise](std::future fut) { diff --git a/src/libstore/gc.cc b/src/libstore/gc.cc index 8e997c8e2471..f5ea569cda24 100644 --- a/src/libstore/gc.cc +++ b/src/libstore/gc.cc @@ -81,7 +81,7 @@ void LocalStore::createTempRootsFile() } } -void LocalStore::addTempRoot(const StorePath & path) +void LocalStore::addTempRoots(const StorePathSet & paths, bool _skipIfSlow) { if (config->readOnly) { debug( @@ -130,12 +130,14 @@ void LocalStore::addTempRoot(const StorePath & path) } try { - debug("sending GC root '%s'", printStorePath(path)); - writeFull(fdRootsSocket->get(), printStorePath(path) + "\n", false); - char c; - readFull(fdRootsSocket->get(), &c, 1); - assert(c == '1'); - debug("got ack for GC root '%s'", printStorePath(path)); + for (auto & path : paths) { + debug("sending GC root '%s'", printStorePath(path)); + writeFull(fdRootsSocket->get(), printStorePath(path) + "\n", false); + char c; + readFull(fdRootsSocket->get(), &c, 1); + assert(c == '1'); + debug("got ack for GC root '%s'", printStorePath(path)); + } } catch (SystemError & e) { /* The garbage collector may have exited, so we need to restart. */ @@ -152,10 +154,22 @@ void LocalStore::addTempRoot(const StorePath & path) } } - /* Record the store path in the temporary roots file so it will be + /* Record the store paths in the temporary roots file so they will be seen by a future run of the garbage collector. */ - auto s = printStorePath(path) + '\0'; - writeFull(_fdTempRoots.lock()->get(), s); + + std::string s; + + for (auto & path : paths) + s += printStorePath(path) + '\0'; + + { + auto fdTempRoots(_fdTempRoots.lock()); + + /* This might not be atomic, but that's fine. Writes go in-order, and if + we partially write a store path, findTempRoots() will just ignore it, + and we'll send it the new temproots below if it's still running. */ + writeFull(fdTempRoots->get(), s); + } } static std::string censored = "{censored}"; @@ -208,7 +222,7 @@ void LocalStore::findTempRoots(Roots & tempRoots, bool censor) while ((end = contents.find((char) 0, pos)) != std::string::npos) { auto root = std::string_view(contents).substr(pos, end - pos); debug("got temporary root '%s'", root); - tempRoots[parseStorePath(root)].emplace(censor ? censored : fmt("{temp:%s}", name)); + tempRoots[parseStorePath(root)].emplace(censor ? censored : fmt("{nix-process:%s}", name)); pos = end + 1; } } @@ -360,7 +374,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) bool shouldDelete = options.action == GCOptions::gcDeleteDead || options.action == GCOptions::gcDeleteSpecific; - boost::unordered_flat_set> roots, dead, alive; + Roots roots; + boost::unordered_flat_set> dead, alive; /* Return early if nothing to delete */ if (std::visit( @@ -374,7 +389,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) { // The temp roots only store the hash part to make it easier to // ignore suffixes like '.lock', '.chroot' and '.check'. - boost::unordered_flat_set> tempRoots; + boost::unordered_flat_map tempRoots; // Hash part of the store path currently being deleted, if // any. @@ -450,7 +465,17 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) debug("GC roots server accepted new client"); - /* Process the connection in a separate thread. */ + /* Process the connection in a separate thread. Hold + the `connections` lock while starting the thread + and registering it, so that the thread's cleanup + handler (which removes it from `connections`) + can't run before it has been inserted. Otherwise + a client that disconnects immediately would leave + a stale entry, and a later client that gets the + same fd number would cause a duplicate key insert, + destroying a joinable std::thread and thus calling + std::terminate(). */ + auto conn(connections.lock()); auto fdClient_ = fdClient.get(); std::thread clientThread([&, fdClient = std::move(fdClient)]() { Finally cleanup([&]() { @@ -477,7 +502,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) debug("got new GC root '%s'", path); auto hashPart = storePath->hashPart(); auto shared(_shared.lock()); - shared->tempRoots.emplace(hashPart); + // FIXME: could get the PID from the socket. + shared->tempRoots.insert_or_assign(std::string(hashPart), "{nix-process:unknown}"); /* If this path is currently being deleted, then we have to wait until deletion is finished to ensure that @@ -493,14 +519,15 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) } else printError("received garbage instead of a root from client"); writeFull(fdClient.get(), "1", false); - } catch (Error & e) { + } catch (BaseError & e) { debug("reading GC root from client: %s", e.msg()); break; } } }); - connections.lock()->insert({fdClient_, std::move(clientThread)}); + auto [it, inserted] = conn->insert({fdClient_, std::move(clientThread)}); + assert(inserted); } } }); @@ -517,20 +544,16 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) /* Find the roots. Since we've grabbed the GC lock, the set of permanent roots cannot increase now. */ printInfo("finding garbage collector roots..."); - Roots rootMap; if (!options.ignoreLiveness) - findRootsNoTemp(rootMap, true); - - for (auto & i : rootMap) - roots.insert(i.first); + findRootsNoTemp(roots, options.censor); /* Read the temporary roots created before we acquired the global GC root. Any new roots will be sent to our socket. */ - Roots tempRoots; - findTempRoots(tempRoots, true); - for (auto & root : tempRoots) { - _shared.lock()->tempRoots.emplace(root.first.hashPart()); - roots.insert(root.first); + { + Roots tempRoots; + findTempRoots(tempRoots, options.censor); + for (auto & root : tempRoots) + _shared.lock()->tempRoots.insert_or_assign(std::string(root.first.hashPart()), *root.second.begin()); } /* Synchronisation point for testing, see tests/functional/gc-non-blocking.sh. */ @@ -561,6 +584,14 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) results.paths.insert(path); + /* If this path has a leftover `.unpacked` marker (from an + interrupted `addMultipleToStore()`), delete the marker + *before* the path itself. Deleting a directory is not atomic, + so if we were interrupted partway through, a marker left next + to a partially-deleted path would make `addMultipleToStore()` + reuse that corrupt path. */ + deletePath(unpackedMarkerFor(realPath)); + uint64_t bytesFreed; deleteStorePath(realPath, bytesFreed, isKnownPath); @@ -640,35 +671,40 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) if (dead.contains(*path)) continue; + if (auto pathsToDelete = std::get_if(&options.pathsToDelete)) { + if (!pathsToDelete->deleteReferrers && !pathsToDelete->paths.contains(*path)) { + if (options.action != GCOptions::gcDeleteDead) + throw Error( + "Cannot delete path '%s' because it's referenced by path '%s'.", + printStorePath(start), + printStorePath(*path)); + debug( + "cannot delete '%s' because '%s' is not in the specified paths to delete", + printStorePath(start), + printStorePath(*path)); + return; + } + } /* If this is a root, bail out. */ - if (roots.contains(*path)) { + if (auto i = roots.find(*path); i != roots.end()) { + if (options.action == GCOptions::gcDeleteSpecific) + throw Error( + "Cannot delete path '%s' because it's referenced by the GC root '%s'.", + printStorePath(start), + *i->second.begin()); debug("cannot delete '%s' because it's a root", printStorePath(*path)); alive.insert(start); return markAlive(*path); } - if (std::visit( - overloaded{ - [&](const GCOptions::SpecificPaths & pathsToDelete) { - if (!pathsToDelete.deleteReferrers && !pathsToDelete.paths.contains(*path)) { - debug( - "cannot delete '%s' because '%s' is not in the specified paths to delete", - printStorePath(start), - printStorePath(*path)); - return true; - } - return false; - }, - [](const GCOptions::WholeStore & _) { return false; }, - }, - options.pathsToDelete)) - return; - - { + static bool inTest = getEnv("_NIX_IN_TEST").has_value(); + if (!(inTest && options.ignoreLiveness)) { auto hashPart = path->hashPart(); auto shared(_shared.lock()); - if (shared->tempRoots.contains(hashPart)) { - debug("cannot delete '%s' because it's a temporary root", printStorePath(*path)); + if (auto i = shared->tempRoots.find(std::string(hashPart)); i != shared->tempRoots.end()) { + if (options.action == GCOptions::gcDeleteSpecific) + throw Error( + "Cannot delete path '%s' because it's in use by '%s'.", printStorePath(start), i->second); alive.insert(start); return markAlive(*path); } @@ -774,15 +810,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) for (auto & i : pathsToDelete.paths) { maybeDeleteReferrersClosure(i); - - if (options.action == GCOptions::gcDeleteSpecific && !dead.contains(i)) - throw Error( - "Cannot delete path '%1%' since it is still alive. " - "To find out why, use: " - "nix-store --query --roots and nix-store --query --referrers", - printStorePath(i)); - else if (!dead.contains(i)) - debug("cannot delete '%s' because it's still alive", printStorePath(i)); + assert(options.action == GCOptions::gcDeleteDead || dead.count(i)); } }, [&](const GCOptions::WholeStore & _) { diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc index 5e62408847bf..0a770ae40c6b 100644 --- a/src/libstore/globals.cc +++ b/src/libstore/globals.cc @@ -19,6 +19,8 @@ #include #include +#include + #ifndef _WIN32 # include #endif @@ -288,6 +290,20 @@ const ExternalBuilder * LocalSettings::findExternalDerivationBuilderIfSupported( return nullptr; } +std::optional WorkerSettings::getHostName() +{ + if (hostName != "") + return hostName; + +#ifndef _WIN32 + char hostname[_POSIX_HOST_NAME_MAX + 1]; + if (gethostname(hostname, sizeof(hostname)) == 0) + return std::string(hostname); +#endif + + return std::nullopt; +} + ProfileDirsOptions Settings::getProfileDirsOptions() const { return { @@ -298,6 +314,8 @@ ProfileDirsOptions Settings::getProfileDirsOptions() const std::string nixVersion = PACKAGE_VERSION; +const std::string determinateNixVersion = DETERMINATE_NIX_VERSION; + NLOHMANN_JSON_SERIALIZE_ENUM( SandboxMode, { @@ -375,6 +393,27 @@ void from_json(const nlohmann::json & j, ChrootPath & cp) cp.optional = j.at("optional").get(); } +static nlohmann::json pathsInChrootToJSON(const PathsInChroot & paths) +{ + auto j = nlohmann::json::object(); + for (auto & [target, chrootPath] : paths) { + nlohmann::json cp; + to_json(cp, chrootPath); + j[target.string()] = std::move(cp); + } + return j; +} + +template<> +std::map BaseSetting::toJSONObject() const +{ + auto obj = AbstractSetting::toJSONObject(); + obj.emplace("value", pathsInChrootToJSON(value)); + obj.emplace("defaultValue", pathsInChrootToJSON(defaultValue)); + obj.emplace("documentDefault", documentDefault); + return obj; +} + template<> PathsInChroot BaseSetting::parse(const std::string & str) const { @@ -444,6 +483,24 @@ std::string BaseSetting::to_string() const return nlohmann::json(value).dump(); } +template +T JSONSetting::parse(const std::string & str) const +{ + try { + return nlohmann::json::parse(str).template get(); + } catch (std::exception & e) { + throw UsageError("parsing setting '%s': %s", BaseSetting::name, e.what()); + } +} + +template +std::string JSONSetting::to_string() const +{ + return nlohmann::json(BaseSetting::get()).dump(); +} + +template class JSONSetting; + template<> void BaseSetting::appendOrSet(PathsInChroot newValue, bool append) { diff --git a/src/libstore/http-binary-cache-store.cc b/src/libstore/http-binary-cache-store.cc index d881e696b756..eb4613ecc24d 100644 --- a/src/libstore/http-binary-cache-store.cc +++ b/src/libstore/http-binary-cache-store.cc @@ -76,16 +76,16 @@ void HttpBinaryCacheStore::init() auto cacheKey = config->getReference().render(/*withParams=*/false); if (auto cacheInfo = diskCache->upToDateCacheExists(cacheKey)) { - config->wantMassQuery.setDefault(cacheInfo->wantMassQuery); - config->priority.setDefault(cacheInfo->priority); + applyCacheInfoFields(cacheInfo->fields); } else { + std::map fields; try { - BinaryCacheStore::init(); + fields = parseNixCacheInfo(); } catch (UploadToHTTP &) { throw Error("'%s' does not appear to be a binary cache", config->cacheUri.to_string()); } - diskCache->createCache( - cacheKey, config->storeDir, {.wantMassQuery = config->wantMassQuery, .priority = config->priority}); + applyCacheInfoFields(fields); + diskCache->createCache(cacheKey, config->storeDir, {.fields = std::move(fields)}); } } diff --git a/src/libstore/include/nix/store/active-builds.hh b/src/libstore/include/nix/store/active-builds.hh new file mode 100644 index 000000000000..a56b33f5b5ff --- /dev/null +++ b/src/libstore/include/nix/store/active-builds.hh @@ -0,0 +1,116 @@ +#pragma once + +#include "nix/util/util.hh" +#include "nix/util/json-impls.hh" +#include "nix/store/path.hh" + +#include +#include + +namespace nix { + +/** + * A uid and optional corresponding user name. + */ +struct UserInfo +{ + uid_t uid = -1; + std::optional name; + + /** + * Create a UserInfo from a UID, looking up the username if possible. + */ + static UserInfo fromUid(uid_t uid); +}; + +struct ActiveBuild +{ + pid_t nixPid; + + std::optional clientPid; + std::optional clientUid; + + pid_t mainPid; + UserInfo mainUser; + std::optional cgroup; + + time_t startTime; + + StorePath derivation; +}; + +struct ActiveBuildInfo : ActiveBuild +{ + struct ProcessInfo + { + pid_t pid = 0; + pid_t parentPid = 0; + UserInfo user; + std::vector argv; + std::optional utime, stime, cutime, cstime; + }; + + // User/system CPU time for the entire cgroup, if available. + std::optional utime, stime; + + std::vector processes; +}; + +struct TrackActiveBuildsStore +{ + struct BuildHandle + { + TrackActiveBuildsStore & tracker; + uint64_t id; + + BuildHandle(TrackActiveBuildsStore & tracker, uint64_t id) + : tracker(tracker) + , id(id) + { + } + + BuildHandle(BuildHandle && other) noexcept + : tracker(other.tracker) + , id(other.id) + { + other.id = 0; + } + + ~BuildHandle() + { + if (id) { + try { + tracker.buildFinished(*this); + } catch (...) { + ignoreExceptionInDestructor(); + } + } + } + }; + + virtual ~TrackActiveBuildsStore() = default; + + virtual BuildHandle buildStarted(const ActiveBuild & build) = 0; + + virtual void buildFinished(const BuildHandle & handle) = 0; + + virtual void anchor(); +}; + +struct QueryActiveBuildsStore +{ + inline static std::string operationName = "Querying active builds"; + + virtual ~QueryActiveBuildsStore() = default; + + virtual std::vector queryActiveBuilds() = 0; + + virtual void anchor(); +}; + +} // namespace nix + +JSON_IMPL(UserInfo) +JSON_IMPL(ActiveBuild) +JSON_IMPL(ActiveBuildInfo) +JSON_IMPL(ActiveBuildInfo::ProcessInfo) diff --git a/src/libstore/include/nix/store/async-path-writer.hh b/src/libstore/include/nix/store/async-path-writer.hh new file mode 100644 index 000000000000..49d1ecbc69fd --- /dev/null +++ b/src/libstore/include/nix/store/async-path-writer.hh @@ -0,0 +1,27 @@ +#pragma once + +#include "nix/store/store-api.hh" + +namespace nix { + +struct AsyncPathWriter +{ + virtual void anchor(); + + virtual ~AsyncPathWriter() = default; + + virtual StorePath addPath( + std::string contents, + std::string name, + StorePathSet references, + RepairFlag repair, + std::shared_ptr provenance = {}) = 0; + + virtual void waitForPath(const StorePath & path) = 0; + + virtual void waitForAllPaths() = 0; + + static ref make(ref store); +}; + +} // namespace nix diff --git a/src/libstore/include/nix/store/binary-cache-store.hh b/src/libstore/include/nix/store/binary-cache-store.hh index 0cc5d1f3ff3f..1544c4bcec82 100644 --- a/src/libstore/include/nix/store/binary-cache-store.hh +++ b/src/libstore/include/nix/store/binary-cache-store.hh @@ -121,6 +121,16 @@ protected: BinaryCacheStore(Config &); + /** + * Fetch and parse `nix-cache-info`. + */ + std::map parseNixCacheInfo(); + + /** + * Apply the known `nix-cache-info` fields from `fields` to this store. + */ + void applyCacheInfoFields(const std::map & fields); + /** * Compute the path to the given realisation * @@ -130,6 +140,11 @@ protected: public: + bool includeInProvenance() override + { + return true; + } + virtual bool fileExists(const std::string & path) = 0; virtual void upsertFile( @@ -230,7 +245,8 @@ public: ContentAddressMethod hashMethod, HashAlgorithm hashAlgo, const StorePathSet & references, - RepairFlag repair) override; + RepairFlag repair, + std::shared_ptr provenance) override; StorePath addToStore( std::string_view name, diff --git a/src/libstore/include/nix/store/build-result.hh b/src/libstore/include/nix/store/build-result.hh index 808fcc829eda..c3bdb2b09973 100644 --- a/src/libstore/include/nix/store/build-result.hh +++ b/src/libstore/include/nix/store/build-result.hh @@ -13,6 +13,8 @@ namespace nix { +struct Provenance; + /** * Names must be disjoint with `BuildResultFailureStatus`. * @@ -50,6 +52,7 @@ enum struct BuildResultFailureStatus : uint8_t { /// know about this one, so change it back to `OutputRejected` /// before serialization. HashMismatch, + Cancelled, }; /** @@ -76,6 +79,11 @@ public: */ bool isNonDeterministic = false; + /** + * The provenance of the derivation, if any. + */ + std::shared_ptr provenance; + public: /** * Variadic constructor for throwing with format strings. @@ -93,6 +101,7 @@ public: Status status; HintFmt msg; bool isNonDeterministic = false; + std::shared_ptr provenance; }; /** @@ -103,7 +112,7 @@ public: : CloneableError(std::move(args.msg)) , status{args.status} , isNonDeterministic{args.isNonDeterministic} - + , provenance{args.provenance} { } @@ -133,6 +142,12 @@ struct BuildResult */ SingleDrvOutputs builtOutputs; + /** + * The provenance of the derivation, if any. Note that this is the provenance of the current build, not + * necessarily of previously existing outputs. + */ + std::shared_ptr provenance; + bool operator==(const BuildResult::Success &) const noexcept; std::strong_ordering operator<=>(const BuildResult::Success &) const noexcept; }; @@ -195,6 +210,13 @@ struct BuildResult bool operator==(const BuildResult &) const noexcept; std::strong_ordering operator<=>(const BuildResult &) const noexcept; + + bool isCancelled() const + { + auto failure = tryGetFailure(); + // FIXME: remove MiscFailure eventually. + return failure && (failure->status == Failure::Cancelled || failure->status == Failure::MiscFailure); + } }; /** diff --git a/src/libstore/include/nix/store/build/build-log.hh b/src/libstore/include/nix/store/build/build-log.hh index cdc9125734d1..4fd71a65f5d2 100644 --- a/src/libstore/include/nix/store/build/build-log.hh +++ b/src/libstore/include/nix/store/build/build-log.hh @@ -3,6 +3,7 @@ #include "nix/util/logging.hh" #include "nix/util/serialise.hh" +#include "nix/util/ref.hh" #include #include @@ -35,9 +36,9 @@ private: public: /** - * The build activity. Owned by BuildLog. + * The build activity. */ - std::unique_ptr act; + ref act; /** * Map for tracking nested activities from JSON messages. @@ -48,7 +49,7 @@ public: * @param maxTailLines Maximum number of tail lines to keep * @param act Activity for this build */ - BuildLog(size_t maxTailLines, std::unique_ptr act); + BuildLog(size_t maxTailLines, ref act); /** * Process output data from child process. diff --git a/src/libstore/include/nix/store/build/derivation-builder.hh b/src/libstore/include/nix/store/build/derivation-builder.hh index ea2e46445142..2562e29da743 100644 --- a/src/libstore/include/nix/store/build/derivation-builder.hh +++ b/src/libstore/include/nix/store/build/derivation-builder.hh @@ -76,6 +76,11 @@ struct DerivationBuilderParams /** The path of the derivation. */ const StorePath & drvPath; + /** + * The provenance of the derivation, if known + */ + const std::shared_ptr drvProvenance; + BuildResult & buildResult; /** @@ -117,6 +122,11 @@ struct DerivationBuilderParams StringSet systemFeatures; DesugaredEnv desugaredEnv; + + /** + * The activity corresponding to the build. + */ + ref act; }; /** diff --git a/src/libstore/include/nix/store/build/derivation-building-goal.hh b/src/libstore/include/nix/store/build/derivation-building-goal.hh index 6a17f73eeb51..7d8c489719cd 100644 --- a/src/libstore/include/nix/store/build/derivation-building-goal.hh +++ b/src/libstore/include/nix/store/build/derivation-building-goal.hh @@ -115,9 +115,13 @@ private: */ std::pair checkPathValidity(std::map & initialOutputs); - Done doneSuccess(BuildResult::Success::Status status, SingleDrvOutputs builtOutputs); + Done doneSuccess( + BuildResult::Success::Status status, + SingleDrvOutputs builtOutputs, + ActivityId act = 0, + std::shared_ptr provenance = nullptr); - Done doneFailure(BuildError ex); + Done doneFailure(BuildError ex, ActivityId act = 0); BuildError fixupBuilderFailureErrorMessage(BuilderFailureError msg, BuildLog & buildLog); diff --git a/src/libstore/include/nix/store/build/goal.hh b/src/libstore/include/nix/store/build/goal.hh index f0861ddb4794..121d2b058d63 100644 --- a/src/libstore/include/nix/store/build/goal.hh +++ b/src/libstore/include/nix/store/build/goal.hh @@ -169,7 +169,7 @@ public: /** * Build result. */ - BuildResult buildResult; + BuildResult buildResult{BuildError(BuildResult::Failure::Cancelled, "")}; /** * Suspend our goal and wait until we get `work`-ed again. diff --git a/src/libstore/include/nix/store/build/substitution-goal.hh b/src/libstore/include/nix/store/build/substitution-goal.hh index 27585e348d98..73c54819bf7d 100644 --- a/src/libstore/include/nix/store/build/substitution-goal.hh +++ b/src/libstore/include/nix/store/build/substitution-goal.hh @@ -17,6 +17,11 @@ struct PathSubstitutionGoal : public Goal */ StorePath storePath; + /** + * Whether, if there are not substituters, to return ecNoSubstituters or ecFailed. + */ + bool pathRequired; + /** * Whether to try to repair a valid path. */ @@ -39,6 +44,7 @@ public: PathSubstitutionGoal( const StorePath & storePath, Worker & worker, + bool pathRequired, RepairFlag repair = NoRepair, std::optional ca = std::nullopt); ~PathSubstitutionGoal(); @@ -54,7 +60,11 @@ public: Co init(); Co gotInfo(); Co tryToRun( - StorePath subPath, nix::ref sub, std::shared_ptr info, bool & substituterFailed); + StorePath subPath, + nix::ref sub, + std::shared_ptr info, + bool & substituterFailed, + ActivityId parentAct); Co finished(); /* Called by destructor, can't be overridden */ @@ -64,6 +74,8 @@ public: { return JobCategory::Substitution; }; + + Done doneFailure(ExitCode result, BuildResult::Failure failure, ActivityId act = 0); }; } // namespace nix diff --git a/src/libstore/include/nix/store/build/worker.hh b/src/libstore/include/nix/store/build/worker.hh index 861a86b485b9..8bb35221e72a 100644 --- a/src/libstore/include/nix/store/build/worker.hh +++ b/src/libstore/include/nix/store/build/worker.hh @@ -277,7 +277,10 @@ public: * @ref PathSubstitutionGoal "substitution goal" */ std::shared_ptr makePathSubstitutionGoal( - const StorePath & storePath, RepairFlag repair = NoRepair, std::optional ca = std::nullopt); + const StorePath & storePath, + bool pathRequired = false, + RepairFlag repair = NoRepair, + std::optional ca = std::nullopt); std::shared_ptr makeDrvOutputSubstitutionGoal(const DrvOutput & id); /** diff --git a/src/libstore/include/nix/store/builtins.hh b/src/libstore/include/nix/store/builtins.hh index e2caba3f1839..feeb6b49897c 100644 --- a/src/libstore/include/nix/store/builtins.hh +++ b/src/libstore/include/nix/store/builtins.hh @@ -8,8 +8,12 @@ # include "nix/store/aws-creds.hh" #endif +#include + namespace nix { +struct StructuredAttrs; + struct BuiltinBuilderContext { const BasicDerivation & drv; diff --git a/src/libstore/include/nix/store/common-protocol.hh b/src/libstore/include/nix/store/common-protocol.hh index 2ef0f795b4ea..375e1d0338d5 100644 --- a/src/libstore/include/nix/store/common-protocol.hh +++ b/src/libstore/include/nix/store/common-protocol.hh @@ -35,6 +35,7 @@ struct CommonProto struct ReadConn { Source & from; + bool shortStorePaths = false; }; /** @@ -44,6 +45,7 @@ struct CommonProto struct WriteConn { Sink & to; + bool shortStorePaths = false; }; template diff --git a/src/libstore/include/nix/store/daemon.hh b/src/libstore/include/nix/store/daemon.hh index 4d550696e877..45346950033e 100644 --- a/src/libstore/include/nix/store/daemon.hh +++ b/src/libstore/include/nix/store/daemon.hh @@ -4,10 +4,27 @@ #include "nix/util/serialise.hh" #include "nix/store/store-api.hh" +#include + namespace nix::daemon { enum RecursiveFlag : bool { NotRecursive = false, Recursive = true }; -void processConnection(ref store, FdSource && from, FdSink && to, TrustedFlag trusted, RecursiveFlag recursive); +/** + * Serve a client on the given file descriptors. + * + * @param setupTelemetry Called once after the handshake and logger + * setup, with the W3C `traceparent` string received from the client + * (empty if the client sent none or the protocol feature was not + * negotiated). Allows the caller to set up distributed tracing for + * the connection, e.g. by adding a tracing logger. + */ +void processConnection( + ref store, + FdSource && from, + FdSink && to, + TrustedFlag trusted, + RecursiveFlag recursive, + std::function setupTelemetry = {}); } // namespace nix::daemon diff --git a/src/libstore/include/nix/store/derivations.hh b/src/libstore/include/nix/store/derivations.hh index 6d860ee1840c..5371c114e3f1 100644 --- a/src/libstore/include/nix/store/derivations.hh +++ b/src/libstore/include/nix/store/derivations.hh @@ -17,6 +17,7 @@ namespace nix { struct StoreDirConfig; +struct Provenance; /* Abstract syntax of derivations. */ @@ -199,7 +200,7 @@ struct DerivationType /** * Impure derivation type * - * This is similar at build-time to the content addressed, not standboxed, not fixed + * This is similar at build-time to the content addressed, not sandboxed, not fixed * type, but has some restrictions on its usage. */ struct Impure @@ -208,7 +209,16 @@ struct DerivationType auto operator<=>(const Impure &) const = default; }; - typedef std::variant Raw; + /** + * A builtin:substitute derivation. + */ + struct Substituted + { + bool operator==(const Substituted &) const = default; + auto operator<=>(const Substituted &) const = default; + }; + + typedef std::variant Raw; Raw raw; diff --git a/src/libstore/include/nix/store/export-import.hh b/src/libstore/include/nix/store/export-import.hh index 15092202f1f6..4ea696f992f9 100644 --- a/src/libstore/include/nix/store/export-import.hh +++ b/src/libstore/include/nix/store/export-import.hh @@ -4,16 +4,11 @@ namespace nix { -/** - * Magic header of exportPath() output (obsolete). - */ -const uint32_t exportMagic = 0x4558494e; - /** * Export multiple paths in the format expected by `nix-store * --import`. The paths will be sorted topologically. */ -void exportPaths(Store & store, const StorePathSet & paths, Sink & sink); +void exportPaths(Store & store, const StorePathSet & paths, Sink & sink, unsigned int version); /** * Import a sequence of NAR dumps created by `exportPaths()` into the diff --git a/src/libstore/include/nix/store/filetransfer.hh b/src/libstore/include/nix/store/filetransfer.hh index 774a4e1403b3..1456b79fb47c 100644 --- a/src/libstore/include/nix/store/filetransfer.hh +++ b/src/libstore/include/nix/store/filetransfer.hh @@ -439,14 +439,14 @@ public: * exception. */ virtual ItemHandle - enqueueFileTransfer(const FileTransferRequest & request, Callback callback) = 0; + enqueueFileTransfer(const FileTransferRequest & request, Callback callback) noexcept = 0; /** * Unpause a transfer that has been previously paused by a dataCallback. */ virtual void unpauseTransfer(ItemHandle handle) = 0; - std::future enqueueFileTransfer(const FileTransferRequest & request); + std::future enqueueFileTransfer(const FileTransferRequest & request) noexcept; /** * Synchronously download a file. diff --git a/src/libstore/include/nix/store/gc-store.hh b/src/libstore/include/nix/store/gc-store.hh index 0a0335016b74..1e4e620eedca 100644 --- a/src/libstore/include/nix/store/gc-store.hh +++ b/src/libstore/include/nix/store/gc-store.hh @@ -7,9 +7,13 @@ namespace nix { +// FIXME: should turn this into an std::variant to represent the +// several root types. +using GcRootInfo = std::string; + typedef boost::unordered_flat_map< StorePath, - boost::unordered_flat_set>, + boost::unordered_flat_set>, std::hash> Roots; @@ -72,6 +76,12 @@ struct GCOptions * Stop after at least `maxFreed` bytes have been freed. */ uint64_t maxFreed{std::numeric_limits::max()}; + + /** + * Whether to hide potentially sensitive information about GC + * roots (such as PIDs). + */ + bool censor = false; }; struct GCResults @@ -97,7 +107,7 @@ struct GCResults * * The notion of GC roots actually not part of this class. * - * - The base `Store` class has `Store::addTempRoot()` because for a store + * - The base `Store` class has `Store::addTempRoots()` because for a store * that doesn't support garbage collection at all, a temporary GC root is * safely implementable as no-op. * diff --git a/src/libstore/include/nix/store/globals.hh b/src/libstore/include/nix/store/globals.hh index 34eab7f88469..7841bc5949f9 100644 --- a/src/libstore/include/nix/store/globals.hh +++ b/src/libstore/include/nix/store/globals.hh @@ -15,6 +15,16 @@ namespace nix { +template<> +StoreReference BaseSetting::parse(const std::string & str) const; +template<> +std::string BaseSetting::to_string() const; + +template<> +std::set BaseSetting>::parse(const std::string & str) const; +template<> +std::string BaseSetting>::to_string() const; + struct ProfileDirsOptions; struct LogFileSettings : public virtual Config @@ -66,8 +76,8 @@ public: To wipe the lookup cache completely: ```shell-session - $ rm $HOME/.cache/nix/binary-cache-v*.sqlite* - # rm /root/.cache/nix/binary-cache-v*.sqlite* + $ rm $HOME/.cache/nix/binary-cache-*.sqlite* + # rm /root/.cache/nix/binary-cache-*.sqlite* ``` )"}; @@ -220,12 +230,8 @@ public: The following system types are widely used, as Nix is actively supported on these platforms: - `x86_64-linux` - - `x86_64-darwin` - - `i686-linux` - `aarch64-linux` - `aarch64-darwin` - - `armv6l-linux` - - `armv7l-linux` In general, you do not have to modify this setting. While you can force Nix to run a Darwin-specific `builder` executable on a Linux machine, the result would obviously be wrong. @@ -425,6 +431,8 @@ public: * Get the options needed for profile directory functions. */ ProfileDirsOptions getProfileDirsOptions() const; + + const ExternalBuilder * findExternalDerivationBuilderIfSupported(const Derivation & drv); }; // FIXME: don't use a global variable. @@ -448,6 +456,8 @@ void loadConfFile(AbstractConfig & config); */ extern std::string nixVersion; +extern const std::string determinateNixVersion; + /** * @param loadConfig Whether to load configuration from `nix.conf`, `NIX_CONFIG`, etc. May be disabled for unit tests. * @note When using libexpr, and/or libmain, This is not sufficient. See initNix(). diff --git a/src/libstore/include/nix/store/legacy-ssh-store.hh b/src/libstore/include/nix/store/legacy-ssh-store.hh index 839cc8df74b2..40351014bf43 100644 --- a/src/libstore/include/nix/store/legacy-ssh-store.hh +++ b/src/libstore/include/nix/store/legacy-ssh-store.hh @@ -85,6 +85,11 @@ public: ref openConnection(); + bool includeInProvenance() override + { + return true; + } + void queryPathInfoUncached( const StorePath & path, Callback> callback) noexcept override; @@ -124,11 +129,12 @@ public: StorePath addToStoreFromDump( Source & dump, std::string_view name, - FileSerialisationMethod dumpMethod = FileSerialisationMethod::NixArchive, - ContentAddressMethod hashMethod = FileIngestionMethod::NixArchive, - HashAlgorithm hashAlgo = HashAlgorithm::SHA256, - const StorePathSet & references = StorePathSet(), - RepairFlag repair = NoRepair) override + FileSerialisationMethod dumpMethod, + ContentAddressMethod hashMethod, + HashAlgorithm hashAlgo, + const StorePathSet & references, + RepairFlag repair, + std::shared_ptr provenance) override { unsupported("addToStore"); } diff --git a/src/libstore/include/nix/store/local-overlay-store.hh b/src/libstore/include/nix/store/local-overlay-store.hh index 10b04937c3aa..4fd8ee29d79a 100644 --- a/src/libstore/include/nix/store/local-overlay-store.hh +++ b/src/libstore/include/nix/store/local-overlay-store.hh @@ -142,6 +142,11 @@ private: /** * Check lower store if upper DB does not have. + * + * In addition, copy up metadata for lower store objects (and their + * closure). (I.e. Optimistically cache in the upper DB.) The + * returned info is the upper store's own, so that its DB id is + * the upper DB's. */ void queryPathInfoUncached( const StorePath & path, Callback> callback) noexcept override; @@ -154,6 +159,12 @@ private: */ bool isValidPathUncached(const StorePath & path) override; + /** + * Register a lower store object (and its closure) in the upper + * DB. + */ + void registerLowerPath(const ValidPathInfo & info); + /** * Check the lower store and upper DB. */ diff --git a/src/libstore/include/nix/store/local-settings.hh b/src/libstore/include/nix/store/local-settings.hh index 8e4833591cfa..88198802b66d 100644 --- a/src/libstore/include/nix/store/local-settings.hh +++ b/src/libstore/include/nix/store/local-settings.hh @@ -37,6 +37,14 @@ struct BaseSetting::trait template<> void BaseSetting::appendOrSet(PathsInChroot newValue, bool append); +template<> +std::map BaseSetting::toJSONObject() const; + +template<> +std::vector BaseSetting>::parse(const std::string & str) const; +template<> +std::string BaseSetting>::to_string() const; + struct GCSettings : public virtual Config { private: @@ -564,10 +572,24 @@ public: captured by the derivation model itself and are too variable between different versions of the same system to be hard-coded into nix. - The hook is passed the derivation path and, if sandboxes are - enabled, the sandbox directory. It can then modify the sandbox and - send a series of commands to modify various settings to stdout. The - currently recognized commands are: + The hook receives the derivation to be built as JSON in the file + pointed to by the environment variable `NIX_DERIVATION_V4`. See + [@docroot@/protocols/json/derivation/index.md](@docroot@/protocols/json/derivation/index.md) + for the format. For example, to read the `requiredSystemFeatures` + attribute: + + ```sh + jq -r '.env.requiredSystemFeatures' < "$NIX_DERIVATION_V4" + ``` + + > **Deprecated** + > Using the derivation store path passed as `argv[1]` to inspect the + > derivation is deprecated and not recommended. This path may not + > exist when Nix is invoked as a remote builder. + + If sandboxes are enabled, the hook also receives the sandbox + directory as `argv[2]`. It can send a series of commands to modify + various settings to stdout. The currently recognized commands are: - `extra-sandbox-paths`\ Pass a list of files and directories to be included in the diff --git a/src/libstore/include/nix/store/local-store.hh b/src/libstore/include/nix/store/local-store.hh index 0851795ff3d2..872add714857 100644 --- a/src/libstore/include/nix/store/local-store.hh +++ b/src/libstore/include/nix/store/local-store.hh @@ -6,12 +6,14 @@ #include "nix/store/pathlocks.hh" #include "nix/store/store-api.hh" #include "nix/store/indirect-root-store.hh" +#include "nix/store/active-builds.hh" #include "nix/util/sync.hh" +#include #include #include #include -#include +#include namespace nix { @@ -27,8 +29,8 @@ const int nixSchemaVersion = 10; struct OptimiseStats { - unsigned long filesLinked = 0; - uint64_t bytesFreed = 0; + std::atomic filesLinked = 0; + std::atomic bytesFreed = 0; }; struct LocalSettings; @@ -179,7 +181,10 @@ public: MakeError(PathInUse, Error); -class LocalStore : public virtual IndirectRootStore, public virtual GcStore +class LocalStore : public virtual IndirectRootStore, + public virtual GcStore, + public virtual TrackActiveBuildsStore, + public virtual QueryActiveBuildsStore { void anchor() override; @@ -295,6 +300,9 @@ public: void addToStore(const ValidPathInfo & info, Source & source, RepairFlag repair, CheckSigsFlag checkSigs) override; + void + addMultipleToStore(PathsSource && pathsToCopy, Activity & act, RepairFlag repair, CheckSigsFlag checkSigs) override; + StorePath addToStoreFromDump( Source & dump, std::string_view name, @@ -302,12 +310,38 @@ public: ContentAddressMethod hashMethod, HashAlgorithm hashAlgo, const StorePathSet & references, - RepairFlag repair) override; + RepairFlag repair, + std::shared_ptr provenance) override; - void addTempRoot(const StorePath & path) override; + void addTempRoots(const StorePathSet & paths, bool skipIfSlow) override; private: + /** + * Restore the contents of `info` from `source` (a NAR) into the + * store, verifying the NAR hash/size and (if applicable) the + * content address, then canonicalise, optimise and optionally fsync + * the path. + * + * This does *not* acquire a lock on the path or register it as + * valid: the caller must already hold the path lock and is + * responsible for registering the path afterwards. Shared by + * `addToStore()` and `addMultipleToStore()`. + */ + void doAddToStore(const ValidPathInfo & info, Source & source, RepairFlag repair); + + /** + * The path of the marker file (`.unpacked`) used to record + * that the NAR for the store path at `realPath` has been unpacked + * completely and successfully. See `addMultipleToStore()`. + */ + static std::filesystem::path unpackedMarkerFor(const std::filesystem::path & realPath) + { + auto marker = realPath; + marker += ".unpacked"; + return marker; + } + void createTempRootsFile(); /** @@ -473,12 +507,7 @@ private: void openDB(State & state, bool create); - /** - * Perform or check if a database schema upgrade is needed. - * @param dryRun only check if an upgrade is needed. - * @return true if an upgrade is needed or was performed, false otherwise. - */ - bool upgradeDBSchema(State & state, bool dryRun); + void upgradeDBSchema(State & state, bool isNew); void makeStoreWritable(); @@ -505,7 +534,7 @@ private: std::pair createTempDirInStore(); - typedef boost::unordered_flat_set InodeHash; + typedef boost::concurrent_flat_set InodeHash; InodeHash loadInodeHash(); Strings readDirectoryIgnoringInodes(const std::filesystem::path & path, const InodeHash & inodeHash); @@ -526,6 +555,24 @@ private: friend struct DerivationGoal; /* Only used for createTempDirInStore. */ friend class DerivationBuilderImpl; + +private: + + std::filesystem::path activeBuildsDir; + + struct ActiveBuildFile + { + AutoCloseFD fd; + AutoDelete del; + }; + + Sync> activeBuilds; + + std::vector queryActiveBuilds() override; + + BuildHandle buildStarted(const ActiveBuild & build) override; + + void buildFinished(const BuildHandle & handle) override; }; } // namespace nix diff --git a/src/libstore/include/nix/store/meson.build b/src/libstore/include/nix/store/meson.build index 3ccf0adc2fa1..132a9067558e 100644 --- a/src/libstore/include/nix/store/meson.build +++ b/src/libstore/include/nix/store/meson.build @@ -10,6 +10,8 @@ config_pub_h = configure_file( ) headers = [ config_pub_h ] + files( + 'active-builds.hh', + 'async-path-writer.hh', 'aws-creds.hh', 'binary-cache-store.hh', 'build-result.hh', @@ -73,6 +75,7 @@ headers = [ config_pub_h ] + files( 'pathlocks.hh', 'posix-fs-canonicalise.hh', 'profiles.hh', + 'provenance.hh', 'realisation.hh', 'references.hh', 'remote-fs-accessor.hh', diff --git a/src/libstore/include/nix/store/nar-info-disk-cache.hh b/src/libstore/include/nix/store/nar-info-disk-cache.hh index f7bfcd035a23..a191353fac3a 100644 --- a/src/libstore/include/nix/store/nar-info-disk-cache.hh +++ b/src/libstore/include/nix/store/nar-info-disk-cache.hh @@ -5,6 +5,9 @@ #include "nix/store/nar-info.hh" #include "nix/store/realisation.hh" +#include +#include + namespace nix { struct SQLiteSettings; @@ -33,8 +36,13 @@ public: struct CacheInfo { int id = 0; - bool wantMassQuery = false; - int priority = 0; + + /** + * The `nix-cache-info` fields other than `StoreDir`, stored + * verbatim (e.g. `WantMassQuery`). Keeping these generic + * means fields we don't (yet) understand are still recorded. + */ + std::map fields; }; /** diff --git a/src/libstore/include/nix/store/path-info.hh b/src/libstore/include/nix/store/path-info.hh index 0dd19b81ecf1..c399fa23abc4 100644 --- a/src/libstore/include/nix/store/path-info.hh +++ b/src/libstore/include/nix/store/path-info.hh @@ -13,6 +13,7 @@ namespace nix { class Store; struct StoreDirConfig; +struct Provenance; /** * JSON format version for path info output. @@ -117,6 +118,12 @@ struct UnkeyedValidPathInfo */ std::optional ca; + /** + * The provenance of this store path, i.e. a link back to the Nix + * expression used to create it. + */ + std::shared_ptr provenance; + UnkeyedValidPathInfo(const UnkeyedValidPathInfo & other) = default; UnkeyedValidPathInfo(const StoreDirConfig & store, Hash narHash); diff --git a/src/libstore/include/nix/store/provenance.hh b/src/libstore/include/nix/store/provenance.hh new file mode 100644 index 000000000000..5f9773e724ea --- /dev/null +++ b/src/libstore/include/nix/store/provenance.hh @@ -0,0 +1,77 @@ +#pragma once + +#include "nix/util/provenance.hh" +#include "nix/util/types.hh" +#include "nix/store/path.hh" +#include "nix/store/outputs-spec.hh" + +namespace nix { + +struct BuildProvenance : Provenance +{ + /** + * The derivation that built this path. + */ + StorePath drvPath; + + /** + * The output of the derivation that corresponds to this path. + */ + OutputName output; + + /** + * The hostname of the machine on which the derivation was built, if known. + */ + std::optional buildHost; + + /** + * User-defined tags from the build host. + */ + StringMap tags; + + /** + * The system type of the derivation. + */ + std::string system; + + /** + * The provenance of the derivation, if known. + */ + std::shared_ptr next; + + // FIXME: do we need anything extra for CA derivations? + + BuildProvenance( + const StorePath & drvPath, + const OutputName & output, + std::optional buildHost, + StringMap tags, + std::string system, + std::shared_ptr next); + + nlohmann::json to_json() const override; +}; + +struct CopiedProvenance : Provenance +{ + /** + * Store URL (typically a binary cache) from which this store + * path was copied. + */ + std::string from; + + /** + * Provenance of the store path in the upstream store, if any. + */ + std::shared_ptr next; + + CopiedProvenance(std::string_view from, std::shared_ptr next) + : from(from) + , next(std::move(next)) + { + } + + nlohmann::json to_json() const override; +}; + +} // namespace nix diff --git a/src/libstore/include/nix/store/remote-store.hh b/src/libstore/include/nix/store/remote-store.hh index 144b4b8e4355..0769afd6b34b 100644 --- a/src/libstore/include/nix/store/remote-store.hh +++ b/src/libstore/include/nix/store/remote-store.hh @@ -10,6 +10,7 @@ #include "nix/util/file-descriptor.hh" #include "nix/store/gc-store.hh" #include "nix/store/log-store.hh" +#include "nix/store/active-builds.hh" namespace nix { @@ -33,7 +34,7 @@ public: } Setting maxConnections{ - this, 1, "max-connections", "Maximum number of concurrent connections to the Nix daemon."}; + this, 64, "max-connections", "Maximum number of concurrent connections to the Nix daemon."}; Setting maxConnectionAge{ this, @@ -46,7 +47,10 @@ public: * \todo RemoteStore is a misnomer - should be something like * DaemonStore. */ -struct RemoteStore : public virtual Store, public virtual GcStore, public virtual LogStore +struct RemoteStore : public virtual Store, + public virtual GcStore, + public virtual LogStore, + public virtual QueryActiveBuildsStore { private: void anchor() override; @@ -69,6 +73,10 @@ public: void queryPathInfoUncached( const StorePath & path, Callback> callback) noexcept override; + asio::awaitable queryPathInfos( + const std::set & paths, + fun>>)> callback) override; + void queryReferrers(const StorePath & path, StorePathSet & referrers) override; StorePathSet queryValidDerivers(const StorePath & path) override; @@ -92,7 +100,8 @@ public: ContentAddressMethod caMethod, HashAlgorithm hashAlgo, const StorePathSet & references, - RepairFlag repair); + RepairFlag repair, + std::shared_ptr provenance); /** * Add a content-addressable store path. `dump` will be drained. @@ -100,11 +109,12 @@ public: StorePath addToStoreFromDump( Source & dump, std::string_view name, - FileSerialisationMethod dumpMethod = FileSerialisationMethod::NixArchive, - ContentAddressMethod hashMethod = FileIngestionMethod::NixArchive, - HashAlgorithm hashAlgo = HashAlgorithm::SHA256, - const StorePathSet & references = StorePathSet(), - RepairFlag repair = NoRepair) override; + FileSerialisationMethod dumpMethod, + ContentAddressMethod hashMethod, + HashAlgorithm hashAlgo, + const StorePathSet & references, + RepairFlag repair, + std::shared_ptr provenance) override; void addToStore(const ValidPathInfo & info, Source & nar, RepairFlag repair, CheckSigsFlag checkSigs) override; @@ -126,7 +136,7 @@ public: void ensurePath(const StorePath & path) override; - void addTempRoot(const StorePath & path) override; + void addTempRoots(const StorePathSet & paths, bool skipIfSlow) override; Roots findRoots(bool censor) override; @@ -155,6 +165,8 @@ public: void addBuildLog(const StorePath & drvPath, std::string_view log) override; + std::vector queryActiveBuilds() override; + std::optional getVersion() override; void connect() override; diff --git a/src/libstore/include/nix/store/sqlite.hh b/src/libstore/include/nix/store/sqlite.hh index d11515b29618..e416c44bd04b 100644 --- a/src/libstore/include/nix/store/sqlite.hh +++ b/src/libstore/include/nix/store/sqlite.hh @@ -74,9 +74,26 @@ struct SQLite */ void isCache(); + /** + * Execute a statement. Retry if the database is busy. Do not call this inside a transaction; use execNoRetry() + * instead. + */ void exec(const std::string & stmt); + /** + * Execute a statement. + */ + void execNoRetry(const std::string & stmt); + uint64_t getLastInsertedRowId(); + + /** + * Set the value returned by `getLastInsertedRowId()`. Since only + * successful inserts update the last-inserted rowid, this can be + * used to detect whether an upsert statement performed an insert + * or an update. + */ + void setLastInsertedRowId(uint64_t id); }; /** diff --git a/src/libstore/include/nix/store/ssh.hh b/src/libstore/include/nix/store/ssh.hh index e090dba2317f..b9d771230fdd 100644 --- a/src/libstore/include/nix/store/ssh.hh +++ b/src/libstore/include/nix/store/ssh.hh @@ -39,11 +39,11 @@ private: Sync state_; - void addCommonSSHOpts(OsStrings & args); - bool isMasterRunning(); + void addCommonSSHOpts(OsStrings & args, std::optional socketPath); + bool isMasterRunning(std::filesystem::path socketPath); #ifndef _WIN32 // TODO re-enable on Windows, once we can start processes. - std::filesystem::path startMaster(); + std::optional startMaster(); #endif public: diff --git a/src/libstore/include/nix/store/store-api.hh b/src/libstore/include/nix/store/store-api.hh index 68a18f48db67..78a819c21540 100644 --- a/src/libstore/include/nix/store/store-api.hh +++ b/src/libstore/include/nix/store/store-api.hh @@ -14,6 +14,8 @@ #include "nix/store/store-dir-config.hh" #include "nix/store/store-reference.hh" #include "nix/util/source-path.hh" +#include "nix/util/async.hh" +#include "nix/util/fun.hh" #include #include @@ -43,6 +45,8 @@ struct SourceAccessor; struct NarInfoDiskCache; struct NarInfoDiskCacheSettings; class Store; +struct AsyncPathWriter; +struct Provenance; typedef std::map OutputPathMap; @@ -467,7 +471,9 @@ public: StorePath followLinksToStorePath(std::string_view path) const; /** - * Check whether a path is valid. + * Check whether a path is valid. NOTE: this function does not + * generally cache whether a path is valid. You may want to use + * `maybeQueryPathInfo()`, which does cache. */ bool isValidPath(const StorePath & path); @@ -507,15 +513,35 @@ public: /** * Query information about a valid path. It is permitted to omit - * the name part of the store path. + * the name part of the store path. Throws an exception if the + * path is not valid. */ ref queryPathInfo(const StorePath & path); + /** + * Like `queryPathInfo()`, but returns `nullptr` if the path is + * not valid. + */ + std::shared_ptr maybeQueryPathInfo(const StorePath & path); + /** * Asynchronous version of queryPathInfo(). */ void queryPathInfo(const StorePath & path, Callback> callback) noexcept; + /** + * Asynchronously query information about multiple store paths. As + * results arrive (possibly in batches from a remote server), + * `callback` is invoked one or more times with a vector of + * `(path, info)` pairs. A null `info` denotes that the path is + * not valid. Every requested path is reported exactly once across + * all invocations of `callback`. Unlike `queryPathInfo()`, an + * invalid path is not an error. + */ + virtual asio::awaitable queryPathInfos( + const std::set & paths, + fun>>)> callback); + /** * Version of queryPathInfo() that only queries the local narinfo cache and not * the actual store. @@ -720,7 +746,8 @@ public: ContentAddressMethod hashMethod = ContentAddressMethod::Raw::NixArchive, HashAlgorithm hashAlgo = HashAlgorithm::SHA256, const StorePathSet & references = StorePathSet(), - RepairFlag repair = NoRepair) = 0; + RepairFlag repair = NoRepair, + std::shared_ptr provenance = nullptr) = 0; /** * Add a mapping indicating that `deriver!outputName` maps to the output path @@ -859,9 +886,18 @@ public: * ``` * */ - virtual void addTempRoot(const StorePath & path) + void addTempRoot(const StorePath & path) + { + addTempRoots({path}); + } + + /** + * Add multiple store paths as temporary roots of the garbage collector. + * The roots disappears as soon as we exit. + */ + virtual void addTempRoots(const StorePathSet & paths, bool skipIfSlow = false) { - debug("not creating temporary root, store doesn't support GC"); + debug("not creating temporary roots, store doesn't support GC"); } /** @@ -954,7 +990,17 @@ public: /** * Write a derivation to the Nix store, and return its path. */ - virtual StorePath writeDerivation(const Derivation & drv, RepairFlag repair = NoRepair); + virtual StorePath writeDerivation( + const Derivation & drv, RepairFlag repair = NoRepair, std::shared_ptr provenance = nullptr); + + /** + * Asynchronously write a derivation to the Nix store, and return its path. + */ + StorePath writeDerivation( + AsyncPathWriter & asyncPathWriter, + const Derivation & drv, + RepairFlag repair = NoRepair, + std::shared_ptr provenance = nullptr); /** * Read a derivation (which must already be valid). @@ -1080,6 +1126,15 @@ public: return {}; } + /** + * Whether, when copying *from* this store, a "copied" provenance + * record should be added. + */ + virtual bool includeInProvenance() + { + return false; + } + protected: Stats stats; @@ -1098,9 +1153,10 @@ protected: }; /** - * Copy a path from one store to another. + * Copy a path from one store to another. Return the path info of the newly added store path, or nullptr if the path was + * already valid. */ -void copyStorePath( +std::shared_ptr copyStorePath( Store & srcStore, Store & dstStore, const StorePath & storePath, diff --git a/src/libstore/include/nix/store/worker-protocol-connection.hh b/src/libstore/include/nix/store/worker-protocol-connection.hh index 49de3fe67396..d49110ba06c7 100644 --- a/src/libstore/include/nix/store/worker-protocol-connection.hh +++ b/src/libstore/include/nix/store/worker-protocol-connection.hh @@ -94,6 +94,14 @@ struct WorkerProto::BasicClientConnection : WorkerProto::BasicConnection */ ClientHandshakeInfo postHandshake(const StoreDirConfig & store); + /** + * Start sending an operation: write the opcode, followed by the + * trace context of the current activity if the "open-telemetry" + * protocol feature has been negotiated, so that the daemon can + * parent its work under it. + */ + void startOp(WorkerProto::Op op); + void addTempRoot(const StoreDirConfig & remoteStore, bool * daemonException, const StorePath & path); StorePathSet queryValidPaths( diff --git a/src/libstore/include/nix/store/worker-protocol-impl.hh b/src/libstore/include/nix/store/worker-protocol-impl.hh index 605663d2eeb4..8d00f7930a23 100644 --- a/src/libstore/include/nix/store/worker-protocol-impl.hh +++ b/src/libstore/include/nix/store/worker-protocol-impl.hh @@ -47,12 +47,14 @@ struct WorkerProto::Serialise { static T read(const StoreDirConfig & store, WorkerProto::ReadConn conn) { - return CommonProto::Serialise::read(store, CommonProto::ReadConn{.from = conn.from}); + return CommonProto::Serialise::read( + store, CommonProto::ReadConn{.from = conn.from, .shortStorePaths = conn.shortStorePaths}); } static void write(const StoreDirConfig & store, WorkerProto::WriteConn conn, const T & t) { - CommonProto::Serialise::write(store, CommonProto::WriteConn{.to = conn.to}, t); + CommonProto::Serialise::write( + store, CommonProto::WriteConn{.to = conn.to, .shortStorePaths = conn.shortStorePaths}, t); } }; diff --git a/src/libstore/include/nix/store/worker-protocol.hh b/src/libstore/include/nix/store/worker-protocol.hh index d09d30421dcf..8c87b3f10e01 100644 --- a/src/libstore/include/nix/store/worker-protocol.hh +++ b/src/libstore/include/nix/store/worker-protocol.hh @@ -12,6 +12,7 @@ namespace nix { #define WORKER_MAGIC_1 0x6e697863 #define WORKER_MAGIC_2 0x6478696f +#define WORKER_MAGIC_ACCESS_DENIED 0xab9a9ff0 // = 🚫 /* Note: you generally shouldn't change the protocol version. Define a new `WorkerProto::Feature` instead. */ @@ -119,6 +120,12 @@ struct WorkerProto static const Version minimum; + static constexpr std::string_view featureQueryActiveBuilds = "queryActiveBuilds"; + static constexpr std::string_view featureProvenance = "provenance"; + static constexpr std::string_view featureVersionedAddToStoreMultiple = "versionedAddToStoreMultiple"; + static constexpr std::string_view featureAddTempRoots = "addTempRoots"; + static constexpr std::string_view featureQueryPathInfos = "queryPathInfos"; + /** * Feature for transmitting `UnkeyedRealisation` and `DrvOutput` * using drvPath (store path) instead of the old hash-based JSON format. @@ -135,6 +142,13 @@ struct WorkerProto */ static constexpr std::string_view featureDisableSetOptions = "disable-set-options"; + /** + * Feature for propagating OpenTelemetry trace context from the + * client to the daemon: after the handshake, the client sends a + * W3C `traceparent` string (empty if the client is not tracing). + */ + static constexpr std::string_view featureOpenTelemetry = "open-telemetry"; + /** * A unidirectional read connection, to be used by the read half of the * canonical serializers below. @@ -143,6 +157,7 @@ struct WorkerProto { Source & from; const Version & version; + bool shortStorePaths = false; }; /** @@ -153,6 +168,7 @@ struct WorkerProto { Sink & to; const Version & version; + bool shortStorePaths = false; }; /** @@ -256,6 +272,9 @@ enum struct WorkerProto::Op : uint64_t { AddBuildLog = 45, BuildPathsWithResults = 46, AddPermRoot = 47, + QueryActiveBuilds = 48, + AddTempRoots = 49, + QueryPathInfos = 50, }; struct WorkerProto::ClientHandshakeInfo diff --git a/src/libstore/include/nix/store/worker-settings.hh b/src/libstore/include/nix/store/worker-settings.hh index d78ca420f776..bc8a21b5a6f7 100644 --- a/src/libstore/include/nix/store/worker-settings.hh +++ b/src/libstore/include/nix/store/worker-settings.hh @@ -7,6 +7,11 @@ namespace nix { +template<> +std::vector BaseSetting>::parse(const std::string & str) const; +template<> +std::string BaseSetting>::to_string() const; + struct MaxBuildJobsSetting : public BaseSetting { MaxBuildJobsSetting( @@ -364,6 +369,25 @@ public: /nix/store/scz72lskj03ihkcn42ias5mlp4i4gr1k-bash-4.4-p23-man /nix/store/a724znygmd1cac856j3gfsyvih3lw07j-bash-4.4-p23`. )"}; + + Setting hostName{ + this, + "", + "host-name", + R"( + The name of this host for recording build provenance. If unset, the Unix host name is used. + )"}; + + std::optional getHostName(); + + JSONSetting buildProvenanceTags{ + this, + {}, + "build-provenance-tags", + R"( + Arbitrary name/value pairs that are recorded in the build provenance of store paths built by this machine. + This can be used to tag builds with metadata such as the CI job URL, build cluster name, etc. + )"}; }; } // namespace nix diff --git a/src/libstore/keys.cc b/src/libstore/keys.cc index 8b02e7a66819..604b6e36ff40 100644 --- a/src/libstore/keys.cc +++ b/src/libstore/keys.cc @@ -11,14 +11,16 @@ PublicKeys getDefaultPublicKeys() // FIXME: filter duplicates for (const auto & s : settings.trustedPublicKeys.get()) { - PublicKey key(s); - publicKeys.emplace(key.name, key); + auto key = PublicKey::parse(s); + auto name = key->name; + publicKeys.emplace(name, std::move(key)); } + // FIXME: keep secret keys in memory (see Store::signRealisation()). for (const auto & secretKeyFile : settings.secretKeyFiles.get()) { try { - SecretKey secretKey(readFile(secretKeyFile)); - publicKeys.emplace(secretKey.name, secretKey.toPublicKey()); + auto secretKey = SecretKey::parse(readFile(secretKeyFile)); + publicKeys.emplace(secretKey->name, secretKey->toPublicKey()); } catch (SystemError & e) { /* Ignore unreadable key files. That's normal in a multi-user installation. */ diff --git a/src/libstore/linux/build/chroot-linux-derivation-builder.hh b/src/libstore/linux/build/chroot-linux-derivation-builder.hh index bee6e4659758..b5c939bc991a 100644 --- a/src/libstore/linux/build/chroot-linux-derivation-builder.hh +++ b/src/libstore/linux/build/chroot-linux-derivation-builder.hh @@ -59,6 +59,8 @@ struct ChrootLinuxDerivationBuilder : ChrootDerivationBuilder, LinuxDerivationBu void killSandbox(bool getStats) override; void addDependencyImpl(const StorePath & path) override; + + ActiveBuild getActiveBuild() override; }; } // namespace nix diff --git a/src/libstore/linux/build/linux-derivation-builder.cc b/src/libstore/linux/build/linux-derivation-builder.cc index 310f796de854..8970197334f4 100644 --- a/src/libstore/linux/build/linux-derivation-builder.cc +++ b/src/libstore/linux/build/linux-derivation-builder.cc @@ -591,7 +591,12 @@ void ChrootLinuxDerivationBuilder::startChild() sendPid.writeSide.close(); if (auto status = helper.wait(); !statusOk(status)) { + // Ensure the slave side of the pseudoterminal has been opened at least once so that reading from the master + // doesn't hang. + openSlaveNoDup(); + processSandboxSetupMessages(); + // Only reached if the child process didn't send an exception. throw Error("unable to start build process: %s", statusToString(status)); } @@ -986,4 +991,11 @@ void ChrootLinuxDerivationBuilder::addDependencyImpl(const StorePath & path) throw Error("could not add path '%s' to sandbox: %s", store.printStorePath(path), statusToString(status)); } +ActiveBuild ChrootLinuxDerivationBuilder::getActiveBuild() +{ + auto build = DerivationBuilderImpl::getActiveBuild(); + build.cgroup = cgroup; + return build; +} + } // namespace nix diff --git a/src/libstore/local-fs-store.cc b/src/libstore/local-fs-store.cc index a6fd24bf34e6..d2493f717bea 100644 --- a/src/libstore/local-fs-store.cc +++ b/src/libstore/local-fs-store.cc @@ -51,7 +51,7 @@ struct LocalStoreAccessor : SourceAccessor void requireStoreObject(const StorePath & storePath) { - if (requireValidPath && !store->isValidPath(storePath)) + if (requireValidPath && !store->maybeQueryPathInfo(storePath)) throw InvalidPath("path '%1%' is not a valid store path", store->printStorePath(storePath)); } diff --git a/src/libstore/local-overlay-store.cc b/src/libstore/local-overlay-store.cc index c0fee83d7d18..61c4e2fbfc67 100644 --- a/src/libstore/local-overlay-store.cc +++ b/src/libstore/local-overlay-store.cc @@ -104,16 +104,32 @@ void LocalOverlayStore::queryPathInfoUncached( return callbackPtr->rethrow(); } // If we don't have it, check lower store - lowerStore->queryPathInfo(path, {[path, callbackPtr](std::future> fut) { + lowerStore->queryPathInfo(path, {[this, path, callbackPtr](std::future> fut) { try { - (*callbackPtr)(fut.get().get_ptr()); + /* Copy the lower store object up into the + upper DB, so that subsequent queries (such + as derivation output lookups) find it there, + and so that we return an info whose DB id + belongs to the upper DB rather than the + lower one. */ + registerLowerPath(*fut.get()); } catch (...) { return callbackPtr->rethrow(); } + LocalStore::queryPathInfoUncached(path, std::move(*callbackPtr)); }}); }}); } +void LocalOverlayStore::registerLowerPath(const ValidPathInfo & info) +{ + // recur on references, syncing entire closure. + for (auto & r : info.references) + if (r != info.path) + isValidPath(r); + LocalStore::registerValidPath(info); +} + void LocalOverlayStore::queryRealisationUncached( const DrvOutput & drvOutput, Callback> callback) noexcept { @@ -146,15 +162,9 @@ bool LocalOverlayStore::isValidPathUncached(const StorePath & path) if (res) return res; res = lowerStore->isValidPath(path); - if (res) { + if (res) // Get path info from lower store so upper DB genuinely has it. - auto p = lowerStore->queryPathInfo(path); - // recur on references, syncing entire closure. - for (auto & r : p->references) - if (r != path) - isValidPath(r); - LocalStore::registerValidPath(*p); - } + registerLowerPath(*lowerStore->queryPathInfo(path)); return res; } diff --git a/src/libstore/local-store-active-builds.cc b/src/libstore/local-store-active-builds.cc new file mode 100644 index 000000000000..004eb80361f1 --- /dev/null +++ b/src/libstore/local-store-active-builds.cc @@ -0,0 +1,285 @@ +#include "nix/store/local-store.hh" +#include "nix/util/json-utils.hh" +#ifdef __linux__ +# include "nix/util/cgroup.hh" +# include +# include +# include +#endif + +#ifdef __APPLE__ +# include +# include +# include +#endif + +#include +#include + +namespace nix { + +#ifdef __linux__ +static ActiveBuildInfo::ProcessInfo getProcessInfo(pid_t pid) +{ + ActiveBuildInfo::ProcessInfo info; + info.pid = pid; + info.argv = + tokenizeString>(readFile(fmt("/proc/%d/cmdline", pid)), std::string("\000", 1)); + + auto statPath = fmt("/proc/%d/stat", pid); + + AutoCloseFD statFd = open(statPath.c_str(), O_RDONLY | O_CLOEXEC); + if (!statFd) + throw SysError("opening '%s'", statPath); + + // Get the UID from the ownership of the stat file. + struct stat st; + if (fstat(statFd.get(), &st) == -1) + throw SysError("getting ownership of '%s'", statPath); + info.user = UserInfo::fromUid(st.st_uid); + + // Read /proc/[pid]/stat for parent PID and CPU times. + // Format: pid (comm) state ppid ... + // Note that the comm field can contain spaces, so use a regex to parse it. + auto statContent = trim(readFile(statFd.get())); + static std::regex statRegex(R"((\d+) \(([^)]*)\) (.*))"); + std::smatch match; + if (!std::regex_match(statContent, match, statRegex)) + throw Error("failed to parse /proc/%d/stat", pid); + + // Parse the remaining fields after (comm). + auto remainingFields = tokenizeString>(match[3].str()); + + if (remainingFields.size() > 1) + info.parentPid = string2Int(remainingFields[1]).value_or(0); + + static long clkTck = sysconf(_SC_CLK_TCK); + if (remainingFields.size() > 14 && clkTck > 0) { + if (auto utime = string2Int(remainingFields[11])) + info.utime = std::chrono::microseconds((*utime * 1'000'000) / clkTck); + if (auto stime = string2Int(remainingFields[12])) + info.stime = std::chrono::microseconds((*stime * 1'000'000) / clkTck); + if (auto cutime = string2Int(remainingFields[13])) + info.cutime = std::chrono::microseconds((*cutime * 1'000'000) / clkTck); + if (auto cstime = string2Int(remainingFields[14])) + info.cstime = std::chrono::microseconds((*cstime * 1'000'000) / clkTck); + } + + return info; +} + +/** + * Recursively get all descendant PIDs of a given PID using /proc/[pid]/task/[pid]/children. + */ +static std::set getDescendantPids(pid_t pid) +{ + std::set descendants; + + [&](this auto self, pid_t pid) -> void { + try { + descendants.insert(pid); + for (const auto & childPidStr : + tokenizeString>(readFile(fmt("/proc/%d/task/%d/children", pid, pid)))) + if (auto childPid = string2Int(childPidStr)) + self(*childPid); + } catch (...) { + // Process may have exited. + ignoreExceptionExceptInterrupt(); + } + }(pid); + + return descendants; +} +#endif + +#ifdef __APPLE__ +static ActiveBuildInfo::ProcessInfo getProcessInfo(pid_t pid) +{ + ActiveBuildInfo::ProcessInfo info; + info.pid = pid; + + // Get basic process info including ppid and uid. + struct proc_bsdinfo procInfo; + if (proc_pidinfo(pid, PROC_PIDTBSDINFO, 0, &procInfo, sizeof(procInfo)) != sizeof(procInfo)) + throw SysError("getting process info for pid %d", pid); + + info.parentPid = procInfo.pbi_ppid; + info.user = UserInfo::fromUid(procInfo.pbi_uid); + + // Get CPU times. + struct proc_taskinfo taskInfo; + if (proc_pidinfo(pid, PROC_PIDTASKINFO, 0, &taskInfo, sizeof(taskInfo)) == sizeof(taskInfo)) { + + mach_timebase_info_data_t timebase; + mach_timebase_info(&timebase); + auto nanosecondsPerTick = (double) timebase.numer / (double) timebase.denom; + + // Convert nanoseconds to microseconds. + info.utime = + std::chrono::microseconds((uint64_t) ((double) taskInfo.pti_total_user * nanosecondsPerTick / 1000)); + info.stime = + std::chrono::microseconds((uint64_t) ((double) taskInfo.pti_total_system * nanosecondsPerTick / 1000)); + } + + // Get argv using sysctl. + int mib[3] = {CTL_KERN, KERN_PROCARGS2, pid}; + size_t size = 0; + + // First call to get size. + if (sysctl(mib, 3, nullptr, &size, nullptr, 0) == 0 && size > 0) { + std::vector buffer(size); + if (sysctl(mib, 3, buffer.data(), &size, nullptr, 0) == 0) { + // Format: argc (int), followed by executable path, followed by null-terminated args + if (size >= sizeof(int)) { + int argc; + memcpy(&argc, buffer.data(), sizeof(argc)); + + // Skip past argc and executable path (null-terminated). + size_t pos = sizeof(int); + while (pos < size && buffer[pos] != '\0') + pos++; + pos++; // Skip the null terminator + + // Parse the arguments. + while (pos < size && info.argv.size() < (size_t) argc) { + size_t argStart = pos; + while (pos < size && buffer[pos] != '\0') + pos++; + + if (pos > argStart) + info.argv.emplace_back(buffer.data() + argStart, pos - argStart); + + pos++; // Skip the null terminator + } + } + } + } + + return info; +} + +/** + * Recursively get all descendant PIDs using sysctl with KERN_PROC. + */ +static std::set getDescendantPids(pid_t startPid) +{ + // Get all processes. + int mib[4] = {CTL_KERN, KERN_PROC, KERN_PROC_ALL, 0}; + size_t size = 0; + + if (sysctl(mib, 4, nullptr, &size, nullptr, 0) == -1) + return {startPid}; + + std::vector procs(size / sizeof(struct kinfo_proc)); + if (sysctl(mib, 4, procs.data(), &size, nullptr, 0) == -1) + return {startPid}; + + // Get the children of all processes. + std::map> children; + size_t count = size / sizeof(struct kinfo_proc); + for (size_t i = 0; i < count; i++) { + pid_t childPid = procs[i].kp_proc.p_pid; + pid_t parentPid = procs[i].kp_eproc.e_ppid; + children[parentPid].insert(childPid); + } + + // Get all children of `pid`. + std::set descendants; + std::queue todo; + todo.push(startPid); + while (auto pid = pop(todo)) { + if (!descendants.insert(*pid).second) + continue; + for (auto & child : children[*pid]) + todo.push(child); + } + + return descendants; +} +#endif + +std::vector LocalStore::queryActiveBuilds() +{ + std::vector result; + + for (auto & entry : DirectoryIterator{activeBuildsDir}) { + auto path = entry.path(); + + try { + // Open the file. If we can lock it, the build is not active. + auto fd = openLockFile(path, false); + if (!fd || lockFile(fd.get(), ltRead, false)) { + AutoDelete(path, false); + continue; + } + + ActiveBuildInfo info(nlohmann::json::parse(readFile(fd.get())).get()); + +#if defined(__linux__) || defined(__APPLE__) + /* Read process information. */ + try { +# ifdef __linux__ + if (info.cgroup) { + for (auto pid : linux::getPidsInCgroup(*info.cgroup)) + info.processes.push_back(getProcessInfo(pid)); + + /* Read CPU statistics from the cgroup. */ + auto stats = linux::getCgroupStats(*info.cgroup); + info.utime = stats.cpuUser; + info.stime = stats.cpuSystem; + } else +# endif + { + for (auto pid : getDescendantPids(info.mainPid)) + info.processes.push_back(getProcessInfo(pid)); + } + } catch (...) { + ignoreExceptionExceptInterrupt(); + } +#endif + + result.push_back(std::move(info)); + } catch (...) { + ignoreExceptionExceptInterrupt(); + } + } + + return result; +} + +LocalStore::BuildHandle LocalStore::buildStarted(const ActiveBuild & build) +{ + // Write info about the active build to the active-builds directory where it can be read by `queryBuilds()`. + static std::atomic nextId{1}; + + auto id = nextId++; + + auto infoFileName = fmt("%d-%d", getpid(), id); + auto infoFilePath = activeBuildsDir / infoFileName; + + auto infoFd = openLockFile(infoFilePath, true); + + // Lock the file to denote that the build is active. + lockFile(infoFd.get(), ltWrite, true); + + // Don't fsync: the lock above is what marks the build as active, and queryActiveBuilds() deletes any file it + // can lock, so durability buys nothing here. An fsync (F_FULLFSYNC on macOS) would also delay the parent + // between starting the builder and reading its setup messages. + writeFile(infoFilePath, nlohmann::json(build).dump(), 0600, FsSync::No); + + activeBuilds.lock()->emplace( + id, + ActiveBuildFile{ + .fd = std::move(infoFd), + .del = AutoDelete(infoFilePath, false), + }); + + return BuildHandle(*this, id); +} + +void LocalStore::buildFinished(const BuildHandle & handle) +{ + activeBuilds.lock()->erase(handle.id); +} + +} // namespace nix diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 909f22369dc0..d54902dc7510 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -9,14 +9,18 @@ #include "nix/store/references.hh" #include "nix/util/callback.hh" #include "nix/util/topo-sort.hh" +#include "nix/util/thread-pool.hh" +#include "nix/util/util.hh" #include "nix/util/finally.hh" #include "nix/util/compression.hh" #include "nix/util/signals.hh" #include "nix/store/posix-fs-canonicalise.hh" #include "nix/util/source-accessor.hh" #include "nix/store/keys.hh" +#include "nix/store/filetransfer.hh" #include "nix/util/users.hh" #include "nix/store/store-registration.hh" +#include "nix/util/provenance.hh" #include #include @@ -99,6 +103,28 @@ bool LocalStoreConfig::getDefaultRequireSigs() return settings.requireSigs; } +/** + * A `ValidPathInfo` that also records the id of the corresponding row + * in the `ValidPaths` table of the SQLite database. This allows + * `queryValidPathId()` to return the id from `pathInfoCache` without + * consulting the database. + */ +struct LocalStorePathInfo : ValidPathInfo +{ + uint64_t id; + + LocalStorePathInfo(ValidPathInfo info, uint64_t id) + : UnkeyedValidPathInfo(info) + , ValidPathInfo(std::move(info)) + , id(id) + { + } + + void anchor() override; +}; + +void LocalStorePathInfo::anchor() {} + struct LocalStore::State::Stmts { /* Some precompiled SQLite statements. */ @@ -106,6 +132,7 @@ struct LocalStore::State::Stmts SQLiteStmt UpdatePathInfo; SQLiteStmt AddReference; SQLiteStmt QueryPathInfo; + SQLiteStmt QueryValidPathId; SQLiteStmt QueryReferences; SQLiteStmt QueryReferrers; SQLiteStmt InvalidatePath; @@ -117,8 +144,25 @@ struct LocalStore::State::Stmts SQLiteStmt QueryRealisedOutput; SQLiteStmt QueryPathFromHashPart; SQLiteStmt QueryValidPaths; + SQLiteStmt QuerySchemaMigration; }; +/** + * Write the schema version to `schemaPath` atomically: write it to a + * temporary file, fsync it, and atomically rename it into place. This + * ensures the schema file is never empty or partially written. + */ +static void writeSchemaFile(const std::filesystem::path & schemaPath, int version) +{ + auto tmp = schemaPath; + tmp += fmt(".tmp.%d", getpid()); + AutoDelete del(tmp, false); + writeFile(tmp, std::to_string(version), 0666, FsSync::Yes); + std::filesystem::rename(tmp, schemaPath); + del.cancel(); + syncParent(schemaPath); +} + LocalStore::LocalStore(ref config) : Store{*config} , LocalFSStore{*config} @@ -130,6 +174,7 @@ LocalStore::LocalStore(ref config) , schemaPath(dbDir / "schema") , tempRootsDir(config->stateDir.get() / "temproots") , fnTempRoots(tempRootsDir / std::to_string(getpid())) + , activeBuildsDir(config->stateDir.get() / "active-builds") { auto state(_state->lock()); state->stmts = std::make_unique(); @@ -150,6 +195,7 @@ LocalStore::LocalStore(ref config) const auto & localSettings = config->getLocalSettings(); const auto & gcSettings = localSettings.getGCSettings(); createDirs(gcRootsDir); + createDirs(activeBuildsDir); for (auto & perUserDir : {profilesDir / "per-user", gcRootsDir / "per-user"}) { createDirs(perUserDir); @@ -264,16 +310,7 @@ LocalStore::LocalStore(ref config) : "database schema needs migrating, but this cannot be done in read-only mode"); } - auto acquireWriteLock = [&]() { - if (!lockFile(globalLock.get(), ltWrite, false)) { - printInfo("waiting for exclusive access to the Nix store..."); - // We have acquired a shared lock; release it to prevent deadlocks. - // This can happen if someone else is trying to promote their read - // lock into a write lock. - lockFile(globalLock.get(), ltNone, false); - lockFile(globalLock.get(), ltWrite, true); - } - }; + bool isNew = false; if (curSchema > nixSchemaVersion) throw Error("current Nix store schema is version %1%, but I only support %2%", curSchema, nixSchemaVersion); @@ -281,7 +318,8 @@ LocalStore::LocalStore(ref config) else if (curSchema == 0) { /* new store */ curSchema = nixSchemaVersion; openDB(*state, true); - writeFile(schemaPath, fmt("%1%", curSchema), 0666, FsSync::Yes); + writeSchemaFile(schemaPath, curSchema); + isNew = true; } else if (curSchema < nixSchemaVersion) { @@ -297,7 +335,12 @@ LocalStore::LocalStore(ref config) "which is no longer supported. To convert to the new format,\n" "please upgrade Nix to version 1.11 first."); - acquireWriteLock(); + if (!lockFile(globalLock.get(), ltWrite, false)) { + Activity act(*logger, lvlInfo, actUnknown, "waiting for exclusive access to the Nix store"); + lockFile( + globalLock.get(), ltNone, false); // We have acquired a shared lock; release it to prevent deadlocks + lockFile(globalLock.get(), ltWrite, true); + } /* Get the schema version again, because another process may have performed the upgrade already. */ @@ -311,51 +354,45 @@ LocalStore::LocalStore(ref config) if (curSchema < 8) { SQLiteTxn txn(state->db); - state->db.exec("alter table ValidPaths add column ultimate integer"); - state->db.exec("alter table ValidPaths add column sigs text"); + state->db.execNoRetry("alter table ValidPaths add column ultimate integer"); + state->db.execNoRetry("alter table ValidPaths add column sigs text"); txn.commit(); } if (curSchema < 9) { - SQLiteTxn txn(state->db); state->db.exec("drop table FailedPaths"); - txn.commit(); } if (curSchema < 10) { - SQLiteTxn txn(state->db); state->db.exec("alter table ValidPaths add column ca text"); - txn.commit(); } - writeFile(schemaPath, fmt("%1%", nixSchemaVersion), 0666, FsSync::Yes); + writeSchemaFile(schemaPath, nixSchemaVersion); - // Downgrade to a read lock and hold to prevent other processes from - // upgrading the schema while we're using the store lockFile(globalLock.get(), ltRead, true); } else openDB(*state, false); - if (!config->readOnly && upgradeDBSchema(*state, true)) { - acquireWriteLock(); - upgradeDBSchema(*state, false); - // Downgrade to a read lock and hold to prevent other processes from - // upgrading the schema while we're using the store - lockFile(globalLock.get(), ltRead, true); - } + upgradeDBSchema(*state, isNew); /* Prepare SQL statements. */ state->stmts->RegisterValidPath.create( state->db, - "insert into ValidPaths (path, hash, registrationTime, deriver, narSize, ultimate, sigs, ca) values (?, ?, ?, ?, ?, ?, ?, ?);"); + fmt("insert into ValidPaths (path, hash, registrationTime, deriver, narSize, ultimate, sigs, ca%s) values (?, ?, ?, ?, ?, ?, ?, ?%s) " + "on conflict (path) do update set hash = excluded.hash, narSize = excluded.narSize, ultimate = excluded.ultimate, sigs = excluded.sigs, ca = excluded.ca " + "returning id;", + experimentalFeatureSettings.isEnabled(Xp::Provenance) ? ", provenance" : "", + experimentalFeatureSettings.isEnabled(Xp::Provenance) ? ", ?" : "")); state->stmts->UpdatePathInfo.create( state->db, "update ValidPaths set narSize = ?, hash = ?, ultimate = ?, sigs = ?, ca = ? where path = ?;"); state->stmts->AddReference.create(state->db, "insert or replace into Refs (referrer, reference) values (?, ?);"); state->stmts->QueryPathInfo.create( state->db, - "select id, hash, registrationTime, deriver, narSize, ultimate, sigs, ca from ValidPaths where path = ?;"); + fmt("select id, hash, registrationTime, deriver, narSize, ultimate, sigs, ca%s from ValidPaths where path = ?;", + experimentalFeatureSettings.isEnabled(Xp::Provenance) ? ", provenance" : "")); + state->stmts->QueryValidPathId.create(state->db, "select id from ValidPaths where path = ?;"); state->stmts->QueryReferences.create( state->db, "select path from Refs join ValidPaths on reference = id where referrer = ?;"); state->stmts->QueryReferrers.create( @@ -579,24 +616,11 @@ void LocalStore::openDB(State & state, bool create) } } -bool LocalStore::upgradeDBSchema(State & state, bool dryRun) +void LocalStore::upgradeDBSchema(State & state, bool isNew) { - bool ret = false; + state.db.exec("create table if not exists SchemaMigrations (migration text primary key not null);"); - { - SQLiteStmt queryHasSchemaMigrations; - queryHasSchemaMigrations.create( - state.db, "SELECT 1 FROM sqlite_master WHERE type='table' AND name='SchemaMigrations';"); - auto useQueryHasSchemaMigrations(queryHasSchemaMigrations.use()); - if (!useQueryHasSchemaMigrations.next()) { - if (dryRun) - return true; - else { - state.db.exec("create table SchemaMigrations (migration text primary key not null);"); - ret = true; - } - } - } + state.stmts->QuerySchemaMigration.create(state.db, "select 1 from SchemaMigrations where migration = ?;"); StringSet schemaMigrations; @@ -608,36 +632,39 @@ bool LocalStore::upgradeDBSchema(State & state, bool dryRun) schemaMigrations.insert(useQuerySchemaMigrations.getStr(0)); } - auto needsMigration = [&](const std::string & migrationName) -> bool { - return !schemaMigrations.contains(migrationName); - }; - - auto maybeUpgrade = [&](const std::string & migrationName, const std::string & stmt) { - if (!needsMigration(migrationName)) + auto doUpgrade = [&](const std::string & migrationName, const std::string & stmt) { + if (schemaMigrations.contains(migrationName)) return; - ret = true; - if (dryRun) - return; + if (!isNew) + notice("executing Nix database schema migration '%s'...", migrationName); - debug("executing Nix database schema migration '%s'...", migrationName); + retrySQLite([&]() { + SQLiteTxn txn(state.db); - SQLiteTxn txn(state.db); - state.db.exec(stmt + fmt(";\ninsert or ignore into SchemaMigrations values('%s')", migrationName)); - txn.commit(); + /* Another process may have executed this migration in the + meantime, so check for it again within the + transaction. */ + if (state.stmts->QuerySchemaMigration.use().apply(migrationName).next()) + return; + + state.db.execNoRetry(stmt + fmt(";\ninsert or ignore into SchemaMigrations values('%s')", migrationName)); + txn.commit(); + }); schemaMigrations.insert(migrationName); }; if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) - maybeUpgrade( + doUpgrade( "20251017-ca-derivations", #include "ca-specific-schema.sql.gen.hh" ); - maybeUpgrade("20260309-drop-redundant-indexreferrer", "drop index if exists IndexReferrer"); + doUpgrade("20260309-drop-redundant-indexreferrer", "drop index if exists IndexReferrer"); - return ret; + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) + doUpgrade("20241024-provenance", "alter table ValidPaths add column provenance text"); } /* To improve purity, users may want to make the Nix store a read-only @@ -711,17 +738,29 @@ uint64_t LocalStore::addValidPath(State & state, const ValidPathInfo & info) "cannot add path '%s' to the Nix store because it claims to be content-addressed but isn't", printStorePath(info.path)); - state.stmts->RegisterValidPath.use() - .apply(printStorePath(info.path)) - .apply(info.narHash.to_string(HashFormat::Base16, true)) - .apply(info.registrationTime == 0 ? time(nullptr) : info.registrationTime) - .apply(info.deriver ? printStorePath(*info.deriver) : "", (bool) info.deriver) - .apply(info.narSize, info.narSize != 0) - .apply(info.ultimate ? 1 : 0, info.ultimate) - .apply(concatStringsSep(" ", Signature::toStrings(info.sigs)), !info.sigs.empty()) - .apply(renderContentAddress(info.ca), (bool) info.ca) - .exec(); - uint64_t id = state.db.getLastInsertedRowId(); + auto query = state.stmts->RegisterValidPath.use() + .apply(printStorePath(info.path)) + .apply(info.narHash.to_string(HashFormat::Base16, true)) + .apply(info.registrationTime == 0 ? time(nullptr) : info.registrationTime) + .apply(info.deriver ? printStorePath(*info.deriver) : "", (bool) info.deriver) + .apply(info.narSize, info.narSize != 0) + .apply(info.ultimate ? 1 : 0, info.ultimate) + .apply(concatStringsSep(" ", Signature::toStrings(info.sigs)), !info.sigs.empty()) + .apply(renderContentAddress(info.ca), (bool) info.ca); + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) + query.apply(info.provenance ? info.provenance->to_json_str() : "", (bool) info.provenance); + /* Valid ids start at 1, so a last-inserted rowid of 0 signals + that the upsert did an update rather than an insert. */ + state.db.setLastInsertedRowId(0); + if (!query.next()) + throw Error("registering path '%s' in the Nix database did not return an id", printStorePath(info.path)); + uint64_t id = query.getInt(0); + + /* If the path was already valid (i.e. the upsert did an update + rather than an insert), the derivation outputs have already + been registered, so we're done. */ + if (state.db.getLastInsertedRowId() != id) + return id; /* If this is a derivation, then store the derivation outputs in the database. This is useful for the garbage collector: it can @@ -745,7 +784,8 @@ uint64_t LocalStore::addValidPath(State & state, const ValidPathInfo & info) } } - pathInfoCache->lock()->upsert(info.path, PathInfoCacheValue{.value = std::make_shared(info)}); + pathInfoCache->lock()->upsert( + info.path, PathInfoCacheValue{.value = std::make_shared(info, id)}); return id; } @@ -780,7 +820,7 @@ std::shared_ptr LocalStore::queryPathInfoInternal(State & s throw Error("invalid-path entry for '%s': %s", printStorePath(path), e.what()); } - auto info = std::make_shared(path, UnkeyedValidPathInfo(*this, narHash)); + auto info = std::make_shared(ValidPathInfo(path, UnkeyedValidPathInfo(*this, narHash)), id); info->registrationTime = useQueryPathInfo.getInt(2); @@ -807,6 +847,12 @@ std::shared_ptr LocalStore::queryPathInfoInternal(State & s while (useQueryReferences.next()) info->references.insert(parseStorePath(useQueryReferences.getStr(0))); + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) { + auto prov = (const char *) sqlite3_column_text(state.stmts->QueryPathInfo, 8); + if (prov) + info->provenance = Provenance::from_json_str(prov); + } + return info; } @@ -825,7 +871,15 @@ void LocalStore::updatePathInfo(State & state, const ValidPathInfo & info) uint64_t LocalStore::queryValidPathId(State & state, const StorePath & path) { - auto use(state.stmts->QueryPathInfo.use().apply(printStorePath(path))); + /* Avoid a database query if the id is in `pathInfoCache`. */ + { + auto cache(pathInfoCache->lock()); + if (auto res = cache->getOrNullptr(path)) + if (auto info = dynamic_cast(res->value.get())) + return info->id; + } + + auto use(state.stmts->QueryValidPathId.use().apply(printStorePath(path))); if (!use.next()) throw InvalidPath("path '%s' is not valid", printStorePath(path)); return use.getInt(0); @@ -833,7 +887,7 @@ uint64_t LocalStore::queryValidPathId(State & state, const StorePath & path) bool LocalStore::isValidPath_(State & state, const StorePath & path) { - return state.stmts->QueryPathInfo.use().apply(printStorePath(path)).next(); + return state.stmts->QueryValidPathId.use().apply(printStorePath(path)).next(); } bool LocalStore::isValidPathUncached(const StorePath & path) @@ -970,20 +1024,27 @@ void LocalStore::registerValidPaths(const ValidPathInfos & infos) SQLiteTxn txn(state->db); StorePathSet paths; + std::unordered_map ids; for (auto & [_, i] : infos) { assert(i.narHash.algo == HashAlgorithm::SHA256); - if (isValidPath_(*state, i.path)) - updatePathInfo(*state, i); - else - addValidPath(*state, i); + ids.insert_or_assign(i.path, addValidPath(*state, i)); paths.insert(i.path); } for (auto & [_, i] : infos) { - auto referrer = queryValidPathId(*state, i.path); - for (auto & j : i.references) - state->stmts->AddReference.use().apply(referrer).apply(queryValidPathId(*state, j)).exec(); + auto referrer = ids.at(i.path); + for (auto & j : i.references) { + auto k = ids.find(j); + /* Note that queryValidPathId() uses pathInfoCache, so it's possible that path `j` has been + * garbage-collected. But that's fine: the foreign key constraint on Refs will detect that the ID is no + * longer valid. Path IDs are never reused so there is no possibility that ID now refers to another + * path. */ + state->stmts->AddReference.use() + .apply(referrer) + .apply(k != ids.end() ? k->second : queryValidPathId(*state, j)) + .exec(); + } } /* Do a topological sort of the paths. This will throw an @@ -1043,6 +1104,105 @@ bool LocalStore::realisationIsUntrusted(const Realisation & realisation) return config->requireSigs && !realisation.checkSignatures(realisation.id, getPublicKeys()); } +/* Check whether `path` exists and is owned by the current user. This is + used to decide whether to trust a previously-unpacked store path and + its `.unpacked` marker. Otherwise, if sandboxing is disabled, a build + user could plant files to trick us into registering an arbitrary + store path as valid. */ +static bool existsAndIsOwnedBySelf(const std::filesystem::path & path) +{ + auto st = maybeLstat(path); + if (!st) + return false; +#ifndef _WIN32 + if (st->st_uid != geteuid()) + return false; +#endif + return true; +} + +void LocalStore::doAddToStore(const ValidPathInfo & info, Source & source, RepairFlag repair) +{ + const LocalSettings & localSettings = config->getLocalSettings(); + + auto realPath = toRealPath(info.path); + + // TODO: make repair more safe: write to a temporary location first before renaming/deleting. + deletePath(realPath); + + /* Maybe free up some disk space (before writing the NAR) so that + restorePath() below doesn't fail with ENOSPC. */ + autoGC(); + + /* While restoring the path from the NAR, compute the hash + of the NAR. */ + HashSink hashSink(HashAlgorithm::SHA256); + + TeeSource wrapperSource{source, hashSink}; + + restorePath(realPath, wrapperSource, localSettings.fsyncStorePaths); + + auto hashResult = hashSink.finish(); + + if (hashResult.numBytesDigested != info.narSize) + throw Error( + "size mismatch importing path '%s';\n specified: %s\n got: %s", + printStorePath(info.path), + info.narSize, + hashResult.numBytesDigested); + + if (hashResult.hash != info.narHash) + throw Error( + "hash mismatch importing path '%s';\n specified: %s\n got: %s", + printStorePath(info.path), + info.narHash.to_string(HashFormat::SRI, true), + hashResult.hash.to_string(HashFormat::SRI, true)); + + if (info.ca) { + auto & specified = *info.ca; + auto actualHash = ({ + SourcePath sourcePath = requireStoreObjectAccessor(info.path, /*requireValidPath=*/false); + Hash h{HashAlgorithm::SHA256}; // throwaway def to appease C++ + auto fim = specified.method.getFileIngestionMethod(); + switch (fim) { + case FileIngestionMethod::Flat: + case FileIngestionMethod::NixArchive: { + HashModuloSink caSink{ + specified.hash.algo, + std::string{info.path.hashPart()}, + }; + dumpPath(sourcePath, caSink, (FileSerialisationMethod) fim); + h = caSink.finish().hash; + break; + } + case FileIngestionMethod::Git: + h = git::dumpHash(specified.hash.algo, sourcePath).hash; + break; + } + ContentAddress{ + .method = specified.method, + .hash = std::move(h), + }; + }); + if (specified.hash != actualHash.hash) { + throw Error( + "ca hash mismatch importing path '%s';\n specified: %s\n got: %s", + printStorePath(info.path), + specified.hash.to_string(HashFormat::Nix32, true), + actualHash.hash.to_string(HashFormat::Nix32, true)); + } + } + + canonicalisePathMetaData(realPath, {NIX_WHEN_SUPPORT_ACLS(localSettings.ignoredAcls)}); + + optimisePath(realPath, repair); // FIXME: combine with hashPath() + + if (localSettings.fsyncStorePaths) { + recursiveSync(realPath); + syncParent(realPath); + } +} + void LocalStore::addToStore(const ValidPathInfo & info, Source & source, RepairFlag repair, CheckSigsFlag checkSigs) { if (checkSigs && pathInfoIsUntrusted(info)) @@ -1066,77 +1226,7 @@ void LocalStore::addToStore(const ValidPathInfo & info, Source & source, RepairF /* The path may have been created by another process in the meantime, so check again. */ if (repair || !isValidPathUncached(info.path)) { - deletePath(realPath); - - /* While restoring the path from the NAR, compute the hash - of the NAR. */ - HashSink hashSink(HashAlgorithm::SHA256); - - TeeSource wrapperSource{source, hashSink}; - - restorePath(realPath, wrapperSource, config->getLocalSettings().fsyncStorePaths); - - auto hashResult = hashSink.finish(); - - if (hashResult.hash != info.narHash) - throw Error( - "hash mismatch importing path '%s';\n specified: %s\n got: %s", - printStorePath(info.path), - info.narHash.to_string(HashFormat::SRI, true), - hashResult.hash.to_string(HashFormat::SRI, true)); - - if (hashResult.numBytesDigested != info.narSize) - throw Error( - "size mismatch importing path '%s';\n specified: %s\n got: %s", - printStorePath(info.path), - info.narSize, - hashResult.numBytesDigested); - - if (info.ca) { - auto & specified = *info.ca; - auto actualHash = ({ - SourcePath sourcePath = requireStoreObjectAccessor(info.path, /*requireValidPath=*/false); - Hash h{HashAlgorithm::SHA256}; // throwaway def to appease C++ - auto fim = specified.method.getFileIngestionMethod(); - switch (fim) { - case FileIngestionMethod::Flat: - case FileIngestionMethod::NixArchive: { - HashModuloSink caSink{ - specified.hash.algo, - std::string{info.path.hashPart()}, - }; - dumpPath(sourcePath, caSink, (FileSerialisationMethod) fim); - h = caSink.finish().hash; - break; - } - case FileIngestionMethod::Git: - h = git::dumpHash(specified.hash.algo, sourcePath).hash; - break; - } - ContentAddress{ - .method = specified.method, - .hash = std::move(h), - }; - }); - if (specified.hash != actualHash.hash) { - throw Error( - "ca hash mismatch importing path '%s';\n specified: %s\n got: %s", - printStorePath(info.path), - specified.hash.to_string(HashFormat::Nix32, true), - actualHash.hash.to_string(HashFormat::Nix32, true)); - } - } - - autoGC(); - - canonicalisePathMetaData(realPath, {NIX_WHEN_SUPPORT_ACLS(config->getLocalSettings().ignoredAcls)}); - - optimisePath(realPath, repair); // FIXME: combine with hashPath() - - if (config->getLocalSettings().fsyncStorePaths) { - recursiveSync(realPath); - syncParent(realPath); - } + doAddToStore(info, source, repair); registerValidPath(info); } else @@ -1148,6 +1238,156 @@ void LocalStore::addToStore(const ValidPathInfo & info, Source & source, RepairF } } +void LocalStore::addMultipleToStore( + PathsSource && pathsToCopy, Activity & act, RepairFlag repair, CheckSigsFlag checkSigs) +{ + /* Atomic because it's decremented from the worker threads below + (when a download is skipped). */ + std::atomic bytesExpected{0}; + for (auto & [info, _] : pathsToCopy) + bytesExpected += info.narSize; + act.setExpected(actCopyPath, bytesExpected); + + std::atomic nrDone{0}; + std::atomic nrRunning{0}; + auto showProgress = [&, nrTotal = pathsToCopy.size()]() { act.progress(nrDone, nrTotal, nrRunning); }; + + using PathWithSource = std::pair>; + + /* Stage A: figure out which paths are not already valid, lock them, + and re-check their validity under the lock. + + `PathLocks::lockPaths()` acquires the locks in sorted order, so + all processes lock store paths in the same order and we cannot + deadlock against another process writing to the same store. All + locks are held until we've registered the paths as valid + (stage C). */ + std::set pathsToLock; + std::vector maybeToWrite; + + for (auto & item : pathsToCopy) { + auto & info = item.first; + + if (checkSigs && pathInfoIsUntrusted(info)) + throw Error( + "cannot add path '%s' because it lacks a signature by a trusted key", printStorePath(info.path)); + + addTempRoot(info.path); + + if (!repair && isValidPath(info.path)) { + nrDone++; + showProgress(); + continue; + } + + maybeToWrite.push_back(&item); + + /* Lock the output path. But don't lock if we're being called + from a build hook (whose parent process already acquired a + lock on this path). */ + if (!locksHeld.count(printStorePath(info.path))) + pathsToLock.insert(toRealPath(info.path)); + } + + if (maybeToWrite.empty()) + return; + + PathLocks outputLocks; + if (!pathsToLock.empty()) + outputLocks.lockPaths(pathsToLock); + + std::vector toWrite; + + for (auto * item : maybeToWrite) { + auto & info = item->first; + + /* The path may have been created by another process in the meantime. */ + if (!repair && isValidPathUncached(info.path)) { + // We may have a negative cache entry for this path, so get rid of it. + invalidatePathInfoCacheFor(info.path); + nrDone++; + showProgress(); + continue; + } + + info.ultimate = false; + toWrite.push_back(item); + } + + /* Stage B: extract the NARs in parallel, in order of descending NAR + size so that the largest (and typically slowest) NARs are started + first. This does everything `addToStore` does except registering + the paths as valid, which is deferred to stage C. */ + std::sort(toWrite.begin(), toWrite.end(), [](auto * a, auto * b) { return a->first.narSize > b->first.narSize; }); + + /* Use enough threads to saturate both the local CPUs (for + decompression/hashing/optimising) and the incoming connections to + the source store (e.g. a binary cache). */ + ThreadPool pool(std::max((size_t) std::thread::hardware_concurrency(), fileTransferSettings.httpConnections.get())); + + for (auto * item : toWrite) { + pool.enqueue([&, item]() { + checkInterrupt(); + + auto & info = item->first; + + /* Make sure that the Source object is destroyed when + we're done. In particular, a SinkToSource object must + be destroyed to ensure that the destructors on its + stack frame are run; this includes + LegacySSHStore::narFromPath()'s connection lock. */ + auto source = std::move(item->second); + + MaintainCount mc(nrRunning); + showProgress(); + + /* The existence of `.unpacked` marks that the NAR + for this path was previously unpacked (see below). This allows + an interrupted `addMultipleToStore()` to reuse already-extracted + paths instead of re-fetching them. + + Require both the marker and the unpacked path to be owned + by us, so that (when sandboxing is disabled) a build user + can't plant a fake marker to trick us into registering an + arbitrary path as valid. + + Note that GC will delete the marker before the path itself, + so if the marker and the path both exists, the contents of + the path are correct. */ + auto realPath = toRealPath(info.path); + auto unpackedMarker = unpackedMarkerFor(realPath); + + if (!repair && existsAndIsOwnedBySelf(unpackedMarker) && existsAndIsOwnedBySelf(realPath)) { + notice("reusing previously unpacked path at '%s'", printStorePath(info.path)); + act.setExpected(actCopyPath, bytesExpected -= info.narSize); + } else { + doAddToStore(info, *source, repair); + /* The path has been unpacked, so mark it as such. */ + writeFile(unpackedMarker, ""); + } + + nrDone++; + showProgress(); + }); + } + + pool.process(); + + /* Stage C: register all the newly added paths as valid in a single + transaction. */ + ValidPathInfos infos; + for (auto * item : toWrite) + infos.insert_or_assign(item->first.path, item->first); + + registerValidPaths(infos); + + /* Now that the paths are registered as valid, the .unpacked markers are no longer needed. */ + for (auto * item : toWrite) + tryUnlink(unpackedMarkerFor(toRealPath(item->first.path))); + + outputLocks.setDeletion(true); +} + StorePath LocalStore::addToStoreFromDump( Source & source0, std::string_view name, @@ -1155,7 +1395,8 @@ StorePath LocalStore::addToStoreFromDump( ContentAddressMethod hashMethod, HashAlgorithm hashAlgo, const StorePathSet & references, - RepairFlag repair) + RepairFlag repair, + std::shared_ptr provenance) { /* For computing the store path. */ auto hashSink = std::make_unique(hashAlgo); @@ -1192,7 +1433,7 @@ StorePath LocalStore::addToStoreFromDump( dumpBuffer.release(); dumpBuffer.reset((char *) tmp); } else { - throw std::bad_alloc(); + outOfMemory(); } auto got = 0; Finally cleanup([&]() { dump = {dumpBuffer.get(), dump.size() + got}; }); @@ -1302,6 +1543,7 @@ StorePath LocalStore::addToStoreFromDump( auto info = ValidPathInfo::makeFromCA(*this, name, std::move(desc), narHash.hash); info.narSize = narHash.numBytesDigested; + info.provenance = provenance; registerValidPath(info); } else // We may have a negative cache entry for this path, so get rid of it. diff --git a/src/libstore/make.nix b/src/libstore/make.nix new file mode 100644 index 000000000000..76354905ee06 --- /dev/null +++ b/src/libstore/make.nix @@ -0,0 +1,72 @@ +# libnixstore, transcribed from meson.build and its subdirectories. +{ + pkgs, + nixMake, + nix-util, +}: + +let + inherit (pkgs) lib; +in + +nixMake.mkComponent { + name = "determinate-nix-store"; + libName = "nixstore"; + + deps = [ nix-util ]; + + root = ./.; + + excludeSources = [ + "windows" + "freebsd" + "darwin" + ] + ++ lib.optional (!nixMake.config.awsAuth) "aws-creds.cc"; + + includeDirs = [ + "" + "include" + "linux/build" + "linux/include" + "unix/build" + "unix/include" + ]; + + files = nixMake.commonSupportFiles // { + "schema.sql.gen.hh" = nixMake.mkStringHeader ./schema.sql; + "ca-specific-schema.sql.gen.hh" = nixMake.mkStringHeader ./ca-specific-schema.sql; + }; + + configHeaders = { + "include/nix/store/config.hh" = { + NIX_LOCAL_SYSTEM = "x86_64-linux"; + NIX_SUPPORT_ACL = 1; + NIX_WITH_AWS_AUTH = nixMake.config.awsAuth; + }; + "store-config-private.hh" = { + CAN_LINK_SYMLINK = 1; + DETERMINATE_NIX_VERSION = lib.fileContents ../../.version-determinate; + HAVE_EMBEDDED_SANDBOX_SHELL = 0; + HAVE_LANDLOCK = 1; + HAVE_MOVE_MOUNT = 1; + HAVE_OPEN_TREE = 1; + HAVE_POSIX_FALLOCATE = 1; + HAVE_SECCOMP = 1; + HAVE_STATVFS = 1; + IS_STATIC = null; + LSOF = "lsof"; + NIX_CONF_DIR = "/etc/nix"; + NIX_LOG_DIR = "/nix/var/log/nix"; + NIX_STATE_DIR = "/nix/var/nix"; + NIX_STORE_DIR = "/nix/store"; + NIX_USE_WASMTIME = nixMake.config.wasm; + PACKAGE_VERSION = lib.fileContents ../../.version; + SANDBOX_SHELL = "${pkgs.busybox-sandbox-shell}/bin/busybox"; + }; + }; + + extraCxxFlags = nixMake.weakVtablesFlags; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libstore/meson.build b/src/libstore/meson.build index c266f05d4dd1..9f556aace91b 100644 --- a/src/libstore/meson.build +++ b/src/libstore/meson.build @@ -13,6 +13,8 @@ project( license : 'LGPL-2.1-or-later', ) +fs = import('fs') + cxx = meson.get_compiler('cpp') subdir('nix-meson-build-support/deps-lists') @@ -22,6 +24,12 @@ configdata_priv = configuration_data() # TODO rename, because it will conflict with downstream projects configdata_priv.set_quoted('PACKAGE_VERSION', meson.project_version()) +configdata_priv.set('IS_STATIC', get_option('default_library') == 'static') + +configdata_priv.set_quoted( + 'DETERMINATE_NIX_VERSION', + fs.read('../../.version-determinate').strip(), +) subdir('nix-meson-build-support/default-system-cpu') @@ -212,8 +220,6 @@ if get_option('embedded-sandbox-shell') generated_headers += embedded_sandbox_shell_gen endif -fs = import('fs') - prefix = get_option('prefix') # For each of these paths, assume that it is relative to the prefix unless # it is already an absolute path (which is the default for store-dir, localstatedir, and log-dir). @@ -273,6 +279,19 @@ configdata_priv.set_quoted( : 'lsof', ) +link_args = [] + +wasmtime_required = get_option('wasm').disable_if( + get_option('default_library') == 'static', + error_message : 'Building with wasmtime and static linking is not supported', +) + +if wasmtime_required.enabled() + link_args += '-lwasmtime' +endif + +configdata_priv.set('NIX_USE_WASMTIME', wasmtime_required.enabled().to_int()) + config_priv_h = configure_file( configuration : configdata_priv, output : 'store-config-private.hh', @@ -281,6 +300,8 @@ config_priv_h = configure_file( subdir('nix-meson-build-support/common') sources = files( + 'active-builds.cc', + 'async-path-writer.cc', 'binary-cache-store.cc', 'build-result.cc', 'build/build-log.cc', @@ -321,6 +342,7 @@ sources = files( 'local-fs-store.cc', 'local-gc.cc', 'local-overlay-store.cc', + 'local-store-active-builds.cc', 'local-store.cc', 'log-store.cc', 'machines.cc', @@ -340,6 +362,7 @@ sources = files( 'pathlocks.cc', 'posix-fs-canonicalise.cc', 'profiles.cc', + 'provenance.cc', 'realisation.cc', 'references.cc', 'remote-fs-accessor.cc', @@ -394,11 +417,10 @@ this_library = library( generated_headers, sources, config_priv_h, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, - link_args : linker_export_flags, - prelink : true, # For C++ static initializers + link_args : linker_export_flags + link_args, + prelink : prelink, # For C++ static initializers install : true, cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [], ) diff --git a/src/libstore/meson.options b/src/libstore/meson.options index c822133df46e..6bae2ab11f17 100644 --- a/src/libstore/meson.options +++ b/src/libstore/meson.options @@ -39,3 +39,9 @@ option( type : 'feature', description : 'build support for AWS authentication with S3', ) + +option( + 'wasm', + type : 'feature', + description : 'enable wasmtime integration into the Nix derivation builder', +) diff --git a/src/libstore/misc.cc b/src/libstore/misc.cc index 8db977b6fce0..7d42acb9bdb8 100644 --- a/src/libstore/misc.cc +++ b/src/libstore/misc.cc @@ -18,15 +18,11 @@ namespace nix { void Store::computeFSClosure( - const StorePathSet & startPaths, - StorePathSet & paths_, - bool flipDirection, - bool includeOutputs, - bool includeDerivers) + const StorePathSet & startPaths, StorePathSet & out, bool flipDirection, bool includeOutputs, bool includeDerivers) { - std::function(const StorePath & path)> queryDeps; - if (flipDirection) - queryDeps = [this, includeOutputs, includeDerivers](const StorePath & path) -> asio::awaitable { + if (flipDirection) { + std::function(const StorePath & path)> queryDeps = + [this, includeOutputs, includeDerivers](const StorePath & path) -> asio::awaitable { StorePathSet res; StorePathSet referrers; queryReferrers(path, referrers); @@ -44,27 +40,72 @@ void Store::computeFSClosure( res.insert(*maybeOutPath); co_return res; }; - else - queryDeps = [this, includeOutputs, includeDerivers](const StorePath & path) -> asio::awaitable { - StorePathSet res; - auto info = co_await callbackToAwaitable>( - [this, path](Callback> cb) { queryPathInfo(path, std::move(cb)); }); + computeClosure(startPaths, out, GetEdgesAsync(queryDeps)); + } else { - for (auto & ref : info->references) - if (ref != path) - res.insert(ref); + asio::io_context ctx; + std::exception_ptr ex; - if (includeOutputs && path.isDerivation()) - for (auto & [_, maybeOutPath] : queryPartialDerivationOutputMap(path)) - if (maybeOutPath && isValidPath(*maybeOutPath)) - res.insert(*maybeOutPath); + StorePathSet required, done; - if (includeDerivers && info->deriver && isValidPath(*info->deriver)) - res.insert(*info->deriver); - co_return res; + std::function(StorePathSet)> doPaths; + doPaths = [&](StorePathSet paths) -> asio::awaitable { + StorePathSet batch; + for (auto & path : std::exchange(paths, {})) + if (done.insert(path).second) + batch.insert(path); + + auto executor = co_await asio::this_coro::executor; + + co_await queryPathInfos( + batch, [&](std::vector>> infos) { + StorePathSet todo; + + for (auto & [path, info] : infos) { + if (!info) { + if (required.contains(path)) + throw InvalidPath("path '%s' is not valid", printStorePath(path)); + continue; + } + + out.insert(path); + + for (auto & ref : info->references) + if (ref != path) { + required.insert(ref); + todo.insert(ref); + } + + if (includeOutputs && path.isDerivation()) + // FIXME: need an async, multiple-path version of queryPartialDerivationOutputMap(). + for (auto & [_, maybeOutPath] : queryPartialDerivationOutputMap(path)) + if (maybeOutPath) + todo.insert(*maybeOutPath); + + if (includeDerivers && info->deriver) + todo.insert(*info->deriver); + + // FIXME: process partialClosure when we merge + // https://github.com/DeterminateSystems/nix-src/pull/523. + } + + if (!todo.empty()) + asio::co_spawn(executor, std::bind(doPaths, todo), [&](std::exception_ptr e) { + if (e) + ex = e; + }); + }); }; - computeClosure(startPaths, paths_, GetEdgesAsync(queryDeps)); + asio::co_spawn(ctx, std::bind(doPaths, startPaths), [&](std::exception_ptr e) { + if (e) + ex = e; + }); + + ctx.run(); + if (ex) + std::rethrow_exception(ex); + } } void Store::computeFSClosure( @@ -178,7 +219,8 @@ static void collectDerivedPaths( MissingPaths Store::queryMissing(const std::vector & targets) { - Activity act(*logger, lvlDebug, actUnknown, "querying info about missing paths"); + Activity act(*logger, lvlDebug, "QueryMissing", {}, "querying info about missing paths"); + PushActivity pact(act.id); MissingPaths res; @@ -301,7 +343,7 @@ MissingPaths Store::queryMissing(const std::vector & targets) } }, [&](const DerivedPath::Opaque & bo) -> asio::awaitable { - if (isValidPath(bo.path)) + if (maybeQueryPathInfo(bo.path)) co_return; SubstitutablePathInfos infos; diff --git a/src/libstore/nar-info-disk-cache.cc b/src/libstore/nar-info-disk-cache.cc index 6554bd7bf2e4..b66d36d652bc 100644 --- a/src/libstore/nar-info-disk-cache.cc +++ b/src/libstore/nar-info-disk-cache.cc @@ -2,7 +2,9 @@ #include "nix/util/users.hh" #include "nix/util/sync.hh" #include "nix/store/sqlite.hh" +#include "nix/store/pathlocks.hh" #include "nix/store/globals.hh" +#include "nix/store/provenance.hh" #include #include @@ -18,8 +20,7 @@ create table if not exists BinaryCaches ( url text unique not null, timestamp integer not null, storeDir text not null, - wantMassQuery integer not null, - priority integer not null + fields text not null ); create table if not exists NARs ( @@ -36,12 +37,18 @@ create table if not exists NARs ( deriver text, sigs text, ca text, + provenance text, timestamp integer not null, present integer not null, primary key (cache, hashPart), foreign key (cache) references BinaryCaches(id) on delete cascade ); +-- Used by the periodic purge of expired entries. Without it, the +-- purge is a full table scan, which on a multi-GiB cache can hold the +-- SQLite write lock for minutes. +create index if not exists NARsExpiry on NARs(present, timestamp); + create table if not exists BuildTrace ( cache integer not null, @@ -72,6 +79,12 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache /* How often to purge expired entries from the cache. */ const int purgeInterval = 24 * 3600; + /* How many expired entries to delete per transaction when + purging. Each chunk is committed separately so that the SQLite + write lock is never held for long and progress is not lost if + the process is killed. */ + const int purgeChunkSize = 1000; + struct Cache { std::string storeDir; @@ -91,7 +104,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache NarInfoDiskCacheImpl( const Settings & settings, SQLiteSettings sqliteSettings, - std::filesystem::path dbPath = getCacheDir() / "binary-cache-v8.sqlite") + std::filesystem::path dbPath = getCacheDir() / "binary-cache-detsys-v4.sqlite") : NarInfoDiskCache{settings} { auto state(_state.lock()); @@ -106,23 +119,22 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache state->insertCache.create( state->db, - "insert into BinaryCaches(url, timestamp, storeDir, wantMassQuery, priority) values (?1, ?2, ?3, ?4, ?5) on conflict (url) do update set timestamp = ?2, storeDir = ?3, wantMassQuery = ?4, priority = ?5 returning id;"); + "insert into BinaryCaches(url, timestamp, storeDir, fields) values (?1, ?2, ?3, ?4) on conflict (url) do update set timestamp = ?2, storeDir = ?3, fields = ?4 returning id;"); state->queryCache.create( - state->db, - "select id, storeDir, wantMassQuery, priority from BinaryCaches where url = ? and timestamp > ?"); + state->db, "select id, storeDir, fields from BinaryCaches where url = ? and timestamp > ?"); state->insertNAR.create( state->db, "insert or replace into NARs(cache, hashPart, namePart, url, compression, fileHash, fileSize, narHash, " - "narSize, refs, deriver, sigs, ca, timestamp, present) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)"); + "narSize, refs, deriver, sigs, ca, provenance, timestamp, present) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)"); state->insertMissingNAR.create( state->db, "insert or replace into NARs(cache, hashPart, timestamp, present) values (?, ?, ?, 0)"); state->queryNAR.create( state->db, - "select present, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca from NARs where cache = ? and hashPart = ? and ((present = 0 and timestamp > ?) or (present = 1 and timestamp > ?))"); + "select present, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca, provenance from NARs where cache = ? and hashPart = ? and ((present = 0 and timestamp > ?) or (present = 1 and timestamp > ?))"); state->insertRealisation.create( state->db, @@ -148,31 +160,64 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache )"); /* Periodically purge expired entries from the database. */ - retrySQLite([&]() { - auto now = time(nullptr); + auto now = time(nullptr); + auto purgeDue = retrySQLite([&]() { SQLiteStmt queryLastPurge(state->db, "select value from LastPurge"); auto queryLastPurge_(queryLastPurge.use()); + return !queryLastPurge_.next() || queryLastPurge_.getInt(0) < now - purgeInterval; + }); - if (!queryLastPurge_.next() || queryLastPurge_.getInt(0) < now - purgeInterval) { - SQLiteStmt( + if (purgeDue) { + /* Processes that start around the same time will all + find the purge due, so take a lock to ensure only one + of them does it. The others skip the purge. Since + LastPurge is only updated once the purge completes, if + the purging process is killed, the next process to open + the cache will take over. */ + auto purgeLock = openLockFile(dbPath.string() + ".purge-lock", true); + if (!lockFile(purgeLock.get(), ltWrite, false)) { + debug("skipping purge of the NAR info disk cache because another process is doing it"); + } else { + /* Delete in chunks, each in its own autocommit + transaction, so that the write lock is released + between chunks and concurrent processes are only + ever blocked for the duration of one chunk. If this + process is killed, the chunks already committed stay + deleted and the next purge continues where we left + off. */ + SQLiteStmt purge( state->db, - "delete from NARs where ((present = 0 and timestamp < ?) or (present = 1 and timestamp < ?))") - .use() - // Use a minimum TTL to prevent --refresh from - // nuking the entire disk cache. - .apply(now - std::max(settings.ttlNegative.get(), 3600U)) - .apply(now - std::max(settings.ttlPositive.get(), 30 * 24 * 3600U)) - .exec(); + "delete from NARs where rowid in (select rowid from NARs where ((present = 0 and timestamp < ?) or (present = 1 and timestamp < ?)) limit ?)"); + + uint64_t deleted = 0; + while (true) { + auto n = retrySQLite([&]() { + purge + .use() + // Use a minimum TTL to prevent --refresh from + // nuking the entire disk cache. + .apply(now - std::max(settings.ttlNegative.get(), 3600U)) + .apply(now - std::max(settings.ttlPositive.get(), 30 * 24 * 3600U)) + .apply(purgeChunkSize) + .exec(); + return sqlite3_changes(state->db); + }); + deleted += n; + if (n < purgeChunkSize) + break; + } - debug("deleted %d entries from the NAR info disk cache", sqlite3_changes(state->db)); + debug("deleted %d entries from the NAR info disk cache", deleted); - SQLiteStmt(state->db, "insert or replace into LastPurge(dummy, value) values ('', ?)") - .use() - .apply(now) - .exec(); + retrySQLite([&]() { + SQLiteStmt(state->db, "insert or replace into LastPurge(dummy, value) values ('', ?)") + .use() + .apply(now) + .exec(); + }); } - }); + } } Cache & getCache(State & state, const std::string & uri) @@ -200,8 +245,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache .storeDir = queryCache.getStr(1), .info = { .id = (int) queryCache.getInt(0), - .wantMassQuery = queryCache.getInt(2) != 0, - .priority = (int) queryCache.getInt(3), + .fields = nlohmann::json::parse(queryCache.getStr(2)).get>(), }}; state.caches.emplace(uri, cache); } @@ -229,8 +273,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache .apply(uri) .apply(time(nullptr)) .apply(storeDir) - .apply(info.wantMassQuery) - .apply(info.priority)); + .apply(nlohmann::json(info.fields).dump())); if (!r.next()) { unreachable(); } @@ -294,6 +337,8 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache for (auto & sig : tokenizeString(queryNAR.getStr(10), " ")) narInfo->sigs.insert(Signature::parse(sig)); narInfo->ca = ContentAddress::parseOpt(queryNAR.getStr(11)); + if (experimentalFeatureSettings.isEnabled(Xp::Provenance) && !queryNAR.isNull(12)) + narInfo->provenance = Provenance::from_json_str_optional(queryNAR.getStr(12)); return {oValid, narInfo}; }); @@ -374,6 +419,9 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache .apply(info->deriver ? std::string(info->deriver->to_string()) : "", (bool) info->deriver) .apply(concatStringsSep(" ", Signature::toStrings(info->sigs))) .apply(renderContentAddress(info->ca)) + .apply( + info->provenance ? info->provenance->to_json_str() : "", + experimentalFeatureSettings.isEnabled(Xp::Provenance) && info->provenance) .apply(time(nullptr)) .exec(); diff --git a/src/libstore/nar-info.cc b/src/libstore/nar-info.cc index 8a7ab0fae489..b71fa7888a10 100644 --- a/src/libstore/nar-info.cc +++ b/src/libstore/nar-info.cc @@ -2,6 +2,7 @@ #include "nix/store/store-dir-config.hh" #include "nix/util/strings.hh" #include "nix/util/json-utils.hh" +#include "nix/util/provenance.hh" namespace nix { @@ -90,7 +91,8 @@ NarInfo::NarInfo(const StoreDirConfig & store, const std::string & s, const std: throw corrupt("extra CA"); // FIXME: allow blank ca or require skipping field? ca = ContentAddress::parseOpt(value); - } + } else if (name == "Provenance" && experimentalFeatureSettings.isEnabled(Xp::Provenance)) + provenance = Provenance::from_json_str(value); pos = eol + 1; line += 1; @@ -138,6 +140,9 @@ std::string NarInfo::to_string(const StoreDirConfig & store) const if (ca) res += "CA: " + renderContentAddress(*ca) + "\n"; + if (provenance && experimentalFeatureSettings.isEnabled(Xp::Provenance)) + res += "Provenance: " + provenance->to_json_str() + "\n"; + return res; } diff --git a/src/libstore/optimise-store.cc b/src/libstore/optimise-store.cc index fa19b08d6ece..05d35c935e2f 100644 --- a/src/libstore/optimise-store.cc +++ b/src/libstore/optimise-store.cc @@ -1,10 +1,13 @@ #include "nix/store/local-store.hh" #include "nix/store/local-settings.hh" #include "nix/util/signals.hh" +#include "nix/util/thread-pool.hh" #include "nix/store/posix-fs-canonicalise.hh" #include "nix/util/source-accessor.hh" #include "nix/util/file-system.hh" +#include + #include #include #ifdef __APPLE__ @@ -103,7 +106,14 @@ void LocalStore::optimisePath_( { checkInterrupt(); - auto st = lstat(path); + /* Missing valid paths are a problem, but they should not prevent + store optimisation from finishing, so warn and skip them. */ + auto optSt = maybeLstat(path); + if (!optSt) { + warn("skipping missing path %s", PathFmt(path)); + return; + } + auto & st = *optSt; #ifdef __APPLE__ /* HFS/macOS has some undocumented security feature disabling hardlinking for @@ -299,24 +309,36 @@ void LocalStore::optimiseStore(OptimiseStats & stats) { Activity act(*logger, actOptimiseStore); - auto paths = queryAllValidPaths(); - InodeHash inodeHash = loadInodeHash(); + auto paths = [&]() { + Activity act(*logger, lvlTalkative, actUnknown, "querying valid paths"); + return queryAllValidPaths(); + }(); + + InodeHash inodeHash = [&]() { + Activity act(*logger, lvlTalkative, actUnknown, "reading .links directory"); + return loadInodeHash(); + }(); act.progress(0, paths.size()); - uint64_t done = 0; + std::atomic done = 0; + + ThreadPool pool; for (auto & i : paths) { - addTempRoot(i); - if (!isValidPath(i)) - continue; /* path was GC'ed, probably */ - { - Activity act(*logger, lvlTalkative, actUnknown, fmt("optimising path '%s'", printStorePath(i))); - optimisePath_(&act, stats, config->realStoreDir.get() / i.to_string(), inodeHash, NoRepair); - } - done++; - act.progress(done, paths.size()); + pool.enqueue([&, i]() { + addTempRoot(i); + if (!isValidPath(i)) + return; /* path was GC'ed, probably */ + { + Activity act2(*logger, lvlTalkative, actUnknown, fmt("optimising path '%s'", printStorePath(i))); + optimisePath_(&act2, stats, config->realStoreDir.get() / i.to_string(), inodeHash, NoRepair); + } + act.progress(++done, paths.size()); + }); } + + pool.process(); } void LocalStore::optimiseStore() @@ -325,7 +347,7 @@ void LocalStore::optimiseStore() optimiseStore(stats); - printInfo("%s freed by hard-linking %d files", renderSize(stats.bytesFreed), stats.filesLinked); + printInfo("%s freed by hard-linking %d files", renderSize(stats.bytesFreed.load()), stats.filesLinked.load()); } void LocalStore::optimisePath(const std::filesystem::path & path, RepairFlag repair) diff --git a/src/libstore/package.nix b/src/libstore/package.nix index 69f79153ee15..034c06e3824e 100644 --- a/src/libstore/package.nix +++ b/src/libstore/package.nix @@ -4,7 +4,7 @@ mkMesonLibrary, unixtools, - darwin, + apple-sdk, freebsd, nix-util, @@ -16,6 +16,7 @@ nlohmann_json, sqlite, cmake, # for resolving aws-crt-cpp dep + wasmtime, busybox-sandbox-shell ? null, pkgsStatic, @@ -24,7 +25,7 @@ version, - embeddedSandboxShell ? stdenv.hostPlatform.isStatic, + embeddedSandboxShell ? stdenv.hostPlatform.isStatic && !stdenv.hostPlatform.isDarwin, withSandboxShell ? stdenv.hostPlatform.isLinux || stdenv.hostPlatform.isFreeBSD, sandboxShell ? @@ -37,7 +38,9 @@ withAWS ? # Default is this way because there have been issues building this dependency - (lib.meta.availableOn stdenv.hostPlatform aws-c-common), + (lib.meta.availableOn stdenv.hostPlatform aws-c-common) && !stdenv.hostPlatform.isStatic, + + enableWasm ? !stdenv.hostPlatform.isStatic, }: let @@ -45,15 +48,17 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-store"; + pname = "determinate-nix-store"; inherit version; workDir = ./.; fileset = fileset.unions [ ../../nix-meson-build-support ./nix-meson-build-support + # FIXME: get rid of these symlinks. ../../.version ./.version + ../../.version-determinate ./meson.build ./meson.options ./include/nix/store/meson.build @@ -81,7 +86,8 @@ mkMesonLibrary (finalAttrs: { ] ++ lib.optional stdenv.hostPlatform.isLinux libseccomp ++ lib.optional stdenv.hostPlatform.isFreeBSD freebsd.libjail - ++ lib.optional withAWS aws-crt-cpp; + ++ lib.optional withAWS aws-crt-cpp + ++ lib.optional enableWasm wasmtime; propagatedBuildInputs = [ nix-util @@ -92,6 +98,7 @@ mkMesonLibrary (finalAttrs: { (lib.mesonEnable "seccomp-sandboxing" stdenv.hostPlatform.isLinux) (lib.mesonBool "embedded-sandbox-shell" embeddedSandboxShell) (lib.mesonEnable "s3-aws-auth" withAWS) + (lib.mesonEnable "wasm" enableWasm) ] ++ lib.optionals withSandboxShell [ (lib.mesonOption "sandbox-shell" sandboxShell) diff --git a/src/libstore/path-info.cc b/src/libstore/path-info.cc index 9408e9890900..713a33133503 100644 --- a/src/libstore/path-info.cc +++ b/src/libstore/path-info.cc @@ -5,6 +5,7 @@ #include "nix/util/json-utils.hh" #include "nix/util/comparator.hh" #include "nix/util/strings.hh" +#include "nix/util/provenance.hh" namespace nix { @@ -224,6 +225,9 @@ UnkeyedValidPathInfo::toJSON(const StoreDirConfig * store, bool includeImpureInf for (auto & sig : sigs) sigsObj.push_back(sig.to_string()); } + + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) + jsonObject["provenance"] = provenance ? provenance->to_json() : nullptr; } return jsonObject; @@ -299,6 +303,12 @@ UnkeyedValidPathInfo UnkeyedValidPathInfo::fromJSON(const StoreDirConfig * store if (auto * rawSignatures = optionalValueAt(json, "signatures")) res.sigs = *rawSignatures; + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) { + auto prov = json.find("provenance"); + if (prov != json.end() && !prov->second.is_null()) + res.provenance = Provenance::from_json(prov->second); + } + return res; } diff --git a/src/libstore/provenance.cc b/src/libstore/provenance.cc new file mode 100644 index 000000000000..e08614a0e97d --- /dev/null +++ b/src/libstore/provenance.cc @@ -0,0 +1,84 @@ +#include "nix/store/provenance.hh" +#include "nix/util/json-utils.hh" + +#include + +namespace nix { + +static void checkProvenanceTagName(std::string_view name) +{ + static const std::regex tagNameRegex("^[A-Za-z_][A-Za-z0-9_+\\-]*$"); + if (!std::regex_match(name.begin(), name.end(), tagNameRegex)) + throw Error("tag name '%s' is invalid", name); +} + +BuildProvenance::BuildProvenance( + const StorePath & drvPath, + const OutputName & output, + std::optional buildHost, + StringMap tags, + std::string system, + std::shared_ptr next) + : drvPath(drvPath) + , output(output) + , buildHost(std::move(buildHost)) + , tags(std::move(tags)) + , system(std::move(system)) + , next(std::move(next)) +{ + for (const auto & [name, value] : this->tags) + checkProvenanceTagName(name); +} + +nlohmann::json BuildProvenance::to_json() const +{ + return { + {"type", "build"}, + {"drv", drvPath.to_string()}, + {"output", output}, + {"buildHost", buildHost}, + {"system", system}, + {"next", next ? next->to_json() : nlohmann::json(nullptr)}, + {"tags", tags}, + }; +} + +Provenance::Register registerBuildProvenance("build", [](nlohmann::json json) { + auto & obj = getObject(json); + std::shared_ptr next; + if (auto p = optionalValueAt(obj, "next"); p && !p->is_null()) + next = Provenance::from_json(*p); + std::optional buildHost; + if (auto p = optionalValueAt(obj, "buildHost")) + buildHost = p->get>(); + StringMap tags; + if (auto p = optionalValueAt(obj, "tags"); p && !p->is_null()) + tags = p->get(); + auto buildProv = make_ref( + StorePath(getString(valueAt(obj, "drv"))), + getString(valueAt(obj, "output")), + std::move(buildHost), + std::move(tags), + getString(valueAt(obj, "system")), + next); + return buildProv; +}); + +nlohmann::json CopiedProvenance::to_json() const +{ + return { + {"type", "copied"}, + {"from", from}, + {"next", next ? next->to_json() : nlohmann::json(nullptr)}, + }; +} + +Provenance::Register registerCopiedProvenance("copied", [](nlohmann::json json) { + auto & obj = getObject(json); + std::shared_ptr next; + if (auto p = optionalValueAt(obj, "next"); p && !p->is_null()) + next = Provenance::from_json(*p); + return make_ref(getString(valueAt(obj, "from")), next); +}); + +} // namespace nix diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc index f0f672038bb9..579c3551dd88 100644 --- a/src/libstore/remote-store.cc +++ b/src/libstore/remote-store.cc @@ -20,6 +20,8 @@ #include "nix/store/filetransfer.hh" #include "nix/util/signals.hh" #include "nix/util/socket.hh" +#include "nix/util/provenance.hh" + #include #ifndef _WIN32 @@ -93,6 +95,8 @@ void RemoteStore::initConnection(Connection & conn) // which only adds the feature under certain conditions. Adding is easier than removing. auto localVersion = WorkerProto::latest; localVersion.features.insert(std::string{WorkerProto::featureDisableSetOptions}); + if (!experimentalFeatureSettings.isEnabled(Xp::Provenance)) + localVersion.features.erase(std::string(WorkerProto::featureProvenance)); conn.protoVersion = WorkerProto::BasicClientConnection::handshake(conn.to, tee, localVersion); if (conn.protoVersion.number < WorkerProto::minimum.number) @@ -108,6 +112,13 @@ void RemoteStore::initConnection(Connection & conn) throw Error("protocol mismatch, got '%s'", chomp(saved.s)); } + if (conn.protoVersion.features.contains(WorkerProto::featureOpenTelemetry)) { + /* Send our trace context so that the daemon can parent + its spans under ours. */ + conn.to << getTraceparent(logger->getTraceContext(0)); + conn.to.flush(); + } + static_cast(conn) = conn.postHandshake(*this); for (auto & feature : conn.protoVersion.features) @@ -117,7 +128,8 @@ void RemoteStore::initConnection(Connection & conn) if (ex) std::rethrow_exception(ex); } catch (Error & e) { - throw Error("cannot open connection to remote store '%s': %s", config.getHumanReadableURI(), e.what()); + throw Error( + "cannot open connection to remote store '%s': %s", config.getHumanReadableURI(), Uncolored(e.message())); } if (!conn.protoVersion.features.contains(WorkerProto::featureDisableSetOptions)) @@ -126,11 +138,11 @@ void RemoteStore::initConnection(Connection & conn) void RemoteStore::setOptions(Connection & conn) { - conn.to << WorkerProto::Op::SetOptions << settings.keepFailed << settings.getWorkerSettings().keepGoing - << settings.getWorkerSettings().tryFallback << verbosity << settings.getWorkerSettings().maxBuildJobs - << settings.getWorkerSettings().maxSilentTime << true << (settings.verboseBuild ? lvlError : lvlVomit) - << 0 // obsolete log type - << 0 /* obsolete print build trace */ + conn.startOp(WorkerProto::Op::SetOptions); + conn.to << settings.keepFailed << settings.getWorkerSettings().keepGoing << settings.getWorkerSettings().tryFallback + << verbosity << settings.getWorkerSettings().maxBuildJobs << settings.getWorkerSettings().maxSilentTime + << true << (settings.verboseBuild ? lvlError : lvlVomit) << 0 // obsolete log type + << 0 /* obsolete print build trace */ << settings.getLocalSettings().buildCores << settings.getWorkerSettings().useSubstitutes; std::map overrides; @@ -181,7 +193,7 @@ void RemoteStore::setOptions() bool RemoteStore::isValidPathUncached(const StorePath & path) { auto conn(getConnection()); - conn->to << WorkerProto::Op::IsValidPath; + conn->startOp(WorkerProto::Op::IsValidPath); WorkerProto::write(*this, *conn, path); conn.processStderr(); return readInt(conn->from); @@ -196,7 +208,7 @@ StorePathSet RemoteStore::queryValidPaths(const StorePathSet & paths, Substitute StorePathSet RemoteStore::queryAllValidPaths() { auto conn(getConnection()); - conn->to << WorkerProto::Op::QueryAllValidPaths; + conn->startOp(WorkerProto::Op::QueryAllValidPaths); conn.processStderr(); return WorkerProto::Serialise::read(*this, *conn); } @@ -204,7 +216,7 @@ StorePathSet RemoteStore::queryAllValidPaths() StorePathSet RemoteStore::querySubstitutablePaths(const StorePathSet & paths) { auto conn(getConnection()); - conn->to << WorkerProto::Op::QuerySubstitutablePaths; + conn->startOp(WorkerProto::Op::QuerySubstitutablePaths); WorkerProto::write(*this, *conn, paths); conn.processStderr(); return WorkerProto::Serialise::read(*this, *conn); @@ -217,7 +229,7 @@ void RemoteStore::querySubstitutablePathInfos(const StorePathCAMap & pathsMap, S auto conn(getConnection()); - conn->to << WorkerProto::Op::QuerySubstitutablePathInfos; + conn->startOp(WorkerProto::Op::QuerySubstitutablePathInfos); if (conn->protoVersion.number < WorkerProto::Version::Number{1, 22}) { StorePathSet paths; for (auto & path : pathsMap) @@ -253,10 +265,75 @@ void RemoteStore::queryPathInfoUncached( } } +asio::awaitable RemoteStore::queryPathInfos( + const std::set & paths, + fun>>)> callback) +{ + /* Filter out paths that we already have cached. */ + StorePathSet uncached; + { + std::vector>> cached; + for (auto & path : paths) { + if (auto r = queryPathInfoFromClientCache(path)) + cached.emplace_back(path, *r); + else + uncached.insert(path); + } + if (!cached.empty()) + callback(std::move(cached)); + } + + if (uncached.empty()) + co_return; + + { + auto conn(getConnection()); + + if (conn->protoVersion.features.contains(WorkerProto::featureQueryPathInfos)) { + auto cacheResult = [&](const StorePath & path, std::shared_ptr info) { + pathInfoCache->lock()->upsert(path, PathInfoCacheValue{.value = info}); + }; + + conn->startOp(WorkerProto::Op::QueryPathInfos); + WorkerProto::write(*this, *conn, uncached); + conn.processStderr(); + + /* Read the infos for the valid paths. Paths not reported + by the daemon are invalid. */ + auto todo = uncached; + auto n = readNum(conn->from); + std::vector>> results; + for (size_t i = 0; i < n; i++) { + auto info = + std::make_shared(WorkerProto::Serialise::read(*this, *conn)); + if (!todo.erase(info->path)) + throw Error("daemon returned path info for unexpected path '%s'", printStorePath(info->path)); + cacheResult(info->path, info); + results.emplace_back(info->path, info); + } + + for (auto & path : todo) { + cacheResult(path, nullptr); + results.emplace_back(path, nullptr); + } + + callback(std::move(results)); + + co_return; + } + + /* Release the connection to prevent the fallback below from + deadlocking on the connection pool. */ + } + + /* Fallback for daemons that don't support the batched operation. */ + co_await Store::queryPathInfos(uncached, std::move(callback)); +} + void RemoteStore::queryReferrers(const StorePath & path, StorePathSet & referrers) { auto conn(getConnection()); - conn->to << WorkerProto::Op::QueryReferrers; + conn->startOp(WorkerProto::Op::QueryReferrers); WorkerProto::write(*this, *conn, path); conn.processStderr(); for (auto & i : WorkerProto::Serialise::read(*this, *conn)) @@ -266,7 +343,7 @@ void RemoteStore::queryReferrers(const StorePath & path, StorePathSet & referrer StorePathSet RemoteStore::queryValidDerivers(const StorePath & path) { auto conn(getConnection()); - conn->to << WorkerProto::Op::QueryValidDerivers; + conn->startOp(WorkerProto::Op::QueryValidDerivers); WorkerProto::write(*this, *conn, path); conn.processStderr(); return WorkerProto::Serialise::read(*this, *conn); @@ -278,7 +355,7 @@ StorePathSet RemoteStore::queryDerivationOutputs(const StorePath & path) return Store::queryDerivationOutputs(path); } auto conn(getConnection()); - conn->to << WorkerProto::Op::QueryDerivationOutputs; + conn->startOp(WorkerProto::Op::QueryDerivationOutputs); WorkerProto::write(*this, *conn, path); conn.processStderr(); return WorkerProto::Serialise::read(*this, *conn); @@ -290,7 +367,7 @@ RemoteStore::queryPartialDerivationOutputMap(const StorePath & path, Store * eva if (WorkerProto::Version::Number::fromWire(getProtocol()) >= WorkerProto::Version::Number{1, 22}) { if (!evalStore_) { auto conn(getConnection()); - conn->to << WorkerProto::Op::QueryDerivationOutputMap; + conn->startOp(WorkerProto::Op::QueryDerivationOutputMap); WorkerProto::write(*this, *conn, path); conn.processStderr(); return WorkerProto::Serialise>>::read(*this, *conn); @@ -322,7 +399,8 @@ RemoteStore::queryPartialDerivationOutputMap(const StorePath & path, Store * eva std::optional RemoteStore::queryPathFromHashPart(const std::string & hashPart) { auto conn(getConnection()); - conn->to << WorkerProto::Op::QueryPathFromHashPart << hashPart; + conn->startOp(WorkerProto::Op::QueryPathFromHashPart); + conn->to << hashPart; conn.processStderr(); return WorkerProto::Serialise>::read(*this, *conn); } @@ -333,16 +411,21 @@ ref RemoteStore::addCAToStore( ContentAddressMethod caMethod, HashAlgorithm hashAlgo, const StorePathSet & references, - RepairFlag repair) + RepairFlag repair, + std::shared_ptr provenance) { std::optional conn_(getConnection()); auto & conn = *conn_; if (conn->protoVersion >= WorkerProto::Version{.number = {1, 25}}) { - conn->to << WorkerProto::Op::AddToStore << name << caMethod.renderWithAlgo(hashAlgo); + conn->startOp(WorkerProto::Op::AddToStore); + + conn->to << name << caMethod.renderWithAlgo(hashAlgo); WorkerProto::write(*this, *conn, references); conn->to << repair; + if (conn->protoVersion.features.contains(WorkerProto::featureProvenance)) + conn->to << (provenance ? provenance->to_json_str() : ""); // The dump source may invoke the store, so we need to make some room. connections->incCapacity(); @@ -364,7 +447,8 @@ ref RemoteStore::addCAToStore( name, printHashAlgo(hashAlgo)); std::string s = dump.drain(); - conn->to << WorkerProto::Op::AddTextToStore << name << s; + conn->startOp(WorkerProto::Op::AddTextToStore); + conn->to << name << s; WorkerProto::write(*this, *conn, references); conn.processStderr(); break; @@ -374,7 +458,8 @@ ref RemoteStore::addCAToStore( case ContentAddressMethod::Raw::Git: default: { auto fim = caMethod.getFileIngestionMethod(); - conn->to << WorkerProto::Op::AddToStore << name + conn->startOp(WorkerProto::Op::AddToStore); + conn->to << name << ((hashAlgo == HashAlgorithm::SHA256 && fim == FileIngestionMethod::NixArchive) ? 0 : 1) /* backwards compatibility hack */ @@ -420,7 +505,8 @@ StorePath RemoteStore::addToStoreFromDump( ContentAddressMethod hashMethod, HashAlgorithm hashAlgo, const StorePathSet & references, - RepairFlag repair) + RepairFlag repair, + std::shared_ptr provenance) { FileSerialisationMethod fsm; switch (hashMethod.getFileIngestionMethod()) { @@ -439,7 +525,7 @@ StorePath RemoteStore::addToStoreFromDump( } if (fsm != dumpMethod) unsupported("RemoteStore::addToStoreFromDump doesn't support this `dumpMethod` `hashMethod` combination"); - auto storePath = addCAToStore(dump, name, hashMethod, hashAlgo, references, repair)->path; + auto storePath = addCAToStore(dump, name, hashMethod, hashAlgo, references, repair, provenance)->path; invalidatePathInfoCacheFor(storePath); return storePath; } @@ -448,14 +534,17 @@ void RemoteStore::addToStore(const ValidPathInfo & info, Source & source, Repair { auto conn(getConnection()); - conn->to << WorkerProto::Op::AddToStoreNar; + conn->startOp(WorkerProto::Op::AddToStoreNar); WorkerProto::write(*this, *conn, info.path); WorkerProto::write(*this, *conn, info.deriver); conn->to << info.narHash.to_string(HashFormat::Base16, false); WorkerProto::write(*this, *conn, info.references); conn->to << info.registrationTime << info.narSize << info.ultimate; WorkerProto::write(*this, *conn, info.sigs); - conn->to << renderContentAddress(info.ca) << repair << !checkSigs; + conn->to << renderContentAddress(info.ca); + if (conn->protoVersion.features.contains(WorkerProto::featureProvenance)) + conn->to << (info.provenance ? info.provenance->to_json_str() : ""); + conn->to << repair << !checkSigs; if (conn->protoVersion >= WorkerProto::Version{.number = {1, 23}}) { conn.withFramedSink([&](Sink & sink) { copyNAR(source, sink); }); @@ -497,7 +586,9 @@ void RemoteStore::addMultipleToStore( *this, WorkerProto::WriteConn{ .to = sink, - .version = {.number = {.major = 1, .minor = 16}}, + .version = conn->protoVersion.features.contains(WorkerProto::featureVersionedAddToStoreMultiple) + ? conn->protoVersion + : WorkerProto::Version{.number = {.major = 1, .minor = 16}}, }, pathInfo); pathSource->drainInto(sink); @@ -505,14 +596,16 @@ void RemoteStore::addMultipleToStore( } }); - conn->to << WorkerProto::Op::AddMultipleToStore << repair << !checkSigs; + conn->startOp(WorkerProto::Op::AddMultipleToStore); + + conn->to << repair << !checkSigs; conn.withFramedSink([&](Sink & sink) { source->drainInto(sink); }); } void RemoteStore::registerDrvOutput(const Realisation & info) { auto conn(getConnection()); - conn->to << WorkerProto::Op::RegisterDrvOutput; + conn->startOp(WorkerProto::Op::RegisterDrvOutput); WorkerProto::write(*this, *conn, info); conn.processStderr(); } @@ -530,7 +623,7 @@ void RemoteStore::queryRealisationUncached( return callback(nullptr); } - conn->to << WorkerProto::Op::QueryRealisation; + conn->startOp(WorkerProto::Op::QueryRealisation); WorkerProto::write(*this, *conn, id); conn.processStderr(); @@ -571,7 +664,7 @@ void RemoteStore::buildPaths( copyDrvsFromEvalStore(drvPaths, evalStore); auto conn(getConnection()); - conn->to << WorkerProto::Op::BuildPaths; + conn->startOp(WorkerProto::Op::BuildPaths); WorkerProto::write(*this, *conn, drvPaths); conn->to << buildMode; conn.processStderr(); @@ -587,7 +680,7 @@ std::vector RemoteStore::buildPathsWithResults( auto & conn = *conn_; if (conn->protoVersion >= WorkerProto::Version{.number = {1, 34}}) { - conn->to << WorkerProto::Op::BuildPathsWithResults; + conn->startOp(WorkerProto::Op::BuildPathsWithResults); WorkerProto::write(*this, *conn, paths); conn->to << buildMode; conn.processStderr(); @@ -662,22 +755,40 @@ BuildResult RemoteStore::buildDerivation(const StorePath & drvPath, const BasicD void RemoteStore::ensurePath(const StorePath & path) { auto conn(getConnection()); - conn->to << WorkerProto::Op::EnsurePath; + conn->startOp(WorkerProto::Op::EnsurePath); WorkerProto::write(*this, *conn, path); conn.processStderr(); readInt(conn->from); } -void RemoteStore::addTempRoot(const StorePath & path) +void RemoteStore::addTempRoots(const StorePathSet & paths, bool skipIfSlow) { + if (paths.empty()) + return; + auto conn(getConnection()); - conn->addTempRoot(*this, &conn.daemonException, path); + + if (conn->protoVersion.features.contains(WorkerProto::featureAddTempRoots)) { + conn->startOp(WorkerProto::Op::AddTempRoots); + WorkerProto::write(*this, *conn, paths); + conn.processStderr(); + readInt(conn->from); + } else { + if (skipIfSlow) + return; + /* Fallback for daemons that don't support the batched + operation. Note that this is very slow for large sets of + paths on high-latency links, due to a network round-trip per + path. */ + for (auto & path : paths) + conn->addTempRoot(*this, &conn.daemonException, path); + } } Roots RemoteStore::findRoots(bool censor) { auto conn(getConnection()); - conn->to << WorkerProto::Op::FindRoots; + conn->startOp(WorkerProto::Op::FindRoots); conn.processStderr(); size_t count = readNum(conn->from); Roots result; @@ -696,7 +807,7 @@ void RemoteStore::collectGarbage(const GCOptions & options, GCResults & results) conn->protoVersion.features.contains(WorkerProto::featureDeleteDeadSpecificReferrers); if (supportsDeleteSpecificReferrers) { - conn->to << WorkerProto::Op::CollectGarbage; + conn->startOp(WorkerProto::Op::CollectGarbage); WorkerProto::write(*this, *conn, options.action); WorkerProto::write(*this, *conn, options.pathsToDelete); } else { @@ -713,7 +824,7 @@ void RemoteStore::collectGarbage(const GCOptions & options, GCResults & results) [](const GCOptions::WholeStore & _) { return StorePathSet{}; }, }, options.pathsToDelete); - conn->to << WorkerProto::Op::CollectGarbage; + conn->startOp(WorkerProto::Op::CollectGarbage); WorkerProto::write(*this, *conn, options.action); WorkerProto::write(*this, *conn, paths); } @@ -735,7 +846,7 @@ void RemoteStore::collectGarbage(const GCOptions & options, GCResults & results) void RemoteStore::optimiseStore() { auto conn(getConnection()); - conn->to << WorkerProto::Op::OptimiseStore; + conn->startOp(WorkerProto::Op::OptimiseStore); conn.processStderr(); readInt(conn->from); } @@ -743,7 +854,8 @@ void RemoteStore::optimiseStore() bool RemoteStore::verifyStore(bool checkContents, RepairFlag repair) { auto conn(getConnection()); - conn->to << WorkerProto::Op::VerifyStore << checkContents << repair; + conn->startOp(WorkerProto::Op::VerifyStore); + conn->to << checkContents << repair; conn.processStderr(); return readInt(conn->from); } @@ -751,7 +863,7 @@ bool RemoteStore::verifyStore(bool checkContents, RepairFlag repair) void RemoteStore::addSignatures(const StorePath & storePath, const std::set & sigs) { auto conn(getConnection()); - conn->to << WorkerProto::Op::AddSignatures; + conn->startOp(WorkerProto::Op::AddSignatures); WorkerProto::write(*this, *conn, storePath); WorkerProto::write(*this, *conn, sigs); conn.processStderr(); @@ -766,7 +878,7 @@ MissingPaths RemoteStore::queryMissing(const std::vector & targets) // Don't hold the connection handle in the fallback case // to prevent a deadlock. goto fallback; - conn->to << WorkerProto::Op::QueryMissing; + conn->startOp(WorkerProto::Op::QueryMissing); WorkerProto::write(*this, *conn, targets); conn.processStderr(); MissingPaths res; @@ -784,12 +896,23 @@ MissingPaths RemoteStore::queryMissing(const std::vector & targets) void RemoteStore::addBuildLog(const StorePath & drvPath, std::string_view log) { auto conn(getConnection()); - conn->to << WorkerProto::Op::AddBuildLog << drvPath.to_string(); + conn->startOp(WorkerProto::Op::AddBuildLog); + conn->to << drvPath.to_string(); StringSource source(log); conn.withFramedSink([&](Sink & sink) { source.drainInto(sink); }); readInt(conn->from); } +std::vector RemoteStore::queryActiveBuilds() +{ + auto conn(getConnection()); + if (!conn->protoVersion.features.count(WorkerProto::featureQueryActiveBuilds)) + throw Error("remote store does not support querying active builds"); + conn->startOp(WorkerProto::Op::QueryActiveBuilds); + conn.processStderr(); + return nlohmann::json::parse(readString(conn->from)).get>(); +} + std::optional RemoteStore::getVersion() { auto conn(getConnection()); diff --git a/src/libstore/restricted-store.cc b/src/libstore/restricted-store.cc index 1cc0f57af1f1..6ea7752db4f6 100644 --- a/src/libstore/restricted-store.cc +++ b/src/libstore/restricted-store.cc @@ -104,7 +104,8 @@ struct RestrictedStore : public virtual IndirectRootStore, public virtual GcStor ContentAddressMethod hashMethod, HashAlgorithm hashAlgo, const StorePathSet & references, - RepairFlag repair) override; + RepairFlag repair, + std::shared_ptr provenance) override; void narFromPath(const StorePath & path, Sink & sink) override; @@ -129,7 +130,7 @@ struct RestrictedStore : public virtual IndirectRootStore, public virtual GcStor unsupported("buildDerivation"); } - void addTempRoot(const StorePath & path) override {} + void addTempRoots(const StorePathSet & paths, bool skipIfSlow) override {} void addIndirectRoot(const std::filesystem::path & path) override {} @@ -226,9 +227,10 @@ StorePath RestrictedStore::addToStoreFromDump( ContentAddressMethod hashMethod, HashAlgorithm hashAlgo, const StorePathSet & references, - RepairFlag repair) + RepairFlag repair, + std::shared_ptr provenance) { - auto path = next->addToStoreFromDump(dump, name, dumpMethod, hashMethod, hashAlgo, references, repair); + auto path = next->addToStoreFromDump(dump, name, dumpMethod, hashMethod, hashAlgo, references, repair, provenance); goal.addDependency(path); return path; } diff --git a/src/libstore/s3-binary-cache-store.cc b/src/libstore/s3-binary-cache-store.cc index 7910357e0b18..9abc73882c37 100644 --- a/src/libstore/s3-binary-cache-store.cc +++ b/src/libstore/s3-binary-cache-store.cc @@ -376,7 +376,7 @@ void S3BinaryCacheStore::abortMultipartUpload(std::string_view key, std::string_ req.uri = VerbatimURL(url); req.method = HttpMethod::Delete; - fileTransfer->enqueueFileTransfer(req).get(); + (void) fileTransfer->enqueueFileTransfer(req).get(); } catch (...) { ignoreExceptionInDestructor(); } @@ -409,7 +409,7 @@ void S3BinaryCacheStore::completeMultipartUpload( req.data = {payload}; req.mimeType = "text/xml"; - fileTransfer->enqueueFileTransfer(req).get(); + (void) fileTransfer->enqueueFileTransfer(req).get(); debug("S3 multipart upload completed: %d parts uploaded for '%s'", partEtags.size(), key); } diff --git a/src/libstore/sqlite.cc b/src/libstore/sqlite.cc index 5bd3a4975c72..57ea4639dea0 100644 --- a/src/libstore/sqlite.cc +++ b/src/libstore/sqlite.cc @@ -3,6 +3,7 @@ #include "nix/util/util.hh" #include "nix/util/url.hh" #include "nix/util/signals.hh" +#include "nix/util/sync.hh" #ifdef __linux__ # include @@ -10,6 +11,7 @@ #include +#include #include namespace nix { @@ -72,20 +74,36 @@ SQLite::SQLite(const std::filesystem::path & path, Settings && settings) // https://github.com/openzfs/zfs/issues/14290#issuecomment-3074672917. // Remove this workaround when a fix is widely installed, perhaps 2027? Candidate: // https://github.com/search?q=repo%3Aopenzfs%2Fzfs+%22Linux%3A+zfs_putpage%3A+complete+async+page+writeback+immediately%22&type=commits + // + // Note: we only do this the first time that this process opens + // the database. Closing a file descriptor drops all POSIX locks + // that the process holds on that file, including the lock on + // db.sqlite-shm held by SQLite on behalf of other connections to + // the same database. Another process could then conclude that + // it's the only user of the database and truncate + // db.sqlite-shm, causing us to crash with SIGBUS. + // + // We hold the lock on `shmFilesSynced` until the file descriptor + // has been closed, so that a concurrent open of the same database + // in another thread cannot create a SQLite connection (and thus + // acquire POSIX locks on db.sqlite-shm) while we still have the + // file open. #ifdef __linux__ - try { - auto shmFile = path; - shmFile += "-shm"; - AutoCloseFD fd = open(shmFile.string().c_str(), O_RDWR | O_CLOEXEC); - if (fd) { - struct statfs fs; - if (fstatfs(fd.get(), &fs)) - throw SysError("statfs() on %s", PathFmt(shmFile)); - if (fs.f_type == /* ZFS_SUPER_MAGIC */ 801189825 && fdatasync(fd.get()) != 0) - throw SysError("fsync() on %s", PathFmt(shmFile)); + static Sync> shmFilesSynced; + { + auto shmFilesSynced_(shmFilesSynced.lock()); + if (shmFilesSynced_->insert(path).second) { + auto shmFile = path; + shmFile += "-shm"; + AutoCloseFD fd = open(shmFile.string().c_str(), O_RDWR | O_CLOEXEC); + if (fd) { + struct statfs fs; + if (fstatfs(fd.get(), &fs)) + throw SysError("statfs() on %s", PathFmt(shmFile)); + if (fs.f_type == /* ZFS_SUPER_MAGIC */ 801189825 && fdatasync(fd.get()) != 0) + throw SysError("fsync() on %s", PathFmt(shmFile)); + } } - } catch (...) { - throw; } #endif @@ -133,10 +151,13 @@ void SQLite::isCache() void SQLite::exec(const std::string & stmt) { - retrySQLite([&]() { - if (sqlite3_exec(db, stmt.c_str(), 0, 0, 0) != SQLITE_OK) - SQLiteError::throw_(db, "executing SQLite statement '%s'", stmt); - }); + retrySQLite([&]() { execNoRetry(stmt); }); +} + +void SQLite::execNoRetry(const std::string & stmt) +{ + if (sqlite3_exec(db, stmt.c_str(), 0, 0, 0) != SQLITE_OK) + SQLiteError::throw_(db, "executing SQLite statement '%s'", stmt); } uint64_t SQLite::getLastInsertedRowId() @@ -144,6 +165,11 @@ uint64_t SQLite::getLastInsertedRowId() return sqlite3_last_insert_rowid(db); } +void SQLite::setLastInsertedRowId(uint64_t id) +{ + sqlite3_set_last_insert_rowid(db, id); +} + void SQLiteStmt::create(sqlite3 * db, const std::string & sql) { checkInterrupt(); diff --git a/src/libstore/ssh-store.cc b/src/libstore/ssh-store.cc index 4ecf2780e7b2..5303fb82f450 100644 --- a/src/libstore/ssh-store.cc +++ b/src/libstore/ssh-store.cc @@ -61,6 +61,11 @@ struct alignas(8) /* Work around ASAN failures on i686-linux. */ { } + bool includeInProvenance() override + { + return true; + } + // FIXME extend daemon protocol, move implementation to RemoteStore std::optional getBuildLogExact(const StorePath & path) override { @@ -201,7 +206,7 @@ struct MountedSSHStore : virtual SSHStore, virtual LocalFSStore std::filesystem::path addPermRoot(const StorePath & path, const std::filesystem::path & gcRoot) override { auto conn(getConnection()); - conn->to << WorkerProto::Op::AddPermRoot; + conn->startOp(WorkerProto::Op::AddPermRoot); WorkerProto::write(*this, *conn, path); WorkerProto::write(*this, *conn, gcRoot.string()); conn.processStderr(); diff --git a/src/libstore/ssh.cc b/src/libstore/ssh.cc index 3cd3e82a23f4..35692f428719 100644 --- a/src/libstore/ssh.cc +++ b/src/libstore/ssh.cc @@ -92,7 +92,7 @@ SSHMaster::SSHMaster( checkValidAuthority(authority); } -void SSHMaster::addCommonSSHOpts(OsStrings & args) +void SSHMaster::addCommonSSHOpts(OsStrings & args, std::optional socketPath) { auto sshArgs = getNixSshOpts(); args.insert(args.end(), sshArgs.begin(), sshArgs.end()); @@ -116,12 +116,15 @@ void SSHMaster::addCommonSSHOpts(OsStrings & args) // the remote session won't be garbled if the local command is slow. args.push_back(OS_STR("-oPermitLocalCommand=yes")); args.push_back(OS_STR("-oLocalCommand=echo started")); + args.insert(args.end(), {OS_STR("-S"), socketPath ? socketPath->native() : OS_STR("none")}); } -bool SSHMaster::isMasterRunning() +bool SSHMaster::isMasterRunning(std::filesystem::path socketPath) { + assert(useMaster); + OsStrings args = {OS_STR("-O"), OS_STR("check"), string_to_os_string(hostnameAndUser)}; - addCommonSSHOpts(args); + addCommonSSHOpts(args, socketPath); auto res = runProgram(RunOptions{.program = "ssh", .args = std::move(args), .mergeStderrToStdout = true}); return res.first == 0; @@ -153,7 +156,7 @@ std::unique_ptr SSHMaster::startCommand(OsStrings && comm #ifdef _WIN32 // TODO re-enable on Windows, once we can start processes. throw UnimplementedError("cannot yet SSH on windows because spawning processes is not yet implemented"); #else - std::filesystem::path socketPath = startMaster(); + auto socketPath = startMaster(); Pipe in, out; in.create(); @@ -186,12 +189,20 @@ std::unique_ptr SSHMaster::startCommand(OsStrings && comm if (!fakeSSH) { args = {"ssh", hostnameAndUser.c_str(), "-x"}; - addCommonSSHOpts(args); - if (!socketPath.empty()) - args.insert(args.end(), {"-S", socketPath.string()}); + addCommonSSHOpts(args, socketPath); if (verbosity >= lvlChatty) args.push_back("-v"); args.splice(args.end(), std::move(extraSshArgs)); + // Override LocalCommand to no-op on command SSHs; master + // already consumed "started". On fallback, "echo started" + // would leak into the nix protocol stream. #441 + if (useMaster) { + for (auto & arg : args) { + if (arg.starts_with(OS_STR("-oLocalCommand="))) { + arg = OS_STR("-oLocalCommand=true"); + } + } + } args.push_back("--"); } @@ -207,9 +218,9 @@ std::unique_ptr SSHMaster::startCommand(OsStrings && comm in.readSide = INVALID_DESCRIPTOR; out.writeSide = INVALID_DESCRIPTOR; - // Wait for the SSH connection to be established, - // So that we don't overwrite the password prompt with our progress bar. - if (!fakeSSH && !useMaster && !isMasterRunning()) { + // Skip readLine when useMaster: SSH connects via master socket, or + // falls back to direct connection where LocalCommand is now a no-op. + if (!fakeSSH && !useMaster && !(socketPath && isMasterRunning(*socketPath))) { std::string reply; try { reply = readLine(out.readSide.get()); @@ -225,20 +236,32 @@ std::unique_ptr SSHMaster::startCommand(OsStrings && comm conn->out = std::move(out.readSide); conn->in = std::move(in.writeSide); + /* Without real SSH, the child *is* the remote program (e.g. + `nix-daemon --stdio`), rather than an `ssh` client that we can + kill without affecting the remote side. So on teardown, give + it a chance to exit by itself after it sees EOF on its stdin, + e.g. to export its telemetry, instead of killing it right + away. */ + if (fakeSSH) { + conn->sshPid.setKillSignal(0); + conn->sshPid.setKillTimeout(std::chrono::seconds(10)); + } + return conn; #endif } #ifndef _WIN32 // TODO re-enable on Windows, once we can start processes. -std::filesystem::path SSHMaster::startMaster() +std::optional SSHMaster::startMaster() { if (!useMaster) - return {}; + return std::nullopt; auto state(state_.lock()); - if (state->sshMaster != INVALID_DESCRIPTOR) + // Check if the master is still alive before returning the cached socket. + if (state->sshMaster != INVALID_DESCRIPTOR && state->sshMaster.isAlive()) return state->socketPath; state->socketPath = tmpDir->path() / "ssh.sock"; @@ -251,7 +274,7 @@ std::filesystem::path SSHMaster::startMaster() auto suspension = logger->suspend(); - if (isMasterRunning()) + if (isMasterRunning(state->socketPath)) return state->socketPath; state->sshMaster = startProcess( @@ -263,10 +286,10 @@ std::filesystem::path SSHMaster::startMaster() if (dup2(out.writeSide.get(), STDOUT_FILENO) == -1) throw SysError("duping over stdout"); - OsStrings args = {"ssh", hostnameAndUser.c_str(), "-M", "-N", "-S", state->socketPath.string()}; + OsStrings args = {"ssh", hostnameAndUser.c_str(), "-M", "-N", "-oControlPersist=no"}; if (verbosity >= lvlChatty) args.push_back("-v"); - addCommonSSHOpts(args); + addCommonSSHOpts(args, state->socketPath); auto env = createSSHEnv(); nix::execvpe(args.begin()->c_str(), stringsToCharPtrs(args).data(), stringsToCharPtrs(env).data()); diff --git a/src/libstore/store-api.cc b/src/libstore/store-api.cc index fee7c8061efa..b4e5fd11b9ee 100644 --- a/src/libstore/store-api.cc +++ b/src/libstore/store-api.cc @@ -18,6 +18,7 @@ #include "nix/util/signals.hh" #include "nix/util/environment-variables.hh" #include "nix/util/file-system.hh" +#include "nix/store/provenance.hh" #include "store-config-private.hh" @@ -197,9 +198,14 @@ StorePath Store::addToStore( std::optional storePath; auto sink = sourceToSink([&](Source & source) { LengthSource lengthSource(source); - storePath = addToStoreFromDump(lengthSource, name, fsm, method, hashAlgo, references, repair); - if (settings.warnLargePathThreshold && lengthSource.total >= settings.warnLargePathThreshold) - warn("copied large path '%s' to the store (%s)", path, renderSize(lengthSource.total)); + storePath = + addToStoreFromDump(lengthSource, name, fsm, method, hashAlgo, references, repair, path.getProvenance()); + if (settings.warnLargePathThreshold && lengthSource.total >= settings.warnLargePathThreshold) { + static bool failOnLargePath = getEnv("_NIX_TEST_FAIL_ON_LARGE_PATH").value_or("") == "1"; + if (failOnLargePath) + throw Error("doesn't copy large path '%s' to the store (%d)", path, renderSize(lengthSource.total)); + warn("copied large path '%s' to the store (%d)", path, renderSize(lengthSource.total)); + } }); dumpPath(path, *sink, fsm, filter); sink->finish(); @@ -362,6 +368,7 @@ ValidPathInfo Store::addToStoreSlow( }), narHash); info.narSize = narSize; + info.provenance = srcPath.getProvenance(); if (!isValidPath(info.path)) { auto source = sinkToSource([&](Sink & scratchpadSink) { srcPath.dumpPath(scratchpadSink); }); @@ -573,6 +580,23 @@ ref Store::queryPathInfo(const StorePath & storePath) return promise.get_future().get(); } +std::shared_ptr Store::maybeQueryPathInfo(const StorePath & storePath) +{ + std::promise> promise; + + queryPathInfo(storePath, {[&](std::future> result) { + try { + promise.set_value(result.get()); + } catch (InvalidPath &) { + promise.set_value(nullptr); + } catch (...) { + promise.set_exception(std::current_exception()); + } + }}); + + return promise.get_future().get(); +} + static bool goodStorePath(const StorePath & expected, const StorePath & actual) { return expected.hashPart() == actual.hashPart() @@ -651,6 +675,25 @@ void Store::queryPathInfo(const StorePath & storePath, Callback Store::queryPathInfos( + const std::set & paths, + fun>>)> callback) +{ + /* Default implementation: query each path individually, reporting + each result as it arrives. */ + co_await forEachAsync(paths, [&](const StorePath & path) -> asio::awaitable { + std::shared_ptr info; + try { + auto i = co_await callbackToAwaitable>( + [&](Callback> cb) { queryPathInfo(path, std::move(cb)); }); + info = i.get_ptr(); + } catch (InvalidPath &) { + } + std::vector>> result{{path, info}}; + callback(std::move(result)); + }); +} + void Store::queryRealisation( const DrvOutput & id, Callback> callback) noexcept { @@ -900,13 +943,26 @@ makeCopyPathMessage(const StoreConfig & srcCfg, const StoreConfig & dstCfg, std: "copying path '%s' from '%s' to '%s'", storePath, srcCfg.getHumanReadableURI(), dstCfg.getHumanReadableURI()); } -void copyStorePath( +/** + * Wrap upstream provenance in a "copied" provenance record to record + * where the path was copied from. But uninformative origins like + * LocalStore are omitted. + */ +static std::shared_ptr +addCopiedProvenance(std::shared_ptr provenance, Store & srcStore) +{ + if (!srcStore.includeInProvenance()) + return provenance; + return std::make_shared(srcStore.config.getReference().render(false), provenance); +} + +std::shared_ptr copyStorePath( Store & srcStore, Store & dstStore, const StorePath & storePath, RepairFlag repair, CheckSigsFlag checkSigs) { /* Bail out early (before starting a download from srcStore) if dstStore already has this path. */ if (!repair && dstStore.isValidPath(storePath)) - return; + return nullptr; const auto & srcCfg = srcStore.config; const auto & dstCfg = dstStore.config; @@ -919,25 +975,22 @@ void copyStorePath( {storePathS, srcCfg.getHumanReadableURI(), dstCfg.getHumanReadableURI()}); PushActivity pact(act.id); - auto info = srcStore.queryPathInfo(storePath); + auto srcInfo = srcStore.queryPathInfo(storePath); + auto info = std::make_shared(*srcInfo); uint64_t total = 0; // recompute store path on the chance dstStore does it differently if (info->ca && info->references.empty()) { - auto info2 = make_ref(*info); - info2->path = + info->path = dstStore.makeFixedOutputPathFromCA(info->path.name(), info->contentAddressWithReferences().value()); if (dstStore.storeDir == srcStore.storeDir) - assert(info->path == info2->path); - info = info2; + assert(info->path == srcInfo->path); } - if (info->ultimate) { - auto info2 = make_ref(*info); - info2->ultimate = false; - info = info2; - } + info->ultimate = false; + + info->provenance = addCopiedProvenance(info->provenance, srcStore); if (getEnv("_NIX_TEST_CONCURRENT_SUBSTITUTION")) std::this_thread::sleep_for(std::chrono::seconds(1)); @@ -959,6 +1012,8 @@ void copyStorePath( }); dstStore.addToStore(*info, *source, repair, checkSigs); + + return info; } std::map copyPaths( @@ -1011,6 +1066,10 @@ std::map copyPaths( CheckSigsFlag checkSigs, SubstituteFlag substitute) { + /* This is optional to avoid a performance regression when talking to old daemons that don't have the AddTempRoots + * operation. */ + dstStore.addTempRoots(storePaths, true); + auto valid = dstStore.queryValidPaths(storePaths, substitute); StorePathSet missing; @@ -1060,6 +1119,7 @@ std::map copyPaths( ValidPathInfo infoForDst = *info; infoForDst.path = storePathForDst; + infoForDst.provenance = addCopiedProvenance(info->provenance, srcStore); auto source = sinkToSource([&, narSize = info->narSize](Sink & sink) { // We can reasonably assume that the copy will happen whenever we @@ -1242,8 +1302,7 @@ void Store::signPathInfo(ValidPathInfo & info) auto secretKeyFiles = settings.secretKeyFiles; for (auto & secretKeyFile : secretKeyFiles.get()) { - SecretKey secretKey(readFile(secretKeyFile)); - LocalSigner signer(std::move(secretKey)); + LocalSigner signer(SecretKey::parse(readFile(secretKeyFile))); info.sign(*this, signer); } } @@ -1255,8 +1314,7 @@ void Store::signRealisation(Realisation & realisation) auto secretKeyFiles = settings.secretKeyFiles; for (auto & secretKeyFile : secretKeyFiles.get()) { - SecretKey secretKey(readFile(secretKeyFile)); - LocalSigner signer(std::move(secretKey)); + LocalSigner signer(SecretKey::parse(readFile(secretKeyFile))); realisation.sign(realisation.id, signer); } } diff --git a/src/libstore/uds-remote-store.cc b/src/libstore/uds-remote-store.cc index c4137df7c6aa..6133ec7b571f 100644 --- a/src/libstore/uds-remote-store.cc +++ b/src/libstore/uds-remote-store.cc @@ -106,7 +106,8 @@ ref UDSRemoteStore::openConnection() void UDSRemoteStore::addIndirectRoot(const std::filesystem::path & path) { auto conn(getConnection()); - conn->to << WorkerProto::Op::AddIndirectRoot << path.string(); + conn->startOp(WorkerProto::Op::AddIndirectRoot); + conn->to << path.string(); conn.processStderr(); readInt(conn->from); } diff --git a/src/libstore/unix/build/derivation-builder-impl.hh b/src/libstore/unix/build/derivation-builder-impl.hh index cdfe9a901791..f9cd1523176d 100644 --- a/src/libstore/unix/build/derivation-builder-impl.hh +++ b/src/libstore/unix/build/derivation-builder-impl.hh @@ -32,6 +32,11 @@ protected: */ Pid pid; + /** + * Handles to track active builds for `nix ps`. + */ + std::optional activeBuildHandle; + LocalStore & store; const LocalSettings & localSettings = store.config->getLocalSettings(); @@ -201,6 +206,11 @@ protected: return acquireUserLock(settings.nixStateDir, localSettings, 1, false); } + /** + * Construct the `ActiveBuild` object for `ActiveBuildsTracker`. + */ + virtual ActiveBuild getActiveBuild(); + /** * Return the paths that should be made available in the sandbox. * This includes: @@ -251,10 +261,23 @@ protected: return store.toRealPath(store.parseStorePath(p.native())); } + /** + * Open the slave side of the pseudoterminal. + */ + AutoCloseFD openSlaveNoDup(); + /** * Open the slave side of the pseudoterminal and use it as stderr. + * FIXME: This is called in the child. It should really be called in the parent (except for the dup to stderr), + * since if the child fails before opening the slave, the parent will hang forever reading from the master. However, + * opening the slave in the parent causes random failures on macOS. See c536e00c9deeac58bc4b3299dbc702604c32adbe. */ - void openSlave(); + void openSlave() + { + auto builderOut = openSlaveNoDup(); + if (dup2(builderOut.get(), STDERR_FILENO) == -1) + throw SysError("cannot pipe standard error into log file"); + } /** * Called by prepareBuild() to start the child process for the diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 386d3f863c2b..4dc6dc4cbbb8 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -2,6 +2,7 @@ #include "nix/util/file-system-at.hh" #include "nix/util/file-system.hh" #include "nix/store/local-store.hh" +#include "nix/store/active-builds.hh" #include "nix/util/processes.hh" #include "nix/store/builtins.hh" #include "nix/store/path-references.hh" @@ -20,6 +21,9 @@ #include "nix/store/build/derivation-env-desugar.hh" #include "nix/util/terminal.hh" #include "nix/store/filetransfer.hh" +#include "nix/store/provenance.hh" + +#include #include #include @@ -65,6 +69,10 @@ # include "darwin-derivation-builder.hh" #endif +#if NIX_USE_WASMTIME +# include "wasi-derivation-builder.hh" +#endif + #if NIX_WITH_AWS_AUTH # include "nix/store/aws-creds.hh" # include "nix/store/s3-url.hh" @@ -177,6 +185,8 @@ bool DerivationBuilderImpl::killChild() pid.wait(); + activeBuildHandle.reset(); + miscMethods->childTerminated(); } return ret; @@ -211,6 +221,8 @@ SingleDrvOutputs DerivationBuilderImpl::unprepareBuild() root. */ killSandbox(true); + activeBuildHandle.reset(); + /* Terminate the recursive Nix daemon. */ stopDaemon(); @@ -510,17 +522,44 @@ std::optional DerivationBuilderImpl::startBuild() pid.setSeparatePG(true); + /* Make the build visible to `nix ps`. */ + if (auto tracker = dynamic_cast(&store)) + activeBuildHandle.emplace(tracker->buildStarted(getActiveBuild())); + processSandboxSetupMessages(); return builderOut.get(); } +ActiveBuild DerivationBuilderImpl::getActiveBuild() +{ + return { + .nixPid = getpid(), + .clientPid = std::nullopt, // FIXME + .clientUid = std::nullopt, // FIXME + .mainPid = pid, + .mainUser = UserInfo::fromUid(buildUser ? buildUser->getUID() : getuid()), + .startTime = buildResult.startTime, + .derivation = drvPath, + }; +} + PathsInChroot DerivationBuilderImpl::getPathsInSandbox() { /* Allow a user-configurable set of directories from the host file system. */ PathsInChroot pathsInChroot = defaultPathsInChroot; + for (auto & p : pathsInChroot) + if (!p.second.optional +#if HAVE_EMBEDDED_SANDBOX_SHELL + && p.second.source != SANDBOX_SHELL +#endif + && !maybeLstat(p.second.source)) + throw SysError( + "path %s is configured as part of the `sandbox-paths` option, but is inaccessible", + PathFmt(p.second.source)); + if (hasPrefix(store.storeDir, tmpDirInSandbox().native())) { throw Error("`sandbox-build-dir` must not contain the storeDir"); } @@ -561,8 +600,28 @@ PathsInChroot DerivationBuilderImpl::getPathsInSandbox() enum BuildHookState { stBegin, stExtraChrootDirs }; + nlohmann::json drvJson = drv; + + auto [tmpFd, drvJsonPath] = createTempFile("nix-drv-json"); + writeFile(drvJsonPath, drvJson.dump()); + AutoDelete drvJsonFile(drvJsonPath, false); + + auto hookEnv = getEnv(); + static_assert(expectedJsonVersionDerivation == 4); + hookEnv["NIX_DERIVATION_V4"] = drvJsonPath; + + auto [hookStatus, lines] = runProgram( + RunOptions{ + .program = localSettings.preBuildHook.get(), + .lookupPath = false, + .args = getPreBuildHookArgs(), + .environment = std::move(hookEnv), + }); + if (!statusOk(hookStatus)) + throw ExecError( + hookStatus, "pre-build hook '%1%' %2%", localSettings.preBuildHook, statusToString(hookStatus)); + auto state = stBegin; - auto lines = runProgram(localSettings.preBuildHook.get(), false, getPreBuildHookArgs()); auto lastPos = std::string::size_type{0}; for (auto nlPos = lines.find('\n'); nlPos != std::string::npos; nlPos = lines.find('\n', lastPos)) { auto line = lines.substr(lastPos, nlPos - lastPos); @@ -596,7 +655,7 @@ void DerivationBuilderImpl::prepareSandbox() throw Error("feature 'uid-range' is not supported on this platform"); } -void DerivationBuilderImpl::openSlave() +AutoCloseFD DerivationBuilderImpl::openSlaveNoDup() { std::string slaveName = getPtsName(builderOut.get()); @@ -614,8 +673,7 @@ void DerivationBuilderImpl::openSlave() if (tcsetattr(builderOut.get(), TCSANOW, &term)) throw SysError("putting pseudoterminal into raw mode"); - if (dup2(builderOut.get(), STDERR_FILENO) == -1) - throw SysError("cannot pipe standard error into log file"); + return builderOut; } #if NIX_WITH_AWS_AUTH @@ -666,13 +724,13 @@ void DerivationBuilderImpl::processSandboxSetupMessages() try { return readLine(builderOut.get()); } catch (Error & e) { - auto status = pid.wait(); + auto status = pid != -1 ? pid.wait() : 0; e.addTrace( {}, "while waiting for the build environment for '%s' to initialize (%s, previous messages: %s)", store.printStorePath(drvPath), - statusToString(status), - concatStringsSep("|", msgs)); + status ? statusToString(status) : "no status", + concatStringsSep("\n", msgs)); throw; } }(); @@ -1409,10 +1467,8 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() auto tmpOutput = tempDir / "x"; - /* Serialise and create a fresh copy of the output to break - any stale writable file descriptors. Copy through the - serialisation/deserialisation. TODO: Use copyRecursive here and - make use of reflinking. */ + /* Copy files to break stale file descriptors. copyRecursive below will use + reflinking to optimise the copying overhead. */ auto pathAccessor = makeFSSourceAccessor(actualPath); RestoreSink restoreSink{store.config->getLocalSettings().fsyncStorePaths}; restoreSink.dstPath = tmpOutput; @@ -1517,7 +1573,7 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() PathFmt(store.toRealPath(newInfo.path))); deletePath(actualPath); /* Trigger the hash-mismatch error. */ - checkCAFixedOutput(store, drvPath, *output, newInfo); + checkCAFixedOutput(store, drvPath, *output, newInfo, *act); unreachable(); } } @@ -1602,6 +1658,14 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() newInfo.deriver = drvPath; newInfo.ultimate = true; + if (experimentalFeatureSettings.isEnabled(Xp::Provenance)) + newInfo.provenance = std::make_shared( + drvPath, + outputName, + settings.getWorkerSettings().getHostName(), + settings.getWorkerSettings().buildProvenanceTags.get(), + drv.platform, + drvProvenance); store.signPathInfo(newInfo); finish(newInfo.path); @@ -1612,8 +1676,8 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() This is also good so that if a fixed-output produces the wrong path, we still store the result (just don't consider - the derivation sucessful, so if someone fixes the problem by - just changing the wanted hash, the redownload (or whateer + the derivation successful, so if someone fixes the problem by + just changing the wanted hash, the redownload (or whatever possibly quite slow thing it was) doesn't have to be done again. */ if (newInfo.ca) @@ -1628,7 +1692,7 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() /* Apply output checks. This includes checking of the wanted vs got hash of fixed-outputs. */ - checkOutputs(store, drvPath, drv.outputs, drvOptions.outputChecks, infos); + checkOutputs(store, drvPath, drv.outputs, drvOptions.outputChecks, infos, *act); if (buildMode == bmCheck) { return {}; @@ -1777,6 +1841,11 @@ std::unique_ptr makeDerivationBuild useSandbox = params.drv.type().isSandboxed() && !params.drvOptions.noChroot; } +#if NIX_USE_WASMTIME + if (params.drv.platform == "wasm32-wasip1") + return makeWasiDerivationBuilder(store, std::move(miscMethods), std::move(params)); +#endif + const bool isRelocatedStore = store.storeDir != store.config->realStoreDir.get(); if (isRelocatedStore) { diff --git a/src/libstore/unix/build/hook-instance.cc b/src/libstore/unix/build/hook-instance.cc index 64c541e3a9f9..81c19f52ab5f 100644 --- a/src/libstore/unix/build/hook-instance.cc +++ b/src/libstore/unix/build/hook-instance.cc @@ -44,6 +44,12 @@ HookInstance::HookInstance(const Strings & _buildHook) /* Create a pipe to get the output of the builder. */ builderOut.create(); + /* Propagate our trace context to the hook, so that its trace (and, + through it, the remote builder's) is part of ours. Note: this + has to be obtained before forking, since the child gets a + different logger. */ + auto traceContext = logger->getTraceContext(getCurActivity()); + /* Fork the hook. */ pid = startProcess([&]() { if (dup2(fromHook.writeSide.get(), STDERR_FILENO) == -1) @@ -51,6 +57,10 @@ HookInstance::HookInstance(const Strings & _buildHook) commonChildInit(); + for (auto & [name, value] : traceContext) + if (name == "traceparent") + setenv("TRACEPARENT", value.c_str(), 1); + if (chdir("/") == -1) throw SysError("changing into /"); @@ -73,8 +83,11 @@ HookInstance::HookInstance(const Strings & _buildHook) }); /* Give custom build hooks the chance to cleanup. */ - pid.setKillSignal(SIGTERM); - pid.setKillTimeout(500ms); + /* The hook exits by itself when it sees EOF on its stdin, so on + teardown, give it a chance to do so (e.g. to export its + telemetry) instead of signalling it right away. */ + pid.setKillSignal(0); + pid.setKillTimeout(10s); pid.setSeparatePG(true); fromHook.writeSide = -1; @@ -93,7 +106,10 @@ HookInstance::~HookInstance() try { toHook.writeSide = -1; if (pid != -1) { - pid.kill(); + /* Don't allow interrupts, since otherwise `~Pid()` would + call `kill()` a second time while the kill timeout + thread is still running. */ + pid.kill(/*allowInterrupts=*/false); if (onKillChild) onKillChild(); } diff --git a/src/libstore/unix/build/wasi-derivation-builder.cc b/src/libstore/unix/build/wasi-derivation-builder.cc new file mode 100644 index 000000000000..d3968cc2c21c --- /dev/null +++ b/src/libstore/unix/build/wasi-derivation-builder.cc @@ -0,0 +1,96 @@ +#include "store-config-private.hh" + +#if NIX_USE_WASMTIME + +# include "derivation-builder-impl.hh" + +# include + +namespace nix { + +// FIXME: cut&paste +template +T unwrap(wasmtime::Result && res) +{ + if (res) + return res.ok(); + throw Error(res.err().message()); +} + +// FIXME: cut&paste +static std::span string2span(std::string_view s) +{ + return std::span((uint8_t *) s.data(), s.size()); +} + +struct WasiDerivationBuilder : DerivationBuilderImpl +{ + WasiDerivationBuilder( + LocalStore & store, std::shared_ptr miscMethods, DerivationBuilderParams params) + : DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)) + { + experimentalFeatureSettings.require(Xp::WasmDerivations); + } + + void execBuilder(const Strings & args, const Strings & envStrs) override + { + using namespace wasmtime; + + Engine engine; + Linker linker(engine); + unwrap(linker.define_wasi()); + + WasiConfig wasiConfig; + wasiConfig.inherit_stdin(); + wasiConfig.inherit_stdout(); + wasiConfig.inherit_stderr(); + wasiConfig.argv(std::vector(args.begin(), args.end())); + { + std::vector> env2; + for (auto & [k, v] : env) + env2.emplace_back(k, rewriteStrings(v, inputRewrites)); + wasiConfig.env(env2); + } + if (!wasiConfig.preopen_dir( + store.config->realStoreDir.get().string(), + store.storeDir, + /* fs_mutable = */ true)) + throw Error("cannot add store directory to WASI config"); + if (!wasiConfig.preopen_dir( + tmpDir, + tmpDirInSandbox(), + /* fs_mutable = */ true)) + throw Error("cannot add temporary directory to WASI config"); + + auto module = unwrap(Module::compile(engine, string2span(readFile(realPathInHost(drv.builder))))); + wasmtime::Store wasmStore(engine); + unwrap(wasmStore.context().set_wasi(std::move(wasiConfig))); + auto instance = unwrap(linker.instantiate(wasmStore, module)); + + auto startName = "_start"; + auto ext = instance.get(wasmStore, startName); + if (!ext) + throw Error("Wasm module '%s' does not export function '%s'", drv.builder, startName); + auto fun = std::get_if(&*ext); + if (!fun) + throw Error("export '%s' of Wasm module '%s' is not a function", startName, drv.builder); + + unwrap(fun->call(wasmStore.context(), {})); + + _exit(0); + } + + void anchor() override; +}; + +void WasiDerivationBuilder::anchor() {} + +DerivationBuilderUnique makeWasiDerivationBuilder( + LocalStore & store, std::shared_ptr miscMethods, DerivationBuilderParams params) +{ + return DerivationBuilderUnique(new WasiDerivationBuilder(store, std::move(miscMethods), std::move(params))); +} + +} // namespace nix + +#endif // NIX_USE_WASMTIME diff --git a/src/libstore/unix/build/wasi-derivation-builder.hh b/src/libstore/unix/build/wasi-derivation-builder.hh new file mode 100644 index 000000000000..251c397bf99f --- /dev/null +++ b/src/libstore/unix/build/wasi-derivation-builder.hh @@ -0,0 +1,10 @@ +#pragma once + +#include "nix/store/build/derivation-builder.hh" + +namespace nix { + +DerivationBuilderUnique makeWasiDerivationBuilder( + LocalStore & store, std::shared_ptr miscMethods, DerivationBuilderParams params); + +} \ No newline at end of file diff --git a/src/libstore/unix/meson.build b/src/libstore/unix/meson.build index b7a4f566a593..c708529b1157 100644 --- a/src/libstore/unix/meson.build +++ b/src/libstore/unix/meson.build @@ -4,6 +4,7 @@ sources += files( 'build/derivation-builder.cc', 'build/external-derivation-builder.cc', 'build/hook-instance.cc', + 'build/wasi-derivation-builder.cc', 'pathlocks.cc', 'user-lock.cc', ) diff --git a/src/libstore/worker-protocol-connection.cc b/src/libstore/worker-protocol-connection.cc index dcc698293a30..b2999c324640 100644 --- a/src/libstore/worker-protocol-connection.cc +++ b/src/libstore/worker-protocol-connection.cc @@ -1,5 +1,6 @@ #include "nix/store/worker-protocol-connection.hh" #include "nix/store/worker-protocol-impl.hh" +#include "nix/util/logging.hh" #include "nix/store/build-result.hh" #include "nix/store/derivations.hh" @@ -19,10 +20,10 @@ static Logger::Fields readFields(Source & from) Logger::Fields fields; size_t size = readInt(from); for (size_t n = 0; n < size; n++) { - auto type = (decltype(Logger::Field::type)) readInt(from); - if (type == Logger::Field::tInt) + auto type = readInt(from); + if (type == 0) fields.push_back(readNum(from)); - else if (type == Logger::Field::tString) + else if (type == 1) fields.push_back(readString(from)); else throw Error("got unsupported field type %x from Nix daemon", (int) type); @@ -82,11 +83,13 @@ WorkerProto::BasicClientConnection::processStderrReturn(Sink * sink, Source * so auto s = readString(from); auto fields = readFields(from); auto parent = readNum(from); + RemoteLogSource remoteLogSource; logger->startActivity(act, lvl, type, s, fields, parent); } else if (msg == STDERR_STOP_ACTIVITY) { auto act = readNum(from); + RemoteLogSource remoteLogSource; logger->stopActivity(act); } @@ -94,6 +97,7 @@ WorkerProto::BasicClientConnection::processStderrReturn(Sink * sink, Source * so auto act = readNum(from); auto type = (ResultType) readInt(from); auto fields = readFields(from); + RemoteLogSource remoteLogSource; logger->result(act, type, fields); } @@ -161,8 +165,11 @@ WorkerProto::Version WorkerProto::BasicClientConnection::handshake( to.flush(); unsigned int magic = readInt(from); - if (magic != WORKER_MAGIC_2) - throw Error("nix-daemon protocol mismatch from"); + if (magic != WORKER_MAGIC_2) { + if (magic == WORKER_MAGIC_ACCESS_DENIED) + throw Error("access denied by the Nix daemon (check the `allowed-users` setting in the daemon)"); + throw Error("Nix daemon protocol mismatch"); + } auto daemonVersion = WorkerProto::Version::Number::fromWire(readInt(from)); if (daemonVersion.major != WorkerProto::latest.number.major) @@ -230,6 +237,15 @@ WorkerProto::ClientHandshakeInfo WorkerProto::BasicClientConnection::postHandsha return WorkerProto::Serialise::read(store, *this); } +void WorkerProto::BasicClientConnection::startOp(WorkerProto::Op op) +{ + to << op; + if (protoVersion.features.contains(WorkerProto::featureOpenTelemetry)) + /* Send the trace context of the current activity, so that the + daemon can parent its work under it. */ + to << getTraceparent(logger->getTraceContext(getCurActivity())); +} + void WorkerProto::BasicServerConnection::postHandshake(const StoreDirConfig & store, const ClientHandshakeInfo & info) { if (protoVersion >= WorkerProto::Version{.number = {1, 14}} && readInt(from)) { @@ -246,7 +262,8 @@ void WorkerProto::BasicServerConnection::postHandshake(const StoreDirConfig & st std::optional WorkerProto::BasicClientConnection::queryPathInfo( const StoreDirConfig & store, bool * daemonException, const StorePath & path) { - to << WorkerProto::Op::QueryPathInfo << store.printStorePath(path); + startOp(WorkerProto::Op::QueryPathInfo); + to << store.printStorePath(path); try { processStderr(daemonException); } catch (Error & e) { @@ -268,7 +285,7 @@ StorePathSet WorkerProto::BasicClientConnection::queryValidPaths( const StoreDirConfig & store, bool * daemonException, const StorePathSet & paths, SubstituteFlag maybeSubstitute) { assert((protoVersion >= WorkerProto::Version{.number = {1, 12}})); - to << WorkerProto::Op::QueryValidPaths; + startOp(WorkerProto::Op::QueryValidPaths); WorkerProto::write(store, *this, paths); if (protoVersion >= WorkerProto::Version{.number = {1, 27}}) { to << maybeSubstitute; @@ -280,7 +297,8 @@ StorePathSet WorkerProto::BasicClientConnection::queryValidPaths( void WorkerProto::BasicClientConnection::addTempRoot( const StoreDirConfig & store, bool * daemonException, const StorePath & path) { - to << WorkerProto::Op::AddTempRoot << store.printStorePath(path); + startOp(WorkerProto::Op::AddTempRoot); + to << store.printStorePath(path); processStderr(daemonException); readInt(from); } @@ -292,7 +310,8 @@ void WorkerProto::BasicClientConnection::putBuildDerivationRequest( const BasicDerivation & drv, BuildMode buildMode) { - to << WorkerProto::Op::BuildDerivation << store.printStorePath(drvPath); + startOp(WorkerProto::Op::BuildDerivation); + to << store.printStorePath(drvPath); writeDerivation(to, store, drv); to << buildMode; } @@ -306,7 +325,8 @@ WorkerProto::BasicClientConnection::getBuildDerivationResponse(const StoreDirCon void WorkerProto::BasicClientConnection::narFromPath( const StoreDirConfig & store, bool * daemonException, const StorePath & path, fun receiveNar) { - to << WorkerProto::Op::NarFromPath << store.printStorePath(path); + startOp(WorkerProto::Op::NarFromPath); + to << store.printStorePath(path); processStderr(daemonException); receiveNar(from); diff --git a/src/libstore/worker-protocol.cc b/src/libstore/worker-protocol.cc index 37aa4e31bec2..05b941a3e922 100644 --- a/src/libstore/worker-protocol.cc +++ b/src/libstore/worker-protocol.cc @@ -10,6 +10,7 @@ #include "nix/store/path-info.hh" #include "nix/util/json-utils.hh" #include "nix/util/util.hh" +#include "nix/util/provenance.hh" #include #include @@ -29,6 +30,12 @@ const WorkerProto::Version WorkerProto::latest = { WorkerProto::featureRealisationWithPath, }, std::string{WorkerProto::featureDeleteDeadSpecificReferrers}, + std::string{WorkerProto::featureQueryActiveBuilds}, + std::string{WorkerProto::featureProvenance}, + std::string{WorkerProto::featureVersionedAddToStoreMultiple}, + std::string{WorkerProto::featureAddTempRoots}, + std::string{WorkerProto::featureQueryPathInfos}, + std::string{WorkerProto::featureOpenTelemetry}, }, }; @@ -379,6 +386,8 @@ UnkeyedValidPathInfo WorkerProto::Serialise::read(const St info.sigs = WorkerProto::Serialise>::read(store, conn); info.ca = ContentAddress::parseOpt(readString(conn.from)); } + if (conn.version.features.contains(WorkerProto::featureProvenance)) + info.provenance = Provenance::from_json_str_optional(readString(conn.from)); return info; } @@ -394,6 +403,8 @@ void WorkerProto::Serialise::write( WorkerProto::write(store, conn, pathInfo.sigs); conn.to << renderContentAddress(pathInfo.ca); } + if (conn.version.features.contains(WorkerProto::featureProvenance)) + conn.to << (pathInfo.provenance ? pathInfo.provenance->to_json_str() : ""); } WorkerProto::ClientHandshakeInfo diff --git a/src/libutil-c/make.nix b/src/libutil-c/make.nix new file mode 100644 index 000000000000..b52daa84c5f0 --- /dev/null +++ b/src/libutil-c/make.nix @@ -0,0 +1,35 @@ +# libnixutilc, the C API of libnixutil, transcribed from meson.build. +{ + pkgs, + nixMake, + nix-util, +}: + +let + inherit (pkgs) lib; +in + +nixMake.mkComponent { + name = "determinate-nix-util-c"; + libName = "nixutilc"; + + deps = [ nix-util ]; + + root = ./.; + + includeDirs = [ "" ]; + # The headers live in the root, which the default would not export. + publicIncludeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + configHeaders = { + "nix_api_util_config.h" = { + PACKAGE_VERSION = lib.fileContents ../../.version; + }; + }; + + extraCxxFlags = nixMake.weakVtablesFlags; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libutil-c/meson.build b/src/libutil-c/meson.build index 595a51e3cc7e..65ce8aab0367 100644 --- a/src/libutil-c/meson.build +++ b/src/libutil-c/meson.build @@ -53,7 +53,6 @@ subdir('nix-meson-build-support/windows-version') build_both_libraries = get_option('plugin-c-api') library_kwargs = { - 'soversion' : nix_soversion, 'dependencies' : deps_public + deps_private + deps_other, 'include_directories' : include_dirs, 'link_args' : linker_export_flags, diff --git a/src/libutil-c/nix_api_util.cc b/src/libutil-c/nix_api_util.cc index f28a9168e30b..a3fe7614d531 100644 --- a/src/libutil-c/nix_api_util.cc +++ b/src/libutil-c/nix_api_util.cc @@ -1,14 +1,91 @@ #include "nix_api_util.h" #include "nix/util/config-global.hh" #include "nix/util/error.hh" +#include "nix/util/logging.hh" #include "nix_api_util_internal.h" +#include "nix/util/signals.hh" #include "nix/util/util.hh" #include +#include #include #include "nix_api_util_config.h" +/** + * Logger that delegates every supported method to a C callback vtable. + */ +class CallbackLogger : public nix::Logger +{ +public: + nix_logger vtable; + void * userdata; + + CallbackLogger(const nix_logger & vtable, void * userdata) + : vtable(vtable) + , userdata(userdata) + { + } + + ~CallbackLogger() override + { + if (vtable.destroy) + vtable.destroy(userdata); + } + + void log(nix::Verbosity lvl, std::string_view s) noexcept override + { + if (!vtable.log) + return; + std::string buf(s); + vtable.log(userdata, static_cast(lvl), buf.c_str()); + } + + void logEI(const nix::ErrorInfo & ei) noexcept override + { + if (!vtable.log) + return; + std::ostringstream oss; + nix::showErrorInfo(oss, ei, nix::loggerSettings.showTrace.get()); + auto formatted = oss.str(); + vtable.log(userdata, static_cast(ei.level), formatted.c_str()); + } + + void startActivity( + nix::ActivityId act, + nix::Verbosity lvl, + nix::ActivityType type, + const std::string & s, + const Fields & /*fields*/, + nix::ActivityId parent) noexcept override + { + if (!vtable.start_activity) + return; + vtable.start_activity( + userdata, act, static_cast(lvl), static_cast(type), s.c_str(), parent); + } + + void stopActivity(nix::ActivityId act) noexcept override + { + if (vtable.stop_activity) + vtable.stop_activity(userdata, act); + } + + void result(nix::ActivityId act, nix::ResultType type, const Fields & fields) noexcept override + { + if (!vtable.result_string) + return; + if (fields.empty() || !std::get_if(&fields[0].raw)) + return; + vtable.result_string( + userdata, act, static_cast(type), std::get(fields[0].raw).c_str()); + } + + virtual void anchor(); +}; + +void CallbackLogger::anchor() {} + extern "C" { nix_c_context * nix_c_context_create() @@ -111,6 +188,9 @@ nix_err nix_libutil_init(nix_c_context * context) context->last_err_code = NIX_OK; try { nix::initLibUtil(); +#ifndef _WIN32 + nix::unix::startSignalHandlerThread(); +#endif return NIX_OK; } NIXC_CATCH_ERRS @@ -175,4 +255,16 @@ nix_err nix_set_verbosity(nix_c_context * context, nix_verbosity level) NIXC_CATCH_ERRS } +nix_err nix_set_logger(nix_c_context * context, const nix_logger * vtable, void * userdata) +{ + if (context) + context->last_err_code = NIX_OK; + if (vtable == nullptr) + return nix_set_err_msg(context, NIX_ERR_UNKNOWN, "logger vtable is null"); + try { + nix::logger = new CallbackLogger(*vtable, userdata); + } + NIXC_CATCH_ERRS +} + } // extern "C" diff --git a/src/libutil-c/nix_api_util.h b/src/libutil-c/nix_api_util.h index b48d9166d4be..1e0abf817020 100644 --- a/src/libutil-c/nix_api_util.h +++ b/src/libutil-c/nix_api_util.h @@ -14,6 +14,8 @@ * Also contains error handling utilities */ +#include + #ifdef __cplusplus extern "C" { #endif @@ -363,6 +365,169 @@ void nix_clear_err(nix_c_context * context); */ nix_err nix_set_verbosity(nix_c_context * context, nix_verbosity level); +/** + * @} + */ + +/** @defgroup logger Logger + * @brief Capture Nix log output via C callbacks + * + * The functions in this section let an embedder replace Nix's global + * logger with one driven by user-supplied callbacks. This is the + * intended way to surface `builtins.trace` calls, builder output, + * warnings, and other diagnostic messages produced by libnixutil, + * libnixstore and libnixexpr to a host language. + * @{ + */ + +/** + * @brief Activity identifier. + * + * `0` means "no activity" (used as the parent of top-level activities). + */ +typedef uint64_t nix_activity_id; + +/** + * @brief Activity type. + * + * @note Must be kept in sync with `nix::ActivityType`. + */ +enum nix_activity_type { + NIX_ACTIVITY_TYPE_UNKNOWN = 0, + NIX_ACTIVITY_TYPE_COPY_PATH = 100, + NIX_ACTIVITY_TYPE_FILE_TRANSFER = 101, + NIX_ACTIVITY_TYPE_REALISE = 102, + NIX_ACTIVITY_TYPE_COPY_PATHS = 103, + NIX_ACTIVITY_TYPE_BUILDS = 104, + NIX_ACTIVITY_TYPE_BUILD = 105, + NIX_ACTIVITY_TYPE_OPTIMISE_STORE = 106, + NIX_ACTIVITY_TYPE_VERIFY_PATHS = 107, + NIX_ACTIVITY_TYPE_SUBSTITUTE = 108, + NIX_ACTIVITY_TYPE_QUERY_PATH_INFO = 109, + NIX_ACTIVITY_TYPE_POST_BUILD_HOOK = 110, + NIX_ACTIVITY_TYPE_BUILD_WAITING = 111, + NIX_ACTIVITY_TYPE_FETCH_TREE = 112, +}; +typedef enum nix_activity_type nix_activity_type; + +/** + * @brief Activity result type. + * + * @note Must be kept in sync with `nix::ResultType`. + */ +enum nix_result_type { + NIX_RESULT_TYPE_FILE_LINKED = 100, + NIX_RESULT_TYPE_BUILD_LOG_LINE = 101, + NIX_RESULT_TYPE_UNTRUSTED_PATH = 102, + NIX_RESULT_TYPE_CORRUPTED_PATH = 103, + NIX_RESULT_TYPE_SET_PHASE = 104, + NIX_RESULT_TYPE_PROGRESS = 105, + NIX_RESULT_TYPE_SET_EXPECTED = 106, + NIX_RESULT_TYPE_POST_BUILD_LOG_LINE = 107, + NIX_RESULT_TYPE_FETCH_STATUS = 108, + NIX_RESULT_TYPE_HASH_MISMATCH = 109, + NIX_RESULT_TYPE_BUILD_RESULT = 110, +}; +typedef enum nix_result_type nix_result_type; + +/** + * @brief Vtable of callbacks for a custom logger. + * + * Any field may be NULL; missing callbacks are treated as no-ops. + * + * @warning Callbacks may be invoked concurrently from multiple threads + * (notably during parallel builds). Implementations are responsible + * for their own synchronization. + * + * @warning Callbacks must not throw exceptions across the C boundary. + */ +typedef struct nix_logger +{ + /** + * @brief Ordinary log message. + * + * Receives `builtins.trace` output, warnings, errors (after + * formatting), and any message produced through the `printError` / + * `printInfo` / `debug` / etc. macros in the C++ API. + * + * @param[in] userdata The user-supplied opaque pointer. + * @param[in] level Verbosity level of the message. + * @param[in] msg NUL-terminated message body. Borrowed; must not be freed. + */ + void (*log)(void * userdata, nix_verbosity level, const char * msg); + + /** + * @brief An activity (e.g. a build, a substitution) has started. + * + * @param[in] userdata The user-supplied opaque pointer. + * @param[in] activity_id Unique identifier for the new activity. + * @param[in] level Verbosity level associated with this activity. + * @param[in] type Activity type. See ::nix_activity_type. + * @param[in] s NUL-terminated description (may be empty, never NULL). Borrowed. + * @param[in] parent_id ID of the parent activity, or `0` if none. + */ + void (*start_activity)( + void * userdata, + nix_activity_id activity_id, + nix_verbosity level, + nix_activity_type type, + const char * s, + nix_activity_id parent_id); + + /** + * @brief An activity has stopped. + */ + void (*stop_activity)(void * userdata, nix_activity_id activity_id); + + /** + * @brief An activity reported a result whose first field is a string. + * + * Covers in particular: + * - ::NIX_RESULT_TYPE_BUILD_LOG_LINE — a line of builder output + * - ::NIX_RESULT_TYPE_POST_BUILD_LOG_LINE — a line of post-build hook output + * - ::NIX_RESULT_TYPE_SET_PHASE — current build phase + * + * Result types whose first field is not a string (such as + * ::NIX_RESULT_TYPE_PROGRESS) are not delivered through this + * callback. + * + * @param[in] userdata The user-supplied opaque pointer. + * @param[in] activity_id The activity reporting this result. + * @param[in] type The result type. + * @param[in] msg The string field, NUL-terminated. Borrowed. + */ + void (*result_string)(void * userdata, nix_activity_id activity_id, nix_result_type type, const char * msg); + + /** + * @brief Called when the logger is destroyed. + * + * Invoked when the logger is replaced by a subsequent call to + * ::nix_set_logger, or when Nix's global logger is torn + * down at program exit. Use this to free resources owned by + * `userdata`. + */ + void (*destroy)(void * userdata); +} nix_logger; + +/** + * @brief Replace the global Nix logger with one driven by C callbacks. + * + * After this call, log messages, activities and supported results + * produced anywhere in Nix are routed to the supplied callbacks. + * + * The vtable is copied; only the function pointers it contains need to + * remain valid for the lifetime of the logger. `userdata` is borrowed + * and passed unmodified to every callback. When the logger is later + * destroyed (by another call to this function, or at program shutdown), + * `vtable->destroy(userdata)` is invoked if non-NULL. + * + * @param[out] context Optional, stores error information. + * @param[in] vtable Required, callback vtable. + * @param[in] userdata Optional opaque pointer passed to every callback. + * @return NIX_OK on success, an error code otherwise. + */ +nix_err nix_set_logger(nix_c_context * context, const nix_logger * vtable, void * userdata); + /** * @} */ diff --git a/src/libutil-c/package.nix b/src/libutil-c/package.nix index 019fb85d6c2b..d56b1adb01fc 100644 --- a/src/libutil-c/package.nix +++ b/src/libutil-c/package.nix @@ -15,7 +15,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-util-c"; + pname = "determinate-nix-util-c"; inherit version; workDir = ./.; diff --git a/src/libutil-test-support/make.nix b/src/libutil-test-support/make.nix new file mode 100644 index 000000000000..1fb6d3ed8b0d --- /dev/null +++ b/src/libutil-test-support/make.nix @@ -0,0 +1,25 @@ +# libnix-util-test-support, helpers for the unit tests of libnixutil and +# the components built on it, transcribed from meson.build. +{ + nixMake, + nix-util, + nix-util-c, +}: + +nixMake.mkComponent { + name = "determinate-nix-util-test-support"; + libName = "nix-util-test-support"; + + deps = [ + nix-util + nix-util-c + ]; + + root = ./.; + + includeDirs = [ "include" ]; + + files = nixMake.commonSupportFiles; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libutil-test-support/meson.build b/src/libutil-test-support/meson.build index 7299cd65c328..390493c2f5de 100644 --- a/src/libutil-test-support/meson.build +++ b/src/libutil-test-support/meson.build @@ -41,13 +41,12 @@ subdir('nix-meson-build-support/windows-version') this_library = library( 'nix-util-test-support', sources, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, # TODO: Remove `-lrapidcheck` when https://github.com/emil-e/rapidcheck/pull/326 # is available. See also ../libutil/build.meson link_args : linker_export_flags + [ '-lrapidcheck' ], - prelink : true, # For C++ static initializers + prelink : prelink, # For C++ static initializers install : true, ) diff --git a/src/libutil-test-support/package.nix b/src/libutil-test-support/package.nix index f8e92c271137..40ff65d61357 100644 --- a/src/libutil-test-support/package.nix +++ b/src/libutil-test-support/package.nix @@ -17,7 +17,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-util-test-support"; + pname = "determinate-nix-util-test-support"; inherit version; workDir = ./.; diff --git a/src/libutil-tests/compression.cc b/src/libutil-tests/compression.cc index 7978a008fe65..60d2cd29ed35 100644 --- a/src/libutil-tests/compression.cc +++ b/src/libutil-tests/compression.cc @@ -29,8 +29,9 @@ TEST_P(CompressionDecompressionTest, roundtrip) TEST_P(CompressionDecompressionTest, empty) { auto compressed = compress(GetParam(), ""); - if (GetParam() != CompressionAlgo::none) + if (GetParam() != CompressionAlgo::none) { ASSERT_FALSE(compressed.empty()); + } auto o = decompress(GetParam(), compressed); ASSERT_EQ(o, ""); } diff --git a/src/libutil-tests/config.cc b/src/libutil-tests/config.cc index b9eec0c4175f..d512a2460c4c 100644 --- a/src/libutil-tests/config.cc +++ b/src/libutil-tests/config.cc @@ -216,7 +216,7 @@ TEST(Config, toJSONOnNonEmptyConfigWithExperimentalSetting) "description", {}, true, - Xp::Flakes, + Xp::CaDerivations, }; setting.assign("value"); @@ -229,7 +229,7 @@ TEST(Config, toJSONOnNonEmptyConfigWithExperimentalSetting) "description": "description\n", "documentDefault": true, "value": "value", - "experimentalFeature": "flakes" + "experimentalFeature": "ca-derivations" } })#"_json); } diff --git a/src/libutil-tests/cxa-throw.cc b/src/libutil-tests/cxa-throw.cc new file mode 100644 index 000000000000..db002bc2172f --- /dev/null +++ b/src/libutil-tests/cxa-throw.cc @@ -0,0 +1,35 @@ +#include + +#include +#include + +TEST(CxaThrow, catchesLogicErrorFromStdlib) +{ + const char * volatile p = nullptr; + ASSERT_DEATH({ std::string s(p); }, ""); +} + +TEST(CxaThrow, catchesLogicError) +{ + ASSERT_DEATH({ throw std::logic_error("test"); }, ""); +} + +TEST(CxaThrow, catchesOutOfRange) +{ + ASSERT_DEATH({ throw std::out_of_range("test"); }, ""); +} + +TEST(CxaThrow, catchesInvalidArgument) +{ + ASSERT_DEATH({ throw std::invalid_argument("test"); }, ""); +} + +TEST(CxaThrow, catchesDomainError) +{ + ASSERT_DEATH({ throw std::domain_error("test"); }, ""); +} + +TEST(CxaThrow, catchesLengthError) +{ + ASSERT_DEATH({ throw std::length_error("test"); }, ""); +} diff --git a/src/libutil-tests/data/local-keys/mldsa44-message.hex b/src/libutil-tests/data/local-keys/mldsa44-message.hex new file mode 100644 index 000000000000..318cdb0f5b65 --- /dev/null +++ b/src/libutil-tests/data/local-keys/mldsa44-message.hex @@ -0,0 +1 @@ +3B8513E51D75912EA8311333F6E7D16EEBF43515FDDD417ACDA0D5DD97C2E9711BD9A8B92AC448B9BF1B37DBC74636870213E004DDE33DE43E3EA9652E504AE00CDB792D6FD381CCF913F93BC280B87706042EE2A2F64C9506178DAB9041CCB857F95B7FBD3D8C30F582FA2A7944FCCBE93AA7FB09A1D9685A376AFB04764FF43D727FBE4BA2B1AEFF37847C5FA6CD71139122DF360A2BD6B8A7216FEB8B6879FE5AD146B50F5C0C19DE06357892107CA755876CF04D4E0EB934CDA8AECB8C066CFC0E6F2374C79AFD122D8419E40A854FCF3B7792D5760CC9F4BA7CA95A1F47621F1983175F4CE2CD1941665A170373B07702CD12EE26867F26CF6F5694B33F79889C1C58BBFD529D4E66B85BF633986311105D9999CFFF3C723F15B2E93033D19D5D8DB06685C1D49C66E2EE62F8BE3B815886C884DCD6154D83DD7DD1EF0430541A54B56E1ABC98D9E1766FE54CC050BCE7A2ED6088ECF112FCCB221D9AB613D6C73A22487B222446E75071CEE91219A6FB188803DAF88F3D9C30903E028029524459F766664BD56FA5C859172A8E438D0A249DC409582FBFA189945191C90E40689B5C2C9F24CA99DD915B0AB730C73182220810C191EDA185968A31E5B20100D21048F1BFC5B012355E56F7F9B4552BBA790393C9E051B08C1CF14F3700F8338854BE0D5B81AB85B6D1F112427ABD76E6C131DEE68A55911DA4726B4B66E426719871B72B3CECEC57EA1D1E726F54FE873CB51B2669CCFA0CEA49486D0370D3F06381977845B29BDD5847B284617B67ECE06EBB39D58AF22FEAFCDDD7CA449F9CA180EDD641DFF5F7EEF1B15CFCE409C82112B2A30C74715157D8583BAA8ACE09A4AE696B4388FF9311C7AE46DB4A0842388861984E444B0BEB315E407207BF9C378CA60427415082AB01E757805D83B619CA42692606964AB290AC930BAA94F81F2EA35E00210D7AF1889FE58FFD9B98C569EB0600DECBDBEBA62534BC410B4AE676551BF5481F04D14247F2E282F1D8C568D516F57A6F744A98A71F26DEBD387A816880D561009A7FE1816F53157493C64B13797A86CB4EF72B7DF5E39FE652B29A075615B723D8E24F9CC1C9669FC78E8BBF0C3070D382D77E37F939E3EA1CA24BD6F2DC031E855744EA4D08A3CC2E24B7D9CEBC899D898DE45A343D7B1CCD9DA4649695C47E68597B95CEF9616631B2757456B088C107E16BB75B6B7801ED12E4AF6ED57FA4990D89567D907EF5453FC5AC3E692FB14C133A15395D8CF1AFF9264DEEA6A5F75FB3DD8561D4D542F7A515E75D27FA1460D35514CDDE577793EB2B53D3769A5A2723D4397D8F84A22707F491D51FDCDB4CF9C38900CC1B33ED5A15E677FC49EB37EA073A03470D235D56E0E81245D2A8D1A24F1C61D2442E5DD7808AB3E1F808EF06D6A7C8AD7D9C16255C45FB780B6EE5C8C76F0539F20E91D0E880E42449B99CDB611EB3B8709911555653E0D6FE455865AEFC9B94E451248A5F39478FF16C783C1CF9BE63D394C46304E98A1698F1092F0FD690C1F6B8D51F82864C35FC163FE16A78C80874683B3FBE914DBA70A610AC6E43A5314DE3B5E40233CC0ED482F451A4FBBDA99D7872B507DEA5AB9DD7524E9521B8F067E3DC28226580E71D421E633CEA92C3F5A6CD27330D142A0BD84A821AEA50F2E6313C36BFB34F6A9FA6C31B08B183E75A6D38E6139E21A3199BDB72D0208DE15D2714C0D1EC312D8CE0C555B12DC5E74A1BA2A8CBAE19FAF20D23F0E0DC4B65FC7449CE98CE02FD1603942D9913B13C8BBC4E1C858A55A7F5AD8ED4EA4731210E5E69636A51A39837283221550FA780E20D3712FD59C365D8B3203C6639254824EA3D40949449C2CE4B7649F02E5A16E9F50E0F8F560B4B9FFEF7B41472E4AD71224FB0259175F01CC76809A56364CF6734AE4AE7976B749B62D20A84668A37541BAD54FEC24A973FF5A6FBCAA6AE71097C1A9EA938C81E774D737028B5E6C3C233AF53D56D4C82D35783B0B5E2C10ED3FAC06B294C6A046E82C14A441EEFC3A65CDD8CB5816CC9E743EA133D37A13897B88E17798C1F9FBF65D4413734C6BD6BC1E7571B59C6621239F749EB2A6943C4F87E0BDB5A848DB3F8E57BE47E03DF8DD63EA77A653116C36A72FE1AD397A5E0B9C03A135394A96862BC4C7089CD8438111D83CF4AFC8433915E7003EDF070455505B1023BF0AA590DF177EA65A822DA76EB0A44DCFBE66AC37604FEEDB65AD9A7B118ACA8FA231C615FCA306647A9955D5DBA47CA17F66EB259FD5F0D0CB9D822DE320E3E7F383033D45CD36998C3C0D5110AA68E7DB8D9C0562CE02B57F1BD3333C92A087FA7680449C482F1B7C4DE3A28B6FFC9EC0BEB093A8EE5E83C7AF8574D8B2D76FEE88C10D4939A69C1144F3B141DA64F480ABE613F29F10CB83C97E42D43050700F257CE0547D4AA6C2B322C9A9D24D45F269B52922D6955B79258A93E0794F340EAD938F10575E45D508EA97EA89E10E5D5ECC23746F1DA2B48D47B2C32FBFBD5B1D439AE0AA94BE9D3998307949E8C0D84DB390369A4152388439A515EB7F6D3963D1C34AC95C10FEF2CDC5B14626EE325DAE329CF9212CD09E544CBB83B15B0F4976A0381B2E8680409967C721F08CD0E538D7467EB6B01419EE694FEC61886A94F7A1C2F7D36B34E4B143B93546CF0794391D0C1BFFF6EB49A80B86536B419442AD0DF0F8110A0234E346759209BC1574BCBB3DF5CB07A2F899D401D5A0183B36E5B2F15C33C44F7712C713A2EF852DA9D1266BA67879F59E6A24CC5071150C06E965272765F69B2851B67FC0960D1D663482FABC2F149C0CEA5742DEFC912D8297B9B00CEB522CDBE29ABB1486DFE5BD03B8E1AEA793C09962028982C27DB31CDAF6A08BCA211F2621A25FEFBE0E1F63EA155CBFADF88938457A3E0A785E7E327A79229643DFC250689F9730DC0603A01D01E140B03D85FD9CE45B8C8187B5C8D32BF5C1AE3C75CB560CFD23E6162CA0F6BF922E17515E703AC0C2B99B286D65B38D46F8B70A5FBF3FF8920EBDB0BE5FADF6A4EEED69462A14ACCBFDEBA5B3F80BAAD4A475CA97AB9E8F975D7B632CF6CEBE6F4F1079009821BCC35EA5FE4EC7560D71DAB77D0E0D45B557FDF177E6D80B71FC2A7D0A057D859C2183D7ED60F68D98D5524B2878BAD8DC5915F6862443309B3C16F4CC602B6FBD032037E6E99EDB5E8425CB27733788E0E6DD25653A88BC596AC17C3C4501BE9079CE5FF19BCFB98E7B311EE0ACAB6CB84349CE2EBBE98B5321D3A6932D568FA30667C2DFEFCC2AB4FBE14EC5EBDE2D1154A982D4A4C5084CAE3FCDC684B56FC9F70ABED9BCB151D61E72E3B4263992EF9A3B489D2086ADC5418F9131923937EF9574FF8ECDC74CAD36A35B528A49CC4F716FFFBC95054EF6898C91096E887969D56E0DF279145F99E7ED926F7D3741E2890FCA1BA31D4DA37778A31D51348AC912D889CC3C554F4B7BA2C9E2A8B67F3C99F23DF9519C326F3A3A57C85FCB3CB1C04B25FC63DBE4C6F92BA7D57C87DDCF30F2E89DE65B1448C4FDA8112C38672378466A8685A03F7280244475D4F75CD24B73F0B6FEB4F82AC1BB9C935E64EB3D917CDCD96A194331B9D742EE85D5F4619106761775F8A0551FC4AFD8E1F959C302CA4BC958FDE8C18B76BF9F1A07FF61848E8CB837FDF7ED4635C742EADEA879765CF6754788E02DBEBE057DD5F69AD7178AA78A62D02FF93F9342492BA77F90E48585A4EF06EC3C08FF8867F73770987EE2EA5A7F8CFF0BF1D58C33B4E91D4715FCB5568F19CA01E5271751EC21DECBC7072238FE959D99DC14BEB918C15310416489C60347AFA12BE8F00D26DB20B08E950518096EBB9FC2B89E3EBAE5EAD5343E3D0CD26D50DC2F3BA7BA3560427A4379AEF3A6875E7E715C335413CEDCC393257D7945642A3D88ABAF57ED2CFC061CB1FD97B97DFAA1A77270FC9AE977CE8384D721666F2C41015871A138F5F0E258CE4D8B9D50487A46E3D861B52B860A4CD84DDE1F54D45A4D8AE7F1F841857FE4262810E3B496D152E6763AB969DBE94A69E2BE65D50880EF31403421AB60CD1E91DB8114D0643F86B3B317D5B38B8DBF19AD591EA401BFA73CF157F42D85160E4BFE44F79B02E7B301527794AFCDD0894451B6B16F190CE6C31C27C82730E93B57C7939414D210D05EB827A0E882C364B97F2506B06DC5EF8269C9ABE3467B29036027632299F86968771F24F89F409CC80B0E9269EC9B7F2AB62E1D3B79477DED2040391F8B6FC8018EF35BDCF095B07705253AC46859539936FE925209F85BC2FC259E5FB791ED35D2FAB34034B62948701723D51A7A02BEAFF0B178A59B0C3233908319EE309E390A553E6B07F399F86737ED162A9F638F3F7C9055CAF78B0165BB33CEAE80806D3D5E747DF9422B30467C60309C71FD775F219A6E9DCB877EC327BD3333D064C77889881816B2A0D9EB2E2142765E3912EE77AF0347F8E6C5768B30AA1C166E67CED69ED594EBFEFE34E2678490FE0A03D1C949924833AE076F1CF405B0E5BC42B2B24070620AEC3819E746F60045E27A90F18FD859111A1F56F0E0711408E57BD877CA73689AF551A1A2B36242F5084F9E181EA65194CA8D25B91B676C8DDADD4989EB18957082AF97E16D239E4FC862585D0EFED63F05B3906FE3FB4A9D331EC651A21B57609CAD16E4D557585AED19B455F4AF9CA12EFBA53289321D99947D38C58E8B597CC85CFCABA0052AAAF619920194DF0A7BD444ECFB981FBD8948B6FA00024E747F1B4934FF1B70F60BE73D5246BB66508E7D4B03F30A74C5FD4C081CA61111E848505D77E56D77B6E847E7605E1D0991655D0B11EA96D4DC832E9276D2594DE3579B6F405CCBAD998F25AAC7D6F7E49FB2EA65E4046973B7A0BA33830706EC831FBE6842ACCC8BDB311F8B6E4E7586C6A1997DBEDC5DA1CFDE0C0CD896EEA90A93E94D5DCE962920A94241A62951B3AD92CA61A001707C6D0C26D8C5ED251D72B5B4C00223199113497F3F58BABDE6B6D1FA0DB4516EBF53FB60675FA972AFD8AB0C5410089597908EC31EC0D467CD737672489D0DB88F19C4D8A7CAAA1FD6584013C088216D291A870B7CE078842C932B116F8FCAFFFC6B84857D31B02EABFBA38755010AF884E602038CFA24039F420A57E9461B89EF9514CB80A6433B2E140590F417C636FB14578B09AA950127052BC06CDE42A19C97AF1B37F61CDC610BBE220B4B18CB0277363BE7A8218EB05CC19D82B446DC3600EC0CE2D8BE21404F232B39556DE8A2BE74B3BA88B278A481F3F6276D2B8436E7777933CB89872AE0B4AA10C600A442E8FABE488DC3BD8A1FC86C5FD91204E933F149F9B0C19D6CCD317E363C887CEE43DF396214C838217129F558E6445C5EFAD69942FEF812126CC300B5E52EA4152152A43570FFCFDD4438F1C2E3FA3FCDE76364D09DAE30F804AA61E4958C6A72ECE953261243C25FB736E13FF026CCCEA7A6425F9190FE7478C689335F77042578F1F9A0C8E77429B2F6619C12B9CE9DDDB16BA9B1D364D68A1C0D867D755618C303E9BEB49A80034FF432F6777890765D06A8A59FD73952A0AE3B48EAB42DE5FCD914B3301B948656FB70EB360584C554F84A9B6B52F1D7B719DA4920D70EC0DF790BFC4DE57EA79203F891BEF07C7AF76C2FF974FD5F08D55298AEB3369A5F9D2FDBB2808C2174E1CF98CA5BD73A5FC88D8FBB36ABE0BB2B9C886F6A60673ADE5468E94045088717569565B88A955DC2030F2A7168B406192BF7890348DB87355144F124FC52C7B7F123EAA209E723258BB835F7AC8C13559DE5A30CC0783B31476F9FBDCAD97A38F6D70D8C09A3FECF04597B840CDEA4F102CC0E769BCE8EFEAFF445F8B6377EF73C42C6418D1611AC6C4EDB4F66EB259D6855E539D0B68BCD188AA22B405B1DEF98E26F326C0D46E7F5EF3D278BA779421F9643C41717E9AB85325EC40E178B996666F6E03D4523BC1A0C41CA6200FF774EA032DBF474569FC29C879FC950F447A9AF288B2EC7F2426248386DA7BC480CD22825025F35388EF215A82F9795F44279F976BA1DC1AF63A292012D0348B57ACC1F35BDA36E312059365CCE2BA4D473753BA65D27041C07E5C860905581C54EE3D3C47C19698E2056ADBE29D9853C132266BEC887994E908B4F27319E97DE1760264367C806EA24043DF675A497570CF9FF2823A93565373B9C72F39D825E5D2362C1208236124A9A57FDECC08A2CFE32B4B3ECE4F17A2146D59DDA41748C022AC6FFC19589C1BDEC0D251356BC0CC976C46829947914132456BA53E40E47BF7B54A8DD4393B2AAEE4865F9049E77F12971E899340766C3A443654834A006520639B1DEAD364B5F5DAC20FB4CB866899EEF03FA605DE92D2356039EE310570F43B2B17B2816E02D7C48E72D248A19A7E258588BBF4A04C7A4E9E59D53B7A844C94585566058E1F77AC33465159E0718E367568B652054C90921C41A0E6AE0A0BE0CBB2BA30D1F38E3BDBC14031DE978F3F3F844685641847128537F81C5B526D74782FF06FB1CEBBBBEE1A6A4D5E16CA155DFC391C7E840FBC226BFE8CAF7C56450E9F47A8ACEC3C79A1C50974D46A7D84C4A899B2987C13C0D89D5CED475C81F88F3FB7DC9A8EBEF6AFD1989244E8597E5846E83175E585C67BE2366AE2EB4A39574E5C2EE9D6CAADDA0E981E428FA20F083FD89296D946EC6EA0B2643B980447F0DA21E6F761BE30D87F0ABBD1F3662E477203E7323B147934548FA707A22CE4CA6A12A8BEE9810B260ACDAA537B40FE47BA07677DCEB10283DFD0796CE43BA5295D3712F1AEE94ED980B6E33D975905CDFC8C43485542017C0457A7385A8BDDEE0A42AD800DFAD4FD4FBCC0D02D06DA8608D3F8C22AB06B0AB720EC82E69E964086114372E00DA1F54E6DF9C14DB99CF0D98DD6F1CC63004EB48D8EAFF0226B487E0D89CAF7B414DEC5C5252AFE1F9845CFDCA551E7144E0FDC3A26F5EC66BA7498F8E1E6FF6CCD7FDFBD051B73DD89DA827A15AB9603415FF48837C433F39CCAE0990888FA1BC0E5873C552874429814FF59E989319B0DA080DC02BF5255BB34E67A5E0BF188346B034746A4A4C13BF6478919EE2883453EA5F9102F87C32DCEE0F9E9B039DF9E1948110C76D9B39A900B89827F239C2ED02A97EB8FA17C7D0FBD20573F524780CA093D8A57536FD69189B2AC05D3EBEE38393CF7B9AAA756D2685FE5B0442343B78E566A12D4670C21519636F84FE66EA3D09C6295FE0E20734F5FBE5F902727BFAAFBEB418EBFD49CD1F5C8CA1AA63227BE76D76BE3EEB37997F9D1A903DD161961519F820D2D1AA7074683B188B86D2108C23DA7408811497DF86EA866DCE255A7518388C73BEAFB1D18F4F538317206A23886E633BE6AE4D966BDF72837C0FC3CBBE3FBBE0F598FA97065AB6DEBB03BF097E026D91E diff --git a/src/libutil-tests/data/local-keys/mldsa44-signature.hex b/src/libutil-tests/data/local-keys/mldsa44-signature.hex new file mode 100644 index 000000000000..d4952f3420b7 --- /dev/null +++ b/src/libutil-tests/data/local-keys/mldsa44-signature.hex @@ -0,0 +1 @@ +12CFDA699CFF51B6BB67C36C067BB8B26A2AAFFDF75487E7843DE75720F389B01A695206D9558505C4629322889C9073D002CDDD1FDB42D30625544ED22B4D43552732F56C7E06E192B02B9A791C2DB5C605EB3184DAFFDA41FCA4F998FECBEBD4743B43A1FAD7B755A262BCE5D605417AC150C4223B407CAF423ECA1AB8BA1DDA8E72FCC62CBBE67C804A6A82C09395DBA4EC531AAE9F5E4A84632110591CFEDFADAE629F1A168F3E91EFE247F9984C0E6CFC605F7253CFD3248B48BAF98D5A0DA0165599C1A351CEAF32F43F319B4CFCC83A898E92716593607417F0B701103C6335557B10D21D38723CFFC015FF4951CC78582031D84E866646249A85DD0648BC57DD2F1289E27514AA62CED865240A50F76DDE51640D80ECA8F9EC641EF1398925E11B5339AA1B36A4AD059B3B491FF3BDE9C55B8B65685CC62C507018E248182902222D69979159AD2DED90FB33810A542B7685AF4B2F9396A91A251B1CDC2F4B6D6DD6FC60F510ADDF804075A6BE97F267C773B1C23BAB469763A1CAF56E370E0AA3301CA2C96DB07947E9737B42187C3B40200226C787CFD65F021B82788DD8FCA7A97AA7BD1D252CE313843B362497645F80394B811C2F3C64B710F5193F76EB63C5480BDF209BCC347979071738FA028B1529A620280D450B0043EE873B559D147CB4CC1BA01E614D2E4AFC454A1A19526DBF748743FEAA17D754C877F4C7E411BC5CF033C629A15B0D1D740CFA8B223F775CE984E32571FCDA67A67807E0186EBAF978D31ED2336C50A04E4B5EF99502C390542C268004BDC56A1065AC14F9762B59422C225AAFF2EC75E090700A32B3E22C64CF0D98B306147ECE1B38BC88D6ACE621B06E6D668908646532DFA628C5C2B2C10D50442D86A75F2DD7A30D503482A83327F3D2346EDFC51A8187E915A43513E026FF9C460ECD6194B42F9B47D0155124BCFF55E2625640890FF4B138427949B3BC51023A649780C11F4DAD114A3408A125FD9E36F6B28DDF1E070D067C6B51C463059C7448FD0CFC86FB749E0E2E3CF02B88290B5564A3860ECD6C85D128704559C01F791A4429686F21C183D3BDB9984CB77699AA372C5690EEA5A191BBDF8B0942F95751D289F423DF170F176BB661ED8D2AB784BC5B6A344D82E261D5B34B6AFC80B957C1B943648BC46A2A79B4ABA54553F3EB41B9BD8C610D2FF82C34C491E04F678F33E5B383F8E6AC4822586887724F71AC5F161FC25B8FC14D9FC6DEBC25ED2741186080795E253397CD73FB8F27F89E25D60D4B8F40524D9A98B1E3194D2B1C283FB805DB34962EB8AEFE59BBF8444F86E0C644B3589BFC4A3D898A73FEE7326E8C415D5148882F570DFFE375FCC422BDF0F68C456C5484050097BD714DBB3F8777ADF4DF29DA756A28FB2F0A5B919A1D67BC748FC7EF031453C0E3CFBBC963870AE46B7D5B4C06DC828E910DD0622FCC21511B476DA150D9AF7462F4FA634B33260089CC8DBCE5977BED935704EA495F90B569AA8E801F84DBBFE14E8A6C0839B74D917361406BA619BD1FA7085845906C8AED1DC118578CB32A390888FAA7EB234A51ED90601A48ED2A12EF8FC6E743C947399B56D62554E8CDC7687AD9870FFC48F10243E2721031D510443483F7963ED4102625289406311490579A8881F19D520133F95E30CC693C88294F8ECADFB82F7FD20A4BD20DF4210B1ABA449A9521DCC6977DAB0EAEA29306FB7F69E9C4CD54E18149E71CA25D09B76B489C1CC7EE04183EE91214C48C40567C9A9259484C994AC1B38DCC3501DBBBBB4B56C8E442B630C8198F0929DEC00895084F3DB0E02001534146426B88AF58759E7B8ED9A19C5AFF32341931EC8F9018EF84B78BBE67D7873D50CDD8188BF9B8E4785BE4AF4E556A7D51D88CEB8BB1BB7B4F3901BA42CAC5DC0D97A48B19B5B76306DDC1CD71479ECA9A785BAAC6CDAC243F958148A7B4E5088014485593A24008FF164B7AE1B7329F06CEF7BEACE70825A21836A791382C80D35B4B5E3953CFDC53F3411C9F24786F8A14F7567E70C52FF205F1AEDBEBE6006DD10DCFA1968ADE2E53F9651D1C06E3B0E55415DCE2B38601B82433D52108EC8AF384FA54C12175C5B344318CE1152C31601CE86AEAB9BD6C9B960967175A5C947FEA79A458DFA05A27648F5D94304336D7ED633B67A4545C31274427BF3A76F5B4411664D858EE96FC564058CD287030EF45228123D7EAA6A1F4261E94E6C21D13763C7FF4FFBE23DEE2F5805C2E2E32A9549B8F9E7A0BAB1F1D7636132A31BF03389A71A9CD9708EC9BC23F42D42B8E9C51B3AF13893B36FABFBABD0EAF87CACE3B26D8983E70C47312C1255BA030969B55FA05FE7DF89485742A816FA13E7FFCE19C0B2F4B02CE2BA8C6D0E5C2BBED204CBB6D042B244487C60B80D4C2AF150C45D0CDD56C043C14E795EEF4926268F8062F720DEA3EE6D56D0BDB4FE7A7C8E926B08ACAB9F8484C0A71C02DD4606672661A5E32B96A722DEFEDC2CF6853D5E962566D7DF1E8B3E013E427DD8FB18A20ABAF9FD778B12EA781EF293268BBDB21352199FF80D3FE368A838785FFDD66E65AAC6ECBAAEDB5CFF8EA0828300A891333B8F8351C3D5D6FE5C6C51AD15B0FCAD198B0954BBAD081B90C4BAA9AC2059C31F36E8850F3038BA3797E72187A672940BFC68B8F8BFBE4953D97156C3FB79FE43A89079AC4DDEDFF6503921D4752670C832F5EB23B026A5178123C2FB725632550060A91E7AC74D981C69BE2AA6186B42D6234A5D30BD7514C0665F385CC0A014F638B300D82B80CF32E5ABEF0F16AC55BD6A157662ACF38050D8617A339C68C3DC41B263A8385053386F3F837239879D6BF49AA513D1FF7C96D7962ABE3F48F680285D563791DE253187B879B36445842DA6D2D6FEDDEB6AE374EA9A7BE48F62DF175FC207F9E0BDCB173A99A9860975FB35C7E5175A232570B9A013F6605E3165658BC03B32F096F9FB417AE6D8CD1451B8D80022AAE91194F2B270A474A931B8ADD9B0A26B2AA3430D293EEFD911714250E6D95147D3166CB387B948F7FACC76F0B3AD96427AB286C712F30054715B6D788A676A0D95D96CD4D92F4C1EBB57C8A12A93B83F4AA95384B0A4F20A55D200EE76B097F2F4406178CD4D3A0D672A87327A278B65D033217437AED9FFB605CFE7DE6353FB498FEC6C81E1C4EBF0D5CB8121B383ABADEBD720D7D96B52FDEF7B13BDD4A634F6CA6C3D08DC85DFD4472FE53A13CDBE655DE63F1DFF26CEB8FFD7F68FD724970B9DFA60AA0D08B181B20A43E93FD038893D3F0B700031D2F3B4E58627074A9AEB0B3C0CCE0101D27282A3A5D6C8998ACC7D426373849595F81A6B2C0C2CFE40723273E4D5D6F7B7C7E82888F9EA3A4A6B3C2E00000000000000000000000000000000000111E2B3F diff --git a/src/libutil-tests/data/local-keys/mldsa44-sk.hex b/src/libutil-tests/data/local-keys/mldsa44-sk.hex new file mode 100644 index 000000000000..f9ee333878e7 --- /dev/null +++ b/src/libutil-tests/data/local-keys/mldsa44-sk.hex @@ -0,0 +1 @@ +333015CCE8FEFA00B284EE715465E9BACE20202DD5207AE59F6FDC96F8D7F29C504E7D18CAA52B5064FCBAAA44DE8540EE2017CF09C62D4702EAAE198AA01A7EEDD006182392C4A57D2BB9D6A36CABCA8754B248000F5F608D3706FAD1EA062F6FF6B6C404FE1415F6EC45D868D4A60B7A309B8390CAA0376242BCE14E6E853761166C50B22819448592988110362E24B940C4300EDBC22C48368CA088655C020A0AC605641869010725E342841B131053C6048A08850CC14101434808C9806416440008128CC42C9204468BB0884830711320490A188658B28DC4200123392A0CC888020072D9922C240260DC2810611085C0427109200222B2681A4971543031C10290D42044192840A0B46C5392640A450250106814978053826D24C680D9C8915302026108418B4080E4422ACBB42008888421204822C0111CA72940082804148084B20D8CB229C286681C4030049720C096650A446C14274A1BC310E1C2041C18926100041A8925224421132121E0020E142985D8A6041A93005906849CA82964A84883B64193384A10A50DC0361042C68D0432111288314438111C02081AC985132481893666031612C1044E40184AD84452182964C8C64850400DD3162824A42C4AC265DBB06812C58C818011CB9484C2280ECA1202129969A3C4019996619C1026D1482CA10201CC4024E41271A212420C1931DB1205C29228C034820937885924894B3431493082629825D9826C91804D11B27001B448D838281B276CDCA881DB428C9AC48089843193262944446EA318859B140D930812D1340A93080AC0020E1838659C3642E2A09108024ADB9605C2362E64344620B78C1848620A3711033030C388810298289A90610497441C296092C47088960512A689E3447018439293C28C18420061B2809C482E20124D130365C9186CD2344DD914802308045C028082466093C8295AA60914300021C541CB247160B60199942160142299089183464813186C141022881824A3966101887020A78121B509C1360D00A265CC8430A048510A1480A4A851DAB211CA482EE234509B368D10C60401440D51127003C0504B08658A9864D3C83123B7918A14080311040A49410CC805DAA6251007044C025054886901214492344E5116281CC48C041106541811C8C860011911249601D2402C9986014034469C3802A24421524241118971D0104591287160B420611030E4A801142184D184085844299482811036621407081396411401468C9E2D477152D34275ADB0B36638F85512D7D382CE50064F7AD09A348F69FBF34C22DAA2DD185261A3140DEA243DAA235C624AE3CEEDC182A7DFFF60AD24444568FB7DC58DBD70DEAB54FEFDFF742162C282EA39785FF788ACF423309F49B48EC9F7E1C9597FA9595A8BEC343EE8E09964DA239018A274B93A5C39C260C89747E35BA2F7FA6662876A767F6B121F9CEDD48E787A2848736D2C070B0F8C59B55140626BC99609616B34749380C71404F861AFA2407155E0C9E8CA6DEC09D1B3B85593E17BAD5A4B11362E76C0239FEA7FAD482CEDEE9CDFC5C70B72923FA7215F4EA87A4D1E12D60206CE3419E811E41BD36F5022AFE6FA335DCD24B6654B76679CD449A0A49C36FA35D3322AB626A0AB2055039AEA7B93AD06027397D793D617B9EC4D60251559E4FD6841CD40A8BAF1B931BE64477D77F9B9CE593DAF4ECB1F40F3CFFA399ACEB19E3E70BFFED3652B95543763F3C2F796D84B5521C587572EB98B635288A1DF9FA4C7A8C038DB568C0B9F760E308E0C7B60A67EF1AD6D4BE16A9D5746E063F3C48D253390F95FA3CA5A39A44B69E2DF56B737950DC78E5110FA9308F23F5AE472409FE0080C97630CF3425270866D7561BA8AFFDCE6F7F4BF63D8C3F1616485CA6536C1C70590E4585B6C4B572085013DD08511C4938F0910AD317E3FFDC38EF9CEF3024E6CDC13B91169317F2DD347946627C13F260194368C9C08E9E14F2DDEAC3C5BE1CB108644AF3A034C33E496B299E8A905D91292445CDBE4C1A63263CAEFA6F9ABC81E7EECCB4C549D06FDB970B0015FA0496CD35073968E7E0E6453D1D79CB393105F587A60ECB37CAD30A6CD67EFA9DB3C1C42BC44D1BE73AF517727B692044F3412B32DC10330D5EBD46F208417174EB047CFCABFC08A0D4B4E1633E2017FDACB9EE8239F3AB63AF978C14D7AB10D69E07ABDC02FA7F9DFDCF439CF22BF88A7EDDA4314275B8C18D8DF04C81010E9DBAB2A8CE2D4D2B1A4D8B2FE75ED4B51BB09CD8E8E55D9F64FCA3979D158AA29294FBEF61839DEE3849579BC4A0E23E33063C6CD0FE9C429D7D62F3AF2FA33047FFA1BBB8385A99C04DF844A5B2AB21C16908E882ABB74A91C7CAD3EE31FD5DFD0F9726EBD64172DCA81A6A4CBF7670FF460BB21F582367074BA0E61CAE58013446072ED9BA006A51D1AAAB6B25C798CF661C94C7BB6FEB746CF701414ADDF3C8091E90C55399E8DDEB7CDC416CDAE1EDE10253DCE3BADF031BE694372384190307710783D5EA751EA353D408FACC87ABA5E9FDAA1B9C211731B1030603F089D4082F201772FD153D58E647D6869FB528DB28AB7A0F621D5F4C8DB8BBD893C365ED1E4ADBC3153078C4EB0D7EEE7840C755293EBCAE217BC0244D2FAD26A2AF8AA85AA5D710BF48E357E4B8C8021230B5BFD64DAD840D60F7ADC8713F3A3492F606166973465BF05F91A3672D68FD19900EB4DA032C5A674C606871907A3B4D6E328A46DEE3CB9BCA2FE86DF036681037B3777E0D5FC16DF8099F54FAEA59B4ED96835B4ED97BB878F2DA58E3DA24BE2D06B0548A4636C064A014FD083E5AA5CD9075713A4C9A59EB0489D1772FE28D61FA9A0527A5BB2A27BF0A4CAA26437985A7667952C7F6F4EE4E5339CB0324D046FFDB751DA1E891E1FD9F47B38294854576DEC23BCE333641A45AE1F0F1A9936E7BADE9A717EE359A2C96E17F55E3D88AA7EE5AB74545084680370A541FC993F81FA579EBD1E06C7A48C1FA520CC466C830C3D4E3CF84E3342D6583DD06D945FBF6B576904F096916C18D1683B9243D47F21619AE03B266FB168E1F92633CCD654DBA8D2C71F2D817ECA58629C45C71A14ED398432788FDB2D8E6DF48C2BE58D5AFFDBE52718EBEB520BBF902A4F63CF527098743286235FDB71A948D9FD1A7FE1E9E6D3A1F4620B180EB95F2BFEDA3C32701B5203AA6402369CC230E2547E641044A4051B0F9362587F20EEC73DB22116F5EA6E9412227BB85AE030284F4BB50FFC46D00F21E88A718B1933532DAF839EFB0260D2A30E1CEC519C5650A8CBFD2F9D700D433E41545C763856348D7F9BA05DA9DC7DDFD17D66AB5EBD879C6D0D04C26B75985AB3F11242BD15B6B24CC2AB4A1688DC653C6A9B1FC853A29646A5F195E5BC1C972FE0761539A640C561F0B31318852511D39EE6DDD1601C01B942F7E6C242D8DFE54FDA434E142F6CAC60AB6AF30E689F6636C83917E40EC5314B51F15CDC17CFE8701D50A829154EB6323BC40BA7B50357CD2625BE6EB53B49F7EEF13A2D5F7B4C954114CEA1F9CF57A4438BBF0E61E6DB1B6C0C8E1185FD003B142BA301C3B0BBFCA32F3480FE06F562F8D9CC6073 diff --git a/src/libutil-tests/data/local-keys/mldsa65-message.hex b/src/libutil-tests/data/local-keys/mldsa65-message.hex new file mode 100644 index 000000000000..60879c7f5a73 --- /dev/null +++ b/src/libutil-tests/data/local-keys/mldsa65-message.hex @@ -0,0 +1 @@ +283C1E3BD96D843B9B5C508BD42851817D0B8E5BADFEDEC49E850D07B322AAF50EFFD4C2B807AA6B15D31AEF14BC79CE74DA1450EEE9A0FCFED74383D89A0FB54B65474DF6C55C915C7C454DFADC9CE209E5A1C2D8F55B564DCA87149906E85A2115CBB19E82B3159DED464940EDC686ECDC8AC1F430CB1616E6D0AADBF97EA3C6853BC16388BF93A696F3A40C723D1D75630D18AAAFE05DD72E395D165AAB909152BC7D0B3BE9FAF53470D58DA2455043E604878910F5820247FD4C82B4C8166B814652C380037AEA97289F31E2A4D47BBC11DE711712D8A47FFE0FB5D4921DD4A1B73277C10D55C4769E9BE89A010056B32115D19DB822B3BE13D0D6C8F08833113A33E3FA889CD1E37C2939C596B84FD4D7A65DC62E0FD59A2CFF5DDF3841A0D8617F5251D42D597EDB00AD8EBDDFBE71AF0E09906095908D85B728E21AADE429B8F7A55529FFE2E136BE5BBF9D5A5254E2788C1C6B58CF06D57C50B70F903C37C04AE90F34A0E93C2B75E9309C650AD56E377257E4A474A107DAF6133B0E561FEEF825A730C18C55CF71C1DFF9D7E1CB8F8BC566B02A2FE5F119D1C0F6B9ADED54E2077D319961349E2381138A66BAD89BECCDE8FBAB02F675564191F62A7851844B46B4AF7CB75C74C111B320A43B33ABC7EB6E0B32389A97B7F9E8F6D5F6AFD6A2B8C69966A0865AD81A34469AAD6F3E3767ECDA4FF43D11DF1B500212358FF6574DCE137A6A13B4C5BEFF6F4A7AB207FBFB2B65EC072471CC400479544B19D88A7FC04DB56B3A63CDD8E174650A52E395A44E2B0D1A75189B837EB5CDE08CA732657A9D5D48AC4A26397F6120C0F44BF6D15AE1F34B3E051C1E9561B5619C4C3A02345A5648B5EF8541774B1F913631E43416692DC7A966C0D0E51338318B9E849EF357F1D97F0C3B88A89DB2FED50231CD9241FF20EB2E592534D555F1782EDE294AF49368FE7775158A19F5038CF5A63B04B0606AC7CE9387BFEF51F521C6A29EA03EF37B3D367EF3ABC6EF7FB5181585CB76B71ACD7E673E70D19E7D6A21DC7999AA1BA17B71B1BBCB2DE92F0C04585E6DBC16FE403B49BCDF6EECEFF1492195F5FC830D778242E174F755163A47AD5DC090C68BB6D2761CB89324BD5226B220442476B96118EE3348CCD0B658014F042C3F15A46A65B96CB6E52665B68AAE45ADAD188A4BD99DD63431F1703F2C02DEF9DA47016B60CE5CAEC293FEEE2364393D0FE25AE048DE7340810539C2682F49E415891AC3EB549008F9D22A4CDE573A6AB3D86E75E6F382F50BECC0F0D75681ECDFDC465DCEED157D5840D4CAD37E2D41F51018E91C29C15BE7E9B001D2E849CD9DFDCB0B99307ACBC28A9FA74F69DFBCEE92D920231EEAABE8DF8D49B8D808F7E82334B23B1BD3CF303C0807D4A93B743369DADAE031F1D1B006276DFFAB07CA9273518ADE1226C86DBA5215EF9BB2104D8DF5C0A39A6B3AAE12C561E441898351F8A287B98802282A7D754C6CA0E375680BAE892E59C334FA0B13E13B5046E206BEB5D3111A1BFC17241A66CF6645C2F300B22A23904106CBB3A1D584194C44E445B2B162D7761A35D21970608FE89CE098FA88952A28B87018AB873451CE31947AF6B803265BEE8847544C48B21E763BD626534BE1BFDAFDB5D8BDC5FE618A1E62237F06247307EC46695405266E48D5708696DBED93B7F9FCDC762E4734F19C5A72B7FBF4E7C26D0240DE99B9D6DA2C0604FAF0FDE591191E73EEE6E00A2D24020DDC009A4FDA6B7794C17EADD66DEAC356387459F4A93672514ACF533A01C3BF8C6F27C19E823E7953E5A71B046B69434121E1CEECB172349D9F1D8E53BF5D3CDC821940B12BE43806610A67DED4116ED92312973DA042A5AFD8EC83BB45DA5E4C86F016DA426653B81F6E2E57220D8922D8D32E1DAAC2CD98F483177E9D57758F6289EB5E3447AC512F41FF6DDA9A32EF6A2012BA6AC1D6724E4A21EFB783FA1FDC09F155FCF443EB8D98DE80341BE31672CF019776519953DEE85277323AE90EC23489BDB0A5BE6CB8890C0314B45AA1C1542AA192BC5365C7A1FABB61FB74999BF3358FBCA46A9D709354CAB760F5D9A3B373A7D48502F7C345621A432F000D54C3630A8EFE5434FBFF2794B417AB29BC3EEA6109C95DFFFEC38D11D9669D982809E2638AB2178E157C70328501BB51573C81E0C728163727EEBCA58C69B16C31AC96706903663D1EB2891F0084EDBB69197B8BD3CF44BE794B66F0EF36409E9B71B772BC256BD2C754AF0CE78F828EDEACDCCC084DDEE91F5295D4D62E92CEBE71AF8312CC6745E684D8B1B864110E203D1E8105FF104BF5CA3F246B51579F056871F99EA3CBFD2EE1CC1998D3B35E7600262DCE9737306A4E26DEC99627B29D70846BC2880F466AEC193C2F7A991BA808BA9204F4108AD43381AEE936EA481DFB9332EF5B6731932A22DE220B64EBFC1014B4DE45AE31A373C72AB9A2718DCB3C479366962CBDBD99D7B226B4FE9B5E3B515BC3B684658FFAC91B1924CA6ADD489320A6C2274E6BD86A63AA3D2C6C8FE04F840D4ED65E5CEE558EE4989628FC81F052218D7CADC05EAA9915E053E588AEB3AF4CC7ECCE9AB4AE236FE7AC16727F44FD22B41B43909C4CF91C70130A11C57691AE08A7407F13063CD00C490D58C70A28EB64AAFE3F62B8F4086BE1A1F07E2FB9F404DD7DA82F61CDB35AA548905464FE120389417F6A968B864F70E1E1BA967654DA61BCC51D2C75A52A69D5EE3E44784CC25C87F5849456C1A29BD4A407F81D309486D5CDB1C06F8B7FC513C1572E212315E8D9D7C70ADBAF2F6E868DB7F46B3E2B2D1D5F688E349B344F005F43862F28288B987236485EB1E63D5CE65CA3DA7F25574887574AAFAE5B0F2025BCC1ED8B54E2309273909673B574A9183C90AB5474BC213112840A581D0C03CCBC13269055987607F84F3C46885A0B62A2CDBB5EDCDC38E2514224CAE928E1B0D61FCEC0D74CE5F7EE8731AD9F7273364FF481FEBDD9EB8AD3FDDD3B90658F29FCDCB8FC915A5724CC7837C8670CDF2F7DB4307952DD8C8F1B601B2E71B170265099B97EA1F1EB7F6DD4DB496807EBEF2B4EEC008BDFC3B55A5AEE7DDAC4F9A8C2A0CF53FC0A7D6AB5BC581593E7EA0ADDF5F26A5A32C71FECB5891F5CA13BC6EA6F315212C9E51EB9473353FFAF6BE47A4B1E158BE317F4F3E21CE12837 diff --git a/src/libutil-tests/data/local-keys/mldsa65-signature.hex b/src/libutil-tests/data/local-keys/mldsa65-signature.hex new file mode 100644 index 000000000000..ed2ccaea6f59 --- /dev/null +++ b/src/libutil-tests/data/local-keys/mldsa65-signature.hex @@ -0,0 +1 @@ +6068AF79E1C882A7E1C8C872DB3F0B4AA138795600B0C8AE6533ADE2C6B48E008FA0D554E8D99B4F74AFF67E0AFB8375971C21718CD2BDCE658C41AF6E98B48CCE49C3D5516E78550900BAE613F4B359204F19B2CBC0F32905C1935EFBF9CF7FBC7412B152C684BF835AC8336113A468B99FE274455A619AF43A490ED5E865ED9E6446488D861DA1C60BDF0402F9803D123B08E667ED67AFAC005F82092F873FAA7134093FD9CCAFDFED93D2E36C850982E6E2FF7E4BA36F3A46477AE5962E2CFDBFBD228902A73BE084B1BDC6DC0E778E572D000AE68A5371ABA592968B2D5159CEBD64146813201329710CA95D1148998552618687948F8196BE6146C275648715652767210EA791DFA8744781F65FA42493567DC9C70601E0466922DDA717AD52DF58ADA25D6A0043662F45D578BB32CC96A707F8B7F71E31A2E670969ED7452B00131F697381B082A542266FEAD0662DF489EC2540DE3AA7D40E2D3A7820BCD5D8253D709C4A0A98F302F9DCCBA6AE5CC2C02C9C5E9B7A5CBA36CA6F984042DDEEA279E37359C975173C761765035FB13C9E56CEC9E2F4FCC32ED3D8157491A0D34073F4619C3D0007FB5CB1E61B47E2BBA8A7FE261778D3E0EB4EEEA5B3B0F57B699C1BC64D64733F852A3CDA8CFA7740ABB4D5145CB40CAB1F4F4D396337F43AFE8D7CB19C0493BADA9C5D1E50EF1B4768D5C08DE86F3A2C0ED3BC7CA97E4EB8CD5B6CE22CE5A7C2F98E9EC2FD9F5D16EE0FD9A8A0F2D4DE60009525FA1E7FF734F117D546A2F5F5DEF36F4A3366A1D2AD9C4E0A219CE9A9565FD3787ABA67D5E0454E1E325A9F44872BD21680C8D15F7D69ECB3F973568B2CD5E879B5BFA6435F0CD1E1E8E0C310A36644DE927C731073C11D268094D61B8A1F23038505E0A9CCD002084CB7FD1619C03436E95A18B32D449C3BA54BDA86B3FBE85657986A24B7CB9471095557EB2AD5F3BA82D6A2126023B39308EF061465FC266C4DCE877FC2FD79B9E57239AF5532D03B08F5E9F80BE37D0BDAF7391EB7872466EC573A8023259F19A6BDFF23D91A7C2F46C2BF5D23C64F0A8DDD1FE2298496682293B28E15547761913059CC50A4D01E229B8EBA9D9103DDF06907CCDEBA3A8AA04157607F541479620568F46737F957AF3A711BA07D09871AB89E3782A650A6A7FD7ABB4DA2DFEC4D8396B5F94B7FB3B1421E52B185F60B063C3A658BC67AF01D0FD435920E0EA08E4AF65005397BE5016F04CCEFCDFC9CB1B35B6F8B19A6E9D84AE5C1D17F917306D680DFDAB0C1A4B26E6D274B4535F28423C6D353D16376413108DFD83E8DC50554AB8F295C9CC6B8A295E7BBE86AB99D90C08792DC03345B062777B6CB468DD99D771371099FCC65C60B9A68F5B298BBD2F7222784CBED95F59CDE6B9F59574635E9FA2EC4330E40DE8561AC19F9ADA14B9C262C569162B8AB0F7BCCF8B2419EB330A27338699AD527FF64CE4BB66F9AD6EE6EEC7F9FB25A77D482DBAF96B3A5C9DFBD174961A01A03DB2B1214B1707ED13FDBFBA60CD28BCEC942E733714331E0497D393DE2F53B2B64688D461960712EBD546B9A707C8C5851086B8CBC0F0CCE8AE3923EFC57AE8A3CD8595202A1A00EE97FA4C991C62C7A6921A8DA02048AE0569590CBE02B77AEAC474571609DB4C8D3871487D50FC2549D92FB0ECD50B9479AB7CC259F665BBDC329495F774AE046E2A0A438464528FF12F9942CF3078FDCDB1463F15F23403146ACFDEBF54219C408114BDA5D65273D9C5038FEADA233511F6F5A82DEA6ED90436FBCAC08ACB3CB623C5A6250F9094AAB86D694827265B9547CC0E658AD92294DEC5D652F82FB330503F49C20B9D1B0836AF78E000F81FDF62558384D6FF713DE94B11DB282D3DC7975128EC6BCD24C508F9DA27587C2CED52EDBCB71220BBF36AB51D0C2E07F37DAC30D6E739861F2895509F2710756B0CE4DB5C1B5CCF46F919B6F2F71E640808E07CE5BA85124F61322AB47FB77DB7ECEF21D364466170F310607F9E6EA3A0EBF59084897E094075ABCD260C1AFD11A3DED5478F2F26922D6FD29834415FF1AAF9A2F014A8DD794ECA9659B3EA5BB3BD0FBDFE1F980F82CC69BF71BAFA5D58088C3E930F946CA3263DF8F416AD28B96590A0573031110C3627B55E50030740A50839844DDAC0AFF1BA6DBDFFD9AE1D1023EAF7519C37E8E93BBAAFB054D2A23F072FE13B53648DED2619F61A126C65B9BD5784ADD1935269EBA8E29C3ED3408F2C92A3BDFBEA3079B746BEA8B589F9CF8B017E8F0654F203798B3CEBD51934F17AAE95FA1AABF46AEADC26AD3894BC5D11DC0542A6E96389CEE07B5347A4B93F979E01D7A59EFB244D473151DBAAA5B418441394EDFB4AA047454D95DD1968E12C42EB080100009CCC5906150F5DCE08440150DBA5C88C431EC0E11B7C4C32C1A7F0BCC0F12428C02A1A4F4FB11C134EB15CCAAFA1C064D708DADF3E57A6E7F303FE9EAF3BBD3BE58360F2DD301E2AA5F576951B4E47D8287F39270C8259BDFB3B8D21EC2297568AFCC6FF801CA95DA454CE9B2A3EADA4F66A4A335E56F6F30E03EBE3BFB5801039CD85C8EC266402A33994B3979299FF16E7EA209F36553E78F41CE82F6C29D28BD214EB3132971DE05D2B143FEAEA2757FDFBBC2233826C5D4EECF21428A634B9AA76879539E84626BD82DE5DB7930984877FE00E598CB15DD74B88BFCDE50BB627268737D9B5704C117F4042778F8571CB7FB736C0881723D014445A46EF52CFCE8CD4867AB5380D395EEDCF843CBB47C9FA2ADC0483B173E47AA407D05C7DF9A7D64F9DCC469726DFA8FB2FAE8A9DABD028B8495B4F72D85A8331C370C60BA0F1E5E4D1CD88267A250E351447ABB91CF53449BEAD23C29B8D2A6284893BE2D383C6CB38C12305C1A63243A8709B03BAD65C7FC3B15F8CB3DAD600B81D5D807B0A0D7D0DFE8B0616089AAD0E4FBF0375177782983895C7B527FCA4F274A83C10ABA7315E4ED2D915290A2611955B19B45342B9355583ED2CA87C163CF800A053116CEBE54E8BC97EE66B0CD9ABFA7B0AB6DE2666727627A29CC5F5815AE5F4ACAA482C5FFD611D2B67D8AB273BB15F9B2AEEAABB1173F905BF200E7FCA2CD83D0DF48122FA46F71C0FDFB06E0EEC61513A0B2DDB84990CC80FC74FAFB62D5EAB69195B2E3EDFE4604594E91495454A9F66CE33F7602A3B48D7FC0FECEC5BD4D43ADD6AC1CC23C38386CFB5C64DE0F2BA9E4D99A353F4BBFF4BE02C1A9F28EADFF8E7B1FFBC29E204DDE4413C375C36D1F2B40D2957028B1D470ECF8528D2516DD8D9D6827C3C267B6B829456EDC5250B490D6243267D1289448249EB6FC35E4EEAB0662F9644A9B58E421D18EF2D89665A404B7949F9C76189A03C7FFC19DB8F1286A49871F46987F8EE550B92412226FD372E68AF9AA41F58199CAA6ED8F97B359C2F7B1D8F9F0136778D859A92C83179B45C1960E8588F22868EF285286F0B05071169BE5B46F13E49C5CF6BB93B7C80C3FF6FF8FFB6FCCB1791E5BFE2BEBCAC933437A898B4C007D3F3FD5C42CD6E0A7408064A6CDA630D12CCF72D5F962475C260A941EA172901FB79D46D4F31B323D04DCEB7FCDA8EF5B61CC65E978E916127E10CFBE1500D20BD087E88BDF9CD90A21F360079D7E25989D7CAB09E3789FE039CC78CE67BC96A695609DD6A7789DEC80F0BE2CA0A76392D13CBFE4593B5C47018138E352DEB2AC6E41F2B36DD568FB1C863E8429938803A838F06B04B0A9C03E058959BE58C1E1E929100728D56B891902E1F9FD931A0F9533117C62369FFB9A00A80D49B20B30EA238D9007D91CE934873C57ECF74971123FD1644834BA99AADA864B09DBCAAA367D354D0B2A39FF97804E2EFF9C404E05B21EEB0C8DA93928188EAC83E0E36C5C4B8A0D70250BDA41B22C3A2F6F300D3858466AAF6836C9283CB62E7F0D6EDAA1FBA3AA0898E3C603E5B5E17E069149A92EC168384844156BC98405D0078E53EE8B63C0F8F94231FBEF4F692187B5382023371A3A9840C6748A30EF86215CC2A07F20566D488B637C70D06A29153D69FEC3A03E42E66CCEAE8EFD609E50B6FCD7BE4E252CEE22BDECD7EC8022930E167B1A47B1F7BC0E6B51D2F58A6B2E9120418CE857A7547C33BAB8D5B125D171907125B9E6F0A474CF8E70AA17D24B32BFD3FC9F7DEEC5C1ADD3C1E191EE321528E922663C0080D5DF56D4572FC6777DF887D3E57ED52F8FDDF515793441555454E2BAF6660912EA4C100591761F38B2DDE31E814861102F719CDDBD59188F7B734941C43E4FB3A227D4E84A4301880B338B631A7AEEBDF54F4C49689FBAB422519A38461DFE3B6B919D3570B9414254F9611C616D85BBD6027017C1E71FC4885013FA851C0BEDE83A66913863C22128054D431741E339D0C9C8F033383AC5AA0530919C657F3057A8875214351021272F3971A7696FAC1B62B3A438920F307193BD08194919452F4E416AAB86BB322CF48BFE35ACFE5DCBA0B80E75E0CC6809DA136BCAD33F5F26A037FA1577F71079D4D072B0F7FDF1CF5F6009C0BEE128430A4A9436242E0E96947E1DCAC2723EC522413EDBE7B80B4D7896B3BFC4222C4160B0C0D3E1212D4C545E9AADCE0D1024617D7FC3020629596E819BB7F046679C00000000000000000000000000070F171E272A diff --git a/src/libutil-tests/data/local-keys/mldsa65-sk.hex b/src/libutil-tests/data/local-keys/mldsa65-sk.hex new file mode 100644 index 000000000000..8f9cccc06849 --- /dev/null +++ b/src/libutil-tests/data/local-keys/mldsa65-sk.hex @@ -0,0 +1 @@ +EE564C4459FE8CB79A589325F7DA284B0685BE468BCD00439A7FD1EB75027DB22C3C7D2770FC3A4E79275901B32BC4E0EE4D5D884FC88F3FFDB81A9DDD00F3237230A12CF0D54E341285AE80DB80BD862CB21FC582E197C8097A11934A5DE63F3B663CD99BE601656EB9E27D56FBFBBBE38A7B746BD58FA34E65790DA88A030F70176352665727854760745823608022744313876685532623652744768578428337438617405738152074310424824643303541762316746606522673500876402582424231062014318615864371664326545430807247117362061768647140887228548155637813667108317218751042261165872702021201734526616768202017851386182805282338684257756711817037301316076613715010543364068745666462844431341255638064375435052185763302885267265787467273241160368308500544037325353486525717575251843765237572244588015208222037043303784867432458284057071457834018488546568770716155414875865740668060234514635511531250673066534261558437230000731184013200111321553606241330045062675233773288103033106763150884652087885410854273210822581364803555085064232365704775602336343664843341260310526521623852340634457864272746136116180200431145551702148674627241018023845421065675004573078773247041568481607061628830378841668760344070163067747583430870362037684776637525204052215840574131654740885034258448548665812743161135213727830886740735321655550633213675401005282737885020232726568868730511827303333187132051654063466630025031782834370470801865688473736811287187726280705140281047133613210450317004561025210477411068801886300526758257078257765881631878827153368518554583126474744735208465728215002161123065832325034318245363732833316200578534641677238254738430125032457837781834245701843455140812471872428158610157161866368270880056714782710036620562200476612132668524356425670643254032608262111623142301773736256208825188685537231403043140631684750654540275035645286775600557236775545363813633535335434233472227832086422758270346875478474252240164221201666518571286851446530767884454880635646641633738850361250137484325502526664473172685547167828335038352400242141236722417603550140382058487218866442326476102641850631468373647135152227380365638224013370761242854443230528816003523571238881827167775454028367602116562117802008538326831805004033046727857680145426777806171274617505467524313866534740722666531636011416488032683058372734745888670805146168012702105764622180112514176202617362466512076863126342530518568100456237754584137085761574301304221873647015856684537063270402426412188011174253623844525774375415506265526273312335515271240253074831074438432131474833108342228481656781886416050685144448783638723574581172187380550836566108035006184688850785637587486365280874714434614850035716383325065016302527706761565632721061344741184810054411325376215660082103624147781216461553888502178433867475753468813364138808084022063534388681581525737261467070334068388477137142080583424847112441844223064872437821148541631123687413031186634035034112867877265661028776644765453737708350438770604730610858814035262247715578137204463281740471564647355245271081032084767811732687747747036450524583780271267782766511371623341030862360028868580837017610663515666600685300606576215333073557666583D08E9D6AB7411609822EFD78C4FF8242397F7CACC41721F2DEB10546E50931F0A97AEEBB5B28638FE50172947AB9CA35D7BD407002C7A2122CBD1F1155E61DBBFE66787B338D6E5C1D296FFEB8633CD1216BE1502C319319CC40E7E2837261A54F65220849E6B28A50914A53020EA82D74222301B0859DE82543E2528899BB9A45816EF877028F9355B24FAD7754C7C938481684322304561660D01989FB1D318A7D8F05A05A76C902BC91424D2FBE092B974CDC5EE4F3277DF432E63D124009FB13CEEE0482CE2A857AA81D96518CE844F963E24D2D31AA8F4AF8DF0BEE0F55A4E4060D5B9A986943DF3A77FD2D611B8AB652B2F188078BC90A568EC67C682A18E85ED9515879C196693E4A5F752945E1FA0F5A54E47A2BC37ECD1FD84E8A767B7217ACE2488DEAFE710C1BE704179A736792C1071E8289B67B29445258C6C47D5AD4284874278AE24DD2BDF0AC0D4504D0A9A0B1BD0ECAAB288574001C84BE357ACCEEF524BB8D4E37A1136109F51D3971808F2A1B3F00B2C63E9C25AB7EB33E956FE3E1D5E828FACB5C4BB0099E409F31470B84F80785CFD34F44B80728D8C2179DA029EBD336F250C269A0DC0F667AB063B97D71C651F71CBDAD18CA9276C7653125EBE41BE8C30180B96D0424FFC80650C98D5F039FDAED5B3C32859BEF76EAF9E11323AB3BFD0646DB76EC96BEF6DD66A09C6D4ED9CA4F533BCF1DE26F995918B299256A8E46D347832286328749EB391E7730B9D8DF43C5331EDCD95705CB66966AAF171BEA3F37577FFB422F60E59FE925F3368A4858F4DAF0E1B875679583B131F27D1D06A50A55758097CD51E2FB6B99803896D1503342E1131FA1C57E4E9115790BA0B702E02CD0258D285B48F858D48CF9D5C6A05E493161CC652CFD0CAB161AAA0D76EC1B7BA8F69FF5DBF65ACE102579719D409D879EAD09B7D666316116A862341D56B245C4B883C7E18B33DE41BA60131E39D4200214E05E26C1998AABC5B05100AA81930A2E654C656D390F97CBF79256C499583FFEFEFF49BFFE14B7D5840A6408710B4A2D5D4E42468B92415073E30ECA5731CFFD09A4C352D99C6B4A2D20E44E3D5D64EB187DACA4750ACE16D5F1C3116B53D16CA83E66967BA4EE380686D33EC69738BB9E803EB2184B87052DFE821652AD7B61739CD4B064C22972707733914B94C5715E100F967EEA0E5F68EC7671EC81BDC1C81DDBAFBF1625584E3D4B93DC19EA450860C6D4F5E013FD78E4EACC24E1E1A2F6207F95C21CD2ED635E170B955A837706F5FCA2D0303031F880EB878B890ABE4A1BA516ED286B78274467416BBB1BE233FF4995892101A0D39363C160EF05A558FD74400FD579504FC0468A75A30EB1C6FDDBA841500A149151AA360F8A1154EA906CF4E1B166DEAA19A5FCAE8458D261B5FF34220F06509015516009A3A987AB1E860146E99CE85AA5A459BE0469AFD7EA66D4BF63A9A13853AA4C142BFB3887DA8CF307A1C9DA9E05AB97C56B2080F91DC3F8A81BCA52BB04EA85443182214FC72140CB130C2A05BD8A7A5CF0870E534E404346887AE973726316F55C77E921420FB8D993EC62BD119D374C1DDF06FEF2BC67559EC607C0A37F77B6BE2CE30E638B0DDEA0A1CB7608402262615E98B429CAA0B80707E7C9D4B1DD991BC3554C11D59FFA34D66F3B9ED545D82B80B6BE3CCC5DED06B03946FC9763CC57213BA17BE16D0F6F9B788AC83253A0CF7DFA088DB82747AF7E39C4274B0B1F36D2EF636B34E08F2EAE45DE17076B847713A393141A66814D22520A1F46A8E100F2F6F1645FFAB99F6E6EAB91C33BC18146FF7E876434264C1CA19E7804D277AF9D1AB9010B46DD71E137DFCC4C752ABAE55559378EE5C5D025D344BA8D93ECE909C23A1D3371B0ADCE27BB3EBF31473915085339296715528F08E768E310512BB6C70514C0C47054D150F90F0B866DC1BB51A67E1C01110C2E7D4E3F3729D86C91E76F2E5F7D8F5EE75777C19279D1B49BC6CC2B4AA47BCECBB6CC949D1EB3B1130B4F2DB5862C7FBC768587EECB91DED98F146D6D990DD20B6873C5AC13542322B319457B5B47BCD709EA2BCCB3704F12625DCC26BB2806C34A256CBDC1D11BAB7CDC68B07B9DE371AB7782DBA9A120BD4978F28782E8AC67B2093C6886E755EC0C76E648A039D4A7644B0B6C7593D508422219FBABD4939C5DB5EC50D73869E6C7F2AD97D149B03EE67B733FA0206281F7995D1CF1CB384B687A767603986681F84330822CB13E96C1185C5EF70A62321348C4F8AA44AC355C2118CDCFF39391D6751284EB0F2FDDEC5FA8B29D96E11ED47789A8AA1253AC4A8786F927064E63F1B149B47F05CFE48B876B49CA65B384B23DA727BF8C1F5867D325E598FC702D159FD98F4948BA8367F5D9ECA9458C0A08AD9047116AE720B14F69E33905744F1800A53992D14EC60E877447CB54C408F29ABFE290EAD53FD237A3CE4D51731C8EB075B6296F61D9EFE0C7451B7FE0CBD8CF117E58A45DEDFE645109A5C3B78F4B188D44394EE8CC6961CCF11020BD4F89A38EC99892774D62CFE2657AE4796DEE7703022D0378A9EFA33C527817E2402797C4D72139A37ED334047DB781137A736F5BFA7858F970BB2558ECF89389EF469083EDB4D504CFAE350CF7A65619185BE264F01A64FAF6EA33E0DC7B78859C91C5B08265D6F01A0C9F87478FF6F65365C7FB637B746E945DB3E2C894065EF41182AB02DF2A3C3E93A2153CBFE739E638F8303F74A4294EBA9C387BAD384801E3E9C52301A4704B92B3C3A0E36E1B06AF8317B8AA5A65797B30797C29B0A2E6E74AE76E4FBC72DE6FE872D1C551E5A5D2123636EE08F82D98BC0012CBD7F8C61C794FFACDDF1A488E7C2085E8AA879A97E6A5A25E9C19C18AD833D21EF9DD67422F69B19D4AF6664983DC06F043C46622AD391FD3CBF2C68830D1360816186B733E4D81A6BBD8FD36727F2B31CA3FBE66367FA62CE0D066AF28C1F779F1E241ECE21B511F5C73B7BC9369280405536084271110AA2A992ADE03210E24498291AC13D43E57E63EAA187B78F1E75B0A4C69C3D508E5564B074C039E578AC43D811BC1DB37FA85AB33F74C475102548C5F0507E033639E8DEBFEC064DBBE65BD24EC1B19C98700DCD1D5F62D05CB98AC3E41C6FD45A9F8A402D42B0ABF83B995661DDA3F0418AE8D4873840E3B1CD354D52F20B0AEAC365C5D50C6CCAB3C91FFF57AC7704424678AB497348ED3A8F8983FFAAAD5621DF1EB71BF0EE82DD58DFC75D78B3D46C74FE3CC4A2DDB83B64AB72D89F87907AF5DF8A2E35C197922628E1418CCDCDB72E7AF48F47D22DBF3D3F68234B4E9B615C9E613DED3328015093280014650CBE7A876B0F38B21DFE35380071FFE5EA45D0629B08E9033211EDE3935D08871E25B6DDDB95EAF59A1BA5F979C66E61AB6A48B8F5A76A205C4943D0679200371ECD7A8BB63430144AB3510CD0AE4E0DFBD3BCFE9A8240CB9A13D8C64EBC4864EA5849361B3689D98FDAA8106657CC diff --git a/src/libutil-tests/data/local-keys/mldsa87-message.hex b/src/libutil-tests/data/local-keys/mldsa87-message.hex new file mode 100644 index 000000000000..68f0c1f8d64a --- /dev/null +++ b/src/libutil-tests/data/local-keys/mldsa87-message.hex @@ -0,0 +1 @@ +8194B208815B58E017ACC96CE1FDF3E77EA19A092BF4FFAAE288977B62EF19530F3CED556DD2F4DA10EA7C272A0F84BBC6A80EE5CF61D76FEA1FDB753669CB94D3F37C66B8B9908C876A6A0263E9CABEFECDE42D83EC478BA3A60EFE5F7B457BEDE010F5CE8389AAD691D217ED7A15F81D3BF17BE4B874E4E94AE94263661E5981577C79E2E8F9F049C9E6F7D030D68CE872647DD5C7D36A59F3DDC873D273BC89A347C306FD6B82B9C743B822CED7C6125C14D723A7FDC91B83764D6BB2B78C8108C5A8F6E9E9FD4A8725D290EEEDD040487A2B5FF5C54F4B3EACF0FAB50ADE9AD4A6109E180F8D8874CCE8F64807C773710375915161271964C00676717117701C1B98B5614244BE0C7DC01DDFB30B4423626CCC7BFAB56C5C72E05CCD990535EAC7AA544CEBD5DECFF08B07CD848C451CD31077A60138C66FE4A1372013AF97848EA16BF8281834D912CDB3C02451FDAA01707D318AC38B32A27C641FAFF19DA44C2B1F8136468C514201698C6B69EB49C0E711B053E4B0AD23A13FB56976DB7C5A1E2A109BBD911220D1163B70AED4E231CE958A6DF7C3773CBA3FC759AD22583CDA3B3F4B1D1855727FA8175BEC68626998628BCE1857965E762DB3130392C8E87127D9113D7A945A64A3C58D21ED6A6E53CBEB760ACA64B44D7DC6B272BA154436384857D6B63483D995987D0BE4DB3CE026CE19D7FF20BA7FB0C872FE6DF0FC20D172FC294A9521E4C2752A84E36415EFC41F62B73516A83572A16CD819464EC01D6170F9CA01FF89854FEDA7552C201CA08B44BCB01A81416CC62A4A1EA633AF1C80B10480B67A09DB13D1B75814FB9CEB8A7F4CF56B41CE65F1B0440ADB9D560645BB51C90BA71FD83A27F2E8E7331C0B3525BBFD3AF5D8D17A1028D552440305BA9E8B3007D211467C87EB431349751986974FDEA210A5D1EC599B1DEE9C5F266A05325C4CA2A384045E0EDFC6C42A162D1671B16A6D0AECFBD8D2AA5616A58A67B1073E60A22FF849B94CADFDA0A9BF03F2339326C6D7714392FF7C9AEC3D510C0B806F832E0860A9987842DF4E7FFA546D969467B296E7218A6C6803D991C1FB2BFB7C9AFD2C9A869B45BDAEF9FCAFFB02562BAE397D08881B6DAFC98EFCEF56263BDCFD7AFCAA9844CDE5BA1E038E3A35F7FFC9E7E83608F9F797A8C5DF538000DBB18737E5F3021F12967085027FB31675A8E27CF032D8C314E36EFFCAABDB3D6BBD9DB6A5AFDEDA7CB919C0FA1CDD65967370BA91EBBD79672CE7C4CE7ED92C566AA2DFE44989E5637468ADAB3FF0FBA73C7DA3D732A80D9C82FF2E95414FE567C5FFEA1885A2141748ED39F95AFC675B7D061D6340F15654BB65875EA12E50AE6208A8A64C79AE59E1E4C989FDC801095ED0C492FD476F6DA6369133FBF6FC127637C25C06C52C41981AABB05FFD8B4D9882549D41C6E52464AE6B78BB46748D95298688169A8CB26D0659E26BD41C12AE4D7A6F83913CFCBFD8F692779948F2FD17847C0F399A12F2D82E7DC3A4FC5A41E311033871A419A595C1B0AC7F8A9C831C87682E5EF5C050B3E127ABE1928088A3120236CB0F847FB3DA04F9EAC5D4FFD95F552585FE54E647CE48F624CD809ABB9B72538A76E372ACEE2DB2DB081FFBD74B78622C3C2416668DACAE81316EE08DE0010AC1AC4B51C28261AA14BA73F47DCA2892AD0E7BBCA534FFD6C7A5F40E98416C4FFC9DC500DE2570A7F8CA4A6664745476400722FA30FDA786AC2A7733295A745E20D768D140689AD008ABBA4DBAAC013FD81E690DA08D27646E066879F67953B317041265B32C95A914C3875762168C15A46DAC7D0E8B63323D1CFED611AF938F031D6347C1A5AD849C779804BF64806CDC7A28890956D17A4CD30B61B13120ADC747BCE382698FCDF7B5D3DF06B006487DAF08325C5BD7C000E6B15173A53A6F569FB4C6ADFE1BE5AD85E4E016D161FD247E06EBA1774C7C0D6AE19E4F3C53DB96EAB167AE2BC7849BA9AAA9D7E42AE285008DC680D05A080663F17583B78F572FBCC441B4E6FDB24587BCFB7F52DFFEDF6555DE9482985A2B654DACDE0D0C57A529FCB7E30947DBDE2249192B856D540FCF828F0F35812B73A74AFBC1DCD58EA0E881ECA6EFCE61F5641315AE0C0276600E88B39CAD402716A3A6B16B3C40F137F7252EFCCE6297831E3D4F5C98CF1A59B63801CE0C86C6167EBB0986518032B470AE7D265C2C991056C0227357A6D2AD2D03EB6FF462E6744B8A286773EE4340B0A6026A7C475CA1A8F530F2FB11A1A7227C7595C0E35B7318F96EB8CE410587B31D85F84DF42AA298AC524FD5FE6EB943F7A97975F65E359F505FA6EAF54EFA4505F169CBE49ECBF5EB8B1F19FAC3789F58AE1C3C4B8DA48750DCEBC0208E24CB6C4998760E12F113E5C4039BE5867C3B1FC678191A678E9E6877DED6B2B1BAEE26059F76FF9766D1C836A3D92B0DFF2EB39D71DA7C4BD10BBF970E74E4B94A899E2EA8E5B52FD991A0F564EB724E3677ECC290FC3A581DCFEA54A5D82907C72D2216286284EBF14399AC329A1BBC606570387BABE66C9FC6912DD1A672723DD38F7FA7D2B3A6AF686E7768D91EAEFDBCFB174B67DA56ADC9E9DC0833545FBA0286504818785B7DBB9F5502943688A2996125062C75C324D898202D31614EBE700B9E62C1485EE10144C02BFE2E45EA170D6A8212D0C5A1E81290765A5CC17BCA5431D9209FB003E438D7F116AC36680E7A8AF3DD1912A2E4BF02714B4219D836D7D734975464A100479C2572A960EA83DCB97335F853A288C82D8441B04B815D4A9B7797BA44C8EEBFEB30BC9B6B1244432296C47722D24F6FEEDC1C5261EC7759C0DEBBA42DA6737DEA57CAA21277A54AD7A29DC2DA1D28571CC560B746D917B6D2CE12105C1BE5BF599F944A296B344F94632B78A24A28D2EE18C01365CC1C7FA2F40E46C639AD021A3F0D03F0166FDCD8C6846358BAAAE115D94A091E00FF66A2846199480C1519C684324D2B3DAB3922DE136433BF03048D5652DB5274874D8D38419B4F91BBEB2AB8235CBE383767B49D7DA54D8C9FE420B12A04F740D1E9E0F18563A560D69F8460F1D3A423B5B85B4A1C245D673E139CE7FC26313F4B1C8BC78074F092B2D0C1DB0D8B9F3CCC1CD75C43435C5DB0606576F65731977B91EC900F70408D149E8D78810EC8A389923F2EBDADD038AC9B37C00104313F4942EAD62F28CE0B4A9509147B08F8E821677B27977ADA65671BB9DE7B8B231DB2098179F7EA0B04BAD7A8981238546579B34C2522E349CABBE1C03A2D89E2F32D1C4CCF6E719B6D8C9D0BE04496472856E6675E7E5039FFD9EFEA925897A2C6B8ED2E5A61AA4CE011CB755CE322B9A67419D2D04F9AE15B76AE7ABA51AD3901F9EF03FE3CBB4228B6B6F960ACAFAD53AA3C4091999CB5BBEC6B15A3D303970B2B220C917564E3B4AA6B6C0556CDC78B514DBEE9B8642406807962206FEC9423BC502E733BFFA16826F9F6BDA236344F8B85906727535B5996479117CD09ADA1290F3B8B1D5210E2B5E53D14BEBA4D12902F291D8EBD33F7DB4BC64CE2564E630DB477F048C57FB4A21E245A4B36A55C94736E1B6CBB55A0FF83509C07B300DA40884A6416E285FE4B95D9E35D327B2BE8CE00E3308EEC58198ED2CEC4CA4A7B02828DC7628E2B94569FAA58146F2FD2521467522F035DCD50ADF24276DC9DF5EEED528D05CD404F202FE68E5C3C050BD049AA62C03969550F9294A2DE4BC33C2D06951A7AFF1BFC76288B98586D60E44FD08D3A19CBFC8407CD8C20CAB777ACC83EFBBC2155DC5E83C32DBDBD71239CB35E5956F88C7B28C2B670575E9EBFA3E549097B091343A92686C694F3BF3A7725EBBF895B5DF6D2BA04D065FF03E44958207CF8E9488873E8B4B3EE5E362D921B6F5A43E818C46141F2C492872C5ED97424CB205156E7802EA8A21FBE38C57F9657915EDD596E2341CE4ED0C9B61B8BD4712623B5C4EAADCE389C563C052F90C38D48A3B7334B9B4781E43C03B6B8CAFFDE25EDC8880DDFFA499128F4B6A5250A518781811A69F95FEEACC3A3E5F75F61910CE12B3AE24A0FC9BE212CAB8AF3448822FA15195C0BBF5027AAD31DA9B96BBED47FA9E9AE2C4C9D9BDD26294A87997FE04C9E621D3510B6CAD9662EE8ABA7E54A7732237324D4C05C334F6CC28A2A8ED8BBC8509EB63506A21D0E82EDA8D1EA4AE9B37516723BA99C3C18533BB3D4D85C54D7B996F1CEFEEC4895AFCD71FD75878A1F966387E3B0745F416CB5224F38CCDBF4EB1E58756A8017FCF922B008DEA9CD758BEBA1B4513FF5110C5B4E16E842B85FD650A742FD160B3817D822A5866B9DBBCB0B91F12CB2459260558C76B82FD41CC836737C6498801DDDCD6696D2566798D10645BD5E819649CC0BDB7D33E8057E6EF4FCED036FC110C8F072C2FB129E3CD3C05EBDAD4311C42377EC26A9490D4F9062A870923A5830C4F3016F46B9C0B93284C7413895E116658E275CA08603C95FB87CBCCBD0C09A0197538F5A76534BD38DACAC136D080C1E734153D55AC818825BE599059346DE31DAAE41A2008AA6E2FCB6699E85254552815011AEF275B0D8F68FE41D364889F64A3238B25B8E6F85F0454A0AA10E114002CF4D1D3C296313609C316F0F962847C10AAC96654822CF6D4DD85588117427FEF4B35E904A00F7C89DFA7D5EDDD2C89BD31B15351034B62EDE683D212CB15B198F4CE25EC41ADD20E2A699FE50ADD68BF982396C8EB64BD61222AB2720DA6F37F9CE0CB35EC11E0D88A4EC5B0AC95089905DFB9CE855E89C004020E22865C4CF02A4C7A13766E8F41E22C177534B173304D38DE5A972DCC170431947B63C2F4A1C3FA722FC555C1D43A40EC51658DF1987D71F57B0E00FAAC6C4F793832C992DE7D8D032659E0A232B3EC6B81CA38567D79D73506A1723EC720CC00779875C0A5D2B689CC0B6B6FB27896FD3CD81B687CB701AE037F758507490E91C9932AD9A12943AD32A51D7F27DAB47C367977D5C285A3926216E9E98FE93ED97A934A66F4CD76B4ADEDFE105963C4ED68F57E36128925040A1F9717DDE18B30C566D2FCC81D71A4BF8A5EF799ADB1F93EB7DB1B19F4305D8002746672264F0CD597CE0B5D03A53400982A52E6B5167E86193E1CEE60C883623FF06D41724CD9ED35ACB01C38027CD4B97BE287A29D807FB5E8BF002C328257EA436F8079FD8C8AB1F5743596346EC84E3920F3107DB59F80E73EFEC79BD2A9E2465096DABB0DEA7F02F5582D0156F4E7869C0EEC649E3AEA4A4B87D7FB8DB75DA136ED39465EDA380D67BEB89FE9CC6F16FA5543304B193389ED399545F7D6C1ADE806CDB61BD1EBEF3F378DC1C4C773D0DDDFD25F6545A804A345284AB97744E5807D575DAD3CE6DCB4B875F32DD6DF05745EBD3CCC522B4F646AF302BE9097A53AABA893DCF82F9B0CAEA180851C7F9647D6DB511C7F9139012445DCC7F37B5638FB1F9D7D16DC5FAFD9BC6D8B2C73BA47BAF629F6C0A04A4915AAA947DF64B90C2CAD08C6B7513E00D221BCB9DA4EBBDD9DAC3CBC8782E5F3D644CE54DD21C1967647C9D84147E3F7A2B179B6A4108DE40F8679910B3F53F6855A0E9B48E0B24BD860DA6822BB54074D37EBDB797B4DE4E017459F7DA9F750AFFC83A55F212F9E9CF321F2057DFB4525E74AF28FF3E3190F5DCC787F5F3F0A52DFE62E0079FE883A6BDF6C844D42BC9FB4FEBCA44A20962A942F4DB969FA7C5FB64F482A9B0A14A3EF4CB08A1DAD2EAB75D279DB9EB0922800B90A9E9277771C697A317BEE22622ECA4B64EC5896D39EBDE020D116E5D9E7CB265D0D38DEFF54290920E14BEC8B96B380B178985D4E6DB6436D81CBA0159A6B7D6879C925BF7BB40D39A70499D22B71EC812C80355523A02622E84AB35162A0358B2E080D8334BD31ADFD80FE83E92BA8351D12B5DFE69B70775C6A6CBD61FACA9598730DC3B6F993E2502BE0A427728D54F8154C2D35CE0E2BDBB48CD8AF91D9EE3516067680A05A593FFB9DF4C75C3BD9330C4F4BD8FFBB9EF98E741152D3EDFE10A800535BC0514761152CE16CF7BA4FC5DFAC8D8FFA9635795236E1F7BB7E6CECFAEC71AC01B8EF99DF366F2524E120606BDCD130CAF90D29AC6504AFEB26978442164E282ABB92B0ED588907BFD1DE56D6DD1E5302BF034904AC57A469CCE2583E2F070348073D8A2FD3EFDE3211C5F4097E6CDD8ED5A35929962F6835C284DD26CB1985D5E7D543AAAD28B6082CDF34049D66FE9A092357A4DCFB1D4AE8275889F80B1155DA81706D06E6A22D582614A37A405D2B1EC521DC7F506758324A814F560E2E4182BE54EA920D5F3AFFC70534435F956976AE98164C94042F7C026A5611A1E45D5A67F31FB9D3D9A6EB7E70737FD36AACA092F14A6B88B975CE1ECE6BA43AA9703A7517BDFC360C67F903AEF619C23E3A7E1E90BB222ED27215FF45C63BC2AA8A649A45BE1DD8CB315E13694DFACF7513D4FA2E5EC51E0A80332586C7776A7B5241A9C59ED60334AC6D55C92CAF0597ACB6CD20B724FC243A292BC14285389727401853927C3BD105A1534AF94833ABF4780BF00E81CB7AA53C41CAB718D5FFB85F11605B0A1FB8A6DBB2729F91607BE3D2DA2874E0E638A0735DB234D2ACC823F2EC59C4277ECEA080D0F4CA7BEE02EE76AD53D109E294A8D597D9C8C04987C08CEE8D945EF29126AB3267859F1E484D26727E6A7D45028070B8B6B8A579D2F86693E8B43B2960F507C1160C7315619C407F329331EE646592B8CDBFE798F7C13EDA9AF0BA88BC6915B058D1BAA5394E78AE519D3B11C2F194A33CE78276A55092C0B1AC53DA4A921C7DB504EFF918DFA040369CF99A91F086FEDB052AC8D6CAB538CAC374E98EAF4B4B27DD0259F06FA867B069416A996D943695E28995DD344AC75ED058C14750B9880782F826271CD1549B080D218214191384CC4A382742CE1FF33966D1B4437FBD8AD37EA28128F2D02AB0E8AB544DF0C5A21D04B71E345B8625794F19ED0C0573A3516F82D9CB309973587C82D8F399425353DDC9C7131EFA56587DB1B26628A733B19EFD9044298FD6DBDA2D3EB1EE933B018015A33935424F0FC7035B17754D85AB34B796C453D37981030C656DFA2D5BC8C937B3BE42686925C1E4279300E9D83F9B38DF4A6C043943C3F5A19E6810F3FE965BF15D8D10D413F1F2D89D274D76B49ACD7EFE2BF214CA8530DF268C5B189A815ED0779E730D0A270B3C70CEBE84A31C2DF980AB754D76D39FC08A2B0B1F2E63A74BABE092D2945E1A636978F85A45BB0A219B898C31CD0AC11A71075B249A64B57929044FAD4DC9831CDD424847EE00D7D55F9583D3ECC05FEB8DD09C176AF7A5B1B0E65B2D9059C71F6D69022CE65EEB1D2EB62C4026665F8421DFF714D8F085DF6F3E96D91DF68B8BD5AA4A3C5481CB84F57507E0D714E047BF79242BE5ACB9B7A556AEB1D3E08236A7035506BE1EFD45FBECB192A82B360DE06AB9A03D328F3AB0F84B41C7B761B707A01C0D12D0B27919515E74E76AAFF0987669459799D6B31077E7608816A74A5198FD482BDE7459B0413A03E5AFA29868D3687EF1B35004F837DB972A62CA536534148A0318D400E55B08E79EDEEAFC0F11D3526A0C7E66230050FFDF59B4B938D8E2C09BB1AA4C25C27B702973055927BAF0DF2BF32C4804B04BBD5509D360DB326AA43812A70FE10F45965473167EBDB5F9A2CA8FEB937D1AE342C19B4B7B68DFF34FF1B30F215DBB5F1786A962E7F30D4890753C30CE725C67373404BB4016DC3FB76B198C06467EDEFB4E6DEFD45217477CEB624798875E698DC955903CA6AD85767DC4FF4A8A33F2CCD4B81CEBDA209CC43161FDAC42BD7C5A54163A493BA3B36B49CA04EEEB78EED90E9A4603A8BD94039EDBE05A08CF4072A3EAB3691D01571ACBC5997E242BA011CC066B93302035177F574A92CE7F20E348391A8B9AAAEF7A2D7623EDD5035AE96EAED939B288BBBE665E13FC7B8490FFB37E818F150EC095301E1BEBEEAB659343C6176F6ED451C9B82F8A5B401F98495212993AA6AEAF760E381E044263EDF0B9541C03CDF3024CA9283DC484DEDD4E226E4FDCAC1EE76FC9D55271210AE1E5E41F3A1D6DBAA7A1EF35598E0E795B60B93F1F3459D20FE36189CE7594E18202B64C3DE03F868D31A2A9AEBBF258E9985AF752A31D982221FF6993FAFC4A4B8DF15BE6F811B6580BE22CDC8BF40F4D0574CACFE07F736E9D344F9697926F6E7F25A2BB4F6E6E531AE7D2267C3566718601366A6AC97130BCAB555664C14FEE189901E275854E4BDF258961A02009E29AB457D7708807711818409DAA4D889449725676280135EDD8A6E2623A8948FD390FA3ED6589606F1F29F3AB7326F2C197B3EAF92D42E23EA1CD60D0F726AB0BC3B3C4AD0F1058347B83C90A4FD7AB4987D988E5851CB10DACA30DBDCDDBF2D83433731E607A9B10A97005DE408ED1AF8B2EACE9FEDF3F46D235279A20F1B4715209991177266834B387C4C4538332C43D3EBF725D44E1DD44FA44D8BF5EA86E44595173531EDE3A910A3EE0BC8431170DD6F8D95FF8DAB37CFC305561093EE2687148949A409B26081F6B28085EC9855FA5D1FE4110DF8CFDFBB3FE44E65A0C32373F760990BCEFC539EA3F75C8AF30EA853BBB056119A7F752255720606FF25EE0F0A78FA41A9B60D76F6AB8D6DF1342D45EB4D8A8ED22DECF2A242C2DC73494A30654EBF205493A0463E293F86721765174E794F4956535ECA994ED44AFD4C9F3532FF85574C7CBF3A96E0E231F78541768CBF28137F7C30EBA50B9D13CE3727215B370CB48EF1D6778CCEC5863F04D53662A281246DFB6D3EB3134822F44C095C672136CDBCA4190EE3960EECBBE3FEEE2E9160D7938E429105C958A700BA0E9BBB81693DF303C8C37A15B3B08D72080CBB48DD02E8C4DDD15F29D18568932397479B5FC3EFBA370B93489C715FA1BBD64EC03B235155371B4559F72A0F5C0C16871352484957CCF76C2F571FE0222A5754E0E174AAABAA99E4A91BBCB1A43AEB687C948AF80FB63034E5CFFCEE1BC46015A9CF494774921B9D4B58FF8701DD3B2B463AA4834000232929E83DC648E43CA8A3BD8EE06631A22017CAF933E0591D239544D3576D03D337AFB07916B33367448BE0E11E00D170199211861337D3DAA2FFF0F9B029C88984DC8EB1D74B34329EDD3007BAFB1C138FF5FC1FF688AFC66187C44BE6D14575939BDD3B14606516B2080FC2FD3561674BD6CBF59E5B246117A2123D61E63AADA02E3C1C6367EA5AEA9B7B1BE9C76873C42632217FE5A3D0B740A3FFBA50B12D23F88966A9258E7EF84ADB422CDBA2BAA1CC8D13145EA62994E642F1E6E3A4C5DDA7F78C810A40523A477F250F38B6F93189E25F168FD41A840B71EA78C8C57870EA7BE2D4D444842CA4A2E9792B3506D0BAB9AB5B15B0588ACB60C68E6DF6DC30F951453E87B3EB353D74C6E75FE6E5D200576EA5F671C12FB96988BF79C1A06B8A486010070627295EA2CCD6E012F3541F40500E61677959624D45AEDC4AD70A4BA7FA523B04125065D00C6FCB247DFE18650951CBEC12849B9B151A578DF96B1297FB5D110CB929921CFA74EB14F577679C3CD6B5EAA246050B5FF52EA46D6045F884C09BE49B553BBF3B6E321790FA9043BFC4ED3E6396589642AC61F5E9CE8A2D8C7B0F85BB26602A151880FADD47B8C1EA1772F88DD1739346570BFB7F50D8BC68105F48BB7B5B31BD7B0FD72A9EEE4DA8E146C18AF33DEBC4CB0117581EA624EFAB98CC9560A230D8FDEBCBBC385F476E631A2675C86701DCF4A85EAC342FA947CC0F89E65DB1C207BB89F2F41ADF6D29AD4B87C7EEF1FF347F294623AAEC62F0E0E69CA588F5BAEBE11467C79C80053ACC0899961F892D68B412DC189DA31B22508F75201A393147E2A7501B59B4465B41E3C7C50FB37F5606B6DEBA3D9D92322FC17BE3DC471E93CEA94A9EDF341F93CC19B630D64BE4CC72526838CC00F02D45305078471375F8139D11F37FF1A86EEFAABA2DCBD8627F3AC2F651403230DF9DCA8F88A20E3D8746810838CB1E8F69A4A86BFCDF8DF7D5B8F287859ABB791CD5DC110F5A4C175AA799BAC2AE1B72C147910DB94E05BCAED9EAD4BA68E4210143B6543EDFD7FAB161B9029B3CD0C08DADF06ABBBDBDD22A1BF1E654FE7914A935F7936EA91FB1B24CCB762130D7A0BF3FD2337D8E56F9D724FD5E9089467CD8DB3272DC90FF60E06DF812B694F72779816F9B7B52EFDBA8D3246D09001EAA5E820B195750222FDB4E856D69FECCFB92075EAF34A5FFE4F58AC3FEC0DAAE65DA621083908A2FD831A3417A36E5656BD2E448B865F90C6F059CA4862853D31CC8BB52CF699FE0DBC0C3A9411404819910EC5293F676212D317A227166EF9BDBE4567FB77E765724783F2F86263EE9280D0A76C42FB8724773D046C1C3581AEF4AD07FFE2D8FA4 diff --git a/src/libutil-tests/data/local-keys/mldsa87-signature.hex b/src/libutil-tests/data/local-keys/mldsa87-signature.hex new file mode 100644 index 000000000000..dae55effb425 --- /dev/null +++ b/src/libutil-tests/data/local-keys/mldsa87-signature.hex @@ -0,0 +1 @@ +498BF0D1C00BD3E08E30F3C5C821D05A9EA314D15D7D3E884BF4EB919678D7D374ECBF941685894437B23AA18C6E4301082682DF82C9C9B8F78D53A22DAB782FBE141AF50382453D9F0DECA046CA5736E697BDCEB35EC881A61536E2D9682ED4BC7D265A9284A3CEA87EC7B9C4E5E2AD1ED8C4CCC32A7FC7FE47E449B348ABCAF204F590B8AC169FA50E8F7C91A8224EA5E0151234EFD010E0AC05B0133157B63F496BB9D54A6295E43F3826E12F4DCA1D4D1687A74CD2CBC7719282F595035954DF66FAA4E4D2E70BEDE9EE7856DA2AAACFA52E8723056C9D7EA1C0A38C19FACC17884796F0A04CBE1B3E3B5B1B6731EC8ED97474F8B08EFC3930D129CCA86ACF53033563DF7DC73DE741D7C17658D976168E02A56BCB47F3FB985FCF81CEB18E942D29A50A9E15C85EC5550432CA6A45FD25C50410B3CC0326F6DA67F002B90792876C2AF68FF552D6B863EE886B18800221C20F8CBF9A680795280F75F6E902C5E02662A59C8BCF16F129B45A35841D3032B73089736D6A36EE52E99EEC41179E9C8424526E868B1949DD225DF5DAC7994509CB61C0222624F7271D32317DECBFB5597894C082B5EA871F3E706B5BE56620A23D749F2E17060152A6703DAC4DF0D38B29C6704DFDA39773AA2A44D76A4FE662CDA0ACEAAD08F48C217AED4ACAEBD85E2B36D6CCB388AF7B534CED6F8D36340803763E9829FAD3702D8908BC321FA918F96448E5019CD264635E980D0997995CCA04AB302B3F93B25C5F28102D9DE2B285C20D1EF03ACFA7FE4447008A825980F6921969663F4F60285416D44F1BE49BBD64BEF4115AFF8CD4F1E5A2755ACF501CD7CA42F4916EEB6C048C781F12D8446AD13AC4E9955A4D1CB179893D0029F2DBC7131961ED1658ABCD743E0213FE9D45DE2BE92F324A7A797104E8F71C2DCBCD80ADE49C5CF298609F191D037B3883BBACF25CB0A11BCC29DB759DAE8B184B458F039AFF147397C843BB7B4774C7480E20DB4F5319D4C041C0CB618A9AE2E81A3EAA9DCE05F41698456068BFF39674445EAACDF1097F384E336207262A0F6B98879753D3FCCA7F580FB189D580B6A58A1E009CF4F47EB042E4387976C63079BB2A2A2B96F09895AF7227B3D2F3601467B7F70ACEAB6A0FFB04C103E0DEE7E1B59043503B42935D6A8433EAB5B05C03C7AED59A5BC50EC9BA81395D102C6C66B52C34A36301187404D84E809AFF4B8995B9D2129B06683E2E4D7C4591ED5050B465E894A4654F38C3CAEC3D9511BEF9341E14430DF8FEC166616E5D6E124CD6BF515CB280825F1682C0066743D71821DDD4FE579E20657A78E521CAA1012D15D819D3258F4CA24761E7F12EF47C364DF9CA35DE68FEF127AD7D9BD516CA65631AD36BB199CB0ECCE0EBEFADE1F7AF73B7DB3D505D5A27265846251C0642842C19002D28029EB9B226C8B1F4895B19E5B2DCF33E6D3C52F654DBF96D4448C43842DC6FB84C9616B7C6461D6761EE2B8F835A03328C31B42A562AABD28134FEF86541B45579E796A7BB9C4B859DA4693D613AB8B35CEB11E45DF87AC75F281E252B57C2983427F8E1EBBFE8BF38F9F45B1CEF693EF1101098565EC4A8AD55C6B3FEF7621E75A0EAAA2B3E85721213BC722ACBC3BEB34C6FAFCD2EDA4327EB7799C0F62CB308ED84C49913C57BC2A3FAF8AE4957E565370182323441150F1BFB0B87AEBF854BD37DE88CA56D3D0695474C3E4CC3E445C826893B2B5C25045F8B167B90C00A3E03329A8FBB78249BA5CA7AE2B3013D47AC7C109395CF587B3E4BAF8537721FBDF6CD646EFD72DDBC8201700FF85DEBA50EAB1F14C921A6202AD61CE745A5D0F61B542FB920872EFC2113A606BD466963AE97D9A2267860BDDE0F211025ACC6D5FEF6B5222C52662DD39BDB8B07B7EA773D8AE67E2B760C6BF8DCA4443AFC97650E7B27276213E1A347E2CA3AC65FF9BF066F94B644110D6694F79CF39C35A07C4A3AAA6C0DCFD437A01B3BB6D6AE5A78D727E3E73E8BFE8A75CDC394DB20B2787936964264029E471CEA7D27A72E6B68116620D0166B92B41F17417ADA6CD5A5B504DDA85171D233F77E0EC122012CA3A48B896151B14F379D83B87DDE3F9E0795EC5F11CB81E851868D24375E4E3961068C300ABCAC8F41C52024E0504AC773EC380CBFF6D15E8E85C0E49FC5925CDCF36765B2DE261E77C6845B7E0A4B3325C21B6FFF601EA96B25C09BCF4ADBE5E90928AC3343499018384A0ACA8879584A7B6D2206252F4FD0B6A93965B4609AD39553ED25C0CBE1CA120563B446F86D80865B9E086421595E05D9E341065654767A89C058F50E7E3685E1F6F42E4CC06E6AA987C695F077CEBDB8FC8694B116D38D992FD41E7A885DD9F3D0396007914761DAD88C5496F42001BD71E1BD2DC16E62375FB41AB58E8227FA950DD27742C30BF8B5C6D3BD7A45AD34F15C84271D5BB977EF5F65AACB899E840A8AA2139E8DAE9927A8E48CA797907377AE804B6123100802B2D4DE0A14B347D4782E1EAE1148899F56090541C8E5A82EB16F7284B60F73E33627E6C80936601C127C7414779BAB955A4D8B296D6AD9592DBCBC53DD2A22CBF3DB50269068068977CA89A11C33B36436D79D25D8184DC15EA778D417E4751346E57D05F509340733E36D8F93F9BC0DBA9B2268AD5131E2CC83D054F53661D79815001D29F07291B78816F9D632D8AB9306A38D2B972FE98D87DA541047FDA61B5D2B4317771CD92F418F53B50EEF4105639A1AF77204C3FD69B7DC6090F023D9A3810AACE73292BEFAB0A6C4D4B3C8A164C42D4C640290DF6CE262630E30DC31CF582338FDC1FECA6ACE5BAEE7994E00E8867D0B04A8854CBC4AD6D892FCC92F17ECB656AE303D0635DB7AC21CFA5A43B4012C7B7517A318E71E32B062369932FB42BE1A5B787CB78BFA7F00FDF08ED533758395DA85DB640DBE59375842E4072B76DC4736A6C059C3712A0760269F35461FB1341C4B9BD4C16E3F5D5D700601038BE8BBD76165C9CE5A4B0E7AF28224B6182E96B53432CBCE911920DB003D669BC6CEDB4F829E5FA4EA292D3EF16F6E876C8378953455969F45116D0D1EF4FCC986D8B0424BECCA52B09FFE4267C28148BD5F4462BCC70686BE01E159335026BC5FEDD3D8329C5E04D2FFECA97FB43DC88E9A68830CEEDC28E90B1E77ADBCC3C8408DCFF1A5C0CBC62418FAC212EF8C5A71054281F42F01070E03DD5456ECB652AD63AC161D4D1F139CB05DBB9FF18A2B2B3DE2B3C2CF4328D72D9BEDE401C6F3C8CC28816230B9D2CAB2DBAFE17639D180988859C2F6596FA0E766509FD3C6896B85F63FCC68E6096550B2AD180CC192F984BC626B08714627CC7E14FA206BA04C83AD39F0CD7B7F0CBA64E3A08219A5669256152BCABDA7CE13E684D1A79AC6F36668B5EDAF7D1BF16C325AD97893233925F2361C8C01EE280FE1EDD1D46E34ADE090E90A36C47E1CAE686C157C043090B90AE6943AFE03DB5257C1715EBBE2EEAADDC8F9EE827ECFD6E7FE3B5DE7DCBF14D817727DD15343F46C4B5861A30BB61F16F0D634E86371619C5072B01D0AD6693D2CB3AF5CF947346DB3BC9BE78FD945ACD95A67CDA9D3F2B534291028B47C5D0822A358D7D764C4F0479E72503AF9990CA52C61B478F6AD722BB3AE8BDC6BA8164A1F5AB53B2D91DB5512E61D2AF786EBD43DE2A6B66E0B7B2132853CB5C979BA9C1411A41CFBE06494DF2E9A2CC436B32C1F66E969EC597586E9CE2BD40FDFD663B28283C31776E2FB1F08664546DBEABED11B7F65CBCC66A952BC4780682DC0EF4A54E750D74098EB0CF0CD478D8B4EE87F2D4F7965605FA9A6DEF319F6310DC5183D01C755B08EA90E7F1065060BA0F2C978ED1358CD283E58B04B152A6ACEB32D9DB23EDDF635AFB394934353209A5AEC91D15E191FDC13CE670A85F90DC9F829F35E5C015C956AD8EA9DBFC2BE08A683F7970A5E4486835114836784761BBFB52BEE5E1A7EC5A264F8422714BC9EB5419644997456ECAC88C1FFD74712DCDFD7630CFF3129FED31F5D47E7CDBB96F340E836D76760D156CD470DF23C7925AB20B9EB3A91C25F138D281C5E07B9F5A6162F18A68DB9535A2B267523F3A33437EF707FB6A7F47E98CD72DEBB8960D6EF14C0E621C26120B83D0C809F4A5AB916BE6B602C298C119E09E20B7A85254FF871AA0ABDD91B08E0D61535F66DD5C3161205614105FDC784E4D43C5239C3F22B20922FBE4FF4A5EB483030D2BCA541B2D6E0BB8A7C6C253E7C3EC206B710BA5210C641174691455114A4AA212C27BF9E112F606880C9BEA27F9B1A2697943101DDCBE2E976A5F9148375B05378DEB1F84DFAEB82547F586B4AB0E5E019E706A3FC8B22A383D93EFCBA6AC8B974303BA8BCE73C760E61707859E330510FE4D63F6962F82BD3E7E45C86DC47DE8307491BE6066A0946C9B98818988F2DFB09558A1D3B446C3E1A0BE607D4595695419851DB07B1C22A227DC38453DA069AE706F5E9230E533EEC4DDF81E5D1A3C3E6843413774983CDBE9F3B739B8688156B1C01B5C9EC8E94AFDDE6A0EA73FE2D22C5707420A5BBE456CF6A009D72CA38A313A51FFF3A36E5C47DB402DF99BEFA103D7929DD62A0341EC0DF047520174DEF2B52F7C512135768424B6771621C9FC6075E3B6F0B5EAC5C2249E1AA539A132976D6FC6FEE7CDFBFA4FB895E4ACCFD73CD46C675E7A1937FCD3ADB71A6661BE852EEF1A7C7A0140D9877A7A8B50CC324DBA01970DADAF33A8A6D73BF6AD616D61D334695F2DE5CC906D0B680AABDE20AE71B40FF574B2E61EEB3912E630954FFA8B848D00209775FC2B75667A6BBCDD47527191D2C94B4DE2DC97AA830E0B9B52B8ABF4782B69D4061552532F81780947AC2AF25781E78913B044CFD99ABBC6E58F535372F71612CEAA4C51A354BDC40E628B9AD51056DC092CCF291711B0A5850501BB3F40B08E6209563EBA84426B31AF10CCD14E898E9CE2CF8E4BD136578DB7271B6D8442C8F984A76F2BB4BE488837802B2466A277CDC0FF7A24DBCD44D5D74C9EA3E6E20A46D4A4B1A5CD2A3B9975297B22855F4EA6F0B58495819F5708B336AF7CA7C7213887033289535FB15C55D72AC3AE733F0C81DB13E0196932E65CDDF28A5BFAD29BF07ABDC4E78746E79EA62A471B94E127C3CC65C19054A71B75ED7B4E747836FD2091A6A544814CD900D97EAD77DBAF2632E209969DFC8ABF87809A745A293F37C283A51A999A9ADD9A06B355A78C86696F193C4729FB27ABB82C6B4D93FC423FA6402E54F260F12A71EDB0A173D4F31A3F4BFD920AC4102C05BBFA8BCAEEF1325AE16E892C88DFE282193FD506716EE250AB3D9C1A73FBBEE0580436666CAF38D49654D5594182F3B6FF32DDFB0C422CC15A213DE17096AE31A7E8D90F37E89518DC81BFF8E51EA45F46F6CA305909BCB01E7E8171928E427F29B874BD875E74C20B3BC2C2813B0891B7AB403CD1951EC46E23431766D4103C8E3BD643C01694D31C3295AA7133816CD90F77894ECC0D5B901F09EB317722FF17F340AA499A29DB5950545E952317F6051EFEE982DBBCC64F95A207C1A99BF0D2F55F5C5CB58A37B0B31ACBE68D5874AA7CD561FBF66654690A5C0C6FC8AC84B9C0DA8BF6549F2579045E344EA44FF328BBEA1DB2E5F415D3541EFA8E525AD3C8E9C30314065AFB82BFA769FE257CB1CD369500D646C891A6BFB03B6D69833AB72A8F6961F6FDF3EA5D9B98558DFE833554D8D41C0E79236C8F0E352EEAE6A99EA0BAB4F52DA7A92B4698C107DAB1521CFF4FFFCB307E4BE89F56028B349FEA9D36B0995BA1A8B495D03DF7BD9D20B5DACB4D674B46A9BB9461FEEE556E2940C61C709F4EABE8A0206D728FE96C76B1F4EDD5F94625CA104F11A01C87B8ED83DB443D17103925BFC3F77C609C92C441222AC717E19BF1065A159D87328437BA007AB4FA262145610917D4481CA5BAA94F49965C67E08A62EE454FAB3F7F1067EC27D644C19AB16ABF4FA68DA102D185DE15E474C4A004BC52CC7E485CA9BE16B384AD41B31550155718D586AA112B4E2B6FCC1FB9A6765A411AFBFADFACD46055B4B2D1614267157F86A0CC0FC1561FE52DBDD2716C7D394AEE756053075E89A6E2CDBE5BAC1620C53EF1060710F20AFBFB1BDA04DEDC016DF8282D35AE42B06DBB7B618BB13D7401504F2F964DF8DF55919EA5C35710808CDA952F6A9DDD9FE392433EB672BC2506C8F2AAFC88D3DEDCC103C1815CEF8B36EA06CB38B9DBAB6417A7729555117A9CC6D0803814A7AAC5C48D8204053EC837F71AFB68E355B5AF17D57CE929931FD098FB473440BCAAFD22681238C603A3C063ACAEB2542BE9C8A0608EB280762A6D930D354892ED0E9370683EB3177DE3CC8360B5671D86098D652FCF6641B13B551EDE758CC8306083A6B94596C6F83A2BECF2079BCBDC3D4ECFA8DC3E154AFC1C92236979FBE0E1B29304A62828B92EAF2F3F5002FB6E00000000000000000000000000000000000000000000000000000050C14171B202D31 diff --git a/src/libutil-tests/data/local-keys/mldsa87-sk.hex b/src/libutil-tests/data/local-keys/mldsa87-sk.hex new file mode 100644 index 000000000000..025cf4d1626c --- /dev/null +++ b/src/libutil-tests/data/local-keys/mldsa87-sk.hex @@ -0,0 +1 @@ +2AD180C0AD68E081C265C682EE82AD88A5D0F5EB9F5B9DFDF387B175B214F297965468E54AB2B99C74A344B633B1102E5BA80B6372BD5430A208EAC14C5F822A91809D572C4415D402BF56BFFE22FB9FD0A2D59079D0DB408DCABE82ED2FEE0D12E26A27F4B784E56BA8AD02474DD75DB5CB35F79E1862DB95F2602B5C827E2358102E9A0444020592039685DC2812E03442202901A29424E210905B044013C4818C400098A82588320214882104492901264064246C4CB22493A68D02280D98268A80482413228D1C330200C50D5B048210B849C83421510641128561900408A09850D20428C002258A4441D812224842091CA190A1348D114861DC4600838261C1A28813244818116C8A020508B0101009698C06201C044AC4284E10808582A08D613209E2322221346060C41062242E1191218CC26C14C31009212940A2501C134259902459C48C9BA26882A6890BB50023408DE442860A982D04878150428658486449A68901405013B1690221819A0872D4A0495BC249183061C3A009D2360E13004813152424926159964983146412C68D1034308C80498B926D9482801CB36854082A514466618831DBA80481404ADC1412108841C2428A4AA009038071C846204A484D43C625C11885DB808411C49121206E194285C216901B090CD04442DBB091E0360A4BB408D43262584831C2C200188810C186915B2229A3142C0929028B3892149561032468E4006900263050A68CC904840C01660916424BA4311CA28009040861B28904406A0905292143250CB548138501E100228B924058864558068682128681304ED1C224DB2644218705620660E390884A00294232291C4012C2800951B8098B200D88428454424252228C4C38201291295038454A3452D13090E2A64DA20285511880A11604A2968858B28CDB946D04A8710A34614A222158420941442AA036611A00525238611C42841215480BB60150364D02C005CC124E841869211729E4C860A2A6850A278A20154A8B38459B2802409801588631E126620B386A1CB08C24C98504164CA496681A46898336014A489001B32464020CCA828050C62DC008911AA41021214D03410C214582A3364EA12672E2C24D1C130E23A8615408490BB09190266D92866C1034460B9584CA0421D224709CC83192064E82047148440193124963368C53206ACCB28811A3051140261A428E83144C503442842250438401430600DA94441826411922620CA86D98C8285C820011C151814032D22430D88488C2322EC92472DA2471DCA64CC3128500824861A01094162DDCA46D9C8825C34242C446082049200088084330488B9288C9922804966900B39014126022B871C8A82023B2294A880DE31029C9A21051820103806054866DD9926C8A2852E0382A83402514A15001080119B48522924C21C4712448250193001BB18008A04421084C4B0869A4A04549A64089225210118D08228DC828825384084A16844326910991040A820C43325214C670182121583849E192610A452A54284018140E8B4451A4A29149204D9A00928C06319B1080C81812608644583442C01452E1328C0C9561D3442A1AC844504422CB14628316421C184201C82551206A4C420612370900158189C4252294684C848D52042689A66944482924A468840471224071A4B20C52329208882541B01124A20DC9042E50B444534662141722E1888D43860883224112836881104588A64C4C962C5C1825040290131642008828E2A8204B1244124428C9A451D812459B18601411114B1082E2467081124A51148824A76C8A066A93468013A6454448880CC00DD1088A52842054C444C1886599001222428910868C10A84D89B410D32262411010C18644214325C9342013122D94C244C2B82548A00064104C981682DC928D50182D01426CA21090048800A294888AC444108069C3148A1B410A0A0548A4B2480C1721A294700B8040841045D9B2100B425011A360400002633424603224C036228B808D5B182E0B134610B069219345203982D23860C44080143285533068E22882A2C621D9C4250246901A470420C2680AB450D4268611425222A46C434624C4102624938411218662268100246E1A2806184820193151D02885E1946CE4C20519C4711B097158464DA32652DB205012246923A161CC18021A0451E11408C9042514933150006EE9D68E6DF390D82306189DF75CCA2FF9A73B51414FE2658B006D3276473B8441F4F0D545B017BB0F51517801048D8ADEDAEA6BF0EC994F1D1F168F739BB0039AD8ADDA327812A1766E400A870DFE6AE610F994520C2A92336E0EB53ADF88644A81F7BF76FBE33072542FF20C31D771D24AD0591C622DC98C3E407243EC3FD9DD453455719A174C9E682638D69555B4A4B907AE087A7A5C414D6ED6472E6AAAE648F973C41BD28D76B3464413D52C83B1111C67564D38669A925A085D322D6DF6F4153497CBABF5DE5E11082974940012071399CEE5176695EC6774A0D37253724ECAB2B1D26B6A620CCD4ACF66C5C5C18FC7F92DCA1E707655CDC6A89AF423AFE8B39C6B8AED86B1501E1BFBE6ACA628938453AD1E97C4FD567E74DB3842EB5D57D3C2FCF462FC9DBC1CD7E52BA8D87CE7648DADF669389C2D9481E3658BE8F6B41FFB290C89FF9F105F945428F15F5962B83A33BDE918C3469F868D9CF19A3E76178C7A670333A248B42B99AB01CDC231EC6C3AFDC4CD971CBD5AB98756CFC6D11D6C5B766A07381D7305A396D075A0593E89C2353FCA87F86C25E1C48AFD08D0218A4F72C4A075B5E0097CF88ED15D40F25C3C421B3D92C902CA6C92323A97257CB0F9ED3FFF0898DCFAE4D14A17ECE76560540251486EC22F9A00821B7E956ABA4CF7820B9C7F488359CA55752E30EB4590FCB080A22F7CD0E4DB92FD2283398D13B796C5B55EBF3CA0660E1D04C2F93932D258885579FACE2358416EDB0DEE92E7AB59BDFE1F0D12191AC9D1A6A7CF7EFE2FF7B2B23864C802B4A5D0C696779C79D89D058CC9FAF9F9473363DD044B176C22C5EC719A692CBC6A97C13E270CDB5004F20ABBBE75809531B13A581BEDFF6E3BAFB588FF361DAF136B68B440C7B536AE965462F7C0CD0B26C4BD623C36D3BC247E1D49A58769DCB670716F657DA75986BCB1F630311478E54C9AD70C963FDBAFECD1419A15DD4D938A7AAE6E0365B89214316BD81BF28EB646B52F1998C93AC601EEDA721805B57B52790DAAF4DB77351CF4AC2014BD692AF13295B18E307C113F3845BA8EEE88B1A52ED5F6E6E6376807951879D5023E1C41038D023FA8BACAF3704C67A037F768E9755DEDDC61E66DD7A9500B924832E5B5732B40B4CAEB2C34EDFABC3E3EDC9B59A201EAA5441FBE3D8BEF3D94DEDAB0069BBF2D4E769CEB551D1A0A99B13E06D2F2761CC735A4F787E2655C704261D7167D349C60F006979F2BA2E1F5FB65F5F08B2708D94FE3F6871824EBFF2F44D2228381BB7F3ECA69514D1D491C05919A246459E70E9FC94645A05A287234EBC4DF76B6352978A53CE0A0A6B63289A19D6A25069B3A87E578423A033F4906A44B3BEC4202CBD3D528DA02737A897D71E9AD9823D7CC28AA1DB9CA645357F1B07D66C7934C7CF9B64F0B85EECB0C7D3FCABA571868EE83C9B53D7A3277029AFEF9899E09B9515BDEF4BC0FBFB976BEEFC2CE629EA713529ED924EBFD106495062972EAE40E2610FBEA65BA118125C05186CFDAB0EC01C384669323EE4A4BE2CF005438FAA7EDF4EB22CFDA6E14B9F7E03282E3520CD1B5F0A15E47DEB218A747B15DC034B1CAA665DFA619788D2D07E6C58AD120451F67D6E3F310FE44854BD5B035E2A101D4164EEC6EA42AECC9D9A50EC472BDBA4634E2E96C51380037FB2E58731D1B237486C132E15813A5C04525EA29B71F849D8F910FC332BF7839FF602C100CA2CDCD96163B31E3528444C82713FBAD64F990FE33EACA8B8DCADA5B1990BEDED44661FA75543D7121671BCD069B43252398F897D39E2A04B9FEA4926AD773F38EEA1E9C5B23E2656D936453F6ED418375D52EA7B01D56654DC84237D63DA2FB941AE06DA734A9BE27FAC9892C33F284B39BB987F9C41F4394ED236BB930F0973CEBD732C441617C8EB95FBAF7CCF0D894D9A3194579EE485573498BD80B649F50241F80A33119214F0DDF2089A56DC42FE1A90A5D82AC13ACF5418305C1DC6D76A2AFB7903F1E1278EAD84C11652B2BD28B5DBD48E7A081170A3918C2F8DCFA038B67BC99A8B396CBAD4855A9916BA52487CE1995F77F2755A70E398DC65E752BF5D0F888F54A255413824AF9EA976B5CCF4102964C2E0BD0ABC69B834E6BB4BBE32040BA829CEFA2557357E2019BB87AE7A31C42CBC8982BCDB23F2C37D695E52CE1B4728729C216A5A348EBDD01D2CB07D4B30F16303C7AEC30D16AE99C7F9611CF2CC209106D5B4788CDC08A5C98374E56FF3C52A30D7D24C67E2402519CAC5514498372CD81D0D3D6A9B859E58CA3C0DFF22A2CBAE5E321FFA103207400C9C31F6C8E2D9888602EAD5AAD25600A32B9A15761EA5325D10273972430B829785E24754BA2684FFD15028C6F1CCD6F19A3B2F9C32F85F8AEEF72F11E79658CEF99A6A4F091732384D3BAB63EA1202D9D3A7DEA2B2616E9DA4F1660EA80EB50A24D9B07BC060639562C561D070E803211CD93AB9E7D5A07357E3E539CA273FE994B55C869389A84BBC56E8EB708DFF5CAE5F07FA84F5073FD9B777E1A193849F8009B83ACA1C6C97998E6C78D297C708980590BF7C3EDA267C98C0795443E3BEF8BE2308AF6D11FADDE82A1E1B403043A268548673485B2414D8B762D278E75080DD09F4969724DE3FDD869A2F1E2B9A37197C0DFCE2144705E9FD91CD1BF5FF282A454836F35DAEEE2E3D1ED3B0953310A2DBB5FE108AA7E7E1F28A1A3FAC39E042A0FF48C5109628FD968B05A9B20813ADD8AA75585438F2183CAA29F0DC91D4D040669602D0FE19DFB2B9E353E940BD0C74C63BED2467B8630AC1C301E71837544ED17944673CBBCD68E584C60F326B059FAF64C7FAB1AC2DF7B2766FCDD9186755D018B5C19B39673B0293264B877ACE5C123374A680858ACB45AE9475B1B8D95D94A6534B86CBB74813C50E8C30B8B779A5F5225F823FD02368F767AB95B5986F39289E033F45B6742784F39131D96E5D8CDBF225FF6C03684086485B96C839CCBB285BFC1D631CEB87840AE9245440DF46D21CD77E7CFB0124012C1C4875ACE1C4290101074FD7B2CAA25AB4AF68C28C5D48797931CFFE46DDCCDD11C776A4AA0C20B27B7A24107B27184DD3D26F7300B5078572C6F2C724CD0B504C186AE19023D825A59E968D4D4F66E340F2894D603DE58E3A5C5293D6DAB9AD182FB5BC14888CCD698113AF3A5D7039FA7E48FEFFD6942129BCB436026E486EE9AC994AD28CE8D885469A5F7B627A2E62AF68C402518396D9D17A738D0DBFA2F2DB6FD48534B8516F4D01C3A21ED023775E6FB962B0E5322A224E5D86275188D1A59F43749BD3793294F9643A0C08EA08E26351FDD474F5C8B036621D512AF88B169EAA6BB87949E50854E05B78363852E1172776FD88B32832988985564983E120B2DFD06E98541F26E51837C821628645F2AABCB3CBE9DE9AE585434DA2947CD077A243DC43BD4DA2F940B89818FE193D1754EBFDB454702FC9235F46BE9DBE0103A4BD0981AF64EB4A695EDBC4204C1FD2C2C3530C994280A2DD6B9011DA7CB68BABC5C0DBF1222BDE403770CEA1283E7B87764555A9841E86202A6657E5E6948F0A473D8AD902BCE510D5B15086BD34A1BBCB5C3EDB7E0045F1D6D377A6060358917C641DCA7921C761F359A97AB86CD00E6077B4519CC802600A8D17FCD32227D043D78DCD766A4F068F9CF2F58AE5FF58F96721F72FE917366C7B08E98081A7F4C87A669E17E9E0AD0C2C8E9FD6D07496D7FBCD47760EEEEA5F2685A2C98BB211212411CB5D4318CA1B741C8DFA0062E5375C9F199507748C6A76C945718D460710C80FD16847C35373E034D09DFE39ED579AA31F72CC4583D98E37E5D35394848C8BE777CCA9C0B68F452A916FA68A23DFE78A20D4E16EF31EA633468F58CB1280C619721F94C34F5241F15A04AB105F9764F117E71CCFA265387EA47B537150692FCADB6DF1CB0C3F7CB725BE932F0891B343FCB2EDDB7C24F77F296C69541B070E58BCEEE9C08D0EED5FAEDFADF39061C4CB9559E2686D846EF43E19EE920579D70944003BBA729DBCC0197A46ABD9C1B86AF9344B6B5800D573E7FA3CF0C4C20D4E69BE26C902E008CF068C34DC855AFFEAE85F41DD70D003930DECC1064BA8D6459B1CC17BBD1942E733B4B293AA6B3B081BC97B735BF213BB838CE199690E4E876990C0335D05E41D7DE02E6EC54E99E80523B3DC2E3D4AE904D74FAB55D6CA0E31E62BC54885BB6CD9BD81CDCF263F48BA7648427F8A954BD3144E670B0618AEAD6F81E093B35435021F07161926ED9E664D7805344A042BF07DE6DBC837A41D27A4C81C36EB16AB31BB16EA0919DEF73091FF7BD8A6C89250E2C980A5F4B9AF2DFDD66EC23D60F1BCEA538801451C3AAAEC208526AF893A286C55A1648866D0A166709CAD4CAC3E9ADC5D8C6058BE13FA1F11CA65F01AC7FA5E5A99425340B4ADC350EA483AF19E637AD364EA62911680428FC830928EC86B77E82CE39312E8A63FDDA30524452EFCDC6FF0294CF582B00CC2BC2032A742EC2C79E56E698BA9AC6B0D9301AC505242E8FFEED7C335A66A761CC80F07F832B146233C3A567CB5795A472471960043AFD78706B5ACF62782A0CB547595B040682EF11603DC5CD86E2315E4EEF07C49854B069BE248005D00CE0EAD4CC4EFDB7231F5BCDCB3387504D16CD50276918E90303F4A3 diff --git a/src/libutil-tests/file-descriptor.cc b/src/libutil-tests/file-descriptor.cc index c385d2960f01..b6b13fd9ab62 100644 --- a/src/libutil-tests/file-descriptor.cc +++ b/src/libutil-tests/file-descriptor.cc @@ -2,10 +2,14 @@ #include #include "nix/util/file-descriptor.hh" +#include "nix/util/file-system.hh" #include "nix/util/serialise.hh" #include "nix/util/signals.hh" #include +#include +#include +#include #ifndef _WIN32 # include @@ -322,4 +326,123 @@ TEST(WriteFull, RespectsAllowInterrupts) EXPECT_EQ(source.readLine(/*eofOk=*/true), "hello"); } +class CopyFdRangeTest : public ::testing::TestWithParam> +{ +protected: + static std::string makeData() + { + std::mt19937 rng{42}; + std::string data(192 * 1024 + 1024, '\0'); + std::ranges::generate(data, rng); + return data; + } + + bool tryCoW() + { + return std::get<0>(GetParam()); + } + + bool isRegularFile() + { + return std::get<1>(GetParam()); + } +}; + +TEST_P(CopyFdRangeTest, regularToRegular) +{ + auto source = createAnonymousTempFile(); + auto dest = createAnonymousTempFile(); + auto input = makeData(); + writeFull(source.get(), input); + + FdSink sink{dest.get()}; + sink.isRegularFile = isRegularFile(); + copyFdRange(source.get(), 0, input.size(), sink, tryCoW()); + sink.flush(); + ASSERT_EQ(sink.written, input.size()); + ASSERT_EQ(lseek(dest.get(), 0, SEEK_SET), 0); + ASSERT_EQ(readFile(dest.get()), input); +} + +TEST_P(CopyFdRangeTest, offsetWorks) +{ + auto source = createAnonymousTempFile(); + auto dest = createAnonymousTempFile(); + auto input = makeData(); + writeFull(source.get(), input); + + static constexpr off_t offset = 4096; + FdSink sink{dest.get()}; + sink.isRegularFile = isRegularFile(); + copyFdRange(source.get(), offset, input.size() - offset, sink, tryCoW()); + sink.flush(); + ASSERT_EQ(sink.written, input.size() - offset); + ASSERT_EQ(lseek(dest.get(), 0, SEEK_SET), 0); + ASSERT_EQ(readFile(dest.get()), input.substr(offset)); +} + +TEST_P(CopyFdRangeTest, endOfFile) +{ + auto source = createAnonymousTempFile(); + Descriptor dest; + AutoCloseFD dest_ = createAnonymousTempFile(); + Pipe pipe; + if (isRegularFile()) { + dest = dest_.get(); + } else { + pipe.create(); + dest = pipe.writeSide.get(); + } + FdSink sink(dest); + sink.isRegularFile = isRegularFile(); + /* This can't ever deadlock, because nothing should be written to the pipe anyway. */ + EXPECT_THROW(copyFdRange(source.get(), 0, 512 * 1024, sink, tryCoW()), EndOfFile); +} + +TEST_P(CopyFdRangeTest, toPipe) +{ + auto input = makeData(); + auto source = createAnonymousTempFile(); + writeFull(source.get(), input); + + Pipe pipe; + pipe.create(); + + std::string output; + std::thread reader([&] { output = drainFD(pipe.readSide.get()); }); + + FdSink sink{pipe.writeSide.get()}; + /* This is only a hint and isn't load-bearing. */ + sink.isRegularFile = isRegularFile(); + copyFdRange(source.get(), 0, input.size(), sink, tryCoW()); + sink.flush(); + pipe.writeSide.close(); + reader.join(); + ASSERT_EQ(sink.written, input.size()); + ASSERT_EQ(output, input); +} + +TEST_P(CopyFdRangeTest, nonSeekableInput) +{ + Pipe in; + in.create(); + auto dest = createAnonymousTempFile(); + FdSink sink{dest.get()}; + sink.isRegularFile = isRegularFile(); + // TODO: Test the error code accurately. + EXPECT_THROW(copyFdRange(in.readSide.get(), 0, 128 * 1024, sink, tryCoW()), SystemError); +} + +INSTANTIATE_TEST_SUITE_P( + CopyFdRange, + CopyFdRangeTest, + ::testing::Combine(::testing::Values(false, true), ::testing::Values(false, true)), + [](const ::testing::TestParamInfo> & info) { + std::string res = ""; + res += std::get<0>(info.param) ? "fast" : "slow"; + res += "_"; + res += std::get<1>(info.param) ? "regular" : "nonregular"; + return res; + }); + } // namespace nix diff --git a/src/libutil-tests/fun.cc b/src/libutil-tests/fun.cc index e763ac018e7f..166a181d0670 100644 --- a/src/libutil-tests/fun.cc +++ b/src/libutil-tests/fun.cc @@ -2,6 +2,7 @@ #include #include "nix/util/fun.hh" +#include "util-tests-config.hh" namespace nix { @@ -39,19 +40,31 @@ TEST(fun, moveConstructFromStdFunction) TEST(fun, rejectsEmptyStdFunction) { std::function empty; +#if HAVE_CXA_THROW + ASSERT_DEATH((fun{empty}), "invalid_argument"); +#else EXPECT_THROW((fun{empty}), std::invalid_argument); +#endif } TEST(fun, rejectsEmptyStdFunctionMove) { std::function empty; +#if HAVE_CXA_THROW + ASSERT_DEATH((fun{std::move(empty)}), "invalid_argument"); +#else EXPECT_THROW((fun{std::move(empty)}), std::invalid_argument); +#endif } TEST(fun, rejectsNullFunctionPointer) { int (*nullFp)(int) = nullptr; +#if HAVE_CXA_THROW + ASSERT_DEATH((fun{nullFp}), "invalid_argument"); +#else EXPECT_THROW((fun{nullFp}), std::invalid_argument); +#endif } TEST(fun, nullptrDeletedAtCompileTime) diff --git a/src/libutil-tests/local-keys.cc b/src/libutil-tests/local-keys.cc index 665200550cc3..fc83ca8da825 100644 --- a/src/libutil-tests/local-keys.cc +++ b/src/libutil-tests/local-keys.cc @@ -1,5 +1,9 @@ #include "nix/util/signature/local-keys.hh" #include "nix/util/base-n.hh" +#include "nix/util/configuration.hh" +#include "nix/util/file-system.hh" +#include "nix/util/util.hh" +#include "nix/util/tests/test-data.hh" #include @@ -7,23 +11,27 @@ namespace nix { TEST(local_keys, signAndVerify) { - auto sk = SecretKey::generate("test-key-1"); - auto pk = sk.toPublicKey(); + experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::CNSA); - auto sig = sk.signDetached("hello world"); - auto sig2 = sk.signDetached("hello world"); - ASSERT_EQ(sig, sig2); // checks idempotence of signing + for (auto type : {KeyType::Ed25519, KeyType::MLDSA44, KeyType::MLDSA65, KeyType::MLDSA87, KeyType::ECDSAP384}) { + auto sk = SecretKey::generate("test-key-1", type); + auto pk = sk->toPublicKey(); - ASSERT_EQ(sig.keyName, "test-key-1"); - ASSERT_TRUE(pk.verifyDetached("hello world", sig)); + auto sig = sk->signDetached("hello world"); + auto sig2 = sk->signDetached("hello world"); + ASSERT_EQ(sig, sig2); // checks idempotence of signing - auto sk2 = SecretKey(sk.to_string()); - ASSERT_EQ(sk2.name, sk.name); - ASSERT_EQ(sk2.key, sk.key); + ASSERT_EQ(sig.keyName, "test-key-1"); + ASSERT_TRUE(pk->verifyDetached("hello world", sig)); - auto pk2 = PublicKey(pk.to_string()); - ASSERT_EQ(pk2.name, pk.name); - ASSERT_EQ(pk2.key, pk.key); + auto sk2 = SecretKey::parse(sk->to_string()); + ASSERT_EQ(sk2->name, sk->name); + ASSERT_EQ(sk2->key, sk->key); + + auto pk2 = PublicKey::parse(pk->to_string()); + ASSERT_EQ(pk2->name, pk->name); + ASSERT_EQ(pk2->key, pk->key); + } } TEST(local_keys, rfc8032TestVector) @@ -50,15 +58,196 @@ TEST(local_keys, rfc8032TestVector) auto skBytes = seed + pubKeyBytes; auto skString = "test:" + base64::encode(std::as_bytes(std::span{skBytes.data(), skBytes.size()})); - auto sk = SecretKey(skString); - auto sig = sk.signDetached(message); + auto sk = SecretKey::parse(skString); + auto sig = sk->signDetached(message); ASSERT_EQ(sig.keyName, "test"); ASSERT_EQ(sig.sig, expectedSig); - auto pk = sk.toPublicKey(); - ASSERT_EQ(pk.key, pubKeyBytes); - ASSERT_TRUE(pk.verifyDetached(message, sig)); + auto pk = sk->toPublicKey(); + ASSERT_EQ(pk->key, pubKeyBytes); + ASSERT_TRUE(pk->verifyDetached(message, sig)); +} + +TEST(local_keys, ecdsaP384TestVector) +{ + experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::CNSA); + + // ECDSA P-384 / SHA-384 test vector from + // https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Standards-and-Guidelines/documents/examples/P384_SHA384.pdf. + // The signature was produced with an explicit (non-RFC-6979) nonce K, so we can only verify it here, not reproduce + // it via signDetached (which uses deterministic nonces). + // + // Q_x = 3BF701BC9E9D36B4D5F1455343F09126F2564390F2B487365071243C61E6471FB9D2AB74657B82F9086489D9EF0F5CB5 + // Q_y = D1A358EAFBF952E68D533855CCBDAA6FF75B137A5101443199325583552A6295FFE5382D00CFCDA30344A9B5B68DB855 + // R = 30EA514FC0D38D8208756F068113C7CADA9F66A3B40EA3B313D040D9B57DD41A332795D02CC7D507FCEF9FAF01A27088 + // S = CC808E504BE414F46C9027BCBF78ADF067A43922D6FCAA66C4476875FBB7B94EFD1F7D5DBE620BFB821C46D549683AD8 + + // DER-encoded SubjectPublicKeyInfo: ecPublicKey + secp384r1, then 0x04 || Q_x || Q_y. + auto pkDer = base16::decode( + "3076" // SEQUENCE (118) + "3010" // SEQUENCE AlgorithmIdentifier (16) + "06072a8648ce3d0201" // OID 1.2.840.10045.2.1 ecPublicKey + "06052b81040022" // OID 1.3.132.0.34 secp384r1 + "036200" // BIT STRING (98 bytes, 0 unused) + "04" // uncompressed point + "3bf701bc9e9d36b4d5f1455343f09126f2564390f2b487365071243c61e6471fb9d2ab74657b82f9086489d9ef0f5cb5" // Q_x + "d1a358eafbf952e68d533855ccbdaa6ff75b137a5101443199325583552a6295ffe5382d00cfcda30344a9b5b68db855"); // Q_y + + // DER-encoded ECDSA-Sig-Value: SEQUENCE { INTEGER r, INTEGER s }. + // S's top bit is set, so it gets a leading 0x00 to keep the INTEGER positive. + auto sigDer = base16::decode( + "3065" // SEQUENCE (101) + "0230" // INTEGER r (48 bytes) + "30ea514fc0d38d8208756f068113c7cada9f66a3b40ea3b313d040d9b57dd41a332795d02cc7d507fcef9faf01a27088" + "023100" // INTEGER s (49 bytes, leading 0x00) + "cc808e504be414f46c9027bcbf78adf067a43922d6fcaa66c4476875fbb7b94efd1f7d5dbe620bfb821c46d549683ad8"); + + auto pkString = "ecdsa-test:" + base64::encode(std::as_bytes(std::span{pkDer.data(), pkDer.size()})); + auto pk = PublicKey::parse(pkString); + + Signature sig{.keyName = "ecdsa-test", .sig = sigDer}; + ASSERT_TRUE(pk->verifyDetached("Example of ECDSA with P-384", sig)); + + // Tampering with the message must cause verification to fail. + ASSERT_FALSE(pk->verifyDetached("Example of ECDSA with P-385", sig)); +} + +TEST(local_keys, rfc6979EcdsaP384TestVector) +{ + experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::CNSA); + + // Test vector A.2.6 from RFC-6979: ECDSA P-384 / SHA-384, message "sample". + // Our signDetached uses RFC-6979 deterministic nonces, so we can reproduce + // the expected signature exactly. + // + // x = 6B9D3DAD2E1B8C1C05B19875B6659F4DE23C3B667BF297BA9AA47740787137D8 + // 96D5724E4C70A825F872C9EA60D2EDF5 + // Ux = EC3A4E415B4E19A4568618029F427FA5DA9A8BC4AE92E02E06AAE5286B300C64 + // DEF8F0EA9055866064A254515480BC13 + // Uy = 8015D9B72D7D57244EA8EF9AC0C621896708A59367F9DFB9F54CA84B3F1C9DB1 + // 288B231C3AE0D4FE7344FD2533264720 + // r = 94EDBB92A5ECB8AAD4736E56C691916B3F88140666CE9FA73D64C4EA95AD133C + // 81A648152E44ACF96E36DD1E80FABE46 + // s = 99EF4AEB15F178CEA1FE40DB2603138F130E740A19624526203B6351D0A3A94F + // A329C145786E679E7B82C71A38628AC8 + + // DER-encoded SEC1 ECPrivateKey (RFC 5915) with named curve secp384r1 + // and the matching public key embedded. + auto skDer = base16::decode( + "3081a4" // SEQUENCE (164) + "020101" // INTEGER version=1 + "0430" // OCTET STRING (48 bytes) -- private scalar x + "6b9d3dad2e1b8c1c05b19875b6659f4de23c3b667bf297ba9aa47740787137d896d5724e4c70a825f872c9ea60d2edf5" + "a007" // [0] parameters (7 bytes) + "06052b81040022" // OID 1.3.132.0.34 secp384r1 + "a164" // [1] publicKey (100 bytes) + "036200" // BIT STRING (98 bytes, 0 unused) + "04" // uncompressed point + "ec3a4e415b4e19a4568618029f427fa5da9a8bc4ae92e02e06aae5286b300c64def8f0ea9055866064a254515480bc13" // Ux + "8015d9b72d7d57244ea8ef9ac0c621896708a59367f9dfb9f54ca84b3f1c9db1288b231c3ae0d4fe7344fd2533264720"); // Uy + + // Expected DER-encoded ECDSA-Sig-Value. Both r and s have their top bit set, + // so each gets a leading 0x00 to keep the INTEGER positive. + auto expectedSig = base16::decode( + "3066" // SEQUENCE (102) + "023100" // INTEGER r (49 bytes) + "94edbb92a5ecb8aad4736e56c691916b3f88140666ce9fa73d64c4ea95ad133c81a648152e44acf96e36dd1e80fabe46" + "023100" // INTEGER s (49 bytes) + "99ef4aeb15f178cea1fe40db2603138f130e740a19624526203b6351d0a3a94fa329c145786e679e7b82c71a38628ac8"); + + auto skString = "rfc6979-test:" + base64::encode(std::as_bytes(std::span{skDer.data(), skDer.size()})); + auto sk = SecretKey::parse(skString); + + auto sig = sk->signDetached("sample"); + ASSERT_EQ(sig.keyName, "rfc6979-test"); + ASSERT_EQ(sig.sig, expectedSig); + + auto pk = sk->toPublicKey(); + ASSERT_TRUE(pk->verifyDetached("sample", sig)); +} + +/** + * Run an ACVP ML-DSA-sigGen-FIPS204 test vector (external/pure interface, + * deterministic, empty context). For each variant we load the ACVP `sk`, + * `message` and expected `signature` from data files, wrap the expanded-key + * `sk` in a PKCS#8 PrivateKeyInfo DER (per draft-ietf-lamps-dilithium- + * certificates), feed it to SecretKey::parse, sign the message, and assert + * the signature bytes match the ACVP output exactly. + * + * `derPrefixHex` is the PKCS#8 prefix to prepend to the raw expanded sk; it + * encodes the outer SEQUENCE, version, AlgorithmIdentifier (with the + * variant's OID), and the two OCTET STRING headers. + * + * Source: + * https://github.com/usnistgov/ACVP-Server/blob/15c0f3deeefbfa8cb6cd32a99e1ca3b738c66bf0/gen-val/json-files/ML-DSA-sigGen-FIPS204/internalProjection.json + */ +static void +runMlDsaAcvpTest(std::string_view variant, std::string_view derPrefixHex, size_t expectedSkSize, size_t expectedSigSize) +{ + experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::CNSA); + + auto dataDir = getUnitTestData() / "local-keys"; + auto sk = base16::decode(chomp(readFile(dataDir / (std::string(variant) + "-sk.hex")))); + auto message = base16::decode(chomp(readFile(dataDir / (std::string(variant) + "-message.hex")))); + auto expSig = base16::decode(chomp(readFile(dataDir / (std::string(variant) + "-signature.hex")))); + + ASSERT_EQ(sk.size(), expectedSkSize); + ASSERT_EQ(expSig.size(), expectedSigSize); + + auto der = base16::decode(derPrefixHex) + sk; + auto skString = + std::string(variant) + ":" + base64::encode(std::as_bytes(std::span{der.data(), der.size()})); + auto parsed = SecretKey::parse(skString); + + auto sig = parsed->signDetached(message); + ASSERT_EQ(sig.keyName, std::string(variant)); + ASSERT_EQ(sig.sig, expSig); + + auto pk = parsed->toPublicKey(); + ASSERT_TRUE(pk->verifyDetached(message, sig)); +} + +TEST(local_keys, mlDsa44AcvpTestVector) +{ + // ACVP tgId 1 / tcId 4. id-ml-dsa-44 OID = 2.16.840.1.101.3.4.3.17. + runMlDsaAcvpTest( + "mldsa44", + "30820A18" // SEQUENCE (2584) + "020100" // INTEGER version=0 + "300B0609608648016503040311" // AlgorithmIdentifier: id-ml-dsa-44 + "04820A04" // OCTET STRING privateKey (2564) + "04820A00", // OCTET STRING expandedKey (2560) + 2560, + 2420); +} + +TEST(local_keys, mlDsa65AcvpTestVector) +{ + // ACVP tgId 3 / tcId 40. id-ml-dsa-65 OID = 2.16.840.1.101.3.4.3.18. + runMlDsaAcvpTest( + "mldsa65", + "30820FD8" // SEQUENCE (4056) + "020100" // INTEGER version=0 + "300B0609608648016503040312" // AlgorithmIdentifier: id-ml-dsa-65 + "04820FC4" // OCTET STRING privateKey (4036) + "04820FC0", // OCTET STRING expandedKey (4032) + 4032, + 3309); +} + +TEST(local_keys, mlDsa87AcvpTestVector) +{ + // ACVP tgId 5 / tcId 73. id-ml-dsa-87 OID = 2.16.840.1.101.3.4.3.19. + runMlDsaAcvpTest( + "mldsa87", + "30821338" // SEQUENCE (4920) + "020100" // INTEGER version=0 + "300B0609608648016503040313" // AlgorithmIdentifier: id-ml-dsa-87 + "04821324" // OCTET STRING privateKey (4900) + "04821320", // OCTET STRING expandedKey (4896) + 4896, + 4627); } } // namespace nix diff --git a/src/libutil-tests/make.nix b/src/libutil-tests/make.nix new file mode 100644 index 000000000000..3a41a1715ab8 --- /dev/null +++ b/src/libutil-tests/make.nix @@ -0,0 +1,82 @@ +# The unit tests of libnixutil, transcribed from meson.build and +# package.nix: the `nix-util-tests` executable, with the derivations that +# run it as `tests.run` and `tests.run-without-new-syscalls`. +{ + pkgs, + nixMake, + nix-util, + nix-util-c, + nix-util-test-support, +}: + +let + inherit (pkgs) lib; + + tests = nixMake.mkComponent { + name = "determinate-nix-util-tests"; + type = "executable"; + exeName = "nix-util-tests"; + + deps = [ + nix-util + nix-util-c + nix-util-test-support + ]; + + root = ./.; + + includeDirs = [ "" ]; + + files = nixMake.commonSupportFiles; + + configHeaders = { + "util-tests-config.hh" = { + PACKAGE_VERSION = lib.fileContents ../../.version; + # libnixutil interposes __cxa_throw (see src/libutil/make.nix). + HAVE_CXA_THROW = 1; + }; + }; + + # The tests have no main() of their own; it comes from gtest_main, + # which no #include refers to, so it is linked explicitly (after the + # -lgtest from pkg-config, which also provides its -L). + extraLinkLibs = [ "-lgtest_main" ]; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; + }; + + exe = "${tests}/bin/${tests.exeName}"; + + env = { + _NIX_TEST_UNIT_DATA = ./data; + }; +in + +tests +// { + tests = { + run = nixMake.runTest { + name = "${tests.name}-run"; + command = [ exe ]; + inherit env; + }; + + # Run the same tests with newer syscalls disabled via seccomp, to + # exercise fallback paths (iterative openat for openat2, /proc/self/fd + # for fchmodat2). + run-without-new-syscalls = nixMake.runTest { + name = "${tests.name}-run-without-new-syscalls"; + command = [ + "enosys" + "--syscall" + "openat2" + "--syscall" + "fchmodat2" + "--" + exe + ]; + inherit env; + nativeBuildInputs = [ pkgs.util-linux ]; + }; + }; +} diff --git a/src/libutil-tests/meson.build b/src/libutil-tests/meson.build index 1a5b24761fce..f7004c1c160d 100644 --- a/src/libutil-tests/meson.build +++ b/src/libutil-tests/meson.build @@ -45,16 +45,17 @@ if host_machine.system() == 'windows' deps_other += socket endif +subdir('nix-meson-build-support/common') + configdata = configuration_data() configdata.set_quoted('PACKAGE_VERSION', meson.project_version()) +configdata.set('HAVE_CXA_THROW', have_cxa_throw.to_int()) config_priv_h = configure_file( configuration : configdata, output : 'util-tests-config.hh', ) -subdir('nix-meson-build-support/common') - sources = files( 'alignment.cc', 'archive.cc', @@ -103,6 +104,10 @@ sources = files( 'xml-writer.cc', ) +if have_cxa_throw + sources += files('cxa-throw.cc') +endif + if host_machine.system() != 'windows' subdir('unix') endif diff --git a/src/libutil-tests/nix_api_util.cc b/src/libutil-tests/nix_api_util.cc index 14e23138b7ce..3dffc2ad4d7d 100644 --- a/src/libutil-tests/nix_api_util.cc +++ b/src/libutil-tests/nix_api_util.cc @@ -1,4 +1,6 @@ #include "nix/util/config-global.hh" +#include "nix/util/finally.hh" +#include "nix/util/logging.hh" #include "nix_api_util.h" #include "nix/util/tests/nix_api_util.hh" #include "nix/util/tests/string_callback.hh" @@ -6,6 +8,9 @@ #include #include +#include +#include +#include #include "util-tests-config.hh" @@ -76,4 +81,130 @@ TEST_F(nix_api_util_context, nix_err_code) ASSERT_EQ(nix_err_code(ctx), NIX_ERR_UNKNOWN); } +struct LogCapture +{ + std::vector> logs; + std::vector> starts; + std::vector stops; + std::vector> results; + bool destroyed = false; +}; + +extern "C" void capture_log(void * userdata, nix_verbosity level, const char * msg) +{ + auto * c = static_cast(userdata); + c->logs.emplace_back(level, msg); +} + +extern "C" void capture_start_activity( + void * userdata, + nix_activity_id id, + nix_verbosity level, + nix_activity_type type, + const char * s, + nix_activity_id parent) +{ + auto * c = static_cast(userdata); + c->starts.emplace_back(id, level, type, s, parent); +} + +extern "C" void capture_stop_activity(void * userdata, nix_activity_id id) +{ + auto * c = static_cast(userdata); + c->stops.push_back(id); +} + +extern "C" void capture_result_string(void * userdata, nix_activity_id id, nix_result_type type, const char * msg) +{ + auto * c = static_cast(userdata); + c->results.emplace_back(id, type, msg); +} + +extern "C" void capture_destroy(void * userdata) +{ + auto * c = static_cast(userdata); + c->destroyed = true; +} + +const nix_logger captureLoggerVtable{ + .log = capture_log, + .start_activity = capture_start_activity, + .stop_activity = capture_stop_activity, + .result_string = capture_result_string, + .destroy = capture_destroy, +}; + +TEST_F(nix_api_util_context, nix_set_logger_rejects_null_vtable) +{ + ASSERT_EQ(nix_set_logger(ctx, nullptr, nullptr), NIX_ERR_UNKNOWN); +} + +TEST_F(nix_api_util_context, nix_set_logger_routes_log_calls) +{ + LogCapture capture; + // Restore the default logger before `capture` goes out of scope so + // the destroy callback runs while `capture` is still alive. + Finally restoreLogger([] { nix::logger = nix::makeSimpleLogger().release(); }); + + ASSERT_EQ(nix_set_logger(ctx, &captureLoggerVtable, &capture), NIX_OK); + + nix::logger->log(nix::lvlInfo, "hello world"); + nix::logger->warn("be careful"); + + ASSERT_EQ(capture.logs.size(), 2u); + EXPECT_EQ(capture.logs[0].first, NIX_LVL_INFO); + EXPECT_EQ(capture.logs[0].second, "hello world"); + EXPECT_EQ(capture.logs[1].first, NIX_LVL_WARN); + // warn() prefixes "warning: " (with ANSI escapes). + EXPECT_NE(capture.logs[1].second.find("be careful"), std::string::npos); +} + +TEST_F(nix_api_util_context, nix_set_logger_routes_activities_and_results) +{ + LogCapture capture; + Finally restoreLogger([] { nix::logger = nix::makeSimpleLogger().release(); }); + + ASSERT_EQ(nix_set_logger(ctx, &captureLoggerVtable, &capture), NIX_OK); + + nix::ActivityId actId; + { + nix::Activity act(*nix::logger, nix::lvlInfo, nix::actBuild, "building foo"); + actId = act.id; + nix::logger->result(act.id, nix::resBuildLogLine, nix::Logger::Fields{"line of build output"}); + // Integer-valued result; should be filtered out by result_string. + act.progress(10, 100); + } + + ASSERT_EQ(capture.starts.size(), 1u); + EXPECT_EQ(std::get<0>(capture.starts[0]), actId); + EXPECT_EQ(std::get<1>(capture.starts[0]), NIX_LVL_INFO); + EXPECT_EQ(std::get<2>(capture.starts[0]), NIX_ACTIVITY_TYPE_BUILD); + EXPECT_EQ(std::get<3>(capture.starts[0]), "building foo"); + + ASSERT_EQ(capture.results.size(), 1u) << "result_string should only fire for string-valued results"; + EXPECT_EQ(std::get<0>(capture.results[0]), actId); + EXPECT_EQ(std::get<1>(capture.results[0]), NIX_RESULT_TYPE_BUILD_LOG_LINE); + EXPECT_EQ(std::get<2>(capture.results[0]), "line of build output"); + + ASSERT_EQ(capture.stops.size(), 1u); + EXPECT_EQ(capture.stops[0], actId); +} + +TEST_F(nix_api_util_context, nix_set_logger_does_not_invoke_destroy_when_replaced) +{ + // Both captures must outlive the Finally so that the logger held + // by `nix::logger` at teardown can call destroy on a live capture. + LogCapture capture; + LogCapture capture2; + Finally restoreLogger([] { nix::logger = nix::makeSimpleLogger().release(); }); + + ASSERT_EQ(nix_set_logger(ctx, &captureLoggerVtable, &capture), NIX_OK); + EXPECT_FALSE(capture.destroyed); + + // Replacing the logger won't destroy the old logger. + ASSERT_EQ(nix_set_logger(ctx, &captureLoggerVtable, &capture2), NIX_OK); + EXPECT_FALSE(capture.destroyed); + EXPECT_FALSE(capture2.destroyed); +} + } // namespace nixC diff --git a/src/libutil/args.cc b/src/libutil/args.cc index 4a85a08cb26e..6d63e9c13ad2 100644 --- a/src/libutil/args.cc +++ b/src/libutil/args.cc @@ -274,7 +274,8 @@ void RootArgs::parseCmdline(const Strings & _cmdline, bool allowShebang) if (auto s = getEnv("NIX_GET_COMPLETIONS")) { size_t n = std::stoi(*s); - assert(n > 0 && n <= cmdline.size()); + if (n < 1 || n > cmdline.size()) + throw UsageError("NIX_GET_COMPLETIONS must be between 1 and the number of arguments"); *std::next(cmdline.begin(), n - 1) += completionMarker; completions = std::make_shared(); verbosity = lvlError; @@ -514,7 +515,7 @@ void Args::checkArgs() { for (auto & [name, flag] : longFlags) { if (flag->required && flag->timesUsed == 0) - throw UsageError("required argument '--%s' is missing", name); + throw UsageError("required argument '%s' is missing", "--" + name); } } @@ -608,7 +609,7 @@ Strings argvToStrings(int argc, char ** argv) std::optional Command::experimentalFeature() { - return {Xp::NixCommand}; + return {}; } MultiCommand::MultiCommand(std::string_view commandName, const Commands & commands_) @@ -633,7 +634,7 @@ MultiCommand::MultiCommand(std::string_view commandName, const Commands & comman }}, .completer = {[&](AddCompletions & completions, size_t, std::string_view prefix) { for (auto & [name, command] : commands) - if (hasPrefix(name, prefix)) + if (hasPrefix(name, prefix) && !hasPrefix(name, "__")) completions.add(name); }}}); @@ -672,6 +673,8 @@ nlohmann::json MultiCommand::toJSON() auto command = commandFun(); auto j = command->toJSON(); auto cat = nlohmann::json::object(); + if (command->category() == Command::catUndocumented) + continue; cat["id"] = command->category(); cat["description"] = trim(categories[command->category()]); cat["experimental-feature"] = command->experimentalFeature(); diff --git a/src/libutil/caching-source-accessor.cc b/src/libutil/caching-source-accessor.cc index 13efc1b8bdf3..ebdccfb6b450 100644 --- a/src/libutil/caching-source-accessor.cc +++ b/src/libutil/caching-source-accessor.cc @@ -4,6 +4,7 @@ namespace nix { +// FIXME: Use ForwardingSourceAccessor. namespace { class CachingSourceAccessor : public SourceAccessor @@ -96,6 +97,11 @@ class CachingSourceAccessor : public SourceAccessor { return next->getFingerprint(path); } + + std::shared_ptr getProvenance(const CanonPath & path) override + { + return next->getProvenance(path); + } }; } // namespace diff --git a/src/libutil/configuration.cc b/src/libutil/configuration.cc index 39e21fc0f80f..45a7a505234c 100644 --- a/src/libutil/configuration.cc +++ b/src/libutil/configuration.cc @@ -427,11 +427,11 @@ std::set BaseSetting>::parse( { std::set res; for (auto & s : tokenizeString(str)) { - if (auto thisXpFeature = parseExperimentalFeature(s); thisXpFeature) { + if (auto thisXpFeature = parseExperimentalFeature(s)) res.insert(thisXpFeature.value()); - if (thisXpFeature.value() == Xp::Flakes) - res.insert(Xp::FetchTree); - } else if (s == "no-url-literals") + else if (stabilizedFeatures.count(s)) + debug("experimental feature '%s' is now stable", s); + else if (s == "no-url-literals") warn( "experimental feature '%s' has been stabilized and renamed; use 'lint-url-literals = fatal' setting instead", s); diff --git a/src/libutil/current-process.cc b/src/libutil/current-process.cc index acb6e52dd733..d99b5063d677 100644 --- a/src/libutil/current-process.cc +++ b/src/libutil/current-process.cc @@ -10,6 +10,7 @@ #ifdef __APPLE__ # include +# include #endif #ifdef __linux__ @@ -113,6 +114,15 @@ void restoreProcessContext(bool restoreMounts) } } #endif + +#ifdef __APPLE__ + /* Reset the Mach exception ports. Otherwise, if a crashpad_handler is attached to this process, it will be + inherited across execve() and receive spurious crash reports from unrelated programs (e.g. in `nix run`). + FIXME: it would be better to have Sentry tell crashpad_handler to quit, but it doesn't appear to have an API for + that. */ + task_set_exception_ports( + mach_task_self(), EXC_MASK_ALL | EXC_MASK_CRASH, MACH_PORT_NULL, EXCEPTION_DEFAULT, THREAD_STATE_NONE); +#endif } ////////////////////////////////////////////////////////////////////// diff --git a/src/libutil/error.cc b/src/libutil/error.cc index 182621b66ac3..7bf916529e7d 100644 --- a/src/libutil/error.cc +++ b/src/libutil/error.cc @@ -6,6 +6,7 @@ #include "nix/util/signals.hh" #include "nix/util/terminal.hh" #include "nix/util/position.hh" +#include "nix/util/sentry.hh" #include #include @@ -109,7 +110,8 @@ void printCodeLines(std::ostream & out, const std::string & prefix, const Pos & out << std::endl << fmt("%1% |%2%" ANSI_RED "%3%" ANSI_NORMAL, prefix, spaces, arrows); } - } + } else + out << std::endl << fmt("%1% %|2$5d|| " ANSI_ITALIC "(empty file)" ANSI_NORMAL, prefix, (errPos.line)); // next line of code. if (loc.nextLineOfCode.has_value()) { @@ -154,9 +156,10 @@ static bool printPosMaybe(std::ostream & oss, std::string_view indent, const std { bool hasPos = pos && *pos; if (hasPos) { - oss << indent << ANSI_BLUE << "at " ANSI_WARNING << *pos << ANSI_NORMAL << ":"; + auto loc = pos->getCodeLines(); + oss << indent << ANSI_BLUE << "at " ANSI_WARNING << *pos << ANSI_NORMAL << (loc ? ":" : ""); - if (auto loc = pos->getCodeLines()) { + if (loc) { printCodeLines(oss, "", *pos, *loc); oss << "\n"; } @@ -445,13 +448,13 @@ static void writeErr(std::string_view buf) #ifdef _WIN32 DWORD n; if (!WriteFile(fd, buf.data(), buf.size(), &n, NULL)) - abort(); + return; #else auto n = ::write(fd, buf.data(), buf.size()); if (n < 0) { if (errno == EINTR) continue; - abort(); + return; } #endif buf = buf.substr(n); @@ -463,9 +466,19 @@ void panic(std::string_view msg) writeErr("\n\n" ANSI_RED "terminating due to unexpected unrecoverable internal error: " ANSI_NORMAL); writeErr(msg); writeErr("\n"); + setSentryTag("panic_msg", std::string(msg).c_str()); std::terminate(); } +void outOfMemory() +{ + /* Running out of memory is not a bug, so don't treat it as a + crash. Use `_exit()` since the process may not be in a state + where destructors and `atexit` handlers can run. */ + writeErr("\n" ANSI_RED "error:" ANSI_NORMAL " ran out of memory\n"); + _exit(outOfMemoryExitStatus); +} + void unreachable(std::source_location loc) { char buf[512]; @@ -487,32 +500,37 @@ int handleExceptions(const std::string & programName, fun body) ErrorInfo::programName = baseNameOf(programName); - auto doLog = [&](BaseError & e) { - try { - logError(e.info()); - } catch (...) { - printError(ANSI_RED "error:" ANSI_NORMAL " Exception while printing an exception."); - } - }; + /* Note: this must happen after `printException()` below, so that + loggers that record the exception (like the OpenTelemetry + logger) can still do so. `flush()` must come after `stop()`, + which ends any open spans. */ + Finally stopLogger([]() { + logger->stop(); + logger->flush(); + }); - std::string error = ANSI_RED "error:" ANSI_NORMAL " "; try { body(); - } catch (Exit & e) { - return e.status; - } catch (UsageError & e) { - doLog(e); - printError("Try '%1% --help' for more information.", programName); - return 1; - } catch (BaseError & e) { - doLog(e); - return e.info().status; - } catch (std::bad_alloc & e) { - printError(error + "out of memory"); - return 1; - } catch (std::exception & e) { - printError(error + e.what()); - return 1; + } catch (...) { + auto ex = std::current_exception(); + + /* Report the exception to the logger, which prints it and, in + the case of the OpenTelemetry logger, records it on the + root span. */ + logger->printException(ex, programName); + + /* Determine the exit status. */ + try { + std::rethrow_exception(ex); + } catch (Exit & e) { + return e.status; + } catch (UsageError & e) { + return 1; + } catch (BaseError & e) { + return e.info().status; + } catch (...) { + return 1; + } } return 0; diff --git a/src/libutil/experimental-features.cc b/src/libutil/experimental-features.cc index 04a6ef38afc9..faa0926da1b3 100644 --- a/src/libutil/experimental-features.cc +++ b/src/libutil/experimental-features.cc @@ -19,7 +19,7 @@ void MissingExperimentalFeature::anchor() {} /** * If two different PRs both add an experimental feature, and we just - * used a number for this, we *woudln't* get merge conflict and the + * used a number for this, we *wouldn't* get merge conflict and the * counter will be incremented once instead of twice, causing a build * failure. * @@ -27,7 +27,7 @@ void MissingExperimentalFeature::anchor() {} * feature, we either have no issue at all if few features are not added * at the end of the list, or a proper merge conflict if they are. */ -constexpr size_t numXpFeatures = 1 + static_cast(Xp::BLAKE3Hashes); +constexpr size_t numXpFeatures = 1 + static_cast(Xp::BakedDerivations); constexpr std::array xpFeatureDetails = {{ { @@ -73,38 +73,21 @@ constexpr std::array xpFeatureDetails )", .trackingUrl = "https://github.com/NixOS/nix/milestone/42", }, - { - .tag = Xp::Flakes, - .name = "flakes", - .description = R"( - Enable flakes. See the manual entry for [`nix - flake`](@docroot@/command-ref/new-cli/nix3-flake.md) for details. - )", - .trackingUrl = "https://github.com/NixOS/nix/milestone/27", - }, { .tag = Xp::FetchTree, .name = "fetch-tree", .description = R"( + *Enabled for Determinate Nix Installer users since 2.24* + Enable the use of the [`fetchTree`](@docroot@/language/builtins.md#builtins-fetchTree) built-in function in the Nix language. `fetchTree` exposes a generic interface for fetching remote file system trees from different types of remote sources. - The [`flakes`](#xp-feature-flakes) feature flag always enables `fetch-tree`. This built-in was previously guarded by the `flakes` experimental feature because of that overlap. Enabling just this feature serves as a "release candidate", allowing users to try it out in isolation. )", .trackingUrl = "https://github.com/NixOS/nix/milestone/31", }, - { - .tag = Xp::NixCommand, - .name = "nix-command", - .description = R"( - Enable the new `nix` subcommands. See the manual on - [`nix`](@docroot@/command-ref/new-cli/nix.md) for details. - )", - .trackingUrl = "https://github.com/NixOS/nix/milestone/28", - }, { .tag = Xp::GitHashing, .name = "git-hashing", @@ -281,6 +264,70 @@ constexpr std::array xpFeatureDetails )", .trackingUrl = "https://github.com/NixOS/nix/milestone/60", }, + { + .tag = Xp::BuildTimeFetchTree, + .name = "build-time-fetch-tree", + .description = R"( + Enable the built-in derivation `builtin:fetch-tree`, as well as the flake input attribute `buildTime`. + )", + .trackingUrl = "", + }, + { + .tag = Xp::ParallelEval, + .name = "parallel-eval", + .description = R"( + Enable built-in functions for parallel evaluation. + )", + .trackingUrl = "", + }, + { + .tag = Xp::WasmBuiltin, + .name = "wasm-builtin", + .description = R"( + Enable the use of the [`builtins.wasm`](@docroot@/language/builtins.md) built-in function in the Nix language. + `builtins.wasm` allows calling WebAssembly functions from Nix expressions. + )", + .trackingUrl = "", + }, + { + .tag = Xp::WasmDerivations, + .name = "wasm-derivations", + .description = R"( + Allow derivations to target the WebAssembly system type (`wasm32-wasip1`). + When enabled, derivations with `system = "wasm32-wasip1"` can be built locally + using a WASI runtime environment. + )", + .trackingUrl = "", + }, + { + .tag = Xp::Provenance, + .name = "provenance", + .description = R"( + Enable keeping track of the provenance of store paths. + )", + .trackingUrl = "", + }, + { + .tag = Xp::CNSA, + .name = "cnsa", + .description = R"( + Enable support for keys and signatures using algorithms from the + Commercial National Security Algorithm (CNSA) Suite: ECDSA P-384 + and ML-DSA-44, ML-DSA-65, and ML-DSA-87. + )", + .trackingUrl = "", + }, + { + .tag = Xp::BakedDerivations, + .name = "baked-derivations", + .description = R"( + Enable *baked derivations*: derivations with the builder + `builtin:substitute` that are never built, but whose outputs are + obtained by substitution. This enables the `builtins.bakedDerivation` + function and the `nix flake bake` command. + )", + .trackingUrl = "", + }, }}; static_assert( @@ -292,6 +339,12 @@ static_assert( }(), "array order does not match enum tag order"); +/** + * A set of previously experimental features that are now considered + * stable. We don't warn if users have these in `experimental-features`. + */ +std::set stabilizedFeatures{"flakes", "nix-command"}; + const std::optional parseExperimentalFeature(const std::string_view & name) { using ReverseXpMap = std::map; diff --git a/src/libutil/file-descriptor-private.hh b/src/libutil/file-descriptor-private.hh new file mode 100644 index 000000000000..d934fe5cb245 --- /dev/null +++ b/src/libutil/file-descriptor-private.hh @@ -0,0 +1,23 @@ +#pragma once + +#include "nix/util/file-descriptor.hh" + +namespace nix { + +struct FdSink; + +/** + * Try to copy @p nbytes bytes from @p from fd to @p to fd, starting at @p offset. + * The intent is to try to use optimized file copies like copy_file_range (on Linux) first + * and only fall back to the slower copying when the fast path fails or is unsupported. + * + * @throws SystemError in case the copy was started, but failed halfway. + * + * @return true if the copy succeeded, false if it's unsupported. Partial copies + * should be reported as errors. + * + * @pre written should be set to 0 before calling this function. + */ +bool tryCopyFdRangeFast(Descriptor from, Descriptor to, off_t offset, size_t nbytes, size_t & written); + +} // namespace nix diff --git a/src/libutil/file-descriptor.cc b/src/libutil/file-descriptor.cc index f5f5ee05a69d..d0d8fa1ffbed 100644 --- a/src/libutil/file-descriptor.cc +++ b/src/libutil/file-descriptor.cc @@ -3,6 +3,8 @@ #include "nix/util/util.hh" #include "nix/util/signals.hh" +#include "file-descriptor-private.hh" + #include #include #include @@ -202,10 +204,30 @@ std::string drainFD(Descriptor fd, DrainFdOpts opts) return std::move(sink.s); } -void copyFdRange(Descriptor fd, off_t offset, size_t nbytes, Sink & sink) +void copyFdRange(Descriptor fd, off_t offset, size_t nbytes, Sink & sink, bool tryCoW) { + /* Nothing to do. */ + if (!nbytes) + return; + + static constexpr size_t copyBufferSize = 64 * 1024; + + if (tryCoW) { + /* dynamic_cast here is slightly annoying, but it's much cheaper than + doing I/O so it doesn't really matter. */ + FdSink * fdSink = dynamic_cast(&sink); + /* Destination is a regular file which hasn't been written to and we've + been hinted that we should try block cloning. Try it and in case we + can't do block cloning then fall back to the loop below. Also, don't + bother with files smaller than the copy buffer. */ + if (fdSink && !fdSink->hasData() && fdSink->written == 0 && fdSink->isRegularFile && nbytes > copyBufferSize + && tryCopyFdRangeFast(fd, fdSink->fd, offset, nbytes, fdSink->written)) { + return; + } + } + auto left = nbytes; - std::array buf; + std::array buf; while (left) { auto limit = std::min(left, buf.size()); diff --git a/src/libutil/forwarding-source-accessor.cc b/src/libutil/forwarding-source-accessor.cc new file mode 100644 index 000000000000..bbe2a309e927 --- /dev/null +++ b/src/libutil/forwarding-source-accessor.cc @@ -0,0 +1,7 @@ +#include "nix/util/forwarding-source-accessor.hh" + +namespace nix { + +void ForwardingSourceAccessor::anchor() {} + +} // namespace nix \ No newline at end of file diff --git a/src/libutil/fs-sink.cc b/src/libutil/fs-sink.cc index c95265e63703..bf71770311e0 100644 --- a/src/libutil/fs-sink.cc +++ b/src/libutil/fs-sink.cc @@ -210,6 +210,7 @@ struct RestoreRegularFile : CreateRegularFileSink, FdSink , fd(std::move(fd_)) , startFsync(startFSync_) { + isRegularFile = true; /* Hint for copying contents via CoW copy_file_range. */ } void anchor() override; diff --git a/src/libutil/include/nix/util/args.hh b/src/libutil/include/nix/util/args.hh index 42ccb960525d..8f39336b764d 100644 --- a/src/libutil/include/nix/util/args.hh +++ b/src/libutil/include/nix/util/args.hh @@ -377,6 +377,7 @@ struct Command : virtual public Args using Category = int; static constexpr Category catDefault = 0; + static constexpr Category catUndocumented = 1; virtual std::optional experimentalFeature(); diff --git a/src/libutil/include/nix/util/configuration.hh b/src/libutil/include/nix/util/configuration.hh index dc6d8a811b39..2f6236d614f5 100644 --- a/src/libutil/include/nix/util/configuration.hh +++ b/src/libutil/include/nix/util/configuration.hh @@ -488,6 +488,20 @@ public: } }; +/** + * A setting whose value is represented as JSON. The type `T` must be supported by `nlohmann::json`'s `get()`. + */ +template +class JSONSetting : public Setting +{ +public: + using Setting::Setting; + + T parse(const std::string & str) const override; + + std::string to_string() const override; +}; + /** * `AbsolutePath` wraps `std::filesystem::path`, so implicit conversion * from `Setting` to `const path &` requires two @@ -564,7 +578,7 @@ public: Example: ``` - experimental-features = nix-command flakes + experimental-features = ca-derivations ``` The following experimental features are available: diff --git a/src/libutil/include/nix/util/coroutine-gc.hh b/src/libutil/include/nix/util/coroutine-gc.hh new file mode 100644 index 000000000000..7b6d166cc1dd --- /dev/null +++ b/src/libutil/include/nix/util/coroutine-gc.hh @@ -0,0 +1,79 @@ +#pragma once +///@file + +#include + +namespace nix { + +/** + * Hooks that make the boost::coroutines2 coroutines in libutil (see + * `sourceToSink`/`sinkToSource` in `serialise.hh`) visible to a + * conservative garbage collector. libutil itself does not depend on + * any GC; these default to null and are installed by the evaluator + * (`libexpr`), which implements them in terms of bdwgc's registered + * stacks (`GC_register_stack` etc.). + * + * The GC side keeps track of the "current stack" of each thread (the + * registered stack the thread is executing on) and, for every stack + * that is not being executed on, a saved stack pointer from which it + * is scanned. The hooks keep that bookkeeping in sync with the + * coroutine switches. + * + * Note that a coroutine's continuation does not necessarily live on + * its own stack: when coroutine B's body resumes coroutine C, and C's + * body in turn writes to B's sink, B yields *from C's stack*, and the + * next resumption of B lands on C's stack as well. (This happens + * e.g. when `copyStorePath()`'s `sinkToSource` coroutine streams a + * NAR through a `sourceToSink` decompression coroutine.) Hence the + * yield-side hooks operate on the current stack rather than on the + * coroutine's own stack, and the resume-side hooks only provisionally + * set the current stack, which the resumed code corrects. + * + * The contract: + * + * - `coroStackRegister(base, size)` is called when a coroutine stack + * is allocated, with `base` its cold (hi) end. It returns an opaque + * cookie identifying the stack. `coroStackUnregister(cookie)` is + * called when the stack is deallocated. + * + * - `coroSwitchTo(cookie)` is called just before the current thread + * resumes the coroutine identified by `cookie` (including the + * initial start of its body and the forced unwinding of a suspended + * one). It records the stack pointer of the stack being left (so + * that everything from there up to that stack's base is scannable), + * provisionally makes the coroutine's own stack the current one, + * and returns an opaque handle for the stack being left, to be + * passed to `coroSwitchBack` when the switch returns (because the + * coroutine yielded or finished). + * + * - `coroEnter(cookie)` is called at the start of the coroutine body + * (which does run on the coroutine's own stack) and makes that stack + * the current one. This is needed because the cookie is only known + * once the stack has been allocated, which happens after the + * `coroSwitchTo` for the initial start. + * + * - `coroYield()` is called just before a coroutine yields back to its + * resumer. It records the stack pointer of the current stack + * (whichever registered stack the yield is executed on) and returns + * a handle for it; `coroResume(handle)` is called when the yield + * returns (normally or by exception) and makes that stack the + * current one again. + * + * All hooks are invoked with the corresponding stack switches + * strictly balanced. + * + * The hooks may be reset to null at any time, in particular in a + * forked child process (which inherits the parent's bookkeeping but + * doesn't execute on the stack it describes and never runs the + * collector), so every use must check for null, including the + * "closing" half of a pair whose "opening" half did run. + */ +extern void * (*coroStackRegister)(void * base, size_t size); +extern void (*coroStackUnregister)(void * cookie); +extern void * (*coroSwitchTo)(void * cookie); +extern void (*coroSwitchBack)(void * prevHandle); +extern void (*coroEnter)(void * cookie); +extern void * (*coroYield)(); +extern void (*coroResume)(void * handle); + +} // namespace nix diff --git a/src/libutil/include/nix/util/error.hh b/src/libutil/include/nix/util/error.hh index 686ca708656b..89f04c88ba25 100644 --- a/src/libutil/include/nix/util/error.hh +++ b/src/libutil/include/nix/util/error.hh @@ -470,6 +470,28 @@ void throwExceptionSelfCheck(); [[noreturn]] void panic(std::string_view msg); +/** + * The exit status of the process when it runs out of memory. This is + * the value of `ENOMEM` on Linux. + */ +constexpr int outOfMemoryExitStatus = 12; + +/** + * Print an error message and exit with status + * `outOfMemoryExitStatus`. Installed as the `std::new_handler` so + * that failing memory allocations produce a clear "out of memory" + * error instead of an opaque abort (in particular when `operator new` + * is overridden by mimalloc, which cannot throw `std::bad_alloc`). + */ +[[noreturn]] +void outOfMemory(); + +/** + * Log the current exception (if any) and call abort(). + */ +[[noreturn]] +void onTerminate(); + /** * Run a function, printing an error and returning on exception. * Useful for wrapping a `main` function that may throw diff --git a/src/libutil/include/nix/util/experimental-features.hh b/src/libutil/include/nix/util/experimental-features.hh index 057a3b0064ed..18374246e37f 100644 --- a/src/libutil/include/nix/util/experimental-features.hh +++ b/src/libutil/include/nix/util/experimental-features.hh @@ -19,9 +19,7 @@ namespace nix { enum struct ExperimentalFeature { CaDerivations, ImpureDerivations, - Flakes, FetchTree, - NixCommand, GitHashing, RecursiveNix, FetchClosure, @@ -38,8 +36,17 @@ enum struct ExperimentalFeature { PipeOperators, ExternalBuilders, BLAKE3Hashes, + BuildTimeFetchTree, + ParallelEval, + WasmBuiltin, + WasmDerivations, + Provenance, + CNSA, + BakedDerivations, }; +extern std::set stabilizedFeatures; + /** * Just because writing `ExperimentalFeature::CaDerivations` is way too long */ diff --git a/src/libutil/include/nix/util/file-descriptor.hh b/src/libutil/include/nix/util/file-descriptor.hh index a6796f22691e..a0df4222b416 100644 --- a/src/libutil/include/nix/util/file-descriptor.hh +++ b/src/libutil/include/nix/util/file-descriptor.hh @@ -115,11 +115,13 @@ size_t readOffset(Descriptor fd, off_t offset, std::span buffer); /** * Read \ref nbytes starting at \ref offset from a seekable file into a sink. * - * @throws SystemError if fd is not seekable or any operation fails + * @param tryCoW Used as a hint to use optimised file copying like copy_file_range. + * + * @throws SystemError if @p fd is not seekable or any operation fails * @throws Interrupted if the operation was interrupted * @throws EndOfFile if an EOF was reached before reading \ref nbytes */ -void copyFdRange(Descriptor fd, off_t offset, size_t nbytes, Sink & sink); +void copyFdRange(Descriptor fd, off_t offset, size_t nbytes, Sink & sink, bool tryCoW = false); /** * Wrappers around read()/write() that read/write exactly the diff --git a/src/libutil/include/nix/util/forwarding-source-accessor.hh b/src/libutil/include/nix/util/forwarding-source-accessor.hh new file mode 100644 index 000000000000..1e4500f95f9d --- /dev/null +++ b/src/libutil/include/nix/util/forwarding-source-accessor.hh @@ -0,0 +1,54 @@ +#pragma once + +#include "source-accessor.hh" + +namespace nix { + +/** + * A source accessor that just forwards every operation to another + * accessor. This is not useful in itself but can be used as a + * superclass for accessors that do change some operations. + */ +struct ForwardingSourceAccessor : SourceAccessor +{ + ref next; + + ForwardingSourceAccessor(ref next) + : next(next) + { + } + + void readFile(const CanonPath & path, Sink & sink, fun sizeCallback) override + { + next->readFile(path, sink, sizeCallback); + } + + std::optional maybeLstat(const CanonPath & path) override + { + return next->maybeLstat(path); + } + + DirEntries readDirectory(const CanonPath & path) override + { + return next->readDirectory(path); + } + + std::string readLink(const CanonPath & path) override + { + return next->readLink(path); + } + + std::string showPath(const CanonPath & path) override + { + return next->showPath(path); + } + + std::optional getPhysicalPath(const CanonPath & path) override + { + return next->getPhysicalPath(path); + } + + void anchor() override; +}; + +} // namespace nix diff --git a/src/libutil/include/nix/util/logging.hh b/src/libutil/include/nix/util/logging.hh index dc69edde40bc..1e069e72cf4f 100644 --- a/src/libutil/include/nix/util/logging.hh +++ b/src/libutil/include/nix/util/logging.hh @@ -3,11 +3,13 @@ #include "nix/util/error.hh" #include "nix/util/configuration.hh" +#include "nix/util/types.hh" #include "nix/util/file-descriptor.hh" #include "nix/util/finally.hh" #include "nix/util/fun.hh" #include +#include #include @@ -28,6 +30,13 @@ typedef enum { actPostBuildHook = 110, actBuildWaiting = 111, actFetchTree = 112, + /** + * The type under which string-named activities (created via the + * name-based `Activity` constructor) are reported on legacy code + * paths, cf. the name-based `Logger::startActivity()`. Do not use + * directly. + */ + actStringly = 10113, } ActivityType; typedef enum { @@ -40,6 +49,16 @@ typedef enum { resSetExpected = 106, resPostBuildLogLine = 107, resFetchStatus = 108, + resHashMismatch = 109, + resBuildResult = 110, + /** + * Emitted via the JSON `result()` overload: an object describing + * the response to an HTTP request, with the fields `httpStatus` + * (the status code, absent for non-HTTP protocols) and `bodySize` + * (the number of body bytes received). More fields may be added + * in the future. + */ + resHttpStatus = 10111, } ResultType; typedef uint64_t ActivityId; @@ -63,12 +82,21 @@ public: {}, "json-log-path", R"( - A file or unix socket to which JSON records of Nix's log output are + A file or Unix domain socket to which JSON records of Nix's log output are written, in the same format as `--log-format internal-json` (without the `@nix ` prefixes on each line). Concurrent writes to the same file by multiple Nix processes are not supported and may result in interleaved or corrupted log records. )"}; + + Setting sessionId{ + this, + "", + "session-id", + R"( + An identifier for the current Nix session, which is included in JSON log output to + allow grouping of log messages from the same session. This defaults to a random UUID. + )"}; }; extern LoggerSettings loggerSettings; @@ -81,37 +109,43 @@ public: struct Field { - // FIXME: use std::variant. - enum { tInt = 0, tString = 1 } type; - - uint64_t i = 0; - std::string s; + std::variant raw; Field(const std::string & s) - : type(tString) - , s(s) + : raw(s) { } Field(const char * s) - : type(tString) - , s(s) + : raw(std::string(s)) { } Field(const uint64_t & i) - : type(tInt) - , i(i) + : raw(i) { } }; typedef std::vector Fields; + /** + * Key/value meta-information about a string-named activity, cf. + * the name-based `startActivity()`. + */ + using ActivityMetadata = std::span>; + virtual ~Logger(); virtual void stop() {}; + /** + * Flush any buffered state to its final destination. Loggers that + * upload to a remote service (such as the OpenTelemetry logger) use + * this to perform the upload on exit. + */ + virtual void flush() {}; + /** * Guard object to resume the logger when done. */ @@ -133,22 +167,35 @@ public: return false; } - virtual void log(Verbosity lvl, std::string_view s) = 0; + /* Note: logging functions must be noexcept, since they're often + called in contexts where exceptions cannot be handled (such as + in completion callbacks or destructors). Implementations should + handle failure to write the message (e.g. by ignoring it or + disabling the logger). */ + virtual void log(Verbosity lvl, std::string_view s) noexcept = 0; - void log(std::string_view s) + void log(std::string_view s) noexcept { log(lvlInfo, s); } - virtual void logEI(const ErrorInfo & ei) = 0; + virtual void logEI(const ErrorInfo & ei) noexcept = 0; - void logEI(Verbosity lvl, ErrorInfo ei) + /** + * Report the exception that terminated the program (see + * `handleExceptions()`). The default implementation prints it; + * loggers that report to a monitoring system can override this to + * record it there as well (or instead). + */ + virtual void printException(const std::exception_ptr & ex, std::string_view programName) noexcept; + + void logEI(Verbosity lvl, ErrorInfo ei) noexcept { ei.level = lvl; logEI(ei); } - virtual void warn(const std::string & msg); + virtual void warn(const std::string & msg) noexcept; virtual void startActivity( ActivityId act, @@ -156,11 +203,46 @@ public: ActivityType type, const std::string & s, const Fields & fields, - ActivityId parent) {}; + ActivityId parent) noexcept {}; - virtual void stopActivity(ActivityId act) {}; + /** + * Start a string-named activity carrying key/value + * meta-information. The default implementation reports it via the + * `ActivityType`-based overload as `actStringly`, discarding the + * name and metadata, so legacy loggers work unchanged. Metadata + * keys follow the OpenTelemetry attribute naming conventions + * where applicable (e.g. `http.request.method`); Nix-specific + * keys use a `nix.` prefix. + */ + virtual void startActivity( + ActivityId act, + Verbosity lvl, + std::string_view name, + ActivityMetadata metadata, + std::string_view s, + ActivityId parent) noexcept + { + startActivity(act, lvl, actStringly, std::string(s), {}, parent); + }; + + virtual void stopActivity(ActivityId act) noexcept {}; + + virtual void result(ActivityId act, ResultType type, const Fields & fields) noexcept {}; - virtual void result(ActivityId act, ResultType type, const Fields & fields) {}; + virtual void result(ActivityId act, ResultType type, const nlohmann::json & json) noexcept {}; + + /** + * Return distributed tracing context for the given activity as + * W3C Trace Context headers (`traceparent`, `tracestate`), for + * propagation to another process (e.g. as HTTP request headers). + * `act == 0` denotes the root context of this process. Returns an + * empty list if this logger does not do tracing or has no context + * for `act`. + */ + virtual Headers getTraceContext(ActivityId act) + { + return {}; + } virtual void writeToStdout(std::string_view s); @@ -212,6 +294,18 @@ struct Activity Logger & logger, ActivityType type, const Logger::Fields & fields = {}, ActivityId parent = getCurActivity()) : Activity(logger, lvlError, type, "", fields, parent) {}; + /** + * Start a string-named activity carrying key/value + * meta-information, cf. the name-based `Logger::startActivity()`. + */ + Activity( + Logger & logger, + Verbosity lvl, + std::string_view name, + Logger::ActivityMetadata metadata = {}, + std::string_view s = {}, + ActivityId parent = getCurActivity()); + Activity(const Activity & act) = delete; ~Activity(); @@ -226,6 +320,11 @@ struct Activity result(resSetExpected, type2, expected); } + void result(ResultType type, const nlohmann::json & json) const + { + logger.result(id, type, json); + } + template void result(ResultType type, const Args &... args) const { @@ -262,20 +361,45 @@ extern Logger * logger; std::unique_ptr makeSimpleLogger(bool printBuildLogs = true); -/** - * Create a logger that sends log messages to `mainLogger` and the - * list of loggers in `extraLoggers`. Only `mainLogger` is used for - * writing to stdout and getting user input. - */ -std::unique_ptr -makeTeeLogger(std::unique_ptr mainLogger, std::vector> && extraLoggers); - std::unique_ptr makeJSONLogger(Descriptor fd, bool includeNixPrefix = true); std::unique_ptr makeJSONLogger(const std::filesystem::path & path, bool includeNixPrefix = true); +/** + * Add an additional logger to the global `logger` by combining them + * into a `TeeLogger`. The current logger keeps responsibility for + * stdout and user interaction. + */ +void applyExtraLogger(std::unique_ptr extraLogger); + void applyJSONLogger(); +/** + * Extract the value of the `traceparent` header from trace context + * headers returned by `Logger::getTraceContext()`, or the empty + * string if there is none. + */ +std::string getTraceparent(const Headers & headers); + +/** + * Marks, for the duration of its existence, Logger calls made on this + * thread as replaying messages that originated in another process + * (e.g. activities forwarded from the daemon to its client). Loggers + * that export telemetry should ignore such messages, since the + * originating process is responsible for exporting them. + */ +struct RemoteLogSource +{ + RemoteLogSource(); + ~RemoteLogSource(); +}; + +/** + * Whether Logger calls on this thread are currently replaying + * messages from another process, cf. `RemoteLogSource`. + */ +bool isRemoteLogSource(); + /** * @param source A noun phrase describing the source of the message, e.g. "the builder". */ @@ -360,6 +484,6 @@ inline void warn(const std::string & fs, const Args &... args) warn(args); \ } -void writeToStderr(std::string_view s); +void writeToStderr(std::string_view s) noexcept; } // namespace nix diff --git a/src/libutil/include/nix/util/meson.build b/src/libutil/include/nix/util/meson.build index 4350be5a8bf6..1379fd8d2f2a 100644 --- a/src/libutil/include/nix/util/meson.build +++ b/src/libutil/include/nix/util/meson.build @@ -32,6 +32,7 @@ headers = [ config_pub_h ] + files( 'config-global.hh', 'config-impl.hh', 'configuration.hh', + 'coroutine-gc.hh', 'current-process.hh', 'deleter.hh', 'demangle.hh', @@ -50,6 +51,7 @@ headers = [ config_pub_h ] + files( 'file-system.hh', 'finally.hh', 'fmt.hh', + 'forwarding-source-accessor.hh', 'fs-sink.hh', 'fun.hh', 'git.hh', @@ -63,19 +65,23 @@ headers = [ config_pub_h ] + files( 'memo.hh', 'memory-source-accessor.hh', 'mounted-source-accessor.hh', + 'move-only-function.hh', 'muxable-pipe.hh', 'nar-accessor.hh', 'nar-cache.hh', 'nar-listing.hh', 'os-string.hh', + 'override-provenance-source-accessor.hh', 'pool.hh', 'pos-idx.hh', 'pos-table.hh', 'position.hh', 'processes.hh', + 'provenance.hh', 'ref.hh', 'regex-combinators.hh', 'repair-flag.hh', + 'sentry.hh', 'serialise.hh', 'signals.hh', 'signature/local-keys.hh', diff --git a/src/libutil/include/nix/util/move-only-function.hh b/src/libutil/include/nix/util/move-only-function.hh new file mode 100644 index 000000000000..479812207822 --- /dev/null +++ b/src/libutil/include/nix/util/move-only-function.hh @@ -0,0 +1,83 @@ +#pragma once +///@file + +#include +#include +#include +#include + +namespace nix { + +#ifdef __cpp_lib_move_only_function + +template +using MoveOnlyFunction = std::move_only_function; + +#else + +/** + * Fallback implementation of `std::move_only_function` for standard + * libraries that don't provide it yet (e.g. libc++ as of version 21): + * a type-erased callable wrapper that, unlike `std::function`, only + * requires the callable to be move-constructible, so it can hold + * lambdas that capture move-only types. + */ +template +class MoveOnlyFunction; + +template +class MoveOnlyFunction +{ + struct Base + { + virtual Ret call(Args &&... args) = 0; + virtual ~Base() = default; + }; + + template + struct Impl final : Base + { + F f; + + template + requires std::is_same_v, F> + Impl(G && g) + : f(std::forward(g)) + { + } + + Ret call(Args &&... args) override + { + return f(std::forward(args)...); + } + }; + + std::unique_ptr impl; + +public: + MoveOnlyFunction() = default; + + template + requires(!std::is_same_v, MoveOnlyFunction> && std::is_invocable_r_v) + MoveOnlyFunction(F && f) + : impl(std::make_unique>>(std::forward(f))) + { + } + + MoveOnlyFunction(MoveOnlyFunction &&) noexcept = default; + MoveOnlyFunction & operator=(MoveOnlyFunction &&) noexcept = default; + + Ret operator()(Args... args) + { + return impl->call(std::forward(args)...); + } + + explicit operator bool() const + { + return (bool) impl; + } +}; + +#endif + +} // namespace nix diff --git a/src/libutil/include/nix/util/override-provenance-source-accessor.hh b/src/libutil/include/nix/util/override-provenance-source-accessor.hh new file mode 100644 index 000000000000..2b61d1d82b54 --- /dev/null +++ b/src/libutil/include/nix/util/override-provenance-source-accessor.hh @@ -0,0 +1,23 @@ +#pragma once + +#include "nix/util/forwarding-source-accessor.hh" + +namespace nix { + +struct OverrideProvenanceSourceAccessor : ForwardingSourceAccessor +{ + OverrideProvenanceSourceAccessor(ref next, std::shared_ptr provenance) + : ForwardingSourceAccessor(std::move(next)) + { + this->provenance = std::move(provenance); + } + + std::shared_ptr getProvenance(const CanonPath & path) override + { + return provenance; + } + + void anchor() override; +}; + +} // namespace nix diff --git a/src/libutil/include/nix/util/pos-idx.hh b/src/libutil/include/nix/util/pos-idx.hh index 8e668176c619..7b7d16ca3a4d 100644 --- a/src/libutil/include/nix/util/pos-idx.hh +++ b/src/libutil/include/nix/util/pos-idx.hh @@ -15,12 +15,12 @@ class PosIdx private: uint32_t id; +public: explicit PosIdx(uint32_t id) : id(id) { } -public: PosIdx() : id(0) { @@ -45,6 +45,11 @@ public: { return std::hash{}(id); } + + uint32_t get() const + { + return id; + } }; inline PosIdx noPos = {}; diff --git a/src/libutil/include/nix/util/pos-table.hh b/src/libutil/include/nix/util/pos-table.hh index 755bfbbe2201..8c455c59736b 100644 --- a/src/libutil/include/nix/util/pos-table.hh +++ b/src/libutil/include/nix/util/pos-table.hh @@ -49,6 +49,9 @@ private: */ using LinesCache = LRUCache; + // FIXME: this could be made lock-free (at least for access) if we + // have a data structure where pointers to existing positions are + // never invalidated. Sync> origins_; mutable Sync linesCache; diff --git a/src/libutil/include/nix/util/processes.hh b/src/libutil/include/nix/util/processes.hh index 64053fab4935..6e0c8ad34c0e 100644 --- a/src/libutil/include/nix/util/processes.hh +++ b/src/libutil/include/nix/util/processes.hh @@ -23,7 +23,7 @@ #include #include #include -#include +#include namespace nix { @@ -36,8 +36,7 @@ class Pid pid_t pid = -1; bool separatePG = false; int killSignal = SIGKILL; - std::chrono::milliseconds killTimeout; - std::thread killThread; + std::chrono::milliseconds killTimeout{0}; #else AutoCloseFD pid = INVALID_DESCRIPTOR; #endif @@ -61,6 +60,13 @@ public: // TODO: Implement for Windows #ifndef _WIN32 + /** + * Check whether the child process is still running, without + * blocking. If it has exited (or was reaped elsewhere), the child + * is reaped if necessary and this object is reset so that the + * destructor won't kill()/wait() an already-dead process. + */ + bool isAlive(); void setSeparatePG(bool separatePG); void setKillSignal(int signal); void setKillTimeout(std::chrono::milliseconds duration); @@ -74,6 +80,7 @@ public: swap(lhs.pid, rhs.pid); swap(lhs.separatePG, rhs.separatePG); swap(lhs.killSignal, rhs.killSignal); + swap(lhs.killTimeout, rhs.killTimeout); #else swap(lhs.pid, rhs.pid); #endif @@ -104,6 +111,31 @@ struct ProcessOptions int cloneFlags = 0; }; +/** + * Register a callback to be run by `startProcess()` in the forked + * child, before the child's main function. This is for state that + * doesn't survive a `fork()`, in particular objects owning a thread: + * the thread doesn't exist in the child, so such objects can be + * neither used nor destroyed there. Typical usage: + * + * static RegisterForkCallback resetFoo([]() { ... }); + * + * Note that callbacks are not run for `vfork()`ed children, since + * those share the parent's memory. Exceptions thrown by a callback + * are ignored, since there's not much the child can do about them. + */ +struct RegisterForkCallback +{ + typedef std::vector> Callbacks; + + static Callbacks & callbacks(); + + RegisterForkCallback(fun callback) + { + callbacks().push_back(std::move(callback)); + } +}; + #ifndef _WIN32 pid_t startProcess(fun processMain, const ProcessOptions & options = ProcessOptions()); #endif diff --git a/src/libutil/include/nix/util/provenance.hh b/src/libutil/include/nix/util/provenance.hh new file mode 100644 index 000000000000..1e45aca015ac --- /dev/null +++ b/src/libutil/include/nix/util/provenance.hh @@ -0,0 +1,61 @@ +#pragma once + +#include "nix/util/ref.hh" +#include "nix/util/canon-path.hh" + +#include + +#include + +namespace nix { + +struct Provenance +{ + virtual void anchor(); + + virtual ~Provenance() = default; + + static ref from_json_str(std::string_view); + + static std::shared_ptr from_json_str_optional(std::string_view); + + static ref from_json(const nlohmann::json & json); + + std::string to_json_str() const; + + virtual nlohmann::json to_json() const = 0; + +protected: + + using ProvenanceFactory = std::function(nlohmann::json)>; + + using RegisteredTypes = std::map; + + static RegisteredTypes & registeredTypes(); + +public: + + struct Register + { + Register(const std::string & type, ProvenanceFactory && factory) + { + registeredTypes().insert_or_assign(type, std::move(factory)); + } + }; +}; + +struct SubpathProvenance : public Provenance +{ + std::shared_ptr next; + CanonPath subpath; + + SubpathProvenance(std::shared_ptr next, const CanonPath & subpath) + : next(std::move(next)) + , subpath(subpath) + { + } + + nlohmann::json to_json() const override; +}; + +} // namespace nix diff --git a/src/libutil/include/nix/util/sentry.hh b/src/libutil/include/nix/util/sentry.hh new file mode 100644 index 000000000000..8da0fe867517 --- /dev/null +++ b/src/libutil/include/nix/util/sentry.hh @@ -0,0 +1,9 @@ +#pragma once + +#include "nix/util/fun.hh" + +namespace nix { + +extern fun setSentryTag; + +} diff --git a/src/libutil/include/nix/util/serialise.hh b/src/libutil/include/nix/util/serialise.hh index 67d8d3afb946..b0bb0fd49f25 100644 --- a/src/libutil/include/nix/util/serialise.hh +++ b/src/libutil/include/nix/util/serialise.hh @@ -79,6 +79,14 @@ public: void flush(); + /** + * Check whether the buffer has unflushed data. + */ + bool hasData() const noexcept + { + return bufPos; + } + protected: virtual void writeUnbuffered(std::string_view data) = 0; @@ -194,6 +202,12 @@ private: public: Descriptor fd; size_t written = 0; + /** + * A hint to the sink as to whether the destination is a regular file. + * Specifying a true for a non regular file doesn't affect correctness - only + * efficiency. + */ + bool isRegularFile:1 = false; FdSink() : fd(INVALID_DESCRIPTOR) @@ -781,6 +795,8 @@ public: { try { finish(); + } catch (EndOfFile &) { + // Ignore this, since an EOF will generally already have been encountered by the user of EnsureRead. } catch (...) { ignoreExceptionInDestructor(); } diff --git a/src/libutil/include/nix/util/signals.hh b/src/libutil/include/nix/util/signals.hh index 5124ebaccafd..0cfa7f3754dd 100644 --- a/src/libutil/include/nix/util/signals.hh +++ b/src/libutil/include/nix/util/signals.hh @@ -50,6 +50,37 @@ struct InterruptCallback virtual ~InterruptCallback(); }; +/** + * Portable identifiers for the signal events that callbacks can be + * registered for via `createSignalCallback()`. + */ +enum class SignalType { + /** + * SIGINT/SIGTERM/SIGHUP, or a programmatic `triggerInterrupt()`. + */ + Int, + /** + * SIGTSTP, called just before the process stops itself. + */ + Stop, + /** + * SIGCONT, i.e. the process was resumed after being stopped. + */ + Cont, + /** + * SIGWINCH, i.e. the terminal size changed. + */ + Winch, +}; + +/** + * Register a function that gets called from the signal handler thread + * (in a non-signal context) when the given signal type is received. + * + * @note Does nothing on Windows + */ +std::unique_ptr createSignalCallback(SignalType type, fun callback); + /** * Register a function that gets called on SIGINT (in a non-signal * context). diff --git a/src/libutil/include/nix/util/signature/local-keys.hh b/src/libutil/include/nix/util/signature/local-keys.hh index 789fb831f0f3..6d7620335174 100644 --- a/src/libutil/include/nix/util/signature/local-keys.hh +++ b/src/libutil/include/nix/util/signature/local-keys.hh @@ -41,24 +41,28 @@ struct Signature auto operator<=>(const Signature &) const = default; }; +enum KeyType { + Ed25519, + MLDSA44, + MLDSA65, + MLDSA87, + ECDSAP384, +}; + +KeyType parseKeyType(std::string_view s); + +const StringSet & getKeyTypes(); + +// FIXME: remove this class. struct Key { - std::string name; - std::string key; + const std::string name; + const std::string key; std::string to_string() const; protected: - /** - * Construct Key from a string in the format - * ‘:’. - * - * @param sensitiveValue Avoid displaying the raw Base64 in error - * messages to avoid leaking private keys. - */ - Key(std::string_view s, bool sensitiveValue); - Key(std::string_view name, std::string && key) : name(name) , key(std::move(key)) @@ -70,27 +74,35 @@ struct PublicKey; struct SecretKey : Key { - SecretKey(std::string_view s); + using Key::Key; + + virtual ~SecretKey() {}; + + static std::unique_ptr parse(std::string_view s); /** * Return a detached signature of the given string. */ - Signature signDetached(std::string_view s) const; + virtual Signature signDetached(std::string_view s) const; - PublicKey toPublicKey() const; + virtual std::unique_ptr toPublicKey() const; - static SecretKey generate(std::string_view name); + /** + * Return a PEM PKCS#8 encoding of this secret key. The Nix-specific + * key name is not included. Only OpenSSL-backed keys (ML-DSA and ECDSA P-384) are supported. + */ + virtual std::string toPEM() const; -private: - SecretKey(std::string_view name, std::string && key) - : Key(name, std::move(key)) - { - } + static std::unique_ptr generate(std::string_view name, KeyType type); }; struct PublicKey : Key { - PublicKey(std::string_view data); + using Key::Key; + + virtual ~PublicKey() {}; + + static std::unique_ptr parse(std::string_view s); /** * @return true iff `sig` and this key's names match, and `sig` is a @@ -104,20 +116,20 @@ struct PublicKey : Key * * @param sig the raw signature bytes (not Base64 encoded). */ - bool verifyDetachedAnon(std::string_view data, const Signature & sig) const; + virtual bool verifyDetachedAnon(std::string_view data, const Signature & sig) const; -private: - PublicKey(std::string_view name, std::string && key) - : Key(name, std::move(key)) - { - } - friend struct SecretKey; + /** + * Return a PEM SubjectPublicKeyInfo encoding of this public key. + * The Nix-specific key name is not included. Only OpenSSL-backed keys (ML-DSA and ECDSA P-384) are + * supported. + */ + virtual std::string toPEM() const; }; /** * Map from key names to public keys */ -typedef std::map PublicKeys; +typedef std::map> PublicKeys; /** * @return true iff ‘sig’ is a correct signature over ‘data’ using one diff --git a/src/libutil/include/nix/util/signature/signer.hh b/src/libutil/include/nix/util/signature/signer.hh index ebeedc87a765..9c8858075617 100644 --- a/src/libutil/include/nix/util/signature/signer.hh +++ b/src/libutil/include/nix/util/signature/signer.hh @@ -52,7 +52,7 @@ using Signers = std::map; */ struct LocalSigner : Signer { - LocalSigner(SecretKey && privateKey); + LocalSigner(std::unique_ptr && privateKey); Signature signDetached(std::string_view s) const override; @@ -60,8 +60,8 @@ struct LocalSigner : Signer private: - SecretKey privateKey; - PublicKey publicKey; + const std::unique_ptr privateKey; + const std::unique_ptr publicKey; }; } // namespace nix diff --git a/src/libutil/include/nix/util/source-accessor.hh b/src/libutil/include/nix/util/source-accessor.hh index cc904a720d2f..d00ba91373ff 100644 --- a/src/libutil/include/nix/util/source-accessor.hh +++ b/src/libutil/include/nix/util/source-accessor.hh @@ -10,6 +10,7 @@ namespace nix { struct Sink; +struct Provenance; /** * Note there is a decent chance this type soon goes away because the problem is solved another way. @@ -243,6 +244,13 @@ public: return std::nullopt; } + std::shared_ptr provenance; + + /** + * Return the provenance of the specified path, or `nullptr` if not available. + */ + virtual std::shared_ptr getProvenance(const CanonPath & path); + /** * Drop any cached state that could go stale across external filesystem * mutation (e.g. cached directory fds). @@ -302,7 +310,8 @@ ref makeFSSourceAccessor( * Construct an accessor that presents a "union" view of a vector of * underlying accessors. Earlier accessors take precedence over later. */ -ref makeUnionSourceAccessor(std::vector> && accessors); +ref +makeUnionSourceAccessor(std::vector> && accessors, std::shared_ptr displayAccessor); /** * Make a wrapper source accessor that caches positive lookup results. diff --git a/src/libutil/include/nix/util/source-path.hh b/src/libutil/include/nix/util/source-path.hh index f15b44fdd437..80e53c2677b9 100644 --- a/src/libutil/include/nix/util/source-path.hh +++ b/src/libutil/include/nix/util/source-path.hh @@ -114,6 +114,11 @@ struct SourcePath return {accessor, accessor->resolveSymlinks(path, mode)}; } + std::shared_ptr getProvenance() const + { + return accessor->getProvenance(path); + } + friend class std::hash; }; diff --git a/src/libutil/include/nix/util/table.hh b/src/libutil/include/nix/util/table.hh index 13e4506d5a30..0af33b66cc3b 100644 --- a/src/libutil/include/nix/util/table.hh +++ b/src/libutil/include/nix/util/table.hh @@ -2,10 +2,26 @@ #include "nix/util/types.hh" +#include + namespace nix { -typedef std::vector> Table; +struct TableCell +{ + std::string content; + + enum Alignment { Left, Right } alignment = Left; + + TableCell(std::string content, Alignment alignment = Left) + : content(std::move(content)) + , alignment(alignment) + { + } +}; + +using TableRow = std::vector; +using Table = std::vector; -void printTable(std::ostream & out, Table & table); +void printTable(std::ostream & out, Table & table, unsigned int width = std::numeric_limits::max()); } // namespace nix diff --git a/src/libutil/include/nix/util/tarfile.hh b/src/libutil/include/nix/util/tarfile.hh index 68eaa0823dda..2bfd11d2e056 100644 --- a/src/libutil/include/nix/util/tarfile.hh +++ b/src/libutil/include/nix/util/tarfile.hh @@ -36,6 +36,8 @@ struct TarArchive ~TarArchive(); }; +void unpackTarfile(Source & source, const std::filesystem::path & destDir); + void unpackTarfile(const std::filesystem::path & tarFile, const std::filesystem::path & destDir); time_t unpackTarfileToSink(TarArchive & archive, ExtendedFileSystemObjectSink & parseSink); diff --git a/src/libutil/include/nix/util/terminal.hh b/src/libutil/include/nix/util/terminal.hh index 5e35cbb95408..c70006bc51e3 100644 --- a/src/libutil/include/nix/util/terminal.hh +++ b/src/libutil/include/nix/util/terminal.hh @@ -44,6 +44,11 @@ void updateWindowSize(); */ std::pair getWindowSize(); +/** + * @return The number of columns of the terminal, or std::numeric_limits::max() if unknown. + */ +unsigned int getWindowWidth(); + /** * Get the slave name of a pseudoterminal in a thread-safe manner. * diff --git a/src/libutil/include/nix/util/thread-pool.hh b/src/libutil/include/nix/util/thread-pool.hh index d41e7a6f76e2..a6bb7cf02d1c 100644 --- a/src/libutil/include/nix/util/thread-pool.hh +++ b/src/libutil/include/nix/util/thread-pool.hh @@ -87,21 +87,27 @@ private: * its dependencies have been processed. */ template -void processGraph(const std::set & nodes, fun(const T &)> getEdges, fun processNode) +void processGraph( + const std::set & nodes, + fun(const T &)> getEdges, + fun processNode, + bool discoverNodes = false, + size_t maxThreads = 0) { struct Graph { + std::set known; std::set left; std::map> refs, rrefs; }; - Sync graph_(Graph{nodes, {}, {}}); + Sync graph_(Graph{nodes, nodes, {}, {}}); std::function worker; - /* Create pool last to ensure threads are stopped before other destructors - * run */ - ThreadPool pool; + /* Create pool last to ensure threads are stopped before other + destructors run. */ + ThreadPool pool(maxThreads); worker = [&](const T & node) { { @@ -118,11 +124,19 @@ void processGraph(const std::set & nodes, fun(const T &)> getEdge { auto graph(graph_.lock()); - for (auto & ref : refs) + for (auto & ref : refs) { + if (discoverNodes) { + auto [i, inserted] = graph->known.insert(ref); + if (inserted) { + pool.enqueue(std::bind(worker, std::ref(*i))); + graph->left.insert(ref); + } + } if (graph->left.count(ref)) { graph->refs[node].insert(ref); graph->rrefs[ref].insert(node); } + } if (graph->refs[node].empty()) goto doWork; } diff --git a/src/libutil/include/nix/util/url-parts.hh b/src/libutil/include/nix/util/url-parts.hh index c57c32e61cf1..0c4e7b6be2f6 100644 --- a/src/libutil/include/nix/util/url-parts.hh +++ b/src/libutil/include/nix/util/url-parts.hh @@ -8,7 +8,7 @@ namespace nix { // URI stuff. const static std::string pctEncoded = "(?:%[0-9a-fA-F][0-9a-fA-F])"; -const static std::string unreservedRegex = "(?:[a-zA-Z0-9-._~])"; +const static std::string unreservedRegex = "(?:[a-zA-Z0-9._~-])"; const static std::string subdelimsRegex = "(?:[!$&'\"()*+,;=])"; const static std::string pcharRegex = "(?:" + unreservedRegex + "|" + pctEncoded + "|" + subdelimsRegex + "|[:@])"; const static std::string fragmentRegex = "(?:" + pcharRegex + "|[/? \"^])*"; diff --git a/src/libutil/include/nix/util/url.hh b/src/libutil/include/nix/util/url.hh index 990f6d02b2d0..e3322cfb0d28 100644 --- a/src/libutil/include/nix/util/url.hh +++ b/src/libutil/include/nix/util/url.hh @@ -229,6 +229,11 @@ struct ParsedURL */ std::string renderPath(bool encode = false) const; + /** + * Like to_string(), but removes query strings and passwords. + */ + std::string renderSanitized() const; + auto operator<=>(const ParsedURL & other) const noexcept = default; /** diff --git a/src/libutil/include/nix/util/util.hh b/src/libutil/include/nix/util/util.hh index 021da6484d49..f0c3eb40d086 100644 --- a/src/libutil/include/nix/util/util.hh +++ b/src/libutil/include/nix/util/util.hh @@ -251,6 +251,11 @@ void ignoreExceptionInDestructor(Verbosity lvl = lvlError); */ void ignoreExceptionExceptInterrupt(Verbosity lvl = lvlError); +/** + * Like ignoreExceptionExceptInterrupt(), but specifies the error prefix. + */ +void logExceptionExceptInterrupt(std::string_view prefix = "error: ", Verbosity lvl = lvlError); + /** * Tree formatting. */ @@ -322,9 +327,15 @@ typename T::mapped_type * get(T & map, const K & key) template typename T::mapped_type * get(T && map, const K & key) = delete; -/** - * Look up a value in a `boost::concurrent_flat_map`. - */ +template +std::optional getOptional(const T & map, const typename T::key_type & key) +{ + auto i = map.find(key); + if (i == map.end()) + return std::nullopt; + return {i->second}; +} + template std::optional getConcurrent(const T & map, const typename T::key_type & key) { diff --git a/src/libutil/linux/cgroup.cc b/src/libutil/linux/cgroup.cc index 9ebc3ab777d7..151b55d44b1e 100644 --- a/src/libutil/linux/cgroup.cc +++ b/src/libutil/linux/cgroup.cc @@ -174,4 +174,23 @@ CanonPath getRootCgroup() return rootCgroup; } +std::set getPidsInCgroup(const std::filesystem::path & cgroup) +{ + if (!pathExists(cgroup)) + return {}; + + auto procsFile = cgroup / "cgroup.procs"; + + std::set result; + + for (auto & pidStr : tokenizeString>(readFile(procsFile))) { + if (auto o = string2Int(pidStr)) + result.insert(*o); + else + throw Error("invalid PID '%s'", pidStr); + } + + return result; +} + } // namespace nix::linux diff --git a/src/libutil/linux/include/nix/util/cgroup.hh b/src/libutil/linux/include/nix/util/cgroup.hh index 7b50ffe8daec..7e54dbb8885a 100644 --- a/src/libutil/linux/include/nix/util/cgroup.hh +++ b/src/libutil/linux/include/nix/util/cgroup.hh @@ -41,4 +41,9 @@ CanonPath getCurrentCgroup(); */ CanonPath getRootCgroup(); +/** + * Get the PIDs of all processes in the given cgroup. + */ +std::set getPidsInCgroup(const std::filesystem::path & cgroup); + } // namespace nix::linux diff --git a/src/libutil/logging.cc b/src/libutil/logging.cc index 02e238e5712d..323de64105c3 100644 --- a/src/libutil/logging.cc +++ b/src/libutil/logging.cc @@ -7,11 +7,16 @@ #include "nix/util/position.hh" #include "nix/util/sync.hh" #include "nix/util/unix-domain-socket.hh" +#include "nix/util/exit.hh" #include #include #include +#include +#include +#include + namespace nix { void LoggerSettings::anchor() {} @@ -20,7 +25,7 @@ LoggerSettings loggerSettings; static GlobalConfig::Register rLoggerSettings(&loggerSettings); -static thread_local ActivityId curActivity = 0; +[[gnu::tls_model("initial-exec")]] static thread_local ActivityId curActivity = 0; ActivityId getCurActivity() { @@ -32,6 +37,23 @@ void setCurActivity(const ActivityId activityId) curActivity = activityId; } +[[gnu::tls_model("initial-exec")]] static thread_local unsigned int remoteLogSourceDepth = 0; + +RemoteLogSource::RemoteLogSource() +{ + remoteLogSourceDepth++; +} + +RemoteLogSource::~RemoteLogSource() +{ + remoteLogSourceDepth--; +} + +bool isRemoteLogSource() +{ + return remoteLogSourceDepth > 0; +} + /** * This is a raw pointer to allow it to leak. * Avoids races in activity teardown. @@ -40,11 +62,43 @@ Logger * logger = makeSimpleLogger(true).release(); Logger::~Logger() {} -void Logger::warn(const std::string & msg) +void Logger::warn(const std::string & msg) noexcept { log(lvlWarn, ANSI_WARNING "warning:" ANSI_NORMAL " " + msg); } +void Logger::printException(const std::exception_ptr & ex, std::string_view programName) noexcept +{ + /* Note: we log to `this` rather than to the global `logger`, so + that a `TeeLogger` can forward this to each of its loggers + without the message being printed once per logger. */ + auto doLog = [&](const BaseError & e) { + try { + logEI(e.info()); + } catch (...) { + log(lvlError, ANSI_RED "error:" ANSI_NORMAL " Exception while printing an exception."); + } + }; + + constexpr std::string_view error = ANSI_RED "error:" ANSI_NORMAL " "; + + try { + std::rethrow_exception(ex); + } catch (Exit &) { + } catch (UsageError & e) { + doLog(e); + log(lvlError, fmt("\nTry '%1% --help' for more information.", programName)); + } catch (BaseError & e) { + doLog(e); + } catch (std::bad_alloc & e) { + log(lvlError, std::string(error) + "out of memory"); + } catch (std::exception & e) { + log(lvlError, std::string(error) + e.what()); + } catch (...) { + log(lvlError, std::string(error) + "unknown exception"); + } +} + void Logger::writeToStdout(std::string_view s) { Descriptor standard_out = getStandardOutput(); @@ -86,7 +140,7 @@ class SimpleLogger : public Logger return printBuildLogs; } - void log(Verbosity lvl, std::string_view s) override + void log(Verbosity lvl, std::string_view s) noexcept override { if (lvl > verbosity) return; @@ -124,7 +178,7 @@ class SimpleLogger : public Logger writeToStderr(prefix + filterANSIEscapes(s, !tty) + "\n"); } - void logEI(const ErrorInfo & ei) override + void logEI(const ErrorInfo & ei) noexcept override { std::ostringstream oss; showErrorInfo(oss, ei, loggerSettings.showTrace.get()); @@ -138,21 +192,18 @@ class SimpleLogger : public Logger ActivityType type, const std::string & s, const Fields & fields, - ActivityId parent) override + ActivityId parent) noexcept override { if (lvl <= verbosity && !s.empty()) log(lvl, s + "..."); } - void result(ActivityId act, ResultType type, const Fields & fields) override + void result(ActivityId act, ResultType type, const Fields & fields) noexcept override { - if (type == resBuildLogLine && printBuildLogs) { - auto lastLine = fields[0].s; - printError(lastLine); - } else if (type == resPostBuildLogLine && printBuildLogs) { - auto lastLine = fields[0].s; - printError("post-build-hook: " + lastLine); - } + if (type == resBuildLogLine && printBuildLogs) + printError(std::get(fields[0].raw)); + else if (type == resPostBuildLogLine && printBuildLogs) + printError("post-build-hook: " + std::get(fields[0].raw)); } }; @@ -160,7 +211,7 @@ class SimpleLogger : public Logger Verbosity verbosity = lvlInfo; -static void writeFullLogging(Descriptor fd, std::string_view s) +static void writeFullLogging(Descriptor fd, std::string_view s) noexcept { try { writeFull(fd, s, false); @@ -172,7 +223,7 @@ static void writeFullLogging(Descriptor fd, std::string_view s) } } -void writeToStderr(std::string_view s) +void writeToStderr(std::string_view s) noexcept { writeFullLogging(getStandardError(), s); } @@ -206,6 +257,19 @@ Activity::Activity( logger.startActivity(id, lvl, type, s, fields, parent); } +Activity::Activity( + Logger & logger, + Verbosity lvl, + std::string_view name, + Logger::ActivityMetadata metadata, + std::string_view s, + ActivityId parent) + : logger(logger) + , id(nextId++ + (((uint64_t) getPid()) << 32)) +{ + logger.startActivity(id, lvl, name, metadata, s, parent); +} + void to_json(nlohmann::json & json, std::shared_ptr pos) { if (pos) { @@ -223,6 +287,16 @@ void to_json(nlohmann::json & json, std::shared_ptr pos) namespace { +std::string getSessionId() +{ + if (!loggerSettings.sessionId.get().empty()) + return loggerSettings.sessionId.get(); + + // Generate a UUIDv7 as the session ID. + static std::string uuid = boost::uuids::to_string(boost::uuids::time_generator_v7()()); + return uuid; +} + struct JSONLogger : Logger { Descriptor fd; @@ -245,10 +319,10 @@ struct JSONLogger : Logger return; auto & arr = json["fields"] = nlohmann::json::array(); for (auto & f : fields) - if (f.type == Logger::Field::tInt) - arr.push_back(f.i); - else if (f.type == Logger::Field::tString) - arr.push_back(f.s); + if (auto p = std::get_if(&f.raw)) + arr.push_back(*p); + else if (auto p = std::get_if(&f.raw)) + arr.push_back(*p); else unreachable(); } @@ -260,8 +334,10 @@ struct JSONLogger : Logger Sync _state; - void write(const nlohmann::json & json) + void write(nlohmann::json json) { + json["sid"] = getSessionId(); + auto line = (includeNixPrefix ? "@nix " : "") + json.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace) + "\n"; @@ -281,16 +357,16 @@ struct JSONLogger : Logger } } - void log(Verbosity lvl, std::string_view s) override + void log(Verbosity lvl, std::string_view s) noexcept override { nlohmann::json json; json["action"] = "msg"; json["level"] = lvl; json["msg"] = s; - write(json); + write(std::move(json)); } - void logEI(const ErrorInfo & ei) override + void logEI(const ErrorInfo & ei) noexcept override { std::ostringstream oss; showErrorInfo(oss, ei, loggerSettings.showTrace.get()); @@ -314,7 +390,7 @@ struct JSONLogger : Logger json["trace"] = traces; } - write(json); + write(std::move(json)); } void startActivity( @@ -323,7 +399,7 @@ struct JSONLogger : Logger ActivityType type, const std::string & s, const Fields & fields, - ActivityId parent) override + ActivityId parent) noexcept override { nlohmann::json json; json["action"] = "start"; @@ -333,25 +409,58 @@ struct JSONLogger : Logger json["text"] = s; json["parent"] = parent; addFields(json, fields); - write(json); + write(std::move(json)); } - void stopActivity(ActivityId act) override + void startActivity( + ActivityId act, + Verbosity lvl, + std::string_view name, + ActivityMetadata metadata, + std::string_view s, + ActivityId parent) noexcept override + { + nlohmann::json json; + json["action"] = "start"; + json["id"] = act; + json["level"] = lvl; + json["type"] = actStringly; + json["text"] = s; + json["parent"] = parent; + json["name"] = name; + auto payload = nlohmann::json::object(); + for (auto & [key, value] : metadata) + std::visit([&](auto & v) { payload[std::string(key)] = v; }, value.raw); + json["payload"] = std::move(payload); + write(std::move(json)); + } + + void stopActivity(ActivityId act) noexcept override { nlohmann::json json; json["action"] = "stop"; json["id"] = act; - write(json); + write(std::move(json)); } - void result(ActivityId act, ResultType type, const Fields & fields) override + void result(ActivityId act, ResultType type, const Fields & fields) noexcept override { nlohmann::json json; json["action"] = "result"; json["id"] = act; json["type"] = type; addFields(json, fields); - write(json); + write(std::move(json)); + } + + void result(ActivityId act, ResultType type, const nlohmann::json & j) noexcept override + { + nlohmann::json json; + json["action"] = "result"; + json["id"] = act; + json["type"] = type; + json["payload"] = j; + write(std::move(json)); } }; @@ -390,18 +499,19 @@ std::unique_ptr makeJSONLogger(const std::filesystem::path & path, bool return std::make_unique(std::move(fd), includeNixPrefix); } +std::string getTraceparent(const Headers & headers) +{ + for (auto & [name, value] : headers) + if (name == "traceparent") + return value; + return ""; +} + void applyJSONLogger() { if (auto & opt = loggerSettings.jsonLogPath.get()) { try { - std::vector> loggers; - loggers.push_back(makeJSONLogger(*opt, false)); - try { - logger = makeTeeLogger(std::unique_ptr(logger), std::move(loggers)).release(); - } catch (...) { - // `logger` is now gone so give up. - abort(); - } + applyExtraLogger(makeJSONLogger(*opt, false)); } catch (...) { ignoreExceptionExceptInterrupt(); } diff --git a/src/libutil/make.nix b/src/libutil/make.nix new file mode 100644 index 000000000000..0682a08a799e --- /dev/null +++ b/src/libutil/make.nix @@ -0,0 +1,59 @@ +# libnixutil, transcribed from meson.build and its subdirectories. +{ nixMake }: + +nixMake.mkComponent { + name = "determinate-nix-util"; + libName = "nixutil"; + + root = ./.; + + # Every .cc file under the root is a compilation unit, except these. + excludeSources = [ + "windows" + "freebsd" + ]; + + includeDirs = [ + "" + "include" + "unix" + "unix/include" + "linux/include" + "widecharwidth" + ]; + + files = nixMake.commonSupportFiles // { + # Interposes __cxa_throw; linked into libnixutil only. + "nix-meson-build-support/common/cxa-throw/interpose-cxa-throw.cc" = + ../../nix-meson-build-support/common/cxa-throw/interpose-cxa-throw.cc; + "nix-meson-build-support/common/cxa-throw/is-logic-error.hh" = + ../../nix-meson-build-support/common/cxa-throw/is-logic-error.hh; + + }; + + configHeaders = { + "include/nix/util/config.hh" = { + NIX_UBSAN_ENABLED = 0; + NIX_ASAN_ENABLED = 0; + }; + "util-config-private.hh" = { + HAVE_LIBCPUID = 1; + HAVE_POSIX_FALLOCATE = 1; + }; + "unix/util-unix-config-private.hh" = { + HAVE_CLOSE_RANGE = 1; + HAVE_COPY_FILE_RANGE = 1; + HAVE_DECL_AT_SYMLINK_NOFOLLOW = 1; + HAVE_F_GETPATH = 0; + HAVE_LUTIMES = 1; + HAVE_PIPE2 = 1; + HAVE_STRSIGNAL = 1; + HAVE_SYSCONF = 1; + HAVE_UTIMENSAT = 1; + }; + }; + + extraCxxFlags = nixMake.weakVtablesFlags; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; +} diff --git a/src/libutil/memory-source-accessor.cc b/src/libutil/memory-source-accessor.cc index ba2816fc0f2d..aac4c14358da 100644 --- a/src/libutil/memory-source-accessor.cc +++ b/src/libutil/memory-source-accessor.cc @@ -73,7 +73,7 @@ void MemorySourceAccessor::readFile(const CanonPath & path, Sink & sink, fun(&f->raw)) return s->target; else diff --git a/src/libutil/meson.build b/src/libutil/meson.build index b61403b51c4f..bf3a1f3b0833 100644 --- a/src/libutil/meson.build +++ b/src/libutil/meson.build @@ -136,6 +136,10 @@ config_priv_h = configure_file( subdir('nix-meson-build-support/common') +if have_cxa_throw + deps_other += cxa_throw_dep +endif + sources = [ config_priv_h ] + files( 'archive.cc', 'args.cc', @@ -161,6 +165,7 @@ sources = [ config_priv_h ] + files( 'file-content-address.cc', 'file-descriptor.cc', 'file-system.cc', + 'forwarding-source-accessor.cc', 'fs-sink.cc', 'git.cc', 'hash.cc', @@ -177,6 +182,7 @@ sources = [ config_priv_h ] + files( 'position.cc', 'posix-source-accessor.cc', 'processes.cc', + 'provenance.cc', 'serialise.cc', 'signature/local-keys.cc', 'signature/signer.cc', @@ -224,11 +230,10 @@ subdir('nix-meson-build-support/windows-version') this_library = library( 'nixutil', sources, - soversion : nix_soversion, dependencies : deps_public + deps_private + deps_other, include_directories : include_dirs, link_args : linker_export_flags, - prelink : true, # For C++ static initializers + prelink : prelink, # For C++ static initializers install : true, cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [], ) diff --git a/src/libutil/mounted-source-accessor.cc b/src/libutil/mounted-source-accessor.cc index 68e01ea6659f..0346f980d873 100644 --- a/src/libutil/mounted-source-accessor.cc +++ b/src/libutil/mounted-source-accessor.cc @@ -110,6 +110,12 @@ struct MountedSourceAccessorImpl : MountedSourceAccessor auto [accessor, subpath] = resolve(path); return accessor->getFingerprint(subpath); } + + std::shared_ptr getProvenance(const CanonPath & path) override + { + auto [accessor, subpath] = resolve(path); + return accessor->getProvenance(subpath); + } }; } // namespace diff --git a/src/libutil/package.nix b/src/libutil/package.nix index 049c75c5805d..fe6fcb75f454 100644 --- a/src/libutil/package.nix +++ b/src/libutil/package.nix @@ -23,7 +23,7 @@ let in mkMesonLibrary (finalAttrs: { - pname = "nix-util"; + pname = "determinate-nix-util"; inherit version; workDir = ./.; diff --git a/src/libutil/posix-source-accessor.cc b/src/libutil/posix-source-accessor.cc index 55fa6342f451..5fe44d4e5403 100644 --- a/src/libutil/posix-source-accessor.cc +++ b/src/libutil/posix-source-accessor.cc @@ -150,7 +150,7 @@ void PosixFileSourceAccessor::readFile(const CanonPath & path, Sink & sink, fun< /* The most important invariant we care about here is writing exactly size bytes to the sink. copyFdRange should throw an EndOfFile if we fail to read `size` bytes. */ - copyFdRange(fd.get(), /*offset=*/0, size, sink); + copyFdRange(fd.get(), /*offset=*/0, size, sink, /*tryCoW=*/true); } bool PosixFileSourceAccessor::pathExists(const CanonPath & path) @@ -468,7 +468,7 @@ void PosixDirectorySourceAccessor::readFile(const CanonPath & path, Sink & sink, if (!fileFd) { if (errno == ENOENT) - throw FileNotFound("file '%s' does not exist", showPath(path)); + throw FileNotFound("path '%s' does not exist", showPath(path)); throw SysError("opening '%s'", showPath(path)); } diff --git a/src/libutil/processes.cc b/src/libutil/processes.cc index 0211557034cf..4c79e55b926c 100644 --- a/src/libutil/processes.cc +++ b/src/libutil/processes.cc @@ -5,6 +5,12 @@ namespace nix { void ExecError::anchor() {} +RegisterForkCallback::Callbacks & RegisterForkCallback::callbacks() +{ + static Callbacks callbacks; + return callbacks; +} + Pid & Pid::operator=(Pid && other) noexcept { swap(*this, other); diff --git a/src/libutil/provenance.cc b/src/libutil/provenance.cc new file mode 100644 index 000000000000..f83fa624c0e9 --- /dev/null +++ b/src/libutil/provenance.cc @@ -0,0 +1,83 @@ +#include "nix/util/provenance.hh" +#include "nix/util/json-utils.hh" +#include "nix/util/override-provenance-source-accessor.hh" + +namespace nix { + +void Provenance::anchor() {} + +struct UnknownProvenance : Provenance +{ + nlohmann::json payload; + + UnknownProvenance(nlohmann::json payload) + : payload(std::move(payload)) + { + } + + nlohmann::json to_json() const override + { + return payload; + } + + void anchor() override; +}; + +void UnknownProvenance::anchor() {} + +Provenance::RegisteredTypes & Provenance::registeredTypes() +{ + static Provenance::RegisteredTypes types; + return types; +} + +ref Provenance::from_json_str(std::string_view s) +{ + return from_json(nlohmann::json::parse(s)); +} + +std::shared_ptr Provenance::from_json_str_optional(std::string_view s) +{ + if (s.empty()) + return nullptr; + return Provenance::from_json_str(s); +} + +ref Provenance::from_json(const nlohmann::json & json) +{ + auto & obj = getObject(json); + + auto type = getString(valueAt(obj, "type")); + + auto it = registeredTypes().find(type); + if (it == registeredTypes().end()) + return make_ref(obj); + + return it->second(obj); +} + +std::string Provenance::to_json_str() const +{ + return to_json().dump(); +} + +nlohmann::json SubpathProvenance::to_json() const +{ + return { + {"type", "subpath"}, + {"subpath", subpath.abs()}, + {"next", next ? next->to_json() : nlohmann::json(nullptr)}, + }; +} + +Provenance::Register registerSubpathProvenance("subpath", [](nlohmann::json json) { + auto & obj = getObject(json); + std::shared_ptr next; + if (auto p = optionalValueAt(obj, "next"); p && !p->is_null()) + next = Provenance::from_json(*p); + return make_ref(next, CanonPath(getString(valueAt(obj, "subpath")))); +}); + +void OverrideProvenanceSourceAccessor::anchor() {} + +} // namespace nix diff --git a/src/libutil/serialise.cc b/src/libutil/serialise.cc index eb1209843179..f89af274202f 100644 --- a/src/libutil/serialise.cc +++ b/src/libutil/serialise.cc @@ -1,4 +1,5 @@ #include "nix/util/serialise.hh" +#include "nix/util/coroutine-gc.hh" #include "nix/util/file-descriptor.hh" #include "nix/util/signals.hh" #include "nix/util/socket.hh" @@ -372,10 +373,107 @@ void StringSource::skip(size_t len) /* 512KiB is a conservative estimate for deeply nested NARs, which are limited to 64 levels. We also tend to allocate rather large buffers on the stack, so we should leave plenty of headroom. Note that no evaluation is supposed to - happen on sourceToSink/sinkToSource coroutine stacks (for Boehm GC reasons), - which requires much more stack space. */ + happen on sourceToSink/sinkToSource coroutine stacks (they are too small for + that), though the GC hooks below do make them scannable. */ static constexpr size_t defaultCoroutineStackSize = 512 * 1024; +void * (*coroStackRegister)(void * base, size_t size) = nullptr; +void (*coroStackUnregister)(void * cookie) = nullptr; +void * (*coroSwitchTo)(void * cookie) = nullptr; +void (*coroSwitchBack)(void * prevHandle) = nullptr; +void (*coroEnter)(void * cookie) = nullptr; +void * (*coroYield)() = nullptr; +void (*coroResume)(void * handle) = nullptr; + +namespace { + +/** + * A stack allocator that reports the coroutine stacks to the GC hooks + * (see `coroutine-gc.hh`), storing the resulting cookie in the owning + * object (which must outlive the coroutine). + */ +struct GCTrackedStackAllocator +{ + void ** cookie; + + boost::context::stack_context allocate() + { + auto sctx = boost::coroutines2::protected_fixedsize_stack(defaultCoroutineStackSize).allocate(); + *cookie = coroStackRegister ? coroStackRegister(sctx.sp, sctx.size) : nullptr; + return sctx; + } + + void deallocate(boost::context::stack_context & sctx) + { + /* Note: the hooks may have been reset to null since the + allocation (see `coroutine-gc.hh`). */ + if (*cookie && coroStackUnregister) + coroStackUnregister(*cookie); + *cookie = nullptr; + boost::coroutines2::protected_fixedsize_stack(defaultCoroutineStackSize).deallocate(sctx); + } +}; + +/** + * Notify the GC hooks that the current thread is about to switch onto + * (and, on destruction, back off) the coroutine stack identified by + * `cookie`. Construct just before resuming a coroutine + * (this includes destroying a suspended one, which unwinds on its own + * stack); the switch back happens when the coroutine yields or + * finishes. + */ +struct CoroutineGuard +{ + void * prev = nullptr; + bool active = false; + + CoroutineGuard(void * cookie) + { + if (coroSwitchTo) { + prev = coroSwitchTo(cookie); + active = true; + } + } + + ~CoroutineGuard() + { + if (active && coroSwitchBack) + coroSwitchBack(prev); + } +}; + +/** + * Notify the GC hooks that the current thread is about to yield from + * a coroutine (and, on destruction, that the yield has returned). + * Construct just before calling the coroutine's `yield`, in the same + * scope, so that the destructor also runs when the yield throws + * (e.g. `forced_unwind` when a suspended coroutine is destroyed). + * Note that this deliberately doesn't refer to the coroutine's own + * stack: the yield may be executed on another coroutine's stack (see + * `coroutine-gc.hh`). + */ +struct CoroutineYieldGuard +{ + void * handle = nullptr; + bool active = false; + + CoroutineYieldGuard() + { + if (coroYield) { + handle = coroYield(); + active = true; + } + } + + ~CoroutineYieldGuard() + { + if (active && coroResume) + coroResume(handle); + } +}; + +} // namespace + std::unique_ptr sourceToSink(fun reader) { struct SourceToSink : FinishSink @@ -384,12 +482,22 @@ std::unique_ptr sourceToSink(fun reader) fun reader; std::optional coro; + void * stackCookie = nullptr; SourceToSink(fun reader) : reader(reader) { } + ~SourceToSink() + { + /* Destroying a suspended coroutine unwinds it on its own + stack, so this too is a switch onto the coroutine + stack. */ + CoroutineGuard guard{stackCookie}; + coro.reset(); + } + std::string_view cur; void operator()(std::string_view in) override @@ -398,23 +506,30 @@ std::unique_ptr sourceToSink(fun reader) return; cur = in; + CoroutineGuard guard{stackCookie}; + if (!coro) { - coro = coro_t::push_type( - boost::coroutines2::protected_fixedsize_stack(defaultCoroutineStackSize), - [&](coro_t::pull_type & yield) { - LambdaSource source([&](char * out, size_t out_len) { - if (cur.empty()) { + coro = coro_t::push_type(GCTrackedStackAllocator{&stackCookie}, [&](coro_t::pull_type & yield) { + /* Note: `stackCookie` has been set by the stack + allocator before the body started. */ + if (coroEnter) + coroEnter(stackCookie); + LambdaSource source([&](char * out, size_t out_len) { + if (cur.empty()) { + { + CoroutineYieldGuard guard; yield(); - if (yield.get()) - throw EndOfFile("coroutine has finished"); } + if (yield.get()) + throw EndOfFile("coroutine has finished"); + } - size_t n = cur.copy(out, out_len); - cur.remove_prefix(n); - return n; - }); - reader(source); + size_t n = cur.copy(out, out_len); + cur.remove_prefix(n); + return n; }); + reader(source); + }); } if (!*coro) { @@ -428,8 +543,10 @@ std::unique_ptr sourceToSink(fun reader) void finish() override { - if (coro && *coro) + if (coro && *coro) { + CoroutineGuard guard{stackCookie}; (*coro)(true); + } } }; @@ -445,6 +562,7 @@ std::unique_ptr sinkToSource(fun writer, fun eof) fun writer; fun eof; std::optional coro; + void * stackCookie = nullptr; SinkToSource(fun writer, fun eof) : writer(writer) @@ -452,38 +570,55 @@ std::unique_ptr sinkToSource(fun writer, fun eof) { } + ~SinkToSource() + { + /* See `SourceToSink::~SourceToSink()`. */ + CoroutineGuard guard{stackCookie}; + coro.reset(); + } + std::string_view cur; size_t read(char * data, size_t len) override { + CoroutineGuard guard{stackCookie}; + bool hasCoro = coro.has_value(); if (!hasCoro) { - coro = coro_t::pull_type( - boost::coroutines2::protected_fixedsize_stack(defaultCoroutineStackSize), - [&](coro_t::push_type & yield) { - /* Feed the consumer in chunks, instead of on each write - to avoid excessive context switching. parseDump does - lots of small writes to the sink, which we should - accumulate. */ - struct CoroBufferedSink : BufferedSink + coro = coro_t::pull_type(GCTrackedStackAllocator{&stackCookie}, [&](coro_t::push_type & yield) { + /* Note: `stackCookie` has been set by the stack + allocator before the body started. */ + if (coroEnter) + coroEnter(stackCookie); + + /* Feed the consumer in chunks, instead of on each write + to avoid excessive context switching. parseDump does + lots of small writes to the sink, which we should + accumulate. */ + struct CoroBufferedSink : BufferedSink + { + coro_t::push_type & yield; + + /* Note: `writer()` may hand this sink to a + nested coroutine (e.g. a `sourceToSink` + decompressor), in which case this runs, and + yields, on that coroutine's stack. */ + void writeUnbuffered(std::string_view data) override { - coro_t::push_type & yield; - - void writeUnbuffered(std::string_view data) override - { - yield(data); - } + CoroutineYieldGuard guard; + yield(data); + } - CoroBufferedSink(coro_t::push_type & yield) - : yield(yield) - { - } - }; + CoroBufferedSink(coro_t::push_type & yield) + : yield(yield) + { + } + }; - CoroBufferedSink sink(yield); - writer(sink); - sink.flush(); - }); + CoroBufferedSink sink(yield); + writer(sink); + sink.flush(); + }); } if (cur.empty()) { diff --git a/src/libutil/signature/local-keys.cc b/src/libutil/signature/local-keys.cc index 8afa3eb728f2..207aea85a719 100644 --- a/src/libutil/signature/local-keys.cc +++ b/src/libutil/signature/local-keys.cc @@ -1,16 +1,31 @@ #include #include #include +#include +#include +#include #include "nix/util/base-n.hh" #include "nix/util/signature/local-keys.hh" #include "nix/util/json-utils.hh" #include "nix/util/util.hh" +#include "nix/util/deleter.hh" namespace nix { namespace { +using AutoEVP_PKEY = std::unique_ptr>; +using AutoEVP_PKEY_CTX = std::unique_ptr>; +using AutoEVP_MD_CTX = std::unique_ptr>; +using AutoBIO = std::unique_ptr>; + +std::string_view keyNamePart(std::string_view s) +{ + auto colon = s.find(':'); + return colon == std::string_view::npos ? std::string_view{} : s.substr(0, colon); +} + /** * Parse a colon-separated string where the second part is Base64-encoded. * @@ -45,6 +60,99 @@ std::string serializeColonBase64(std::string_view name, std::string_view data) return std::string(name) + ":" + base64::encode(std::as_bytes(std::span{data.data(), data.size()})); } +const char * toOpenSSLKeyType(KeyType type) +{ + switch (type) { + case KeyType::MLDSA44: + return "ML-DSA-44"; + case KeyType::MLDSA65: + return "ML-DSA-65"; + case KeyType::MLDSA87: + return "ML-DSA-87"; + case KeyType::ECDSAP384: + return "EC"; + case KeyType::Ed25519: + default: + throw Error("key type is not supported by OpenSSL"); + } +} + +/** + * Return the digest algorithm to use with `EVP_DigestSign` / `EVP_DigestVerify` + * for a given key type, or `nullptr` if the algorithm performs hashing + * internally (as ML-DSA does). + */ +const char * digestForKeyType(KeyType type) +{ + switch (type) { + case KeyType::MLDSA44: + case KeyType::MLDSA65: + case KeyType::MLDSA87: + return nullptr; + case KeyType::ECDSAP384: + return "SHA384"; + case KeyType::Ed25519: + default: + unreachable(); + } +} + +/** + * Determine the `KeyType` of a parsed OpenSSL key, or return `std::nullopt` if + * we don't support it. + */ +std::optional detectOpenSSLKeyType(EVP_PKEY * pkey) +{ + if (EVP_PKEY_is_a(pkey, "ML-DSA-44") == 1) + return KeyType::MLDSA44; + if (EVP_PKEY_is_a(pkey, "ML-DSA-65") == 1) + return KeyType::MLDSA65; + if (EVP_PKEY_is_a(pkey, "ML-DSA-87") == 1) + return KeyType::MLDSA87; + if (EVP_PKEY_is_a(pkey, "EC") == 1) { + char curve[64] = {0}; + size_t curveLen = 0; + if (EVP_PKEY_get_utf8_string_param(pkey, OSSL_PKEY_PARAM_GROUP_NAME, curve, sizeof(curve), &curveLen) == 1 + && std::string_view(curve, curveLen) == "secp384r1") + return KeyType::ECDSAP384; + } + return std::nullopt; +} + +/** + * Parse a DER-encoded PKCS#8 `PrivateKeyInfo`. + */ +AutoEVP_PKEY parsePrivateKey(std::string_view der) +{ + auto p = (const unsigned char *) der.data(); + AutoEVP_PKEY pkey(d2i_AutoPrivateKey(nullptr, &p, der.size())); + return pkey; +} + +/** + * Parse a DER-encoded `SubjectPublicKeyInfo`. + */ +AutoEVP_PKEY parsePublicKey(std::string_view der) +{ + auto p = (const unsigned char *) der.data(); + AutoEVP_PKEY pkey(d2i_PUBKEY(nullptr, &p, der.size())); + return pkey; +} + +AutoEVP_PKEY parsePublicKey(std::string_view der, KeyType type) +{ + auto pkey = parsePublicKey(der); + + auto detected = pkey ? detectOpenSSLKeyType(pkey.get()) : std::nullopt; + if (!detected || *detected != type) + throw Error( + "public key is not '%s' (got '%s')", + toOpenSSLKeyType(type), + pkey ? EVP_PKEY_get0_type_name(pkey.get()) : ""); + + return pkey; +} + } // anonymous namespace Signature Signature::parse(std::string_view s) @@ -81,19 +189,32 @@ Strings Signature::toStrings(const std::set & sigs) return res; } -Key::Key(std::string_view s, bool sensitiveValue) +static std::unordered_map keyTypeMap{ + {"ed25519", KeyType::Ed25519}, + {"ml-dsa-44", KeyType::MLDSA44}, + {"ml-dsa-65", KeyType::MLDSA65}, + {"ml-dsa-87", KeyType::MLDSA87}, + {"ecdsa-p384", KeyType::ECDSAP384}, +}; + +const StringSet & getKeyTypes() { - try { - auto [parsedName, parsedKey] = parseColonBase64(s, "key"); - name = std::move(parsedName); - key = std::move(parsedKey); - } catch (Error & e) { - std::string extra; - if (!sensitiveValue) - extra = fmt(" with raw value '%s'", s); - e.addTrace({}, "while decoding key named '%s'%s", name, extra); - throw; - } + static StringSet validKeyTypes = [] { + StringSet s; + for (const auto & [k, _] : keyTypeMap) { + s.insert(std::string(k)); + } + return s; + }(); + return validKeyTypes; +} + +KeyType parseKeyType(std::string_view s) +{ + auto i = keyTypeMap.find(s); + if (i != keyTypeMap.end()) + return i->second; + throw UsageError("unknown key type '%s'; valid key types are %s", s, concatStringsSep(", ", getKeyTypes())); } std::string Key::to_string() const @@ -101,46 +222,334 @@ std::string Key::to_string() const return serializeColonBase64(name, key); } -SecretKey::SecretKey(std::string_view s) - : Key{s, true} +Signature SecretKey::signDetached(std::string_view s) const { - if (key.size() != crypto_sign_SECRETKEYBYTES) - throw Error("secret key is not valid"); + throw Error("signing is not implemented for this key type"); } -Signature SecretKey::signDetached(std::string_view data) const +std::unique_ptr SecretKey::toPublicKey() const { - unsigned char sig[crypto_sign_BYTES]; - unsigned long long sigLen; - crypto_sign_detached(sig, &sigLen, (unsigned char *) data.data(), data.size(), (unsigned char *) key.data()); - return Signature{ - .keyName = name, - .sig = std::string((char *) sig, sigLen), - }; + throw Error("conversion to public key is not implemented for this key type"); } -PublicKey SecretKey::toPublicKey() const +std::string SecretKey::toPEM() const +{ + throw Error("conversion to PEM is not implemented for this key type"); +} + +struct Ed25519PublicKey : PublicKey +{ + Ed25519PublicKey(std::string_view name, std::string && _key) + : PublicKey(name, std::move(_key)) + { + assert(key.size() == crypto_sign_PUBLICKEYBYTES); + } + + bool verifyDetachedAnon(std::string_view data, const Signature & sig) const override + { + if (sig.sig.size() != crypto_sign_BYTES) + return false; + + return crypto_sign_verify_detached( + (unsigned char *) sig.sig.data(), + (unsigned char *) data.data(), + data.size(), + (unsigned char *) key.data()) + == 0; + } + + virtual void anchor(); +}; + +void Ed25519PublicKey::anchor() {} + +struct Ed25519SecretKey : SecretKey { - unsigned char pk[crypto_sign_PUBLICKEYBYTES]; - crypto_sign_ed25519_sk_to_pk(pk, (unsigned char *) key.data()); - return PublicKey(name, std::string((char *) pk, crypto_sign_PUBLICKEYBYTES)); + Ed25519SecretKey(std::string_view name, std::string && _key) + : SecretKey(name, std::move(_key)) + { + assert(key.size() == crypto_sign_SECRETKEYBYTES); + } + + static std::unique_ptr generate(std::string_view name) + { + unsigned char pk[crypto_sign_PUBLICKEYBYTES]; + unsigned char sk[crypto_sign_SECRETKEYBYTES]; + if (crypto_sign_keypair(pk, sk) != 0) + throw Error("key generation failed"); + + return std::make_unique(name, std::string((char *) sk, crypto_sign_SECRETKEYBYTES)); + } + + Signature signDetached(std::string_view data) const override + { + unsigned char sig[crypto_sign_BYTES]; + unsigned long long sigLen; + crypto_sign_detached(sig, &sigLen, (unsigned char *) data.data(), data.size(), (unsigned char *) key.data()); + return Signature{ + .keyName = name, + .sig = std::string((char *) sig, sigLen), + }; + } + + std::unique_ptr toPublicKey() const override + { + unsigned char pk[crypto_sign_PUBLICKEYBYTES]; + crypto_sign_ed25519_sk_to_pk(pk, (unsigned char *) key.data()); + return std::make_unique(name, std::string((char *) pk, crypto_sign_PUBLICKEYBYTES)); + } + + virtual void anchor(); +}; + +void Ed25519SecretKey::anchor() {} + +struct OpenSSLPublicKey : PublicKey +{ + KeyType type; + AutoEVP_PKEY pkey; + + OpenSSLPublicKey(KeyType type, std::string_view name, std::string && key, AutoEVP_PKEY && pkey) + : PublicKey(name, std::move(key)) + , type(type) + , pkey(std::move(pkey)) + { + assert( + type == KeyType::MLDSA44 || type == KeyType::MLDSA65 || type == KeyType::MLDSA87 + || type == KeyType::ECDSAP384); + } + + bool verifyDetachedAnon(std::string_view data, const Signature & sig) const override + { + if (!experimentalFeatureSettings.isEnabled(Xp::CNSA)) + return false; + + AutoEVP_MD_CTX ctx(EVP_MD_CTX_new()); + if (!ctx) + throw Error("EVP_MD_CTX_new failed"); + + if (EVP_DigestVerifyInit_ex(ctx.get(), nullptr, digestForKeyType(type), nullptr, nullptr, pkey.get(), nullptr) + <= 0) + throw Error("EVP_DigestVerifyInit_ex failed"); + + return EVP_DigestVerify( + ctx.get(), + (const unsigned char *) sig.sig.data(), + sig.sig.size(), + (const unsigned char *) data.data(), + data.size()) + == 1; + } + + std::string toPEM() const override + { + AutoBIO bio(BIO_new(BIO_s_mem())); + if (!bio) + throw Error("BIO_new failed"); + + if (PEM_write_bio_PUBKEY(bio.get(), pkey.get()) <= 0) + throw Error("PEM_write_bio_PUBKEY failed"); + + char * data = nullptr; + long len = BIO_get_mem_data(bio.get(), &data); + return std::string(data, len); + } + + virtual void anchor(); +}; + +void OpenSSLPublicKey::anchor() {} + +struct OpenSSLSecretKey : SecretKey +{ + KeyType type; + AutoEVP_PKEY pkey; + + OpenSSLSecretKey(KeyType type, std::string_view name, std::string && key, AutoEVP_PKEY && pkey) + : SecretKey(name, std::move(key)) + , type(type) + , pkey(std::move(pkey)) + { + assert( + type == KeyType::MLDSA44 || type == KeyType::MLDSA65 || type == KeyType::MLDSA87 + || type == KeyType::ECDSAP384); + } + + static std::unique_ptr generate(std::string_view name, KeyType type) + { + experimentalFeatureSettings.require(Xp::CNSA); + + auto typeS = toOpenSSLKeyType(type); + AutoEVP_PKEY_CTX ctx(EVP_PKEY_CTX_new_from_name(nullptr, typeS, nullptr)); + if (!ctx) + throw Error("EVP_PKEY_CTX_new_from_name failed for '%s'", typeS); + + if (EVP_PKEY_keygen_init(ctx.get()) <= 0) + throw Error("EVP_PKEY_keygen_init failed"); + + if (type == KeyType::ECDSAP384) { + char curve[] = "secp384r1"; + OSSL_PARAM params[] = { + OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME, curve, 0), + OSSL_PARAM_construct_end(), + }; + if (EVP_PKEY_CTX_set_params(ctx.get(), params) <= 0) + throw Error("EVP_PKEY_CTX_set_params failed"); + } + + EVP_PKEY * rawPkey = nullptr; + if (EVP_PKEY_generate(ctx.get(), &rawPkey) <= 0) + throw Error("EVP_PKEY_generate failed"); + AutoEVP_PKEY pkey(rawPkey); + + unsigned char * derBuf = nullptr; + int derLen = i2d_PrivateKey(pkey.get(), &derBuf); + if (derLen < 0) + throw Error("i2d_PrivateKey failed"); + std::string der((const char *) derBuf, derLen); + OPENSSL_free(derBuf); + + return std::make_unique(type, name, std::move(der), std::move(pkey)); + } + + Signature signDetached(std::string_view data) const override + { + experimentalFeatureSettings.require(Xp::CNSA); + + AutoEVP_MD_CTX ctx(EVP_MD_CTX_new()); + if (!ctx) + throw Error("EVP_MD_CTX_new failed"); + + /* Generate a deterministic signature (i.e. only depending on the key and the data) since Ed25519 is also + deterministic. For ML-DSA we set OSSL_SIGNATURE_PARAM_DETERMINISTIC; for ECDSA we request RFC-6979 + deterministic nonces via OSSL_SIGNATURE_PARAM_NONCE_TYPE. Note from RFC-9882: "The signer SHOULD NOT + use the deterministic variant of ML-DSA on platforms where side-channel attacks or fault attacks are a + concern." */ + int deterministic = 1; + unsigned int nonceType = 1; + OSSL_PARAM mlDsaParams[] = { + OSSL_PARAM_construct_int(OSSL_SIGNATURE_PARAM_DETERMINISTIC, &deterministic), + OSSL_PARAM_construct_end(), + }; + OSSL_PARAM ecdsaParams[] = { + OSSL_PARAM_construct_uint(OSSL_SIGNATURE_PARAM_NONCE_TYPE, &nonceType), + OSSL_PARAM_construct_end(), + }; + OSSL_PARAM * params = type == KeyType::ECDSAP384 ? ecdsaParams : mlDsaParams; + + if (EVP_DigestSignInit_ex(ctx.get(), nullptr, digestForKeyType(type), nullptr, nullptr, pkey.get(), params) + <= 0) + throw Error("EVP_DigestSignInit_ex failed"); + + size_t sigLen = 0; + if (EVP_DigestSign(ctx.get(), nullptr, &sigLen, (const unsigned char *) data.data(), data.size()) <= 0) + throw Error("EVP_DigestSign (get length) failed"); + + std::string sig(sigLen, '\0'); + if (EVP_DigestSign( + ctx.get(), (unsigned char *) sig.data(), &sigLen, (const unsigned char *) data.data(), data.size()) + <= 0) + throw Error("EVP_DigestSign failed"); + sig.resize(sigLen); + + return Signature{ + .keyName = name, + .sig = std::move(sig), + }; + } + + std::unique_ptr toPublicKey() const override + { + experimentalFeatureSettings.require(Xp::CNSA); + + unsigned char * derBuf = nullptr; + int derLen = i2d_PUBKEY(pkey.get(), &derBuf); + if (derLen < 0) + throw Error("i2d_PUBKEY failed"); + std::string der((const char *) derBuf, derLen); + OPENSSL_free(derBuf); + + auto pubKey = parsePublicKey(der, type); + + return std::make_unique(type, name, std::move(der), std::move(pubKey)); + } + + std::string toPEM() const override + { + AutoBIO bio(BIO_new(BIO_s_mem())); + if (!bio) + throw Error("BIO_new failed"); + + if (PEM_write_bio_PrivateKey(bio.get(), pkey.get(), nullptr, nullptr, 0, nullptr, nullptr) <= 0) + throw Error("PEM_write_bio_PrivateKey failed"); + + char * data = nullptr; + long len = BIO_get_mem_data(bio.get(), &data); + return std::string(data, len); + } + + virtual void anchor(); +}; + +void OpenSSLSecretKey::anchor() {} + +std::unique_ptr SecretKey::parse(std::string_view s) +{ + try { + auto [name, key] = parseColonBase64(s, "key"); + + if (key.size() == crypto_sign_SECRETKEYBYTES) + return std::make_unique(name, std::move(key)); + else if (auto pkey = parsePrivateKey(key); experimentalFeatureSettings.isEnabled(Xp::CNSA) && pkey) { + auto type = detectOpenSSLKeyType(pkey.get()); + if (!type) + throw Error("secret key has unsupported type '%s'", EVP_PKEY_get0_type_name(pkey.get())); + return std::make_unique(*type, name, std::move(key), std::move(pkey)); + } else + throw Error("secret key is not valid"); + + } catch (Error & e) { + e.addTrace({}, "while decoding key '%s'", keyNamePart(s)); + throw; + } } -SecretKey SecretKey::generate(std::string_view name) +std::unique_ptr SecretKey::generate(std::string_view name, KeyType type) { - unsigned char pk[crypto_sign_PUBLICKEYBYTES]; - unsigned char sk[crypto_sign_SECRETKEYBYTES]; - if (crypto_sign_keypair(pk, sk) != 0) - throw Error("key generation failed"); + switch (type) { - return SecretKey(name, std::string((char *) sk, crypto_sign_SECRETKEYBYTES)); + case KeyType::Ed25519: + return Ed25519SecretKey::generate(name); + + case KeyType::MLDSA44: + case KeyType::MLDSA65: + case KeyType::MLDSA87: + case KeyType::ECDSAP384: + return OpenSSLSecretKey::generate(name, type); + + default: + unreachable(); + } } -PublicKey::PublicKey(std::string_view s) - : Key{s, false} +std::unique_ptr PublicKey::parse(std::string_view s) { - if (key.size() != crypto_sign_PUBLICKEYBYTES) - throw Error("public key is not valid"); + try { + auto [name, key] = parseColonBase64(s, "key"); + + if (key.size() == crypto_sign_PUBLICKEYBYTES) + return std::make_unique(name, std::move(key)); + else if (auto pkey = parsePublicKey(key); experimentalFeatureSettings.isEnabled(Xp::CNSA) && pkey) { + auto type = detectOpenSSLKeyType(pkey.get()); + if (!type) + throw Error("public key has unsupported type '%s'", EVP_PKEY_get0_type_name(pkey.get())); + return std::make_unique(*type, name, std::move(key), std::move(pkey)); + } else + throw Error("public key is not valid"); + } catch (Error & e) { + e.addTrace({}, "while decoding key '%s'", keyNamePart(s)); + throw; + } } bool PublicKey::verifyDetached(std::string_view data, const Signature & sig) const @@ -153,15 +562,13 @@ bool PublicKey::verifyDetached(std::string_view data, const Signature & sig) con bool PublicKey::verifyDetachedAnon(std::string_view data, const Signature & sig) const { - if (sig.sig.size() != crypto_sign_BYTES) - throw Error("signature is not valid"); + // Unsupported key type, can't verify. + return false; +} - return crypto_sign_verify_detached( - (unsigned char *) sig.sig.data(), - (unsigned char *) data.data(), - data.size(), - (unsigned char *) key.data()) - == 0; +std::string PublicKey::toPEM() const +{ + throw Error("conversion to PEM is not implemented for this key type"); } bool verifyDetached(std::string_view data, const Signature & sig, const PublicKeys & publicKeys) @@ -170,7 +577,7 @@ bool verifyDetached(std::string_view data, const Signature & sig, const PublicKe if (key == publicKeys.end()) return false; - return key->second.verifyDetachedAnon(data, sig); + return key->second->verifyDetachedAnon(data, sig); } } // namespace nix diff --git a/src/libutil/signature/signer.cc b/src/libutil/signature/signer.cc index bb9bbad04f05..334b9b528200 100644 --- a/src/libutil/signature/signer.cc +++ b/src/libutil/signature/signer.cc @@ -6,20 +6,20 @@ namespace nix { void Signer::anchor() {} -LocalSigner::LocalSigner(SecretKey && privateKey) - : privateKey(privateKey) - , publicKey(privateKey.toPublicKey()) +LocalSigner::LocalSigner(std::unique_ptr && _privateKey) + : privateKey(std::move(_privateKey)) + , publicKey(privateKey->toPublicKey()) { } Signature LocalSigner::signDetached(std::string_view s) const { - return privateKey.signDetached(s); + return privateKey->signDetached(s); } const PublicKey & LocalSigner::getPublicKey() { - return publicKey; + return *publicKey; } } // namespace nix diff --git a/src/libutil/source-accessor.cc b/src/libutil/source-accessor.cc index a7cc160a6dc5..09c2493820e0 100644 --- a/src/libutil/source-accessor.cc +++ b/src/libutil/source-accessor.cc @@ -1,5 +1,7 @@ #include + #include "nix/util/source-accessor.hh" +#include "nix/util/provenance.hh" namespace nix { @@ -136,4 +138,9 @@ CanonPath SourceAccessor::resolveSymlinks(const CanonPath & path, SymlinkResolut return res; } +std::shared_ptr SourceAccessor::getProvenance(const CanonPath & path) +{ + return provenance && !path.isRoot() ? std::make_shared(provenance, path) : provenance; +} + } // namespace nix diff --git a/src/libutil/table.cc b/src/libutil/table.cc index fa1bf110d93f..215171dc02fc 100644 --- a/src/libutil/table.cc +++ b/src/libutil/table.cc @@ -1,4 +1,5 @@ #include "nix/util/table.hh" +#include "nix/util/terminal.hh" #include #include @@ -7,7 +8,7 @@ namespace nix { -void printTable(std::ostream & out, Table & table) +void printTable(std::ostream & out, Table & table, unsigned int width) { auto nrColumns = table.size() > 0 ? table.front().size() : 0; @@ -18,19 +19,31 @@ void printTable(std::ostream & out, Table & table) assert(i.size() == nrColumns); size_t column = 0; for (auto j = i.begin(); j != i.end(); ++j, ++column) - if (j->size() > widths[column]) - widths[column] = j->size(); + // TODO: take ANSI escapes into account when calculating width. + widths[column] = std::max(widths[column], j->content.size()); } for (auto & i : table) { size_t column = 0; + std::string line; for (auto j = i.begin(); j != i.end(); ++j, ++column) { - std::string s = *j; + std::string s = j->content; replace(s.begin(), s.end(), '\n', ' '); - out << s; - if (column < nrColumns - 1) - out << std::string(widths[column] - s.size() + 2, ' '); + + auto padding = std::string(widths[column] - s.size(), ' '); + if (j->alignment == TableCell::Right) { + line += padding; + line += s; + } else { + line += s; + if (column + 1 < nrColumns) + line += padding; + } + + if (column + 1 < nrColumns) + line += " "; } + out << filterANSIEscapes(line, false, width); out << std::endl; } } diff --git a/src/libutil/tarfile.cc b/src/libutil/tarfile.cc index 7149d73a5fbd..3916d8050025 100644 --- a/src/libutil/tarfile.cc +++ b/src/libutil/tarfile.cc @@ -199,6 +199,14 @@ static void extract_archive(TarArchive & archive, const std::filesystem::path & #undef NIX_LIBARCHIVE_NATIVE_PATH_FUNC +void unpackTarfile(Source & source, const std::filesystem::path & destDir) +{ + auto archive = TarArchive(source); + + createDirs(destDir); + extract_archive(archive, destDir); +} + void unpackTarfile(const std::filesystem::path & tarFile, const std::filesystem::path & destDir) { auto archive = TarArchive(tarFile); diff --git a/src/libutil/tee-logger.cc b/src/libutil/tee-logger.cc index 8e27714e3f40..7b40a036adda 100644 --- a/src/libutil/tee-logger.cc +++ b/src/libutil/tee-logger.cc @@ -20,6 +20,12 @@ class TeeLogger final : public Logger logger->stop(); }; + void flush() override + { + for (auto & logger : loggers) + logger->flush(); + }; + void pause() override { for (auto & logger : loggers) @@ -32,42 +38,81 @@ class TeeLogger final : public Logger logger->resume(); }; - void log(Verbosity lvl, std::string_view s) override + void log(Verbosity lvl, std::string_view s) noexcept override { for (auto & logger : loggers) logger->log(lvl, s); } - void logEI(const ErrorInfo & ei) override + void logEI(const ErrorInfo & ei) noexcept override { for (auto & logger : loggers) logger->logEI(ei); } + void printException(const std::exception_ptr & ex, std::string_view programName) noexcept override + { + for (auto & logger : loggers) + logger->printException(ex, programName); + } + void startActivity( ActivityId act, Verbosity lvl, ActivityType type, const std::string & s, const Fields & fields, - ActivityId parent) override + ActivityId parent) noexcept override { for (auto & logger : loggers) logger->startActivity(act, lvl, type, s, fields, parent); } - void stopActivity(ActivityId act) override + void startActivity( + ActivityId act, + Verbosity lvl, + std::string_view name, + ActivityMetadata metadata, + std::string_view s, + ActivityId parent) noexcept override + { + for (auto & logger : loggers) + logger->startActivity(act, lvl, name, metadata, s, parent); + } + + void stopActivity(ActivityId act) noexcept override { for (auto & logger : loggers) logger->stopActivity(act); } - void result(ActivityId act, ResultType type, const Fields & fields) override + void result(ActivityId act, ResultType type, const Fields & fields) noexcept override { for (auto & logger : loggers) logger->result(act, type, fields); } + void result(ActivityId act, ResultType type, const nlohmann::json & json) noexcept override + { + for (auto & logger : loggers) + logger->result(act, type, json); + } + + Headers getTraceContext(ActivityId act) override + { + for (auto & logger : loggers) { + auto headers = logger->getTraceContext(act); + if (!headers.empty()) + return headers; + } + return {}; + } + + void addLogger(std::unique_ptr logger) + { + loggers.push_back(std::move(logger)); + } + void writeToStdout(std::string_view s) override { for (auto & logger : loggers) { @@ -99,14 +144,21 @@ class TeeLogger final : public Logger } // namespace -std::unique_ptr -makeTeeLogger(std::unique_ptr mainLogger, std::vector> && extraLoggers) +void applyExtraLogger(std::unique_ptr extraLogger) { - std::vector> allLoggers; - allLoggers.push_back(std::move(mainLogger)); - for (auto & l : extraLoggers) - allLoggers.push_back(std::move(l)); - return std::make_unique(std::move(allLoggers)); + auto teeLogger = dynamic_cast(logger); + if (!teeLogger) { + try { + std::vector> loggers; + loggers.push_back(std::unique_ptr(logger)); + teeLogger = new TeeLogger(std::move(loggers)); + } catch (...) { + // `logger` is now gone so give up. + abort(); + } + logger = teeLogger; + } + teeLogger->addLogger(std::move(extraLogger)); } } // namespace nix diff --git a/src/libutil/terminal.cc b/src/libutil/terminal.cc index ebdc783e4abb..a56114c43de3 100644 --- a/src/libutil/terminal.cc +++ b/src/libutil/terminal.cc @@ -190,6 +190,14 @@ std::pair getWindowSize() return *windowSize->lock(); } +unsigned int getWindowWidth() +{ + unsigned int width = getWindowSize().second; + if (width <= 0) + width = std::numeric_limits::max(); + return width; +} + #ifndef _WIN32 std::string getPtsName(int fd) { diff --git a/src/libutil/union-source-accessor.cc b/src/libutil/union-source-accessor.cc index dd68a2d8838d..33dd90d8f413 100644 --- a/src/libutil/union-source-accessor.cc +++ b/src/libutil/union-source-accessor.cc @@ -11,22 +11,22 @@ struct UnionSourceAccessor : SourceAccessor public: std::vector> accessors; + std::shared_ptr displayAccessor; - UnionSourceAccessor(std::vector> _accessors) + UnionSourceAccessor(std::vector> _accessors, std::shared_ptr _displayAccessor) : accessors(std::move(_accessors)) + , displayAccessor(std::move(_displayAccessor)) { displayPrefix.clear(); } void readFile(const CanonPath & path, Sink & sink, fun sizeCallback) override { - for (auto & accessor : accessors) { - auto st = accessor->maybeLstat(path); - if (st) { + for (const auto & [last, accessor] : markLast(accessors)) + if (last || accessor->maybeLstat(path)) { accessor->readFile(path, sink, sizeCallback); return; } - } throw FileNotFound("path '%s' does not exist", showPath(path)); } @@ -60,16 +60,16 @@ struct UnionSourceAccessor : SourceAccessor std::string readLink(const CanonPath & path) override { - for (auto & accessor : accessors) { - auto st = accessor->maybeLstat(path); - if (st) + for (const auto & [last, accessor] : markLast(accessors)) + if (last || accessor->maybeLstat(path)) return accessor->readLink(path); - } throw FileNotFound("path '%s' does not exist", showPath(path)); } std::string showPath(const CanonPath & path) override { + if (displayAccessor) + return displayAccessor->showPath(path); for (auto & accessor : accessors) return accessor->showPath(path); return SourceAccessor::showPath(path); @@ -102,13 +102,24 @@ struct UnionSourceAccessor : SourceAccessor } return {path, std::nullopt}; } + + std::shared_ptr getProvenance(const CanonPath & path) override + { + for (auto & accessor : accessors) { + auto prov = accessor->getProvenance(path); + if (prov) + return prov; + } + return nullptr; + } }; } // namespace -ref makeUnionSourceAccessor(std::vector> && accessors) +ref +makeUnionSourceAccessor(std::vector> && accessors, std::shared_ptr displayAccessor) { - return make_ref(std::move(accessors)); + return make_ref(std::move(accessors), displayAccessor); } } // namespace nix diff --git a/src/libutil/unix/file-descriptor.cc b/src/libutil/unix/file-descriptor.cc index 545f4f2b7040..5f6dbed39ff3 100644 --- a/src/libutil/unix/file-descriptor.cc +++ b/src/libutil/unix/file-descriptor.cc @@ -5,8 +5,10 @@ #include #include #include +#include #include "util-unix-config-private.hh" +#include "../file-descriptor-private.hh" namespace nix { @@ -186,4 +188,72 @@ void unix::SelfPipe::drain() } } +bool tryCopyFdRangeFast(Descriptor from, Descriptor to, off_t offset, size_t nbytes, size_t & written) +{ +#if HAVE_COPY_FILE_RANGE + + /* Otherwise trying block cloning is pretty pointless. Also used to error + out on partial copies (unless we happen to hit block alignment boundary, + but that seems unlikely to be useful). */ + assert(written == 0); + size_t left = nbytes; + static std::atomic_flag copyFileRangeUnsupported = {}; + + if (copyFileRangeUnsupported.test(std::memory_order_relaxed)) + return false; + + while (left) { + checkInterrupt(); + + ssize_t n = ::copy_file_range( + from, + &offset, + to, + nullptr, + left, + 0 +# ifdef COPY_FILE_RANGE_CLONE /* FreeBSD */ + | COPY_FILE_RANGE_CLONE +# endif + ); + + /* Reached EOF too early. */ + if (n == 0) + throw EndOfFile( + "unexpected end-of-file copying from %1% to %2%", + PathFmt(descriptorToPath(from)), + PathFmt(descriptorToPath(to))); + + if (n == -1) { + if (errno == EINTR) + continue; /* Loop over to hit checkInterrupt. */ + + if (written == 0 && (errno == EINVAL || errno == EOPNOTSUPP || errno == EXDEV || errno == EBADF)) { + /* Retry with fallback. */ + } else if (written == 0 && errno == ENOSYS) { + /* Cache ENOSYS. */ + copyFileRangeUnsupported.test_and_set(); + } else { + throw SysError([&] { + return HintFmt( + "copying contents of %1% to %2% via copy_file_range", + PathFmt(descriptorToPath(from)), + PathFmt(descriptorToPath(to))); + }); + } + + return false; + } + + assert(static_cast(n) <= left); + left -= n; + written += n; + } + + return true; +#else + return false; +#endif +} + } // namespace nix diff --git a/src/libutil/unix/include/nix/util/signals-impl.hh b/src/libutil/unix/include/nix/util/signals-impl.hh index e22268214478..3e92aeb63300 100644 --- a/src/libutil/unix/include/nix/util/signals-impl.hh +++ b/src/libutil/unix/include/nix/util/signals-impl.hh @@ -37,7 +37,7 @@ namespace unix { extern std::atomic _isInterrupted; -extern thread_local std::function interruptCheck; +[[gnu::tls_model("initial-exec")]] extern thread_local std::function interruptCheck; void _interrupted(); diff --git a/src/libutil/unix/meson.build b/src/libutil/unix/meson.build index dbe142d3a88c..c78896b71394 100644 --- a/src/libutil/unix/meson.build +++ b/src/libutil/unix/meson.build @@ -48,6 +48,23 @@ foreach funcspec : check_funcs_unix configdata_unix.set(define_name, define_value, description : funcspec[1]) endforeach +# Detects the Linux signature of the copy_file_range. +have_copy_file_range = cxx.links( + '''#include +int main() { + static_assert(sizeof(off_t) == 8); + copy_file_range(0, nullptr, 0, nullptr, 0, 0); +} + ''', + name : 'have copy_file_range', +) + +configdata_unix.set( + 'HAVE_COPY_FILE_RANGE', + have_copy_file_range.to_int(), + description : 'Optionally used for optimised file copying', +) + config_unix_priv_h = configure_file( configuration : configdata_unix, output : 'util-unix-config-private.hh', diff --git a/src/libutil/unix/processes.cc b/src/libutil/unix/processes.cc index 32d59535883a..b845c87dcc96 100644 --- a/src/libutil/unix/processes.cc +++ b/src/libutil/unix/processes.cc @@ -14,6 +14,7 @@ #include #include #include +#include using namespace std::chrono_literals; #include @@ -40,6 +41,7 @@ Pid::Pid(Pid && other) noexcept : pid(other.pid) , separatePG(other.separatePG) , killSignal(other.killSignal) + , killTimeout(other.killTimeout) { other.release(); } @@ -80,8 +82,23 @@ int Pid::kill(bool allowInterrupts) std::atomic killed = false; + std::thread killThread; + + /* Make sure that the thread is joined even if `wait()` throws + an exception. */ + Finally joinKillThread([&]() { + if (killThread.joinable()) { + killed = true; + killThread.join(); + } + }); + + /* Note: the thread must not use `pid`, since `wait()` sets it to + -1 when the child has exited. */ + pid_t target = separatePG ? -pid : pid; + if (killTimeout > 0ms && killSignal != SIGKILL) - killThread = std::thread([&]() { + killThread = std::thread([&, target]() { auto elapsed = 0ms; while (elapsed < killTimeout) { std::this_thread::sleep_for(25ms); @@ -89,13 +106,13 @@ int Pid::kill(bool allowInterrupts) if (killed) return; } - ::kill(separatePG ? -pid : pid, SIGKILL); + ::kill(target, SIGKILL); }); /* Send the requested signal to the child. If it has its own process group, send the signal to every process in the child process group (which hopefully includes *all* its children). */ - if (::kill(separatePG ? -pid : pid, killSignal) != 0) { + if (::kill(target, killSignal) != 0) { /* On BSDs, killing a process group will return EPERM if all processes in the group are zombies (or something like that). So try to detect and ignore that situation. */ @@ -105,12 +122,7 @@ int Pid::kill(bool allowInterrupts) logError(SysError("killing process %d", pid).info()); } - int ret = wait(allowInterrupts); - if (killThread.joinable()) { - killed = true; - killThread.join(); - } - return ret; + return wait(allowInterrupts); } int Pid::wait(bool allowInterrupts) @@ -130,6 +142,25 @@ int Pid::wait(bool allowInterrupts) } } +bool Pid::isAlive() +{ + assert(pid != -1); + pid_t res = waitpid(pid, nullptr, WNOHANG); + if (res == 0) + return true; + if (res == -1) { + if (errno == EINTR) + /* Assume it's still alive; the caller can check again later. */ + return true; + if (errno != ECHILD) + warn("waitpid failed for PID %d: %s", pid, strerror(errno)); + } + /* The process has exited and been reaped (or was already reaped + elsewhere). Reset so the destructor doesn't kill()/wait() it. */ + release(); + return false; +} + void Pid::setSeparatePG(bool separatePG) { this->separatePG = separatePG; @@ -244,6 +275,15 @@ pid_t startProcess(fun processMain, const ProcessOptions & options) ~ProgressBar() tries to join a thread that doesn't exist. */ logger = newLogger; + + /* Discard other state that doesn't survive the fork, + such as objects owning a thread. */ + for (auto & callback : RegisterForkCallback::callbacks()) { + try { + callback(); + } catch (...) { + } + } } try { #ifdef __linux__ diff --git a/src/libutil/unix/signals.cc b/src/libutil/unix/signals.cc index 38030a77ddfd..9611bdcf365a 100644 --- a/src/libutil/unix/signals.cc +++ b/src/libutil/unix/signals.cc @@ -15,7 +15,7 @@ void Cancelled::anchor() {} std::atomic unix::_isInterrupted = false; -thread_local std::function unix::interruptCheck; +[[gnu::tls_model("initial-exec")]] thread_local std::function unix::interruptCheck; void unix::_interrupted() { @@ -29,11 +29,11 @@ void unix::_interrupted() ////////////////////////////////////////////////////////////////////// -/* We keep track of interrupt callbacks using integer tokens, so we can iterate +/* We keep track of signal callbacks using integer tokens, so we can iterate safely without having to lock the data structure while executing arbitrary functions. */ -struct InterruptCallbacks +struct SignalCallbacks { typedef int64_t Token; @@ -41,21 +41,47 @@ struct InterruptCallbacks handler because of an erroneous double delete. */ Token nextToken = 0; - /* Used as a list, see InterruptCallbacks comment. */ - std::map> callbacks; + /* Each per-signal map is used as a list, see SignalCallbacks comment. */ + std::map>> callbacks; }; InterruptCallback::~InterruptCallback() {} /* Required to avoid static initialization order fiasco. This allows global objects to safely register callbacks. */ -static Sync & getInterruptCallbacks() +static Sync & getSignalCallbacks() { /* Intentionally leak, according to the Construct On First Use Idiom. An alternative is to use the Nifty Counter Idiom, but - InterruptCallbacks' destructor is not very important. */ - static Sync * _interruptCallbacks = new Sync(); - return *_interruptCallbacks; + SignalCallbacks' destructor is not very important. */ + static Sync * _signalCallbacks = new Sync(); + return *_signalCallbacks; +} + +static void triggerSignalCallbacks(SignalType type) +{ + SignalCallbacks::Token i = 0; + while (true) { + std::function callback; + { + auto signalCallbacks(getSignalCallbacks().lock()); + auto it = signalCallbacks->callbacks.find(type); + if (it == signalCallbacks->callbacks.end()) + break; + auto lb = it->second.lower_bound(i); + if (lb == it->second.end()) + break; + + callback = lb->second; + i = lb->first + 1; + } + + try { + callback(); + } catch (...) { + ignoreExceptionInDestructor(); + } + } } static void signalHandlerThread(sigset_t set) @@ -70,7 +96,29 @@ static void signalHandlerThread(sigset_t set) else if (signal == SIGWINCH) { updateWindowSize(); + triggerSignalCallbacks(SignalType::Winch); } + + else if (signal == SIGCONT) { + /* The terminal may have been resized while we were + stopped, and we won't have received SIGWINCH for it, + since the kernel only sends it to the foreground process + group. */ + updateWindowSize(); + triggerSignalCallbacks(SignalType::Cont); + } + +#ifndef __FreeBSD__ + else if (signal == SIGTSTP) { + triggerSignalCallbacks(SignalType::Stop); + /* Since we consumed SIGTSTP via sigwait(), its default + action (stopping the process) no longer happens, so stop + ourselves explicitly. SIGSTOP cannot be blocked. On + resumption, SIGCONT is delivered via sigwait() as + usual. */ + kill(getpid(), SIGSTOP); + } +#endif } } @@ -78,27 +126,7 @@ void unix::triggerInterrupt() { _isInterrupted = true; - { - InterruptCallbacks::Token i = 0; - while (true) { - std::function callback; - { - auto interruptCallbacks(getInterruptCallbacks().lock()); - auto lb = interruptCallbacks->callbacks.lower_bound(i); - if (lb == interruptCallbacks->callbacks.end()) - break; - - callback = lb->second; - i = lb->first + 1; - } - - try { - callback(); - } catch (...) { - ignoreExceptionInDestructor(); - } - } - } + triggerSignalCallbacks(SignalType::Int); } static sigset_t savedSignalMask; @@ -125,6 +153,12 @@ void unix::startSignalHandlerThread() sigaddset(&set, SIGHUP); sigaddset(&set, SIGPIPE); sigaddset(&set, SIGWINCH); + sigaddset(&set, SIGCONT); +#ifndef __FreeBSD__ + /* Not on FreeBSD, where SIGTSTP doubles as NIX_SIG_MULTI_INT and is + delivered to specific threads using pthread_kill(). */ + sigaddset(&set, SIGTSTP); +#endif if (pthread_sigmask(SIG_BLOCK, &set, nullptr)) throw SysError("blocking signals"); @@ -156,10 +190,12 @@ namespace { /* RAII helper to automatically deregister a callback. */ struct InterruptCallbackImpl : InterruptCallback { - InterruptCallbacks::Token token; + SignalType type; + SignalCallbacks::Token token; - InterruptCallbackImpl(InterruptCallbacks::Token token) - : token(token) + InterruptCallbackImpl(SignalType type, SignalCallbacks::Token token) + : type(type) + , token(token) { } @@ -170,19 +206,24 @@ struct InterruptCallbackImpl : InterruptCallback ~InterruptCallbackImpl() override { - auto interruptCallbacks(getInterruptCallbacks().lock()); - interruptCallbacks->callbacks.erase(token); + auto signalCallbacks(getSignalCallbacks().lock()); + signalCallbacks->callbacks[type].erase(token); } }; } // namespace +std::unique_ptr createSignalCallback(SignalType type, fun callback) +{ + auto signalCallbacks(getSignalCallbacks().lock()); + auto token = signalCallbacks->nextToken++; + signalCallbacks->callbacks[type].emplace(token, callback); + return std::make_unique(type, token); +} + std::unique_ptr createInterruptCallback(fun callback) { - auto interruptCallbacks(getInterruptCallbacks().lock()); - auto token = interruptCallbacks->nextToken++; - interruptCallbacks->callbacks.emplace(token, callback); - return std::make_unique(token); + return createSignalCallback(SignalType::Int, std::move(callback)); } } // namespace nix diff --git a/src/libutil/url.cc b/src/libutil/url.cc index c09c9e09180b..f6f2209aef70 100644 --- a/src/libutil/url.cc +++ b/src/libutil/url.cc @@ -337,6 +337,15 @@ std::string ParsedURL::renderPath(bool encode) const return renderUrlPathNoPctEncoding(path); } +std::string ParsedURL::renderSanitized() const +{ + auto url = *this; + if (url.authority) + url.authority->password.reset(); + url.query.clear(); + return url.to_string(); +} + std::string ParsedURL::renderAuthorityAndPath() const { std::string res; diff --git a/src/libutil/util.cc b/src/libutil/util.cc index e6b2fffa25fa..6c885f3ec07a 100644 --- a/src/libutil/util.cc +++ b/src/libutil/util.cc @@ -5,7 +5,9 @@ #include #include +#include +#include #include #include #include @@ -22,6 +24,10 @@ bad_ref_cast::~bad_ref_cast() {} void initLibUtil() { + static std::atomic_flag done = ATOMIC_FLAG_INIT; + if (done.test_and_set()) + return; + // Check that exception handling works. Exception handling has been observed // not to work on darwin when the linker flags aren't quite right. // In this case we don't want to expose the user to some unrelated uncaught @@ -43,6 +49,26 @@ void initLibUtil() if (sodium_init() == -1) throw Error("could not initialise libsodium"); + + /* Prevent OpenSSL from registering its atexit() handler + (OPENSSL_cleanup()). If we exit() while other threads that use + OpenSSL are still running, OPENSSL_cleanup() frees OpenSSL's + thread-local state handlers; when those threads then exit, their + thread-specific-data destructors (init_thread_stop()) crash on + the freed state. This happens in particular in nix-daemon + connection children, where library destructors run by _dl_fini() + (e.g. aws-crt-cpp's) stop their worker threads *after* + OPENSSL_cleanup() has already run. Since we're exiting anyway, + skipping the cleanup is harmless. This must run before any other + use of OpenSSL, since only the first initialisation takes + effect. */ + if (OPENSSL_init_crypto(OPENSSL_INIT_NO_ATEXIT, nullptr) != 1) + throw Error("could not initialise OpenSSL"); + + /* Make sure that failing memory allocations don't result in an + opaque abort() (e.g. from mimalloc's `operator new` override, + which cannot throw `std::bad_alloc`). */ + std::set_new_handler(outOfMemory); } ////////////////////////////////////////////////////////////////////// @@ -244,6 +270,11 @@ void ignoreExceptionInDestructor(Verbosity lvl) } void ignoreExceptionExceptInterrupt(Verbosity lvl) +{ + logExceptionExceptInterrupt("error (ignored): ", lvl); +} + +void logExceptionExceptInterrupt(std::string_view prefix, Verbosity lvl) { try { throw; @@ -254,9 +285,9 @@ void ignoreExceptionExceptInterrupt(Verbosity lvl) cancellation. */ throw; } catch (Error & e) { - printMsg(lvl, ANSI_RED "error (ignored):" ANSI_NORMAL " %s", e.info().msg); + printMsg(lvl, ANSI_RED "%s" ANSI_NORMAL "%s", prefix, e.info().msg); } catch (std::exception & e) { - printMsg(lvl, ANSI_RED "error (ignored):" ANSI_NORMAL " %s", e.what()); + printMsg(lvl, ANSI_RED "%s" ANSI_NORMAL "%s", prefix, e.what()); } } @@ -312,4 +343,6 @@ std::pair getLine(std::string_view s) } } +fun setSentryTag = [](const char *, const char *) {}; + } // namespace nix diff --git a/src/libutil/windows/file-descriptor.cc b/src/libutil/windows/file-descriptor.cc index c571d0a3e9f7..00932de2a754 100644 --- a/src/libutil/windows/file-descriptor.cc +++ b/src/libutil/windows/file-descriptor.cc @@ -3,6 +3,8 @@ #include "nix/util/finally.hh" #include "nix/util/serialise.hh" +#include "../file-descriptor-private.hh" + #include #include @@ -45,12 +47,19 @@ size_t readOffset(Descriptor fd, off_t offset, std::span buffer) ov.Offset = static_cast(offset); if constexpr (sizeof(offset) > 4) /* We don't build with 32 bit off_t, but let's be safe. */ ov.OffsetHigh = static_cast(offset >> 32); + // TODO: Don't do this on each call maybe? + if (::GetFileType(fd) != FILE_TYPE_DISK) + // Not the most accurate error code, but it will do. + throw windows::WinError( + DWORD(ERROR_SEEK_ON_DEVICE), "reading at offset %1% from a non-seekable handle", offset); DWORD n; if (!ReadFile(fd, buffer.data(), static_cast(buffer.size()), &n, &ov)) { - throw windows::WinError([&] { - return HintFmt( - "reading %1% bytes at offset %2% from %3%", buffer.size(), offset, PathFmt(descriptorToPath(fd))); - }); + auto err = ::GetLastError(); + // We report EOF as 0 return code, not an actual error. + if (err == ERROR_HANDLE_EOF) + return 0; + throw windows::WinError( + err, "reading %1% bytes at offset %2% from %3%", buffer.size(), offset, PathFmt(descriptorToPath(fd))); } return static_cast(n); } @@ -133,4 +142,11 @@ void syncDescriptor(Descriptor fd) } } +bool tryCopyFdRangeFast(Descriptor from, Descriptor to, off_t offset, size_t nbytes, size_t & written) +{ + /* TODO: Implement if this becomes relevant. Seems like windows maybe has some reflinking capabilities: + https://devblogs.microsoft.com/engineering-at-microsoft/dev-drive-and-copy-on-write-for-developer-performance/ */ + return false; +} + } // namespace nix diff --git a/src/libutil/windows/include/nix/util/signals-impl.hh b/src/libutil/windows/include/nix/util/signals-impl.hh index 379397c72c35..db899b292191 100644 --- a/src/libutil/windows/include/nix/util/signals-impl.hh +++ b/src/libutil/windows/include/nix/util/signals-impl.hh @@ -44,6 +44,11 @@ struct ReceiveInterrupts ~ReceiveInterrupts() {} }; +inline std::unique_ptr createSignalCallback(SignalType type, fun callback) +{ + return {}; +} + inline std::unique_ptr createInterruptCallback(fun callback) { return {}; diff --git a/src/nix/app.cc b/src/nix/app.cc index 167ffd8d86f8..4a8ca815c0fa 100644 --- a/src/nix/app.cc +++ b/src/nix/app.cc @@ -74,6 +74,7 @@ UnresolvedApp InstallableValue::toApp(EvalState & state) std::visit( overloaded{ [&](const NixStringContextElem::DrvDeep & d) -> DerivedPath { + state.waitForPath(d.drvPath); /* We want all outputs of the drv */ return DerivedPath::Built{ .drvPath = makeConstantStorePathRef(d.drvPath), @@ -81,6 +82,7 @@ UnresolvedApp InstallableValue::toApp(EvalState & state) }; }, [&](const NixStringContextElem::Built & b) -> DerivedPath { + state.waitForPath(*b.drvPath); return DerivedPath::Built{ .drvPath = b.drvPath, .outputs = OutputsSpec::Names{b.output}, @@ -88,18 +90,19 @@ UnresolvedApp InstallableValue::toApp(EvalState & state) }, [&](const NixStringContextElem::Opaque & o) -> DerivedPath { return DerivedPath::Opaque{ - .path = o.path, + .path = state.devirtualize(o.path), }; }, + [&](const NixStringContextElem::Path & p) -> DerivedPath { + throw Error("'program' attribute of an 'app' output cannot have no context"); + }, }, c.raw)); } - state.ensureLazyPathsCopied(context); - return UnresolvedApp{App{ .context = std::move(context2), - .program = program, + .program = state.devirtualize(program, context), }}; } diff --git a/src/nix/baked-flake.nix b/src/nix/baked-flake.nix new file mode 100644 index 000000000000..c4505bca7e79 --- /dev/null +++ b/src/nix/baked-flake.nix @@ -0,0 +1,150 @@ +{ + outputs = + { self }: + let + data = builtins.fromJSON (builtins.readFile ./outputs.json); + + optionalAttrs = cond: attrs: if cond then attrs else { }; + + # `setAttrByPath [ "a" "b" ] x` returns `{ a.b = x; }`. + setAttrByPath = + path: value: + if path == [ ] then + value + else + { ${builtins.head path} = setAttrByPath (builtins.tail path) value; }; + + # The inventory of each output. `nix flake bake` only records + # nodes that can be baked: derivation leaves, and non-leaf nodes + # that contain at least one of them. + inventories = builtins.mapAttrs (name: output: output.output) data; + + # Convert an inventory node into the corresponding flake output. + convert = + node: + if node ? children then + builtins.mapAttrs (name: child: convert child) node.children + # Unlike in Nixpkgs, every attribute set in a baked flake only + # contains things worth recursing into, so mark them all with the + # Nixpkgs `recurseForDerivations` convention for `nix search`. + // { + recurseForDerivations = true; + } + else if node.derivation.failed or false then + # The derivation could not be baked (e.g. because it is marked as + # broken, or is a content-addressed derivation). Provide a stub + # that can be listed (e.g. by `nix search`) but fails when built, + # like the original. + let + error = throw "package '${node.derivation.name}' cannot be built because it could not be baked (e.g. because it is marked as broken)"; + in + setAttrByPath (node.derivationAttrPath or [ ]) ( + { + type = "derivation"; + name = node.derivation.name; + meta = + optionalAttrs (node ? shortDescription) { description = node.shortDescription; } + // optionalAttrs (node.derivation ? mainProgram) { mainProgram = node.derivation.mainProgram; }; + drvPath = error; + outPath = error; + outputName = node.derivation.outputName or "out"; + } + // optionalAttrs (node.derivation ? system) { inherit (node.derivation) system; } + ) + else + let + baked = builtins.bakedDerivation { + name = node.derivation.name; + outputs = builtins.mapAttrs (outputName: path: { inherit path; }) node.derivation.outputs; + }; + + outputNames = builtins.attrNames node.derivation.outputs; + + # The output that this attribute refers to. + outputName = node.derivation.outputName or (builtins.head outputNames); + + # Mirror the attribute set produced by `derivation`: the common + # attributes, plus one attribute per output. + commonAttrs = { + type = "derivation"; + name = node.derivation.name; + system = node.derivation.system or (builtins.head node.forSystems); + meta = + optionalAttrs (node ? shortDescription) { description = node.shortDescription; } + // optionalAttrs (node.derivation ? mainProgram) { mainProgram = node.derivation.mainProgram; }; + drvPath = baked.drvPath; + outputs = outputNames; + all = map (outputName: outputAttrs.${outputName}) outputNames; + } + // outputAttrs; + + outputAttrs = builtins.listToAttrs ( + map (outputName: { + name = outputName; + value = commonAttrs // { + inherit outputName; + outPath = baked.${outputName}; + }; + }) outputNames + ); + in + # The derivation may live at a sub-path of the output attribute + # (e.g. `config.system.build.toplevel` for `nixosConfigurations`). + setAttrByPath (node.derivationAttrPath or [ ]) outputAttrs.${outputName}; + + # Convert an inventory node into a flake schema inventory node. The + # original flake's schemas may need attributes that aren't present + # in the baked flake (e.g. `pkgs.stdenv.system` for NixOS + # configurations), so the baked flake provides its own schemas that + # just reproduce the recorded inventory. + mkInventory = + node: + optionalAttrs (node ? forSystems) { inherit (node) forSystems; } + // optionalAttrs (node ? isLegacy) { inherit (node) isLegacy; } + // ( + if node ? children then + { children = builtins.mapAttrs (name: child: mkInventory child) node.children; } + else + { + derivationAttrPath = node.derivationAttrPath or [ ]; + } + // optionalAttrs (node ? what) { inherit (node) what; } + // optionalAttrs (node ? shortDescription) { inherit (node) shortDescription; } + // optionalAttrs (node ? isFlakeCheck) { inherit (node) isFlakeCheck; } + ); + + schemas = builtins.mapAttrs ( + name: node: + let + schema = data.${name}; + in + { + version = 1; + inherit (schema) doc; + inventory = output: mkInventory node; + } + // optionalAttrs (schema ? roles) { + roles = builtins.listToAttrs ( + map (role: { + name = role; + value = { }; + }) schema.roles + ); + } + // optionalAttrs (schema ? appendSystem) { inherit (schema) appendSystem; } + // optionalAttrs (schema ? defaultAttrPath) { inherit (schema) defaultAttrPath; } + ) inventories; + in + builtins.mapAttrs (name: node: convert node) inventories + // { + schemas = schemas // { + schemas = { + version = 1; + doc = "The `schemas` flake output defines the flake schemas of this flake."; + inventory = output: { + children = builtins.mapAttrs (name: schema: { what = "flake schema"; }) output; + }; + }; + }; + }; +} diff --git a/src/nix/build-remote/build-remote.cc b/src/nix/build-remote/build-remote.cc index 8f8f21a8c39c..5c3dea683039 100644 --- a/src/nix/build-remote/build-remote.cc +++ b/src/nix/build-remote/build-remote.cc @@ -18,6 +18,10 @@ #include "nix/util/serialise.hh" #include "nix/store/build-result.hh" #include "nix/store/store-open.hh" +#include "nix/util/config-global.hh" +#include "nix/util/environment-variables.hh" + +#include "otel-logger.hh" #include "nix/util/strings.hh" #include "nix/store/derivations.hh" #include "nix/store/local-store.hh" @@ -86,11 +90,21 @@ static int main_build_remote(int argc, char ** argv) FdSource source(STDIN_FILENO); - /* Read the parent's settings. */ + /* Read the parent's settings. The parent sends the settings of + all `Config`s, not just `Settings`, so apply them + accordingly. Only apply those that differ from ours, though: + we've read the same configuration files, so the rest are + defaults, and setting a deprecated setting warns even if + it's to its default. (Note that we can't just have the + parent send its overridden settings: things like `--store` + set the setting without marking it as overridden.) */ + std::map ourSettings; + globalConfig.getSettings(ourSettings); while (readInt(source)) { auto name = readString(source); auto value = readString(source); - settings.set(name, value); + if (auto i = ourSettings.find(name); i == ourSettings.end() || i->second.value != value) + globalConfig.set(name, value); } auto maxBuildJobs = settings.getWorkerSettings().maxBuildJobs; @@ -109,6 +123,7 @@ static int main_build_remote(int argc, char ** argv) std::shared_ptr sshStore; AutoCloseFD bestSlotLock; + bool tracing = false; auto machines = Machine::parseConfig({settings.thisSystem}, settings.getWorkerSettings().builders); debug("got %d remote builders", machines.size()); @@ -245,6 +260,18 @@ static int main_build_remote(int argc, char ** argv) lock = -1; + /* Set up tracing only now that we're actually going to + use a remote builder, so that a build hook that + turns out to be unused (which is the common case, + since we're started for every build) doesn't upload + a trace. Our root span is parented to the trace of + the `nix` process that runs us, which it passes in + `TRACEPARENT`. */ + if (!tracing) { + tracing = true; + initOtel("nix-build-remote", "nix-build-remote", getEnv("TRACEPARENT").value_or("")); + } + try { storeUri = bestMachine->storeUri.render(); diff --git a/src/nix/bundle.cc b/src/nix/bundle.cc index 4808a2fc716f..659fc8315d17 100644 --- a/src/nix/bundle.cc +++ b/src/nix/bundle.cc @@ -54,21 +54,9 @@ struct CmdBundle : InstallableValueCommand return catSecondary; } - // FIXME: cut&paste from CmdRun. - Strings getDefaultFlakeAttrPaths() override + StringSet getRoles() override { - Strings res{"apps." + settings.thisSystem.get() + ".default", "defaultApp." + settings.thisSystem.get()}; - for (auto & s : SourceExprCommand::getDefaultFlakeAttrPaths()) - res.push_back(s); - return res; - } - - Strings getDefaultFlakeAttrPathPrefixes() override - { - Strings res{"apps." + settings.thisSystem.get() + "."}; - for (auto & s : SourceExprCommand::getDefaultFlakeAttrPathPrefixes()) - res.push_back(s); - return res; + return {"nix-run"}; } void run(ref store, ref installable) override @@ -86,9 +74,9 @@ struct CmdBundle : InstallableValueCommand std::move(bundlerFlakeRef), bundlerName, std::move(extendedOutputsSpec), - {"bundlers." + settings.thisSystem.get() + ".default", "defaultBundler." + settings.thisSystem.get()}, - {"bundlers." + settings.thisSystem.get() + "."}, - lockFlags}; + {"nix-bundler"}, + lockFlags, + getDefaultFlakeSchemas()}; auto vRes = evalState->allocValue(); evalState->callFunction(*bundler.toValue(*evalState).first, *val, *vRes, noPos); @@ -103,6 +91,8 @@ struct CmdBundle : InstallableValueCommand NixStringContext context2; auto drvPath = evalState->coerceToStorePath(attr1->pos, *attr1->value, context2, ""); + evalState->waitForAllPaths(); + drvPath.requireDerivation(); auto attr2 = vRes->attrs()->get(evalState->s.outPath); @@ -111,6 +101,8 @@ struct CmdBundle : InstallableValueCommand auto outPath = evalState->coerceToStorePath(attr2->pos, *attr2->value, context2, ""); + evalState->waitForAllPaths(); + store->buildPaths({ DerivedPath::Built{ .drvPath = makeConstantStorePathRef(drvPath), diff --git a/src/nix/crash-handler.cc b/src/nix/crash-handler.cc index 17c948dab143..4a6c83ba55bb 100644 --- a/src/nix/crash-handler.cc +++ b/src/nix/crash-handler.cc @@ -31,10 +31,12 @@ void logFatal(std::string const & s) #endif } +} // namespace + void onTerminate() { logFatal( - "Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:\n"); + "\nDeterminate Nix crashed. This is a bug. Please report this at https://github.com/DeterminateSystems/nix-src/issues with the following information included:\n"); try { std::exception_ptr eptr = std::current_exception(); if (eptr) { @@ -55,7 +57,6 @@ void onTerminate() std::abort(); } -} // namespace void registerCrashHandler() { diff --git a/src/nix/crash.cc b/src/nix/crash.cc new file mode 100644 index 000000000000..edaf325a9875 --- /dev/null +++ b/src/nix/crash.cc @@ -0,0 +1,79 @@ +#include "nix/cmd/command.hh" + +#include + +using namespace nix; + +static void doThrow() +{ + throw Error("exception in destructor"); +} + +struct CmdCrash : Command +{ + std::string type; + + CmdCrash() + { + expectArg("type", &type); + } + + Category category() override + { + return catUndocumented; + } + + std::string description() override + { + return "crash the program to test crash reporting"; + } + + void run() override + { + if (type == "segfault") { + printError("Triggering a segfault..."); + volatile int * p = nullptr; + *p = 123; + } + + else if (type == "assert") { + printError("Triggering an assertion failure..."); + assert(false && "This is an assertion failure"); + } + + else if (type == "logic-error") { + printError("Triggering a C++ logic error..."); + std::bitset<4>{"012"}; + } + + else if (type == "panic") { + printError("Triggering a panic..."); + panic("test panic"); + } + + else if (type == "abort") { + printError("Triggering an abort..."); + std::abort(); + } + + else if (type == "terminate") { + printError("Triggering an std::terminate..."); + + struct Foo + { + ~Foo() + { + doThrow(); + } + }; + + Foo foo; + } + + else { + throw Error("unknown crash type '%s'", type); + } + } +}; + +static auto rCrash = registerCommand("__crash"); diff --git a/src/nix/develop.cc b/src/nix/develop.cc index 3105efa82536..ec190d26f6fe 100644 --- a/src/nix/develop.cc +++ b/src/nix/develop.cc @@ -1,6 +1,7 @@ #include "nix/cmd/command.hh" #include "nix/util/config-global.hh" #include "nix/expr/eval.hh" +#include "nix/fetchers/fetch-settings.hh" #include "nix/cmd/installable-flake.hh" #include "nix/main/common-args.hh" #include "nix/main/shared.hh" @@ -456,22 +457,9 @@ struct Common : InstallableCommand, MixProfile rewrites.insert({BuildEnvironment::getString(fileInBuilderEnv->second), targetFilePath.string()}); } - Strings getDefaultFlakeAttrPaths() override + StringSet getRoles() override { - Strings paths{ - "devShells." + settings.thisSystem.get() + ".default", - "devShell." + settings.thisSystem.get(), - }; - for (auto & p : SourceExprCommand::getDefaultFlakeAttrPaths()) - paths.push_back(p); - return paths; - } - - Strings getDefaultFlakeAttrPathPrefixes() override - { - auto res = SourceExprCommand::getDefaultFlakeAttrPathPrefixes(); - res.emplace_front("devShells." + settings.thisSystem.get() + "."); - return res; + return {"nix-develop"}; } StorePath getShellOutPath(ref store, ref installable) @@ -507,6 +495,11 @@ struct Common : InstallableCommand, MixProfile shellOutPath, }; } + + void preRun(ref store) override + { + fetchSettings.warnDirty = false; + } }; struct CmdDevelop : Common, MixEnvironment @@ -660,9 +653,9 @@ struct CmdDevelop : Common, MixEnvironment std::move(nixpkgs), "bashInteractive", ExtendedOutputsSpec::Default(), - Strings{}, - Strings{"legacyPackages." + settings.thisSystem.get() + "."}, - nixpkgsLockFlags); + StringSet{"nix-build"}, + nixpkgsLockFlags, + std::nullopt); for (auto & path : Installable::toStorePathSet( getEvalStore(), store, Realise::Outputs, OperateOn::Output, {bashInstallable})) { diff --git a/src/nix/diff-closures.cc b/src/nix/diff-closures.cc index 72108399d12f..08104aab0109 100644 --- a/src/nix/diff-closures.cc +++ b/src/nix/diff-closures.cc @@ -53,10 +53,10 @@ GroupedPaths getClosureInfo(ref store, const StorePath & toplevel) std::string showVersions(const StringSet & versions) { if (versions.empty()) - return "∅"; + return "(absent)"; StringSet versions2; for (auto & version : versions) - versions2.insert(version.empty() ? "ε" : version); + versions2.insert(version.empty() ? "(no version)" : version); return concatStringsSep(", ", versions2); } @@ -103,8 +103,13 @@ void printClosureDiff( if (showDelta || !removed.empty() || !added.empty()) { std::vector items; - if (!removed.empty() || !added.empty()) + if (!removed.empty() && !added.empty()) { items.push_back(fmt("%s → %s", showVersions(removed), showVersions(added))); + } else if (!removed.empty()) { + items.push_back(fmt("%s removed", showVersions(removed))); + } else if (!added.empty()) { + items.push_back(fmt("%s added", showVersions(added))); + } if (showDelta) items.push_back(fmt("%s%s" ANSI_NORMAL, sizeDelta > 0 ? ANSI_RED : ANSI_GREEN, renderSize(sizeDelta))); logger->cout("%s%s: %s", indent, name, concatStringsSep(", ", items)); diff --git a/src/nix/diff-closures.md b/src/nix/diff-closures.md index 0294c0d8def7..6b07af28f958 100644 --- a/src/nix/diff-closures.md +++ b/src/nix/diff-closures.md @@ -11,8 +11,8 @@ R""( baloo-widgets: 20.08.1 → 20.08.2 bluez-qt: +12.6 KiB dolphin: 20.08.1 → 20.08.2, +13.9 KiB - kdeconnect: 20.08.2 → ∅, -6597.8 KiB - kdeconnect-kde: ∅ → 20.08.2, +6599.7 KiB + kdeconnect: 20.08.2 removed, -6597.8 KiB + kdeconnect-kde: 20.08.2 added, +6599.7 KiB … ``` @@ -34,9 +34,9 @@ dolphin: 20.08.1 → 20.08.2, +13.9 KiB No size change is shown if it's below the threshold. If the package does not exist in either the *before* or *after* closures, it is -represented using `∅` (empty set) on the appropriate side of the -arrow. If a package has an empty version string, the version is -rendered as `ε` (epsilon). +represented using `added` or `removed`. +If a package has an empty version string, the version is +rendered as `(no version)`. There may be multiple versions of a package in each closure. In that case, only the changed versions are shown. Thus, diff --git a/src/nix/eval.cc b/src/nix/eval.cc index 6b8c0ba12234..d5db8e4b1dc9 100644 --- a/src/nix/eval.cc +++ b/src/nix/eval.cc @@ -2,6 +2,7 @@ #include "nix/main/common-args.hh" #include "nix/main/shared.hh" #include "nix/store/store-api.hh" +#include "nix/store/local-fs-store.hh" #include "nix/expr/eval.hh" #include "nix/expr/eval-inline.hh" #include "nix/expr/value-to-json.hh" @@ -15,6 +16,7 @@ struct CmdEval : MixJSON, InstallableValueCommand, MixReadOnlyOption bool raw = false; std::optional apply; std::optional writeTo; + std::optional drvLink; CmdEval() : InstallableValueCommand() @@ -38,6 +40,15 @@ struct CmdEval : MixJSON, InstallableValueCommand, MixReadOnlyOption .labels = {"path"}, .handler = {&writeTo}, }); + + addFlag({ + .longName = "drv-link", + .description = + "Use *path* as prefix for symlinks to derivations produced by the evaluation at top-level. By default, no symlinks are created.", + .labels = {"path"}, + .handler = {&drvLink}, + .completer = completePath, + }); } std::string description() override @@ -62,6 +73,9 @@ struct CmdEval : MixJSON, InstallableValueCommand, MixReadOnlyOption if (raw && json) throw UsageError("--raw and --json are mutually exclusive"); + if (drvLink && !raw && !json) + throw UsageError("--drv-link requires --raw or --json"); + auto state = getEvalState(); auto [v, pos] = installable->toValue(*state); @@ -83,10 +97,10 @@ struct CmdEval : MixJSON, InstallableValueCommand, MixReadOnlyOption [&](this const auto & recurse, Value & v, const PosIdx pos, const std::filesystem::path & path) -> void { state->forceValue(v, pos); - if (v.type() == nString) { - copyContext(v, context); + if (v.type() == nString) + // FIXME: disallow strings with contexts? writeFile(path, v.string_view()); - } else if (v.type() == nAttrs) { + else if (v.type() == nAttrs) { [[maybe_unused]] bool directoryCreated = std::filesystem::create_directory(path); // Directory should not already exist assert(directoryCreated); @@ -110,20 +124,46 @@ struct CmdEval : MixJSON, InstallableValueCommand, MixReadOnlyOption else if (raw) { logger->stop(); - auto string = state->coerceToString(noPos, *v, context, "while generating the eval command output"); - writeFull(getStandardOutput(), *string); + writeFull( + getStandardOutput(), + state->devirtualize( + *state->coerceToString(noPos, *v, context, "while generating the eval command output"), context)); } else if (json) { - printJSON(printValueAsJSON(*state, true, *v, pos, context, false)); + // FIXME: use printJSON + auto j = printValueAsJSON(*state, true, *v, pos, context, false); + logger->cout("%s", state->devirtualize(outputPretty ? j.dump(2) : j.dump(), context)); } else { - ValuePrinter printer(*state, *v, PrintOptions{.force = true, .derivationPaths = true}, &context); - logger->cout("%s", printer); + logger->cout("%s", ValuePrinter(*state, *v, PrintOptions{.force = true, .derivationPaths = true})); } - state->ensureLazyPathsCopied(context); + if (drvLink) { + // Create out links for the store paths in `context`. Similar to createOutLinks(), except that for + // derivations, we create symlinks to the .drv files rather than the outputs (since we didn't build + // anything). + if (auto store2 = store.dynamic_pointer_cast()) { + std::set roots; + for (auto & c : context) + std::visit( + overloaded{ + [&](const NixStringContextElem::Opaque & o) { roots.insert(state->devirtualize(o.path)); }, + [&](const NixStringContextElem::DrvDeep & d) { roots.insert(d.drvPath); }, + [&](const NixStringContextElem::Built & b) { roots.insert(b.drvPath->getBaseStorePath()); }, + [&](const NixStringContextElem::Path &) {}, + }, + c.raw); + for (const auto & [i, path] : enumerate(roots)) { + auto symlink = *drvLink; + if (i) + symlink += fmt("-%d", i); + state->waitForPath(path); + store2->addPermRoot(path, absPath(symlink).string()); + } + } + } } }; diff --git a/src/nix/flake-bake.cc b/src/nix/flake-bake.cc new file mode 100644 index 000000000000..1c47c1087966 --- /dev/null +++ b/src/nix/flake-bake.cc @@ -0,0 +1,75 @@ +#include "flake-command.hh" +#include "nix/cmd/flake-schemas.hh" +#include "nix/util/file-system.hh" + +#include + +using namespace nix; +using namespace nix::flake; + +struct CmdFlakeBake : FlakeCommand, MixFlakeSchemas, MixReadOnlyOption +{ + std::filesystem::path destDir; + + flake_schemas::FlakeInventoryOptions options{ + .showLegacy = true, + .showDrvNames = true, + .bake = true, + }; + + CmdFlakeBake() + { + addFlag({ + .longName = "dest-dir", + .description = "Directory in which to write the baked `flake.nix`.", + .labels = {"path"}, + .handler = {&destDir}, + .completer = completePath, + .required = true, + }); + addFlag({ + .longName = "all-systems", + .description = "Bake the outputs for all systems, not just the current system.", + .handler = {&options.showAllSystems, true}, + }); + } + + std::string description() override + { + return "bake a flake"; + } + + std::optional experimentalFeature() override + { + return Xp::BakedDerivations; + } + + std::string doc() override + { + return +#include "flake-bake.md" + ; + } + + void run(nix::ref store) override + { + auto state = getEvalState(); + auto evalStore = getEvalStore(); + auto flake = make_ref(lockFlake()); + + /* Don't use the eval cache: baking evaluates everything exactly once, so caching every attribute in + SQLite is pure overhead (and it serialises parallel evaluation on the database writer). */ + auto cache = flake_schemas::call(*state, flake, getDefaultFlakeSchemas(), /*allowEvalCache=*/false); + + auto inv = flake_schemas::getFlakeInventory(*state, *getEvalStore(), *flake, cache, options); + + std::filesystem::create_directories(destDir); + writeFile(destDir / "outputs.json", inv.dump()); + writeFile( + destDir / "flake.nix", +#include "baked-flake.nix.gen.hh" + ); + } +}; + +static auto rCmdFlakeBake = registerCommand2({"flake", "bake"}); diff --git a/src/nix/flake-bake.md b/src/nix/flake-bake.md new file mode 100644 index 000000000000..c4bc62cd2c1b --- /dev/null +++ b/src/nix/flake-bake.md @@ -0,0 +1,46 @@ +R""( + +# Examples + +* Bake the `patchelf` flake into the directory `./patchelf-baked`: + + ```console + # nix flake bake github:NixOS/patchelf --dest-dir ./patchelf-baked + ``` + + The baked flake can be used like any other flake: + + ```console + # nix flake show path:./patchelf-baked + # nix build path:./patchelf-baked#patchelf + ``` + +* Bake the outputs for all systems, not just the current one: + + ```console + # nix flake bake github:NixOS/patchelf --dest-dir ./patchelf-baked --all-systems + ``` + +# Description + +This command evaluates the outputs of the flake specified by flake reference *flake-url* and writes a new, *baked* flake to the directory *path* (specified by `--dest-dir`). The baked flake provides the same outputs as the original flake, but they are pre-evaluated: every derivation is replaced by a [baked derivation](@docroot@/language/builtins.md#builtins-bakedDerivation), that is, a derivation with the builder `builtin:substitute` that is never built but whose outputs are obtained by substitution from a binary cache. As a result, evaluating a baked flake is much faster than evaluating the original, since it does (almost) no evaluation. + +The baked flake has no inputs, so it can be used without access to the inputs of the original flake. It provides its own [flake schemas](@docroot@/protocols/flake-schemas.md) (in the `schemas` output) that reproduce the structure of the original flake's outputs, so `nix flake show` and `nix flake check` work on it without needing the original flake's schemas. + +Derivation attributes in the baked flake have the same `name`, `system`, `outPath`, `outputName`, `outputs` and `meta.mainProgram` as in the original flake, so commands such as `nix build` and `nix run` behave the same. Their `drvPath` is different, however: it refers to the baked derivation rather than the original one. The outputs of a baked flake can also be used as inputs of other derivations, e.g. by another flake that has the baked flake as an input. + +Building the outputs of a baked flake requires that their store paths can be [substituted](@docroot@/command-ref/conf-file.md#conf-substituters) (or are already present in the Nix store). If a store path cannot be substituted, the build fails; Nix will not fall back to building the original derivation. + +By default, only the outputs for the current system are baked. Use `--all-systems` to bake the outputs for all systems. + +# Limitations + +* Only outputs that are recognised by the [flake schemas](@docroot@/protocols/flake-schemas.md) are baked. Non-derivation outputs, such as `overlays`, `nixosModules` and `templates`, are omitted from the baked flake. + +* Derivations whose output paths are not known at evaluation time cannot be baked. This includes [content-addressed derivations](@docroot@/store/derivation/outputs/content-address.md) and impure derivations. `nix flake bake` prints a warning for each such derivation and omits it from the baked flake. + +* Outputs whose derivation is nested inside the output attribute (such as `nixosConfigurations..config.system.build.toplevel`) are baked at that nested attribute. Other attributes of the output are not included. + +This command requires the [`baked-derivations`](@docroot@/development/experimental-features.md#xp-feature-baked-derivations) experimental feature, both in the Nix CLI and in the Nix daemon. + +)"" diff --git a/src/nix/flake-check.md b/src/nix/flake-check.md index 007640c27c94..a700c5cbe6fe 100644 --- a/src/nix/flake-check.md +++ b/src/nix/flake-check.md @@ -18,66 +18,20 @@ R""( # Description This command verifies that the flake specified by flake reference -*flake-url* can be evaluated successfully (as detailed below), and -that the derivations specified by the flake's `checks` output can be -built successfully. +*flake-url* can be evaluated and built successfully according to its +`schemas` flake output. For every flake output that has a schema +definition, `nix flake check` uses the schema to extract the contents +of the output. Then, for every item in the contents: + +* It evaluates the elements of the `evalChecks` attribute set returned + by the schema for that item, printing an error or warning for every + check that fails to evaluate or that evaluates to `false`. + +* It builds the `derivation` attribute returned by the schema for that + item, if the item has the `isFlakeCheck` attribute. If the `keep-going` option is set to `true`, Nix will keep evaluating as much as it can and report the errors as it encounters them. Otherwise it will stop at the first error. -# Evaluation checks - -The following flake output attributes must be derivations: - -* `checks.`*system*`.`*name* -* `devShells.`*system*`.default` -* `devShells.`*system*`.`*name* -* `nixosConfigurations.`*name*`.config.system.build.toplevel` -* `packages.`*system*`.default` -* `packages.`*system*`.`*name* - -The following flake output attributes must be [app -definitions](./nix3-run.md): - -* `apps.`*system*`.default` -* `apps.`*system*`.`*name* - -The following flake output attributes must be [template -definitions](./nix3-flake-init.md): - -* `templates.default` -* `templates.`*name* - -The following flake output attributes must be *Nixpkgs overlays*: - -* `overlays.default` -* `overlays.`*name* - -The following flake output attributes must be *NixOS modules*: - -* `nixosModules.default` -* `nixosModules.`*name* - -The following flake output attributes must be -[bundlers](./nix3-bundle.md): - -* `bundlers.default` -* `bundlers.`*name* - -Old default attributes are renamed, they will work but will emit a warning: - -* `defaultPackage.` → `packages.`*system*`.default` -* `defaultApps.` → `apps.`*system*`.default` -* `defaultTemplate` → `templates.default` -* `defaultBundler.` → `bundlers.`*system*`.default` -* `overlay` → `overlays.default` -* `devShell.` → `devShells.`*system*`.default` -* `nixosModule` → `nixosModules.default` - -In addition, the `hydraJobs` output is evaluated in the same way as -Hydra's `hydra-eval-jobs` (i.e. as a arbitrarily deeply nested -attribute set of derivations). Similarly, the -`legacyPackages`.*system* output is evaluated like `nix-env --query --available `. - )"" diff --git a/src/nix/flake-metadata.md b/src/nix/flake-metadata.md index b99770049950..6625a9356f92 100644 --- a/src/nix/flake-metadata.md +++ b/src/nix/flake-metadata.md @@ -69,7 +69,7 @@ data. This includes: * `Description`: A one-line description of the flake, taken from the `description` field in `flake.nix`. -* `Path`: The store path containing the source code of the flake. +* `Path`: The store path containing the source code of the flake. This field is absent if the [`lazy-trees`](@docroot@/command-ref/conf-file.md#conf-lazy-trees) setting is enabled, since then the flake is not required to reside in the Nix store. * `Revision`: The Git or Mercurial commit hash of the locked flake. @@ -97,7 +97,7 @@ With `--json`, the output is a JSON object with the following fields: * `description`: See `Description` above. -* `path`: See `Path` above. +* `path`: See `Path` above. This field is absent if the [`lazy-trees`](@docroot@/command-ref/conf-file.md#conf-lazy-trees) setting is enabled. * `revision`: See `Revision` above. diff --git a/src/nix/flake-prefetch-inputs.cc b/src/nix/flake-prefetch-inputs.cc index f1eb7d773868..3ed800e12ed5 100644 --- a/src/nix/flake-prefetch-inputs.cc +++ b/src/nix/flake-prefetch-inputs.cc @@ -42,14 +42,19 @@ struct CmdFlakePrefetchInputs : FlakeCommand if (!state_.lock()->done.insert(&node).second) return; + auto lockedNode = dynamic_cast(&node); + + if (lockedNode && lockedNode->buildTime) + return; + // Relative path inputs are part of their parent's source tree. - if (auto lockedNode = dynamic_cast(&node); - lockedNode && !lockedNode->lockedRef.input.isRelative()) { + if (lockedNode && !lockedNode->lockedRef.input.isRelative()) { try { Activity act(*logger, lvlInfo, actUnknown, fmt("fetching '%s'", lockedNode->lockedRef)); auto accessor = lockedNode->lockedRef.input.getAccessor(fetchSettings, *store).first; - fetchToStore( - fetchSettings, *store, accessor, FetchMode::Copy, lockedNode->lockedRef.input.getName()); + if (!evalSettings.lazyTrees) + fetchToStore( + fetchSettings, *store, accessor, FetchMode::Copy, lockedNode->lockedRef.input.getName()); } catch (Error & e) { printError("%s", e.what()); nrFailed++; diff --git a/src/nix/flake-prefetch-inputs.md b/src/nix/flake-prefetch-inputs.md index a69f7d367915..b571fa348370 100644 --- a/src/nix/flake-prefetch-inputs.md +++ b/src/nix/flake-prefetch-inputs.md @@ -12,6 +12,6 @@ R""( Fetch the inputs of a flake. This ensures that they are already available for any subsequent evaluation of the flake. -This operation is recursive: it will fetch not just the direct inputs of the top-level flake, but also transitive inputs. +This operation is recursive: it fetches not just the direct inputs of the top-level flake, but also transitive inputs. It skips build-time inputs, i.e. inputs that have the attribute `buildTime = true`. )"" diff --git a/src/nix/flake.cc b/src/nix/flake.cc index e56c3d86a8c3..f94815eb88ad 100644 --- a/src/nix/flake.cc +++ b/src/nix/flake.cc @@ -9,7 +9,6 @@ #include "nix/store/derived-path.hh" #include "nix/util/os-string.hh" #include "nix/util/signals.hh" -#include "nix/util/mounted-source-accessor.hh" #include "nix/store/store-open.hh" #include "nix/store/derivations.hh" #include "nix/store/outputs-spec.hh" @@ -23,6 +22,10 @@ #include "nix/fetchers/fetch-to-store.hh" #include "nix/store/local-fs-store.hh" #include "nix/store/globals.hh" +#include "nix/expr/parallel-eval.hh" +#include "nix/util/exit.hh" +#include "nix/cmd/flake-schemas.hh" +#include "nix/store/names.hh" #include #include @@ -135,6 +138,7 @@ struct CmdFlakeUpdate : FlakeCommand lockFlags.recreateLockFile = updateAll; lockFlags.writeLockFile = true; lockFlags.applyNixConfig = true; + lockFlags.requireLockable = false; lockFlake(); } @@ -167,38 +171,12 @@ struct CmdFlakeLock : FlakeCommand lockFlags.writeLockFile = true; lockFlags.failOnUnlocked = true; lockFlags.applyNixConfig = true; + lockFlags.requireLockable = false; lockFlake(); } }; -static void enumerateOutputs( - EvalState & state, - Value & vFlake, - std::function callback) -{ - auto pos = vFlake.determinePos(noPos); - state.forceAttrs(vFlake, pos, "while evaluating a flake to get its outputs"); - - auto aOutputs = vFlake.attrs()->get(state.symbols.create("outputs")); - assert(aOutputs); - - state.forceAttrs(*aOutputs->value, pos, "while evaluating the outputs of a flake"); - - auto sHydraJobs = state.symbols.create("hydraJobs"); - - /* Hack: ensure that hydraJobs is evaluated before anything - else. This way we can disable IFD for hydraJobs and then enable - it for other outputs. */ - if (auto attr = aOutputs->value->attrs()->get(sHydraJobs)) - callback(state.symbols[attr->name], *attr->value, attr->pos); - - for (auto & attr : *aOutputs->value->attrs()) { - if (attr.name != sHydraJobs) - callback(state.symbols[attr.name], *attr.value, attr.pos); - } -} - struct CmdFlakeMetadata : FlakeCommand, MixJSON { std::string description() override @@ -215,13 +193,17 @@ struct CmdFlakeMetadata : FlakeCommand, MixJSON void run(nix::ref store) override { + lockFlags.requireLockable = false; auto lockedFlake = lockFlake(); auto & flake = lockedFlake.flake; - /* Flakes do not get copied to the store, but are instead mounted at - their expected store paths in storeFS. Querying metadata does not - force copying to the store, as one would expect. */ - auto storePath = store->toStorePath(flake.path.path.abs()).first; + /* Hack to show the store path if available. */ + std::optional storePath; + if (store->isInStore(flake.path.path.abs())) { + auto path = store->toStorePath(flake.path.path.abs()).first; + if (store->isValidPath(path)) + storePath = path; + } if (json) { nlohmann::json j; @@ -243,7 +225,8 @@ struct CmdFlakeMetadata : FlakeCommand, MixJSON j["revCount"] = *revCount; if (auto lastModified = flake.lockedRef.input.getLastModified()) j["lastModified"] = *lastModified; - j["path"] = store->printStorePath(storePath); + if (storePath) + j["path"] = store->printStorePath(*storePath); j["locks"] = lockedFlake.lockFile.toJSON().first; if (auto fingerprint = lockedFlake.getFingerprint(*store, fetchSettings)) j["fingerprint"] = fingerprint->to_string(HashFormat::Base16, false); @@ -254,7 +237,8 @@ struct CmdFlakeMetadata : FlakeCommand, MixJSON logger->cout(ANSI_BOLD "Locked URL:" ANSI_NORMAL " %s", flake.lockedRef.to_string()); if (flake.description) logger->cout(ANSI_BOLD "Description:" ANSI_NORMAL " %s", *flake.description); - logger->cout(ANSI_BOLD "Path:" ANSI_NORMAL " %s", store->printStorePath(storePath)); + if (storePath) + logger->cout(ANSI_BOLD "Path:" ANSI_NORMAL " %s", store->printStorePath(*storePath)); if (auto rev = flake.lockedRef.input.getRev()) logger->cout(ANSI_BOLD "Revision:" ANSI_NORMAL " %s", rev->to_string(HashFormat::Base16, false)); if (auto dirtyRev = fetchers::maybeGetStrAttr(flake.lockedRef.toAttrs(), "dirtyRev")) @@ -284,7 +268,7 @@ struct CmdFlakeMetadata : FlakeCommand, MixJSON "%s" ANSI_BOLD "%s" ANSI_NORMAL ": %s%s", prefix + (last ? treeLast : treeConn), input.first, - (*lockedNode)->lockedRef, + (*lockedNode)->lockedRef.to_string(true), lastModifiedStr); bool firstVisit = visited.insert(*lockedNode).second; @@ -313,9 +297,39 @@ struct CmdFlakeInfo : CmdFlakeMetadata } }; -struct CmdFlakeCheck : FlakeCommand, MixPrintOutPaths, MixOutLinkBase +/** + * Log the exception `error`, after forcing cached evaluation errors. + * + * Note: this must not be called from inside a catch handler. + * `CachedEvalError::force()` evaluates the original attribute, which + * may have to wait on a thunk owned by another fiber, and a fiber + * must not suspend while an exception is being handled. + */ +static void logEvalError(std::exception_ptr error) +{ + std::optional cached; + try { + std::rethrow_exception(error); + } catch (eval_cache::CachedEvalError & e) { + cached.emplace(e); + } catch (Error & e) { + logError(e.info()); + return; + } + + assert(!std::current_exception()); + + try { + cached->force(); + } catch (Error & e) { + logError(e.info()); + } +} + +struct CmdFlakeCheck : FlakeCommand, MixPrintOutPaths, MixOutLinkBase, MixFlakeSchemas { bool build = true; + bool buildAll = false; bool checkAllSystems = false; CmdFlakeCheck() @@ -326,6 +340,11 @@ struct CmdFlakeCheck : FlakeCommand, MixPrintOutPaths, MixOutLinkBase .description = "Do not build checks.", .handler = {&build, false}, }); + addFlag({ + .longName = "build-all", + .description = "Build all derivations, not just checks.", + .handler = {&buildAll, true}, + }); addFlag({ .longName = "all-systems", .description = "Check the outputs for all systems.", @@ -364,442 +383,133 @@ struct CmdFlakeCheck : FlakeCommand, MixPrintOutPaths, MixOutLinkBase auto state = getEvalState(); lockFlags.applyNixConfig = true; - auto flake = lockFlake(); + auto flake = std::make_shared(lockFlake()); auto localSystem = std::string(settings.thisSystem.get()); - bool hasErrors = false; - auto reportError = [&](const Error & e) { - try { - throw e; - } catch (Interrupted & e) { - throw; - } catch (Error & e) { - if (settings.getWorkerSettings().keepGoing) { - logError(e.info()); - hasErrors = true; - } else - throw; - } - }; - - StringSet omittedSystems; - - // FIXME: rewrite to use EvalCache. - - auto resolve = [&](PosIdx p) { return state->positions[p]; }; - - auto argHasName = [&](Symbol arg, std::string_view expected) { - std::string_view name = state->symbols[arg]; - return name == expected || name == "_" || (hasPrefix(name, "_") && name.substr(1) == expected); - }; - - auto checkSystemName = [&](std::string_view system, const PosIdx pos) { - // FIXME: what's the format of "system"? - if (system.find('-') == std::string::npos) - reportError(Error("'%s' is not a valid system type, at %s", system, resolve(pos))); - }; - - auto checkSystemType = [&](std::string_view system, const PosIdx pos) { - if (!checkAllSystems && system != localSystem) { - omittedSystems.insert(std::string(system)); - return false; - } else { - return true; - } - }; - - auto checkDerivation = - [&](const std::string & attrPath, Value & v, const PosIdx pos) -> std::optional { - try { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking derivation %s", attrPath)); - auto packageInfo = getDerivation(*state, v, false); - if (!packageInfo) - throw Error("flake attribute '%s' is not a derivation", attrPath); - else { - // FIXME: check meta attributes - auto storePath = packageInfo->queryDrvPath(); - if (storePath) { - logger->log( - lvlInfo, fmt("derivation evaluated to %s", store->printStorePath(storePath.value()))); - } - return storePath; - } - } catch (Error & e) { - e.addTrace(resolve(pos), HintFmt("while checking the derivation '%s'", attrPath)); - reportError(e); - } - return std::nullopt; - }; - - std::map> attrPathsByDrv; - - auto checkApp = [&](const std::string & attrPath, Value & v, const PosIdx pos) { - try { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking app '%s'", attrPath)); - state->forceAttrs(v, pos, ""); - if (auto attr = v.attrs()->get(state->symbols.create("type"))) - state->forceStringNoCtx(*attr->value, attr->pos, ""); - else - throw Error("app '%s' lacks attribute 'type'", attrPath); - - if (auto attr = v.attrs()->get(state->symbols.create("program"))) { - if (attr->name == state->symbols.create("program")) { - NixStringContext context; - state->forceString(*attr->value, context, attr->pos, ""); - } - } else - throw Error("app '%s' lacks attribute 'program'", attrPath); - - if (auto attr = v.attrs()->get(state->symbols.create("meta"))) { - state->forceAttrs(*attr->value, attr->pos, ""); - if (auto dAttr = attr->value->attrs()->get(state->symbols.create("description"))) - state->forceStringNoCtx(*dAttr->value, dAttr->pos, ""); - else - logWarning({ - .msg = HintFmt("app '%s' lacks attribute 'meta.description'", attrPath), - }); - } else - logWarning({ - .msg = HintFmt("app '%s' lacks attribute 'meta'", attrPath), - }); - - for (auto & attr : *v.attrs()) { - std::string_view name(state->symbols[attr.name]); - if (name != "type" && name != "program" && name != "meta") - throw Error("app '%s' has unsupported attribute '%s'", attrPath, name); - } - } catch (Error & e) { - e.addTrace(resolve(pos), HintFmt("while checking the app definition '%s'", attrPath)); - reportError(e); - } - }; - - auto checkOverlay = [&](std::string_view attrPath, Value & v, const PosIdx pos) { - try { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking overlay '%s'", attrPath)); - state->forceValue(v, pos); - if (!v.isLambda()) { - throw Error("overlay is not a function, but %s instead", showType(v)); - } - if (v.lambda().fun->getFormals() || !argHasName(v.lambda().fun->arg, "final")) - throw Error("overlay does not take an argument named 'final'"); - // FIXME: if we have a 'nixpkgs' input, use it to - // evaluate the overlay. - } catch (Error & e) { - e.addTrace(resolve(pos), HintFmt("while checking the overlay '%s'", attrPath)); - reportError(e); - } - }; - - auto checkModule = [&](std::string_view attrPath, Value & v, const PosIdx pos) { - try { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking NixOS module '%s'", attrPath)); - state->forceValue(v, pos); - } catch (Error & e) { - e.addTrace(resolve(pos), HintFmt("while checking the NixOS module '%s'", attrPath)); - reportError(e); - } - }; - - std::function checkHydraJobs; + auto cache = flake_schemas::call(*state, flake, getDefaultFlakeSchemas()); - checkHydraJobs = [&](std::string_view attrPath, Value & v, const PosIdx pos) { - try { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking Hydra job '%s'", attrPath)); - state->forceAttrs(v, pos, ""); + auto inventory = cache->getRoot()->getAttr("inventory"); + auto outputs = cache->getRoot()->getAttr("outputs"); - if (state->isDerivation(v)) - throw Error("jobset should not be a derivation at top-level"); + FutureVector futures(*state->executor); - for (auto & attr : *v.attrs()) { - state->forceAttrs(*attr.value, attr.pos, ""); - auto attrPath2 = concatStrings(attrPath, ".", state->symbols[attr.name]); - if (state->isDerivation(*attr.value)) { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking Hydra job '%s'", attrPath2)); - checkDerivation(attrPath2, *attr.value, attr.pos); - } else - checkHydraJobs(attrPath2, *attr.value, attr.pos); - } + Sync> drvPaths_; + Sync> uncheckedOutputs; + Sync> omittedSystems; + Sync>> derivedPathToAttrPaths_; - } catch (Error & e) { - e.addTrace(resolve(pos), HintFmt("while checking the Hydra jobset '%s'", attrPath)); - reportError(e); - } - }; + std::function node)> visit; - auto checkNixOSConfiguration = [&](const std::string & attrPath, Value & v, const PosIdx pos) { - try { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking NixOS configuration '%s'", attrPath)); - Bindings & bindings = Bindings::emptyBindings; - auto vToplevel = findAlongAttrPath(*state, "config.system.build.toplevel", bindings, v).first; - state->forceValue(*vToplevel, pos); - if (!state->isDerivation(*vToplevel)) - throw Error("attribute 'config.system.build.toplevel' is not a derivation"); - } catch (Error & e) { - e.addTrace(resolve(pos), HintFmt("while checking the NixOS configuration '%s'", attrPath)); - reportError(e); - } - }; + std::atomic_bool hasErrors = false; - auto checkTemplate = [&](std::string_view attrPath, Value & v, const PosIdx pos) { - try { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking template '%s'", attrPath)); + visit = [&](ref node) { + flake_schemas::visit( + checkAllSystems ? std::optional() : localSystem, + false, // FIXME: add a --legacy flag? + node, + flake->flake.provenance, - state->forceAttrs(v, pos, ""); + [&](const flake_schemas::Leaf & leaf) { + /* Set if the check failed and we're in `--keep-going` + mode. Logged below, outside the catch handler. */ + std::exception_ptr error; - if (auto attr = v.attrs()->get(state->symbols.create("path"))) { - if (attr->name == state->symbols.create("path")) { - NixStringContext context; - auto path = state->coerceToPath(attr->pos, *attr->value, context, ""); - if (!path.pathExists()) - throw Error("template '%s' refers to a non-existent path '%s'", attrPath, path); - // TODO: recursively check the flake in 'path'. - } - } else - throw Error("template '%s' lacks attribute 'path'", attrPath); - - if (auto attr = v.attrs()->get(state->symbols.create("description"))) - state->forceStringNoCtx(*attr->value, attr->pos, ""); - else - throw Error("template '%s' lacks attribute 'description'", attrPath); - - for (auto & attr : *v.attrs()) { - std::string_view name(state->symbols[attr.name]); - if (name != "path" && name != "description" && name != "welcomeText") - throw Error("template '%s' has unsupported attribute '%s'", attrPath, name); - } - } catch (Error & e) { - e.addTrace(resolve(pos), HintFmt("while checking the template '%s'", attrPath)); - reportError(e); - } - }; - - auto checkBundler = [&](const std::string & attrPath, Value & v, const PosIdx pos) { - try { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking bundler '%s'", attrPath)); - state->forceValue(v, pos); - if (!v.isLambda()) - throw Error("bundler must be a function"); - // TODO: check types of inputs/outputs? - } catch (Error & e) { - e.addTrace(resolve(pos), HintFmt("while checking the template '%s'", attrPath)); - reportError(e); - } - }; - - { - Activity act(*logger, lvlInfo, actUnknown, "evaluating flake"); - - auto vFlake = state->allocValue(); - flake::callFlake(*state, flake, *vFlake); - - enumerateOutputs(*state, *vFlake, [&](std::string_view name, Value & vOutput, const PosIdx pos) { - Activity act(*logger, lvlInfo, actUnknown, fmt("checking flake output '%s'", name)); - - try { - evalSettings.enableImportFromDerivation.setDefault(name != "hydraJobs"); - - state->forceValue(vOutput, pos); - - std::string_view replacement = name == "defaultPackage" ? "packages..default" - : name == "defaultApp" ? "apps..default" - : name == "defaultTemplate" ? "templates.default" - : name == "defaultBundler" ? "bundlers..default" - : name == "overlay" ? "overlays.default" - : name == "devShell" ? "devShells..default" - : name == "nixosModule" ? "nixosModules.default" - : ""; - if (replacement != "") - warn("flake output attribute '%s' is deprecated; use '%s' instead", name, replacement); - - if (name == "checks") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) { - std::string_view attr_name = state->symbols[attr.name]; - checkSystemName(attr_name, attr.pos); - if (checkSystemType(attr_name, attr.pos)) { - state->forceAttrs(*attr.value, attr.pos, ""); - for (auto & attr2 : *attr.value->attrs()) { - auto drvPath = checkDerivation( - fmt("%s.%s.%s", name, attr_name, state->symbols[attr2.name]), - *attr2.value, - attr2.pos); - if (drvPath && attr_name == settings.thisSystem.get()) { - auto path = DerivedPath::Built{ - .drvPath = makeConstantStorePathRef(*drvPath), - .outputs = OutputsSpec::All{}, - }; - - // Build and store the attribute path for error reporting - AttrPath attrPath{state->symbols.create(name), attr.name, attr2.name}; - attrPathsByDrv[path].push_back(std::move(attrPath)); - } - } + try { + bool done = true; + bool buildSkipped = false; + + if (auto evalChecks = leaf.node->maybeGetAttr("evalChecks")) { + auto checkNames = evalChecks->getAttrs(); + for (auto & checkName : checkNames) { + auto cursor = evalChecks->getAttr(checkName); + Activity act( + *logger, + lvlInfo, + actUnknown, + fmt("running flake check '%s'", cursor->getAttrPathStr())); + auto b = cursor->getBool(); + if (!b) + throw Error("Evaluation check '%s' failed.", cursor->getAttrPathStr()); } } - } - else if (name == "formatter") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) { - const auto & attr_name = state->symbols[attr.name]; - checkSystemName(attr_name, attr.pos); - if (checkSystemType(attr_name, attr.pos)) { - checkDerivation(fmt("%s.%s", name, attr_name), *attr.value, attr.pos); - }; + if (auto drv = leaf.derivation(outputs)) { + + /* Check whether this is a valid derivation. */ + if (!drv->maybeGetAttr("drvPath") || drv->getAttr("type")->getString() != "derivation") + throw Error("Flake output '%s' is not a derivation.", drv->getAttrPathStr()); + + DrvName parsedDrvName(drv->getAttr("name")->getString()); + + if (buildAll || leaf.isFlakeCheck()) { + auto drvPath = drv->forceDerivation(); + auto derivedPath = DerivedPath::Built{ + .drvPath = makeConstantStorePathRef(drvPath), + .outputs = OutputsSpec::All{}, + }; + (*derivedPathToAttrPaths_.lock())[derivedPath].push_back(leaf.node->getAttrPath()); + drvPaths_.lock()->push_back(std::move(derivedPath)); + if (build) + done = false; + } else + buildSkipped = true; } - } - else if (name == "packages" || name == "devShells") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) { - const auto & attr_name = state->symbols[attr.name]; - checkSystemName(attr_name, attr.pos); - if (checkSystemType(attr_name, attr.pos)) { - state->forceAttrs(*attr.value, attr.pos, ""); - for (auto & attr2 : *attr.value->attrs()) - checkDerivation( - fmt("%s.%s.%s", name, attr_name, state->symbols[attr2.name]), - *attr2.value, - attr2.pos); - }; - } - } - - else if (name == "apps") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) { - const auto & attr_name = state->symbols[attr.name]; - checkSystemName(attr_name, attr.pos); - if (checkSystemType(attr_name, attr.pos)) { - state->forceAttrs(*attr.value, attr.pos, ""); - for (auto & attr2 : *attr.value->attrs()) - checkApp( - fmt("%s.%s.%s", name, attr_name, state->symbols[attr2.name]), - *attr2.value, - attr2.pos); - }; - } - } - - else if (name == "defaultPackage" || name == "devShell") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) { - const auto & attr_name = state->symbols[attr.name]; - checkSystemName(attr_name, attr.pos); - if (checkSystemType(attr_name, attr.pos)) { - checkDerivation(fmt("%s.%s", name, attr_name), *attr.value, attr.pos); - }; - } - } - - else if (name == "defaultApp") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) { - const auto & attr_name = state->symbols[attr.name]; - checkSystemName(attr_name, attr.pos); - if (checkSystemType(attr_name, attr.pos)) { - checkApp(fmt("%s.%s", name, attr_name), *attr.value, attr.pos); - }; - } - } - - else if (name == "legacyPackages") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) { - checkSystemName(state->symbols[attr.name], attr.pos); - checkSystemType(state->symbols[attr.name], attr.pos); - // FIXME: do getDerivations? - } - } - - else if (name == "overlay") - checkOverlay(name, vOutput, pos); - - else if (name == "overlays") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) - checkOverlay(fmt("%s.%s", name, state->symbols[attr.name]), *attr.value, attr.pos); - } - - else if (name == "nixosModule") - checkModule(name, vOutput, pos); - - else if (name == "nixosModules") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) - checkModule(fmt("%s.%s", name, state->symbols[attr.name]), *attr.value, attr.pos); - } - - else if (name == "nixosConfigurations") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) - checkNixOSConfiguration( - fmt("%s.%s", name, state->symbols[attr.name]), *attr.value, attr.pos); + if (done) + notice( + "✅ " ANSI_BOLD "%s" ANSI_NORMAL "%s", + leaf.node->getAttrPathStr(), + buildSkipped ? ANSI_ITALIC ANSI_FAINT " (build skipped)" : ""); + } catch (Interrupted & e) { + throw; + } catch (Error & e) { + printError("❌ " ANSI_RED "%s" ANSI_NORMAL, leaf.node->getAttrPathStr()); + if (settings.getWorkerSettings().keepGoing) { + error = std::current_exception(); + hasErrors = true; + } else + throw; } - else if (name == "hydraJobs") - checkHydraJobs(name, vOutput, pos); + if (error) + logEvalError(error); + }, - else if (name == "defaultTemplate") - checkTemplate(name, vOutput, pos); + [&](std::function forEachChild) { + forEachChild([&](Symbol attrName, ref node, bool isLast) { + state->spawn(futures, 2, [&visit, node]() { visit(node); }); + }); + }, - else if (name == "templates") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) - checkTemplate(fmt("%s.%s", name, state->symbols[attr.name]), *attr.value, attr.pos); - } + [&](ref node, const std::vector & systems) { + for (auto & s : systems) + omittedSystems.lock()->insert(s); + }, - else if (name == "defaultBundler") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) { - const auto & attr_name = state->symbols[attr.name]; - checkSystemName(attr_name, attr.pos); - if (checkSystemType(attr_name, attr.pos)) { - checkBundler(fmt("%s.%s", name, attr_name), *attr.value, attr.pos); - }; - } - } + [&](ref node) {}); + }; - else if (name == "bundlers") { - state->forceAttrs(vOutput, pos, ""); - for (auto & attr : *vOutput.attrs()) { - const auto & attr_name = state->symbols[attr.name]; - checkSystemName(attr_name, attr.pos); - if (checkSystemType(attr_name, attr.pos)) { - state->forceAttrs(*attr.value, attr.pos, ""); - for (auto & attr2 : *attr.value->attrs()) { - checkBundler( - fmt("%s.%s.%s", name, attr_name, state->symbols[attr2.name]), - *attr2.value, - attr2.pos); - } - }; - } - } + flake_schemas::forEachOutput( + inventory, + [&](Symbol outputName, + std::shared_ptr output, + const std::string & doc, + bool isLast) { + if (output) + state->spawn(futures, 1, [&visit, output(ref(output))]() { visit(output); }); + else + uncheckedOutputs.lock()->insert(std::string(state->symbols[outputName])); + }); - else if ( - name == "lib" || name == "darwinConfigurations" || name == "darwinModules" - || name == "flakeModule" || name == "flakeModules" || name == "herculesCI" - || name == "homeConfigurations" || name == "homeModule" || name == "homeModules" - || name == "nixopsConfigurations") - // Known but unchecked community attribute - ; + futures.finishAll(); - else - warn("unknown flake output '%s'", name); + if (!uncheckedOutputs.lock()->empty()) + warn("The following flake outputs are unchecked: %s.", concatStringsSep(", ", *uncheckedOutputs.lock())); - } catch (Error & e) { - e.addTrace(resolve(pos), HintFmt("while checking flake output '%s'", name)); - reportError(e); - } - }); - } + auto drvPaths(drvPaths_.lock()); + auto derivedPathToAttrPaths(derivedPathToAttrPaths_.lock()); - std::vector results; - if (build && !attrPathsByDrv.empty()) { - auto keys = std::views::keys(attrPathsByDrv); - std::vector drvPaths(keys.begin(), keys.end()); + if (build && !drvPaths->empty()) { // TODO: This filtering of substitutable paths is a temporary workaround until // https://github.com/NixOS/nix/issues/5025 (union stores) is implemented. // @@ -811,71 +521,85 @@ struct CmdFlakeCheck : FlakeCommand, MixPrintOutPaths, MixOutLinkBase // For now, we skip building derivations whose outputs are already available // via substitution, as `nix flake check` only needs to verify buildability, // not actually produce the outputs. - auto missing = store->queryMissing(drvPaths); + state->waitForAllPaths(); std::vector toBuild; - for (auto & path : missing.willBuild) { - toBuild.emplace_back( - DerivedPath::Built{ + + if (printOutputPaths || outLink) { + toBuild = *drvPaths; + } else { + std::set toBuildSet; + for (auto & path : store->queryMissing(*drvPaths).willBuild) { + auto derivedPath = DerivedPath::Built{ .drvPath = makeConstantStorePathRef(path), .outputs = OutputsSpec::All{}, - }); + }; + toBuild.emplace_back(derivedPath); + toBuildSet.insert(std::move(derivedPath)); + } + + for (auto & [derivedPath, attrPaths] : *derivedPathToAttrPaths) + if (!toBuildSet.contains(derivedPath)) + for (auto & attrPath : attrPaths) + notice( + "✅ " ANSI_BOLD "%s" ANSI_NORMAL ANSI_ITALIC ANSI_FAINT + " (previously built)" ANSI_NORMAL, + attrPath.to_string(*state)); } + // FIXME: should start building while evaluating. Activity act(*logger, lvlInfo, actUnknown, fmt("running %d flake checks", toBuild.size())); - // once we get rid of the temporary hack above, this tenary operator will also go away - results = store->buildPathsWithResults((printOutputPaths || outLink) ? drvPaths : toBuild); - - // Report build failures with attribute paths - for (auto & result : results) { - if (auto * failure = result.tryGetFailure()) { - auto it = attrPathsByDrv.find(result.path); - if (it != attrPathsByDrv.end() && !it->second.empty()) { - for (auto & attrPath : it->second) { - reportError(Error( - "failed to build attribute '%s', build of '%s' failed: %s", - attrPath.to_string(*state), - result.path.to_string(*store), - failure->message())); - } - } else { - // Derivation has no attribute path (e.g., a build dependency) - reportError( - Error("build of '%s' failed: %s", result.path.to_string(*store), failure->message())); + auto buildResults = store->buildPathsWithResults(toBuild); + assert(buildResults.size() == toBuild.size()); + + for (auto & buildResult : buildResults) { + if (auto failure = buildResult.tryGetFailure()) + try { + hasErrors = true; + for (auto & attrPath : (*derivedPathToAttrPaths)[buildResult.path]) + if (failure->status == BuildResult::Failure::Cancelled) + notice( + "❓ " ANSI_BOLD "%s" ANSI_NORMAL ANSI_FAINT " (cancelled)", + attrPath.to_string(*state)); + else + printError("❌ " ANSI_RED "%s" ANSI_NORMAL, attrPath.to_string(*state)); + if (failure->status != BuildResult::Failure::Cancelled) + throw *failure; + } catch (Error & e) { + logError(e.info()); } - } + else + for (auto & attrPath : (*derivedPathToAttrPaths)[buildResult.path]) + notice("✅ " ANSI_BOLD "%s" ANSI_NORMAL, attrPath.to_string(*state)); } - } - if (hasErrors) - throw Error("some errors were encountered during the evaluation"); - logger->log(lvlInfo, ANSI_GREEN "all checks passed!" ANSI_NORMAL); + auto builtPaths = + buildResults | std::views::filter([](auto & result) { return result.tryGetSuccess(); }) + | std::views::transform([&](auto & result) { return toBuiltPath(result, getEvalStore(), store); }) + | std::ranges::to(); - if (!omittedSystems.empty()) { + printOutPathsMaybe(builtPaths, store); + + createOutLinksMaybe(builtPaths, store); + } + + if (!omittedSystems.lock()->empty()) { // TODO: empty system is not visible; render all as nix strings? warn( "The check omitted these incompatible systems: %s\n" "Use '--all-systems' to check all.", - concatStringsSep(", ", omittedSystems)); - }; - - auto builtPaths = - results | std::views::transform([&](auto & result) { return toBuiltPath(result, getEvalStore(), store); }) - | std::ranges::to(); - - printOutPathsMaybe(builtPaths, store); + concatStringsSep(", ", *omittedSystems.lock())); + } - createOutLinksMaybe(builtPaths, store); + if (hasErrors) + throw Exit(1); }; }; -static Strings defaultTemplateAttrPathsPrefixes{"templates."}; -static Strings defaultTemplateAttrPaths = {"templates.default", "defaultTemplate"}; - -struct CmdFlakeInitCommon : virtual Args, EvalCommand +struct CmdFlakeInitCommon : virtual Args, EvalCommand, MixFlakeSchemas { - std::string templateUrl = "templates"; + std::string templateUrl = "https://flakehub.com/f/DeterminateSystems/flake-templates/0.1"; std::filesystem::path destDir; const flake::LockFlags lockFlags{.writeLockFile = false}; @@ -889,13 +613,7 @@ struct CmdFlakeInitCommon : virtual Args, EvalCommand .labels = {"template"}, .handler = {&templateUrl}, .completer = {[&](AddCompletions & completions, size_t, std::string_view prefix) { - completeFlakeRefWithFragment( - completions, - getEvalState(), - lockFlags, - defaultTemplateAttrPathsPrefixes, - defaultTemplateAttrPaths, - prefix); + completeFlakeRefWithFragment(completions, getEvalState(), lockFlags, {"nix-template"}, prefix); }}, }); } @@ -915,9 +633,9 @@ struct CmdFlakeInitCommon : virtual Args, EvalCommand std::move(templateFlakeRef), templateName, ExtendedOutputsSpec::Default(), - defaultTemplateAttrPaths, - defaultTemplateAttrPathsPrefixes, - lockFlags); + {"nix-template"}, + lockFlags, + {}); auto cursor = installable.getCursor(*evalState); @@ -1114,25 +832,22 @@ struct CmdFlakeArchive : FlakeCommand, MixJSON, MixDryRun, MixNoCheckSigs StorePathSet sources; - auto getStorePath = [&](const FlakeRef & lockedRef) { - return dryRun ? lockedRef.input.computeStorePath(*store) - : std::get(lockedRef.input.fetchToStore(fetchSettings, *store)); - }; - - auto storePath = getStorePath(flake.flake.lockedRef); + auto storePath = dryRun ? flake.flake.lockedRef.input.computeStorePath(*store) + : std::get(flake.flake.lockedRef.input.fetchToStore(fetchSettings, *store)); sources.insert(storePath); // FIXME: use graph output, handle cycles. - auto traverse = [&store, json = json, &sources, &getStorePath]( - this const auto & self, const flake::Node & node) -> nlohmann::json { + auto traverse = + [&, json = json, dryRun = dryRun](this const auto & self, const flake::Node & node) -> nlohmann::json { nlohmann::json jsonObj2 = json ? nlohmann::json::object() : nlohmann::json(nullptr); for (auto & [inputName, input] : node.inputs) { if (auto inputNode = std::get_if<0>(&input)) { std::optional storePath; - const auto & lockedRef = (*inputNode)->lockedRef; - if (!lockedRef.input.isRelative()) { - storePath = getStorePath(lockedRef); + if (!(*inputNode)->lockedRef.input.isRelative()) { + storePath = dryRun ? (*inputNode)->lockedRef.input.computeStorePath(*store) + : std::get( + (*inputNode)->lockedRef.input.fetchToStore(fetchSettings, *store)); sources.insert(*storePath); } if (json) { @@ -1165,22 +880,36 @@ struct CmdFlakeArchive : FlakeCommand, MixJSON, MixDryRun, MixNoCheckSigs } }; -struct CmdFlakeShow : FlakeCommand, MixJSON +struct CmdFlakeShow : FlakeCommand, MixJSON, MixFlakeSchemas { - bool showLegacy = false; - bool showAllSystems = false; + flake_schemas::FlakeInventoryOptions options; CmdFlakeShow() { addFlag({ .longName = "legacy", .description = "Show the contents of the `legacyPackages` output.", - .handler = {&showLegacy, true}, + .handler = {&options.showLegacy, true}, }); addFlag({ .longName = "all-systems", .description = "Show the contents of outputs for all systems.", - .handler = {&showAllSystems, true}, + .handler = {&options.showAllSystems, true}, + }); + addFlag({ + .longName = "output-paths", + .description = "Include the store paths of derivation outputs in the JSON output.", + .handler = {&options.showOutputPaths, true}, + }); + addFlag({ + .longName = "drv-paths", + .description = "Include the store paths of derivations in the JSON output.", + .handler = {&options.showDrvPaths, true}, + }); + addFlag({ + .longName = "drv-names", + .description = "Show the names and versions of derivations.", + .handler = {&options.showDrvNames, true}, }); } @@ -1198,298 +927,86 @@ struct CmdFlakeShow : FlakeCommand, MixJSON void run(nix::ref store) override { - evalSettings.enableImportFromDerivation.setDefault(false); + if (options.showOutputPaths && !json) + throw UsageError("The '--output-paths' flag requires '--json'."); + + if (options.showDrvPaths && !json) + throw UsageError("The '--drv-paths' flag requires '--json'."); + + if (json) + options.showDrvNames = true; auto state = getEvalState(); auto flake = make_ref(lockFlake()); - auto localSystem = std::string(settings.thisSystem.get()); - std::function - hasContent; - - // For frameworks it's important that structures are as lazy as possible - // to prevent infinite recursions, performance issues and errors that - // aren't related to the thing to evaluate. As a consequence, they have - // to emit more attributes than strictly (sic) necessary. - // However, these attributes with empty values are not useful to the user - // so we omit them. - hasContent = [&](eval_cache::AttrCursor & visitor, const AttrPath & attrPath, const Symbol & attr) -> bool { - auto attrPath2(attrPath); - attrPath2.push_back(attr); - auto attrPathS = attrPath2.resolve(*state); - const auto & attrName = state->symbols[attr]; - - auto visitor2 = visitor.getAttr(attrName); - - try { - if ((attrPathS[0] == "apps" || attrPathS[0] == "checks" || attrPathS[0] == "devShells" - || attrPathS[0] == "legacyPackages" || attrPathS[0] == "packages") - && (attrPathS.size() == 1 || attrPathS.size() == 2)) { - for (const auto & subAttr : visitor2->getAttrs()) { - if (hasContent(*visitor2, attrPath2, subAttr)) { - return true; - } - } - return false; - } + auto cache = flake_schemas::call(*state, flake, getDefaultFlakeSchemas()); - if ((attrPathS.size() == 1) - && (attrPathS[0] == "formatter" || attrPathS[0] == "nixosConfigurations" - || attrPathS[0] == "nixosModules" || attrPathS[0] == "overlays")) { - for (const auto & subAttr : visitor2->getAttrs()) { - if (hasContent(*visitor2, attrPath2, subAttr)) { - return true; - } - } - return false; - } + auto inv = flake_schemas::getFlakeInventory(*state, *getEvalStore(), *flake, cache, options); - // If we don't recognize it, it's probably content - return true; - } catch (EvalError & e) { - // Some attrs may contain errors, e.g. legacyPackages of - // nixpkgs. We still want to recurse into it, instead of - // skipping it at all. - return true; - } - }; + if (json) { + auto res = nlohmann::json{{"version", 2}, {"inventory", std::move(inv)}}; + printJSON(res); + } else { - std::function - visit; - - visit = [&](eval_cache::AttrCursor & visitor, - const AttrPath & attrPath, - const std::string & headerPrefix, - const std::string & nextPrefix) -> nlohmann::json { - auto j = nlohmann::json::object(); - - auto attrPathS = attrPath.resolve(*state); - - Activity act(*logger, lvlInfo, actUnknown, fmt("evaluating '%s'", attrPath.to_string(*state))); - - try { - auto recurse = [&]() { - if (!json) - logger->cout("%s", headerPrefix); - std::vector attrs; - for (const auto & attr : visitor.getAttrs()) { - if (hasContent(visitor, attrPath, attr)) - attrs.push_back(attr); - } + // Render the JSON into a tree representation. + std::function + render; - for (const auto & [last, attr] : markLast(attrs)) { - const auto & attrName = state->symbols[attr]; - auto visitor2 = visitor.getAttr(attrName); - auto attrPath2(attrPath); - attrPath2.push_back(attr); - auto j2 = visit( - *visitor2, - attrPath2, - fmt(ANSI_GREEN "%s%s" ANSI_NORMAL ANSI_BOLD "%s" ANSI_NORMAL, - nextPrefix, - last ? treeLast : treeConn, - attrName), - nextPrefix + (last ? treeNull : treeLine)); - if (json) - j.emplace(attrName, std::move(j2)); - } - }; + render = [&](nlohmann::json j, const std::string & headerPrefix, const std::string & nextPrefix) { + auto what = j.find("what"); + auto filtered = j.find("filtered"); + auto isLegacy = j.find("isLegacy"); + auto derivation = j.find("derivation"); - auto showDerivation = [&]() { - auto name = visitor.getAttr(state->s.name)->getString(); + auto s = headerPrefix; - if (json) { - std::optional description; - if (auto aMeta = visitor.maybeGetAttr(state->s.meta)) { - if (auto aDescription = aMeta->maybeGetAttr(state->s.description)) - description = aDescription->getString(); - } - j.emplace("type", "derivation"); - j.emplace("name", name); - j.emplace("description", description ? *description : ""); - } else { - logger->cout( - "%s: %s '%s'", - headerPrefix, - attrPath.size() == 2 && attrPathS[0] == "devShell" ? "development environment" - : attrPath.size() >= 2 && attrPathS[0] == "devShells" ? "development environment" - : attrPath.size() == 3 && attrPathS[0] == "checks" ? "derivation" - : attrPath.size() >= 1 && attrPathS[0] == "hydraJobs" ? "derivation" - : "package", - name); - } - }; - - if (attrPath.size() == 0 - || (attrPath.size() == 1 - && (attrPathS[0] == "defaultPackage" || attrPathS[0] == "devShell" - || attrPathS[0] == "formatter" || attrPathS[0] == "nixosConfigurations" - || attrPathS[0] == "nixosModules" || attrPathS[0] == "defaultApp" - || attrPathS[0] == "templates" || attrPathS[0] == "overlays")) - || ((attrPath.size() == 1 || attrPath.size() == 2) - && (attrPathS[0] == "checks" || attrPathS[0] == "packages" || attrPathS[0] == "devShells" - || attrPathS[0] == "apps"))) { - recurse(); - } + if (what != j.end()) + s += fmt(": %s", (std::string) *what); - else if ( - (attrPath.size() == 2 - && (attrPathS[0] == "defaultPackage" || attrPathS[0] == "devShell" || attrPathS[0] == "formatter")) - || (attrPath.size() == 3 - && (attrPathS[0] == "checks" || attrPathS[0] == "packages" || attrPathS[0] == "devShells"))) { - if (!showAllSystems && std::string(attrPathS[1]) != localSystem) { - if (!json) - logger->cout( - fmt("%s " ANSI_WARNING "omitted" ANSI_NORMAL " (use '--all-systems' to show)", - headerPrefix)); - else { - logger->warn(fmt("%s omitted (use '--all-systems' to show)", attrPath.to_string(*state))); - } - } else { - try { - if (visitor.isDerivation()) - showDerivation(); - else { - auto name = visitor.getAttrPathStr(state->s.name); - logger->warn(fmt("%s is not a derivation", name)); - } - } catch (IFDError & e) { - if (!json) { - logger->cout( - fmt("%s " ANSI_WARNING "omitted due to use of import from derivation" ANSI_NORMAL, - headerPrefix)); - } else { - logger->warn( - fmt("%s omitted due to use of import from derivation", attrPath.to_string(*state))); - } - } - } + if (derivation != j.end()) { + auto name = derivation->find("name"); + if (name != derivation->end()) + s += fmt(ANSI_ITALIC " [%s]" ANSI_NORMAL, (std::string) *name); } - else if (attrPath.size() > 0 && attrPathS[0] == "hydraJobs") { - try { - if (visitor.isDerivation()) - showDerivation(); - else - recurse(); - } catch (IFDError & e) { - if (!json) { - logger->cout( - fmt("%s " ANSI_WARNING "omitted due to use of import from derivation" ANSI_NORMAL, - headerPrefix)); - } else { - logger->warn( - fmt("%s omitted due to use of import from derivation", attrPath.to_string(*state))); - } - } - } + if (filtered != j.end() && (bool) *filtered) + s += " " ANSI_WARNING "omitted" ANSI_NORMAL " (use '--all-systems' to show)"; - else if (attrPath.size() > 0 && attrPathS[0] == "legacyPackages") { - if (attrPath.size() == 1) - recurse(); - else if (!showLegacy) { - if (!json) - logger->cout(fmt( - "%s " ANSI_WARNING "omitted" ANSI_NORMAL " (use '--legacy' to show)", headerPrefix)); - else { - logger->warn(fmt("%s omitted (use '--legacy' to show)", attrPath.to_string(*state))); - } - } else if (!showAllSystems && std::string(attrPathS[1]) != localSystem) { - if (!json) - logger->cout( - fmt("%s " ANSI_WARNING "omitted" ANSI_NORMAL " (use '--all-systems' to show)", - headerPrefix)); - else { - logger->warn(fmt("%s omitted (use '--all-systems' to show)", attrPath.to_string(*state))); - } - } else { - try { - if (visitor.isDerivation()) - showDerivation(); - else if (attrPath.size() <= 2) - // FIXME: handle recurseIntoAttrs - recurse(); - } catch (IFDError & e) { - if (!json) { - logger->cout( - fmt("%s " ANSI_WARNING "omitted due to use of import from derivation" ANSI_NORMAL, - headerPrefix)); - } else { - logger->warn( - fmt("%s omitted due to use of import from derivation", attrPath.to_string(*state))); - } - } - } - } + if (isLegacy != j.end() && (bool) *isLegacy) + s += " " ANSI_WARNING "omitted" ANSI_NORMAL " (use '--legacy' to show)"; - else if ( - (attrPath.size() == 2 && attrPathS[0] == "defaultApp") - || (attrPath.size() == 3 && attrPathS[0] == "apps")) { - auto aType = visitor.maybeGetAttr("type"); - std::optional description; - if (auto aMeta = visitor.maybeGetAttr(state->s.meta)) { - if (auto aDescription = aMeta->maybeGetAttr(state->s.description)) - description = aDescription->getString(); - } - if (!aType || aType->getString() != "app") - state->error("not an app definition").debugThrow(); - if (json) { - j.emplace("type", "app"); - if (description) - j.emplace("description", *description); - } else { - logger->cout( - "%s: app: " ANSI_BOLD "%s" ANSI_NORMAL, - headerPrefix, - description ? *description : "no description"); - } - } + logger->cout(s); - else if ( - (attrPath.size() == 1 && attrPathS[0] == "defaultTemplate") - || (attrPath.size() == 2 && attrPathS[0] == "templates")) { - auto description = visitor.getAttr("description")->getString(); - if (json) { - j.emplace("type", "template"); - j.emplace("description", description); - } else { - logger->cout("%s: template: " ANSI_BOLD "%s" ANSI_NORMAL, headerPrefix, description); - } - } + auto children = j.find("children"); - else { - auto [type, description] = (attrPath.size() == 1 && attrPathS[0] == "overlay") - || (attrPath.size() == 2 && attrPathS[0] == "overlays") - ? std::make_pair("nixpkgs-overlay", "Nixpkgs overlay") - : attrPath.size() == 2 && attrPathS[0] == "nixosConfigurations" - ? std::make_pair("nixos-configuration", "NixOS configuration") - : (attrPath.size() == 1 && attrPathS[0] == "nixosModule") - || (attrPath.size() == 2 && attrPathS[0] == "nixosModules") - ? std::make_pair("nixos-module", "NixOS module") - : std::make_pair("unknown", "unknown"); - if (json) { - j.emplace("type", type); - } else { - logger->cout("%s: " ANSI_WARNING "%s" ANSI_NORMAL, headerPrefix, description); + if (children != j.end()) { + for (const auto & [i, child] : enumerate(children->items())) { + bool last = i + 1 == children->size(); + render( + child.value(), + fmt(ANSI_GREEN "%s%s" ANSI_NORMAL ANSI_BOLD "%s" ANSI_NORMAL, + nextPrefix, + last ? treeLast : treeConn, + child.key()), + nextPrefix + (last ? treeNull : treeLine)); } } - } catch (EvalError & e) { - if (!(attrPath.size() > 0 && attrPathS[0] == "legacyPackages")) - throw; - } - - return j; - }; - - auto cache = openEvalCache(*state, ref(flake)); + }; - auto j = visit(*cache->getRoot(), {}, fmt(ANSI_BOLD "%s" ANSI_NORMAL, flake->flake.lockedRef), ""); - if (json) - printJSON(j); + logger->cout("%s", fmt(ANSI_BOLD "%s" ANSI_NORMAL, flake->flake.lockedRef)); + + for (const auto & [i, child] : enumerate(inv.items())) { + bool last = i + 1 == inv.size(); + auto nextPrefix = last ? treeNull : treeLine; + auto output = child.value().find("output"); + auto headerPrefix = fmt( + ANSI_GREEN "%s" ANSI_NORMAL ANSI_BOLD "%s" ANSI_NORMAL, last ? treeLast : treeConn, child.key()); + if (output != child.value().end()) + render(*output, headerPrefix, nextPrefix); + else if (child.value().contains("unknown")) + logger->cout(headerPrefix + ANSI_WARNING " unknown flake output" ANSI_NORMAL); + } + } } }; @@ -1525,9 +1042,7 @@ struct CmdFlakePrefetch : FlakeCommand, MixJSON { auto originalRef = getFlakeRef(); auto resolvedRef = originalRef.resolve(fetchSettings, *store); - auto [accessor, lockedRef] = resolvedRef.lazyFetch(getEvalState()->fetchSettings, *store); - auto storePath = - fetchToStore(getEvalState()->fetchSettings, *store, accessor, FetchMode::Copy, lockedRef.input.getName()); + auto [storePath, accessor, lockedRef] = resolvedRef.input.fetchToStore(fetchSettings, *store); auto hash = store->queryPathInfo(storePath)->narHash; if (json) { @@ -1536,7 +1051,6 @@ struct CmdFlakePrefetch : FlakeCommand, MixJSON res["hash"] = hash.to_string(HashFormat::SRI, true); res["original"] = fetchers::attrsToJSON(resolvedRef.toAttrs()); res["locked"] = fetchers::attrsToJSON(lockedRef.toAttrs()); - res["locked"].erase("__final"); // internal for now printJSON(res); } else { notice( @@ -1573,12 +1087,6 @@ struct CmdFlake : NixMultiCommand #include "flake.md" ; } - - void run() override - { - experimentalFeatureSettings.require(Xp::Flakes); - NixMultiCommand::run(); - } }; static auto rCmdFlake = registerCommand("flake"); diff --git a/src/nix/formatter.cc b/src/nix/formatter.cc index 19d473863570..2fd4c472f1cc 100644 --- a/src/nix/formatter.cc +++ b/src/nix/formatter.cc @@ -32,14 +32,9 @@ static auto rCmdFormatter = registerCommand("formatter"); /** Common implementation bits for the `nix formatter` subcommands. */ struct MixFormatter : SourceExprCommand { - Strings getDefaultFlakeAttrPaths() override + StringSet getRoles() override { - return Strings{"formatter." + settings.thisSystem.get()}; - } - - Strings getDefaultFlakeAttrPathPrefixes() override - { - return Strings{}; + return {"nix-fmt"}; } }; diff --git a/src/nix/get-env.sh b/src/nix/get-env.sh index 39fa6f9ac8f5..5b6162b4ba26 100644 --- a/src/nix/get-env.sh +++ b/src/nix/get-env.sh @@ -17,6 +17,7 @@ __functions="$(declare -F)" __dumpEnv() { printf '{\n' + printf ' "version": 1,\n' printf ' "bashFunctions": {\n' local __first=1 diff --git a/src/nix/key-convert-public-to-pem.md b/src/nix/key-convert-public-to-pem.md new file mode 100644 index 000000000000..010886341aba --- /dev/null +++ b/src/nix/key-convert-public-to-pem.md @@ -0,0 +1,29 @@ +R""( + +# Examples + +* Convert a Nix public key to PEM: + + ```console + # nix key convert-secret-to-public < secret-key \ + | nix key convert-public-to-pem + -----BEGIN PUBLIC KEY----- + … + -----END PUBLIC KEY----- + ``` + +* Convert a public key to PEM and decode it using OpenSSL: + + ```console + # nix key convert-public-to-pem < public-key \ + | openssl pkey -pubin -text -noout + ML-DSA-87 Public-Key: + pub: + … + ``` + +# Description + +This command reads a Nix public verification key (as produced by `nix key convert-secret-to-public`) from standard input and writes the corresponding PEM `SubjectPublicKeyInfo` to standard output. The key name is not included in the PEM output. + +)"" diff --git a/src/nix/key-convert-secret-to-pem.md b/src/nix/key-convert-secret-to-pem.md new file mode 100644 index 000000000000..f1ec2ca32491 --- /dev/null +++ b/src/nix/key-convert-secret-to-pem.md @@ -0,0 +1,29 @@ +R""( + +# Examples + +* Convert an ML-DSA-65 secret key to PEM: + + ```console + # nix key generate-secret --key-name cache.example.org-1 --key-type ml-dsa-65 \ + | nix key convert-secret-to-pem + -----BEGIN PRIVATE KEY----- + … + -----END PRIVATE KEY----- + ``` + +* Convert a secret key to PEM and decode it using OpenSSL: + + ```console + # nix key convert-secret-to-pem < secret-key \ + | openssl pkey -text -noout + ML-DSA-87 Private-Key: + seed: + … + ``` + +# Description + +This command reads a Nix signing key generated by `nix key generate-secret` from standard input and writes the corresponding PEM PKCS#8 private key to standard output. The key name is not included in the PEM output. + +)"" diff --git a/src/nix/key-generate-secret.md b/src/nix/key-generate-secret.md index 2b8459da592f..bedb356257f2 100644 --- a/src/nix/key-generate-secret.md +++ b/src/nix/key-generate-secret.md @@ -24,7 +24,7 @@ R""( # Description -This command generates a new Ed25519 secret key for signing store +This command generates a new secret key for signing store paths and prints it on standard output. Use `nix key convert-secret-to-public` to get the corresponding public key for verifying signed store paths. @@ -36,10 +36,16 @@ the host name of your cache (e.g. `cache.example.org`) with a suffix denoting the number of the key (to be incremented every time you need to revoke a key). +Nix supports keys in the following formats (specified using the `--key-type` option): + +* `ed25519` (libsodium). This is the default key type. It produces compact keys and signatures, but may not be resistant to attacks using quantum computers. +* `ml-dsa-44`, `ml-dsa-65`, `ml-dsa-87` (OpenSSL). These generate much larger keys and signatures, but are believed to be resistant to quantum attacks. +* `ecdsa-p384` (OpenSSL). ECDSA over the NIST P-384 curve using SHA-384. This is a CNSA-approved classical algorithm but is not believed to be resistant to quantum attacks. + # Format Both secret and public keys are represented as the key name followed -by a [Base64](https://en.wikipedia.org/wiki/Base64) encoding of the Ed25519 key data, e.g. +by a [Base64](https://en.wikipedia.org/wiki/Base64) encoding of the key data, e.g. ``` cache.example.org-0:E7lAO+MsPwTFfPXsdPtW8GKui/5ho4KQHVcAGnX+Tti1V4dUxoVoqLyWJ4YESuZJwQ67GVIksDt47og+tPVUZw== diff --git a/src/nix/ls.cc b/src/nix/ls.cc index 410f4f420e05..efcd4d7341d2 100644 --- a/src/nix/ls.cc +++ b/src/nix/ls.cc @@ -4,12 +4,13 @@ #include "nix/main/common-args.hh" #include +#include "ls.hh" + namespace nix { -struct MixLs : virtual Args, MixJSON +struct MixLs : virtual Args, MixJSON, MixLongListing { bool recursive = false; - bool verbose = false; bool showDirectory = false; MixLs() @@ -21,13 +22,6 @@ struct MixLs : virtual Args, MixJSON .handler = {&recursive, true}, }); - addFlag({ - .longName = "long", - .shortName = 'l', - .description = "Show detailed file information.", - .handler = {&verbose, true}, - }); - addFlag({ .longName = "directory", .shortName = 'd', @@ -41,13 +35,13 @@ struct MixLs : virtual Args, MixJSON std::function doPath; auto showFile = [&](const CanonPath & curPath, std::string_view relPath) { - if (verbose) { + if (longListing) { auto st = accessor->lstat(curPath); std::string tp = st.type == SourceAccessor::Type::tRegular ? (st.isExecutable ? "-r-xr-xr-x" : "-r--r--r--") : st.type == SourceAccessor::Type::tSymlink ? "lrwxrwxrwx" : "dr-xr-xr-x"; - auto line = fmt("%s %20d %s", tp, st.fileSize.value_or(0), relPath); + auto line = fmt("%s %9d %s", tp, st.fileSize.value_or(0), relPath); if (st.type == SourceAccessor::Type::tSymlink) line += " -> " + accessor->readLink(curPath); logger->cout(line); diff --git a/src/nix/ls.hh b/src/nix/ls.hh new file mode 100644 index 000000000000..36e61162035f --- /dev/null +++ b/src/nix/ls.hh @@ -0,0 +1,22 @@ +#pragma once + +#include "nix/util/args.hh" + +namespace nix { + +struct MixLongListing : virtual Args +{ + bool longListing = false; + + MixLongListing() + { + addFlag({ + .longName = "long", + .shortName = 'l', + .description = "Show detailed file information.", + .handler = {&longListing, true}, + }); + } +}; + +} // namespace nix diff --git a/src/nix/main.cc b/src/nix/main.cc index 963bc07ce274..a47ee44647ee 100644 --- a/src/nix/main.cc +++ b/src/nix/main.cc @@ -13,6 +13,7 @@ #include "nix/store/store-registration.hh" #include "nix/store/filetransfer.hh" #include "nix/util/finally.hh" +#include "nix/util/json-utils.hh" #include "nix/main/loggers.hh" #include "nix/cmd/markdown.hh" #include "nix/util/memory-source-accessor.hh" @@ -22,19 +23,30 @@ #include "nix/expr/eval-cache.hh" #include "nix/flake/flake.hh" #include "nix/flake/settings.hh" +#include "nix/util/exit.hh" +#include "nix/util/sentry.hh" #include "self-exe.hh" #include "crash-handler.hh" +#include "otel-logger.hh" #include "cli-config-private.hh" #include #include +#if HAVE_SENTRY +# include +#endif + +#if HAVE_MIMALLOC +# include +#endif #ifndef _WIN32 # include # include # include # include +# include #endif #ifdef __linux__ @@ -86,6 +98,22 @@ static bool haveInternet() #endif } +static void disableNet() +{ + // FIXME: should check for command line overrides only. + if (!settings.getWorkerSettings().useSubstitutes.overridden) + // FIXME: should not disable local substituters (like file:///). + settings.getWorkerSettings().useSubstitutes = false; + if (!fetchSettings.tarballTtl.overridden) + fetchSettings.tarballTtl = std::numeric_limits::max(); + if (!settings.getNarInfoDiskCacheSettings().ttlMeta.overridden) + settings.getNarInfoDiskCacheSettings().ttlMeta = std::numeric_limits::max(); + if (!fileTransferSettings.tries.overridden) + fileTransferSettings.tries = 0; + if (!fileTransferSettings.connectTimeout.overridden) + fileTransferSettings.connectTimeout = 1; +} + std::string programPath; struct NixArgs : virtual MultiCommand, virtual MixCommonArgs, virtual RootArgs @@ -102,6 +130,7 @@ struct NixArgs : virtual MultiCommand, virtual MixCommonArgs, virtual RootArgs categories.clear(); categories[catHelp] = "Help commands"; categories[Command::catDefault] = "Main commands"; + categories[Command::catUndocumented] = "Undocumented commands"; categories[catSecondary] = "Infrequently used commands"; categories[catUtility] = "Utility/scripting commands"; categories[catNixInstallation] = "Commands for upgrading or troubleshooting your Nix installation"; @@ -119,7 +148,6 @@ struct NixArgs : virtual MultiCommand, virtual MixCommonArgs, virtual RootArgs .description = "Print full build logs on standard error.", .category = loggingCategory, .handler = {[&]() { logger->setPrintBuildLogs(true); }}, - .experimentalFeature = Xp::NixCommand, }); addFlag({ @@ -135,7 +163,6 @@ struct NixArgs : virtual MultiCommand, virtual MixCommonArgs, virtual RootArgs .description = "Disable substituters and consider all previously downloaded files up-to-date.", .category = miscCategory, .handler = {[&]() { useNet = false; }}, - .experimentalFeature = Xp::NixCommand, }); addFlag({ @@ -143,7 +170,6 @@ struct NixArgs : virtual MultiCommand, virtual MixCommonArgs, virtual RootArgs .description = "Consider all previously downloaded files out-of-date.", .category = miscCategory, .handler = {[&]() { refresh = true; }}, - .experimentalFeature = Xp::NixCommand, }); aliases = { @@ -366,14 +392,37 @@ struct CmdHelpStores : Command static auto rCmdHelpStores = registerCommand("help-stores"); +static void terminateHandler() +{ + // Add the exception type and message to the Sentry crash report. + auto ex = std::current_exception(); + if (ex) { + try { + std::rethrow_exception(ex); + } catch (BaseError & e) { + try { + nix::setSentryTag("terminate_exception_type", typeid(e).name()); + nix::setSentryTag("terminate_exception_msg", e.message().c_str()); + } catch (...) { + } + } catch (const std::exception & e) { + try { + nix::setSentryTag("terminate_exception_type", typeid(e).name()); + nix::setSentryTag("terminate_exception_msg", e.what()); + } catch (...) { + } + } + } + + onTerminate(); +} + void mainWrapped(int argc, char ** argv) { savedArgv = argv; - registerCrashHandler(); - /* The chroot helper needs to be run before any threads have been - started. */ + started (including Sentry's worker thread). */ #ifndef _WIN32 if (argc > 0 && argv[0] == chrootHelperName) { chrootHelper(argc, argv); @@ -381,6 +430,32 @@ void mainWrapped(int argc, char ** argv) } #endif +#if HAVE_MIMALLOC + /* Make allocation failures print a proper error message instead + of aborting silently. Note that the `std::new_handler` installed + by `initLibUtil()` does not suffice, since mimalloc's `operator + new` override never sees it: libmimalloc is linked with + `-Bsymbolic`, so its weak null stub of `std::get_new_handler()` + shadows the real one from libstdc++. Register this before + `initLibUtil()` so allocation failures during initialization + are handled as well. */ + mi_register_error( + [](int err, void *) { + if (err == ENOMEM || err == EOVERFLOW) + outOfMemory(); + /* EFAULT means mimalloc detected heap corruption (e.g. a + double free) in secure mode; don't continue with a + corrupted heap. EINVAL (bad pointer passed to + `mi_free()`) is non-fatal, so just return. */ + if (err == EFAULT) + panic("mimalloc detected heap corruption"); + }, + nullptr); +#endif + + /* This must be called before Sentry since both initialize OpenSSL. */ + initLibUtil(); + /* Set the build hook location For builds we perform a self-invocation, so Nix has to be @@ -394,6 +469,64 @@ void mainWrapped(int argc, char ** argv) }); initNix(); + + /* Initialize Sentry only after initNix(), i.e. after + startSignalHandlerThread() has blocked SIGINT etc. in the + calling thread. The worker threads spawned by `sentry_init()` + inherit that signal mask; if they were started earlier, the + kernel could deliver a Ctrl-C's SIGINT to one of them, where its + default disposition would kill the process immediately (without + printing an error or restoring the terminal cursor). */ + bool sentryEnabled = false; + +#if HAVE_SENTRY + auto sentryEndpoint = getEnv("NIX_SENTRY_ENDPOINT"); + + if (!sentryEndpoint && getEnv("DETSYS_IDS_TELEMETRY") != "disabled") { + try { + auto p = nixConfDir() / "sentry-endpoint"; + if (pathExists(p)) + sentryEndpoint = trim(readFile(p)); + } catch (...) { + ignoreExceptionExceptInterrupt(); + } + } + + if (sentryEndpoint && sentryEndpoint != "") { + sentry_options_t * options = sentry_options_new(); + sentry_options_set_dsn(options, sentryEndpoint->c_str()); + sentry_options_set_database_path(options, (getCacheDir() / "sentry").string().c_str()); + sentry_options_set_release(options, fmt("nix@%s", determinateNixVersion).c_str()); + sentry_options_set_traces_sample_rate(options, 0); + sentry_options_set_auto_session_tracking(options, false); + sentry_options_set_handler_path(options, CRASHPAD_HANDLER_PATH); + sentry_init(options); + setSentryTag = [](const char * key, const char * value) { sentry_set_tag(key, value); }; + setSentryTag("nix_command", argc > 0 ? std::string(baseNameOf(argv[0])).c_str() : ""); + std::set_terminate(terminateHandler); + sentryEnabled = true; + + /* Reset SIGQUIT, for which `sentry_init()` installed a crash + handler, to its default disposition: SIGQUIT is a + user-initiated "quit with core dump" action (e.g. Ctrl-\ at + a terminal), not a crash, so it should not be reported. */ + struct sigaction act; + sigemptyset(&act.sa_mask); + act.sa_flags = 0; + act.sa_handler = SIG_DFL; + if (sigaction(SIGQUIT, &act, 0)) + throw SysError("handling SIGQUIT"); + } + + Finally cleanupSentry([&]() { + if (sentryEnabled) + sentry_shutdown(); + }); +#endif + + if (!sentryEnabled) + registerCrashHandler(); + initGC(); flakeSettings.configureEvalSettings(evalSettings); @@ -402,8 +535,6 @@ void mainWrapped(int argc, char ** argv) tryEnterPrivateMountNamespace(); #endif - Finally f([] { logger->stop(); }); - programPath = argv[0]; auto programName = std::string(baseNameOf(programPath)); auto extensionPos = programName.find_last_of("."); @@ -417,8 +548,18 @@ void mainWrapped(int argc, char ** argv) } { - if (auto legacy = get(RegisterLegacyCommand::commands(), programName)) + if (auto legacy = get(RegisterLegacyCommand::commands(), programName)) { + /* Legacy commands don't have subcommands, so we can set up + the root span right away. Note that they parse their + arguments themselves, so unlike for the commands below, + `--option` cannot configure tracing here. The daemon and + the build hook are the exceptions: they install their + own logger, so they set up tracing themselves. */ + if (programName != "nix-daemon" && programName != "build-remote") { + initOtel(programName, programName, getEnv("TRACEPARENT").value_or("")); + } return (*legacy)(argc, argv); + } } evalSettings.pureEval = true; @@ -444,10 +585,11 @@ void mainWrapped(int argc, char ** argv) if (argc == 2 && std::string(argv[1]) == "__dump-language") { experimentalFeatureSettings.experimentalFeatures = { - Xp::Flakes, Xp::FetchClosure, Xp::DynamicDerivations, Xp::FetchTree, + Xp::BakedDerivations, + Xp::Provenance, }; evalSettings.pureEval = false; auto statePtr = std::make_shared( @@ -479,6 +621,8 @@ void mainWrapped(int argc, char ** argv) b["type"] = showType(info.type, false); if (info.impureOnly) b["impure-only"] = true; + if (info.experimentalFeature) + b["experimental-feature"] = info.experimentalFeature; builtinsJson[name] = std::move(b); } logger->cout("%s", builtinsJson); @@ -491,23 +635,36 @@ void mainWrapped(int argc, char ** argv) } Finally printCompletions([&]() { - if (args.completions) { - switch (args.completions->type) { - case Completions::Type::Normal: - logger->cout("normal"); - break; - case Completions::Type::Filenames: - logger->cout("filenames"); - break; - case Completions::Type::Attrs: - logger->cout("attrs"); - break; + /* Don't let a failure to print completions (e.g. EPIPE because + the shell closed its end of the pipe) propagate, since this + may run while another exception is being unwound. */ + try { + if (args.completions) { + switch (args.completions->type) { + case Completions::Type::Normal: + logger->cout("normal"); + break; + case Completions::Type::Filenames: + logger->cout("filenames"); + break; + case Completions::Type::Attrs: + logger->cout("attrs"); + break; + } + for (auto & s : args.completions->completions) + logger->cout(s.completion + "\t" + trim(s.description)); } - for (auto & s : args.completions->completions) - logger->cout(s.completion + "\t" + trim(s.description)); + } catch (...) { + ignoreExceptionInDestructor(); } }); + if (getEnv("NIX_GET_COMPLETIONS")) + /* Avoid fetching stuff during tab completion. We have to this + early because we haven't checked `haveInternet()` yet + (below). */ + disableNet(); + try { auto isNixCommand = programName.ends_with("nix"); auto allowShebang = isNixCommand && argc > 1; @@ -519,16 +676,19 @@ void mainWrapped(int argc, char ** argv) applyJSONLogger(); + printTalkative("Nix %s", version()); + + std::vector subcommand; + MultiCommand * command = &args; + while (command) { + if (command && command->command) { + subcommand.push_back(command->command->first); + command = dynamic_cast(&*command->command->second); + } else + break; + } + if (args.helpRequested) { - std::vector subcommand; - MultiCommand * command = &args; - while (command) { - if (command && command->command) { - subcommand.push_back(command->command->first); - command = dynamic_cast(&*command->command->second); - } else - break; - } showHelp(subcommand, args); return; } @@ -544,27 +704,21 @@ void mainWrapped(int argc, char ** argv) if (!args.command) throw UsageError("no subcommand specified"); - experimentalFeatureSettings.require(args.command->second->experimentalFeature()); + { + MultiCommand * command = &args; + while (command && command->command) { + experimentalFeatureSettings.require(command->command->second->experimentalFeature()); + command = dynamic_cast(&*command->command->second); + } + } if (args.useNet && !haveInternet()) { warn("you don't have Internet access; disabling some network-dependent features"); args.useNet = false; } - if (!args.useNet) { - // FIXME: should check for command line overrides only. - if (!settings.getWorkerSettings().useSubstitutes.overridden) - settings.getWorkerSettings().useSubstitutes = false; - if (!fetchSettings.tarballTtl.overridden) - fetchSettings.tarballTtl = std::numeric_limits::max(); - if (!fileTransferSettings.tries.overridden) - fileTransferSettings.tries = 0; - if (!fileTransferSettings.connectTimeout.overridden) - fileTransferSettings.connectTimeout = 1; - auto & ttlMeta = settings.getNarInfoDiskCacheSettings().ttlMeta; - if (!ttlMeta.overridden) - ttlMeta = std::numeric_limits::max(); - } + if (!args.useNet) + disableNet(); if (args.refresh) { fetchSettings.tarballTtl = 0; @@ -577,6 +731,21 @@ void mainWrapped(int argc, char ** argv) evalSettings.pureEval = false; } + setSentryTag("nix_subcommand", concatStringsSep(" ", subcommand).c_str()); + + /* Map activities to OpenTelemetry spans, under a root span named + after the subcommand. Note: this must happen after + `parseCmdline()`, so that `--option` can configure tracing. The + daemon is the exception: it sets up tracing itself, per + connection, parented to the client's trace. + + The root span is parented to the trace context in `TRACEPARENT`, + if any, so that a parent process (such as `nix` running + `build-remote`) can include us in its trace. */ + if (subcommand != std::vector{"daemon"}) { + initOtel(programName, "nix " + concatStringsSep(" ", subcommand), getEnv("TRACEPARENT").value_or("")); + } + try { args.command->second->run(); } catch (eval_cache::CachedEvalError & e) { @@ -591,7 +760,10 @@ void mainWrapped(int argc, char ** argv) int main(int argc, char ** argv) { + using namespace nix; + // The CLI has a more detailed version than the libraries; see nixVersion. nix::nixVersion = NIX_CLI_VERSION; + return nix::handleExceptions(argv[0], [&]() { nix::mainWrapped(argc, argv); }); } diff --git a/src/nix/make.nix b/src/nix/make.nix new file mode 100644 index 000000000000..db409bf4cea2 --- /dev/null +++ b/src/nix/make.nix @@ -0,0 +1,91 @@ +# The `nix` executable, transcribed from meson.build. +{ + pkgs, + nixMake, + nix-util, + nix-store, + nix-fetchers, + nix-expr, + nix-flake, + nix-main, + nix-cmd, +}: + +let + inherit (pkgs) lib; + # The overridden dependencies used by the Meson build, layered over Nixpkgs. + nixDeps = pkgs // pkgs.nixDependencies2; +in + +nixMake.mkComponent { + name = "determinate-nix"; + type = "executable"; + exeName = "nix"; + + deps = [ + nix-util + nix-store + nix-fetchers + nix-expr + nix-flake + nix-main + nix-cmd + ]; + + root = ./.; + + # Every .cc file under the root (including the legacy commands in + # subdirectories and unix/) is a compilation unit. The `doc`, `misc` + # and `scripts` symlinks are ignored by the scanner. + includeDirs = [ "" ]; + + files = nixMake.commonSupportFiles // { + "generate-manpage.nix.gen.hh" = nixMake.mkStringHeader ../../doc/manual/generate-manpage.nix; + "generate-settings.nix.gen.hh" = nixMake.mkStringHeader ../../doc/manual/generate-settings.nix; + "generate-store-info.nix.gen.hh" = nixMake.mkStringHeader ../../doc/manual/generate-store-info.nix; + "utils.nix.gen.hh" = nixMake.mkStringHeader ../../doc/manual/utils.nix; + "get-env.sh.gen.hh" = nixMake.mkStringHeader ./get-env.sh; + "help-stores.md.gen.hh" = nixMake.mkStringHeader ../../doc/manual/source/store/types/index.md.in; + "profiles.md.gen.hh" = nixMake.mkStringHeader ../../doc/manual/source/command-ref/files/profiles.md; + "unpack-channel.nix.gen.hh" = nixMake.mkStringHeader ./nix-channel/unpack-channel.nix; + "buildenv.nix.gen.hh" = nixMake.mkStringHeader ./nix-env/buildenv.nix; + "baked-flake.nix.gen.hh" = nixMake.mkStringHeader ./baked-flake.nix; + }; + + configHeaders = { + "cli-config-private.hh" = { + NIX_CLI_VERSION = lib.fileContents ../../.version; + # Fallbacks only: Nix normally locates itself via /proc/self/exe. + NIX_BIN_DIR = "/nix/var/nix/profiles/default/bin"; + NIX_MAN_DIR = "/nix/var/nix/profiles/default/share/man"; + HAVE_MIMALLOC = 1; + HAVE_SENTRY = nixMake.config.sentry; + HAVE_OTEL = nixMake.config.otel; + } + // lib.optionalAttrs nixMake.config.sentry { + CRASHPAD_HANDLER_PATH = "${nixDeps.sentry-native}/bin/crashpad_handler"; + }; + }; + + linkFlags = [ "-Wl,--wrap=__assert_fail" ]; + + binSymlinks = [ + "nix-build" + "nix-channel" + "nix-collect-garbage" + "nix-copy-closure" + "nix-daemon" + "nix-env" + "nix-hash" + "nix-instantiate" + "nix-prefetch-url" + "nix-shell" + "nix-store" + ]; + + # A Python snippet, with `out` bound. + postInstall = '' + os.makedirs(f"{out}/libexec/nix") + os.symlink("../../bin/nix", f"{out}/libexec/nix/build-remote") + ''; +} diff --git a/src/nix/man-pages.cc b/src/nix/man-pages.cc index 9b01307a13c9..0641c89e46ee 100644 --- a/src/nix/man-pages.cc +++ b/src/nix/man-pages.cc @@ -3,6 +3,7 @@ #include "nix/util/file-system.hh" #include "nix/util/current-process.hh" #include "nix/util/environment-variables.hh" +#include "nix/util/logging.hh" namespace nix { @@ -15,6 +16,9 @@ void showManPage(const std::string & name) { restoreProcessContext(); setEnv("MANPATH", (getNixManDir().string() + ":").c_str()); + /* We're about to exec, so end and export any telemetry. */ + logger->stop(); + logger->flush(); execlp("man", "man", name.c_str(), nullptr); if (errno == ENOENT) { // Not SysError because we don't want to suffix the errno, aka No such file or directory. diff --git a/src/nix/meson.build b/src/nix/meson.build index 346f7e4a5c85..01601fb1b252 100644 --- a/src/nix/meson.build +++ b/src/nix/meson.build @@ -50,6 +50,9 @@ endif subdir('nix-meson-build-support/export-all-symbols') subdir('nix-meson-build-support/windows-version') +libmicrohttpd = dependency('libmicrohttpd') +deps_private += libmicrohttpd + configdata = configuration_data() # The CLI has a more detailed version string than the libraries; see `nixVersion` @@ -66,6 +69,26 @@ mandir = get_option('mandir') mandir = fs.is_absolute(mandir) ? mandir : prefix / mandir configdata.set_quoted('NIX_MAN_DIR', mandir) +configdata.set('HAVE_MIMALLOC', mimalloc.found().to_int()) + +sentry_required = get_option('sentry') +configdata.set('HAVE_SENTRY', sentry_required.enabled().to_int()) +if sentry_required.enabled() + crashpad_handler_path = get_option('crashpad-handler') + assert( + crashpad_handler_path != '', + 'crashpad-handler path must be set when sentry is enabled', + ) + configdata.set_quoted('CRASHPAD_HANDLER_PATH', crashpad_handler_path) +endif + +otel_required = get_option('otel') +# Note: we don't need the OTLP exporter from opentelemetry-cpp (which +# would pull in protobuf); we serialize spans to OTLP/JSON ourselves. +otel_trace = dependency('opentelemetry_trace', required : otel_required) +configdata.set('HAVE_OTEL', otel_trace.found().to_int()) +deps_private += otel_trace + config_priv_h = configure_file( configuration : configdata, output : 'cli-config-private.hh', @@ -85,6 +108,7 @@ nix_sources = [ config_priv_h ] + files( 'config.cc', 'copy.cc', 'crash-handler.cc', + 'crash.cc', 'derivation-add.cc', 'derivation-show.cc', 'derivation.cc', @@ -94,6 +118,8 @@ nix_sources = [ config_priv_h ] + files( 'edit.cc', 'env.cc', 'eval.cc', + 'flake-bake.cc', + 'flake-prefetch-inputs.cc', 'flake-prefetch-inputs.cc', 'flake.cc', 'formatter.cc', @@ -104,16 +130,21 @@ nix_sources = [ config_priv_h ] + files( 'make-content-addressed.cc', 'man-pages.cc', 'nar.cc', + 'nario.cc', 'optimise-store.cc', + 'otel-logger.cc', 'path-from-hash-part.cc', 'path-info.cc', 'prefetch.cc', 'profile.cc', + 'provenance.cc', + 'ps.cc', 'registry.cc', 'repl.cc', 'run.cc', 'search.cc', 'self-exe.cc', + 'serve.cc', 'sigs.cc', 'store-copy-log.cc', 'store-delete.cc', @@ -141,6 +172,7 @@ nix_sources += [ gen_header.process('get-env.sh'), gen_header.process('profiles.md'), gen_header.process('help-stores.md'), + gen_header.process('baked-flake.nix'), ] # The rest of the subdirectories aren't separate components, @@ -203,7 +235,9 @@ this_exe = executable( sources, dependencies : deps_private_subproject + deps_private + deps_other + deps_plugin_c_apis, include_directories : include_dirs, - link_args : linker_export_flags, + link_args : linker_export_flags + ( + sentry_required.enabled() ? [ '-lsentry' ] : [] + ), export_dynamic : get_option('plugin-c-api'), install : true, cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [], diff --git a/src/nix/meson.options b/src/nix/meson.options index 76c940d299cc..eefdddbce346 100644 --- a/src/nix/meson.options +++ b/src/nix/meson.options @@ -21,3 +21,23 @@ option( value : false, description : 'Embed the public C API into nix so plugins can resolve those symbols from the executable', ) + +option( + 'sentry', + type : 'feature', + value : 'auto', + description : 'Enable Sentry crash reporting', +) + +option( + 'otel', + type : 'feature', + description : 'distributed tracing via OpenTelemetry (OTLP/HTTP exporter)', +) + +option( + 'crashpad-handler', + type : 'string', + value : '', + description : 'Path to the crashpad_handler binary (required when sentry is enabled)', +) diff --git a/src/nix/nario-export.md b/src/nix/nario-export.md new file mode 100644 index 000000000000..2480733c1cae --- /dev/null +++ b/src/nix/nario-export.md @@ -0,0 +1,29 @@ +R""( + +# Examples + +* Export the closure of the build of `nixpkgs#hello`: + + ```console + # nix nario export --format 2 -r nixpkgs#hello > dump.nario + ``` + + It can be imported into another store: + + ```console + # nix nario import --no-check-sigs < dump.nario + ``` + +# Description + +This command prints to standard output a serialization of the specified store paths in `nario` format. This serialization can be imported into another store using `nix nario import`. + +References of a path are not exported by default; use `-r` to export a complete closure. +Paths are exported in topologically sorted order (i.e. if path `X` refers to `Y`, then `Y` appears before `X`). +You must specify the desired `nario` version. Currently the following versions are supported: + +* `1`: This version is compatible with the legacy `nix-store --export` and `nix-store --import` commands. It should be avoided because it is not memory-efficient on import. It does not support signatures, so you have to use `--no-check-sigs` on import. + +* `2`: The latest version. Recommended. + +)"" diff --git a/src/nix/nario-import.md b/src/nix/nario-import.md new file mode 100644 index 000000000000..9cba60c62203 --- /dev/null +++ b/src/nix/nario-import.md @@ -0,0 +1,15 @@ +R""( + +# Examples + +* Import store paths from the file named `dump`: + + ```console + # nix nario import < dump.nario + ``` + +# Description + +This command reads from standard input a serialization of store paths produced by `nix nario export` and adds them to the Nix store. + +)"" diff --git a/src/nix/nario-list.md b/src/nix/nario-list.md new file mode 100644 index 000000000000..c050457b3657 --- /dev/null +++ b/src/nix/nario-list.md @@ -0,0 +1,43 @@ +R""( + +# Examples + +* List the contents of a nario file: + + ```console + # nix nario list < dump.nario + /nix/store/f671jqvjcz37fsprzqn5jjsmyjj69p9b-xgcc-14.2.1.20250322-libgcc: 201856 bytes + /nix/store/n7iwblclbrz20xinvy4cxrvippdhvqll-libunistring-1.3: 2070240 bytes + … + ``` + +* Use `--json` to get detailed information in JSON format: + + ```console + # nix nario list --json < dump.nario + { + "paths": { + "/nix/store/m1r53pnn…-hello-2.12.1": { + "ca": null, + "deriver": "/nix/store/qa8is0vm…-hello-2.12.1.drv", + "narHash": "sha256-KSCYs4J7tFa+oX7W5M4D7ZYNvrWtdcWTdTL5fQk+za8=", + "narSize": 234672, + "references": [ + "/nix/store/g8zyryr9…-glibc-2.40-66", + "/nix/store/m1r53pnn…-hello-2.12.1" + ], + "registrationTime": 1756900709, + "signatures": [ "cache.nixos.org-1:QbG7A…" ], + "ultimate": false + }, + … + }, + "version": 1 + } + ``` + +# Description + +This command lists the contents of a nario file read from standard input. + +)"" diff --git a/src/nix/nario.cc b/src/nix/nario.cc new file mode 100644 index 000000000000..39b18d20403c --- /dev/null +++ b/src/nix/nario.cc @@ -0,0 +1,350 @@ +#include "nix/cmd/command.hh" +#include "nix/main/shared.hh" +#include "nix/store/store-api.hh" +#include "nix/store/export-import.hh" +#include "nix/util/callback.hh" +#include "nix/util/fs-sink.hh" +#include "nix/util/archive.hh" + +#include "ls.hh" + +#include + +using namespace nix; + +struct CmdNario : NixMultiCommand +{ + CmdNario() + : NixMultiCommand("nario", RegisterCommand::getCommandsFor({"nario"})) + { + } + + std::string description() override + { + return "operations for manipulating nario files"; + } + + Category category() override + { + return catUtility; + } +}; + +static auto rCmdNario = registerCommand("nario"); + +struct CmdNarioExport : StorePathsCommand +{ + unsigned int version = 0; + + CmdNarioExport() + { + addFlag({ + .longName = "format", + .description = "Version of the nario format to use. Must be `1` or `2`.", + .labels = {"nario-format"}, + .handler = {&version}, + .required = true, + }); + } + + std::string description() override + { + return "serialize store paths to standard output in nario format"; + } + + std::string doc() override + { + return +#include "nario-export.md" + ; + } + + void run(ref store, StorePaths && storePaths) override + { + auto fd = getStandardOutput(); + if (isatty(fd)) + throw UsageError("refusing to write nario to a terminal"); + FdSink sink(std::move(fd)); + exportPaths(*store, StorePathSet(storePaths.begin(), storePaths.end()), sink, version); + } +}; + +static auto rCmdNarioExport = registerCommand2({"nario", "export"}); + +static FdSource getNarioSource() +{ + auto fd = getStandardInput(); + if (isatty(fd)) + throw UsageError("refusing to read nario from a terminal"); + return FdSource(std::move(fd)); +} + +struct CmdNarioImport : StoreCommand, MixNoCheckSigs +{ + std::string description() override + { + return "import store paths from a nario file on standard input"; + } + + std::string doc() override + { + return +#include "nario-import.md" + ; + } + + void run(ref store) override + { + auto source{getNarioSource()}; + importPaths(*store, source, checkSigs); + } +}; + +static auto rCmdNarioImport = registerCommand2({"nario", "import"}); + +nlohmann::json listNar(Source & source) +{ + struct : FileSystemObjectSink + { + nlohmann::json root = nlohmann::json::object(); + + nlohmann::json & makeObject(const CanonPath & path, std::string_view type) + { + auto * cur = &root; + for (auto & c : path) { + assert((*cur)["type"] == "directory"); + auto i = (*cur)["entries"].emplace(c, nlohmann::json::object()).first; + cur = &i.value(); + } + auto inserted = cur->emplace("type", type).second; + assert(inserted); + return *cur; + } + + void createDirectory(const CanonPath & path) override + { + auto & j = makeObject(path, "directory"); + j["entries"] = nlohmann::json::object(); + } + + void createRegularFile(const CanonPath & path, fun func) override + { + struct : CreateRegularFileSink + { + bool executable = false; + std::optional size; + + void operator()(std::string_view data) override {} + + void preallocateContents(uint64_t s) override + { + size = s; + } + + void isExecutable() override + { + executable = true; + } + } crf; + + crf.skipContents = true; + + func(crf); + + auto & j = makeObject(path, "regular"); + j.emplace("size", crf.size.value()); + if (crf.executable) + j.emplace("executable", true); + } + + void createSymlink(const CanonPath & path, const std::string & target) override + { + auto & j = makeObject(path, "symlink"); + j.emplace("target", target); + } + + } parseSink; + + parseDump(parseSink, source); + + return parseSink.root; +} + +void renderNarListing(const CanonPath & prefix, const nlohmann::json & root, bool longListing) +{ + std::function recurse; + recurse = [&](const nlohmann::json & json, const CanonPath & path) { + auto type = json["type"]; + + if (longListing) { + auto tp = type == "regular" ? (json.find("executable") != json.end() ? "-r-xr-xr-x" : "-r--r--r--") + : type == "symlink" ? "lrwxrwxrwx" + : "dr-xr-xr-x"; + auto line = fmt("%s %9d %s", tp, type == "regular" ? (uint64_t) json["size"] : 0, prefix / path); + if (type == "symlink") + line += " -> " + (std::string) json["target"]; + logger->cout(line); + } else + logger->cout(fmt("%s", prefix / path)); + + if (type == "directory") { + for (auto & entry : json["entries"].items()) { + recurse(entry.value(), path / entry.key()); + } + } + }; + + recurse(root, CanonPath::root); +} + +struct CmdNarioList : Command, MixJSON, MixLongListing +{ + bool listContents = false; + + CmdNarioList() + { + addFlag({ + .longName = "recursive", + .shortName = 'R', + .description = "List the contents of NARs inside the nario.", + .handler = {&listContents, true}, + }); + } + + std::string description() override + { + return "list the contents of a nario file"; + } + + std::string doc() override + { + return +#include "nario-list.md" + ; + } + + void run() override + { + struct Config : StoreConfig + { + Config(const Params & params) + : StoreConfig(params, FilePathType::Unix) + { + } + + ref openStore() const override + { + abort(); + } + + void anchor() override {} + }; + + struct ListingStore : Store + { + std::optional json; + CmdNarioList & cmd; + + ListingStore(ref config, CmdNarioList & cmd) + : Store{*config} + , cmd(cmd) + { + } + + void queryPathInfoUncached( + const StorePath & path, Callback> callback) noexcept override + { + callback(nullptr); + } + + std::optional isTrustedClient() override + { + return Trusted; + } + + std::optional queryPathFromHashPart(const std::string & hashPart) override + { + return std::nullopt; + } + + void + addToStore(const ValidPathInfo & info, Source & source, RepairFlag repair, CheckSigsFlag checkSigs) override + { + std::optional contents; + if (cmd.listContents) + contents = listNar(source); + else + source.skip(info.narSize); + + if (json) { + // FIXME: make the JSON format configurable. + auto obj = info.toJSON(this, true, PathInfoJsonFormat::V1); + if (contents) + obj.emplace("contents", *contents); + json->emplace(printStorePath(info.path), std::move(obj)); + } else { + if (contents) + renderNarListing(CanonPath(printStorePath(info.path)), *contents, cmd.longListing); + else + logger->cout(fmt("%s: %d bytes", printStorePath(info.path), info.narSize)); + } + } + + StorePath addToStoreFromDump( + Source & dump, + std::string_view name, + FileSerialisationMethod dumpMethod, + ContentAddressMethod hashMethod, + HashAlgorithm hashAlgo, + const StorePathSet & references, + RepairFlag repair, + std::shared_ptr provenance) override + { + unsupported("addToStoreFromDump"); + } + + void narFromPath(const StorePath & path, Sink & sink) override + { + unsupported("narFromPath"); + } + + void queryRealisationUncached( + const DrvOutput &, Callback> callback) noexcept override + { + callback(nullptr); + } + + ref getFSAccessor(bool requireValidPath) override + { + return makeEmptySourceAccessor(); + } + + std::shared_ptr getFSAccessor(const StorePath & path, bool requireValidPath) override + { + unsupported("getFSAccessor"); + } + + void registerDrvOutput(const Realisation & output) override + { + unsupported("registerDrvOutput"); + } + + void anchor() override {} + }; + + auto source{getNarioSource()}; + auto config = make_ref(StoreConfig::Params()); + ListingStore lister(config, *this); + if (json) + lister.json = nlohmann::json::object(); + importPaths(lister, source, NoCheckSigs); + if (json) { + auto j = nlohmann::json::object(); + j["version"] = 1; + j["paths"] = std::move(*lister.json); + printJSON(j); + } + } +}; + +static auto rCmdNarioList = registerCommand2({"nario", "list"}); diff --git a/src/nix/nix-build/nix-build.cc b/src/nix/nix-build/nix-build.cc index e68e775e024f..76081bcc8146 100644 --- a/src/nix/nix-build/nix-build.cc +++ b/src/nix/nix-build/nix-build.cc @@ -459,7 +459,9 @@ static void main_nix_build(int argc, char ** argv) throw UsageError("nix-shell requires a single derivation"); auto & packageInfo = drvs.front(); - auto drv = evalStore->derivationFromPath(packageInfo.requireDrvPath()); + auto drvPath = packageInfo.requireDrvPath(); + state->waitForPath(drvPath); + auto drv = evalStore->derivationFromPath(drvPath); std::vector pathsToBuild; RealisedPath::Set pathsToCopy; @@ -483,6 +485,7 @@ static void main_nix_build(int argc, char ** argv) throw Error("the 'bashInteractive' attribute in did not evaluate to a derivation"); auto bashDrv = drv->requireDrvPath(); + state->waitForPath(bashDrv); pathsToBuild.push_back( DerivedPath::Built{ .drvPath = makeConstantStorePathRef(bashDrv), @@ -675,7 +678,9 @@ static void main_nix_build(int argc, char ** argv) restoreProcessContext(); + /* We're about to exec, so end and export any telemetry. */ logger->stop(); + logger->flush(); execvp(shell->c_str(), argPtrs.data()); @@ -692,6 +697,7 @@ static void main_nix_build(int argc, char ** argv) for (auto & packageInfo : drvs) { auto drvPath = packageInfo.requireDrvPath(); + state->waitForPath(drvPath); auto outputName = packageInfo.queryOutputName(); if (outputName == "") diff --git a/src/nix/nix-channel/nix-channel.cc b/src/nix/nix-channel/nix-channel.cc index aa58beb2436f..76e4ff8a20db 100644 --- a/src/nix/nix-channel/nix-channel.cc +++ b/src/nix/nix-channel/nix-channel.cc @@ -202,6 +202,11 @@ static void update(const StringSet & channelNames) static int main_nix_channel(int argc, char ** argv) { + warn( + "nix-channel is deprecated in favor of flakes in Determinate Nix. \ +See https://zero-to-nix.com for a guide to Nix flakes. \ +For details and to offer feedback on the deprecation process, see: https://github.com/DeterminateSystems/nix-src/issues/34."); + { // Figure out the name of the `.nix-channels' file to use auto home = getHome(); diff --git a/src/nix/nix-env/nix-env.cc b/src/nix/nix-env/nix-env.cc index 7fc518d21534..4f623640a813 100644 --- a/src/nix/nix-env/nix-env.cc +++ b/src/nix/nix-env/nix-env.cc @@ -243,6 +243,7 @@ static std::strong_ordering comparePriorities(EvalState & state, PackageInfo & d static bool isPrebuilt(EvalState & state, PackageInfo & elem) { auto path = elem.queryOutPath(); + state.waitForPath(path); if (state.store->isValidPath(path)) return true; return state.store->querySubstitutablePaths({path}).count(path); @@ -496,16 +497,17 @@ static void printMissing(EvalState & state, PackageInfos & elems) std::vector targets; for (auto & i : elems) if (auto drvPath = i.queryDrvPath()) { + state.waitForPath(*drvPath); auto path = DerivedPath::Built{ .drvPath = makeConstantStorePathRef(*drvPath), .outputs = OutputsSpec::All{}, }; targets.emplace_back(std::move(path)); - } else - targets.emplace_back( - DerivedPath::Opaque{ - .path = i.queryOutPath(), - }); + } else { + auto path = i.queryOutPath(); + state.waitForPath(path); + targets.emplace_back(DerivedPath::Opaque{.path = path}); + } printMissing(state.store, targets); } @@ -781,6 +783,10 @@ static void opSet(Globals & globals, Strings opFlags, Strings opArgs) drv.setName(globals.forceName); auto drvPath = drv.queryDrvPath(); + if (drvPath) + globals.state->waitForPath(*drvPath); + else + globals.state->waitForPath(drv.queryOutPath()); std::vector paths{ drvPath ? (DerivedPath) (DerivedPath::Built{ .drvPath = makeConstantStorePathRef(*drvPath), @@ -1065,6 +1071,10 @@ static void opQuery(Globals & globals, Strings opFlags, Strings opArgs) lvlTalkative, "skipping derivation named '%s' which gives an assertion failure", i.queryName()); i.setFailed(); } + /* The paths may still be being written asynchronously (e.g. by + `builtins.toFile`). */ + for (auto & path : paths) + globals.state->waitForPath(path); validPaths = store.queryValidPaths(paths); substitutablePaths = store.querySubstitutablePaths(paths); } @@ -1096,7 +1106,7 @@ static void opQuery(Globals & globals, Strings opFlags, Strings opArgs) continue; /* For table output. */ - std::vector columns; + TableRow columns; /* For XML output. */ XMLAttrs attrs; diff --git a/src/nix/nix-env/user-env.cc b/src/nix/nix-env/user-env.cc index cb74a36d0d4c..5cd3bba9f430 100644 --- a/src/nix/nix-env/user-env.cc +++ b/src/nix/nix-env/user-env.cc @@ -40,8 +40,10 @@ bool createUserEnv( exist already. */ std::vector drvsToBuild; for (auto & i : elems) - if (auto drvPath = i.queryDrvPath()) + if (auto drvPath = i.queryDrvPath()) { + state.waitForPath(*drvPath); drvsToBuild.push_back({*drvPath}); + } debug("building user environment dependencies"); state.store->buildPaths(toDerivedPaths(drvsToBuild), state.repair ? bmRepair : bmNormal); @@ -154,6 +156,7 @@ bool createUserEnv( debug("building user environment"); std::vector topLevelDrvs; topLevelDrvs.push_back({topLevelDrv}); + state.waitForPath(topLevelDrv); state.store->buildPaths(toDerivedPaths(topLevelDrvs), state.repair ? bmRepair : bmNormal); /* Switch the current user environment to the output path. */ diff --git a/src/nix/nix-instantiate/nix-instantiate.cc b/src/nix/nix-instantiate/nix-instantiate.cc index dee79bcbfc2c..7fa0c99de74c 100644 --- a/src/nix/nix-instantiate/nix-instantiate.cc +++ b/src/nix/nix-instantiate/nix-instantiate.cc @@ -54,19 +54,23 @@ void processExpr( else state.autoCallFunction(autoArgs, v, vRes); if (output == okRaw) - std::cout << *state.coerceToString(noPos, vRes, context, "while generating the nix-instantiate output"); + std::cout << state.devirtualize( + *state.coerceToString(noPos, vRes, context, "while generating the nix-instantiate output"), + context); // We intentionally don't output a newline here. The default PS1 for Bash in NixOS starts with a newline // and other interactive shells like Zsh are smart enough to print a missing newline before the prompt. - else if (output == okXML) - printValueAsXML(state, strict, location, vRes, std::cout, context, noPos); - else if (output == okJSON) { - printValueAsJSON(state, strict, vRes, v.determinePos(noPos), std::cout, context); - std::cout << std::endl; + else if (output == okXML) { + std::ostringstream s; + printValueAsXML(state, strict, location, vRes, s, context, noPos); + std::cout << state.devirtualize(s.str(), context); + } else if (output == okJSON) { + auto j = printValueAsJSON(state, strict, vRes, v.determinePos(noPos), context); + std::cout << state.devirtualize(j.dump(), context) << std::endl; } else { if (strict) state.forceValueDeep(vRes); std::set seen; - printAmbiguous(state, vRes, std::cout, &seen, &context); + printAmbiguous(state, vRes, std::cout, &seen); std::cout << std::endl; } } else { @@ -94,8 +98,6 @@ void processExpr( std::cout << fmt("%s%s\n", drvPathS, (outputName != "out" ? "!" + outputName : "")); } } - - state.ensureLazyPathsCopied(context); } } diff --git a/src/nix/nix-store/nix-store.cc b/src/nix/nix-store/nix-store.cc index 2bb56eb51bca..59928479aba6 100644 --- a/src/nix/nix-store/nix-store.cc +++ b/src/nix/nix-store/nix-store.cc @@ -773,7 +773,7 @@ static void opExport(Strings opFlags, Strings opArgs) paths.insert(store->followLinksToStorePath(i)); FdSink sink(getStandardOutput()); - exportPaths(*store, paths, sink); + exportPaths(*store, paths, sink, 1); sink.flush(); } @@ -954,8 +954,7 @@ static void opServe(Strings opFlags, Strings opArgs) bool substitute = readInt(in); auto paths = ServeProto::Serialise::read(*store, rconn); if (lock && writeAllowed) - for (auto & path : paths) - store->addTempRoot(path); + store->addTempRoots(paths); if (substitute && writeAllowed) { store->substitutePaths(paths); @@ -1102,10 +1101,10 @@ static void opGenerateBinaryCacheKey(Strings opFlags, Strings opArgs) std::string secretKeyFile = *i++; std::string publicKeyFile = *i++; - auto secretKey = SecretKey::generate(keyName); + auto secretKey = SecretKey::generate(keyName, KeyType::Ed25519); - writeFile(publicKeyFile, secretKey.toPublicKey().to_string(), 0666, FsSync::Yes); - writeFile(secretKeyFile, secretKey.to_string(), 0600, FsSync::Yes); + writeFile(publicKeyFile, secretKey->toPublicKey()->to_string(), 0666, FsSync::Yes); + writeFile(secretKeyFile, secretKey->to_string(), 0600, FsSync::Yes); } static void opVersion(Strings opFlags, Strings opArgs) diff --git a/src/nix/nix.md b/src/nix/nix.md index fed781aa77fa..cc31dabbab41 100644 --- a/src/nix/nix.md +++ b/src/nix/nix.md @@ -48,11 +48,6 @@ manual](https://nix.dev/manual/nix/stable/). # Installables -> **Warning** \ -> Installables are part of the unstable -> [`nix-command` experimental feature](@docroot@/development/experimental-features.md#xp-feature-nix-command), -> and subject to change without notice. - Many `nix` subcommands operate on one or more *installables*. These are command line arguments that represent something that can be realised in the Nix store. @@ -72,13 +67,6 @@ That is, Nix will operate on the default flake output attribute of the flake in ### Flake output attribute -> **Warning** \ -> Flake output attribute installables depend on both the -> [`flakes`](@docroot@/development/experimental-features.md#xp-feature-flakes) -> and -> [`nix-command`](@docroot@/development/experimental-features.md#xp-feature-nix-command) -> experimental features, and subject to change without notice. - Example: `nixpkgs#hello` These have the form *flakeref*[`#`*attrpath*], where *flakeref* is a diff --git a/src/nix/otel-logger.cc b/src/nix/otel-logger.cc new file mode 100644 index 000000000000..71ac06fa798a --- /dev/null +++ b/src/nix/otel-logger.cc @@ -0,0 +1,965 @@ +#include "otel-logger.hh" + +#include "cli-config-private.hh" + +#if HAVE_OTEL +# include "nix/store/derivations.hh" +# include "nix/store/filetransfer.hh" +# include "nix/store/names.hh" +# include "nix/util/base-n.hh" +# include "nix/util/compression.hh" +# include "nix/util/config-global.hh" +# include "nix/util/environment-variables.hh" +# include "nix/util/exit.hh" +# include "nix/util/serialise.hh" +# include "nix/util/signals.hh" +# include "nix/util/sync.hh" +# include "nix/util/terminal.hh" +# include "nix/util/url.hh" + +# include +# include +# include + +# include + +# include + +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +# include +#endif + +namespace nix { + +#if HAVE_OTEL + +struct OtelSettings : Config +{ + Setting enable{ + this, + false, + "otlp", + R"( + Whether to export [OpenTelemetry](https://opentelemetry.io/) traces of + Nix's activities (such as evaluation, builds, substitutions and HTTP + requests). Requires `otlp-endpoint` to be set as well. + + Setting the `OTEL_EXPORTER_OTLP_ENDPOINT` or + `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` environment variable enables + tracing regardless of this setting. + )"}; + + Setting endpoint{ + this, + "", + "otlp-endpoint", + R"( + The base URL of the OpenTelemetry collector to which traces are sent, + e.g. `https://otel.example.org`. The path `/v1/traces` is appended + automatically. + + Only used if `otlp` is enabled. Overridden by the + `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable. + )"}; + + Setting headers{ + this, + "", + "otlp-headers", + R"( + Extra HTTP headers to send to the collector, e.g. to pass an + authorization token. + + This uses the same syntax as the `OTEL_EXPORTER_OTLP_HEADERS` + environment variable (which overrides it): a comma-separated list of + `name=value` pairs, where the values are percent-encoded. For + instance, + + ``` + otlp-headers = authorization=Bearer%20secret123 + ``` + + sends the header `authorization: Bearer secret123`. + )"}; + + Setting compression{ + this, + "gzip", + "otlp-compression", + R"( + The compression to apply to exported traces: `gzip` (the default) or + `none`. + + Overridden by the `OTEL_EXPORTER_OTLP_COMPRESSION` environment + variable. + )"}; +}; + +/* Note: deliberately not applied from the client's `setOptions` in the + daemon, so that a client can't redirect the daemon's telemetry. */ +static OtelSettings otelSettings; + +static GlobalConfig::Register rOtelSettings(&otelSettings); + +namespace { + +/** + * The name of the activity under which we upload spans. Since the + * upload itself creates activities (namely the file transfer), we + * must not create spans for this activity or its children, since + * that would create an infinite regress. + */ +constexpr std::string_view uploadActivityName = "UploadOpenTelemetry"; + +struct ExtractCarrier : opentelemetry::context::propagation::TextMapCarrier +{ + std::string_view traceparent, tracestate; + + std::string_view Get(std::string_view key) const noexcept override + { + if (key == "traceparent") + return traceparent; + if (key == "tracestate") + return tracestate; + return {}; + } + + void Set(std::string_view, std::string_view) noexcept override {} +}; + +/** + * Parse a W3C `traceparent` value into a span context usable as a + * remote parent. Returns std::nullopt on an empty or invalid value. + * (Extract() returns the input context unchanged on a parse failure, + * leaving an invalid SpanContext.) + */ +std::optional parseTraceparent(std::string_view traceparent) +{ + if (traceparent.empty()) + return std::nullopt; + ExtractCarrier carrier; + carrier.traceparent = traceparent; + opentelemetry::context::Context emptyCtx; + auto ctx = opentelemetry::trace::propagation::HttpTraceContext{}.Extract(carrier, emptyCtx); + auto spanContext = opentelemetry::trace::GetSpan(ctx)->GetContext(); + if (!spanContext.IsValid()) + return std::nullopt; + return spanContext; +} + +struct InjectCarrier : opentelemetry::context::propagation::TextMapCarrier +{ + Headers headers; + + std::string_view Get(std::string_view) const noexcept override + { + return {}; + } + + void Set(std::string_view key, std::string_view value) noexcept override + { + /* Copy immediately: `value` may point into a stack buffer of + the propagator. */ + headers.emplace_back(std::string(key), std::string(value)); + } +}; + +/** + * The span name for an activity: the name of the enum value without + * the `act` prefix, e.g. `OptimiseStore`. An empty result means the + * activity's text should be used instead. + * + * TODO: Use C++26 reflection to derive this from the enum + * definition generically instead of enumerating the values here. + */ +std::string_view activityName(ActivityType type) +{ + switch (type) { + case actUnknown: + return {}; +# define ACTIVITY_NAME(name) \ + case act##name: \ + return #name; + ACTIVITY_NAME(CopyPath) + ACTIVITY_NAME(FileTransfer) + ACTIVITY_NAME(Realise) + ACTIVITY_NAME(CopyPaths) + ACTIVITY_NAME(Builds) + ACTIVITY_NAME(Build) + ACTIVITY_NAME(OptimiseStore) + ACTIVITY_NAME(VerifyPaths) + ACTIVITY_NAME(Substitute) + ACTIVITY_NAME(QueryPathInfo) + ACTIVITY_NAME(PostBuildHook) + ACTIVITY_NAME(BuildWaiting) + ACTIVITY_NAME(FetchTree) +# undef ACTIVITY_NAME + case actStringly: + /* Only reached via a Logger that flattens string-named + activities; the text fallback is the best we can do. */ + return {}; + } + return {}; +} + +/* Defensive field accessors, like the progress bar's. */ +std::string_view getS(const Logger::Fields & fields, size_t n) +{ + if (n < fields.size()) { + if (auto p = std::get_if(&fields[n].raw)) + return *p; + } + return {}; +} + +/** + * Set the `.path` attribute to a store path, along with + * `.name` and `.version` giving its name and version + * separately, e.g. `patchelf` and `0.18.0` for + * `/nix/store/…-patchelf-0.18.0` or `…-patchelf-0.18.0.drv`. Those are + * much easier to filter on than the path itself. A path that cannot + * be parsed just doesn't get these extra attributes. + */ +void setPathAttributes(opentelemetry::trace::Span & span, std::string_view prefix, std::string_view path) +{ + auto key = [&](std::string_view suffix) { return std::string(prefix) + "." + std::string(suffix); }; + + span.SetAttribute(key("path"), path); + + try { + StorePath storePath(baseNameOf(path)); + DrvName drvName(storePath.isDerivation() ? BasicDerivation::nameFromPath(storePath) : storePath.name()); + span.SetAttribute(key("name"), drvName.name); + if (!drvName.version.empty()) + span.SetAttribute(key("version"), drvName.version); + } catch (...) { + } +} + +/** + * Create the sampler selected by the standard `OTEL_TRACES_SAMPLER` / + * `OTEL_TRACES_SAMPLER_ARG` environment variables, which the C++ SDK + * does not read itself. "parentbased" samplers follow the sampling + * decision of the parent span, which propagates in the sampled flag + * of the W3C trace context — so the daemon follows the client's + * decision. + */ +std::unique_ptr makeSampler() +{ + namespace sdktrace = opentelemetry::sdk::trace; + + auto ratio = [&]() -> double { + auto arg = getEnv("OTEL_TRACES_SAMPLER_ARG"); + if (!arg) + return 1.0; + try { + return std::stod(*arg); + } catch (...) { + warn("invalid OTEL_TRACES_SAMPLER_ARG '%s'; assuming 1.0", *arg); + return 1.0; + } + }; + auto parentBased = [](std::shared_ptr delegate) -> std::unique_ptr { + return std::make_unique(std::move(delegate)); + }; + auto name = getEnv("OTEL_TRACES_SAMPLER").value_or("parentbased_always_on"); + if (name == "always_on") + return std::make_unique(); + if (name == "always_off") + return std::make_unique(); + if (name == "traceidratio") + return std::make_unique(ratio()); + if (name == "parentbased_always_off") + return parentBased(std::make_shared()); + if (name == "parentbased_traceidratio") + return parentBased(std::make_shared(ratio())); + if (name != "parentbased_always_on") + warn("unknown OTEL_TRACES_SAMPLER '%s'; assuming 'parentbased_always_on'", name); + return parentBased(std::make_shared()); +} + +/** + * A `Logger` that maps Nix activities onto OpenTelemetry spans, + * under a root span created in the constructor. It's added to the + * global logger by `initOtel()`. + * + * Activities replayed from another process (cf. `RemoteLogSource`) + * are ignored, since the originating process is responsible for + * exporting them. + */ +class OpenTelemetryLogger : public Logger +{ + using SpanPtr = std::shared_ptr; + + std::unique_ptr provider; + + std::shared_ptr tracer; + + SpanPtr rootSpan; + + /** + * Whether `NIX_DEBUG_OTEL` is set. + */ + bool debug = false; + + /** + * The root span's trace ID, to be printed by `flush()` if + * `debug` is set. + */ + std::string debugTraceId; + + Sync> spans_; + + /** + * Activities for which we don't create spans, namely the + * `UploadOpenTelemetry` activities and their children. + */ + boost::concurrent_flat_set ignoredActs; + + /** + * Whether this activity should be ignored, i.e. whether it's an + * `UploadOpenTelemetry` activity or a child of one. If so, record + * it so that its children are ignored as well. + */ + bool ignoreActivity(ActivityId act, std::string_view name, ActivityId parent) + { + if (name != uploadActivityName && !ignoredActs.contains(parent)) + return false; + ignoredActs.insert(act); + return true; + } + + static Headers injectContext(const SpanPtr & span) + { + if (!span->GetContext().IsValid()) + return {}; + InjectCarrier carrier; + opentelemetry::context::Context ctx; + auto withSpan = opentelemetry::trace::SetSpan(ctx, span); + opentelemetry::trace::propagation::HttpTraceContext{}.Inject(carrier, withSpan); + return std::move(carrier.headers); + } + + /** + * Start a span. When debugging (`NIX_DEBUG_OTEL`), spans carry + * `sampling.priority = 1`, which tells a collector's probabilistic + * sampler to keep them, so that the trace whose ID we print can + * actually be found. + */ + SpanPtr startSpan(std::string_view name, const opentelemetry::trace::StartSpanOptions & options) + { + auto span = tracer->StartSpan(name, options); + if (debug) + span->SetAttribute("sampling.priority", 1); + return span; + } + +public: + OpenTelemetryLogger( + std::string_view serviceName, + std::unique_ptr exporter, + std::string_view rootSpanName, + std::string_view remoteParentTraceparent, + bool isServer) + { + namespace sdktrace = opentelemetry::sdk::trace; + + auto processor = + sdktrace::BatchSpanProcessorFactory::Create(std::move(exporter), sdktrace::BatchSpanProcessorOptions{}); + auto resource = opentelemetry::sdk::resource::Resource::Create({ + {opentelemetry::semconv::service::kServiceName, std::string(serviceName)}, + }); + provider = sdktrace::TracerProviderFactory::Create(std::move(processor), resource, makeSampler()); + tracer = provider->GetTracer("nix"); + + debug = getEnv("NIX_DEBUG_OTEL").has_value(); + + opentelemetry::trace::StartSpanOptions options; + if (isServer) + options.kind = opentelemetry::trace::SpanKind::kServer; + if (auto spanContext = parseTraceparent(remoteParentTraceparent)) + options.parent = *spanContext; + rootSpan = startSpan(rootSpanName, options); + + if (debug) { + char buf[2 * opentelemetry::trace::TraceId::kSize]; + rootSpan->GetContext().trace_id().ToLowerBase16(buf); + debugTraceId = std::string(buf, sizeof(buf)); + } + } + + void log(Verbosity lvl, std::string_view s) noexcept override {} + + void logEI(const ErrorInfo & ei) noexcept override {} + + void printException(const std::exception_ptr & ex, std::string_view programName) noexcept override + { + try { + std::rethrow_exception(ex); + } catch (Exit &) { + /* Not a failure: this is how commands like `--version` + return. */ + } catch (std::exception & e) { + // FIXME: privacy + rootSpan->SetStatus(opentelemetry::trace::StatusCode::kError, filterANSIEscapes(e.what(), true)); + } catch (...) { + rootSpan->SetStatus(opentelemetry::trace::StatusCode::kError, "unknown exception"); + } + } + + void startActivity( + ActivityId act, + Verbosity lvl, + ActivityType type, + const std::string & s, + const Fields & fields, + ActivityId parent) noexcept override + { + try { + if (isRemoteLogSource()) + return; + + opentelemetry::trace::StartSpanOptions options; + + auto name = activityName(type); + bool textIsName = name.empty(); + if (textIsName) + name = s.empty() ? "activity" : std::string_view(s); + + if (ignoreActivity(act, name, parent)) + return; + + auto spans(spans_.lock()); + + if (auto i = spans->find(parent); i != spans->end()) + options.parent = i->second->GetContext(); + else + options.parent = rootSpan->GetContext(); + + auto span = startSpan(name, options); + + if (!s.empty() && !textIsName) + span->SetAttribute("nix.activity.text", filterANSIEscapes(s, true)); + +// Allow handling a subset of enum values +# pragma GCC diagnostic push +# pragma GCC diagnostic ignored "-Wswitch-enum" + switch (type) { + case actFileTransfer: + span->SetAttribute("url.full", getS(fields, 0)); + break; + case actBuild: + case actPostBuildHook: + setPathAttributes(*span, "nix.drv", getS(fields, 0)); + if (auto machine = getS(fields, 1); !machine.empty()) + span->SetAttribute("nix.machine", machine); + break; + case actSubstitute: + case actQueryPathInfo: + setPathAttributes(*span, "nix.store", getS(fields, 0)); + span->SetAttribute("nix.substituter", getS(fields, 1)); + break; + case actCopyPath: + setPathAttributes(*span, "nix.store", getS(fields, 0)); + span->SetAttribute("nix.src.store", getS(fields, 1)); + span->SetAttribute("nix.dst.store", getS(fields, 2)); + break; + default: + break; + } +# pragma GCC diagnostic pop + + spans->emplace(act, std::move(span)); + } catch (...) { + } + } + + void startActivity( + ActivityId act, + Verbosity lvl, + std::string_view name, + ActivityMetadata metadata, + std::string_view s, + ActivityId parent) noexcept override + { + try { + if (isRemoteLogSource()) + return; + + if (ignoreActivity(act, name, parent)) + return; + + /* An activity carrying a `traceparent` metadata field + exists only to link its child activities to a span in + another process (e.g. the client activity on whose + behalf the daemon is performing an operation). Don't + emit a span for the activity itself — there can be very + many of them (one per daemon operation) — but record + the remote context, or the local parent if the trace + context is absent or invalid, for parent lookups by + child activities. */ + for (auto & [key, value] : metadata) { + if (key != "traceparent") + continue; + std::optional spanContext; + if (auto str = std::get_if(&value.raw)) + spanContext = parseTraceparent(*str); + auto spans(spans_.lock()); + if (!spanContext) { + if (auto i = spans->find(parent); i != spans->end()) + spanContext = i->second->GetContext(); + else + spanContext = rootSpan->GetContext(); + } + spans->emplace(act, SpanPtr(new opentelemetry::trace::DefaultSpan(*spanContext))); + return; + } + + opentelemetry::trace::StartSpanOptions options; + + /* Per the OpenTelemetry semantic conventions, HTTP client + spans are named after the request method. */ + auto spanName = name; + for (auto & [key, value] : metadata) + if (key == "http.request.method") + if (auto method = std::get_if(&value.raw)) { + options.kind = opentelemetry::trace::SpanKind::kClient; + spanName = *method; + } + + auto spans(spans_.lock()); + + if (auto i = spans->find(parent); i != spans->end()) + options.parent = i->second->GetContext(); + else + options.parent = rootSpan->GetContext(); + + auto span = startSpan(spanName, options); + + if (!s.empty()) + span->SetAttribute("nix.activity.text", filterANSIEscapes(s, true)); + + for (auto & [key, value] : metadata) { + if (auto str = std::get_if(&value.raw)) + span->SetAttribute(key, *str); + else if (auto n = std::get_if(&value.raw)) + span->SetAttribute(key, (int64_t) *n); + } + + spans->emplace(act, std::move(span)); + } catch (...) { + } + } + + void stopActivity(ActivityId act) noexcept override + { + try { + if (isRemoteLogSource()) + return; + if (ignoredActs.erase(act)) + return; + auto spans(spans_.lock()); + if (auto i = spans->find(act); i != spans->end()) { + /* If the activity is being stopped while an exception + is in flight, i.e. the `Activity` is being destroyed + by stack unwinding, assume that the activity + failed. */ + if (std::uncaught_exceptions()) + i->second->SetStatus( + opentelemetry::trace::StatusCode::kError, "activity terminated by an exception"); + i->second->End(); + spans->erase(i); + } + } catch (...) { + } + } + + void result(ActivityId act, ResultType type, const nlohmann::json & json) noexcept override + { + try { + if (isRemoteLogSource()) + return; + if (type == resHttpStatus) { + auto spans(spans_.lock()); + if (auto i = spans->find(act); i != spans->end()) { + if (auto status = json.find("httpStatus"); status != json.end() && status->is_number()) + i->second->SetAttribute("http.response.status_code", status->get()); + if (auto bodySize = json.find("bodySize"); bodySize != json.end() && bodySize->is_number()) + i->second->SetAttribute("http.response.body.size", bodySize->get()); + } + } else if (type == resBuildResult) { + /* A failed build or substitution doesn't throw, so + `stopActivity()` can't tell; the result tells us. */ + auto spans(spans_.lock()); + if (auto i = spans->find(act); i != spans->end()) { + auto status = json.value("status", ""); + if (!status.empty()) + i->second->SetAttribute("nix.build.status", status); + /* Note: a substitution goal "failing" because there + is no substituter for the path is the normal + prelude to building it, so don't call that an + error. */ + if (!json.value("success", true) && status != "NoSubstituters") { + // FIXME: privacy (the message can include the build log tail) + auto msg = json.value("errorMsg", "build failed"); + i->second->SetStatus(opentelemetry::trace::StatusCode::kError, filterANSIEscapes(msg, true)); + } + } + } + } catch (...) { + } + } + + Headers getTraceContext(ActivityId act) override + { + try { + if (act) { + /* Don't propagate any context from an ignored + activity: the telemetry upload must not be part of + the trace it's carrying. */ + if (ignoredActs.contains(act)) + return {}; + auto spans(spans_.lock()); + if (auto i = spans->find(act); i != spans->end()) + return injectContext(i->second); + } + return injectContext(rootSpan); + } catch (...) { + return {}; + } + } + + void stop() override + { + try { + auto spans(spans_.lock()); + for (auto & [_, span] : *spans) + span->End(); + spans->clear(); + rootSpan->End(); + } catch (...) { + } + } + + void flush() override + { + /* Bound the timeout: the SDK default is microseconds::max(), + and a hung collector must not hang process exit. */ + provider->ForceFlush(std::chrono::microseconds(std::chrono::seconds(5))); + + if (!debugTraceId.empty()) + writeToStderr(fmt("OpenTelemetry trace ID: %s\n", debugTraceId)); + } +}; + +/** + * Serialize an attribute value to its OTLP/JSON representation, i.e. + * an `AnyValue` object such as `{"stringValue": "foo"}`. Note that + * 64-bit integers are represented as strings in OTLP/JSON. + */ +nlohmann::json toAnyValue(const opentelemetry::sdk::common::OwnedAttributeValue & value) +{ + return std::visit( + [](const auto & v) -> nlohmann::json { + using T = std::decay_t; + if constexpr (std::is_same_v) + return {{"boolValue", v}}; + else if constexpr (std::is_same_v) + return {{"doubleValue", v}}; + else if constexpr (std::is_same_v) + return {{"stringValue", v}}; + else if constexpr (std::is_integral_v) + return {{"intValue", std::to_string(v)}}; + else if constexpr (std::is_same_v>) + return {{"bytesValue", base64::encode(std::as_bytes(std::span{v}))}}; + else { + /* Any other vector: an OTLP array of AnyValues. Note + that we can't take the elements by reference, since + `std::vector` yields proxy references. */ + auto values = nlohmann::json::array(); + for (auto x : v) { + using E = std::decay_t; + if constexpr (std::is_same_v) + values.push_back({{"boolValue", (bool) x}}); + else if constexpr (std::is_same_v) + values.push_back({{"doubleValue", x}}); + else if constexpr (std::is_same_v) + values.push_back({{"stringValue", x}}); + else + values.push_back({{"intValue", std::to_string(x)}}); + } + return {{"arrayValue", {{"values", std::move(values)}}}}; + } + }, + value); +} + +/** + * Serialize an attribute map to an OTLP/JSON `KeyValue` array. + */ +template +nlohmann::json toAttributes(const Map & map) +{ + auto res = nlohmann::json::array(); + for (auto & [key, value] : map) + res.push_back({{"key", key}, {"value", toAnyValue(value)}}); + return res; +} + +template +std::string toHex(const Id & id) +{ + char buf[2 * Id::kSize]; + id.ToLowerBase16(buf); + return std::string(buf, sizeof(buf)); +} + +std::string toUnixNano(std::chrono::nanoseconds t) +{ + return std::to_string(t.count()); +} + +/** + * A span exporter that serializes spans to OTLP/JSON and uploads them + * using Nix's own `FileTransfer`. Compared to the exporter that comes + * with opentelemetry-cpp, this avoids a dependency on protobuf (which + * is very large), and it reuses the HTTP client that Nix already has. + */ +class OtlpJsonSpanExporter final : public opentelemetry::sdk::trace::SpanExporter +{ + std::string endpoint; + Headers headers; + bool compress; + + std::atomic isShutdown{false}; + + /* Only ever touched from the batch processor's worker thread. */ + bool warned = false; + +public: + + OtlpJsonSpanExporter(std::string endpoint, Headers headers, bool compress) + : endpoint(std::move(endpoint)) + , headers(std::move(headers)) + , compress(compress) + { + } + + std::unique_ptr MakeRecordable() noexcept override + { + return std::make_unique(); + } + + opentelemetry::sdk::common::ExportResult + Export(const std::span> & recordables) noexcept override + { + using opentelemetry::sdk::common::ExportResult; + + if (isShutdown.load(std::memory_order_acquire)) + return ExportResult::kFailure; + + try { + if (recordables.empty()) + return ExportResult::kSuccess; + + /* All spans in a batch come from the same tracer provider, + so they share a resource and (in our case) a scope. */ + auto spans = nlohmann::json::array(); + const opentelemetry::sdk::resource::Resource * resource = nullptr; + const opentelemetry::sdk::trace::InstrumentationScope * scope = nullptr; + + for (auto & recordable : recordables) { + /* Safe: `MakeRecordable()` only ever returns `SpanData`. */ + auto & span = static_cast(*recordable); + + resource = &span.GetResource(); + scope = &span.GetInstrumentationScope(); + + nlohmann::json json{ + {"traceId", toHex(span.GetTraceId())}, + {"spanId", toHex(span.GetSpanId())}, + {"name", std::string(span.GetName())}, + /* `SpanKind` is declared in the same order as in + OTLP, which starts counting at `unspecified`. */ + {"kind", (int) span.GetSpanKind() + 1}, + {"startTimeUnixNano", toUnixNano(span.GetStartTime().time_since_epoch())}, + {"endTimeUnixNano", toUnixNano(span.GetStartTime().time_since_epoch() + span.GetDuration())}, + {"attributes", toAttributes(span.GetAttributes())}, + {"flags", span.GetFlags().flags()}, + }; + + if (span.GetParentSpanId().IsValid()) + json["parentSpanId"] = toHex(span.GetParentSpanId()); + + if (span.GetStatus() != opentelemetry::trace::StatusCode::kUnset) { + json["status"] = { + {"code", (int) span.GetStatus()}, + {"message", std::string(span.GetDescription())}, + }; + } + + spans.push_back(std::move(json)); + } + + nlohmann::json doc{ + {"resourceSpans", + {{ + {"resource", {{"attributes", toAttributes(resource->GetAttributes())}}}, + {"scopeSpans", + {{ + {"scope", {{"name", scope->GetName()}, {"version", scope->GetVersion()}}}, + {"spans", std::move(spans)}, + }}}, + }}}, + }; + + upload(doc.dump()); + + return ExportResult::kSuccess; + } catch (Interrupted & e) { + /* Not a problem with the collector, so not worth a + warning. */ + printMsg(lvlDebug, "OpenTelemetry export interrupted: %s", e.what()); + return ExportResult::kFailure; + } catch (std::exception & e) { + /* Note that nothing retries a failed export, so all we can + do is drop the spans. Don't let the exception escape, + since this method is noexcept. Only warn about the first + failure: the batch processor exports every few seconds, + so an unreachable collector would otherwise flood the + output with the same warning. */ + warnOnce(warned, "unable to export OpenTelemetry traces: %s", e.what()); + return ExportResult::kFailure; + } + } + + void upload(std::string payload) + { + /* The upload creates activities of its own, which would be + exported as spans, which would create more activities, ad + infinitum. So do it inside an activity that the + OpenTelemetryLogger ignores, along with its children. */ + Activity act(*logger, lvlDebug, uploadActivityName, {}, "", 0); + PushActivity pact(act.id); + + FileTransferRequest request(parseURL(endpoint)); + request.method = HttpMethod::Post; + request.mimeType = "application/json"; + request.headers = headers; + + if (compress) { + payload = nix::compress(CompressionAlgo::gzip, payload); + request.headers.emplace_back("Content-Encoding", "gzip"); + } + + StringSource source{payload}; + request.data = {source}; + + /* Don't hold up the process at exit retrying telemetry. */ + request.retryAttempts = 0; + + getFileTransfer()->upload(request); + } + + bool ForceFlush(std::chrono::microseconds) noexcept override + { + /* We upload synchronously in `Export()`, so there is never + anything buffered here. */ + return true; + } + + bool Shutdown(std::chrono::microseconds) noexcept override + { + isShutdown.store(true, std::memory_order_release); + return true; + } +}; + +/** + * Parse `OTEL_EXPORTER_OTLP_HEADERS`, a comma-separated list of + * percent-encoded `name=value` pairs. + */ +Headers parseOtlpHeaders(std::string_view s) +{ + Headers headers; + for (auto & item : tokenizeString(s, ",")) { + auto eq = item.find('='); + if (eq == std::string::npos) + continue; + auto name = trim(item.substr(0, eq)); + auto value = trim(item.substr(eq + 1)); + if (!name.empty()) + headers.emplace_back(percentDecode(name), percentDecode(value)); + } + return headers; +} + +} // namespace + +void initOtel( + std::string_view serviceName, + std::string_view rootSpanName, + std::string_view remoteParentTraceparent, + bool isServer) +{ + /* An endpoint in the environment enables tracing by itself, and + takes precedence over the settings. */ + auto endpoint = [&]() -> std::string { + if (auto s = getEnv("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT")) + return *s; + if (auto s = getEnv("OTEL_EXPORTER_OTLP_ENDPOINT")) + return *s + "/v1/traces"; + /* Unlike the environment variables, the setting is subject to + `otlp`, so that tracing can be turned off without removing + the endpoint from the configuration. */ + if (otelSettings.enable && !otelSettings.endpoint.get().empty()) + return otelSettings.endpoint.get() + "/v1/traces"; + return ""; + }(); + + /* Without an explicitly configured endpoint, stay off; we don't + want to export to some default endpoint behind the user's + back. */ + if (endpoint.empty()) { + if (otelSettings.enable) + warn("OpenTelemetry tracing is enabled but no endpoint is configured; see the 'otlp-endpoint' setting"); + return; + } + + auto exporter = std::make_unique( + endpoint, + parseOtlpHeaders(getEnv("OTEL_EXPORTER_OTLP_HEADERS").value_or(otelSettings.headers)), + getEnv("OTEL_EXPORTER_OTLP_COMPRESSION").value_or(otelSettings.compression) == "gzip"); + + applyExtraLogger( + std::make_unique( + serviceName, std::move(exporter), rootSpanName, remoteParentTraceparent, isServer)); +} + +#else + +void initOtel(std::string_view, std::string_view, std::string_view, bool) {} + +#endif // HAVE_OTEL + +} // namespace nix diff --git a/src/nix/otel-logger.hh b/src/nix/otel-logger.hh new file mode 100644 index 000000000000..8c30fa414f99 --- /dev/null +++ b/src/nix/otel-logger.hh @@ -0,0 +1,40 @@ +#pragma once +///@file + +#include + +namespace nix { + +/** + * Initialize OpenTelemetry tracing for this process, and add a logger + * to the global logger (using `applyExtraLogger()`) that maps Nix + * activities onto OpenTelemetry spans, under a root span named + * `rootSpanName`, and exports them via OTLP/HTTP. + * + * Does nothing (and is cheap) if tracing isn't configured (see the + * `otlp` settings and the `OTEL_EXPORTER_OTLP_*` environment + * variables), or if tracing support is not compiled in. + * + * If `remoteParentTraceparent` is non-empty, the root span's parent is + * the given W3C trace context, e.g. received from the client during + * the daemon handshake, or from a parent process via the `TRACEPARENT` + * environment variable. An invalid value yields an unparented root + * span. If `isServer` is set, the root span is a server span (i.e. it + * handles a request from another process, as in the daemon). + * + * The logger owns the exporter and its worker thread, so a forked + * child (which gets a fresh global logger, cf. `startProcess()`) + * doesn't trace unless it calls this itself. + * + * The logger's `flush()` exports all pending spans, with bounded + * timeouts. Since loggers are generally not destroyed, it has to be + * called explicitly before the process exits; `handleExceptions()` + * does so. + */ +void initOtel( + std::string_view serviceName, + std::string_view rootSpanName, + std::string_view remoteParentTraceparent = {}, + bool isServer = false); + +} // namespace nix diff --git a/src/nix/package.nix b/src/nix/package.nix index b48ce53c6a71..a078fc156529 100644 --- a/src/nix/package.nix +++ b/src/nix/package.nix @@ -2,11 +2,16 @@ stdenv, lib, mkMesonExecutable, + llvmPackages, nix-store, nix-expr, nix-main, nix-cmd, + opentelemetry-cpp, + sentry-native, + + libmicrohttpd, nix-expr-c, nix-fetchers-c, @@ -26,7 +31,12 @@ # workloads (~10-15% on large evaluations). # mimalloc is disabled on FreeBSD due to a crash in nixpkgs 25.11. # Once the nixpkgs flake is updated, mimalloc can be enabled again. - withMimalloc ? !stdenv.hostPlatform.isWindows && !stdenv.hostPlatform.isFreeBSD, + # It's also disabled on static aarch64-darwin because of a duplicate + # `reallocarray` symbol in libmimalloc.a and lowdown's compats.o. + withMimalloc ? + !stdenv.hostPlatform.isWindows + && !stdenv.hostPlatform.isFreeBSD + && !(stdenv.hostPlatform.isStatic && stdenv.hostPlatform.isDarwin && stdenv.hostPlatform.isAarch64), # Whether to embed the public C API into the `nix` executable so plugins can # resolve those symbols without linking Nix libraries directly. @@ -35,10 +45,12 @@ let inherit (lib) fileset; + enableSentry = !stdenv.hostPlatform.isStatic; + withOtel = !stdenv.hostPlatform.isStatic && stdenv.buildPlatform.canExecute stdenv.hostPlatform; in mkMesonExecutable (finalAttrs: { - pname = "nix"; + pname = "determinate-nix"; inherit version; workDir = ./.; @@ -74,6 +86,7 @@ mkMesonExecutable (finalAttrs: { ../../doc/manual/source/store/types/index.md.in ./profiles.md ../../doc/manual/source/command-ref/files/profiles.md + ./baked-flake.nix # Files ] @@ -89,6 +102,7 @@ mkMesonExecutable (finalAttrs: { nix-expr nix-main nix-cmd + libmicrohttpd ] ++ lib.optionals withPluginCApi [ nix-expr-c @@ -98,18 +112,42 @@ mkMesonExecutable (finalAttrs: { nix-store-c nix-util-c ] - ++ lib.optional withMimalloc mimalloc; + ++ lib.optional withMimalloc mimalloc + ++ lib.optional ( + stdenv.cc.isClang + && stdenv.hostPlatform.isStatic + && stdenv.cc.libcxx != null + && stdenv.cc.libcxx.isLLVM + ) llvmPackages.libunwind + ++ lib.optional enableSentry sentry-native + ++ lib.optional withOtel opentelemetry-cpp; mesonFlags = [ (lib.mesonEnable "mimalloc" withMimalloc) (lib.mesonBool "plugin-c-api" withPluginCApi) - ]; + (lib.mesonEnable "sentry" enableSentry) + (lib.mesonEnable "otel" withOtel) + ] + ++ lib.optional enableSentry ( + lib.mesonOption "crashpad-handler" "${sentry-native}/bin/crashpad_handler" + ); postInstall = lib.optionalString stdenv.hostPlatform.isStatic '' mkdir -p $out/nix-support echo "file binary-dist $out/bin/nix" >> $out/nix-support/hydra-build-products ''; + # Fixes a problem with the "nix-cli-libcxxStdenv-static" package output. + # For some reason that is not clear, it is wanting to use libgcc_eh which is not available. + # Force this to be built with compiler-rt & libunwind over libgcc_eh works. + # Issue: https://github.com/NixOS/nixpkgs/issues/177129 + NIX_CFLAGS_COMPILE = lib.optionalString ( + stdenv.cc.isClang + && stdenv.hostPlatform.isStatic + && stdenv.cc.libcxx != null + && stdenv.cc.libcxx.isLLVM + ) "-rtlib=compiler-rt -unwindlib=libunwind"; + passthru = { exportsPluginCApi = withPluginCApi; }; diff --git a/src/nix/prefetch.cc b/src/nix/prefetch.cc index 2cd4377dd3c0..f254e556505d 100644 --- a/src/nix/prefetch.cc +++ b/src/nix/prefetch.cc @@ -13,6 +13,8 @@ #include "nix/util/environment-variables.hh" #include "nix/util/url.hh" #include "nix/store/path.hh" +#include "nix/util/override-provenance-source-accessor.hh" +#include "nix/fetchers/provenance.hh" #include "man-pages.hh" @@ -143,7 +145,15 @@ std::tuple prefetchFile( Activity act(*logger, lvlChatty, actUnknown, fmt("adding '%s' to the store", url.to_string())); - auto info = store->addToStoreSlow(name, makeFSSourceAccessor(tmpFile), method, hashAlgo, {}, expectedHash); + auto info = store->addToStoreSlow( + name, + {make_ref( + makeFSSourceAccessor(tmpFile), + unpack ? nullptr : std::make_shared(url.to_string()))}, + method, + hashAlgo, + {}, + expectedHash); storePath = info.path; assert(info.ca); hash = info.ca->hash; diff --git a/src/nix/profile-history.md b/src/nix/profile-history.md index f0bfe5037912..463e8624ff98 100644 --- a/src/nix/profile-history.md +++ b/src/nix/profile-history.md @@ -7,7 +7,7 @@ R""( ```console # nix profile history Version 508 (2020-04-10): - flake:nixpkgs#legacyPackages.x86_64-linux.awscli: ∅ -> 1.17.13 + flake:nixpkgs#legacyPackages.x86_64-linux.awscli: 1.17.13 added Version 509 (2020-05-16) <- 508: flake:nixpkgs#legacyPackages.x86_64-linux.awscli: 1.17.13 -> 1.18.211 @@ -20,7 +20,22 @@ between subsequent versions of a profile. It only shows top-level packages, not dependencies; for that, use [`nix profile diff-closures`](./nix3-profile-diff-closures.md). -The addition of a package to a profile is denoted by the string `∅ ->` -*version*, whereas the removal is denoted by *version* `-> ∅`. +The addition of a package to a profile is denoted by the string +*version* `added`, whereas the removal is denoted by *version* ` removed`. + +If a package was replaced by a different store path with the same +version (e.g. because it was rebuilt against a newer version of a +dependency), this is denoted by *version* `changed`. + +With `--show-source`, the locked flake reference of each package +is shown as well. For a package that was upgraded or changed, the old +and new references are shown: + + ```console + # nix profile history --show-source + ... + Version 510 (2020-05-20) <- 509: + flake:nixpkgs#legacyPackages.x86_64-linux.awscli: 1.18.211 changed (github:NixOS/nixpkgs/6fd8c75 -> github:NixOS/nixpkgs/ffc5e35) + ``` )"" diff --git a/src/nix/profile.cc b/src/nix/profile.cc index e63d0150c08e..0df0bfb54728 100644 --- a/src/nix/profile.cc +++ b/src/nix/profile.cc @@ -280,27 +280,49 @@ struct ProfileManifest return std::move(info.path); } - static void printDiff(const ProfileManifest & prev, const ProfileManifest & cur, std::string_view indent) + static void + printDiff(const ProfileManifest & prev, const ProfileManifest & cur, std::string_view indent, bool showSource) { auto i = prev.elements.begin(); auto j = cur.elements.begin(); bool changes = false; + auto flakeRef = [&](const ProfileElement & element) -> std::string { + if (!showSource || !element.source) + return ""; + return fmt(" (%s)", element.source->lockedRef.to_string(true)); + }; + + auto flakeRefChange = [&](const ProfileElement & e1, const ProfileElement & e2) -> std::string { + if (!showSource || !e1.source || !e2.source) + return ""; + return fmt(" (%s -> %s)", e1.source->lockedRef.to_string(true), e2.source->lockedRef.to_string(true)); + }; + while (i != prev.elements.end() || j != cur.elements.end()) { if (j != cur.elements.end() && (i == prev.elements.end() || i->first > j->first)) { - logger->cout("%s%s: ∅ -> %s", indent, j->second.identifier(), j->second.versions()); + auto & e = j->second; + logger->cout("%s%s: %s added%s", indent, e.identifier(), e.versions(), flakeRef(e)); changes = true; ++j; } else if (i != prev.elements.end() && (j == cur.elements.end() || i->first < j->first)) { - logger->cout("%s%s: %s -> ∅", indent, i->second.identifier(), i->second.versions()); + auto & e = i->second; + logger->cout("%s%s: %s removed%s", indent, e.identifier(), e.versions(), flakeRef(e)); changes = true; ++i; } else { - auto v1 = i->second.versions(); - auto v2 = j->second.versions(); + auto & e1 = i->second; + auto & e2 = j->second; + auto v1 = e1.versions(); + auto v2 = e2.versions(); if (v1 != v2) { - logger->cout("%s%s: %s -> %s", indent, i->second.identifier(), v1, v2); + logger->cout("%s%s: %s -> %s%s", indent, e1.identifier(), v1, v2, flakeRefChange(e1, e2)); + changes = true; + } else if (e1.storePaths != e2.storePaths) { + /* Same version, but a different store path (e.g. a + rebuild due to a dependency change). */ + logger->cout("%s%s: %s changed%s", indent, e1.identifier(), v1, flakeRefChange(e1, e2)); changes = true; } ++i; @@ -314,11 +336,11 @@ struct ProfileManifest }; static std::map>> -builtPathsPerInstallable(const std::vector, BuiltPathWithResult>> & builtPaths) +builtPathsPerInstallable(const std::vector & builtPaths) { std::map>> res; - for (auto & [installable, builtPath] : builtPaths) { - auto & r = res.insert({&*installable, + for (auto & b : builtPaths) { + auto & r = res.insert({&*b.installable, { {}, make_ref(), @@ -328,6 +350,7 @@ builtPathsPerInstallable(const std::vector, BuiltPath (e.g. meta.priority fields) if the installable returned multiple derivations. So pick one arbitrarily. FIXME: print a warning? */ + auto builtPath = b.getSuccess(); r.first.push_back(builtPath.path); r.second = builtPath.info; } @@ -364,8 +387,10 @@ struct CmdProfileAdd : InstallablesCommand, MixDefaultProfile { ProfileManifest manifest(*getEvalState(), *profile); - auto builtPaths = builtPathsPerInstallable( - Installable::build2(getEvalStore(), store, Realise::Outputs, installables, bmNormal)); + auto buildResults = Installable::build2(getEvalStore(), store, Realise::Outputs, installables, bmNormal); + Installable::throwBuildErrors(buildResults, *store); + + auto builtPaths = builtPathsPerInstallable(buildResults); for (auto & installable : installables) { ProfileElement element; @@ -745,11 +770,11 @@ struct CmdProfileUpgrade : virtual SourceExprCommand, MixProfileElementMatchers, this, getEvalState(), FlakeRef(element.source->originalRef), - "", + "." + element.source->attrPath, // absolute lookup element.source->outputs, - Strings{element.source->attrPath}, - Strings{}, - lockFlags); + StringSet{}, + lockFlags, + getDefaultFlakeSchemas()); auto derivedPaths = installable->toDerivedPaths(); if (derivedPaths.empty()) @@ -788,8 +813,10 @@ struct CmdProfileUpgrade : virtual SourceExprCommand, MixProfileElementMatchers, if (dryRun) return; - auto builtPaths = builtPathsPerInstallable( - Installable::build2(getEvalStore(), store, Realise::Outputs, installables, bmNormal)); + auto buildResults = Installable::build2(getEvalStore(), store, Realise::Outputs, installables, bmNormal); + Installable::throwBuildErrors(buildResults, *store); + + auto builtPaths = builtPathsPerInstallable(buildResults); for (size_t i = 0; i < installables.size(); ++i) { auto & installable = installables.at(i); @@ -884,6 +911,17 @@ struct CmdProfileDiffClosures : virtual StoreCommand, MixDefaultProfile struct CmdProfileHistory : virtual StoreCommand, EvalCommand, MixDefaultProfile { + bool showSource = false; + + CmdProfileHistory() + { + addFlag({ + .longName = "show-source", + .description = "Show the locked flake reference of each added, removed or changed package.", + .handler = {&showSource, true}, + }); + } + std::string description() override { return "show all versions of a profile"; @@ -917,7 +955,7 @@ struct CmdProfileHistory : virtual StoreCommand, EvalCommand, MixDefaultProfile std::put_time(std::gmtime(&gen.creationTime), "%Y-%m-%d"), prevGen ? fmt(" <- %d", prevGen->first.number) : ""); - ProfileManifest::printDiff(prevGen ? prevGen->second : ProfileManifest(), manifest, " "); + ProfileManifest::printDiff(prevGen ? prevGen->second : ProfileManifest(), manifest, " ", showSource); prevGen = {gen, std::move(manifest)}; } diff --git a/src/nix/provenance-show.md b/src/nix/provenance-show.md new file mode 100644 index 000000000000..526fbd54c8ea --- /dev/null +++ b/src/nix/provenance-show.md @@ -0,0 +1,29 @@ +R""( + +# Examples + +* Show the provenance of a store path: + + ```console + # nix provenance show /run/current-system + /nix/store/k145bdxhdb89i4fkvgdisdz1yh2wiymm-nixos-system-machine-25.05.20251210.d2b1213 + ← copied from cache.flakehub.com + ← built from derivation /nix/store/w3p3xkminq61hs00kihd34w1dglpj5s9-nixos-system-machine-25.05.20251210.d2b1213.drv (output out) on build-machine for x86_64-linux + ← instantiated from flake output github:my-org/my-repo/6b03eb949597fe96d536e956a2c14da9901dbd21?dir=machine#nixosConfigurations.machine.config.system.build.toplevel + ``` + +# Description + +Show the provenance chain of one or more store paths. For each store path, this displays where it came from: what binary cache it was copied from, what flake it was built from, and so on. + +The provenance chain shows the history of how the store path came to exist, including: + +- **Copied**: The path was copied from another Nix store, typically a binary cache. +- **Built**: The path was built from a derivation. +- **Flake evaluation**: The derivation was instantiated during the evaluation of a flake output. +- **Fetched**: The path was obtained by fetching a source tree. +- **Meta**: Metadata associated with the derivation. + +Note: if you want provenance in JSON format, use the `provenance` field returned by `nix path-info --json`. + +)"" diff --git a/src/nix/provenance-verify.md b/src/nix/provenance-verify.md new file mode 100644 index 000000000000..e05cb95bf249 --- /dev/null +++ b/src/nix/provenance-verify.md @@ -0,0 +1,21 @@ +R""( + +# Examples + +* Verify the provenance of a store path: + + ```console + # nix provenance verify /run/current-system + ``` + +# Description + +Verify the provenance of one or more store paths. This checks whether the store paths can be rebuilt from source. Specifically, it verifies the following: + +* That source trees can be fetched. +* That flake evaluations result in the instantiation of the desired store paths (most commonly, store derivations). +* That derivations can be successfully rebuilt, producing identical outputs. + +A non-zero exit code is returned if any of the verifications fail. + +)"" diff --git a/src/nix/provenance.cc b/src/nix/provenance.cc new file mode 100644 index 000000000000..438eb050a81b --- /dev/null +++ b/src/nix/provenance.cc @@ -0,0 +1,590 @@ +#include "nix/cmd/command.hh" +#include "nix/store/store-api.hh" +#include "nix/store/store-open.hh" +#include "nix/expr/provenance.hh" +#include "nix/store/provenance.hh" +#include "nix/flake/provenance.hh" +#include "nix/fetchers/provenance.hh" +#include "nix/util/provenance.hh" +#include "nix/util/json-utils.hh" +#include "nix/fetchers/fetch-to-store.hh" +#include "nix/util/exit.hh" +#include "nix/cmd/installable-flake.hh" +#include "nix/store/derivations.hh" +#include "nix/store/filetransfer.hh" +#include "nix/util/callback.hh" +#include "nix/util/terminal.hh" + +#include +#include +#include +#include +#include + +#define TAB " " + +using namespace nix; + +struct CmdProvenance : NixMultiCommand +{ + CmdProvenance() + : NixMultiCommand("provenance", RegisterCommand::getCommandsFor({"provenance"})) + { + } + + std::string description() override + { + return "query and check the provenance of store paths"; + } + + std::optional experimentalFeature() override + { + return Xp::Provenance; + } + + Category category() override + { + return catSecondary; + } +}; + +static auto rCmdProvenance = registerCommand("provenance"); + +struct CmdProvenanceShow : StorePathsCommand +{ + std::string description() override + { + return "show the provenance chain of store paths"; + } + + std::string doc() override + { + return +#include "provenance-show.md" + ; + } + + void displayProvenance(Store & store, const StorePath & path, std::shared_ptr provenance) + { + while (provenance) { + if (auto copied = std::dynamic_pointer_cast(provenance)) { + logger->cout("← copied from " ANSI_BOLD "%s" ANSI_NORMAL, copied->from); + provenance = copied->next; + } + + else if (auto build = std::dynamic_pointer_cast(provenance)) { + logger->cout( + "← built from derivation " ANSI_BOLD "%s" ANSI_NORMAL " (output " ANSI_BOLD "%s" ANSI_NORMAL + ") on " ANSI_BOLD "%s" ANSI_NORMAL " for " ANSI_BOLD "%s" ANSI_NORMAL, + store.printStorePath(build->drvPath), + build->output, + build->buildHost.value_or("unknown host").c_str(), + build->system); + for (auto & [tagName, tagValue] : build->tags) + logger->cout( + " tag " ANSI_BOLD "%s" ANSI_NORMAL ": %s", tagName, filterANSIEscapes(tagValue, true)); + provenance = build->next; + } + + else if (auto flake = std::dynamic_pointer_cast(provenance)) { + // Collapse subpath/tree provenance into the flake provenance for legibility. + auto next = flake->next; + CanonPath flakePath("/flake.nix"); + if (auto subpath = std::dynamic_pointer_cast(next)) { + next = subpath->next; + flakePath = subpath->subpath; + } + if (auto tree = std::dynamic_pointer_cast(next)) { + FlakeRef flakeRef( + fetchers::Input::fromAttrs(fetchSettings, fetchers::jsonToAttrs(*tree->attrs)), + std::string(flakePath.parent().value_or(CanonPath::root).rel())); + logger->cout( + "← %sinstantiated from %sflake output " ANSI_BOLD "%s#%s" ANSI_NORMAL, + flake->pure ? "" : ANSI_RED "impurely" ANSI_NORMAL " ", + flakeRef.input.isLocked(fetchSettings) ? "" : ANSI_RED "unlocked" ANSI_NORMAL " ", + flakeRef.to_string(), + flake->flakeOutput); + break; + } else { + logger->cout("← instantiated from flake output " ANSI_BOLD "%s" ANSI_NORMAL, flake->flakeOutput); + provenance = flake->next; + } + } + + else if (auto tree = std::dynamic_pointer_cast(provenance)) { + auto input = fetchers::Input::fromAttrs(fetchSettings, fetchers::jsonToAttrs(*tree->attrs)); + logger->cout( + "← from %stree " ANSI_BOLD "%s" ANSI_NORMAL, + input.isLocked(fetchSettings) ? "" : ANSI_RED "unlocked" ANSI_NORMAL " ", + input.to_string()); + break; + } + + else if (auto subpath = std::dynamic_pointer_cast(provenance)) { + logger->cout("← from file " ANSI_BOLD "%s" ANSI_NORMAL, subpath->subpath.abs()); + provenance = subpath->next; + } + + else if (auto drv = std::dynamic_pointer_cast(provenance)) { + logger->cout("← with derivation metadata"); + std::istringstream stream((*drv->meta).dump(2)); + for (std::string line; std::getline(stream, line);) { + logger->cout(" %s", line); + } + provenance = drv->next; + } + + else if (auto fetchurl = std::dynamic_pointer_cast(provenance)) { + logger->cout("← fetched from URL " ANSI_BOLD "%s" ANSI_NORMAL, fetchurl->url); + break; + } + + else { + // Unknown or unhandled provenance type + auto json = provenance->to_json(); + auto typeIt = json.find("type"); + if (typeIt != json.end() && typeIt->is_string()) + logger->cout("← " ANSI_RED "unknown provenance type '%s'" ANSI_NORMAL, typeIt->get()); + else + logger->cout("← " ANSI_RED "unknown provenance type" ANSI_NORMAL); + break; + } + } + } + + void run(ref store, StorePaths && storePaths) override + { + bool first = true; + + for (auto & storePath : storePaths) { + auto info = store->queryPathInfo(storePath); + if (!first) + logger->cout(""); + first = false; + logger->cout(ANSI_BOLD "%s" ANSI_NORMAL, store->printStorePath(info->path)); + + if (info->provenance) + displayProvenance(*store, storePath, info->provenance); + else + logger->cout(ANSI_RED " (no provenance information available)" ANSI_NORMAL); + } + } +}; + +static auto rCmdProvenanceShow = registerCommand2({"provenance", "show"}); + +/** + * A wrapper around an arbitrary store that intercepts `addToStore()` + * and `addToStoreFromDump()` calls to keep track of added paths. + */ +struct TrackingStore : public Store +{ + ref next; + boost::concurrent_flat_set instantiatedPaths; + + TrackingStore(ref next) + : Store(next->config) + , next(next) + { + } + + void addToStore(const ValidPathInfo & info, Source & narSource, RepairFlag repair, CheckSigsFlag checkSigs) override + { + next->addToStore(info, narSource, repair, checkSigs); + instantiatedPaths.insert(info.path); + // FIXME: we should really just disable the path info cache, since the underlying store already does caching. + invalidatePathInfoCacheFor(info.path); + } + + StorePath addToStore( + std::string_view name, + const SourcePath & path, + ContentAddressMethod method, + HashAlgorithm hashAlgo, + const StorePathSet & references, + PathFilter & filter, + RepairFlag repair) override + { + auto storePath = next->addToStore(name, path, method, hashAlgo, references, filter, repair); + instantiatedPaths.insert(storePath); + invalidatePathInfoCacheFor(storePath); + return storePath; + } + + StorePath addToStoreFromDump( + Source & dump, + std::string_view name, + FileSerialisationMethod dumpMethod, + ContentAddressMethod hashMethod, + HashAlgorithm hashAlgo, + const StorePathSet & references, + RepairFlag repair, + std::shared_ptr provenance) override + { + auto storePath = + next->addToStoreFromDump(dump, name, dumpMethod, hashMethod, hashAlgo, references, repair, provenance); + instantiatedPaths.insert(storePath); + invalidatePathInfoCacheFor(storePath); + return storePath; + } + + void queryPathInfoUncached( + const StorePath & path, Callback> callback) noexcept override + { + try { + auto info = std::make_shared(*next->queryPathInfo(path)); + /* The evaluator checks whether a path already exists + before adding it to the store. So if the underlying + store has the path, count it as instantiated, since + the evaluator would otherwise have written it. */ + instantiatedPaths.insert(path); + callback(std::move(info)); + } catch (InvalidPath &) { + callback(nullptr); + } catch (...) { + callback.rethrow(); + } + } + + void queryRealisationUncached( + const DrvOutput & output, Callback> callback) noexcept override + { + next->queryRealisation(output, std::move(callback)); + } + + std::optional queryPathFromHashPart(const std::string & hashPart) override + { + return next->queryPathFromHashPart(hashPart); + } + + void registerDrvOutput(const Realisation & output) override + { + next->registerDrvOutput(output); + } + + ref getFSAccessor(bool requireValidPath) override + { + return next->getFSAccessor(requireValidPath); + } + + std::shared_ptr getFSAccessor(const StorePath & path, bool requireValidPath) override + { + return next->getFSAccessor(path, requireValidPath); + } + + std::optional isTrustedClient() override + { + return next->isTrustedClient(); + } + + void anchor() override {} +}; + +struct CmdProvenanceVerify : StorePathsCommand +{ + bool noRebuild = false; + + CmdProvenanceVerify() + { + addFlag({ + .longName = "no-rebuild", + .description = "Skip rebuilding derivations to verify reproducibility.", + .handler = {&noRebuild, true}, + }); + } + + std::string description() override + { + return "verify the provenance of store paths"; + } + + std::string doc() override + { + return +#include "provenance-verify.md" + ; + } + + bool verifySourcePath(Store & store, const StorePath & expectedPath, const SourcePath & sourcePath) + { + auto computedPath = fetchToStore2(fetchSettings, store, sourcePath, FetchMode::Copy, expectedPath.name()).first; + if (computedPath != expectedPath) { + logger->cout( + "❌ " ANSI_RED "store path mismatch for source '%s': expected '%s' but got '%s'" ANSI_NORMAL, + sourcePath.to_string(), + store.printStorePath(expectedPath), + store.printStorePath(computedPath)); + return false; + } else { + logger->cout("✅ verified store path for source '%s'", sourcePath.to_string()); + return true; + } + } + + using CheckResult = std::variant< + std::pair>, + std::pair, + std::monostate>; + + std::pair + verify(Store & store, std::optional path, std::shared_ptr provenance) + { + if (auto copied = std::dynamic_pointer_cast(provenance)) { + if (!path) { + logger->cout("❌ " ANSI_RED "cannot verify copied provenance without a store path" ANSI_NORMAL); + return {false, std::monostate{}}; + } + bool success = true; + auto fromStore = openStore(copied->from); + auto localInfo = store.queryPathInfo(*path); + auto fromInfo = fromStore->queryPathInfo(*path); + if (localInfo->narHash != fromInfo->narHash) { + logger->cout( + "❌ " ANSI_RED "NAR hash mismatch in origin store '%s': should be '%s' but is '%s'" ANSI_NORMAL, + copied->from, + localInfo->narHash.to_string(HashFormat::SRI, true), + fromInfo->narHash.to_string(HashFormat::SRI, true)); + success = false; + } else + logger->cout("✅ verified NAR hash in origin store '%s'", copied->from); + auto [nextSuccess, result] = verify(store, path, copied->next); + return {success && nextSuccess, std::move(result)}; + } + + else if (auto build = std::dynamic_pointer_cast(provenance)) { + auto success = verify(store, build->drvPath, build->next).first; + + // Verify that `path` is the expected output of the derivation. + auto outputMap = store.queryPartialDerivationOutputMap(build->drvPath); + auto it = outputMap.find(build->output); + if (it == outputMap.end()) { + logger->cout( + "❌ " ANSI_RED "derivation '%s' does not have expected output '%s'" ANSI_NORMAL, + store.printStorePath(build->drvPath), + build->output); + return {false, std::monostate{}}; + } else if (!it->second) { + // Note: this is not an error, should we even print a message? + logger->cout( + "❓ output '%s' of derivation '%s' is not statically known", + build->output, + store.printStorePath(build->drvPath)); + } else if (*it->second != path) { + logger->cout( + "❌ " ANSI_RED "output '%s' of derivation '%s' is '%s', expected '%s'" ANSI_NORMAL, + build->output, + store.printStorePath(build->drvPath), + store.printStorePath(*it->second), + store.printStorePath(*path)); + return {false, std::monostate{}}; + } + + // Do a check rebuild to verify that the derivation + // produces the same output. + if (noRebuild) { + logger->cout( + "⏭️ skipped rebuild of derivation '%s^%s'", store.printStorePath(build->drvPath), build->output); + } else { + try { + store.buildPaths( + {DerivedPath::Built{ + .drvPath = make_ref(SingleDerivedPath::Opaque{build->drvPath}), + .outputs = OutputsSpec::Names{build->output}, + }}, + bmCheck); + logger->cout("✅ rebuilt derivation '%s^%s'", store.printStorePath(build->drvPath), build->output); + } catch (Error & e) { + logger->cout( + "❌ " ANSI_RED "rebuild of derivation '%s^%s' failed: %s" ANSI_NORMAL, + store.printStorePath(build->drvPath), + build->output, + e.what()); + success = false; + } + } + + return {success, std::monostate{}}; + } + + else if (auto flake = std::dynamic_pointer_cast(provenance)) { + // Fetch the flake source. + auto [success, _res] = verify(store, {}, flake->next); + + auto res = std::get_if>(&_res); + if (!res) + return {false, std::monostate{}}; + + // Evaluate the flake output. + flake::LockFlags lockFlags{ + .updateLockFile = false, + .failOnUnlocked = true, + .useRegistries = false, + .allowUnlocked = false, + }; + + if (res->second.path.baseName() != "flake.nix") { + logger->cout( + "❌ " ANSI_RED "expected flake source to be a 'flake.nix' file, but got '%s'" ANSI_NORMAL, + res->second.path.abs()); + return {false, std::monostate{}}; + } + + auto trackingStore = make_ref(getEvalStore()); + + auto evalState = + ref(std::allocate_shared( + traceable_allocator(), + LookupPath{}, + ref(trackingStore), + fetchSettings, + evalSettings, + getStore())); + + InstallableFlake installable{ + nullptr, + evalState, + FlakeRef{std::move(res->first), std::string(res->second.path.parent().value().rel())}, + "." + flake->flakeOutput, + ExtendedOutputsSpec::Default{}, // FIXME: record this in the provenance? + {}, + lockFlags, + {}}; + + // We have to disable the eval cache to ensure that we see which store paths get instantiated. + installable.useEvalCache = false; + + installable.toDerivedPaths(); + + evalState->waitForAllPaths(); + + logger->cout("✅ evaluated '%s#%s'", installable.flakeRef.to_string(true), flake->flakeOutput); + + if (path) { + if (!trackingStore->instantiatedPaths.contains(*path)) { + logger->cout( + "❌ " ANSI_RED "evaluation did not re-instantiate path '%s'" ANSI_NORMAL, + store.printStorePath(*path)); + return {false, std::monostate{}}; + } + + logger->cout("✅ re-instantiated path '%s'", store.printStorePath(*path)); + } + + return {success, std::monostate{}}; + } + + else if (auto tree = std::dynamic_pointer_cast(provenance)) { + auto input = fetchers::Input::fromAttrs(fetchSettings, fetchers::jsonToAttrs(*tree->attrs)); + try { + auto [accessor, final] = input.getAccessor(fetchSettings, store); + if (!input.isLocked(fetchSettings)) + logger->cout("❓ fetched tree '%s', but it's unlocked", input.to_string()); + else + // FIXME: check NAR hash? + logger->cout("✅ fetched tree '%s'", input.to_string()); + + bool success = !path || verifySourcePath(store, *path, SourcePath(accessor, CanonPath::root)); + + return {success, std::make_pair(std::move(final), accessor)}; + } catch (Error & e) { + logger->cout("❌ " ANSI_RED "failed to fetch tree '%s': %s" ANSI_NORMAL, input.to_string(), e.what()); + return {false, std::monostate{}}; + } + } + + else if (auto subpath = std::dynamic_pointer_cast(provenance)) { + auto [success, result] = verify(store, {}, subpath->next); + if (auto p = std::get_if>>(&result)) { + + auto sourcePath = SourcePath(p->second, subpath->subpath); + + if (path && !verifySourcePath(store, *path, sourcePath)) + success = false; + + return {success, std::make_pair(std::move(p->first), std::move(sourcePath))}; + } else + return {false, std::monostate{}}; + } + + else if (auto drv = std::dynamic_pointer_cast(provenance)) + return verify(store, path, drv->next); + + else if (auto fetchurl = std::dynamic_pointer_cast(provenance)) { + if (!path) + return {false, std::monostate{}}; + + auto info = store.queryPathInfo(*path); + + if (!info->ca) { + logger->cout( + "❌ " ANSI_RED "cannot verify URL '%s' without a content address for path '%s'" ANSI_NORMAL, + fetchurl->url, + store.printStorePath(*path)); + return {false, std::monostate{}}; + } + + if (info->ca->method != ContentAddressMethod::Raw::Flat) { + logger->cout( + "❌ " ANSI_RED + "cannot verify URL '%s' with unsupported content address method for path '%s'" ANSI_NORMAL, + fetchurl->url, + store.printStorePath(*path)); + return {false, std::monostate{}}; + } + + HashSink hashSink{info->ca->hash.algo}; + FileTransferRequest req(fetchurl->url); + req.decompress = false; + getFileTransfer()->download(std::move(req), hashSink); + auto hash = hashSink.finish().hash; + + if (hash != info->ca->hash) { + logger->cout( + "❌ " ANSI_RED "hash mismatch for URL '%s': expected '%s' but got '%s'" ANSI_NORMAL, + fetchurl->url, + info->ca->hash.to_string(HashFormat::SRI, true), + hash.to_string(HashFormat::SRI, true)); + return {false, std::monostate{}}; + } + + logger->cout("✅ verified hash of URL '%s'", fetchurl->url); + return {true, std::monostate{}}; + } + + else if (!provenance) { + logger->cout("❓ " ANSI_RED "missing further provenance" ANSI_NORMAL); + return {false, std::monostate{}}; + } + + else { + logger->cout("❓ " ANSI_RED "unknown provenance type" ANSI_NORMAL); + return {false, std::monostate{}}; + } + } + + void run(ref store, StorePaths && storePaths) override + { + bool first = true; + bool success = true; + + for (auto & storePath : storePaths) { + auto info = store->queryPathInfo(storePath); + if (!first) + logger->cout(""); + first = false; + logger->cout(ANSI_BOLD "%s" ANSI_NORMAL, store->printStorePath(info->path)); + + if (info->provenance) + success &= verify(*store, storePath, info->provenance).first; + else { + logger->cout(ANSI_RED " (no provenance information available)" ANSI_NORMAL); + success = false; + } + } + + if (!success) + throw Exit(1); + } +}; + +static auto rCmdProvenanceVerify = registerCommand2({"provenance", "verify"}); diff --git a/src/nix/ps.cc b/src/nix/ps.cc new file mode 100644 index 000000000000..9ae9d97bf98c --- /dev/null +++ b/src/nix/ps.cc @@ -0,0 +1,146 @@ +#include "nix/cmd/command.hh" +#include "nix/main/common-args.hh" +#include "nix/main/shared.hh" +#include "nix/store/store-api.hh" +#include "nix/store/store-cast.hh" +#include "nix/store/active-builds.hh" +#include "nix/util/table.hh" +#include "nix/util/terminal.hh" + +#include + +using namespace nix; + +struct CmdPs : MixJSON, StoreCommand +{ + std::string description() override + { + return "list active builds"; + } + + Category category() override + { + return catUtility; + } + + std::string doc() override + { + return +#include "ps.md" + ; + } + + void run(ref store) override + { + auto & tracker = require(*store); + + auto builds = tracker.queryActiveBuilds(); + + if (json) { + printJSON(nlohmann::json(builds)); + return; + } + + if (builds.empty()) { + notice("No active builds."); + return; + } + + /* Helper to format user info: show name if available, else UID */ + auto formatUser = [](const UserInfo & user) -> std::string { + return user.name ? *user.name : std::to_string(user.uid); + }; + + Table table; + + /* Add column headers. */ + table.push_back({{"USER"}, {"PID"}, {"CPU", TableCell::Alignment::Right}, {"DERIVATION/COMMAND"}}); + + for (const auto & build : builds) { + /* Calculate CPU time - use cgroup stats if available, otherwise sum process times. */ + std::chrono::microseconds cpuTime = build.utime && build.stime ? *build.utime + *build.stime : [&]() { + std::chrono::microseconds total{0}; + for (const auto & process : build.processes) + total += process.utime.value_or(std::chrono::microseconds(0)) + + process.stime.value_or(std::chrono::microseconds(0)) + + process.cutime.value_or(std::chrono::microseconds(0)) + + process.cstime.value_or(std::chrono::microseconds(0)); + return total; + }(); + + /* Add build summary row. */ + table.push_back( + {formatUser(build.mainUser), + std::to_string(build.mainPid), + {fmt("%.1fs", + std::chrono::duration_cast>(cpuTime) + .count()), + TableCell::Alignment::Right}, + fmt(ANSI_BOLD "%s" ANSI_NORMAL " (wall=%ds)", + store->printStorePath(build.derivation), + time(nullptr) - build.startTime)}); + + if (build.processes.empty()) { + table.push_back( + {formatUser(build.mainUser), + std::to_string(build.mainPid), + {"", TableCell::Alignment::Right}, + fmt("%s" ANSI_ITALIC "(no process info)" ANSI_NORMAL, treeLast)}); + } else { + /* Recover the tree structure of the processes. */ + std::set pids; + for (auto & process : build.processes) + pids.insert(process.pid); + + using Processes = std::set; + std::map children; + Processes rootProcesses; + for (auto & process : build.processes) { + if (pids.contains(process.parentPid)) + children[process.parentPid].insert(&process); + else + rootProcesses.insert(&process); + } + + /* Render the process tree. */ + [&](this auto const & visit, const Processes & processes, std::string_view prefix) -> void { + for (const auto & [n, process] : enumerate(processes)) { + bool last = n + 1 == processes.size(); + + // Format CPU time if available + std::string cpuInfo; + if (process->utime || process->stime || process->cutime || process->cstime) { + auto totalCpu = process->utime.value_or(std::chrono::microseconds(0)) + + process->stime.value_or(std::chrono::microseconds(0)) + + process->cutime.value_or(std::chrono::microseconds(0)) + + process->cstime.value_or(std::chrono::microseconds(0)); + auto totalSecs = + std::chrono::duration_cast>( + totalCpu) + .count(); + cpuInfo = fmt("%.1fs", totalSecs); + } + + // Format argv with tree structure + auto argv = concatStringsSep( + " ", tokenizeString>(concatStringsSep(" ", process->argv))); + + table.push_back( + {formatUser(process->user), + std::to_string(process->pid), + {cpuInfo, TableCell::Alignment::Right}, + fmt("%s%s%s", prefix, last ? treeLast : treeConn, argv)}); + + visit(children[process->pid], last ? prefix + treeNull : prefix + treeLine); + } + }(rootProcesses, ""); + } + } + + auto width = isTTY() && isatty(STDOUT_FILENO) ? getWindowWidth() : std::numeric_limits::max(); + + printTable(std::cout, table, width); + } +}; + +static auto rCmdPs = registerCommand2({"ps"}); diff --git a/src/nix/ps.md b/src/nix/ps.md new file mode 100644 index 000000000000..e48a308e6965 --- /dev/null +++ b/src/nix/ps.md @@ -0,0 +1,27 @@ +R"( + +# Examples + +* Show all active builds: + + ```console + # nix ps + USER PID CPU DERIVATION/COMMAND + nixbld11 3534394 110.2s /nix/store/lzvdxlbr6xjd9w8py4nd2y2nnqb9gz7p-nix-util-tests-3.13.2.drv (wall=8s) + nixbld11 3534394 0.8s └───bash -e /nix/store/jwqf79v5p51x9mv8vx20fv9mzm2x7kig-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02 + nixbld11 3534751 36.3s └───ninja -j24 + nixbld11 3535637 0.0s ├───/nix/store/0v2jfvx71l1zn14l97pznvbqnhiq3pyd-gcc-14.3.0/bin/g++ -fPIC -fstack-clash-protection -O2 -U_ + nixbld11 3535639 0.1s │ └───/nix/store/0v2jfvx71l1zn14l97pznvbqnhiq3pyd-gcc-14.3.0/libexec/gcc/x86_64-unknown-linux-gnu/14.3. + nixbld11 3535658 0.0s └───/nix/store/0v2jfvx71l1zn14l97pznvbqnhiq3pyd-gcc-14.3.0/bin/g++ -fPIC -fstack-clash-protection -O2 -U_ + nixbld1 3534377 1.8s /nix/store/nh2dx9cqcy9lw4d4rvd0dbsflwdsbzdy-patchelf-0.18.0.drv (wall=5s) + nixbld1 3534377 1.8s └───bash -e /nix/store/xk05lkk4ij6pc7anhdbr81appiqbcb01-default-builder.sh + nixbld1 3535074 0.0s └───/nix/store/21ymxxap3y8hb9ijcfah8ani9cjpv8m6-bash-5.2p37/bin/bash ./configure --disable-dependency-trackin + ``` + +# Description + +This command lists all currently running Nix builds. +For each build, it shows the derivation path and the main process ID. +On Linux and macOS, it also shows the child processes of each build. + +)" diff --git a/src/nix/repl.cc b/src/nix/repl.cc index bc7cdedfdce1..ef56e56ab6ee 100644 --- a/src/nix/repl.cc +++ b/src/nix/repl.cc @@ -44,9 +44,9 @@ struct CmdRepl : RawInstallablesCommand std::vector files; - Strings getDefaultFlakeAttrPaths() override + StringSet getRoles() override { - return {""}; + return {"nix-repl"}; } bool forceImpureByDefault() override diff --git a/src/nix/repl.md b/src/nix/repl.md index 32c08e24b240..e608dabf6f9c 100644 --- a/src/nix/repl.md +++ b/src/nix/repl.md @@ -36,7 +36,7 @@ R""( Loading Installable ''... Added 1 variables. - # nix repl --extra-experimental-features 'flakes' nixpkgs + # nix repl nixpkgs Loading Installable 'flake:nixpkgs#'... Added 5 variables. diff --git a/src/nix/run.cc b/src/nix/run.cc index d9056ba91df1..fc763802cfee 100644 --- a/src/nix/run.cc +++ b/src/nix/run.cc @@ -43,7 +43,9 @@ void execProgramInStore( std::optional system, std::optional env) { + /* We're about to exec, so end and export any telemetry. */ logger->stop(); + logger->flush(); char ** envp; Strings envStrs; @@ -129,23 +131,9 @@ struct CmdRun : InstallableValueCommand, MixEnvironment ; } - Strings getDefaultFlakeAttrPaths() override + StringSet getRoles() override { - Strings res{ - "apps." + settings.thisSystem.get() + ".default", - "defaultApp." + settings.thisSystem.get(), - }; - for (auto & s : SourceExprCommand::getDefaultFlakeAttrPaths()) - res.push_back(s); - return res; - } - - Strings getDefaultFlakeAttrPathPrefixes() override - { - Strings res{"apps." + settings.thisSystem.get() + "."}; - for (auto & s : SourceExprCommand::getDefaultFlakeAttrPathPrefixes()) - res.push_back(s); - return res; + return {"nix-run"}; } void run(ref store, ref installable) override diff --git a/src/nix/search.cc b/src/nix/search.cc index d7616bcfb1d0..074f01690203 100644 --- a/src/nix/search.cc +++ b/src/nix/search.cc @@ -10,6 +10,7 @@ #include "nix/expr/attr-path.hh" #include "nix/util/hilite.hh" #include "nix/util/strings-inline.hh" +#include "nix/expr/parallel-eval.hh" #include #include @@ -55,9 +56,9 @@ struct CmdSearch : InstallableValueCommand, MixJSON ; } - Strings getDefaultFlakeAttrPaths() override + StringSet getRoles() override { - return {"packages." + settings.thisSystem.get(), "legacyPackages." + settings.thisSystem.get()}; + return {"nix-search"}; } void run(ref store, ref installable) override @@ -83,11 +84,14 @@ struct CmdSearch : InstallableValueCommand, MixJSON auto state = getEvalState(); - std::optional jsonOut; + std::optional> jsonOut; if (json) - jsonOut = json::object(); + jsonOut.emplace(json::object()); - uint64_t results = 0; + std::atomic results = 0; + std::atomic total = 0; + + FutureVector futures(*state->executor); std::function visit; @@ -95,18 +99,33 @@ struct CmdSearch : InstallableValueCommand, MixJSON auto attrPathS = state->symbols.resolve({attrPath}); auto attrPathStr = attrPath.to_string(*state); + /* Activity act(*logger, lvlInfo, actUnknown, fmt("evaluating '%s'", attrPathStr)); + */ try { auto recurse = [&]() { + Executor::WorkItems work; for (const auto & attr : cursor.getAttrs()) { auto cursor2 = cursor.getAttr(state->symbols[attr]); auto attrPath2(attrPath); attrPath2.push_back(attr); - visit(*cursor2, attrPath2, false); + state->addWork( + work, + std::string_view(state->symbols[attr]).find("Packages") != std::string_view::npos ? 0 : 2, + [cursor2, attrPath2, visit]() { visit(*cursor2, attrPath2, false); }); + /* Spawn incrementally rather than after the whole + enumeration, so that idle worker threads can start + on the first attributes while we're still + enumerating the rest. */ + if (work.size() >= 256) + futures.spawn(std::exchange(work, {})); } + futures.spawn(std::move(work)); }; if (cursor.isDerivation()) { + total++; + DrvName name(cursor.getAttr(state->s.name)->getString()); auto aMeta = cursor.maybeGetAttr(state->s.meta); @@ -147,21 +166,21 @@ struct CmdSearch : InstallableValueCommand, MixJSON if (found) { results++; if (json) { - (*jsonOut)[attrPathStr] = { + (*jsonOut->lock())[attrPathStr] = { {"pname", name.name}, {"version", name.version}, {"description", description}, }; } else { - if (results > 1) - logger->cout(""); - logger->cout( - "* %s%s", - wrap("\e[0;1m", hiliteMatches(attrPathStr, attrPathMatches, ANSI_GREEN, "\e[0;1m")), - optionalBracket(" (", name.version, ")")); + auto out = + fmt("%s* %s%s", + results > 1 ? "\n" : "", + wrap("\e[0;1m", hiliteMatches(attrPathStr, attrPathMatches, ANSI_GREEN, "\e[0;1m")), + optionalBracket(" (", name.version, ")")); if (description != "") - logger->cout( - " %s", hiliteMatches(description, descriptionMatches, ANSI_GREEN, ANSI_NORMAL)); + out += fmt( + "\n %s", hiliteMatches(description, descriptionMatches, ANSI_GREEN, ANSI_NORMAL)); + logger->cout(out); } } } @@ -186,14 +205,20 @@ struct CmdSearch : InstallableValueCommand, MixJSON } }; - for (auto & cursor : installable->getCursors(*state)) - visit(*cursor, cursor->getAttrPath(), true); + Executor::WorkItems work; + for (auto & cursor : installable->getCursors(*state, false)) + state->addWork(work, 1, [cursor, visit]() { visit(*cursor, cursor->getAttrPath(), true); }); + + futures.spawn(std::move(work)); + futures.finishAll(); if (json) - printJSON(*jsonOut); + printJSON(*(jsonOut->lock())); if (!json && !results) throw Error("no results for the given search term(s)!"); + + notice("Found %d matching packages out of %d.", results, total); } }; diff --git a/src/nix/serve.cc b/src/nix/serve.cc new file mode 100644 index 000000000000..fd9810fa48a4 --- /dev/null +++ b/src/nix/serve.cc @@ -0,0 +1,294 @@ +#include "nix/cmd/command.hh" +#include "nix/util/file-system.hh" +#include "nix/util/serialise.hh" +#include "nix/util/signals.hh" +#include "nix/util/deleter.hh" +#include "nix/store/nar-info.hh" +#include "nix/store/binary-cache-store.hh" +#include "nix/store/log-store.hh" +#include "nix/util/environment-variables.hh" + +#include +#include + +#include +#include + +using namespace nix; + +using Response = std::unique_ptr>; + +struct CmdServe : StoreCommand +{ + uint16_t port = 8080; + std::string listenAddress = "127.0.0.1"; + std::optional priority; + std::optional portFile; + + CmdServe() + { + addFlag({ + .longName = "port", + .shortName = 'p', + .description = "Port to listen on (default: 8080). Use 0 to dynamically allocate a free port.", + .labels = {"port"}, + .handler = {&port}, + }); + addFlag({ + .longName = "listen-address", + .description = "IP address to listen on (default: `127.0.0.1`). " + "Use `0.0.0.0` or `::` to listen on all interfaces.", + .labels = {"address"}, + .handler = {&listenAddress}, + }); + addFlag({ + .longName = "port-file", + .description = "Write the bound port number to this file.", + .labels = {"path"}, + .handler = {[this](std::string s) { portFile = s; }}, + }); + addFlag({ + .longName = "priority", + .description = "Priority of this cache (overrides the store's default).", + .labels = {"priority"}, + .handler = {[this](std::string s) { priority = std::stoi(s); }}, + }); + } + + std::string description() override + { + return "serve a Nix store as a HTTP binary cache"; + } + + Category category() override + { + return catSecondary; + } + + std::string doc() override + { + return +#include "serve.md" + ; + } + + MHD_Result + handleRequest(Store & store, MHD_Connection * connection, const std::string & url, std::string_view method) + try { + std::string clientAddr = "unknown"; + if (auto * info = MHD_get_connection_info(connection, MHD_CONNECTION_INFO_CLIENT_ADDRESS)) { + char buf[INET6_ADDRSTRLEN]; + auto * addr = info->client_addr; + const void * src = addr->sa_family == AF_INET6 ? (void *) &((sockaddr_in6 *) addr)->sin6_addr + : (void *) &((sockaddr_in *) addr)->sin_addr; + if (inet_ntop(addr->sa_family, src, buf, sizeof(buf))) + clientAddr = buf; + } + + notice("request: client=%s, method=%s, url=%s", clientAddr, method, url); + + Response response; + + auto notFound = [&] { + static constexpr std::string_view body = "404 not found\n"; + response.reset(MHD_create_response_from_buffer(body.size(), (void *) body.data(), MHD_RESPMEM_PERSISTENT)); + return MHD_queue_response(connection, MHD_HTTP_NOT_FOUND, response.get()); + }; + + static const std::regex narInfoUrlRegex{R"(^/([0-9a-z]+)\.narinfo$)"}; + static const std::regex narUrlRegex{R"(^/nar/([0-9a-z]+)-([0-9a-z]+)\.nar$)"}; + static const std::regex logUrlRegex{R"(^/log/([0-9a-z]+-[0-9a-zA-Z+\-._?=]+)$)"}; + + if (method != MHD_HTTP_METHOD_GET && method != MHD_HTTP_METHOD_HEAD) { + std::string_view body = "405 method not allowed\n"; + response.reset(MHD_create_response_from_buffer(body.size(), (void *) body.data(), MHD_RESPMEM_PERSISTENT)); + MHD_add_response_header(response.get(), "Allow", "GET, HEAD"); + return MHD_queue_response(connection, MHD_HTTP_METHOD_NOT_ALLOWED, response.get()); + } + + if (url == "/nix-cache-info") { + auto body = std::make_unique( + "StoreDir: " + store.storeDir + "\n" + "WantMassQuery: " + (store.config.wantMassQuery ? "1" : "0") + "\n" + "Priority: " + std::to_string(priority.value_or(store.config.priority)) + "\n"); + response.reset(MHD_create_response_from_buffer(body->size(), body->data(), MHD_RESPMEM_MUST_COPY)); + MHD_add_response_header(response.get(), "Content-Type", "text/x-nix-cache-info"); + + } else if (std::smatch m; std::regex_match(url, m, narInfoUrlRegex)) { + auto hashPart = m[1].str(); + auto path = store.queryPathFromHashPart(hashPart); + if (!path) + return notFound(); + + auto info = store.queryPathInfo(*path); + NarInfo ni(*info); + ni.compression = CompressionAlgo::none; + // FIXME: would be nicer to use just the NAR hash, but we can't look up NARs by NAR hash. + ni.url = "nar/" + std::string(info->path.hashPart()) + "-" + + info->narHash.to_string(HashFormat::Nix32, false) + ".nar"; + ni.fileSize = info->narSize; + auto body = ni.to_string(store); + response.reset(MHD_create_response_from_buffer(body.size(), body.data(), MHD_RESPMEM_MUST_COPY)); + MHD_add_response_header(response.get(), "Content-Type", "text/x-nix-narinfo"); + + } else if (std::smatch m; std::regex_match(url, m, narUrlRegex)) { + auto hashPart = m[1].str(); + auto expectedNarHash = m[2].str(); + auto path = store.queryPathFromHashPart(hashPart); + if (!path) + return notFound(); + + auto info = store.queryPathInfo(*path); + if (info->narHash.to_string(HashFormat::Nix32, false) != expectedNarHash) + return notFound(); + + struct State + { + std::unique_ptr source; + std::optional firstByte; + }; + + auto state = std::make_unique(); + + state->source = sinkToSource([&store, p = *path](Sink & sink) { store.narFromPath(p, sink); }); + + // Read the first byte so we can return a 404 if the store path / NAR doesn't exist anymore. + try { + char firstByte; + if (state->source->read(&firstByte, 1)) + state->firstByte = firstByte; + } catch (InvalidPath &) { + return notFound(); + } catch (SubstituteGone &) { + return notFound(); + } catch (EndOfFile &) { + } + + // Stream the NAR. + auto reader = [](void * cls, uint64_t pos, char * buf, size_t max) -> ssize_t { + auto & state = *static_cast(cls); + size_t read = 0; + if (pos == 0 && state.firstByte) { + *buf++ = *state.firstByte; + pos++; + max--; + read++; + state.firstByte.reset(); + } + + static bool truncate = getEnv("_NIX_TEST_NIX_SERVE_TRUNCATE_NAR").value_or("") == "1"; + static std::atomic truncatePoint{200000}; + if (truncate && pos > truncatePoint) { + truncatePoint += 200000; + return MHD_CONTENT_READER_END_WITH_ERROR; + } + + try { + read += state.source->read(buf, max); + return read; + } catch (EndOfFile &) { + return MHD_CONTENT_READER_END_OF_STREAM; + } catch (...) { + return MHD_CONTENT_READER_END_WITH_ERROR; + } + }; + + auto freeCb = [](void * cls) { delete static_cast(cls); }; + + response.reset(MHD_create_response_from_callback(MHD_SIZE_UNKNOWN, 65536, reader, state.release(), freeCb)); + MHD_add_response_header(response.get(), "Content-Type", "application/x-nix-nar"); + + } else if (std::smatch m; std::regex_match(url, m, logUrlRegex)) { + auto * logStore = dynamic_cast(&store); + if (!logStore) + return notFound(); + + StorePath path{m[1].str()}; + + auto log = logStore->getBuildLog(path); + if (!log) + return notFound(); + + response.reset(MHD_create_response_from_buffer(log->size(), log->data(), MHD_RESPMEM_MUST_COPY)); + MHD_add_response_header(response.get(), "Content-Type", "text/plain; charset=utf-8"); + } else + return notFound(); + + return MHD_queue_response(connection, MHD_HTTP_OK, response.get()); + + } catch (const Error & e) { + auto body = fmt("500 Internal Server Error\n\nError: %s", e.message()); + Response response; + response.reset(MHD_create_response_from_buffer(body.size(), body.data(), MHD_RESPMEM_MUST_COPY)); + MHD_add_response_header(response.get(), "Content-Type", "text/plain"); + return MHD_queue_response(connection, MHD_HTTP_INTERNAL_SERVER_ERROR, response.get()); + } + + void run(ref store) override + { + struct Ctx + { + Store & store; + CmdServe & cmd; + }; + + Ctx ctx{*store, *this}; + + auto handler = [](void * cls, + MHD_Connection * connection, + const char * url, + const char * method, + const char * version, + const char * upload_data, + size_t * upload_data_size, + void ** con_cls) -> MHD_Result { + auto & ctx = *static_cast(cls); + auto & store = ctx.store; + auto & cmd = ctx.cmd; + return cmd.handleRequest(store, connection, std::string(url), method); + }; + + sockaddr_in addr4{}; + sockaddr_in6 addr6{}; + const sockaddr * sockAddr = nullptr; + unsigned int flags = MHD_USE_INTERNAL_POLLING_THREAD; + + if (inet_pton(AF_INET, listenAddress.c_str(), &addr4.sin_addr) == 1) { + addr4.sin_family = AF_INET; + addr4.sin_port = htons(port); + sockAddr = (const sockaddr *) &addr4; + } else if (inet_pton(AF_INET6, listenAddress.c_str(), &addr6.sin6_addr) == 1) { + addr6.sin6_family = AF_INET6; + addr6.sin6_port = htons(port); + sockAddr = (const sockaddr *) &addr6; + flags |= MHD_USE_IPv6; + } else + throw Error("invalid listen address '%s'", listenAddress); + + auto * daemon = MHD_start_daemon( + flags, port, nullptr, nullptr, handler, &ctx, MHD_OPTION_SOCK_ADDR, sockAddr, MHD_OPTION_END); + + if (!daemon) + throw Error("failed to start HTTP daemon on %s:%d", listenAddress, port); + + Finally _stopDaemon{[&] { + notice("Shutting down..."); + MHD_stop_daemon(daemon); + }}; + + auto * info = MHD_get_daemon_info(daemon, MHD_DAEMON_INFO_BIND_PORT); + uint16_t boundPort = info ? info->port : port; + notice("Listening on http://%s:%d/", listenAddress, boundPort); + + if (portFile) + writeFile(*portFile, std::to_string(boundPort) + "\n"); + + /* Wait for Ctrl-C. */ + std::promise interruptPromise; + std::future interruptFuture = interruptPromise.get_future(); + auto callback = createInterruptCallback([&]() { interruptPromise.set_value(); }); + interruptFuture.get(); + } +}; + +static auto rCmdServe = registerCommand("serve"); diff --git a/src/nix/serve.md b/src/nix/serve.md new file mode 100644 index 000000000000..bd9dff83f938 --- /dev/null +++ b/src/nix/serve.md @@ -0,0 +1,48 @@ +R""( + +# Examples + +* Serve the local Nix store on the default port (8080), listening on localhost only: + + ```console + # nix serve + Listening on http://127.0.0.1:8080/ + ``` + +* Serve the local store on all interfaces, on port 9000: + + ```console + # nix serve --listen-address 0.0.0.0 --port 9000 + ``` + + On another machine, you can then use this server as a substituter: + + ```console + # nix copy --from http://other-host:8080 /nix/store/...-hello-2.12.1 + ``` + +* Serve a chroot store as a binary cache: + + ```console + # nix serve --store /tmp/my-store + ``` + +# Description + +`nix serve` runs an HTTP server that exposes a Nix store as a [binary +cache](@docroot@/glossary.md#gloss-binary-cache). Clients can fetch +store paths from this server by adding its URL to their list of +[substituters](@docroot@/command-ref/conf-file.md#conf-substituters). + +NARs are served uncompressed. + +> **Note** +> +> `nix serve` only handles `GET` and `HEAD` requests; it cannot be +> used to upload paths to the store. It also does not implement +> authentication or TLS — if you want to expose it to the public +> internet, run it behind a reverse proxy such as nginx. + +To shut down the server, send it a `SIGINT` signal. + +)"" diff --git a/src/nix/sigs.cc b/src/nix/sigs.cc index bcc577317983..7e23aba5816a 100644 --- a/src/nix/sigs.cc +++ b/src/nix/sigs.cc @@ -121,8 +121,7 @@ struct CmdSign : StorePathsCommand void run(ref store, StorePaths && storePaths) override { - SecretKey secretKey(readFile(secretKeyFile)); - LocalSigner signer(std::move(secretKey)); + LocalSigner signer(SecretKey::parse(readFile(secretKeyFile))); size_t added{0}; @@ -149,6 +148,7 @@ static auto rCmdSign = registerCommand2({"store", "sign"}); struct CmdKeyGenerateSecret : Command { std::string keyName; + std::string keyType = "ed25519"; CmdKeyGenerateSecret() { @@ -159,6 +159,13 @@ struct CmdKeyGenerateSecret : Command .handler = {&keyName}, .required = true, }); + + addFlag({ + .longName = "key-type", + .description = fmt("Type of key: one of %s.", concatStringsSep(", ", getKeyTypes())), + .labels = {"type"}, + .handler = {&keyType}, + }); } std::string description() override @@ -176,7 +183,7 @@ struct CmdKeyGenerateSecret : Command void run() override { logger->stop(); - writeFull(getStandardOutput(), SecretKey::generate(keyName).to_string()); + writeFull(getStandardOutput(), SecretKey::generate(keyName, parseKeyType(keyType))->to_string()); } }; @@ -196,9 +203,50 @@ struct CmdKeyConvertSecretToPublic : Command void run() override { - SecretKey secretKey(drainFD(STDIN_FILENO)); logger->stop(); - writeFull(getStandardOutput(), secretKey.toPublicKey().to_string()); + writeFull(getStandardOutput(), SecretKey::parse(drainFD(STDIN_FILENO))->toPublicKey()->to_string()); + } +}; + +struct CmdKeyConvertSecretToPem : Command +{ + std::string description() override + { + return "convert a secret key read from standard input to PEM PKCS#8 format"; + } + + std::string doc() override + { + return +#include "key-convert-secret-to-pem.md" + ; + } + + void run() override + { + logger->stop(); + writeFull(getStandardOutput(), SecretKey::parse(drainFD(STDIN_FILENO))->toPEM()); + } +}; + +struct CmdKeyConvertPublicToPem : Command +{ + std::string description() override + { + return "convert a public key read from standard input to PEM SubjectPublicKeyInfo format"; + } + + std::string doc() override + { + return +#include "key-convert-public-to-pem.md" + ; + } + + void run() override + { + logger->stop(); + writeFull(getStandardOutput(), PublicKey::parse(drainFD(STDIN_FILENO))->toPEM()); } }; @@ -210,6 +258,8 @@ struct CmdKey : NixMultiCommand { {"generate-secret", []() { return make_ref(); }}, {"convert-secret-to-public", []() { return make_ref(); }}, + {"convert-secret-to-pem", []() { return make_ref(); }}, + {"convert-public-to-pem", []() { return make_ref(); }}, }) { } diff --git a/src/nix/unix/daemon.cc b/src/nix/unix/daemon.cc index 954a2eea3858..4b1e0074822e 100644 --- a/src/nix/unix/daemon.cc +++ b/src/nix/unix/daemon.cc @@ -9,6 +9,7 @@ #include "nix/store/remote-store-connection.hh" #include "nix/store/store-open.hh" #include "nix/util/serialise.hh" +#include "nix/util/strings.hh" #include "nix/store/globals.hh" #include "nix/util/config-global.hh" #include "nix/store/derivations.hh" @@ -16,6 +17,7 @@ #include "nix/cmd/unix-socket-server.hh" #include "nix/store/daemon.hh" #include "man-pages.hh" +#include "otel-logger.hh" #include "nix/util/socket.hh" #include @@ -28,7 +30,7 @@ #include #include #include -#include +#include #include #include #include @@ -40,6 +42,22 @@ namespace nix { +/** + * Set up distributed tracing for a daemon connection: create a span + * covering the connection's lifetime, parented under the trace + * context received from the client, if any. No-op unless + * OpenTelemetry export is configured. + * + * Note that `main()` deliberately doesn't set up tracing for the + * daemon, so this is where it happens, once per connection (which is + * once per process, since connections are served by forked children + * or over stdio). + */ +static void setupConnectionTelemetry(std::string_view traceparent) +{ + initOtel("nix-daemon", "daemon connection", traceparent, /*isServer=*/true); +} + /** * Settings related to authenticating clients for the Nix daemon. * @@ -271,12 +289,32 @@ static void daemonLoop( auto rootCgroupPath = *cgroupFS / rootCgroup.rel(); if (!pathExists(rootCgroupPath)) throw Error("expected cgroup directory %s", PathFmt(rootCgroupPath)); - auto daemonCgroupPath = rootCgroupPath + "/nix-daemon"; + auto daemonCgroupPath = rootCgroupPath / "nix-daemon"; // Create new sub-cgroup for the daemon. if (mkdir(daemonCgroupPath.c_str(), 0755) != 0 && errno != EEXIST) - throw SysError("creating cgroup '%s'", daemonCgroupPath); + throw SysError("creating cgroup %s", PathFmt(daemonCgroupPath)); // Move daemon into the new cgroup. - writeFile(daemonCgroupPath + "/cgroup.procs", fmt("%d", getpid())); + writeFile(daemonCgroupPath / "cgroup.procs", fmt("%d", getpid())); + + /* Now that the root cgroup has no processes, enable controllers + for the per-build sibling cgroups so they get memory/io stats. */ + try { + auto available = tokenizeString(readFile(rootCgroupPath / "cgroup.controllers")); + Strings enable; + for (auto & c : {"cpu", "memory", "io", "pids"}) + if (available.count(c)) + enable.push_back(fmt("+%s", c)); + if (!enable.empty()) + writeFile(rootCgroupPath / "cgroup.subtree_control", concatStringsSep(" ", enable)); + } catch (SystemError & e) { + /* This is what we get when trying to violate the "no internal processes" rule. */ + if (e.is(std::errc::device_or_resource_busy)) + warn( + "could not enable cgroup controllers because current cgroup (%s) is not process-free", + PathFmt(rootCgroupPath)); + else + warn("could not enable cgroup controllers for builds: %s", e.ec().message()); + } } #endif @@ -339,9 +377,16 @@ static void daemonLoop( trusted = *forceTrustClientOpt; else { peer = unix::getPeerInfo(remote.get()); - auto [_trusted, _userName] = authPeer(peer); - trusted = _trusted; - userName = _userName; + try { + auto [_trusted, _userName] = authPeer(peer); + trusted = _trusted; + userName = _userName; + } catch (const Error & e) { + // Don't just hang up on the user. + FdSink sink(remote.get()); + sink << WORKER_MAGIC_ACCESS_DENIED; + throw; + } }; printInfo( @@ -357,6 +402,8 @@ static void daemonLoop( options.allowVfork = false; startProcess( [&, storeConfig, closeListeners = std::move(closeListeners)]() { + setInterrupted(false); + closeListeners(); // Background the daemon. @@ -380,7 +427,14 @@ static void daemonLoop( FdSource(remote.get()), FdSink(remote.get()), trusted, - RecursiveFlag::NotRecursive); + RecursiveFlag::NotRecursive, + setupConnectionTelemetry); + + /* End the connection span and export all + telemetry. This has to be done explicitly, + since exit() does not unwind the stack. */ + logger->stop(); + logger->flush(); exit(0); }, @@ -405,23 +459,27 @@ static void forwardStdioConnection(RemoteStore & store) int from = conn->from.fd; int to = conn->to.fd; - Socket fromSock = toSocket(from), stdinSock = toSocket(getStandardInput()); - auto nfds = std::max(fromSock, stdinSock) + 1; while (true) { - fd_set fds; - FD_ZERO(&fds); - FD_SET(fromSock, &fds); - FD_SET(stdinSock, &fds); - if (select(nfds, &fds, nullptr, nullptr, nullptr) == -1) + struct pollfd pfds[2]; + pfds[0].fd = from; + pfds[0].events = POLLIN; + pfds[0].revents = 0; + pfds[1].fd = getStandardInput(); + pfds[1].events = POLLIN; + pfds[1].revents = 0; + if (poll(pfds, 2, -1) == -1) { + if (errno == EINTR) + continue; throw SysError("waiting for data from client or server"); - if (FD_ISSET(fromSock, &fds)) { + } + if (pfds[0].revents) { auto res = splice(from, nullptr, STDOUT_FILENO, nullptr, SSIZE_MAX, SPLICE_F_MOVE); if (res == -1) throw SysError("splicing data from daemon socket to stdout"); else if (res == 0) throw EndOfFile("unexpected EOF from daemon socket"); } - if (FD_ISSET(stdinSock, &fds)) { + if (pfds[1].revents) { auto res = splice(STDIN_FILENO, nullptr, to, nullptr, SSIZE_MAX, SPLICE_F_MOVE); if (res == -1) throw SysError("splicing data from stdin to daemon socket"); @@ -442,7 +500,13 @@ static void forwardStdioConnection(RemoteStore & store) */ static void processStdioConnection(ref store, TrustedFlag trustClient) { - processConnection(store, FdSource(STDIN_FILENO), FdSink(STDOUT_FILENO), trustClient, daemon::NotRecursive); + processConnection( + store, + FdSource(STDIN_FILENO), + FdSink(STDOUT_FILENO), + trustClient, + daemon::NotRecursive, + setupConnectionTelemetry); } /** @@ -505,6 +569,8 @@ static void runDaemon( processStdioConnection(store, forceTrustClientOpt.value_or(Trusted)); }, [&](UnixSocket socketPathOverride) { + /* Note: we don't trace in this process; the forked + children set up tracing themselves, per connection. */ auto socketPath = std::move(socketPathOverride) .or_else([&]() -> std::optional { return getDaemonSocketPath(*storeConfig); diff --git a/src/nix/upgrade-nix.cc b/src/nix/upgrade-nix.cc index 3136f4d5fd15..6d161b89fb53 100644 --- a/src/nix/upgrade-nix.cc +++ b/src/nix/upgrade-nix.cc @@ -15,14 +15,6 @@ namespace nix { -/** - * Check whether a path has a "profiles" component. - */ -static bool hasProfilesComponent(const std::filesystem::path & path) -{ - return std::ranges::contains(path, OS_STR("profiles")); -} - /** * Settings related to upgrading Nix itself. */ @@ -33,11 +25,12 @@ struct UpgradeSettings : Config */ Setting storePathUrl{ this, - "https://github.com/NixOS/nixpkgs/raw/master/nixos/modules/installer/tools/nix-fallback-paths.nix", + "", "upgrade-nix-store-path-url", R"( - Used by `nix upgrade-nix`, the URL of the file that contains the - store paths of the latest Nix release. + Deprecated. This option was used to configure how `nix upgrade-nix` operated. + + Using this setting has no effect. It will be removed in a future release of Determinate Nix. )"}; }; @@ -47,26 +40,6 @@ static GlobalConfig::Register rUpgradeSettings(&upgradeSettings); struct CmdUpgradeNix : MixDryRun, StoreCommand { - std::filesystem::path profileDir; - - CmdUpgradeNix() - { - addFlag({ - .longName = "profile", - .shortName = 'p', - .description = "The path to the Nix profile to upgrade.", - .labels = {"profile-dir"}, - .handler = {&profileDir}, - }); - - addFlag({ - .longName = "nix-store-paths-url", - .description = "The URL of the file that contains the store paths of the latest Nix release.", - .labels = {"url"}, - .handler = {&(std::string &) upgradeSettings.storePathUrl}, - }); - } - /** * This command is stable before the others */ @@ -77,7 +50,7 @@ struct CmdUpgradeNix : MixDryRun, StoreCommand std::string description() override { - return "upgrade Nix to the latest stable version"; + return "deprecated in favor of determinate-nixd upgrade"; } std::string doc() override @@ -94,118 +67,9 @@ struct CmdUpgradeNix : MixDryRun, StoreCommand void run(ref store) override { - evalSettings.pureEval = true; - - if (profileDir == "") - profileDir = getProfileDir(store); - - printInfo("upgrading Nix in profile %s", PathFmt(profileDir)); - - auto storePath = getLatestNix(store); - - auto version = DrvName(storePath.name()).version; - - if (dryRun) { - logger->stop(); - warn("would upgrade to version %s", version); - return; - } - - { - Activity act(*logger, lvlInfo, actUnknown, fmt("downloading '%s'...", store->printStorePath(storePath))); - store->ensurePath(storePath); - } - - { - Activity act( - *logger, lvlInfo, actUnknown, fmt("verifying that '%s' works...", store->printStorePath(storePath))); - auto program = store->printStorePath(storePath) + "/bin/nix-env"; - auto s = runProgram(program, false, {OS_STR("--version")}); - if (s.find("Nix") == std::string::npos) - throw Error("could not verify that '%s' works", program); - } - - logger->stop(); - - { - Activity act( - *logger, - lvlInfo, - actUnknown, - fmt("installing '%s' into profile %s...", store->printStorePath(storePath), PathFmt(profileDir))); - - // FIXME: don't call an external process. - runProgram( - getNixBin("nix-env"), - false, - { - OS_STR("--profile"), - profileDir.native(), - OS_STR("-i"), - string_to_os_string(store->printStorePath(storePath)), - OS_STR("--no-sandbox"), - }); - } - - printInfo(ANSI_GREEN "upgrade to version %s done" ANSI_NORMAL, version); - } - - /* Return the profile in which Nix is installed. */ - std::filesystem::path getProfileDir(ref store) - { - auto whereOpt = ExecutablePath::load().findName(OS_STR("nix-env")); - if (!whereOpt) - throw Error("couldn't figure out how Nix is installed, so I can't upgrade it"); - const auto & where = whereOpt->parent_path(); - - printInfo("found Nix in %s", PathFmt(where)); - - if (hasPrefix(where.string(), "/run/current-system")) - throw Error("Nix on NixOS must be upgraded via 'nixos-rebuild'"); - - auto profileDir = where.parent_path(); - - // Chase symlinks until we find a path under a "profiles" - // directory, or we run out of symlinks. - auto resolved = profileDir; - while (!hasProfilesComponent(canonPath(resolved)) && std::filesystem::is_symlink(resolved)) - // Note that operator/ replaces lhs when rhs is absolute. - resolved = resolved.parent_path() / readLink(resolved); - printInfo("found profile %s", PathFmt(resolved)); - - if (std::filesystem::exists(profileDir / "manifest.json")) - throw Error( - "directory %s is managed by 'nix profile' and currently cannot be upgraded by 'nix upgrade-nix'", - PathFmt(profileDir)); - - if (!std::filesystem::exists(profileDir / "manifest.nix")) - throw Error("directory %s does not appear to be part of a Nix profile", PathFmt(profileDir)); - - auto userEnv = store->followLinksToStorePath(profileDir.string()); - - if (!store->isValidPath(userEnv)) - throw Error("directory %s is not in the Nix store", PathFmt(profileDir)); - - return profileDir; - } - - /* Return the store path of the latest stable Nix. */ - StorePath getLatestNix(ref store) - { - Activity act(*logger, lvlInfo, actUnknown, "querying latest Nix version"); - - // FIXME: use nixos.org? - auto req = FileTransferRequest(parseURL(upgradeSettings.storePathUrl.get())); - auto res = getFileTransfer()->download(req); - - auto state = std::make_shared(LookupPath{}, store, fetchSettings, evalSettings); - auto v = state->allocValue(); - state->eval(state->parseExprFromString(res.data, state->rootPath(CanonPath("/no-such-path"))), *v); - Bindings & bindings = Bindings::emptyBindings; - auto v2 = findAlongAttrPath(*state, settings.thisSystem, bindings, *v).first; - - return store->parseStorePath( - state->forceString(*v2, noPos, "while evaluating the path tho latest nix version")); + throw Error( + "The upgrade-nix command isn't available in Determinate Nix; use %s instead", + "sudo determinate-nixd upgrade"); } }; diff --git a/src/nix/upgrade-nix.md b/src/nix/upgrade-nix.md index 3a3bf61b9b05..bb5157175826 100644 --- a/src/nix/upgrade-nix.md +++ b/src/nix/upgrade-nix.md @@ -1,33 +1,11 @@ R""( -# Examples - -* Upgrade Nix to the stable version declared in Nixpkgs: - - ```console - # nix upgrade-nix - ``` - -* Upgrade Nix in a specific profile: - - ```console - # nix upgrade-nix --profile ~alice/.local/state/nix/profiles/profile - ``` - # Description -This command upgrades Nix to the stable version. - -By default, the latest stable version is defined by Nixpkgs, in -[nix-fallback-paths.nix](https://github.com/NixOS/nixpkgs/raw/master/nixos/modules/installer/tools/nix-fallback-paths.nix) -and updated manually. It may not always be the latest tagged release. - -By default, it locates the directory containing the `nix` binary in the `$PATH` -environment variable. If that directory is a Nix profile, it will -upgrade the `nix` package in that profile to the latest stable binary -release. +This command isn't available in Determinate Nix but is present in order to guide +users to the new upgrade path. -You cannot use this command to upgrade Nix in the system profile of a -NixOS system (that is, if `nix` is found in `/run/current-system`). +Use `sudo determinate-nixd upgrade` to upgrade Determinate Nix on systems that manage it imperatively. +In practice, this is any system that isn't running NixOS. )"" diff --git a/tests/functional/attrset.wat b/tests/functional/attrset.wat new file mode 100644 index 000000000000..7a32e2f318c1 --- /dev/null +++ b/tests/functional/attrset.wat @@ -0,0 +1,73 @@ +(module + ;; A module that copies an attrset into Wasm memory using `get_attrset` + ;; and returns it as a list `[ name1 value1 name2 value2 ... ]`. + (import "env" "make_string" (func $make_string (param i32 i32) (result i32))) + (import "env" "make_list" (func $make_list (param i32 i32) (result i32))) + (import "env" "get_attrset" (func $get_attrset (param i32 i32 i32) (result i64))) + + (memory (export "memory") 1) + + ;; Memory layout: a buffer for `get_attrset` at 1024, the list + ;; elements at 8192, and the heap for `nix_wasm_alloc` from 16384. + (global $heap (mut i32) (i32.const 16384)) + + (func (export "nix_wasm_init_v1")) + + ;; A bump allocator. The heap pointer stays aligned to 4 bytes, which is + ;; the only alignment that `get_attrset` asks for. + (func (export "nix_wasm_alloc") (param $size i32) (param $align i32) (result i32) + (local $ptr i32) + (local.set $ptr (global.get $heap)) + (global.set $heap + (i32.and (i32.add (i32.add (global.get $heap) (local.get $size)) (i32.const 3)) (i32.const -4))) + (local.get $ptr)) + + ;; Decode the buffer written by `get_attrset`. + (func $decode (param $buf i32) (result i32) + (local $n i32) + (local $i i32) + (local $name i32) + (local $end i32) + (local $out i32) + (local.set $n (i32.load (local.get $buf))) + ;; The names start after the count and the value IDs. + (local.set $name + (i32.add (local.get $buf) (i32.add (i32.const 4) (i32.mul (local.get $n) (i32.const 4))))) + (block $done + (loop $next + (br_if $done (i32.ge_u (local.get $i) (local.get $n))) + ;; Find the null terminator of the name. + (local.set $end (local.get $name)) + (block $found + (loop $scan + (br_if $found (i32.eqz (i32.load8_u (local.get $end)))) + (local.set $end (i32.add (local.get $end) (i32.const 1))) + (br $scan))) + (local.set $out (i32.add (i32.const 8192) (i32.mul (local.get $i) (i32.const 8)))) + (i32.store + (local.get $out) + (call $make_string (local.get $name) (i32.sub (local.get $end) (local.get $name)))) + (i32.store + (i32.add (local.get $out) (i32.const 4)) + (i32.load + (i32.add (local.get $buf) (i32.add (i32.const 4) (i32.mul (local.get $i) (i32.const 4)))))) + (local.set $name (i32.add (local.get $end) (i32.const 1))) + (local.set $i (i32.add (local.get $i) (i32.const 1))) + (br $next))) + (call $make_list (i32.const 8192) (i32.mul (local.get $n) (i32.const 2)))) + + ;; Pass a buffer that's large enough, so the host must use it. + (func (export "attrs_inline") (param $value i32) (result i32) + (local $res i64) + (local.set $res (call $get_attrset (local.get $value) (i32.const 1024) (i32.const 4096))) + (if (i32.ne (i32.wrap_i64 (local.get $res)) (i32.const 1024)) + (then unreachable)) + (call $decode (i32.wrap_i64 (local.get $res)))) + + ;; Pass a buffer that's too small, so the host must call `nix_wasm_alloc`. + (func (export "attrs_alloc") (param $value i32) (result i32) + (local $res i64) + (local.set $res (call $get_attrset (local.get $value) (i32.const 1024) (i32.const 0))) + (if (i32.ne (i32.wrap_i64 (local.get $res)) (i32.const 16384)) + (then unreachable)) + (call $decode (i32.wrap_i64 (local.get $res))))) diff --git a/tests/functional/bad-alloc.wat b/tests/functional/bad-alloc.wat new file mode 100644 index 000000000000..fca23548a7c0 --- /dev/null +++ b/tests/functional/bad-alloc.wat @@ -0,0 +1,12 @@ +(module + ;; A module whose `nix_wasm_alloc` export has the wrong type (the size is + ;; an i64 rather than an i32). The host must reject it at instantiation. + (memory (export "memory") 1) + + (func (export "nix_wasm_init_v1")) + + (func (export "nix_wasm_alloc") (param i64 i32) (result i32) + (i32.const 0)) + + (func (export "id") (param i32) (result i32) + (local.get 0))) diff --git a/tests/functional/binary-cache.sh b/tests/functional/binary-cache.sh index 1dea9878b988..6a29d6be9429 100755 --- a/tests/functional/binary-cache.sh +++ b/tests/functional/binary-cache.sh @@ -26,6 +26,13 @@ for path in "${paths[@]}"; do || [[ "$path" =~ -dependencies-top$ ]] done +echo foobar > "$TEST_ROOT/big-file" +for ((i = 0; i < 16; i++)); do + cat "$TEST_ROOT/big-file" "$TEST_ROOT/big-file" > "$TEST_ROOT/big-file.tmp" + mv "$TEST_ROOT/big-file.tmp" "$TEST_ROOT/big-file" +done +bigFile="$(nix store add-file --store "file://$cacheDir" "$TEST_ROOT/big-file")" + # Test copying build logs to the binary cache. expect 1 nix log --store "file://$cacheDir" "$outPath" 2>&1 | grep 'is not available' nix store copy-log --to "file://$cacheDir" "$outPath" @@ -42,7 +49,41 @@ nix log "$outPath" | grep FOO cacheDir2="$TEST_ROOT/binary+cache" nix copy --to "file://$cacheDir2" "$outPath" && [[ -d "$cacheDir2" ]] +nixServePid= + +stopNixServe() { + if [[ -n "$nixServePid" ]]; then + kill "$nixServePid" + wait "$nixServePid" + unset nixServePid + fi +} + +startNixServe() { + local portFile="$TEST_ROOT/nix-serve-port" + rm -f "$portFile" + nix serve --port 0 --port-file "$portFile" "$@" & + nixServePid="$!" + while [[ ! -e "$portFile" ]]; do + if ! kill -0 "$nixServePid" 2>/dev/null; then + echo "nix serve exited before writing $portFile" >&2 + return 1 + fi + sleep 0.1 + done + nixServePort=$(cat "$portFile") + httpBinaryCacheUrl="http://localhost:$nixServePort" + trap "stopNixServe" EXIT +} + +restartNixServe() { + stopNixServe + startNixServe --store "file://$cacheDir" +} + basicDownloadTests() { + binaryCache="$1" + # No uploading tests because upload with force HTTP doesn't work. # By default, a binary cache doesn't support "nix-env -qas", but does @@ -50,25 +91,27 @@ basicDownloadTests() { clearStore clearCacheCache - nix-env --substituters "file://$cacheDir" -f dependencies.nix -qas \* | grep -- "---" + nix-env --substituters "$binaryCache" -f dependencies.nix -qas \* | grep -- "---" - nix-store --substituters "file://$cacheDir" --no-require-sigs -r "$outPath" + nix-store --substituters "$binaryCache" --no-require-sigs -r "$outPath" [ -x "$outPath/program" ] # But with the right configuration, "nix-env -qas" should also work. - clearStore - clearCacheCache - echo "WantMassQuery: 1" >> "$cacheDir/nix-cache-info" + if [[ "$binaryCache" == file:// ]]; then + clearStore + clearCacheCache + echo "WantMassQuery: 1" >> "$cacheDir/nix-cache-info" - nix-env --substituters "file://$cacheDir" -f dependencies.nix -qas \* | grep -- "--S" - nix-env --substituters "file://$cacheDir" -f dependencies.nix -qas \* | grep -- "--S" + nix-env --substituters "$binaryCache" -f dependencies.nix -qas \* | grep -- "--S" + nix-env --substituters "$binaryCache" -f dependencies.nix -qas \* | grep -- "--S" - x=$(nix-env -f dependencies.nix -qas \* --prebuilt-only) - [ -z "$x" ] + x=$(nix-env -f dependencies.nix -qas \* --prebuilt-only) + [ -z "$x" ] + fi - nix-store --substituters "file://$cacheDir" --no-require-sigs -r "$outPath" + nix-store --substituters "$binaryCache" --no-require-sigs -r "$outPath" nix-store --check-validity "$outPath" nix-store -qR "$outPath" | grep input-2 @@ -78,36 +121,64 @@ basicDownloadTests() { # Test LocalBinaryCacheStore. -basicDownloadTests +basicDownloadTests "file://$cacheDir" # Test HttpBinaryCacheStore. -export _NIX_FORCE_HTTP=1 -basicDownloadTests +restartNixServe +basicDownloadTests "$httpBinaryCacheUrl" + + +# Test whether resuming from a binary cache that doesn't support ranged requests works. +export _NIX_TEST_NIX_SERVE_TRUNCATE_NAR=1 +restartNixServe +clearStore +nix-store -r "$bigFile" --substituters "$httpBinaryCacheUrl" 2>&1 | tee "$TEST_ROOT/log" +[[ $(grep -c "curl error: Transferred a partial file" "$TEST_ROOT/log") -eq 2 ]] +unset _NIX_TEST_NIX_SERVE_TRUNCATE_NAR + + +# Regression test: queryMissing() should cache narinfos. +restartNixServe +nix path-info -vvvv --store "$httpBinaryCacheUrl" "$bigFile" 2> "$TEST_ROOT/log" +[[ $(grep -c "downloading.*narinfo'" "$TEST_ROOT/log") -eq 1 ]] # Test that multiple concurrent substitutions do only one download. clearStore nix-store --init # needed because concurrent creation of the store can give SQLite errors -_NIX_TEST_CONCURRENT_SUBSTITUTION=1 nix-store -r "$depPath" --substituters "file://$cacheDir" --no-require-sigs -vvvv 2> "$TEST_ROOT/log1" & +_NIX_TEST_CONCURRENT_SUBSTITUTION=1 nix-store -r "$depPath" --substituters "$httpBinaryCacheUrl" --no-require-sigs -vvvv 2> "$TEST_ROOT/log1" & pid1="$!" -_NIX_TEST_CONCURRENT_SUBSTITUTION=1 nix-store -r "$depPath" --substituters "file://$cacheDir" --no-require-sigs -vvvv 2> "$TEST_ROOT/log2" & +_NIX_TEST_CONCURRENT_SUBSTITUTION=1 nix-store -r "$depPath" --substituters "$httpBinaryCacheUrl" --no-require-sigs -vvvv 2> "$TEST_ROOT/log2" & pid2="$!" wait "$pid1" wait "$pid2" [[ $(cat "$TEST_ROOT/log1" "$TEST_ROOT/log2" | grep -c "copying path ") -eq 2 ]] -[[ $(cat "$TEST_ROOT/log1" "$TEST_ROOT/log2" | grep -c "downloading.*nar.xz") -eq 1 ]] +[[ $(cat "$TEST_ROOT/log1" "$TEST_ROOT/log2" | grep -c "downloading.*nar'") -eq 1 ]] -# Test whether Nix notices if the NAR doesn't match the hash in the NAR info. +# Test whether Nix notices if the NAR doesn't match the file size in the NAR info. clearStore -nar=$(find "$cacheDir/nar/" -type f -name "*.nar.xz" | head -n1) +nar=$cacheDir/$(nix path-info --json --store "file://$cacheDir" "$depPath" | jq -r .[].url) mv "$nar" "$nar".good mkdir -p "$TEST_ROOT/empty" nix-store --dump "$TEST_ROOT/empty" | xz > "$nar" -expect 1 nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" 2>&1 | tee "$TEST_ROOT/log" +expect 1 nix-build --substituters "$httpBinaryCacheUrl" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" 2>&1 | tee "$TEST_ROOT/log" +grepQuiet "size mismatch" "$TEST_ROOT/log" + +mv "$nar".good "$nar" + + +# Test whether Nix notices if the NAR doesn't match the hash in the NAR info. +clearStore + +nar=$cacheDir/$(nix path-info --json --store "file://$cacheDir" "$depPath" | jq -r .[].url) +mv "$nar" "$nar".good +xz -cd <"$nar".good | sed 's/foo/Foo/' | xz -c >"$nar" + +expect 1 nix-build --substituters "$httpBinaryCacheUrl" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" 2>&1 | tee "$TEST_ROOT/log" grepQuiet "hash mismatch" "$TEST_ROOT/log" mv "$nar".good "$nar" @@ -117,7 +188,7 @@ mv "$nar".good "$nar" clearStore clearCacheCache -if nix-store --substituters "file://$cacheDir" -r "$outPath"; then +if nix-store --substituters "$httpBinaryCacheUrl" -r "$outPath"; then echo "unsigned binary cache incorrectly accepted" exit 1 fi @@ -128,7 +199,7 @@ clearStore mv "$cacheDir/nar" "$cacheDir/nar2" -nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" 2>&1 | tee "$TEST_ROOT/log" +nix-build --substituters "$httpBinaryCacheUrl" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" 2>&1 | tee "$TEST_ROOT/log" # Verify that missing NARs produce warnings, not errors # The build should succeed despite the warnings @@ -146,9 +217,9 @@ mv "$cacheDir/nar" "$cacheDir/nar2" mkdir "$cacheDir/nar" for i in $(cd "$cacheDir/nar2" && echo *); do touch "$cacheDir"/nar/"$i"; done -(! nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result") +(! nix-build --substituters "$httpBinaryCacheUrl" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result") -nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" --fallback +nix-build --substituters "$httpBinaryCacheUrl" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" --fallback rm -rf "$cacheDir/nar" mv "$cacheDir/nar2" "$cacheDir/nar" @@ -160,7 +231,7 @@ clearStore rm -v "$(grep -l "StorePath:.*dependencies-input-2" "$cacheDir"/*.narinfo)" -nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" 2>&1 | tee "$TEST_ROOT/log" +nix-build --substituters "$httpBinaryCacheUrl" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" 2>&1 | tee "$TEST_ROOT/log" grepQuiet "copying path.*input-0" "$TEST_ROOT/log" grepQuiet "copying path.*input-2" "$TEST_ROOT/log" grepQuiet "copying path.*top" "$TEST_ROOT/log" @@ -169,8 +240,9 @@ grepQuiet "copying path.*top" "$TEST_ROOT/log" # Idem, but without cached .narinfo. clearStore clearCacheCache +restartNixServe -nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" 2>&1 | tee "$TEST_ROOT/log" +nix-build --substituters "$httpBinaryCacheUrl" --no-require-sigs dependencies.nix -o "$TEST_ROOT/result" 2>&1 | tee "$TEST_ROOT/log" grepQuiet "don't know how to build" "$TEST_ROOT/log" grepQuiet "building.*input-1" "$TEST_ROOT/log" grepQuiet "building.*input-2" "$TEST_ROOT/log" @@ -194,25 +266,26 @@ badKey=$(nix key convert-secret-to-public < "$TEST_ROOT/sk2") nix key generate-secret --key-name foo.nixos.org-1 > "$TEST_ROOT/sk3" otherKey=$(nix key convert-secret-to-public < "$TEST_ROOT/sk3") -_NIX_FORCE_HTTP='' nix copy --to "file://$cacheDir"?secret-key="$TEST_ROOT"/sk1 "$outPath" +nix copy --to "file://$cacheDir"?secret-key="$TEST_ROOT"/sk1 "$outPath" # Downloading should fail if we don't provide a key. clearStore clearCacheCache +restartNixServe -(! nix-store -r "$outPath" --substituters "file://$cacheDir") +(! nix-store -r "$outPath" --substituters "$httpBinaryCacheUrl") # And it should fail if we provide an incorrect key. clearStore clearCacheCache -(! nix-store -r "$outPath" --substituters "file://$cacheDir" --trusted-public-keys "$badKey") +(! nix-store -r "$outPath" --substituters "$httpBinaryCacheUrl" --trusted-public-keys "$badKey") # It should succeed if we provide the correct key. -nix-store -r "$outPath" --substituters "file://$cacheDir" --trusted-public-keys "$otherKey $publicKey" +nix-store -r "$outPath" --substituters "$httpBinaryCacheUrl" --trusted-public-keys "$otherKey $publicKey" # It should fail if we corrupt the .narinfo. @@ -233,7 +306,7 @@ clearCacheCache # If we provide a bad and a good binary cache, it should succeed. -nix-store -r "$outPath" --substituters "file://$cacheDir2 file://$cacheDir" --trusted-public-keys "$publicKey" +nix-store -r "$outPath" --substituters "file://$cacheDir2 $httpBinaryCacheUrl" --trusted-public-keys "$publicKey" unset _NIX_FORCE_HTTP diff --git a/tests/functional/build.sh b/tests/functional/build.sh index f3c1dabb3c0b..705759d64143 100755 --- a/tests/functional/build.sh +++ b/tests/functional/build.sh @@ -17,6 +17,22 @@ nix build -f multiple-outputs.nix --json a b --no-link | jq --exit-status ' (.out | match(".*multiple-outputs-b")))) ' +# Duplicate installables should yield one result per command-line +# argument (not one per argument squared), in command-line order even +# when duplicates are interleaved with other installables. +[[ $(nix build -f multiple-outputs.nix --no-link --print-out-paths b b b | wc -l) = 3 ]] +out=$(nix build -f multiple-outputs.nix --no-link --print-out-paths b nothing-to-install b) +[[ $(echo "$out" | wc -l) = 3 ]] +[[ $(echo "$out" | sed -n 1p) = $(echo "$out" | sed -n 3p) ]] +echo "$out" | sed -n 1p | grepQuiet "multiple-outputs-b" +echo "$out" | sed -n 2p | grepQuiet "nothing-to-install" +nix build -f multiple-outputs.nix --no-link --json b nothing-to-install b | jq --exit-status ' + length == 3 + and (.[0].drvPath | match(".*multiple-outputs-b.drv")) + and (.[1].drvPath | match(".*nothing-to-install.drv")) + and (.[2].drvPath | match(".*multiple-outputs-b.drv")) +' + # Test output selection using the '^' syntax. nix build -f multiple-outputs.nix --json a^first --no-link | jq --exit-status ' (.[0] | @@ -156,33 +172,22 @@ printf "" | nix build --no-link --stdin --json | jq --exit-status '. == []' printf "%s\n" "$drv^*" | nix build --no-link --stdin --json | jq --exit-status '.[0]|has("drvPath")' # --keep-going and FOD -if isDaemonNewer "2.34pre"; then - # With the fix, cancelled goals are not reported as failures. - # Use -j1 so only x1 starts and fails; x2, x3, x4 are cancelled. - out="$(nix build -f fod-failing.nix -j1 -L 2>&1)" && status=0 || status=$? - test "$status" = 1 - # Only the hash mismatch error for x1. Cancelled goals not reported. - test "$(<<<"$out" grep -cE '^error:')" = 1 - # Regression test: error messages should not be empty (end with just "failed:") - <<<"$out" grepQuietInverse -E "^error:.*failed: *$" -else - out="$(nix build -f fod-failing.nix -L 2>&1)" && status=0 || status=$? - test "$status" = 1 - # At minimum, check that x1 is reported as failing - <<<"$out" grepQuiet -E "error:.*-x1" -fi +out="$(nix build -f fod-failing.nix -j1 -L 2>&1)" && status=0 || status=$? +test "$status" = 1 +# Only the hash mismatch error for the first failing goal (x1). +# The other goals (x2, x3, x4) are cancelled and not reported as failures. +test "$(<<<"$out" grep -cE '^error:')" = 1 <<<"$out" grepQuiet -E "hash mismatch in fixed-output derivation '.*-x1\\.drv'" <<<"$out" grepQuiet -vE "hash mismatch in fixed-output derivation '.*-x3\\.drv'" <<<"$out" grepQuiet -vE "hash mismatch in fixed-output derivation '.*-x2\\.drv'" out="$(nix build -f fod-failing.nix -L x1 x2 x3 --keep-going 2>&1)" && status=0 || status=$? test "$status" = 1 -# three "hash mismatch" errors - for each failing fod, one "build of ... failed" -test "$(<<<"$out" grep -cE '^error:')" = 4 +# three "hash mismatch" errors - for each failing fod +test "$(<<<"$out" grep -cE '^error:')" = 3 <<<"$out" grepQuiet -E "hash mismatch in fixed-output derivation '.*-x1\\.drv'" <<<"$out" grepQuiet -E "hash mismatch in fixed-output derivation '.*-x3\\.drv'" <<<"$out" grepQuiet -E "hash mismatch in fixed-output derivation '.*-x2\\.drv'" -<<<"$out" grepQuiet -E "error: build of '.*-x[1-3]\\.drv\\^out', '.*-x[1-3]\\.drv\\^out', '.*-x[1-3]\\.drv\\^out' failed" out="$(nix build -f fod-failing.nix -L x4 2>&1)" && status=0 || status=$? test "$status" = 1 @@ -216,7 +221,7 @@ fi # Only fast-fail should be reported as a failure. # Uses fifo for synchronization to ensure deterministic behavior. # Requires -j2 so slow and fast-fail run concurrently (fifo deadlocks if serialized). -if isDaemonNewer "2.34pre" && canUseSandbox; then +if isDaemonNewer "2.34pre" && canUseSandbox && unprivilegedUserNamespacesSupported; then fifoDir="$TEST_ROOT/cancelled-builds-fifo" mkdir -p "$fifoDir" mkfifo "$fifoDir/fifo" @@ -229,7 +234,7 @@ if isDaemonNewer "2.34pre" && canUseSandbox; then if ! isTestOnNixOS; then sandboxPathsArg=(--option sandbox-paths "/nix/store") fi - out="$(nix flake check ./cancelled-builds --impure -L -j2 \ + out="$(nix flake check ./cancelled-builds --no-sandbox-fallback --impure -L -j2 \ --option sandbox true \ "${sandboxPathsArg[@]}" \ --option sandbox-build-dir /build-tmp \ @@ -255,7 +260,7 @@ if isDaemonNewer "2.34pre" && canUseSandbox; then sandboxPathsArg=(--option sandbox-paths "/nix/store") fi system=$(nix eval --raw --impure --expr builtins.currentSystem) - out="$(nix build --impure -L -j2 \ + out="$(nix build --no-sandbox-fallback --impure -L -j2 \ --option sandbox true \ "${sandboxPathsArg[@]}" \ --option sandbox-build-dir /build-tmp \ diff --git a/tests/functional/ca/build-with-garbage-path.sh b/tests/functional/ca/build-with-garbage-path.sh index 298cd469a928..b610f8e2b1c8 100755 --- a/tests/functional/ca/build-with-garbage-path.sh +++ b/tests/functional/ca/build-with-garbage-path.sh @@ -9,7 +9,7 @@ requireDaemonNewerThan "2.4pre20210621" # Get the output path of `rootCA`, and put some garbage instead outPath="$(nix-build ./content-addressed.nix -A rootCA --no-out-link)" # shellcheck disable=SC2046 # Multiple store paths need to become individual args -nix-store --delete $(nix-store -q --referrers-closure "$outPath") +nix-store --delete $(nix-store -q --referrers-closure "$outPath") --ignore-liveness touch "$outPath" # The build should correctly remove the garbage and put the expected path instead diff --git a/tests/functional/ca/derivation-json.sh b/tests/functional/ca/derivation-json.sh index f94c5a72c3f0..eb1d949676a7 100644 --- a/tests/functional/ca/derivation-json.sh +++ b/tests/functional/ca/derivation-json.sh @@ -19,7 +19,7 @@ drvPath3=$(nix derivation add --dry-run < "$TEST_HOME"/foo.json) [[ ! -e "$drvPath3" ]] # But the JSON is rejected without the experimental feature -expectStderr 1 nix derivation add < "$TEST_HOME"/foo.json --experimental-features nix-command | grepQuiet "experimental Nix feature 'ca-derivations' is disabled" +expectStderr 1 nix derivation add < "$TEST_HOME"/foo.json --experimental-features '' | grepQuiet "experimental Nix feature 'ca-derivations' is disabled" # Without --dry-run it is actually written drvPath4=$(nix derivation add < "$TEST_HOME"/foo.json) diff --git a/tests/functional/ca/selfref-gc.sh b/tests/functional/ca/selfref-gc.sh index 7ac9ec9f78d0..fdee6b07ab09 100755 --- a/tests/functional/ca/selfref-gc.sh +++ b/tests/functional/ca/selfref-gc.sh @@ -4,7 +4,7 @@ source common.sh requireDaemonNewerThan "2.4pre20210626" -enableFeatures "ca-derivations nix-command flakes" +enableFeatures "ca-derivations" export NIX_TESTS_CA_BY_DEFAULT=1 cd .. diff --git a/tests/functional/common/functions.sh b/tests/functional/common/functions.sh index dd1850137191..a515cb16ba43 100644 --- a/tests/functional/common/functions.sh +++ b/tests/functional/common/functions.sh @@ -73,6 +73,7 @@ startDaemon() { fi # Start the daemon, wait for the socket to appear. rm -f "$NIX_DAEMON_SOCKET_PATH" + # TODO: remove the nix-command feature when we're no longer testing against old daemons. PATH=$DAEMON_PATH nix --extra-experimental-features 'nix-command' daemon & _NIX_TEST_DAEMON_PID=$! export _NIX_TEST_DAEMON_PID @@ -132,11 +133,11 @@ restartDaemon() { } isDaemonNewer () { - [[ -n "${NIX_DAEMON_PACKAGE:-}" ]] || return 0 - local requiredVersion="$1" - local daemonVersion - daemonVersion=$("$NIX_DAEMON_PACKAGE/bin/nix" daemon --version | cut -d' ' -f3) - [[ $(nix eval --expr "builtins.compareVersions ''$daemonVersion'' ''$requiredVersion''") -ge 0 ]] + [[ -n "${NIX_DAEMON_PACKAGE:-}" ]] || return 0 + local requiredVersion="$1" + local daemonVersion + daemonVersion=$("$NIX_DAEMON_PACKAGE/bin/nix" daemon --version | sed 's/.*) //') + [[ $(nix eval --expr "builtins.compareVersions ''$daemonVersion'' ''$requiredVersion''") -ge 0 ]] } skipTest () { @@ -352,8 +353,14 @@ count() { trap onError ERR +# Note: with newer AppArmor stacks, this restriction doesn't prevent *creating* +# a user namespace, but denies mounting inside it, which Nix's sandbox needs. +unprivilegedUserNamespacesSupported() { + ! { [[ -f /proc/sys/kernel/apparmor_restrict_unprivileged_userns ]] && [[ $(< /proc/sys/kernel/apparmor_restrict_unprivileged_userns) -eq 1 ]]; } +} + requiresUnprivilegedUserNamespaces() { - if [[ -f /proc/sys/kernel/apparmor_restrict_unprivileged_userns ]] && [[ $(< /proc/sys/kernel/apparmor_restrict_unprivileged_userns) -eq 1 ]]; then + if ! unprivilegedUserNamespacesSupported; then skipTest "Unprivileged user namespaces are disabled. Run 'sudo sysctl -w /proc/sys/kernel/apparmor_restrict_unprivileged_userns=0' to allow, and run these tests." fi } @@ -368,7 +375,7 @@ initGitRepo() { local extraArgs="${2-}" # shellcheck disable=SC2086 # word splitting of extraArgs is intended - git -C "$repo" init $extraArgs + git -C "$repo" init --initial-branch=master $extraArgs git -C "$repo" config user.email "foobar@example.com" git -C "$repo" config user.name "Foobar" } diff --git a/tests/functional/common/init.sh b/tests/functional/common/init.sh index 5aa71c0b62bf..5d65ba894fd6 100755 --- a/tests/functional/common/init.sh +++ b/tests/functional/common/init.sh @@ -3,6 +3,9 @@ # for shellcheck : "${test_nix_conf_dir?}" "${test_nix_conf?}" "${config_nix?}" +# Don't upload crashes from tests to Sentry. +export NIX_SENTRY_ENDPOINT= + if isTestOnNixOS; then mkdir -p "$test_nix_conf_dir" "$TEST_HOME" @@ -12,7 +15,7 @@ if isTestOnNixOS; then ! test -e "$test_nix_conf" cat > "$test_nix_conf" < "$NIX_CONF_DIR"/nix.conf < "$NIX_CONF_DIR"/nix.conf.extra <"$TEST_ROOT"/stdout 2>"$TEST_ROOT"/stderr +xpFeature=auto-allocate-uids +gatedSetting=auto-allocate-uids + +# Experimental feature is disabled before, ignore and warn. +NIX_CONFIG=" + experimental-features = + $gatedSetting = true +" expect 1 nix config show $gatedSetting 1>"$TEST_ROOT"/stdout 2>"$TEST_ROOT"/stderr [[ $(cat "$TEST_ROOT/stdout") = '' ]] -grepQuiet "Ignoring setting 'accept-flake-config' because experimental feature 'flakes' is not enabled" "$TEST_ROOT/stderr" -grepQuiet "error: could not find setting 'accept-flake-config'" "$TEST_ROOT/stderr" +grepQuiet "error: could not find setting '$gatedSetting'" "$TEST_ROOT/stderr" -# 'flakes' experimental-feature is disabled after, ignore and warn -NIX_CONFIG=' - accept-flake-config = true - experimental-features = nix-command -' expect 1 nix config show accept-flake-config 1>"$TEST_ROOT"/stdout 2>"$TEST_ROOT"/stderr +# Experimental feature is disabled after, ignore and warn. +NIX_CONFIG=" + $gatedSetting = true + experimental-features = +" expect 1 nix config show $gatedSetting 1>"$TEST_ROOT"/stdout 2>"$TEST_ROOT"/stderr [[ $(cat "$TEST_ROOT/stdout") = '' ]] -grepQuiet "Ignoring setting 'accept-flake-config' because experimental feature 'flakes' is not enabled" "$TEST_ROOT/stderr" -grepQuiet "error: could not find setting 'accept-flake-config'" "$TEST_ROOT/stderr" +grepQuiet "error: could not find setting '$gatedSetting'" "$TEST_ROOT/stderr" -# 'flakes' experimental-feature is enabled before, process -NIX_CONFIG=' - experimental-features = nix-command flakes - accept-flake-config = true -' nix config show accept-flake-config 1>"$TEST_ROOT"/stdout 2>"$TEST_ROOT"/stderr +# Experimental feature is enabled before, process. +NIX_CONFIG=" + experimental-features = $xpFeature + $gatedSetting = true +" nix config show $gatedSetting 1>"$TEST_ROOT"/stdout 2>"$TEST_ROOT"/stderr grepQuiet "true" "$TEST_ROOT/stdout" -grepQuietInverse "Ignoring setting 'accept-flake-config'" "$TEST_ROOT/stderr" -# 'flakes' experimental-feature is enabled after, process -NIX_CONFIG=' - accept-flake-config = true - experimental-features = nix-command flakes -' nix config show accept-flake-config 1>"$TEST_ROOT"/stdout 2>"$TEST_ROOT"/stderr +# Experimental feature is enabled after, process. +NIX_CONFIG=" + $gatedSetting = true + experimental-features = $xpFeature +" nix config show $gatedSetting 1>"$TEST_ROOT"/stdout 2>"$TEST_ROOT"/stderr grepQuiet "true" "$TEST_ROOT/stdout" -grepQuietInverse "Ignoring setting 'accept-flake-config'" "$TEST_ROOT/stderr" +grepQuietInverse "Ignoring setting '$gatedSetting'" "$TEST_ROOT/stderr" function exit_code_both_ways { - expect 1 nix --experimental-features 'nix-command' "$@" 1>/dev/null - nix --experimental-features 'nix-command flakes' "$@" 1>/dev/null + expect 1 nix --experimental-features '' "$@" 1>/dev/null + nix --experimental-features "$xpFeature" "$@" 1>/dev/null # Also, the order should not matter - expect 1 nix "$@" --experimental-features 'nix-command' 1>/dev/null - nix "$@" --experimental-features 'nix-command flakes' 1>/dev/null + expect 1 nix "$@" --experimental-features '' 1>/dev/null + nix "$@" --experimental-features "$xpFeature" 1>/dev/null } -exit_code_both_ways show-config --flake-registry 'https://no' +exit_code_both_ways config show --auto-allocate-uids # Double check these are stable nix --experimental-features '' --help 1>/dev/null nix --experimental-features '' doctor --help 1>/dev/null nix --experimental-features '' repl --help 1>/dev/null nix --experimental-features '' upgrade-nix --help 1>/dev/null - -# These 3 arguments are currently given to all commands, which is wrong (as not -# all care). To deal with fixing later, we simply make them require the -# nix-command experimental features --- it so happens that the commands we wish -# stabilizing to do not need them anyways. -for arg in '--print-build-logs' '--offline' '--refresh'; do - nix --experimental-features 'nix-command' "$arg" --help 1>/dev/null - expect 1 nix --experimental-features '' "$arg" --help 1>/dev/null -done diff --git a/tests/functional/export.sh b/tests/functional/export.sh index 9294395cee83..4e69a8e62688 100755 --- a/tests/functional/export.sh +++ b/tests/functional/export.sh @@ -5,18 +5,24 @@ source common.sh TODO_NixOS outPath=$(nix-build dependencies.nix --no-out-link) +drvPath=$(nix path-info --json "$outPath" | jq -r .\""$outPath"\".deriver) nix-store --export "$outPath" > "$TEST_ROOT"/exp +expectStderr 1 nix nario export "$outPath" | grepQuiet "required argument.*missing" +nix nario export --format 1 "$outPath" > "$TEST_ROOT/exp2" +cmp "$TEST_ROOT/exp" "$TEST_ROOT/exp2" # shellcheck disable=SC2046 nix-store --export $(nix-store -qR "$outPath") > "$TEST_ROOT"/exp_all +nix nario export --format 1 -r "$outPath" > "$TEST_ROOT"/exp_all2 +cmp "$TEST_ROOT/exp_all" "$TEST_ROOT/exp_all2" + if nix-store --export "$outPath" >/dev/full ; then echo "exporting to a bad file descriptor should fail" exit 1 fi - clearStore if nix-store --import < "$TEST_ROOT"/exp; then @@ -24,7 +30,6 @@ if nix-store --import < "$TEST_ROOT"/exp; then exit 1 fi - clearStore nix-store --import < "$TEST_ROOT"/exp_all @@ -32,9 +37,42 @@ nix-store --import < "$TEST_ROOT"/exp_all # shellcheck disable=SC2046 nix-store --export $(nix-store -qR "$outPath") > "$TEST_ROOT"/exp_all2 - clearStore # Regression test: the derivers in exp_all2 are empty, which shouldn't # cause a failure. nix-store --import < "$TEST_ROOT"/exp_all2 + +# Test `nix nario import` on files created by `nix-store --export`. +clearStore +expectStderr 1 nix nario import < "$TEST_ROOT"/exp_all | grepQuiet "lacks a signature" +nix nario import --no-check-sigs < "$TEST_ROOT"/exp_all +nix path-info "$outPath" + +# Test `nix nario list`. +nix nario list < "$TEST_ROOT"/exp_all +nix nario list < "$TEST_ROOT"/exp_all | grepQuiet ".*dependencies-input-0.*bytes" +nix nario list -lR < "$TEST_ROOT"/exp_all | grepQuiet "dr-xr-xr-x .*0 $outPath" +nix nario list -lR < "$TEST_ROOT"/exp_all | grepQuiet "lrwxrwxrwx .*0 $outPath/self -> $outPath" +nix nario list -lR < "$TEST_ROOT"/exp_all | grepQuiet -- "-r--r--r-- .*7 $outPath/foobar" + +# Test format 2 (including signatures). +nix key generate-secret --key-name my-key > "$TEST_ROOT"/secret +public_key=$(nix key convert-secret-to-public < "$TEST_ROOT"/secret) +nix store sign --key-file "$TEST_ROOT/secret" -r "$outPath" +nix nario export --format 2 -r "$outPath" > "$TEST_ROOT"/exp_all +clearStore +expectStderr 1 nix nario import < "$TEST_ROOT"/exp_all | grepQuiet "lacks a signature" +nix nario import --trusted-public-keys "$public_key" < "$TEST_ROOT"/exp_all +[[ $(nix path-info --json "$outPath" | jq -r .[].signatures[]) =~ my-key: ]] + +# Test json listing. +json=$(nix nario list --json -R < "$TEST_ROOT/exp_all") +[[ $(printf "%s" "$json" | jq -r ".paths.\"$outPath\".deriver") = "$drvPath" ]] +[[ $(printf "%s" "$json" | jq -r ".paths.\"$outPath\".contents.type") = directory ]] +[[ $(printf "%s" "$json" | jq -r ".paths.\"$outPath\".contents.entries.foobar.type") = regular ]] +[[ $(printf "%s" "$json" | jq ".paths.\"$outPath\".contents.entries.foobar.size") = 7 ]] + +json=$(nix nario list --json < "$TEST_ROOT/exp_all") +[[ $(printf "%s" "$json" | jq -r ".paths.\"$outPath\".deriver") = "$drvPath" ]] +[[ $(printf "%s" "$json" | jq -r ".paths.\"$outPath\".contents.type") = null ]] diff --git a/tests/functional/fetchGit.sh b/tests/functional/fetchGit.sh index b3aab573b11b..f50ec0011413 100755 --- a/tests/functional/fetchGit.sh +++ b/tests/functional/fetchGit.sh @@ -308,3 +308,71 @@ git -C "$empty" config user.name "Foobar" git -C "$empty" commit --allow-empty --allow-empty-message --message "" nix eval --impure --expr "let attrs = builtins.fetchGit $empty; in assert attrs.lastModified != 0; assert attrs.rev != \"0000000000000000000000000000000000000000\"; assert attrs.revCount == 1; true" + +# Test backward compatibility hack for Nix < 2.20 locks / fetchTree calls that expect Git filters to be applied. +eol="$TEST_ROOT/git-eol" +createGitRepo "$eol" +mkdir -p "$eol/dir" +printf "Hello\nWorld\n" > "$eol/dir/crlf" +printf "ignore me" > "$eol/dir/ignored" +git -C "$eol" add dir/crlf dir/ignored +git -C "$eol" commit -a -m Initial +echo "Version: \$Format:%s\$" > "$eol/dir/version" +printf "crlf text eol=crlf\nignored export-ignore\nversion export-subst\n" > "$eol/dir/.gitattributes" +git -C "$eol" add dir/.gitattributes dir/version +git -C "$eol" commit -a -m 'Apply gitattributes' + +rev="$(git -C "$eol" rev-parse HEAD)" + +export _NIX_TEST_BARF_ON_UNCACHEABLE=1 + +oldHash="sha256-fccLx4BSC7e/PzQM4JnixstJQnd4dzgm73BqKhV3KRs=" +newHash="sha256-Ns7sLZOvpacagAPNun1+jBovMpo+zM7PUJ9x+lW3cIU=" + +expectStderr 0 nix eval --expr \ + "let tree = builtins.fetchTree { type = \"git\"; url = \"file://$eol\"; rev = \"$rev\"; narHash = \"$oldHash\"; }; in assert builtins.readFile \"\${tree}/dir/crlf\" == \"Hello\r\nWorld\r\n\"; assert !builtins.pathExists \"\${tree}/dir/ignored\"; assert builtins.readFile \"\${tree}/dir/version\" == \"Version: Apply gitattributes\n\"; true" \ + | grepQuiet "Please update the NAR hash to '$newHash'" + +nix eval --expr \ + "let tree = builtins.fetchTree { type = \"git\"; url = \"file://$eol\"; rev = \"$rev\"; narHash = \"$newHash\"; }; in assert builtins.readFile \"\${tree}/dir/crlf\" == \"Hello\nWorld\n\"; assert builtins.pathExists \"\${tree}/dir/ignored\"; assert builtins.readFile \"\${tree}/dir/version\" == \"Version: \$Format:%s\$\n\"; true" + +expectStderr 102 nix eval --expr \ + "builtins.fetchTree { type = \"git\"; url = \"file://$eol\"; rev = \"$rev\"; narHash = \"sha256-DLDvcwdcwCxnuPTxSQ6gLAyopB20lD0bOQoQB3i2hsA=\"; }" \ + | grepQuiet "NAR hash mismatch" + +mkdir -p "$TEST_ROOT"/flake +cat > "$TEST_ROOT"/flake/flake.nix << EOF +{ + inputs.eol = { type = "git"; url = "file://$eol"; rev = "$rev"; flake = false; }; + outputs = { self, eol }: rec { + crlf = builtins.readFile "\${eol}/dir/crlf"; + isLegacy = assert crlf == "Hello\r\nWorld\r\n"; true; + isModern = assert crlf == "Hello\nWorld\n"; true; + }; +} +EOF + +# Test locking with Nix < 2.20 semantics (i.e. using `git archive`). +nix eval --nix-219-compat "path:$TEST_ROOT/flake"#isLegacy +nix eval "path:$TEST_ROOT/flake"#isLegacy +[[ $(jq -r .nodes.eol.locked.narHash < "$TEST_ROOT"/flake/flake.lock) = "$oldHash" ]] + +# Test locking with Nix >= 2.20 semantics (i.e. using libgit2). +rm "$TEST_ROOT"/flake/flake.lock +nix eval "path:$TEST_ROOT/flake"#isModern +nix eval --nix-219-compat "path:$TEST_ROOT/flake"#isModern +[[ $(jq -r .nodes.eol.locked.narHash < "$TEST_ROOT"/flake/flake.lock) = "$newHash" ]] + + +# Test that builtins.hashString devirtualizes lazy paths (https://github.com/DeterminateSystems/determinate/issues/160). +hashStringRepo="$TEST_ROOT/hashString" +createGitRepo "$hashStringRepo" +echo hello > "$hashStringRepo"/hello +git -C "$hashStringRepo" add hello +git -C "$hashStringRepo" commit -m 'Initial' +hashStringRev=$(git -C "$hashStringRepo" rev-parse HEAD) + +hash1=$(nix eval --lazy-trees --raw --expr "builtins.hashString \"sha256\" (toString ((builtins.fetchGit { url = file://$hashStringRepo; rev = \"$hashStringRev\"; })))") +hash2=$(nix eval --lazy-trees --raw --expr "builtins.hashString \"sha256\" (toString ((builtins.fetchGit { url = file://$hashStringRepo; rev = \"$hashStringRev\"; })))") + +[[ "$hash1" = "$hash2" ]] diff --git a/tests/functional/fetchGitShallow.sh b/tests/functional/fetchGitShallow.sh index 0da36ee9bac9..f196c3b9e6c1 100644 --- a/tests/functional/fetchGitShallow.sh +++ b/tests/functional/fetchGitShallow.sh @@ -30,7 +30,7 @@ git -C "$TEST_ROOT/shallow-parent" commit -m "Branch commit" git clone --depth 1 "file://$TEST_ROOT/shallow-parent" "$TEST_ROOT/shallow-clone" # Test 1: Fetching a shallow repo succeeds for outPath because revCount is lazy. -path1=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = \"$TEST_ROOT/shallow-clone\"; ref = \"dev\"; }).outPath") +path1=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = \"$TEST_ROOT/shallow-clone\"; ref = \"dev\"; shallow = true; }).outPath") [[ -d "$path1" ]] # But accessing revCount on a shallow clone fails. (! nix eval --impure --expr "(builtins.fetchGit { url = \"$TEST_ROOT/shallow-clone\"; ref = \"dev\"; }).revCount" 2>/dev/null) @@ -93,4 +93,4 @@ git -C "$TEST_ROOT/shallow-build-parent" add flake.nix git -C "$TEST_ROOT/shallow-build-parent" commit -m "add flake" git clone --depth 1 "file://$TEST_ROOT/shallow-build-parent" "$TEST_ROOT/shallow-build-clone" rm -rf "$TEST_ROOT/shallow-build-parent" -nix build --dry-run "git+file://$TEST_ROOT/shallow-build-clone" +nix build --dry-run "git+file://$TEST_ROOT/shallow-build-clone?shallow=1" diff --git a/tests/functional/fetchGitSubmodules.sh b/tests/functional/fetchGitSubmodules.sh index d3f9e4da57bb..db34dedfc00d 100755 --- a/tests/functional/fetchGitSubmodules.sh +++ b/tests/functional/fetchGitSubmodules.sh @@ -265,3 +265,62 @@ test_gitlink_without_gitmodules() { done } test_gitlink_without_gitmodules + +# The backward compatibility hack for Nix < 2.20 locks must also work for repos +# fetched with `submodules = true`. Since the NAR hash of such a repo covers the +# submodule contents, deciding whether a lock was produced with Nix < 2.20 +# semantics requires hashing the *mounted* tree, and the submodules have to be +# exported using those same semantics. +test_legacy_export_with_submodules() { + local root=$TEST_ROOT/legacySubmodulesRoot + local sub=$TEST_ROOT/legacySubmodulesSub + + rm -rf "$TEST_HOME"/.cache/nix + + createGitRepo "$sub" + # `text eol=crlf` is a Git filter, so it's applied by `git checkout` (which is + # what Nix < 2.20 used for repos with submodules) but not by libgit2. + printf "crlf text eol=crlf\n" > "$sub"/.gitattributes + printf "Hello\nWorld\n" > "$sub"/crlf + git -C "$sub" add .gitattributes crlf + git -C "$sub" commit -m "Add crlf" + + createGitRepo "$root" + git -C "$root" submodule add "$sub" sub + git -C "$root" commit -m "Add submodule" + + local rev + rev=$(git -C "$root" rev-parse HEAD) + + local input="{ type = \"git\"; url = \"file://$root\"; rev = \"$rev\"; submodules = true; }" + + # Determine the two NAR hashes in a throwaway store: `builtins.fetchTree` marks + # inputs with a `narHash` as final, so if a tree with the expected hash is + # already in the store, it is returned without running the fetcher at all, + # which would mask the behaviour we want to test below. + local scratch=$TEST_ROOT/legacySubmodulesStore + local legacyHash modernHash + legacyHash=$(nix eval --store "$scratch" --nix-219-compat --raw --expr "(builtins.fetchTree $input).narHash") + modernHash=$(nix eval --store "$scratch" --raw --expr "(builtins.fetchTree $input).narHash") + + # If the Git filter in the submodule didn't make a difference, this test + # wouldn't be testing anything. + [[ $legacyHash != "$modernHash" ]] + + # A NAR hash produced by Nix < 2.20 must still be accepted by default (with a + # warning), and must yield the filtered submodule contents. + expectStderr 0 nix eval --expr \ + "let tree = builtins.fetchTree ($input // { narHash = \"$legacyHash\"; }); in assert builtins.readFile \"\${tree}/sub/crlf\" == \"Hello\r\nWorld\r\n\"; true" \ + | grepQuiet "Please update the NAR hash to '$modernHash'" + + # Conversely, a NAR hash produced by Nix >= 2.20 must be accepted even when + # `nix-219-compat` is enabled. + nix eval --nix-219-compat --expr \ + "let tree = builtins.fetchTree ($input // { narHash = \"$modernHash\"; }); in assert builtins.readFile \"\${tree}/sub/crlf\" == \"Hello\nWorld\n\"; true" + + # A NAR hash that matches neither must still be an error. + expectStderr 102 nix eval --expr \ + "builtins.fetchTree ($input // { narHash = \"sha256-DLDvcwdcwCxnuPTxSQ6gLAyopB20lD0bOQoQB3i2hsA=\"; })" \ + | grepQuiet "NAR hash mismatch" +} +test_legacy_export_with_submodules diff --git a/tests/functional/fetchPath.sh b/tests/functional/fetchPath.sh index 1df895b61662..2784afb0388e 100755 --- a/tests/functional/fetchPath.sh +++ b/tests/functional/fetchPath.sh @@ -3,9 +3,9 @@ source common.sh touch "$TEST_ROOT/foo" -t 202211111111 -# We only check whether 2022-11-1* **:**:** is the last modified date since -# `lastModified` is transformed into UTC in `builtins.fetchTarball`. -[[ "$(nix eval --impure --raw --expr "(builtins.fetchTree \"path://$TEST_ROOT/foo\").lastModifiedDate")" =~ 2022111.* ]] + +# The path fetcher does not return lastModified. +[[ "$(nix eval --impure --expr "(builtins.fetchTree \"path://$TEST_ROOT/foo\") ? lastModifiedDate")" = false ]] # Check that we can override lastModified for "path:" inputs. [[ "$(nix eval --impure --expr "(builtins.fetchTree { type = \"path\"; path = \"$TEST_ROOT/foo\"; lastModified = 123; }).lastModified")" = 123 ]] diff --git a/tests/functional/fetchurl.sh b/tests/functional/fetchurl.sh index 452ac043a5b3..ffc3125709ec 100755 --- a/tests/functional/fetchurl.sh +++ b/tests/functional/fetchurl.sh @@ -69,7 +69,7 @@ echo "$outPath" | grepQuiet 'xyzzy' test -x "$outPath/fetchurl.sh" test -L "$outPath/symlink" -nix-store --delete "$outPath" +nix-store --delete "$outPath" --ignore-liveness # Test unpacking a compressed NAR. narxz="$TEST_ROOT/archive.nar.xz" diff --git a/tests/functional/fib.wasm b/tests/functional/fib.wasm new file mode 100644 index 000000000000..4ea3dd40f9e2 Binary files /dev/null and b/tests/functional/fib.wasm differ diff --git a/tests/functional/fib.wat b/tests/functional/fib.wat new file mode 100644 index 000000000000..3f39a08902b1 --- /dev/null +++ b/tests/functional/fib.wat @@ -0,0 +1,35 @@ +(module + ;; Import host functions from the Nix env module + (import "env" "get_int" (func $get_int (param i32) (result i64))) + (import "env" "make_int" (func $make_int (param i64) (result i32))) + + ;; The host requires an exported memory to read/write data + (memory (export "memory") 1) + + ;; Called once when the module is instantiated; nothing to initialize here + (func (export "nix_wasm_init_v1")) + + ;; Pure wasm: compute fib(n) recursively + ;; fib(0) = 1 + ;; fib(1) = 1 + ;; fib(n) = fib(n-1) + fib(n-2) + (func $fib (param $n i64) (result i64) + (if (i64.le_s (local.get $n) (i64.const 1)) + (then (return (i64.const 1))) + ) + (i64.add + (call $fib (i64.sub (local.get $n) (i64.const 1))) + (call $fib (i64.sub (local.get $n) (i64.const 2))) + ) + ) + + ;; Entry point: receives ValueId of the input integer, returns ValueId of fib(n) + ;; Type: fn(arg: u32) -> u32 (ValueId = u32) + (func (export "fib") (param $arg i32) (result i32) + (call $make_int + (call $fib + (call $get_int (local.get $arg)) + ) + ) + ) +) diff --git a/tests/functional/flakes/bake.sh b/tests/functional/flakes/bake.sh new file mode 100755 index 000000000000..477f4ac5a339 --- /dev/null +++ b/tests/functional/flakes/bake.sh @@ -0,0 +1,309 @@ +#!/usr/bin/env bash + +source ./common.sh + +# Substituting from a file:// cache with --no-require-sigs requires a +# trusted user, which the NixOS tests don't provide. +TODO_NixOS + +flakeDir=$TEST_ROOT/flake +bakedDir=$TEST_ROOT/baked +cacheDir=$TEST_ROOT/binary-cache + +mkdir -p "$flakeDir" +writeSimpleFlake "$flakeDir" + +# Baking requires the `baked-derivations` experimental feature. +expectStderr 1 nix flake bake "$flakeDir" --dest-dir "$bakedDir" | grepQuiet "experimental Nix feature 'baked-derivations' is disabled" +enableFeatures "baked-derivations" + +# Bake the flake. +nix flake bake "$flakeDir" --dest-dir "$bakedDir" +[[ -e $bakedDir/flake.nix ]] +[[ -e $bakedDir/outputs.json ]] + +# Only outputs that can be baked are recorded: unknown outputs (like +# `number`) and outputs for other systems are omitted. +[[ $(jq -c 'keys' < "$bakedDir/outputs.json") = '["legacyPackages","packages"]' ]] +[[ $(jq -c '.packages.output.children | keys' < "$bakedDir/outputs.json") = "[\"$system\"]" ]] + +# In read-only mode, evaluating the baked flake doesn't write the baked derivations to the store. +[[ ! -e $(nix eval --read-only --no-eval-cache --raw "path:$bakedDir#foo.drvPath") ]] + +# The baked flake has the same outputs as the original for the current +# system. Outputs for other systems are not included by default. +nix flake show --json "path:$bakedDir" > "$TEST_ROOT/show-baked.json" +[[ $(jq -r ".inventory.packages.output.children.\"$system\".children.foo.derivation.name" < "$TEST_ROOT/show-baked.json") = simple ]] +[[ $(jq -r ".inventory.packages.output.children.\"$system\".children.default.derivation.name" < "$TEST_ROOT/show-baked.json") = simple ]] +[[ $(jq -r ".inventory.packages.output.children | has(\"someOtherSystem\")" < "$TEST_ROOT/show-baked.json") = false ]] + +# The baked derivations have the same output paths as the originals. +fooPath=$(nix eval --raw "$flakeDir#foo.outPath") +[[ $(nix eval --raw "path:$bakedDir#foo.outPath") = "$fooPath" ]] +[[ $(nix eval --raw "path:$bakedDir#foo.system") = "$system" ]] +[[ $(nix eval --raw "path:$bakedDir#foo.name") = simple ]] + +# The baked derivations are `builtin:substitute` derivations without inputs. +drvPath=$(nix eval --raw "path:$bakedDir#foo.drvPath") +nix derivation show "$drvPath" > "$TEST_ROOT/baked-drv.json" +[[ $(jq -r '.derivations[].builder' < "$TEST_ROOT/baked-drv.json") = builtin:substitute ]] +[[ $(jq -r '.derivations[].system' < "$TEST_ROOT/baked-drv.json") = builtin ]] +[[ $(jq -r '.derivations[].inputs.drvs | length' < "$TEST_ROOT/baked-drv.json") = 0 ]] +[[ $(jq -r '.derivations[].outputs.out.path' < "$TEST_ROOT/baked-drv.json") = "$(basename "$fooPath")" ]] + +# A `builtin:substitute` derivation must have input-addressed outputs with known paths. +jq '.derivations[]' < "$TEST_ROOT/baked-drv.json" > "$TEST_ROOT/baked-drv-single.json" +[[ $(nix derivation add < "$TEST_ROOT/baked-drv-single.json") = "$drvPath" ]] +jq '.outputs.out = {}' < "$TEST_ROOT/baked-drv-single.json" | expectStderr 1 nix derivation add | grepQuiet "must have input-addressed outputs" +jq '.outputs.out = {"method": "nar", "hash": "sha256-iUUXyRY8iW7DGirb0zwGgf1fRbLA7wimTJKgP7l/OQ8="}' < "$TEST_ROOT/baked-drv-single.json" | expectStderr 1 nix derivation add | grepQuiet "must have input-addressed outputs" + +# Building a baked derivation fails if its outputs cannot be substituted. +expectStderr 1 nix build --no-link "path:$bakedDir#foo" | grepQuiet "failed to substitute" + +# Build the original, copy it to a binary cache, and delete it from the store. +nix build --no-link "$flakeDir#foo" +[[ -e $fooPath/hello ]] +nix copy --to "file://$cacheDir" "$fooPath" +nix store delete "$fooPath" +[[ ! -e $fooPath ]] + +# Now the baked derivation can be realised by substituting its output +# from the binary cache. (`--substitute` is needed because substitution +# is disabled automatically when there is no network access.) +nix build --no-link --substitute --substituters "file://$cacheDir" --no-require-sigs "path:$bakedDir#foo" +[[ -e $fooPath/hello ]] + +# Once the output is valid, building the baked derivation succeeds even without substituters. +nix build --no-link "path:$bakedDir#foo" + +# With `--all-systems`, outputs for other systems are baked as well. +nix flake bake "$flakeDir" --dest-dir "$bakedDir-all" --all-systems +nix flake show --json --all-systems "path:$bakedDir-all" > "$TEST_ROOT/show-baked-all.json" +[[ $(jq -r ".inventory.packages.output.children.someOtherSystem.children.foo.derivation.name" < "$TEST_ROOT/show-baked-all.json") = simple ]] +[[ $(nix eval --raw "path:$bakedDir-all#packages.someOtherSystem.foo.outPath") = $(nix eval --raw "$flakeDir#packages.someOtherSystem.foo.outPath") ]] + +# A derivation in another flake can depend on the outputs of a baked flake. +depDir=$TEST_ROOT/dep +mkdir -p "$depDir" +cp "${config_nix}" "$depDir/" +cat > "$depDir/flake.nix" < "$TEST_ROOT/show-baked-legacy.json" +[[ $(jq -r ".inventory.legacyPackages.output.children.\"$system\".children.hello.derivation.name" < "$TEST_ROOT/show-baked-legacy.json") = simple ]] + +# `meta.mainProgram` is preserved, so `nix run` works on baked packages +# whose binary isn't named after the package. +enableFeatures "ca-derivations" +greeterDir=$TEST_ROOT/greeter +mkdir -p "$greeterDir" +cp "${config_nix}" "$greeterDir/" +cat > "$greeterDir/flake.nix" < \$out/bin/hi + echo 'echo hello from hi' >> \$out/bin/hi + chmod +x \$out/bin/hi + ''; + }; + packages.$system.multi = with import ./config.nix; mkDerivation { + name = "multi"; + outputs = [ "bin" "dev" ]; + buildCommand = "mkdir \$bin \$dev; echo bin > \$bin/what; echo dev > \$dev/what"; + }; + packages.$system.multiDev = self.packages.$system.multi.dev; + # A package that fails to evaluate when built (like a package marked as broken). + packages.$system.broken = { + type = "derivation"; + name = "broken"; + system = "$system"; + meta.description = "a broken package"; + outPath = throw "this package is broken"; + drvPath = throw "this package is broken"; + outputName = "out"; + }; + # Content-addressed derivations cannot be baked. + packages.$system.ca = with import ./config.nix; mkDerivation { + name = "ca"; + __contentAddressed = true; + outputHashMode = "recursive"; + outputHashAlgo = "sha256"; + buildCommand = "echo ca > \$out"; + }; + }; +} +EOF + +# Derivations that cannot be baked produce a warning and are omitted. +nix flake bake "$greeterDir" --dest-dir "$bakedDir-greeter" 2>&1 | grepQuiet "warning: cannot bake 'packages.$system.ca'" +# Unbakeable derivations are recorded as failed. The baked flake still +# lists them (so `nix search` and `nix flake show` behave like the original)... +[[ $(jq -r ".packages.output.children.\"$system\".children.ca.derivation.failed" < "$bakedDir-greeter/outputs.json") = true ]] +[[ $(jq -r ".packages.output.children.\"$system\".children.broken.derivation.failed" < "$bakedDir-greeter/outputs.json") = true ]] +[[ $(jq -r ".packages.output.children.\"$system\".children.greeter.derivation | has(\"failed\")" < "$bakedDir-greeter/outputs.json") = false ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#ca.name") = ca ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#broken.meta.description") = "a broken package" ]] +nix search --json "path:$bakedDir-greeter" broken | jq -e ".\"packages.$system.broken\"" +nix flake show --json "path:$bakedDir-greeter" | jq -e ".inventory.packages.output.children.\"$system\".children.broken.derivation.name == \"broken\"" +# ... but building them fails. +expectStderr 1 nix build --no-link "path:$bakedDir-greeter#ca" | grepQuiet "package 'ca' cannot be built because it could not be baked" +expectStderr 1 nix build --no-link "path:$bakedDir-greeter#broken" | grepQuiet "package 'broken' cannot be built because it could not be baked" +[[ $(jq -r ".packages.output.children.\"$system\".children.greeter.derivation.mainProgram" < "$bakedDir-greeter/outputs.json") = hi ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#greeter.meta.mainProgram") = hi ]] + +# The system of each derivation is recorded, so the baked flake doesn't depend on the schema's `forSystems`. +[[ $(jq -r ".packages.output.children.\"$system\".children.greeter.derivation.system" < "$bakedDir-greeter/outputs.json") = "$system" ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#greeter.system") = "$system" ]] + +# Build the original so that the baked package's output is valid, then run it. +nix build --no-link "$greeterDir#greeter" +[[ $(nix run "path:$bakedDir-greeter#greeter") = "hello from hi" ]] + +# The default output isn't necessarily `out`: multi-output derivations +# and attributes that refer to a specific output are baked faithfully. +[[ $(jq -r ".packages.output.children.\"$system\".children.multi.derivation.outputName" < "$bakedDir-greeter/outputs.json") = bin ]] +[[ $(jq -r ".packages.output.children.\"$system\".children.multiDev.derivation.outputName" < "$bakedDir-greeter/outputs.json") = dev ]] +multiBin=$(nix eval --raw "$greeterDir#multi.outPath") +multiDev=$(nix eval --raw "$greeterDir#multi.dev.outPath") +[[ $multiBin != "$multiDev" ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#multi.outputName") = bin ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#multi.outPath") = "$multiBin" ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#multi.bin.outPath") = "$multiBin" ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#multi.dev.outPath") = "$multiDev" ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#multi.dev.outputName") = dev ]] +[[ $(nix eval --json "path:$bakedDir-greeter#multi.outputs") = '["bin","dev"]' ]] +[[ $(nix eval "path:$bakedDir-greeter#multi" --apply 'x: builtins.length x.all') = 2 ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#multiDev.outputName") = dev ]] +[[ $(nix eval --raw "path:$bakedDir-greeter#multiDev.outPath") = "$multiDev" ]] + +# Building a specific output of a baked derivation works. +nix build --no-link "$greeterDir#multi^*" +[[ $(nix build --no-link --json "path:$bakedDir-greeter#multi^dev" | jq -r '.[0].outputs | keys | join(",")') = dev ]] +[[ $(nix build --no-link --json "path:$bakedDir-greeter#multi^dev" | jq -r '.[0].outputs.dev') = "$multiDev" ]] +[[ $(cat "$multiDev/what") = dev ]] + +# With --read-only, baking doesn't write any derivations to the store, +# and produces the same result. +roDir=$TEST_ROOT/ro +mkdir -p "$roDir" +cp "${config_nix}" "$roDir/" +cat > "$roDir/flake.nix" <.config.system.build.toplevel`) are baked +# at that sub-path, not at the output attribute itself. +configsDir=$TEST_ROOT/configs +mkdir -p "$configsDir" +cp ../simple.nix ../simple.builder.sh "${config_nix}" "$configsDir/" +cat > "$configsDir/flake.nix" < "$TEST_ROOT/show-configs.json" +[[ $(jq -r '.inventory.nixosConfigurations.output.children.foo.derivation.name' < "$TEST_ROOT/show-configs.json") = simple ]] +[[ $(jq -r '.inventory.homeConfigurations.output.children.bar.derivation.name' < "$TEST_ROOT/show-configs.json") = simple ]] + +# The inventory written by `nix flake bake` records the derivation attribute path. +nix flake bake "$configsDir" --dest-dir "$bakedDir-configs" +[[ $(jq -r '.nixosConfigurations.output.children.foo.derivation.name' < "$bakedDir-configs/outputs.json") = simple ]] +[[ $(jq -c '.nixosConfigurations.output.children.foo.derivationAttrPath' < "$bakedDir-configs/outputs.json") = '["config","system","build","toplevel"]' ]] +[[ $(jq -c '.homeConfigurations.output.children.bar.derivationAttrPath' < "$bakedDir-configs/outputs.json") = '["activationPackage"]' ]] + +# The baked derivations live at the nested path and have the same +# output paths as the originals... +toplevelPath=$(nix eval --raw "$configsDir#nixosConfigurations.foo.config.system.build.toplevel.outPath") +[[ $(nix eval --raw "path:$bakedDir-configs#nixosConfigurations.foo.config.system.build.toplevel.outPath") = "$toplevelPath" ]] +[[ $(nix eval --raw "path:$bakedDir-configs#nixosConfigurations.foo.config.system.build.toplevel.system") = "$system" ]] +activationPath=$(nix eval --raw "$configsDir#homeConfigurations.bar.activationPackage.outPath") +[[ $(nix eval --raw "path:$bakedDir-configs#homeConfigurations.bar.activationPackage.outPath") = "$activationPath" ]] + +# ... and not at the output attribute itself. +[[ $(nix eval "path:$bakedDir-configs#nixosConfigurations.foo" --apply 'x: x ? drvPath') = false ]] +[[ $(nix eval "path:$bakedDir-configs#homeConfigurations.bar" --apply 'x: x ? drvPath') = false ]] + +# The baked flake provides its own flake schemas, so `nix flake show` +# and `nix flake check` work even for outputs whose original schema +# needs attributes that the baked flake doesn't have (e.g. +# `pkgs.stdenv.system` for `nixosConfigurations`). +nix flake show --json "path:$bakedDir-configs" > "$TEST_ROOT/show-baked-configs.json" +[[ $(jq -r '.inventory.nixosConfigurations.output.children.foo.what' < "$TEST_ROOT/show-baked-configs.json") = "NixOS configuration" ]] +[[ $(jq -c '.inventory.nixosConfigurations.output.children.foo.forSystems' < "$TEST_ROOT/show-baked-configs.json") = "[\"$system\"]" ]] +[[ $(jq -r '.inventory.nixosConfigurations.output.children.foo.derivation.name' < "$TEST_ROOT/show-baked-configs.json") = simple ]] +nix flake show "path:$bakedDir-configs" | grepQuiet "NixOS configuration" +nix flake check --no-build "path:$bakedDir-configs" +nix flake check "path:$bakedDir-greeter" + +# Schema-level attributes are preserved, so e.g. `foo` still resolves to `packages..foo`... +[[ $(jq -c '.packages.defaultAttrPath' < "$bakedDir/outputs.json") = '["default"]' ]] +[[ $(jq -c '.packages.roles | sort' < "$bakedDir/outputs.json") = '["nix-build","nix-develop","nix-run","nix-search"]' ]] +[[ $(nix eval --raw "path:$bakedDir#default.outPath") = "$fooPath" ]] +# ... and `legacyPackages` is still hidden without `--legacy`. +[[ $(jq -r ".inventory.legacyPackages.output.children.\"$system\".isLegacy" < "$TEST_ROOT/show-baked.json") = true ]] + +homeDir=$TEST_ROOT/home +mkdir -p "$homeDir" +cp ../simple.nix ../simple.builder.sh "${config_nix}" "$homeDir/" +cat > "$homeDir/flake.nix" < "$TEST_ROOT/show-baked-home.json" +[[ $(jq -r '.inventory.homeConfigurations.output.children.bar.derivation.name' < "$TEST_ROOT/show-baked-home.json") = simple ]] +[[ $(jq -c '.inventory.homeConfigurations.output.children.bar.forSystems' < "$TEST_ROOT/show-baked-home.json") = "[\"$system\"]" ]] diff --git a/tests/functional/flakes/build-time-flake-inputs.sh b/tests/functional/flakes/build-time-flake-inputs.sh new file mode 100644 index 000000000000..d1fc1c453603 --- /dev/null +++ b/tests/functional/flakes/build-time-flake-inputs.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash + +source ./common.sh + +TODO_NixOS +enableFeatures "build-time-fetch-tree" +restartDaemon +requireGit + +lazy="$TEST_ROOT/lazy" +createGitRepo "$lazy" +echo world > "$lazy/who" +git -C "$lazy" add who +git -C "$lazy" commit -a -m foo + +repo="$TEST_ROOT/repo" + +createGitRepo "$repo" + +cat > "$repo/flake.nix" < "$lazy/who" +git -C "$lazy" commit -a -m foo + +nix flake update --flake "$repo" + +clearStore + +nix build --out-link "$TEST_ROOT/result" -L "$repo" +[[ $(cat "$TEST_ROOT/result") = utrecht ]] + +rm -rf "$lazy" + +clearStore + +expectStderr 1 nix build --out-link "$TEST_ROOT/result" -L "$repo" | grepQuiet "Cannot build.*source.drv" + +# `nix flake prefetch-inputs` should ignore build-time inputs. +depDir=$TEST_ROOT/dep +createGitRepo "$depDir" +createSimpleGitFlake "$depDir" + +cat > "$repo/flake.nix" < "$flakeDir"/flake.nix < "$flakeDir"/flake.nix <&1 && fail "nix flake check --all-systems should have failed" || true) -echo "$checkRes" | grepQuiet "error: overlay is not a function, but a set instead" +echo "$checkRes" | grepQuiet "error: Overlay is not a function." cat > "$flakeDir"/flake.nix <&1 && fail "nix flake check --all-systems should have failed" || true) -echo "$checkRes" | grepQuiet "unknown-attr" +echo "$checkRes" | grepQuiet "Evaluation check.*apps.system-1.default.isValidApp.*failed" cat > "$flakeDir"/flake.nix < "$flakeDir"/flake.nix < "$flakeDir/flake.nix" <"$TEST_HOME/flake.nix" +{ + inputs.nixpkgs.url = "$TEST_HOME/nixpkgs"; + outputs = {self, nixpkgs}: { + legacyPackages.$system.default = (import ./config.nix).mkDerivation { + name = "hello"; + buildCommand = "set -x; mkdir \$out"; + x = "foo"; + }; + }; +} +EOF + +(! nix develop . -L --command sh -c "echo \$x") +[[ $(nix develop .#default -L --command sh -c "echo \$x") == "foo" ]] diff --git a/tests/functional/flakes/dubious-query.sh b/tests/functional/flakes/dubious-query.sh index c147034c3dd9..bda5b8bf50f4 100644 --- a/tests/functional/flakes/dubious-query.sh +++ b/tests/functional/flakes/dubious-query.sh @@ -17,7 +17,7 @@ expectStderr 0 nix --offline build --dry-run "git+file://$repoDir?bar#foo" \ # Check that the anchor (#) is taken as a whole, not split, and throws an error. expectStderr 1 nix --offline build --dry-run "git+file://$repoDir#foo?bar" \ - | grepQuiet "error: flake 'git+file://$repoDir' does not provide attribute 'packages.$system.foo?bar', 'legacyPackages.$system.foo?bar' or 'foo?bar'" + | grepQuiet "error: flake 'git+file://$repoDir' does not provide attribute 'packages.$system.\"foo?bar\"', 'defaultPackage.$system.\"foo?bar\"', 'legacyPackages.$system.\"foo?bar\"' or '\"foo?bar\"'" # Check that a literal `?` in the query doesn't print dubious query warning. expectStderr 0 nix --offline build --dry-run "git+file://$repoDir?#foo" \ diff --git a/tests/functional/flakes/eval-cache.sh b/tests/functional/flakes/eval-cache.sh index 75a2c8cacbf9..450e59e95a0a 100755 --- a/tests/functional/flakes/eval-cache.sh +++ b/tests/functional/flakes/eval-cache.sh @@ -4,6 +4,42 @@ source ./common.sh requireGit +# Test basic caching: trace should only appear on first evaluation +basicCacheDir="$TEST_ROOT/basic-cache-flake" +createGitRepo "$basicCacheDir" "" +cp "${config_nix}" "$basicCacheDir/" +git -C "$basicCacheDir" add config.nix +git -C "$basicCacheDir" commit -m "config.nix" + +cat >"$basicCacheDir/flake.nix" < throw "pure-eval not enabled"; + builtins.trace "basic-test-evaluating" (mkDerivation { + name = "cached-build"; + buildCommand = "echo hello > \\\$out"; + }); + }; +} +EOF + +git -C "$basicCacheDir" add flake.nix +git -C "$basicCacheDir" commit -m "Init" + +clearBinaryCache + +# First build should show trace +nix build --no-link "$basicCacheDir" 2>&1 | grepQuiet 'trace: basic-test-evaluating' + +# Second build should use cache, no trace output +nix build --no-link "$basicCacheDir" 2>&1 | grepQuietInverse 'trace: basic-test-evaluating' + +# Third build should also use cache, no trace output +nix build --no-link "$basicCacheDir" 2>&1 | grepQuietInverse 'trace: basic-test-evaluating' + +# Test edge cases with separate flake flake1Dir="$TEST_ROOT/eval-cache-flake" createGitRepo "$flake1Dir" "" @@ -35,11 +71,11 @@ EOF git -C "$flake1Dir" add flake.nix git -C "$flake1Dir" commit -m "Init" -expect 1 nix build "$flake1Dir#foo.bar" 2>&1 | grepQuiet 'error: breaks' -expect 1 nix build "$flake1Dir#foo.bar" 2>&1 | grepQuiet 'error: breaks' +expect 1 nix build --no-link "$flake1Dir#foo.bar" 2>&1 | grepQuiet 'error: breaks' +expect 1 nix build --no-link "$flake1Dir#foo.bar" 2>&1 | grepQuiet 'error: breaks' # Stack overflow error must not be cached -expect 1 nix build --max-call-depth 50 "$flake1Dir#stack-depth" 2>&1 \ +expect 1 nix build --no-link --max-call-depth 50 "$flake1Dir#stack-depth" 2>&1 \ | grepQuiet 'error: stack overflow; max-call-depth exceeded' # If the SO is cached, the following invocation will produce a cached failure; we expect it to succeed nix build --no-link "$flake1Dir#stack-depth" @@ -48,3 +84,11 @@ nix build --no-link "$flake1Dir#stack-depth" expect 1 nix build "$flake1Dir#ifd" --option allow-import-from-derivation false 2>&1 \ | grepQuiet 'error: cannot build .* during evaluation because the option '\''allow-import-from-derivation'\'' is disabled' nix build --no-link "$flake1Dir#ifd" + +# Test that a store derivation is recreated when it has been deleted +# but the corresponding attribute is still cached. +if ! isTestOnNixOS; then + nix build --no-link "$flake1Dir#drv" + clearStore + nix build --no-link "$flake1Dir#drv" +fi diff --git a/tests/functional/flakes/flakes.sh b/tests/functional/flakes/flakes.sh index 948289a40fef..b9963e47f55a 100755 --- a/tests/functional/flakes/flakes.sh +++ b/tests/functional/flakes/flakes.sh @@ -66,6 +66,9 @@ nix flake metadata "$flake1Dir" | grepQuiet 'URL:.*flake1.*' # Test 'nix flake metadata --json'. json=$(nix flake metadata flake1 --json | jq .) [[ $(echo "$json" | jq -r .description) = 'Bla bla' ]] +if [[ $(nix config show lazy-trees) = false ]]; then + [[ -d $(echo "$json" | jq -r .path) ]] +fi [[ $(echo "$json" | jq -r .lastModified) = $(git -C "$flake1Dir" log -n1 --format=%ct) ]] hash1=$(echo "$json" | jq -r .revision) [[ -n $(echo "$json" | jq -r .fingerprint) ]] @@ -73,6 +76,7 @@ hash1=$(echo "$json" | jq -r .revision) echo foo > "$flake1Dir/foo" git -C "$flake1Dir" add "$flake1Dir"/foo [[ $(nix flake metadata flake1 --json --refresh | jq -r .dirtyRevision) == "$hash1-dirty" ]] +[[ $(_NIX_TEST_FAIL_ON_LARGE_PATH=1 nix flake metadata flake1 --json --refresh --warn-large-path-threshold 1 --lazy-trees | jq -r .dirtyRevision) == "$hash1-dirty" ]] [[ "$(nix flake metadata flake1 --json | jq -r .fingerprint)" != null ]] echo -n '# foo' >> "$flake1Dir/flake.nix" @@ -98,6 +102,16 @@ nix build -o "$TEST_ROOT/result" "git+file://$flake1Dir" (cd "$flake1Dir" && nix build -o "$TEST_ROOT/result" "path:.") (cd "$flake1Dir" && nix build -o "$TEST_ROOT/result" "git+file:.") +# Test 'nix eval --drv-link'. +drvPath=$(nix eval --raw --drv-link "$TEST_ROOT/drv" flake1#foo.drvPath) +[[ $(readlink "$TEST_ROOT/drv") = "$drvPath" ]] +[[ $(nix path-info "$TEST_ROOT/drv") = "$drvPath" ]] +rm "$TEST_ROOT/drv" + +# '--drv-link' requires '--raw' or '--json'. +expectStderr 1 nix eval --drv-link "$TEST_ROOT/drv" flake1#foo.drvPath | grepQuiet -- "--drv-link requires --raw or --json" +[[ ! -e "$TEST_ROOT/drv" ]] + # Test explicit packages.default. nix build -o "$TEST_ROOT/result" "$flake1Dir#default" nix build -o "$TEST_ROOT/result" "git+file://$flake1Dir#default" @@ -106,6 +120,11 @@ nix build -o "$TEST_ROOT/result" "git+file://$flake1Dir#default" nix build -o "$TEST_ROOT/result" "$flake1Dir?ref=HEAD#default" nix build -o "$TEST_ROOT/result" "git+file://$flake1Dir?ref=HEAD#default" +# Check that the fetcher cache works. +if [[ $(nix config show lazy-trees) = false ]]; then + nix build -o "$TEST_ROOT/result" "git+file://$flake1Dir?ref=HEAD#default" -vvvvv 2>&1 | grepQuiet "source path.*cache hit" +fi + # Check that relative paths are allowed for git flakes. # This may change in the future once git submodule support is refined. # See: https://discourse.nixos.org/t/57783 and #9708. @@ -157,7 +176,12 @@ expect 1 nix build -o "$TEST_ROOT/result" "$flake2Dir#bar" --no-update-lock-file nix build -o "$TEST_ROOT/result" "$flake2Dir#bar" --commit-lock-file [[ -e "$flake2Dir/flake.lock" ]] [[ -z $(git -C "$flake2Dir" diff main || echo failed) ]] -[[ $(jq --indent 0 --compact-output . < "$flake2Dir/flake.lock") =~ ^'{"nodes":{"flake1":{"locked":{"lastModified":'.*',"narHash":"sha256-'.*'","ref":"refs/heads/master","rev":"'.*'","revCount":2,"type":"git","url":"file:///'.*'"},"original":{"id":"flake1","type":"indirect"}},"root":{"inputs":{"flake1":"flake1"}}},"root":"root","version":7}'$ ]] +[[ $(jq --indent 0 --compact-output . < "$flake2Dir/flake.lock") =~ ^'{"nodes":{"flake1":{"locked":{"lastModified":'[0-9]*',"narHash":"sha256-'.*'","ref":"refs/heads/master","rev":"'.*'","revCount":2,"type":"git","url":"file:///'.*'"},"original":{"id":"flake1","type":"indirect"}},"root":{"inputs":{"flake1":"flake1"}}},"root":"root","version":7}'$ ]] +if [[ $(nix config show lazy-trees) = true ]]; then + # Test that `lazy-locks` causes NAR hashes to be omitted from the lock file. + nix flake update --flake "$flake2Dir" --commit-lock-file --lazy-locks + [[ $(jq --indent 0 --compact-output . < "$flake2Dir/flake.lock") =~ ^'{"nodes":{"flake1":{"locked":{"lastModified":'[0-9]*',"ref":"refs/heads/master","rev":"'.*'","revCount":2,"type":"git","url":"file:///'.*'"},"original":{"id":"flake1","type":"indirect"}},"root":{"inputs":{"flake1":"flake1"}}},"root":"root","version":7}'$ ]] +fi # Rerunning the build should not change the lockfile. nix build -o "$TEST_ROOT/result" "$flake2Dir#bar" @@ -266,6 +290,9 @@ nix registry remove flake1 [[ $(nix registry list | wc -l) == 4 ]] [[ $(nix registry resolve flake1) = "git+file://$flake1Dir" ]] +# Test the builtin fallback registry. +[[ $(nix registry resolve nixpkgs --flake-registry http://fail.invalid.org/sdklsdklsd --download-attempts 1) = https://flakehub.com/f/DeterminateSystems/nixpkgs-weekly/0.1 ]] + # Test 'nix registry list' with a disabled global registry. nix registry add user-flake1 git+file://"$flake1Dir" nix registry add user-flake2 "git+file://$percentEncodedFlake2Dir" @@ -431,7 +458,7 @@ nix flake metadata "$flake3Dir" --json --eval-store "dummy://?read-only=false" | rm -rf "$badFlakeDir" mkdir "$badFlakeDir" echo INVALID > "$badFlakeDir"/flake.nix -nix store delete "$(nix store add-path "$badFlakeDir")" +nix store delete --ignore-liveness "$(nix store add-path "$badFlakeDir")" [[ $(nix path-info "$(nix store add-path "$flake1Dir")") =~ flake1 ]] [[ $(nix path-info path:"$(nix store add-path "$flake1Dir")") =~ simple ]] diff --git a/tests/functional/flakes/get-flake.sh b/tests/functional/flakes/get-flake.sh index e462607db8f7..704f770e9a9c 100644 --- a/tests/functional/flakes/get-flake.sh +++ b/tests/functional/flakes/get-flake.sh @@ -21,6 +21,26 @@ cat > "$flake1Dir/subflake/flake.nix" < "$flake1Dir/flake.nix" < "$flakeDir/a" -(cd "$flakeDir" && nix flake init) # check idempotence +(cd "$flakeDir" && nix flake init --template "git+file://$templatesDir") # check idempotence # Test 'nix flake init' with conflicts createGitRepo "$flakeDir" echo b > "$flakeDir/a" pushd "$flakeDir" -(! nix flake init) |& grep "refusing to overwrite existing file \"$flakeDir/a\"" +(! nix flake init --template "git+file://$templatesDir") |& grep "refusing to overwrite existing file \"$flakeDir/a\"" popd git -C "$flakeDir" commit -a -m 'Changed' diff --git a/tests/functional/flakes/meson.build b/tests/functional/flakes/meson.build index 16660f6d015b..a6f864f0b1cf 100644 --- a/tests/functional/flakes/meson.build +++ b/tests/functional/flakes/meson.build @@ -36,6 +36,12 @@ suites += { 'trace-ifd.sh', 'get-flake.sh', 'type-error-fallback.sh', + 'build-time-flake-inputs.sh', + 'substitution.sh', + 'shallow.sh', + 'provenance.sh', + 'search.sh', + 'bake.sh', ], 'workdir' : meson.current_source_dir(), } diff --git a/tests/functional/flakes/provenance.sh b/tests/functional/flakes/provenance.sh new file mode 100644 index 000000000000..2004ccd5e693 --- /dev/null +++ b/tests/functional/flakes/provenance.sh @@ -0,0 +1,372 @@ +#!/usr/bin/env bash + +experimental_features="provenance" + +source common.sh + +TODO_NixOS + +createFlake1 + +outPath=$(nix build --print-out-paths --no-link "$flake1Dir#packages.$system.default") +drvPath=$(nix eval --raw "$flake1Dir#packages.$system.default.drvPath") +rev=$(nix flake metadata --json "$flake1Dir" | jq -r .locked.rev) +lastModified=$(nix flake metadata --json "$flake1Dir" | jq -r .locked.lastModified) +treePath=$(nix flake prefetch --json "$flake1Dir" | jq -r .storePath) +builder=$(nix eval --raw "$flake1Dir#packages.$system.default._builder") + +# Building a derivation should have tree+subpath+flake+meta+build provenance. +[[ "$(nix path-info --json --json-format 1 "$outPath" | jq ".\"$outPath\".provenance")" == "$(cat < "$flake1Dir/somefile" +git -C "$flake1Dir" add somefile +nix build --impure --print-out-paths --no-link "$flake1Dir#packages.$system.default" +[[ $(nix path-info --json --json-format 1 "$builder" | jq ".\"$builder\".provenance") != null ]] + +[[ "$(nix provenance show "$outPath")" = "$(cat < "$TEST_ROOT/counter" +cat > "$flake1Dir/flake.nix" < \$out + echo x >> "$TEST_ROOT/counter" + ''; + }; + }; + }; +} +EOF +outPath=$(nix build --print-out-paths --no-link "$flake1Dir") + +expectStderr 1 nix provenance verify --all | grepQuiet "derivation .* may not be deterministic: output .* differs" + +# Test various types of source files. +clearStore +echo x > "$TEST_ROOT/counter" +cat > "$flake1Dir/flake.nix" < "$TEST_ROOT/hello.txt" + +path="$(nix store prefetch-file --json "file://$TEST_ROOT/hello.txt" | jq -r .storePath)" + +[[ "$(nix provenance show "$path")" = $(cat < "$TEST_ROOT/hello.txt" + +expectStderr 1 nix provenance verify "$path" | grepQuiet "hash mismatch for URL" + +# Test invalid tag names +for name in "123-invalid" "invalid tag" "invalid@tag" "-invalid" " foo"; do + expectStderr 1 nix build --build-provenance-tags "{\"$name\": \"value\"}" --no-link "$flake1Dir#packages.$system.default" 2>&1 | grepQuiet "tag name '$name' is invalid" +done diff --git a/tests/functional/flakes/search.sh b/tests/functional/flakes/search.sh new file mode 100644 index 000000000000..5bce484e2025 --- /dev/null +++ b/tests/functional/flakes/search.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash + +source common.sh + +createFlake1 + +cat > "$flake1Dir/flake.nix" <> "$flake1Dir/flake.nix" +git -C "$flake1Dir" commit -a -m bla + +cat > "$repoDir"/flake.nix < show-output.json nix eval --impure --expr ' let show_output = builtins.fromJSON (builtins.readFile ./show-output.json); in -assert show_output.packages.someOtherSystem.default == {}; -assert show_output.packages.${builtins.currentSystem}.default.name == "simple"; -assert show_output.legacyPackages.${builtins.currentSystem} == {}; +assert show_output.inventory.packages.output.children.someOtherSystem.filtered; +assert show_output.inventory.packages.output.children.${builtins.currentSystem}.children.default.derivation.name == "simple"; +assert show_output.inventory.legacyPackages.output.children.${builtins.currentSystem}.isLegacy; true ' @@ -28,8 +28,8 @@ nix flake show --json --all-systems > show-output.json nix eval --impure --expr ' let show_output = builtins.fromJSON (builtins.readFile ./show-output.json); in -assert show_output.packages.someOtherSystem.default.name == "simple"; -assert show_output.legacyPackages.${builtins.currentSystem} == {}; +assert show_output.inventory.packages.output.children.someOtherSystem.children.default.derivation.name == "simple"; +assert show_output.inventory.legacyPackages.output.children.${builtins.currentSystem}.isLegacy; true ' @@ -39,34 +39,7 @@ nix flake show --json --legacy > show-output.json nix eval --impure --expr ' let show_output = builtins.fromJSON (builtins.readFile ./show-output.json); in -assert show_output.legacyPackages.${builtins.currentSystem}.hello.name == "simple"; -true -' - -# Test that attributes are only reported when they have actual content -cat >flake.nix < show-output.json -nix eval --impure --expr ' -let show_output = builtins.fromJSON (builtins.readFile ./show-output.json); -in -assert show_output == { }; +assert show_output.inventory.legacyPackages.output.children.${builtins.currentSystem}.children.hello.derivation.name == "simple"; true ' @@ -87,8 +60,8 @@ nix flake show --json --legacy --all-systems > show-output.json nix eval --impure --expr ' let show_output = builtins.fromJSON (builtins.readFile ./show-output.json); in -assert show_output.legacyPackages.${builtins.currentSystem}.AAAAAASomeThingsFailToEvaluate == { }; -assert show_output.legacyPackages.${builtins.currentSystem}.simple.name == "simple"; +assert show_output.inventory.legacyPackages.output.children.${builtins.currentSystem}.children.AAAAAASomeThingsFailToEvaluate.failed; +assert show_output.inventory.legacyPackages.output.children.${builtins.currentSystem}.children.simple.derivation.name == "simple"; true ' @@ -98,35 +71,4 @@ popd writeIfdFlake "$flakeDir" pushd "$flakeDir" - -nix flake show --json > show-output.json -# shellcheck disable=SC2016 -nix eval --impure --expr ' -let show_output = builtins.fromJSON (builtins.readFile ./show-output.json); -in -assert show_output.packages.${builtins.currentSystem}.default == { }; -true -' - - -# Test that nix keeps going even when packages.$SYSTEM contains not derivations -cat >flake.nix < show-output.json -# shellcheck disable=SC2016 -nix eval --impure --expr ' -let show_output = builtins.fromJSON (builtins.readFile ./show-output.json); -in -assert show_output.packages.${builtins.currentSystem}.not-a-derivation == {}; -true -' - +[[ $(nix flake show --json | jq -r ".inventory.packages.output.children.\"$system\".children.default.derivation.name") = top ]] diff --git a/tests/functional/flakes/source-paths.sh b/tests/functional/flakes/source-paths.sh index 3aa3683c27cf..759c637a5207 100644 --- a/tests/functional/flakes/source-paths.sh +++ b/tests/functional/flakes/source-paths.sh @@ -30,10 +30,10 @@ expectStderr 1 nix eval "$repo#y" | grepQuiet "at $repo/flake.nix:" git -C "$repo" commit -a -m foo -expectStderr 1 nix eval "git+file://$repo?ref=master#y" | grepQuiet "at «git+file://$repo?ref=master&rev=.*»/flake.nix:" +expectStderr 1 nix eval "git+file://$repo?ref=master#y" | grepQuiet "at «git+file://$repo?rev=.*»/flake.nix:" expectStderr 1 nix eval "$repo#z" | grepQuiet "error: Path 'foo' does not exist in Git repository \"$repo\"." -expectStderr 1 nix eval "git+file://$repo?ref=master#z" | grepQuiet "error: '«git+file://$repo?ref=master&rev=.*»/foo' does not exist" +expectStderr 1 nix eval "git+file://$repo?ref=master#z" | grepQuiet "error: path '«git+file://$repo?rev=.*»/foo' does not exist" expectStderr 1 nix eval "$repo#a" | grepQuiet "error: Path 'foo' does not exist in Git repository \"$repo\"." echo 123 > "$repo/foo" @@ -55,3 +55,11 @@ expectStderr 1 nix eval "$repo#b" | grepQuiet "error: Path 'dir' in the reposito git -C "$repo" add "$repo/dir/default.nix" [[ $(nix eval "$repo#b") = 456 ]] + +# Check that error messages in impure mode correctly refer to the original path, not a store path. +expectStderr 1 nix eval --expr "builtins.readFile ((builtins.fetchTree { type = \"git\"; url = \"file://$repo\"; }) + \"/README.md\")" --impure | grepQuiet "Path 'README.md' does not exist in Git repository \"$repo\"." + +# This should still be the case if the input is in the store. +narHash=$(nix flake prefetch --json "$repo" | jq -r .locked.narHash) +expectStderr 1 nix eval --expr "builtins.readFile ((builtins.fetchTree { type = \"git\"; url = \"file://$repo\"; narHash = \"$narHash\"; }) + \"/README.md\")" --impure | grepQuiet "path '.*git+file://$repo.*/README.md' does not exist" +expectStderr 1 nix eval --expr "builtins.readFile ((builtins.fetchTree { type = \"git\"; url = \"file://$repo\"; narHash = \"$narHash\"; }) + \"/README.md\")" | grepQuiet "path '.*git+file://$repo.*/README.md' does not exist" diff --git a/tests/functional/flakes/substitution.sh b/tests/functional/flakes/substitution.sh new file mode 100644 index 000000000000..97a04931abfc --- /dev/null +++ b/tests/functional/flakes/substitution.sh @@ -0,0 +1,31 @@ +#! /usr/bin/env bash + +# Test that inputs are substituted if they cannot be fetched from their original location. + +source ./common.sh + +if [[ $(nix config show lazy-trees) = true ]]; then + exit 0 +fi + +TODO_NixOS + +createFlake1 +createFlake2 + +nix build --no-link "$flake2Dir#bar" + +path1="$(nix flake metadata --json "$flake1Dir" | jq -r .path)" + +# Building after an input disappeared should succeed, because it's still in the Nix store. +mv "$flake1Dir" "$flake1Dir-tmp" +nix build --no-link "$flake2Dir#bar" --no-eval-cache + +# Check that Nix will fall back to fetching the input from a substituter. +cache="file://$TEST_ROOT/binary-cache" +nix copy --to "$cache" "$path1" +clearStore +nix build --no-link "$flake2Dir#bar" --no-eval-cache --substitute --substituters "$cache" + +clearStore +expectStderr 1 nix build --no-link "$flake2Dir#bar" --no-eval-cache | grepQuiet "Git repository.*does not exist" diff --git a/tests/functional/flakes/unlocked-override.sh b/tests/functional/flakes/unlocked-override.sh index ed05440de03b..ed4d131b7ad0 100755 --- a/tests/functional/flakes/unlocked-override.sh +++ b/tests/functional/flakes/unlocked-override.sh @@ -36,6 +36,7 @@ expectStderr 1 nix flake lock "$flake2Dir" --override-input flake1 "$TEST_ROOT/f grepQuiet "Not writing lock file.*because it has an unlocked input" nix flake lock "$flake2Dir" --override-input flake1 "$TEST_ROOT/flake1" --allow-dirty-locks +_NIX_TEST_FAIL_ON_LARGE_PATH=1 nix flake lock "$flake2Dir" --override-input flake1 "$TEST_ROOT/flake1" --allow-dirty-locks --warn-large-path-threshold 1 --lazy-trees # Using a lock file with a dirty lock does not require --allow-dirty-locks, but should print a warning. expectStderr 0 nix eval "$flake2Dir#x" | diff --git a/tests/functional/formatter.sh b/tests/functional/formatter.sh index 444973e752d3..898982f51840 100755 --- a/tests/functional/formatter.sh +++ b/tests/functional/formatter.sh @@ -82,4 +82,6 @@ rm ./my-result # Flake outputs check. nix flake check -nix flake show | grep -P "package 'formatter'" + +clearStore +expectStderr 0 nix flake show | grepQuiet ": formatter" diff --git a/tests/functional/gc-closure.sh b/tests/functional/gc-closure.sh index ee91667cd21f..e2b8b3fbd88b 100755 --- a/tests/functional/gc-closure.sh +++ b/tests/functional/gc-closure.sh @@ -12,7 +12,7 @@ nix_gc_closure() { input0=$(realpath "$TEST_ROOT/gc-root") input1=$(nix build -f dependencies2.nix input1_drv --no-link --print-out-paths) input2=$(nix build -f dependencies2.nix input2_drv --no-link --print-out-paths) - input2_out=$(printf "%s" "$input2" | head -n1) + input2_out=$(head -n1 <<< "$input2") input2_out2=$(printf "%s" "$input2" | tail -n1) top=$(nix build -f dependencies2.nix --no-link --print-out-paths) something_else=$(nix store add-path ./dependencies2.nix) diff --git a/tests/functional/gc-runtime.nix b/tests/functional/gc-runtime.nix index 10dc5e29f56d..05ce67562fe1 100644 --- a/tests/functional/gc-runtime.nix +++ b/tests/functional/gc-runtime.nix @@ -20,6 +20,7 @@ with import ./config.nix; cat > $out/program << 'EOF' #! ${shell} + echo x > "$2" sleep 10000 < "$1" EOF diff --git a/tests/functional/gc-runtime.sh b/tests/functional/gc-runtime.sh index ac788406dea1..9b675cb97329 100755 --- a/tests/functional/gc-runtime.sh +++ b/tests/functional/gc-runtime.sh @@ -9,18 +9,25 @@ case $system in skipTest "Not running Linux"; esac +TODO_NixOS + set -m # enable job control, needed for kill programPath=$(nix-build --no-link ./gc-runtime.nix -A program) environPath=$(nix-build --no-link ./gc-runtime.nix -A environ) openPath=$(nix-build --no-link ./gc-runtime.nix -A open) +fifo="$TEST_ROOT/fifo" +mkfifo "$fifo" + echo "backgrounding program..." export environPath -"$programPath"/program "$openPath"/open & -sleep 2 # hack - wait for the program to get started +"$programPath"/program "$openPath"/open "$fifo" & child=$! echo PID=$child +cat "$fifo" + +expectStderr 1 nix-store --delete "$openPath" | grepQuiet "Cannot delete path.*because it's referenced by the GC root '/proc/" nix-store --gc diff --git a/tests/functional/gc.sh b/tests/functional/gc.sh index aa20c5a0339f..6eb66ee4dc01 100755 --- a/tests/functional/gc.sh +++ b/tests/functional/gc.sh @@ -11,7 +11,7 @@ outPath=$(nix-store -rvv "$drvPath") rm -f "$NIX_STATE_DIR/gcroots/foo" ln -sf "$outPath" "$NIX_STATE_DIR/gcroots/foo" -[ "$(nix-store -q --roots "$outPath")" = "$NIX_STATE_DIR/gcroots/foo -> $outPath" ] +expectStderr 0 nix-store -q --roots "$outPath" | grepQuiet "$NIX_STATE_DIR/gcroots/foo -> $outPath" nix-store --gc --print-roots | grep "$outPath" nix-store --gc --print-live | grep "$outPath" @@ -21,10 +21,10 @@ if nix-store --gc --print-dead | grep -E "$outPath"$; then false; fi nix-store --gc --print-dead inUse=$(readLink "$outPath/reference-to-input-2") -if nix-store --delete "$inUse"; then false; fi +expectStderr 1 nix-store --delete "$inUse" | grepQuiet "Cannot delete path.*because it's referenced by path '" test -e "$inUse" -if nix-store --delete "$outPath"; then false; fi +expectStderr 1 nix-store --delete "$outPath" | grepQuiet "Cannot delete path.*because it's referenced by the GC root " test -e "$outPath" for i in "$NIX_STORE_DIR"/*; do diff --git a/tests/functional/git/exact-rev-fetch-bucketed-refs.sh b/tests/functional/git/exact-rev-fetch-bucketed-refs.sh new file mode 100644 index 000000000000..ee01e64b00f6 --- /dev/null +++ b/tests/functional/git/exact-rev-fetch-bucketed-refs.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash + +# Rev-pinned fetches used to leave the fetched commit un-refed (only in +# FETCH_HEAD), so a later rev-pinned fetch of the same URL had nothing to +# negotiate from and re-downloaded objects already on disk. Nix now fetches +# into refs//tip-, so a later fetch of a +# rev in the same bucket can negotiate against it. + +source ../common.sh + +requireGit + +repo=$TEST_ROOT/repo +trace=$TEST_ROOT/git-trace + +createGitRepo "$repo" + +echo first > "$repo/first" +git -C "$repo" add first +git -C "$repo" commit -m 'First commit.' +rev1=$(git -C "$repo" rev-parse HEAD) + +echo second > "$repo/second" +git -C "$repo" add second +git -C "$repo" commit -m 'Second commit.' +rev2=$(git -C "$repo" rev-parse HEAD) + +export _NIX_FORCE_HTTP=1 + +nix-instantiate --eval --raw -E "(builtins.fetchGit { url = \"file://$repo\"; rev = \"$rev1\"; }).outPath" >/dev/null + +reservedRefNamespace="__nix_internal_fetchers_48ae34d6b380c5ec__"; + +cacheDir=$(find "$TEST_HOME/.cache/nix/gitv3" -maxdepth 1 -mindepth 1 -type d) + +bucket1=${rev1:0:2} +if ! git -C "$cacheDir" rev-parse --verify -q "refs/$reservedRefNamespace/tip-$bucket1" >/dev/null; then + echo "Expected refs/nix/tip-$bucket1 to exist after fetching $rev1." >&2 + exit 1 +fi +if [[ $(git -C "$cacheDir" rev-parse "refs/$reservedRefNamespace/tip-$bucket1") != "$rev1" ]]; then + echo "Expected refs/nix/tip-$bucket1 to point at $rev1." >&2 + exit 1 +fi + +export GIT_TRACE_PACKET=$trace +nix-instantiate --eval --raw -E "(builtins.fetchGit { url = \"file://$repo\"; rev = \"$rev2\"; }).outPath" >/dev/null +unset GIT_TRACE_PACKET + +if ! grep -q "fetch> have $rev1" "$trace"; then + echo "Expected the second fetch to advertise $rev1 (via refs/$reservedRefNamespace/tip-$bucket1) as a negotiation tip." >&2 + cat "$trace" >&2 + exit 1 +fi + +totals=$(grep -oE 'sideband< .*Total [0-9]+' "$trace" | sed -E 's/.*Total ([0-9]+)/\1/') +if [[ $totals -ge 6 ]]; then + echo "Expected the second fetch to receive only new objects (got Total $totals, same as a full re-fetch would)." >&2 + exit 1 +fi + +bucket2=${rev2:0:2} +if [[ $(git -C "$cacheDir" rev-parse "refs/$reservedRefNamespace/tip-$bucket2") != "$rev2" ]]; then + echo "Expected refs/nix/tip-$bucket2 to point at $rev2 after the second fetch." >&2 + exit 1 +fi + +if ! git -C "$cacheDir" cat-file -e "$rev1"; then + echo "Expected $rev1 to remain resolvable after the second fetch." >&2 + exit 1 +fi diff --git a/tests/functional/git/meson.build b/tests/functional/git/meson.build index 73c4bc782871..d1c5686f042b 100644 --- a/tests/functional/git/meson.build +++ b/tests/functional/git/meson.build @@ -2,6 +2,7 @@ suites += { 'name' : 'git', 'deps' : [], 'tests' : [ + 'exact-rev-fetch-bucketed-refs.sh', 'packed-refs-no-cache.sh', 'symlinked-repo.sh', ], diff --git a/tests/functional/git/packed-refs-no-cache.sh b/tests/functional/git/packed-refs-no-cache.sh index 33ffd5fa33e9..fd369e6b1e44 100644 --- a/tests/functional/git/packed-refs-no-cache.sh +++ b/tests/functional/git/packed-refs-no-cache.sh @@ -21,11 +21,7 @@ repo=$TEST_ROOT/./git export _NIX_FORCE_HTTP=1 -rm -rf "$repo" "${repo}-tmp" "$TEST_HOME/.cache/nix" - -git init --initial-branch="master" "$repo" -git -C "$repo" config user.email "nix-tests@example.com" -git -C "$repo" config user.name "Nix Tests" +createGitRepo "$repo" echo "hello world" > "$repo/hello_world" git -C "$repo" add hello_world diff --git a/tests/functional/impure-derivations.sh b/tests/functional/impure-derivations.sh index 2c15e628dd42..b7a0d0ac2aa1 100755 --- a/tests/functional/impure-derivations.sh +++ b/tests/functional/impure-derivations.sh @@ -19,7 +19,7 @@ drvPath2=$(nix derivation add < "$TEST_HOME"/impure-drv.json) [[ "$drvPath" = "$drvPath2" ]] # But only with the experimental feature! -expectStderr 1 nix derivation add < "$TEST_HOME"/impure-drv.json --experimental-features nix-command | grepQuiet "experimental Nix feature 'impure-derivations' is disabled" +expectStderr 1 nix derivation add < "$TEST_HOME"/impure-drv.json --experimental-features '' | grepQuiet "experimental Nix feature 'impure-derivations' is disabled" nix build --dry-run --json --file ./impure-derivations.nix impure.all json=$(nix build -L --no-link --json --file ./impure-derivations.nix impure.all) diff --git a/tests/functional/lang/eval-fail-blackhole.err.exp b/tests/functional/lang/eval-fail-blackhole.err.exp index 95e33a5fe456..6866c58dd414 100644 --- a/tests/functional/lang/eval-fail-blackhole.err.exp +++ b/tests/functional/lang/eval-fail-blackhole.err.exp @@ -7,8 +7,8 @@ error: 3| x = y; error: infinite recursion encountered - at /pwd/lang/eval-fail-blackhole.nix:3:7: + at /pwd/lang/eval-fail-blackhole.nix:2:10: + 1| let { 2| body = x; + | ^ 3| x = y; - | ^ - 4| y = x; diff --git a/tests/functional/lang/eval-fail-deepseq-stack-overflow.err.exp b/tests/functional/lang/eval-fail-deepseq-stack-overflow.err.exp index 4cc43ca095e1..f142b5c4d45d 100644 --- a/tests/functional/lang/eval-fail-deepseq-stack-overflow.err.exp +++ b/tests/functional/lang/eval-fail-deepseq-stack-overflow.err.exp @@ -23,8 +23,3 @@ error: 7| in error: stack overflow; max-call-depth exceeded - at /pwd/lang/eval-fail-deepseq-stack-overflow.nix:5:28: - 4| let - 5| long = builtins.genList (x: x) 100000; - | ^ - 6| reverseLinkedList = builtins.foldl' (tail: head: { inherit head tail; }) null long; diff --git a/tests/functional/lang/eval-fail-derivation-name.err.exp b/tests/functional/lang/eval-fail-derivation-name.err.exp index 76d0575195ba..8f29ec12c7c5 100644 --- a/tests/functional/lang/eval-fail-derivation-name.err.exp +++ b/tests/functional/lang/eval-fail-derivation-name.err.exp @@ -9,7 +9,7 @@ error: … while calling the 'derivationStrict' builtin at «nix-internal»/derivation-internal.nix::: | - | strict = derivationStrict drvAttrs; + | strict = drvFunc drvAttrs; | ^ | diff --git a/tests/functional/lang/eval-fail-derivation-structuredAttrs-stack-overflow.err.exp b/tests/functional/lang/eval-fail-derivation-structuredAttrs-stack-overflow.err.exp index ec849e15d1fe..b8fe074bf13c 100644 --- a/tests/functional/lang/eval-fail-derivation-structuredAttrs-stack-overflow.err.exp +++ b/tests/functional/lang/eval-fail-derivation-structuredAttrs-stack-overflow.err.exp @@ -1,17 +1,17 @@ error: … while evaluating the attribute 'outPath' - at «nix-internal»/derivation-internal.nix:50:7: - 49| value = commonAttrs // { - 50| outPath = strict.${outputName}; + at «nix-internal»/derivation-internal.nix:51:7: + 50| value = commonAttrs // { + 51| outPath = strict.${outputName}; | ^ - 51| drvPath = strict.drvPath; + 52| drvPath = strict.drvPath; … while calling the 'derivationStrict' builtin - at «nix-internal»/derivation-internal.nix:37:12: - 36| - 37| strict = derivationStrict drvAttrs; + at «nix-internal»/derivation-internal.nix:38:12: + 37| + 38| strict = drvFunc drvAttrs; | ^ - 38| + 39| … while evaluating derivation 'test' whose name attribute is located at /pwd/lang/eval-fail-derivation-structuredAttrs-stack-overflow.nix:5:3 diff --git a/tests/functional/lang/eval-fail-genList-3.err.exp b/tests/functional/lang/eval-fail-genList-3.err.exp index 62878d88eb5c..0ab17037b598 100644 --- a/tests/functional/lang/eval-fail-genList-3.err.exp +++ b/tests/functional/lang/eval-fail-genList-3.err.exp @@ -1,12 +1,6 @@ error: … while evaluating list element at index 0 - … from call site - at /pwd/lang/eval-fail-genList-3.nix:1:19: - 1| builtins.genList (x: x + "foo") 2 - | ^ - 2| - … while calling anonymous lambda at /pwd/lang/eval-fail-genList-3.nix:1:19: 1| builtins.genList (x: x + "foo") 2 diff --git a/tests/functional/lang/eval-fail-hashString-2.err.exp b/tests/functional/lang/eval-fail-hashString-2.err.exp index edbf5e23ff78..a633274076d7 100644 --- a/tests/functional/lang/eval-fail-hashString-2.err.exp +++ b/tests/functional/lang/eval-fail-hashString-2.err.exp @@ -6,4 +6,5 @@ error: 2| error: unknown hash algorithm 'foo', expect 'blake3', 'md5', 'sha1', 'sha256', or 'sha512' + Try 'nix-instantiate --help' for more information. diff --git a/tests/functional/lang/eval-fail-memoised-error-trace-not-mutated.err.exp b/tests/functional/lang/eval-fail-memoised-error-trace-not-mutated.err.exp index f8807b6bd9a5..9327371bf228 100644 --- a/tests/functional/lang/eval-fail-memoised-error-trace-not-mutated.err.exp +++ b/tests/functional/lang/eval-fail-memoised-error-trace-not-mutated.err.exp @@ -15,6 +15,8 @@ error: … forcing d + … forcing c + … forcing b … while calling the 'throw' builtin diff --git a/tests/functional/lang/eval-fail-recursion.err.exp b/tests/functional/lang/eval-fail-recursion.err.exp index ee41ff46bea9..21bf7a695bdd 100644 --- a/tests/functional/lang/eval-fail-recursion.err.exp +++ b/tests/functional/lang/eval-fail-recursion.err.exp @@ -1,14 +1,14 @@ error: … in the right operand of the update (//) operator - at /pwd/lang/eval-fail-recursion.nix:2:14: + at /pwd/lang/eval-fail-recursion.nix:2:11: 1| let 2| a = { } // a; - | ^ + | ^ 3| in error: infinite recursion encountered - at /pwd/lang/eval-fail-recursion.nix:2:14: - 1| let - 2| a = { } // a; - | ^ + at /pwd/lang/eval-fail-recursion.nix:4:1: 3| in + 4| a.foo + | ^ + 5| diff --git a/tests/functional/lang/eval-fail-scope-5.err.exp b/tests/functional/lang/eval-fail-scope-5.err.exp index 6edc85f4f161..20027a7f1f57 100644 --- a/tests/functional/lang/eval-fail-scope-5.err.exp +++ b/tests/functional/lang/eval-fail-scope-5.err.exp @@ -21,8 +21,8 @@ error: 8| x ? y, error: infinite recursion encountered - at /pwd/lang/eval-fail-scope-5.nix:8:11: - 7| { - 8| x ? y, - | ^ - 9| y ? x, + at /pwd/lang/eval-fail-scope-5.nix:11:5: + 10| }: + 11| x + y; + | ^ + 12| diff --git a/tests/functional/lang/eval-fail-zipAttrsWith-3.err.exp b/tests/functional/lang/eval-fail-zipAttrsWith-3.err.exp index 780643b9df28..505a7a7b43d4 100644 --- a/tests/functional/lang/eval-fail-zipAttrsWith-3.err.exp +++ b/tests/functional/lang/eval-fail-zipAttrsWith-3.err.exp @@ -2,7 +2,3 @@ error: … while evaluating the attribute 'foo' error: attempt to call something which is not a function but an integer: 1 - at /pwd/lang/eval-fail-zipAttrsWith-3.nix:1:24: - 1| builtins.zipAttrsWith (_: 1) [ { foo = 1; } ] - | ^ - 2| diff --git a/tests/functional/lang/eval-fail-zipAttrsWith-4.err.exp b/tests/functional/lang/eval-fail-zipAttrsWith-4.err.exp index 034d84254198..72c559fbc4d0 100644 --- a/tests/functional/lang/eval-fail-zipAttrsWith-4.err.exp +++ b/tests/functional/lang/eval-fail-zipAttrsWith-4.err.exp @@ -1,12 +1,6 @@ error: … while evaluating the attribute 'foo' - … from call site - at /pwd/lang/eval-fail-zipAttrsWith-4.nix:1:24: - 1| builtins.zipAttrsWith (a: b: a + b) [ - | ^ - 2| { foo = 1; } - … while calling anonymous lambda at /pwd/lang/eval-fail-zipAttrsWith-4.nix:1:27: 1| builtins.zipAttrsWith (a: b: a + b) [ diff --git a/tests/functional/lang/eval-okay-filterattrs-names.exp b/tests/functional/lang/eval-okay-filterattrs-names.exp new file mode 100644 index 000000000000..3f07d6e1a028 --- /dev/null +++ b/tests/functional/lang/eval-okay-filterattrs-names.exp @@ -0,0 +1 @@ +{ a = 3; } diff --git a/tests/functional/lang/eval-okay-filterattrs-names.nix b/tests/functional/lang/eval-okay-filterattrs-names.nix new file mode 100644 index 000000000000..94108fbefdaf --- /dev/null +++ b/tests/functional/lang/eval-okay-filterattrs-names.nix @@ -0,0 +1,5 @@ +builtins.filterAttrs (name: value: name == "a") { + a = 3; + b = 6; + c = 10; +} diff --git a/tests/functional/lang/eval-okay-filterattrs.exp b/tests/functional/lang/eval-okay-filterattrs.exp new file mode 100644 index 000000000000..74b9825e9c42 --- /dev/null +++ b/tests/functional/lang/eval-okay-filterattrs.exp @@ -0,0 +1 @@ +{ b = 6; c = 10; } diff --git a/tests/functional/lang/eval-okay-filterattrs.nix b/tests/functional/lang/eval-okay-filterattrs.nix new file mode 100644 index 000000000000..28d37bbe7843 --- /dev/null +++ b/tests/functional/lang/eval-okay-filterattrs.nix @@ -0,0 +1,5 @@ +builtins.filterAttrs (name: value: value > 5) { + a = 3; + b = 6; + c = 10; +} diff --git a/tests/functional/linux-sandbox.sh b/tests/functional/linux-sandbox.sh index d2c64df2f300..dd83bc4bc7a5 100755 --- a/tests/functional/linux-sandbox.sh +++ b/tests/functional/linux-sandbox.sh @@ -146,3 +146,30 @@ nix-sandbox-build symlink-derivation.nix -A test_sandbox_paths \ --option extra-sandbox-paths "//dir/=$TEST_ROOT/" \ --option extra-sandbox-paths "///./symlinkDir=$symlinkDir" \ --option extra-sandbox-paths "/symlink=$symlinkcert" + +# Nonexistent sandbox paths should error early in the build process +# shellcheck disable=SC2016 +expectStderr 1 nix-sandbox-build --option extra-sandbox-paths '/does-not-exist' \ + -E 'with import '"${config_nix}"'; mkDerivation { name = "trivial"; buildCommand = "echo > $out"; }' | + grepQuiet "path \"/does-not-exist\" is configured as part of the \`sandbox-paths\` option, but is inaccessible" + +# Test pre-build-hook. +DEST="$TEST_ROOT/hook-output" +HOOK="$TEST_ROOT/pre-build-hook" + +echo foo > "$TEST_ROOT"/fnord + +cat > "$HOOK" < "$DEST" +echo "hello from hook!" >&2 +echo "extra-sandbox-paths" +echo "/foo/bar=$TEST_ROOT/fnord" +EOF +chmod +x "$HOOK" + +outPath=$(nix-build --no-out-link --sandbox-paths /nix/store --pre-build-hook "$HOOK" symlink-derivation.nix -A test_sandbox_paths_2) + +[[ $(cat "$TEST_ROOT/store0/nix/store/$(basename "$outPath")/xyzzy") = foo ]] + +[[ "$(cat "$DEST")" == "test-sandbox-paths-2" ]] diff --git a/tests/functional/local-overlay-store/delete-refs-inner.sh b/tests/functional/local-overlay-store/delete-refs-inner.sh index f54ef2bb6b49..708e8c5a8dfd 100644 --- a/tests/functional/local-overlay-store/delete-refs-inner.sh +++ b/tests/functional/local-overlay-store/delete-refs-inner.sh @@ -23,14 +23,14 @@ input2=$(nix-build ../hermetic.nix --no-out-link --arg busybox "$busybox" --arg input3=$(nix-build ../hermetic.nix --no-out-link --arg busybox "$busybox" --arg withFinalRefs true --arg seed 2 -A passthru.input3 -j0) # Can't delete because referenced -expectStderr 1 nix-store --delete "$input1" | grepQuiet "Cannot delete path" -expectStderr 1 nix-store --delete "$input2" | grepQuiet "Cannot delete path" -expectStderr 1 nix-store --delete "$input3" | grepQuiet "Cannot delete path" +expectStderr 1 nix-store --delete "$input1" | grepQuiet "Cannot delete path.*because it's referenced by path" +expectStderr 1 nix-store --delete "$input2" | grepQuiet "Cannot delete path.*because it's referenced by path" +expectStderr 1 nix-store --delete "$input3" | grepQuiet "Cannot delete path.*because it's referenced by path" # These same paths are referenced in the lower layer (by the seed 1 # build done in `initLowerStore`). -expectStderr 1 nix-store --store "$storeA" --delete "$input2" | grepQuiet "Cannot delete path" -expectStderr 1 nix-store --store "$storeA" --delete "$input3" | grepQuiet "Cannot delete path" +expectStderr 1 nix-store --store "$storeA" --delete "$input2" | grepQuiet "Cannot delete path.*because it's referenced by path" +expectStderr 1 nix-store --store "$storeA" --delete "$input3" | grepQuiet "Cannot delete path.*because it's referenced by path" # Can delete nix-store --delete "$hermetic" diff --git a/tests/functional/logging.sh b/tests/functional/logging.sh index b932752458a2..3ebd51fe528b 100755 --- a/tests/functional/logging.sh +++ b/tests/functional/logging.sh @@ -33,10 +33,22 @@ if isDaemonNewer "2.26"; then fi # Test json-log-path. -if [[ "$NIX_REMOTE" != "daemon" ]]; then - clearStore - nix build -vv --file dependencies.nix --no-link --json-log-path "$TEST_ROOT/log.json" 2>&1 | grepQuiet 'building.*dependencies-top.drv' - jq < "$TEST_ROOT/log.json" - grep '{"action":"start","fields":\[".*-dependencies-top.drv","",1,1\],"id":.*,"level":3,"parent":0' "$TEST_ROOT/log.json" >&2 - (( $(grep -c '{"action":"msg","level":5,"msg":"executing builder .*"}' "$TEST_ROOT/log.json" ) == 5 )) -fi +clearStore +nix build -vv --file dependencies.nix --no-link --json-log-path "$TEST_ROOT/log.json" 2>&1 | grepQuiet 'building.*dependencies-top.drv' +grep '{"action":"start","fields":\[".*-dependencies-top.drv","",1,1\],"id":.*,"level":3,"parent":[1-9]' "$TEST_ROOT/log.json" >&2 +grep -E '{"action":"result","id":[^,]+,"payload":{"builtOutputs":{"out":{"outPath":"[^-]+-dependencies-top".*"status":"Built".*"success":true' "$TEST_ROOT/log.json" >&2 +(( $(grep -c '{"action":"msg","level":5,"msg":"executing builder .*"}' "$TEST_ROOT/log.json" ) == 5 )) + +rm "$TEST_ROOT/log.json" +expect 1 nix build -vv --file timeout.nix silent --timeout 1 --no-link --json-log-path "$TEST_ROOT/log.json" +grep -E '{"action":"result","id":[^,]+,"payload":{"errorMsg":"timed out.*",.*"startTime":[1-9][0-9]*,"status":"TimedOut","stopTime":0,"success":false,' "$TEST_ROOT/log.json" >&2 + +# Check that all log entries have the same session ID. +sid=$(head -n1 < "$TEST_ROOT/log.json" | jq -r '.sid') +[[ -n "$sid" && "$sid" != "null" ]] +(( $(jq -s --arg sid "$sid" '[.[] | select(.sid != $sid)] | length' < "$TEST_ROOT/log.json") == 0 )) + +# Test whether setting an explicit session ID works. +nix store info --json-log-path "$TEST_ROOT/log2.json" --session-id "foo" +(( $(jq -s 'length' < "$TEST_ROOT/log2.json") > 0 )) +(( $(jq -s --arg sid foo '[.[] | select(.sid != $sid)] | length' < "$TEST_ROOT/log2.json") == 0 )) diff --git a/tests/functional/make.nix b/tests/functional/make.nix new file mode 100644 index 000000000000..32673452dde8 --- /dev/null +++ b/tests/functional/make.nix @@ -0,0 +1,52 @@ +# Run the functional tests against the nix-make-built `nix`, using +# the test runner (no Meson). The output holds the test logs. +{ + lib, + stdenv, + test-runner, + nix, + + # The test scripts to run, relative to this directory; empty means all + # tests found by the runner. (Not named `tests`, since `callPackage` would + # then pass nixpkgs' `tests` attribute.) + testScripts ? [ ], +}: + +derivation { + name = "${nix.name}-functional-tests"; + system = stdenv.hostPlatform.system; + builder = stdenv.shell; + args = [ + "-e" + (builtins.toFile "builder.sh" '' + if [ -e "$NIX_ATTRS_SH_FILE" ]; then . "$NIX_ATTRS_SH_FILE"; fi + source $stdenv/setup >/dev/null + # The tests write next to their sources (e.g. `result` symlinks and + # the expected/actual output of lang tests), so use a writable copy. + cp -r "$src" "$TMPDIR/src" + chmod -R u+w "$TMPDIR/src" + run-tests \ + --nix-bin-dir "$nix/bin" \ + --source-dir "$TMPDIR/src/tests/functional" \ + --build-dir "$TMPDIR/build" \ + "''${testScripts[@]}" + mkdir -p "$out" + cp -r "$TMPDIR/build/logs" "$out/" + '') + ]; + __structuredAttrs = true; + inherit stdenv nix testScripts; + # Rooted at the repository so that the `.version` symlink resolves and + # the files outside this directory that tests use can be included. + src = lib.fileset.toSource { + root = ../..; + fileset = lib.fileset.unions [ + ./. + ../../.version + ../../scripts/nix-profile.sh.in + ]; + }; + nativeBuildInputs = [ test-runner ]; + # Required for HTTP `nix serve`-based tests in binary-cache.sh + __darwinAllowLocalNetworking = true; +} diff --git a/tests/functional/meson.build b/tests/functional/meson.build index 8fdaf4ffe018..1e43c200068d 100644 --- a/tests/functional/meson.build +++ b/tests/functional/meson.build @@ -144,12 +144,15 @@ suites = [ 'nix_path.sh', 'no-url-literals.sh', 'optimise-store.sh', + 'otel.sh', 'output-normalization.sh', + 'parallel-eval.sh', 'pass-as-file.sh', 'path-from-hash-part.sh', 'path-info.sh', 'placeholders.sh', 'post-hook.sh', + 'ps.sh', 'pure-eval.sh', 'read-only-store.sh', 'readfile-context.sh', @@ -162,6 +165,7 @@ suites = [ 'search.sh', 'secure-drv-outputs.sh', 'selfref-gc.sh', + 'sentry.sh', 'shell.sh', 'short-path-literals.sh', 'signing.sh', @@ -180,6 +184,7 @@ suites = [ 'toString-path.sh', 'user-envs-migration.sh', 'user-envs.sh', + 'wasm.sh', 'why-depends.sh', ], 'workdir' : meson.current_source_dir(), diff --git a/tests/functional/misc.sh b/tests/functional/misc.sh index 131b63323e57..b8bbb74dddd7 100755 --- a/tests/functional/misc.sh +++ b/tests/functional/misc.sh @@ -23,11 +23,11 @@ expect 1 nix-env -q --foo 2>&1 | grep "unknown flag" # Eval Errors. eval_arg_res=$(nix-instantiate --eval -E 'let a = {} // a; in a.foo' 2>&1 || true) -echo "$eval_arg_res" | grep "at «string»:1:15:" +echo "$eval_arg_res" | grep "at «string»:1:12:" echo "$eval_arg_res" | grep "infinite recursion encountered" eval_stdin_res=$(echo 'let a = {} // a; in a.foo' | nix-instantiate --eval -E - 2>&1 || true) -echo "$eval_stdin_res" | grep "at «stdin»:1:15:" +echo "$eval_stdin_res" | grep "at «stdin»:1:12:" echo "$eval_stdin_res" | grep "infinite recursion encountered" # Attribute path errors diff --git a/tests/functional/nix-profile.sh b/tests/functional/nix-profile.sh index 556754a5338f..5444d84b7f21 100755 --- a/tests/functional/nix-profile.sh +++ b/tests/functional/nix-profile.sh @@ -4,12 +4,14 @@ source common.sh TODO_NixOS +requireGit + enableFeatures "ca-derivations" restartDaemon # Make a flake. flake1Dir=$TEST_ROOT/flake1 -mkdir -p "$flake1Dir" +createGitRepo "$flake1Dir" # shellcheck disable=SC2154,SC1039 cat > "$flake1Dir"/flake.nix < "$flake1Dir"/ca.nix cp "${config_nix}" "$flake1Dir"/ +git -C "$flake1Dir" add flake.nix config.nix who version ca.nix +git -C "$flake1Dir" commit -m 'Initial' + # Test upgrading from nix-env. nix-env -f ./user-envs.nix -i foo-1.0 nix profile list | grep -A2 'Name:.*foo' | grep 'Store paths:.*foo-1.0' nix profile add "$flake1Dir" -L -nix profile list | grep -A4 'Name:.*flake1' | grep 'Locked flake URL:.*narHash' +#nix profile list | grep -A4 'Name:.*flake1' | grep 'Locked flake URL:.*narHash' [[ $("$TEST_HOME"/.nix-profile/bin/hello) = "Hello World" ]] [ -e "$TEST_HOME"/.nix-profile/share/man ] # shellcheck disable=SC2235 (! [ -e "$TEST_HOME"/.nix-profile/include ]) nix profile history -nix profile history | grep "packages.$system.default: ∅ -> 1.0" -nix profile diff-closures | grep 'env-manifest.nix: ε → ∅' +nix profile history | grep "packages.$system.default: 1.0, 1.0-man added" +nix profile diff-closures | grep 'env-manifest.nix: (no version) removed' # Test XDG Base Directories support export NIX_CONFIG="use-xdg-base-directories = true" @@ -89,6 +94,24 @@ echo "$completion_output" | grep -q "^normal$" echo "$completion_output" | grep -q "^flake1" echo "$completion_output" | grep -q "^foo" +# Test upgrading a package to a different store path with the same +# version. With --show-source, the locked flake reference change +# (from a clean to a dirty tree) should be shown. +printf Nix > "$flake1Dir"/who +nix profile upgrade flake1 +[[ $("$TEST_HOME"/.nix-profile/bin/hello) = "Hello Nix" ]] +nix profile history | grep "packages.$system.default: 1.0, 1.0-man changed$" +nix profile history --show-source | grep "packages.$system.default: 1.0, 1.0-man changed (git+file://.*rev=[0-9a-f]* -> git+file://.*flake1)" +nix profile history --show-source | grep "packages.$system.default: 1.0, 1.0-man added (git+file://.*rev=[0-9a-f]*)" + +# Same, but now the locked flake reference doesn't change (the tree is +# still dirty), so the same reference should be shown twice. +printf Nixers > "$flake1Dir"/who +nix profile upgrade flake1 +[[ $("$TEST_HOME"/.nix-profile/bin/hello) = "Hello Nixers" ]] +nix profile history | grep "packages.$system.default: 1.0, 1.0-man changed$" +nix profile history --show-source | grep "packages.$system.default: 1.0, 1.0-man changed (git+file://[^ ]*flake1 -> git+file://[^ ]*flake1)$" + # Test upgrading a package. printf NixOS > "$flake1Dir"/who printf 2.0 > "$flake1Dir"/version @@ -112,6 +135,7 @@ printf 1.0 > "$flake1Dir"/version # Test --all exclusivity. assertStderr nix --offline profile upgrade --all foo << EOF error: --all cannot be used with package names or regular expressions. + Try 'nix --help' for more information. EOF @@ -146,9 +170,8 @@ nix profile rollback [ -e "$TEST_HOME"/.nix-profile/bin/foo ] # shellcheck disable=SC2235 nix profile remove foo 2>&1 | grep 'removed 1 packages' -# shellcheck disable=SC2235 -(! [ -e "$TEST_HOME"/.nix-profile/bin/foo ]) -nix profile history | grep 'foo: 1.0 -> ∅' +[[ ! -e "$TEST_HOME"/.nix-profile/bin/foo ]] +nix profile history | grep 'foo: 1.0 removed' nix profile diff-closures | grep 'Version 3 -> 4' # Test installing a non-flake package. @@ -241,11 +264,11 @@ error: An existing package already provides the following file: The conflicting packages have a priority of 5. To prioritise the new package: - nix profile add path:${flake2Dir}#packages.${system}.default --priority 4 + nix profile add git+file://${flake2Dir}#packages.${system}.default --priority 4 To prioritise the existing package: - nix profile add path:${flake2Dir}#packages.${system}.default --priority 6 + nix profile add git+file://${flake2Dir}#packages.${system}.default --priority 6 EOF ) [[ $("$TEST_HOME"/.nix-profile/bin/hello) = "Hello World" ]] diff --git a/tests/functional/oob.wat b/tests/functional/oob.wat new file mode 100644 index 000000000000..dd5da01d7eec --- /dev/null +++ b/tests/functional/oob.wat @@ -0,0 +1,12 @@ +(module + ;; A module that passes an out-of-range pointer to a host function. The + ;; host must reject it rather than access memory outside the Wasm memory. + (import "env" "make_string" (func $make_string (param i32 i32) (result i32))) + + (memory (export "memory") 1) + + (func (export "nix_wasm_init_v1")) + + ;; The memory is one page (64 KiB); this offset is far beyond it. + (func (export "oob") (param i32) (result i32) + (call $make_string (i32.const 0xFFFFFF00) (i32.const 16)))) diff --git a/tests/functional/otel-sink.py b/tests/functional/otel-sink.py new file mode 100644 index 000000000000..b6c4603399bd --- /dev/null +++ b/tests/functional/otel-sink.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""A minimal OTLP/HTTP collector for testing: saves every POST body it +receives, unmodified, to a numbered file in the given directory, along +with the request headers. Listens on an ephemeral port on 127.0.0.1 +and writes the port number to `/port` once it's ready.""" + +import http.server +import json +import os +import sys + +out_dir = sys.argv[1] +count = 0 + + +class Handler(http.server.BaseHTTPRequestHandler): + def do_POST(self): + global count + body = self.rfile.read(int(self.headers.get("content-length", 0))) + with open(os.path.join(out_dir, f"{count}.body"), "wb") as f: + f.write(body) + with open(os.path.join(out_dir, f"{count}.headers"), "w") as f: + json.dump({k.lower(): v for k, v in self.headers.items()}, f) + count += 1 + self.send_response(200) + self.end_headers() + + def log_message(self, *args): + pass + + +server = http.server.HTTPServer(("127.0.0.1", 0), Handler) + +# Write the port atomically, so the test never sees a partial file. +port_file = os.path.join(out_dir, "port") +with open(port_file + ".tmp", "w") as f: + f.write(str(server.server_address[1])) +os.rename(port_file + ".tmp", port_file) + +server.serve_forever() diff --git a/tests/functional/otel.sh b/tests/functional/otel.sh new file mode 100644 index 000000000000..bbaa9eb2e4bf --- /dev/null +++ b/tests/functional/otel.sh @@ -0,0 +1,155 @@ +#!/usr/bin/env bash + +# Test OpenTelemetry trace export: run a few commands against a local +# collector that just saves whatever it receives, then check the +# resulting OTLP/JSON. + +source common.sh + +# Only meaningful if Nix was built with OpenTelemetry support. +nix config show | grepQuiet '^otlp ' || skipTest "Nix was built without OpenTelemetry support" + +[[ $(type -p python3) ]] || skipTest "Python is not installed" + +sinkDir=$TEST_ROOT/otel +mkdir -p "$sinkDir" + +python3 ./otel-sink.py "$sinkDir" & +sinkPid=$! +trap 'kill "$sinkPid"' EXIT + +for ((i = 0; i < 100; i++)); do + [[ -e $sinkDir/port ]] && break + kill -0 "$sinkPid" || fail "the collector died unexpectedly" + sleep 0.1 +done +[[ -e $sinkDir/port ]] || fail "the collector didn't start" + +cat >> "$test_nix_conf" < /dev/null +for ((i = 0; i < 100; i++)); do + [[ $(uploads) -ge 2 ]] && break + sleep 0.1 +done +[[ $(services) = $'nix\nnix-daemon' ]] +[[ $(span "nix store info" .kind) = 1 ]] +[[ $(span "daemon connection" .kind) = 2 ]] # SERVER +[[ $(span "daemon connection" .traceId) = $(span "nix store info" .traceId) ]] +[[ $(span "daemon connection" .parentSpanId) = $(span "nix store info" .spanId) ]] + +# The root span is parented to the trace context in `TRACEPARENT`, +# which is how remote builds via `build-remote` are linked to the +# trace of the `nix` process that runs it. +clearUploads +[[ $(TRACEPARENT=00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01 nix eval --expr '1 + 2') = 3 ]] +[[ $(span "nix eval" .traceId) = 0af7651916cd43dd8448eb211c80319c ]] +[[ $(span "nix eval" .parentSpanId) = b7ad6b7169203331 ]] +[[ $(span "nix eval" .kind) = 1 ]] # INTERNAL, not SERVER diff --git a/tests/functional/package.nix b/tests/functional/package.nix index cf5eea94f432..5204a52c0ff1 100644 --- a/tests/functional/package.nix +++ b/tests/functional/package.nix @@ -10,6 +10,7 @@ jq, git, mercurial, + python3, unixtools, util-linux, @@ -26,6 +27,9 @@ # For running the functional tests against a different pre-built Nix. test-daemon ? null, + + # Whether to run tests with lazy trees enabled. + lazyTrees ? false, }: let @@ -54,6 +58,8 @@ mkMesonDerivation ( jq git mercurial + # For the OpenTelemetry collector in `otel.sh`. + python3 unixtools.script # Explicitly splice the hostHost variant to fix LLVM tests. The nix-cli @@ -99,6 +105,11 @@ mkMesonDerivation ( mkdir $out ''; + _NIX_TEST_EXTRA_CONFIG = lib.optionalString lazyTrees "lazy-trees = true"; + + # Required for HTTP `nix serve`-based tests in binary-cache.sh + __darwinAllowLocalNetworking = true; + meta = { platforms = lib.platforms.unix; }; diff --git a/tests/functional/parallel-eval.sh b/tests/functional/parallel-eval.sh new file mode 100755 index 000000000000..f9c280de2e90 --- /dev/null +++ b/tests/functional/parallel-eval.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash + +# Tests for the multi-threaded evaluator (`eval-cores`). + +source common.sh + +# `builtins.addErrorContext` evaluates its message after the body has +# thrown. If the message depends on a thunk that another fiber is +# evaluating, the current fiber has to wait for it. That must not +# happen inside the exception handler (which the fiber cannot suspend +# in), so the message is evaluated after leaving the handler. +cat > "$TEST_ROOT/add-error-context.nix" <<'NIX' +let + big = builtins.foldl' (a: b: a + b) 0 (builtins.genList (x: x) 3000000); +in +{ + a = big; + b = builtins.addErrorContext "context mentioning ${toString big}" (throw "boom"); +} +NIX + +expectStderr 1 nix eval --eval-cores 4 --json --file "$TEST_ROOT/add-error-context.nix" | grepQuiet "context mentioning 4499998500000" diff --git a/tests/functional/path-info.sh b/tests/functional/path-info.sh index 712b5267878c..3d6041914ed6 100755 --- a/tests/functional/path-info.sh +++ b/tests/functional/path-info.sh @@ -13,7 +13,7 @@ barBase=$(basename "$bar") echo baz > "$TEST_ROOT"/baz baz=$(nix store add-file "$TEST_ROOT"/baz) bazBase=$(basename "$baz") -nix-store --delete "$baz" +nix-store --delete --ignore-liveness "$baz" diff --unified --color=always \ <(nix path-info --json --json-format 2 "$foo" "$bar" "$baz" | diff --git a/tests/functional/ps.sh b/tests/functional/ps.sh new file mode 100755 index 000000000000..7ae89956abb5 --- /dev/null +++ b/tests/functional/ps.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash + +source common.sh + +TODO_NixOS + +needLocalStore "the active-builds directory is per-store" + +clearStore + +# Two FIFOs: the builder writes to "started" once it is running, and +# reads from "finish" to know when to exit. +started=$TEST_ROOT/nix-ps-started.fifo +finish=$TEST_ROOT/nix-ps-finish.fifo +mkfifo "$started" +mkfifo "$finish" + +# Start a build in the background that blocks until we tell it to finish. +nix-build --max-silent-time 60 -o "$TEST_ROOT/result" -E " + with import ${config_nix}; + mkDerivation { + name = \"nix-ps-test\"; + buildCommand = '' + sleep 600 > /dev/null 2>&1 & + sleep 1 + echo started > $started + read line < $finish + mkdir \$out + ''; + }" >"$TEST_ROOT/nix-ps-build.out" 2>&1 & +buildPid=$! + +# Wait for the build to signal that it has started. +read -r _ < "$started" + +# The build is now blocked on $finish. Check that `nix ps` sees it and +# the backgrounded `sleep 600` child process. +out=$(nix ps) +echo "$out" +echo "$out" | grepQuiet 'nix-ps-test.drv' +echo "$out" | grepQuiet 'sleep 600' + +# Check the JSON output as well. +json=$(nix ps --json) +echo "$json" | jq -e 'length == 1' >/dev/null +echo "$json" | jq -e '.[0].derivation | test("nix-ps-test\\.drv$")' >/dev/null +echo "$json" | jq -e '.[0].mainPid | type == "number"' >/dev/null +echo "$json" | jq -e '[.[0].processes[].argv | join(" ")] | any(. | test("bash"))' >/dev/null +echo "$json" | jq -e '[.[0].processes[].argv | join(" ")] | any(. | test("sleep 600"))' >/dev/null +echo "$json" | jq -e '.[0].processes | length == 2' >/dev/null + +# Release the build and wait for it to finish successfully. +echo "done" > "$finish" +wait "$buildPid" + +# After the build finishes, `nix ps` should be empty again. +nix ps --json | jq -e 'length == 0' >/dev/null diff --git a/tests/functional/recursive.nix b/tests/functional/recursive.nix index be9e55da37ec..aa2aa26c5494 100644 --- a/tests/functional/recursive.nix +++ b/tests/functional/recursive.nix @@ -17,7 +17,7 @@ mkDerivation rec { buildCommand = '' mkdir $out - opts="--experimental-features nix-command ${ + opts="${ if (NIX_TESTS_CA_BY_DEFAULT == "1") then "--extra-experimental-features ca-derivations" else "" }" diff --git a/tests/functional/recursive.sh b/tests/functional/recursive.sh index 605c6c5bb0eb..a1fd27178e7d 100755 --- a/tests/functional/recursive.sh +++ b/tests/functional/recursive.sh @@ -10,7 +10,7 @@ restartDaemon unreachable=$(nix store add-path ./recursive.sh) export unreachable -NIX_BIN_DIR=$(dirname "$(type -p nix)") nix --extra-experimental-features 'nix-command recursive-nix' build -o "$TEST_ROOT"/result -L --impure --file ./recursive.nix +NIX_BIN_DIR=$(dirname "$(type -p nix)") nix --extra-experimental-features 'recursive-nix' build -o "$TEST_ROOT"/result -L --impure --file ./recursive.nix [[ $(cat "$TEST_ROOT"/result/inner1) =~ blaat ]] diff --git a/tests/functional/repl.sh b/tests/functional/repl.sh index f3e9975a802d..e5d0ad6088e6 100755 --- a/tests/functional/repl.sh +++ b/tests/functional/repl.sh @@ -143,7 +143,7 @@ EOF testReplResponse ' foo + baz ' "3" \ - ./flake ./flake\#bar --experimental-features 'flakes' + ./flake ./flake\#bar # Test the `:reload` mechanism with flakes: # - Eval `./flake#changingThing` @@ -152,7 +152,7 @@ foo + baz # - Check that the result has changed mkfifo repl_fifo touch repl_output -nix repl ./flake --experimental-features 'flakes' < repl_fifo >> repl_output 2>&1 & +nix repl ./flake < repl_fifo >> repl_output 2>&1 & repl_pid=$! exec 3>repl_fifo # Open fifo for writing echo "changingThing" >&3 @@ -233,7 +233,7 @@ EDITOR='cat' nix repl <<< ':e ' 2>&1 | grepQuiet 'builtin:fetc badDiff=0 badExitCode=0 -nixVersion="$(nix eval --impure --raw --expr 'builtins.nixVersion' --extra-experimental-features nix-command)" +nixVersion="$(nix --version | sed 's/nix //')" # TODO: write a repl interacter for testing. Papering over the differences between readline / editline and between platforms is a pain. diff --git a/tests/functional/repl/debugger-okay-ignore-try.expected b/tests/functional/repl/debugger-okay-ignore-try.expected index 7d4e72fdaeab..e89c31159b67 100644 --- a/tests/functional/repl/debugger-okay-ignore-try.expected +++ b/tests/functional/repl/debugger-okay-ignore-try.expected @@ -58,9 +58,6 @@ abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit nix-repl> :p bricked error: infinite recursion encountered - at «string»:1:1: - 1| bricked - | ^ nix-repl> :p tried { @@ -89,9 +86,6 @@ abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit nix-repl> :p bricked error: infinite recursion encountered - at «string»:1:1: - 1| bricked - | ^ nix-repl> :p tried { @@ -114,9 +108,6 @@ abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit nix-repl> :p bricked error: infinite recursion encountered - at «string»:1:1: - 1| bricked - | ^ nix-repl> :p tried { diff --git a/tests/functional/run-tests.py b/tests/functional/run-tests.py new file mode 100644 index 000000000000..515dd6d4f64f --- /dev/null +++ b/tests/functional/run-tests.py @@ -0,0 +1,294 @@ +#!/usr/bin/env python3 +"""Run the functional tests without Meson. + +This provides what `meson test` provides to the test scripts: the generated +`common/subst-vars.sh` and `config.nix`, the environment variables the +harness in `common/` expects, and a per-test `TEST_ROOT` (derived by the +harness from `TEST_SUITE_NAME` and `TEST_NAME`) so that tests can run in +parallel. Tests are not modified; each is run as `bash -x -e -u -o pipefail +