Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,10 @@ tools/ @DataDog/apm-common-components-core
windows/ @DataDog/libdatadog-core
fuzz/ @DataDog/chaos-platform

# Nix
*.nix @DataDog/nix-guild @DataDog/apm-common-components-core

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL there's a Nix guild 😄

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a small but growing group. Join us at #nix!

flake.* @DataDog/nix-guild @DataDog/apm-common-components-core

# Specific overrides (must come after their general patterns above)
bin_tests/tests/test_the_tests.rs @DataDog/libdatadog-core
bin_tests/src/bin/test_the_tests.rs @DataDog/libdatadog-core
Expand All @@ -105,3 +109,4 @@ libdd-data-pipeline/tests/test_trace_exporter_otlp_export.rs @DataDog/apm-sdk-ca
libdd-trace-utils/src/otlp_encoder/ @DataDog/apm-sdk-capabilities-rust
datadog-sidecar/src/service/ffe_exposures_flusher.rs @DataDog/libdatadog-php @DataDog/libdatadog-apm @DataDog/feature-flagging-and-experimentation-sdk
datadog-sidecar/src/service/ffe_metrics_flusher.rs @DataDog/libdatadog-php @DataDog/libdatadog-apm @DataDog/feature-flagging-and-experimentation-sdk
.github/workflows/nix.yml @DataDog/nix-guild @DataDog/apm-common-components-core
86 changes: 86 additions & 0 deletions .github/workflows/nix.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
name: Test Nix

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add documentation (maybe here and / or inline in this file) with instructions on what to do if there is a failure? I suspect it'll only come up on MSRV bumps.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too sure what the expectation of the doc would be: some basic usage examples to onboard the Nix-unaware / a LLM agent maybe?

I did not want to be too invasive as a first PR but seems like you folks are quite on board with this so I feel like the doc you pointed to would be a good place. Will add.


on: # yamllint disable-line rule:truthy
push:
branches:
- main
- mq-working-branch-*
Comment on lines +4 to +7

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
push:
branches:
- main
- mq-working-branch-*
# Nightly safeguard: catches devshell drift from changes that don't match the
# paths above (e.g. a dependency or build-script change that needs new native
# tooling). Scheduled runs only fire from the default branch, against its HEAD.
schedule:
- cron: "0 4 * * *" # daily at 04:00 UTC

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of triggering it in every PR I think it would be better to have a nightly schedule. That way we can save some capacity for the actual pipelines.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But wouldn't you want to check tings on PRs that change Nix files at least?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely.

As is it's not going to trigger in every PR anyway, only those that change nix files or that have an impact on nix (the toolchain files).

The schedule can be nice though as a sanity check for when those don't change often.

# Also run on PRs that touch the devshell or what it reads. Paths mirror the
# Nix CODEOWNERS entries, plus the toolchain files (read by the flake) and
# this workflow itself.
pull_request:
paths:
- "*.nix"
- "flake.*"
- "rust-toolchain.toml"
- "nightly-toolchain.toml"
- ".github/workflows/nix.yml"

# Default permissions for all jobs
permissions: {}

concurrency:
group: ci-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}-nix
cancel-in-progress: true

env:
CARGO_TERM_COLOR: always

jobs:
test:
strategy:
fail-fast: false
matrix:
platform:
- os: darwin
cpu: arm64
base: macos-15 # always arm64-darwin

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is macos strictly needed? I ask because macos runners are scarce and ocuppying one for this might hurt us a bit in terms of merge times

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it guarantees that it consistently works across OSes for any dev that uses Nix on a macOS machine

Should be mostly mitigated by conditional running mentioned above.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could also use a concurrency group (example from our test workflow)and cancel in progress if you don't need commit level granularity.

Not necessarily a hill I want to die on, but I think using a concurrency group is a good compromise between getting signal that something broke and not potentially backing up the MergeQueue waiting for a macOS runner for a workflow that doesn't indicate an actual code issue.

- os: linux
cpu: x86_64
base: ubuntu-24.04 # always x86_64-linux-gnu
- os: linux
cpu: aarch64
base: ubuntu-24.04-arm # always aarch64-linux-gnu
Comment thread
lloeki marked this conversation as resolved.

name: Test Nix (${{ matrix.platform.cpu }}-${{ matrix.platform.os }})
runs-on: ${{ matrix.platform.base }}

permissions:
contents: read

steps:
- name: Check CPU arch
run: |
test "$(uname -m)" = "${{ matrix.platform.cpu }}"
- name: Free Disk Space (Linux only)
if: runner.os == 'Linux'
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # 1.3.1
with:
tool-cache: true
android: true
dotnet: true
haskell: true
large-packages: false
docker-images: false
swap-storage: false
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: cachix/install-nix-action@ab739621df7a23f52766f9ccc97f38da6b7af14f # v31.10.5
- name: Print toolchain versions
run: |
nix develop --command rustc --version
nix develop --command cargo --version
nix develop --command cbindgen --version
- name: Check nightly formatter toolchain
run: nix develop .#nightly --command cargo fmt --version
- name: Build workspace
run: nix develop --command cargo build --workspace --exclude builder

complete:
name: Nix (complete)
runs-on: ubuntu-24.04
needs:
- test
steps:
- run: echo "DONE!"
46 changes: 46 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,52 @@ We provide two Dev Container configurations:

The container includes all necessary dependencies for building and testing `libdatadog`.

#### Nix development shell

The Nix flake provides a reproducible, pinned shell with Rust, `cbindgen`, and native build tools.

This works natively under both Linux and Darwin.

##### Prerequisite: install Nix

This one-liner installs Nix isolated in `/nix`:

```bash
curl --proto '=https' --tlsv1.2 --location https://nixos.org/nix/install | sh -s -- --daemon
```

Enable the modern CLI and flakes in `/etc/nix/nix.conf`:

```bash
echo "experimental-features = nix-command flakes" | sudo tee -a /etc/nix/nix.conf
```

See the [Nix manual](https://nix.dev/manual/nix/2.28/installation/index.html) for more information.

##### Spawn an interactive shell

Spawn a shell with an environment set up to expose the tooling:

```bash
nix develop
```

Note: legacy `nix-shell` and `nix-build` are also available via the `flake-compat` shims.

##### Run commands

Alternatively, run individual commands:

```bash
nix develop --command cargo build --workspace --exclude builder
nix develop .#nightly --command cargo fmt --all -- --check
```

##### Debugging CI failures

- Reproduce the Nix CI build with `nix develop --command cargo build --workspace --exclude builder`.
- After an MSRV or nightly bump, update `rust-toolchain.toml` or `nightly-toolchain.toml`, then refresh `flake.lock` with `nix flake update`.

#### Docker container
A dockerfile is provided to run tests in a Ubuntu linux environment. This is particularly useful for running and debugging linux-only tests on macOS.

Expand Down
14 changes: 14 additions & 0 deletions default.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Copyright 2026-Present Datadog, Inc. https://www.datadoghq.com/
# SPDX-License-Identifier: Apache-2.0

# flake-compat shim for usage without flakes
(import
(
let lock = builtins.fromJSON (builtins.readFile ./flake.lock); in
fetchTarball {
url = lock.nodes.flake-compat.locked.url or "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz";
sha256 = lock.nodes.flake-compat.locked.narHash;
}
)
{ src = ./.; }
).defaultNix
97 changes: 97 additions & 0 deletions flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

62 changes: 62 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Copyright 2026-Present Datadog, Inc. https://www.datadoghq.com/
# SPDX-License-Identifier: Apache-2.0

{
Comment thread
lloeki marked this conversation as resolved.
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/release-25.11";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the 26.05 channel is the latest stable, why not use it instead ?
And what about using unstable instead ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unstable is nice to have the very latest version, but I think you won't hit the binary cache, which might mean a lot of full local rebuilds from source. The latest stable makes sense though 👍

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does unstable really makes a difference with binary cache ? I never hit that issue personally.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unstable is actually cached, but it moves fast. I vaguely remember situations such as user A updates the flake.lock, 1 month passes, new user B checks the project out from scratch and inherits the same flake.lock, but stuff isn't in the unstable cache anymore. If you are the one updating you shouldn't have issues ofc, because when you check out the package, they are the most recent available and thus in cache. I think. I am not sure. Both latest stable and unstable are probably reasonable.

Another reason could be if you run NixOS, you usually target a specific release, and you probably don't want to duplicate many versions of core packages (the stable and the unstable one).


# cross-platform convenience
flake-utils.url = "github:numtide/flake-utils";

# backwards compatibility with nix-build and nix-shell
flake-compat.url = "https://flakehub.com/f/edolstra/flake-compat/1.tar.gz";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Is this necessary? shell.nix or default.nix aren't included in this repo (maybe they will be in the future?).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ha, I forgot to add the shims, will add.


# pinned, exact upstream Rust toolchains
rust-overlay = {
url = "github:oxalica/rust-overlay";
inputs.nixpkgs.follows = "nixpkgs";
};
};

outputs = { self, nixpkgs, flake-utils, flake-compat, rust-overlay }:
# resolve for all platforms in turn
flake-utils.lib.eachDefaultSystem (system:
let
# packages for this system platform, with the rust-overlay applied
pkgs = import nixpkgs {
inherit system;
overlays = [ (import rust-overlay) ];
};

# A devshell for a given Rust toolchain (read from a toolchain file via
# rust-overlay), with the rest of the build dependencies.
mkDevShell = rust: pkgs.stdenv.mkDerivation {
name = "libdatadog-devshell";

# The stdenv cc-wrapper injects -D_FORTIFY_SOURCE, which glibc rejects
# when compiling without optimization. Some build scripts (e.g.
# spawn_worker's trampoline.c) compile C at -O0 with -Werror, so the
# resulting fortify #warning becomes a hard error. Disable fortify
# hardening in the shell so those builds succeed.
hardeningDisable = [ "fortify" "fortify3" ];

nativeBuildInputs = [
rust # rustc + cargo + rustfmt + clippy, pinned via toolchain file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One slightly annoying thing here is that we need to cargo fmt with nightly. I wonder if we should provide a nightly toolchain pinned as well, or even maybe propose a "libdd-fmt" alias that uses the right nightly toolchain under the hood. It's a detail though, and could be worked out in a follow up PR.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coincidentally I just added a nightly toolchain as .#nightly.

Maybe we can even work out something that would have a closure over the nightly toolchain as an internal dependency for a nightly-based cargo fmt - and only that - to be exposed. That's exactly the kind of thing Nix can solve.

Indeed probably for a later PR.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify usage as of today:

  • nix develop .#nightly to enter a shell with the nightly toolchain
  • nix develop .#nightly -c cargo fmt to run its nightly cargo right away

pkgs.rust-cbindgen
Comment thread
lloeki marked this conversation as resolved.
pkgs.cmake
pkgs.autoconf
pkgs.automake
pkgs.libtool
];
};
in {
# Default: the pinned stable toolchain (single source of truth is
# ./rust-toolchain.toml), matching CI and rustup.
devShells.default = mkDevShell (pkgs.rust-bin.fromRustupToolchainFile ./rust-toolchain.toml);

# Nightly toolchain (./nightly-toolchain.toml) for the jobs that
# genuinely need a nightly compiler. Use with `nix develop .#nightly`.
devShells.nightly = mkDevShell (pkgs.rust-bin.fromRustupToolchainFile ./nightly-toolchain.toml);
}
);
}
14 changes: 14 additions & 0 deletions shell.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Copyright 2026-Present Datadog, Inc. https://www.datadoghq.com/
# SPDX-License-Identifier: Apache-2.0

# flake-compat shim for usage without flakes
(import
(
let lock = builtins.fromJSON (builtins.readFile ./flake.lock); in
fetchTarball {
url = lock.nodes.flake-compat.locked.url or "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz";
sha256 = lock.nodes.flake-compat.locked.narHash;
}
)
{ src = ./.; }
).shellNix
Loading