feat: fast_llvm_repo

README.md

@@ -38,6 +38,7 @@ validates the setup, and how QNX-specific authentication and licensing fit in.
 - [QNX integration](docs/qnx_integration.md)
 - [Test Suite](docs/test_suite.md)
 - [Maintenance](docs/maintenance.md)
+- [Fast LLVM repository](docs/fast_llvm_repo.md)
 
 ## Quick Summary
 
@@ -54,4 +55,5 @@ validates the setup, and how QNX-specific authentication and licensing fit in.
 - Define Linux and QNX toolchains through a Bzlmod extension.
 - Resolve default package metadata through `packages/version_matrix.bzl`.
 - Generate toolchain repositories from platform-specific templates.
+- Optionally materialize prebuilt LLVM distributions with fast extraction.
 - Validate toolchain selections through the workspace test matrix.

docs/fast_llvm_repo.md

@@ -0,0 +1,111 @@
+<!--
+# *******************************************************************************
+# Copyright (c) 2026 Contributors to the Eclipse Foundation
+#
+# See the NOTICE file(s) distributed with this work for additional
+# information regarding copyright ownership.
+#
+# This program and the accompanying materials are made available under the
+# terms of the Apache License Version 2.0 which is available at
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# SPDX-License-Identifier: Apache-2.0
+# *******************************************************************************
+-->
+
+# Fast LLVM repository
+
+## Purpose
+
+`fast_llvm_repo` creates the prebuilt LLVM distribution repository consumed by
+[`toolchains_llvm`](https://github.com/bazel-contrib/toolchains_llvm). It is a
+performance-oriented alternative to the repository that `toolchains_llvm`
+normally generates internally.
+
+The rule is useful when LLVM setup is a significant part of the developer or
+CI feedback time. In the measurements that motivated it, LLVM setup decreased
+from roughly 220 seconds to 70 seconds. The exact result depends on the host;
+the rule is not required for correctness.
+
+## Usage with Bzlmod
+
+The consuming module needs both `toolchains_llvm` and this module. The LLVM
+version must currently be kept in sync in the two tags below:
+
+```starlark
+bazel_dep(name = "score_bazel_cpp_toolchains", version = "0.5.4", dev_dependency = True)
+bazel_dep(name = "toolchains_llvm", version = "1.8.0", dev_dependency = True)
+
+llvm = use_extension(
+    "@toolchains_llvm//toolchain/extensions:llvm.bzl",
+    "llvm",
+    dev_dependency = True,
+)
+llvm.toolchain(
+    name = "llvm_toolchain",
+    llvm_version = "22.1.7",
+)
+
+fast_llvm_repo = use_repo_rule(
+    "@score_bazel_cpp_toolchains//rules:fast_llvm_repo.bzl",
+    "fast_llvm_repo",
+)
+fast_llvm_repo(
+    name = "llvm_toolchain_llvm",
+    llvm_version = "22.1.7",
+)
+
+# Keep the generated toolchain configuration, but replace its LLVM root.
+llvm.toolchain_root(label = "@llvm_toolchain_llvm//:BUILD")
+
+use_repo(llvm, "llvm_toolchain")
+```
+
+The repository name `llvm_toolchain_llvm` is intentional: it is the name that
+`toolchains_llvm` normally derives for the LLVM distribution. The generated
+`BUILD.bazel` is based on `toolchains_llvm`'s `BUILD.llvm_repo.tpl`, so the
+existing toolchain configuration can continue to refer to the same layout.
+
+## What the rule does
+
+For the requested LLVM version, the rule:
+
+1. rejects non-Linux hosts and selects the archive for the host architecture,
+2. downloads it with a pinned SHA-256 checksum,
+3. extracts it with parallel `xz -T0` output streamed to `tar`, using Bash's
+   `pipefail` to check both process exit codes,
+4. renders the BUILD file expected by `toolchains_llvm`.
+
+The parallel XZ pipeline is the main optimization. It changes the extraction
+implementation while leaving the LLVM archive, checksum verification, and
+toolchain target layout intact.
+
+## Why not `http_archive`?
+
+`http_archive` can functionally download and unpack a `tar.xz`. However,
+`toolchains_llvm` already uses Bazel's standard repository download-and-extract
+mechanism for the LLVM repository it creates internally. That
+`download_and_extract()` path is the slow part this rule is intended to
+replace.
+
+`http_archive` does not provide a switch to use the parallel `xz -T0` pipeline,
+and `patch_cmds` run only after extraction. Using `http_archive` would therefore
+not provide the performance improvement.
+
+If the additional extraction time is acceptable, the original
+`toolchains_llvm` setup remains the simpler option. This rule exists for the
+case where the measured setup time justifies the additional repository-rule
+maintenance.
+
+## Scope and trade-offs
+
+This is intentionally a small, specialized rule rather than a replacement for
+`http_archive` or `toolchains_llvm`:
+
+- only the versions and Linux architectures listed in
+  [`fast_llvm_repo.bzl`](../rules/fast_llvm_repo.bzl) are supported;
+- the fast path requires `bash`, `tar`, and `xz` on the host;
+- the version/checksum table must be kept aligned with `toolchains_llvm`;
+- the rule uses `toolchains_llvm`'s internal BUILD template, so upgrades of
+  that dependency need to be checked; and
+- the `llvm_version` values in the two module tags must not diverge.

docs/repository_layout.md

@@ -40,8 +40,10 @@ into repository rule invocations.
 
 `rules/`
 
-Contains the repository rules that materialize a toolchain repository.
-`rules/gcc.bzl` renders BUILD and configuration files from templates.
+Contains the repository rules that materialize toolchain repositories.
+`rules/gcc.bzl` renders BUILD and configuration files from templates, while
+`rules/fast_llvm_repo.bzl` provides the optional optimized LLVM distribution
+repository.
 
 `packages/`
 
@@ -69,4 +71,4 @@ feature-verification and language-standard test suites.
 `tools/`
 
 Contains utility scripts that Bazel executes directly, most notably the QNX
-credential helper used for authenticated downloads from `qnx.com`.
+credential helper used for authenticated downloads from `qnx.com`.

rules/BUILD

@@ -0,0 +1,17 @@
+# *******************************************************************************
+# Copyright (c) 2026 Contributors to the Eclipse Foundation
+#
+# See the NOTICE file(s) distributed with this work for additional
+# information regarding copyright ownership.
+#
+# This program and the accompanying materials are made available under the
+# terms of the Apache License Version 2.0 which is available at
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# SPDX-License-Identifier: Apache-2.0
+# *******************************************************************************
+
+exports_files(
+    ["fast_llvm_repo.bzl"],
+    visibility = ["//visibility:public"],
+)

rules/fast_llvm_repo.bzl

@@ -0,0 +1,178 @@
+# *******************************************************************************
+# Copyright (c) 2026 Contributors to the Eclipse Foundation
+#
+# See the NOTICE file(s) distributed with this work for additional
+# information regarding copyright ownership.
+#
+# This program and the accompanying materials are made available under the
+# terms of the Apache License Version 2.0 which is available at
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# SPDX-License-Identifier: Apache-2.0
+# *******************************************************************************
+
+"""Repository rule for fast extraction of prebuilt LLVM distributions.
+
+This rule is intended to be used together with `toolchains_llvm`. It creates
+the LLVM distribution repository that `toolchains_llvm` normally generates,
+but uses the host's parallel `xz` decompressor before streaming into `tar`.
+"""
+
+# The supported LLVM releases. Each entry maps a requested version and host
+# architecture to the upstream archive Bazel downloads and verifies.
+_LLVM_DISTRIBUTIONS = {
+    "19.1.0": {
+        "aarch64": {
+            # Official prebuilt LLVM package for 64-bit ARM Linux hosts.
+            "url": "https://github.com/llvm/llvm-project/releases/download/llvmorg-19.1.0/clang+llvm-19.1.0-aarch64-linux-gnu.tar.xz",
+            # SHA-256 pinned by toolchains_llvm for this exact upstream archive.
+            "sha256": "7bb54afd330fe1a1c2d4c593fa1e2dbe2abd9bf34fb3597994ff41e443cf144b",
+        },
+        "x86_64": {
+            # Official prebuilt LLVM package for 64-bit x86 Linux hosts.
+            "url": "https://github.com/llvm/llvm-project/releases/download/llvmorg-19.1.0/LLVM-19.1.0-Linux-X64.tar.xz",
+            # SHA-256 of that archive.
+            "sha256": "cee77d641690466a193d9b88c89705de1c02bbad46bde6a3b126793c0a0f2923",
+        },
+    },
+    "19.1.1": {
+        "aarch64": {
+            # Official prebuilt LLVM package for 64-bit ARM Linux hosts.
+            "url": "https://github.com/llvm/llvm-project/releases/download/llvmorg-19.1.1/clang+llvm-19.1.1-aarch64-linux-gnu.tar.xz",
+            # SHA-256 pinned by toolchains_llvm for this exact upstream archive.
+            "sha256": "414d2ebef10c5035e9df10a224e81b484dbe17d319373050d0c1b3b1467040d2",
+        },
+        "x86_64": {
+            # Official prebuilt LLVM package for 64-bit x86 Linux hosts.
+            "url": "https://github.com/llvm/llvm-project/releases/download/llvmorg-19.1.1/LLVM-19.1.1-Linux-X64.tar.xz",
+            "sha256": "8204de000b6a6921f0572e038336601e3225898e9a253c8aaa43b0a5fae8a4ce",
+        },
+    },
+    "22.1.7": {
+        "aarch64": {
+            # Official prebuilt LLVM package for 64-bit ARM Linux hosts.
+            "url": "https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.7/LLVM-22.1.7-Linux-ARM64.tar.xz",
+            # SHA-256 from toolchains_llvm's distribution metadata for this
+            # exact upstream archive. `ctx.download` rejects altered or corrupt
+            # downloads.
+            "sha256": "118ca2d3ad9da34367e05735317854e7977db45dc4c02a32af58da64c23b8789",
+        },
+        "x86_64": {
+            # Official prebuilt LLVM package for 64-bit x86 Linux hosts.
+            "url": "https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.7/LLVM-22.1.7-Linux-X64.tar.xz",
+            "sha256": "edb0522b41e261819c06ea437d249f9b8acfa413d3805bc9920eec6fb76ff830",
+        },
+    },
+}
+
+_ARCH_ALIASES = {
+    "aarch64": "aarch64",
+    "amd64": "x86_64",
+    "arm64": "aarch64",
+    "x86_64": "x86_64",
+}
+
+def _log(message):
+    print("[fast_llvm_repo] " + message)
+
+# Implementation of `fast_llvm_repo`. Repository rules run while Bazel is
+# preparing external dependencies, before analysing or building project code.
+def _fast_llvm_repo_impl(ctx):
+    # The distributions below contain Linux binaries and are used as the
+    # execution root by `toolchains_llvm`; do not create a repository that
+    # cannot run on the host.
+    if ctx.os.name.lower() != "linux":
+        fail("Unsupported host operating system: %s (fast_llvm_repo supports Linux hosts only)" % ctx.os.name)
+
+    # Verify tools needed for extraction are available in the host's PATH. The rule
+    # will fail with a clear message if any of them is missing.
+    tar = ctx.which("tar")
+    if not tar:
+        fail("tar not found in PATH")
+    xz = ctx.which("xz")
+    if not xz:
+        fail("xz not found in PATH")
+    bash = ctx.which("bash")
+    if not bash:
+        fail("bash not found in PATH")
+
+    # Select the archive native to the host which will execute this toolchain.
+    host_arch = _ARCH_ALIASES.get(ctx.os.arch)
+    if not host_arch:
+        fail("Unsupported host architecture: %s" % ctx.os.arch)
+
+    # Look up the requested release and its host-specific archive.
+    distributions = _LLVM_DISTRIBUTIONS.get(ctx.attr.llvm_version)
+    dist = distributions.get(host_arch) if distributions else None
+
+    # Fail with a useful message if MODULE.bazel asks for a version that has no
+    # URL and checksum entry yet.
+    if not dist:
+        fail("Unsupported LLVM version/host architecture: %s/%s" % (ctx.attr.llvm_version, host_arch))
+
+    # Choose a temporary file inside this external repository for the archive.
+    archive = ctx.path("llvm.tar.xz")
+
+    # Download the archive and verify its SHA-256 before using it.
+    _log("downloading LLVM %s for %s" % (ctx.attr.llvm_version, host_arch))
+    ctx.download(
+        url = dist["url"],
+        output = archive,
+        sha256 = dist["sha256"],
+    )
+    _log("download completed")
+
+    # LLVM's archive has many independently decompressible XZ blocks. Let xz
+    # decode them in parallel, then stream the resulting tar file directly to
+    # tar; materializing the uncompressed archive would be wasteful. Bash's
+    # `pipefail` makes a decompressor failure fail the whole pipeline.
+
+    _log("extracting LLVM with xz -T0 and tar")
+
+    result = ctx.execute(
+        [
+            bash,
+            "-o",
+            "pipefail",
+            "-c",
+            "set -e; \"$1\" -T0 -dc \"$2\" | \"$3\" -xf - --strip-components=1 -C \"$4\"",
+            "fast_llvm_repo",
+            xz,
+            archive,
+            tar,
+            ctx.path("."),
+        ],
+        # Allow the relatively large archive up to 30 minutes to unpack.
+        timeout = 1800,
+        # Do not suppress command output from Bazel's repository-rule log.
+        quiet = False,
+    )
+
+    # Stop repository creation if extraction reported an error.
+    if result.return_code:
+        fail(result.stderr)
+    _log("extraction completed")
+
+    # The archive is no longer needed after extraction; leave only the LLVM
+    # distribution files in the external repository.
+    ctx.delete(archive)
+
+    # Generate the BUILD file expected by toolchains_llvm. LLVM 16 and later
+    # store compiler resources in a directory named by the major version.
+    _log("generating BUILD.bazel")
+    ctx.template(
+        "BUILD.bazel",
+        Label("@toolchains_llvm//toolchain:BUILD.llvm_repo.tpl"),
+        substitutions = {"{LLVM_VERSION}": ctx.attr.llvm_version.split(".")[0]},
+    )
+    _log("llvm setup completed")
+
+# Public repository rule used from MODULE.bazel. Its only user-facing input
+# is an LLVM version.
+fast_llvm_repo = repository_rule(
+    implementation = _fast_llvm_repo_impl,
+    attrs = {
+        # Required version key used to select an entry in `_LLVM_DISTRIBUTIONS`.
+        "llvm_version": attr.string(mandatory = True),
+    },
+)