diff --git a/Changelog.md b/Changelog.md index 8dd1a08..b44c889 100644 --- a/Changelog.md +++ b/Changelog.md @@ -1,4 +1,9 @@ ### v1.2-dev +- `--capture rllvm` (or `REACHABILITY_CAPTURE=rllvm`) records bitcode with + rllvm 0.7.1 or newer instead of gllvm. Rust and mixed targets then take + their bitcode from one build and one extraction from the linked artifact, + so crates that only run at build time (`syn`, `quote`, `cc`, ...) are no + longer analyzed as part of the target. - Support cargo 1.100's new build layout - C/C++ analysis bitcode is now compiled at `-O0` in addition to `-fno-inline -fno-inline-functions` (bitcode compile only, via diff --git a/README.md b/README.md index 765e021..0e90ccc 100644 --- a/README.md +++ b/README.md @@ -64,15 +64,22 @@ flowchart TD driver (Python) analyzer (C++ / LLVM) ─────────────── ───────────────────── acquire bitcode ─┐ - C/C++ : gllvm │ llvm-link load .bc → build call graph → - Rust : rustc ├─► merge .bc ─► resolve indirect calls → BFS from - --emit=llvm-bc ─┘ entry → JSON report + coverage lists + C/C++ : gllvm │ + Rust : rustc │ llvm-link load .bc → build call graph → + --emit=llvm-bc ├─► merge .bc ─► resolve indirect calls → BFS from + │ entry → JSON report + coverage lists + or, with │ + --capture rllvm: │ + C/C++/Rust from ─┘ + one build ``` The driver builds the project so that it emits bitcode, and merges it with -`llvm-link`. The analyzer builds the call graph, resolves indirect calls by -type and by tracking where function addresses flow, and walks the graph from -the entries; see [docs/analysis.md](docs/analysis.md). +`llvm-link`. With `--capture rllvm`, one build records the C, C++ and Rust +bitcode together, and `rllvm-get-bc` extracts it from the linked artifact. The +analyzer builds the call graph, resolves indirect calls by type and by +tracking where function addresses flow, and walks the graph from the entries; +see [docs/analysis.md](docs/analysis.md). ## Prerequisites @@ -83,6 +90,9 @@ the entries; see [docs/analysis.md](docs/analysis.md). especially for Rust. - **Go** (to install gllvm), **Python 3.10 or newer**, and a **C++17** compiler. + - Instead of gllvm: [rllvm](https://github.com/h1994st/rllvm) 0.7.1 or + newer, with `--capture rllvm`; see + [Capturing with rllvm](docs/usage.md#capturing-with-rllvm). - For Rust targets: a nightly **rustc** and **cargo** whose LLVM is not newer than yours. - [AFL++](https://github.com/AFLplusplus/AFLplusplus) from commit @@ -94,6 +104,7 @@ the entries; see [docs/analysis.md](docs/analysis.md). ```bash bash scripts/setup.sh # gllvm, rust-src, .venv, analyzer build +REACHABILITY_CAPTURE=rllvm bash scripts/setup.sh # rllvm instead of gllvm ``` Or step by step: `make venv`, then `make build` (or `make build LLVM_MAJOR=23`). diff --git a/docs/analysis.md b/docs/analysis.md index 03e3df3..225a860 100644 --- a/docs/analysis.md +++ b/docs/analysis.md @@ -9,9 +9,14 @@ metric fields mean, and where the analysis is known to fall short. driver (Python) analyzer (C++ / LLVM) ─────────────── ───────────────────── acquire bitcode ─┐ - C/C++ : gllvm │ llvm-link load .bc → build call graph → - Rust : rustc ├─► merge .bc ─► resolve indirect calls → BFS from - --emit=llvm-bc ─┘ entry → JSON report + coverage lists + C/C++ : gllvm │ + Rust : rustc │ llvm-link load .bc → build call graph → + --emit=llvm-bc ├─► merge .bc ─► resolve indirect calls → BFS from + │ entry → JSON report + coverage lists + or, with │ + --capture rllvm: │ + C/C++/Rust from ─┘ + one build ``` The **driver** (`driver/reachability/`) checks that the LLVM tools agree on one diff --git a/docs/development.md b/docs/development.md index d843978..8685a79 100644 --- a/docs/development.md +++ b/docs/development.md @@ -6,7 +6,9 @@ bash scripts/setup.sh # gllvm 1.3.1, rust-src, .venv, analyzer build ``` -`scripts/setup.sh` installs gllvm with `go install`, adds the `rust-src` +`scripts/setup.sh` installs gllvm with `go install` (with +`REACHABILITY_CAPTURE=rllvm`, rllvm 0.7.1 or newer with `brew install`, or +`cargo install` without Homebrew, unless a new enough one is on `PATH`), adds the `rust-src` component, creates `.venv` with the driver (editable) and the pinned test dependencies from `requirements-dev.txt`, and builds the analyzer. It does not upgrade pip. `make venv` only creates the virtual environment. diff --git a/docs/rust.md b/docs/rust.md index b7f404e..a10b400 100644 --- a/docs/rust.md +++ b/docs/rust.md @@ -9,7 +9,9 @@ Rust projects are built in one of two ways: Both add `--emit=llvm-bc` to every rustc call and collect one bitcode file per crate and codegen unit. rustc's LLVM must not be newer than the analyzer's; -see [llvm-support.md](llvm-support.md). +see [llvm-support.md](llvm-support.md). With `--capture rllvm`, both instead +take the bitcode from the linked artifact; see +[Capturing with rllvm](usage.md#capturing-with-rllvm). The outputs on this page come from reachability 1.2-dev (commit `f5adeab`) with LLVM 23 and rustc 1.99.0-nightly (2026-07-12, LLVM 22.1.8). diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 61890f7..d0cb3d3 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -62,6 +62,11 @@ The run checks for `gclang`, `gclang++` and `get-bc` before it builds and lists the missing ones. `scripts/setup.sh` installs gllvm; then put it on `PATH`: `export PATH="$(go env GOPATH)/bin:$PATH"`. +With `--capture rllvm` it checks for `rllvm-cc`, `rllvm-cxx`, `rllvm-get-bc` +and `rllvm-info` instead, plus `rllvm-rustc` for Rust targets. +`REACHABILITY_CAPTURE=rllvm bash scripts/setup.sh` installs rllvm 0.7.1 or +newer with `brew install`, or with `cargo install` without Homebrew. + Test: `test_acquire_c.py::test_acquire_checks_gllvm_tools_before_building`. ### The build failed diff --git a/docs/usage.md b/docs/usage.md index 7331de1..8fd5b38 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -50,8 +50,9 @@ reachability run --lang --project [options] | `--dot FILE` | none | Also write the reachable call graph as Graphviz DOT | | `--no-edges` | off | Leave the `edges` array out of the report | | `--build-cmd CMD` | detected | Shell command that builds the project, see [C and C++ builds](#c-and-c-builds) and [rust.md](rust.md#native-harness-builds) | -| `--artifact PATH` | detected | C/C++: the built file to take bitcode from, see [Artifact selection](#artifact-selection) | -| `--static-libs {auto,none,all}` | `auto` | C/C++: which static archives to analyze in full, see [Static libraries](#static-libraries) | +| `--capture {gllvm,rllvm}` | `gllvm` | Wrappers that record the bitcode, see [Capturing with rllvm](#capturing-with-rllvm) | +| `--artifact PATH` | detected | C/C++, or any target with `--capture rllvm`: the built file to take bitcode from, see [Artifact selection](#artifact-selection) | +| `--static-libs {auto,none,all}` | `auto` | C/C++, or any target with `--capture rllvm`: which static archives to analyze in full, see [Static libraries](#static-libraries) | | `--optimize` | off | Analyze at the build's own optimization level, see [Source-faithful build](#source-faithful-build) | | `--profile {debug,release}` | tool default | Rust build profile, see [rust.md](rust.md#profile-and-codegen-units) | | `--codegen-units N` | detected | `rust` and `mixed`: rustc `-Ccodegen-units`, see [rust.md](rust.md#profile-and-codegen-units) | @@ -85,7 +86,8 @@ only starts from `main`; a `staticlib` harness needs `libfuzzer`, `ziggy` and `afl` build through the fuzzer's own command. `rust` and `mixed` run a plain `cargo build`. Both are described in -[rust.md](rust.md). +[rust.md](rust.md). With `--capture rllvm`, every target takes its bitcode +from one build instead; see [Capturing with rllvm](#capturing-with-rllvm). Test: `test_cli.py::test_target_entry_defaults`. @@ -303,6 +305,62 @@ Tests: `test_acquire_c.py::test_build_env_injects_opt0_by_default`, `::test_acquire_forwards_optimize_to_build_env`, and the `test_report.py::test_optimization_advisory_*` tests. +## Capturing with rllvm + +`--capture rllvm`, or `REACHABILITY_CAPTURE=rllvm`, records the bitcode with +[rllvm](https://github.com/h1994st/rllvm) 0.7.1 or newer instead of gllvm +(`brew install h1994st/tap/rllvm` or `cargo install rllvm`). `rllvm-cc`, +`rllvm-cxx`, `rllvm-rustc`, `rllvm-get-bc` and `rllvm-info` must be on `PATH`. +gllvm stays the default. + +**C and C++.** The build runs with `CC=rllvm-cc` and `CXX=rllvm-cxx`. The +driver writes a private `RLLVM_CONFIG` that names the same `clang`, `clang++` +and `llvm-link` as for gllvm, and passes the +[source-faithful flags](#source-faithful-build), together with any +`LLVM_BITCODE_GENERATION_FLAGS`, as rllvm's `bitcode_generation_flags`, which +also apply to the bitcode compile only. Build command detection, artifact +selection and static libraries work as with gllvm. Archive members are matched +by the exact bitcode paths rllvm records rather than by object name: rllvm +names a bitcode file after its source and compilation +(`.parser_.o.bc`), not after the member object (`parser.o`). + +**Rust and mixed.** rllvm also wraps rustc, so every target takes its bitcode +from one build and one extraction from the linked artifact, with no separate +Rust collection and no merge of the two: + +| `--lang` | Build command | Wrapped | +|---|---|---| +| `rust` | `cargo build` | rustc | +| `libfuzzer`, `ziggy`, `afl` | the fuzzer's own command | rustc | +| `mixed` | the C/C++ build command | rustc, `CC`, `CXX` | + +`--build-cmd` replaces the command in each case. A `mixed` build must link +the C/C++ code and the Rust crates into the artifact itself, for example +`--build-cmd 'cargo build && make && $CXX glue.o target/debug/libfoo.a -o fuzzer' --artifact fuzzer`. +Harness builds leave `CC` and `CXX` alone, so libFuzzer's C++ runtime stays +out of the report, and still run the clean command afterwards. + +The rustc flags are the ones the per-crate collection adds: `-Copt-level=0` +unless `--optimize` (for `rust` and `mixed` with the profile's debug +assertions and overflow checks pinned), the `--mangling` flags, and +`--codegen-units` when given. Cargo applies the profile itself. + +The bitcode is what the linker put into the artifact. Crates that only run at +build time, the dependencies of build scripts and proc macros such as `syn`, +`quote` or `cc`, are therefore not analyzed, whereas the per-crate collection +merges them in. A library-only crate yields only its own bitcode; point +`--artifact` at a binary that links it, such as the fuzz harness. The artifact +search also covers a cargo workspace's target directory and skips cargo's own +`deps/`, `build/` and `incremental/` copies in favor of the uplifted output. + +Tests: the `test_acquire_c.py::test_rllvm_*` tests, +`::test_extract_bc_rllvm_writes_manifest_from_catalog`, +`::test_include_static_libs_rllvm_matches_members_by_bitcode_path`, +`::test_find_artifacts_prefers_cargo_uplifted_output`, +`::test_find_artifacts_searches_a_workspace_target_outside_the_project`, +`test_cli.py::test_run_static_archives_end_to_end` and +`test_reachable_sets.py::test_mixed_c_rust_reachable_from_one_rllvm_extraction`. + ## Output files The file formats are described in [output.md](output.md). @@ -364,7 +422,8 @@ Tests: `test_cli.py::test_clean_c_removes_build_artifacts_and_outputs`, |---|---| | `REACHABILITY_ANALYZER` | Analyzer binary; default `analyzer/build/reachability-analyzer` in the repository | | `CLANG`, `CLANGXX`, `LLVM_LINK` | Override one tool; otherwise the version matching the analyzer's LLVM is used (`clang-23`, …), then the plain name | -| `PATH` | Must contain gllvm's `gclang`, `gclang++` and `get-bc` for `c`, `cpp` and `mixed` | +| `PATH` | Must contain gllvm's `gclang`, `gclang++` and `get-bc` for `c`, `cpp` and `mixed`, or rllvm's tools with `--capture rllvm` | +| `REACHABILITY_CAPTURE` | Default for `--capture`, see [Capturing with rllvm](#capturing-with-rllvm) | | `LLVM_BITCODE_GENERATION_FLAGS` | Extra flags for gllvm's bitcode compile, see [Source-faithful build](#source-faithful-build) | | `RUSTFLAGS`, `CARGO_ENCODED_RUSTFLAGS` | Kept and merged with the driver's flags, see [rust.md](rust.md#plain-cargo-builds) | | `CARGO_TARGET_DIR` | Honoured when collecting Rust bitcode | diff --git a/driver/reachability/acquire_c.py b/driver/reachability/acquire_c.py index 87aafd7..032c8e2 100644 --- a/driver/reachability/acquire_c.py +++ b/driver/reachability/acquire_c.py @@ -1,8 +1,8 @@ -"""C / C++ bitcode acquisition via gllvm. +"""C / C++ bitcode acquisition via gllvm or rllvm. -Build the project with the gclang/gclang++ wrappers (which embed bitcode-path -metadata into each object), then run get-bc on the built artifact to extract a -whole-program .bc. The artifact is auto-detected from the build output when not +Build the project with the gclang/gclang++ wrappers (or rllvm-cc/rllvm-cxx, with +capture="rllvm"; both embed bitcode-path metadata into each object), then run +get-bc (or rllvm-get-bc) on the built artifact to extract a whole-program .bc. The artifact is auto-detected from the build output when not given explicitly. Independent of the project's own LTO setup. When the build command is auto-detected (no explicit --build-cmd), it also forces @@ -26,6 +26,7 @@ import atexit from collections import deque import hashlib +import json import mmap import os import re @@ -48,10 +49,21 @@ class AcquireError(RuntimeError): ".cache"} _PROBE_DIRS = {"CMakeFiles", "meson-private"} _PROBE_PREFIXES = ("conftest", ".conftest") +_CARGO_INTERNAL_DIRS = {".fingerprint", "fingerprint", "build", "deps", "incremental"} BUILD_DIR = "build-reachability" -_BC_MARKERS = (b".llvm_bc", b"__llvm_bc") +# Sections that record the per-object bitcode paths: gllvm's `.llvm_bc` (Mach-O +# `__llvm_bc`) and rllvm's `.rllvm_bc` (`__rllvm_bc`). rllvm uses names of its +# own because LLVM's `.llvmbc`, which it once used for COFF and WebAssembly, is +# the section `clang -fembed-bitcode` writes and wasm-ld drops at link time. +# Neither name contains the other, so both are listed. +_BC_MARKERS = (b".llvm_bc", b"__llvm_bc", b".rllvm_bc", b"__rllvm_bc") _ARCHIVE_MAGIC = b"!\n" _THIN_ARCHIVE_MAGIC = b"!\n" +CAPTURES = ("gllvm", "rllvm") +_CAPTURE_TOOLS = { + "gllvm": ("gclang", "gclang++", "get-bc"), + "rllvm": ("rllvm-cc", "rllvm-cxx", "rllvm-get-bc", "rllvm-info"), +} _KIND_RANK = {"exec": 3, "shared": 2, "archive": 1, "object": 0} _MACHO_MAGIC = (0xFEEDFACE, 0xFEEDFACF, 0xCEFAEDFE, 0xCFFAEDFE, 0xBEBAFECA, 0xBFBAFECA) @@ -64,6 +76,16 @@ class AcquireError(RuntimeError): _OPT_LEVEL_FLAG = re.compile(r"(?:\A|\s)-O") +def _bitcode_generation_flags(inherited: str, optimize: bool) -> str: + """Flags for the bitcode compile only. An inherited -O level is kept and + only the no-inlining flags are added; optimize=True adds nothing.""" + if optimize: + return inherited + injected = (_BITCODE_NO_INLINE_FLAGS if _OPT_LEVEL_FLAG.search(inherited) + else _BITCODE_SOURCE_FAITHFUL_FLAGS) + return (inherited + " " + injected).strip() + + def _build_env(clang_bindir: str, optimize: bool = False) -> dict: """Environment for the gllvm build. By default (optimize=False) it asks gllvm to emit source-faithful bitcode via LLVM_BITCODE_GENERATION_FLAGS, which gllvm @@ -80,10 +102,74 @@ def _build_env(clang_bindir: str, optimize: bool = False) -> dict: env["LLVM_COMPILER_PATH"] = clang_bindir env["LC_ALL"] = "C" if not optimize: - inherited = env.get("LLVM_BITCODE_GENERATION_FLAGS", "") - injected = (_BITCODE_NO_INLINE_FLAGS if _OPT_LEVEL_FLAG.search(inherited) - else _BITCODE_SOURCE_FAITHFUL_FLAGS) - env["LLVM_BITCODE_GENERATION_FLAGS"] = (inherited + " " + injected).strip() + env["LLVM_BITCODE_GENERATION_FLAGS"] = _bitcode_generation_flags( + env.get("LLVM_BITCODE_GENERATION_FLAGS", ""), optimize) + return env + + +_RLLVM_RUSTC_WRAPPER = """#!/bin/sh +exec rllvm-rustc "$@" $REACH_EXTRA_RUSTFLAGS +""" + + +def _rllvm_rustc_env(env, work_dir, rustc_flags): + """Wrap every rustc call in rllvm-rustc, appending `rustc_flags` after the + ones cargo or a fuzzing tool passes so they win over the profile.""" + wrapper = os.path.join(work_dir, "rllvm-rustc-wrapper.sh") + try: + with open(wrapper, "w") as fh: + fh.write(_RLLVM_RUSTC_WRAPPER) + os.chmod(wrapper, 0o755) + except OSError as exc: + raise AcquireError(f"cannot create rllvm-rustc wrapper: {exc}") from exc + env["RUSTC_WRAPPER"] = wrapper + env["REACH_EXTRA_RUSTFLAGS"] = " ".join(rustc_flags) + return env + + +def _toml_string(value): + return '"' + value.replace("\\", "\\\\").replace('"', '\\"') + '"' + + +def _rllvm_env(env, tc, work_dir, optimize=False, wrap_c=True): + """Point rllvm-cc/rllvm-cxx at the exact toolchain through a private + RLLVM_CONFIG. The tools rllvm needs beyond clang and llvm-link come from + clang's own (symlink-resolved) bin directory. The bitcode-only flags are the + ones _build_env gives gllvm, including an inherited + LLVM_BITCODE_GENERATION_FLAGS.""" + def resolved(value): + return os.path.abspath(shutil.which(value) or value) + + clang = resolved(tc.clang) + bindir = os.path.dirname(os.path.realpath(clang)) + keys = { + "llvm_config_filepath": os.path.join(bindir, "llvm-config"), + "clang_filepath": clang, + "clangxx_filepath": resolved(getattr(tc, "clangxx", clang)), + "llvm_ar_filepath": os.path.join(bindir, "llvm-ar"), + "llvm_link_filepath": resolved(getattr(tc, "llvm_link", "llvm-link")), + "llvm_bindir": bindir, + } + objcopy = os.path.join(bindir, "llvm-objcopy") + if os.path.exists(objcopy): + keys["llvm_objcopy_filepath"] = objcopy + lines = [f"{key} = {_toml_string(value)}" for key, value in keys.items()] + flags = _bitcode_generation_flags( + env.get("LLVM_BITCODE_GENERATION_FLAGS", ""), optimize).split() + if flags: + lines.append("bitcode_generation_flags = [" + + ", ".join(_toml_string(f) for f in flags) + "]") + config = os.path.join(work_dir, "rllvm.toml") + try: + with open(config, "w") as fh: + fh.write("\n".join(lines) + "\n") + except OSError as exc: + raise AcquireError(f"cannot write rllvm config: {exc}") from exc + if wrap_c: + env["CC"] = "rllvm-cc" + env["CXX"] = "rllvm-cxx" + env["RLLVM_CONFIG"] = config + env.pop("LLVM_BITCODE_GENERATION_FLAGS", None) return env @@ -276,9 +362,28 @@ def _classify(path): return None -def _built_files(project_dir): - for root, dirs, files in os.walk(project_dir): - dirs[:] = [d for d in dirs if d not in _SKIP_DIRS and d not in _PROBE_DIRS] +def _search_roots(project_dir, extra_dirs=()): + """`project_dir` plus each existing directory in `extra_dirs` outside it, + such as a cargo workspace's target directory.""" + roots = [project_dir] + base = os.path.realpath(project_dir) + for directory in extra_dirs: + real = os.path.realpath(directory) + if os.path.isdir(real) and os.path.commonpath([base, real]) != base: + roots.append(directory) + return list(dict.fromkeys(roots)) + + +def _built_files(project_dir, skip_cargo_internals=False, extra_dirs=()): + """Files under `project_dir` and `extra_dirs`. skip_cargo_internals also + prunes cargo's own directories inside a profile directory (the one holding + `.cargo-lock`); the final artifacts are uplifted next to them.""" + for root, dirs, files in (entry for top in _search_roots(project_dir, extra_dirs) + for entry in os.walk(top)): + skip = _SKIP_DIRS | _PROBE_DIRS + if skip_cargo_internals and ".cargo-lock" in files: + skip = skip | _CARGO_INTERNAL_DIRS + dirs[:] = [d for d in dirs if d not in skip] for f in files: if f.startswith(_PROBE_PREFIXES): continue @@ -288,7 +393,7 @@ def _built_files(project_dir): yield p -def find_artifacts(project_dir, newer_than=None): +def find_artifacts(project_dir, newer_than=None, extra_dirs=()): """Walk `project_dir` for built files get-bc can read, ranked best-first: files carrying gllvm's bitcode section come first, then ones built by this run, then by kind (executable > shared lib > archive > object), then newest. @@ -296,10 +401,13 @@ def find_artifacts(project_dir, newer_than=None): gclang too: CMake's `CMakeFiles/` (compiler-ID and ABI-check executables), Meson's `meson-private/` (sanity-check executable) and autoconf's `conftest*` files, including the hidden `.conftest.c.o` gllvm leaves behind, which - configure's `rm -f conftest*` does not remove. + configure's `rm -f conftest*` does not remove. Cargo's internal copies + (`deps/`, build scripts, incremental objects) are skipped in favour of the + artifacts cargo uplifts. """ cands = [] - for p in _built_files(project_dir): + for p in _built_files(project_dir, skip_cargo_internals=True, + extra_dirs=extra_dirs): kind = _classify(p) if kind is None: continue @@ -323,28 +431,61 @@ def _artifact_score(path, newer_than=None): return (_has_bitcode_marker(path), fresh, _KIND_RANK.get(kind, -1)) -def _extract_bc(art, out, archive=False, manifest=False, env=None): +def _rllvm_modules(art, env=None): + """Bitcode paths rllvm recorded in `art`, from its `rllvm-info --json` + catalog. Returns (paths, error); paths is None on failure.""" + try: + r = subprocess.run(["rllvm-info", "--json", art], capture_output=True, + env=env) + except OSError as exc: + return None, f"cannot run rllvm-info: {exc}" + if r.returncode != 0: + return None, tail(r.stderr or r.stdout) or f"rllvm-info exited {r.returncode}" + try: + modules = json.loads(r.stdout)["modules"] + except (ValueError, KeyError, TypeError) as exc: + return None, f"unreadable rllvm-info catalog: {exc}" + return [m["path"] for m in modules if m.get("path")], "" + + +def _extract_bc(art, out, archive=False, manifest=False, env=None, + capture="gllvm"): """Run get-bc on `art` into `out`. `archive` uses -b to build one whole-archive module (instead of a lazy bitcode archive); `manifest` also writes the linked - object list to `.llvm.manifest`. Returns (ok, stderr).""" - cmd = ["get-bc"] - if archive: - cmd.append("-b") - if manifest: - cmd.append("-m") - cmd += ["-o", out, art] + object list to `.llvm.manifest`. Returns (ok, stderr). + + With capture="rllvm", rllvm-get-bc already links an archive into one module, + and the manifest is written from rllvm's catalog of `art`.""" + if capture == "rllvm": + cmd = ["rllvm-get-bc", "-o", out, art] + else: + cmd = ["get-bc"] + if archive: + cmd.append("-b") + if manifest: + cmd.append("-m") + cmd += ["-o", out, art] try: r = subprocess.run(cmd, capture_output=True, env=env) except OSError as exc: - return False, f"cannot run get-bc: {exc}" + return False, f"cannot run {cmd[0]}: {exc}" try: exists = os.path.exists(out) and os.path.getsize(out) > 0 except OSError as exc: - return False, f"cannot inspect get-bc output {out}: {exc}" + return False, f"cannot inspect {cmd[0]} output {out}: {exc}" ok = r.returncode == 0 and exists detail = tail(r.stderr or r.stdout) if not ok and not detail: - detail = f"get-bc exited {r.returncode} without usable output" + detail = f"{cmd[0]} exited {r.returncode} without usable output" + if ok and manifest and capture == "rllvm": + paths, err = _rllvm_modules(art, env) + if paths is None: + return False, err + try: + with open(out + ".llvm.manifest", "w") as fh: + fh.write("".join(p + "\n" for p in paths)) + except OSError as exc: + return False, f"cannot write manifest: {exc}" return ok, detail @@ -391,17 +532,28 @@ def _archive_members(archive_path): return {os.path.basename(line) for line in _archive_listing(archive_path)} -def _bitcode_archives(project_dir): - """Static archives under `project_dir` that carry gllvm bitcode.""" - return [p for p in _built_files(project_dir) +def _rllvm_archive_members(archive_path, env=None): + """Real paths of the bitcode rllvm recorded for each member of a static + archive, which the target's manifest names exactly.""" + paths, err = _rllvm_modules(archive_path, env) + if paths is None: + raise AcquireError(f"cannot list static archive members: {archive_path}: {err}") + return {os.path.realpath(p) for p in paths} + + +def _bitcode_archives(project_dir, extra_dirs=()): + """Static archives under `project_dir` and `extra_dirs` that carry gllvm or + rllvm bitcode.""" + return [p for p in _built_files(project_dir, extra_dirs=extra_dirs) if _classify(p) == "archive" and _has_bitcode_marker(p)] -def _plan_static_libs(manifest, archive_members, mode): +def _plan_static_libs(manifest, archive_members, mode, key=_member_name): """Decide which static archives to fully include. manifest: per-object .bc paths the linker pulled into the target. - archive_members: {archive_path: {member object name, ...}}. + archive_members: {archive_path: {member key, ...}}, where `key` maps a + manifest path to the same kind of member key (gllvm: the member object name). mode: "auto" includes only archives the target links (members intersect the manifest); "all" includes every archive given. @@ -409,7 +561,7 @@ def _plan_static_libs(manifest, archive_members, mode): (non-archive) objects itself, so merging them with the full archives produces no duplicate symbols. """ - manifest_member_names = {_member_name(p) for p in manifest} + manifest_member_names = {key(p) for p in manifest} chosen = [] ordered = sorted(archive_members.items(), key=lambda kv: len(kv[1]), reverse=True) @@ -422,11 +574,12 @@ def _plan_static_libs(manifest, archive_members, mode): def _include_static_libs(project_dir, art, kind, primary_bc, mode, - work_dir=None, env=None): + work_dir=None, env=None, capture="gllvm", + extra_dirs=()): """Replace `primary_bc` with the target's own objects plus the full contents of the static archives it links. Returns the replacement bc list, or None when there is no relevant archive.""" - archives = [a for a in _bitcode_archives(project_dir) + archives = [a for a in _bitcode_archives(project_dir, extra_dirs) if os.path.realpath(a) != os.path.realpath(art)] if not archives: return None @@ -440,11 +593,16 @@ def _include_static_libs(project_dir, art, kind, primary_bc, mode, "cannot complete static-library expansion" ) - members = {a: _archive_members(a) for a in archives} + if capture == "rllvm": + members = {a: _rllvm_archive_members(a, env) for a in archives} + key = os.path.realpath + else: + members = {a: _archive_members(a) for a in archives} + key = _member_name for archive, names in sorted(members.items()): if not names: print(f"static library (genuinely empty): {os.path.relpath(archive, project_dir)}") - chosen = _plan_static_libs(manifest, members, mode) + chosen = _plan_static_libs(manifest, members, mode, key=key) if not chosen: return None @@ -456,7 +614,8 @@ def _include_static_libs(project_dir, art, kind, primary_bc, mode, out = os.path.join(work_dir, f"{os.path.basename(a)}-{tag}.full.bc") else: out = a + ".full.bc" - ok, err = _extract_bc(a, out, archive=True, manifest=True, env=env) + ok, err = _extract_bc(a, out, archive=True, manifest=True, env=env, + capture=capture) if ok: objects = _manifest_objects(out + ".llvm.manifest") if objects: @@ -550,8 +709,9 @@ def _run_build(cmd, project_dir, env, verbose): def acquire_c_bitcode(project_dir, tc, artifact=None, build_cmd=None, static_libs="auto", verbose=False, optimize=False, - work_dir=None): - """Build `project_dir` with gllvm wrappers and extract its bitcode. + work_dir=None, capture="gllvm", rustc_flags=None, + wrap_c=True, extra_env=None, search_dirs=()): + """Build `project_dir` with gllvm (or rllvm) wrappers and extract its bitcode. artifact: path (relative to project_dir) of the built binary/object/archive. When None, the build product is auto-detected. An explicit path is strict. @@ -563,23 +723,39 @@ def acquire_c_bitcode(project_dir, tc, artifact=None, build_cmd=None, -fno-inline in the bitcode compile only, via LLVM_BITCODE_GENERATION_FLAGS, so no function the source defines is optimized away); when True leave the build's own optimization in place. + capture: "gllvm" (default) or "rllvm", the wrappers that record bitcode. + rustc_flags: rllvm only; when given, rustc is wrapped in rllvm-rustc with + these flags appended, so Rust crates linked into the artifact are extracted + with it. wrap_c=False leaves CC/CXX alone (a Rust-only target). + extra_env: further variables for the build. + search_dirs: directories outside project_dir that also hold build output, + searched for the artifact and static archives (a cargo target directory). Returns a list of absolute .bc paths to be linked together. """ - missing = [t for t in ("gclang", "gclang++", "get-bc") if not shutil.which(t)] + tools = _CAPTURE_TOOLS[capture] + (("rllvm-rustc",) if rustc_flags is not None else ()) + missing = [t for t in tools if not shutil.which(t)] if missing: - raise AcquireError( - f"{', '.join(missing)} not found on PATH; run scripts/setup.sh and " - 'export PATH="$(go env GOPATH)/bin:$PATH"' - ) + hint = ("; install rllvm 0.7.1 or newer, e.g. with " + "REACHABILITY_CAPTURE=rllvm bash scripts/setup.sh" if capture == "rllvm" + else '; run scripts/setup.sh and export PATH="$(go env GOPATH)/bin:$PATH"') + raise AcquireError(f"{', '.join(missing)} not found on PATH{hint}") if work_dir is None: work_dir = tempfile.mkdtemp(prefix="reach-c-") atexit.register(shutil.rmtree, work_dir, ignore_errors=True) - clang_bindir = os.path.dirname(os.path.abspath(tc.clang)) - env = _gllvm_env( - _build_env(clang_bindir, optimize=optimize), tc, - tool_dir=os.path.join(work_dir, "gllvm-tools"), - ) + if capture == "rllvm": + env = dict(os.environ) + env["LC_ALL"] = "C" + env = _rllvm_env(env, tc, work_dir, optimize=optimize, wrap_c=wrap_c) + if rustc_flags is not None: + env = _rllvm_rustc_env(env, work_dir, rustc_flags) + else: + clang_bindir = os.path.dirname(os.path.abspath(tc.clang)) + env = _gllvm_env( + _build_env(clang_bindir, optimize=optimize), tc, + tool_dir=os.path.join(work_dir, "gllvm-tools"), + ) + env.update(extra_env or {}) cmd = build_cmd or ["make"] before = time.time() if verbose: @@ -611,7 +787,8 @@ def _no_bitcode(base): candidates = [explicit] else: cutoff = None if cached else before - candidates = find_artifacts(project_dir, newer_than=cutoff) + candidates = find_artifacts(project_dir, newer_than=cutoff, + extra_dirs=search_dirs) if len(candidates) > 1: best = _artifact_score(candidates[0], cutoff) tied = [p for p in candidates if _artifact_score(p, cutoff) == best] @@ -632,7 +809,8 @@ def _no_bitcode(base): tag = hashlib.sha256(os.path.realpath(cand).encode()).hexdigest()[:16] out = os.path.join(work_dir, f"{os.path.basename(cand)}-{tag}.bc") ok, err = _extract_bc(cand, out, archive=(ck == "archive"), - manifest=(ck in ("exec", "shared")), env=env) + manifest=(ck in ("exec", "shared")), env=env, + capture=capture) if ok: art, kind, primary = cand, ck, out if not explicit: @@ -657,7 +835,8 @@ def _no_bitcode(base): if static_libs != "none": expanded = _include_static_libs( project_dir, art, kind, primary, static_libs, - work_dir=work_dir, env=env, + work_dir=work_dir, env=env, capture=capture, + extra_dirs=search_dirs, ) if expanded is not None: return expanded diff --git a/driver/reachability/acquire_rust.py b/driver/reachability/acquire_rust.py index 2d11df2..e5df72d 100644 --- a/driver/reachability/acquire_rust.py +++ b/driver/reachability/acquire_rust.py @@ -140,6 +140,23 @@ def _cargo_metadata(project_dir): raise AcquireError(f"cargo metadata returned malformed JSON: {exc}") from exc +def cargo_target_dirs(project_dir): + """Target directories of the cargo projects at `project_dir` and, for + cargo-fuzz, `project_dir/fuzz`. A workspace member builds into the + workspace's target directory, outside the member itself.""" + dirs = [] + for directory in (project_dir, os.path.join(project_dir, "fuzz")): + if not os.path.exists(os.path.join(directory, "Cargo.toml")): + continue + try: + target = _cargo_metadata(directory).get("target_directory") + except AcquireError: + continue + if target: + dirs.append(target) + return dirs + + def _profile_manifest(project_dir): directory = os.path.realpath(project_dir) fallback = None @@ -276,15 +293,38 @@ def _compose_rustflags(project_dir, codegen_units=1, optimize=False, existing = _config_rustflags(project_dir) flags = _emit_flags(codegen_units) + existing if not optimize: - da, oc = _resolve_assertions(project_dir, profile) - flags = flags + [ - "-Copt-level=0", - f"-Cdebug-assertions={'on' if da else 'off'}", - f"-Coverflow-checks={'on' if oc else 'off'}", - ] + flags = flags + _source_faithful_flags(project_dir, profile) + return flags + _mangling_flags(mangling) + + +def _source_faithful_flags(project_dir, profile): + """-Copt-level=0 with debug-assertions and overflow-checks pinned to the + profile's values (see _compose_rustflags).""" + da, oc = _resolve_assertions(project_dir, profile) + return [ + "-Copt-level=0", + f"-Cdebug-assertions={'on' if da else 'off'}", + f"-Coverflow-checks={'on' if oc else 'off'}", + ] + + +def rllvm_rustc_flags(project_dir, profile="debug", optimize=False, + mangling="auto", codegen_units=None): + """Flags appended to every rustc call when rllvm-rustc captures a cargo + build. Cargo already applies the profile and the project's own flags, so + only the source-faithful and mangling flags are added, plus codegen-units + when given explicitly.""" + flags = [] if codegen_units is None else [f"-Ccodegen-units={codegen_units}"] + if not optimize: + flags += _source_faithful_flags(project_dir, profile) return flags + _mangling_flags(mangling) +def native_rustc_flags(optimize=False, mangling="auto"): + """Flags appended to every rustc call of a fuzzing tool's own build.""" + return ([] if optimize else ["-Copt-level=0"]) + _mangling_flags(mangling) + + def _mangling_flags(mangling): """rustc flags that force the symbol-mangling scheme. rustc accepts an explicit legacy only together with -Zunstable-options, so forcing legacy @@ -736,7 +776,7 @@ def acquire_rust_bitcode_native(project_dir, build_cmd, shell=False, verbose=Fal except OSError as exc: raise AcquireError(f"cannot create rustc bitcode wrapper: {exc}") from exc - extra = ([] if optimize else ["-Copt-level=0"]) + _mangling_flags(mangling) + extra = native_rustc_flags(optimize, mangling) env = dict(os.environ) env["RUSTC_WRAPPER"] = wrapper diff --git a/driver/reachability/cli.py b/driver/reachability/cli.py index 5e4916b..10c55a8 100644 --- a/driver/reachability/cli.py +++ b/driver/reachability/cli.py @@ -7,6 +7,7 @@ import argparse import json import os +import shlex import shutil import subprocess import sys @@ -58,14 +59,85 @@ def default_analyzer(): return path +def _rllvm_rust_build(args, mode): + """(shell build command, rustc flags, wrap C) for a Rust or mixed target + captured with rllvm: the fuzzing tool's own build, `cargo build`, or the + project's C/C++ build (which links the Rust crates itself).""" + profile = args.profile or "debug" + if mode == "mixed": + flags = acquire_rust.rllvm_rustc_flags( + args.project, profile, args.optimize, args.mangling, args.codegen_units) + return (args.build_cmd or acquire_c.detect_build_cmd(args.project) + or "make"), flags, True + if args.lang in _RUST_NATIVE: + flags = acquire_rust.native_rustc_flags(args.optimize, args.mangling) + return (args.build_cmd + or shlex.join(_native_build_cmd(args.lang, args.profile))), flags, False + flags = acquire_rust.rllvm_rustc_flags( + args.project, profile, args.optimize, args.mangling, args.codegen_units) + cmd = ["cargo", "build"] + if profile == "release": + cmd.append("--release") + if args.build_std: + cmd += ["-Zbuild-std", "--target", acquire_rust._rustc_host()] + return args.build_cmd or shlex.join(cmd), flags, False + + +def _acquire_rllvm_rust(args, tc, mode, verbose=False, work_dir=None): + """One build with rustc (and, for mixed, CC/CXX) wrapped by rllvm, then one + extraction from the linked artifact: C, C++ and Rust bitcode come out + together, so neither a separate Rust collection nor a merge of the two is + needed.""" + build, flags, wrap_c = _rllvm_rust_build(args, mode) + print(f"build command: {build}" + f"{' (from --build-cmd)' if args.build_cmd else ''}") + extra_env = {} + if args.lang in _RUST_NATIVE: + extra_env["AFLRS_REQUIRE_PLUGINS"] = os.environ.get( + "AFLRS_REQUIRE_PLUGINS", "1") + before = _snapshot_c_artifacts(args.project) + try: + bcs = acquire_c.acquire_c_bitcode( + args.project, tc, args.artifact, ["sh", "-c", build], + static_libs=args.static_libs, verbose=verbose, + optimize=args.optimize, work_dir=work_dir, capture="rllvm", + rustc_flags=flags, wrap_c=wrap_c, extra_env=extra_env, + search_dirs=acquire_rust.cargo_target_dirs(args.project), + ) + finally: + _record_c_artifacts(args.project, before) + if args.lang in _RUST_NATIVE and not args.optimize: + # Static-library expansion can return bitcode from the build tree, + # which the clean deletes. + bcs = [_keep_bitcode(path, index, work_dir) for index, path in enumerate(bcs)] + _native_clean(args.project, args.lang, verbose) + return bcs + + +def _keep_bitcode(path, index, work_dir): + """`path`, copied into `work_dir` unless it is already there.""" + root = os.path.realpath(work_dir) + if os.path.commonpath([root, os.path.realpath(path)]) == root: + return path + kept = os.path.join(work_dir, f"kept-{index}-{os.path.basename(path)}") + try: + shutil.copyfile(path, kept) + except OSError as exc: + raise acquire_c.AcquireError(f"cannot keep bitcode {path}: {exc}") from exc + return kept + + def _acquire(args, tc, verbose=False, work_dir=None): """Return the list of .bc files for the project per --lang.""" mode = TARGETS[args.lang][0] + capture = getattr(args, "capture", "gllvm") + if capture == "rllvm" and mode in ("rust", "mixed"): + return _acquire_rllvm_rust(args, tc, mode, verbose, work_dir) bcs = [] if mode in ("c", "cpp", "mixed"): # An explicit --build-cmd wins; otherwise auto-detect the build system # from the project's files. Either runs under a shell so it can be a - # compound command; gllvm wrappers are injected via env. + # compound command; gllvm or rllvm wrappers are injected via env. build = args.build_cmd or acquire_c.detect_build_cmd(args.project) if args.build_cmd: if verbose: @@ -81,6 +153,7 @@ def _acquire(args, tc, verbose=False, work_dir=None): args.project, tc, args.artifact, build_cmd, static_libs=args.static_libs, verbose=verbose, optimize=args.optimize, work_dir=work_dir, + capture=capture, ) ) finally: @@ -346,6 +419,9 @@ def _run_info(args): if mode in ("c", "cpp", "mixed"): info["artifact"] = args.artifact info["static_libs"] = args.static_libs + info["capture"] = args.capture + elif args.capture == "rllvm": + info["capture"] = args.capture if mode in ("rust", "mixed"): info["profile"] = args.profile info["codegen_units"] = args.codegen_units @@ -459,7 +535,7 @@ def build_parser(): "fuzz harness (libfuzzer/ziggy/afl). The harness types set " "the default entry: libfuzzer->fuzz_target!, ziggy/afl->main") r.add_argument("--artifact", default=None, - help="C/C++: built binary/object/archive to extract bitcode " + help="C/C++ (any target with --capture rllvm): built binary/object/archive to extract bitcode " "from, relative to --project (default: auto-detect the " "build product)") r.add_argument("--build-cmd", default=None, dest="build_cmd", @@ -477,6 +553,13 @@ def build_parser(): "are reported, not silently dropped); 'none' keeps only " "the linker's view; 'all' includes every bitcode archive " "in the tree") + r.add_argument("--capture", default=os.environ.get("REACHABILITY_CAPTURE", "gllvm"), + choices=list(acquire_c.CAPTURES), + help="wrappers that record bitcode during the build: gllvm " + "(gclang/get-bc, C/C++ only) or rllvm (rllvm-cc, " + "rllvm-rustc, rllvm-get-bc), which also captures Rust " + "and extracts every language from the linked artifact " + "(default: $REACHABILITY_CAPTURE, else gllvm)") r.add_argument("--entry", action="append", default=None, help="entry function (repeatable; overrides the --lang default). " "Accepts a mangled symbol, a demangled name, a '::name' " diff --git a/driver/tests/test_acquire_c.py b/driver/tests/test_acquire_c.py index a6aabad..af12fc5 100644 --- a/driver/tests/test_acquire_c.py +++ b/driver/tests/test_acquire_c.py @@ -1,3 +1,4 @@ +import json import os import shutil import struct @@ -6,6 +7,11 @@ import pytest +try: + import tomllib +except ModuleNotFoundError: + import tomli as tomllib + from reachability import acquire_c @@ -100,6 +106,99 @@ def test_build_env_preserves_inherited_bcgen_flags(monkeypatch): assert "-fno-inline" in flags +def _rllvm_toolchain(tmp_path): + bindir = tmp_path / "llvm" / "bin" + bindir.mkdir(parents=True) + for name in ("clang", "clang++", "llvm-link", "llvm-config", "llvm-ar", + "llvm-objcopy"): + _write(bindir / name, b"", executable=True) + return bindir, type("TC", (), { + "clang": str(bindir / "clang"), + "clangxx": str(bindir / "clang++"), + "llvm_link": str(bindir / "llvm-link"), + })() + + +def test_rllvm_env_pins_exact_tools(tmp_path): + bindir, tc = _rllvm_toolchain(tmp_path) + env = acquire_c._rllvm_env({}, tc, str(tmp_path)) + assert env["CC"] == "rllvm-cc" + assert env["CXX"] == "rllvm-cxx" + with open(env["RLLVM_CONFIG"], "rb") as fh: + config = tomllib.load(fh) + assert config["clang_filepath"] == tc.clang + assert config["clangxx_filepath"] == tc.clangxx + assert config["llvm_link_filepath"] == tc.llvm_link + assert config["llvm_config_filepath"] == str(bindir / "llvm-config") + assert config["llvm_ar_filepath"] == str(bindir / "llvm-ar") + assert config["llvm_objcopy_filepath"] == str(bindir / "llvm-objcopy") + assert config["bitcode_generation_flags"] == [ + "-O0", "-fno-inline", "-fno-inline-functions"] + + +def test_rllvm_env_keeps_inherited_opt_level(tmp_path): + _, tc = _rllvm_toolchain(tmp_path) + env = acquire_c._rllvm_env( + {"LLVM_BITCODE_GENERATION_FLAGS": "-O1"}, tc, str(tmp_path)) + with open(env["RLLVM_CONFIG"], "rb") as fh: + flags = tomllib.load(fh)["bitcode_generation_flags"] + assert flags == ["-O1", "-fno-inline", "-fno-inline-functions"] + assert "LLVM_BITCODE_GENERATION_FLAGS" not in env + + +def test_rllvm_env_optimize_omits_flags(tmp_path): + _, tc = _rllvm_toolchain(tmp_path) + env = acquire_c._rllvm_env({}, tc, str(tmp_path), optimize=True) + with open(env["RLLVM_CONFIG"], "rb") as fh: + assert "bitcode_generation_flags" not in tomllib.load(fh) + + +def test_extract_bc_rllvm_writes_manifest_from_catalog(tmp_path, monkeypatch): + out = tmp_path / "app.bc" + calls = [] + + def run(cmd, **kwargs): + calls.append(cmd) + stdout = b"" + if cmd[0] == "rllvm-get-bc": + out.write_bytes(b"BC") + else: + stdout = json.dumps({"modules": [ + {"path": "/p/.main_1.o.bc"}, {"path": "/p/.util_2.o.bc"}, + ]}).encode() + return type("R", (), {"returncode": 0, "stdout": stdout, "stderr": b""})() + + monkeypatch.setattr(acquire_c.subprocess, "run", run) + ok, _ = acquire_c._extract_bc("/p/app", str(out), archive=True, + manifest=True, capture="rllvm") + assert ok + assert calls == [["rllvm-get-bc", "-o", str(out), "/p/app"], + ["rllvm-info", "--json", "/p/app"]] + assert (tmp_path / "app.bc.llvm.manifest").read_text() == ( + "/p/.main_1.o.bc\n/p/.util_2.o.bc\n") + + +def test_include_static_libs_rllvm_matches_members_by_bitcode_path(monkeypatch): + # rllvm names bitcode after the source and a hash, not the member object. + primary = ["/p/app/.main_1.o.bc", "/p/lib/.parser_2.o.bc"] + members = ["/p/lib/.parser_2.o.bc", "/p/lib/.tree_3.o.bc"] + manifests = { + "/p/app.bc.llvm.manifest": primary, + "/p/lib/libxml2.a.full.bc.llvm.manifest": members, + } + monkeypatch.setattr(acquire_c, "_bitcode_archives", + lambda p, *a, **k: ["/p/lib/libxml2.a", "/p/other/libother.a"]) + monkeypatch.setattr(acquire_c, "_rllvm_modules", lambda a, env=None: ( + members if a == "/p/lib/libxml2.a" else ["/p/other/.other_4.o.bc"], "")) + monkeypatch.setattr(acquire_c, "_extract_bc", lambda *a, **k: (True, "")) + monkeypatch.setattr(acquire_c, "_manifest_objects", + lambda p: manifests.get(p, [])) + result = acquire_c._include_static_libs( + "/p", "/p/app", "exec", "/p/app.bc", "auto", capture="rllvm", + ) + assert result == ["/p/app/.main_1.o.bc", "/p/lib/libxml2.a.full.bc"] + + def test_build_looks_cached(): assert acquire_c._build_looks_cached("make: Nothing to be done for 'all'.") assert acquire_c._build_looks_cached("ninja: no work to do.") @@ -283,6 +382,42 @@ def test_find_artifacts_skips_build_system_probes(tmp_path, probe): assert acquire_c.find_artifacts(str(tmp_path)) == [str(real)] +@pytest.mark.parametrize("internal", [ + "deps/app-0123456789abcdef", + "build/foo-0123456789abcdef/build-script-build", + "build/app/0123456789abcdef/out/app", +]) +def test_find_artifacts_prefers_cargo_uplifted_output(tmp_path, internal): + profile = tmp_path / "target" / "release" + profile.mkdir(parents=True) + (profile / ".cargo-lock").write_bytes(b"") + _write(str(profile / "app"), _fake_elf(2), executable=True) + path = profile / internal + path.parent.mkdir(parents=True, exist_ok=True) + _write(str(path), _fake_elf(2), executable=True) + assert acquire_c.find_artifacts(str(tmp_path)) == [str(profile / "app")] + + +def test_find_artifacts_searches_a_workspace_target_outside_the_project(tmp_path): + project = tmp_path / "workspace" / "member" + project.mkdir(parents=True) + profile = tmp_path / "workspace" / "target" / "release" + profile.mkdir(parents=True) + (profile / ".cargo-lock").write_bytes(b"") + _write(str(profile / "app"), _fake_elf(2), executable=True) + assert acquire_c.find_artifacts(str(project)) == [] + found = acquire_c.find_artifacts( + str(project), extra_dirs=[str(profile.parent), str(project)]) + assert found == [str(profile / "app")] + + +def test_find_artifacts_keeps_c_build_directories(tmp_path): + app = tmp_path / "build" / "app" + app.parent.mkdir() + _write(str(app), _fake_elf(2), executable=True) + assert acquire_c.find_artifacts(str(tmp_path)) == [str(app)] + + def test_classify_recognizes_thin_archive(tmp_path): thin = tmp_path / "libfoo.a" thin.write_bytes(b"!\n" + b"\x00" * 16) @@ -402,7 +537,7 @@ def test_include_static_libs_uses_exact_manifest_paths(monkeypatch): "/p/b/libsame.a.full.bc.llvm.manifest": ["/p/b/.same.o.bc"], "/p/app.bc.llvm.manifest": primary, } - monkeypatch.setattr(acquire_c, "_bitcode_archives", lambda p: archives) + monkeypatch.setattr(acquire_c, "_bitcode_archives", lambda p, *a, **k: archives) monkeypatch.setattr(acquire_c, "_archive_members", lambda p: {"same.o"}) monkeypatch.setattr(acquire_c, "_extract_bc", lambda *a, **k: (True, "")) monkeypatch.setattr( @@ -416,7 +551,7 @@ def test_include_static_libs_uses_exact_manifest_paths(monkeypatch): def test_include_static_libs_partial_failure_is_atomic(monkeypatch): archives = ["/p/a.a", "/p/b.a"] - monkeypatch.setattr(acquire_c, "_bitcode_archives", lambda p: archives) + monkeypatch.setattr(acquire_c, "_bitcode_archives", lambda p, *a, **k: archives) monkeypatch.setattr( acquire_c, "_manifest_objects", lambda p: ["/p/.main.o.bc", "/p/.a.o.bc", "/p/.b.o.bc"] @@ -438,7 +573,7 @@ def test_include_static_libs_partial_failure_is_atomic(monkeypatch): def test_include_static_libs_distinguishes_empty_target_manifest(monkeypatch): - monkeypatch.setattr(acquire_c, "_bitcode_archives", lambda p: ["/p/lib.a"]) + monkeypatch.setattr(acquire_c, "_bitcode_archives", lambda p, *a, **k: ["/p/lib.a"]) monkeypatch.setattr(acquire_c, "_manifest_objects", lambda p: []) with pytest.raises(acquire_c.AcquireError, match="empty target manifest"): acquire_c._include_static_libs( @@ -447,7 +582,7 @@ def test_include_static_libs_distinguishes_empty_target_manifest(monkeypatch): def test_include_static_libs_reports_genuinely_empty_archive(monkeypatch, capsys): - monkeypatch.setattr(acquire_c, "_bitcode_archives", lambda p: ["/p/empty.a"]) + monkeypatch.setattr(acquire_c, "_bitcode_archives", lambda p, *a, **k: ["/p/empty.a"]) monkeypatch.setattr(acquire_c, "_archive_members", lambda p: set()) result = acquire_c._include_static_libs( "/p", "/p/app", "archive", "/p/app.bc", "all", diff --git a/driver/tests/test_cli.py b/driver/tests/test_cli.py index 40c2c7e..74bc0b8 100644 --- a/driver/tests/test_cli.py +++ b/driver/tests/test_cli.py @@ -10,6 +10,7 @@ from reachability import cli, outputs, toolchain HAVE_GLLVM = shutil.which("gclang") is not None +HAVE_RLLVM = shutil.which("rllvm-cc") is not None class _Ok: @@ -613,15 +614,20 @@ def test_run_c_direct(analyzer, tmp_path, monkeypatch): assert "dead_fn" not in reachable -@pytest.mark.skipif(not HAVE_GLLVM, reason="gllvm not installed") -def test_run_static_archives_end_to_end(analyzer, tmp_path, monkeypatch): +@pytest.mark.parametrize("capture", [ + pytest.param("gllvm", marks=pytest.mark.skipif( + not HAVE_GLLVM, reason="gllvm not installed")), + pytest.param("rllvm", marks=pytest.mark.skipif( + not HAVE_RLLVM, reason="rllvm not installed")), +]) +def test_run_static_archives_end_to_end(analyzer, tmp_path, monkeypatch, capture): monkeypatch.setenv("REACHABILITY_ANALYZER", analyzer) work = tmp_path / "c_static_archives" shutil.copytree(os.path.join(FIXTURES, "c_static_archives"), work) out = tmp_path / "report.json" rc = cli.main([ "run", "--project", str(work), "--lang", "c", "--artifact", "app", - "--static-libs", "auto", "--out", str(out), + "--static-libs", "auto", "--capture", capture, "--out", str(out), ]) assert rc == 0 result = json.load(open(out)) diff --git a/driver/tests/test_reachable_sets.py b/driver/tests/test_reachable_sets.py index 01460f5..e5e758f 100644 --- a/driver/tests/test_reachable_sets.py +++ b/driver/tests/test_reachable_sets.py @@ -13,6 +13,7 @@ from reachability import acquire_c, acquire_rust, analyze, link, toolchain HAVE_GLLVM = shutil.which("gclang") is not None +HAVE_RLLVM = shutil.which("rllvm-rustc") is not None def assert_soundness(result, expected): @@ -140,6 +141,28 @@ def test_mixed_c_rust_reachable(analyzer, tmp_path): assert_soundness(result, _expected("mixed_c_rust")) +@pytest.mark.skipif( + not (HAVE_RLLVM and shutil.which("cargo")), reason="needs rllvm + cargo" +) +def test_mixed_c_rust_reachable_from_one_rllvm_extraction(analyzer, tmp_path): + # One build links the C++ glue and the Rust staticlib; one extraction from + # the linked binary yields both sides, with no separate Rust collection. + work = tmp_path / "mixed_c_rust" + shutil.copytree(os.path.join(FIXTURES, "mixed_c_rust"), work, + ignore=shutil.ignore_patterns("target")) + tc = _tc(analyzer) + _require_rust_readable(tc) + build = ("cargo build && make && $CXX glue.o target/debug/libmixed_rust.a " + "-shared -o libglue.so") + bcs = acquire_c.acquire_c_bitcode( + str(work), tc, "libglue.so", ["sh", "-c", build], capture="rllvm", + rustc_flags=acquire_rust.rllvm_rustc_flags(str(work)), + ) + merged = link.link_bitcode(bcs, str(work / "merged.bc"), tc) + result = analyze.analyze(merged, tc, ["LLVMFuzzerTestOneInput"]) + assert_soundness(result, _expected("mixed_c_rust")) + + @pytest.mark.skipif(not shutil.which("cargo"), reason="cargo not installed") def test_rust_main_entry(analyzer, tmp_path): # ziggy/afl harness shape: a Rust bin rooted at `main`, resolved flexibly diff --git a/scripts/setup.sh b/scripts/setup.sh index 2cdd696..f51bf46 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -9,8 +9,32 @@ ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" LLVM_MAJOR="${LLVM_MAJOR:-$(bash "$ROOT/scripts/select_llvm.sh")}" LLVM_CONFIG="llvm-config-${LLVM_MAJOR}" -# 1. gllvm (C/C++ whole-program bitcode extraction) via Go. -if ! command -v gclang >/dev/null 2>&1; then +# 1. The bitcode capture `reachability run` uses by default: gllvm via Go, or +# with REACHABILITY_CAPTURE=rllvm, rllvm via Homebrew or else cargo. +RLLVM_MIN=0.7.1 +rllvm_ok() { + local have + have="$(rllvm-cc --rllvm-version 2>/dev/null | awk '{print $2}')" || return 1 + [ -n "$have" ] \ + && [ "$(printf '%s\n' "$RLLVM_MIN" "$have" | sort -V | head -n 1)" = "$RLLVM_MIN" ] +} +if [ "${REACHABILITY_CAPTURE:-gllvm}" = rllvm ]; then + if ! rllvm_ok; then + if command -v brew >/dev/null 2>&1; then + echo "[setup] installing rllvm via brew install..." + brew install h1994st/tap/rllvm + elif command -v cargo >/dev/null 2>&1; then + echo "[setup] installing rllvm >= ${RLLVM_MIN} via cargo install..." + cargo install --locked rllvm --version ">=${RLLVM_MIN}" + else + echo "error: Homebrew or cargo is required to install rllvm" >&2 + exit 1 + fi + if ! rllvm_ok; then + echo "warning: the rllvm-cc on PATH is still older than ${RLLVM_MIN}" >&2 + fi + fi +elif ! command -v gclang >/dev/null 2>&1; then if ! command -v go >/dev/null 2>&1; then echo "error: Go is required to install gllvm; install go and re-run setup" >&2 exit 1