Skip to content

Commit 8a67fab

Browse files
redsun82Copilot
redsun82
and
Copilot
committed
Merge origin/main into Actions owner distrust branch
This picks up the current inline expectations test framework, which now expects each implementation to provide relative URLs for result locations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2 parents 14f38e6 + 050bd41 commit 8a67fab

2,249 files changed

Lines changed: 87653 additions & 42955 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/skills/bazel/SKILL.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
---
2+
name: bazel
3+
description: Conventions for editing Bazel files in the github/codeql repository: the shared `misc/bazel` helpers, `codeql_platform_select` and the `{CODEQL_PLATFORM}` packaging placeholder, adding `MODULE.bazel` dependencies, and the `semmle_code` stub that keeps the standalone build working. Use when editing any `BUILD.bazel`, `*.bzl`, `MODULE.bazel` or `.bazelrc` here, and before validating such an edit.
4+
---
5+
6+
# Bazel in the codeql repository
7+
8+
A bzlmod module named `ql`, repo name `@codeql` ([`MODULE.bazel`](../../../MODULE.bazel)). It builds standalone, and is
9+
also consumed by an internal module; standalone builds replace that module with a stub.
10+
11+
## Traps
12+
13+
Things that will waste your time or produce a wrong edit here. Read this section even if you skip the rest.
14+
15+
* **`//...` does not work.** `bazel build //...`, and even `bazel query //...`, fail at the repo root: the patched
16+
modules under [`misc/bazel/registry`](../../../misc/bazel/registry) are real packages referencing repos that are not
17+
visible from the main repo, and [`.bazelrc`](../../../.bazelrc) notes separately that transitions break `...` builds.
18+
The error names a registry directory unrelated to your edit, so it is easy to misdiagnose. **Validate the specific
19+
target or package you changed**, not a recursive pattern.
20+
* **There is no `MODULE.bazel.lock`, deliberately.** [`.bazelrc`](../../../.bazelrc) sets `--lockfile_mode=off` because
21+
the workspace-relative module override makes a lockfile unstable. Do not add one, and do not "fix" its absence.
22+
* **`linux_arm64` vs `linux-arm64`.** The keyword argument and config setting use an underscore; the platform *string*
23+
substituted into paths and zip names uses a hyphen. They are not interchangeable.
24+
* **Do not stub your way out of a missing internal dependency.** See [Building standalone](#building-standalone) for the
25+
one narrow case where extending the stub is correct.
26+
27+
## Conventions
28+
29+
* **Copy a neighboring target rather than inventing a shape.** Packaging is not uniform: some packages use the
30+
`codeql_*` wrappers, others still use `pkg_files` directly. Copy the closest *working* neighbor, and prefer the
31+
wrapper for new code.
32+
* **Pin anything fetched over the network** with `sha256` or `integrity`. Bazel only *warns* on an unpinned download, so
33+
nothing fails loudly, but the build stops being reproducible and a retagged upstream release silently changes what you
34+
build. `lfs_archive` is the exception: content is pinned by git object.
35+
* **Format with `bazel run //misc/bazel/buildifier`.** It rewrites in place, so do not hand-tune formatting. Also wired
36+
as a `pre-commit` hook ([`.pre-commit-config.yaml`](../../../.pre-commit-config.yaml)).
37+
38+
## Where new code goes
39+
40+
Bazel's own macro / rule / repository-rule distinction applies as usual. What is repo-specific:
41+
42+
| Adding | Goes in |
43+
| --- | --- |
44+
| a new packaging shape | extend [`misc/bazel/pkg.bzl`](../../../misc/bazel/pkg.bzl), do not fork `pkg_files` |
45+
| a new OS or arch split | [`misc/bazel/os.bzl`](../../../misc/bazel/os.bzl), do not hand-roll a `select()` over `@platforms//` |
46+
| a fetch of something external | a repository rule ([`lfs.bzl`](../../../misc/bazel/lfs.bzl), [`ripunzip.bzl`](../../../misc/ripunzip/ripunzip.bzl)), not a `genrule` |
47+
| a wrapper used by one language | next to that language ([`swift/rules.bzl`](../../../swift/rules.bzl)) |
48+
| a wrapper used across languages | `misc/bazel/` |
49+
50+
Prefer inline rules in `BUILD.bazel`. A `.bzl` file earns its `load()` only when the shape repeats across packages or a
51+
value must be computed: [`rust.bzl`](../../../misc/bazel/rust.bzl) is worth it because every Rust binary that ships in a
52+
pack must get the same universal-binary wrapper and symbols test, and forgetting either is a release bug. A local
53+
debugging aid opts out and declares a plain `rust_binary`; see `swift-syntax-parse` in
54+
[`unified/swift-syntax-rs/BUILD.bazel`](../../../unified/swift-syntax-rs/BUILD.bazel). The `_gen_binaries` list in
55+
[`go/BUILD.bazel`](../../../go/BUILD.bazel) does not earn a `.bzl`, because it is shared within a single file, where a
56+
local variable does the job.
57+
58+
Macros here are typically a thin public wrapper around a private rule (`codeql_csharp_binary`, `swift_cc_binary`). Keep
59+
the rule narrow and the ergonomics in the macro. Each macro decorates the caller's `name` to mint its helper targets,
60+
for example `internal/<name>` or `single_arch/<name>`. Their visibility is that macro's choice (private, package
61+
default, or the caller's own), so read the macro instead of assuming. When an error names a target you cannot find in
62+
any source file, a macro minted it: grep the suffix under `misc/bazel/`.
63+
64+
## Shared helpers
65+
66+
Frequently-used pieces, so you load the existing one instead of rewriting it. Read the file for its actual exports.
67+
68+
| `load()` path | Covers |
69+
| --- | --- |
70+
| `//misc/bazel:pkg.bzl` | CodeQL packs and packaging |
71+
| `//misc/bazel:os.bzl` | platform and architecture selection |
72+
| `//misc/bazel:lfs.bzl` | on-demand git-LFS repositories |
73+
| `//misc/bazel:rust.bzl` | Rust binary wrapper |
74+
| `//misc/bazel:csharp.bzl` | C# binary/library/test wrappers |
75+
| `//misc/bazel:utils.bzl` | `select_os`; prefer `os.bzl`'s `os_select` in new code |
76+
77+
[`defs.bzl`](../../../defs.bzl) at the root exports `codeql_platform` for *dependent* modules. It is not the way to get
78+
the platform string here; use `os.bzl`.
79+
80+
## Platform selection
81+
82+
[`codeql_platform_select`](../../../misc/bazel/os.bzl) takes one keyword argument per CodeQL platform: `linux64`,
83+
`linux_arm64`, `osx64` and `win64`. `otherwise` supplies the value for whichever of those you leave unset; it is **not**
84+
a `//conditions:default`. **There is deliberately no fallback from `linux_arm64` to `linux64`.** If you only care about
85+
the OS, use `os_select`, which gives Linux the same value on both architectures and has a `posix` shorthand for the
86+
shared Linux/macOS value.
87+
88+
In a macro (no `ctx`) it returns a `select()`:
89+
90+
```python
91+
load("//misc/bazel:os.bzl", "codeql_platform_select")
92+
load("//misc/bazel:pkg.bzl", "codeql_pkg_files")
93+
94+
codeql_pkg_files(
95+
name = "extractor-arch",
96+
exes = codeql_platform_select(
97+
otherwise = ["//unified/extractor"],
98+
win64 = ["//unified/extractor-unsupported-os:extractor"],
99+
),
100+
prefix = "tools/{CODEQL_PLATFORM}",
101+
)
102+
```
103+
104+
If implementation code needs to *branch* on the value rather than pass it through, pass `ctx` and add
105+
`OS_DETECTION_ATTRS` to the rule's attributes. The value is then resolved eagerly instead of being an opaque `select()`:
106+
107+
```python
108+
load("//misc/bazel:os.bzl", "OS_DETECTION_ATTRS", "os_select")
109+
110+
def _impl(ctx):
111+
ext = os_select(ctx, windows = ".exe", posix = "")
112+
...
113+
114+
my_rule = rule(
115+
implementation = _impl,
116+
attrs = {"src": attr.label()} | OS_DETECTION_ATTRS,
117+
)
118+
```
119+
120+
## Packs
121+
122+
`codeql_pack` assembles the files that become an extractor pack. See [`unified/BUILD.bazel`](../../../unified/BUILD.bazel)
123+
for a minimal complete example and [`pkg.bzl`](../../../misc/bazel/pkg.bzl) for the arguments. The non-obvious parts:
124+
125+
* **`{CODEQL_PLATFORM}` in a destination path is the routing mechanism**, not just a substitution. A path containing it
126+
is *arch-specific* and lands in the per-architecture zip; every other path is *common*. So `prefix =
127+
"tools/{CODEQL_PLATFORM}"` both places the file and marks it arch-specific. `arch_overrides` forces named
128+
destinations into the arch-specific part without a placeholder.
129+
* **`codeql_pkg_files` splits `srcs` (plain) from `exes` (mode 755)** and **rejects `attributes =`** with an explicit
130+
error. Use `exes` rather than hand-rolling `pkg_attributes(mode = "755")`.
131+
* **`pkg_dirs` and `pkg_symlinks` are unsupported** and fail at analysis time.
132+
* `codeql_pack` also generates an installer and an `install` alias, hence `bazel run //unified:install`. Pass
133+
`installer_alias = None` if one package defines several packs.
134+
* `codeql_pack_group` exists for bundling packs into distribution zips, but nothing in this repo instantiates it.
135+
136+
## Adding a dependency
137+
138+
In order of preference:
139+
140+
1. **A [Bazel Central Registry](https://registry.bazel.build/) module.** Add a `bazel_dep` in
141+
[`MODULE.bazel`](../../../MODULE.bazel).
142+
2. **A patched upstream module.** Add it under [`misc/bazel/registry`](../../../misc/bazel/registry), which `.bazelrc`
143+
puts ahead of the BCR. Put patches in `modules/<repo>/<version>/patches`, rename the version with a `-codeql.N`
144+
suffix, and run [`fix.py`](../../../misc/bazel/registry/fix.py) to realign the metadata.
145+
3. **A raw archive.** Use `http_archive` via `use_repo_rule`, or a repository rule. Copy an adjacent declaration and
146+
keep its checksum field populated.
147+
148+
Vendored Rust crates under [`misc/bazel/3rdparty`](../../../misc/bazel/3rdparty) are generated. Regenerate with
149+
[`update_cargo_deps.sh`](../../../misc/bazel/3rdparty/update_cargo_deps.sh) rather than editing, and keep the
150+
`use_repo` lists in sync, which `bazel mod tidy` does for module extensions.
151+
152+
## Building standalone
153+
154+
[`MODULE.bazel`](../../../MODULE.bazel) declares `semmle_code` with a `local_path_override` pointing at `..`, which
155+
resolves when this repo is checked out inside the internal module. [`.bazelrc`](../../../.bazelrc), which Bazel reads
156+
when invoked in *this* workspace, overrides that with a stub. This line is the whole reason a standalone build resolves:
157+
158+
```
159+
common --override_module=semmle_code=%workspace%/misc/bazel/semmle_code_stub
160+
```
161+
162+
Do not change either the override path or the `local_path_override`; they work as a pair.
163+
164+
[`misc/bazel/semmle_code_stub`](../../../misc/bazel/semmle_code_stub) is an otherwise empty module supplying no-op
165+
versions of the internal helpers that shared `.bzl` files load *unconditionally*. That is its only job, and it is small
166+
enough to read.
167+
168+
**Extend the stub only when a `.bzl` file every standalone target loads gains a new internal `load()`**, which breaks
169+
package loading outright, for everyone. Prefer not needing one. **Never add a stub so that an internal-only target
170+
appears to build**: some targets depend on internal libraries (`grep -rl @semmle_code --include=*.bazel` finds them) and
171+
are correctly unbuildable here. Unlike a `load()`, such a dependency only fails when that target is actually requested.
172+
173+
[`.bazelrc.internal`](../../../.bazelrc.internal) is **not** read here; it carries settings for the internal build. A
174+
setting needed by both has to be written in both files, with paths differing because this repo sits at a different depth
175+
there.

‎.github/workflows/label-external-contributions.yml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
name: Label external contributions
2+
3+
on:
4+
schedule:
5+
- cron: "7,22,37,52 * * * *"
6+
workflow_dispatch:
7+
8+
permissions: {}
9+
10+
concurrency:
11+
group: label-external-contributions
12+
cancel-in-progress: false
13+
14+
jobs:
15+
label:
16+
if: github.ref_name == github.event.repository.default_branch
17+
runs-on: ubuntu-latest
18+
timeout-minutes: 10
19+
permissions:
20+
pull-requests: write
21+
22+
steps:
23+
- name: Label external contributions
24+
env:
25+
GH_TOKEN: ${{ github.token }}
26+
REPO: ${{ github.repository }}
27+
run: |
28+
set -euo pipefail
29+
30+
label="external-contribution"
31+
updated_cutoff=$(date -u -d "1 hour ago" "+%Y-%m-%dT%H:%M:%SZ")
32+
33+
while IFS= read -r pr_number; do
34+
if [[ ! "$pr_number" =~ ^[1-9][0-9]*$ ]]; then
35+
echo "Skipping malformed pull request number."
36+
continue
37+
fi
38+
39+
pr_json=$(gh api "repos/$REPO/pulls/$pr_number")
40+
if ! jq -e \
41+
--arg repo "$REPO" \
42+
--arg label "$label" \
43+
'.state == "open" and
44+
.draft == false and
45+
.base.repo.full_name == $repo and
46+
(.head.repo.full_name | type == "string") and
47+
.head.repo.full_name != $repo and
48+
.user.type == "User" and
49+
.author_association != "MEMBER" and
50+
.author_association != "OWNER" and
51+
(any(.labels[]?; .name == $label) | not)' \
52+
>/dev/null <<<"$pr_json"; then
53+
continue
54+
fi
55+
56+
events=$(gh api --paginate \
57+
"repos/$REPO/issues/$pr_number/events?per_page=100" |
58+
jq -cs 'add')
59+
60+
if jq -e --arg label "$label" \
61+
'any(.[]; .event == "labeled" and .label.name == $label)' \
62+
>/dev/null <<<"$events"; then
63+
continue
64+
fi
65+
66+
jq -n --arg label "$label" '{labels: [$label]}' |
67+
gh api --method POST \
68+
"repos/$REPO/issues/$pr_number/labels" \
69+
--input - \
70+
>/dev/null
71+
echo "Labelled pull request #$pr_number."
72+
done < <(
73+
gh api --method GET --paginate "repos/$REPO/issues" \
74+
-f state=open \
75+
-f since="$updated_cutoff" \
76+
-f sort=updated \
77+
-f direction=desc \
78+
-f per_page=100 |
79+
jq -r '.[] | select(.pull_request != null) | .number'
80+
)

‎.pre-commit-config.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,15 @@ default_language_version:
44
python: python3.12
55
repos:
66
- repo: https://github.com/pre-commit/pre-commit-hooks
7-
rev: v3.2.0
7+
rev: v6.0.0
88
hooks:
99
- id: trailing-whitespace
1010
exclude: /test([^/]*)/.*$(?<!\.qlref)|.*\.patch$|.*\.qll?$
1111
- id: end-of-file-fixer
1212
exclude: Cargo.lock$|/test([^/]*)/.*$(?<!\.qlref)|.*\.patch$|.*\.qll?$
1313

1414
- repo: https://github.com/pre-commit/mirrors-clang-format
15-
rev: v17.0.6
15+
rev: v20.1.8
1616
hooks:
1717
- id: clang-format
1818

‎BUILD.bazel‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,11 @@ exports_files([
33
"Cargo.lock",
44
"Cargo.toml",
55
])
6+
7+
constraint_setting(name = "swift_runtime_linkage")
8+
9+
constraint_value(
10+
name = "static_swift_runtime",
11+
constraint_setting = ":swift_runtime_linkage",
12+
visibility = ["//visibility:public"],
13+
)

0 commit comments

Comments
 (0)