Build the dep-graph reverse index lazily, per DepKind by xmakro · Pull Request #157319 · rust-lang/rust

xmakro · 2026-06-02T15:42:42Z

Replace the eager per-DepKind fingerprint-to-index map built at decode with a counting sort into per-kind ranges plus a lazily-built map per kind.

cjgillot

I like the idea. Could you fix tidy so we can test and perf it?

View changes since this review

xmakro · 2026-06-03T23:20:56Z

Thanks for the review! I addressed the comments

cjgillot · 2026-06-04T10:37:55Z

@bors try @rust-timer queue

Build the dep-graph reverse index lazily, per DepKind

rust-bors · 2026-06-04T12:47:22Z

☀️ Try build successful (CI)
Build commit: 4754d42 (4754d42b1ab5ad44b221d4c4a7b09bee66237a16, parent: 49b19d32b9f01a5aa606f3bf2e90e6e0aa462c03)

rust-timer · 2026-06-04T13:57:09Z

Finished benchmarking commit (4754d42): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.1%	[0.0%, 0.3%]	7
Improvements ✅ (primary)	-0.4%	[-0.7%, -0.2%]	50
Improvements ✅ (secondary)	-0.7%	[-2.7%, -0.0%]	40
All ❌✅ (primary)	-0.4%	[-0.7%, -0.2%]	50

Max RSS (memory usage)

Results (primary -2.3%, secondary -1.9%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.1%	[0.7%, 1.8%]	3
Regressions ❌ (secondary)	4.0%	[2.1%, 7.7%]	4
Improvements ✅ (primary)	-2.6%	[-5.1%, -0.8%]	37
Improvements ✅ (secondary)	-3.1%	[-4.9%, -1.8%]	20
All ❌✅ (primary)	-2.3%	[-5.1%, 1.8%]	40

Cycles

Results (primary -2.5%, secondary -1.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	5.0%	[3.9%, 5.6%]	3
Improvements ✅ (primary)	-2.5%	[-3.1%, -2.2%]	3
Improvements ✅ (secondary)	-4.3%	[-6.7%, -3.3%]	8
All ❌✅ (primary)	-2.5%	[-3.1%, -2.2%]	3

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 511.14s -> 512.503s (0.27%)
Artifact size: 398.59 MiB -> 400.66 MiB (0.52%)

cjgillot

Perf is great, the implementation looks good. A few nits on comments.
r=me afterwards

View changes since this review

cjgillot · 2026-06-04T15:00:46Z

+        nodes: &IndexSlice<SerializedDepNodeIndex, DepNode>,
+        nodes_by_kind: &[Option<SerializedDepNodeIndex>],
+    ) -> &UnhashMap<PackedFingerprint, SerializedDepNodeIndex> {
+        self.map.get_or_init(|| {


Suggested change

self.map.get_or_init(|| {

self.map.get_or_init(|| {

let _prof_timer = self.profiler.generic_activity("incr_comp_load_dep_graph_reverse_index");

Let's add a timer to for detailed perf output. This will make easier to understand why regression happen in some queries.

I added the timer in fingerprint_map but I put the profiler on SerializedDepGraph rather than on LazyKindIndex, since there's one of those per DepKind and it would be a lot of clones. Annoying is that this needs a manual Debug impl. Please let me know if you see a better way

xmakro · 2026-06-04T16:49:01Z

Thanks for the thorough review! I made a separate commit so it is easier to review. If it looks good, I would squash the commits into one before the merge

cjgillot · 2026-06-05T10:06:25Z

Thanks!
@bors r+

rust-bors · 2026-06-05T10:06:27Z

📌 Commit 62b140a has been approved by cjgillot

It is now in the queue for this repository.

rust-bors · 2026-06-05T11:24:04Z

⚠️ A new commit b62dfa6c25ac1ba30a086f36c464959c5bfe448b was pushed.

This pull request was unapproved.

JonathanBrouwer · 2026-06-10T15:51:59Z

@bors retry
@bors treeclosed=5
GitHub is still super unstable, some builds are succeeding but I'd rather just keep the tree closed while things are like this

…illot Build the dep-graph reverse index lazily, per DepKind Replace the eager per-DepKind fingerprint-to-index map built at decode with a counting sort into per-kind ranges plus a lazily-built map per kind.

jhpratt · 2026-06-10T20:55:18Z

stalled

@bors yield

@bors try jobs=dist-i686-msvc

rust-bors · 2026-06-10T20:55:21Z

Auto build was cancelled. Cancelled workflows:

https://github.com/rust-lang/rust/actions/runs/27292225253

The next pull request likely to be tested is #157319.

Build the dep-graph reverse index lazily, per DepKind try-job: dist-i686-msvc

…illot Build the dep-graph reverse index lazily, per DepKind Replace the eager per-DepKind fingerprint-to-index map built at decode with a counting sort into per-kind ranges plus a lazily-built map per kind.

jhpratt · 2026-06-10T20:55:44Z

@bors p=4

@bors yield

rust-bors · 2026-06-10T20:55:47Z

Auto build was cancelled. Cancelled workflows:

https://github.com/rust-lang/rust/actions/runs/27305756573

The next pull request likely to be tested is #157726.

…illot Build the dep-graph reverse index lazily, per DepKind Replace the eager per-DepKind fingerprint-to-index map built at decode with a counting sort into per-kind ranges plus a lazily-built map per kind.

jhpratt · 2026-06-11T00:53:52Z

stalled yet again, including the try job

@bors yield

@bors try cancel

@bors try jobs=dist-i686-msvc

rust-bors · 2026-06-11T00:53:55Z

Auto build was cancelled. Cancelled workflows:

https://github.com/rust-lang/rust/actions/runs/27306178177

The next pull request likely to be tested is #157726.

rust-bors · 2026-06-11T00:53:56Z

Try build cancelled. Cancelled workflows:

https://github.com/rust-lang/rust/actions/runs/27305752284

Build the dep-graph reverse index lazily, per DepKind try-job: dist-i686-msvc

…illot Build the dep-graph reverse index lazily, per DepKind Replace the eager per-DepKind fingerprint-to-index map built at decode with a counting sort into per-kind ranges plus a lazily-built map per kind.

rust-bors · 2026-06-11T02:38:35Z

💔 Test for a4dd565 failed: CI. Failed jobs:

auto - i686-gnu-1 (web logs, enhanced plaintext logs)
auto - pr-check-1 (web logs, enhanced plaintext logs)

rust-log-analyzer · 2026-06-11T02:49:02Z

A job failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

jhpratt · 2026-06-11T02:51:26Z

@bors retry

rust-bors · 2026-06-11T04:09:01Z

💔 Test for 729af82 failed: CI. Failed job:

try - dist-i686-msvc (web logs, enhanced plaintext logs)

jieyouxu · 2026-06-11T05:40:32Z

@bors try jobs=dist-i686-msvc

rust-bors · 2026-06-11T05:40:38Z

⌛ Trying commit 0f7c98f with merge 853d73e…

To cancel the try build, run the command @bors try cancel.

Workflow: https://github.com/rust-lang/rust/actions/runs/27326398758

Build the dep-graph reverse index lazily, per DepKind try-job: dist-i686-msvc

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jun 2, 2026

This comment has been minimized.

Sign in to view

cjgillot reviewed Jun 3, 2026

View reviewed changes

cjgillot self-assigned this Jun 3, 2026

xmakro force-pushed the perf/lazy-depkind-reverse-index branch from c801cd6 to 20be77e Compare June 3, 2026 21:52

xmakro marked this pull request as ready for review June 3, 2026 23:20

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 3, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jun 4, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 4, 2026

Auto merge of #157319 - xmakro:perf/lazy-depkind-reverse-index, r=<try>

4754d42

Build the dep-graph reverse index lazily, per DepKind

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Jun 4, 2026

cjgillot approved these changes Jun 4, 2026

View reviewed changes

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 5, 2026

xmakro force-pushed the perf/lazy-depkind-reverse-index branch from 62b140a to b62dfa6 Compare June 5, 2026 11:24

rust-bors Bot added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jun 5, 2026

rust-bors Bot removed the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jun 5, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 10, 2026

Auto merge of #157319 - xmakro:perf/lazy-depkind-reverse-index, r=<try>

1162991

Build the dep-graph reverse index lazily, per DepKind try-job: dist-i686-msvc

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 11, 2026

Auto merge of #157319 - xmakro:perf/lazy-depkind-reverse-index, r=<try>

729af82

Build the dep-graph reverse index lazily, per DepKind try-job: dist-i686-msvc

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 11, 2026

Auto merge of #157319 - xmakro:perf/lazy-depkind-reverse-index, r=<try>

853d73e

Build the dep-graph reverse index lazily, per DepKind try-job: dist-i686-msvc

	self.map.get_or_init(\|\| {
	self.map.get_or_init(\|\| {
	let _prof_timer = self.profiler.generic_activity("incr_comp_load_dep_graph_reverse_index");

Uh oh!

Conversation

xmakro commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

cjgillot left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xmakro commented Jun 3, 2026

Uh oh!

cjgillot commented Jun 4, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Jun 4, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Jun 4, 2026

Overall result: ❌✅ regressions and improvements - please read:

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

cjgillot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cjgillot Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

xmakro Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

xmakro commented Jun 4, 2026

Uh oh!

cjgillot commented Jun 5, 2026

Uh oh!

rust-bors Bot commented Jun 5, 2026

Uh oh!

rust-bors Bot commented Jun 5, 2026

Uh oh!

JonathanBrouwer commented Jun 10, 2026

Uh oh!

This comment has been minimized.

jhpratt commented Jun 10, 2026

Uh oh!

rust-bors Bot commented Jun 10, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

jhpratt commented Jun 10, 2026

Uh oh!

rust-bors Bot commented Jun 10, 2026

Uh oh!

This comment has been minimized.

jhpratt commented Jun 11, 2026

Uh oh!

rust-bors Bot commented Jun 11, 2026

Uh oh!

rust-bors Bot commented Jun 11, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

xmakro commented Jun 2, 2026 •

edited

Loading

cjgillot left a comment •

edited by rustbot

Loading

cjgillot left a comment •

edited

Loading

rust-bors Bot commented Jun 11, 2026 •

edited

Loading