Index and hash HIR as part of lowering

[cjgillot]

Part of #88186
Based on #88880 (see merge commit).

Once HIR is lowered, it is later indexed by the index_hir query and hashed for crate_hash. This PR moves those post-processing steps to lowering itself. As a side objective, the HIR crate data structure is refactored as an IndexVec<LocalDefId, Option<OwnerInfo<'hir>>> where OwnerInfo stores all the relevant information for an HIR owner.

r? @michaelwoerister
cc @petrochenkov

[rust-highfive]

Some changes occurred in src/tools/clippy.

cc @rust-lang/clippy

[cjgillot]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit 701098f9476dcab4555517d26abe5d1050c2b899 with merge 5395b5fa9aea18700afc4e5ac3e94a260e124494...

[bors]

☀️ Try build successful - checks-actions
Build commit: 5395b5fa9aea18700afc4e5ac3e94a260e124494 (5395b5fa9aea18700afc4e5ac3e94a260e124494)

[rust-timer]

Queued 5395b5fa9aea18700afc4e5ac3e94a260e124494 with parent 49c0861, future comparison URL.

[rust-timer]

Finished benchmarking commit (5395b5fa9aea18700afc4e5ac3e94a260e124494): comparison url.

Summary: This change led to very large relevant mixed results 🤷 in compiler performance.

• Very large improvement in instruction counts (up to -5.1% on full builds of cranelift-codegen)
• Large regression in instruction counts (up to 3.2% on incr-patched: dummy fn builds of unused-warnings)

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR led to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf +perf-regression

[michaelwoerister]

Thanks, @cjgillot, I've blocked off some time to review this tomorrow.

[michaelwoerister]

This looks pretty promising :)

The first commits are other pending PRs (see merge commits).

Is this still based on pending PRs? If yes, would you mind listing them? If no, please rebase.

[michaelwoerister]

This is not a complete review. Just leaving a few comments.

[michaelwoerister]

Do we have a high-level description of what the lowering process and query setup will look like after rust-lang/compiler-team#452?

[cjgillot]

I added a summary description in #88186 description.

[michaelwoerister]

@bors r+

[bors]

📌 Commit 1e2dbb5 has been approved by michaelwoerister

[bors]

⌛ Testing commit 1e2dbb5 with merge bd41e09...

[bors]

☀️ Test successful - checks-actions
Approved by: michaelwoerister
Pushing bd41e09 to master...

[rust-timer]

Finished benchmarking commit (bd41e09): comparison url.

Summary: This change led to very large relevant mixed results 🤷 in compiler performance.

• Very large improvement in instruction counts (up to -5.4% on full builds of cranelift-codegen)
• Very large regression in instruction counts (up to 5.3% on full builds of unused-warnings)

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression

[rylev]

@cjgillot @michaelwoerister this indeed looks like a pretty significant regression. The unused-warnings benchmark is showing large regressions in hir_lowering. I noticed that the perf run done before merging shows similar results. Was there a reason to merge despite the poor perf results?

[michaelwoerister]

My reading of the results was: this regresses two small, synthetic benchmarks while improving performance for many real-world benchmarks. That seemed like a good tradeoff.

[rylev]

@michaelwoerister that's certainly reasonable, but two thoughts to add there:

• We'd like to make reasoning like that explicit, so it would be helpful next time to add your reasoning in a comment along with @rustbot label +perf-regression-triaged so that we can track that this perf regression is acceptable.
• While it certainly makes sense to provide more weight to real-world crates than to "synthetic" stress test crates, the regressions are very large (at times 60 times the significance threshold). It seems unwise to me to shrug off regressions of that magnitude even if they aren't in real-world crates.
• While this change may not impact a real-world crate that we use in performance testing, it is highly unlikely that the regression of the magnitude we see in unused-warnings or externs won't negatively impact some real world crates.

In my opinion, we should look into this regression or at the very least have a larger discussion about how to interpret these types of regressions. In the future, the plan is to make regressions such as this one block bors from merging without an explicit opting out.

[the8472]

This PR also regressed max-rss across several benchmarks. It's not noise, as can be seen in the stm32f4 timelines which are low-noise for max-rss benchmarks.