HirIdification: rework Map by ljedrz · Pull Request #59042 · rust-lang/rust

ljedrz · 2019-03-09T08:42:15Z

The next iteration of HirIdification (#57578).

remove NodeId from Entry
change Map::map to an FxHashMap<HirId, Entry>
base the NodeId Map methods on HirId ones (reverses the current state)
HirIdify librustdoc a little bit (some NodeId Map methods were converted to work on HirIds)

The second change might have performance implications, so I'd do a perf run to be sure it's fine; it simplifies the codebase and shouldn't have an impact as long as the Map searches are cached (which is now possible thanks to using HirIds).

r? @Zoxc

ljedrz · 2019-03-09T08:42:37Z

@bors try (for perf)

bors · 2019-03-09T08:43:00Z

⌛ Trying commit 12c55fd with merge 920010c...

@Zoxc

HirIdification: rework Map The next iteration of HirIdification (#57578). - remove `NodeId` from `Entry` - change `Map::map` to an `FxHashMap<HirId, Entry>` - base the `NodeId` `Map` methods on `HirId` ones (reverses the current state) - HirIdify `librustdoc` a little bit (some `NodeId` `Map` methods were converted to work on `HirId`s) The second change might have performance implications, so I'd do a perf run to be sure it's fine; it simplifies the codebase and shouldn't have an impact as long as the `Map` searches are cached (which is now possible thanks to using `HirId`s). r? @Zoxc

rust-highfive · 2019-03-09T09:45:00Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:03f1a5dc:start=1552121008554060787,finish=1552121082514661400,duration=73960600613
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
Setting environment variables from .travis.yml
---
[00:59:57]    Compiling rustdoc v0.0.0 (/checkout/src/librustdoc)
[01:00:03] error[E0308]: mismatched types
[01:00:03]   --> src/librustdoc/passes/collect_intra_doc_links.rs:82:49
[01:00:03]    |
[01:00:03] 82 |                                     .with_scope(id,
[01:00:03]    |                                                 ^^ expected struct `syntax::ast::NodeId`, found struct `rustc::hir::HirId`
[01:00:03]    = note: expected type `syntax::ast::NodeId`
[01:00:03]               found type `rustc::hir::HirId`
[01:00:03] 
[01:00:03] error[E0308]: mismatched types
[01:00:03] error[E0308]: mismatched types
[01:00:03]    --> src/librustdoc/passes/collect_intra_doc_links.rs:146:45
[01:00:03]     |
[01:00:03] 146 |                                 .with_scope(id,
[01:00:03]     |                                             ^^ expected struct `syntax::ast::NodeId`, found struct `rustc::hir::HirId`
[01:00:03]     = note: expected type `syntax::ast::NodeId`
[01:00:03]                found type `rustc::hir::HirId`
[01:00:03] 
[01:00:04] error[E0308]: mismatched types
[01:00:04] error[E0308]: mismatched types
[01:00:04]    --> src/librustdoc/visit_ast.rs:275:39
[01:00:04]     |
[01:00:04] 275 |                 if cx.tcx.hir().attrs(node).lists("doc").has_word("hidden") {
[01:00:04]     |                                       ^^^^ expected struct `syntax::ast::NodeId`, found struct `rustc::hir::HirId`
[01:00:04]     = note: expected type `syntax::ast::NodeId`
[01:00:04]                found type `rustc::hir::HirId`
[01:00:04] 
[01:00:04] error: aborting due to 3 previous errors
---
travis_time:end:097e7ef8:start=1552124696918302332,finish=1552124696923750692,duration=5448360
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:2aca16b6
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1568c58c
travis_time:start:1568c58c
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2019-03-09T10:07:23Z

💔 Test failed - checks-travis

rust-highfive · 2019-03-09T10:07:24Z

The job dist-x86_64-linux of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:15d26819
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
[01:16:00]    Compiling rustdoc v0.0.0 (/checkout/src/librustdoc)
[01:16:05] error[E0308]: mismatched types
[01:16:05]   --> src/librustdoc/passes/collect_intra_doc_links.rs:82:49
[01:16:05]    |
[01:16:05] 82 |                                     .with_scope(id,
[01:16:05]    |                                                 ^^ expected struct `syntax::ast::NodeId`, found struct `rustc::hir::HirId`
[01:16:05]    = note: expected type `syntax::ast::NodeId`
[01:16:05]               found type `rustc::hir::HirId`
[01:16:05] 
[01:16:05] error[E0308]: mismatched types
[01:16:05] error[E0308]: mismatched types
[01:16:05]    --> src/librustdoc/passes/collect_intra_doc_links.rs:146:45
[01:16:05]     |
[01:16:05] 146 |                                 .with_scope(id,
[01:16:05]     |                                             ^^ expected struct `syntax::ast::NodeId`, found struct `rustc::hir::HirId`
[01:16:05]     = note: expected type `syntax::ast::NodeId`
[01:16:05]                found type `rustc::hir::HirId`
[01:16:05] 
[01:16:06] error[E0308]: mismatched types
[01:16:06] error[E0308]: mismatched types
[01:16:06]    --> src/librustdoc/visit_ast.rs:275:39
[01:16:06]     |
[01:16:06] 275 |                 if cx.tcx.hir().attrs(node).lists("doc").has_word("hidden") {
[01:16:06]     |                                       ^^^^ expected struct `syntax::ast::NodeId`, found struct `rustc::hir::HirId`
[01:16:06]     = note: expected type `syntax::ast::NodeId`
[01:16:06]                found type `rustc::hir::HirId`
[01:16:06] 
[01:16:06] error: aborting due to 3 previous errors
---
travis_time:end:29603c5a:start=1552125624405458878,finish=1552125624414692049,duration=9233171
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:060a2f09
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0e52fe18
travis_time:start:0e52fe18
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:057ad52b
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

ljedrz · 2019-03-09T10:24:09Z

Oof

@bors try (for perf, for real this time)

bors · 2019-03-09T10:24:21Z

⌛ Trying commit 16b3081 with merge 3c13d8d...

@Zoxc

HirIdification: rework Map The next iteration of HirIdification (#57578). - remove `NodeId` from `Entry` - change `Map::map` to an `FxHashMap<HirId, Entry>` - base the `NodeId` `Map` methods on `HirId` ones (reverses the current state) - HirIdify `librustdoc` a little bit (some `NodeId` `Map` methods were converted to work on `HirId`s) The second change might have performance implications, so I'd do a perf run to be sure it's fine; it simplifies the codebase and shouldn't have an impact as long as the `Map` searches are cached (which is now possible thanks to using `HirId`s). r? @Zoxc

rust-highfive · 2019-03-09T12:18:01Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:111a9abc:start=1552127067054046443,finish=1552127142662952951,duration=75608906508
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
Setting environment variables from .travis.yml
---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:27:42] 
[01:27:42] running 119 tests
[01:28:10] .iiiii...i.....i..i...i..i.i..i.ii...i.....i..i....i..........iiii..........i...ii...i.......ii.i.i. 100/119
[01:28:14] i......iii.i.....ii
[01:28:14] 
[01:28:14]  finished in 32.648
[01:28:14] travis_fold:end:test_debuginfo

---
[01:52:03]     Finished release [optimized] target(s) in 47.88s
[01:52:03]      Running build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_driver-72f48af8359eb542
[01:52:03] 
[01:52:03] running 9 tests
[01:52:03] ...F.....
[01:52:03] 
[01:52:03] ---- test::contravariant_region_ptr_err stdout ----
[01:52:03] ---- test::contravariant_region_ptr_err stdout ----
[01:52:03] thread 'test::contravariant_region_ptr_err' panicked at 'index out of bounds: the len is 1 but the index is 4294967040', /checkout/src/libcore/slice/mod.rs:2539:10
[01:52:03] 
[01:52:03] 
[01:52:03] failures:
[01:52:03]     test::contravariant_region_ptr_err
[01:52:03]     test::contravariant_region_ptr_err
[01:52:03] 
[01:52:03] test result: FAILED. 8 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:52:03] 
[01:52:03] error: test failed, to rerun pass '--lib'
[01:52:03] 
[01:52:03] 
[01:52:03] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" "" "--manifest-path" "/checkout/src/rustc/Cargo.toml" "-p" "rustc_driver" "--" "--quiet"
[01:52:03] 
[01:52:03] 
[01:52:03] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:52:03] Build completed unsuccessfully in 0:37:41
[01:52:03] Build completed unsuccessfully in 0:37:41
[01:52:03] make: *** [check] Error 1
[01:52:03] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0bfdefd4
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sat Mar  9 12:17:55 UTC 2019
---
travis_time:end:06567592:start=1552133877784669710,finish=1552133877790069673,duration=5399963
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:020e05ea
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2019-03-09T12:38:28Z

☀️ Try build successful - checks-travis
Build commit: 3c13d8d

ljedrz · 2019-03-09T12:41:38Z

Can I get a perf run pls?

Xanewok · 2019-03-09T13:04:50Z

Assuming I remember the invocation...

@rust-timer build 3c13d8d

Mark-Simulacrum · 2019-03-09T13:10:43Z

@rust-timer build 3c13d8d

rust-timer · 2019-03-09T13:10:45Z

Success: Queued 3c13d8d with parent c9f8304, comparison URL.

Xanewok · 2019-03-09T13:12:33Z

@Mark-Simulacrum huh so only bare command works in the comment body?

Mark-Simulacrum · 2019-03-09T13:15:27Z

No, it should've responded to you somehow. I need to investigate why it didn't.

rust-timer · 2019-03-09T16:52:46Z

Finished benchmarking try commit 3c13d8d

ljedrz · 2019-03-09T16:57:11Z

Hmm, looks like there are some instructions and max-rss losses.

@Zoxc do you think we should also give it a shot together with #57173 or that the map should remain a Vec? I'm not sure how dense HirIds are; if they are not dense enough, perf results would probably be similar. Actually I'm not even sure if it's possible to index a Vec with HirIds; we could always perform a HirId to NodeId conversion for that, but I'm not sure if that's the best course of action - we would want to deprecate those as much as possible after all.

Zoxc · 2019-03-09T19:48:20Z

I would wait until #58623 lands before trying to replace the existing node map. Until then I'd just focus on getting rid of NodeId uses outside the HIR map.

ljedrz · 2019-03-09T19:51:35Z

Ok; can you mark this PR as blocked so it doesn't pop up during PR queue cleanups?

Amanieu · 2019-03-10T00:43:15Z

Is it really necessary to wait for #58623 to land? That PR only touches libstd and doesn't really interact with this PR. The HashMap API is completely unchanged, just the implementation.

bors · 2019-04-24T18:03:28Z

📌 Commit 37954df has been approved by Zoxc

bors · 2019-04-24T18:15:40Z

⌛ Testing commit 37954df with merge 49c193172ef3ae229b5c8149a67ff5e31c0d4672...

ljedrz · 2019-04-24T18:29:27Z

I mean the instruction results did seem to improve in comparison to the previous perf run; did you expect greens in the incremental department?

Zoxc · 2019-04-24T18:36:49Z

@ljedrz wall-time is the metric which counts, which seems pretty much as bad as before.

emilyalbini · 2019-04-24T19:09:18Z

Yielding priority to the stable release.

@bors retry

bors · 2019-04-24T20:47:35Z

⌛ Testing commit 37954df with merge 04480fd5dd4945b4a4bb31e46934bff1739df63b...

emilyalbini · 2019-04-24T21:04:52Z

Yielding priority to the stable release, again.

@bors retry

bors · 2019-04-25T00:53:35Z

⌛ Testing commit 37954df with merge 3d21124...

@Zoxc

HirIdification: rework Map The next iteration of HirIdification (#57578). - remove `NodeId` from `Entry` - change `Map::map` to an `FxHashMap<HirId, Entry>` - base the `NodeId` `Map` methods on `HirId` ones (reverses the current state) - HirIdify `librustdoc` a little bit (some `NodeId` `Map` methods were converted to work on `HirId`s) The second change might have performance implications, so I'd do a perf run to be sure it's fine; it simplifies the codebase and shouldn't have an impact as long as the `Map` searches are cached (which is now possible thanks to using `HirId`s). r? @Zoxc

bors · 2019-04-25T03:44:15Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: Zoxc
Pushing 3d21124 to master...

rust-highfive · 2019-04-25T03:45:52Z

📣 Toolstate changed by #59042!

Tested on commit 3d21124.
Direct link to PR: #59042

💔 clippy-driver on windows: test-pass → build-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
💔 clippy-driver on linux: test-pass → build-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
💔 rls on windows: test-fail → build-fail (cc @Xanewok, @rust-lang/infra).
💔 rls on linux: test-fail → build-fail (cc @Xanewok, @rust-lang/infra).

@Manishearth

Tested on commit rust-lang/rust@3d21124. Direct link to PR: <rust-lang/rust#59042> 💔 clippy-driver on windows: test-pass → build-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra). 💔 clippy-driver on linux: test-pass → build-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra). 💔 rls on windows: test-fail → build-fail (cc @Xanewok, @rust-lang/infra). 💔 rls on linux: test-fail → build-fail (cc @Xanewok, @rust-lang/infra).

Rustup for rust-lang/rust#59042 changelog: none

Changes: ```` Rustup for rust-lang#59042 Update pulldown_cmark to 0.5 Only run AppVeyor on r+, try and the master branch Remove approx_constant known problems Suppress let_and_return if let has attributes Add test for or_fun_call macro suggestion UI test cleanup: Extract needless_range_loop tests Change "if types change" to "if you later change the type" ````

@oli-obk

submodules: update clippy from 9897442 to 8c0e038 Should fix clippy/rls toolstate breakage Changes: ```` Rustup for #59042 Update pulldown_cmark to 0.5 Only run AppVeyor on r+, try and the master branch Remove approx_constant known problems Suppress let_and_return if let has attributes Add test for or_fun_call macro suggestion UI test cleanup: Extract needless_range_loop tests Change "if types change" to "if you later change the type" ```` r? @oli-obk

@oli-obk

submodules: update clippy from 9897442 to 8c0e038 Should fix clippy/rls toolstate breakage Changes: ```` Rustup for #59042 Update pulldown_cmark to 0.5 Only run AppVeyor on r+, try and the master branch Remove approx_constant known problems Suppress let_and_return if let has attributes Add test for or_fun_call macro suggestion UI test cleanup: Extract needless_range_loop tests Change "if types change" to "if you later change the type" ```` r? @oli-obk

@ljedrz

Optimize HIR map Builds on #59042 cc @ljedrz r? @eddyb

Changes: ```` Rustup for rust-lang/rust#59042 Update pulldown_cmark to 0.5 Only run AppVeyor on r+, try and the master branch Remove approx_constant known problems Suppress let_and_return if let has attributes Add test for or_fun_call macro suggestion UI test cleanup: Extract needless_range_loop tests Change "if types change" to "if you later change the type" ````

rust-highfive assigned Zoxc Mar 9, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 9, 2019

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 9, 2019

ljedrz force-pushed the HirIdification_rework_map branch from 12c55fd to 16b3081 Compare March 9, 2019 10:23

ljedrz mentioned this pull request Mar 9, 2019

Remove metadata only codegen backend #58847

Merged

Zoxc added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 9, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Apr 24, 2019

Zoxc mentioned this pull request Apr 24, 2019

Optimize HIR map #60246

Merged

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 25, 2019

bors merged commit 37954df into rust-lang:master Apr 25, 2019

bors mentioned this pull request Apr 25, 2019

Allocate HIR on an arena #57173

Closed

phansch added a commit to phansch/rust-clippy that referenced this pull request Apr 25, 2019

Rustup for rust-lang/rust#59042

1f5a3c6

bors added a commit to rust-lang/rust-clippy that referenced this pull request Apr 25, 2019

Auto merge of #4030 - phansch:rustup190425, r=Manishearth

d84d3f8

Rustup for rust-lang/rust#59042 changelog: none

bors added a commit that referenced this pull request May 8, 2019

Auto merge of #60246 - Zoxc:hir-map-vec, r=eddyb

d7f4a86

Optimize HIR map Builds on #59042 cc @ljedrz r? @eddyb

pnkfelix mentioned this pull request Jun 21, 2019

Panic with Layout mismatch when copying! from macro #61663

Closed

ljedrz deleted the HirIdification_rework_map branch June 24, 2019 19:05

Uh oh!

Conversation

ljedrz commented Mar 9, 2019

Uh oh!

ljedrz commented Mar 9, 2019

Uh oh!

bors commented Mar 9, 2019

Uh oh!

rust-highfive commented Mar 9, 2019

Uh oh!

bors commented Mar 9, 2019

Uh oh!

rust-highfive commented Mar 9, 2019

Uh oh!

ljedrz commented Mar 9, 2019

Uh oh!

bors commented Mar 9, 2019

Uh oh!

rust-highfive commented Mar 9, 2019

Uh oh!

bors commented Mar 9, 2019

Uh oh!

ljedrz commented Mar 9, 2019

Uh oh!

Xanewok commented Mar 9, 2019

Uh oh!

Mark-Simulacrum commented Mar 9, 2019

Uh oh!

rust-timer commented Mar 9, 2019

Uh oh!

Xanewok commented Mar 9, 2019

Uh oh!

Mark-Simulacrum commented Mar 9, 2019

Uh oh!

rust-timer commented Mar 9, 2019

Uh oh!

ljedrz commented Mar 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zoxc commented Mar 9, 2019

Uh oh!

ljedrz commented Mar 9, 2019

Uh oh!

Amanieu commented Mar 10, 2019

Uh oh!

bors commented Apr 24, 2019

Uh oh!

bors commented Apr 24, 2019

Uh oh!

ljedrz commented Apr 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zoxc commented Apr 24, 2019

Uh oh!

emilyalbini commented Apr 24, 2019

Uh oh!

bors commented Apr 24, 2019

Uh oh!

emilyalbini commented Apr 24, 2019

Uh oh!

bors commented Apr 25, 2019

Uh oh!

bors commented Apr 25, 2019

Uh oh!

rust-highfive commented Apr 25, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

ljedrz commented Mar 9, 2019 •

edited

Loading

ljedrz commented Apr 24, 2019 •

edited

Loading