Properly capture trailing 'unglued' token

[Aaron1011]

If we try to capture the Vec<u8> in Option<Vec<u8>>, we'll
need to capture a > token which was 'unglued' from a >> token.
The processing of unglueing a token for parsing purposes bypasses the
usual capturing infrastructure, so we currently lose the trailing >.
As a result, we fall back to the reparsed TokenStream, causing us to
lose spans.

This commit makes token capturing keep track of a trailing 'unglued'
token. Note that we don't need to care about unglueing except at the end
of the captured tokens - if we capture both the first and second unglued
tokens, then we'll end up capturing the full 'glued' token, which
already works correctly.

[rust-highfive]

r? @petrochenkov

(rust-highfive has picked a reviewer for you, use r? to override)

[petrochenkov]

What a coincidence, I reviewed #79912 today and started thinking - how does token collection works with cases like tuple.0.0 where we break tokens (a float 0.0 in this case) into parts?
I didn't have a chance to test anything yet though.

Could you actually add tuple.0.0 to test cases in addition to >>?

[petrochenkov]

All this stuff will go away once lexer starts producing "fine-grained" tokens, e.g. two tokens for >> and three tokens for 0.0, and parser learns to work with them.

[Aaron1011]

@petrochenkov: We can't run into this kind of issue with tuple.0.0. This issue occurs when we try to capture part of the TokenStream through a macro_rules! matcher. In the case of tuple.0.0, you would need to capture tuple.0 as an expression to trigger the tuple unglueing behavior - however, macro_rules! explicitly prevent you from doing this by disallowing . (and other tokens) after an :expr matcher).

[petrochenkov]

@bors r+

[bors]

📌 Commit e6fa633 has been approved by petrochenkov

[bors]

⌛ Testing commit e6fa633 with merge 803c602...

[bors]

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing 803c602 to master...