Fix macro matching for NoDelim delimited sequences.

[nnethercote]

PR #95159 unintentionally changed the behaviour of declarative macro
matching for NoDelim delimited sequences.

• Before Introduce TtParser #95159, delimiters were implicit in mbe::Delimited. When
  doing macro matching, normal delimiters were generated out of thin air
  as necessary, but NoDelim delimiters were not. This was done within
  TokenTree::{get_tt,len}.
• After Introduce TtParser #95159, the delimiters were explicit. There was an unintentional
  change whereby NoDelim delimiters were represented explicitly just
  like normal delimeters.
• A new matcher representation for use in parse_tt #95555 introduced a new matcher representation (MatcherLoc) and the
  NoDelim delimiters were made explicit within it, just like
  mbe::Delimited.
• Remove explicit delimiter token trees from Delimited. #95797 then changed mbe::Delimited back to having implicit
  delimiters, but because matching is now being done on MatcherLoc,
  the behavioural change persisted.

The fix is simple: remove the explicit NoDelim delimiters in the
MatcherLoc representation. This gives back the original behaviour.

As for why this took so long to be found: it seems that NoDelim
sequences are unusual. It took a macro that generates another macro
(from the yarte_lexer crate, found via a crater run) to uncover this.

Fixes #96305.

r? @petrochenkov

[nnethercote]

The only problem here is there is no test case added, because the only test case I have spans two crates, and I don't know how to incorporate that into rustc's testing structure.

[Aaron1011]

You can use an // aux-build comment to make compile test build another crate.

[petrochenkov]

This is unfortunate case, I'd say that #95159 fixed a bug.
If #96305 is the sole regression found by crater (or there are few similar regressions), then it would be better to keep things as is.
NoDelims are regular brackets like () or {}, just invisible, so they are supposed to match or not match in the same way.

[petrochenkov]

In general, nonterminal tokens from macro_rules are not supposed to be usable in left-hand sides of nested macro_rules (with exception of single-token nonterminals).
#49326 introduced this restriction (for slightly different reasons), but not in a sufficiently reliable way, unfortunately - after being converted to token streams for metadata encoding-decoding they become usable again.

It would be better to report this case early, at macro definition site, when parsing the LHS of the macro definition.

[petrochenkov]

An alternative fix for the regression from #96305 is to stop wrapping single-token literal matchers into groups, like it was done in #92472 with ident matchers.

[petrochenkov]

Given that the macro in #96305 correctly reports an error if used locally, I's say that

It would be better to report this case early, at macro definition site, when parsing the LHS of the macro definition.

is the right course of action here.

See Nonterminal/NoDelim when parsing a macro LHS => report an error (or a deny-by-default deprecation lint if there are too many regressions).

[nnethercote]

Given that the macro in #96305 correctly reports an error if used locally, I's say that

It would be better to report this case early, at macro definition site, when parsing the LHS of the macro definition.

is the right course of action here.

See Nonterminal/NoDelim when parsing a macro LHS => report an error (or a deny-by-default deprecation lint if there are too many regressions).

Fair enough. I've added this information to #96305, and I will close this PR.