Store NtLiteral without generalizing to Expr

[dtolnay]

The discussion under #91166 raised the question of whether a NtLiteral (the value of a $:literal matcher in macro_rules) can be conceptualized as a single token, the way that $:ident is. Today it isn't — a negative literal is passed to procedural macros as a pair of proc macro tokens:

// main.rs

macro_rules! fwd {
    ($literal:literal) => {
        repro!($literal);
    };
}

fwd!(-1);

// lib.rs

#[proc_macro]
pub fn repro(input: TokenStream) -> TokenStream {
    println!("{:#?}", input);
    TokenStream::new()
}

TokenStream [
    Group {
        delimiter: None,
        stream: TokenStream [
            Punct {
                ch: '-',
                spacing: Alone,
            },
            Literal {
                kind: Integer,
                symbol: "1",
                suffix: None,
            },
        ],
    },
]

However proc macro literals already support negative literals in a single token, and it's easy to construct one: proc_macro::Literal::i32_unsuffixed(-1). As such, given that macro_rules's $:literal and proc_macro::Literal are both already capable of representing negative literals in one piece, I think having those literals fall apart on entry into the proc macro is not a great behavior. My expectation would be that the proc macro call in the snippet above would pass a single token which is equivalent to proc_macro::Literal::i32_unsuffixed(-1), and I'd like to start making changes in that direction.

This PR does not make any intentional observable behavior change, but it replaces the way that $:literal is represented internally from P<Expr> (which happens to be always either ExprKind::Lit or ExprKind::Unary(UnOp::Neg, ExprKind::Lit) if rustc is correctly implemented) to a dedicated SignedLiteral type. This makes illegal states unrepresentable and will make it more straightforward to map $:literal one-to-one to proc_macro::Literal whenever that comes up.

[rust-highfive]

r? @davidtwco

(rust-highfive has picked a reviewer for you, use r? to override)

[petrochenkov]

r? @petrochenkov

[petrochenkov]

I want to check some things before proceeding with changes in this area, cc #92472.

[petrochenkov]

Some random thoughts:

• I think it's just fundamentally wrong to represent fragments like

  - /* any whitespace tokens including comments possibly with
        newlines */ 1

  as a single token, and the literal matcher accepts such fragments.
• Literal::i32_unsuffixed and friends were a mistake, IMO. During the major pre-stabilization proc macro API audit (Review proc macro API 1.2 #50473) I remember having something against them, but I don't remember why exactly I didn't actively tried to block their stabilization in their current - accepting form, maybe I thought it was too breaking. The rustc parser used separate tokens for - and 1 at that point, and those methods deliberately introduced a mismatch between internal tokens (also used for macro_rules) and proc macro tokens in the direction of less fine grained proc macro tokens, while the rest of the API made tokens more fine grained compared to the internal version, if there was a mismatch at all.
• I don't want the internal tokens to have both unglued literal forms (coming from the lexer), and glued forms (coming from Literal::i32_unsuffixed and friends), normalizing at proc macro boundary seems preferable.
• The plan in (Accept tuple.0.0 as tuple indexing (take 2) #71322) assumed that the internal literal tokens will become more fine-grained, with floating point literals being broken into multiple tokens. That again would make the idea of literal matcher always being a single token incorrect even if we are gluing minuses into the literal tokens now.
• I don't really like this PR as a refactoring, yes it makes impossible state unrepresentable, but it makes the logic of working with literal fragments more complex and ugly either.

[dtolnay]

Thanks for the review! I can tell I'm not going to change your mind on this so I'll go ahead and close.