impl FromStr for proc_macro::Literal

[dtolnay]

Note that unlike impl FromStr for proc_macro::TokenStream, this impl does not permit whitespace or comments. The input string must consist of nothing but your literal.

• "1".parse::<Literal>() ⟶ ok

• "1.0".parse::<Literal>() ⟶ ok

• "'a'".parse::<Literal>() ⟶ ok

• "\"\n\"".parse::<Literal>() ⟶ ok

• "0 1".parse::<Literal>() ⟶ LexError

• " 0".parse::<Literal>() ⟶ LexError

• "0 ".parse::<Literal>() ⟶ LexError

• "/* comment */0".parse::<Literal>() ⟶ LexError

• "0/* comment */".parse::<Literal>() ⟶ LexError

• "0// comment".parse::<Literal>() ⟶ LexError

---

Use case

let hex_int: Literal = format!("0x{:x}", int).parse().unwrap();

The only way this is expressible in the current API is significantly worse.

let hex_int = match format!("0x{:x}", int)
    .parse::<TokenStream>()
    .unwrap()
    .into_iter()
    .next()
    .unwrap()
{
    TokenTree::Literal(literal) => literal,
    _ => unreachable!(),
};

[rust-highfive]

r? @petrochenkov

(rust-highfive has picked a reviewer for you, use r? to override)

[petrochenkov]

Aren't APIs like this provided by higher-level crates like proc_macro2?
But looks like this case cannot be done precisely using the existing public API due to possible whitespace and comments around the literal.

The main alternative that I can see here is implementing FromStr for TokenTree instead.
(Which is the intermediate step from more general TokenStream to more specific Literal.)
In that case the literal would be obtainable as

let hex_int = match format!("0x{:x}", int).parse::<TokenTree>().unwrap() {
    TokenTree::Literal(literal) => literal,
    _ => unreachable!(),
};

and impl FromStr for Literal would became purely a convenience function.

[dtolnay]

Aren't APIs like this provided by higher-level crates like proc_macro2?

Nope. The proc-macro2 API is intended to be an exact clone of the proc-macro API and is not meant to expose anything beyond that.

It backports new proc-macro APIs to older versions of Rust where possible, though. Still it's not for something that isn't in proc-macro. New APIs land in proc-macro first (such as this PR) and then go into proc-macro2.

The main alternative that I can see here is implementing FromStr for TokenTree instead.

I prefer not to do this because of the ambiguity around whether whitespace/comments are permitted in the input. If not, Group parsing gets very weird. If yes, there's no longer a way to parse a literal that is only allowed to be a literal, other than writing your own whitespace/comment lexer. And if yes for groups and no for other TokenTree variants, ... doesn't seem great.

Besides, from an API design perspective FromStr for TokenTree does not strike me as useful, whereas FromStr for Literal is directly commonly useful. I feel it would be quite unusual to have input which you know is exactly one "token tree" but do not know which kind of token tree, and would be fine with it being a Group containing multiple tokens, but not fine with it being non-grouped multiple tokens. Also the choice of what Spacing to put on parsed Punct tokens is not good. Presumably it would just unconditionally put an Alone spacing, but it seems bizarre and I don't really know use cases where that would be always appropriate behavior.

I'd like to have FromStr for TokenStream to mean parsing arbitrary tokens/whitespace, and FromStr for Literal that is just effectively equivalent to a checked Symbol::intern(string) i.e. what you give it is the actual representation of the literal and nothing else.

[dtolnay]

@rust-lang/libs:
@rfcbot fcp merge

Previous comment has some discussion of this impl in relation to impl FromStr for TokenStream (which is long stable) and impl FromStr for TokenTree (which I think I would prefer not to have).

[rfcbot]

Team member @dtolnay has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Amanieu
• @BurntSushi
• @dtolnay
• @joshtriplett
• @m-ou-se
• @sfackler
• @yaahc

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[m-ou-se]

There's an accepted RFC to turn prefix".." into a single token in Rust 2021, such that a later RFC could define e.g. z".." as a new type of literal. Looking at this PR, I'm now wondering what edition Literal::from_str("z\"\"") should use, since the string does not carry any span information. This is already a problem for TokenStream::from_str though, so not really a new concern for this PR.

I'll post a comment on that RFC PR about this.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.