Reduce unnecessary escaping in proc_macro::Literal::character/string

[dtolnay]

I noticed that https://doc.rust-lang.org/proc_macro/struct.Literal.html#method.character is producing unreadable literals that make macro-expanded code unnecessarily hard to read. Since the proc macro server was using escape_unicode(), every char is escaped using \u{…} regardless of whether there is any need to do so. For example Literal::character('=') would previously produce '\u{3d}' which unnecessarily obscures the meaning when reading the macro-expanded code.

I've changed Literal::string also in this PR because str's Debug impl is also smarter than just calling escape_debug on every char. For example Literal::string("ferris's") would previously produce "ferris\'s" but will now produce "ferris's".

[rust-highfive]

r? @lcnr

(rust-highfive has picked a reviewer for you, use r? to override)

[petrochenkov]

cc #60495

[lcnr]

don't know the areas actually impacted by this change, so

r? @petrochenkov

[petrochenkov]

I'd like to use this opportunity to address #60495 (maybe not to make all the relevant code changes, but at least to make a decision).
Updated status of literal escaping in the compiler - #60495 (comment).

• Question 1: Why we are doing the escaping at all?
  • Answer: If we are producing a literal token, such token needs to result in a string that is semantically the same after it's unescaped back by rustc's parser.
  • The alternative is to produce a raw string token instead of doing escaping and producing a cooked string token.
    macro_rules do this with doc comments, @dtolnay you mentioned this in Document compatibility between declarative and procedural macro tokens reference#1169 (comment), but that example doesn't actually apply to proc macros, proc macros escape doc comments instead.
    Maybe macro_rules should escape them too, because right now they are the odd one, all other places in Inconsistent literal escaping in proc macros #60495 (comment) use escaping.
• Question 2: What are the requirements for this escaping?
  • Apparently the only strong requirement is that the roundtrip through the rustc parser's unescaping should work?
  • From this PR it looks like readability of the output is also a soft requirement.
    It means that we should probably prefer debug escaping everywhere, because escape_unicode is entirely unreadable, and escape_default is less readable.
• Question 3 (unanswered): Why on earth Debug impl for str and escape_debug do different things?
  • Which one of these is the canonical debug escaping? I assume that here we should also make a choice and use it consistently?

@dtolnay
Could you comment on these questions?

[dtolnay]

The alternative is to produce a raw string token instead of doing escaping and producing a cooked string token.

I don't have a preference between these. They both come out equally readable in typical usage, from what I've seen.

The fact that proc macros and macro_rules do the opposite is a good catch. Here is what I tried:

// main.rs

macro_rules! decl {
    (#[doc = $tt:tt]) => {
        println!("decl macro: #[doc = {}]", stringify!($tt));
    };
}

fn main() {
    decl! {
        ///"
    }
    proc! {
        ///"
    }
}

// lib.rs

#[proc_macro]
pub fn proc(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    if let proc_macro::TokenTree::Group(group) = input.into_iter().nth(1).unwrap() {
        println!("proc macro: #[doc = {}]", group.stream().into_iter().nth(2).unwrap());
    }
    proc_macro::TokenStream::new()
}

proc macro: #[doc = "\""]
decl macro: #[doc = r#"""#]

I think it would be reasonable to make these two behave the same. It's not important to me which behavior we pick.

What are the requirements for this escaping?

I think you got it right in your response to this above.

Why on earth Debug impl for str and escape_debug do different things?

From what I can tell, this divergence was introduced by #83079.

The current behavior of Debug is the newer/better formatting, while escape_debug is the older. I think we should update escape_debug to change to the new behavior where ' does not get a backslash.

[petrochenkov]

the only strong requirement is that the roundtrip through the rustc parser's unescaping should work

So one more alternative is to introduce a new escape_minimal_for_rust_literal doing only the escaping that is required for the roundtrip to work.
That's would be pretty much \, " or ' depending on the literal kind, and maybe also bare CR, need to check the lexer code.

At least it will be predictable in that case, unlike debug escaping.
It would also be readable enough because no unicode would be mangled (maybe with some whitespace parts being less readable than with debug escaping).

[petrochenkov]

Ok, I think this is good to go.
I'll unify the remaining cases in the compiler myself, and then maybe try to migrate from debug escaping to escape_minimal_for_rust_literal after that.
@bors r+

[bors]

📌 Commit f383134 has been approved by petrochenkov