Ergonomics of String handling

[kbknapp]

From the survey, string ergonomics are huge. Many, many CLI applications deal heavily with strings. In Rust, strings can be...difficult.

Granted, (IMO) Rust handles them correctly, sometimes the correctness doesn't actually matter for a given problem domain and just adds unnecessary gyration.

For example, we have:

• (&)('static)str
• Cow
• (&)String
• (&)OsStr
• (&)OsString

It's understanding it can be overwhelming. My personal opinion is we should first tackle/discuss the ergonomics of using OsStr(ing) as it's heavily used on Linux (where paths may not contain valid UTF-8).

IMO OsStr should have the same user experience as &str/String.

We could also probably start by either listing known issues/inconsistencies or any current issue links/RFCs on the matter.

[kbknapp]

One of the quotes from the survey:

OsStr(ing) lacks starts_with/ends_with/contains APIs. This is a hard problem, I think, but also seems like something we're perfectly capable of solving if we wanted to, and seeing to_string_lossy().starts_with(...) or, worse in some ways, to_str().unwrap().starts_with() looks really bad to me from a compatibility standpoint. Like, why am I using OsStr if I have to pay this cost anyway?

[BurntSushi]

This RFC extends the Pattern API to OsStr: rust-lang/rfcs#2295

[epage]

Is there anything like format! for OsString? If I want to pass a path as a command line argument via process::Command, I either need to build it up manually (joining "--flag" and my path) or convert the path to a String and use format!. So I'm going from OsString to String back to OsString, losing the ability to track files that need the OsString.

[epage]

There was easy_strings. For rapid prototyping, I wonder if we should evaluate making "as easy as python" (e.g. internally clone or Arc everything, reducing borrows) versions of common vocabulary terms. failure::Error and easy_strings are some examples. Paths are another opportunity for this.

Are there any others?

[kbknapp]

If I want to pass a path as a command line argument via process::Command, I either need to build it up manually (joining "--flag" and my path) or convert the path to a String and use format!. So I'm going from OsString to String back to OsString, losing the ability to track files that need the OsString.

I agree - but in this particular case your only option is to build it manually, as using format! and going from OsStr->String->OsStr doesn't do anything. If you went to String at all you're either losing invalid UTF-8 (using _lossy()), or panic!ing, meaning you didn't need OsStr in the first place.

easy_strings

This looks awesome and I wasn't aware of it! There's also

• string-wrapper for stack allocated strings.
• string for the storage backing being generic (and/or stack allocated)

IMO, if there was a single "easy by default" string which I could later tune/drop-in replace with std Strings I'd be all for it. Drop in replace is the hard part though, as all functionality must be replicated (traits, etc.) for the "easy don't care" string.

[kbknapp]

Also, I've found OsStr lacking in clap for some more cases in the wild. There's been a few issues related them as well:

• validator_os takes &OsStr, but validator takes String clap-rs/clap#1165
• validator_os returns Result<(), OsString> clap-rs/clap#848
• convert Result<(), OsString> -> Result<(), String> clap-rs/clap#981
• use cases for value_of/values_of clap-rs/clap#751

[jeekobu]

There is some discussion of the hurdles of mixing Path/PathBuf with String in rust-lang/rust#49868

[epage]

There was easy_strings. For rapid prototyping, I wonder if we should evaluate making "as easy as python" (e.g. internally clone or Arc everything, reducing borrows) versions of common vocabulary terms. failure::Error and easy_strings are some examples. Paths are another opportunity for this.

Along these lines, an "ergonomic" string type that supports multiplying against integers would be useful like in Python

CC @Aaronepower

[epage]

... I always assumed OsStr let you convert to/from bytes.

• It turns out it only supports converting to/from str. This means there is no underlying type to use to manipulate them.
• It doesn't support any kind of way to walk the parts and manipulate them.

[BurntSushi]

I always assumed OsStr let you convert to/from bytes.

You can, on Unix only though: https://doc.rust-lang.org/std/os/unix/ffi/trait.OsStrExt.html

On Windows, all you can get are the UTF-16 bytes (which is the raw representation): https://doc.rust-lang.org/std/os/windows/ffi/trait.OsStrExt.html

On Unix, OsStrs are, AIUI, zero cost in that they represent the bytes from the platform as-is. On Windows, OsStrs are always transcoded to and from UTF-16 at the boundaries (with WTF-8 as the internal representation).

[BurntSushi]

I've written a bit about this topic as it relates to byte strings: https://docs.rs/bstr/0.1.2/bstr/#file-paths-and-os-strings

While it seems like we will eventually get string-like APIs on OsStr, that still won't be enough. Consider the perhaps somewhat common case of trying to match a file path against a regex. The regex machinery cannot know the internal representation of an OsStr, so you're only real choice is to lossily convert it to UTF-8 on Windows and use the raw bytes on Unix. But the standard library doesn't make this particular use case easy and currently requires writing platform specific code.