fix(stream): decode lossily instead of dropping lines on invalid UTF-8#2997
Open
albatrossflyon-coder wants to merge 1 commit into
Open
Conversation
BufRead::lines().map_while(Result::ok) returns Err on a non-UTF-8 line (e.g. OEM/ANSI bytes from a non-English-locale Windows tool), and map_while stops at the first None — silently discarding every line after the bad one too, not just the bad one. A failing build command whose error text contains a single non-UTF-8 byte produces output that looks like a clean success. Add read_lines_lossy(), which reads raw bytes and decodes each line with String::from_utf8_lossy (invalid bytes become U+FFFD) instead of erroring, and use it at all 6 call sites in run_streaming instead of the truncating lines()/map_while(Result::ok) pattern. Fixes rtk-ai#2994
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2994.
run_streaming's six call sites all read child stdout/stderr with:BufRead::lines()returnsErrfor a line that isn't valid UTF-8 (OEM/ANSIbytes from a non-English-locale Windows tool, per the issue's cp850 repro).
map_whilestops iteration at the firstNone— so one non-UTF-8 linedoesn't just get dropped, it silently discards every line emitted after
it too. A failing build whose error text happens to contain one non-UTF-8
byte can end up looking like a clean, silent success even though the exit
code is non-zero.
Fix
Added
read_lines_lossy(), which reads raw bytes viaBufRead::read_untiland decodes each line with
String::from_utf8_lossy(invalid bytes becomeU+FFFD) instead of erroring out of the iterator. Replaced all 6
.lines().map_while(Result::ok)call sites inrun_streamingwith it —this is the single shared path every filter routes through, so the fix
covers all callers at once.
rtk pipe(cmds/system/pipe_cmd.rs) already fails loudly on invalidUTF-8 stdin rather than silently, per the issue's own note — left as-is,
out of scope here.
Testing
cargo test: 2448 passed, 8 ignored, 0 failed (no regressions)read_lines_lossydirectly, including one thatreproduces the exact failure mode from the issue: an ASCII line, a line
with an embedded invalid byte, then another ASCII line — asserts all 3
lines survive instead of the third silently vanishing
cargo clippy --all-targets -- -D warnings: cleancargo fmt --check: clean