[Fix]: Preserve UTF-8 codepoints across PTY 4 KiB read boundaries

[matiaspalmac]

Description

The PTY reader thread was calling String::from_utf8_lossy on each raw 4096-byte chunk returned by reader.read. That works for pure ASCII, but breaks on any multi-byte UTF-8 sequence that happens to straddle a chunk boundary: the last byte(s) of a codepoint get decoded as the lead of an "incomplete" sequence and are substituted with U+FFFD, so the terminal shows � for accented letters, CJK text, emoji, box-drawing characters, and most non-English shell prompts whenever the read lands on a bad offset.

The fix is a small leftover buffer: on every read we combine any held-back bytes with the new chunk, find the longest valid UTF-8 prefix, emit that, and keep the 1–3 trailing truncated bytes for the next read. Actually invalid sequences are still replaced with U+FFFD so one bad byte cannot stall the stream forever, and the leftover is flushed on EOF/error so the final prompt byte is never silently swallowed.

Related Issue

Fixes #70

Context / Problem

On Windows PowerShell a prompt containing a non-ASCII character such as », 日, or the spinner emoji from e.g. npm/pnpm reliably produced � in the Trixty terminal whenever the output happened to flush at a 4096-byte boundary that landed inside the codepoint. It was intermittent and looked like a font issue, but it was the decoder discarding bytes that hadn't arrived yet.

Change

• Extracted the decoding rule into a small pure helper split_utf8_safely(buf: &[u8]) -> usize that returns how many bytes of buf are safe to emit now.
• Rewrote the reader loop to hold back the truncated tail and prepend it to the next read.
• Flush any trailing leftover on EOF/error so the final output is never dropped.
• Added six unit tests covering ASCII, 4-byte codepoints split at every internal boundary (1/2/3), mixed multi-byte runs with 1-byte reads, invalid bytes (U+FFFD recovery, no stall), empty input, and truncated tail on close.

Verification

• cargo test --lib → 33 passed, 0 failed (includes the 6 new tests).
• cargo clippy --lib --no-deps -- -D warnings → clean.
• OS: Windows 11
• Version: v1.0.10

Checklist

• I have tested this fix thoroughly.
• I have followed the project's coding guidelines.
• My changes generate no new warnings or errors.
• I have verified the fix on:
  • OS: Windows
  • Version: v1.0.10

[github-actions]

Thanks for the contribution! I'll review it as soon as possible. If you have still changes, please mark this PR as draft and all reviews will be cancelled. Tests reviews will be re-run only when the PR is marked as ready for review.

[Copilot]

Pull request overview

Fixes PTY output decoding so UTF-8 multi-byte codepoints aren’t corrupted when reads split across the fixed 4 KiB boundary (fixes #70).

Changes:

• Added split_utf8_safely helper to identify an emit-safe UTF-8 prefix and retain truncated tails.
• Updated the PTY reader thread to prepend any leftover bytes to the next read and flush leftovers on EOF/error.
• Added unit tests covering boundary splits, invalid bytes, empty input, and truncated tails.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[matiaspalmac]

Addressed all three Copilot observations:

1. Allocation per read — reader now keeps a single Vec<u8> assembly buffer with capacity 4096 + 4 and calls drain(..split) in place, so no std::mem::take + to_vec churn on each chunk.
2. Post-invalid bytes delayed until the next read — split_utf8_safely no longer uses valid_up_to + error_len as the boundary. On any mid-buffer invalid sequence it now returns buf.len() and the caller emits the whole chunk through from_utf8_lossy (one U+FFFD per bad sequence, no stalled suffix).
3. Docstring vs implementation mismatch — rewrote the doc comment to reflect the new "only truncated tails are held back" contract.

Added invalid_byte_mid_chunk_does_not_delay_valid_trailing_bytes as a regression test that would fail under the old implementation. 7/7 pty tests pass, clippy clean. (5ca2037)