Move HTML→Markdown and Unicode-abuse sanitization into Post#21
Merged
Conversation
SwiftMail no longer performs HTML→Markdown conversion; Post now owns that responsibility end-to-end. Changes: - Bump SwiftText minimum to 1.1.7 (required for SwiftTextCore product) - Add SwiftTextCore to PostServer target (provides UnicodeAbuseSanitizer) - MessageDetail.markdown(): apply UnicodeAbuseSanitizer to the converted markdown (and to plain-text fallback), matching the sanitization that SwiftMail's now-removed markdownContent() used to perform - Update Package.resolved (SwiftText 1.1.6 → 1.1.7) Behavior is equivalent or better: bidi-override scalars and zalgo-style combining-mark clusters are stripped before the string is returned to callers/AI agents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SwiftTextCoreproduct)SwiftTextCoreto thePostServertargetunicodeAbusedescription to JSON/MCP/hook outputs when sanitization modified contentMotivation
SwiftMail's
MessagePart.markdownContent()(which performed HTML→Markdown + sanitization) is being removed in the companion PR Cocoanetics/SwiftMail#remove-swifttext. Post already ownedHTMLToMarkdownviaSwiftTextHTML; this PR completes the move by also preserving Unicode-abuse sanitization inside Post itself.Oliver also wanted sanitization to apply consistently anywhere Post exposes extracted content — not just markdown conversion — and to surface a simple machine-readable/human-readable hint when content had to be cleaned.
Behavior
Sanitization
UnicodeAbuseSanitizernow removes abusive Unicode from exported content, including:Covered outputs
This now applies to:
fetch(text,html, andmarkdownbody modes)eml(text,html, andmarkdownbody modes)list --jsonandsearch --jsonlistMessagessearchMessagesfetchMessagedraft --replying-toNew metadata
When sanitization modifies subject and/or body, outputs may include:
No field is emitted when nothing had to be changed.
Notes
unicodeAbuseBuild / test
swift build -c release✅swift test✅ (8 tests, 0 failures)