Describe the bug
Comet's parse_utf8_to_boolean (native/spark-expr/src/conversion_funcs/string.rs) uses Rust str::trim, which strips only Unicode char::is_whitespace code points. Spark's UTF8String.trimAll() (referenced by Cast for stringToBool) trims every byte where Character.isWhitespace(b) || Character.isISOControl(b) (UTF8String.java:201-203, UTF8String.java:994-1010).
The two sets differ on the ASCII control bytes 0x00-0x08, 0x0E-0x1F, and 0x7F: Spark trims them, Comet does not.
`\" \x01true\x01 \"` -> Spark casts to `true`, Comet casts to `null` (or ANSI-errors).
Scope
Same divergence likely exists in other trim-based cast paths in conversion_funcs/string.rs. UTF8String.trimAll parity should be assessed systematically.
Prior context
Reported in the review of #4914; that PR intentionally left the pre-existing behavior alone to keep it a pure perf change.
Describe the bug
Comet's
parse_utf8_to_boolean(native/spark-expr/src/conversion_funcs/string.rs) uses Ruststr::trim, which strips only Unicodechar::is_whitespacecode points. Spark'sUTF8String.trimAll()(referenced byCastforstringToBool) trims every byte whereCharacter.isWhitespace(b) || Character.isISOControl(b)(UTF8String.java:201-203,UTF8String.java:994-1010).The two sets differ on the ASCII control bytes
0x00-0x08,0x0E-0x1F, and0x7F: Spark trims them, Comet does not.Repro (per @mbutrovich on #4914)
Scope
Same divergence likely exists in other
trim-based cast paths inconversion_funcs/string.rs.UTF8String.trimAllparity should be assessed systematically.Prior context
Reported in the review of #4914; that PR intentionally left the pre-existing behavior alone to keep it a pure perf change.