Skip to content

cast string to boolean: trim ISO control bytes to match Spark's UTF8String.trimAll #4959

Description

@andygrove

Describe the bug

Comet's parse_utf8_to_boolean (native/spark-expr/src/conversion_funcs/string.rs) uses Rust str::trim, which strips only Unicode char::is_whitespace code points. Spark's UTF8String.trimAll() (referenced by Cast for stringToBool) trims every byte where Character.isWhitespace(b) || Character.isISOControl(b) (UTF8String.java:201-203, UTF8String.java:994-1010).

The two sets differ on the ASCII control bytes 0x00-0x08, 0x0E-0x1F, and 0x7F: Spark trims them, Comet does not.

Repro (per @mbutrovich on #4914)

`\" \x01true\x01 \"` -> Spark casts to `true`, Comet casts to `null` (or ANSI-errors).

Scope

Same divergence likely exists in other trim-based cast paths in conversion_funcs/string.rs. UTF8String.trimAll parity should be assessed systematically.

Prior context

Reported in the review of #4914; that PR intentionally left the pre-existing behavior alone to keep it a pure perf change.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions