Skip to content

Commit 1f3b4ea

Browse files
timsaucerclaude
andcommitted
feat(spark): accept native Python literals for literal-friendly args
Audit the functions.spark namespace against pyspark.sql.functions and widen arguments that pyspark types as a non-column literal so callers can pass bare int/float/str instead of wrapping in lit(): - int args: array_repeat count, slice start/length, shiftleft/shiftright/ shiftrightunsigned numBits, sha2 numBits, round scale, substring pos/len, width_bucket numBucket - int32-coerced args (binding requires int32): add_months months, date_add/date_sub days, space n, make_dt_interval/make_interval parts - float args: modulus/pmod operands; make_*_interval secs - str args: next_day dayOfWeek, date_trunc/trunc format, date_part field, from_utc_timestamp/to_utc_timestamp tz, spark_cast type_str, json_tuple *fields - Any: array_contains value, if_ if_true/if_false Arguments that pyspark types as ColumnOrName (str means column name, not a literal) are left as Expr to avoid diverging from pyspark semantics: ilike/like pattern, parse_url partToExtract/key, str_to_map delimiters, bit_get pos, time_trunc unit. Also rename str_to_map's delimiter params to pairDelim/keyValueDelim to match pyspark exactly (they were pair_delim/key_value_delim). Add a coercion test matrix and update docstring examples to show the native-literal calling convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 86a135e commit 1f3b4ea

2 files changed

Lines changed: 297 additions & 147 deletions

File tree

0 commit comments

Comments
 (0)