Commit 1f3b4ea
feat(spark): accept native Python literals for literal-friendly args
Audit the functions.spark namespace against pyspark.sql.functions and
widen arguments that pyspark types as a non-column literal so callers can
pass bare int/float/str instead of wrapping in lit():
- int args: array_repeat count, slice start/length, shiftleft/shiftright/
shiftrightunsigned numBits, sha2 numBits, round scale, substring pos/len,
width_bucket numBucket
- int32-coerced args (binding requires int32): add_months months,
date_add/date_sub days, space n, make_dt_interval/make_interval parts
- float args: modulus/pmod operands; make_*_interval secs
- str args: next_day dayOfWeek, date_trunc/trunc format, date_part field,
from_utc_timestamp/to_utc_timestamp tz, spark_cast type_str,
json_tuple *fields
- Any: array_contains value, if_ if_true/if_false
Arguments that pyspark types as ColumnOrName (str means column name, not a
literal) are left as Expr to avoid diverging from pyspark semantics:
ilike/like pattern, parse_url partToExtract/key, str_to_map delimiters,
bit_get pos, time_trunc unit.
Also rename str_to_map's delimiter params to pairDelim/keyValueDelim to
match pyspark exactly (they were pair_delim/key_value_delim).
Add a coercion test matrix and update docstring examples to show the
native-literal calling convention.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 86a135e commit 1f3b4ea
2 files changed
Lines changed: 297 additions & 147 deletions
0 commit comments