Skip to content

Codegen dispatch for YearMonth and DayTimeInterval with shuffle and nested element #4975

Description

@peterxcli

Describe the bug

#4541 (review)

Steps to reproduce

import java.nio.file.Files
import spark.implicits._

spark.conf.set("spark.sql.adaptive.enabled", "false")

val path = Files.createTempDirectory("comet-ym-").resolve("data").toString
Seq((1, 2), (0, -1)).toDF("y", "m").write.parquet(path)

// Case 1: nested YearMonthIntervalType fallback
val nested = spark.sql(
  """SELECT transform(a, x -> x) AS result
    |FROM VALUES
    |  (array(make_ym_interval(1, 2),
    |         CAST(NULL AS INTERVAL YEAR TO MONTH)))
    |AS t(a)""".stripMargin)

nested.explain("extended")
nested.show(false)

// Case 2: native-shuffle fallback
val shuffled = spark.sql(
  s"""SELECT y, m, make_ym_interval(y, m) AS i
     |FROM parquet.`$path`
     |DISTRIBUTE BY y""".stripMargin)

shuffled.explain("extended")
shuffled.show(false)

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingenhancementNew feature or requestpriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions