Untracked memory mode without spark.memory.offHeap.size throws NoSuchElementException during driver init

[LuciferYang]

Backend

VL (Velox) — the entry point is in gluten-core; the end-to-end failure also reaches the Velox driver-start path.

Bug description

GlutenDriverPlugin.setPredefinedConfigs reads the off-heap size in its non-dynamic-sizing branch with the single-arg conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY). checkOffHeapSettings returns early for untracked memory mode (spark.gluten.memory.untracked=true) and skips the requirement that the size be set, so in that mode the key can be absent. SparkConf.get(String) throws NoSuchElementException on a missing key (it reads the raw settings map and does not consult the ConfigEntry default), so driver plugin init crashes instead of proceeding under untracked semantics.

The same single-arg read exists on the Velox driver-start path: VeloxListenerApi.onDriverStart also calls conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY), and it runs after setPredefinedConfigs (via GlutenDriverPlugin.init → Component.onDriverStart). The dynamic-sizing branch already shields that reader by doing conf.set(SPARK_OFFHEAP_SIZE_KEY, "0"); untracked mode did not, so it was the one unshielded path.

The fix normalizes the key in the untracked branch: when spark.memory.offHeap.size is absent, set it to 0, mirroring the dynamic-sizing branch. One change covers both readers (the plugin and VeloxListenerApi), and normal mode is unaffected because checkOffHeapSettings already guarantees the key is set there.

Gluten version

main (1.8.0-SNAPSHOT)

Spark version

Version-agnostic (applies to spark-3.3 / 3.4 / 3.5 / 4.0 / 4.1).

Spark configurations

• spark.plugins=org.apache.gluten.GlutenPlugin
• spark.gluten.memory.untracked=true
• spark.memory.offHeap.size not set (dynamic off-heap sizing left at its default of false).

System information

N/A — logic issue in gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala, independent of OS/hardware.

Relevant logs

java.util.NoSuchElementException: spark.memory.offHeap.size
  at org.apache.spark.SparkConf.$anonfun$get$1(SparkConf.scala:...)
  at org.apache.spark.SparkConf.getSizeAsBytes(SparkConf.scala:...)
  at org.apache.gluten.GlutenDriverPlugin$.setPredefinedConfigs(GlutenPlugin.scala:141)
  at org.apache.gluten.GlutenDriverPlugin.init(GlutenPlugin.scala:61)

Fix direction

In the untracked/non-dynamic branch, set spark.memory.offHeap.size to 0 when it is absent, so neither setPredefinedConfigs nor VeloxListenerApi.onDriverStart hits NoSuchElementException.