From 545df02a2c3469864f42f8a8feeee93a2892cd74 Mon Sep 17 00:00:00 2001 From: Brent Gardner Date: Sun, 19 Jul 2026 17:29:26 -0600 Subject: [PATCH] docs(config): rewrite max_partitions_per_task description to be timeless Drop PR-relative phrasing ("pre-multi-partition-tasks execution model", "preserving master's behaviour on merge") and the "Spark user persona" jargon. Describe what the setting does in terms a config-doc reader can act on. Addresses andygrove's review comment on #2038. Co-Authored-By: Claude Opus 4.7 (1M context) --- ballista/core/src/config.rs | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/ballista/core/src/config.rs b/ballista/core/src/config.rs index c3f841adf..bef5556fa 100644 --- a/ballista/core/src/config.rs +++ b/ballista/core/src/config.rs @@ -359,13 +359,12 @@ static CONFIG_ENTRIES: LazyLock> = LazyLock::new(|| ConfigEntry::new( BALLISTA_SCHEDULER_MAX_PARTITIONS_PER_TASK.to_string(), "Upper bound on the number of input partitions packed into a single \ - task's `partition_slice`. `1` (default) is the pre-multi-partition-tasks \ - execution model — one task per partition, matching the Spark user \ - persona and preserving master's behaviour on merge. Raise to enable \ - multi-partition tasks (fewer tasks, parallel-sort / parallel-join wins); \ - `0` means unbounded — the scheduler fills each task up to the executor's \ - free vcore count. Does not apply to collapse stages, which must pack \ - their full pending queue into a single task for correctness." + task's `partition_slice`. `1` (default) means one task per input \ + partition. Raise to enable multi-partition tasks (fewer tasks, \ + parallel-sort / parallel-join wins); `0` means unbounded — the \ + scheduler fills each task up to the executor's free vcore count. \ + Does not apply to collapse stages, which must pack their full \ + pending queue into a single task for correctness." .to_string(), DataType::UInt64, Some(1.to_string()),