Skip to content

fix: propagate sort shuffle memory_limit_per_task_bytes to executors#2090

Merged
andygrove merged 4 commits into
apache:mainfrom
andygrove:worktree-fix-2089-sort-shuffle-mem-limit
Jul 20, 2026
Merged

fix: propagate sort shuffle memory_limit_per_task_bytes to executors#2090
andygrove merged 4 commits into
apache:mainfrom
andygrove:worktree-fix-2089-sort-shuffle-mem-limit

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #2089.

Rationale for this change

ballista.shuffle.sort_based.memory_limit_per_task_bytes
(SortShuffleConfig::memory_limit_per_task_bytes) controls the point at which
the sort shuffle writer spills its in-memory batches to disk. The scheduler
built SortShuffleWriterExec with the configured value, but the value was not
carried in the SortShuffleWriterExecNode protobuf message. On decode, the
executor reconstructed the config with SortShuffleConfig::new(true, batch_size),
which discards the override and uses the built-in default. Because the writer
actually runs on the executor, the knob was effectively inert in distributed
execution.

What changes are included in this PR?

  • Add memory_limit_per_task_bytes to the SortShuffleWriterExecNode message
    as an optional uint64 and populate it on encode. Presence lets a plan
    produced before this field existed decode to the built-in default rather than
    to 0.
  • Apply the value on decode via with_memory_limit_per_task_bytes, so the
    session override reaches the executor.
  • Treat a value of 0 as "disable the per-task budget": the writer then spills
    only under runtime memory-pool pressure. Previously a 0 limit would have
    forced a spill on every batch.
  • Refresh the tuning guide and configs docs, which still documented removed keys
    (buffer_size, memory_limit, spill_threshold), and describe the per-task
    budget and the 0-disables behavior.

Are there any user-facing changes?

The configuration setting now takes effect on executors as documented. Setting
it to 0 is newly meaningful: it disables the per-task budget and relies on
memory-pool pressure to trigger spilling. No public Rust API changes.

The sort shuffle writer's per-task buffered-bytes budget
(`ballista.shuffle.sort_based.memory_limit_per_task_bytes`) was built into
`SortShuffleConfig` by the scheduler but never carried in the
`SortShuffleWriterExecNode` protobuf message. Executors, where the writer
actually runs, reconstructed the config with the built-in default, so the
session override was inert in distributed execution.

Carry the value on the wire as an optional field (absent decodes to the
built-in default) and apply it on decode. Also treat a value of 0 as
"disable the per-task budget", leaving runtime memory-pool pressure as the
sole spill trigger, and refresh the tuning guide and configs docs, which
still referenced removed keys.

Closes apache#2089
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 18, 2026

@milenkovicm milenkovicm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove

@phillipleblanc phillipleblanc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great!

@metegenez metegenez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only nit: adding an OOM warning to the config docs for the 0 + unbounded-pool combo. Other than that, LGTM. Test coverage looks good.

One question about having 2 spill triggers: In a world we mostly cover and account the bytes into memory pool, this local operator count and trigger wouldnt be necessary, right?

@andygrove
andygrove marked this pull request as ready for review July 19, 2026 13:34
@andygrove

Copy link
Copy Markdown
Member Author

Only nit: adding an OOM warning to the config docs for the 0 + unbounded-pool combo. Other than that, LGTM. Test coverage looks good.

Added. Thanks.

One question about having 2 spill triggers: In a world we mostly cover and account the bytes into memory pool, this local operator count and trigger wouldnt be necessary, right?

Yes, if we had accurate memory accounting (perhaps backed by the real memory allocator - see #2033) I think we would revisit these configs.

@andygrove
andygrove marked this pull request as draft July 19, 2026 14:30
@andygrove
andygrove marked this pull request as ready for review July 19, 2026 20:53
@andygrove
andygrove merged commit 752d4de into apache:main Jul 20, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sort shuffle writer's memory_limit_per_task_bytes config is not propagated to executors

4 participants