Rename task_slots -> vcores (executor spec + CLI)#2045
Merged
avantgardnerio merged 3 commits intoJul 16, 2026
Merged
Conversation
github-actions
Bot
added
documentation
5 tasks
The name `task_slots` implies "how many task instances can run concurrently
on this executor," but it's actually been used two different ways:
- concurrent-task capacity (scheduler-side accounting)
- virtual-core count that a task can drive through one plan-Arc
(what `--concurrent-tasks` actually controlled — the CLI help even
hinted at this)
Rename to `vcores`, following the YARN precedent
(`yarn.nodemanager.resource.cpu-vcores`) and Spark's `spark.executor.cores`.
The `v` prefix is self-documenting: it isn't physical `nproc`, and readers
grepping in a year don't have to hunt for a doc comment to know that.
The new name also leaves room for future execution models that decouple
"how many tasks" from "how many cores per task."
Mechanical rename, no behavior change. Field numbers in the proto are
preserved for wire compat.
Highlights:
- `ExecutorSpecification.task_slots` -> `vcores` (u32), with a doc
comment citing YARN/Spark.
- Proto: `ExecutorResource.task_slots` -> `vcores`,
`AvailableTaskSlots` -> `AvailableVcores`, `ExecutorTaskSlots` ->
`ExecutorVcores`, `PollWorkParams.num_free_slots` -> `num_free_vcores`.
- CLI: `--concurrent-tasks` -> `--vcores` (help text explains it's
scheduler-facing accounting, not physical nproc).
- Cluster bookkeeping: `InMemoryClusterState.task_slots` ->
`available_vcores`. Local variables in bind helpers renamed to match.
- TUI: "Task Slots" column -> "vCores"; `SortColumn::TaskSlots` ->
`Vcores`.
- Executor process: startup log line and doc comment on `vcores`
reworded to drop stale "concurrent tasks" prose.
- Docs, docker-compose, tpch workflow, and CLI README updated so
`--concurrent-tasks` invocations don't fail at boot under the new
flag name. Historical changelogs left as-is.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
avantgardnerio
force-pushed
the
brent/rename-task-slots-to-vcores
branch
from
July 15, 2026 22:45
e5a1ef1 to
4b0b99f
Compare
Member
|
@lukekim fyi |
Member
|
This LGTM because it is a more generic approach to advertising executor compute capacity and it is not tied into any specific task/partition execution model. I'd like to see feedback from other contributors/users before approving, since technically it is a breaking change. |
phillipleblanc
approved these changes
Jul 16, 2026
phillipleblanc
left a comment
phillipleblanc
left a comment
Contributor
There was a problem hiding this comment.
I think this makes sense, my one concern would be I think we should keep the ability to gracefully upgrade existing cluster by having this release understand both protobuf message types, and then in the release after we can remove the old messages.
Keeps the pre-rename operator surface working for one release so in-place
cluster upgrades don't break:
- `--concurrent-tasks` on ballista-executor and ballista-cli: hidden clap
alias that folds into `--vcores` with a stderr deprecation warning.
- `ExecutorSpecification::with_task_slots` / `task_slots()`,
`ExecutorData::{total,available}_task_slots()`, and
`ExecutorDataChange::task_slots()`: `#[deprecated]` shims returning the
renamed fields.
gRPC wire is already binary-compatible (field tags and types unchanged);
no shim needed there.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@phillipleblanc , thanks for the approval! I think the wire protocol will actually be okay for this one, but I added fallbacks for the CLI & API which should help. They work for this PR, but I'm not sure if they will work for more complex ones like multi-partition-tasks. So I've filed an issue where we can discuss what Ballista's live upgrade strategy might be. |
andygrove
reviewed
Jul 16, 2026
andygrove
approved these changes
Jul 16, 2026
andygrove
left a comment
andygrove
left a comment
Member
There was a problem hiding this comment.
LGTM - with one request to revert the change to the user persona documentation. I agree that the wire format should be unchanged. Thanks @avantgardnerio!
The persona doc is a stable contract about what each user type relies on; naming refactors shouldn't churn it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to review feedback on #2038: split the
task_slots→vcoresrename into its own PR so it can land independently.Mechanical rename, no behavior change. Follows the YARN precedent (
yarn.nodemanager.resource.cpu-vcores) and Spark'sspark.executor.cores. Thevprefix is self-documenting — readers grepping in a year don't have to hunt for a doc comment to know these aren't physicalnproc. It also leaves room for future execution models that decouple "how many tasks" from "how many cores per task."What changed
ExecutorSpecification.task_slots→vcores(u32), with a YARN/Spark doc commentExecutorResource.task_slots→vcores,AvailableTaskSlots→AvailableVcores,ExecutorTaskSlots→ExecutorVcores,PollWorkParams.num_free_slots→num_free_vcores. Field numbers preserved for wire compat.--concurrent-tasks→--vcoreson bothballista-executorandballista-cliInMemoryClusterState.task_slots→available_vcoresdocker-compose.yml, tpch workflow, and CLI README updated so--concurrent-tasksinvocations don't fail at boot under the new flag nameTest plan
🤖 Generated with Claude Code