Live upgrade strategy

[avantgardnerio]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Ballista users are now running clusters 24/7, and can't afford downtime during upgrades. However, Ballista is still young & flexible, and we want to maintain velocity even as we make architectural changes. It would be good to have an official upgrade strategy that protects both.

Describe the solution you'd like

Add a ballista_protocol_version: u32 field to ExecutorRegistration / HeartbeatFromExecutor. The scheduler holds its own compiled-in BALLISTA_PROTOCOL_VERSION constant. On every
heartbeat or registration:

• If executor.protocol_version == scheduler.protocol_version → proceed normally.
• Otherwise → drop the heartbeat with an info! log and never dispatch tasks to that executor.

We bump BALLISTA_PROTOCOL_VERSION only on releases that break wire compat, not once per major version. Most releases won't bump it; some will.

Rolling upgrade flow

1. New scheduler comes up with bumped BALLISTA_PROTOCOL_VERSION.
2. Old executors heartbeat in; scheduler drops them; they drain naturally as their existing tasks finish (or the operator kills them).
3. New executors with the bumped version come up and register normally.
4. No dual-implementation window, no code carrying old + new proto types.

Both scheduler versions are still up during the transition, so operators can roll executors and schedulers in either order — mismatched pairs just don't talk to each other, they don't corrupt state.

Prior art

DQE (Coralogix' Ballista fork) has been doing this for a while with reasonable success.

Describe alternatives you've considered

3 release upgades: add, change impl & deprecate, remove

Additional context

The downsides:

1. cluster capacity is at worst halved during upgrades - mitigation: upgrade off hours, custom k8s operator
2. more operational complexity

[avantgardnerio]

@phillipleblanc @andygrove PTAL

[andygrove]

LGTM

[milenkovicm]

It definitely makes sense to capture proto version, i just wonder if we need to capture datafusion version.

ballista major version tracks datafusion upgrade, we might have background compatibility in ballista, but datafusion proto might be incompatible, so executors might fail to decode proto. do we need to take extra care when ballista is released for datafusion proto compatibility ?

[phillipleblanc]

I think we might also need an extensibility hook that allows users to also have their own application version that works in a similar way to the protocol version you are proposing - even if ballista itself is compatible, the application may have made changes that render previous executors imcompatible with the new deployment, and being able to hook into this same mechanism would be nice.

Otherwise LGTM!

[phillipleblanc]

It definitely makes sense to capture proto version, i just wonder if we need to capture datafusion version.

ballista major version tracks datafusion upgrade, we might have background compatibility in ballista, but datafusion proto might be incompatible, so executors might fail to decode proto. do we need to take extra care when ballista is released for datafusion proto compatibility ?

We could take the conservative approach of bumping the BALLISTA_PROTOCOL_VERSION on every DF upgrade.