feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes

[avantgardnerio]

Summary

Closes #2071.

My notes:

1. caution, I do not run upstream ballista in production, and I do not (yet) have a k8s cluster to test this on
2. however, this is the strategy that until recently Coralogix used with mostly success
3. I still feel confident submitting this, because it does not appear we have this functionality at all yet?
4. my motivation is purely (well, mostly) selfish: I would like to release multi-partition-tasks without a multi-major-version-upgrade cycle

With that out of the way, onto the Claude ramblings:

Adds a compile-time BALLISTA_PROTOCOL_VERSION: u32 = 1 in
ballista_core that gates executor↔scheduler traffic, plus
Kubernetes-style /healthz and /readyz probes on both binaries so
live upgrades have something to observe.

Protocol version

• New uint32 ballista_protocol_version = 7 on ExecutorRegistration.
• Executor populates it on registration, heartbeat, and poll_work.
• Scheduler rejects mismatches (or missing metadata on heartbeat) with
  Status::failed_precondition, an info! log, and increments
  ballista_scheduler_rejected_by_protocol_version_total (prometheus
  feature).
• Executor /readyz flips off on RPC failure, and after 5 consecutive
  heartbeat failures the executor self-terminates so a mismatched pod
  crash-loops until its image is bumped rather than sitting idle.

Health probes

• Scheduler: mounted on the existing axum router (same port as gRPC/REST).
  /healthz is always 200. /readyz returns 200 iff at least
  --min-ready-executors executors are registered (default 1; set to 0
  for dev/single-node).
• Executor: new HTTP server on --bind-health-port (default 50053),
  since Arrow Flight isn't HTTP. /healthz is always 200 — deliberately
  independent of scheduler connectivity to prevent restart cascades.
  /readyz reflects the last heartbeat.

Rolling upgrade semantics (documented in kubernetes.md)

scheduler ↔ executor	outcome
old ↔ old	works
old ↔ new	works (old scheduler ignores the unknown field)
new ↔ new	works
new ↔ old	rejected — the case the check exists for

Bump both Deployments' image tags together in the same release; each
rolls independently under its own maxSurge, and the scheduler's
/readyz gate keeps Service traffic on the old scheduler until the
new one has matching-version executors registered.

Bump policy: bump BALLISTA_PROTOCOL_VERSION only on releases that
change the executor↔scheduler wire format. Most releases won't.

Test plan

• cargo test -p ballista-scheduler --lib (236 pass, incl. 8 new)
• cargo test -p ballista-executor --lib (45 pass)
• cargo clippy --all-targets -p ballista-core -p ballista-executor -p ballista-scheduler (clean)
• cargo fmt --all
• End-to-end smoke against a 2-executor local cluster: all four
  /healthz and /readyz endpoints return 200; scheduler /readyz
  shows ready: 2 executors registered.

New unit tests cover:

• Register rejected on mismatched protocol version
• Heartbeat rejected on mismatched protocol version
• Heartbeat rejected when metadata is missing entirely
• poll_work rejected on mismatched protocol version
• /healthz always OK
• /readyz fails with no executors
• /readyz OK when min_ready_executors = 0
• /readyz OK after an active heartbeat is recorded

Notes for reviewers

• The zero default of the new proto field naturally means "old
  executor" and is rejected once we start at
  BALLISTA_PROTOCOL_VERSION = 1. This is intentional per the issue's
  "no dual-implementation window" design.
• The old scheduler is permissive because it doesn't know the field
  exists — so during a rolling upgrade, old-scheduler ↔ new-executor
  keeps working. Only new-scheduler ↔ old-executor is rejected.
• Health probes are unconditional: they mount regardless of the
  rest-api feature or --disable-rest-api, since they're
  operational, not part of the REST API surface.

🤖 Generated with Claude Code

[avantgardnerio]

@phillipleblanc FYI

[phillipleblanc]

I think the protocol version makes sense. I have one comment about the health service specifically below, but in general I think this points to a tension in what Ballista is and how we expect consumers to use it.

We use Ballista as a library that we embed into our existing process (i.e. we call scheduler_process::create_scheduler() and serve this on our existing gRPC port rather than let Ballista create it).

I think it would be useful to have a split between the Ballista standalone processes (i.e. ballista-scheduler, ballista-executor) vs the libraries that those binaries (and other consumers) can use to build their own distributed query engines. So for the health probes specifically, I think it would make sense to have it live in the binary/process startup, not as part of the library component. That way its still usable out of the box and is a good reference implementation for anyone that is integrating with ballista, but doesn't force them to take this specific implementation.

[phillipleblanc]

In our project we embed Ballista in our process that already defines health/readiness probes that are specific to our implementation. This would add a second health surface that we can't turn off without removing this code in our fork.

Perhaps similar to the rest-api feature flag, we could add a health-probes flag that controls this? (or bundle this with the rest-api feature flag?) Alternatively a separate option to control whether ballista starts the health service at all that we could disable.