cluster-controller: ON REFRESH scheduling as a strategy

[aljoscha]

NOTE: This is the next PR in the cluster autoscaling stack. The full branch is at #36738.

Three commits. The first ports ON REFRESH scheduling into the cluster controller as a pure OnRefreshStrategy. The second flips the controller gates on in the sqllogictest binary's system-parameter defaults and migrates the legacy scheduling assertions to controller behavior, which is only green once the strategy exists and the legacy scheduler is gated off, so it rides in this PR. The third (from review) replaces the string-matched audit-reason attribution with a typed CreateReason the strategies attach per desired slot: the kernel merges per shape by explicit precedence, the adapter maps variants 1:1 (a new variant is a compile error instead of a silent Manual fallthrough), and the on-refresh reason embeds its window decision, making "the audit detail appears iff the create audits schedule" structural.

With this, the controller framework owns all three existing behaviours (baseline, graceful reconfiguration, on-refresh). The enable_cluster_controller gate defaults on, so this flips scheduling ownership to the controller out of the box. The legacy cluster_scheduling.rs policy stays as the gate-off break-glass fallback: check_scheduling_policies and handle_scheduling_decisions no-op while the gate is on, so the two never both write a scheduled cluster's replica set. The legacy path is removed in the final cleanup PR.

Strategy

The strategy contributes one replica at the cluster's realized shape while the cluster is inside a refresh window and nothing otherwise. The window decision (an MV still needs a refresh, or is estimated to still need Persist compaction) is ported verbatim from check_refresh_policy. A scheduled cluster's replication_factor is the controller's domain: ALTER ... SET (SCHEDULE = <non-MANUAL>) now normalizes it to 0 at planning time, exactly like CREATE CLUSTER does, and the strategy self-heals a stale non-zero value at runtime, so no migration is needed. The implicit baseline contributes nothing on a scheduled cluster, leaving the on-refresh strategy as the sole replica-set contributor there.

Signals

The refresh-window signals (read timestamp, compaction estimate, and the write frontiers and schedules of the bound REFRESH MVs) are pulled through a new refresh_window_inputs ctx method, on demand and only for scheduled clusters, the same pay-for-what-you-use way as hydration. The catalog- and storage-derived inputs are gathered on the coordinator loop, but the oracle read timestamp is fetched on a spawned task that completes the reply, so the serial coordinator loop never blocks on the oracle round-trip, the same split the legacy check_refresh_policy makes. The MV write frontier is carried as a full Antichain<Timestamp> to preserve the exact frontier semantics, which adds a timely dependency to the pure crate. The cluster schedule is added to the compare-and-append witness so a concurrent SET (SCHEDULE = ...) rejects an in-flight on-refresh decision.

Schedule vs graceful reconfiguration

The schedule decides which strategy owns the replica set, so the two ownership regimes are kept from overlapping: a SCHEDULE change is refused while a reconfiguration record is in flight (a new AlterClusterScheduleWhileReconfiguring error, mirroring the replication-factor guard), and a config-shape ALTER on a scheduled cluster takes the direct (non-record) path, per the design doc: the realized config advances immediately and the controller reconciles the in-window replica to the new shape on its next tick. A record on a scheduled cluster can then only pre-date the schedule (pre-upgrade catalog state); for that case the on-refresh strategy defers its replication-factor normalization until the record settles, keeping new_replication_factor single-writer at any moment.

Audit attribution

A window-open create carries a new ReplicaCreateDropReason::OnRefresh(RefreshWindowDecision), which maps to the schedule reason and renders the RefreshWindowDecision into the existing SchedulingDecisionsWithReasonsV2 scheduling_policies detail, the same detail the legacy scheduler records. So an on-refresh create keeps the window reasons behind it (which MVs needed a refresh or a compaction time, and the hydration-time estimate), while the matching window-close drop is an ordinary retired drop with no detail. The refresh-strategies user doc is updated for the new drop shape.

Tests

On-refresh kernel and seam tests in mz-cluster-controller (window in/out decision, the read-ts boundary, the hydration estimate and compaction windows for both REFRESH AT and REFRESH EVERY schedules, replication-factor normalization and its deferral under an in-flight record, the schedule compare-and-append witness, audit-detail merging across strategies, and end-to-end create/drop and record-settlement through a fake ctx), the refresh_window_decision_to_audit_log mapping in mz-adapter, an ON REFRESH section in cluster-controller.td covering scheduling, the schedule-mid-record guard, the direct-path resize, and the ALTER-side normalization. materialized-view-refresh-options.td and materialized_views.slt are retargeted to read the replica set (replication_factor now reads 0 on a scheduled cluster).

Motivation

Implements ON REFRESH scheduling as a strategy within the controller framework, per doc/developer/design/20260522_cluster_autoscaling.md.

User-visible behavior changes: mz_clusters.replication_factor reads 0 for a scheduled cluster (mz_cluster_replicas is authoritative for what is running), window-close replica drops are audited retired instead of schedule, ALTER ... SET (SCHEDULE = <non-MANUAL>) normalizes the replication factor to 0, ALTER ... SET (SCHEDULE = MANUAL) leaves the cluster at replication factor 0 until the user sets one explicitly, and a SCHEDULE change is refused while a cluster reconfiguration is in progress.

[aljoscha]

this makes it sound generic, but why let the window decision think leak into here?

[aljoscha]

Fair, fixed in the latest push: the kernel no longer knows the window-decision type. Strategies now attach an opaque AuditDetail payload (a ctx enum, today one variant, OnRefresh(RefreshWindowDecision)) to a desired slot, the kernel merges it per shape without interpreting it, and the environment matches on the variant when it renders the audit event. The window decision only exists in the on-refresh strategy, the ctx vocabulary, and the coordinator's audit conversion.

[aljoscha]

this whole thing make me think: why even match on the string reason, and not just have the strategy create a ReplicaCratedDropReason with the right enum variant? Side-steps this dance here, no? Only take a look and report in our chat, don't change it yet.

[aljoscha]

Done, pushed as a separate commit on top ("cluster-controller: typed CreateReason instead of string-matched audit reasons"). Strategies attach a CreateReason per desired slot (Baseline, GracefulReconfiguration, HydrationBurst, OnRefresh(RefreshWindowDecision)), the kernel merges per shape by an explicit precedence() rank, and the adapter maps variants 1:1 with an exhaustive match, so reason_from_strategies and the string dance are gone. Bonus: embedding the window decision in the OnRefresh variant made the "detail iff schedule reason" invariant structural and deleted the AuditDetail side-channel from the previous round, and ReplicaCreateDropReason::OnRefresh tightened to a non-optional decision. Creates no longer carry the full name list (nothing consumed it), and the now-dead *_STRATEGY_NAME/Strategy::name() machinery is removed.

[aljoscha]

@mtabebe you might want to take a peek as well since you saw all the other PRs in the stack, I tagged @ggevay because he worked on the cluster scheduling feature AFAIK

[aljoscha]

Why don't all callers re-use the ClusterScheduling variant here?

[aljoscha]

Deliberate: reusing ClusterScheduling would not delete a conversion, it would re-target it through the legacy intermediate (RefreshWindowDecision -> SchedulingDecision::Refresh(RefreshDecision { cluster_on: true, .. }) -> blob instead of directly -> blob). The legacy shape also encodes semantics the controller does not have: cluster_on exists because the legacy scheduler audits off-decisions on drops (controller drops are uniformly retired), and the per-policy Vec exists for the multi-policy design that only ever grew one policy (reasons_to_audit_log_reasons asserts the singleton via into_element). And it would invert the cleanup arrow: today ClusterScheduling + SchedulingDecision/RefreshDecision die with the legacy scheduler in the final cleanup PR, leaving one variant and one conversion. Added a doc sentence on the variant saying exactly this. The one real wart is that both paths render the same RefreshDecisionWithReasonV2 independently (drift risk while both are live), happy to extract a shared render helper if you want, though it also just dies in the cleanup PR.