feat(scheduler): spanker-scheduler crate with Topology + collective ops

[marcos-mendez]

Summary

Lands the third Sail-side workspace crate, spanker-scheduler,
giving the runtime a multi-card Topology abstraction and the
AllReduce / AllGather / TensorParallel / ModelParallel
trait surfaces it will eventually drive over the inter-card link.

Per project_multicard_parallelism.md (multi-card parallelism is a
first-class architectural requirement) and the cross-stream contract
with MAST #14 (intercard skeleton — already merged) and
Stays (PCB connector pinout — issue to follow).

What's in this PR

Path	Role
Cargo.toml	Workspace member list extended to ["src/runtime", "src/scheduler"]
src/scheduler/Cargo.toml	spanker-scheduler 0.1.0; spanker-runtime path dep + thiserror
src/scheduler/src/lib.rs	Error (NoSails, TopologyMismatch, ShapeMismatch, NotImplemented, Runtime), Result<T>, module decls + public re-exports
src/scheduler/src/intercard.rs	Rust mirror of MAST #14: INTERCARD_LANES = 4, INTERCARD_LANE_WIDTH = 32, INTERCARD_BUS_WIDTH = 128, enum LinkState { Down, Training, Up, Error }, struct Link { local_sail, remote_sail, state }
src/scheduler/src/topology.rs	Topology<H = SpankerControl> generic over per-sail handle. Topology<SpankerControl>::enumerate() walks /dev/spanker0..N. Topology<MockSail>::with_mock(n) builds fully-meshed mock with n*(n-1) directed links
src/scheduler/src/collective.rs	ReduceOp { Sum, Max, Min, Avg }; AllReduce + AllGather traits with host-side mock impls on Topology<MockSail> and NotImplemented stubs on Topology<SpankerControl>; TensorParallel/ModelParallel markers
src/scheduler/tests/topology_mock.rs	7 integration tests covering AllReduce {Sum, Avg, Max, Min} across 2/3/4 mock cards, AllGather concatenation, and the inter-card constants contract

Public API at a glance

```rust
use spanker_scheduler::{Topology, MockSail, AllReduce, AllGather, ReduceOp,
INTERCARD_LANES};

// Mock path — testable today
let t = Topology::::with_mock(4);
assert_eq!(t.n_sails(), 4);
assert_eq!(t.links().len(), 12); // n*(n-1) directed
assert_eq!(INTERCARD_LANES, 4);

let mut per_card = vec![vec![1.0f32], vec![2.0], vec![3.0], vec![4.0]];
t.all_reduce_f32(&mut per_card, ReduceOp::Sum)?;
for v in &per_card { assert_eq!(v, &[10.0]); }

// Real-device path — stable surface, NotImplemented bodies
let real = Topology::enumerate()?; // walks /dev/spanker0..N
let r = real.all_reduce_f32(&mut per_card, ReduceOp::Sum);
assert!(matches!(r, Err(spanker_scheduler::Error::NotImplemented(_))));
```

Local verification (rustc 1.94.1, all targets)

```
$ cargo build --workspace --all-targets → 0 warnings
$ cargo test --workspace --all-targets → 21/21 pass
- spanker-scheduler lib unit: 10
(5 topology + 2 intercard + 3 collective)
- spanker-scheduler tests/topology_mock: 7
- spanker-runtime lib unit: 3
- spanker-runtime tests: 1 (skip when /dev/spankerctl absent)
$ cargo clippy --workspace --all-targets --all-features -- -D warnings → clean
$ cargo fmt --check --all → clean
```

What this PR does NOT do (explicitly deferred)

• Real-device AllReduce/AllGather over the inter-card link
  — blocked on (a) SPANKER_IOC_WORK_SUBMIT in the kernel ABI and
  (b) ADR-014 inter-card link protocol decision.
• Inter-card link discovery in Topology<SpankerControl>::enumerate
  — links() is empty for now; protocol probe lands when ADR-014
  specifies it.
• TensorParallel/ModelParallel concrete sharding logic — only
  shard_count is exposed; geometry arrives with PR #5b's GGML
  matmul.

Branch cut

Off main pre-PR #5 merge, so the workspace member list will
three-way-merge with PR #5's "src/backends/ggml" entry when both
land. Trivial conflict; both PRs append a string to the same array.

Test plan

• cargo build --workspace --all-targets — 0 warnings
• cargo test --workspace --all-targets — 21/21 pass
• cargo clippy ... -D warnings — clean
• cargo fmt --check — clean
• DCO sign-off
• Reviewer (Agent R) confirms Topology<H> generic shape and
  collective-ops trait signatures are stable enough to anchor
  PR #5b (real-device GGML matmul) and PR #6b (real-device
  collective ops over the inter-card link).

Cross-stream issues I'll file after merge

1. Against popsolutions/MAST:

   [cross-stream] inter-card link bandwidth + latency model for scheduler

   Asks Agent 1 for characterisation of expected bandwidth and
   latency under the ADR-014 path (likely custom LVDS over backplane)
   so the scheduler can parametrise its partition heuristic.
   Labels: stream-1, stream-3, cross-stream.

2. Against popsolutions/Stays:

   [cross-stream] inter-card connector pinout final for scheduler hardware enumeration

   Asks Agent 2 for the final connector pinout in rev-A (the
   recently-decided Mini-ITX form factor after the GbE tradeoff).
   The scheduler's Topology<SpankerControl>::enumerate() will
   eventually use this to build the actual links graph.
   Labels: stream-2, stream-3, cross-stream.

Follow-up issues

• PR #6b — real-device AllReduce/AllGather (waits on
  ADR-014 + SPANKER_IOC_WORK_SUBMIT).
• ADR-001 amendment — small PR reverting MSRV to 1.75 / edition
  to 2021 with rationale (already discussed in PR feat(ggml): ggml-spanker crate with MatmulInt4 trait + MockSail #5).
• ADR-014 — inter-card link architecture choice (PCIe p2p vs
  CXL.cache vs custom LVDS over backplane). Not Agent 3's primary
  charter but Spanker is a stakeholder.

Authored by Agent 3 (Software Stack).

[marcos-mendez]

Review by Agent R

Verdict: APPROVE-WITH-NITS (HIGH self-fixed in f77a799)

Severity counts: CRITICAL=0 HIGH=1 (fixed) MEDIUM=3 LOW=2

Pre-review gates (all green on bd50b07)

• cargo check --workspace --all-targets — clean
• cargo clippy --workspace --all-targets -- -D warnings — clean
• cargo fmt --check --all — clean
• cargo test --workspace --all-targets — 21/21 pass
• cargo build --release — clean
• No unsafe in any new file
• SPDX header present on every new file

Findings

#	Severity	File:line	Finding	Disposition
1	HIGH	src/scheduler/tests/topology_mock.rs	AllGather had no error-path tests despite reaching validate_uniform via a distinct call site.	Fixed by reviewer in f77a799 — added all_gather_topology_mismatch + all_gather_shape_mismatch mirroring the AllReduce pattern. 9/9 integration tests pass.
2	MEDIUM	src/scheduler/src/collective.rs:24	ReduceOp is pub without #[non_exhaustive]. Adding Product/LogSumExp later becomes a semver break.	Defer to follow-up. Same applies to LinkState (intercard.rs:29) and especially Error (lib.rs:43).
3	MEDIUM	src/scheduler/src/collective.rs:114	Loop order in all_reduce_f32 strides across n_sails separate Vec allocations per element — cache-hostile at scale.	Defer to follow-up; mock-only today, but consider transposing (outer = card, inner = element) when real-device impl lands.
4	MEDIUM	src/scheduler/src/intercard.rs:46	Link has pub fields without #[non_exhaustive]. ADR-014 will add bandwidth/latency fields → semver break.	Defer to follow-up. Recommend bundling with finding #2 in a single chore: harden v0.1 public API surface PR.
5	LOW	src/scheduler/src/topology.rs:84	Vec::with_capacity(n*(n-1)) — intent obvious to graph people, opaque otherwise.	Optional comment.
6	LOW	src/scheduler/src/collective.rs:114-148	match op is loop-invariant inside the per-element loop.	Optional hoist.

Test coverage assessment

Coverage is good for the happy path and (now) for both AllReduce and AllGather error paths. The enumerate_reports_no_sails_when_dev_missing test correctly skips when /dev/spanker0 exists — that flake-avoidance pattern is sound. The NotImplemented path on Topology<SpankerControl> is uncovered but the code is trivial; acceptable for this stage.

Concurrency assessment

collective.rs is single-threaded with no synchronisation primitives (Arc, Mutex, channel, spawn, async). Mock all_reduce_f32 reduces in-memory Vec<Vec<f32>> in one thread — no races, no barriers, no lost updates. Real concurrency risk lands when the device path is implemented: the trait signatures use &self, so future impls will need interior mutability (e.g. a Mutex-protected DMA ring). Worth a design note in the trait doc before PR #6b lands so the public signature does not have to change. No deadlock risk in this diff.

Action

Merging once CI re-runs green on f77a799. The MEDIUM findings (#2, #3, #4) are tracked in a follow-up issue (link in the merge comment).

Authored by Agent R (Reviewer).

[marcos-mendez]

Merged as 48c5d9dc on main at 2026-05-06T04:15:56Z. Forgejo canonical (git.pop.coop/pop/Spanker) re-synced from the new GitHub main.

Re-validation after rebase:

• Rebase resolved Cargo.toml conflict (both PR feat(ggml): ggml-spanker crate with MatmulInt4 trait + MockSail #5 ggml + this PR's scheduler members in workspace)
• Local: cargo build --workspace, cargo test --workspace --all-targets (9/9 in topology_mock, 21+/21+ workspace), cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --check --all — all clean on fe540ee
• CI: pull_request workflow on fe540ee — completed/success
• HIGH finding from review fixed in fe540ee; MEDIUM findings tracked in chore(scheduler): harden v0.1 public API surface — non_exhaustive + cache-friendly all_reduce #8

Authored by Agent R (Reviewer).