docs(upstream): day-1 recon — LiteEth ECP5 SGMII status (production-mature, host-link-bandwidth-constraint flagged)#34
Merged
Conversation
Day-1 reconnaissance on the LiteEth + LiteICLink + 88E1512 SGMII path that rev-A adopts for its GbE host link (per ADR-001, after PCIe was deferred to rev-B). Status: production-mature, integration-only. - LiteEth MAC + IP/UDP stack: BSD-2-Clause, 9+ years in production. - SGMII path driven by liteiclink/serdes/serdes_ecp5.py wrapping the ECP5 DCU primitive — same wrapper rev-A uses for the inter-card link, validated at 1.25 Gbps SGMII on Versa-ECP5 + ECPIX-5. - 88E1512 Marvell PHY is the canonical transceiver; reused verbatim from the two -85F-class reference designs. - Real-world throughput: 800-940 Mbps measured (UDP iperf) — 80-94% of GbE line rate. - ECP5-5G suffix matters: SGMII 1.25 Gbps requires LFE5UM5G, which rev-A already targets. Cross-stream constraint flagged: the GbE host link is the slowest hop in rev-A's bandwidth hierarchy (~100 MB/s after IP/UDP overhead vs ~500 MB/s inter-card vs ~2 GB/s local DDR). The Spanker scheduler bandwidth model needs a third constant HOST_LINK_BW_BYTES_PER_SEC = 100_000_000. Filed as a cross-stream Spanker issue; Stream 3 owns the implementation. Updates 0001-rev-a-known-upstream-issues.md with a LiteEth section that captures the Day-1 finding and the cross-stream constraint. Authored by Agent 4 (Open FPGA Upstream Contributions). Signed-off-by: Marcos (Agent 4) <popsolutions.co@gmail.com>
marcos-mendez
added
stream-4
Member
Author
Review by Agent R — APPROVECI 2/2 SUCCESS (Docs/ADR + KiCad/ERC+DRC). Doc-only, mirrors prior recon format (yosys, prjtrellis, litedram). Headline: integration-only, production-mature. Versa-ECP5 + ECPIX-5 ship this exact path daily; 800-940 Mbps measured. No upstream contribution gap for rev-A. Cross-stream HOST_LINK_BW finding correctly surfaced as Spanker #21 (Stream 3 owns implementation). Bandwidth hierarchy now triple-pinned: HOST_LINK_BW (100 MB/s) < INTERCARD_BW (500 MB/s) < LOCAL_DDR_BW (2 GB/s). Merging via two-step. Forgejo sync follows. Authored by Agent R (Reviewer). |
marcos-mendez
added a commit
to popsolutions/Spanker
that referenced
this pull request
May 6, 2026
…loses #21) (#22) Add `HOST_LINK_BW_BYTES_PER_SEC = 100_000_000` (100 MB/s) to the bandwidth model, capturing the rev-A GbE host link as the third tier of the bandwidth hierarchy: Local DDR (per card) ~2.0 GB/s LOCAL_DDR_BW Inter-card (per direction) ~500 MB/s INTERCARD_BW Host link (GbE) ~100 MB/s HOST_LINK_BW (NEW) Source-of-truth: Stays `docs/upstream-contributions/2026-05-06-liteeth-ecp5-sgmii.md` (Stays PR #34, merged 2026-05-06). Community measurements on Versa-ECP5 and ECPIX-5 land at 800-940 Mbps UDP iperf3, i.e. 80-94 % of GbE line rate. The 100 MB/s number is the realistic post-IP/UDP/Ethernet-header steady-state ceiling. The host link is 5x slower than inter-card and 20x slower than local DDR — it is the dominant cost when collective ops must reach the host (model load, gradient checkpoint to host RAM, dataset streaming, prompt-embedding upload). ## Scope Minimal — per the issue spec's "if pick_strategy already handles this" branch: - `pick_strategy` is the per-token TP/MP decision and most decode tokens stay on-card; host-link cost is small per-token and only matters at session boundaries. - No callers exist today for a session-level cost-budget API, so introducing `bytes_per_second_per_token_estimate` would be speculative generality (YAGNI). Defer until the runtime needs it. - This PR keeps the public surface to a constant + module-level doctest update + tests. ## Tests 3 new unit tests in `bandwidth.rs`: - `host_link_bw_constant_matches_recon_doc` — pins value to 100_000_000 (guards against silent "round up to 125 MB/s line rate" drift). - `host_link_bw_is_slowest_hop` — pins the three-tier ordering HOST_LINK < INTERCARD < LOCAL_DDR. - `host_link_bw_is_inside_observed_range` — pins 80-125 MB/s envelope (community recon range, with line-rate ceiling). Plus the existing `constants_are_positive` test extended to cover the new constant. Module-level doctest in `bandwidth.rs` updated to demonstrate all three constants. Crate-root doctest in `lib.rs` updated to assert the three-tier ordering. ## Cargo gates - `cargo build -p spanker-scheduler`: green - `cargo test -p spanker-scheduler`: 27 unit + 9 integration + 6 doctests, all green (delta: +3 unit tests vs PR #19 baseline) - `cargo clippy -p spanker-scheduler --all-targets -- -D warnings`: green - `cargo fmt -p spanker-scheduler -- --check`: clean Refs: - #21 (this issue) - popsolutions/Stays#34 (LiteEth ECP5 SGMII recon, source-of-truth) - #17 (PR that landed initial 2-tier model) - #19 (PR that landed pick_strategy) Authored by Agent 3 (Software Stack — Spanker). Signed-off-by: Marcos <m@pop.coop> Co-authored-by: Marcos <m@pop.coop>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Day-1 reconnaissance on the LiteEth + LiteICLink + 88E1512 SGMII path that rev-A adopts for its GbE host link (per ADR-001, after PCIe was deferred to rev-B with CertusPro-NX).
enjoy-digital/liteeth+enjoy-digital/liteiclink(BSD-2-Clause). Same author (Florent Kermarrec) as LiteDRAM / LitePCIe / LiteX.liteiclink/serdes/serdes_ecp5.pySerDes wrapper rev-A uses for the inter-card link (4 lanes × 1.25 Gbps) is the same wrapper that drives the SGMII host link (1 lane × 1.25 Gbps).Cross-stream constraint flagged
The GbE host link is the slowest hop in rev-A's bandwidth hierarchy:
The Spanker scheduler bandwidth model needs a third constant:
HOST_LINK_BW_BYTES_PER_SEC = 100_000_000(100 MB/s after IP/UDP overhead).Filed as Spanker cross-stream issue: popsolutions/Spanker#21 — Stream 3 owns the implementation PR.
Files
docs/upstream-contributions/2026-05-06-liteeth-ecp5-sgmii.md(new, mirrors prior recon format)docs/upstream-contributions/0001-rev-a-known-upstream-issues.md(LiteEth section added between LitePCIe and "What is NOT in this survey")License + DCO
CC-BY-SA-4.0on the new doc, consistent with sibling recons.-s).docs(upstream): day-1 recon — LiteEth ECP5 SGMII status.Test plan
bandwidth.rsconstants.Out of scope
bandwidth.rsimplementation (filed as [cross-stream] Add HOST_LINK_BW_BYTES_PER_SEC to bandwidth.rs (rev-A GbE 100 MB/s) Spanker#21; Stream 3 owns the PR).Authored by Agent 4 (Open FPGA Upstream Contributions).