feat: NVL576 - Multi Level Topology Support - Design and Prototype

[ravisoundar]

Description

Design and Prototype for the NVL576 multi level block topology support.
Addresses #415

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.
• All commits are signed off per DCO (git commit -s).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ravisoundar]

/ok-to-test 24e80de

[codecov]

Codecov Report

❌ Patch coverage is 92.04545% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.60%. Comparing base (8468abe) to head (507afeb).
⚠️ Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/translate/block_tree.go	84.44%	6 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #416      +/-   ##
==========================================
+ Coverage   72.15%   73.60%   +1.44%     
==========================================
  Files          89       94       +5     
  Lines        5689     6349     +660     
==========================================
+ Hits         4105     4673     +568     
- Misses       1382     1434      +52     
- Partials      202      242      +40     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[greptile-apps]

Greptile Summary

This PR introduces multi-level NVLink domain hierarchy support for NVL576 clusters, extending the block topology engine to represent up to three named grouping levels (Level1/Level2/Level3) above the accelerator domain. The tree builder is refactored into three explicit phases — per-domain packing, level-by-level aggregation, and a root fallback — so that Slurm's BlockSizes can express multi-tier boundaries (e.g. 9,144 for 9-node racks grouped into 16-rack NVL576 Scalable Units) without hand-editing topology.conf.

• Data model: HostInfo gains Level1/Level2/Level3 string fields; DomainMap.GetLevelInfo provides membership maps at each tier; pkg/models reads the corresponding network.topology.nvidia.com/level* labels from capacity-block YAML entries.
• Tree builder: buildBlockTree in block_tree.go is restructured into Phase 1 (domain packing), Phase 2 (Level3→Level2→Level1 aggregation using fanout from blockSizes), and Phase 3 (root fallback for backward compat); per-node identification is unified behind a new levelIdentifier() interface method.
• Tests & model: TestComplementNVL576 validates end-to-end output for a 2-room, 32-rack model with 3 absent racks, and a new nvl576.yaml simulation fixture drives the test.

Confidence Score: 4/5

Safe to merge for the documented NVL576 BlockSizes=[9,144] use case, but has a node-loss defect when Level fields are only partially populated.

The three-phase tree builder correctly handles the target NVL576 configuration and all existing tests pass. However, when at least one domain has Level3 set but another domain in the same DomainMap does not, the un-labelled domain is silently dropped from currentNodes in Phase 2 and never appears in the emitted topology.conf with no error or warning. This would cause Slurm to be unaware of those nodes.

pkg/translate/block_tree.go — the Phase 2 levelMap construction and Phase 3 fallback both need a check for domain nodes that were in domainNodes but are absent from any level group.

Important Files Changed

Filename	Overview
pkg/translate/block_tree.go	Core multi-phase tree builder refactored for NVL576 multi-level aggregation. Two issues found: domain nodes without Level3 are silently dropped when any Level3 data is present; desiredGroupSize uses remaining[last] instead of remaining[0], skipping intermediate block size boundaries for blockSizes with 3+ entries.
pkg/topology/domain.go	Adds Level1/Level2/Level3 fields to HostInfo and a new GetLevelInfo method that groups domains by level value with sorted children. Logic is correct; the two-return signature (present, members) is clean.
pkg/models/model.go	Adds three new LabelTopologyLevel* constants and propagates level labels from capacity block entries into HostInfo via AddHostInfo. Change is minimal and correct.
pkg/topology/domain_test.go	Adds comprehensive TestGetLevelInfo table-driven tests covering all four levels, empty maps, absent levels, and partial level population. Coverage is thorough.
pkg/translate/block_complement_test.go	Adds TestComplementNVL576 end-to-end integration test validating the full block topology output for the NVL576 simulation model with BlockSizes=[9,144]. Expected output is correct and well-documented.
tests/models/nvl576.yaml	New NVL576 simulation model YAML with 2 rooms, 16 racks each (3 absent), correct block-level nvidia.com labels. Header comment incorrectly attributes topology.kubernetes.io/level* labels to the metadata used by the block engine.
docs/design/nvl576-multi-level-topology-sdd.md	New SDD accurately describes the three-phase algorithm and lists known limitations, but omits two: partial Level3 population causes silent node loss, and blockSizes[last] fanout formula skips intermediate block sizes for 3+ entry configs.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[buildBlockTree] --> B[Phase 1: packDomainNodes]
    B --> C{domCapacity >= blockSizes last?}
    C -- yes --> D[Return flat list as root children]
    C -- no --> E[Phase 2: loop levels 3 to 1]
    E --> F{remaining empty?}
    F -- yes --> G[Early return currentNodes as root children]
    F -- no --> H{GetLevelInfo present?}
    H -- absent --> I[Phase 3]
    H -- present --> J[Build nodesMap from currentNodes]
    J --> K[Build levelMap from members]
    K --> L[packAggregateNodes desiredGroupSize = remaining LAST / currentCapacity]
    L --> M[Update currentNodes and currentCapacity]
    M --> E
    I --> N{remaining empty after Phase 2?}
    N -- yes --> G
    N -- no --> O[Root fallback pack currentNodes under root]
    O --> P[Return aggregateBlockNode]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[buildBlockTree] --> B[Phase 1: packDomainNodes]
    B --> C{domCapacity >= blockSizes last?}
    C -- yes --> D[Return flat list as root children]
    C -- no --> E[Phase 2: loop levels 3 to 1]
    E --> F{remaining empty?}
    F -- yes --> G[Early return currentNodes as root children]
    F -- no --> H{GetLevelInfo present?}
    H -- absent --> I[Phase 3]
    H -- present --> J[Build nodesMap from currentNodes]
    J --> K[Build levelMap from members]
    K --> L[packAggregateNodes desiredGroupSize = remaining LAST / currentCapacity]
    L --> M[Update currentNodes and currentCapacity]
    M --> E
    I --> N{remaining empty after Phase 2?}
    N -- yes --> G
    N -- no --> O[Root fallback pack currentNodes under root]
    O --> P[Return aggregateBlockNode]

Loading

_{Reviews (3): Last reviewed commit: "feat: NVL576 - Multi Level Topology Supp..." | Re-trigger Greptile}

[ravisoundar]

/ok-to-test 507afeb

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pull-request-416.docs.buildwithfern.com/topograph

[dmitsh]

Are you mixing network fabric and accelerated domains?
"Level-N" isn't very descriptive. It may mean different things to different providers.
Also, I'd use map[string]string instead of a fixed number of levels. The key could resemble a k8s label name, which we could then use in the k8s engine.