From 664c0018273b5f1f39f58c3a6c410c8a2c021937 Mon Sep 17 00:00:00 2001 From: Ravi Shankar Date: Tue, 21 Jul 2026 18:04:55 -0700 Subject: [PATCH] feat: NVL576 - Multi Level Topology Support - Design and Prototype Signed-off-by: Ravi Shankar --- .../design/nvl576-multi-level-topology-sdd.md | 277 ++++++++++++++++++ go.mod | 2 +- go.sum | 4 +- pkg/models/model.go | 12 +- pkg/topology/domain.go | 50 ++++ pkg/topology/domain_test.go | 143 +++++++++ pkg/translate/block_complement_test.go | 103 +++++++ pkg/translate/block_tree.go | 149 ++++++++-- tests/models/nvl576.yaml | 276 +++++++++++++++++ 9 files changed, 990 insertions(+), 26 deletions(-) create mode 100644 docs/design/nvl576-multi-level-topology-sdd.md create mode 100644 tests/models/nvl576.yaml diff --git a/docs/design/nvl576-multi-level-topology-sdd.md b/docs/design/nvl576-multi-level-topology-sdd.md new file mode 100644 index 00000000..8fb6499c --- /dev/null +++ b/docs/design/nvl576-multi-level-topology-sdd.md @@ -0,0 +1,277 @@ +# Software Design Document: NVL576 Multi-Level Topology Support + +## Status + +In Progress + +## Summary + +Extend Topograph's block topology engine to represent multi-level NVLink domain +hierarchies — specifically NVL576 Scalable Units (SUs) — so that Slurm and +Kubernetes schedulers can distinguish placement within a rack from placement +within a full NVL576 SU. The feature introduces up to three named hierarchy +levels (Level1, Level2, Level3) carried in `topology.HostInfo`, and a new +multi-phase tree-building algorithm in `pkg/translate` that converts those levels +into contiguous, consistently-padded block groups. + +## Background + +### NVL576 hardware topology + +An NVL576 Scalable Unit (SU) contains 16 NVL36 racks. Each rack holds 9 compute +nodes (36 GPUs via NVLink). The logical grouping is therefore: + +``` +SU (NVL576) +└── rack-01 … rack-16 (16 racks per SU) + └── node-01 … node-09 (9 nodes per rack, 36 GPUs) +``` + +A cluster may have multiple SUs under a common network domain (building, data +center zone, etc.), making the full hierarchy up to four levels deep: +Level1 (datacenter) → Level2 (building/SU group) → Level3 (SU/room) → +Level4 (rack/accelerator domain) → nodes. + +### Existing limitation + +Before this change Topograph treated each accelerator domain (rack) independently. +`BlockSizes` in `topology.conf` could express two tiers (base block and top +level), but the tree builder had no awareness of intermediate groupings. As a +result: + +- Slurm could not differentiate intra-rack placement from intra-SU placement. +- Incremental deployment or maintenance (missing racks) could shift base-block + positions, breaking Slurm's position-based aggregate inference. +- An NVL576 cluster required manual `topology.conf` authoring to express the + 16-rack SU boundary. + +## Goals + +- Emit `BlockSizes=9,144` for NVL576 deployments from topology provider data + alone, without hand-editing `topology.conf`. +- Emit exactly 16 consecutive base-block entries per SU, inserting empty + `BlockName` placeholder entries for missing or offline racks. +- Carry Level1/Level2/Level3 hierarchy data from providers through the canonical + `topology.Graph` and into the block topology engine. +- Preserve full backward compatibility: clusters that do not use Level fields + must produce identical output to the pre-change behavior. + +## Non-Goals + +- Custom rack naming schemes (e.g., `nvl576_d1_r05`) are a provider + responsibility and are out of scope for the translate layer. +- Kubernetes label changes for NVL576 topology (separate from the existing + `network.topology.nvidia.com/accelerator` and `tier-N` labels) are not part + of this change. +- Changes to any provider (`aws`, `gcp`, `oci`, `netq`, etc.); those providers + must opt in separately by populating the Level fields. + +## Data Model Changes + +### `topology.HostInfo` + +`pkg/topology/domain.go` — three new string fields: + +```go +type HostInfo struct { + Domain string // accelerator/NVLink domain name (rack) + InstanceID string + HostName string + Level1 string // optional: top-level grouping (e.g. datacenter) + Level2 string // optional: mid-level grouping (e.g. building or SU group) + Level3 string // optional: lowest named level above the domain (e.g. SU / room) +} +``` + +Level fields are optional and populated only when the provider or simulation +model has corresponding topology data. Their absence leaves behavior unchanged. + +### `DomainMap.GetLevelInfo` + +A new method on `DomainMap` (`pkg/topology/domain.go`) returns, for a given +level number (1–4), whether any hosts carry that level's value and a map from +each group name to its sorted list of child names at the level below: + +```go +func (m DomainMap) GetLevelInfo(level int) (present bool, members map[string][]string) +``` + +`level=3` groups hosts by `Level3` value and returns child domain IDs. +`level=2` groups by `Level2` and returns child `Level3` values. +`level=1` groups by `Level1` and returns child `Level2` values. + + +### Simulation model YAML + +`pkg/models/model.go` already reads `network.topology.nvidia.com/level1`, +`network.topology.nvidia.com/level2`, and `network.topology.nvidia.com/level3` +labels from the capacity-block label map and writes them into `HostInfo` when +calling `DomainMap.AddHostInfo`. No structural changes to the YAML schema are +needed; providers assign the labels on their `Instance` objects, and the model +loader propagates them automatically. + +## Algorithm: `buildBlockTree` + +The tree builder in `pkg/translate/block_tree.go` is refactored into three +explicit phases. + +### Phase 1 – Domain packing + +`packDomainNodes` splits each accelerator domain's hosts into base blocks of +`baseBlockSize` nodes, pads each domain to a multiple of `groupSize` base +blocks, and wraps the result in one `aggregateBlockNode` per domain. `groupSize` +is the smallest power of two such that `groupSize × baseBlockSize ≥ maxDomainSize` +across all domains. + +If every domain's padded capacity already meets `blockSizes[last]`, the flat list +of domain nodes is returned immediately as the tree root's children. + +### Phase 2 – Level aggregation (Level3 → Level2 → Level1) + +When `HostInfo` carries Level3/Level2/Level1 values, the builder iterates over +levels from closest to the node outward: + +``` +for level in [3, 2, 1]: + remaining = getRemainingBlocks(blockSizes, currentCapacity) + if len(remaining) == 0: break + present, members = domains.GetLevelInfo(level) + if not present: break + desiredGroupSize = blockSizes[last] / currentCapacity + if desiredGroupSize <= 0: break + + nodesMap = { node.levelIdentifier() → node for node in currentNodes } + levelMap = { levelName → [nodesMap[child] for child in members[levelName]] } + packed = packAggregateNodes(levelMap, completed, desiredGroupSize) + currentNodes = packed + currentCapacity = desiredGroupSize * currentCapacity // slot capacity, not live-node count + completed.append(currentCapacity) +``` + +Key design decisions: + +**Fanout from `blockSizes`, not from observed maximum.** `desiredGroupSize` is +`blockSizes[last] / currentCapacity`, not `maxNodes / currentCapacity`. This +ensures every group is padded to the declared block-size boundary. In the NVL576 +example, 15 active racks in a room produce `desiredGroupSize = 144/9 = 16`; the +room is padded to 16 rack slots regardless of how many racks are currently +populated. + +**Capacity from slot arithmetic.** After packing, `currentCapacity` is updated +to `desiredGroupSize × prevCapacity` — the slot capacity of one packed group. +Using slot arithmetic ensures the next tier's fanout is computed against the declared block-size boundary. + +**Membership from `GetLevelInfo`.** Level names (e.g. `"room-1"`, `"room-2"`) +and their children (sorted rack IDs) come from `GetLevelInfo`. The builder looks +up each child in `nodesMap` by `levelIdentifier()`. Domain nodes carry their +domain ID as their identifier; packed groups carry the level-name key set by +`packAggregateNodes` when it creates the outer aggregate per map key. + +**Early exit.** The loop exits as soon as `remaining` becomes empty (blockSizes +satisfied) or a level is absent from the DomainMap (no further hierarchy to +climb). + +### Phase 3 – Fallback "root" aggregation + +When no Level fields are present in the DomainMap, Phase 2 exits on the first +iteration and all domain nodes are packed under a single `"root"` key, preserving +the pre-change behavior exactly. The same path handles clusters where level +processing exhausts the available levels before satisfying `blockSizes`: the +remaining nodes are packed under `"root"` using `blockSizes[last] / currentCapacity`. + +### Empty placeholder handling + +`newAggregateBlock` (called by `packAggregateNodes`) pads incomplete groups up +to `desiredGroupSize` using `newEmptyAggregateBlock`. Each empty slot becomes an +empty `baseBlockNode` (no ID, no leaves), which `baseBlockToBlockInfo` converts +to a `blockInfo` with an empty name and no nodes. `toBlockTopology` writes these +as bare `BlockName=blockNNN` entries in `topology.conf` — exactly the placeholder +semantics required by Slurm. + +## NVL576 Example + +**Topology:** building-1 → room-1 (16 rack switches, 14 with nodes), room-2 +(15 rack switches, 15 with nodes). Each rack has 9 nodes. `BlockSizes=[9, 144]`. + +**Phase 1:** 29 domain aggregate nodes, each with `nodeCount = 9`. +`domCapacity = 9`, `remaining = [144]`. + +**Phase 2 Level3:** +- `GetLevelInfo(3)` → `members = { "room-1": [14 rack IDs], "room-2": [15 rack IDs] }`. +- `desiredGroupSize = 144/9 = 16`. +- room-1: 14 real racks packed into a group of 16 → 2 empty placeholder slots. +- room-2: 15 real racks packed into a group of 16 → 1 empty placeholder slot. +- `currentCapacity = 16 × 9 = 144 = blockSizes[last]`. + +**Phase 2 Level2 check:** `remaining = getRemainingBlocks([9,144], 144) = nil` → break. + +**Phase 3:** `remaining` is empty → early return; no "root" wrapping tier. + +**Output** (32 blocks): + +``` +# block001=rack-1-01 +BlockName=block001 Nodes=node[0001-0009] +... +# block014=rack-1-16 +BlockName=block014 Nodes=node[0136-0144] +BlockName=block015 ← empty placeholder (rack-1-03 missing) +BlockName=block016 ← empty placeholder (rack-1-13 missing) +# block017=rack-2-01 +BlockName=block017 Nodes=node[0145-0153] +... +# block031=rack-2-16 +BlockName=block031 Nodes=node[0280-0288] +BlockName=block032 ← empty placeholder (rack-2-11 missing) +BlockSizes=9,144 +``` + +## Known Limitations + +### Non-contiguous level hierarchies + +Phase 2 stops at the first absent intermediate level and does not apply any outer +levels above it. `GetLevelInfo(N)` returns `Level{N+1}` values as child names; if +an intermediate level is absent, those child lists are empty and `packAggregateNodes` +would produce all-placeholder groups, so the loop breaks instead of continuing. + +As a result, a topology that sets `Level3` and `Level1` but omits `Level2`, or one +that only sets `Level2`, will fall through to Phase 3 root aggregation without the +outer level being applied. A `klog.Warningf` is emitted when data at an outer level +is dropped, so the condition is not silent. + +Providers populating Level fields should use contiguous level assignments +(Level3 → Level2 → Level1 without gaps). + +### Placeholder placement + +Empty placeholder blocks are appended at the end of each level group, not inserted +at their logical position within a sorted name sequence. If a room has 16 racks +(rack-01 through rack-16) and rack-05 is absent, the placeholder appears after +rack-15 rather than in the fifth slot. + +The current design has no per-element ordering information; all ordering within a +level is alphabetical by name, and missing entries fill trailing slots. Positional +ordering within groups would require providers to supply an explicit slot index and +a corresponding design update to the tree builder. + +## Backward Compatibility + +The Level fields in `HostInfo` are zero-valued strings by default. `GetLevelInfo` +returns `present=false` when no host carries a given level value. Phase 2's first +iteration immediately breaks on `!present`, and Phase 3 falls through to the +original single-tier `"root"` aggregation with `desiredGroupSize = blockSizes[last] +/ domCapacity`. All existing tests pass without modification. + + +## Test Plan + +- `TestComplementNVL576` in `pkg/translate/block_complement_test.go`: loads the + `nvl576.yaml` simulation model, generates block topology with `BlockSizes=[9, 144]`, + and asserts 32 blocks — 16 per room, with correct node ranges and placeholder + entries for the 3 missing racks. +- All pre-existing complement tests (`TestComplementMissingBaseBlock`, + `TestComplementMissingLeafSegment`, `TestComplementKeepsSeparateAccelerators`, + etc.) continue to pass, verifying backward compatibility. +- `pkg/topology` unit tests cover `GetLevelInfo` across all four level values + including absent levels. diff --git a/go.mod b/go.mod index 0c15e1f9..0add231a 100644 --- a/go.mod +++ b/go.mod @@ -107,7 +107,7 @@ require ( golang.org/x/oauth2 v0.36.0 // indirect golang.org/x/sys v0.46.0 // indirect golang.org/x/term v0.44.0 // indirect - golang.org/x/text v0.38.0 // indirect + golang.org/x/text v0.39.0 // indirect golang.org/x/time v0.15.0 // indirect google.golang.org/genproto v0.0.0-20260414002931-afd174a4e478 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260414002931-afd174a4e478 // indirect diff --git a/go.sum b/go.sum index e4e5ebbd..cc4801d2 100644 --- a/go.sum +++ b/go.sum @@ -223,8 +223,8 @@ golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw= golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc= golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y= -golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE= -golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4= +golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus= +golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= diff --git a/pkg/models/model.go b/pkg/models/model.go index cb09022b..7db0af6f 100644 --- a/pkg/models/model.go +++ b/pkg/models/model.go @@ -32,6 +32,9 @@ import ( const ( LabelTopologyRegion = "topology.kubernetes.io/region" LabelTopologyZone = "topology.kubernetes.io/zone" + LabelTopologyLevel1 = "network.topology.nvidia.com/level1" + LabelTopologyLevel2 = "network.topology.nvidia.com/level2" + LabelTopologyLevel3 = "network.topology.nvidia.com/level3" ) // Switch is a switch vertex in a simulation model YAML tree (tests/models). @@ -353,7 +356,14 @@ func (model *Model) ToGraph(instances []topology.ComputeInstances) (*topology.Gr for hostName, instance := range model.Nodes { if domain := instance.AcceleratorID(); domain != "" { instanceID := getInstanceID(hostName) - domainMap.AddHost(domain, instanceID, instance2node[instanceID]) + domainMap.AddHostInfo(&topology.HostInfo{ + Domain: domain, + InstanceID: instanceID, + HostName: instance2node[instanceID], + Level1: instance.Labels[LabelTopologyLevel1], + Level2: instance.Labels[LabelTopologyLevel2], + Level3: instance.Labels[LabelTopologyLevel3], + }) } } diff --git a/pkg/topology/domain.go b/pkg/topology/domain.go index 6cbc9aaa..f61f1cda 100644 --- a/pkg/topology/domain.go +++ b/pkg/topology/domain.go @@ -18,6 +18,7 @@ package topology import ( "fmt" + "slices" "strings" "k8s.io/klog/v2" @@ -27,6 +28,9 @@ type HostInfo struct { Domain string InstanceID string HostName string + Level1 string + Level2 string + Level3 string } // DomainMap maps accelerator domain name to host metadata. @@ -64,3 +68,49 @@ func (m DomainMap) AddHostInfo(hostInfo *HostInfo) { m[hostInfo.Domain] = map[string]*HostInfo{hostInfo.HostName: hostInfo} } } + +func (m DomainMap) GetLevelInfo(level int) (present bool, members map[string][]string) { + children := make(map[string]map[string]struct{}) + for _, hosts := range m { + for _, host := range hosts { + var key, child string + switch level { + case 1: + key = host.Level1 + child = host.Level2 + case 2: + key = host.Level2 + child = host.Level3 + case 3: + key = host.Level3 + child = host.Domain + case 4: + key = host.Domain + child = host.HostName + default: + continue + } + if len(key) > 0 { + if _, ok := children[key]; !ok { + children[key] = make(map[string]struct{}) + } + if len(child) > 0 { + children[key][child] = struct{}{} + } + } + } + } + if len(children) == 0 { + return false, nil + } + members = make(map[string][]string, len(children)) + for key, childMap := range children { + childList := make([]string, 0, len(childMap)) + for child := range childMap { + childList = append(childList, child) + } + slices.Sort(childList) + members[key] = childList + } + return true, members +} diff --git a/pkg/topology/domain_test.go b/pkg/topology/domain_test.go index a764f43e..350be28f 100644 --- a/pkg/topology/domain_test.go +++ b/pkg/topology/domain_test.go @@ -22,6 +22,149 @@ import ( "github.com/stretchr/testify/require" ) +// nvl576DomainMap builds a small DomainMap that mirrors the NVL576 level +// hierarchy used across tests: +// +// Level1 = "dc-1" +// Level2 = "building-1" +// Level3 = "room-1" (rack-a: 3 nodes) or "room-2" (rack-b: 2 nodes, rack-c: 3 nodes) +// Domain = "rack-a" | "rack-b" | "rack-c" +func nvl576DomainMap() DomainMap { + m := NewDomainMap() + for _, h := range []HostInfo{ + {Domain: "rack-a", HostName: "n01", InstanceID: "i01", Level1: "dc-1", Level2: "building-1", Level3: "room-1"}, + {Domain: "rack-a", HostName: "n02", InstanceID: "i02", Level1: "dc-1", Level2: "building-1", Level3: "room-1"}, + {Domain: "rack-a", HostName: "n03", InstanceID: "i03", Level1: "dc-1", Level2: "building-1", Level3: "room-1"}, + {Domain: "rack-b", HostName: "n04", InstanceID: "i04", Level1: "dc-1", Level2: "building-1", Level3: "room-2"}, + {Domain: "rack-b", HostName: "n05", InstanceID: "i05", Level1: "dc-1", Level2: "building-1", Level3: "room-2"}, + {Domain: "rack-c", HostName: "n06", InstanceID: "i06", Level1: "dc-1", Level2: "building-1", Level3: "room-2"}, + {Domain: "rack-c", HostName: "n07", InstanceID: "i07", Level1: "dc-1", Level2: "building-1", Level3: "room-2"}, + {Domain: "rack-c", HostName: "n08", InstanceID: "i08", Level1: "dc-1", Level2: "building-1", Level3: "room-2"}, + } { + hCopy := h + m.AddHostInfo(&hCopy) + } + return m +} + +func TestGetLevelInfo(t *testing.T) { + tests := []struct { + name string + domainMap DomainMap + level int + wantPresent bool + wantMembers map[string][]string // full members map (children sorted) + }{ + { + name: "empty DomainMap returns not present", + domainMap: NewDomainMap(), + level: 3, + wantPresent: false, + }, + { + name: "invalid level returns not present", + domainMap: nvl576DomainMap(), + level: 0, + wantPresent: false, + }, + { + name: "level absent from all hosts returns not present", + domainMap: func() DomainMap { + m := NewDomainMap() + // hosts have no Level1 set + m.AddHostInfo(&HostInfo{Domain: "rack-a", HostName: "n01", InstanceID: "i01", Level3: "room-1"}) + return m + }(), + level: 1, + wantPresent: false, + }, + { + name: "level 4 groups by domain; children are sorted hostnames", + domainMap: nvl576DomainMap(), + level: 4, + wantPresent: true, + wantMembers: map[string][]string{ + "rack-a": {"n01", "n02", "n03"}, + "rack-b": {"n04", "n05"}, + "rack-c": {"n06", "n07", "n08"}, + }, + }, + { + name: "level 3 groups by Level3; children are sorted domain IDs", + domainMap: nvl576DomainMap(), + level: 3, + wantPresent: true, + wantMembers: map[string][]string{ + "room-1": {"rack-a"}, + "room-2": {"rack-b", "rack-c"}, + }, + }, + { + name: "level 2 groups by Level2; children are sorted Level3 values", + domainMap: nvl576DomainMap(), + level: 2, + wantPresent: true, + wantMembers: map[string][]string{ + "building-1": {"room-1", "room-2"}, + }, + }, + { + name: "level 1 groups by Level1; children are sorted Level2 values", + domainMap: nvl576DomainMap(), + level: 1, + wantPresent: true, + wantMembers: map[string][]string{ + "dc-1": {"building-1"}, + }, + }, + { + name: "multiple top-level groups", + domainMap: func() DomainMap { + m := NewDomainMap() + m.AddHostInfo(&HostInfo{Domain: "r1", HostName: "a", InstanceID: "a", Level1: "dc-west", Level2: "bldg-w"}) + m.AddHostInfo(&HostInfo{Domain: "r1", HostName: "b", InstanceID: "b", Level1: "dc-west", Level2: "bldg-w"}) + m.AddHostInfo(&HostInfo{Domain: "r2", HostName: "c", InstanceID: "c", Level1: "dc-east", Level2: "bldg-e"}) + m.AddHostInfo(&HostInfo{Domain: "r2", HostName: "d", InstanceID: "d", Level1: "dc-east", Level2: "bldg-e"}) + m.AddHostInfo(&HostInfo{Domain: "r2", HostName: "e", InstanceID: "e", Level1: "dc-east", Level2: "bldg-e"}) + return m + }(), + level: 1, + wantPresent: true, + wantMembers: map[string][]string{ + "dc-west": {"bldg-w"}, + "dc-east": {"bldg-e"}, + }, + }, + { + name: "hosts with empty level value are excluded from that level", + domainMap: func() DomainMap { + m := NewDomainMap() + m.AddHostInfo(&HostInfo{Domain: "rack-a", HostName: "n1", InstanceID: "i1", Level3: "room-1"}) + // n2 has no Level3; it must not appear in level-3 results + m.AddHostInfo(&HostInfo{Domain: "rack-a", HostName: "n2", InstanceID: "i2"}) + return m + }(), + level: 3, + wantPresent: true, + wantMembers: map[string][]string{ + "room-1": {"rack-a"}, + }, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + present, members := tc.domainMap.GetLevelInfo(tc.level) + require.Equal(t, tc.wantPresent, present) + if !tc.wantPresent { + require.Nil(t, members) + return + } + require.Equal(t, tc.wantMembers, members) + }) + } +} + func TestDomainMapAddHost(t *testing.T) { domainMap := NewDomainMap() diff --git a/pkg/translate/block_complement_test.go b/pkg/translate/block_complement_test.go index 14ad8fc5..3934383f 100644 --- a/pkg/translate/block_complement_test.go +++ b/pkg/translate/block_complement_test.go @@ -11,6 +11,7 @@ import ( "strings" "testing" + "github.com/NVIDIA/topograph/pkg/models" "github.com/NVIDIA/topograph/pkg/topology" "github.com/stretchr/testify/require" ) @@ -399,6 +400,108 @@ func TestGetBlockTopologyUnitSingleBlockSize(t *testing.T) { require.Equal(t, expected, buf.String()) } +// TestComplementNVL576 validates multi-level block tree construction using the nvl576 +// simulation model. The topology is: building-1 → room-1 (16 switches, 14 with nodes) and +// room-2 (15 switches, all with nodes) → racks → 9 nodes each. +// rack-1-03, rack-1-13, and rack-2-11 have no block entries and are absent from domains. +// +// With BlockSizes=[9,144]: +// +// - Domain capacity: each rack holds exactly 9 nodes (1 base block), so domCapacity=9. +// +// - Level3 (room) fanout: blockSizes[last]/domCapacity = 144/9 = 16. Both rooms are +// padded to 16 slots: room-1 gets 2 empty padding slots (blocks 015–016); room-2 +// gets 1 empty padding slot (block 032). +// +// - After Level3, currentCapacity = 16×9 = 144 = blockSizes[last], so remaining is +// empty and the loop exits. No Level2 or "root" aggregation tier is added. +// +// - Total output: 32 blocks (16 room-1 + 16 room-2) followed by BlockSizes=9,144. +func TestComplementNVL576(t *testing.T) { + model, err := models.NewModelFromFile("nvl576.yaml") + require.NoError(t, err) + + graph, _ := model.ToGraph(nil) + require.NotNil(t, graph.Domains) + + cfg := &Config{ + Plugin: topology.TopologyBlock, + BlockSizes: []int{9, 144}, + } + nt, err := NewNetworkTopology(graph, cfg) + require.NoError(t, err) + + var buf bytes.Buffer + require.Nil(t, nt.toBlockTopology(&buf, false)) + + expected := strings.Join([]string{ + "# block001=rack-1-01", + "BlockName=block001 Nodes=node[0001-0009]", + "# block002=rack-1-02", + "BlockName=block002 Nodes=node[0010-0018]", + "# block003=rack-1-04", + "BlockName=block003 Nodes=node[0028-0030,0032,0034-0036]", + "# block004=rack-1-05", + "BlockName=block004 Nodes=node[0037-0045]", + "# block005=rack-1-06", + "BlockName=block005 Nodes=node[0046-0054]", + "# block006=rack-1-07", + "BlockName=block006 Nodes=node[0055-0063]", + "# block007=rack-1-08", + "BlockName=block007 Nodes=node[0064-0072]", + "# block008=rack-1-09", + "BlockName=block008 Nodes=node[0073-0081]", + "# block009=rack-1-10", + "BlockName=block009 Nodes=node[0082-0090]", + "# block010=rack-1-11", + "BlockName=block010 Nodes=node[0091-0099]", + "# block011=rack-1-12", + "BlockName=block011 Nodes=node[0100-0108]", + "# block012=rack-1-14", + "BlockName=block012 Nodes=node[0118-0126]", + "# block013=rack-1-15", + "BlockName=block013 Nodes=node[0127-0135]", + "# block014=rack-1-16", + "BlockName=block014 Nodes=node[0136-0144]", + "BlockName=block015", + "BlockName=block016", + "# block017=rack-2-01", + "BlockName=block017 Nodes=node[0145-0153]", + "# block018=rack-2-02", + "BlockName=block018 Nodes=node[0154-0162]", + "# block019=rack-2-03", + "BlockName=block019 Nodes=node[0163-0171]", + "# block020=rack-2-04", + "BlockName=block020 Nodes=node[0172-0180]", + "# block021=rack-2-05", + "BlockName=block021 Nodes=node[0181-0189]", + "# block022=rack-2-06", + "BlockName=block022 Nodes=node[0190-0198]", + "# block023=rack-2-07", + "BlockName=block023 Nodes=node[0199-0207]", + "# block024=rack-2-08", + "BlockName=block024 Nodes=node[0208-0216]", + "# block025=rack-2-09", + "BlockName=block025 Nodes=node[0217-0225]", + "# block026=rack-2-10", + "BlockName=block026 Nodes=node[0226-0234]", + "# block027=rack-2-12", + "BlockName=block027 Nodes=node[0244-0252]", + "# block028=rack-2-13", + "BlockName=block028 Nodes=node[0253-0261]", + "# block029=rack-2-14", + "BlockName=block029 Nodes=node[0262-0270]", + "# block030=rack-2-15", + "BlockName=block030 Nodes=node[0271-0275]", + "# block031=rack-2-16", + "BlockName=block031 Nodes=node[0280-0288]", + "BlockName=block032", + "BlockSizes=9,144", + "", + }, "\n") + require.Equal(t, expected, buf.String()) +} + // getBlockWithIBAsymmetricSpineTestSet models two spines with four leaf switches on the // left spine and three on the right, each leaf switch hosting one accelerator domain. func getBlockWithIBAsymmetricSpineTestSet() (*topology.Graph, map[string]string) { diff --git a/pkg/translate/block_tree.go b/pkg/translate/block_tree.go index 6b68ac0b..7530cbbb 100644 --- a/pkg/translate/block_tree.go +++ b/pkg/translate/block_tree.go @@ -17,6 +17,7 @@ import ( // blockTreeNode is implemented by host, base, and aggregate block nodes. type blockTreeNode interface { blockTreeNode() + levelIdentifier() string } // hostNode is the lowermost tree level: a host slot or an empty placeholder (host == nil). @@ -25,6 +26,12 @@ type hostNode struct { } func (*hostNode) blockTreeNode() {} +func (n *hostNode) levelIdentifier() string { + if n.host == nil { + return "" + } + return n.host.HostName +} // baseBlockNode is the Slurm base block level. It always holds exactly baseBlockSize // host nodes; missing positions or hosts are nil-host placeholders. @@ -37,16 +44,18 @@ type baseBlockNode struct { func (*baseBlockNode) blockTreeNode() {} -func (n *baseBlockNode) domainIdentifier() string { return n.domain } +func (n *baseBlockNode) levelIdentifier() string { return n.domain } // aggregateBlockNode groups base blocks or other aggregates. An domain with // multiple base blocks is represented as an aggregate of baseBlockNode children. type aggregateBlockNode struct { + id string children []blockTreeNode nodeCount int // sum of nodeCount across all children } -func (*aggregateBlockNode) blockTreeNode() {} +func (*aggregateBlockNode) blockTreeNode() {} +func (n *aggregateBlockNode) levelIdentifier() string { return n.id } // splitIntoBaseBlocks splits a sorted host list into one or more base blocks of at // most baseBlockSize leaves each. Overflow blocks get a "#N" suffix on the ID. @@ -110,7 +119,7 @@ func isEmptyBlock(b *blockInfo) bool { // 4. Empty blockInfo (tree slot was never filled) func baseBlockToBlockInfo(bb *baseBlockNode, byName map[string]*blockInfo, seq int) *blockInfo { id := fmt.Sprintf("block%03d", seq) - domainID := bb.domainIdentifier() + domainID := bb.levelIdentifier() nodes := hostNamesFromLeaves(bb.leaves) if len(nodes) > 0 { return &blockInfo{id: id, name: blockDisplayName(bb.id, domainID), nodes: nodes} @@ -213,27 +222,58 @@ func newEmptyBaseBlock(baseBlockSize int) *baseBlockNode { return &baseBlockNode{leaves: leaves} } -// buildBlockTree packs domains into a padded block tree shaped by blockSizes. +// buildBlockTree constructs a padded aggregate tree from domain nodes, shaped by +// blockSizes and the optional Level3/Level2/Level1 hierarchy in the DomainMap. +// +// # Phase 1 – domain nodes +// +// packDomainNodes splits each domain's hosts into base blocks of baseBlockSize, +// pads each domain to a multiple of groupSize blocks, and returns one aggregateBlockNode +// per domain. groupSize is derived from the maximum domain size relative to baseBlockSize; +// when all domains fit within a single base block, groupSize is 1. +// +// If every domain's padded capacity already meets blockSizes[last], no further +// aggregation is needed and the flat list is returned immediately. +// +// # Phase 2 – level aggregation (Level3 → Level2 → Level1) +// +// When HostInfo carries Level3/Level2/Level1 values, the function builds one tier +// per level using GetLevelInfo to discover the parent→children membership at each +// level. The fanout per level is: +// +// desiredGroupSize = blockSizes[last] / currentCapacity // -// packDomainNodes handles all domain-level work (packing, equalization, root -// padding) and returns both the fully padded domain nodes and the live bb count. -// buildBlockTree uses the capacity recorded on each domain node to decide whether -// higher-tier aggregation is needed, then delegates to packAggregateNodes. +// This pads each group to the blockSize boundary rather than just to the observed +// maximum, so a level group with fewer nodes than the next blockSize receives +// empty placeholder slots to make up the difference. +// +// After packing, currentCapacity is updated to the actual padded nodeCount of one +// group (desiredGroupSize × currentCapacity), which is then used as the basis for +// the next tier's fanout. The loop stops early when: +// - blockSizes is already satisfied (remaining is empty), or +// - a level is absent from the DomainMap (!present). +// +// # Phase 3 – fallback "root" aggregation +// +// When no level fields are set in the DomainMap, Phase 2 exits on the first +// iteration and the remaining domain nodes are packed under a single "root" key. +// This is also the path taken when level processing leaves blockSizes unsatisfied +// (e.g. fewer levels than blockSizes entries): the leftover tiers are collapsed +// into one root group using the same desiredGroupSize formula. func buildBlockTree(domains topology.DomainMap, blockSizes []int) *aggregateBlockNode { baseBlockSize := blockSizes[0] + // groupSize aligns each domain to a power-of-two multiple of base blocks so + // that all domains occupy the same number of slots within a group. groupSize := groupSizeFromDomains(domains, baseBlockSize, blockSizes[len(blockSizes)-1]) - //Pad each domain to a multiple of groupSize base blocks, - //then pack those blocks into aggregate nodes of size groupSize until we reach the top tier or satisfy blockSizes[last]. + // Phase 1: pack domains into per-domain aggregate nodes. domainNodes := packDomainNodes(domains, baseBlockSize, groupSize) if len(domainNodes) == 0 { return nil } - // All domain nodes returned by packDomainNodes are aggregateBlockNodes whose - // nodeCount reflects the slot capacity of that domain (not just live hosts). - // When all domains are complete, their capacity is already >= lastBS so - // getRemainingBlocks returns nil and we return the flat tree unchanged. + // Each domain node's nodeCount is its slot capacity (padded), not its live + // host count. If that capacity already meets blockSizes[last], return flat. domCapacity := getNodeCount(domainNodes[0]) remaining := getRemainingBlocks(blockSizes, domCapacity) if len(remaining) == 0 { @@ -244,13 +284,78 @@ func buildBlockTree(domains topology.DomainMap, blockSizes []int) *aggregateBloc return &aggregateBlockNode{children: domainNodes, nodeCount: total} } - //Build a higher-tier tree above the domain nodes with groupSize fanout and enough levels to satisfy blockSizes[last]. - //The tree is padded with empty aggregate nodes as needed to reach the next power-of-two multiple of groupSize, - //so every domain occupies complete groups at every tier. - //The final tree may be wider than blockSizes[last] but is guaranteed to have at least that capacity. + // completed tracks the node capacity at each completed tier and is passed to + // newEmptyAggregateBlock so that padding nodes have the correct internal shape. completed := []int{baseBlockSize, domCapacity} - desiredGroupSize := remaining[len(remaining)-1] / domCapacity - nodesMap := map[string][]blockTreeNode{"root": domainNodes} + + // Phase 2: build one aggregate tier per level (Level3 → Level2 → Level1). + currentCapacity := domCapacity + currentNodes := domainNodes + for _, level := range []int{3, 2, 1} { + // Recompute remaining at the start of each iteration so the fanout uses + // the capacity that reflects any padding added in the previous tier. + remaining = getRemainingBlocks(blockSizes, currentCapacity) + if len(remaining) == 0 { + break + } + present, members := domains.GetLevelInfo(level) + if !present { + //When a level is absent, we assume there are no more top levels and skip aggregation. + //Provider components should ensure the levels are contiguous and set the levels correctly + //so that aggregation can proceed correctly. + break + } + // Fanout is derived from blockSizes[last] so that each group is padded + // to the blockSize boundary, not merely to the observed maximum. + desiredGroupSize := remaining[len(remaining)-1] / currentCapacity + if desiredGroupSize <= 0 { + break + } + + // Map each current node's level identifier to the node itself so that + // levelMap can look up child nodes by the names returned by GetLevelInfo. + nodesMap := make(map[string]blockTreeNode) + for _, node := range currentNodes { + nodesMap[node.levelIdentifier()] = node + } + + // Build a per-level-name list of child nodes using the membership map + // from GetLevelInfo (e.g. "room-1" → ["rack-1-01", "rack-1-02", ...]). + levelMap := make(map[string][]blockTreeNode) + for levelName, children := range members { + childNodes := []blockTreeNode{} + for _, child := range children { + if childNode, exists := nodesMap[child]; exists { + childNodes = append(childNodes, childNode) + } + } + levelMap[levelName] = childNodes + } + + //Pack the current level into aggregate nodes according to the desired group size. + packed, _ := packAggregateNodes(levelMap, completed, desiredGroupSize) + + //Reset the variables for the next iteration + currentNodes = packed + currentCapacity = desiredGroupSize * currentCapacity // Slot capacity = desiredGroupSize × prevCapacity. + completed = append(completed, currentCapacity) + } + + // Phase 3: check whether level processing satisfied blockSizes. + remaining = getRemainingBlocks(blockSizes, currentCapacity) + if len(remaining) == 0 { + // All blockSizes entries are covered; return current nodes as the root's + // direct children without an extra wrapping tier. + total := 0 + for _, n := range currentNodes { + total += getNodeCount(n) + } + return &aggregateBlockNode{children: currentNodes, nodeCount: total} + } + // blockSizes not yet satisfied (no level fields, or fewer levels than blockSizes + // entries): pack the remaining nodes under "root" with the standard fanout. + desiredGroupSize := remaining[len(remaining)-1] / currentCapacity + nodesMap := map[string][]blockTreeNode{"root": currentNodes} aggregateNodes, aggCount := packAggregateNodes(nodesMap, completed, desiredGroupSize) return &aggregateBlockNode{children: aggregateNodes, nodeCount: aggCount} } @@ -273,7 +378,7 @@ func packDomainNodes(domains topology.DomainMap, baseBlockSize, groupSize int) [ blocks = append(blocks, newEmptyBaseBlock(baseBlockSize)) } - aggregateNode := &aggregateBlockNode{} + aggregateNode := &aggregateBlockNode{id: domainID} for _, b := range blocks { aggregateNode.nodeCount += baseBlockSize aggregateNode.children = append(aggregateNode.children, b) @@ -310,7 +415,7 @@ func packAggregateNodes(nodesMap map[string][]blockTreeNode, completed []int, gr for _, b := range blocks { localCount += getNodeCount(b) } - aggregateNodes = append(aggregateNodes, &aggregateBlockNode{children: blocks, nodeCount: localCount}) + aggregateNodes = append(aggregateNodes, &aggregateBlockNode{id: nodeID, children: blocks, nodeCount: localCount}) total += localCount } return aggregateNodes, total diff --git a/tests/models/nvl576.yaml b/tests/models/nvl576.yaml new file mode 100644 index 00000000..9c6345be --- /dev/null +++ b/tests/models/nvl576.yaml @@ -0,0 +1,276 @@ +# building-1 +# / \ +# room-1 room-2 +# / ... \ / ... \ +# rack-1-01 rack-1-16 rack-2-01 rack-2-16 +# (16 switches) (16 switches) +# | | +# ------- ------- +# 0001 0145 +# ... ... +# 0009 0153 +# ------- ------- +# +# room-1: rack-1-{01..16}, 9 nodes/rack; rack-1-03 and rack-1-13 have no nodes +# room-2: rack-2-{01..16}, 9 nodes/rack +# +# Metadata: +# building-1: topology.kubernetes.io/level2: building-1 +# room-1: topology.kubernetes.io/level3: room-1 +# room-2: topology.kubernetes.io/level3: room-2 +# + +switches: + building-1: + labels: + topology.kubernetes.io/level2: building-1 + switches: + - room-1 + - room-2 + room-1: + labels: + topology.kubernetes.io/level3: room-1 + switches: + - rack-1-01 + - rack-1-02 + - rack-1-03 + - rack-1-04 + - rack-1-05 + - rack-1-06 + - rack-1-07 + - rack-1-08 + - rack-1-09 + - rack-1-10 + - rack-1-11 + - rack-1-12 + - rack-1-13 + - rack-1-14 + - rack-1-15 + - rack-1-16 + room-2: + labels: + topology.kubernetes.io/level3: room-2 + switches: + - rack-2-01 + - rack-2-02 + - rack-2-03 + - rack-2-04 + - rack-2-05 + - rack-2-06 + - rack-2-07 + - rack-2-08 + - rack-2-09 + - rack-2-10 + - rack-2-12 + - rack-2-13 + - rack-2-14 + - rack-2-15 + - rack-2-16 +blocks: +- switch: rack-1-01 + nodes: + - node[0001-0009] + labels: + network.topology.nvidia.com/accelerator: rack-1-01 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-02 + nodes: + - node[0010-0018] + labels: + network.topology.nvidia.com/accelerator: rack-1-02 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-04 + nodes: + - node[0028-0030] + - node0032 + - node[0034-0036] + labels: + network.topology.nvidia.com/accelerator: rack-1-04 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-05 + nodes: + - node[0037-0045] + labels: + network.topology.nvidia.com/accelerator: rack-1-05 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-06 + nodes: + - node[0046-0054] + labels: + network.topology.nvidia.com/accelerator: rack-1-06 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-07 + nodes: + - node[0055-0063] + labels: + network.topology.nvidia.com/accelerator: rack-1-07 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-08 + nodes: + - node[0064-0072] + labels: + network.topology.nvidia.com/accelerator: rack-1-08 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-09 + nodes: + - node[0073-0081] + labels: + network.topology.nvidia.com/accelerator: rack-1-09 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-10 + nodes: + - node[0082-0090] + labels: + network.topology.nvidia.com/accelerator: rack-1-10 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-11 + nodes: + - node[0091-0099] + labels: + network.topology.nvidia.com/accelerator: rack-1-11 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-12 + nodes: + - node[0100-0108] + labels: + network.topology.nvidia.com/accelerator: rack-1-12 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-14 + nodes: + - node[0118-0126] + labels: + network.topology.nvidia.com/accelerator: rack-1-14 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-15 + nodes: + - node[0127-0135] + labels: + network.topology.nvidia.com/accelerator: rack-1-15 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-1-16 + nodes: + - node[0136-0144] + labels: + network.topology.nvidia.com/accelerator: rack-1-16 + network.topology.nvidia.com/level3: room-1 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-01 + nodes: + - node[0145-0153] + labels: + network.topology.nvidia.com/accelerator: rack-2-01 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-02 + nodes: + - node[0154-0162] + labels: + network.topology.nvidia.com/accelerator: rack-2-02 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-03 + nodes: + - node[0163-0171] + labels: + network.topology.nvidia.com/accelerator: rack-2-03 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-04 + nodes: + - node[0172-0180] + labels: + network.topology.nvidia.com/accelerator: rack-2-04 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-05 + nodes: + - node[0181-0189] + labels: + network.topology.nvidia.com/accelerator: rack-2-05 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-06 + nodes: + - node[0190-0198] + labels: + network.topology.nvidia.com/accelerator: rack-2-06 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-07 + nodes: + - node[0199-0207] + labels: + network.topology.nvidia.com/accelerator: rack-2-07 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-08 + nodes: + - node[0208-0216] + labels: + network.topology.nvidia.com/accelerator: rack-2-08 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-09 + nodes: + - node[0217-0225] + labels: + network.topology.nvidia.com/accelerator: rack-2-09 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-10 + nodes: + - node[0226-0234] + labels: + network.topology.nvidia.com/accelerator: rack-2-10 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-12 + nodes: + - node[0244-0252] + labels: + network.topology.nvidia.com/accelerator: rack-2-12 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-13 + nodes: + - node[0253-0261] + labels: + network.topology.nvidia.com/accelerator: rack-2-13 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-14 + nodes: + - node[0262-0270] + labels: + network.topology.nvidia.com/accelerator: rack-2-14 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-15 + nodes: + - node[0271-0275] + labels: + network.topology.nvidia.com/accelerator: rack-2-15 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 +- switch: rack-2-16 + nodes: + - node[0280-0288] + labels: + network.topology.nvidia.com/accelerator: rack-2-16 + network.topology.nvidia.com/level3: room-2 + network.topology.nvidia.com/level2: building-1 + +