docs: document scale_m1 GB200/GB300 workflow with block topology

Author: ryanhamel

README.md

@@ -370,24 +370,75 @@ This either prints out the topology in slurm topology format or creates an outpu
 
 It is installed on the scheduler node automatically by the azure-slurm installer (alongside `azslurm`) and runs under the same virtual environment at `/opt/azurehpc/slurm/venv`. It is not installed on execute nodes. Logs are written to `/opt/azurehpc/slurm/logs/scale_m1.log`.
 
+#### Required slurm.conf settings
+To use GB200/GB300, the block topology plugin must be enabled and the gbx00 partition must be excluded from auto-suspend:
+
+```
+TopologyParam=BlockAsNodeRank
+TopologyPlugin=topology/block
+# exclude any partition that uses gbx00
+SuspendExcParts=gpu
+```
+
+**Do not** add these lines to the template's **Slurm Configuration** parameter (`additional_slurm_config` / `AdditionalSlurmConfig`). `TopologyPlugin=topology/block` requires a populated `topology.conf` to exist before `slurmctld` starts; applying it at cluster creation, before any nodes are scaled and the topology is generated, prevents `slurmctld` from starting.
+
+Instead, apply them manually on the scheduler **after** the nodes are scaled and the block topology file is in place. Add the lines to `/etc/slurm/site_specific.conf` (which is included by `slurm.conf`) and run `scontrol reconfigure`. Replace `gpu` in `SuspendExcParts` with the name of the partition that uses gbx00 nodes if it differs. The full ordering is shown in the workflow below.
+
 #### Workflow
-Scaling is a three-step workflow: create a reservation, power up to the target (optionally with a buffer), then prune back down to the target. Once the nodes are ready, delete the reservation to release the nodes back to the cluster so jobs can be scheduled on them.
+With the cluster running and an SSH session open on the scheduler node, the end-to-end process is:
+
+1. **Scale the partition** to the target size — reserve the nodes, then power up to the target.
+2. **Generate the block topology and put it in place** at `/sched/<clustername>/topology.conf`.
+3. **Enable the block topology plugin** by adding the required settings to `/etc/slurm/site_specific.conf`, then run `scontrol reconfigure`.
+4. **Release the ready nodes** from the reservation so jobs can be scheduled on them.
+
+When releasing nodes in the final step, remove only the nodes that are powered up and ready to run jobs. Leave all powered-off or unhealthy nodes in the reservation, and do **not** delete the reservation — deleting it would also release the not-yet-ready nodes back to the cluster.
 
 ```bash
 # 1. Reserve the partition's nodes under the scale_m1 reservation
 scale_m1 create_reservation -p gpu
 
-# 2. Power up 28 racks (504 nodes), overprovisioning 6 racks (108 nodes) to absorb unhealthy hardware
+# 2. Power up 28 racks (504 nodes) to the target size
+scale_m1 power_up --racks 28
+
+# 3. Generate the block topology for the partition (--block_size 18 = one rack per block)
+azslurm topology -n --block --block_size 18 -p gpu > topology.conf
+
+# 4. Review topology.conf, then copy it into place at /sched/<clustername>/topology.conf.
+#    For a cluster named "ccw":
+cp topology.conf /sched/ccw/topology.conf
+
+# 5. Enable the block topology plugin and exclude the gbx00 partition from auto-suspend by adding
+#    the following lines to /etc/slurm/site_specific.conf (included by slurm.conf):
+#      TopologyParam=BlockAsNodeRank
+#      TopologyPlugin=topology/block
+#      SuspendExcParts=gpu
+$EDITOR /etc/slurm/site_specific.conf
+
+# 6. Apply the configuration
+scontrol reconfigure
+
+# 7. Remove only the powered-up, ready nodes from the reservation so workloads can run on them.
+#    Leave all powered-off or unhealthy nodes in the reservation; do not delete the reservation.
+#    Update the reservation so it contains only the nodes that are NOT yet ready. For example,
+#    if ccw-1-3-gpu-[1-3] are still powering up or unhealthy, keep just those reserved:
+scontrol update ReservationName=scale_m1 Nodes=ccw-1-3-gpu-[1-3]
+```
+
+#### Overprovisioning to absorb unhealthy hardware (advanced)
+In rare cases you may want to power up extra nodes so that unhealthy hardware can be discarded while still reaching the target size. `power_up` accepts an overprovision buffer, and `prune` / `prune_now` then trim back to the target, keeping only fully populated racks:
+
+```bash
+# Power up 28 racks plus a 6-rack (108-node) buffer to absorb unhealthy hardware
 scale_m1 power_up --racks 28 --overprovision-racks 6
 
-# 3. Prune back to 28 racks. `prune` prints the nodes to terminate; `prune_now` also terminates them
+# Prune back to 28 racks. `prune` prints the nodes to terminate; `prune_now` also terminates them
 scale_m1 prune --racks 28 > to_terminate.txt
 scale_m1 prune_now --racks 28
-
-# 4. Release the reserved nodes to the cluster so workloads can run on them
-scontrol delete reservation scale_m1
 ```
 
+After pruning to the target size, continue with the topology generation, configuration, and reservation steps from the workflow above.
+
 #### Subcommands
 | Subcommand | Purpose |
 | --- | --- |