CI: require N consecutive nvidia-smi successes after Windows device cycle (#2195)

* CI: require N consecutive nvidia-smi successes after device cycle

Multi-GPU Windows rows (observed on 2x H100 MCDM after #2176 landed)
keep failing the "Ensure GPU is working" step with `Failed to
initialize NVML: Not Found`. Root cause: after `pnputil` cycles both
display devices, NVML briefly reports success mid-init then flaps back
to "Not Found" a couple seconds later. The existing poll exits on the
*first* `nvidia-smi` exit code 0, so the loop bails ~2 seconds in and
the next workflow step hits the flap window.

Scale the consecutive-success requirement to the number of cycled
NVIDIA devices (1 for single-GPU rows, 2 for the H100 pair) and bump
the inter-iteration sleep from 2 to 3 seconds. Single-GPU rows pay an
extra 1-sec floor; multi-GPU rows now require ~6 sec of stable NVML
before moving on.

The 60-sec deadline is unchanged; the loop still bails (and the script
fails loudly) if NVML doesn't settle in time.

* CI: restore the pre-#2176 5-sec unconditional settle before the poll

Pre-#2176, every Windows row ran install_gpu_driver.ps1 unconditionally
and that script ended with a fixed `Start-Sleep -Seconds 5` after the
pnputil cycle. #2176 dropped that floor (the poll exits on the first
nvidia-smi success at ~2 sec on single-GPU, ~2 sec mid-flap on the
H100 pair). Put the 5-sec floor back, before the consecutive-success
poll, so we never settle for less than the known-good baseline.

* CI: trim configure_driver_mode.ps1 comments for portability

* CI: drop redundant @(...) comment

Author: leofang

ci/tools/configure_driver_mode.ps1

@@ -30,26 +30,30 @@ function Set-DriverMode {
         exit 1
     }
 
-    # Only restart NVIDIA display adapters, not other display devices (e.g. QEMU VGA)
-    $nvidia_devices = Get-PnpDevice -Class Display -FriendlyName "NVIDIA*"
+    # Only restart NVIDIA display adapters, not other display devices (e.g. QEMU VGA).
+    $nvidia_devices = @(Get-PnpDevice -Class Display -FriendlyName "NVIDIA*")
+    $gpu_count = $nvidia_devices.Count
     foreach ($device in $nvidia_devices) {
         Write-Output "Restarting device: $($device.FriendlyName) ($($device.InstanceId))"
         pnputil /disable-device "$($device.InstanceId)"
         pnputil /enable-device "$($device.InstanceId)"
     }
 
-    # Poll nvidia-smi until NVML can initialize, or give up after ~60s.
-    # A fixed sleep is not enough on slower-coming-back-up multi-GPU rows
-    # (e.g. 2x H100 MCDM) where pnputil enable returns before NVML is
-    # ready. Pattern borrowed from the runner-team `nvgha-driver.ps1`.
+    # Initial settle after the device cycle.
+    Start-Sleep -Seconds 5
+
+    # Poll nvidia-smi for N consecutive successes (N == cycled GPUs)
+    # so a mid-init "ok" flap doesn't fool the loop; bail after ~60s.
     Write-Output "Waiting for nvidia-smi/NVML to come back up after device cycle..."
     $deadline = (Get-Date).AddSeconds(60)
+    $consecutive_ok = 0
     do {
-        Start-Sleep -Seconds 2
+        Start-Sleep -Seconds 3
         & nvidia-smi.exe 2>&1 | Out-Null
-    } while ($LASTEXITCODE -ne 0 -and (Get-Date) -lt $deadline)
-    if ($LASTEXITCODE -ne 0) {
-        Write-Error "nvidia-smi did not return cleanly within 60s of the device cycle"
+        if ($LASTEXITCODE -eq 0) { $consecutive_ok++ } else { $consecutive_ok = 0 }
+    } while ($consecutive_ok -lt $gpu_count -and (Get-Date) -lt $deadline)
+    if ($consecutive_ok -lt $gpu_count) {
+        Write-Error "nvidia-smi did not return cleanly $gpu_count times in a row within 60s of the device cycle"
         exit 1
     }
 }