System Erase's post-reset "wait for iDRAC, then power on" step: PowerShell's fixed wait has ~0 margin in practice; Python's retry loop looks unguarded against connection-level failures

[tejeszacsko]

Scope: PowerShell (Set-DeviceFirmwareSimpleUpdateREDFISH's System Erase path is not involved here — this is specifically Invoke-SystemEraseREDFISH.psm1 and its Python counterpart SystemEraseREDFISH.py, both under Redfish PowerShell/ / Redfish Python/). Related to, but distinct from, the break-escaping-caller's-loop issue reported separately.

Summary

After a System Erase job that includes the BIOS component completes, both implementations wait a fixed amount of time for iDRAC to come back up before attempting to power the server back on:

• PowerShell (Invoke-SystemEraseREDFISH.psm1): waits a fixed 5 minutes, then attempts to reconnect essentially once (two closely-spaced POST attempts with no delay or retry loop between them), giving up immediately on failure.
• Python (SystemEraseREDFISH.py): waits a fixed 6 minutes, then retries up to 5 times with a 60-second delay between attempts if the connection succeeds but returns a non-success HTTP status — but the requests.post(...) calls in that retry loop aren't wrapped in a try/except, so a connection-level failure (iDRAC not yet listening at all, rather than iDRAC responding with an error) looks like it would raise an unhandled exception and crash the script outright, rather than being retried the way an HTTP-level error is.

Real timing data

Ran a System Erase (CryptographicErasePD,IDRAC,BIOS) against a PowerEdge R6615 and captured full timestamped logs. Two independent runs both hit the PowerShell 5-minute wait's failure case; here's the exact breakdown from the more recent one:

18:13:51.447  Stage 1 starts - about to call Invoke-SystemEraseREDFISH
18:22:59.430  Cmdlet's buffered output appears (call has returned) -
              includes "System Erase job execution time: Minutes: 4, Seconds: 0"
              and two "Unable to connect to the remote server" failures
              after the documented "wait 5 minutes for iDRAC" step

• Total call duration: 547.98s
• Erase job's own reported polling time: 240s (4:00)
• Remainder (setup + the 5-minute wait + 2 failed reconnects + overhead): 307.98s — only 7.98 seconds of that is not the nominal 300s wait, meaning both reconnect attempts fired right at the 5:00 mark with essentially no margin.
• Our own follow-up connectivity check, run immediately after the cmdlet returned (2.35s later, after closing an unrelated browser window), succeeded on its first attempt in 42ms.

So iDRAC was reachable only a few seconds after the module gave up — the wait wasn't drastically too short, it just had zero margin, and across 2/2 real attempts that margin wasn't enough.

Suggested fix

Rather than picking a new fixed number (hardware/network variability means any constant will eventually be violated again), replace the fixed-wait-then-give-up pattern with a bounded polling loop: check every 15–30s for up to some generous ceiling (10–15 minutes), succeeding as soon as iDRAC responds instead of gambling on one guessed duration. This is the same pattern already used successfully elsewhere for "wait for iDRAC" scenarios.

For the Python side specifically, independent of the wait duration: wrapping the requests.post(...) calls in that retry loop in a try/except for connection errors (treating them the same as a non-success status code — log and retry after the existing 60s delay) would close what looks like a real gap, since the scenario the retry loop exists to handle (iDRAC not up yet) is exactly the case most likely to raise a connection error rather than return an HTTP status at all.

Environment

• Windows PowerShell 5.1, Dell PowerEdge R6615, iDRAC9
• Checked against the current master branch of both the PowerShell and Python implementations

[tejeszacsko]

Follow-up: the same root cause also affects the completion-polling loop itself, not just the post-completion wait — in both implementations

While chasing a downstream issue in our own integration, we found a related-but-distinct symptom of the same underlying behavior described above (iDRAC becoming unreachable while it resets itself as part of erasing the IDRAC component). It affects the job-status polling loop itself, in both languages, though the two implementations fail differently.

What we saw

A real run (CryptographicErasePD,IDRAC,BIOS on a PowerEdge R6615) hit an HTTP 404 during the PowerShell module's polling loop, well before any completion message was ever reported:

{"error": {"@Message.ExtendedInfo": [{"Message": "Unable to complete the operation because the resource
/redfish/v1/Managers/iDRAC.Embedded.1/Oem/Dell/Jobs/JID_851782951199 entered is not found.", "MessageArgs":
["/redfish/v1/Managers/iDRAC.Embedded.1/Oem/Dell/Jobs/JID_851782951199"], "MessageArgs@odata.count": 1, "MessageId": "IDRAC.2.9.SYS403",
"RelatedProperties": [], "RelatedProperties@odata.count": 0, "Resolution": "Enter the correct resource and retry the operation. For resources
with numeric ID in the URI, enable the \"Redfish.1#NumericDynamicSegmentsEnable\" attribute and retry the operation. For information about
valid resource, see the Redfish Users Guide available on the support site.", "Severity": "Critical"}, {"Message": "The resource at the URI
/redfish/v1/Managers/iDRAC.Embedded.1/Oem/Dell/Jobs/JID_851782951199 was not found.", "MessageArgs":
["/redfish/v1/Managers/iDRAC.Embedded.1/Oem/Dell/Jobs/JID_851782951199"], "MessageArgs@odata.count": 1, "MessageId":
"Base.1.8.ResourceMissingAtURI", "RelatedProperties": [], "RelatedProperties@odata.count": 0, "Resolution": "Place a valid resource at the URI
or correct the URI and resubmit the request.", "Severity": "Critical"}], "code": "Base.1.8.GeneralError", "message": "A general error has
occurred. See ExtendedInfo for more information"}}

We confirmed via the iDRAC GUI afterward that this exact job (JID_851782951199) went on to show System_Erase - Completed (100%) — the job itself was never wiped or lost, iDRAC was just transiently unreachable partway through the erase.

PowerShell (Invoke-SystemEraseREDFISH.psm1)

Every Invoke-WebRequest call inside loop_job_status's while ($true) loop is wrapped in:

catch {
    $RespErr
    return
}

return here exits loop_job_status entirely and silently — not just that one iteration. So any request-level failure while polling (a 404 for the job's own resource, same as a plain connection failure) causes the whole completion check to be abandoned with no success/failure signal whatsoever, however early it happens. The caller then proceeds straight into the "wait for iDRAC, then power on" step this issue already describes, exactly as if the job had genuinely completed — it has no way to tell the difference.

Python (SystemEraseREDFISH.py)

loop_job_status()'s polling loop (current master) is actually more resilient than PowerShell for one failure mode, and less for another:

while True:
    try:
        response = requests.get(...)
    except:
        if retry_count == 10:
            logging.info("- INFO, retry count of 10 has been reached to communicate with iDRAC, script will exit")
            sys.exit(0)
        else:
            logging.info("- INFO, lost iDRAC network connection, retry GET request after 10 second sleep delay")
            retry_count += 1
            time.sleep(15)
            continue
    current_time = (datetime.now()-start_time)
    if response.status_code == 200:
        current_job_status = data['Message']
    else:
        logging.error("\n- FAIL, Command failed to check job status, return code is %s" % response.status_code)
        sys.exit(0)

A genuine connection-level exception (refused/timeout) is retried here, up to 10 times. But requests doesn't raise an exception for a non-2xx HTTP response, so a 404 like the one above skips the try/except entirely and lands on the else branch, which calls sys.exit(0) immediately. That's a harder failure than the PowerShell side: the whole process exits on the spot, with no retry and no chance for the power-on step to ever run - whereas PowerShell at least silently continues into it (based on a wrong assumption, but continues).

Why this matters together with the timing issue above

The fixed-wait problem already reported here assumes the wait starts counting from genuine job completion. In PowerShell, this bug means the wait can just as easily start counting from "the polling loop gave up early" instead — which, per the timing data above, already has ~0 margin. In Python, the equivalent scenario doesn't even get that far; the script just exits.

Suggested fix (same shape as above, extended to the polling loop)

For both implementations, a request failure while polling for job status — whether a connection-level exception or a non-2xx HTTP response — should be treated as transient and retried with a short delay, bounded by the loop's own overall timeout (the existing 50-minute $end_time in PowerShell, the existing 2-hour ceiling in Python), the same way a non-terminal job status is already handled. That would let both implementations naturally ride out iDRAC's own reset window and observe the real completion message, rather than treating "iDRAC isn't reachable yet" — the exact condition a System Erase involving the IDRAC component is expected to cause — as an immediate, unrecoverable failure.

[texroemer]

@tejeszacsko

Fixed system erase cmdlet and tested it on both iDRAC9 and 10, no issues with erasing BIOS component and powering on the server after iDRAC reboot to complete BIOS RTD process.

Example of running updated cmdlet:

PS C:\>Invoke-SystemEraseREDFISH -idrac_ip 192.168.0.120  -idrac_username root -idrac_password calvin -execute_system_erase BIOS -power_on_server

- Keys and Values being passed in for POST action 'SystsemErase' -


Name                           Value                                                                                                                                                                                                
----                           -----                                                                                                                                                                                                
Component                      {BIOS}                                                                                                                                                                                               

- PASS, statuscode 202 returned successfully for POST command to create update job ID 'JID_178524817666'

- INFO, script will now loop polling the job status until marked completed

- Job ID 'JID_178524817666' not marked completed, checking job status again. Current job message: Starting System Erase, waiting for host to shutdown
- Job ID 'JID_178524817666' not marked completed, checking job status again. Current job message: Turning on the server to perform System Erase tasks.

- PASS, job ID 'JID_178524817666' successfully marked as completed

System Erase job execution time:

Minutes : 13
Seconds : 53

- INFO, BIOS component detected. Cmdlet will wait 7 minutes for iDRAC to come back up, then power ON server to complete BIOS reset to default process
- PASS, iDRAC in ready state, both LC status and server status properties report Ready
- PASS, statuscode 204 returned successfully to power ON the server. Cmdlet will now loop checking the server power status until reported OFF
- INFO, server still in ON state, checking server power status again
- INFO, server still in ON state, checking server power status again
- INFO, server still in ON state, checking server power status again
- PASS, verified server is in OFF state, BIOS reset to default process is complete

- User selected to power ON server after System Erase process completes
- PASS, statuscode 204 returned successfully to power ON the server

Thanks
Tex

[tejeszacsko]

Thanks Tex, appreciate you getting to this one too. Re-checked the current master branch of both Invoke-SystemEraseREDFISH.psm1 and SystemEraseREDFISH.py against everything in this thread (the original fixed-wait timing report, and the follow-up about the polling loop). It's a bit of a mixed picture: the timing side genuinely got better, but the "a request failure during polling is treated as final instead of transient" issue is still around in both languages, and it looks like it came along for the ride into the new function that was added for the timing fix.

Genuinely better: the fixed-wait itself

PowerShell now does a Start-Sleep 420 (7 min, up from 5) followed by a real bounded readiness poll — a new check_iDRAC_API_status_check function, up to 10 retries × 30s, checking LCStatus/Status -eq "Ready" — before moving on to finish_bios_reset_to_default/power_on_server. That's exactly the kind of thing we were hoping for (poll for real readiness instead of one fixed guess), and it matches the "PASS, iDRAC in ready state..." line in your example output. Nice improvement over the old blind-wait-then-one-attempt approach.

Still open: loop_job_status's completion-polling loop (both languages)

PowerShell — the catch blocks around both Invoke-WebRequest calls in loop_job_status look the same as before:

catch
{
    $RespErr
    return
}

return here still exits loop_job_status entirely and silently on any request-level exception. A new branch was added for a non-exception, non-200 response:

if ($result.StatusCode -eq 200)
{
    #- Write-Host "PASS, GET request passed to check job status"
}
else
{
    [String]::Format("- FAIL, GET command failed to check job status, statuscode {0} returned. Detail error message: {1}",$result.StatusCode,$result)
    Start-Sleep 10
    continue
}

but as far as we can tell this branch wouldn't actually catch the 404 scenario we originally reported: Invoke-WebRequest throws a terminating exception on any non-2xx status by default in Windows PowerShell (and in this file's PS7+ branch too — we didn't see -SkipHttpErrorCheck used anywhere in the repo), so a 404 would still land in the catch/return above rather than this new branch. This new branch looks like it'd only fire for a response that completes successfully but isn't recognized as success, which doesn't quite match how Invoke-WebRequest behaves for actual error codes — so it may not end up doing much in practice.

Python's loop_job_status() GET-retry already looked solid and is unchanged (it already wrapped the request in try/except with a 10x/15s retry, as quoted in our earlier comment) — but the non-200 branch still exits right away, the same as before:

if response.status_code == 200:
    current_job_status = data['Message']
else:
    logging.error("\n- FAIL, GET command failed to check job status, return code %s" % response.status_code)
    sys.exit(0)

requests doesn't raise for a non-2xx response, so a 404 here would still skip the try/except and exit the whole script.

Heads up: the new readiness-check functions seem to carry a similar gap

PowerShell's new check_iDRAC_API_status_check (added to implement the bounded-poll idea above — nice to see!) wraps its own Invoke-WebRequest/POST calls the same way loop_job_status does:

catch
{
Write-Host
$RespErr
return
}

— bare return on any exception, with no way to tell "confirmed ready" apart from "request failed early." Its caller doesn't check the return value either:

Start-Sleep 420
check_iDRAC_API_status_check
finish_bios_reset_to_default

So if iDRAC happens to still be unreachable right when the 7-minute sleep ends — the exact situation this issue is about — the readiness check would give up quietly on its first failed request, and the code would move straight into finish_bios_reset_to_default as if readiness had been confirmed, with nothing to indicate it wasn't.

Python's equivalent new function, check_idrac_connection(), takes a different approach and covers part of what we flagged, though not quite all of it. It pings the iDRAC IP first; if that ping already succeeds, the function returns right away without making an HTTP request at all, so it wouldn't catch "ICMP's up but the Redfish/HTTPS listener isn't yet" — a pretty plausible ordering during iDRAC's own boot. If the initial ping fails, it does retry both the ping and a follow-up GET with proper except requests.ConnectionError handling — though that retry has no overall timeout, so it could in theory run indefinitely if iDRAC never comes back. Worth knowing about separately, even if it's a different question from the one we raised. Either way, check_idrac_connection() is only called once, before the actual retry loop:

time.sleep(360)
check_idrac_connection()
count = 0
while True:
    ...
    response = requests.post(url, data=json.dumps(payload), headers=headers, verify=False, auth=...)
    if count == 5:
        sys.exit(0)
    if response.status_code == 204 or response.status_code == 202 or response.status_code == 200:
        ...

That requests.post(...) call (and the matching one a bit further down, inside the nested BIOS-status-check loop for GetRemoteServicesAPIStatus) still isn't wrapped in try/except — which was the specific thing we'd flagged originally. If the connection drops again between the pre-check and this POST (or the ping-succeeds fast path skipped verifying HTTP in the first place), it'd still raise an unhandled requests.ConnectionError and crash the script, the same way it would have before check_idrac_connection() was added.

Net effect on our side: we'll hang onto our tolerance logic around Invoke-SystemEraseREDFISH's post-completion polling (treating a 404-during-poll on a job we know exists as "probably fine, keep going" rather than a hard failure) — the reason we needed it originally still applies, and now it's relevant to the new readiness-check step too, not just the original fixed-wait.

Thanks again for the work on this one — happy to re-test whenever there's a follow-up to either language's retry handling.