terraform apply: mssqlwin1 ConfigureSqlServer CustomScript extension failed (503 downloading MSI)

[doherty100]

Summary

A fresh terraform apply of the full sandbox (all base modules enabled) failed on a single resource: the configure_sql_server CustomScript extension on VM mssqlwin1 (vm_mssql_win module). All other resources — including the vwan P2S VPN gateway (31m26s) — provisioned successfully.

Failing command

terraform apply sandbox.tfplan

Step: Scenario 1 (fresh sandbox from scratch), unattended execution phase.

Full error output

Error: creating Extension (Subscription: "8e93423f-d08c-4539-ba3d-8cbc20bc6aa5"
Resource Group Name: "rg-sand-dev-dczipt6p"
Virtual Machine Name: "mssqlwin1"
Extension Name: "vmx-sand-dev-mssqlwin1-ConfigureSqlServer"): polling after CreateOrUpdate: polling failed: the Azure API returned the following error:

Status: "VMExtensionProvisioningError"
Code: ""
Message: "VM has reported a user failure when processing extension 'vmx-sand-dev-mssqlwin1-ConfigureSqlServer'. Please correct the error and try again. (publisher 'Microsoft.Compute' and type 'CustomScriptExtension'). Error code: '72'. Error message: 'Unable to download the msi. This may be due to firewall rules or a networking error. The exception was: The remote server returned an error: (503) Server Unavailable.'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot."

  with module.vm_mssql_win[0].azurerm_virtual_machine_extension.configure_sql_server,
  on modules/vm-mssql-win/compute.tf line 119, in resource "azurerm_virtual_machine_extension" "configure_sql_server":
 119: resource "azurerm_virtual_machine_extension" "configure_sql_server" {

API response timing: startTime 2026-07-08T23:56:01Z, endTime 2026-07-08T23:59:32Z, status Failed.

Context

• Branch: vnext (commit ca7e52c)
• Terraform: v1.15.8; providers: azurerm v4.80.0, azapi v2.10.0
• Modules enabled: all base (vnet_app, vm_jumpbox_linux, vm_mssql_win, mssql, mysql, vwan); vnet_shared always on. Extras disabled.
• Resource group: rg-sand-dev-dczipt6p; location centralus
• Only 1 error in the run; all other resources created successfully.

Assessment

The extension reported (503) Server Unavailable while downloading an MSI — a transient networking / remote-server availability error during the CustomScript download, not obviously a config defect. Per the repo error-handling policy, the workflow was stopped on first error without retry or self-patch.

[doherty100]

Corrected root cause — IMDS managed-identity token 503, NOT a storage/MSI-installer download

After pulling the CustomScriptExtension handler logs off mssqlwin1 via az vm run-command, the earlier "transient 503 downloading an MSI" framing is wrong and is retracted.

Evidence (from C:\WindowsAzure\Logs\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.22\CustomScriptHandler.log)

[07/08/2026 23:56:17.32] [INFO]  Downloading files specified in configuration...
[07/08/2026 23:56:17.63] [INFO]  Attempting to get MSI from IMDS
[07/08/2026 23:59:11.79] [ERROR] WebClient: retry attempts expired
[07/08/2026 23:59:11.79] [ERROR] Unknown exception occurred while attempting to get MSI token
                                 System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
   at ...MsiProvider.GetMsiHelper(NameValueCollection queries)
[07/08/2026 23:59:11.80] [FATAL] Failed to download all specified files. Exiting.
                                 Exception: Unable to download the msi. ... (503) Server Unavailable.

Handler telemetry: durationOfFileDownload: 174439 ms, filesDownloaded: 0.

What actually happened

• The extension sets protected_settings.managedIdentity = {}, so the CSE handler authenticates the fileUris download with the VM's system-assigned managed identity.
• To do that it first calls IMDS (169.254.169.254/metadata/identity/oauth2/token) for a token. IMDS returned HTTP 503 for the full ~174 s handler retry window, then the handler gave up.
• "MSI" in the log = Managed Service Identity token, not a Windows .msi installer. The download of the .ps1 scripts from the storage account was never attempted — it failed at token acquisition.

Not the cause

• Not the storage public-access barrier (an earlier hypothesis in this investigation — retracted; the handler never contacted storage).
• Not the type_handler_version (1.10 is the newest line; handler ran as 1.10.22).
• Not a script URL / firewall / storage-key issue.

Current state

IMDS on mssqlwin1 is healthy again (verified ~90 min later — it issued a https://storage.azure.com/ token successfully). This was a transient IMDS token-endpoint outage at the VM/platform level. It has now hit this exact step twice (see #553 comment + this issue), and the CSE handler's fixed ~174 s IMDS retry window is not long enough to ride out a longer IMDS blip.

Next steps

• Retrying the convergence apply now (IMDS healthy) — expected to succeed.
• Researching a durable resilience fix that removes the CSE handler's hard IMDS-token dependency at download time (SAS is not an option: storage account shared-key access is disabled). Findings to follow.

[doherty100]

Resilience analysis — durable fixes for the CSE / IMDS-503 download failure

Research into removing the fragile IMDS dependency in module.vm_mssql_win's configure_sql_server CustomScript extension. SAS is intentionally excluded from every option (the storage account has shared-key access disabled by the MCAPSGovDeployPolicies / StorageAccount_DisableLocalAuth_Modify management-group policy), and every option works over the existing blob private endpoint with the VM's system-assigned identity (Storage Blob Data Reader).

Why the current design can't just be tuned

The CSE handler performs the IMDS managed-identity token fetch itself, with a hard-coded ~174 s retry window. There is no settings/protectedSettings property to extend it (CSE property values). Microsoft's IMDS retry guidance (5 attempts / ~30 s, backoff to 60 s — error handling) is aimed at application code, not extension handlers. The observed IMDS 503 outage far exceeded 174 s, so no amount of CSE tuning (including the type_handler_version) can prevent this failure. Bumping the extension version does not help — 1.10 is already the newest line (ran as 1.10.22).

Key architectural fact

azurerm_virtual_machine_run_command with an inline source { script = file(...) } delivers the script body over the VM Guest Agent control channel — no IMDS token fetch is made to deliver the script (Run Command overview). This module already uses that exact pattern successfully for configure_firewall_rules and configure_sql_login (modules/vm-mssql-win/compute.tf:67-77), neither of which has ever exhibited this failure.

Ranked options (most → least robust)

Rank 1 — Inline run-command + user-controlled IMDS retry bootstrap.
Replace the CSE with one azurerm_virtual_machine_run_command whose inline bootstrap polls IMDS with a long (30–90 min) exponential backoff, then downloads the sibling scripts over the private endpoint and runs the orchestrator. Removes the fixed 174 s window entirely (retry duration/backoff become tunable in our own code). Native azurerm.

Rank 2 — Replace CSE with inline run-commands (write-then-run). Simplest; pattern already proven in this module.
Chain azurerm_virtual_machine_run_command resources using source { script = file(...) }. Scripts arrive via the VM-Agent channel, so the download-time IMDS dependency is eliminated. Only Connect-AzAccount -Identity inside the orchestrator still calls IMDS — now in user code that can be wrapped in a retry loop.

Rank 3 — azapi run-command with scriptUri + scriptUriManagedIdentity + artifacts.
Closest to today's design; managed-identity downloads, no SAS, no fixed 174 s window. But it retains the IMDS dependency (VM Agent fetches the token for each artifact), and the artifacts property is not yet surfaced in the azurerm provider — it requires azapi (already pinned at ~> 2.10.0).

Shared complication for Ranks 1–2

Invoke-MssqlConfiguration.ps1 locates its sibling scripts via $PSCommandPath. With inline delivery there is no co-located download directory, so the siblings (Set-MssqlConfiguration.ps1, Set-MssqlStartupConfiguration.ps1) must be written to a fixed path (e.g. C:\OrchestratorScripts\) and the orchestrator adjusted to reference that path. Inline source.script has a ~256 KB limit; the largest script is ~33 KB, so this is not a constraint.

Recommendation

Rank 2 — eliminates the download-time IMDS dependency, reuses a pattern already working in this module, and needs no new providers or secrets. Choose Rank 1 additionally if we also want to harden the orchestrator's own Connect-AzAccount -Identity IMDS call against a long outage.

Note on reproduction/verification: re-enabling public access to run a convergence apply is currently blocked in this subscription by a real-time management-group modify policy (MCAPSGovDeployPolicies → StorageAccount_PublicNetwork_Modify / Key Vault equivalent) that reverts publicNetworkAccess to Disabled on write. This does not affect the fix itself (the VM uses the private endpoint), only the operational retry path.