Azure CLI azure-devops: az artifacts universal download to network share reports success but intermittently writes file with all zero byte content.

[sarathys]

Below is a repro of az artifacts universal download that intermittently writes all zero byte file which is expected to have non-zero byte content.

Of particular note and maybe related to the problem:

• File length is exactly 4194816 bytes = 4096.5 KB (exact power of two)
• .vmgs file is Hyper-V file, first 256+ bytes of file content are all zero bytes (in case that may confuse some check?)
• Intermittent problem seems to occur when saving to network share; do not see problem when saving to local drive (C:, D:, E:).
• Problem NOT seen when downloading other files in the same universal feed package (larger VHD files and such).
• Main problem is artifacttool.exe always reports SUCCESS whether content is incorrectly saved as all zero bytes or correctly with non-zero bytes.

az version

{
  "azure-cli": "2.45.0",
  "azure-cli-core": "2.45.0",
  "azure-cli-telemetry": "1.0.8",
  "extensions": {
    "azure-devops": "0.26.0"
  }
}

function CountBytes
{
    [OutputType([int[]])]
    param
    (
        [byte[]]$bytes
    )

    $count = [int[]]::new(256)
    foreach ($byte in $bytes)
    {
        $count[$byte]++
    }
    $count
}

#$destination = "D:\ArcA\arca.onenode.complete\23.307.10327.test"
$destination = "\\N35R0403-CL\ClusterStorage$\Infrastructure_1\Downloads\arca.onenode.complete\23.307.10327.test"

$results = @()

for ($n = 1; $n -le 10; $n++)
{
    [void](Remove-Item -Path $destination -Recurse -Force)
    [void](New-Item -Path $destination -ItemType Directory -Force)

    & az artifacts universal download --organization https://msazure.visualstudio.com/ --feed AzureStackUniversalBuddy --name arca.onenode.complete --version 23.307.10327 --path $destination --file-filter 'IRVM01/Virtual Machines/*.vmgs'
    Write-Verbose -Message "LASTEXITCODE: $LASTEXITCODE" -Verbose

    $file = Get-Item -Path "$destination\IRVM01\Virtual Machines\23F3E1A9-16FC-478C-B8F6-F35BACFBF989.vmgs"

    Write-Verbose -Message $file -Verbose
    $h = @{}
    $bytes = [System.IO.File]::ReadAllBytes($file.FullName) # Get-Content -Path $vmgsFile -Encoding Byte
    $count = CountBytes -bytes $bytes
    $isZeros = $count[0] -eq $file.Length
    if ($isZeros) { Write-Warning -Message "IsZeros: $isZeros" } else { Write-Verbose -Message "IsZeros: $isZeros" -Verbose }
    
    $results += [PSCustomObject]@{
        Attempt = $n
        IsZeros = $isZeros
    }
}

Write-Verbose -Message "Results:" -Verbose
$results | Format-Table -AutoSize

Output

VERBOSE: LASTEXITCODE: 0
VERBOSE: \\N35R0403-CL\ClusterStorage$\Infrastructure_1\Downloads\arca.onenode.complete\23.307.10327.test\IRVM01\Virtual Machines\23F3E1A9-16FC-478C-B8F6-F35BACFBF989.vmgs
WARNING: IsZeros: True
VERBOSE: Results:

Attempt IsZeros
------- -------
      1    True
      2    True
      3    True
      4    True
      5   False
      6    True
      7    True
      8   False
      9    True
     10    True

[azure-client-tools-bot-prd]

Hi @sarathys,

2.45.0 is not the latest Azure CLI(2.46.0).

Please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

Thank you for your feedback. This has been routed to the support team for assistance.

[yonzhan]

route to CXP team

[navba-MSFT]

Adding Service team to look into this issue.

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @v-anvashist, @V-hmusukula.

Issue Details

---

Below is a repro of az artifacts universal download that intermittently writes all zero byte file which is expected to have non-zero byte content.

Of particular note and maybe related to the problem:

• File length is exactly 4194816 bytes = 4096.5 KB (exact power of two)
• .vmgs file is Hyper-V file, first 256+ bytes of file content are all zero bytes (in case that may confuse some check?)
• Intermittent problem seems to occur when saving to network share; do not see problem when saving to local drive (C:, D:, E:).
• Problem NOT seen when downloading other files in the same universal feed package (larger VHD files and such).
• Main problem is artifacttool.exe always reports SUCCESS whether content is incorrectly saved as all zero bytes or correctly with non-zero bytes.

az version

{
  "azure-cli": "2.45.0",
  "azure-cli-core": "2.45.0",
  "azure-cli-telemetry": "1.0.8",
  "extensions": {
    "azure-devops": "0.26.0"
  }
}

function CountBytes
{
    [OutputType([int[]])]
    param
    (
        [byte[]]$bytes
    )

    $count = [int[]]::new(256)
    foreach ($byte in $bytes)
    {
        $count[$byte]++
    }
    $count
}

#$destination = "D:\ArcA\arca.onenode.complete\23.307.10327.test"
$destination = "\\N35R0403-CL\ClusterStorage$\Infrastructure_1\Downloads\arca.onenode.complete\23.307.10327.test"

$results = @()

for ($n = 1; $n -le 10; $n++)
{
    [void](Remove-Item -Path $destination -Recurse -Force)
    [void](New-Item -Path $destination -ItemType Directory -Force)

    & az artifacts universal download --organization https://msazure.visualstudio.com/ --feed AzureStackUniversalBuddy --name arca.onenode.complete --version 23.307.10327 --path $destination --file-filter 'IRVM01/Virtual Machines/*.vmgs'
    Write-Verbose -Message "LASTEXITCODE: $LASTEXITCODE" -Verbose

    $file = Get-Item -Path "$destination\IRVM01\Virtual Machines\23F3E1A9-16FC-478C-B8F6-F35BACFBF989.vmgs"

    Write-Verbose -Message $file -Verbose
    $h = @{}
    $bytes = [System.IO.File]::ReadAllBytes($file.FullName) # Get-Content -Path $vmgsFile -Encoding Byte
    $count = CountBytes -bytes $bytes
    $isZeros = $count[0] -eq $file.Length
    if ($isZeros) { Write-Warning -Message "IsZeros: $isZeros" } else { Write-Verbose -Message "IsZeros: $isZeros" -Verbose }
    
    $results += [PSCustomObject]@{
        Attempt = $n
        IsZeros = $isZeros
    }
}

Write-Verbose -Message "Results:" -Verbose
$results | Format-Table -AutoSize

Output

VERBOSE: LASTEXITCODE: 0
VERBOSE: \\N35R0403-CL\ClusterStorage$\Infrastructure_1\Downloads\arca.onenode.complete\23.307.10327.test\IRVM01\Virtual Machines\23F3E1A9-16FC-478C-B8F6-F35BACFBF989.vmgs
WARNING: IsZeros: True
VERBOSE: Results:

Attempt IsZeros
------- -------
      1    True
      2    True
      3    True
      4    True
      5   False
      6    True
      7    True
      8   False
      9    True
     10    True

Author:	sarathys
Assignees:	-
Labels:	Service Attention, Artifacts, DevOps, needs-team-attention
Milestone:	-

[navba-MSFT]

@v-anvashist, @V-hmusukula. Could you please look into this once you get a chance ?

[aluty]

Attached --debug log with the repro of the problem.

1. First download (incorrectly) reports success but creates file with all zero byte content.
2. Second download overwrites file, sets correct non-zero byte content, and reports success.

Note this is an intermittent issue and may only repro on cluster network share (with reparse point https://learn.microsoft.com/en-us/windows-hardware/drivers/ifs/reparse-points).

az artifacts universal download to network share bug.txt