Consider use UTF-8 by default for Azure CLI

[doggy8088]

Describe the bug

I was reported a bug on StackOverflow: https://stackoverflow.com/q/78008939/910074

When I have to use UTF-8 for my default console output encoding ([Console]::OutputEncoding), the Azure CLI unable to handle Chinese characters because Encoding issue. It cause either Chinese chars missing or produce messy code.

Related command

$(az account list -o json)

az account list -o json | jq '.'

Errors

Issue script & Debug output

It's an encoding issue.

Expected behavior

I expected Azure CLI can handle Chinese characters correctly.

Environment Summary

azure-cli 2.57.0

core 2.57.0
telemetry 1.1.0

Extensions:
account 0.2.3
azure-devops 0.25.0
front-door 1.0.16
interactive 0.4.5
k8s-extension 1.2.4
managementpartner 0.1.3

Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\wakau.azure\cliextensions'

Python (Windows) 3.11.7 (tags/v3.11.7:fa7a6f2, Dec 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

I have a workaround by now. Just edit C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmd file. Add -X utf8 to the python arguments.

::
:: Microsoft Azure CLI - Windows Installer - Author file components script
:: Copyright (C) Microsoft Corporation. All Rights Reserved.
::

@IF EXIST "%~dp0\..\python.exe" (
  SET AZ_INSTALLER=MSI
  "%~dp0\..\python.exe" -X utf8 -IBm azure.cli %*
) ELSE (
  echo Failed to load python executable.
  exit /b 1
)

[yonzhan]

Thank you for opening this issue, we will look into it.

[jiasli]

I am able to repro with the latest PowerShell 7.4.1. My system locale is English (United States):

Printing to console is fine:

> az group show -n testrg
{
  ...
  "tags": {
    ...
    "key1": "测试"
  },
  ...
}

But a warning is shown when redirecting:

> az group show -n testrg > out.txt
WARNING: Unable to encode the output with cp1252 encoding. Unsupported characters are discarded.

(Actually, I wrote that warning in microsoft/knack#178.)

According to https://docs.python.org/3/library/sys.html#sys.stdout

sys.stdout
Non-character devices such as disk files and pipes use the system locale encoding (i.e. the ANSI codepage).

So changing the console's encoding with [Console]::OutputEncoding = [Text.UTF8Encoding]::new() won't affect Python's output encoding.

I would recommend changing your system encoding to UTF-8 (follow microsoft/knack#178), so that you won't need to modify the az.cmd entry script every time you update Azure CLI.

Also see: python/cpython#74595