{CI} Fix extension release scripts by wangzelin007 · Pull Request #7275 · Azure/azure-cli-extensions

wangzelin007 · 2024-02-08T04:28:28Z

Modify the code according to #7273

This checklist is used to make sure that common guidelines for a pull request are followed.

Related command

General Guidelines

Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required)
My extension version conforms to the Extension version schema

For new extensions:

My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish

There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

azure-client-tools-bot-prd · 2024-02-08T04:28:31Z

️✔️Azure CLI Extensions Breaking Change Test

️✔️Non Breaking Changes

azure-client-tools-bot-prd · 2024-02-08T04:28:34Z

Hi @wangzelin007,
Please write the description of changes which can be perceived by customers into HISTORY.rst.
If you want to release a new extension version, please update the version in setup.py as well.

yonzhan · 2024-02-08T04:28:38Z

CI

scripts/ci/sync_extensions.py

jiasli · 2024-02-08T06:38:12Z

I would recommend we postpone this PR until we are back from holiday.

github-actions · 2024-05-31T07:30:42Z

For more info about extension versioning, please refer to Extension version schema

This reverts commit 08f3b0a.

Copilot

Pull request overview

Updates CI/release helper scripts used to sync/publish Azure CLI extensions and generate/upload the extension command tree, aligning with #7273 (UTF-8 command tree session + idempotent blob uploads) and adding additional build-time adjustments.

Changes:

Introduce run_az_cmd wrapper and use it for az storage blob operations (including --overwrite) in sync/upload flows.
Update command-tree generation to use UTF-8 session encoding and adjust command-tree build blocklist / dependencies.
Add runtime package inspection/upgrade logic in update_ext_cmd_tree.py for the ml extension path.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
scripts/ci/util.py	Adds `run_az_cmd` wrapper and switches to `subprocess.check_output` usage.
scripts/ci/update_ext_cmd_tree.py	Uses UTF-8 session for cmd tree, switches upload/url calls to `run_az_cmd`, and adds package upgrade helpers/logic.
scripts/ci/sync_extensions.py	Replaces direct `subprocess.run` calls with `run_az_cmd` for storage operations.
scripts/ci/build_ext_cmd_tree.sh	Adjusts blocklist and changes when dependencies are installed (plus adds `regex` install).

Comments suppressed due to low confidence (1)

scripts/ci/build_ext_cmd_tree.sh:35

build_ext_cmd_tree.sh now runs az extension list-available / az extension add before installing azure-cli/azure-cli-core. This makes the script dependent on az already being present on the agent image; previously it bootstrapped those dependencies itself. Consider either restoring the early install (or adding a fast preflight check for az) to avoid failures on images without Azure CLI preinstalled.

echo "Listing Available Extensions:"
az extension list-available -otable

# turn off telemetry as it crowds output
export AZURE_CORE_COLLECT_TELEMETRY=False

# use index.json in master branch
export AZURE_EXTENSION_INDEX_URL=https://raw.githubusercontent.com/Azure/azure-cli-extensions/master/src/index.json

output=$(az extension list-available --query [].name -otsv)
# azure-cli-ml is replaced by ml
# disable alias which relies on Jinja2 2.10
# disable rdbms-connect and deploy-to-azure which cause cmd tree build failures
blocklist=("azure-cli-ml" "alias" "rdbms-connect" "deploy-to-azure")

rm -f ~/.azure/extCmdTreeToUpload.json

filter_exts=""
for ext in $output; do
    ext=${ext%$'\r'}   # Remove a trailing newline when running on Windows.
    if [[ " ${blocklist[@]} " =~ " ${ext} " ]]; then
        continue
    fi
    filter_exts="${filter_exts} ${ext}"
    echo "Adding extension:" $ext
    az extension add --upgrade -n $ext
    if [ $? != 0 ]

Copilot · 2026-04-01T09:06:29Z

scripts/ci/util.py

+
+    if message:


run_az_cmd prints twice when message is True: it prints the default "Running: ..." line and then prints the boolean value (True) because the subsequent if message: branch also runs. Consider using elif message is not False: (or checking for isinstance(message, str)) so True triggers only the default message and never prints the literal boolean.

Suggested change

if message:

elif isinstance(message, str):

Copilot · 2026-04-01T09:06:30Z

scripts/ci/util.py

+    try:
+        result = subprocess.run(cmd, capture_output=True, check=True)
+        return result
+    except subprocess.CalledProcessError as ex:
+        error_message = ex.stderr if ex.stderr else str(ex)
+        if raise_error:
+            raise Exception(f"Failed to run command: {cmd}\nError: {error_message}") from ex


On failure, error_message = ex.stderr will usually be bytes (because subprocess.run is not using text=True). This makes the raised Exception message hard to read (e.g., b'...'). Consider running with text=True (or decoding ex.stderr) so error output is readable and consistent.

Copilot · 2026-04-01T09:06:30Z

scripts/ci/update_ext_cmd_tree.py

+    """Execute a shell command and return the output."""
+    try:
+        result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+        if result.returncode == 0:
+            return result.stdout.strip()
+        else:
+            return f"Error: {result.stderr.strip()}"
+    except Exception as e:
+        return f"Exception: {str(e)}"


The new execute_command helper swallows failures by returning strings like "Error: ..." / "Exception: ...", but callers (e.g., upgrade_package) don't check these and the script continues. For CI tooling, it would be safer to raise on non-zero return codes (or return (rc, stdout, stderr) and explicitly fail) so command-tree generation doesn’t silently proceed with a broken environment.

Suggested change

"""Execute a shell command and return the output."""

try:

result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

if result.returncode == 0:

return result.stdout.strip()

else:

return f"Error: {result.stderr.strip()}"

except Exception as e:

return f"Exception: {str(e)}"

"""Execute a shell command and return the output.

Raises a RuntimeError if the command exits with a non-zero status.

"""

try:

result = subprocess.run(

command,

stdout=subprocess.PIPE,

stderr=subprocess.PIPE,

text=True,

check=True,

)

return result.stdout.strip()

except subprocess.CalledProcessError as e:

# Raise an explicit error so callers cannot accidentally ignore failures.

stderr = (e.stderr or "").strip()

raise RuntimeError(

f"Command '{' '.join(command)}' failed with exit code {e.returncode}: {stderr}"

) from e

Copilot · 2026-04-01T09:06:31Z

scripts/ci/update_ext_cmd_tree.py

+def get_package_version(package_name):
+    """Get the current version of a Python package."""
+    command = ["pip", "show", package_name]
+    output = execute_command(command)
+    if "Version:" in output:
+        for line in output.splitlines():
+            if line.startswith("Version:"):
+                version = line.split(":")[1].strip()
+                print(f"{package_name} current version: {version}")
+
+
+def upgrade_package(package_name):
+    """Upgrade a Python package to the latest version."""
+    command = ["pip", "install", "--upgrade", package_name]
+    print(f"{command}")
+    return execute_command(command)


get_package_version / upgrade_package invoke pip directly (e.g., ["pip", "install", ...]). In CI this can target a different interpreter than the one running this script. Prefer using sys.executable -m pip to ensure packages are queried/installed into the same environment used to import azure.cli.*.

Copilot · 2026-04-01T09:06:31Z

scripts/ci/update_ext_cmd_tree.py

+    if ext_name == 'ml':
+        get_package_version("azure-storage-blob")
+        upgrade_package("azure-storage-blob")
+        get_package_version("azure-storage-blob")
+        get_package_version("rpds")
+        upgrade_package("rpds")
+        get_package_version("rpds")
+        get_package_version("rpds-py")
+        upgrade_package("rpds-py")
+        get_package_version("rpds-py")


update_cmd_tree now conditionally upgrades packages at runtime when ext_name == 'ml' (azure-storage-blob, rpds, rpds-py). This is a significant behavior change that can make command-tree builds non-reproducible and can introduce unexpected dependency changes mid-run. If this is required, consider moving these pins/upgrades into the CI environment setup (e.g., the shell script) and documenting why only ml needs it.

Suggested change

if ext_name == 'ml':

get_package_version("azure-storage-blob")

upgrade_package("azure-storage-blob")

get_package_version("azure-storage-blob")

get_package_version("rpds")

upgrade_package("rpds")

get_package_version("rpds")

get_package_version("rpds-py")

upgrade_package("rpds-py")

get_package_version("rpds-py")

Copilot · 2026-04-01T09:06:31Z

scripts/ci/update_ext_cmd_tree.py

 def update_cmd_tree(ext_name):
    print(f"Processing {ext_name}")
+    if ext_name == 'ml':
+        get_package_version("azure-storage-blob")
+        upgrade_package("azure-storage-blob")
+        get_package_version("azure-storage-blob")
+        get_package_version("rpds")
+        upgrade_package("rpds")
+        get_package_version("rpds")
+        get_package_version("rpds-py")
+        upgrade_package("rpds-py")
+        get_package_version("rpds-py")


This PR is described as updating scripts per #7273 (encoding + --overwrite), but it also adds runtime package upgrades for ml and new helper functions. If these additions are intentional, it would help to update the PR description (or link the specific issue) so reviewers understand the extra scope and rationale.

wangzelin007 added 8 commits February 7, 2024 16:01

update

2517e69

update

bc3aed8

update

f2b7bab

update

87f519a

update

8906e0b

Update update_ext_cmd_tree.py

b6017d4

Update update_ext_cmd_tree.py

2c3d7a3

update

5de410e

azure-client-tools-bot-prd bot added this to the February 2024 (2024-03-05) milestone Feb 8, 2024

microsoft-github-policy-service bot requested review from bebound, jiasli, jsntcy, kairu-ms and yonzhan February 8, 2024 04:28

microsoft-github-policy-service bot added Auto-Assign Auto assign by bot CI labels Feb 8, 2024

microsoft-github-policy-service bot assigned wangzelin007 Feb 8, 2024

Merge branch 'main' into release-scripts

0c3bb06

wangzelin007 marked this pull request as ready for review February 8, 2024 04:50

jiasli reviewed Feb 8, 2024

View reviewed changes

scripts/ci/sync_extensions.py Outdated Show resolved Hide resolved

jiasli reviewed Feb 8, 2024

View reviewed changes

scripts/ci/sync_extensions.py Outdated Show resolved Hide resolved

wangzelin007 added 5 commits February 8, 2024 15:14

update

bfe5ed1

Update sync_extensions.py

8a844b7

Update util.py

29f00c6

Merge branch 'main' into release-scripts

148eaf3

Update util.py

e8c767a

wangzelin007 added 12 commits May 31, 2024 16:14

Update util.py

212f082

Create build_all_ext.sh

08f3b0a

Revert "Create build_all_ext.sh"

bb4a4e2

This reverts commit 08f3b0a.

Update build_ext_cmd_tree.sh

218ca95

Update build_ext_cmd_tree.sh

690b7ce

Update build_ext_cmd_tree.sh

e8572a7

Update build_ext_cmd_tree.sh

7447487

Update build_ext_cmd_tree.sh

868a7b2

Update build_ext_cmd_tree.sh

80a3c1c

Update build_ext_cmd_tree.sh

6ac4465

Update build_ext_cmd_tree.sh

9beaa14

Merge branch 'main' into release-scripts

8578db6

wangzelin007 modified the milestones: February 2024 (2024-03-05), Backlog Nov 26, 2024

wangzelin007 added 6 commits December 20, 2024 12:27

generate command tree for ml

ac63598

Merge branch 'main' into release-scripts

50d18b4

Merge branch 'main' into release-scripts

6d0db41

generate command tree for ml

9c84141

generate command tree for ml

eac3fb4

fix: add rdbms-connect and deploy-to-azure to cmd tree blocklist

49d26d1

Copilot AI review requested due to automatic review settings April 1, 2026 08:57

Copilot started reviewing on behalf of wangzelin007 April 1, 2026 09:00 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Conversation

wangzelin007 commented Feb 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related command

General Guidelines

About Extension Publish

Uh oh!

azure-client-tools-bot-prd bot commented Feb 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azure-client-tools-bot-prd bot commented Feb 8, 2024

Uh oh!

yonzhan commented Feb 8, 2024

Uh oh!

Uh oh!

Uh oh!

jiasli commented Feb 8, 2024

Uh oh!

github-actions bot commented May 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wangzelin007 commented Feb 8, 2024 •

edited

Loading

azure-client-tools-bot-prd bot commented Feb 8, 2024 •

edited

Loading

github-actions bot commented May 31, 2024 •

edited

Loading