az ml batch-endpoint invoke wants an AccountKey or SasToken for registered Datastores for output

[neilmca-inc]

From here https://learn.microsoft.com/en-us/cli/azure/ml/batch-endpoint?view=azure-cli-latest#az-ml-batch-endpoint-invoke

This command works as the output is to the registered internal Azure Machine Learning Default Datastore...

az ml batch-endpoint invoke --name mybatchendpoint --input https://azuremlexampledata.blob.core.windows.net/data/mnist/sample --input-type uri_folder --output-path azureml://datastores/workspaceblobstore/paths/mybatchendpoint --set output_file_name=predictions.csv --query name -o tsv --resource-group myrg --workspace-name myamlworkspace

When I want my output to go to another Storage Account location - I have pre-registered it in AML Studio as follows...(under Data > Datastores)

Datastore name: data_mystorageaccount
Datastore type: Azure Blob Storage
Subscription ID: {redactedmyAzureSub}
Storage account: mystorageaccount
Blob container: output
Save credentials with the datastore for data access - enabled
Authentication type: Account key
Account key: {the account key from the storage account mystorageaccount}
Clicked Create

From looking in the Studio it can browse to that output location - see image - it can see a file already called ServiceTags_Public_20230306.json in the right hand pane...

This proves connection to the Storage Account via the AccountKey as successful.

When I run the following command...

az ml batch-endpoint invoke --name mybatchendpoint --input https://azuremlexampledata.blob.core.windows.net/data/mnist/sample --input-type uri_folder --output-path azureml://datastores/data_mystorageaccount/paths/mybatchendpoint --set output_file_name=predictions.csv --query name -o tsv --resource-group myrg --workspace-name myamlworkspace

...it fails with the following output error Missing AccountKey or SasToken

ERROR: {
  "error": {
    "code": "UserError",
    "severity": null,
    "message": "Missing AccountKey or SasToken",
    "messageFormat": null,
    "messageParameters": null,
    "referenceCode": null,
    "detailsUri": null,
    "target": null,
    "details": [],
    "innerError": null,
    "debugInfo": null,
    "additionalInfo": null
  },
  "correlation": {
    "operation": "{redacted}",
    "request": "{redacted}"
  },
  "environment": "{redacted}",
  "location": "{redacted}",
  "time": "2023-03-16T13:41:12.1918203+00:00",
  "componentName": "managementfrontend"
}

It states on the Datastores screen in AML Studio the following...

Datastores securely connect to a storage service on Azure by storing connection information. With datastores, you no longer need to provide credential information in your scripts to access your data

...so I'm not sure why this invoke command fails? Why does it need a AccountKey or SasToken passed to it as part of the az ml batch-endpoint invoke command?

Even if I did need to pass a AccountKey or SasToken as part of this command, is it supported here? There are no examples listed in the documentation for output types

---

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

• ID: 5233a95c-48d7-d49b-3e21-b42c94a31290
• Version Independent ID: 60a98b3a-a7b9-ae58-ab5e-4b1d086a7dd0
• Content: az ml batch-endpoint
• Content Source: latest/docs-ref-autogen/ml/batch-endpoint.yml
• GitHub Login: @rloutlaw
• Microsoft Alias: routlaw

[yonzhan]

route to CXP team

[kartikeyavs]

Are there any findings on this particular issue? I am seeing this issue in our Azure tenent.

[santiagxf]

Thanks for reporting this issue to us. We are happy to help. Can you please run the same command but with the flag --debug? That would be:

az ml batch-endpoint invoke --name mybatchendpoint \
                            --input https://azuremlexampledata.blob.core.windows.net/data/mnist/sample \
                            --input-type uri_folder \
                            --output-path azureml://datastores/workspaceblobstore/paths/mybatchendpoint \
                            --set output_file_name=predictions.csv \
                            --resource-group myrg \
                            --workspace-name myamlworkspace  \
                            --debug

Please remove any PII information before sharing it. Also, it will help to know if thi workspace is under a private VNET?

[jsfsn]

We are having the same issue.

[borgewi]

We have the same issue @santiagxf. It stopped working around October 5th and seems to be an issue with Azure CLI.

[yacuzo]

Same issue. Both on an API calling AML endpoint using REST and in Azure CLI. Seems like the issue is on AML itself. Ouput (and input) storage are in another VNET with service-endpoint.

Scrubbed Log: aml output auth error.txt

[santiagxf]

We are investigating this issue. I will provide an update within the day.

[borgewi]

@santiagxf Any update here?

[santiagxf]

This issue is under investigation and we are trying to identify the root of the issue with the logs provided. I will update the thread as soon as we have an update. As a workaround, can you try out on a new created endpoint? I can't reproduce the issue from my side so it may be related with existing ones.