SubType value ParquetReadSettings has no mapping. Synapse workspace Pipeline.

[jonathanhMS]

Describe the bug

Issue: The property ParquetReadSettings is missed in the JSON Synapse Pipeline definition after doing a CLI deployment process in his DevOps.

Error from Synapse after deployment

It is showed fine in the JSON definition but it removed this property after the deployment.

Before CLI

After CLI.

Troubleshoot.

Checking on CLI console we did see a warning during the deployment.

This completes but after it happens the JSON in the Synapse changed removing this property in it.

I was able to reproduce the issue using the metadata activity where is this property is located.

Steps to reproduce:
Create a pipeline with a metadata activity
Linked sevice to ADLS2
Parquet Data Set
Field List Exist
Save JSON as a JsonFIle in a local folder in the PC
run CLI command az synapse pipeline update --workspace-name synapseWorksapceName --name CLI_Test --file "@C:/Users/jonathanh/Documents/SYnapseCLITest/CLI_Pipeline.json"
A warning will be thrown
check the Pipeline in the Synapse UI It will show a JSON parse error
Verify the JSON definition, the ParqueReadSettinng porperty has been removed.__

Related command

az synapse pipeline update --workspace-name synapseWorkspaceName --name CLI_Test --file "@C:/Users/jonathanh/Documents/SYnapseCLITest/CLI_Pipeline.json"

Errors

Issue script & Debug output

{
"name": "CLI_Test",
"properties": {
"activities": [
{
"name": "Wait1",
"type": "Wait",
"dependsOn": [
{
"activity": "Get Metadata1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"waitTimeInSeconds": 1
}
},
{
"name": "Get Metadata1",
"type": "GetMetadata",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "Parquet1",
"type": "DatasetReference"
},
"fieldList": [
"exists"
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "ParquetReadSettings"
}
}
}
],
"annotations": []
}
}

Expected behavior

{
"name": "CLI_Test",
"properties": {
"activities": [
{
"name": "Wait1",
"type": "Wait",
"dependsOn": [
{
"activity": "Get Metadata1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"waitTimeInSeconds": 1
}
},
{
"name": "Get Metadata1",
"type": "GetMetadata",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "Parquet1",
"type": "DatasetReference"
},
"fieldList": [
"exists"
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"enablePartitionDiscovery": false
},
"formatSettings": { }
}
}
],
"annotations": []
}
}

Environment Summary

azure-cli 2.58.0

core 2.58.0
telemetry 1.1.0

Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\jonathanh.azure\cliextensions'

Python (Windows) 3.11.7 (tags/v3.11.7:fa7a6f2, Dec 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @wonner, @v-yanjungao.

[jonathanhMS]

Thanks for checking on this @yonzhan , is there any update on this? @

[jonathanhMS]

Hi @yonzhan , could you please share an update on this, this issue has been stopped our customer deployment process and it has been passed 2 weeks with no update, any comment will be useful and action plan from your side. Customer is asking for an ETA.