az synapse pipeline show does not return full json

[ross-p-smith]

I should be able to download the json of a pipeline and then use that very same json to create a pipeline

For instance if I run

artifact_json=$(az synapse ${artifact_type} show \
                --workspace-name "${SYNAPSE_WORKSPACE_NAME}" \
                --name ${artifact} \
                --output json)

I will be missing the top level "properties" key that contains "activities".

This means I cannot use this command

az synapse ${artifact_type} create --file "${artifact_json}" --name "${artifact_name}"\
                --workspace-name "${SYNAPSE_WORKSPACE_NAME}" "${args[@]}" -o json

you will get an error

  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/synapse/manual/operations/artifacts.py", line 80, in create_or_update_pipeline
    properties = PipelineResource.from_dict(definition_file['properties'])
KeyError: 'properties'

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 951246e6-c415-7f63-5655-8e590ba37171
• Version Independent ID: c9fe58c0-2a4f-6f46-43d7-c6942852923a
• Content: az synapse pipeline
• Content Source: latest/docs-ref-autogen/synapse/pipeline.yml
• GitHub Login: @rloutlaw
• Microsoft Alias: routlaw

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @wonner, @zesluo.

Issue Details

---

I should be able to download the json of a pipeline and then use that very same json to create a pipeline

For instance if I run

artifact_json=$(az synapse ${artifact_type} show \
                --workspace-name "${SYNAPSE_WORKSPACE_NAME}" \
                --name ${artifact} \
                --output json)

I will be missing the top level "properties" key that contains "activities".

This means I cannot use this command

az synapse ${artifact_type} create --file "${artifact_json}" --name "${artifact_name}"\
                --workspace-name "${SYNAPSE_WORKSPACE_NAME}" "${args[@]}" -o json

you will get an error

  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/synapse/manual/operations/artifacts.py", line 80, in create_or_update_pipeline
    properties = PipelineResource.from_dict(definition_file['properties'])
KeyError: 'properties'

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 951246e6-c415-7f63-5655-8e590ba37171
• Version Independent ID: c9fe58c0-2a4f-6f46-43d7-c6942852923a
• Content: az synapse pipeline
• Content Source: latest/docs-ref-autogen/synapse/pipeline.yml
• GitHub Login: @rloutlaw
• Microsoft Alias: routlaw

Author:	ross-p-smith
Assignees:	-
Labels:	Service Attention, needs-triage, Synapse
Milestone:	-

[yonzhan]

route to service team

[ross-p-smith]

I've looked into this and I think what is actually needed is an "export" for pipeline and trigger; like we have for notebook.

[ross-p-smith]

Sorry - another data point is that "show" does not return parameters or variables, just activities

[zesluo]

Thanks for your feedback! how do you download? like this?

[ross-p-smith]

Thanks @zesluo - but we require an api to do this so that is part of our DevOps process. Our pipelines our uploaded as part of our build and part of our dev cycle is to use Synapse and then we extract our workspace and commit to git. I know there is git support but that is focussed around central synapse instance rather than individual synapse instance per developer.

It still stands that the az cli does not have an export for the pipeline and triggers

[dayland]

As a follow up to this... az synapse pipeline show and az synapse trigger show provides all the necessary JSON components now. However you cannot simply take the JSON from the show option and use it in a create command. We were able to use some fairly simple jQuery to convert the output into a valid format...

Example:

artifact_json=$(az synapse pipeline show \
                --workspace-name "${SYNAPSE_WORKSPACE_NAME}" \
                --name ${artifact} \
                --output json)

artifact_json=$(echo "$artifact_json" | jq '. | { name: .name, properties: { description: .description, activities: .activities, parameters: .parameters, variables: .variables, annotations: .annotations }, type: .type }')

echo ${artifact_json} | jq -S "del(.etag, .id, .resourceGroup)" > "${DIR}/${artifact}.json"

The output of this is then usable for:

az synapse pipeline create \
                --file ${DIR}/${artifact}.json \
                --name ${artifact} \
                --workspace-name "${SYNAPSE_WORKSPACE_NAME}" \
                --output json

[microsoft-github-policy-service]

🔔 Routing this issue to @act-codegen-extensibility-squad.