delete step fails to serialize dataset response

[trivedi234]

While deployment in ADO the process fails at the step where deletion of objects not existing in code happens with the error:

`STEP: Deleting objects not in source ...
Azure Data Factory (instance) loaded.
The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Unable to deserialize the response.
At /home/agent/.local/share/powershell/Modules/azure.datafactory.tools/1.14.0/public/Get-AdfFromService.ps1:40 char:21

• … .DataSets = Get-AzDataFactoryV2Dataset -ResourceGroupName "$ResourceG …

•           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

##[error]PowerShell exited with code '1'.
`
The problem seems to be related to Issue 473

As a consequence:

1. If we keep the DeleteNotInSource to true, then post the failure all the stopped triggers are not turned on causing an unexpected discontinuity of changed pipelines
2. If we keep the DeleteNotInSource to false, then already deployed pipelines could possibly overlap with their new incoming versions as the older ones cant be deleted anymore. AN causes a general code maintenance problems as nothing can be deleted in our set-up manually (no publish button enabled)

Here is the ADO power-shell task that we are using:

- task: AzurePowerShell@5 inputs: azureSubscription: 'xxx' ScriptType: 'InlineScript' Inline: | $opt = New-AdfPublishOption $opt.IncrementalDeployment = $true $opt.IncrementalDeploymentStorageUri = '$(adfStateStorageUri)' $opt.TriggerStopMethod = 'DeployableOnly' # $opt.DeleteNotInSource = $true Publish-AdfV2FromJson -RootFolder '$(System.ArtifactsDirectory)/factory' -ResourceGroupName "$(adfResourceGroup)" -DataFactoryName "$(adfName)" -Location "xxx" -Stage "tst" -Option $opt pwsh: true azurePowerShellVersion: 'LatestVersion' failOnStandardError: true

Expected behaviour is to be able to delete all the objects from datafactory not in the code.

PowerShell Module version: 1.14.0

[trivedi234]

related to issue 473 #473

[NowinskiK]

Hi @trivedi234, could you provide an example of a file that causes the deserialization issue?

[trivedi234]

Hi @NowinskiK Thanks for the swift response.

Since the trace doesnot indicate which file it encountered problem with (I assume since the response wasnt read it couldnt print the name of the dataset), let me provide you one of the dataset JSON as an example.

{ "name": "ds_adls_csv", "properties": { "parameters": { "storage_url": { "type": "String" }, "container": { "type": "String" }, "folder_path": { "type": "String" } }, "folder": { "name": "src_systems" }, "annotations": [], "type": "DelimitedText", "typeProperties": { "location": { "type": "AzureBlobFSLocation", "folderPath": { "value": "@dataset().folder_path", "type": "Expression" }, "fileSystem": { "value": "@dataset().container", "type": "Expression" } }, "columnDelimiter": ";", "escapeChar": "\\", "firstRowAsHeader": true, "quoteChar": "\"" }, "schema": [] } }

Why Dataset? Well thats what the error message pointed out, and looking at the PS src code for adf-tools I saw that the first line of code for retrieving the objects in the delete module is getting factory, followed by dataset. Seems the factory file was easily retrieved but the dataset deserialization failed, indicating a general problem (as in it might as well fail for other objects like pipeline and LS) with deserializing complex factory objects.

[trivedi234]

short update here, we found out that the problem is particularyl related to one specifc object which is a servicenow-v2 dataset, and the problem is at the az.datafactory level, but somehow there isnt a way to confirm if the servicenowv2 object is in general not supported by az. datafactory, nothing in the change logs.

[NowinskiK]

What version of Az.DataFactory do you have installed on the agent's machine?
The latest one is 1.19.7.

[trivedi234]

Yes, we have the latest version installed, I even forced installed 1.19.7 in debug run and the get-* cmdlet still fails for the servicenow dataset, it seems only this dataset is not supported everything else seems to be fine

[NowinskiK]

@trivedi234 thanks for your patience on this issue.
Preview version which includes the fix is release as 1.16.0-preview (https://www.powershellgallery.com/packages/azure.datafactory.tools/1.16.0-preview)
Please check, test and let me know if it works correctly.

[trivedi234]

@trivedi234 thanks for your patience on this issue. Preview version which includes the fix is release as 1.16.0-preview (https://www.powershellgallery.com/packages/azure.datafactory.tools/1.16.0-preview) Please check, test and let me know if it works correctly.

We have tested the 1.16.0 version which is now default latest version, and it still encounters the same failure. The root-cause continues to be serialization failure for servicenow dataset type which the az.datafactory clases dont fully support. Somehow the Azure CLI cmdlet are able to serialize them.