Skip to content

In workflow, enrichment listed in apply are not replayed if they are listed in skipped #438

@tifennlegoff

Description

@tifennlegoff

Hey, I made this PR to fix the issue that I currently have, it's related with this Slack Thread.

Currently have an issue, all enrichments listed in apply meta are not replayed. We added this processor to replay all enrichment, even skipped ones because, some of them can be skipped because of a dependencies issue with the failed one. We added this processor before the workflow:

- label: replay_skipped
  switch:
    - check: this.meta.enrichments.exists("skipped") && this.meta.enrichments.skipped.length() > 0
      processors:
        - label: replay_skipped_enrichments
          bloblang: |-
            root = this
            root.meta.enrichments.apply = this.meta.enrichments.failed.keys().merge(this.meta.enrichments.skipped)

Our workflow looks like that

---
processor_resources:
  - label: enrichments
    workflow:
      meta_path: meta.enrichments
      order:
...

But, in the document resulted, the enrichments listed in apply are not replayed, only the one previously in failed is replayed. They are not listed in succeeded, they are in skipped and the field added by enrichment are not present.

I'm not expert in Go but in my PR, if I well understand, in function skipFromMeta, the if skipped section should be before the if apply one because the if apply will remove the enrichment from the list skipList filled by the if skipped section.

I wrote test to validate if we have the expected result and it's works 🍾

I run tests for others functions TestWorkflows, TestWorkflowsDeps, TestWorkflowsParallel and TestWorkflowsWithOrderResources

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions