Skip to content

Build skipList before exclude enrichment in apply section#437

Open
tifennlegoff wants to merge 1 commit into
redpanda-data:mainfrom
tifennlegoff:workflow_switch_apply_retry
Open

Build skipList before exclude enrichment in apply section#437
tifennlegoff wants to merge 1 commit into
redpanda-data:mainfrom
tifennlegoff:workflow_switch_apply_retry

Conversation

@tifennlegoff
Copy link
Copy Markdown

@tifennlegoff tifennlegoff commented May 21, 2026

(This PR is related with this issue)

Currently have an issue, all enrichments listed in apply meta are not replayed. We added this processor to replay all enrichment, even skipped ones because, some of them can be skipped because of a dependencies issue with the failed one. We added this processor before the workflow:

- label: replay_skipped
  switch:
    - check: this.meta.enrichments.exists("skipped") && this.meta.enrichments.skipped.length() > 0
      processors:
        - label: replay_skipped_enrichments
          bloblang: |-
            root = this
            root.meta.enrichments.apply = this.meta.enrichments.failed.keys().merge(this.meta.enrichments.skipped)

Our workflow looks like that

---
processor_resources:
  - label: enrichments
    workflow:
      meta_path: meta.enrichments
      order:
...

But, in the document resulted, the enrichments listed in apply are not replayed, only the one previously in failed is replayed. They are not listed in succeeded, they are in skipped and the field added by enrichment are not present.

I'm not expert in Go but in my PR, if I well understand, in function skipFromMeta, the if skipped section should be before the if apply one because the if apply will remove the enrichment from the list skipList filled by the if skipped section.

I wrote test to validate if we have the expected result and it's works 🍾

I run tests for others functions TestWorkflows, TestWorkflowsDeps, TestWorkflowsParallel and TestWorkflowsWithOrderResources

I run workflow tests TestWorkflows, TestWorkflowsDeps, TestWorkflowsParallel and TestWorkflowsWithOrderResources

⋊> ~/D/benthos on workflow_switch_apply_retry  go test -v ./internal/impl/pure/ -run TestWorkflows                                               
=== RUN   TestWorkflows
=== RUN   TestWorkflows/0
=== RUN   TestWorkflows/1
=== RUN   TestWorkflows/2
=== RUN   TestWorkflows/3
=== RUN   TestWorkflows/4
--- PASS: TestWorkflows (0.01s)
    --- PASS: TestWorkflows/0 (0.00s)
    --- PASS: TestWorkflows/1 (0.00s)
    --- PASS: TestWorkflows/2 (0.00s)
    --- PASS: TestWorkflows/3 (0.00s)
    --- PASS: TestWorkflows/4 (0.00s)
=== RUN   TestWorkflowsWithResources
=== RUN   TestWorkflowsWithResources/0
=== RUN   TestWorkflowsWithResources/1
=== RUN   TestWorkflowsWithResources/2
=== RUN   TestWorkflowsWithResources/3
--- PASS: TestWorkflowsWithResources (0.00s)
    --- PASS: TestWorkflowsWithResources/0 (0.00s)
    --- PASS: TestWorkflowsWithResources/1 (0.00s)
    --- PASS: TestWorkflowsWithResources/2 (0.00s)
    --- PASS: TestWorkflowsWithResources/3 (0.00s)
=== RUN   TestWorkflowsParallel
--- PASS: TestWorkflowsParallel (0.46s)
=== RUN   TestWorkflowsWithOrderResources
=== RUN   TestWorkflowsWithOrderResources/0
=== RUN   TestWorkflowsWithOrderResources/1
=== RUN   TestWorkflowsWithOrderResources/2
=== RUN   TestWorkflowsWithOrderResources/3
--- PASS: TestWorkflowsWithOrderResources (0.01s)
    --- PASS: TestWorkflowsWithOrderResources/0 (0.00s)
    --- PASS: TestWorkflowsWithOrderResources/1 (0.00s)
    --- PASS: TestWorkflowsWithOrderResources/2 (0.00s)
    --- PASS: TestWorkflowsWithOrderResources/3 (0.00s)
PASS
ok      github.com/redpanda-data/benthos/v4/internal/impl/pure  0.997s

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 21, 2026

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants