Skip to content

Add pipeline DLQ support for rds source#6817

Merged
oeyh merged 4 commits intoopensearch-project:mainfrom
oeyh:rds-source-pipeline-dlq
May 7, 2026
Merged

Add pipeline DLQ support for rds source#6817
oeyh merged 4 commits intoopensearch-project:mainfrom
oeyh:rds-source-pipeline-dlq

Conversation

@oeyh
Copy link
Copy Markdown
Collaborator

@oeyh oeyh commented May 4, 2026

Description

Adds pipeline DLQ support to the RDS source plugin for both the stream path (binlog/WAL) and the export path.

When a dlq_pipeline is configured, failed events are sent to it with _failure_metadata containing pluginId, pluginName, pipelineName, and errorMessage.

Stream path (BinlogEventListener / LogicalReplicationEventProcessor):

  • Row processing failures (data type conversion, record conversion)
  • Buffer write/flush failures

Export path (DataFileLoader):

  • Record processing failures during parquet file loading

When no dlq_pipeline is configured, behavior is unchanged — errors are logged and counters incremented.

Testing

Tested end-to-end locally against Aurora MySQL with a dlq_pipeline configured with a stdout sink.

Stream path:

  • Verified normal INSERT/UPDATE/DELETE events flow to the main pipeline with no DLQ output
  • Triggered row processing error and confirmed the failed event was sent to the DLQ with correct _failure_metadata

Export path:

  • Ran a full RDS export and verified records are processed normally
  • Triggered row processing error and confirmed failed events were sent to the DLQ with correct _failure_metadata

No DLQ configured:

  • Verified that when no dlq_pipeline is defined, failurePipeline is null and sendToFailurePipeline returns immediately with no side effects

Issues Resolved

N/A

Check List

  • New functionality includes testing
  • New functionality has been documented
  • Commits are signed off

@oeyh oeyh changed the title Add pipeline DLQ support for rds source stream path WIP: Add pipeline DLQ support for rds source stream path May 4, 2026
oeyh added 3 commits May 6, 2026 03:19
Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: Hai Yan <oeyh@amazon.com>
@oeyh oeyh force-pushed the rds-source-pipeline-dlq branch from b2c7aed to 108da49 Compare May 6, 2026 05:55
@oeyh oeyh changed the title WIP: Add pipeline DLQ support for rds source stream path Add pipeline DLQ support for rds source May 6, 2026
@oeyh oeyh marked this pull request as ready for review May 6, 2026 06:01
divbok
divbok previously approved these changes May 6, 2026
…gured

Signed-off-by: Hai Yan <oeyh@amazon.com>
@oeyh oeyh merged commit f190c3b into opensearch-project:main May 7, 2026
140 of 142 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants