Create Messaging Queue Microservice for Error Handling Using Kafka #144
Description
Overview
Build a dedicated messaging queue microservice leveraging Kafka to handle errors within a data-driven architecture.
Motivation
Current error handling mechanisms are distributed or inefficient for large-scale, data-driven workflows. Centralizing error capture, retry, and alerts through Kafka provides resilience and improves observability.
Requirements
- Design and implement a microservice using Kafka as the backbone for the message queue
- Ensure the service can:
- Capture errors from various services
- Queue and retry failed data events
- Integrate with existing data-driven architecture
- Expose monitoring/alerting for error topics
Acceptance Criteria
- Kafka-based microservice is deployed and integrated
- Documentation describing error queue protocols and integrations
- Error event flow diagrams and service architecture details provided
- Robust testing and rollback procedures in place
Additional Notes
- Evaluate use of Kafka Streams or connectors if beneficial
- Service should be language-agnostic for message formats where possible