Fix EventOutCh deadlock by making publish non-blocking#663
Fix EventOutCh deadlock by making publish non-blocking#663openshift-merge-bot[bot] merged 1 commit intoredhat-cne:mainfrom
Conversation
|
/gemini review |
PublishEventViaAPI sends to EventOutCh with a blocking channel send. When ProcessOutChannel (the consumer) is slow due to HTTP POST failures (e.g., unreachable subscribers), the 100-slot buffer fills up and the next publish blocks forever. This deadlocks the processMessages goroutine, freezing socket I/O and causing cloud-event-proxy to hang silently. Fix by using select with default to drop events when the channel is full instead of blocking indefinitely. Dropped events are logged as warnings and counted as failures in metrics. Signed-off-by: Jack Ding <jackding@gmail.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jzding, nocturnalastro The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/verified later @dpopsuev |
|
@jzding: This PR has been marked to be verified later by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-merge-bot
Bot
merged commit 409bff2
into
redhat-cne:main
|
/cherry-pick release-4.22 |
|
@jzding: new pull request created: #664 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
4 participants
PublishEventViaAPI sends to EventOutCh with a blocking channel send. When ProcessOutChannel (the consumer) is slow due to HTTP POST failures (e.g., unreachable subscribers), the 100-slot buffer fills up and the next publish blocks forever. This deadlocks the processMessages goroutine, freezing socket I/O and causing cloud-event-proxy to hang silently.
Fix by using select with default to drop events when the channel is full instead of blocking indefinitely. Dropped events are logged as warnings and counted as failures in metrics.