[logstash-input] Handle connection resets correctly#536
Conversation
| } | ||
| } else { | ||
| final Throwable realCause = extractCause(cause, 0); | ||
| if (logger.isDebugEnabled()) { |
There was a problem hiding this comment.
Move this into the if clause, otherwise it is nonsensical to log a "Handling" statement if no handling happens (i.e. when you end up in the "else" clause)
| } | ||
| // when execution tasks rejected, no need to forward the exception to netty channel handlers | ||
| if (cause instanceof RejectedExecutionException) { | ||
| if (logger.isDebugEnabled()) { |
There was a problem hiding this comment.
State you are handling the exception when you are handling the exception, i.e. inside the if clause.
| this.isQuietPeriod.compareAndSet(false, true); | ||
| } | ||
| } else { | ||
| if (logger.isDebugEnabled()) { |
There was a problem hiding this comment.
WARN that you are not handling the exception in all other cases.
| if ("Connection reset by peer".equals(message)) { | ||
| return true; | ||
| } | ||
| } else if (ex instanceof SocketException) { |
There was a problem hiding this comment.
Detect "Connection reset"s in form of a Socket Exception. I do not know if IOException is even correct but using an else clause should handle this.
| import org.apache.logging.log4j.Logger; | ||
|
|
||
| import java.io.IOException; | ||
| import java.net.SocketException; |
There was a problem hiding this comment.
Import necessary exception for handling.
There was a problem hiding this comment.
Pull request overview
Updates the Beats input’s Netty exception handling to better classify/handle connection reset scenarios and to log handled vs. unhandled exceptions more clearly, avoiding misleading “handled” logs followed by Netty pipeline warnings.
Changes:
- Adjusts
exceptionCaughtlogging to log “Handling exception” only for handledRejectedExecutionExceptions, and “Unhandled exception” when forwarding to Netty. - Extends noisy-exception detection to treat
SocketException: Connection resetas a connection reset case.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| private boolean isNoisyException(final Throwable ex) { | ||
| if (ex instanceof IOException) { | ||
| if (ex instanceof SocketException) { |
There was a problem hiding this comment.
Socket Exception is a subclass of IOException with a different Error message. Check first.
Connection resets can be thrown not only as IOExceptions but SocketExceptions too, which is unhandled. Current handling results in a log message that the case is handled, followed by an error and stack trace because it is not handled.
This change handles both cases,
a) detecting cases of connection resets, thrown as SocketException and closing the connection
b) corrects logging to indicate when an Error is handled and when it isn't.
Example of connection resets, observed in the wild:
Added 05.2026: This is caused by excessive health checks, done by e.g. monitoring solutions or loadbalancers, who back out of the connection with a RST instead of FIN. Confirmed with TCP/SSL check of checkmk.
P.S. Please backport to 8.x