Skip to content

[logstash-input] Handle connection resets correctly#536

Open
ash-darin wants to merge 4 commits into
logstash-plugins:mainfrom
ash-darin:patch-2
Open

[logstash-input] Handle connection resets correctly#536
ash-darin wants to merge 4 commits into
logstash-plugins:mainfrom
ash-darin:patch-2

Conversation

@ash-darin

@ash-darin ash-darin commented Apr 14, 2026

Copy link
Copy Markdown

Connection resets can be thrown not only as IOExceptions but SocketExceptions too, which is unhandled. Current handling results in a log message that the case is handled, followed by an error and stack trace because it is not handled.

This change handles both cases,

a) detecting cases of connection resets, thrown as SocketException and closing the connection
b) corrects logging to indicate when an Error is handled and when it isn't.

Example of connection resets, observed in the wild:

[2026-04-14T10:01:18,143][INFO ][org.logstash.beats.BeatsHandler][pipeline-name][0091cc855ef1087f43a697aa15a65059dde76fb10db72eac9d82c3a650a9376e] [local: 192.168.0.2:5001, remote: 192.168.0.11:36316] Handling exception: java.net.SocketException: Connection reset (caused by: java.net.SocketException: Connection reset)
[2026-04-14T10:01:18,143][WARN ][io.netty.channel.DefaultChannelPipeline][pipeline-name][0091cc855ef1087f43a697aa15a65059dde76fb10db72eac9d82c3a650a9376e] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.net.SocketException: Connection reset
[..]

Added 05.2026: This is caused by excessive health checks, done by e.g. monitoring solutions or loadbalancers, who back out of the connection with a RST instead of FIN. Confirmed with TCP/SSL check of checkmk.

P.S. Please backport to 8.x

}
} else {
final Throwable realCause = extractCause(cause, 0);
if (logger.isDebugEnabled()) {

@ash-darin ash-darin Apr 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this into the if clause, otherwise it is nonsensical to log a "Handling" statement if no handling happens (i.e. when you end up in the "else" clause)

}
// when execution tasks rejected, no need to forward the exception to netty channel handlers
if (cause instanceof RejectedExecutionException) {
if (logger.isDebugEnabled()) {

@ash-darin ash-darin Apr 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

State you are handling the exception when you are handling the exception, i.e. inside the if clause.

this.isQuietPeriod.compareAndSet(false, true);
}
} else {
if (logger.isDebugEnabled()) {

@ash-darin ash-darin Apr 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARN that you are not handling the exception in all other cases.

if ("Connection reset by peer".equals(message)) {
return true;
}
} else if (ex instanceof SocketException) {

@ash-darin ash-darin Apr 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Detect "Connection reset"s in form of a Socket Exception. I do not know if IOException is even correct but using an else clause should handle this.

import org.apache.logging.log4j.Logger;

import java.io.IOException;
import java.net.SocketException;

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import necessary exception for handling.

@ash-darin ash-darin marked this pull request as ready for review April 14, 2026 10:56
@robbavey robbavey requested a review from Copilot April 16, 2026 20:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Beats input’s Netty exception handling to better classify/handle connection reset scenarios and to log handled vs. unhandled exceptions more clearly, avoiding misleading “handled” logs followed by Netty pipeline warnings.

Changes:

  • Adjusts exceptionCaught logging to log “Handling exception” only for handled RejectedExecutionExceptions, and “Unhandled exception” when forwarding to Netty.
  • Extends noisy-exception detection to treat SocketException: Connection reset as a connection reset case.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/main/java/org/logstash/beats/BeatsHandler.java Outdated
Comment thread src/main/java/org/logstash/beats/BeatsHandler.java Outdated

private boolean isNoisyException(final Throwable ex) {
if (ex instanceof IOException) {
if (ex instanceof SocketException) {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Socket Exception is a subclass of IOException with a different Error message. Check first.

@robbavey robbavey self-assigned this May 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants