Skip to content

Rework Syslog pool for less "connection errors" noise#3325

Merged
amokan merged 5 commits intoLogflare:mainfrom
ruslandoga:rd/syslog-no-connect-error
Apr 1, 2026
Merged

Rework Syslog pool for less "connection errors" noise#3325
amokan merged 5 commits intoLogflare:mainfrom
ruslandoga:rd/syslog-no-connect-error

Conversation

@ruslandoga
Copy link
Copy Markdown
Contributor

@ruslandoga ruslandoga commented Mar 31, 2026

This PR refactors the Syslog pool to prevent excessive log noise during connection failures.

Previously, the pool used NimblePool's async worker initialization. Since failing an async init in NimblePool requires raising or exiting, every failed connection attempt resulted in a logged crash -- noisy.

I shifted from {:async, ...} initialization to an :idle worker state on init. Sockets are now opened on-demand during checkout in the caller, similar to Finch HTTP1 pool.

CC @amokan :)

# close and remove sockets on any error
def handle_info({tag, socket, reason}, {socket, _backend_config})
def handle_info({tag, socket, reason}, {:connected, socket, _config})
when tag in [:tcp_error, :ssl_error] do
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a race condition here when :idle conn (in pool) becomes {:connected, ...} in the caller during checkout and for some reason errors out and sends the error message to its owner (pool). It gets resolved automatically on the subsequent checkout, but the caller gets a "socket is closed" error.

@ruslandoga ruslandoga changed the title Rework Syslog pool for no connection error noise Rework Syslog pool for less connection errors noise Apr 1, 2026
@ruslandoga ruslandoga changed the title Rework Syslog pool for less connection errors noise Rework Syslog pool for less "connection errors" noise Apr 1, 2026
@ruslandoga ruslandoga force-pushed the rd/syslog-no-connect-error branch from dc4e0fc to 1d5eb5a Compare April 1, 2026 13:12
Copy link
Copy Markdown
Contributor

@amokan amokan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@amokan amokan merged commit 234be27 into Logflare:main Apr 1, 2026
12 checks passed
@ruslandoga ruslandoga deleted the rd/syslog-no-connect-error branch April 1, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants