Skip to content

Handle :EXIT messages in ingestion WriteBuffer#6454

Open
SAY-5 wants to merge 5 commits into
plausible:masterfrom
SAY-5:fix-writebuffer-exit-trap
Open

Handle :EXIT messages in ingestion WriteBuffer#6454
SAY-5 wants to merge 5 commits into
plausible:masterfrom
SAY-5:fix-writebuffer-exit-trap

Conversation

@SAY-5

@SAY-5 SAY-5 commented Jun 18, 2026

Copy link
Copy Markdown

Changes

Plausible.Ingestion.WriteBuffer calls Process.flag(:trap_exit, true) so terminate/2 can flush the buffer on shutdown, but handle_info/2 only matched :tick. When a linked port or process (the ClickHouse connection socket) exits, the GenServer receives {:EXIT, port, :normal} and crashes with a FunctionClauseError, then restarts. This adds a handle_info/2 clause that ignores :EXIT messages so the buffer keeps running. Fixes #6382.

Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
@CLAassistant

CLAassistant commented Jun 18, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@aerosol

aerosol commented Jun 29, 2026

Copy link
Copy Markdown
Member

Thanks for your contribution. With regards to the test, it seems a bit pointless to send a message to an artificially spawned buffer - you could do that against the primary buffer as well, since nothing is expected to happen - except no crash. You could experiment with triggering sending the actual EXIT message via Process.exit if you like the test to be more meaningful.

But the PR states that this is ClickHouse socket dying - in which case terminate will try to flush with a dead connection, which may lead to another crash, right?

@SAY-5

SAY-5 commented Jul 2, 2026

Copy link
Copy Markdown
Author

Good points, thanks. I reworked the test to spawn a process, link it to the buffer, and kill it, so the buffer now receives a genuine {:EXIT, from, reason} through the trap-exit link rather than a hand-crafted send/2. Confirmed it fails with the FunctionClauseError from #6382 without the new clause and passes with it.

On the second point: you're right that the socket dying is a separate failure mode. The buffer isn't linked to the ClickHouse connection (IngestRepo.query! checks a connection out of the pool per call), so what #6382 actually reports is the {:EXIT, port, :normal} reaching an unmatched handle_info. A flush over a dead connection during terminate/2 would raise, but that path is independent of this crash and would be its own change, so I kept this PR scoped to swallowing the stray :EXIT. Happy to open a follow-up for the terminate/flush resiliency if you'd like it handled here.

@aerosol aerosol left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last round of changes if you don't mind and we're good to go

end

def handle_info({:EXIT, _from, _reason}, state) do
# We trap exits so terminate/2 can flush on shutdown. Linked ports and

@aerosol aerosol Jul 8, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is unnecessary, let's log the reason instead (warning level is fine).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bonus points for asserting the log entry in test 🤗

|> Map.to_list()
|> Keyword.put(:name, :"write_buffer_exit_#{System.unique_integer([:positive])}")
# large interval so the empty buffer never flushes to ClickHouse during the test
|> Keyword.put(:flush_interval_ms, 60_000)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about making this a bit more readable? Since compile_time_prepare result is a map, let's build a keyword list verbatim and only fetch necessary fields via the dot-notation access.

Something like:

test "...", %{test: test} do
opts = [
  header: @compile_time_opts.header,
  insert_sql: @compile_time_opts.insert_sql,
  insert_opts: @compile_time_opts.insert_opts,
  name: test, # unique atom is at our disposal already
  flush_interval_ms: 1_000_000_000 # no need to ever flush automatically on slow CI
]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FunctionClauseError in WriteBuffer.handle_info/2 on {:EXIT, port, :normal}

3 participants