Memory leak with version 1.9.2

[leigh-capa]

Hello, first time raising a defect like this, apologies if anything is incorrect.

We have some fluentd running in k8s, that takes in a bunch of data via the http endpoint. We noticed recently that the pods were being killed due to a memory leak. The primary developer was busy, so I (not a ruby dev) investigated with the aid of AI tooling.

Here’s some of the output from the AI:

Summary

Root cause: A bug in cool.io (the async I/O library used by fluentd) where disabled write watchers can never be properly detached.

The lifecycle for every HTTP connection:

1. Client sends request → fluentd reads it
2. Fluentd writes response → write watcher fires → buffer drains → write watcher is disabled (enabled=0, but stays in the event loop's @watchers hash)
3. Connection closes → close() tries to detach the write watcher → cool.io's C extension sees enabled==0 and silently returns without removing the watcher from @watchers
4. The write watcher in @watchers keeps the entire connection object graph alive: Watcher → EventHandler::TCPServer → TCPCallbackSocket → closures → Handler → Http::Parser (with request body strings) → TCPSocket → Coolio::Buffer

At ~100KB per request body, 5,318 leaked connections = ~530MB in strings alone, matching the 571MB we saw in the sigdump.

The fix: Re-enable the write watcher before detaching it, so the C-level guard doesn't fire and the watcher is properly removed from @watchers.

The changes are:

• src/patches/coolio_write_watcher_leak_fix.rb - monkey-patch for Coolio::IO#detach_write_watcher
• Dockerfile - loads src/patches/*.rb before plugins

After deploying the provided monkey patch, my pods seem to be much better:

2 hours in, zero restarts, memory completely flat:

┌───────┬────────┬────────┬────────┬────────┐
│ Pod │ 15:13 │ 15:32 │ 15:45 │ 16:10 │
├───────┼────────┼────────┼────────┼────────┤
│ d4mlm │ 960Mi │ 964Mi │ 960Mi │ 961Mi │
├───────┼────────┼────────┼────────┼────────┤
│ tlb5c │ 1004Mi │ 1001Mi │ 1006Mi │ 1004Mi │
├───────┼────────┼────────┼────────┼────────┤
│ zgldv │ 946Mi │ 949Mi │ 952Mi │ 947Mi │
└───────┴────────┴────────┴────────┴────────┘

At this point on the old image, these pods would have been ~200Mi higher and climbing. The leak is gone.

When I asked for a reference to the bug in cool.io, it gave me the following output:

It's a regression introduced just two weeks ago. The bug was added by PR #88 in cool.io, merged on 2026-02-09 and released in v1.9.2.

The backstory:

• Issue TypeError: wrong argument type nil (expected Data) when processing a detached watcher #87 reported a TypeError crash when a watcher was detached from one thread while the event loop processed an event for it in another
• PR Fix TypeError when processing a detached watcher #88 fixed that by adding an enabled == 0 guard in the Watcher_Detach macro to skip already-detached watchers
• But the guard conflates "disabled" (enabled=0, still in @watchers) with "detached" (loop=Qnil, removed from @watchers), causing our leak

The correct fix upstream would be to check watcher_data->loop == Qnil instead of watcher_data->enabled == 0. This hasn't been reported to cool.io yet - it's worth filing an issue at https://github.com/socketry/cool.io/issues referencing PR #88.

[2ZZ]

I also experienced this in FluentD v1.19.2, resolved by pinning cool.io to 1.9.1.

[Watson1978]

I've been investigating this issue and just submitted PR #90 to modernize our memory check CI using ruby_memcheck.

The good news is that the CI is now fully green. Assuming our test coverage is sufficient, this indicates that there are no pure C-level memory leaks (i.e., missing free() for malloc allocations).

[Watson1978]

If the fix introduced in PR #88 was the root cause of the memory leak,
we would expect a memory leak to occur when continuously running the script that reproduces the race condition reported in #87.

However, I verified this by running the race condition script in a loop, and no such memory leak occurred.

As shared in the benchmark results above, the memory usage remained completely stable across versions 1.9.1, 1.9.2, and 1.9.3 even when forcing the race condition repeatedly.

Verify script

require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'cool.io', '= 1.9.1'
  # gem 'cool.io', '= 1.9.2'
  # gem 'cool.io', '= 1.9.3'
end

class Victim < Cool.io::IOWatcher
  def initialize(io)
    super
    @io = io
  end

  def on_readable
    begin
      @io.read_nonblock(1024)
    rescue
    end
  end
end

event_loop = Cool.io::Loop.default
iteration = 0

puts "Starting Race Condition Leak Test... Cool.io: #{Cool.io::VERSION}"

loop do
  iteration += 1
  r_victim, w_victim = IO.pipe
  victim_watcher = Victim.new(r_victim)

  victim_watcher.attach(event_loop)

  Thread.new do
    sleep 0.01
    w_victim.write("dummy\n") rescue nil
  end
  Thread.new do
    sleep 0.01
    victim_watcher.detach rescue nil
  end

  event_loop.run_once rescue nil
  r_victim.close rescue nil
  w_victim.close rescue nil

  if iteration % 1000 == 0
    rss = `ps -o rss= -p #{Process.pid}`.to_i
    puts "#{iteration},#{rss}"
  end
end

Result

Here is the memory usage result at 20000 iterations.

cool.io 1.9.1	cool.io 1.9.2	cool.io 1.9.3
30976 KB	30688 KB	30832 KB

Environment

• OS: Linux 6.19.2-1-MANJARO
• ruby 4.0.1 (2026-01-13 revision e04267a14b) +PRISM [x86_64-linux]

[leigh-capa]

Once again, I feel the need to preface my comments with "I am not a ruby dev", there is a lot of AI assistance here.

@Watson1978 is investigating on issue #89 but hasn't reproduced it with their test script — their script exercises a simple IOWatcher attach/detach cycle, not the fluentd-specific write-then-disable-then-close path that triggers the leak. The
issue is still open.

I got Claude to put together a script to mimic how fluentd calls cool.io in a similar method to fluentd

require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'cool.io', '= 1.9.1'    # no leak
  # gem 'cool.io', '= 1.9.2'  # leaks
  # gem 'cool.io', '= 1.9.3'  # leaks
end

require 'socket'

# Reproduces the write watcher memory leak in cool.io >= 1.9.2.
#
# This exercises the exact path that fluentd's HTTP input takes:
#   1. Accept a TCP connection and wrap it in Coolio::IO
#   2. Write a response (attaches and enables the write watcher)
#   3. Event loop fires on_writable, buffer drains, disable_write_watcher
#      is called (write watcher: enabled=0, still in @watchers)
#   4. Connection closes -> close() calls detach_write_watcher
#      In v1.9.2+, detach sees enabled==0 and silently returns,
#      leaving the watcher in @watchers forever, pinning the entire
#      connection object graph in memory.
#
# Watson1978's script uses IOWatcher with explicit attach/detach, which
# never goes through this disable-then-detach path.

class HttpLikeHandler < Coolio::IO
  # Pad the response to make leaked memory more visible (~1KB per connection)
  RESPONSE = "HTTP/1.1 200 OK\r\nContent-Length: 1024\r\nConnection: close\r\n\r\n" + ("X" * 1024)

  def on_connect
  end

  def on_read(data)
    # Simulate fluentd: read request, write response.
    # write() -> schedule_write -> enable_write_watcher (attaches write watcher)
    # Event loop fires on_writable -> buffer drains -> disable_write_watcher
    # on_write_complete fires -> we close the connection
    write(RESPONSE)
  end

  def on_write_complete
    close
  end
end

PORT = 18989
loop_instance = Coolio::Loop.default

server = Coolio::TCPServer.new('127.0.0.1', PORT, HttpLikeHandler)
server.attach(loop_instance)

# Run the event loop in a background thread
event_thread = Thread.new { loop_instance.run }

REQUEST = "GET / HTTP/1.1\r\nHost: localhost\r\n\r\n"

puts "cool.io #{Coolio::VERSION} — write watcher leak test"
puts "iteration,rss_kb,watchers"

iteration = 0
loop do
  iteration += 1

  # Simulate an HTTP client: connect, send request, read response, disconnect
  begin
    sock = TCPSocket.new('127.0.0.1', PORT)
    sock.write(REQUEST)
    sock.read  # read until EOF (server closes after response)
    sock.close
  rescue => e
    sleep 0.01
    retry
  end

  if iteration % 500 == 0
    GC.start
    rss = `ps -o rss= -p #{Process.pid}`.to_i
    watchers = loop_instance.instance_variable_get(:@watchers)&.size || 0
    puts "#{iteration},#{rss},#{watchers}"
  end
end

Results:

Iterations	cool.io 1.9.1	cool.io 1.9.2	cool.io 1.9.3
500	36 MB / 1 watcher	36 MB / 501 watchers	36 MB / 501 watchers
5,000	39 MB / 1 watcher	49 MB / 5,001 watchers	49 MB / 5,001 watchers
10,000	39 MB / 1 watcher	58 MB / 10,001 watchers	58 MB / 10,001 watchers
20,000	37 MB / 1 watcher	76 MB / 20,001 watchers	76 MB / 20,001 watchers
50,000	37 MB / 1 watcher	136 MB / 50,001 watchers	137 MB / 50,001 watchers
100,000	37 MB / 1 watcher	239 MB / 100,001 watchers	239 MB / 100,001 watchers