[Windows Only] Agent v5+ APPCRASH (Heap Corruption 0xc0000374) and TCP Socket Hang with Upstream Swarm

[Kruzerson]

1. Environment

OS: Windows Server (This issue is exclusively observed on Windows)

Fluent Bit Agent: v5.x (Tested on 5.0.3+)

Fluent Bit Server (Upstream): v5.0.5 running behind a Docker Swarm VIP

Use Case: Heavy multiline application logs buffering to filesystem and forwarding to a central aggregator.

2. Description
   There is a critical Windows-specific stability issue in the v5+ branch. The agent exhibits two distinct failure modes under moderate-to-heavy loads:

Issue A: Heap Corruption (APPCRASH)
The fluent-bit.exe service crashes silently. The Windows Event Viewer registers an APPCRASH event with exception code 0xc0000374 (Heap Corruption) in ntdll.dll. This is typically triggered when processing large multiline log blocks using the tail plugin combined with storage.type filesystem. When the crash occurs, in-flight memory chunks are lost.

Issue B: TCP Socket Hang (Deadlock)
When the upstream aggregator container (Docker Swarm) is restarted, the Windows agent completely freezes log transmission. The forward output plugin fails to detect the dropped connection (likely missing TCP RST from Swarm), keeps the "dead" socket open, and waits indefinitely for an ACK. The agent ignores net.io_timeout and net.keepalive_idle_timeout. Log flow only resumes after manually restarting the Windows service.

3. Agent Configuration to Reproduce

YAML

service:
  flush: 5
  storage.type: filesystem
  storage.path: "C:/fluent-bit/buffer/"
  storage.sync: normal
  storage.backlog.mem_limit: 50M
  storage.max_chunks_up: 256
  
pipeline:
  inputs:
    - name: tail
      path: "C:/AppLogs/*/*/*/*.log"
      tag: app.logs
      multiline.parser: custom_multiline
      read_from_head: false
      db: "C:/fluent-bit/buffer/tail.db"
      buffer_chunk_size: 5M
      buffer_max_size: 10M

  outputs:
    - name: forward
      match: "*"
      host: 10.0.0.100
      port: 24224
      require_ack_response: true
      retry_limit: no_retries
      net.connect_timeout: 5s
      net.io_timeout: 15s
      net.keepalive: on
      net.keepalive_idle_timeout: 10s

4. Expected Behavior

The tail plugin on Windows must handle large multiline logs without corrupting the heap memory (0xc0000374).

The forward plugin must strictly enforce net.io_timeout, forcefully close dead sockets when the upstream server goes away silently, and autonomously re-establish the connection without requiring a manual service restart.

[cosmo0920]

One additional Windows-specific hypothesis is that this issue may not be caused solely by the forward output connection itself, but rather by significantly increased internal event throughput introduced in the Fluent Bit v5 series.

Prior to v5, there were several bottlenecks in the ingestion and scheduling pipeline which effectively limited the maximum internal event rate. As a result, Windows-specific limitations in the internal event transport layer were much harder to trigger in practice.

On Windows, Fluent Bit still emulates some UNIX pipe/socketpair behavior using TCP loopback socket pairs. This mechanism consumes ephemeral ports, socket handles, and kernel networking resources, and has historically been a source of operational issues for Windows users under high load.

With the v5 engine optimizations, heavy tail + multiline + filesystem buffering workloads may now generate substantially higher internal signaling traffic. If the bundled event backend still falls back to a TCP-based socketpair implementation on Windows, this could dramatically increase:

• loopback TCP connections
• TIME_WAIT sockets
• ephemeral port consumption
• socket handle pressure
• WSA/kernel networking resource usage

Under these conditions, the observed symptoms may appear as:

• forward output hangs
• stalled ACK waits
• reconnection failures
• deadlocked upstream sockets
• general instability during downstream restarts

even if the root cause is actually exhaustion or degradation of the internal socketpair/event mechanism.

It may therefore be useful to compare the number of loopback TCP connections and TIME_WAIT sockets owned by fluent-bit.exe during reproduction, especially before and after restarting the upstream Swarm service.

For example, monitoring:

Get-NetTCPConnection -OwningProcess <PID>

may help determine whether the process is accumulating large numbers of loopback sockets under load.

A possible long-term mitigation would be to prefer AF_UNIX-based socketpair implementations on Windows 10+ (where supported), and only fall back to TCP loopback socketpairs on older systems where AF_UNIX is unavailable.

So, this might be related issue #11790.