OA-1090: silent write failures and stale idle connections in ServerChannels

[siddharthteotia]

Summary

Fixes two independent bugs in ServerChannels where dead or stale Netty connections caused queries to silently hang until timeout instead of failing fast.

Bug 1 — writeAndFlush() failure not propagated

Before (broken path):

Broker thread
    │
    ├─ connectWithoutLocking()
    │       └─ _channel.isActive() == true  ✓
    │
    ├─ writeAndFlush(requestBytes)
    │       └─ ChannelFuture listener fires:
    │               f.isSuccess()  ← NEVER CHECKED
    │               markRequestSent()  ← always called, even on failure
    │                       │
    │                       ▼
    │               AsyncQueryResponse thinks request was sent
    │               Server never received it → query hangs until timeout
    │
    └─ [query times out after 30s+ with no useful error]

After (fixed path):

Broker thread
    │
    ├─ writeAndFlush(requestBytes)
    │       └─ ChannelFuture listener fires:
    │               if (!f.isSuccess())
    │                   markServerDown()  ← all in-flight queries fail fast
    │                   return
    │               markRequestSent()     ← only on success
    │
    └─ [query fails immediately with a clear error]

---

Bug 2 — No idle connection eviction (silent hung TCP)

Before (broken path):

[Network partition / server frozen / firewall drop]
        │
        ▼
  TCP connection hangs — no RST, no FIN sent
        │
        ├─ Netty never fires channelInactive()
        ├─ _channel.isActive() still returns true
        └─ SO_KEEPALIVE default idle = ~2 hours on Linux
                │
                ▼
  Next query arrives for this server
        │
        ├─ connectWithoutLocking(): isActive() == true → no reconnect
        ├─ writeAndFlush(): bytes go into kernel buffer, never delivered
        └─ [query hangs until timeout; repeats for every query for up to 2h]

After (fixed path):

[Network partition / server frozen / firewall drop]
        │
        ▼
  TCP connection hangs — no RST, no FIN sent
        │
  IdleStateHandler fires after 300s of no read OR write
        │
        ▼
  ctx.close() called
        │
        ▼
  channelInactive() fires on DataTableHandler
        │
        ▼
  QueryRouter.markServerDown() → in-flight queries fail fast
        │
  Next query: isActive() == false → connectWithoutLocking() reconnects
        └─ If server still down: connect() throws → query fails immediately

---

Changes

File	Change
NettyConfig.java	Add channelIdleTimeoutSeconds (default 300s) and channelConnectTimeoutMs (default 10s)
ServerChannels.java	Fix writeAndFlush listener; add IdleStateHandler; add CONNECT_TIMEOUT_MILLIS to Bootstrap
ServerChannelsTest.java	Two new tests covering both fixed scenarios

Configuration (via NettyConfig):

• channel.idle.timeout.seconds — how long a fully idle connection lives before eviction (default: 300s)
• channel.connect.timeout.ms — max time bootstrap.connect().sync() can block (default: 10s)

Testing Done

• testWriteFailureMarksServerDown — mocks a failed ChannelFuture; verifies QueryRouter.markServerDown() is called and markRequestSent() is NOT called
• testIdleConnectionEviction — connects with a 1s idle timeout; verifies the channel is closed by IdleStateHandler and markServerDown() fires via channelInactive() within 5s
• Existing testConnect tests (both NIO and native transport) still pass

Fixes: OA-1090