fix(gateway): threaded acceptor survives fd exhaustion instead of tearing down#143
Conversation
…ring down Round-2 follow-up to #140: that PR hardened the epoll transport's fd-exhaustion handling (disarm the listener, keep serving, re-arm when a descriptor frees) but left the portable threaded TcpServer asymmetric. transient_accept_errno() deliberately omits EMFILE/ENFILE, so when the process hits RLIMIT_NOFILE (reachable in the default config — max_active_connections defaults to 0, i.e. uncapped, so the server's own slow/idle clients can exhaust descriptors), accept_retry() returned -1, serve_listen_socket() broke out, and the acceptor stopped permanently even though existing connections were fine and the condition is transient (it clears the moment any client disconnects and frees an fd). The epoll path survives the identical condition. Fix: accept_retry() now detects EMFILE/ENFILE/ENOBUFS/ENOMEM separately and backs off briefly (10ms) before retrying instead of returning fatal. It cannot be added to the immediate-retry transient set — the pending connection stays in the backlog, so an instant retry would just re-fail and busy-spin at 100% CPU. The back-off lets worker threads finish and close their sockets, freeing descriptors; the acceptor auto-resumes, symmetric with the epoll listener pause/resume. make check 270/270, make tsan 20/20, CodeScene delta clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 24 minutes and 34 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![codescene-delta-analysis[bot]](https://avatars.githubusercontent.com/in/53921?s=60&v=4){kind=small}
There was a problem hiding this comment.
Gates Passed
See analysis details in CodeScene
Quality Gate Profile: Pay Down Tech Debt
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.
…ct (#147) * docs: overhaul all stale docs for the post-v0.2.1 (v0.2.2) state Full staleness audit of every prose doc against current main. The anchors were frozen at v0.2.1 / 263 tests / "no active milestone" while 12 PRs (#135-#146) had merged and are unreleased. - Resume anchors (PROGRESS.md, HANDOFF.md): Current state / Last action / Next action / test count (263->270) brought current; the two duplicate frozen anchors and the stale macOS benchmark numbers fixed; a dated v0.2.2 log entry. - CLAUDE.md + AGENTS.md: the post-M35 roadmap-memory section now records the post-v0.2.1 hardening + perf wave (identical edit in both). - CHANGELOG.md: new [0.2.2] section (decoder enum rejection #136, network/CLI hardening #137/#140/#141/#143, real UBSan abort gate #142, ocaml diff_report #144, try_emplace ~+5% #138, index load-factor ~+18.6% #145). CMakeLists version 0.2.1 -> 0.2.2. - README: benchmark/flamegraph/limitations sections reflect the engine wins (measured on the profile workload, not the micro-bench table) and the gateway hardening; release_readiness 270/270 + UBSan gate + v0.2.2 scope. - Networking docs (socket_gateway, socket_hardening): connection cap, EINTR retry, transient-accept survival, fd-exhaustion handling, UDP send-error counting. replay_and_recovery: decode_command now rejects out-of-domain enums. binary_protocol/differential_testing/fix_protocol/SECURITY/recruiting_notes/ CONTRIBUTING: smaller accuracy updates. results/README: add the three socket artifacts. make check 270/270. Stale results/*.txt provenance digests regenerated separately. pool_backed_storage.md table follows its artifact regeneration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * results: regenerate stale provenance artifacts for the v0.2.2 source The post-v0.2.1 source changes (#135-#146) left 14 results/*.txt with stale Source digests (the authoritative staleness signal per the provenance rules, not commit-hash drift). Regenerated via their make targets so each declares Dirty inputs: no against current HEAD: differential, pool_backed_storage, allocator_experiment, recovery_benchmarks, false_sharing_study, perf_stat_linux (partial PMU, QSL_PERF_ALLOW_PARTIAL), perf_report_linux, numa_affinity_study (linux-constrained), socket_load_summary, socket_profile_loopback, socket_stress_summary, dpdk_environment, nic_offload_environment. docs/pool_backed_storage.md: refreshed the median table, digest reference, and qualitative ordering from the regenerated artifact (contiguous fastest on four of five workloads; intrusive leads dense). The baseline rows now include the try_emplace (#138) and index load-factor (#145) wins. Honesty notes: these were measured on a thermally-warmed M2 from a long session, so absolute values run higher than a cool-host snapshot — the relative orderings and provenance digests are the load-bearing content, and every artifact carries a hardware/build-dependence caveat. latest.txt is regenerated separately on a cooled host to keep its headline micro-benchmark numbers representative. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * results: regenerate latest.txt (cool host) and sync README/PROGRESS numbers latest.txt regenerated on a thermally-recovered host so the headline micro-bench numbers are representative (protocol canary 16.1 ns/op), with a fresh Source digest (Dirty inputs: no) matching current source. The README benchmark table and the PROGRESS measured-results section are aligned to it: order_book ~90, protocol ~16, gateway ~102, matching ~91, replay ~101 ns. The matching/replay rows are slightly faster than the prior committed run (the v0.2.2 engine wins showing on the resting-order paths); the order_book micro-bench is unchanged (near-empty index, so it does not exercise the load-factor win). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * results: re-regenerate env-check artifacts against the final v0.2.2 docs dpdk_environment.txt and nic_offload_environment.txt include README.md (and CLAUDE.md/AGENTS.md/results/README.md) in their digest scope, so the README number-sync re-staled them. Regenerated against the final committed docs; Dirty inputs: no. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
Summary
Round-2 follow-up to #140. That PR hardened the epoll transport's fd-exhaustion handling (disarm the listener, keep serving, re-arm when a descriptor frees) but left the portable threaded
TcpServerasymmetric.transient_accept_errno()deliberately omitsEMFILE/ENFILE, so when the process hitsRLIMIT_NOFILE— reachable in the default config (max_active_connectionsdefaults to 0, i.e. uncapped, so the server's own slow/idle clients can exhaust descriptors) —accept_retry()returned-1,serve_listen_socket()broke out, and the acceptor stopped permanently, even though existing connections were fine and the condition is transient (it clears the moment any client disconnects and frees an fd). The epoll path survives the identical condition.Fix
accept_retry()now detectsEMFILE/ENFILE/ENOBUFS/ENOMEMseparately and backs off briefly (10 ms) before retrying instead of returning fatal. It deliberately is not added to the immediate-retry transient set — the pending connection stays in the backlog, so an instant retry would just re-fail and busy-spin at 100% CPU. The back-off lets worker threads finish and close their sockets, freeing descriptors; the acceptor auto-resumes, symmetric with the epoll listener pause/resume.Notes
RLIMIT_NOFILEin-process, which risks the test harness's own descriptors; the logic mirrors the epoll path's handling and all existing gateway tests pass.Verification
make check270/270,make tsan20/20, CodeScene delta clean.🤖 Generated with Claude Code