feat(cli): chfx proxy — capturing native-protocol proxy for external clients#45
Merged
Conversation
…clients Add `chfx proxy`, a standalone TCP proxy that any native client connects through (clickhouse-client, Go/JDBC/Python drivers, …) — the proxy never spawns a client itself. It forwards every connection to --target and tees the native packet stream into a capture. - Single-shot (default): capture the first connection then exit, emitting the raw .chproto to stdout, writing it (--out), or decoding it to JSON (--decode). - Persistent (--persistent): serve many connections until Ctrl-C, writing one conn-NNNN.chproto per connection (--save-dir) and/or streaming a decoded JSON doc per connection (--decode). - Plaintext/uncompressed only, same constraint as query/capture. Backed by a new startCaptureProxy() in scripts/native-proxy.mjs (fixed listen port, many connections), distinct from the one-shot ephemeral startProxy() used by query/capture. Adds parseHostPort() for --listen / --target and a 'none' CommandOutput variant for streaming commands. Tests: - Unit (cli.test.ts): parseHostPort, all single-shot output variants, persistent save-dir/decode, the usage-error matrix, and real-socket coverage of startCaptureProxy (bidirectional forward + no-hang on a refused upstream). - Integration (proxy.integration.test.ts): real ClickHouse via testcontainers + a real in-container clickhouse-client connecting back through the host proxy via host.testcontainers.internal (--compression 0 keeps it plaintext). Covers raw/--out/--decode/ --out+--decode/--no-node-bytes+--compact, host:port --listen, and both persistent modes. Runs in CI under the existing `npm test`. Docs: README proxy section + command table, AGENTS.md, docs/cli-spec.md, todo.md (item 5 marked implemented). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a new chfx proxy CLI command that runs a standalone TCP capturing proxy for ClickHouse Native protocol clients, forwarding to a target endpoint while recording the bidirectional packet stream for raw dump output and/or JSON decoding.
Changes:
- Introduces
chfx proxy(single-shot by default,--persistentfor multi-connection) with output modes: raw.chproto,--out, and--decode(including streaming decode in persistent mode). - Adds
startCaptureProxy()inscripts/native-proxy.mjsto support fixed listen ports and multiple connections, plus CLI plumbing and a newstdout: 'none'command output variant for streaming commands. - Adds unit + integration tests (including real sockets and a real ClickHouse container/client), and updates docs/specs to describe the new command.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| todo.md | Marks item 5 implemented and documents the proxy command at a high level. |
| src/cli/registry.ts | Registers the new proxy command and its --help metadata. |
| src/cli/proxy.integration.test.ts | End-to-end integration coverage using Testcontainers + in-container clickhouse-client via host proxy. |
| src/cli/output.ts | Adds NoneOutput (stdout: 'none') for commands that stream their own output. |
| src/cli/index.ts | Wires proxy into the CLI entrypoint and handles the new none output kind. |
| src/cli/connection.ts | Adds parseHostPort() for --listen / --target. |
| src/cli/commands/proxy.ts | Implements proxyCommand (single-shot + persistent) and output behavior. |
| src/cli/cli.test.ts | Adds unit tests for parseHostPort, proxy output modes, usage errors, and real-socket proxy behavior. |
| scripts/native-proxy.mjs | Adds startCaptureProxy() multi-connection capture proxy implementation. |
| scripts/native-proxy.d.mts | Adds TypeScript declarations for startCaptureProxy() and its options. |
| README.md | Documents chfx proxy usage and options. |
| docs/cli-spec.md | Updates the CLI spec to reflect the implemented proxy command and flags. |
| AGENTS.md | Updates repo structure notes for the new proxy command and parsing helper. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ressure) Address review findings on the chfx proxy feature: - startCaptureProxy now flushes a partial capture for any connection still open when close() runs (and on `once`'s self-stop), instead of dropping it. Single-shot registers a Ctrl-C handler too, so a long-lived/pooled client (which never closes its socket) can be finalized with Ctrl-C rather than hanging. Fixes the persistent-shutdown drop of in-flight connections as well. - Forward with backpressure (pause/resume) so a slow consumer can't bloat the socket buffers. - parseHostPort: handle bracketed and bare IPv6 (`[::1]:9000`, `::1`), previously mis-split on the last colon. - Tests: real-socket flush-on-close test, IPv6 parse cases, and a distinct pre-exposed listen port per integration test (no more shared-port EADDRINUSE/TOCTOU flakiness). - Docs: document that a capture completes on client disconnect (Ctrl-C to finalize otherwise), that --save-dir filenames reset/overwrite per run, and that --persistent --decode pairs with --compact for NDJSON. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rs, help) - runPersistent: track in-flight connection handlers in a Set that removes each task on settlement, so the collection stays bounded for a long-running proxy (was an unbounded array). - startCaptureProxy: route post-listen server errors to onError. Once the server is listening the returned promise is already resolved, so the old `reject(err)` path silently dropped later errors (e.g. accept failures). - registry: clarify that proxy --decode switches stdout to a JSON envelope and combines with --out (which still writes the dump file). (The earlier Copilot note on parseHostPort IPv6 was already addressed in the previous commit: bracketed [::1]:9000 and bare ::1 are handled.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
chfx proxy, a standalone TCP proxy that any native client connects through (clickhouse-client, Go/JDBC/Python drivers, …) — the proxy never spawns a client itself. It forwards every connection to --target and tees the native packet stream into a capture.Backed by a new startCaptureProxy() in scripts/native-proxy.mjs (fixed listen port, many connections), distinct from the one-shot ephemeral startProxy() used by query/capture. Adds parseHostPort() for --listen / --target and a 'none' CommandOutput variant for streaming commands.
Tests:
npm test.Docs: README proxy section + command table, AGENTS.md, docs/cli-spec.md, todo.md (item 5 marked implemented).