From 5ef6041129ac9c2355e497ac84879055f14c4b34 Mon Sep 17 00:00:00 2001 From: Alex Soffronow-Pagonidis Date: Fri, 12 Jun 2026 10:00:43 +0200 Subject: [PATCH 1/2] docs: add CLI/tooling roadmap and spec Captures the requirements-session decisions for the agent-usable CLI (chfx), configurable bundled server version, binary-dump import, and the standalone capture proxy. Lands on main so subsequent per-item branches inherit the roadmap. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/cli-spec.md | 95 ++++++++++++++++++++++++++++++++++++++++++++++++ todo.md | 23 ++++++++++++ 2 files changed, 118 insertions(+) create mode 100644 docs/cli-spec.md create mode 100644 todo.md diff --git a/docs/cli-spec.md b/docs/cli-spec.md new file mode 100644 index 0000000..867c299 --- /dev/null +++ b/docs/cli-spec.md @@ -0,0 +1,95 @@ +# CLI & Tooling Spec + +Captures the decisions from the requirements session for the agent-usable CLI +and related work (todo items 1, 2, 4, 5). **Item 3 (configurable TCP protocol +version) is dropped** — clickhouse-client negotiates `min(client, server)` on +its own and exposes no way to force a version, and clamping it in the proxy is +out of scope. + +## 1. CLI — `chfx` + +A single npm `bin` named **`chfx`**, run via Node/tsx. Decoders are imported +directly from `src/core` (one source of truth — see §Output). The package is +prepared **publish-ready**: a build step compiles `src/cli` + the needed core to +`dist`, and `package.json` (`bin`, `files`, `exports`) is wired so `npm publish` +works. Local use via `npx chfx` / `npm link`. + +### Agent-usability requirements (item 1) +- **Deterministic JSON on stdout.** Primary output is one well-formed JSON + document with stable key ordering. Diagnostics/logs go to **stderr** only. +- **Non-interactive by default.** No prompts, spinners, color, or progress on + stdout. Everything is flag-controllable so an agent can script it. +- **Self-describing output.** The JSON carries a tool/schema version marker, the + format, the (for captures) negotiated protocol version, and documents the + `byteRange` convention inline. +- (Errors are emitted as JSON on stderr with a sane exit code — not a raw stack + trace — but elaborate exit-code taxonomy was not prioritized.) + +### Commands + +#### `chfx decode` +Import a binary dump from a file **or stdin** and emit structured JSON. +- Inputs: `.chproto` captures, raw **Native** bodies, raw **RowBinary** + (`RowBinaryWithNamesAndTypes`) bodies. +- Format detection: **autodetect with override.** `.chproto` is detected by its + magic header; raw bodies are autodetected best-effort with + `--format native|rowbinary|chproto` to force it (the reliable path when the + Native/RowBinary heads are ambiguous). +- Accepts binary on **stdin** (e.g. piped from clickhouse-client) as well as a + file path argument. + +#### `chfx query` +Run SQL against a server and decode the result in one step. +- Transport: **both, `--transport http|native`.** + - `http`: POST to ClickHouse HTTP, request the chosen format, decode the body. + - `native`: drive clickhouse-client through the capture proxy and decode the + full `.chproto` packet stream. +- Default `--format native` (richest). **Experimental type settings** + (Variant/Dynamic/JSON enablement) are **sent by default**, with + `--no-experimental-settings` to disable for read-only/strict servers that + reject them. +- Remote connection flags: `--host/--port/--user/--password`, env-var fallbacks, + and HTTPS/TLS where applicable. + +#### `chfx proxy` (item 5 — standalone capture proxy) +A listener that forwards to a target server and captures the native TCP stream. +**Any** native client (clickhouse-client, Go/JDBC/Python drivers, …) connects +through it — the proxy does not spawn the client itself. +- Configurable lifecycle: + - **Default: single-shot** — accept one connection, capture, write a + `.chproto` (and decode if asked), then exit. + - Flags opt into **persistent** mode (long-running, one `.chproto` per + connection to an output dir) and **live decode** (`--decode` streams decoded + JSON per connection to stdout). +- Flags: `--listen`, `--target host:port`, `--out`/`--save-dir`, `--decode`, + `--persistent`/`--once`, plus remote `--user/--password`/TLS where applicable. +- **Plaintext/uncompressed only** (same constraint as today). TLS/compressed + streams are unsupported — error clearly and document it. + +#### `chfx schema` / `--help` +Machine-readable description of commands, flags, and the output JSON shape for +agent discovery. + +### Output (item 4) +- **Reuse the web `ParsedData`/`AstNode` shape verbatim**, serialized to JSON, + wrapped with top-level metadata: tool/schema version, format, negotiated + protocol version (for captures), and the raw bytes. +- **Raw bytes inline, once**, as a top-level **hex** string. Agents read a node's + `byteRange {start, end}` (exclusive end) and slice the hex to inspect bytes — + no second command or sidecar file. + +### Tests & docs (cross-cutting) +- **Thorough CLI tests/fixtures**: decode against the existing + `src/core/decoder/fixtures/protocol/*.chproto`, golden-JSON output assertions, + plus query/proxy coverage. +- **README**: a short, practical **quick-start** near the top, and further down a + full reference of all commands, options, settings, and defaults. + +## 2. Docker — configurable server version (item 2) +- `ARG CH_VERSION=latest` → `FROM clickhouse/clickhouse-server:${CH_VERSION}`. +- Surfaced through docker-compose, e.g. `CH_VERSION=24.3 docker compose build`. +- Version is baked at **build time** (no runtime/CI-matrix publishing for now). + +## 3. Configurable TCP/native protocol version — **dropped** +No reliable way to force the negotiated version via clickhouse-client; the +existing Native HTTP `client_protocol_version` selector is sufficient. diff --git a/todo.md b/todo.md new file mode 100644 index 0000000..3cf1418 --- /dev/null +++ b/todo.md @@ -0,0 +1,23 @@ +# TODO + +Spec for items 1, 2, 4, 5: [docs/cli-spec.md](docs/cli-spec.md) + +1. **CLI (`chfx`) usable by agent** — npm bin via Node/tsx, publish-ready. + Deterministic JSON on stdout, non-interactive, self-describing. Commands: + `decode`, `query`, `proxy`, `schema`/`--help`. +2. **Configurable server version in the image** — `ARG CH_VERSION` → + `FROM clickhouse/clickhouse-server:${CH_VERSION}`, surfaced via docker-compose. +3. ~~Configurable protocol version in TCP + native web interface~~ — **dropped** + (clickhouse-client can't force the negotiated version; HTTP selector suffices). +4. **Import binary dump in CLI → structured output** — `chfx decode` (and + `query`): autodetect `.chproto`/Native/RowBinary with `--format` override, + stdin supported. Emits the web `ParsedData`/`AstNode` JSON plus the full raw + buffer inline as one hex string; agents slice it via each node's `byteRange`. +5. **Use with external clients, in the CLI** — `chfx proxy`: standalone capture + proxy any native client connects through. Single-shot by default; persistent + and live-decode via flags. Plaintext/uncompressed only. + +Cross-cutting: thorough CLI tests/fixtures; README quick-start + full options +reference; remote auth/TLS flags for `query`/`proxy`. + +6. Deploy similar to play.clickhouse.com? From d19bc07877ddd8ab64b0278dbac10814df1c3070 Mon Sep 17 00:00:00 2001 From: Alex Soffronow-Pagonidis Date: Fri, 12 Jun 2026 10:00:44 +0200 Subject: [PATCH 2/2] feat(docker): configurable bundled ClickHouse server version Add a global CH_VERSION build arg (default latest) consumed by the runtime stage's FROM, surfaced through docker-compose (CH_VERSION=24.3 docker compose build). Declared before the first FROM so it is in scope for the runtime FROM. Documents usage in the README quick start. Verified: `CH_VERSION=24.3 docker compose build` produces an image whose clickhouse-server --version reports 24.3.18.7. Co-Authored-By: Claude Opus 4.8 (1M context) --- Dockerfile | 9 ++++++++- README.md | 16 ++++++++++++++++ docker-compose.yml | 34 ++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+), 1 deletion(-) create mode 100644 docker-compose.yml diff --git a/Dockerfile b/Dockerfile index f31fe40..8cc8711 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,3 +1,10 @@ +# CH_VERSION selects the bundled ClickHouse server image tag, baked at build +# time. Override with `--build-arg CH_VERSION=24.3` or via docker-compose +# (`CH_VERSION=24.3 docker compose build`). Defaults to the latest release. +# Declared before the first FROM so it is a global ARG usable in the runtime +# stage's FROM below. +ARG CH_VERSION=latest + # Build stage FROM node:20-alpine AS builder WORKDIR /app @@ -7,7 +14,7 @@ COPY . . RUN npm run build # Runtime stage -FROM clickhouse/clickhouse-server:latest +FROM clickhouse/clickhouse-server:${CH_VERSION} # Install nginx and supervisor RUN apt-get update && apt-get install -y nginx supervisor && rm -rf /var/lib/apt/lists/* diff --git a/README.md b/README.md index 1c00f30..a69934f 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,22 @@ docker run -d -p 8080:80 rowbinary-explorer Open http://localhost:8080 +### Bundled ClickHouse version + +The image bundles a ClickHouse server, pinned to `latest` by default. Choose a +specific version at **build time** with the `CH_VERSION` build argument (it maps +to the `clickhouse/clickhouse-server:` image tag): + +```bash +# docker build +docker build --build-arg CH_VERSION=24.3 -t rowbinary-explorer . + +# docker compose +CH_VERSION=24.3 docker compose build +``` + +The version is baked into the image — rebuild to change it. + ## Desktop App For developers who already run ClickHouse locally. Download the latest release for your platform from the [Releases](../../releases) page: diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..1686612 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,34 @@ +# Self-contained deployment: bundled ClickHouse + nginx + capture server. +# +# docker compose up -d --build # build and start (http://:8080) +# docker compose logs -f # follow logs +# docker compose down # stop and remove +# +# Override the web port, read-only mode, or bundled ClickHouse version without +# editing this file: +# WEB_PORT=9000 docker compose up -d --build +# READONLY=0 docker compose up -d --build # allow INSERT/DDL (internal use) +# CH_VERSION=24.3 docker compose build # pin the bundled server version +services: + explorer: + build: + context: . + args: + # Bundled ClickHouse server image tag, baked at build time (default latest). + CH_VERSION: "${CH_VERSION:-latest}" + image: rowbinary-explorer + ports: + # host:container — the app/nginx listens on 80 inside the container. + - "${WEB_PORT:-8080}:80" + environment: + # READONLY=1 (default) serves the read-only viewer user. + # Set READONLY=0 for an internal deployment that may INSERT / run DDL. + READONLY: "${READONLY:-1}" + restart: unless-stopped + volumes: + # Persist the bundled ClickHouse data across restarts (drop this line for + # a fully ephemeral instance). + - clickhouse-data:/var/lib/clickhouse + +volumes: + clickhouse-data: