Expose new API fields in CLI (#289)#301
Open
sdairs wants to merge 5 commits into
Open
Conversation
Wire up the four groups of API surface landed in the drift fixes (#298/#299/#300) to user-facing CLI commands: 1. ClickPipe object-storage ingestion control: `--skip-initial-load` and `--start-after` (continuous/SQS pipes only). `--skip-initial-load` requires `--queue-url`; `--start-after` conflicts with `--skip-initial-load`. 2. Service horizontal autoscaling on `service create`/`service scale`: `--min-replicas`/`--max-replicas`/`--autoscaling-mode`. The horizontal pair is mutually exclusive with the vertical flags. Infers `horizontal` when the replica pair is present without an explicit mode, and rejects min-without-max (and vice versa) before any network call. 3. ClickPipe MySQL `--server-id` for CDC replication. 4. ClickPipe schema discovery (beta): new `clickpipe schema-discover` command with `kafka`/`kinesis` subcommands that POST to `/clickpipes/schemaDiscovery` and render the inferred fields. Source connection flags are shared with the `create` subcommands via `KafkaSourceFields`/`KinesisSourceFields` flatten structs, so there is no `service_id` positional clash and no destination fields. Adds `AutoscalingMode::VALUES` to the API library (drift-guarded by the analyzer enum-values check) and a `click_pipe_schema_discovery` wrapper on `CloudClient`. Tests: - bin: build_*_request unit tests (minimal + maximal) for horizontal autoscaling on create and scale, plus resolve_horizontal_autoscaling edge cases; Cli::try_parse_from coverage for all new flags. - wiremock: object-storage skipInitialLoad/startAfter, MySQL serverId, and schema-discovery POST body shape for kafka + kinesis.
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
- resolve_horizontal_autoscaling no longer injects autoscalingMode "horizontal" when --min-replicas/--max-replicas are given without an explicit --autoscaling-mode. The API resolves an omitted mode itself, and an equal band with the mode omitted is accepted as a vertical fixed replica count without the horizontal entitlement — inferring horizontal changed those semantics. - service scale human output now reports the autoscaling mode and, for horizontal services, the replica band and per-replica memory instead of only the vertical fields. - --iterator-timestamp values above i64::MAX now fail with a clear error instead of silently wrapping negative via `as i64`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified against the live API with an OAuth bearer token: the gateway rejects POST /clickpipes/schemaDiscovery with "This endpoint is not available for JWT authentication". The command is side-effect-free but requires API-key auth, so is_write_command must return true to fail fast with the API-key guidance (exit code 4) instead of a doomed network round-trip. Moves the test case from the read-only list to the destructive list accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9510dc4. Configure here.
Relax the autoscaling clap conflicts so `--autoscaling-mode` and the memory flags combine with either mode's fields — the scaling PATCH accepts a mode switch with its target fields in one request, and horizontal requires equal min/max memory in the same call. Only the contradictory `--num-replicas` vs `--min-replicas`/`--max-replicas` conflict remains. Print no mode-specific scale output for an unrecognized autoscaling mode instead of falling through to the vertical fields. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rndD
approved these changes
Jul 20, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR exposes newly-added ClickHouse Cloud API surface in the clickhousectl CLI, covering ClickPipes ingestion controls, MySQL replication server ID, schema discovery (beta), and service horizontal autoscaling—aligning the CLI with recent OpenAPI drift remediation work.
Changes:
- Added ClickPipe object-storage ingestion control flags (
--skip-initial-load,--start-after) and MySQL--server-id, including wire-shape coverage. - Added service horizontal autoscaling flags (
--min-replicas,--max-replicas,--autoscaling-mode) with shared request-building logic forservice createandservice scale. - Introduced
clickpipe schema-discover(Kafka/Kinesis) and aCloudClientwrapper for the schema-discovery endpoint; updated README usage/docs.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents new autoscaling, ClickPipe ingestion-control, MySQL server-id, and schema-discover usage. |
| crates/clickhousectl/tests/cli_request_shape_test.rs | Adds wiremock tests asserting new request-body fields for S3 ingestion control, MySQL serverId, and schema discovery POST shape. |
| crates/clickhousectl/src/main.rs | Wires the new clickpipe schema-discover command into cloud dispatch. |
| crates/clickhousectl/src/cloud/commands.rs | Implements request building/serialization for new flags and adds schema discovery handler; refactors Kafka/Kinesis source construction. |
| crates/clickhousectl/src/cloud/client.rs | Adds a CloudClient::click_pipe_schema_discovery wrapper. |
| crates/clickhousectl/src/cloud/cli.rs | Adds new CLI flags/commands and clap parsing tests; classifies schema-discover as “write” for auth gating. |
| crates/clickhouse-cloud-api/src/models.rs | Adds AutoscalingMode::VALUES for CLI validation / analyzer drift-guarding. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Correct build_kafka_source/build_kinesis_source doc comments to the actual positional command shape (no --source flag). - Align the clickpipe_schema_discover doc comment with its write classification: side-effect-free, but the gateway rejects OAuth/Bearer so API key auth is required. - Drop redundant after_help lines (auth note, --json hint) from the schema-discover clap command. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exposes the four groups of API surface landed in the OpenAPI drift fixes (#298 / #299 / #300) as user-facing CLI commands, per the plan in #289.
Closes #289.
Changes
1. ClickPipe object-storage ingestion control
--skip-initial-load: skip the initial snapshot load, only ingest new objects (continuous/SQS pipes only; requires--queue-url).--start-after: resume ingestion after a specific object key (conflicts with--skip-initial-load).skipInitialLoad/startAfteron the object-storage source body.2. Service horizontal autoscaling (
service create/service scale)--min-replicas/--max-replicas/--autoscaling-mode.--num-replicas/--min-replica-memory-gb/--max-replica-memory-gb) via clapconflicts_with_all.--autoscaling-modeis given explicitly; the API resolves an omitted mode itself (an equal--min-replicas/--max-replicasband with the mode omitted is a vertical fixed replica count needing no horizontal entitlement, so the CLI must not inferhorizontal).--min-replicaswithout--max-replicas(and vice versa) with a clear error before any network call.resolve_horizontal_autoscaling→HorizontalAutoscalingstruct (used by both create and scale viabuild_create_service_request/build_service_scale_request).3. ClickPipe MySQL
--server-idserverIdon the MySQL source body.4. ClickPipe schema discovery (beta)
clickpipe schema-discover <SERVICE_ID> <kafka|kinesis>command./clickpipes/schemaDiscoveryand renders the inferred fields (table by default, JSON with--json/ agent detection).createsubcommands viaKafkaSourceFields/KinesisSourceFields#[command(flatten)]structs — noservice_idpositional clash, no destination (--name/--database/--table/--column) flags needed.API library
AutoscalingMode::VALUESconst (drift-guarded by the analyzer's existing enum-values check).click_pipe_schema_discoverywrapper onCloudClient.Tests
build_create_service_request/build_service_scale_requestunit tests (minimal + maximal) for horizontal autoscaling;resolve_horizontal_autoscalingedge cases (explicit vertical with no replicas, min-without-max rejection, invalid mode);Cli::try_parse_fromcoverage for all new flags (object-storage ingestion, MySQL server-id, autoscaling on create/scale, schema-discover kafka/kinesis).skipInitialLoad/startAfteron the wire (present + absent); MySQLserverId(present + absent); schema-discovery POST body shape for kafka + kinesis.cargo clippy -p clickhouse-cloud-api -p clickhouse-openapi-analyzer --all-targets -- -D warningsclean.python3 scripts/check-openapi-drift.py --dry-run→ 0 actionable drift.cargo check --workspace --all-featuresclean.Docs
service create/scaleexamples; object-storage continuous ingestion example with--skip-initial-load/--start-after; MySQL--server-idexample; new "Discovering a source schema (beta)" subsection under ClickPipes.