Expose stage/release status in schema#5443
Merged
Merged
Conversation
ilyakuz-db
reviewed
Jun 5, 2026
ilyakuz-db
approved these changes
Jun 5, 2026
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
requested review from
andrewnester,
denik,
pietern and
shreyas-goenka
Collaborator
Integration test reportCommit: 102150d
24 interesting tests: 15 SKIP, 7 KNOWN, 2 flaky
Top 29 slowest tests (at least 2 minutes):
|
pietern
requested changes
Jun 9, 2026
Generate the workspace and account command stubs from a checked-in .codegen/cli.json instead of reflecting over an external API spec, so codegen is reproducible from data committed to this repo. - internal/cligen: a self-contained, data-driven generator (model, templates, name/casing derivation) that renders the command stubs from cli.json - bundle/internal/schema: read API annotations (descriptions, enums, field behaviors) directly from cli.json; drop the OpenAPI pseudo-spec reconstruction and the raw-spec environment path - Taskfile: wire the cli.json-based generators into `task generate` - regenerate the committed codegen artifacts from cli.json Co-authored-by: Isaac
splitASCII drives every casing function and is a faithful port of genkit's Named.splitASCII, whose nearest-letter scanning produces several non-obvious splits (acronym head/tail handling, digits never starting a word). Pin those as a curated table so the port can't drift. Co-authored-by: Isaac
Pure move of the name-splitting engine (splitASCII and its condAtNearestLetters/searchNearest helpers) out of names.go so it sits next to split_test.go. The casing functions stay in names.go. Co-authored-by: Isaac
Co-authored-by: Isaac
Add table cases for the special-cased early returns in camelName/snakeName and the '$'-skip branch in splitASCII, bringing the pure name/casing logic to full statement coverage. Co-authored-by: Isaac
Switch the CLI to genkit's native cli_v1 producer (`genkit update-sdk`, mode cli_v1). The cli.json schema keys/refs now use the Go SDK's package.PascalName rendering and the file carries every message — including request/response wrappers, which are inert for the CLI since it reads only the schemas its commands reference. - derive DoNotSuggest from launch_stage == PRIVATE_PREVIEW instead of the dropped preview flag (cli_json.go, parser.go), sourcing the signal from the single launch-stage source of truth - Taskfile generate-cli-json: build genkit from universe HEAD and run update-sdk, since the cli_v1 producer is decoupled from the spec - regenerate annotations, jsonschema (+ for-docs), and docs; compute.Kind's description now resolves via the package.PascalName key - cli_json_test: assert the spec sha via the _openapi_sha file (genkit no longer writes it into cli.json metadata) Co-authored-by: Isaac
The cli_v1 producer emits go_v0 package.PascalName schema keys that match the Go SDK type names directly (e.g. serving.OpenAiConfig), so the Go-name-to-spec-name fixup is never reached. Regenerating the annotations, json schemas, and docs is byte-identical with it removed. Co-authored-by: Isaac
- drop the .codegen/cli.json entry from the test-task sources; cli.json changes are already covered by the dedicated cli.json/cligen tasks - restore the "Generating CLI code..." echo to match origin/main Co-authored-by: Isaac
janniklasrose
force-pushed
the
janniklasrose/sdk-field-status-in-schema
branch
from
a4089fd to
1a4d7eb
Compare
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
auto-merge was automatically disabled
June 9, 2026 23:04
Merge commits are not allowed on this repository
janniklasrose
commented
Jun 9, 2026
Read launch_stage, enum_launch_stages and enum_descriptions from the checked-in .codegen/cli.json and surface them in the generated bundle schema, docs and Python models: - Prefix field descriptions with [Public Preview] / [Beta] / [Private Preview] so the stage shows in editor hover tooltips and the generated docstrings. - Emit the parallel enumDescriptions array so VSCode renders a per-value stage label and description in autocomplete dropdowns. - Carry the raw x-databricks-launch-stage alongside the existing x-databricks-preview, and hide PRIVATE_PREVIEW fields via doNotSuggest. PreviewTag lives in bundle/internal/annotation as the single source of truth shared by the schema and docs generators. The PUBLIC_BETA label is "[Beta]" to match Databricks public documentation. cli.json is filtered upstream at min-stage=PRIVATE_PREVIEW, so no DEVELOPMENT surface reaches the extractor and launch_stage is already clean (GA plus the three preview stages). The OpenAPI-spec-based DEVELOPMENT filtering and preview normalization the earlier version carried are no longer needed. assignAnnotation flags an open question (in a comment) on whether either x-databricks-preview or x-databricks-launch-stage is consumed downstream of the published schema, pending confirmation with the team. Regenerated jsonschema.json, jsonschema_for_docs.json, annotations_openapi.yml, the bundle docs and the Python (pydabs) models. Co-authored-by: Isaac
Map ./bundle/schema/jsonschema.json onto databricks.yml / bundle.yml in .vscode/settings.json so the schema generated in this repo is the one the editor validates and autocompletes against. Co-authored-by: Isaac
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
The pydabs-codegen run during the merge resolution re-synced the lockfile through a locally-configured pypi proxy (pypi-proxy.dev.databricks.com), swapping every registry URL and dropping the size fields. Only python/codegen/uv.lock was caught and reverted at the time; this one slipped into the merge commit staged. Co-authored-by: Isaac
Resolved the open question: neither x-databricks-preview nor x-databricks-launch-stage has external downstream consumers. In-repo, pydabs does read x-databricks-preview from jsonschema.json (experimental markers and private-field exclusion), so that extension stays. The raw launch stage had no consumer at all, so drop it from jsonschema.json and jsonschema_for_docs.json; the stage still surfaces as the description prefix and the per-value enumDescriptions labels. Both extensions originate solely from the parser-generated annotations_openapi.yml, and the parser pairs Preview PRIVATE with stage PRIVATE_PREVIEW, so the PRIVATE_PREVIEW doNotSuggest branches were redundant with the Preview ones and are gone too (doNotSuggest count in the schema is unchanged). Extension.LaunchStage had no remaining writer and is removed; the annotation Descriptor keeps its launch-stage fields as the pipeline input. Docs output is byte-identical. Co-authored-by: Isaac
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
pietern
reviewed
Jun 12, 2026
| @@ -340,7 +340,7 @@ class JobDict(TypedDict, total=False): | |||
| """ | |||
| :meta private: [EXPERIMENTAL] | |||
Contributor
There was a problem hiding this comment.
This emits this meta tag. Can we map our launch stages into similar meta tags instead of including it in the field description? cc @kanterov
Main removed the unused bundle/docsgen generator (#5568); the docs-only schema jsonschema_for_docs.json is the published-docs source of truth and already carries this branch's launch-stage prefixes and enum labels, so the branch's docsgen changes are dropped with the package (modify/delete conflicts resolved by deleting). PreviewTag's comment no longer claims to be shared with the docs generator. Co-authored-by: Isaac
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Clarify comments in PreviewTag function regarding launch stages.
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
pietern
approved these changes
Jun 15, 2026
| return "[Beta]" | ||
| case "PUBLIC_PREVIEW": | ||
| return "[Public Preview]" | ||
| } |
Contributor
There was a problem hiding this comment.
Would be nice if we had a "LaunchStage" type we can use for this (for ex in internal/clijson/types.go).
Some kind of enforcement or check that if a new stage is added we don't forget to update this mapping.
Contributor
Author
There was a problem hiding this comment.
will do as follow-up
| // The raw launch stage is intentionally not emitted into the published | ||
| // schema — nothing consumes it there. It surfaces only as the description | ||
| // prefix below and the per-value enumDescriptions labels. | ||
| // x-databricks-preview does stay: the parser derives it from launch_stage |
Contributor
There was a problem hiding this comment.
The field x-databricks-preview exists only as a remnant of previous codegen.
Would be great if we make also downstream codegen use the launch stage exclusively.
Can be a follow up.
| "$ref": "#/$defs/string" | ||
| }, | ||
| "evaluation": { | ||
| "description": "[Public Preview]", | ||
| "$ref": "#/$defs/github.com/databricks/databricks-sdk-go/service/sql.AlertV2Evaluation" |
Contributor
There was a problem hiding this comment.
How does this render when you hover over the field. Does it compose this description with the type's?
Contributor
Author
There was a problem hiding this comment.
this is the description (there isn't one for alert.evaluation or the others)
See #5588 for backfilling missing annotations as follow-up
pietern
added a commit
that referenced
this pull request
Jun 15, 2026
Reconciles #5443 (expose launch stage/release status in the schema) with the consolidated single-file annotation system on this branch: - Brought in unchanged: annotation.Descriptor's new LaunchStage / EnumLaunchStages / EnumDescriptions fields, the PreviewTag helper (preview.go), jsonschema.Extension's enumDescriptions, and assignAnnotation's launch-stage description prefixing + enumDescriptions emission (these merged cleanly onto our annotations.go). - Reapplied in our in-memory extractor: parser.go now populates the launch stage on each field and the enum launch stages/descriptions on the type's Self descriptor via SetField/SetSelf, instead of the deleted annotations_openapi.yml. The launch-stage data is inherited from cli.json at generation time and never persisted to annotations.yml. - Kept annotations_openapi.yml deleted (our system computes that layer in memory). bundle/schema/jsonschema.json and jsonschema_for_docs.json are byte-for-byte identical to origin/main after regeneration. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 9d003d2
516 interesting tests: 472 MISS, 32 FAIL, 4 KNOWN, 2 PANIC, 2 flaky, 2 SKIP, 2 RECOVERED
Top 50 slowest tests (at least 2 minutes):
|
artchen-db
pushed a commit
to artchen-db/cli
that referenced
this pull request
Jun 18, 2026
databricks#5611) ## Changes Two follow-ups requested in the review of databricks#5443: 1. **`LaunchStage` type + enforcement** ([comment](databricks#5443 (comment))). Add a `LaunchStage` type in `internal/clijson` (the cli.json contract package) with the known stages (`GA`, `PUBLIC_PREVIEW`, `PUBLIC_BETA`, `PRIVATE_PREVIEW`) as constants and a closed-set `ParseLaunchStage`. Schema generation validates every `launch_stage` read from `.codegen/cli.json` and fails with an actionable error on any unrecognized stage, so a stage introduced upstream can't silently render as GA. `bundle/internal/annotation` keeps only the presentation mapping (`PreviewTag`), keyed by `clijson.LaunchStage`. 2. **Drop the `x-databricks-preview` remnant** ([comment](databricks#5443 (comment))). `x-databricks-preview` existed only as a remnant of previous codegen. The published schema now carries `x-databricks-launch-stage` instead, and the Python (pydabs) codegen reads the launch stage exclusively (its `LaunchStage` class mirrors `clijson.LaunchStage`). ## Behavior Behavior-preserving refactor — the set of hidden/experimental fields is unchanged: - `jsonschema.json` / `jsonschema_for_docs.json`: each `x-databricks-preview: PRIVATE` entry becomes `x-databricks-launch-stage: PRIVATE_PREVIEW`; `doNotSuggest` is unchanged. - `annotations_openapi.yml`: the derived `x-databricks-preview` entries are dropped (`x-databricks-launch-stage` was already present). - Generated pydabs models under `python/databricks/bundles/**` are **byte-identical**. ## Design notes - `x-databricks-launch-stage` is emitted into the published schema only for the private-preview stage — the only stage any consumer acts on (`doNotSuggest` and pydabs `experimental`). Other stages still surface as the description prefix and the per-value `enumDescriptions` labels, matching the original rationale in databricks#5443 for keeping the published schema lean. - The contract still types `launch_stage` as a plain string (the `clijson` wire types are generated by genkit). Fully guaranteeing that no value outside the closed set can be produced requires upstreaming the producer to emit the `LaunchStage` enum directly; until then `clijson.ParseLaunchStage` enforces the set where the contract is consumed. ## Testing - `go test ./internal/clijson/... ./bundle/internal/... ./bundle/schema/... ./libs/jsonschema/...` - `go test ./acceptance -run TestAccept/bundle/refschema` - Python codegen unit tests (`python/codegen/codegen_tests/`) - Re-ran `./task generate-schema`, `generate-schema-docs`, and `pydabs-codegen`; verified the only schema change is the `x-databricks-preview` → `x-databricks-launch-stage` swap and that the generated pydabs models are unchanged. Follow-up to databricks#5443. This pull request and its description were written by Isaac, an AI coding agent.
artchen-db
pushed a commit
to artchen-db/cli
that referenced
this pull request
Jun 18, 2026
…atabricks#5588) ## Changes `description: PLACEHOLDER` markers are now dropped when cli.json documents the field: `newAnnotationHandler` strips them before the merge, so the upstream description flows into the schema, and the sync step rewrites `annotations.yml` without them in the same run. - Placeholders for genuinely undocumented fields are kept (still the TODO marker). - Entries that also carry hand-written data (e.g. `deprecation_message`) lose only the placeholder. - Most of the diff is regeneration: `annotations.yml` −484 lines (deletions only); both schema artifacts gain 128 description lines (23 of which replace a bare launch-stage tag). ## Why A PLACEHOLDER means "no documentation anywhere". It is added when a field first appears undocumented, but nothing removed it once upstream docs arrived. The annotations file overrides upstream docs in the merge and PLACEHOLDER is skipped when assigning descriptions, so the schema ended up with no description at all — the stale marker hides the real text and then discards itself. 128 fields in the published schema were affected (e.g. `vector_search_endpoints.*.target_qps`). The file cannot self-heal: a shadowed field never registers as missing, so the backlog only grows. This also fixes the bare `[Public Preview]` prefixes from the launch-stage work (databricks#5443): the prefix was there, but the description it should prefix was swallowed by this bug. ## Tests - Table test for the strip plus a handler-level test asserting the merged view resolves the upstream description while preserving hand-written fields. - `TestRequiredAnnotationsForNewFields` now also records deletes/updates during regeneration (previously a nil-visitor panic) and asserts regeneration of the committed file is a no-op, enforcing idempotency. - Re-verified after merging main: build, the `bundle/internal/schema` + `annotation` + `bundle/...` unit tests, the `refschema` acceptance test, and `lint` all pass; regeneration is byte-idempotent across runs. This pull request and its description were written by Isaac, an AI coding agent. <!-- isaac --resume 4ecd889d-ebf8-47d3-ad07-e35c3342e5f0 --> --------- Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Expose launch stage/release status in the bundle schema
Why
Non-GA fields should be annotated as such when viewing a bundle in IDEs.
Tests
Example vector search endpoint:
Enum with one GA, one PuPr:
PuPr field without any annotation:
