Stabilize since-versions and publish the docs schema only to docgen#5401
Closed
shreyas-goenka wants to merge 2 commits into
Closed
Stabilize since-versions and publish the docs schema only to docgen#5401shreyas-goenka wants to merge 2 commits into
shreyas-goenka wants to merge 2 commits into
Conversation
`x-since-version` was recomputed from git history on every schema generation, keyed by the Go type path. That made versions drift: when a type is renamed or a shared struct is split into per-resource typed structs (e.g. permissions), the field re-keys and gets re-stamped with a newer version. Persist the computed map as an append-only state file and treat stored entries as authoritative so a recorded version never changes, even across refactors. The state lives on the `docgen` branch (next to jsonschema_for_docs.json), not in the main source tree: - since_version.go reads/writes the state at the path in DATABRICKS_SINCE_VERSIONS_FILE. When unset (local `task generate`, regular CI) behavior is unchanged: versions are computed from history and nothing is persisted, so main stays clean and the workflow's "only the docs schema changed" assertion still holds. - When set, computeSinceVersions loads the stored map, refreshes it from git history to discover new fields, merges with stored entries winning, and writes it back. sinceVersionAliases lets a renamed/retyped field inherit its previous key's version. - The update-schema-docs workflow checks out docgen, points the env var at its since_versions.json, regenerates, and commits both the schema and the refreshed state back to docgen. It runs on every release tag, so the state is updated on each release; the first run seeds it from history. The map is written deterministically (sorted, trailing newline) so it stays diff-stable. No existing annotation changes; this only prevents future drift. Co-authored-by: Isaac
shreyas-goenka
force-pushed
the
shreyas-goenka/stable-since-versions
branch
from
b243a6e to
2a3b079
Compare
shreyas-goenka
changed the title
bundle/schema/jsonschema_for_docs.json was a checked-in generated artifact on main, but nothing in the CLI reads it — the binary embeds jsonschema.json, and bundle/docsgen regenerates from Go types. Its only consumer is downstream docs tooling, which now reads from the docgen branch. Remove it from main and generate it only on release: - Delete bundle/schema/jsonschema_for_docs.json and drop generate-schema-docs from the `all`/`generate` aggregators and the Taskfile (so `task generate` no longer recreates it on main). - The update-schema-docs workflow now regenerates the schema straight into the docgen worktree (next to since_versions.json) and commits both to docgen, instead of generating on main and copying. Drops the now-unneeded "assert only the docs schema changed on main" / capture / copy steps. - Drop the file from .wsignore, .gitattributes, and the auto-generated-files rule. Co-authored-by: Isaac
Contributor
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
shreyas-goenka
changed the title
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
x-since-versionstable and moves the docs schema + its state offmain.Two commits
1. Persist since-versions on docgen for stable annotations.
x-since-versionwas recomputed from git history on every generation, keyed by Go type path, so versions drifted when a type was renamed or a shared struct was split into per-resource typed structs (e.g. permissions). Now a since-versions map is persisted as append-only state — stored entries are authoritative and never change. It lives on thedocgenbranch (viaDATABRICKS_SINCE_VERSIONS_FILE), so local/regular generation is unchanged andmainstays clean.sinceVersionAliaseslets a renamed field inherit its previous key's version.2. Stop keeping the docs schema on main; publish it only to docgen.
bundle/schema/jsonschema_for_docs.jsonwas a checked-in generated artifact onmain, but nothing in the CLI reads it (the binary embedsjsonschema.json;bundle/docsgenregenerates from Go types). Its only consumer is downstream docs tooling, which now reads fromdocgen. So:bundle/schema/jsonschema_for_docs.jsonand dropgenerate-schema-docsfrom theall/generateaggregators and the Taskfile.update-schema-docsworkflow now regenerates the schema straight into the docgen worktree (next tosince_versions.json) and commits both todocgen, instead of generating onmainand copying. Drops the now-unneeded assert/capture/copy steps..wsignore,.gitattributes, and the auto-generated-files rule.Sequencing (done)
The downstream universe docs-tool previously fetched the schema from the latest release tag; it was switched to read from the
docgenbranch and that change is merged to universe master (databricks-eng/universe#2008134). So removing the file frommain(and hence future tags) no longer affects it.Behavior on release
update-schema-docsruns on everyv*tag: it regenerates the schema, refreshes the append-only state, and commits both todocgen. The first run seeds the state from history.Test
go build/go vet/gofmtclean; unit tests for the merge/load/save logic pass.--docsmode withDATABRICKS_SINCE_VERSIONS_FILEset — a frozen stored version won in the output schema and the state was written back.This pull request and its description were written by Isaac.