feat(docs): guided adoption IA — theme, hubs, section folders, and redirects by rederik76 · Pull Request #121 · databricks-solutions/lakeflow_framework

rederik76 · 2026-07-12T08:57:39Z

Summary

Redesigns the Lakeflow Framework documentation for guided adoption — moving from reference-heavy pages to a hub-based, step-oriented experience aligned with how teams deploy and build (Closes #119).

Theme & landing — Migrate docs to sphinx_immaterial with Databricks-branded CSS, LFF logo, WAF-style hero landing, and nav-aligned home cards (Architecture · Samples · Build · Deploy · Features)
Information architecture — Top-level tabs: Get Started, Architecture, Samples, Build, Deploy, Features, Contributors; caption-only section indexes land on hub pages (Quick Start, Deploy hub, etc.)
Section folders — Move flat docs/source/*.rst into topic folders (get-started/, architecture/, build/, deploy/, features/, contributors/, samples/) with index.rst hubs and kebab-case page names
URL redirects — Add docs/redirects/*.yaml and docs/scripts/write_redirects.py; post-build HTML stubs preserve legacy URLs for current and local-branch-preview builds
Deploy — Deploy hub with before-you-deploy, split local deploy (Framework Bundle vs Pipeline Bundle), framework options/wheel guides, and streamlined CI/CD page (numbered shared workflow)
Build & Features — Build hub with spec reference and patterns nested under Build; Features restructured into job-based sub-hubs plus A–Z index; platform hub intro clarifies LFF vs SDP coverage
Contributors — Contributors hub with card grid; env, branching/releases, workflow, imports, and docs authoring guides rewritten in Quick Start / Deploy style; contributors/dev-docs.rst documents section-folder layout and redirect YAML maintenance
Authoring UX — Interactive .. task-list:: prerequisites on Quick Start, CI/CD, and local deploy pages; copy buttons on lf-command-block command snippets; Mermaid deploy-order diagram on before-you-deploy
Content alignment — Home tagline and What is page updated for two-bundle model, flat DAB + lakeflow-framework wheel, and optional contrib; Quick Start and samples/README.md aligned on deploy steps
Versioning — Bump VERSION to v0.21.0; harden multiversion docs build (select_versions.py, build_versioned_docs.py); add sphinx-immaterial to requirements-docs.txt

How to preview this branch (recommended)

This PR changes theme, navigation, and folder layout — use the multiversion preview build, not plain make html, so the version switcher and redirect stubs are exercised.

Check out this branch (must not be main or detached HEAD — preview builds the current branch as local-branch-preview):
```
git checkout feature/docs-guided-adoption-ia
```
Build the versioned site with branch preview:
```
make -C docs html-multiversion-preview
```
- current is built from main
- This branch is also built as local-branch-preview (includes uncommitted working-tree edits if any)
Serve the built site locally (required for the version dropdown — do not rely on file://):
```
cd docs/build/html
python3 -m http.server 8000
```
Open the site root and select this branch in the version menu:
- Browse to http://localhost:8000/index.html
- Use the version selector in the header and choose local-branch-preview
- Confirm you are reviewing this PR’s IA (new top nav, folder URLs, landing cards) — not current (main)
Spot-check key paths under local-branch-preview:
- Home landing — hero, tagline, architecture diagram, onboarding cards
- Get Started → Quick Start (task-list prerequisites, copy buttons on commands)
- Deploy → Before you deploy (Mermaid diagram), CI/CD (numbered workflow)
- Features hub — category cards + A–Z
- Legacy redirect — e.g. open local-branch-preview/quick_start.html and confirm redirect to get-started/quick-start.html

Optional quick iteration (single-version, no switcher): make -C docs html → open docs/build/html/index.html (layout only; no version menu).

Navigation (before → after)

Before	After
12+ flat top tabs, long labels	7 tabs + Home
What is as top-level tab	What is under Get Started sidebar
Samples, Spec Reference as top tabs	Nested under Samples / Build
Deploy scattered across multiple pages	Deploy hub with ordered cards
Features as flat list	Features hub + category sub-hubs + A–Z
Contributors as long toctree	Contributors hub + Contrib sidebar
Flat `docs/source/*.rst` filenames	Section folders with hub `index.rst` pages

Key new / restructured pages

Area	Pages
Home	`_landing.rst`, architecture diagram
Get Started	`get-started/quick-start.rst`, `get-started/what-is-lakeflow-framework.rst`
Deploy	`deploy/index.rst`, `deploy/before-you-deploy.rst`, `deploy/framework/*`, `deploy/pipeline-bundle/local.rst`, `deploy/ci-cd.rst`
Build	`build/index.rst`, `build/spec-reference/`, `build/patterns/`
Features	`features/*/index.rst` category hubs, `features/a-z.rst`
Contributors	`contributors/index.rst` hub + rewritten `contributors/dev-*.rst`
Redirects	`docs/redirects/*.yaml`, `docs/scripts/write_redirects.py`

Removed / retired

deploy_local.rst (split into framework vs pipeline guides)
deploy_framework_bundle.rst, deploy_pipeline_bundle.rst
pipeline_execution.rst (stub removed from nav)
orchestration.rst removed from top nav
RTD theme templates (docs/_templates/)
Superseded docs IA ADR (0008-docs-ia-home-architecture.md)

Other changes

docs/conf.py — immaterial theme, nav patching for section-index pages, nested hub wiring, copy-button and mermaid hooks, SEO/canonical preserved
docs/source/_static/databricks-theme.css — header/tabs/cards/hub grid/landing onboarding styles
Feature pages — consistent Overview headings and metadata strip link labels
samples/README.md — deploy cross-links, task-list prerequisites, logical-environment admonition
tests/test_select_versions.py — coverage for version selection logic

Notes for reviewers

~16 duplicate-label warnings in spec-reference shared fragments are pre-existing (shared .. include:: anchors). Build succeeds; not introduced by the folder restructure.

Test plan

Version

v0.20.0 → v0.21.0

…ilds Switch the docs theme to sphinx-immaterial with Databricks styling, a landing/quick-start IA, and the native version dropdown. Build each version from that ref's own conf/source so historical RTD docs stay intact, emit a shared superset versions.json for both selectors, and extend 0.x selection back to 0.12 (latest patch per minor).

Keep historical docs on main's conf so the version selector survives old tags, fix immaterial version_json resolution, and standardize feature pages with Overview sections, titled Databricks Docs links, and soft brand metadata badges.

Replace the flat/nested Framework parent with nine top-level Features sections, collapse inactive sidebar branches, and keep an A–Z index for lookup. Checkpoint before comparing hub layout E.

…rome Replace the split deploy pages with a Deploy hub (before you deploy, local, CI/CD), promote Samples and Architecture in top nav, and reorder Build (spec reference before patterns, Deploy after Build). Fix scroll-spy header title swap so the site name and version selector stay visible; drop the orchestration stub and retired deploy entry points.

Hub-based Deploy and Contributors sections, split framework vs pipeline local deploy guides, and contributor env/workflow/docs rewrites. Refresh home landing (nav-aligned cards, tagline, onboarding layout) and align What is with two-bundle and wheel/contrib messaging. Merge main for package namespace (#87), tpch sample rebuild, and unit tests. Bump VERSION to v0.21.0.

Drop duplicate ADR-0008 for docs information architecture; package ADR-0008 remains the canonical 0008 decision record.

Refresh the docs information architecture ahead of the planned site-wide move to folder/index hubs and post-build redirects: Features hub copy, category hub renames, schema management split, SDP flows and target catalog/schema terminology, shared inline architecture diagram, Quick Start RST migration, and sphinx-immaterial theme and navigation updates.

Move flat Sphinx sources into topic folders (get-started, architecture, build, deploy, features, contributors, samples) and update navigation, cross-references, and conf wiring. Add redirect YAML and post-build HTML stubs so legacy URLs keep working for current and local previews. Also refresh guided-adoption content: streamline CI/CD page layout, align Quick Start and samples README with deploy steps, and update the platform features hub intro.

Replace PNG topology diagrams with Mermaid flowcharts on pattern pages, drop the legacy PNG comparison blocks, and update Change Flow wording to auto CDC flow terminology. Restructure Example Data Flow sections from nested bullet lists to headings so sample tables no longer render as blockquotes, and extend lf-content-table CSS for raw HTML tables. Use absolute :doc: paths across build, deploy, features, contributors, get-started, and samples docs so links resolve reliably after the IA folder move. Remove a duplicate empty toctree from the site index and ignore docs/build/ in .gitignore.

…er logo Restructure Build > Patterns into a catalog hub with Base Patterns and Advanced composition stubs, plus standalone scaling and mix-and-match pages. Replace static pattern PNGs with lf-arch-styled Mermaid flowcharts, client-side rendering via mermaid-zoom.js, and legacy URL redirects. Fix immaterial search on flat builds by gating version_dropdown and normalizing stopwords in language_data.js. Update the header logo to the agreed block LFF monogram at nav-friendly scale on #EB1600.

CI installs docs deps with --require-hashes --no-deps from the lockfile. sphinx-immaterial was listed in requirements-docs.txt but missing from the lock, so conf.py failed to import the theme in GitHub Actions.

Load sphinx-immaterial mermaid and task-list directives during spelling builds, stop orphaning Mermaid source as spell-checked text, and skip non-prose diagram/raw nodes. Extend spelling_wordlist.txt with framework and docs vocabulary flagged by the checker.

Use main conf for current and tags >= v0.21.0 while keeping v0.20.0 conf for older release tags so legacy RTD layouts still build after IA rebrand.

Raise the header inner bar above sticky in-header tabs when the version dropdown is open, and override visited/hover link colors so preview and release links stay readable in dark mode. Also retitle the deploy local guide to “Deploy pipeline bundles”.

Convert UK spellings across docs and ADRs to US forms and update the spelling wordlist accordingly. Add a “Why teams use it” bullet on the What is page for centralized framework configuration inheritance at runtime.

rederik76 added 8 commits July 11, 2026 10:00

feat(docs): restructure Features hub into job-based sections (layout D)

c46eb05

Replace the flat/nested Framework parent with nine top-level Features sections, collapse inactive sidebar branches, and keep an A–Z index for lookup. Checkpoint before comparing hub layout E.

docs: remove superseded docs IA ADR

792b7ab

Drop duplicate ADR-0008 for docs information architecture; package ADR-0008 remains the canonical 0008 decision record.

rederik76 requested a review from haillew as a code owner July 12, 2026 08:57

rederik76 self-assigned this Jul 12, 2026

rederik76 added 8 commits July 12, 2026 21:15

fix(docs): add sphinx-immaterial to requirements-docs.lock

fe14e37

CI installs docs deps with --require-hashes --no-deps from the lockfile. sphinx-immaterial was listed in requirements-docs.txt but missing from the lock, so conf.py failed to import the theme in GitHub Actions.

fix(docs): cut over versioned builds to v0.21.0 conf at release

1dbbc76

Use main conf for current and tags >= v0.21.0 while keeping v0.20.0 conf for older release tags so legacy RTD layouts still build after IA rebrand.

fix get started paragraph

86c4efd

haillew approved these changes Jul 13, 2026

View reviewed changes

rederik76 merged commit d474150 into main Jul 13, 2026
5 checks passed

rederik76 deleted the feature/docs-guided-adoption-ia branch July 13, 2026 03:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(docs): guided adoption IA — theme, hubs, section folders, and redirects#121

feat(docs): guided adoption IA — theme, hubs, section folders, and redirects#121
rederik76 merged 16 commits into
mainfrom
feature/docs-guided-adoption-ia

rederik76 commented Jul 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rederik76 commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to preview this branch (recommended)

Navigation (before → after)

Key new / restructured pages

Removed / retired

Other changes

Notes for reviewers

Test plan

Version

Related

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rederik76 commented Jul 12, 2026 •

edited

Loading