cli_reference.md

Kindling CLI Reference

The kindling CLI is installed via spark-kindling-cli. Run any command with -h or --help for inline help. Use -V / --version to print CLI, SDK, and runtime versions.

kindling -V
kindling <group> --help
kindling <group> <command> --help

---

pipeline

Locally run and inspect medallion pipeline layers.

pipeline run <PIPE_ID>

Run a single registered pipe locally against already-populated upstream entity storage. Primary dev iteration loop — run one layer at a time without re-running upstream layers.

Entity data sources follow the priority stack: tests/entities/ fixture CSV → kindling.yaml env mapping → registered provider.

Option	Default	Description
--app PATH	auto	Path to app.py (auto-discovered when omitted)
--env TEXT	KINDLING_ENV or local	Config environment overlay
--config PATH	—	Config directory override (if app.py supports it)
--quiet / -q	—	Suppress INFO logs; show WARNING and above only
--no-watermark	—	Bypass watermark tracking; process full dataset

kindling pipeline run bronze.ingest_myproject --app apps/my_pipeline --env dev
kindling pipeline run silver.stage_myproject --env dev

pipeline list

List all registered pipe IDs for an app.

Option	Default	Description
--app PATH	auto	Path to app.py
--env TEXT	KINDLING_ENV or local	Config environment overlay

---

migrate

Inspect and apply entity schema migrations. Requires a running Spark session — call these from within a Kindling notebook or pipeline context.

migrate plan

Show pending schema changes for all registered entities without applying them. Flags destructive changes (type changes, column removal, partition changes) so you can review before applying.

migrate apply

Apply pending schema migrations.

• Non-destructive changes (column additions) are always applied.
• Destructive changes require --destructive.
• CATALOG entities use a blue-green strategy: old table archived as <name>_migration_blue until cleanup.
• STORAGE entities rewrite in place using Delta's ACID guarantees.

Option	Default	Description
--destructive	—	Allow destructive changes
--backup none|snapshot	none	Backup strategy before destructive changes

migrate rollback <ENTITY_ID>

Restore a CATALOG entity to its pre-migration state by promoting the <name>_migration_blue archive back to live. Only valid after a blue-green apply that has not yet been cleaned up.

migrate cleanup <ENTITY_ID>

Drop blue-green artifacts (_migration_blue, _migration_green) after a confirmed successful migration.

---

config

Manage Kindling configuration files.

config init

Generate an initial settings.yaml file.

Option	Default	Description
--output PATH	settings.yaml	Output file path
--name TEXT	inferred from pyproject.toml	App name written into the config
--force	—	Overwrite if already exists

config set <KEY> <VALUE>

Set a configuration value using dot-notation keys. Supports base, platform, and env scopes.

Option	Default	Description
--level base|platform|env	base	Config scope
--platform databricks|fabric|synapse	—	Required when --level=platform
--env TEXT	—	Environment name (required when --level=env)
--app TEXT	—	App name; targets the app-specific config directory
--config-dir PATH	.	Root directory containing config files

kindling config set kindling.telemetry.logging.level DEBUG
kindling config set kindling.bootstrap.load_lake false --level platform --platform fabric
kindling config set kindling.secrets.secret_scope my-scope --level env --env prod

---

env

Validate and maintain local environment prerequisites.

env check

Check whether the local environment is ready for Kindling.

Option	Default	Description
--config PATH	settings.yaml	Settings file to validate
--local	—	Also check Java, PySpark, delta-spark, and hadoop-azure JARs
--platform databricks|fabric|synapse	auto-detected	Check platform pre-flight readiness: required vars are reported as SET or MISSING with export hints, and authentication alternatives are listed in the SDK's resolution order. Exits 1 unless the platform is ready

Platform pre-flight requirements:

Platform	Required vars	Auth — one of
databricks	DATABRICKS_HOST	DATABRICKS_TOKEN; the AZURE_TENANT_ID + AZURE_CLIENT_ID + AZURE_CLIENT_SECRET service principal triple; or a usable Azure CLI session (az login)
fabric	FABRIC_WORKSPACE_ID, FABRIC_LAKEHOUSE_ID	the service principal triple, or az login / managed identity at run time
synapse	SYNAPSE_WORKSPACE_NAME, SYNAPSE_SPARK_POOL_NAME	the service principal triple, or az login / managed identity at run time

A partially set service principal triple fails the check naming the missing vars. For Databricks workspaces with personal access tokens disabled, mint a short-lived Microsoft Entra ID token instead:

export DATABRICKS_TOKEN=$(az account get-access-token \
  --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d \
  --query accessToken -o tsv)

kindling env check --local
kindling env check --platform fabric

env ensure

Download local Spark + ABFSS support JARs into /tmp/hadoop-jars/. Safe to re-run; existing JARs are skipped.

kindling env ensure

env update

Update Kindling packages inside a domain devcontainer without rebuilding the container. This downloads wheel assets from the public Kindling GitHub release, refreshes /opt/kindling-packages/wheels/ and the local PEP 503 index at /opt/kindling-packages/simple/, reinstalls Kindling into the current Python environment, then runs poetry update and poetry install for the current project.

Option	Default	Description
--version TEXT	latest	Kindling release version or tag
--repo TEXT	sep/spark-kindling-framework	GitHub repository containing release wheels
--package-dir PATH	/opt/kindling-packages	Local wheel cache used by generated projects
--project PATH	.	Poetry project to update
--no-project	—	Refresh the devcontainer cache only
--no-global	—	Skip reinstalling Kindling into the current Python environment
--no-sync	—	Run poetry install without --sync
--no-sudo	—	Do not use sudo for image-owned cache paths

kindling env update
kindling env update --version 0.10.35

---

workspace

Manage and deploy platform workspace configuration.

workspace init

Initialize the platform workspace: deploy settings.yaml and overlay configs to {base}/config/ in storage. With --notebook-bootstrap, also generates and imports notebook bootstrap files into the platform workspace via platform APIs.

Use this for first-time workspace setup.

Option	Default	Description
--platform databricks|fabric|synapse	auto-detected	Target platform
--storage-account TEXT	AZURE_STORAGE_ACCOUNT	Storage account name, name.domain, or full URL
--container TEXT	AZURE_CONTAINER or artifacts	Blob container name
--base-path TEXT	AZURE_BASE_PATH	Base path prefix within container
--config PATH	settings.yaml	Settings file to deploy
--notebook-bootstrap	—	Generate and import notebook bootstrap files into the platform workspace
--workspace TEXT	—	Platform workspace for notebook import (required with --notebook-bootstrap)
--overwrite	—	Overwrite existing config and notebooks

kindling workspace init --platform fabric --storage-account myacct
kindling workspace init --platform fabric --storage-account myacct \
    --notebook-bootstrap --workspace <workspace-id>

workspace deploy

Re-deploy config to Azure Storage after settings.yaml changes. Deploys settings.yaml + overlays to {base}/config/ in storage only.

Option	Default	Description
--platform databricks|fabric|synapse	auto-detected	Target platform
--config PATH	settings.yaml	Settings file to deploy
--storage-account TEXT	AZURE_STORAGE_ACCOUNT	Storage account name, name.domain, or full URL
--container TEXT	AZURE_CONTAINER or artifacts	Blob container name
--base-path TEXT	AZURE_BASE_PATH	Base path prefix within container
--skip-config	—	Skip config upload
--overwrite	—	Overwrite existing config
--allow-missing-config	—	Don't fail if settings file is not found

kindling workspace deploy --platform synapse --storage-account mystorageacct
kindling workspace deploy --platform fabric --storage-account mystorageacct --overwrite

---

notebook

Round-trip workspace notebooks as local python source files. Local files use the Databricks source format — cells separated by # COMMAND ----------, markdown cells as # MAGIC blocks — which is git-friendly and is the same format the standalone platform reads as local workspace notebooks, so pulled files run locally unchanged. Outputs and execution counts are not preserved; the round-trip carries code and markdown.

The underlying operations are available programmatically via kindling_sdk.notebooks (create_notebook_client(platform, workspace) → list_notebooks / get_notebook_cells / import_notebook / delete_notebook), paired with kindling.notebook_source for the cells ↔ python-source conversion.

All three commands share these options:

Option	Default	Description
--platform databricks|fabric|synapse	auto-detected	Target platform
--workspace TEXT	platform env var	Workspace: URL (Databricks), name (Synapse), id (Fabric). Falls back to DATABRICKS_HOST / SYNAPSE_WORKSPACE_NAME / FABRIC_WORKSPACE_ID
--folder TEXT	/Shared/kindling	Workspace folder (Databricks only)

notebook list

List notebooks in the platform workspace.

kindling notebook list --platform databricks
kindling notebook list --platform fabric --workspace <workspace-id>

notebook pull

Export workspace notebooks to local .py source files. Refuses to replace existing local files unless --overwrite is passed.

Option	Default	Description
NAMES...	—	Notebook names to pull
--all	—	Pull every notebook in the workspace
--out DIR	notebooks	Local directory for the .py files
--overwrite	—	Replace existing local files

kindling notebook pull my_pipeline --platform databricks
kindling notebook pull --all --out notebooks/ --overwrite

notebook push

Import local .py source files into the workspace as notebooks (created or replaced by name; the notebook name defaults to the file stem).

Option	Default	Description
FILES...	—	Local .py source files to push
--name TEXT	file stem	Workspace notebook name (single file only)

kindling notebook push notebooks/my_pipeline.py --platform databricks
kindling notebook push notebooks/*.py --platform synapse

---

runtime

Manage kindling runtime artifacts.

runtime deploy

Deploy kindling runtime artifacts (wheels + bootstrap script) to Azure Data Lake Storage. This is the primary path for installing kindling into a new environment or promoting between environments (e.g. staging → prod).

Source types:

Source	Example	Description
github:VERSION	github:latest, github:0.10.15	Download from the public Kindling GitHub release
local:PATH	local:./dist	Read wheels from a local directory
abfss://…	abfss://artifacts@staging.dfs.core.windows.net/k	Copy from another ADLS path

Destination layout under --dest:

{dest}/packages/   ← spark_kindling-*.whl
{dest}/scripts/    ← kindling_bootstrap.py

The --dest root is your artifacts_storage_path in BOOTSTRAP_CONFIG.

Option	Default	Description
--source TEXT	required	Source specifier (see above)
--dest TEXT	required	Destination abfss:// URI
--version TEXT	—	Release version for github: source; overrides the version embedded in --source
--skip-bootstrap	—	Skip the bootstrap script
--overwrite	—	Overwrite existing scripts (wheels always overwrite)

# Install the latest release into a storage account
kindling runtime deploy \
  --source github:latest \
  --dest abfss://artifacts@myacct.dfs.core.windows.net/kindling

# Install a specific version
kindling runtime deploy \
  --source github:0.10.15 \
  --dest abfss://artifacts@myacct.dfs.core.windows.net/kindling

# Deploy from a local build
kindling runtime deploy \
  --source local:./dist \
  --dest abfss://artifacts@mydev.dfs.core.windows.net/kindling

# Promote staging → prod (ADLS to ADLS)
kindling runtime deploy \
  --source abfss://artifacts@staging.dfs.core.windows.net/kindling \
  --dest abfss://artifacts@prod.dfs.core.windows.net/kindling

Azure auth uses DefaultAzureCredential — az login or AZURE_TENANT_ID / AZURE_CLIENT_ID / AZURE_CLIENT_SECRET both work. GitHub auth is not required for public Kindling releases.

---

app

Validate, package, deploy, run, and inspect Kindling applications.

app init <APP_NAME>

Create a Kindling app under apps/ in an existing repo. Generates app.py, app.yaml, lake-reqs.txt, settings.yaml, settings.local.yaml, and QUICKSTART.md.

Option	Default	Description
--package TEXT	app name	Domain package imported by the app
--auth oauth|key|cli	oauth	ABFSS auth method for generated local config comments
--layers medallion|minimal	medallion	Entity/pipe template style
--repo-root PATH	.	Repo root that will receive the app under apps/
--pattern batch|streaming|file-ingestion	—	Execution pattern to scaffold; omit for a minimal hello-world app
--template-dir PATH	—	Directory of custom Jinja2 templates overlaying the built-ins

app validate

Validate entity and pipe definitions without starting Spark. Checks entity and pipe registration, pipe input/output entity existence, and delta entity merge_columns presence.

Option	Default	Description
--app PATH	auto	Path to app.py
--env TEXT	KINDLING_ENV or local	Config environment to load

app package <APP_NAME>

Package an app directory into a .kda archive. Looks up apps/<app_name>/ by convention; use --local-folder for non-standard layouts.

Option	Default	Description
--local-folder PATH	—	Override convention lookup with this directory
--output PATH	dist/<app>.kda	Destination .kda file or directory
--json	—	Machine-readable JSON output

app deploy <APP_NAME>

Deploy an app to a remote platform. Looks up apps/<app_name>/ by convention and packages on the fly; use --local-folder to override or --kda-package for a pre-built archive.

Option	Default	Description
--local-folder PATH	—	Override convention lookup. Packaged on the fly.
--kda-package PATH	—	Pre-built .kda archive. Skips convention lookup.
--remote-name TEXT	source dir stem	Remote app name
--platform databricks|fabric|synapse	auto-detected	Target platform
--json	—	Machine-readable JSON output

app run <APP_NAME>

Run an app locally (standalone) or submit a run of a deployed app (remote).

Standalone (--platform standalone, the default): looks up apps/<app_name>/ by convention and runs locally with embedded Spark. Use --local-folder to override for non-standard layouts.

Remote (--platform databricks|fabric|synapse): submits a run of the already-deployed app. The app must have been deployed first with kindling app deploy. --local-folder has no meaning for remote runs and will error.

Option	Default	Description
--platform standalone|databricks|fabric|synapse	standalone	Execution platform
--local-folder PATH	—	Override convention lookup (standalone only)
--app-name TEXT	—	Remote app name override (remote only)
--env TEXT	—	Runtime environment
--config PATH	—	Config directory override (standalone only)
--quiet / -q	—	Suppress INFO logs (standalone only)
--local-package PATH	—	Prepend a local package root to PYTHONPATH (repeatable; standalone only)
--parameters PATH	—	YAML/JSON file of runtime parameters
--param KEY=VALUE	—	Runtime parameter override (repeatable)
--no-wait	—	Return immediately after starting (remote)
--no-logs	—	Skip log streaming (remote)
--poll-interval FLOAT	5.0	Status poll interval in seconds
--timeout FLOAT	3600.0	Max wait time in seconds
--fail-on-error / --no-fail-on-error	fail	Exit non-zero on failed run
--dotenv PATH	.env	Dotenv file to load (repeatable)
--no-dotenv	—	Do not load .env
--json	—	Machine-readable JSON output

# Local standalone — convention lookup
kindling app run my-pipeline
kindling app run my-pipeline --local-package packages/my_pipeline --env local

# Local standalone — non-standard layout
kindling app run my-pipeline --local-folder path/to/app

# Remote — deploy first, then run
kindling app deploy my-pipeline --platform synapse
kindling app run my-pipeline --platform synapse

app status <RUN_ID>

Fetch the current status for a remote app run.

Option	Default	Description
--platform databricks|fabric|synapse	auto-detected	Target platform

app logs <RUN_ID>

Fetch or stream logs for a remote app run.

Option	Default	Description
--from-line INT	0	Starting line number
--size INT	1000	Number of lines to fetch
--stream	—	Tail logs until completion or timeout
--poll-interval FLOAT	5.0	Poll interval for streaming
--max-wait FLOAT	300.0	Max streaming wait in seconds
--platform databricks|fabric|synapse	auto-detected	Target platform

app cancel <RUN_ID>

Cancel a remote app run.

Option	Default	Description
--platform databricks|fabric|synapse	auto-detected	Target platform
--json	—	Machine-readable JSON output

app cleanup <APP_NAME>

Delete a previously deployed remote application. Pass the app name directly — the same name used when deploying.

Option	Default	Description
--platform databricks|fabric|synapse	auto-detected	Target platform
--json	—	Machine-readable JSON output

app inspect <APP_NAME>

Show entity resolution information — provider type, storage path, and whether a tests/entities/ fixture override is active.

Option	Default	Description
--env TEXT	local	Config environment
--app PATH	auto	Path to app.py
--entities	—	Show full entity resolution table

---

entity

Inspect and validate entity data during local development.

entity show <ENTITY_ID>

Print entity contents from its data source. Reads via the priority stack: tests/entities/ fixture CSV first, then the registered provider.

Option	Default	Description
--env TEXT	local	Config environment
--app PATH	auto	Path to app.py
--limit INT	20	Max rows to display
--count	—	Print row count only

entity validate <ENTITY_ID>

Run basic data quality checks: row count (ERROR if zero), null check (WARN on nulls in key/required columns), schema match (WARN if fixture columns differ from entity schema). Exits 0 on pass or warnings; exits 1 on any ERROR.

Option	Default	Description
--env TEXT	local	Config environment
--app PATH	auto	Path to app.py

---

runner

Register and manage per-app job definitions for pipeline/workflow integration.

kindling app run submits one-time runs directly — no job definition is needed for that. Use runner register only when you want to expose an app as a named job definition that an external orchestrator (Synapse Pipeline, Databricks Workflow, Fabric Pipeline) can trigger by name.

runner register

Register an app as a named job definition on the platform. Creates or updates the definition in place (idempotent). Config overrides supplied with --config are baked into the job definition as bootstrap args.

Option	Default	Description
--app TEXT	required	App name to register as a job definition
--config KEY=VALUE	—	Config override baked into the job definition (repeatable)
--platform databricks|fabric|synapse	auto-detected	Target platform
--json	—	Machine-readable JSON output

kindling runner register --app my-app --platform synapse
kindling runner register --app my-app --config env=prod --config region=eastus

runner status

Show which apps have registered job definitions on the platform. Without --app the command scans the current directory for apps and reports their registration state.

Option	Default	Description
--app TEXT	all discovered	App name to check
--platform databricks|fabric|synapse	auto-detected	Target platform
--json	—	Machine-readable JSON output

kindling runner status
kindling runner status --app my-app --platform synapse --json

runner delete

Delete a registered job definition from the platform workspace. Prompts for confirmation unless --yes is supplied.

Option	Default	Description
--app TEXT	required	App name whose job definition should be deleted
--platform databricks|fabric|synapse	auto-detected	Target platform
--yes	—	Skip interactive confirmation prompt
--json	—	Machine-readable JSON output

kindling runner delete --app my-app --platform fabric --yes
kindling runner delete --app my-app  # prompts for confirmation

runner invoke

Invoke a registered job directly with a raw parameters YAML/JSON file. Advanced/debug use only — normal execution should use kindling app run.

The parameters file is passed directly to the platform job run API. Include job_id or app_name in the file to identify the target job.

Option	Default	Description
--params PATH	required	YAML or JSON parameters file
--platform databricks|fabric|synapse	auto-detected	Target platform
--wait	—	Poll until the run completes
--poll-interval FLOAT	10.0	Poll interval in seconds
--timeout FLOAT	3600.0	Timeout in seconds
--json	—	Machine-readable JSON output

kindling runner invoke --params params.yaml
kindling runner invoke --params params.yaml --wait --platform fabric

---

test

Run Kindling project test suites.

test run

Run pytest through Kindling's portable test wrapper.

Option	Default	Description
--suite unit|component|integration|system|extension|all	unit	Logical test suite
--path PATH	tests/<suite>	Pytest path to run (repeatable)
--platform databricks|fabric|synapse	—	Target platform
--test TEXT	—	Pytest -k expression
--marker TEXT	—	Pytest -m expression
--ci	—	Emit junit/json reports and fail fast
--results-dir PATH	test-results	Results directory
--workers TEXT	—	pytest-xdist worker count
--coverage TEXT	—	Coverage target (--cov=<target>; repeatable)
--no-cov	—	Pass --no-cov to pytest
--preflight none|local|system	none	Optional preflight check
--dotenv PATH	.env	Dotenv file to load (repeatable)
--no-dotenv	—	Do not load .env
--pytest-arg TEXT	—	Extra argument passed through to pytest (repeatable)

test check

Run a Kindling test preflight check without pytest.

Option	Default	Description
--preflight local|system	local	Preflight type
--platform databricks|fabric|synapse	—	Target platform

test cleanup

Run project cleanup hooks for system-test resources.

Option	Default	Description
--platform databricks|fabric|synapse	—	Target platform
--all	—	Clean all configured platforms
--skip-packages	—	Skip cleanup of old package artifacts

---

repo

Scaffold and manage multi-package Kindling repos.

repo init <REPO_NAME>

Create a Kindling repo root with shared dev tooling (.devcontainer/, .github/workflows/ci.yml, .gitignore).

Option	Default	Description
--output-dir PATH	.	Directory to initialize as the repo root
--template-dir PATH	—	Custom Jinja2 templates overlaying the built-ins
--overwrite-devcontainer	—	Replace an existing devcontainer.json

---

package

Create and deploy Kindling domain packages.

package init <PACKAGE_NAME>

Create a Kindling package under an existing multi-package repo at packages/<name>/.

Option	Default	Description
--auth oauth|key|cli	oauth	Auth style for generated test/config examples
--layers medallion|minimal	medallion	Package template style
--no-integration	—	Omit tests/integration/
--repo-root PATH	.	Repo root to receive the new package
--template-dir PATH	—	Custom Jinja2 templates

package deploy <PACKAGE_NAME>

Build a package wheel with Poetry and upload it to artifact storage at {base}/packages/. Looks up packages/<package_name>/ by convention; use --local-folder for non-standard layouts.

Option	Default	Description
--local-folder PATH	—	Override convention lookup with this directory
--dist-dir PATH	dist	Directory where Poetry writes the wheel
--storage-account TEXT	AZURE_STORAGE_ACCOUNT	Storage account
--container TEXT	AZURE_CONTAINER or artifacts	Container name
--base-path TEXT	AZURE_BASE_PATH	Base path prefix
--json	—	Machine-readable JSON output

package add entity <ENTITY_ID>

Scaffold an entity definition and a CSV fixture stub.

• Appends a DataEntities.entity() skeleton to <path>/entities.py
• Creates tests/entities/<ns>/<name>.csv

Option	Default	Description
--package PATH	required	Package root

package add pipe <PIPE_ID>

Scaffold a DataPipes pipe and matching unit/integration test stubs.

Option	Default	Description
--inputs TEXT	—	Comma-separated input entity IDs
--package PATH	required	Package root

package add ingestion <ENTITY_ID>

Scaffold a file-ingestion pipe and matching test stubs. The --source-pattern is matched against the filename (not the full ABFSS path); named groups are automatically extracted as columns by the framework.

Option	Default	Description
--source-pattern TEXT	—	Regex for matching filenames; named groups become columns
--filename-metadata TEXT	—	Field name to extract from a named capture group (ignored when --source-pattern is set)
--package PATH	required	Package root

---

agent

Manage agent instruction files for Claude Code, Copilot, and Codex.

agent setup

Generate (or update) agent instruction files from the Kindling reference doc:

• CLAUDE.md — Claude Code
• .github/copilot-instructions.md — GitHub Copilot
• AGENTS.md — Codex / OpenAI agents

Re-run after pulling a new devcontainer image to pick up updated documentation.

Option	Default	Description
--force	—	Regenerate even if version is unchanged
--check	—	Report whether files are up to date without writing
--project PATH	.	Project root directory

---

Environment variables

Variable	Used by	Description
AZURE_STORAGE_ACCOUNT	workspace init, workspace deploy, package deploy, runtime deploy	Storage account name
AZURE_CONTAINER	same	Blob container (default: artifacts)
AZURE_BASE_PATH	same	Base path prefix within container
AZURE_TENANT_ID	all Azure auth	Service principal tenant
AZURE_CLIENT_ID	all Azure auth	Service principal client ID
AZURE_CLIENT_SECRET	all Azure auth	Service principal secret
AZURE_CLOUD	Azure auth	Cloud environment (AzureUSGovernment, AzureChinaCloud, etc.)
DATABRICKS_HOST	databricks platform	Databricks workspace URL
DATABRICKS_TOKEN	databricks platform	Databricks PAT (or Entra ID token) — one auth alternative; the AZURE_* service principal triple or an az login session also works
FABRIC_WORKSPACE_ID	fabric platform	Fabric workspace GUID
FABRIC_LAKEHOUSE_ID	fabric platform	Fabric lakehouse GUID
SYNAPSE_WORKSPACE_NAME	synapse platform	Synapse workspace name
SYNAPSE_SPARK_POOL_NAME	synapse platform	Spark pool name
KINDLING_ENV	app/pipeline/config	Default environment overlay