Upgrade to DBR 18 LTS, enforce CLI version, and improve CI/CD traceability

[andre-salvati]

Summary

Upgrades the template to the DBR 18 line and tightens reproducibility by pinning/enforcing a minimum Databricks CLI version. Also improves CI/CD deployments by propagating PR/branch/actor metadata into the workflow generation step, and refreshes docs + SDK demo scripts.

Key changes

• Runtime/tooling refresh

  • Move to Databricks Runtime 18.0 LTS + PySpark 4.1 .
  • Enforced bundle CLI version via _databricks_.yml (_databricks__cli_version: ">=0.286.0").

• CI/CD improvements (GitHub Actions)

  • Updated onpush.yml to capture branch name, PR number, and developer (actor) and pass them into generate_template_workflow.py for staging/prod serverless deploys.
  • This supports richer job metadata (see “add tags for job observability” commit) and makes runs easier to trace back to a PR/branch/author.

• Repo hygiene / docs

  • .gitignore cleanup and Makefile simplification (removes some deploy targets).
  • README edits and docs updates (including CI/CD diagram assets).
  • Added/updated SDK demo scripts (e.g., system tables / workspace-account / cost analysis demos) per commit history + file list.

Why

• Keep the template aligned with the newer DBR/PySpark line.
• Avoid “works on my machine” issues by ensuring a compatible Databricks CLI version.
• Improve job observability and traceability by embedding PR/branch/actor context during workflow generation and deploy steps.