skills: split into databricks-model-serving (ops) + databricks-ml-training (experimental)

[QuentinAmbard]

Stacks on top of #84. Merge #84 first; this PR includes those commits at the base and adds the new skill on top.

Why this PR exists

PR #84 lands the model-serving content (endpoint create, query, update, traffic config, AI Gateway, Foundation Model API discovery) into databricks-model-serving. That's the right shape for a serving-ops skill, and it's what reviewers should expect a skill called "model serving" to contain.

The remaining a-d-k content — training a model with MLflow autolog, registering it to Unity Catalog, promoting versions via @prod aliases, custom PyFunc authoring, hand-rolled ResponsesAgent code — is a different lifecycle. It runs before an endpoint exists, often in a notebook submitted as a serverless job, and an agent asked to "train an XGBoost model and deploy it" needs both concerns surfaced cleanly rather than blended into one skill description.

This PR lands the dev-side content as a separate databricks-ml-training experimental skill, and weaves a few small but high-leverage serving-side fixes from the original a-d-k content into databricks-model-serving where they belong.

What this PR improves

A focused dev-side skill. New experimental/databricks-ml-training/ owns the training → register → consume narrative: MLflow autolog with Optuna for hyperparameter tuning, mlflow.set_registry_uri(\"databricks-uc\") + experiment-parent-folder pre-creation, alias-based promotion (@prod / @challenger), batch scoring via mlflow.pyfunc.spark_udf, custom PyFunc with the file-based "Models from Code" pattern, hand-rolled ResponsesAgent with LangGraph + UC Function + Vector Search tools, and the databricks jobs submit --no-wait train-and-deploy pattern.

Frontmatter triggers that actually triage. Each skill's description lists what it IS for and what it explicitly is NOT for, with cross-pointers (databricks-ml-training says "use databricks-model-serving for endpoint ops"; databricks-model-serving says "use databricks-ml-training for training and PyFunc authoring"). When the user says "train a model and deploy it," the orchestrator pulls both skills exactly once each.

Cross-skill links that resolve. Every databricks-model-serving → databricks-ml-training link and every reverse link uses the right relative path for the stable-skills/ ↔ experimental-experimental/ layout. No broken anchors, no stale paths to the old training-and-serving.md filename anywhere.

Five small but high-leverage gaps closed in databricks-model-serving. The original a-d-k port left a few non-obvious serving behaviors implicit. Each fix is woven into the existing section that already covers the topic — no new mega-sections, no duplication of MLflow boilerplate an LLM already knows from training data. The result: serving-side behavior an agent would otherwise have to guess at is now explicit and signposted.

Summary of changes

Area	What changed
New experimental skill	experimental/databricks-ml-training/ with SKILL.md + agents/ + assets/ + references/{custom-pyfunc.md, genai-agents.md}. Owns the full dev-side narrative (autolog + Optuna, UC registration, alias promotion, batch scoring, custom PyFunc, custom ResponsesAgent, train-and-deploy serverless job pattern).
Frontmatter scoping	Both skills' descriptions list scope + explicit NOT-for callouts pointing at their sibling. Triggers triage cleanly when the user mentions both training and deployment.
Cross-links	All cross-skill paths fixed for the stable ↔ experimental layout. Foundation Model API discovery moved into databricks-model-serving/SKILL.md inline (was previously linked into the relocated training file).
Serving gaps closed	Five small fixes in databricks-model-serving/SKILL.md: MLflow Deployments client gotchas (tags= top-level, served_model_name derivation), zero-downtime version-swap pattern (alias-repoint AND update_endpoint), two-state-field readiness rationale (state.ready lies during version-swap), classical-ML dataframe_records query example, Serving-UI "Owned by me" SP-filter troubleshooting row. Each merged into the existing section that already covered the topic.
DAS-only content preserved	PR #84's idempotency check before agent deploy, AppKit integration section, off-platform AI SDK v6 streaming, endpoint-structure ASCII diagram, OpenAPI schema section — all kept.
Versioning	databricks-model-serving bumped to 0.4.0 (description retightened, gaps closed). New databricks-ml-training at 0.1.0 under experimental/. Manifest regenerated; 27 skills total.

Reviewer aid

The split is on the natural seam — anything that runs before mlflow.deployments.get_deploy_client(...).create_endpoint(...) is dev-side and lives in databricks-ml-training, anything from create_endpoint onward is ops-side and lives in databricks-model-serving. The Python create_endpoint(...) / update_endpoint(...) call itself is canonically a serving operation and is documented there with the two non-obvious gotchas.

Validation: python3 scripts/skills.py validate passes; zero broken links across both touched skills.

This pull request and its description were written by Isaac.

[simonfaltum]

Reviewed the proposed end state. Note this PR is one commit stacked on the still-open #84, so what shows here is the combined ~712-line delta against main; merge #84 first, then re-check this PR's true delta (relevant to the vector-search link below).

Verdict: fix-then-merge - no blockers, but a few things to address, flagged inline.

Headline items:

• The HPO "train and register" example silently promotes the wrong model (autolog registers every trial; promotion picks latest-by-version, not best-by-metric).
• A cross-link to databricks-jobs points at a section (and content) that doesn't exist.
• The databricks-vector-search link will break once rebased onto current main (vector-search moved to skills/).
• MLflow pins (mlflow==2.22.0) contradict the "MLflow 3" text and the skill's own pin-to-runtime rule.

Verified clean (so you see coverage):

• python3 scripts/skills.py validate passes; manifest / Codex metadata / icons in sync. (The model-serving manifest description staying short is by design - stable skills get a curated description, experimental ones derive from frontmatter.)
• The MLflow APIs are real, not invented: ResponsesAgent, the ResponsesAgentRequest/Response/StreamEvent classes, output_to_responses_items_stream / to_chat_completions_input (match the official ResponsesAgent docs), and DatabricksLakebase(database_instance_name=...).
• CLI flags verified against the CLI: jobs submit --no-wait, jobs list-runs --active-only -o json.
• No real credentials / workspace IDs; placeholders throughout; no destructive defaults.
• Strategy is strong: climbs to MLflow + UC registry + serverless jobs + serving, lands a durable governed gold UC table, delegates no-code agents to databricks-agent-bricks, public APIs only.

The model-serving additions (Deployments-client gotchas, alias + update_endpoint version swap, two-field readiness, dataframe_records query, runtime FM-API listing, SP-filter troubleshooting) are high-signal - one small client-variable issue noted inline.

Posted as a COMMENT (advisory, non-blocking).

[simonfaltum]

Promotes the wrong model. With autolog(registered_model_name=FULL_NAME) (line 85), every trial's .fit() logs and registers a version, so 20 trials produce ~20 versions. max(..., key=version) here then picks the last trial to finish, not the best by AUC, so @prod lands on an arbitrary model and the Optuna search is wasted. The prose at line 45 ("the best one is what gets registered") is inaccurate.

Fix: after study.optimize, either retrain once on study.best_params in a single parent run and register that, or select the winning run explicitly, e.g. client.search_runs(experiment_ids=[...], order_by=["metrics.<auc> DESC"], max_results=1) and alias that version.

[simonfaltum]

This anchor doesn't resolve. databricks-jobs/SKILL.md has no heading matching #one-time-runs-jobs-submit--async-pattern-for-notebooks, and the "four jobs submit traps / full debugging flow" content isn't in that skill at all (grep for jobs submit, --no-wait, notebook_output finds nothing; only spec.client: "4" exists, and it's in its references/task-types.md). An agent following the link lands at the top of databricks-jobs and never finds what's promised here.

Fix: inline the four traps here, point at the real location, or add the section to databricks-jobs.

[simonfaltum]

This link will break once the PR rebases onto current main. databricks-vector-search was promoted from experimental/ to skills/ on main (it's now skills/databricks-vector-search/, gone from experimental/). ../databricks-vector-search/SKILL.md resolves on this branch only because it's based on an older main.

Fix: ../../skills/databricks-vector-search/SKILL.md.

[simonfaltum]

This pin contradicts the surrounding text and the skill's own rule. The text calls ResponsesAgent "MLflow 3's standardized agent interface" and notes "DBR 16.1+ has mlflow 3.x", but pins mlflow==2.22.0 here (and in custom-pyfunc.md:72, SKILL.md:182). The Gotchas table (SKILL.md:236) warns that a pip_requirements mismatch crashes the endpoint at load and says to pin to the runtime. Logging from a DBR-3.x runtime but forcing serving to 2.22.0 is exactly that skew.

Fix: pin the MLflow 3.x version DBR ships, or use the live f"mlflow=={version('mlflow')}" pattern the skill already recommends. (ResponsesAgent does exist in 2.22.0, so it's not a guaranteed import error, but the version skew with databricks-langchain / langgraph is the real risk.)

[simonfaltum]

At ~850 characters this is the longest skill description in the repo (the experimental median is ~250; the next-longest is ~740). It's the agent's routing input. The explicit "Use when / NOT for" triage is genuinely useful and worth keeping, but the framing around it could be trimmed to bring it closer to siblings.

[simonfaltum]

Stale reference. training-and-serving.md doesn't exist anywhere in the repo (the PR description itself says no such paths remain). Drop the comment or point at the live source for resolving the LLM endpoint at runtime.

[simonfaltum]

This won't run as written. The block reads parameters from sys.argv[1..3], but line 220 says to submit it "as the notebook" via jobs submit, and the submit JSON passes no parameters. Notebook tasks receive parameters via dbutils.widgets, not sys.argv, so sys.argv[1] raises IndexError. (dbutils.notebook.exit at line 214 is notebook-only, confirming this is meant as a notebook.)

Fix: read params via dbutils.widgets.get(...) and pass base_parameters in the submit, or run it as a spark_python_task.

[simonfaltum]

These two calls are on different client types but share one client variable. set_registered_model_alias(...) is a method on mlflow.tracking.MlflowClient; update_endpoint(...) is on the MLflow Deployments client (mlflow.deployments.get_deploy_client("databricks")). As written, a single client can't do both - whichever it is, the other call raises AttributeError.

Fix: use two distinctly-named clients, e.g. mlflow_client.set_registered_model_alias(...) and deploy_client.update_endpoint(...).

[simonfaltum]

Two copy-paste snags: feature_cols is used but never defined, and F.current_timestamp() needs from pyspark.sql import functions as F (not auto-imported). Worth fixing so the example runs verbatim.

[QuentinAmbard]

Thanks for the thorough review, @simonfaltum — addressed in 68bf9e5:

experimental/databricks-ml-training/SKILL.md

• Autolog now runs without registered_model_name. A separate best run retrains with study.best_params and registers that single artifact, so @prod points at the actual winning trial rather than whichever trial happened to finish last.
• Added the missing from pyspark.sql import functions as F import and defined feature_cols in the batch-scoring block.
• Dropped the bogus #one-time-runs-jobs-submit--async-pattern-for-notebooks anchor and reworded the jobs-submit traps inline. Also normalized ../../skills/X/SKILL.md → ../X/SKILL.md everywhere — the install-flatten convention puts experimental/ and skills/ in the same flat directory at install time.
• Bumped mlflow==2.22.0 → mlflow==3.1.0.

references/custom-pyfunc.md

• mlflow==3.1.0; same install-flatten path normalization.

references/genai-agents.md

• mlflow==3.1.0.
• Replaced the stale training-and-serving.md pointer with databricks-model-serving.
• sys.argv[...] → dbutils.widgets for the deploy_agent notebook params.
• Default deploy tag is now ai_generated_source=databricks-agent-skills.

skills/databricks-model-serving/SKILL.md

• Split the shared client into registry = MlflowClient(...) and deploy = get_deploy_client(...). The alias call and the endpoint update target two different SDK surfaces — reusing one variable name was misleading.

Description length: left as-is per discussion — the long discriminator helps Claude pick the right skill when there are several adjacent ML skills.

[QuentinAmbard]

@simonfaltum @dustin-anchorage when you get a chance, could you do a final pass? Conflict with upstream/main is resolved (merge brought in #84's pre-split refs which I removed since the dev-side content lives in the new ml-training skill), and the two skill frontmatter descriptions were also tightened to cover OpenAPI schemas, off-platform streaming, serving logs/metrics, and the jobs-submit serverless pattern — body content already covered these but the description didn't, so an LLM orchestrator would have missed those user phrasings. Tip is e9692af. Thanks!

[QuentinAmbard]

@simonfaltum @dustin-anchorage when you have a minute, could you do a final pass?

Since the last round:

• Resolved the conflict with upstream/main (the merge brought in PR skills(model-serving): merge dev-side training/agent flows from a-d-k experimental #84's pre-split refs, which I removed since the dev-side content now lives in databricks-ml-training).
• Densified both frontmatter descriptions per Simon's earlier comment (model-serving 895→717 chars, ml-training 949→640). The 'Use when / NOT for' triage stays since it's the routing input, but the framing around it is tighter — CRUD verbs collapsed, redundant qualifiers dropped.
• ml-training description now surfaces task + framework keywords (classification/regression, XGBoost, scikit-learn, LightGBM, PyTorch) so phrasings like 'train an XGBoost classifier' hit the trigger.
• Audit caught a stale 'What's Next' refs table left by the merge — dropped it.

Tip is a8ebfbb. Thanks!

[dustinvannoy-db]

One suggestion I'd like you to accept to fix a statement that contradicts docs. Then you can merge.

[dustinvannoy-db]

Suggested change

	`mlflow.autolog()` captures params, metrics, code, and the model artifact for every run; `registered_model_name=...` auto-registers the best run to UC (auto-incremented version). Wrap training with **Optuna** so each trial is a child run and the best one is what gets registered.
	`mlflow.autolog()` captures params, metrics, code, and the model artifact for every run. Wrap training with **Optuna** so each trial is a nested run. **Don't** set `registered_model_name=…` on autolog — it registers a new UC version on *every* trial; instead retrain once on `study.best_params` and register only that winning model (below).