diff --git a/docs/ai-mcp-workflow.md b/docs/ai-mcp-workflow.md
index c147e173..7aa73ef7 100644
--- a/docs/ai-mcp-workflow.md
+++ b/docs/ai-mcp-workflow.md
@@ -57,6 +57,26 @@ dataset preparation, rule labeling, measured replay, feature extraction,
baseline ladder, and holdout error analysis are all script-backed and
reproducible.
+## Concrete Project Effect
+
+AI agents were used as orchestration and review assistants around deterministic
+scripts, not as labelers or runtime optimizers. The measurable outputs of that
+workflow are:
+
+- a 20 235-record ClickHouse query corpus assembled from public and project
+ sources;
+- 9 837 successfully replayed records enriched with measured ClickHouse
+ metrics;
+- a complete labeling pipeline with `rule_only`, `measured_only`, and `both`
+ provenance fields;
+- a five-step baseline ladder plus holdout error analysis for the risk model;
+- a public Streamable HTTP MCP demo endpoint that exposes the same analyzer
+ tools used locally.
+
+The important boundary is unchanged: AI helped build, inspect, and document the
+system; ClickAdvisor's product output still comes from deterministic code and
+script-generated metrics.
+
## Safety Rule
If a recommendation is not backed by a ClickAdvisor finding, measured workload
diff --git a/docs/evaluation.md b/docs/evaluation.md
index 30f98c16..43374258 100644
--- a/docs/evaluation.md
+++ b/docs/evaluation.md
@@ -52,12 +52,26 @@ Artifacts:
- `eval/results/classifier_ablation_current/metrics.csv`
- `docs/experiments/classifier_ablation.md`
+Methodology note: this experiment is not the same task as the risk-label
+baseline ladder in section 4. Classifier ablation is a small synthetic
+multi-label experiment over `expected_rules_to_fire` (`144` train / `36` test).
+The risk-label ladder is a 20 235-record triage experiment over
+`low` / `medium` / `high` labels created from rule signals plus measured replay
+metrics. The Random Forest train/test gap here is a useful warning about
+overfitting and rare labels in the small synthetic benchmark; it does not
+contradict the risk-label holdout score because the labels, split, dataset size,
+and target definition are different.
+
## 3. Retrieval Ablation
Retrieval quality is measured with `MRR@3` over explicit query-to-document gold
references. A result is relevant only when its URL/path or text matches the
gold reference for one of the expected rules.
+The current metric is stricter than early exploratory retrieval runs: generic
+near-topic ClickHouse chunks no longer count as relevant. Older MRR values
+around `0.61` are therefore not directly comparable with this repaired run.
+
Current command:
```bash
@@ -98,6 +112,23 @@ Current dataset:
| Feature rows | 20 235 |
| Numeric features | 115 |
+Dataset composition and limitations:
+
+| Source | Records | Note |
+|---|---:|---|
+| `clickhouse_functional_tests` / `clickhouse_stateless` | 16 845 | Diverse ClickHouse SQL corpus from upstream functional tests, not production user workload |
+| `clickhouse_performance_tests` | 1 825 | Upstream performance-oriented SQL/XML workload material |
+| `expert_synthetic_antipatterns` | 1 145 | Project-curated anti-pattern and edge-case examples |
+| Other benchmarks and project cases | 420 | ClickBench, TPCH/TPCDS, JOB, rule benchmark, bug reproducers |
+
+The corpus should be described as a diverse ClickHouse query corpus with
+rule- and replay-derived weak labels, not as DBA-adjudicated ground truth.
+The label method is `benchmark_expected_rules_plus_feature_weak_labels_v1`:
+static rule findings are reconciled with percentile-based measured metrics from
+local ClickHouse replay where execution succeeded. This is suitable for
+triage-model research and prioritization experiments, while production findings
+remain deterministic rule-engine output.
+
Current baseline ladder:
| Model | CV macro-F1 | Test macro-F1 | Holdout macro-F1 |
@@ -122,11 +153,36 @@ The `measured_only` drop is the key sanity check: it shows where ML has less
rule-derived signal to inherit. The `both` result shows the strongest core,
where static rules and measured replay agree.
+Error analysis for the high class:
+
+- High false negatives: `27`; `26` of them are `measured_only`, so most misses
+ come from replay-heavy cases with weak or absent rule signal.
+- High false positives: `32`; `17` are `rule_only` and `13` are
+ `measured_only`, showing both over-prioritized rule patterns and noisy metric
+ thresholds.
+- `both` is the most stable slice: `132` records, `0.975` macro-F1, `1.000`
+ high recall, and only `2` total errors.
+
+Learning-curve diagnostic for the Random Forest risk model:
+
+| Train fraction | Train records | Holdout macro-F1 | Holdout high recall |
+|---:|---:|---:|---:|
+| 0.10 | 1 415 | 0.893 | 0.675 |
+| 0.25 | 3 539 | 0.916 | 0.796 |
+| 0.50 | 7 078 | 0.934 | 0.850 |
+| 0.75 | 10 616 | 0.944 | 0.871 |
+| 1.00 | 14 155 | 0.949 | 0.887 |
+
+The fixed holdout score rises with more training data and then stabilizes,
+which is a stronger sign than a single holdout number. Train macro-F1 remains
+higher, so the model is not claimed as an independent replacement for rules.
+
Artifacts:
- `data/ml/expert_dataset/eda/ds_report.md`
- `docs/experiments/risk_labeling_ds_summary.md`
- `eval/results/risk_baseline_ladder_current/metrics.json`
+- `eval/results/risk_learning_curve_current/summary.md`
- `data/ml/expert_dataset/eda/risk_error_analysis/error_analysis.md`
## 5. Workload Analyzer Prototype
diff --git a/docs/experiments/classifier_ablation.md b/docs/experiments/classifier_ablation.md
index a87d1349..4fabe79c 100644
--- a/docs/experiments/classifier_ablation.md
+++ b/docs/experiments/classifier_ablation.md
@@ -32,3 +32,22 @@ is small and several labels have only one or two positive examples in test.
Random Forest is the strongest baseline in this run, but the train/test gap
shows why these numbers should be reported as synthetic classifier ablation
metrics, not as production generalization claims.
+
+The gap is expected for this specific setup: the split has only `36` test cases,
+and rare rule labels may have one or two positives. Tree models can memorize
+the training side of this compact benchmark (`train macro-F1` about `0.97`)
+while test macro-F1 stays around `0.67`-`0.69`.
+
+## Relationship to Risk-Label Baselines
+
+This experiment must not be compared directly with
+`risk_baseline_ladder_current`.
+
+| Experiment | Dataset | Target | Split | What it answers |
+|---|---:|---|---|---|
+| Classifier ablation | 180 synthetic benchmark cases | multi-label `expected_rules_to_fire` | 144 train / 36 test | Can ML features predict expected rule IDs on a compact rule benchmark? |
+| Risk baseline ladder | 20 235 query records | `low` / `medium` / `high` risk | group-aware train/test/holdout | Can SQL/text/rule/measured features triage query risk? |
+
+The classifier ablation gap is a warning about rare-label generalization in the
+small rule benchmark. The risk-label holdout score is a separate triage result
+with different labels, features, split semantics, and dataset size.
diff --git a/docs/experiments/retrieval_ablation.md b/docs/experiments/retrieval_ablation.md
index 1094ee68..02d5b6b6 100644
--- a/docs/experiments/retrieval_ablation.md
+++ b/docs/experiments/retrieval_ablation.md
@@ -30,6 +30,10 @@ The repaired metric no longer counts an arbitrary high-scoring ClickHouse chunk
as relevant. A retrieved chunk must match an explicit gold reference for one of
the expected rules.
+This explains the apparent drop from earlier exploratory `MRR@3` values around
+`0.61`: those runs used a looser relevance check and are not directly
+comparable with the repaired metric shown here.
+
MiniLM-L6 is strongest on this English-heavy KB sample. The default can still
remain multilingual E5 when Russian queries and multilingual KB growth matter
more than this small English ablation.
diff --git a/docs/experiments/risk_labeling_ds_summary.md b/docs/experiments/risk_labeling_ds_summary.md
index bc14439a..80b944a3 100644
--- a/docs/experiments/risk_labeling_ds_summary.md
+++ b/docs/experiments/risk_labeling_ds_summary.md
@@ -23,6 +23,25 @@ Artifacts:
- `data/ml/expert_dataset/splits/risk_split_v1.json`
- `data/ml/expert_dataset/eda/ds_report.md`
+## Dataset Composition and Limitations
+
+This is a diverse ClickHouse query corpus with replay- and rule-derived weak
+risk labels. It is not a DBA-adjudicated production incident dataset.
+
+| Source | Records | Why it is useful | Limitation |
+|---|---:|---|---|
+| `clickhouse_functional_tests` / `clickhouse_stateless` | 16 845 | Broad SQL grammar, functions, joins, subqueries, edge cases | Functional tests are not real user workloads |
+| `clickhouse_performance_tests` | 1 825 | Performance-oriented upstream workload material | Still benchmark/test data, not customer query logs |
+| `expert_synthetic_antipatterns` | 1 145 | Explicit anti-patterns and product scenarios | Synthetic by design |
+| Other benchmark/project sources | 420 | ClickBench, TPCH/TPCDS, JOB, rule benchmark, bug reproducers | Smaller slices with different semantics |
+
+All 20 235 records use
+`label_method = benchmark_expected_rules_plus_feature_weak_labels_v1`. The
+labels combine static rule findings with percentile-based measured metrics from
+local ClickHouse replay where execution succeeded. This is a valid DS setup for
+triage and prioritization research, but the labels should be treated as weak
+supervision, not as absolute ground truth.
+
## Label Sources
`label_source` is the most important methodological fact:
@@ -65,10 +84,35 @@ The `measured_only` drop confirms that the headline score is partly driven by
rule-derived features. The `both` result confirms that the model is strongest
where both sources agree.
+High-class errors are concentrated in the weaker label-source slices:
+
+- High false negatives: `27`; `26` are `measured_only`.
+- High false positives: `32`; `17` are `rule_only` and `13` are
+ `measured_only`.
+- `both` has only `2` errors over `132` holdout records and `1.000` high recall.
+
+## Learning Curve
+
+The Random Forest risk model was retrained on increasing fractions of the
+group-aware train split and evaluated on the same fixed holdout split:
+
+| Train fraction | Train records | Train macro-F1 | Holdout macro-F1 | Holdout high recall |
+|---:|---:|---:|---:|---:|
+| 0.10 | 1 415 | 0.978 | 0.893 | 0.675 |
+| 0.25 | 3 539 | 0.963 | 0.916 | 0.796 |
+| 0.50 | 7 078 | 0.970 | 0.934 | 0.850 |
+| 0.75 | 10 616 | 0.972 | 0.944 | 0.871 |
+| 1.00 | 14 155 | 0.973 | 0.949 | 0.887 |
+
+The holdout curve rises and stabilizes instead of collapsing on the full train
+split. This supports the risk-label result as a stable triage experiment while
+still showing that train performance is higher than holdout performance.
+
Product conclusion: ML adds value as triage, prioritization, confidence
grouping, and review-queue ordering. Runtime recommendations remain rule-first.
Full artifacts:
- `eval/results/risk_baseline_ladder_current/summary.md`
+- `eval/results/risk_learning_curve_current/summary.md`
- `data/ml/expert_dataset/eda/risk_error_analysis/error_analysis.md`
diff --git a/eval/results/risk_learning_curve_current/learning_curve.csv b/eval/results/risk_learning_curve_current/learning_curve.csv
new file mode 100644
index 00000000..f1b3f9fd
--- /dev/null
+++ b/eval/results/risk_learning_curve_current/learning_curve.csv
@@ -0,0 +1,6 @@
+fraction,train_records,train_label_counts,train_macro_f1,train_mcc,train_high_recall,holdout_macro_f1,holdout_mcc,holdout_high_recall,holdout_high_precision
+0.1,1415,"{'medium': 996, 'low': 299, 'high': 120}",0.9777124859551805,0.9753319573238626,0.975,0.8932861794550314,0.8944344270005464,0.675,0.81
+0.25,3539,"{'medium': 2491, 'low': 748, 'high': 300}",0.962934106881713,0.9600182272047234,0.9366666666666666,0.9158942499671004,0.9144457309051403,0.7958333333333333,0.7925311203319502
+0.5,7078,"{'medium': 4982, 'low': 1497, 'high': 599}",0.9704071521957314,0.9669318505264164,0.9499165275459098,0.9343644420417401,0.9301935928346581,0.85,0.8360655737704918
+0.75,10616,"{'medium': 7472, 'low': 2246, 'high': 898}",0.9721708654897511,0.9688436088650182,0.9643652561247216,0.944120222307378,0.9390426931758646,0.8708333333333333,0.8636363636363636
+1.0,14155,"{'medium': 9963, 'low': 2994, 'high': 1198}",0.9727389104646452,0.9698296739116526,0.9616026711185309,0.9490414301820191,0.945035207694112,0.8875,0.8693877551020408
diff --git a/eval/results/risk_learning_curve_current/learning_curve.json b/eval/results/risk_learning_curve_current/learning_curve.json
new file mode 100644
index 00000000..e53fba61
--- /dev/null
+++ b/eval/results/risk_learning_curve_current/learning_curve.json
@@ -0,0 +1,93 @@
+{
+ "metadata": {
+ "created_at": "2026-07-01T11:35:56.376283+00:00",
+ "script": "scripts/lab/generate_risk_learning_curve.py",
+ "features_path": "data/ml/expert_dataset/features/features.jsonl",
+ "split_path": "data/ml/expert_dataset/splits/risk_split_v1.json",
+ "random_state": 42,
+ "model": "random_forest_all_features",
+ "note": "Learning curve uses the risk-label train split and evaluates every fraction on the fixed holdout split. It is a stability diagnostic, not a separate model claim."
+ },
+ "rows": [
+ {
+ "fraction": 0.1,
+ "train_records": 1415,
+ "train_label_counts": {
+ "medium": 996,
+ "low": 299,
+ "high": 120
+ },
+ "train_macro_f1": 0.9777124859551805,
+ "train_mcc": 0.9753319573238626,
+ "train_high_recall": 0.975,
+ "holdout_macro_f1": 0.8932861794550314,
+ "holdout_mcc": 0.8944344270005464,
+ "holdout_high_recall": 0.675,
+ "holdout_high_precision": 0.81
+ },
+ {
+ "fraction": 0.25,
+ "train_records": 3539,
+ "train_label_counts": {
+ "medium": 2491,
+ "low": 748,
+ "high": 300
+ },
+ "train_macro_f1": 0.962934106881713,
+ "train_mcc": 0.9600182272047234,
+ "train_high_recall": 0.9366666666666666,
+ "holdout_macro_f1": 0.9158942499671004,
+ "holdout_mcc": 0.9144457309051403,
+ "holdout_high_recall": 0.7958333333333333,
+ "holdout_high_precision": 0.7925311203319502
+ },
+ {
+ "fraction": 0.5,
+ "train_records": 7078,
+ "train_label_counts": {
+ "medium": 4982,
+ "low": 1497,
+ "high": 599
+ },
+ "train_macro_f1": 0.9704071521957314,
+ "train_mcc": 0.9669318505264164,
+ "train_high_recall": 0.9499165275459098,
+ "holdout_macro_f1": 0.9343644420417401,
+ "holdout_mcc": 0.9301935928346581,
+ "holdout_high_recall": 0.85,
+ "holdout_high_precision": 0.8360655737704918
+ },
+ {
+ "fraction": 0.75,
+ "train_records": 10616,
+ "train_label_counts": {
+ "medium": 7472,
+ "low": 2246,
+ "high": 898
+ },
+ "train_macro_f1": 0.9721708654897511,
+ "train_mcc": 0.9688436088650182,
+ "train_high_recall": 0.9643652561247216,
+ "holdout_macro_f1": 0.944120222307378,
+ "holdout_mcc": 0.9390426931758646,
+ "holdout_high_recall": 0.8708333333333333,
+ "holdout_high_precision": 0.8636363636363636
+ },
+ {
+ "fraction": 1.0,
+ "train_records": 14155,
+ "train_label_counts": {
+ "medium": 9963,
+ "low": 2994,
+ "high": 1198
+ },
+ "train_macro_f1": 0.9727389104646452,
+ "train_mcc": 0.9698296739116526,
+ "train_high_recall": 0.9616026711185309,
+ "holdout_macro_f1": 0.9490414301820191,
+ "holdout_mcc": 0.945035207694112,
+ "holdout_high_recall": 0.8875,
+ "holdout_high_precision": 0.8693877551020408
+ }
+ ]
+}
diff --git a/eval/results/risk_learning_curve_current/learning_curve.svg b/eval/results/risk_learning_curve_current/learning_curve.svg
new file mode 100644
index 00000000..b9ba5559
--- /dev/null
+++ b/eval/results/risk_learning_curve_current/learning_curve.svg
@@ -0,0 +1,41 @@
+
\ No newline at end of file
diff --git a/eval/results/risk_learning_curve_current/summary.md b/eval/results/risk_learning_curve_current/summary.md
new file mode 100644
index 00000000..456a2969
--- /dev/null
+++ b/eval/results/risk_learning_curve_current/summary.md
@@ -0,0 +1,17 @@
+# Risk Learning Curve
+
+Learning curve uses the risk-label train split and evaluates every fraction on the fixed holdout split. It is a stability diagnostic, not a separate model claim.
+
+| Train fraction | Train records | Train macro-F1 | Holdout macro-F1 | Holdout MCC | Holdout high recall |
+|---:|---:|---:|---:|---:|---:|
+| 0.10 | 1415 | 0.978 | 0.893 | 0.894 | 0.675 |
+| 0.25 | 3539 | 0.963 | 0.916 | 0.914 | 0.796 |
+| 0.50 | 7078 | 0.970 | 0.934 | 0.930 | 0.850 |
+| 0.75 | 10616 | 0.972 | 0.944 | 0.939 | 0.871 |
+| 1.00 | 14155 | 0.973 | 0.949 | 0.945 | 0.887 |
+
+## Interpretation
+
+Holdout macro-F1 rises early and then stabilizes, while train macro-F1 remains higher. This is expected for a Random Forest over sparse SQL/text/rule features. The fixed holdout curve is the important signal: it does not collapse when the model sees the full train split, so the 20k risk-label result is more stable than the separate small synthetic classifier ablation.
+
+
diff --git a/scripts/lab/generate_risk_learning_curve.py b/scripts/lab/generate_risk_learning_curve.py
new file mode 100644
index 00000000..034c0e6b
--- /dev/null
+++ b/scripts/lab/generate_risk_learning_curve.py
@@ -0,0 +1,270 @@
+from __future__ import annotations
+
+import argparse
+import csv
+import html
+import json
+import random
+from collections import defaultdict
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+from sklearn.ensemble import RandomForestClassifier
+
+from scripts.lab.run_risk_baseline_ladder import (
+ DEFAULT_FEATURES_PATH,
+ DEFAULT_SPLIT_PATH,
+ LABELS,
+ all_feature_matrices,
+ labels_for,
+ load_rows,
+ metrics_for,
+ rows_for_ids,
+)
+
+DEFAULT_OUTPUT_DIR = Path("eval/results/risk_learning_curve_current")
+DEFAULT_FRACTIONS = [0.10, 0.25, 0.50, 0.75, 1.00]
+
+
+def main() -> None:
+ args = parse_args()
+ rows = load_rows(args.features)
+ split = json.loads(args.split.read_text(encoding="utf-8"))
+ rows_by_id = {str(row["id"]): row for row in rows}
+ train_rows = rows_for_ids(rows_by_id, split["train_ids"])
+ holdout_rows = rows_for_ids(rows_by_id, split["holdout_ids"])
+
+ learning_rows = build_learning_curve(
+ train_rows,
+ holdout_rows,
+ fractions=args.fractions,
+ random_state=args.random_state,
+ )
+ write_outputs(learning_rows, args.output_dir, args)
+ print(f"Saved risk learning curve to {args.output_dir}")
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description="Generate risk-label learning curve artifacts.")
+ parser.add_argument("--features", type=Path, default=DEFAULT_FEATURES_PATH)
+ parser.add_argument("--split", type=Path, default=DEFAULT_SPLIT_PATH)
+ parser.add_argument("--output-dir", type=Path, default=DEFAULT_OUTPUT_DIR)
+ parser.add_argument("--random-state", type=int, default=42)
+ parser.add_argument("--fractions", type=float, nargs="*", default=DEFAULT_FRACTIONS)
+ return parser.parse_args()
+
+
+def build_learning_curve(
+ train_rows: list[dict[str, Any]],
+ holdout_rows: list[dict[str, Any]],
+ *,
+ fractions: list[float],
+ random_state: int,
+) -> list[dict[str, Any]]:
+ results: list[dict[str, Any]] = []
+ for fraction in fractions:
+ sampled_train = stratified_sample(train_rows, fraction=fraction, random_state=random_state)
+ x_train, x_holdout = all_feature_matrices(sampled_train, holdout_rows)
+ estimator = RandomForestClassifier(
+ n_estimators=220,
+ min_samples_leaf=2,
+ class_weight="balanced",
+ n_jobs=-1,
+ random_state=random_state,
+ )
+ y_train = labels_for(sampled_train)
+ y_holdout = labels_for(holdout_rows)
+ estimator.fit(x_train, y_train)
+ train_pred = np.asarray(estimator.predict(x_train), dtype=object)
+ holdout_pred = np.asarray(estimator.predict(x_holdout), dtype=object)
+ train_metrics = metrics_for(y_train, train_pred, split_name="train_subset")
+ holdout_metrics = metrics_for(y_holdout, holdout_pred, split_name="holdout")
+ results.append(
+ {
+ "fraction": float(fraction),
+ "train_records": len(sampled_train),
+ "train_label_counts": train_metrics["true_label_counts"],
+ "train_macro_f1": train_metrics["macro_f1"],
+ "train_mcc": train_metrics["mcc"],
+ "train_high_recall": train_metrics["per_label"]["high"]["recall"],
+ "holdout_macro_f1": holdout_metrics["macro_f1"],
+ "holdout_mcc": holdout_metrics["mcc"],
+ "holdout_high_recall": holdout_metrics["per_label"]["high"]["recall"],
+ "holdout_high_precision": holdout_metrics["per_label"]["high"]["precision"],
+ }
+ )
+ return results
+
+
+def stratified_sample(
+ rows: list[dict[str, Any]],
+ *,
+ fraction: float,
+ random_state: int,
+) -> list[dict[str, Any]]:
+ if not 0 < fraction <= 1:
+ raise ValueError(f"fraction must be in (0, 1], got {fraction}")
+ by_label: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ for row in rows:
+ by_label[str(row["target"])].append(row)
+
+ rng = random.Random(random_state)
+ sampled: list[dict[str, Any]] = []
+ for label in LABELS:
+ label_rows = list(by_label[label])
+ rng.shuffle(label_rows)
+ count = max(1, round(len(label_rows) * fraction))
+ sampled.extend(label_rows[:count])
+ sampled.sort(key=lambda row: str(row["id"]))
+ return sampled
+
+
+def write_outputs(rows: list[dict[str, Any]], output_dir: Path, args: argparse.Namespace) -> None:
+ output_dir.mkdir(parents=True, exist_ok=True)
+ metadata = {
+ "created_at": datetime.now(UTC).isoformat(),
+ "script": "scripts/lab/generate_risk_learning_curve.py",
+ "features_path": str(args.features),
+ "split_path": str(args.split),
+ "random_state": args.random_state,
+ "model": "random_forest_all_features",
+ "note": (
+ "Learning curve uses the risk-label train split and evaluates every fraction on "
+ "the fixed holdout split. It is a stability diagnostic, not a separate model claim."
+ ),
+ }
+ (output_dir / "learning_curve.json").write_text(
+ json.dumps({"metadata": metadata, "rows": rows}, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ with (output_dir / "learning_curve.csv").open("w", encoding="utf-8", newline="") as handle:
+ writer = csv.DictWriter(handle, fieldnames=list(rows[0]), lineterminator="\n")
+ writer.writeheader()
+ writer.writerows(rows)
+ (output_dir / "learning_curve.svg").write_text(render_svg(rows), encoding="utf-8")
+ (output_dir / "summary.md").write_text(render_markdown(rows, metadata), encoding="utf-8")
+
+
+def render_markdown(rows: list[dict[str, Any]], metadata: dict[str, Any]) -> str:
+ lines = [
+ "# Risk Learning Curve",
+ "",
+ metadata["note"],
+ "",
+ "| Train fraction | Train records | Train macro-F1 | Holdout macro-F1 | "
+ "Holdout MCC | Holdout high recall |",
+ "|---:|---:|---:|---:|---:|---:|",
+ ]
+ for row in rows:
+ lines.append(
+ f"| {row['fraction']:.2f} | {row['train_records']} | "
+ f"{row['train_macro_f1']:.3f} | {row['holdout_macro_f1']:.3f} | "
+ f"{row['holdout_mcc']:.3f} | {row['holdout_high_recall']:.3f} |"
+ )
+ lines.extend(
+ [
+ "",
+ "## Interpretation",
+ "",
+ "Holdout macro-F1 rises early and then stabilizes, while train macro-F1 remains "
+ "higher. This is expected for a Random Forest over sparse SQL/text/rule features. "
+ "The fixed holdout curve is the important signal: it does not collapse when the "
+ "model sees the full train split, so the 20k risk-label result is more stable than "
+ "the separate small synthetic classifier ablation.",
+ "",
+ "",
+ "",
+ ]
+ )
+ return "\n".join(lines)
+
+
+def render_svg(rows: list[dict[str, Any]]) -> str:
+ width = 780
+ height = 420
+ margin_left = 72
+ margin_right = 34
+ margin_top = 42
+ margin_bottom = 58
+ plot_width = width - margin_left - margin_right
+ plot_height = height - margin_top - margin_bottom
+ min_y = 0.75
+ max_y = 1.0
+
+ def x_pos(fraction: float) -> float:
+ return margin_left + (fraction - 0.1) / 0.9 * plot_width
+
+ def y_pos(value: float) -> float:
+ return margin_top + (max_y - value) / (max_y - min_y) * plot_height
+
+ def points(metric_name: str) -> str:
+ return " ".join(
+ f"{x_pos(float(row['fraction'])):.1f},{y_pos(float(row[metric_name])):.1f}"
+ for row in rows
+ )
+
+ grid_lines = []
+ for value in [0.75, 0.80, 0.85, 0.90, 0.95, 1.00]:
+ y = y_pos(value)
+ grid_lines.append(
+ f''
+ )
+ grid_lines.append(
+ f'{value:.2f}'
+ )
+
+ x_labels = []
+ for row in rows:
+ x = x_pos(float(row["fraction"]))
+ x_labels.append(
+ f'{float(row["fraction"]):.2f}'
+ )
+
+ circles = []
+ for metric_name, color in [
+ ("train_macro_f1", "#0f766e"),
+ ("holdout_macro_f1", "#2563eb"),
+ ]:
+ for row in rows:
+ circles.append(
+ f''
+ )
+
+ title = html.escape("Risk-label learning curve: Random Forest")
+ return "\n".join(
+ [
+ '",
+ ]
+ )
+
+
+if __name__ == "__main__":
+ main()