Document DS limitations and learning curve

docs/ai-mcp-workflow.md

@@ -57,6 +57,26 @@ dataset preparation, rule labeling, measured replay, feature extraction,
 baseline ladder, and holdout error analysis are all script-backed and
 reproducible.
 
+## Concrete Project Effect
+
+AI agents were used as orchestration and review assistants around deterministic
+scripts, not as labelers or runtime optimizers. The measurable outputs of that
+workflow are:
+
+- a 20 235-record ClickHouse query corpus assembled from public and project
+  sources;
+- 9 837 successfully replayed records enriched with measured ClickHouse
+  metrics;
+- a complete labeling pipeline with `rule_only`, `measured_only`, and `both`
+  provenance fields;
+- a five-step baseline ladder plus holdout error analysis for the risk model;
+- a public Streamable HTTP MCP demo endpoint that exposes the same analyzer
+  tools used locally.
+
+The important boundary is unchanged: AI helped build, inspect, and document the
+system; ClickAdvisor's product output still comes from deterministic code and
+script-generated metrics.
+
 ## Safety Rule
 
 If a recommendation is not backed by a ClickAdvisor finding, measured workload

docs/evaluation.md

@@ -52,12 +52,26 @@ Artifacts:
 - `eval/results/classifier_ablation_current/metrics.csv`
 - `docs/experiments/classifier_ablation.md`
 
+Methodology note: this experiment is not the same task as the risk-label
+baseline ladder in section 4. Classifier ablation is a small synthetic
+multi-label experiment over `expected_rules_to_fire` (`144` train / `36` test).
+The risk-label ladder is a 20 235-record triage experiment over
+`low` / `medium` / `high` labels created from rule signals plus measured replay
+metrics. The Random Forest train/test gap here is a useful warning about
+overfitting and rare labels in the small synthetic benchmark; it does not
+contradict the risk-label holdout score because the labels, split, dataset size,
+and target definition are different.
+
 ## 3. Retrieval Ablation
 
 Retrieval quality is measured with `MRR@3` over explicit query-to-document gold
 references. A result is relevant only when its URL/path or text matches the
 gold reference for one of the expected rules.
 
+The current metric is stricter than early exploratory retrieval runs: generic
+near-topic ClickHouse chunks no longer count as relevant. Older MRR values
+around `0.61` are therefore not directly comparable with this repaired run.
+
 Current command:
 
 ```bash
@@ -98,6 +112,23 @@ Current dataset:
 | Feature rows | 20 235 |
 | Numeric features | 115 |
 
+Dataset composition and limitations:
+
+| Source | Records | Note |
+|---|---:|---|
+| `clickhouse_functional_tests` / `clickhouse_stateless` | 16 845 | Diverse ClickHouse SQL corpus from upstream functional tests, not production user workload |
+| `clickhouse_performance_tests` | 1 825 | Upstream performance-oriented SQL/XML workload material |
+| `expert_synthetic_antipatterns` | 1 145 | Project-curated anti-pattern and edge-case examples |
+| Other benchmarks and project cases | 420 | ClickBench, TPCH/TPCDS, JOB, rule benchmark, bug reproducers |
+
+The corpus should be described as a diverse ClickHouse query corpus with
+rule- and replay-derived weak labels, not as DBA-adjudicated ground truth.
+The label method is `benchmark_expected_rules_plus_feature_weak_labels_v1`:
+static rule findings are reconciled with percentile-based measured metrics from
+local ClickHouse replay where execution succeeded. This is suitable for
+triage-model research and prioritization experiments, while production findings
+remain deterministic rule-engine output.
+
 Current baseline ladder:
 
 | Model | CV macro-F1 | Test macro-F1 | Holdout macro-F1 |
@@ -122,11 +153,36 @@ The `measured_only` drop is the key sanity check: it shows where ML has less
 rule-derived signal to inherit. The `both` result shows the strongest core,
 where static rules and measured replay agree.
 
+Error analysis for the high class:
+
+- High false negatives: `27`; `26` of them are `measured_only`, so most misses
+  come from replay-heavy cases with weak or absent rule signal.
+- High false positives: `32`; `17` are `rule_only` and `13` are
+  `measured_only`, showing both over-prioritized rule patterns and noisy metric
+  thresholds.
+- `both` is the most stable slice: `132` records, `0.975` macro-F1, `1.000`
+  high recall, and only `2` total errors.
+
+Learning-curve diagnostic for the Random Forest risk model:
+
+| Train fraction | Train records | Holdout macro-F1 | Holdout high recall |
+|---:|---:|---:|---:|
+| 0.10 | 1 415 | 0.893 | 0.675 |
+| 0.25 | 3 539 | 0.916 | 0.796 |
+| 0.50 | 7 078 | 0.934 | 0.850 |
+| 0.75 | 10 616 | 0.944 | 0.871 |
+| 1.00 | 14 155 | 0.949 | 0.887 |
+
+The fixed holdout score rises with more training data and then stabilizes,
+which is a stronger sign than a single holdout number. Train macro-F1 remains
+higher, so the model is not claimed as an independent replacement for rules.
+
 Artifacts:
 
 - `data/ml/expert_dataset/eda/ds_report.md`
 - `docs/experiments/risk_labeling_ds_summary.md`
 - `eval/results/risk_baseline_ladder_current/metrics.json`
+- `eval/results/risk_learning_curve_current/summary.md`
 - `data/ml/expert_dataset/eda/risk_error_analysis/error_analysis.md`
 
 ## 5. Workload Analyzer Prototype

docs/experiments/classifier_ablation.md

@@ -32,3 +32,22 @@ is small and several labels have only one or two positive examples in test.
 Random Forest is the strongest baseline in this run, but the train/test gap
 shows why these numbers should be reported as synthetic classifier ablation
 metrics, not as production generalization claims.
+
+The gap is expected for this specific setup: the split has only `36` test cases,
+and rare rule labels may have one or two positives. Tree models can memorize
+the training side of this compact benchmark (`train macro-F1` about `0.97`)
+while test macro-F1 stays around `0.67`-`0.69`.
+
+## Relationship to Risk-Label Baselines
+
+This experiment must not be compared directly with
+`risk_baseline_ladder_current`.
+
+| Experiment | Dataset | Target | Split | What it answers |
+|---|---:|---|---|---|
+| Classifier ablation | 180 synthetic benchmark cases | multi-label `expected_rules_to_fire` | 144 train / 36 test | Can ML features predict expected rule IDs on a compact rule benchmark? |
+| Risk baseline ladder | 20 235 query records | `low` / `medium` / `high` risk | group-aware train/test/holdout | Can SQL/text/rule/measured features triage query risk? |
+
+The classifier ablation gap is a warning about rare-label generalization in the
+small rule benchmark. The risk-label holdout score is a separate triage result
+with different labels, features, split semantics, and dataset size.

docs/experiments/retrieval_ablation.md

@@ -30,6 +30,10 @@ The repaired metric no longer counts an arbitrary high-scoring ClickHouse chunk
 as relevant. A retrieved chunk must match an explicit gold reference for one of
 the expected rules.
 
+This explains the apparent drop from earlier exploratory `MRR@3` values around
+`0.61`: those runs used a looser relevance check and are not directly
+comparable with the repaired metric shown here.
+
 MiniLM-L6 is strongest on this English-heavy KB sample. The default can still
 remain multilingual E5 when Russian queries and multilingual KB growth matter
 more than this small English ablation.

docs/experiments/risk_labeling_ds_summary.md

@@ -23,6 +23,25 @@ Artifacts:
 - `data/ml/expert_dataset/splits/risk_split_v1.json`
 - `data/ml/expert_dataset/eda/ds_report.md`
 
+## Dataset Composition and Limitations
+
+This is a diverse ClickHouse query corpus with replay- and rule-derived weak
+risk labels. It is not a DBA-adjudicated production incident dataset.
+
+| Source | Records | Why it is useful | Limitation |
+|---|---:|---|---|
+| `clickhouse_functional_tests` / `clickhouse_stateless` | 16 845 | Broad SQL grammar, functions, joins, subqueries, edge cases | Functional tests are not real user workloads |
+| `clickhouse_performance_tests` | 1 825 | Performance-oriented upstream workload material | Still benchmark/test data, not customer query logs |
+| `expert_synthetic_antipatterns` | 1 145 | Explicit anti-patterns and product scenarios | Synthetic by design |
+| Other benchmark/project sources | 420 | ClickBench, TPCH/TPCDS, JOB, rule benchmark, bug reproducers | Smaller slices with different semantics |
+
+All 20 235 records use
+`label_method = benchmark_expected_rules_plus_feature_weak_labels_v1`. The
+labels combine static rule findings with percentile-based measured metrics from
+local ClickHouse replay where execution succeeded. This is a valid DS setup for
+triage and prioritization research, but the labels should be treated as weak
+supervision, not as absolute ground truth.
+
 ## Label Sources
 
 `label_source` is the most important methodological fact:
@@ -65,10 +84,35 @@ The `measured_only` drop confirms that the headline score is partly driven by
 rule-derived features. The `both` result confirms that the model is strongest
 where both sources agree.
 
+High-class errors are concentrated in the weaker label-source slices:
+
+- High false negatives: `27`; `26` are `measured_only`.
+- High false positives: `32`; `17` are `rule_only` and `13` are
+  `measured_only`.
+- `both` has only `2` errors over `132` holdout records and `1.000` high recall.
+
+## Learning Curve
+
+The Random Forest risk model was retrained on increasing fractions of the
+group-aware train split and evaluated on the same fixed holdout split:
+
+| Train fraction | Train records | Train macro-F1 | Holdout macro-F1 | Holdout high recall |
+|---:|---:|---:|---:|---:|
+| 0.10 | 1 415 | 0.978 | 0.893 | 0.675 |
+| 0.25 | 3 539 | 0.963 | 0.916 | 0.796 |
+| 0.50 | 7 078 | 0.970 | 0.934 | 0.850 |
+| 0.75 | 10 616 | 0.972 | 0.944 | 0.871 |
+| 1.00 | 14 155 | 0.973 | 0.949 | 0.887 |
+
+The holdout curve rises and stabilizes instead of collapsing on the full train
+split. This supports the risk-label result as a stable triage experiment while
+still showing that train performance is higher than holdout performance.
+
 Product conclusion: ML adds value as triage, prioritization, confidence
 grouping, and review-queue ordering. Runtime recommendations remain rule-first.
 
 Full artifacts:
 
 - `eval/results/risk_baseline_ladder_current/summary.md`
+- `eval/results/risk_learning_curve_current/summary.md`
 - `data/ml/expert_dataset/eda/risk_error_analysis/error_analysis.md`

eval/results/risk_learning_curve_current/learning_curve.csv

@@ -0,0 +1,6 @@
+fraction,train_records,train_label_counts,train_macro_f1,train_mcc,train_high_recall,holdout_macro_f1,holdout_mcc,holdout_high_recall,holdout_high_precision
+0.1,1415,"{'medium': 996, 'low': 299, 'high': 120}",0.9777124859551805,0.9753319573238626,0.975,0.8932861794550314,0.8944344270005464,0.675,0.81
+0.25,3539,"{'medium': 2491, 'low': 748, 'high': 300}",0.962934106881713,0.9600182272047234,0.9366666666666666,0.9158942499671004,0.9144457309051403,0.7958333333333333,0.7925311203319502
+0.5,7078,"{'medium': 4982, 'low': 1497, 'high': 599}",0.9704071521957314,0.9669318505264164,0.9499165275459098,0.9343644420417401,0.9301935928346581,0.85,0.8360655737704918
+0.75,10616,"{'medium': 7472, 'low': 2246, 'high': 898}",0.9721708654897511,0.9688436088650182,0.9643652561247216,0.944120222307378,0.9390426931758646,0.8708333333333333,0.8636363636363636
+1.0,14155,"{'medium': 9963, 'low': 2994, 'high': 1198}",0.9727389104646452,0.9698296739116526,0.9616026711185309,0.9490414301820191,0.945035207694112,0.8875,0.8693877551020408

eval/results/risk_learning_curve_current/learning_curve.json

@@ -0,0 +1,93 @@
+{
+  "metadata": {
+    "created_at": "2026-07-01T11:35:56.376283+00:00",
+    "script": "scripts/lab/generate_risk_learning_curve.py",
+    "features_path": "data/ml/expert_dataset/features/features.jsonl",
+    "split_path": "data/ml/expert_dataset/splits/risk_split_v1.json",
+    "random_state": 42,
+    "model": "random_forest_all_features",
+    "note": "Learning curve uses the risk-label train split and evaluates every fraction on the fixed holdout split. It is a stability diagnostic, not a separate model claim."
+  },
+  "rows": [
+    {
+      "fraction": 0.1,
+      "train_records": 1415,
+      "train_label_counts": {
+        "medium": 996,
+        "low": 299,
+        "high": 120
+      },
+      "train_macro_f1": 0.9777124859551805,
+      "train_mcc": 0.9753319573238626,
+      "train_high_recall": 0.975,
+      "holdout_macro_f1": 0.8932861794550314,
+      "holdout_mcc": 0.8944344270005464,
+      "holdout_high_recall": 0.675,
+      "holdout_high_precision": 0.81
+    },
+    {
+      "fraction": 0.25,
+      "train_records": 3539,
+      "train_label_counts": {
+        "medium": 2491,
+        "low": 748,
+        "high": 300
+      },
+      "train_macro_f1": 0.962934106881713,
+      "train_mcc": 0.9600182272047234,
+      "train_high_recall": 0.9366666666666666,
+      "holdout_macro_f1": 0.9158942499671004,
+      "holdout_mcc": 0.9144457309051403,
+      "holdout_high_recall": 0.7958333333333333,
+      "holdout_high_precision": 0.7925311203319502
+    },
+    {
+      "fraction": 0.5,
+      "train_records": 7078,
+      "train_label_counts": {
+        "medium": 4982,
+        "low": 1497,
+        "high": 599
+      },
+      "train_macro_f1": 0.9704071521957314,
+      "train_mcc": 0.9669318505264164,
+      "train_high_recall": 0.9499165275459098,
+      "holdout_macro_f1": 0.9343644420417401,
+      "holdout_mcc": 0.9301935928346581,
+      "holdout_high_recall": 0.85,
+      "holdout_high_precision": 0.8360655737704918
+    },
+    {
+      "fraction": 0.75,
+      "train_records": 10616,
+      "train_label_counts": {
+        "medium": 7472,
+        "low": 2246,
+        "high": 898
+      },
+      "train_macro_f1": 0.9721708654897511,
+      "train_mcc": 0.9688436088650182,
+      "train_high_recall": 0.9643652561247216,
+      "holdout_macro_f1": 0.944120222307378,
+      "holdout_mcc": 0.9390426931758646,
+      "holdout_high_recall": 0.8708333333333333,
+      "holdout_high_precision": 0.8636363636363636
+    },
+    {
+      "fraction": 1.0,
+      "train_records": 14155,
+      "train_label_counts": {
+        "medium": 9963,
+        "low": 2994,
+        "high": 1198
+      },
+      "train_macro_f1": 0.9727389104646452,
+      "train_mcc": 0.9698296739116526,
+      "train_high_recall": 0.9616026711185309,
+      "holdout_macro_f1": 0.9490414301820191,
+      "holdout_mcc": 0.945035207694112,
+      "holdout_high_recall": 0.8875,
+      "holdout_high_precision": 0.8693877551020408
+    }
+  ]
+}

eval/results/risk_learning_curve_current/summary.md

@@ -0,0 +1,17 @@
+# Risk Learning Curve
+
+Learning curve uses the risk-label train split and evaluates every fraction on the fixed holdout split. It is a stability diagnostic, not a separate model claim.
+
+| Train fraction | Train records | Train macro-F1 | Holdout macro-F1 | Holdout MCC | Holdout high recall |
+|---:|---:|---:|---:|---:|---:|
+| 0.10 | 1415 | 0.978 | 0.893 | 0.894 | 0.675 |
+| 0.25 | 3539 | 0.963 | 0.916 | 0.914 | 0.796 |
+| 0.50 | 7078 | 0.970 | 0.934 | 0.930 | 0.850 |
+| 0.75 | 10616 | 0.972 | 0.944 | 0.939 | 0.871 |
+| 1.00 | 14155 | 0.973 | 0.949 | 0.945 | 0.887 |
+
+## Interpretation
+
+Holdout macro-F1 rises early and then stabilizes, while train macro-F1 remains higher. This is expected for a Random Forest over sparse SQL/text/rule features. The fixed holdout curve is the important signal: it does not collapse when the model sees the full train split, so the 20k risk-label result is more stable than the separate small synthetic classifier ablation.
+
+![Risk learning curve](learning_curve.svg)