feat: expand rule catalog and synthetic benchmark

README.md

@@ -12,9 +12,10 @@ ClickAdvisor помогает DBA и разработчикам быстрее 
 он принимает SQL-файл, парсит запрос через `sqlglot`, применяет набор
 детерминированных правил и возвращает отчёт в консоли, JSON или Markdown.
 
-Главный принцип проекта: **LLM и retrieval могут помогать, но ядро анализа —
-это rule engine с явно описанными условиями применимости**. Утилита работает
-локально и не отправляет SQL во внешние сервисы.
+Главный принцип проекта: **rules + ML + retrieval**, где доверенное ядро
+анализа остаётся rule engine с явно описанными условиями применимости. ML
+используется как отдельная evaluation surface, retrieval добавляет локальные
+ссылки на документацию, а утилита не отправляет SQL во внешние сервисы.
 
 [Сайт проекта](https://clickadvisor.lovable.app)
 
@@ -36,18 +37,18 @@ ClickHouse, могут опираться на устаревшие рекоме
 
 - CLI-команда `chadvisor analyze`;
 - MCP-сервер `chadvisor mcp-server`;
-- 18 Tier 1 rewrite-правил `R-001`…`R-018`;
-- 3 детектора антипаттернов `D-003`, `D-004`, `D-007`;
+- 62 rewrite/advisory-правил `R-*`;
+- 15 детекторов антипаттернов `D-*`;
 - version detection через ClickHouse HTTP API;
 - console / JSON / Markdown отчёты;
 - режим объяснений `--mode explain`;
 - optional retrieval advisory через embedded Qdrant KB;
 - optional `EXPLAIN ESTIMATE` сравнение через ClickHouse HTTP API;
 - synthetic benchmark для проверки срабатывания правил.
 
-В каталоге `/docs/rules/cards/` описаны 54 карточки правил. Реализованная
-часть в коде — 21 правило/детектор, перечисленные ниже в разделе
-«Правила и покрытие».
+В каталоге `/docs/rules/cards/` описаны 119 валидируемых карточек правил:
+77 из них уже имеют реализацию и тесты, 42 остаются backlog-карточками для
+следующих итераций.
 
 ## Быстрый старт
 
@@ -253,29 +254,23 @@ poetry run chadvisor mcp-server
 
 ## Правила и покрытие
 
-| Rule ID | Что ищет | Tier |
-|---|---|---|
-| `R-001` | `COUNT(DISTINCT x)` → `uniqExact(x)` | `1A` |
-| `R-002` | `COUNT(DISTINCT x)` → advisory `uniq(x)` | `1B` |
-| `R-003` | `quantileExact(...)` → advisory `quantileTDigest(...)` | `1B` |
-| `R-004` | `COUNT(*) FROM (SELECT DISTINCT ...)` → специализированный агрегат | `1A` |
-| `R-005` | `toDate(col) = ...` → range predicate | `1A` |
-| `R-006` | `toYYYYMM(...)` / `toStartOfMonth(...)` → range predicate | `1A` |
-| `R-007` | `toStartOfHour/Day/FifteenMinutes(...)` → range predicate | `1A` |
-| `R-008` | избыточный `CAST(...)` в фильтре | `1C` |
-| `R-009` | `x IN (one_value)` → `x = one_value` | `1A` |
-| `R-010` | `x = a OR x = b OR x = c` → `x IN (...)` | `1A` |
-| `R-011` | условие без агрегатов в `HAVING` → `WHERE` | `1C` |
-| `R-012` | `WHERE TRUE`, `AND 1=1` и другие константные предикаты | `1A` |
-| `R-013` | `length(x) = 0 / > 0 / != 0` → `empty` / `notEmpty` | `1A` |
-| `R-014` | advisory: hash-based `GROUP BY` для длинных строк | `1B` |
-| `R-015` | лишний `DISTINCT` после эквивалентного `GROUP BY` | `1A` |
-| `R-016` | `ORDER BY` в подзапросе без `LIMIT` | `1C` |
-| `R-017` | pushdown внешнего фильтра во внутренний подзапрос | `1A` |
-| `R-018` | advisory: `UNION` → `UNION ALL`, если множества не пересекаются | `1C` |
-| `D-003` | top-level `SELECT *` | `detector` |
-| `D-004` | top-level `SELECT` без `LIMIT` и без агрегации | `detector` |
-| `D-007` | `FINAL` в `FROM` | `detector` |
+Текущий каталог:
+
+| Surface | Карточки | Реализовано и протестировано | Backlog |
+|---|---:|---:|---:|
+| `R-*` rewrite/advisory rules | 74 | 62 | 12 |
+| `D-*` detectors | 25 | 15 | 10 |
+| `E-*` environment cards | 20 | 0 | 20 |
+| Всего | 119 | 77 | 42 |
+
+Реализованные диапазоны: `R-001`…`R-062`,
+`D-003`, `D-004`, `D-007`, `D-014`…`D-025`. Карточки `R-101`…`R-112`,
+`D-001`, `D-002`, `D-005`…`D-013` и `E-001`…`E-020`
+пока описывают backlog.
+
+Полный список карточек хранится в [`docs/rules/cards/`](docs/rules/cards/),
+а фактическая регистрация правил — в
+[`clickadvisor/rules/registry.py`](clickadvisor/rules/registry.py).
 
 Классификация tier:
 
@@ -286,20 +281,31 @@ poetry run chadvisor mcp-server
 
 ## Метрики качества
 
-Запуск synthetic benchmark:
+Текущие воспроизводимые метрики на 2026-06-30:
+
+- rule detection on synthetic held-out: `36/36` cases, strict precision
+  `1.000`, recall `1.000`, F1 `1.000`;
+- classifier on `synthetic_expanded_v1` test split: best test macro F1
+  `0.691`, best test micro F1 `0.988`;
+- retrieval MRR@3: `0.517` on `20` explicit query-doc pairs with MiniLM-L6
+  (`0.458` for the current multilingual-e5 default).
+
+Rule detection is a deterministic regression metric, not an ML generalization
+claim. Classifier F1 is reported separately because that layer is trained.
+
+Запуск expanded synthetic benchmark:
 
 ```bash
-poetry run python scripts/eval/run_benchmark.py
+poetry run python scripts/eval/run_benchmark.py \
+  --cases-dir benchmark/cases/synthetic_expanded \
+  --mode strict
 ```
 
-На синтетическом бенчмарке из 20 кейсов ожидаемые метрики в `lenient`-режиме:
-
-- Precision: `1.00`;
-- Recall: `1.00`;
-- F1: `1.00`.
+Classifier ablation:
 
-`lenient` означает, что дополнительные валидные находки не штрафуются, если
-ожидаемые правила тоже сработали.
+```bash
+poetry run python scripts/eval/ablation_classifiers.py --run-id classifier_ablation_current
+```
 
 Для retrieval есть отдельный ablation-скрипт:
 
@@ -309,6 +315,9 @@ poetry run python scripts/eval/ablation_embeddings.py
 
 Результаты выбора embedding-модели описаны в
 [`docs/adr/ADR-013-embedding-model-selection.md`](docs/adr/ADR-013-embedding-model-selection.md).
+Подробная методика и текущие результаты: [`docs/evaluation.md`](docs/evaluation.md),
+[`docs/experiments/classifier_ablation.md`](docs/experiments/classifier_ablation.md),
+[`docs/experiments/retrieval_ablation.md`](docs/experiments/retrieval_ablation.md).
 
 ## Архитектура
 
@@ -358,6 +367,13 @@ poetry run python scripts/eval/run_benchmark.py
 Integration test для version detection ожидает ClickHouse HTTP endpoint на
 `localhost:8123`. В GitHub Actions он поднимается как service container.
 
+### AI-assisted development
+
+Codex и Claude использовались системно в разработке: для ревью архитектурных
+решений, генерации вариантов тестов, документации и проверки согласованности
+плана с кодом. Они не входят в trusted runtime path ClickAdvisor: рекомендации
+CLI/MCP формируются rule engine, ML evaluation surface и локальным retrieval.
+
 ## Что не заявляется как готовое в CLI v1
 
 - продуктовый generative LLM в critical path;

benchmark/README.md

@@ -26,7 +26,7 @@ poetry run python scripts/eval/run_benchmark.py
 
 The repository also includes an expanded generated synthetic dataset:
 
-- `benchmark/cases/synthetic_expanded/`: 162 deterministic cases
+- `benchmark/cases/synthetic_expanded/`: 180 deterministic cases
 - `benchmark/splits/synthetic_expanded_v1.yaml`: fixed 80/20 train/test split
 - `scripts/benchmark/generate_synthetic_dataset.py`: reproducible generator
 
@@ -58,7 +58,7 @@ poetry run python scripts/eval/ablation_embeddings.py
 
 ## Planned scope
 
-The benchmark is expected to grow beyond the generated 162-case synthetic set
+The benchmark is expected to grow beyond the generated 180-case synthetic set
 with manually reviewed real-query cases made up
 of:
 
@@ -113,7 +113,7 @@ Core fields:
 - `cases/clickbench/`: ClickBench query seeds
 - `cases/job/`: selected JOIN-heavy JOB seeds
 - `cases/synthetic/`: 20 validated targeted rule cases
-- `cases/synthetic_expanded/`: 162 generated rule-regression cases with
+- `cases/synthetic_expanded/`: 180 generated rule-regression cases with
   negative examples and a fixed split
 - `cases/github-issues/`: placeholder for manually curated issue-derived cases
 

benchmark/cases/synthetic_expanded/synthetic_expanded_r057_001.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r057_001
+source: synthetic
+sql: SELECT EXTRACT(MONTH FROM ts) AS mon FROM events LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_month_to_tomonth
+  detected_by_rule: R-057
+  severity: low
+  description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-057
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-057.

benchmark/cases/synthetic_expanded/synthetic_expanded_r057_002.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r057_002
+source: synthetic
+sql: SELECT EXTRACT(MONTH FROM created_at) AS mon FROM orders LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_month_to_tomonth
+  detected_by_rule: R-057
+  severity: low
+  description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-057
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-057.

benchmark/cases/synthetic_expanded/synthetic_expanded_r057_003.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r057_003
+source: synthetic
+sql: SELECT EXTRACT(MONTH FROM started_at) AS mon FROM sessions LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_month_to_tomonth
+  detected_by_rule: R-057
+  severity: low
+  description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-057
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-057.

benchmark/cases/synthetic_expanded/synthetic_expanded_r057_004.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r057_004
+source: synthetic
+sql: SELECT EXTRACT(MONTH FROM request_time) AS mon FROM requests LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_month_to_tomonth
+  detected_by_rule: R-057
+  severity: low
+  description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-057
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-057.

benchmark/cases/synthetic_expanded/synthetic_expanded_r057_005.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r057_005
+source: synthetic
+sql: SELECT EXTRACT(MONTH FROM processed_at) AS mon FROM payments LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_month_to_tomonth
+  detected_by_rule: R-057
+  severity: low
+  description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-057
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-057.

benchmark/cases/synthetic_expanded/synthetic_expanded_r057_006.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r057_006
+source: synthetic
+sql: SELECT EXTRACT(MONTH FROM event_time) AS mon FROM query_log LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_month_to_tomonth
+  detected_by_rule: R-057
+  severity: low
+  description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-057
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-057.

benchmark/cases/synthetic_expanded/synthetic_expanded_r058_001.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r058_001
+source: synthetic
+sql: SELECT EXTRACT(DAY FROM ts) AS day_num FROM events LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_day_to_todayofmonth
+  detected_by_rule: R-058
+  severity: low
+  description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-058
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-058.

benchmark/cases/synthetic_expanded/synthetic_expanded_r058_002.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r058_002
+source: synthetic
+sql: SELECT EXTRACT(DAY FROM created_at) AS day_num FROM orders LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_day_to_todayofmonth
+  detected_by_rule: R-058
+  severity: low
+  description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-058
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-058.

benchmark/cases/synthetic_expanded/synthetic_expanded_r058_003.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r058_003
+source: synthetic
+sql: SELECT EXTRACT(DAY FROM started_at) AS day_num FROM sessions LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_day_to_todayofmonth
+  detected_by_rule: R-058
+  severity: low
+  description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-058
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-058.

benchmark/cases/synthetic_expanded/synthetic_expanded_r058_004.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r058_004
+source: synthetic
+sql: SELECT EXTRACT(DAY FROM request_time) AS day_num FROM requests LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_day_to_todayofmonth
+  detected_by_rule: R-058
+  severity: low
+  description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-058
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-058.

benchmark/cases/synthetic_expanded/synthetic_expanded_r058_005.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r058_005
+source: synthetic
+sql: SELECT EXTRACT(DAY FROM processed_at) AS day_num FROM payments LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_day_to_todayofmonth
+  detected_by_rule: R-058
+  severity: low
+  description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-058
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-058.

benchmark/cases/synthetic_expanded/synthetic_expanded_r058_006.yaml

@@ -0,0 +1,17 @@
+status: validated
+case_id: synthetic_expanded_r058_006
+source: synthetic
+sql: SELECT EXTRACT(DAY FROM event_time) AS day_num FROM query_log LIMIT 100
+schema_files: []
+known_issues:
+- issue_id: I-1
+  type: extract_day_to_todayofmonth
+  detected_by_rule: R-058
+  severity: low
+  description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse.
+expected_rules_to_fire:
+- R-058
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Generated positive synthetic case for R-058.