Complete rule catalog implementation

README.md

@@ -37,8 +37,9 @@ ClickHouse, могут опираться на устаревшие рекоме
 
 - CLI-команда `chadvisor analyze`;
 - MCP-сервер `chadvisor mcp-server`;
-- 62 rewrite/advisory-правил `R-*`;
-- 15 детекторов антипаттернов `D-*`;
+- 74 rewrite/advisory-правила `R-*`;
+- 25 детекторов антипаттернов `D-*`;
+- 20 environment-правил `E-*`;
 - version detection через ClickHouse HTTP API;
 - console / JSON / Markdown отчёты;
 - режим объяснений `--mode explain`;
@@ -47,8 +48,8 @@ ClickHouse, могут опираться на устаревшие рекоме
 - synthetic benchmark для проверки срабатывания правил.
 
 В каталоге `/docs/rules/cards/` описаны 119 валидируемых карточек правил:
-77 из них уже имеют реализацию и тесты, 42 остаются backlog-карточками для
-следующих итераций.
+119 имеют зарегистрированную реализацию и прямое тестовое покрытие, backlog
+по карточкам закрыт.
 
 ## Быстрый старт
 
@@ -171,19 +172,20 @@ poetry run chadvisor analyze --sql query.sql \
 пользовательские данные не читаются. Используется только оценка планировщика
 ClickHouse (`rows`, `marks`).
 
-### Schema и EXPLAIN как дополнительные входы
+### Schema, EXPLAIN и environment как дополнительные входы
 
 CLI уже принимает опциональные файлы:
 
 ```bash
 poetry run chadvisor analyze --sql query.sql \
   --schema schema.sql \
+  --environment environment.json \
   --explain explain.json
 ```
 
-На текущем этапе основная часть реализованных правил работает по SQL AST.
-Схема и EXPLAIN зарезервированы для правил, которым нужен дополнительный
-контекст.
+`environment.json` передаёт настройки, hardware, cluster/workload facts и
+system metrics для `E-*` и части Tier 2 advisory rules. Если файл окружения не
+передан, environment-правила не срабатывают и SQL-only анализ остаётся прежним.
 
 ## Knowledge base и retrieval advisory
 
@@ -258,15 +260,13 @@ poetry run chadvisor mcp-server
 
 | Surface | Карточки | Реализовано и протестировано | Backlog |
 |---|---:|---:|---:|
-| `R-*` rewrite/advisory rules | 74 | 62 | 12 |
-| `D-*` detectors | 25 | 15 | 10 |
-| `E-*` environment cards | 20 | 0 | 20 |
-| Всего | 119 | 77 | 42 |
+| `R-*` rewrite/advisory rules | 74 | 74 | 0 |
+| `D-*` detectors | 25 | 25 | 0 |
+| `E-*` environment cards | 20 | 20 | 0 |
+| Всего | 119 | 119 | 0 |
 
-Реализованные диапазоны: `R-001`…`R-062`,
-`D-003`, `D-004`, `D-007`, `D-014`…`D-025`. Карточки `R-101`…`R-112`,
-`D-001`, `D-002`, `D-005`…`D-013` и `E-001`…`E-020`
-пока описывают backlog.
+Реализованные диапазоны: `R-001`…`R-062`, `R-101`…`R-112`,
+`D-001`…`D-025` и `E-001`…`E-020`.
 
 Полный список карточек хранится в [`docs/rules/cards/`](docs/rules/cards/),
 а фактическая регистрация правил — в
@@ -278,13 +278,14 @@ poetry run chadvisor mcp-server
 - `1B` — приближённые или opt-in рекомендации;
 - `1C` — условные рекомендации, зависящие от схемы или контекста;
 - `detector` — диагностика антипаттерна без автоматического rewrite.
+- `env` — environment-aware настройки по hardware, settings, cluster и workload.
 
 ## Метрики качества
 
 Текущие воспроизводимые метрики на 2026-06-30:
 
-- rule detection on synthetic held-out: `36/36` cases, strict precision
-  `1.000`, recall `1.000`, F1 `1.000`;
+- rule detection on expanded synthetic benchmark: `222/222` cases, strict
+  precision `1.000`, recall `1.000`, F1 `1.000`;
 - classifier on `synthetic_expanded_v1` test split: best test macro F1
   `0.691`, best test micro F1 `0.988`;
 - retrieval MRR@3: `0.517` on `20` explicit query-doc pairs with MiniLM-L6
@@ -322,7 +323,7 @@ poetry run python scripts/eval/ablation_embeddings.py
 ## Архитектура
 
 ```text
-SQL + optional Schema / EXPLAIN / CH version
+SQL + optional Schema / EXPLAIN / Environment / CH version
         ↓
    SQL Parser (sqlglot, ClickHouse dialect)
         ↓
@@ -380,7 +381,6 @@ CLI/MCP формируются rule engine, ML evaluation surface и локал
 - автоматический анализ `query_log`;
 - автоматические DDL-изменения;
 - выполнение `ANALYZE` или реальный replay запросов на данных;
-- полноценные environment rules по hardware/config.
+- автоматическое применение Tier 2 design/storage рекомендаций без проверки DBA.
 
-Эти направления описаны в ADR и backlog, но README отражает именно то, что
-поддерживает текущий код репозитория.
+README отражает именно то, что поддерживает текущий код репозитория.

benchmark/README.md

@@ -26,14 +26,17 @@ poetry run python scripts/eval/run_benchmark.py
 
 The repository also includes an expanded generated synthetic dataset:
 
-- `benchmark/cases/synthetic_expanded/`: 180 deterministic cases
+- `benchmark/cases/synthetic_expanded/`: 222 deterministic cases, including
+  SQL-only, schema-aware, and env-aware rule probes
 - `benchmark/splits/synthetic_expanded_v1.yaml`: fixed 80/20 train/test split
+  for the original 180-case ML dataset
 - `scripts/benchmark/generate_synthetic_dataset.py`: reproducible generator
 
 The expanded dataset exists to avoid presenting a suspicious metric on only 20
 hand-authored probes. It contains positive variations for implemented rules,
-multi-label overlaps where the current analyzer really emits several findings,
-and negative queries where no implemented rule should fire.
+targeted backlog-closure probes for `D-*`, `E-*`, and `R-101+`, multi-label
+overlaps where the current analyzer really emits several findings, and
+negative queries where no implemented rule should fire.
 
 Run the expanded benchmark:
 
@@ -101,6 +104,9 @@ Core fields:
 - `source`
 - `sql`
 - `schema_files`
+- `schema_ddl` for inline schema-aware cases
+- `environment` for env-aware cases
+- `context_type`: `sql-only`, `schema-aware`, `env-aware`, or `explain-aware`
 - `known_issues`
 - `expected_rules_to_fire`
 - `expected_improvement`
@@ -113,8 +119,9 @@ Core fields:
 - `cases/clickbench/`: ClickBench query seeds
 - `cases/job/`: selected JOIN-heavy JOB seeds
 - `cases/synthetic/`: 20 validated targeted rule cases
-- `cases/synthetic_expanded/`: 180 generated rule-regression cases with
-  negative examples and a fixed split
+- `cases/synthetic_expanded/`: 222 rule-regression cases with negative examples,
+  fixed split metadata for the original generated subset, and targeted
+  schema/env-aware probes for newly implemented rules
 - `cases/github-issues/`: placeholder for manually curated issue-derived cases
 
 ## Validation

benchmark/SCHEMA.yaml

@@ -29,6 +29,13 @@ properties:
     type: array
     items:
       type: string
+  schema_ddl:
+    type: ["string", "null"]
+  environment:
+    type: ["object", "null"]
+  context_type:
+    type: string
+    enum: ["sql-only", "schema-aware", "env-aware", "explain-aware"]
   known_issues:
     type: array
     items:

benchmark/TEMPLATE.yaml

@@ -8,6 +8,9 @@ source: "TODO: tpch | clickbench | job | synthetic | github-issues"
 sql: |
   -- TODO: paste SQL here
 schema_files: [] # TODO: add relative schema file paths if relevant
+schema_ddl: null # TODO: inline DDL for schema-aware cases when no schema file is needed
+environment: null # TODO: structured environment context for env-aware cases
+context_type: sql-only # sql-only | schema-aware | env-aware | explain-aware
 known_issues: [] # TODO: list of {issue_id, type, detected_by_rule, severity, description}
 expected_rules_to_fire: [] # TODO: expert-only labeling with DBA review
 expected_findings_count: 0 # TODO: expected number of findings in the final report

benchmark/cases/synthetic_expanded/synthetic_expanded_d001_001.yaml

@@ -0,0 +1,19 @@
+status: validated
+case_id: synthetic_expanded_d001_001
+source: synthetic
+sql: SELECT count() FROM events WHERE user_id = 42
+schema_files: []
+schema_ddl: CREATE TABLE events (ts DateTime, user_id UInt64) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) PRIMARY KEY user_id ORDER BY user_id
+context_type: schema-aware
+known_issues:
+- issue_id: I-1
+  type: full_scan_on_partitioned_table
+  detected_by_rule: D-001
+  severity: high
+  description: Partitioned table query does not filter the partition key.
+expected_rules_to_fire:
+- D-001
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-001.

benchmark/cases/synthetic_expanded/synthetic_expanded_d002_001.yaml

@@ -0,0 +1,18 @@
+status: validated
+case_id: synthetic_expanded_d002_001
+source: synthetic
+sql: SELECT count() FROM events CROSS JOIN users
+schema_files: []
+context_type: sql-only
+known_issues:
+- issue_id: I-1
+  type: cross_join_risk
+  detected_by_rule: D-002
+  severity: high
+  description: CROSS JOIN can create a large Cartesian product.
+expected_rules_to_fire:
+- D-002
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-002.

benchmark/cases/synthetic_expanded/synthetic_expanded_d005_001.yaml

@@ -0,0 +1,24 @@
+status: validated
+case_id: synthetic_expanded_d005_001
+source: synthetic
+sql: SELECT count() FROM logs WHERE message LIKE '%timeout%'
+schema_files: []
+context_type: sql-only
+known_issues:
+- issue_id: I-1
+  type: leading_wildcard_like
+  detected_by_rule: D-005
+  severity: medium
+  description: Leading wildcard LIKE cannot use ordered pruning efficiently.
+- issue_id: I-2
+  type: skip_index_recommendation
+  detected_by_rule: R-102
+  severity: medium
+  description: Substring search may need token or ngram skip indexes.
+expected_rules_to_fire:
+- D-005
+- R-102
+expected_findings_count: 2
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-005 with the related R-102 advisory.

benchmark/cases/synthetic_expanded/synthetic_expanded_d006_001.yaml

@@ -0,0 +1,18 @@
+status: validated
+case_id: synthetic_expanded_d006_001
+source: synthetic
+sql: SELECT arrayJoin(tags) AS tag FROM events WHERE user_id = 42 LIMIT 10
+schema_files: []
+context_type: sql-only
+known_issues:
+- issue_id: I-1
+  type: arrayjoin_before_filter
+  detected_by_rule: D-006
+  severity: high
+  description: arrayJoin appears before WHERE and may expand rows too early.
+expected_rules_to_fire:
+- D-006
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-006.

benchmark/cases/synthetic_expanded/synthetic_expanded_d008_001.yaml

@@ -0,0 +1,19 @@
+status: validated
+case_id: synthetic_expanded_d008_001
+source: synthetic
+sql: SELECT count() FROM events SAMPLE 0.1
+schema_files: []
+schema_ddl: CREATE TABLE events (id UInt64) ENGINE = MergeTree ORDER BY id
+context_type: schema-aware
+known_issues:
+- issue_id: I-1
+  type: sample_without_sample_by
+  detected_by_rule: D-008
+  severity: low
+  description: Query uses SAMPLE but table DDL has no SAMPLE BY expression.
+expected_rules_to_fire:
+- D-008
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-008.

benchmark/cases/synthetic_expanded/synthetic_expanded_d009_001.yaml

@@ -0,0 +1,22 @@
+status: validated
+case_id: synthetic_expanded_d009_001
+source: synthetic
+sql: SELECT count() FROM events
+schema_files: []
+schema_ddl: CREATE TABLE events (status Nullable(String)) ENGINE = MergeTree ORDER BY status
+environment:
+  null_counts:
+    status: 0
+context_type: schema-aware
+known_issues:
+- issue_id: I-1
+  type: nullable_without_need
+  detected_by_rule: D-009
+  severity: low
+  description: Nullable column has no observed NULL values in supplied context.
+expected_rules_to_fire:
+- D-009
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-009.

benchmark/cases/synthetic_expanded/synthetic_expanded_d010_001.yaml

@@ -0,0 +1,18 @@
+status: validated
+case_id: synthetic_expanded_d010_001
+source: synthetic
+sql: SELECT id FROM (SELECT id, payload FROM events) LIMIT 10
+schema_files: []
+context_type: sql-only
+known_issues:
+- issue_id: I-1
+  type: unused_columns_in_select
+  detected_by_rule: D-010
+  severity: low
+  description: Inner subquery projects a column unused by the outer query.
+expected_rules_to_fire:
+- D-010
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-010.

benchmark/cases/synthetic_expanded/synthetic_expanded_d011_001.yaml

@@ -0,0 +1,30 @@
+status: validated
+case_id: synthetic_expanded_d011_001
+source: synthetic
+sql: SELECT count() FROM events e JOIN users u ON toUInt64(e.user_id) = u.id
+schema_files: []
+context_type: sql-only
+known_issues:
+- issue_id: I-1
+  type: implicit_type_coercion_in_join
+  detected_by_rule: D-011
+  severity: high
+  description: JOIN key comparison contains an explicit cast.
+- issue_id: I-2
+  type: redundant_cast_on_filter_removal
+  detected_by_rule: R-008
+  severity: medium
+  description: Cast around a column can block index use.
+- issue_id: I-3
+  type: throwing_cast_to_or_default_cast
+  detected_by_rule: R-020
+  severity: medium
+  description: Throwing toType conversion on a column can interrupt the query.
+expected_rules_to_fire:
+- D-011
+- R-008
+- R-020
+expected_findings_count: 3
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-011; explicit JOIN cast also exercises cast advisories.

benchmark/cases/synthetic_expanded/synthetic_expanded_d012_001.yaml

@@ -0,0 +1,18 @@
+status: validated
+case_id: synthetic_expanded_d012_001
+source: synthetic
+sql: SELECT row_number() OVER (ORDER BY ts) FROM events LIMIT 10
+schema_files: []
+context_type: sql-only
+known_issues:
+- issue_id: I-1
+  type: window_function_without_partition
+  detected_by_rule: D-012
+  severity: medium
+  description: Window function has no PARTITION BY clause.
+expected_rules_to_fire:
+- D-012
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-012.

benchmark/cases/synthetic_expanded/synthetic_expanded_d013_001.yaml

@@ -0,0 +1,18 @@
+status: validated
+case_id: synthetic_expanded_d013_001
+source: synthetic
+sql: SELECT id FROM (SELECT id FROM (SELECT id FROM (SELECT id FROM (SELECT id FROM events)))) LIMIT 10
+schema_files: []
+context_type: sql-only
+known_issues:
+- issue_id: I-1
+  type: deeply_nested_subqueries
+  detected_by_rule: D-013
+  severity: low
+  description: Query contains deeply nested subqueries.
+expected_rules_to_fire:
+- D-013
+expected_findings_count: 1
+expected_improvement: null
+synthetic_explain_path: null
+notes: Synthetic positive case for D-013.