diff --git a/README.md b/README.md index d95face2..2c8881c0 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,10 @@ ClickAdvisor помогает DBA и разработчикам быстрее он принимает SQL-файл, парсит запрос через `sqlglot`, применяет набор детерминированных правил и возвращает отчёт в консоли, JSON или Markdown. -Главный принцип проекта: **LLM и retrieval могут помогать, но ядро анализа — -это rule engine с явно описанными условиями применимости**. Утилита работает -локально и не отправляет SQL во внешние сервисы. +Главный принцип проекта: **rules + ML + retrieval**, где доверенное ядро +анализа остаётся rule engine с явно описанными условиями применимости. ML +используется как отдельная evaluation surface, retrieval добавляет локальные +ссылки на документацию, а утилита не отправляет SQL во внешние сервисы. [Сайт проекта](https://clickadvisor.lovable.app) @@ -36,8 +37,8 @@ ClickHouse, могут опираться на устаревшие рекоме - CLI-команда `chadvisor analyze`; - MCP-сервер `chadvisor mcp-server`; -- 18 Tier 1 rewrite-правил `R-001`…`R-018`; -- 3 детектора антипаттернов `D-003`, `D-004`, `D-007`; +- 62 rewrite/advisory-правил `R-*`; +- 15 детекторов антипаттернов `D-*`; - version detection через ClickHouse HTTP API; - console / JSON / Markdown отчёты; - режим объяснений `--mode explain`; @@ -45,9 +46,9 @@ ClickHouse, могут опираться на устаревшие рекоме - optional `EXPLAIN ESTIMATE` сравнение через ClickHouse HTTP API; - synthetic benchmark для проверки срабатывания правил. -В каталоге `/docs/rules/cards/` описаны 54 карточки правил. Реализованная -часть в коде — 21 правило/детектор, перечисленные ниже в разделе -«Правила и покрытие». +В каталоге `/docs/rules/cards/` описаны 119 валидируемых карточек правил: +77 из них уже имеют реализацию и тесты, 42 остаются backlog-карточками для +следующих итераций. ## Быстрый старт @@ -253,29 +254,23 @@ poetry run chadvisor mcp-server ## Правила и покрытие -| Rule ID | Что ищет | Tier | -|---|---|---| -| `R-001` | `COUNT(DISTINCT x)` → `uniqExact(x)` | `1A` | -| `R-002` | `COUNT(DISTINCT x)` → advisory `uniq(x)` | `1B` | -| `R-003` | `quantileExact(...)` → advisory `quantileTDigest(...)` | `1B` | -| `R-004` | `COUNT(*) FROM (SELECT DISTINCT ...)` → специализированный агрегат | `1A` | -| `R-005` | `toDate(col) = ...` → range predicate | `1A` | -| `R-006` | `toYYYYMM(...)` / `toStartOfMonth(...)` → range predicate | `1A` | -| `R-007` | `toStartOfHour/Day/FifteenMinutes(...)` → range predicate | `1A` | -| `R-008` | избыточный `CAST(...)` в фильтре | `1C` | -| `R-009` | `x IN (one_value)` → `x = one_value` | `1A` | -| `R-010` | `x = a OR x = b OR x = c` → `x IN (...)` | `1A` | -| `R-011` | условие без агрегатов в `HAVING` → `WHERE` | `1C` | -| `R-012` | `WHERE TRUE`, `AND 1=1` и другие константные предикаты | `1A` | -| `R-013` | `length(x) = 0 / > 0 / != 0` → `empty` / `notEmpty` | `1A` | -| `R-014` | advisory: hash-based `GROUP BY` для длинных строк | `1B` | -| `R-015` | лишний `DISTINCT` после эквивалентного `GROUP BY` | `1A` | -| `R-016` | `ORDER BY` в подзапросе без `LIMIT` | `1C` | -| `R-017` | pushdown внешнего фильтра во внутренний подзапрос | `1A` | -| `R-018` | advisory: `UNION` → `UNION ALL`, если множества не пересекаются | `1C` | -| `D-003` | top-level `SELECT *` | `detector` | -| `D-004` | top-level `SELECT` без `LIMIT` и без агрегации | `detector` | -| `D-007` | `FINAL` в `FROM` | `detector` | +Текущий каталог: + +| Surface | Карточки | Реализовано и протестировано | Backlog | +|---|---:|---:|---:| +| `R-*` rewrite/advisory rules | 74 | 62 | 12 | +| `D-*` detectors | 25 | 15 | 10 | +| `E-*` environment cards | 20 | 0 | 20 | +| Всего | 119 | 77 | 42 | + +Реализованные диапазоны: `R-001`…`R-062`, +`D-003`, `D-004`, `D-007`, `D-014`…`D-025`. Карточки `R-101`…`R-112`, +`D-001`, `D-002`, `D-005`…`D-013` и `E-001`…`E-020` +пока описывают backlog. + +Полный список карточек хранится в [`docs/rules/cards/`](docs/rules/cards/), +а фактическая регистрация правил — в +[`clickadvisor/rules/registry.py`](clickadvisor/rules/registry.py). Классификация tier: @@ -286,20 +281,31 @@ poetry run chadvisor mcp-server ## Метрики качества -Запуск synthetic benchmark: +Текущие воспроизводимые метрики на 2026-06-30: + +- rule detection on synthetic held-out: `36/36` cases, strict precision + `1.000`, recall `1.000`, F1 `1.000`; +- classifier on `synthetic_expanded_v1` test split: best test macro F1 + `0.691`, best test micro F1 `0.988`; +- retrieval MRR@3: `0.517` on `20` explicit query-doc pairs with MiniLM-L6 + (`0.458` for the current multilingual-e5 default). + +Rule detection is a deterministic regression metric, not an ML generalization +claim. Classifier F1 is reported separately because that layer is trained. + +Запуск expanded synthetic benchmark: ```bash -poetry run python scripts/eval/run_benchmark.py +poetry run python scripts/eval/run_benchmark.py \ + --cases-dir benchmark/cases/synthetic_expanded \ + --mode strict ``` -На синтетическом бенчмарке из 20 кейсов ожидаемые метрики в `lenient`-режиме: - -- Precision: `1.00`; -- Recall: `1.00`; -- F1: `1.00`. +Classifier ablation: -`lenient` означает, что дополнительные валидные находки не штрафуются, если -ожидаемые правила тоже сработали. +```bash +poetry run python scripts/eval/ablation_classifiers.py --run-id classifier_ablation_current +``` Для retrieval есть отдельный ablation-скрипт: @@ -309,6 +315,9 @@ poetry run python scripts/eval/ablation_embeddings.py Результаты выбора embedding-модели описаны в [`docs/adr/ADR-013-embedding-model-selection.md`](docs/adr/ADR-013-embedding-model-selection.md). +Подробная методика и текущие результаты: [`docs/evaluation.md`](docs/evaluation.md), +[`docs/experiments/classifier_ablation.md`](docs/experiments/classifier_ablation.md), +[`docs/experiments/retrieval_ablation.md`](docs/experiments/retrieval_ablation.md). ## Архитектура @@ -358,6 +367,13 @@ poetry run python scripts/eval/run_benchmark.py Integration test для version detection ожидает ClickHouse HTTP endpoint на `localhost:8123`. В GitHub Actions он поднимается как service container. +### AI-assisted development + +Codex и Claude использовались системно в разработке: для ревью архитектурных +решений, генерации вариантов тестов, документации и проверки согласованности +плана с кодом. Они не входят в trusted runtime path ClickAdvisor: рекомендации +CLI/MCP формируются rule engine, ML evaluation surface и локальным retrieval. + ## Что не заявляется как готовое в CLI v1 - продуктовый generative LLM в critical path; diff --git a/benchmark/README.md b/benchmark/README.md index 110bb06e..87fb99ba 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -26,7 +26,7 @@ poetry run python scripts/eval/run_benchmark.py The repository also includes an expanded generated synthetic dataset: -- `benchmark/cases/synthetic_expanded/`: 162 deterministic cases +- `benchmark/cases/synthetic_expanded/`: 180 deterministic cases - `benchmark/splits/synthetic_expanded_v1.yaml`: fixed 80/20 train/test split - `scripts/benchmark/generate_synthetic_dataset.py`: reproducible generator @@ -58,7 +58,7 @@ poetry run python scripts/eval/ablation_embeddings.py ## Planned scope -The benchmark is expected to grow beyond the generated 162-case synthetic set +The benchmark is expected to grow beyond the generated 180-case synthetic set with manually reviewed real-query cases made up of: @@ -113,7 +113,7 @@ Core fields: - `cases/clickbench/`: ClickBench query seeds - `cases/job/`: selected JOIN-heavy JOB seeds - `cases/synthetic/`: 20 validated targeted rule cases -- `cases/synthetic_expanded/`: 162 generated rule-regression cases with +- `cases/synthetic_expanded/`: 180 generated rule-regression cases with negative examples and a fixed split - `cases/github-issues/`: placeholder for manually curated issue-derived cases diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_001.yaml new file mode 100644 index 00000000..896bdc64 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_001.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r057_001 +source: synthetic +sql: SELECT EXTRACT(MONTH FROM ts) AS mon FROM events LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_month_to_tomonth + detected_by_rule: R-057 + severity: low + description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-057 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-057. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_002.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_002.yaml new file mode 100644 index 00000000..31a3ae34 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_002.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r057_002 +source: synthetic +sql: SELECT EXTRACT(MONTH FROM created_at) AS mon FROM orders LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_month_to_tomonth + detected_by_rule: R-057 + severity: low + description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-057 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-057. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_003.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_003.yaml new file mode 100644 index 00000000..5458678e --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_003.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r057_003 +source: synthetic +sql: SELECT EXTRACT(MONTH FROM started_at) AS mon FROM sessions LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_month_to_tomonth + detected_by_rule: R-057 + severity: low + description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-057 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-057. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_004.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_004.yaml new file mode 100644 index 00000000..15d597db --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_004.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r057_004 +source: synthetic +sql: SELECT EXTRACT(MONTH FROM request_time) AS mon FROM requests LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_month_to_tomonth + detected_by_rule: R-057 + severity: low + description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-057 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-057. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_005.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_005.yaml new file mode 100644 index 00000000..b1055aec --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_005.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r057_005 +source: synthetic +sql: SELECT EXTRACT(MONTH FROM processed_at) AS mon FROM payments LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_month_to_tomonth + detected_by_rule: R-057 + severity: low + description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-057 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-057. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_006.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_006.yaml new file mode 100644 index 00000000..c321f721 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r057_006.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r057_006 +source: synthetic +sql: SELECT EXTRACT(MONTH FROM event_time) AS mon FROM query_log LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_month_to_tomonth + detected_by_rule: R-057 + severity: low + description: EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-057 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-057. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_001.yaml new file mode 100644 index 00000000..e4bf859d --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_001.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r058_001 +source: synthetic +sql: SELECT EXTRACT(DAY FROM ts) AS day_num FROM events LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_day_to_todayofmonth + detected_by_rule: R-058 + severity: low + description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-058 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-058. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_002.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_002.yaml new file mode 100644 index 00000000..808abc21 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_002.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r058_002 +source: synthetic +sql: SELECT EXTRACT(DAY FROM created_at) AS day_num FROM orders LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_day_to_todayofmonth + detected_by_rule: R-058 + severity: low + description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-058 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-058. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_003.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_003.yaml new file mode 100644 index 00000000..a80449a4 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_003.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r058_003 +source: synthetic +sql: SELECT EXTRACT(DAY FROM started_at) AS day_num FROM sessions LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_day_to_todayofmonth + detected_by_rule: R-058 + severity: low + description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-058 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-058. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_004.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_004.yaml new file mode 100644 index 00000000..9704f127 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_004.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r058_004 +source: synthetic +sql: SELECT EXTRACT(DAY FROM request_time) AS day_num FROM requests LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_day_to_todayofmonth + detected_by_rule: R-058 + severity: low + description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-058 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-058. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_005.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_005.yaml new file mode 100644 index 00000000..1c9f1c95 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_005.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r058_005 +source: synthetic +sql: SELECT EXTRACT(DAY FROM processed_at) AS day_num FROM payments LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_day_to_todayofmonth + detected_by_rule: R-058 + severity: low + description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-058 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-058. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_006.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_006.yaml new file mode 100644 index 00000000..182b8d61 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r058_006.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r058_006 +source: synthetic +sql: SELECT EXTRACT(DAY FROM event_time) AS day_num FROM query_log LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_day_to_todayofmonth + detected_by_rule: R-058 + severity: low + description: EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse. +expected_rules_to_fire: +- R-058 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-058. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_001.yaml new file mode 100644 index 00000000..20fa50cd --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_001.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r059_001 +source: synthetic +sql: SELECT EXTRACT(HOUR FROM ts) AS hr FROM events LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_hour_to_tohour + detected_by_rule: R-059 + severity: low + description: EXTRACT(HOUR FROM ts) эквивалентна toHour(ts) в ClickHouse. +expected_rules_to_fire: +- R-059 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-059. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_002.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_002.yaml new file mode 100644 index 00000000..66588e53 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_002.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r059_002 +source: synthetic +sql: SELECT EXTRACT(HOUR FROM created_at) AS hr FROM orders LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_hour_to_tohour + detected_by_rule: R-059 + severity: low + description: EXTRACT(HOUR FROM ts) эквивалентна toHour(ts) в ClickHouse. +expected_rules_to_fire: +- R-059 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-059. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_003.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_003.yaml new file mode 100644 index 00000000..d61a25c0 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_003.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r059_003 +source: synthetic +sql: SELECT EXTRACT(HOUR FROM started_at) AS hr FROM sessions LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_hour_to_tohour + detected_by_rule: R-059 + severity: low + description: EXTRACT(HOUR FROM ts) эквивалентна toHour(ts) в ClickHouse. +expected_rules_to_fire: +- R-059 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-059. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_004.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_004.yaml new file mode 100644 index 00000000..13ad54f1 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_004.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r059_004 +source: synthetic +sql: SELECT EXTRACT(HOUR FROM request_time) AS hr FROM requests LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_hour_to_tohour + detected_by_rule: R-059 + severity: low + description: EXTRACT(HOUR FROM ts) эквивалентна toHour(ts) в ClickHouse. +expected_rules_to_fire: +- R-059 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-059. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_005.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_005.yaml new file mode 100644 index 00000000..1d0aec49 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_005.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r059_005 +source: synthetic +sql: SELECT EXTRACT(HOUR FROM processed_at) AS hr FROM payments LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_hour_to_tohour + detected_by_rule: R-059 + severity: low + description: EXTRACT(HOUR FROM ts) эквивалентна toHour(ts) в ClickHouse. +expected_rules_to_fire: +- R-059 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-059. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_006.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_006.yaml new file mode 100644 index 00000000..5de135f5 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r059_006.yaml @@ -0,0 +1,17 @@ +status: validated +case_id: synthetic_expanded_r059_006 +source: synthetic +sql: SELECT EXTRACT(HOUR FROM event_time) AS hr FROM query_log LIMIT 100 +schema_files: [] +known_issues: +- issue_id: I-1 + type: extract_hour_to_tohour + detected_by_rule: R-059 + severity: low + description: EXTRACT(HOUR FROM ts) эквивалентна toHour(ts) в ClickHouse. +expected_rules_to_fire: +- R-059 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Generated positive synthetic case for R-059. diff --git a/benchmark/splits/synthetic_expanded_v1.yaml b/benchmark/splits/synthetic_expanded_v1.yaml index b111e114..297b481a 100644 --- a/benchmark/splits/synthetic_expanded_v1.yaml +++ b/benchmark/splits/synthetic_expanded_v1.yaml @@ -5,37 +5,35 @@ train_case_ids: - synthetic_expanded_d003_001 - synthetic_expanded_d003_002 - synthetic_expanded_d003_003 -- synthetic_expanded_d003_004 - synthetic_expanded_d003_005 - synthetic_expanded_d003_006 - synthetic_expanded_d004_001 -- synthetic_expanded_d004_002 - synthetic_expanded_d004_003 - synthetic_expanded_d004_004 - synthetic_expanded_d004_005 +- synthetic_expanded_d004_006 - synthetic_expanded_d007_001 - synthetic_expanded_d007_002 -- synthetic_expanded_d007_003 -- synthetic_expanded_d007_004 -- synthetic_expanded_d007_005 - synthetic_expanded_d007_006 -- synthetic_expanded_d014_001 +- synthetic_expanded_d014_002 - synthetic_expanded_d014_003 +- synthetic_expanded_d014_004 - synthetic_expanded_d014_005 - synthetic_expanded_d014_006 - synthetic_expanded_negative_001 +- synthetic_expanded_negative_002 - synthetic_expanded_negative_003 -- synthetic_expanded_negative_004 -- synthetic_expanded_negative_005 +- synthetic_expanded_negative_006 - synthetic_expanded_negative_007 - synthetic_expanded_negative_008 -- synthetic_expanded_negative_009 -- synthetic_expanded_negative_012 +- synthetic_expanded_negative_010 +- synthetic_expanded_negative_011 +- synthetic_expanded_negative_013 - synthetic_expanded_negative_014 - synthetic_expanded_negative_015 - synthetic_expanded_negative_016 - synthetic_expanded_negative_017 -- synthetic_expanded_negative_018 +- synthetic_expanded_negative_019 - synthetic_expanded_negative_020 - synthetic_expanded_r001_001 - synthetic_expanded_r001_003 @@ -131,16 +129,35 @@ train_case_ids: - synthetic_expanded_r020_004 - synthetic_expanded_r020_005 - synthetic_expanded_r020_006 +- synthetic_expanded_r057_001 +- synthetic_expanded_r057_002 +- synthetic_expanded_r057_003 +- synthetic_expanded_r057_004 +- synthetic_expanded_r057_005 +- synthetic_expanded_r057_006 +- synthetic_expanded_r058_001 +- synthetic_expanded_r058_002 +- synthetic_expanded_r058_003 +- synthetic_expanded_r058_004 +- synthetic_expanded_r058_005 +- synthetic_expanded_r059_001 +- synthetic_expanded_r059_002 +- synthetic_expanded_r059_003 +- synthetic_expanded_r059_004 +- synthetic_expanded_r059_005 +- synthetic_expanded_r059_006 test_case_ids: -- synthetic_expanded_d004_006 -- synthetic_expanded_d014_002 -- synthetic_expanded_d014_004 -- synthetic_expanded_negative_002 -- synthetic_expanded_negative_006 -- synthetic_expanded_negative_010 -- synthetic_expanded_negative_011 -- synthetic_expanded_negative_013 -- synthetic_expanded_negative_019 +- synthetic_expanded_d003_004 +- synthetic_expanded_d004_002 +- synthetic_expanded_d007_003 +- synthetic_expanded_d007_004 +- synthetic_expanded_d007_005 +- synthetic_expanded_d014_001 +- synthetic_expanded_negative_004 +- synthetic_expanded_negative_005 +- synthetic_expanded_negative_009 +- synthetic_expanded_negative_012 +- synthetic_expanded_negative_018 - synthetic_expanded_r001_002 - synthetic_expanded_r001_007 - synthetic_expanded_r001_008 @@ -165,7 +182,8 @@ test_case_ids: - synthetic_expanded_r019_002 - synthetic_expanded_r019_003 - synthetic_expanded_r020_003 +- synthetic_expanded_r058_006 counts: - total: 162 - train: 129 - test: 33 + total: 180 + train: 144 + test: 36 diff --git a/clickadvisor/ml/__init__.py b/clickadvisor/ml/__init__.py new file mode 100644 index 00000000..2e864413 --- /dev/null +++ b/clickadvisor/ml/__init__.py @@ -0,0 +1,3 @@ +"""Feature extraction and classification utilities for ClickAdvisor's ML +evaluation surface. Independent from the deterministic rule engine in +clickadvisor/rules/.""" diff --git a/clickadvisor/ml/classifier.py b/clickadvisor/ml/classifier.py new file mode 100644 index 00000000..f1dd43a3 --- /dev/null +++ b/clickadvisor/ml/classifier.py @@ -0,0 +1,199 @@ +from __future__ import annotations + +from collections.abc import Iterable, Mapping, Sequence +from dataclasses import dataclass +from typing import Any, Protocol + +import numpy as np +from sklearn.ensemble import RandomForestClassifier +from sklearn.linear_model import LogisticRegression +from sklearn.metrics import f1_score, precision_score, recall_score +from sklearn.multiclass import OneVsRestClassifier +from sklearn.multioutput import MultiOutputClassifier +from sklearn.preprocessing import MultiLabelBinarizer + +from clickadvisor.ml.dataset import BenchmarkExample +from clickadvisor.ml.features import ordered_feature_names + +try: # CatBoost is optional for library imports, but used by the ablation script when available. + from catboost import CatBoostClassifier +except ImportError: # pragma: no cover - exercised only in lean environments. + CatBoostClassifier = None + + +class EstimatorProtocol(Protocol): + def fit(self, x: np.ndarray, y: np.ndarray) -> Any: ... + + def predict(self, x: np.ndarray) -> np.ndarray: ... + + +@dataclass(frozen=True, slots=True) +class DatasetMatrix: + x: np.ndarray + y: np.ndarray + case_ids: list[str] + feature_names: list[str] + label_names: list[str] + + +@dataclass(frozen=True, slots=True) +class ClassifierMetrics: + model: str + train_f1_macro: float + train_f1_micro: float + test_f1_macro: float + test_f1_micro: float + test_precision_macro: float + test_recall_macro: float + test_precision_micro: float + test_recall_micro: float + train_cases: int + test_cases: int + labels: int + + def as_row(self) -> dict[str, str | float | int]: + return { + "model": self.model, + "train_f1_macro": self.train_f1_macro, + "train_f1_micro": self.train_f1_micro, + "test_f1_macro": self.test_f1_macro, + "test_f1_micro": self.test_f1_micro, + "test_precision_macro": self.test_precision_macro, + "test_recall_macro": self.test_recall_macro, + "test_precision_micro": self.test_precision_micro, + "test_recall_micro": self.test_recall_micro, + "train_cases": self.train_cases, + "test_cases": self.test_cases, + "labels": self.labels, + } + + +def build_label_binarizer(examples: Sequence[BenchmarkExample]) -> MultiLabelBinarizer: + label_names = sorted({label for example in examples for label in example.labels}) + return MultiLabelBinarizer(classes=label_names) + + +def vectorize_examples( + examples: Sequence[BenchmarkExample], + *, + feature_names: Sequence[str] | None = None, + label_binarizer: MultiLabelBinarizer | None = None, +) -> DatasetMatrix: + rows: list[Mapping[str, float]] = [example.features for example in examples] + names = list(feature_names or ordered_feature_names(rows)) + labels = label_binarizer or build_label_binarizer(examples) + if not hasattr(labels, "classes_"): + labels.fit([example.labels for example in examples]) + + x = np.array( + [[float(example.features.get(feature_name, 0.0)) for feature_name in names] for example in examples], + dtype=float, + ) + y = labels.transform([example.labels for example in examples]).astype(int) + return DatasetMatrix( + x=x, + y=y, + case_ids=[example.case_id for example in examples], + feature_names=names, + label_names=list(labels.classes_), + ) + + +def make_classifiers(random_state: int = 42) -> dict[str, EstimatorProtocol]: + classifiers: dict[str, EstimatorProtocol] = { + "logistic_regression": OneVsRestClassifier( + LogisticRegression( + max_iter=1000, + class_weight="balanced", + solver="liblinear", + random_state=random_state, + ) + ), + "random_forest": RandomForestClassifier( + n_estimators=250, + max_depth=None, + min_samples_leaf=1, + class_weight="balanced", + random_state=random_state, + n_jobs=-1, + ), + } + if CatBoostClassifier is not None: + classifiers["catboost"] = MultiOutputClassifier( + CatBoostClassifier( + iterations=120, + depth=4, + learning_rate=0.08, + loss_function="Logloss", + random_seed=random_state, + verbose=False, + allow_writing_files=False, + ) + ) + return classifiers + + +def evaluate_classifier( + model_name: str, + estimator: EstimatorProtocol, + train: DatasetMatrix, + test: DatasetMatrix, +) -> ClassifierMetrics: + estimator.fit(train.x, train.y) + train_pred = _normalize_predictions(estimator.predict(train.x)) + test_pred = _normalize_predictions(estimator.predict(test.x)) + + return ClassifierMetrics( + model=model_name, + train_f1_macro=f1_score(train.y, train_pred, average="macro", zero_division=0), + train_f1_micro=f1_score(train.y, train_pred, average="micro", zero_division=0), + test_f1_macro=f1_score(test.y, test_pred, average="macro", zero_division=0), + test_f1_micro=f1_score(test.y, test_pred, average="micro", zero_division=0), + test_precision_macro=precision_score(test.y, test_pred, average="macro", zero_division=0), + test_recall_macro=recall_score(test.y, test_pred, average="macro", zero_division=0), + test_precision_micro=precision_score(test.y, test_pred, average="micro", zero_division=0), + test_recall_micro=recall_score(test.y, test_pred, average="micro", zero_division=0), + train_cases=len(train.case_ids), + test_cases=len(test.case_ids), + labels=len(train.label_names), + ) + + +def evaluate_classifiers( + train_examples: Sequence[BenchmarkExample], + test_examples: Sequence[BenchmarkExample], + *, + random_state: int = 42, + model_names: Iterable[str] | None = None, +) -> list[ClassifierMetrics]: + all_examples = [*train_examples, *test_examples] + label_binarizer = build_label_binarizer(all_examples) + label_binarizer.fit([example.labels for example in all_examples]) + all_feature_rows: list[Mapping[str, float]] = [example.features for example in all_examples] + feature_names = ordered_feature_names(all_feature_rows) + + train = vectorize_examples( + train_examples, + feature_names=feature_names, + label_binarizer=label_binarizer, + ) + test = vectorize_examples( + test_examples, + feature_names=feature_names, + label_binarizer=label_binarizer, + ) + + classifiers = make_classifiers(random_state=random_state) + selected = set(model_names) if model_names is not None else set(classifiers) + return [ + evaluate_classifier(name, estimator, train, test) + for name, estimator in classifiers.items() + if name in selected + ] + + +def _normalize_predictions(predictions: np.ndarray) -> np.ndarray: + array = np.asarray(predictions) + if array.ndim == 3 and array.shape[-1] == 1: + array = array[:, :, 0] + return (array > 0).astype(int) diff --git a/clickadvisor/ml/dataset.py b/clickadvisor/ml/dataset.py new file mode 100644 index 00000000..30d9b589 --- /dev/null +++ b/clickadvisor/ml/dataset.py @@ -0,0 +1,81 @@ +from __future__ import annotations + +from collections.abc import Iterable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import yaml + +from clickadvisor.ml.features import FeatureMap, QueryFeatureExtractor + + +@dataclass(frozen=True, slots=True) +class BenchmarkExample: + case_id: str + sql: str + labels: tuple[str, ...] + features: FeatureMap + + +def load_benchmark_cases(cases_dir: Path) -> list[dict[str, Any]]: + cases: list[dict[str, Any]] = [] + for path in sorted(cases_dir.glob("*.yaml")): + payload = yaml.safe_load(path.read_text(encoding="utf-8")) + if isinstance(payload, dict): + cases.append(payload) + return cases + + +def load_split(split_path: Path) -> dict[str, set[str]]: + payload = yaml.safe_load(split_path.read_text(encoding="utf-8")) + if not isinstance(payload, dict): + raise ValueError("split metadata must be a YAML object") + return { + "train": set(_as_string_list(payload.get("train_case_ids"), "train_case_ids")), + "test": set(_as_string_list(payload.get("test_case_ids"), "test_case_ids")), + } + + +def build_examples( + cases: Iterable[dict[str, Any]], + extractor: QueryFeatureExtractor | None = None, +) -> list[BenchmarkExample]: + feature_extractor = extractor or QueryFeatureExtractor() + examples: list[BenchmarkExample] = [] + for case in cases: + case_id = _required_string(case, "case_id") + sql = _required_string(case, "sql") + labels = tuple(_as_string_list(case.get("expected_rules_to_fire"), "expected_rules_to_fire")) + features = feature_extractor.extract(sql).features + examples.append(BenchmarkExample(case_id=case_id, sql=sql, labels=labels, features=features)) + return examples + + +def split_examples( + examples: Iterable[BenchmarkExample], + split: dict[str, set[str]], +) -> tuple[list[BenchmarkExample], list[BenchmarkExample]]: + train_ids = split["train"] + test_ids = split["test"] + train: list[BenchmarkExample] = [] + test: list[BenchmarkExample] = [] + for example in examples: + if example.case_id in train_ids: + train.append(example) + elif example.case_id in test_ids: + test.append(example) + return train, test + + +def _required_string(payload: dict[str, Any], key: str) -> str: + value = payload.get(key) + if not isinstance(value, str) or not value: + raise ValueError(f"{key} must be a non-empty string") + return value + + +def _as_string_list(value: Any, key: str) -> list[str]: + if not isinstance(value, list) or not all(isinstance(item, str) for item in value): + raise ValueError(f"{key} must be a list of strings") + return value diff --git a/clickadvisor/ml/features.py b/clickadvisor/ml/features.py new file mode 100644 index 00000000..909426d5 --- /dev/null +++ b/clickadvisor/ml/features.py @@ -0,0 +1,428 @@ +from __future__ import annotations + +import logging +import re +from collections.abc import Mapping +from dataclasses import asdict, dataclass + +import sqlglot +import sqlglot.expressions as exp + +from clickadvisor.core.sql_parser import SQLParser + +FeatureMap = dict[str, float] + +_ASYNC_INSERT_RE = re.compile(r"\basync_insert\s*=\s*1\b", re.IGNORECASE) +_WAIT_FLAG_RE = re.compile(r"\bwait_for_async_insert\s*=\s*1\b", re.IGNORECASE) +_INSERT_RE = re.compile(r"\bINSERT\b", re.IGNORECASE) +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +_UINT64_COL_RE = re.compile(r"`?(\w+)`?\s+(?:UInt64|Int64)\b", re.IGNORECASE) +_LOW_CARDINALITY_PARTS = frozenset( + ["type", "status", "category", "flag", "level", "kind", "state", "mode", "priority", "rank", "class"] +) +_TO_TYPE_PREFIXES = ( + "touint", "toint", "tofloat", "tostring", "todecimal", + "todate", "todatetime", "tofixedstring", "touuid", +) +_OR_SAFE_SUFFIXES = ("orzero", "ornull", "ordefault") + + +@dataclass(slots=True) +class QueryFeatures: + """Structured AST/text feature vector for a single SQL query. + + Boolean fields map 1-to-1 to patterns from the deterministic rule engine + but are computed independently so the classifier can generalise beyond + exact rule boundaries. + """ + + # --- count / distinct --- + has_count_distinct: bool + # --- projection --- + has_select_star: bool + # --- table modifiers --- + has_final_modifier: bool + # --- set operators --- + has_union: bool + has_union_all: bool + # --- grouping --- + has_group_by: bool + has_group_by_string_column: bool + # --- having --- + has_having: bool + has_having_without_aggregate: bool + # --- filter functions --- + has_function_on_filter_column: bool + # --- subquery --- + has_subquery: bool + has_subquery_with_orderby_no_limit: bool + has_nested_subquery_filter: bool + # --- predicate patterns --- + has_or_chain_same_column: bool + has_in_with_single_value: bool + # --- quantile --- + has_quantile_exact: bool + # --- cast --- + has_cast: bool + has_cast_without_default: bool + # --- async insert --- + has_async_insert_setting: bool + has_async_insert_without_wait: bool + # --- limit --- + has_limit: bool + has_no_limit: bool + # --- misc predicates --- + has_constant_predicate: bool + has_length_zero_check: bool + # --- numeric shape features --- + table_count: int + column_count_in_select: int + where_clause_depth: int + query_length_chars: int + + def to_vector(self) -> dict[str, float | int]: + """Convert to numeric dict for model input. Bools become 0/1.""" + result: dict[str, float | int] = {} + for field, value in asdict(self).items(): + result[field] = int(value) if isinstance(value, bool) else value + return result + + +def _node_depth(node: exp.Expression) -> int: + """Recursively compute maximum AST depth below *node*.""" + children: list[exp.Expression] = [] + for val in node.args.values(): + if isinstance(val, exp.Expression): + children.append(val) + elif isinstance(val, list): + children.extend(c for c in val if isinstance(c, exp.Expression)) + if not children: + return 0 + return 1 + max(_node_depth(c) for c in children) + + +def _where_depth(ast: sqlglot.Expression) -> int: + max_depth = 0 + for where in ast.find_all(exp.Where): + max_depth = max(max_depth, _node_depth(where.this)) + return max_depth + + +def _has_cast_without_default(ast: sqlglot.Expression) -> bool: + """True when a throwing CAST (no OrDefault/OrZero/OrNull) is present.""" + for cast in ast.find_all(exp.Cast): + if isinstance(cast.this, exp.Column): + return True + for fn in ast.find_all(exp.Anonymous): + name = fn.name.lower() + if name.endswith(_OR_SAFE_SUFFIXES): + continue + if name.startswith(_TO_TYPE_PREFIXES): + if fn.expressions and isinstance(fn.expressions[0], exp.Column): + return True + return False + + +def _has_cast(ast: sqlglot.Expression) -> bool: + """True when any cast or type-conversion function is present.""" + if any(True for _ in ast.find_all(exp.Cast)): + return True + for fn in ast.find_all(exp.Anonymous): + if fn.name.lower().startswith(_TO_TYPE_PREFIXES): + return True + return False + + +def _has_async_insert_without_wait(sql: str) -> bool: + return bool( + _INSERT_RE.search(sql) + and _ASYNC_INSERT_RE.search(sql) + and not _WAIT_FLAG_RE.search(sql) + ) + + +def _has_oversized_uint_candidate(sql: str) -> bool: + if not _CREATE_TABLE_RE.search(sql): + return False + for match in _UINT64_COL_RE.finditer(sql): + col = match.group(1).lower() + parts = col.split("_") + if _LOW_CARDINALITY_PARTS.intersection(parts): + return True + return False + + +class FeatureExtractor: + """Extract QueryFeatures from raw SQL via sqlglot AST with regex fallback.""" + + def __init__(self) -> None: + self._parser = SQLParser() + + def extract(self, sql: str) -> QueryFeatures: + """Parse *sql* via sqlglot (clickhouse dialect) and extract all features. + + If sqlglot raises ParseError the regex fallback path is used and a + warning is logged. The returned QueryFeatures will have False/0 for any + field that cannot be derived from text alone. + """ + try: + ast = self._parser.parse(sql) + except sqlglot.errors.ParseError: + logging.warning( + "AST parse failed for SQL (len=%d), using regex fallback", + len(sql), + ) + return self._regex_fallback(sql) + return self._from_ast(ast, sql) + + def _from_ast(self, ast: sqlglot.Expression, sql: str) -> QueryFeatures: + p = self._parser + top = ast if isinstance(ast, exp.Select) else None + + has_union = any(True for _ in ast.find_all(exp.Union)) + has_union_all = any( + u.args.get("distinct") is False for u in ast.find_all(exp.Union) + ) + has_having = any(True for _ in ast.find_all(exp.Having)) + has_subquery = any(True for _ in ast.find_all(exp.Subquery)) + has_limit = any(True for _ in ast.find_all(exp.Limit)) + # GROUP BY anywhere in the query tree (including subqueries) + has_group_by = any( + s.args.get("group") is not None for s in ast.find_all(exp.Select) + ) + + col_count = len(top.expressions) if top is not None else 0 + t_count = sum(1 for _ in ast.find_all(exp.Table)) + + return QueryFeatures( + has_count_distinct=p.has_count_distinct(ast), + has_select_star=p.has_top_level_select_star(ast), + has_final_modifier=p.has_final_modifier(ast), + has_union=has_union, + has_union_all=has_union_all, + has_group_by=has_group_by, + has_group_by_string_column=p.has_groupby_string_candidate(ast), + has_having=has_having, + has_having_without_aggregate=p.has_having_without_agg(ast), + has_function_on_filter_column=( + p.has_todate_equality(ast) + or p.has_date_part_equality(ast) + or p.has_interval_start_equality(ast) + or p.has_redundant_cast(ast) + ), + has_subquery=has_subquery, + has_subquery_with_orderby_no_limit=p.has_orderby_without_limit_in_subquery(ast), + has_nested_subquery_filter=p.has_subquery_filter_pushdown(ast), + has_or_chain_same_column=p.has_disjunction_chain(ast), + has_in_with_single_value=p.has_in_singleton(ast), + has_quantile_exact=p.has_quantile_exact(ast), + has_cast=_has_cast(ast), + has_cast_without_default=_has_cast_without_default(ast), + has_async_insert_setting=bool(_ASYNC_INSERT_RE.search(sql)), + has_async_insert_without_wait=_has_async_insert_without_wait(sql), + has_limit=has_limit, + has_no_limit=p.has_missing_limit(ast), + has_constant_predicate=p.has_constant_predicate(ast), + has_length_zero_check=p.has_length_empty_pattern(ast), + table_count=t_count, + column_count_in_select=col_count, + where_clause_depth=_where_depth(ast), + query_length_chars=len(sql), + ) + + def _regex_fallback(self, sql: str) -> QueryFeatures: + has_union = bool(re.search(r"\bUNION\b", sql, re.I)) + has_union_all = bool(re.search(r"\bUNION\s+ALL\b", sql, re.I)) + has_limit = bool(re.search(r"\bLIMIT\b", sql, re.I)) + has_cast = bool( + re.search(r"\bCAST\s*\(", sql, re.I) + or any(re.search(rf"\b{p}", sql, re.I) for p in _TO_TYPE_PREFIXES) + ) + return QueryFeatures( + has_count_distinct=bool(re.search(r"COUNT\s*\(\s*DISTINCT", sql, re.I)), + has_select_star=bool(re.search(r"\bSELECT\s+\*", sql, re.I)), + has_final_modifier=bool(re.search(r"\bFINAL\b", sql, re.I)), + has_union=has_union, + has_union_all=has_union_all, + has_group_by=bool(re.search(r"\bGROUP\s+BY\b", sql, re.I)), + has_group_by_string_column=False, + has_having=bool(re.search(r"\bHAVING\b", sql, re.I)), + has_having_without_aggregate=False, + has_function_on_filter_column=bool( + re.search(r"\btoDate\s*\(|\btoYYYYMM\s*\(|\btoStartOf", sql, re.I) + ), + has_subquery=bool(re.search(r"\bSELECT\b.*\bSELECT\b", sql, re.I | re.DOTALL)), + has_subquery_with_orderby_no_limit=False, + has_nested_subquery_filter=False, + has_or_chain_same_column=False, + has_in_with_single_value=bool(re.search(r"\bIN\s*\(\s*'[^']*'\s*\)", sql, re.I)), + has_quantile_exact=bool(re.search(r"\bquantileExact\b", sql, re.I)), + has_cast=has_cast, + has_cast_without_default=has_cast, + has_async_insert_setting=bool(_ASYNC_INSERT_RE.search(sql)), + has_async_insert_without_wait=_has_async_insert_without_wait(sql), + has_limit=has_limit, + has_no_limit=False, + has_constant_predicate=bool( + re.search(r"\bWHERE\b.*\bTRUE\b", sql, re.I | re.DOTALL) + or re.search(r"\b1\s*=\s*1\b", sql, re.I) + ), + has_length_zero_check=bool(re.search(r"\blength\s*\(", sql, re.I)), + table_count=0, + column_count_in_select=0, + where_clause_depth=0, + query_length_chars=len(sql), + ) + + +# --------------------------------------------------------------------------- +# Legacy classes — kept for backward compatibility with clickadvisor/ml/dataset.py +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True, slots=True) +class FeatureVector: + sql: str + features: FeatureMap + + +class QueryFeatureExtractor: + """Extract deterministic AST/text features for classical ML baselines.""" + + def __init__(self) -> None: + self.parser = SQLParser() + + def extract(self, sql: str) -> FeatureVector: + features = self._base_text_features(sql) + try: + ast = self.parser.parse(sql) + except sqlglot.errors.ParseError: + features["parse_error"] = 1.0 + features.update(self._regex_features(sql)) + return FeatureVector(sql=sql, features=features) + + features["parse_error"] = 0.0 + features.update(self._ast_shape_features(ast)) + features.update(self._rule_pattern_features(ast, sql)) + return FeatureVector(sql=sql, features=features) + + def _base_text_features(self, sql: str) -> FeatureMap: + normalized = sql.strip() + return { + "sql_length_chars": float(len(normalized)), + "sql_line_count": float(max(1, normalized.count("\n") + 1)), + "contains_comment": float("--" in normalized or "/*" in normalized), + "contains_settings_clause": float(bool(re.search(r"\bsettings\b", normalized, re.I))), + } + + def _ast_shape_features(self, ast: sqlglot.Expression) -> FeatureMap: + top_level_select = isinstance(ast, exp.Select) + group = ast.args.get("group") if top_level_select else None + return { + "is_select": float(top_level_select), + "is_insert": float(isinstance(ast, exp.Insert)), + "is_create": float(isinstance(ast, exp.Create)), + "select_count": float(sum(1 for _ in ast.find_all(exp.Select))), + "table_count": float(sum(1 for _ in ast.find_all(exp.Table))), + "join_count": float(sum(1 for _ in ast.find_all(exp.Join))), + "subquery_count": float(sum(1 for _ in ast.find_all(exp.Subquery))), + "where_count": float(sum(1 for _ in ast.find_all(exp.Where))), + "having_count": float(sum(1 for _ in ast.find_all(exp.Having))), + "order_count": float(sum(1 for _ in ast.find_all(exp.Order))), + "limit_count": float(sum(1 for _ in ast.find_all(exp.Limit))), + "groupby_key_count": float(len(group.expressions) if isinstance(group, exp.Group) else 0), + "function_call_count": float(self._function_call_count(ast)), + "aggregate_call_count": float(self._aggregate_call_count(ast)), + "literal_count": float(sum(1 for _ in ast.find_all(exp.Literal))), + } + + def _rule_pattern_features(self, ast: sqlglot.Expression, sql: str) -> FeatureMap: + return { + "has_count_distinct": float(self.parser.has_count_distinct(ast)), + "has_quantile_exact": float(self.parser.has_quantile_exact(ast)), + "has_count_star_distinct_subquery": float( + self.parser.has_count_star_distinct_subquery(ast) + ), + "has_todate_equality": float(self.parser.has_todate_equality(ast)), + "has_date_part_equality": float(self.parser.has_date_part_equality(ast)), + "has_interval_start_equality": float(self.parser.has_interval_start_equality(ast)), + "has_function_on_filter_column": float(self._has_function_on_filter_column(ast)), + "has_redundant_cast": float(self.parser.has_redundant_cast(ast)), + "has_singleton_in": float(self.parser.has_in_singleton(ast)), + "has_disjunction_chain": float(self.parser.has_disjunction_chain(ast)), + "has_having_without_aggregate": float(self.parser.has_having_without_agg(ast)), + "has_constant_predicate": float(self.parser.has_constant_predicate(ast)), + "has_length_empty_pattern": float(self.parser.has_length_empty_pattern(ast)), + "has_groupby_string_candidate": float(self.parser.has_groupby_string_candidate(ast)), + "has_distinct_after_groupby": float(self.parser.has_distinct_after_groupby(ast)), + "has_orderby_without_limit_subquery": float( + self.parser.has_orderby_without_limit_in_subquery(ast) + ), + "has_subquery_filter_pushdown": float(self.parser.has_subquery_filter_pushdown(ast)), + "has_top_level_select_star": float(self.parser.has_top_level_select_star(ast)), + "has_missing_limit": float(self.parser.has_missing_limit(ast)), + "has_union_not_all": float(self.parser.has_union_not_all(ast)), + "has_final_modifier": float(self.parser.has_final_modifier(ast)), + "has_throwing_cast": float(self._has_throwing_cast(ast)), + "has_async_insert_without_wait": float(self._has_async_insert_without_wait(sql)), + "has_oversized_int_candidate": float(self._has_oversized_int_candidate(sql)), + } + + def _regex_features(self, sql: str) -> FeatureMap: + return { + "has_async_insert_without_wait": float(self._has_async_insert_without_wait(sql)), + "has_oversized_int_candidate": float(self._has_oversized_int_candidate(sql)), + } + + def _function_call_count(self, ast: sqlglot.Expression) -> int: + function_types = (exp.Anonymous, exp.Cast, exp.Count, exp.Sum, exp.Avg, exp.Min, exp.Max) + return sum(1 for node in ast.walk() if isinstance(node, function_types)) + + def _aggregate_call_count(self, ast: sqlglot.Expression) -> int: + aggregate_types = (exp.Count, exp.Sum, exp.Avg, exp.Min, exp.Max) + return sum(1 for node in ast.walk() if isinstance(node, aggregate_types)) + + def _has_function_on_filter_column(self, ast: sqlglot.Expression) -> bool: + return ( + self.parser.has_todate_equality(ast) + or self.parser.has_date_part_equality(ast) + or self.parser.has_interval_start_equality(ast) + or self.parser.has_redundant_cast(ast) + ) + + def _has_throwing_cast(self, ast: sqlglot.Expression) -> bool: + for cast in ast.find_all(exp.Cast): + if isinstance(cast.this, exp.Column): + return True + for function in ast.find_all(exp.Anonymous): + name = function.name.lower() + if name.endswith(("orzero", "ornull", "ordefault")): + continue + if name.startswith(("touint", "toint", "tofloat", "tostring", "todecimal", "todate")): + return bool(function.expressions and isinstance(function.expressions[0], exp.Column)) + return False + + def _has_async_insert_without_wait(self, sql: str) -> bool: + return bool( + re.search(r"\binsert\b", sql, re.I) + and re.search(r"\basync_insert\s*=\s*1\b", sql, re.I) + and not re.search(r"\bwait_for_async_insert\s*=\s*1\b", sql, re.I) + ) + + def _has_oversized_int_candidate(self, sql: str) -> bool: + if not re.search(r"\bcreate\s+table\b", sql, re.I): + return False + return bool( + re.search( + r"\b(?:event_type|order_status|log_level|session_type|account_status|payment_type)\s+U?Int64\b", + sql, + re.I, + ) + ) + + +def ordered_feature_names(rows: list[Mapping[str, float]]) -> list[str]: + names: set[str] = set() + for row in rows: + names.update(row) + return sorted(names) diff --git a/clickadvisor/rules/detectors/D015_optimize_table_final.py b/clickadvisor/rules/detectors/D015_optimize_table_final.py new file mode 100644 index 00000000..daf5aa6b --- /dev/null +++ b/clickadvisor/rules/detectors/D015_optimize_table_final.py @@ -0,0 +1,46 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_OPTIMIZE_FINAL_RE = re.compile( + r"\bOPTIMIZE\s+TABLE\s+[\w.`\"]+.*?\bFINAL\b", + re.IGNORECASE | re.DOTALL, +) + + +class D015OptimizeTableFinal(Rule): + rule_id = "D-015" + name = "optimize_table_final" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + if not _OPTIMIZE_FINAL_RE.search(context.sql): + return None + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="high", + description=( + "OPTIMIZE TABLE ... FINAL перезаписывает все data parts в один, " + "игнорируя лимиты слияния. На больших таблицах — часы I/O и риск OOM." + ), + suggestion=( + "Используйте FINAL в SELECT вместо OPTIMIZE FINAL: " + "SELECT ... FROM t FINAL WHERE ... " + "Не запускайте OPTIMIZE FINAL в продакшене." + ), + example_before="OPTIMIZE TABLE events FINAL", + example_after="SELECT event_id FROM events FINAL WHERE dt >= today() - 7", + explain_why=( + "OPTIMIZE FINAL игнорирует max_bytes_to_merge_at_max_space_in_pool " + "и сливает части в один. Может создать часть в сотни ГБ, " + "которую невозможно разбить обратно." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D016_alter_table_mutation.py b/clickadvisor/rules/detectors/D016_alter_table_mutation.py new file mode 100644 index 00000000..f78990c2 --- /dev/null +++ b/clickadvisor/rules/detectors/D016_alter_table_mutation.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_MUTATION_RE = re.compile( + r"\bALTER\s+TABLE\s+[\w.`\"]+\s+(DELETE|UPDATE)\b", + re.IGNORECASE, +) + + +class D016AlterTableMutation(Rule): + rule_id = "D-016" + name = "alter_table_mutation" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _MUTATION_RE.search(context.sql) + if not m: + return None + op = m.group(1).upper() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="high", + description=( + f"ALTER TABLE ... {op} (мутация) перезаписывает весь затронутый data part. " + "Даже изменение одной строки вызывает перезапись сотен ГБ. " + "Нельзя откатить после запуска." + ), + suggestion=( + "Для удаления данных используйте lightweight DELETE (CH >= 22.8) " + "или DROP PARTITION. " + "Для обновлений рассмотрите ReplacingMergeTree или CollapsingMergeTree." + ), + example_before="ALTER TABLE events DELETE WHERE dt < '2023-01-01'", + example_after="DELETE FROM events WHERE dt < '2023-01-01' -- CH >= 22.8", + explain_why=( + "Мутации в ClickHouse — асинхронные фоновые процессы, перезаписывающие " + "целые data parts. Продолжают выполняться после рестарта сервера. " + "При подзапросах (x IN (SELECT ...)) нагрузка на CPU/RAM многократно возрастает." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D017_nullable_column_in_ddl.py b/clickadvisor/rules/detectors/D017_nullable_column_in_ddl.py new file mode 100644 index 00000000..cad0f193 --- /dev/null +++ b/clickadvisor/rules/detectors/D017_nullable_column_in_ddl.py @@ -0,0 +1,54 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +_NULLABLE_COL_RE = re.compile( + r"`?(\w+)`?\s+Nullable\s*\(\s*(\w+)", + re.IGNORECASE, +) + + +class D017NullableColumnInDDL(Rule): + rule_id = "D-017" + name = "nullable_column_in_ddl" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + m = _NULLABLE_COL_RE.search(sql) + if not m: + return None + column = m.group(1) + inner_type = m.group(2) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="medium", + description=( + f"Колонка '{column}' объявлена как Nullable({inner_type}). " + "Nullable создаёт дополнительный файл UInt8-маски, увеличивая хранение " + "и замедляя обработку. Nullable-колонки не могут быть в table indexes." + ), + suggestion=( + f"Рассмотрите замену на {inner_type} DEFAULT 0 (для числовых) " + f"или {inner_type} DEFAULT '' (для строк), если NULL не несёт " + "семантической нагрузки." + ), + example_before=f"CREATE TABLE t ({column} Nullable({inner_type})) ENGINE = MergeTree ORDER BY tuple()", + example_after=f"CREATE TABLE t ({column} {inner_type} DEFAULT 0) ENGINE = MergeTree ORDER BY tuple()", + explain_why=( + "Nullable column consumes additional storage space и almost always " + "negatively affects performance (ClickHouse docs). " + "Nullable type field cannot be included in table indexes." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D018_deprecated_ngrambf_tokenbf_index.py b/clickadvisor/rules/detectors/D018_deprecated_ngrambf_tokenbf_index.py new file mode 100644 index 00000000..a6102566 --- /dev/null +++ b/clickadvisor/rules/detectors/D018_deprecated_ngrambf_tokenbf_index.py @@ -0,0 +1,51 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_DEPRECATED_INDEX_RE = re.compile( + r"\bINDEX\s+(\w+)\s+.*?\bTYPE\s+(ngrambf_v1|tokenbf_v1)\b", + re.IGNORECASE | re.DOTALL, +) + + +class D018DeprecatedNgramBFIndex(Rule): + rule_id = "D-018" + name = "deprecated_ngrambf_tokenbf_index" + tier = "detector" + ch_version_introduced = "0.720" + ch_version_deprecated = "26.2" + + def check(self, context: QueryContext) -> Finding | None: + m = _DEPRECATED_INDEX_RE.search(context.sql) + if not m: + return None + index_name = m.group(1) + index_type = m.group(2) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="medium", + description=( + f"Skip index '{index_name}' использует TYPE {index_type}, " + "который устарел в ClickHouse >= 26.2. " + "Замените на text-индекс (инвертированный индекс)." + ), + suggestion=( + f"Замените на: INDEX {index_name} TYPE text GRANULARITY 1. " + "Text-индекс детерминирован, поддерживает многотерминный поиск " + "и не требует настройки размера n-gram/токенов." + ), + example_before=f"INDEX {index_name} message TYPE {index_type}(4, 1024, 2, 0) GRANULARITY 4", + example_after=f"INDEX {index_name} message TYPE text GRANULARITY 1", + explain_why=( + f"{index_type} использует вероятностный Bloom filter с false positive rate. " + "Text-индекс (инвертированный) более точен, масштабируется лучше " + "и поддерживает детерминированный поиск по токенам." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D019_set_zero_unlimited_skip_index.py b/clickadvisor/rules/detectors/D019_set_zero_unlimited_skip_index.py new file mode 100644 index 00000000..b6071141 --- /dev/null +++ b/clickadvisor/rules/detectors/D019_set_zero_unlimited_skip_index.py @@ -0,0 +1,48 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_SET_ZERO_RE = re.compile( + r"\bINDEX\s+(\w+)\s+.*?\bTYPE\s+set\s*\(\s*0\s*\)", + re.IGNORECASE | re.DOTALL, +) + + +class D019SetZeroUnlimitedSkipIndex(Rule): + rule_id = "D-019" + name = "set_zero_unlimited_skip_index" + tier = "detector" + ch_version_introduced = "0.720" + + def check(self, context: QueryContext) -> Finding | None: + m = _SET_ZERO_RE.search(context.sql) + if not m: + return None + index_name = m.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"Skip index '{index_name}' использует TYPE set(0) — неограниченный размер набора. " + "На высококардинальных колонках индекс не сможет пропускать блоки " + "и только замедлит слияния." + ), + suggestion=( + f"Задайте явный лимит: INDEX {index_name} TYPE set(100) GRANULARITY 4. " + "Используйте set(N) только для колонок с малым числом уникальных значений в грануле." + ), + example_before=f"INDEX {index_name} status TYPE set(0) GRANULARITY 4", + example_after=f"INDEX {index_name} status TYPE set(100) GRANULARITY 4", + explain_why=( + "set(0) означает неограниченный размер набора на гранулу. " + "При высокой кардинальности набор содержит все значения " + "и не может пропустить ни один блок, тратя память при слияниях." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D020_partition_by_non_date.py b/clickadvisor/rules/detectors/D020_partition_by_non_date.py new file mode 100644 index 00000000..190144e0 --- /dev/null +++ b/clickadvisor/rules/detectors/D020_partition_by_non_date.py @@ -0,0 +1,70 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +# Matches PARTITION BY +_PARTITION_BY_RE = re.compile( + r"\bPARTITION\s+BY\s+([^\n]+?)(?=\s*(?:ORDER\s+BY|PRIMARY\s+KEY|SETTINGS|ENGINE|\Z))", + re.IGNORECASE | re.DOTALL, +) +# Date functions that are acceptable as PARTITION BY +_DATE_FUNCTIONS = frozenset([ + "toyyyymm", "tostartofmonth", "tostartofyear", "tostartofquarter", + "tostartofweek", "tostartofday", "toyear", "todate", "todate32", + "tostartofinterval", +]) + + +def _has_date_function(expr: str) -> bool: + low = expr.lower() + return any(fn in low for fn in _DATE_FUNCTIONS) + + +class D020PartitionByNonDate(Rule): + rule_id = "D-020" + name = "partition_by_non_date_column" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + m = _PARTITION_BY_RE.search(sql) + if not m: + return None + partition_expr = m.group(1).strip() + if _has_date_function(partition_expr): + return None + # Skip tuple() — no partition + if re.match(r"tuple\s*\(\s*\)", partition_expr, re.IGNORECASE): + return None + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="medium", + description=( + f"PARTITION BY '{partition_expr}' использует выражение без date-функции. " + "Высококардинальные ключи партиционирования приводят к Part Explosion " + "и деградации INSERT/мерджей." + ), + suggestion=( + "Используйте date-функцию для партиционирования: PARTITION BY toYYYYMM(ts) " + "или PARTITION BY toStartOfMonth(ts). " + "Убедитесь, что кардинальность ключа не превышает сотни уникальных значений." + ), + example_before=f"PARTITION BY {partition_expr}", + example_after="PARTITION BY toYYYYMM(ts)", + explain_why=( + "Partitioning primarily is a data management feature, not a query optimization tool. " + "High-cardinality partition key leads to too many parts, " + "degrading insert performance and merge operations." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D021_select_star_in_mv.py b/clickadvisor/rules/detectors/D021_select_star_in_mv.py new file mode 100644 index 00000000..102d944a --- /dev/null +++ b/clickadvisor/rules/detectors/D021_select_star_in_mv.py @@ -0,0 +1,46 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches CREATE MATERIALIZED VIEW ... AS SELECT * +_MV_SELECT_STAR_RE = re.compile( + r"\bCREATE\s+MATERIALIZED\s+VIEW\b.*?\bAS\s+SELECT\s+\*", + re.IGNORECASE | re.DOTALL, +) + + +class D021SelectStarInMV(Rule): + rule_id = "D-021" + name = "select_star_in_mv_create" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + if not _MV_SELECT_STAR_RE.search(context.sql): + return None + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="medium", + description=( + "CREATE MATERIALIZED VIEW использует SELECT *, что делает MV хрупким: " + "изменения схемы источника (добавление/удаление столбца) сломают MV." + ), + suggestion=( + "Замените SELECT * на явный список нужных колонок. " + "Используйте MIN набор колонок для целевой агрегации." + ), + example_before="CREATE MATERIALIZED VIEW mv AS SELECT * FROM source", + example_after="CREATE MATERIALIZED VIEW mv AS SELECT user_id, count() AS cnt FROM source GROUP BY user_id", + explain_why=( + "SELECT * в MV фиксирует список столбцов на момент создания. " + "Новые столбцы источника не попадут в MV; " + "удалённые столбцы сломают MV при следующем INSERT." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D022_delete_without_where.py b/clickadvisor/rules/detectors/D022_delete_without_where.py new file mode 100644 index 00000000..8a36c98c --- /dev/null +++ b/clickadvisor/rules/detectors/D022_delete_without_where.py @@ -0,0 +1,57 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# DELETE FROM table without WHERE +# Matches: DELETE FROM tablename (with optional whitespace at end) +_DELETE_NO_WHERE_RE = re.compile( + r"\bDELETE\s+FROM\s+[\w.`\"]+\s*(?:;|\Z)", + re.IGNORECASE, +) +# Also check that no WHERE follows +_DELETE_FROM_RE = re.compile(r"\bDELETE\s+FROM\s+[\w.`\"]+", re.IGNORECASE) +_WHERE_RE = re.compile(r"\bWHERE\b", re.IGNORECASE) + + +class D022DeleteWithoutWhere(Rule): + rule_id = "D-022" + name = "delete_from_without_where" + tier = "detector" + ch_version_introduced = "22.8" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql.strip() + if not _DELETE_FROM_RE.search(sql): + return None + if _WHERE_RE.search(sql): + return None + # DELETE FROM without WHERE found + m = _DELETE_FROM_RE.search(sql) + table = m.group(0).replace("DELETE FROM", "").strip() if m else "table" + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="high", + description=( + f"DELETE FROM {table} без WHERE-условия удаляет ВСЕ строки таблицы. " + "Это деструктивная и медленная операция для ClickHouse." + ), + suggestion=( + f"Если цель — очистить таблицу, используйте TRUNCATE TABLE {table} " + "(намного быстрее — удаляет все parts метаданных). " + "Если DELETE задуман для части строк, добавьте WHERE-предикат." + ), + example_before=f"DELETE FROM {table}", + example_after=f"TRUNCATE TABLE {table} -- если нужно очистить всё", + explain_why=( + "DELETE FROM без WHERE помечает все строки как deleted. " + "Фактическое удаление происходит при merge. " + "TRUNCATE TABLE мгновенно удаляет все parts и несравнимо быстрее." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D023_mv_without_to.py b/clickadvisor/rules/detectors/D023_mv_without_to.py new file mode 100644 index 00000000..6483f10c --- /dev/null +++ b/clickadvisor/rules/detectors/D023_mv_without_to.py @@ -0,0 +1,45 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_MV_RE = re.compile(r"\bCREATE\s+MATERIALIZED\s+VIEW\b", re.IGNORECASE) +_TO_RE = re.compile(r"\bTO\s+\w", re.IGNORECASE) + + +class D023MVWithoutTo(Rule): + rule_id = "D-023" + name = "mv_without_to_clause" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _MV_RE.search(sql): + return None + if _TO_RE.search(sql): + return None + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + "CREATE MATERIALIZED VIEW без TO создаёт скрытую целевую таблицу .inner.mvname. " + "Управление схемой, ENGINE и индексами такой таблицы значительно сложнее." + ), + suggestion=( + "Используйте явный TO-синтаксис: сначала создайте target table, " + "затем CREATE MATERIALIZED VIEW mv TO target AS SELECT ..." + ), + example_before="CREATE MATERIALIZED VIEW mv AS SELECT user_id, count() FROM events GROUP BY user_id", + example_after="CREATE TABLE mv_target ... ENGINE=...; CREATE MATERIALIZED VIEW mv TO mv_target AS ...", + explain_why=( + "TO-синтаксис позволяет явно управлять схемой target table: " + "менять ENGINE, добавлять индексы, делать ALTER без пересоздания MV." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D024_mv_with_populate.py b/clickadvisor/rules/detectors/D024_mv_with_populate.py new file mode 100644 index 00000000..558e70c0 --- /dev/null +++ b/clickadvisor/rules/detectors/D024_mv_with_populate.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_MV_POPULATE_RE = re.compile( + r"\bCREATE\s+MATERIALIZED\s+VIEW\b.*?\bPOPULATE\b", + re.IGNORECASE | re.DOTALL, +) + + +class D024MVWithPopulate(Rule): + rule_id = "D-024" + name = "mv_with_populate" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + if not _MV_POPULATE_RE.search(context.sql): + return None + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="medium", + description=( + "CREATE MATERIALIZED VIEW ... POPULATE создаёт окно потери данных: " + "строки, вставленные ВО ВРЕМЯ backfill-а, не попадут в MV. " + "POPULATE не атомарен." + ), + suggestion=( + "Создайте MV без POPULATE, затем вручную залейте исторические данные: " + "INSERT INTO target SELECT ... FROM source." + ), + example_before="CREATE MATERIALIZED VIEW mv TO target AS SELECT ... FROM src POPULATE", + example_after=( + "CREATE MATERIALIZED VIEW mv TO target AS SELECT ... FROM src;\n" + "-- Затем: INSERT INTO target SELECT ... FROM src" + ), + explain_why=( + "POPULATE читает данные из источника в фоне и одновременно " + "начинает триггериться на новые INSERT. " + "Данные, вставленные ВО ВРЕМЯ backfill-а, теряются." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/D025_mv_with_join.py b/clickadvisor/rules/detectors/D025_mv_with_join.py new file mode 100644 index 00000000..c1f5e7cb --- /dev/null +++ b/clickadvisor/rules/detectors/D025_mv_with_join.py @@ -0,0 +1,47 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_MV_RE = re.compile(r"\bCREATE\s+MATERIALIZED\s+VIEW\b", re.IGNORECASE) +_JOIN_IN_BODY_RE = re.compile(r"\b(?:INNER|LEFT|RIGHT|FULL|CROSS)?\s*JOIN\b", re.IGNORECASE) + + +class D025MVWithJoin(Rule): + rule_id = "D-025" + name = "mv_with_join_in_select" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _MV_RE.search(sql): + return None + if not _JOIN_IN_BODY_RE.search(sql): + return None + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="medium", + description=( + "CREATE MATERIALIZED VIEW содержит JOIN в SELECT. " + "При каждом INSERT в левую таблицу выполняется JOIN со всей правой таблицей. " + "Это критически бьёт по производительности вставки." + ), + suggestion=( + "Рассмотрите использование Dictionary, engine=Join или engine=Set для правой таблицы. " + "Они хранятся в памяти и выполняют lookup намного быстрее полного JOIN-а." + ), + example_before="CREATE MATERIALIZED VIEW mv TO t AS SELECT e.uid, u.name FROM events e LEFT JOIN users u ON e.uid = u.uid", + example_after="CREATE MATERIALIZED VIEW mv TO t AS SELECT uid, dictGet('users_dict', 'name', uid) FROM events", + explain_why=( + "MV срабатывает как AFTER INSERT TRIGGER на левую таблицу. " + "JOIN выполняется со ВСЕЙ правой таблицей при каждом INSERT-е, " + "что может увеличить latency вставки в разы." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/detectors/__init__.py b/clickadvisor/rules/detectors/__init__.py index bba29548..f92ccf42 100644 --- a/clickadvisor/rules/detectors/__init__.py +++ b/clickadvisor/rules/detectors/__init__.py @@ -2,5 +2,36 @@ from clickadvisor.rules.detectors.D004_missing_limit import D004MissingLimit from clickadvisor.rules.detectors.D007_final_modifier import D007FinalModifier from clickadvisor.rules.detectors.D014_async_insert_no_wait import D014AsyncInsertNoWait +from clickadvisor.rules.detectors.D015_optimize_table_final import D015OptimizeTableFinal +from clickadvisor.rules.detectors.D016_alter_table_mutation import D016AlterTableMutation +from clickadvisor.rules.detectors.D017_nullable_column_in_ddl import D017NullableColumnInDDL +from clickadvisor.rules.detectors.D018_deprecated_ngrambf_tokenbf_index import ( + D018DeprecatedNgramBFIndex, +) +from clickadvisor.rules.detectors.D019_set_zero_unlimited_skip_index import ( + D019SetZeroUnlimitedSkipIndex, +) +from clickadvisor.rules.detectors.D020_partition_by_non_date import D020PartitionByNonDate +from clickadvisor.rules.detectors.D021_select_star_in_mv import D021SelectStarInMV +from clickadvisor.rules.detectors.D022_delete_without_where import D022DeleteWithoutWhere +from clickadvisor.rules.detectors.D023_mv_without_to import D023MVWithoutTo +from clickadvisor.rules.detectors.D024_mv_with_populate import D024MVWithPopulate +from clickadvisor.rules.detectors.D025_mv_with_join import D025MVWithJoin -__all__ = ["D003SelectStar", "D004MissingLimit", "D007FinalModifier", "D014AsyncInsertNoWait"] +__all__ = [ + "D003SelectStar", + "D004MissingLimit", + "D007FinalModifier", + "D014AsyncInsertNoWait", + "D015OptimizeTableFinal", + "D016AlterTableMutation", + "D017NullableColumnInDDL", + "D018DeprecatedNgramBFIndex", + "D019SetZeroUnlimitedSkipIndex", + "D020PartitionByNonDate", + "D021SelectStarInMV", + "D022DeleteWithoutWhere", + "D023MVWithoutTo", + "D024MVWithPopulate", + "D025MVWithJoin", +] diff --git a/clickadvisor/rules/registry.py b/clickadvisor/rules/registry.py index 5f14f0b0..7957c3ba 100644 --- a/clickadvisor/rules/registry.py +++ b/clickadvisor/rules/registry.py @@ -11,6 +11,17 @@ D004MissingLimit, D007FinalModifier, D014AsyncInsertNoWait, + D015OptimizeTableFinal, + D016AlterTableMutation, + D017NullableColumnInDDL, + D018DeprecatedNgramBFIndex, + D019SetZeroUnlimitedSkipIndex, + D020PartitionByNonDate, + D021SelectStarInMV, + D022DeleteWithoutWhere, + D023MVWithoutTo, + D024MVWithPopulate, + D025MVWithJoin, ) from clickadvisor.rules.tier1 import ( R001CountDistinct, @@ -33,6 +44,48 @@ R018UnionToUnionAll, R019UintNarrowing, R020CastOrDefault, + R021DateTime64ZeroToDateTime, + R022FloatMonetary, + R023StringDatetimeColumn, + R024StringIPColumn, + R025OrderByTupleNoPK, + R026SumCaseToCountIf, + R027SumCaseColToSumIf, + R028CoalesceToIfNull, + R029LowerLikeToILike, + R030NotInSingleton, + R031StringUUIDColumn, + R032Int8BooleanColumn, + R033MaxCaseToMaxIf, + R034MinCaseToMinIf, + R035AvgCaseToAvgIf, + R036NestedIfToMultiIf, + R037EmptyStringEqToEmpty, + R038NonEmptyStringNeqToNotEmpty, + R039LengthGteOneToNotEmpty, + R040TodateComparisonToDatetime, + R041StringCodeColumn, + R042GroupArrayNoLimit, + R043HavingCountGtZero, + R044ToDateTimeToDateToStartOfDay, + R045LikeWithoutWildcardsToEq, + R046NotEmptyToNotEmpty, + R047PositionToLike, + R048PositionCIToILike, + R049SumIfOneToCountIf, + R050ToYYYYMMComparison, + R051DateTruncToNative, + R052FormatDateTimeYMD, + R053ArrayReduceSum, + R054ArrayReduceMax, + R055ArrayReduceMin, + R056ExtractToNative, + R057ExtractMonthToMonth, + R058ExtractDayToDayOfMonth, + R059ExtractHourToHour, + R060HasAndHasToHasAll, + R061HasOrHasToHasAny, + R062ArrayCountZeroToNotHas, ) DEFAULT_CARDS_DIR = Path("docs/rules/cards") @@ -57,10 +110,63 @@ R018UnionToUnionAll(), R019UintNarrowing(), R020CastOrDefault(), + R021DateTime64ZeroToDateTime(), + R022FloatMonetary(), + R023StringDatetimeColumn(), + R024StringIPColumn(), + R025OrderByTupleNoPK(), + R026SumCaseToCountIf(), + R027SumCaseColToSumIf(), + R028CoalesceToIfNull(), + R029LowerLikeToILike(), + R030NotInSingleton(), + R031StringUUIDColumn(), + R032Int8BooleanColumn(), D003SelectStar(), D004MissingLimit(), D007FinalModifier(), D014AsyncInsertNoWait(), + D015OptimizeTableFinal(), + D016AlterTableMutation(), + D017NullableColumnInDDL(), + D018DeprecatedNgramBFIndex(), + D019SetZeroUnlimitedSkipIndex(), + D020PartitionByNonDate(), + D021SelectStarInMV(), + D022DeleteWithoutWhere(), + D023MVWithoutTo(), + D024MVWithPopulate(), + D025MVWithJoin(), + R033MaxCaseToMaxIf(), + R034MinCaseToMinIf(), + R035AvgCaseToAvgIf(), + R036NestedIfToMultiIf(), + R037EmptyStringEqToEmpty(), + R038NonEmptyStringNeqToNotEmpty(), + R039LengthGteOneToNotEmpty(), + R040TodateComparisonToDatetime(), + R041StringCodeColumn(), + R042GroupArrayNoLimit(), + R043HavingCountGtZero(), + R044ToDateTimeToDateToStartOfDay(), + R045LikeWithoutWildcardsToEq(), + R046NotEmptyToNotEmpty(), + R047PositionToLike(), + R048PositionCIToILike(), + R049SumIfOneToCountIf(), + R050ToYYYYMMComparison(), + R051DateTruncToNative(), + R052FormatDateTimeYMD(), + R053ArrayReduceSum(), + R054ArrayReduceMax(), + R055ArrayReduceMin(), + R056ExtractToNative(), + R057ExtractMonthToMonth(), + R058ExtractDayToDayOfMonth(), + R059ExtractHourToHour(), + R060HasAndHasToHasAll(), + R061HasOrHasToHasAny(), + R062ArrayCountZeroToNotHas(), ] RULE_REGISTRY: dict[str, Rule] = {rule.rule_id: rule for rule in RULES} diff --git a/clickadvisor/rules/tier1/R021_datetime64_zero.py b/clickadvisor/rules/tier1/R021_datetime64_zero.py new file mode 100644 index 00000000..8b997981 --- /dev/null +++ b/clickadvisor/rules/tier1/R021_datetime64_zero.py @@ -0,0 +1,51 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +_DT64_ZERO_RE = re.compile( + r"`?(\w+)`?\s+DateTime64\s*\(\s*0\s*(?:,\s*'[^']*')?\s*\)", + re.IGNORECASE, +) + + +class R021DateTime64ZeroToDateTime(Rule): + rule_id = "R-021" + name = "datetime64_zero_to_datetime" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + m = _DT64_ZERO_RE.search(sql) + if not m: + return None + column = m.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"Колонка '{column}' объявлена как DateTime64(0), " + "что семантически эквивалентно DateTime. " + "DateTime занимает 4 байта (UInt32) вместо 8 байт (Int64) DateTime64." + ), + suggestion=( + f"Замените DateTime64(0) на DateTime для колонки '{column}'. " + "Диапазон DateTime: 1970-01-01 — 2106-02-07 (UInt32 epoch)." + ), + example_before=f"CREATE TABLE t ({column} DateTime64(0)) ENGINE = MergeTree ORDER BY {column}", + example_after=f"CREATE TABLE t ({column} DateTime) ENGINE = MergeTree ORDER BY {column}", + explain_why=( + "DateTime64(0) хранит эпох-секунды как Int64 (8 байт). " + "DateTime хранит как UInt32 (4 байта). При precision=0 семантика совпадает." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R022_float_monetary.py b/clickadvisor/rules/tier1/R022_float_monetary.py new file mode 100644 index 00000000..52b9a0c7 --- /dev/null +++ b/clickadvisor/rules/tier1/R022_float_monetary.py @@ -0,0 +1,62 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +_FLOAT_COL_RE = re.compile( + r"`?(\w+)`?\s+(Float32|Float64)\b", + re.IGNORECASE, +) +_MONETARY_PARTS = frozenset( + ["amount", "price", "cost", "revenue", "fee", "total", "balance", "tax", + "payment", "charge", "salary", "income", "expense", "profit", "discount", + "rate", "value"] +) + + +def _is_monetary_name(name: str) -> bool: + parts = name.lower().split("_") + return bool(_MONETARY_PARTS.intersection(parts)) + + +class R022FloatMonetary(Rule): + rule_id = "R-022" + name = "float_monetary_column_to_decimal" + tier = "1B" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + for m in _FLOAT_COL_RE.finditer(sql): + column = m.group(1) + col_type = m.group(2) + if _is_monetary_name(column): + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"Колонка '{column}' объявлена как {col_type}, " + "но по имени предположительно хранит монетарное значение. " + "Float-арифметика не ассоциативна и даёт неточные финансовые расчёты." + ), + suggestion=( + f"Замените {col_type} на Decimal64(2) для '{column}'. " + "Decimal даёт точные результаты для SUM/AVG денежных сумм." + ), + example_before=f"CREATE TABLE t ({column} {col_type}) ENGINE = MergeTree ORDER BY tuple()", + example_after=f"CREATE TABLE t ({column} Decimal64(2)) ENGINE = MergeTree ORDER BY tuple()", + explain_why=( + "(a + b) - a может не равняться b при Float64 из-за рounding. " + "Decimal(p, s) даёт точные целочисленные вычисления со смещённой точкой." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) + return None diff --git a/clickadvisor/rules/tier1/R023_string_datetime_column.py b/clickadvisor/rules/tier1/R023_string_datetime_column.py new file mode 100644 index 00000000..9f10d7b6 --- /dev/null +++ b/clickadvisor/rules/tier1/R023_string_datetime_column.py @@ -0,0 +1,67 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +_STRING_COL_RE = re.compile( + r"`?(\w+)`?\s+String\b", + re.IGNORECASE, +) +_DATE_NAME_PARTS = frozenset( + ["date", "datetime", "timestamp", "time", "created", "updated", "modified", + "deleted", "occurred", "happened", "at", "on", "when", "dt", "ts"] +) + + +def _is_datetime_name(name: str) -> bool: + parts = name.lower().split("_") + # require at least one date-related word in name parts + if _DATE_NAME_PARTS.intersection(parts): + return True + low = name.lower() + # also catch names like 'created_at', 'event_date', 'event_time' + return any(low.endswith(s) for s in ("_at", "_date", "_time", "_ts", "_dt", "_on")) + + +class R023StringDatetimeColumn(Rule): + rule_id = "R-023" + name = "string_datetime_column_to_date" + tier = "1B" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + for m in _STRING_COL_RE.finditer(sql): + column = m.group(1) + if _is_datetime_name(column): + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"Колонка '{column}' объявлена как String, " + "но по имени предположительно хранит дату или время. " + "Числовые типы дат сжимаются Delta-кодеком и допускают date-функции напрямую." + ), + suggestion=( + f"Замените String на Date (если только дата) или DateTime " + f"(если дата и время) для колонки '{column}'. " + "Проверьте реальный формат значений." + ), + example_before=f"CREATE TABLE t ({column} String) ENGINE = MergeTree ORDER BY {column}", + example_after=f"CREATE TABLE t ({column} DateTime) ENGINE = MergeTree ORDER BY {column}", + explain_why=( + "String хранит дату как UTF-8 (~10-19 байт). " + "Date — 2 байта, DateTime — 4 байта. " + "Числовые типы поддерживают Delta-сжатие и range-индексацию." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) + return None diff --git a/clickadvisor/rules/tier1/R024_string_ip_column.py b/clickadvisor/rules/tier1/R024_string_ip_column.py new file mode 100644 index 00000000..bfa5b139 --- /dev/null +++ b/clickadvisor/rules/tier1/R024_string_ip_column.py @@ -0,0 +1,75 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +_STRING_COL_RE = re.compile( + r"`?(\w+)`?\s+String\b", + re.IGNORECASE, +) +_IP_NAME_PARTS = frozenset( + ["ip", "addr", "address", "host", "peer", "remote", "client", "server", + "src", "dst", "source", "destination"] +) + +# Exact column names that strongly indicate IP +_IP_EXACT_SUFFIXES = ("_ip", "_addr", "_address", "_host", "_peer") +_IP_EXACT_NAMES = frozenset( + ["ip", "remote_addr", "client_ip", "server_ip", "src_ip", "dst_ip", + "ip_address", "remote_host", "peer_address", "client_address", + "source_ip", "destination_ip"] +) + + +def _is_ip_name(name: str) -> bool: + low = name.lower() + if low in _IP_EXACT_NAMES: + return True + if any(low.endswith(s) for s in _IP_EXACT_SUFFIXES): + return True + parts = set(low.split("_")) + # Require "ip" or "addr" in parts to avoid false positives + return bool({"ip", "addr"}.intersection(parts)) + + +class R024StringIPColumn(Rule): + rule_id = "R-024" + name = "string_ip_column_to_ipv4" + tier = "1B" + ch_version_introduced = "116.253" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + for m in _STRING_COL_RE.finditer(sql): + column = m.group(1) + if _is_ip_name(column): + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"Колонка '{column}' объявлена как String, " + "но по имени предположительно хранит IP-адрес. " + "Тип IPv4 хранит адрес в 4 байтах (UInt32) вместо ~15 байт строки." + ), + suggestion=( + f"Замените String на IPv4 для '{column}' если это IPv4-адрес, " + "или IPv6 если это IPv6. Проверьте реальный формат значений." + ), + example_before=f"CREATE TABLE t ({column} String) ENGINE = MergeTree ORDER BY tuple()", + example_after=f"CREATE TABLE t ({column} IPv4) ENGINE = MergeTree ORDER BY tuple()", + explain_why=( + "IPv4 хранит адрес как UInt32 (4 байта). " + "Строка '192.168.1.1' занимает ~11 байт. " + "IPv4 поддерживает нативные функции IPv4CIDRToRange, toIPv4." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) + return None diff --git a/clickadvisor/rules/tier1/R025_order_by_tuple.py b/clickadvisor/rules/tier1/R025_order_by_tuple.py new file mode 100644 index 00000000..69d79300 --- /dev/null +++ b/clickadvisor/rules/tier1/R025_order_by_tuple.py @@ -0,0 +1,57 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +# Matches ORDER BY tuple() or ORDER BY () +_ORDER_BY_TUPLE_RE = re.compile( + r"\bORDER\s+BY\s+(?:tuple\s*\(\s*\)|\(\s*\))", + re.IGNORECASE, +) + + +class R025OrderByTupleNoPK(Rule): + rule_id = "R-025" + name = "order_by_tuple_no_pk" + tier = "1B" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + if not _ORDER_BY_TUPLE_RE.search(sql): + return None + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="medium", + description=( + "CREATE TABLE использует ORDER BY tuple() — без первичного ключа. " + "Каждый SELECT с WHERE будет сканировать все данные без возможности " + "использовать sparse primary index." + ), + suggestion=( + "Выберите ORDER BY по колонкам, наиболее часто используемым в WHERE. " + "Например: ORDER BY (user_id, ts) для аналитики по пользователям." + ), + example_before=( + "CREATE TABLE events (user_id UInt64, ts DateTime) " + "ENGINE = MergeTree ORDER BY tuple()" + ), + example_after=( + "CREATE TABLE events (user_id UInt64, ts DateTime) " + "ENGINE = MergeTree ORDER BY (user_id, ts)" + ), + explain_why=( + "ORDER BY tuple() отключает сортировку и sparse primary index. " + "Без primary key каждый SELECT выполняет полный скан — " + "10-1000x медленнее по сравнению с правильным ORDER BY." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R026_sum_case_to_countif.py b/clickadvisor/rules/tier1/R026_sum_case_to_countif.py new file mode 100644 index 00000000..d737a670 --- /dev/null +++ b/clickadvisor/rules/tier1/R026_sum_case_to_countif.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches SUM(CASE WHEN ... THEN 1 ELSE 0 END) or COUNT(CASE WHEN ... THEN 1 END) +_SUM_CASE_ONE_RE = re.compile( + r"\b(?:SUM|COUNT)\s*\(\s*CASE\s+WHEN\s+(.+?)\s+THEN\s+1\b(?:\s+ELSE\s+0\s*)?\s*END\s*\)", + re.IGNORECASE | re.DOTALL, +) + + +class R026SumCaseToCountIf(Rule): + rule_id = "R-026" + name = "sum_case_to_countif" + tier = "1A" + ch_version_introduced = "1.23" + + def check(self, context: QueryContext) -> Finding | None: + m = _SUM_CASE_ONE_RE.search(context.sql) + if not m: + return None + cond = m.group(1).strip() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + "SUM(CASE WHEN ... THEN 1 ELSE 0 END) или COUNT(CASE WHEN ... THEN 1 END) " + "эквивалентно countIf(cond) — встроенному комбинатору ClickHouse." + ), + suggestion=f"Замените на countIf({cond})", + example_before=f"SUM(CASE WHEN {cond} THEN 1 ELSE 0 END)", + example_after=f"countIf({cond})", + explain_why=( + "countIf(cond) — синтаксический сахар, семантически идентичен " + "COUNT(CASE WHEN cond THEN 1 END). Без материализации CASE-выражения." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R027_sum_case_col_to_sumif.py b/clickadvisor/rules/tier1/R027_sum_case_col_to_sumif.py new file mode 100644 index 00000000..43399782 --- /dev/null +++ b/clickadvisor/rules/tier1/R027_sum_case_col_to_sumif.py @@ -0,0 +1,46 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches SUM(CASE WHEN cond THEN col_or_expr ELSE 0 END) +# Does NOT match THEN 1 (handled by R-026) +_SUM_CASE_COL_RE = re.compile( + r"\bSUM\s*\(\s*CASE\s+WHEN\s+(.+?)\s+THEN\s+((?!1\b)[^\s][^)]+?)\s+ELSE\s+0\s*END\s*\)", + re.IGNORECASE | re.DOTALL, +) + + +class R027SumCaseColToSumIf(Rule): + rule_id = "R-027" + name = "sum_case_col_to_sumif" + tier = "1A" + ch_version_introduced = "1.23" + + def check(self, context: QueryContext) -> Finding | None: + m = _SUM_CASE_COL_RE.search(context.sql) + if not m: + return None + cond = m.group(1).strip() + col = m.group(2).strip() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + "SUM(CASE WHEN cond THEN col ELSE 0 END) эквивалентно " + "sumIf(col, cond) — встроенному комбинатору ClickHouse." + ), + suggestion=f"Замените на sumIf({col}, {cond})", + example_before=f"SUM(CASE WHEN {cond} THEN {col} ELSE 0 END)", + example_after=f"sumIf({col}, {cond})", + explain_why=( + "sumIf(col, cond) применяет сложение только к строкам где cond=true. " + "Математически идентично SUM(CASE WHEN ... ELSE 0 END) при ELSE 0." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R028_coalesce_to_ifnull.py b/clickadvisor/rules/tier1/R028_coalesce_to_ifnull.py new file mode 100644 index 00000000..31c2c92d --- /dev/null +++ b/clickadvisor/rules/tier1/R028_coalesce_to_ifnull.py @@ -0,0 +1,45 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches COALESCE(arg1, arg2) with exactly 2 arguments (no nested commas at top level) +_COALESCE_TWO_RE = re.compile( + r"\bCOALESCE\s*\(\s*([^,()]+)\s*,\s*([^,()]+)\s*\)", + re.IGNORECASE, +) + + +class R028CoalesceToIfNull(Rule): + rule_id = "R-028" + name = "coalesce_two_args_to_ifnull" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _COALESCE_TWO_RE.search(context.sql) + if not m: + return None + arg1 = m.group(1).strip() + arg2 = m.group(2).strip() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"COALESCE({arg1}, {arg2}) с двумя аргументами полностью эквивалентен " + f"ifNull({arg1}, {arg2})." + ), + suggestion=f"Замените на ifNull({arg1}, {arg2})", + example_before=f"COALESCE({arg1}, {arg2})", + example_after=f"ifNull({arg1}, {arg2})", + explain_why=( + "ifNull(x, y) — специализированная функция без overhead varargs-перебора. " + "Семантически идентична COALESCE с двумя аргументами." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R029_lower_like_to_ilike.py b/clickadvisor/rules/tier1/R029_lower_like_to_ilike.py new file mode 100644 index 00000000..da6fafda --- /dev/null +++ b/clickadvisor/rules/tier1/R029_lower_like_to_ilike.py @@ -0,0 +1,46 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches lower(col) LIKE 'pattern' +_LOWER_LIKE_RE = re.compile( + r"\blower\s*\(\s*(\w+)\s*\)\s+LIKE\s+('[^']*')", + re.IGNORECASE, +) + + +class R029LowerLikeToILike(Rule): + rule_id = "R-029" + name = "lower_col_like_to_ilike" + tier = "1A" + ch_version_introduced = "22.6" + + def check(self, context: QueryContext) -> Finding | None: + m = _LOWER_LIKE_RE.search(context.sql) + if not m: + return None + column = m.group(1) + pattern = m.group(2) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"lower({column}) LIKE {pattern} эквивалентен " + f"{column} ILIKE {pattern}. " + "ILIKE — нативный оператор ClickHouse (с версии 22.6)." + ), + suggestion=f"Замените на {column} ILIKE {pattern}", + example_before=f"lower({column}) LIKE {pattern}", + example_after=f"{column} ILIKE {pattern}", + explain_why=( + "ILIKE выполняет регистронезависимое сравнение без явного вызова lower(). " + "Упрощает план запроса и улучшает читаемость кода." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R030_not_in_singleton.py b/clickadvisor/rules/tier1/R030_not_in_singleton.py new file mode 100644 index 00000000..7da3e1b2 --- /dev/null +++ b/clickadvisor/rules/tier1/R030_not_in_singleton.py @@ -0,0 +1,45 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches col NOT IN (single_value) — one numeric or string literal, no comma +_NOT_IN_SINGLETON_RE = re.compile( + r"(\w+)\s+NOT\s+IN\s*\(\s*('[^']*'|\d+(?:\.\d+)?)\s*\)", + re.IGNORECASE, +) + + +class R030NotInSingleton(Rule): + rule_id = "R-030" + name = "not_in_singleton_to_not_equal" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _NOT_IN_SINGLETON_RE.search(context.sql) + if not m: + return None + col = m.group(1) + val = m.group(2) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"{col} NOT IN ({val}) с одним значением эквивалентен " + f"{col} != {val} для не-Nullable колонок." + ), + suggestion=f"Замените на {col} != {val}", + example_before=f"WHERE {col} NOT IN ({val})", + example_after=f"WHERE {col} != {val}", + explain_why=( + "NOT IN с одним элементом строит hash set вместо прямого " + "скалярного сравнения. Для не-Nullable колонок результат идентичен !=." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R031_string_uuid_column.py b/clickadvisor/rules/tier1/R031_string_uuid_column.py new file mode 100644 index 00000000..83d0fe12 --- /dev/null +++ b/clickadvisor/rules/tier1/R031_string_uuid_column.py @@ -0,0 +1,67 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +_STRING_COL_RE = re.compile( + r"`?(\w+)`?\s+String\b", + re.IGNORECASE, +) +_UUID_NAME_PARTS = frozenset(["uuid", "guid", "uid"]) +_UUID_EXACT_SUFFIXES = ("_uuid", "_guid", "_uid") +_UUID_COLUMN_NAMES = frozenset( + ["trace_id", "span_id", "request_id", "correlation_id", + "uuid", "guid", "uid", "object_id"] +) + + +def _is_uuid_name(name: str) -> bool: + low = name.lower() + if low in _UUID_COLUMN_NAMES: + return True + if any(low.endswith(s) for s in _UUID_EXACT_SUFFIXES): + return True + parts = set(low.split("_")) + return bool(_UUID_NAME_PARTS.intersection(parts)) + + +class R031StringUUIDColumn(Rule): + rule_id = "R-031" + name = "string_uuid_column_to_uuid_type" + tier = "1B" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + for m in _STRING_COL_RE.finditer(sql): + column = m.group(1) + if _is_uuid_name(column): + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"Колонка '{column}' объявлена как String, " + "но по имени предположительно хранит UUID. " + "UUID-тип хранит 16 байт вместо ~36 байт строки с дефисами." + ), + suggestion=( + f"Замените String на UUID для '{column}'. " + "Проверьте, что все значения — валидные UUID-строки." + ), + example_before=f"CREATE TABLE t ({column} String) ENGINE = MergeTree ORDER BY {column}", + example_after=f"CREATE TABLE t ({column} UUID) ENGINE = MergeTree ORDER BY {column}", + explain_why=( + "UUID хранит 128 бит (16 байт) в бинарном виде. " + "String UUID занимает ~36 байт. Экономия ~2.25x плюс нативные UUID-функции." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) + return None diff --git a/clickadvisor/rules/tier1/R032_int8_boolean_column.py b/clickadvisor/rules/tier1/R032_int8_boolean_column.py new file mode 100644 index 00000000..21182b75 --- /dev/null +++ b/clickadvisor/rules/tier1/R032_int8_boolean_column.py @@ -0,0 +1,69 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +_INT8_COL_RE = re.compile( + r"`?(\w+)`?\s+(UInt8|Int8)\b", + re.IGNORECASE, +) +_BOOL_PREFIXES = ("is_", "has_", "can_", "was_", "did_", "should_", "will_", + "allow_", "enable_", "disable_") +_BOOL_EXACT = frozenset( + ["flag", "enabled", "disabled", "active", "inactive", "deleted", "archived", + "published", "visible", "hidden", "locked", "verified", "confirmed", + "approved", "rejected", "paid", "free", "premium", "suspended"] +) + + +def _is_boolean_name(name: str) -> bool: + low = name.lower() + if low in _BOOL_EXACT: + return True + return any(low.startswith(p) for p in _BOOL_PREFIXES) + + +class R032Int8BooleanColumn(Rule): + rule_id = "R-032" + name = "int8_boolean_column_to_bool" + tier = "1B" + ch_version_introduced = "22.1" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + for m in _INT8_COL_RE.finditer(sql): + column = m.group(1) + col_type = m.group(2) + if _is_boolean_name(column): + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"Колонка '{column}' объявлена как {col_type}, " + "но по имени предположительно хранит булево значение. " + "Тип Bool хранится идентично UInt8 (1 байт), " + "но документирует домен явно." + ), + suggestion=( + f"Замените {col_type} на Bool для '{column}'. " + "Требует CH >= 22.1. " + "Убедитесь, что значения только 0 и 1." + ), + example_before=f"CREATE TABLE t ({column} {col_type}) ENGINE = MergeTree ORDER BY tuple()", + example_after=f"CREATE TABLE t ({column} Bool) ENGINE = MergeTree ORDER BY tuple()", + explain_why=( + "Bool хранится как UInt8 (1 байт). " + "Замена семантически нейтральна для хранения, " + "но добавляет явное документирование домена в схеме." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) + return None diff --git a/clickadvisor/rules/tier1/R033_max_case_to_maxif.py b/clickadvisor/rules/tier1/R033_max_case_to_maxif.py new file mode 100644 index 00000000..a21cab42 --- /dev/null +++ b/clickadvisor/rules/tier1/R033_max_case_to_maxif.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches MAX(CASE WHEN cond THEN col END) without ELSE +_MAX_CASE_RE = re.compile( + r"\bMAX\s*\(\s*CASE\s+WHEN\s+(.+?)\s+THEN\s+([^\s)][^)]+?)\s+END\s*\)", + re.IGNORECASE | re.DOTALL, +) + + +class R033MaxCaseToMaxIf(Rule): + rule_id = "R-033" + name = "max_case_to_maxif" + tier = "1A" + ch_version_introduced = "1.23" + + def check(self, context: QueryContext) -> Finding | None: + m = _MAX_CASE_RE.search(context.sql) + if not m: + return None + cond = m.group(1).strip() + col = m.group(2).strip() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"MAX(CASE WHEN {cond} THEN {col} END) эквивалентен maxIf({col}, {cond})." + ), + suggestion=f"Замените на maxIf({col}, {cond})", + example_before=f"MAX(CASE WHEN {cond} THEN {col} END)", + example_after=f"maxIf({col}, {cond})", + explain_why=( + "maxIf — встроенный комбинатор ClickHouse, семантически идентичный " + "MAX(CASE WHEN cond THEN col END). NULL из CASE игнорируется MAX." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R034_min_case_to_minif.py b/clickadvisor/rules/tier1/R034_min_case_to_minif.py new file mode 100644 index 00000000..13078be3 --- /dev/null +++ b/clickadvisor/rules/tier1/R034_min_case_to_minif.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_MIN_CASE_RE = re.compile( + r"\bMIN\s*\(\s*CASE\s+WHEN\s+(.+?)\s+THEN\s+([^\s)][^)]+?)\s+END\s*\)", + re.IGNORECASE | re.DOTALL, +) + + +class R034MinCaseToMinIf(Rule): + rule_id = "R-034" + name = "min_case_to_minif" + tier = "1A" + ch_version_introduced = "1.23" + + def check(self, context: QueryContext) -> Finding | None: + m = _MIN_CASE_RE.search(context.sql) + if not m: + return None + cond = m.group(1).strip() + col = m.group(2).strip() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"MIN(CASE WHEN {cond} THEN {col} END) эквивалентен minIf({col}, {cond})." + ), + suggestion=f"Замените на minIf({col}, {cond})", + example_before=f"MIN(CASE WHEN {cond} THEN {col} END)", + example_after=f"minIf({col}, {cond})", + explain_why=( + "minIf — встроенный комбинатор ClickHouse. " + "NULL из CASE игнорируется MIN — то же поведение minIf." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R035_avg_case_to_avgif.py b/clickadvisor/rules/tier1/R035_avg_case_to_avgif.py new file mode 100644 index 00000000..885ff9f5 --- /dev/null +++ b/clickadvisor/rules/tier1/R035_avg_case_to_avgif.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_AVG_CASE_RE = re.compile( + r"\bAVG\s*\(\s*CASE\s+WHEN\s+(.+?)\s+THEN\s+([^\s)][^)]+?)\s+END\s*\)", + re.IGNORECASE | re.DOTALL, +) + + +class R035AvgCaseToAvgIf(Rule): + rule_id = "R-035" + name = "avg_case_to_avgif" + tier = "1A" + ch_version_introduced = "1.23" + + def check(self, context: QueryContext) -> Finding | None: + m = _AVG_CASE_RE.search(context.sql) + if not m: + return None + cond = m.group(1).strip() + col = m.group(2).strip() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"AVG(CASE WHEN {cond} THEN {col} END) эквивалентен avgIf({col}, {cond})." + ), + suggestion=f"Замените на avgIf({col}, {cond})", + example_before=f"AVG(CASE WHEN {cond} THEN {col} END)", + example_after=f"avgIf({col}, {cond})", + explain_why=( + "avgIf — встроенный комбинатор. " + "AVG игнорирует NULL из CASE — то же поведение avgIf." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R036_nested_if_to_multiif.py b/clickadvisor/rules/tier1/R036_nested_if_to_multiif.py new file mode 100644 index 00000000..bf4189fd --- /dev/null +++ b/clickadvisor/rules/tier1/R036_nested_if_to_multiif.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches IF(cond1, val1, IF(cond2, val2, val3)) +# Simple regex for one level of nesting +_NESTED_IF_RE = re.compile( + r"\bIF\s*\(\s*([^,()]+?)\s*,\s*([^,()]+?)\s*,\s*IF\s*\(", + re.IGNORECASE, +) + + +class R036NestedIfToMultiIf(Rule): + rule_id = "R-036" + name = "nested_if_to_multiif" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + if not _NESTED_IF_RE.search(context.sql): + return None + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + "Обнаружен вложенный IF(..., ..., IF(...)). " + "multiIf(c1, v1, c2, v2, else_v) читаемее и избавляет от вложенности." + ), + suggestion="Замените IF(c1, v1, IF(c2, v2, v3)) на multiIf(c1, v1, c2, v2, v3)", + example_before="IF(score > 90, 'A', IF(score > 70, 'B', 'C'))", + example_after="multiIf(score > 90, 'A', score > 70, 'B', 'C')", + explain_why=( + "multiIf — нативная функция ClickHouse для множественных условий. " + "Семантически идентична вложенным IF, но более читаема и устраняет вложенность." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R037_empty_string_eq_to_empty.py b/clickadvisor/rules/tier1/R037_empty_string_eq_to_empty.py new file mode 100644 index 00000000..efa97f91 --- /dev/null +++ b/clickadvisor/rules/tier1/R037_empty_string_eq_to_empty.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches col = '' or '' = col (empty string comparison) +_EQ_EMPTY_RE = re.compile( + r"(\w+)\s*=\s*''|''\s*=\s*(\w+)", + re.IGNORECASE, +) + + +class R037EmptyStringEqToEmpty(Rule): + rule_id = "R-037" + name = "empty_string_eq_to_empty" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _EQ_EMPTY_RE.search(context.sql) + if not m: + return None + col = (m.group(1) or m.group(2)).strip() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"'{col} = ''' можно заменить на empty({col}). " + "empty() — специализированный предикат ClickHouse." + ), + suggestion=f"Замените на empty({col})", + example_before=f"WHERE {col} = ''", + example_after=f"WHERE empty({col})", + explain_why=( + "empty(s) ≡ s = '' для String типа. " + "empty() более читаем и выражает намерение напрямую." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R038_nonempty_string_neq.py b/clickadvisor/rules/tier1/R038_nonempty_string_neq.py new file mode 100644 index 00000000..2247b309 --- /dev/null +++ b/clickadvisor/rules/tier1/R038_nonempty_string_neq.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches col != '' or col <> '' (not-empty string comparison) +_NEQ_EMPTY_RE = re.compile( + r"(\w+)\s*(?:!=|<>)\s*''|''\s*(?:!=|<>)\s*(\w+)", + re.IGNORECASE, +) + + +class R038NonEmptyStringNeqToNotEmpty(Rule): + rule_id = "R-038" + name = "nonempty_string_neq_to_notempty" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _NEQ_EMPTY_RE.search(context.sql) + if not m: + return None + col = (m.group(1) or m.group(2)).strip() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"'{col} != ''' можно заменить на notEmpty({col}). " + "notEmpty() — специализированный предикат ClickHouse." + ), + suggestion=f"Замените на notEmpty({col})", + example_before=f"WHERE {col} != ''", + example_after=f"WHERE notEmpty({col})", + explain_why=( + "notEmpty(s) ≡ s != '' для String типа. " + "notEmpty() более читаем и выражает намерение напрямую." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R039_length_gte_one_to_notempty.py b/clickadvisor/rules/tier1/R039_length_gte_one_to_notempty.py new file mode 100644 index 00000000..6acd5ff7 --- /dev/null +++ b/clickadvisor/rules/tier1/R039_length_gte_one_to_notempty.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches length(col) >= 1 +_LENGTH_GTE_ONE_RE = re.compile( + r"\blength\s*\(\s*(\w+)\s*\)\s*>=\s*1\b", + re.IGNORECASE, +) + + +class R039LengthGteOneToNotEmpty(Rule): + rule_id = "R-039" + name = "length_gte_one_to_notempty" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _LENGTH_GTE_ONE_RE.search(context.sql) + if not m: + return None + col = m.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"length({col}) >= 1 эквивалентно notEmpty({col}). " + "notEmpty() — специализированный предикат ClickHouse." + ), + suggestion=f"Замените length({col}) >= 1 на notEmpty({col})", + example_before=f"WHERE length({col}) >= 1", + example_after=f"WHERE notEmpty({col})", + explain_why=( + "length(x) >= 1 ≡ length(x) > 0 ≡ notEmpty(x). " + "notEmpty более читаем и выражает намерение напрямую." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R040_todate_comparison.py b/clickadvisor/rules/tier1/R040_todate_comparison.py new file mode 100644 index 00000000..c6749eb2 --- /dev/null +++ b/clickadvisor/rules/tier1/R040_todate_comparison.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches toDate(col) OP 'YYYY-MM-DD' where OP is >=, >, <=, < +_TODATE_CMP_RE = re.compile( + r"\btoDate\s*\(\s*(\w+)\s*\)\s*(>=|>|<=|<)\s*'(\d{4}-\d{2}-\d{2})'", + re.IGNORECASE, +) + + +class R040TodateComparisonToDatetime(Rule): + rule_id = "R-040" + name = "todate_comparison_to_datetime" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _TODATE_CMP_RE.search(context.sql) + if not m: + return None + col = m.group(1) + op = m.group(2) + date = m.group(3) + suggestion = self._build_suggestion(col, op, date) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="high", + description=( + f"toDate({col}) {op} '{date}' оборачивает колонку в функцию, " + "запрещая использование sparse primary index для DateTime-колонки." + ), + suggestion=suggestion, + example_before=f"WHERE toDate({col}) {op} '{date}'", + example_after=suggestion.replace("Замените на ", "WHERE "), + explain_why=( + "Применение toDate() к колонке делает predicate не-sargable: " + "sparse index по DateTime не может быть использован. " + "Прямое сравнение с DateTime-константой восстанавливает sargability." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) + + def _build_suggestion(self, col: str, op: str, date: str) -> str: + if op == ">=": + return f"Замените на {col} >= '{date} 00:00:00'" + elif op == ">": + return f"Замените на {col} >= (toDateTime('{date}') + toIntervalDay(1))" + elif op == "<=": + return f"Замените на {col} < (toDateTime('{date}') + toIntervalDay(1))" + elif op == "<": + return f"Замените на {col} < '{date} 00:00:00'" + return f"Замените toDate({col}) {op} '{date}' на прямое DateTime-сравнение" diff --git a/clickadvisor/rules/tier1/R041_string_code_column.py b/clickadvisor/rules/tier1/R041_string_code_column.py new file mode 100644 index 00000000..b2d35704 --- /dev/null +++ b/clickadvisor/rules/tier1/R041_string_code_column.py @@ -0,0 +1,63 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_CREATE_TABLE_RE = re.compile(r"\bCREATE\s+TABLE\b", re.IGNORECASE) +_STRING_COL_RE = re.compile(r"`?(\w+)`?\s+String\b", re.IGNORECASE) + +_CODE_SUFFIXES = ("_code", "_iso", "_abbr", "_abbreviation") +_CODE_EXACT = frozenset( + ["country_code", "currency_code", "language_code", "lang_code", + "iso_code", "currency", "country", "lang", "region_code", + "state_code", "status_code", "error_code"] +) + + +def _is_fixed_code_name(name: str) -> bool: + low = name.lower() + if low in _CODE_EXACT: + return True + return any(low.endswith(s) for s in _CODE_SUFFIXES) + + +class R041StringCodeColumn(Rule): + rule_id = "R-041" + name = "string_code_column_to_fixedstring" + tier = "1B" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if not _CREATE_TABLE_RE.search(sql): + return None + for m in _STRING_COL_RE.finditer(sql): + column = m.group(1) + if _is_fixed_code_name(column): + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"Колонка '{column}' объявлена как String, " + "но по имени предположительно хранит код фиксированной длины. " + "FixedString(N) хранит без length-prefix, эффективнее для фиксированных кодов." + ), + suggestion=( + f"Проверьте длину значений: " + f"country_code → FixedString(2), currency_code → FixedString(3). " + f"Замените String на FixedString(N) для '{column}'." + ), + example_before=f"CREATE TABLE t ({column} String) ENGINE = MergeTree ORDER BY tuple()", + example_after=f"CREATE TABLE t ({column} FixedString(2)) ENGINE = MergeTree ORDER BY tuple()", + explain_why=( + "FixedString(N) хранит ровно N байт без prefix длины. " + "Для ISO-кодов (2-3 символа) это ~30% экономия хранилища." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) + return None diff --git a/clickadvisor/rules/tier1/R042_grouparray_no_limit.py b/clickadvisor/rules/tier1/R042_grouparray_no_limit.py new file mode 100644 index 00000000..ff18cb12 --- /dev/null +++ b/clickadvisor/rules/tier1/R042_grouparray_no_limit.py @@ -0,0 +1,48 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches groupArray(col) without a size argument: groupArray(col) but NOT groupArray(N)(col) +# groupArray with limit looks like: groupArray(N)(col) where N is a number +_GROUPARRAY_NO_LIMIT_RE = re.compile( + r"\bgroupArray\s*\(\s*(?!\d)(\w+)\s*\)", + re.IGNORECASE, +) + + +class R042GroupArrayNoLimit(Rule): + rule_id = "R-042" + name = "grouparray_without_limit" + tier = "1B" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _GROUPARRAY_NO_LIMIT_RE.search(context.sql) + if not m: + return None + col = m.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="medium", + description=( + f"groupArray({col}) без ограничения размера накапливает ВСЕ значения в памяти. " + "На больших данных это может вызвать OOM." + ), + suggestion=( + f"Добавьте лимит: groupArray(1000)({col}). " + "Выберите N в зависимости от максимально ожидаемого размера группы." + ), + example_before=f"SELECT user_id, groupArray({col}) FROM log GROUP BY user_id", + example_after=f"SELECT user_id, groupArray(1000)({col}) FROM log GROUP BY user_id", + explain_why=( + "groupArray без лимита накапливает все значения для каждой группы в RAM. " + "groupArray(N)(col) безопасно ограничивает размер массива." + ), + confidence="advisory", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R043_having_count_gt_zero.py b/clickadvisor/rules/tier1/R043_having_count_gt_zero.py new file mode 100644 index 00000000..6aac1a71 --- /dev/null +++ b/clickadvisor/rules/tier1/R043_having_count_gt_zero.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches HAVING count() > 0 or HAVING count(*) > 0 +_HAVING_COUNT_GT_ZERO_RE = re.compile( + r"\bHAVING\s+count\s*\(\s*(?:\*|)\s*\)\s*>\s*0\b", + re.IGNORECASE, +) + + +class R043HavingCountGtZero(Rule): + rule_id = "R-043" + name = "having_count_gt_zero_removal" + tier = "1A" + ch_version_introduced = "0.5" + + def check(self, context: QueryContext) -> Finding | None: + if not _HAVING_COUNT_GT_ZERO_RE.search(context.sql): + return None + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + "HAVING count() > 0 является тавтологией: " + "после GROUP BY каждая группа содержит минимум одну строку. " + "Условие никогда не отфильтровывает строки." + ), + suggestion="Удалите HAVING count() > 0 — оно не меняет результат", + example_before="SELECT user_id, count() FROM events GROUP BY user_id HAVING count() > 0", + example_after="SELECT user_id, count() FROM events GROUP BY user_id", + explain_why=( + "По определению GROUP BY: каждая результирующая строка соответствует " + "хотя бы одной строке источника. Поэтому count() >= 1 всегда." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R044_todatetime_todate.py b/clickadvisor/rules/tier1/R044_todatetime_todate.py new file mode 100644 index 00000000..71841e1c --- /dev/null +++ b/clickadvisor/rules/tier1/R044_todatetime_todate.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches toDateTime(toDate(col)) +_TODATETIME_TODATE_RE = re.compile( + r"\btoDateTime\s*\(\s*toDate\s*\(\s*(\w+)\s*\)\s*\)", + re.IGNORECASE, +) + + +class R044ToDateTimeToDateToStartOfDay(Rule): + rule_id = "R-044" + name = "todatetime_todate_to_startofday" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _TODATETIME_TODATE_RE.search(context.sql) + if not m: + return None + col = m.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"toDateTime(toDate({col})) вычисляет начало дня через двойное преобразование. " + f"toStartOfDay({col}) — нативная функция для той же операции." + ), + suggestion=f"Замените toDateTime(toDate({col})) на toStartOfDay({col})", + example_before=f"toDateTime(toDate({col}))", + example_after=f"toStartOfDay({col})", + explain_why=( + "toDate(ts) = дата без времени. toDateTime(date) = date + 00:00:00. " + "toStartOfDay(ts) = ts отсечённый до полуночи текущего дня. Всё это одно." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R045_like_without_wildcards.py b/clickadvisor/rules/tier1/R045_like_without_wildcards.py new file mode 100644 index 00000000..0ebff24a --- /dev/null +++ b/clickadvisor/rules/tier1/R045_like_without_wildcards.py @@ -0,0 +1,46 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches col LIKE 'literal' where literal has no % or _ +_LIKE_NO_WILDCARD_RE = re.compile( + r"(\w+)\s+LIKE\s+'([^%_']+)'", + re.IGNORECASE, +) + + +class R045LikeWithoutWildcardsToEq(Rule): + rule_id = "R-045" + name = "like_without_wildcards_to_eq" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _LIKE_NO_WILDCARD_RE.search(context.sql) + if not m: + return None + col = m.group(1) + pattern = m.group(2) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"{col} LIKE '{pattern}' без шаблонов (% и _) эквивалентен " + f"{col} = '{pattern}'. " + "LIKE без шаблонов компилируется в regexp, что избыточно." + ), + suggestion=f"Замените на {col} = '{pattern}'", + example_before=f"WHERE {col} LIKE '{pattern}'", + example_after=f"WHERE {col} = '{pattern}'", + explain_why=( + "LIKE 'pattern' без % и _ = exact match = = 'pattern'. " + "= выполняет прямое сравнение без regexp overhead." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R046_not_empty_to_notempty.py b/clickadvisor/rules/tier1/R046_not_empty_to_notempty.py new file mode 100644 index 00000000..d5ad7124 --- /dev/null +++ b/clickadvisor/rules/tier1/R046_not_empty_to_notempty.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches NOT empty(col) +_NOT_EMPTY_RE = re.compile( + r"\bNOT\s+empty\s*\(\s*(\w+)\s*\)", + re.IGNORECASE, +) + + +class R046NotEmptyToNotEmpty(Rule): + rule_id = "R-046" + name = "not_empty_to_notempty" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _NOT_EMPTY_RE.search(context.sql) + if not m: + return None + col = m.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=( + f"NOT empty({col}) эквивалентно notEmpty({col}). " + "notEmpty — нативная функция ClickHouse." + ), + suggestion=f"Замените NOT empty({col}) на notEmpty({col})", + example_before=f"WHERE NOT empty({col})", + example_after=f"WHERE notEmpty({col})", + explain_why=( + "notEmpty(x) ≡ NOT empty(x). " + "Нативная функция устраняет оператор NOT и более читаема." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R047_position_to_like.py b/clickadvisor/rules/tier1/R047_position_to_like.py new file mode 100644 index 00000000..dbaceb9e --- /dev/null +++ b/clickadvisor/rules/tier1/R047_position_to_like.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_POSITION_GT_ZERO_RE = re.compile( + r"\bposition\s*\(\s*(\w+)\s*,\s*'([^']+)'\s*\)\s*>\s*0", + re.IGNORECASE, +) + + +class R047PositionToLike(Rule): + rule_id = "R-047" + name = "position_gt_zero_to_like" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _POSITION_GT_ZERO_RE.search(context.sql) + if not m: + return None + col, pattern = m.group(1), m.group(2) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"position({col}, '{pattern}') > 0 эквивалентно {col} LIKE '%{pattern}%'.", + suggestion=f"Замените на {col} LIKE '%{pattern}%'", + example_before=f"WHERE position({col}, '{pattern}') > 0", + example_after=f"WHERE {col} LIKE '%{pattern}%'", + explain_why="LIKE может использовать skip-индекс (ngrambf_v1, text), position() — нет.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R048_positionci_to_ilike.py b/clickadvisor/rules/tier1/R048_positionci_to_ilike.py new file mode 100644 index 00000000..b259335a --- /dev/null +++ b/clickadvisor/rules/tier1/R048_positionci_to_ilike.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_POSITIONCI_GT_ZERO_RE = re.compile( + r"\bpositionCaseInsensitive\s*\(\s*(\w+)\s*,\s*'([^']+)'\s*\)\s*>\s*0", + re.IGNORECASE, +) + + +class R048PositionCIToILike(Rule): + rule_id = "R-048" + name = "positioncaseinsensitive_to_ilike" + tier = "1A" + ch_version_introduced = "22.6" + + def check(self, context: QueryContext) -> Finding | None: + m = _POSITIONCI_GT_ZERO_RE.search(context.sql) + if not m: + return None + col, pattern = m.group(1), m.group(2) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"positionCaseInsensitive({col}, '{pattern}') > 0 эквивалентно {col} ILIKE '%{pattern}%'.", + suggestion=f"Замените на {col} ILIKE '%{pattern}%'", + example_before=f"WHERE positionCaseInsensitive({col}, '{pattern}') > 0", + example_after=f"WHERE {col} ILIKE '%{pattern}%'", + explain_why="ILIKE более читаем и может использовать case-insensitive skip-индексы.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R049_sumif_one_to_countif.py b/clickadvisor/rules/tier1/R049_sumif_one_to_countif.py new file mode 100644 index 00000000..60e35549 --- /dev/null +++ b/clickadvisor/rules/tier1/R049_sumif_one_to_countif.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_SUMIF_ONE_RE = re.compile( + r"\bsumIf\s*\(\s*1\s*,\s*(.+?)\s*\)", + re.IGNORECASE | re.DOTALL, +) + + +class R049SumIfOneToCountIf(Rule): + rule_id = "R-049" + name = "sumif_one_to_countif" + tier = "1A" + ch_version_introduced = "1.23" + + def check(self, context: QueryContext) -> Finding | None: + m = _SUMIF_ONE_RE.search(context.sql) + if not m: + return None + cond = m.group(1).strip() + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"sumIf(1, {cond}) эквивалентно countIf({cond}).", + suggestion=f"Замените на countIf({cond})", + example_before=f"sumIf(1, {cond})", + example_after=f"countIf({cond})", + explain_why="countIf — нативная функция подсчёта строк по условию; semantically identical.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R050_toyyyymm_comparison.py b/clickadvisor/rules/tier1/R050_toyyyymm_comparison.py new file mode 100644 index 00000000..90b40a7c --- /dev/null +++ b/clickadvisor/rules/tier1/R050_toyyyymm_comparison.py @@ -0,0 +1,69 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches toYYYYMM(col) OP YYYYMM_literal (6-digit number) +_TOYYYYMM_CMP_RE = re.compile( + r"\btoYYYYMM\s*\(\s*(\w+)\s*\)\s*(>=|>|<=|<)\s*(\d{6})\b", + re.IGNORECASE, +) + + +def _yyyymm_to_date(yyyymm: str) -> str: + return f"{yyyymm[:4]}-{yyyymm[4:6]}-01" + + +class R050ToYYYYMMComparison(Rule): + rule_id = "R-050" + name = "toyyyymm_comparison_to_range" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _TOYYYYMM_CMP_RE.search(context.sql) + if not m: + return None + col, op, yyyymm = m.group(1), m.group(2), m.group(3) + date_str = _yyyymm_to_date(yyyymm) + suggestion = self._build_suggestion(col, op, date_str, yyyymm) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="high", + description=( + f"toYYYYMM({col}) {op} {yyyymm} оборачивает колонку в функцию, " + "запрещая использование sparse primary index." + ), + suggestion=suggestion, + example_before=f"WHERE toYYYYMM({col}) {op} {yyyymm}", + example_after=suggestion.replace("Замените на ", "WHERE "), + explain_why=( + "Применение toYYYYMM() к колонке делает predicate не-sargable. " + "Прямое сравнение с DateTime-константой восстанавливает sargability." + ), + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) + + def _build_suggestion(self, col: str, op: str, date_str: str, yyyymm: str) -> str: + if op == ">=": + return f"Замените на {col} >= '{date_str} 00:00:00'" + elif op == ">": + yr, mo = int(yyyymm[:4]), int(yyyymm[4:6]) + mo += 1 + if mo > 12: + mo, yr = 1, yr + 1 + return f"Замените на {col} >= '{yr}-{mo:02d}-01 00:00:00'" + elif op == "<=": + yr, mo = int(yyyymm[:4]), int(yyyymm[4:6]) + mo += 1 + if mo > 12: + mo, yr = 1, yr + 1 + return f"Замените на {col} < '{yr}-{mo:02d}-01 00:00:00'" + elif op == "<": + return f"Замените на {col} < '{date_str} 00:00:00'" + return f"Замените toYYYYMM({col}) {op} {yyyymm} на прямое сравнение" diff --git a/clickadvisor/rules/tier1/R051_date_trunc_to_native.py b/clickadvisor/rules/tier1/R051_date_trunc_to_native.py new file mode 100644 index 00000000..521a1947 --- /dev/null +++ b/clickadvisor/rules/tier1/R051_date_trunc_to_native.py @@ -0,0 +1,47 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_DATE_TRUNC_RE = re.compile( + r"\bDATE_TRUNC\s*\(\s*'(day|hour|month|year|week|quarter)'\s*,\s*(\w+)\s*\)", + re.IGNORECASE, +) + +_UNIT_MAP = { + "day": "toStartOfDay", + "hour": "toStartOfHour", + "month": "toStartOfMonth", + "year": "toStartOfYear", + "week": "toStartOfWeek", + "quarter": "toStartOfQuarter", +} + + +class R051DateTruncToNative(Rule): + rule_id = "R-051" + name = "date_trunc_to_native" + tier = "1A" + ch_version_introduced = "21.4" + + def check(self, context: QueryContext) -> Finding | None: + m = _DATE_TRUNC_RE.search(context.sql) + if not m: + return None + unit, col = m.group(1).lower(), m.group(2) + native_fn = _UNIT_MAP.get(unit, f"toStartOf{unit.capitalize()}") + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"DATE_TRUNC('{unit}', {col}) заменяется {native_fn}({col}).", + suggestion=f"Замените на {native_fn}({col})", + example_before=f"DATE_TRUNC('{unit}', {col})", + example_after=f"{native_fn}({col})", + explain_why="Нативные CH-функции более явны и идиоматичны.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R052_formatdatetime_ymd.py b/clickadvisor/rules/tier1/R052_formatdatetime_ymd.py new file mode 100644 index 00000000..8fce8265 --- /dev/null +++ b/clickadvisor/rules/tier1/R052_formatdatetime_ymd.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_FDATETIME_YMD_RE = re.compile( + r"\bformatDateTime\s*\(\s*(\w+)\s*,\s*'%Y-%m-%d'\s*\)", + re.IGNORECASE, +) + + +class R052FormatDateTimeYMD(Rule): + rule_id = "R-052" + name = "formatdatetime_ymd_to_toString" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _FDATETIME_YMD_RE.search(context.sql) + if not m: + return None + col = m.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"formatDateTime({col}, '%Y-%m-%d') заменяется toString(toDate({col})).", + suggestion=f"Замените на toString(toDate({col}))", + example_before=f"formatDateTime({col}, '%Y-%m-%d')", + example_after=f"toString(toDate({col}))", + explain_why="toString(toDate()) более прямой без overhead парсинга format-строки.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R053_arrayreduce_sum.py b/clickadvisor/rules/tier1/R053_arrayreduce_sum.py new file mode 100644 index 00000000..217d4081 --- /dev/null +++ b/clickadvisor/rules/tier1/R053_arrayreduce_sum.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_ARRAYREDUCE_SUM_RE = re.compile(r"\barrayReduce\s*\(\s*'sum'\s*,\s*(\w+)\s*\)", re.IGNORECASE) + + +class R053ArrayReduceSum(Rule): + rule_id = "R-053" + name = "arrayreduce_sum_to_arraysum" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _ARRAYREDUCE_SUM_RE.search(context.sql) + if not m: + return None + arr = m.group(1) + return Finding( + rule_id=self.rule_id, rule_name=self.name, tier=self.tier, severity="low", + description=f"arrayReduce('sum', {arr}) заменяется arraySum({arr}).", + suggestion=f"Замените на arraySum({arr})", + example_before=f"arrayReduce('sum', {arr})", example_after=f"arraySum({arr})", + explain_why="arraySum — специализированная функция, более читаемая.", + confidence="provable", ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R054_arrayreduce_max.py b/clickadvisor/rules/tier1/R054_arrayreduce_max.py new file mode 100644 index 00000000..ac6c6611 --- /dev/null +++ b/clickadvisor/rules/tier1/R054_arrayreduce_max.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_ARRAYREDUCE_MAX_RE = re.compile(r"\barrayReduce\s*\(\s*'max'\s*,\s*(\w+)\s*\)", re.IGNORECASE) + + +class R054ArrayReduceMax(Rule): + rule_id = "R-054" + name = "arrayreduce_max_to_arraymax" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _ARRAYREDUCE_MAX_RE.search(context.sql) + if not m: + return None + arr = m.group(1) + return Finding( + rule_id=self.rule_id, rule_name=self.name, tier=self.tier, severity="low", + description=f"arrayReduce('max', {arr}) заменяется arrayMax({arr}).", + suggestion=f"Замените на arrayMax({arr})", + example_before=f"arrayReduce('max', {arr})", example_after=f"arrayMax({arr})", + explain_why="arrayMax — специализированная нативная функция.", + confidence="provable", ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R055_arrayreduce_min.py b/clickadvisor/rules/tier1/R055_arrayreduce_min.py new file mode 100644 index 00000000..c8283559 --- /dev/null +++ b/clickadvisor/rules/tier1/R055_arrayreduce_min.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_ARRAYREDUCE_MIN_RE = re.compile(r"\barrayReduce\s*\(\s*'min'\s*,\s*(\w+)\s*\)", re.IGNORECASE) + + +class R055ArrayReduceMin(Rule): + rule_id = "R-055" + name = "arrayreduce_min_to_arraymin" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _ARRAYREDUCE_MIN_RE.search(context.sql) + if not m: + return None + arr = m.group(1) + return Finding( + rule_id=self.rule_id, rule_name=self.name, tier=self.tier, severity="low", + description=f"arrayReduce('min', {arr}) заменяется arrayMin({arr}).", + suggestion=f"Замените на arrayMin({arr})", + example_before=f"arrayReduce('min', {arr})", example_after=f"arrayMin({arr})", + explain_why="arrayMin — специализированная нативная функция.", + confidence="provable", ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R056_extract_year.py b/clickadvisor/rules/tier1/R056_extract_year.py new file mode 100644 index 00000000..6a04b63d --- /dev/null +++ b/clickadvisor/rules/tier1/R056_extract_year.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_EXTRACT_RE = re.compile( + r"\bEXTRACT\s*\(\s*YEAR\s+FROM\s+(\w+)\s*\)", + re.IGNORECASE, +) + + +class R056ExtractToNative(Rule): + rule_id = "R-056" + name = "extract_year_to_toyear" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _EXTRACT_RE.search(context.sql) + if not m: + return None + col = m.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"EXTRACT(YEAR FROM {col}) заменяется toYear({col}).", + suggestion=f"Замените на toYear({col})", + example_before=f"EXTRACT(YEAR FROM {col})", + example_after=f"toYear({col})", + explain_why="Нативные CH-функции более читаемы и идиоматичны.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R057_extract_month.py b/clickadvisor/rules/tier1/R057_extract_month.py new file mode 100644 index 00000000..19b7da30 --- /dev/null +++ b/clickadvisor/rules/tier1/R057_extract_month.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_EXTRACT_MONTH_RE = re.compile( + r"\bEXTRACT\s*\(\s*MONTH\s+FROM\s+(\w+)\s*\)", + re.IGNORECASE, +) + + +class R057ExtractMonthToMonth(Rule): + rule_id = "R-057" + name = "extract_month_to_tomonth" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + match = _EXTRACT_MONTH_RE.search(context.sql) + if not match: + return None + column = match.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"EXTRACT(MONTH FROM {column}) заменяется toMonth({column}).", + suggestion=f"Замените на toMonth({column})", + example_before=f"EXTRACT(MONTH FROM {column})", + example_after=f"toMonth({column})", + explain_why="Нативные CH-функции более читаемы и идиоматичны.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R058_extract_day.py b/clickadvisor/rules/tier1/R058_extract_day.py new file mode 100644 index 00000000..57584a7b --- /dev/null +++ b/clickadvisor/rules/tier1/R058_extract_day.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_EXTRACT_DAY_RE = re.compile( + r"\bEXTRACT\s*\(\s*DAY\s+FROM\s+(\w+)\s*\)", + re.IGNORECASE, +) + + +class R058ExtractDayToDayOfMonth(Rule): + rule_id = "R-058" + name = "extract_day_to_todayofmonth" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + match = _EXTRACT_DAY_RE.search(context.sql) + if not match: + return None + column = match.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"EXTRACT(DAY FROM {column}) заменяется toDayOfMonth({column}).", + suggestion=f"Замените на toDayOfMonth({column})", + example_before=f"EXTRACT(DAY FROM {column})", + example_after=f"toDayOfMonth({column})", + explain_why="Нативные CH-функции более читаемы и идиоматичны.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R059_extract_hour.py b/clickadvisor/rules/tier1/R059_extract_hour.py new file mode 100644 index 00000000..f8fe5c63 --- /dev/null +++ b/clickadvisor/rules/tier1/R059_extract_hour.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_EXTRACT_HOUR_RE = re.compile( + r"\bEXTRACT\s*\(\s*HOUR\s+FROM\s+(\w+)\s*\)", + re.IGNORECASE, +) + + +class R059ExtractHourToHour(Rule): + rule_id = "R-059" + name = "extract_hour_to_tohour" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + match = _EXTRACT_HOUR_RE.search(context.sql) + if not match: + return None + column = match.group(1) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"EXTRACT(HOUR FROM {column}) заменяется toHour({column}).", + suggestion=f"Замените на toHour({column})", + example_before=f"EXTRACT(HOUR FROM {column})", + example_after=f"toHour({column})", + explain_why="Нативные CH-функции более читаемы и идиоматичны.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R060_has_and_has_to_hasall.py b/clickadvisor/rules/tier1/R060_has_and_has_to_hasall.py new file mode 100644 index 00000000..c9086b48 --- /dev/null +++ b/clickadvisor/rules/tier1/R060_has_and_has_to_hasall.py @@ -0,0 +1,38 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches has(arr, x) AND has(arr, y) where arr is the same identifier +_HAS_AND_HAS_RE = re.compile( + r"\bhas\s*\(\s*(\w+)\s*,\s*('[^']*'|\d+)\s*\)\s+AND\s+has\s*\(\s*\1\s*,\s*('[^']*'|\d+)\s*\)", + re.IGNORECASE, +) + + +class R060HasAndHasToHasAll(Rule): + rule_id = "R-060" + name = "has_and_has_to_hasall" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _HAS_AND_HAS_RE.search(context.sql) + if not m: + return None + arr, v1, v2 = m.group(1), m.group(2), m.group(3) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"has({arr}, {v1}) AND has({arr}, {v2}) заменяется hasAll({arr}, [{v1}, {v2}]).", + suggestion=f"Замените на hasAll({arr}, [{v1}, {v2}])", + example_before=f"has({arr}, {v1}) AND has({arr}, {v2})", + example_after=f"hasAll({arr}, [{v1}, {v2}])", + explain_why="hasAll более читаем и легко расширяется для дополнительных значений.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R061_has_or_has_to_hasany.py b/clickadvisor/rules/tier1/R061_has_or_has_to_hasany.py new file mode 100644 index 00000000..892ed668 --- /dev/null +++ b/clickadvisor/rules/tier1/R061_has_or_has_to_hasany.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +_HAS_OR_HAS_RE = re.compile( + r"\bhas\s*\(\s*(\w+)\s*,\s*('[^']*'|\d+)\s*\)\s+OR\s+has\s*\(\s*\1\s*,\s*('[^']*'|\d+)\s*\)", + re.IGNORECASE, +) + + +class R061HasOrHasToHasAny(Rule): + rule_id = "R-061" + name = "has_or_has_to_hasany" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _HAS_OR_HAS_RE.search(context.sql) + if not m: + return None + arr, v1, v2 = m.group(1), m.group(2), m.group(3) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"has({arr}, {v1}) OR has({arr}, {v2}) заменяется hasAny({arr}, [{v1}, {v2}]).", + suggestion=f"Замените на hasAny({arr}, [{v1}, {v2}])", + example_before=f"has({arr}, {v1}) OR has({arr}, {v2})", + example_after=f"hasAny({arr}, [{v1}, {v2}])", + explain_why="hasAny более читаем и легко расширяется для дополнительных значений.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/R062_arraycount_zero_to_not_has.py b/clickadvisor/rules/tier1/R062_arraycount_zero_to_not_has.py new file mode 100644 index 00000000..035010b6 --- /dev/null +++ b/clickadvisor/rules/tier1/R062_arraycount_zero_to_not_has.py @@ -0,0 +1,38 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + +# Matches arrayCount(x -> x = val, arr) = 0 +_ARRAYCOUNT_ZERO_RE = re.compile( + r"\barrayCount\s*\(\s*\w+\s*->\s*\w+\s*=\s*('[^']*'|\d+)\s*,\s*(\w+)\s*\)\s*=\s*0\b", + re.IGNORECASE, +) + + +class R062ArrayCountZeroToNotHas(Rule): + rule_id = "R-062" + name = "arraycount_zero_to_not_has" + tier = "1A" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + m = _ARRAYCOUNT_ZERO_RE.search(context.sql) + if not m: + return None + val, arr = m.group(1), m.group(2) + return Finding( + rule_id=self.rule_id, + rule_name=self.name, + tier=self.tier, + severity="low", + description=f"arrayCount(x -> x = {val}, {arr}) = 0 заменяется NOT has({arr}, {val}).", + suggestion=f"Замените на NOT has({arr}, {val})", + example_before=f"arrayCount(x -> x = {val}, {arr}) = 0", + example_after=f"NOT has({arr}, {val})", + explain_why="has() более эффективная функция поиска по значению в массиве.", + confidence="provable", + ch_version_introduced=self.ch_version_introduced, + ) diff --git a/clickadvisor/rules/tier1/__init__.py b/clickadvisor/rules/tier1/__init__.py index afe6276a..974c4ba5 100644 --- a/clickadvisor/rules/tier1/__init__.py +++ b/clickadvisor/rules/tier1/__init__.py @@ -18,6 +18,48 @@ from clickadvisor.rules.tier1.R018_union_to_union_all import R018UnionToUnionAll from clickadvisor.rules.tier1.R019_uint_narrowing import R019UintNarrowing from clickadvisor.rules.tier1.R020_cast_or_default import R020CastOrDefault +from clickadvisor.rules.tier1.R021_datetime64_zero import R021DateTime64ZeroToDateTime +from clickadvisor.rules.tier1.R022_float_monetary import R022FloatMonetary +from clickadvisor.rules.tier1.R023_string_datetime_column import R023StringDatetimeColumn +from clickadvisor.rules.tier1.R024_string_ip_column import R024StringIPColumn +from clickadvisor.rules.tier1.R025_order_by_tuple import R025OrderByTupleNoPK +from clickadvisor.rules.tier1.R026_sum_case_to_countif import R026SumCaseToCountIf +from clickadvisor.rules.tier1.R027_sum_case_col_to_sumif import R027SumCaseColToSumIf +from clickadvisor.rules.tier1.R028_coalesce_to_ifnull import R028CoalesceToIfNull +from clickadvisor.rules.tier1.R029_lower_like_to_ilike import R029LowerLikeToILike +from clickadvisor.rules.tier1.R030_not_in_singleton import R030NotInSingleton +from clickadvisor.rules.tier1.R031_string_uuid_column import R031StringUUIDColumn +from clickadvisor.rules.tier1.R032_int8_boolean_column import R032Int8BooleanColumn +from clickadvisor.rules.tier1.R033_max_case_to_maxif import R033MaxCaseToMaxIf +from clickadvisor.rules.tier1.R034_min_case_to_minif import R034MinCaseToMinIf +from clickadvisor.rules.tier1.R035_avg_case_to_avgif import R035AvgCaseToAvgIf +from clickadvisor.rules.tier1.R036_nested_if_to_multiif import R036NestedIfToMultiIf +from clickadvisor.rules.tier1.R037_empty_string_eq_to_empty import R037EmptyStringEqToEmpty +from clickadvisor.rules.tier1.R038_nonempty_string_neq import R038NonEmptyStringNeqToNotEmpty +from clickadvisor.rules.tier1.R039_length_gte_one_to_notempty import R039LengthGteOneToNotEmpty +from clickadvisor.rules.tier1.R040_todate_comparison import R040TodateComparisonToDatetime +from clickadvisor.rules.tier1.R041_string_code_column import R041StringCodeColumn +from clickadvisor.rules.tier1.R042_grouparray_no_limit import R042GroupArrayNoLimit +from clickadvisor.rules.tier1.R043_having_count_gt_zero import R043HavingCountGtZero +from clickadvisor.rules.tier1.R044_todatetime_todate import R044ToDateTimeToDateToStartOfDay +from clickadvisor.rules.tier1.R045_like_without_wildcards import R045LikeWithoutWildcardsToEq +from clickadvisor.rules.tier1.R046_not_empty_to_notempty import R046NotEmptyToNotEmpty +from clickadvisor.rules.tier1.R047_position_to_like import R047PositionToLike +from clickadvisor.rules.tier1.R048_positionci_to_ilike import R048PositionCIToILike +from clickadvisor.rules.tier1.R049_sumif_one_to_countif import R049SumIfOneToCountIf +from clickadvisor.rules.tier1.R050_toyyyymm_comparison import R050ToYYYYMMComparison +from clickadvisor.rules.tier1.R051_date_trunc_to_native import R051DateTruncToNative +from clickadvisor.rules.tier1.R052_formatdatetime_ymd import R052FormatDateTimeYMD +from clickadvisor.rules.tier1.R053_arrayreduce_sum import R053ArrayReduceSum +from clickadvisor.rules.tier1.R054_arrayreduce_max import R054ArrayReduceMax +from clickadvisor.rules.tier1.R055_arrayreduce_min import R055ArrayReduceMin +from clickadvisor.rules.tier1.R056_extract_year import R056ExtractToNative +from clickadvisor.rules.tier1.R057_extract_month import R057ExtractMonthToMonth +from clickadvisor.rules.tier1.R058_extract_day import R058ExtractDayToDayOfMonth +from clickadvisor.rules.tier1.R059_extract_hour import R059ExtractHourToHour +from clickadvisor.rules.tier1.R060_has_and_has_to_hasall import R060HasAndHasToHasAll +from clickadvisor.rules.tier1.R061_has_or_has_to_hasany import R061HasOrHasToHasAny +from clickadvisor.rules.tier1.R062_arraycount_zero_to_not_has import R062ArrayCountZeroToNotHas __all__ = [ "R001CountDistinct", @@ -40,4 +82,46 @@ "R018UnionToUnionAll", "R019UintNarrowing", "R020CastOrDefault", + "R021DateTime64ZeroToDateTime", + "R022FloatMonetary", + "R023StringDatetimeColumn", + "R024StringIPColumn", + "R025OrderByTupleNoPK", + "R026SumCaseToCountIf", + "R027SumCaseColToSumIf", + "R028CoalesceToIfNull", + "R029LowerLikeToILike", + "R030NotInSingleton", + "R031StringUUIDColumn", + "R032Int8BooleanColumn", + "R033MaxCaseToMaxIf", + "R034MinCaseToMinIf", + "R035AvgCaseToAvgIf", + "R036NestedIfToMultiIf", + "R037EmptyStringEqToEmpty", + "R038NonEmptyStringNeqToNotEmpty", + "R039LengthGteOneToNotEmpty", + "R040TodateComparisonToDatetime", + "R041StringCodeColumn", + "R042GroupArrayNoLimit", + "R043HavingCountGtZero", + "R044ToDateTimeToDateToStartOfDay", + "R045LikeWithoutWildcardsToEq", + "R046NotEmptyToNotEmpty", + "R047PositionToLike", + "R048PositionCIToILike", + "R049SumIfOneToCountIf", + "R050ToYYYYMMComparison", + "R051DateTruncToNative", + "R052FormatDateTimeYMD", + "R053ArrayReduceSum", + "R054ArrayReduceMax", + "R055ArrayReduceMin", + "R056ExtractToNative", + "R057ExtractMonthToMonth", + "R058ExtractDayToDayOfMonth", + "R059ExtractHourToHour", + "R060HasAndHasToHasAll", + "R061HasOrHasToHasAny", + "R062ArrayCountZeroToNotHas", ] diff --git a/data/ml/features_dataset.jsonl b/data/ml/features_dataset.jsonl new file mode 100644 index 00000000..85886684 --- /dev/null +++ b/data/ml/features_dataset.jsonl @@ -0,0 +1,162 @@ +{"case_id": "synthetic_expanded_d003_001", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 29}, "labels": ["select_star_on_wide_table"], "split": "train"} +{"case_id": "synthetic_expanded_d003_002", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 49}, "labels": ["select_star_on_wide_table"], "split": "train"} +{"case_id": "synthetic_expanded_d003_003", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 51}, "labels": ["select_star_on_wide_table"], "split": "train"} +{"case_id": "synthetic_expanded_d003_004", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 56}, "labels": ["select_star_on_wide_table"], "split": "train"} +{"case_id": "synthetic_expanded_d003_005", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 48}, "labels": ["select_star_on_wide_table"], "split": "train"} +{"case_id": "synthetic_expanded_d003_006", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 54}, "labels": ["select_star_on_wide_table"], "split": "train"} +{"case_id": "synthetic_expanded_d004_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 3, "where_clause_depth": 2, "query_length_chars": 74}, "labels": ["missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_d004_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 3, "where_clause_depth": 2, "query_length_chars": 57}, "labels": ["missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_d004_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 3, "where_clause_depth": 2, "query_length_chars": 70}, "labels": ["missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_d004_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 3, "where_clause_depth": 2, "query_length_chars": 74}, "labels": ["missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_d004_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 3, "where_clause_depth": 2, "query_length_chars": 59}, "labels": ["missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_d004_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 3, "where_clause_depth": 2, "query_length_chars": 75}, "labels": ["missing_limit_on_unbounded_result"], "split": "test"} +{"case_id": "synthetic_expanded_d007_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 1, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 50}, "labels": ["final_modifier_usage"], "split": "train"} +{"case_id": "synthetic_expanded_d007_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 1, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 54}, "labels": ["final_modifier_usage"], "split": "train"} +{"case_id": "synthetic_expanded_d007_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 1, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 56}, "labels": ["final_modifier_usage"], "split": "train"} +{"case_id": "synthetic_expanded_d007_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 1, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 51}, "labels": ["final_modifier_usage"], "split": "train"} +{"case_id": "synthetic_expanded_d007_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 1, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 56}, "labels": ["final_modifier_usage"], "split": "train"} +{"case_id": "synthetic_expanded_d007_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 1, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 52}, "labels": ["final_modifier_usage"], "split": "train"} +{"case_id": "synthetic_expanded_d014_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 1, "has_async_insert_without_wait": 1, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 62}, "labels": ["async_insert_without_wait_flag"], "split": "train"} +{"case_id": "synthetic_expanded_d014_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 1, "has_async_insert_without_wait": 1, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 64}, "labels": ["async_insert_without_wait_flag"], "split": "test"} +{"case_id": "synthetic_expanded_d014_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 1, "has_async_insert_without_wait": 1, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 62}, "labels": ["async_insert_without_wait_flag"], "split": "train"} +{"case_id": "synthetic_expanded_d014_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 1, "has_async_insert_without_wait": 1, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 67}, "labels": ["async_insert_without_wait_flag"], "split": "test"} +{"case_id": "synthetic_expanded_d014_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 1, "has_async_insert_without_wait": 1, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 59}, "labels": ["async_insert_without_wait_flag"], "split": "train"} +{"case_id": "synthetic_expanded_d014_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 1, "has_async_insert_without_wait": 1, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 61}, "labels": ["async_insert_without_wait_flag"], "split": "train"} +{"case_id": "synthetic_expanded_negative_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 107}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 2, "query_length_chars": 82}, "labels": [], "split": "test"} +{"case_id": "synthetic_expanded_negative_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 63}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 2, "query_length_chars": 72}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 71}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 1, "has_union_all": 1, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 2, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 73}, "labels": [], "split": "test"} +{"case_id": "synthetic_expanded_negative_007", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 34}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_008", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 29}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_009", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 49}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_010", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 1, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 74}, "labels": [], "split": "test"} +{"case_id": "synthetic_expanded_negative_011", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 44}, "labels": [], "split": "test"} +{"case_id": "synthetic_expanded_negative_012", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 73}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_013", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 74}, "labels": [], "split": "test"} +{"case_id": "synthetic_expanded_negative_014", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 1, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 87}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_015", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 46}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_016", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 46}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_017", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 56}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_018", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 48}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_negative_019", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 44}, "labels": [], "split": "test"} +{"case_id": "synthetic_expanded_negative_020", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 41}, "labels": [], "split": "train"} +{"case_id": "synthetic_expanded_r001_001", "features": {"has_count_distinct": 1, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 42}, "labels": ["exact_count_distinct_specialization", "approx_count_distinct_advisory"], "split": "train"} +{"case_id": "synthetic_expanded_r001_002", "features": {"has_count_distinct": 1, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 43}, "labels": ["exact_count_distinct_specialization", "approx_count_distinct_advisory"], "split": "test"} +{"case_id": "synthetic_expanded_r001_003", "features": {"has_count_distinct": 1, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 65}, "labels": ["exact_count_distinct_specialization", "approx_count_distinct_advisory"], "split": "train"} +{"case_id": "synthetic_expanded_r001_004", "features": {"has_count_distinct": 1, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 88}, "labels": ["exact_count_distinct_specialization", "approx_count_distinct_advisory"], "split": "train"} +{"case_id": "synthetic_expanded_r001_005", "features": {"has_count_distinct": 1, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 47}, "labels": ["exact_count_distinct_specialization", "approx_count_distinct_advisory"], "split": "train"} +{"case_id": "synthetic_expanded_r001_006", "features": {"has_count_distinct": 1, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 65}, "labels": ["exact_count_distinct_specialization", "approx_count_distinct_advisory"], "split": "train"} +{"case_id": "synthetic_expanded_r001_007", "features": {"has_count_distinct": 1, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 51}, "labels": ["exact_count_distinct_specialization", "approx_count_distinct_advisory"], "split": "test"} +{"case_id": "synthetic_expanded_r001_008", "features": {"has_count_distinct": 1, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 70}, "labels": ["exact_count_distinct_specialization", "approx_count_distinct_advisory"], "split": "test"} +{"case_id": "synthetic_expanded_r003_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 1, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 55}, "labels": ["exact_quantile_candidate"], "split": "test"} +{"case_id": "synthetic_expanded_r003_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 1, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 48}, "labels": ["exact_quantile_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r003_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 1, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 53}, "labels": ["exact_quantile_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r003_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 1, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 81}, "labels": ["exact_quantile_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r003_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 1, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 51}, "labels": ["exact_quantile_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r003_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 1, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 56}, "labels": ["exact_quantile_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r003_007", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 1, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 54}, "labels": ["exact_quantile_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r003_008", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 1, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 51}, "labels": ["exact_quantile_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r004_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 58}, "labels": ["count_star_distinct_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r004_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 59}, "labels": ["count_star_distinct_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r004_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 81}, "labels": ["count_star_distinct_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r004_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 63}, "labels": ["count_star_distinct_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r004_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 63}, "labels": ["count_star_distinct_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r004_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 60}, "labels": ["count_star_distinct_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r005_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 66}, "labels": ["function_on_datetime_filter", "unsafe_cast_without_default"], "split": "test"} +{"case_id": "synthetic_expanded_r005_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 56}, "labels": ["function_on_datetime_filter", "unsafe_cast_without_default"], "split": "test"} +{"case_id": "synthetic_expanded_r005_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 63}, "labels": ["function_on_datetime_filter", "unsafe_cast_without_default"], "split": "train"} +{"case_id": "synthetic_expanded_r005_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 68}, "labels": ["function_on_datetime_filter", "unsafe_cast_without_default"], "split": "train"} +{"case_id": "synthetic_expanded_r005_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 70}, "labels": ["function_on_datetime_filter", "unsafe_cast_without_default"], "split": "test"} +{"case_id": "synthetic_expanded_r005_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 70}, "labels": ["function_on_datetime_filter", "unsafe_cast_without_default"], "split": "train"} +{"case_id": "synthetic_expanded_r006_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 62}, "labels": ["date_part_filter_to_range"], "split": "test"} +{"case_id": "synthetic_expanded_r006_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 56}, "labels": ["date_part_filter_to_range"], "split": "train"} +{"case_id": "synthetic_expanded_r006_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 56}, "labels": ["date_part_filter_to_range"], "split": "train"} +{"case_id": "synthetic_expanded_r006_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 76}, "labels": ["date_part_filter_to_range"], "split": "train"} +{"case_id": "synthetic_expanded_r006_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 64}, "labels": ["date_part_filter_to_range"], "split": "train"} +{"case_id": "synthetic_expanded_r006_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 66}, "labels": ["date_part_filter_to_range"], "split": "train"} +{"case_id": "synthetic_expanded_r007_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 72}, "labels": ["interval_start_filter_to_range"], "split": "train"} +{"case_id": "synthetic_expanded_r007_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 81}, "labels": ["interval_start_filter_to_range"], "split": "test"} +{"case_id": "synthetic_expanded_r007_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 96}, "labels": ["interval_start_filter_to_range"], "split": "train"} +{"case_id": "synthetic_expanded_r007_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 84}, "labels": ["interval_start_filter_to_range"], "split": "train"} +{"case_id": "synthetic_expanded_r007_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 84}, "labels": ["interval_start_filter_to_range"], "split": "train"} +{"case_id": "synthetic_expanded_r007_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 86}, "labels": ["interval_start_filter_to_range"], "split": "test"} +{"case_id": "synthetic_expanded_r008_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 63}, "labels": ["redundant_cast_on_filter", "unsafe_cast_without_default"], "split": "test"} +{"case_id": "synthetic_expanded_r008_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 58}, "labels": ["redundant_cast_on_filter", "unsafe_cast_without_default"], "split": "train"} +{"case_id": "synthetic_expanded_r008_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 63}, "labels": ["redundant_cast_on_filter", "unsafe_cast_without_default"], "split": "train"} +{"case_id": "synthetic_expanded_r008_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 58}, "labels": ["redundant_cast_on_filter", "unsafe_cast_without_default"], "split": "train"} +{"case_id": "synthetic_expanded_r008_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 64}, "labels": ["redundant_cast_on_filter", "unsafe_cast_without_default"], "split": "train"} +{"case_id": "synthetic_expanded_r008_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 62}, "labels": ["redundant_cast_on_filter", "unsafe_cast_without_default"], "split": "train"} +{"case_id": "synthetic_expanded_r009_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 1, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 50}, "labels": ["singleton_in_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r009_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 1, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 50}, "labels": ["singleton_in_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r009_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 1, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 49}, "labels": ["singleton_in_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r009_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 1, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 56}, "labels": ["singleton_in_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r009_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 1, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 49}, "labels": ["singleton_in_predicate"], "split": "test"} +{"case_id": "synthetic_expanded_r009_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 1, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 54}, "labels": ["singleton_in_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r010_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 1, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 4, "query_length_chars": 83}, "labels": ["disjunction_chain_to_in"], "split": "train"} +{"case_id": "synthetic_expanded_r010_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 1, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 4, "query_length_chars": 83}, "labels": ["disjunction_chain_to_in"], "split": "train"} +{"case_id": "synthetic_expanded_r010_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 1, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 4, "query_length_chars": 91}, "labels": ["disjunction_chain_to_in"], "split": "train"} +{"case_id": "synthetic_expanded_r010_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 1, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 4, "query_length_chars": 98}, "labels": ["disjunction_chain_to_in"], "split": "test"} +{"case_id": "synthetic_expanded_r010_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 1, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 4, "query_length_chars": 84}, "labels": ["disjunction_chain_to_in"], "split": "test"} +{"case_id": "synthetic_expanded_r010_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 1, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 4, "query_length_chars": 91}, "labels": ["disjunction_chain_to_in"], "split": "test"} +{"case_id": "synthetic_expanded_r011_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 1, "has_having_without_aggregate": 1, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 94}, "labels": ["having_without_aggregate"], "split": "train"} +{"case_id": "synthetic_expanded_r011_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 1, "has_having_without_aggregate": 1, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 98}, "labels": ["having_without_aggregate"], "split": "test"} +{"case_id": "synthetic_expanded_r011_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 1, "has_having_without_aggregate": 1, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 104}, "labels": ["having_without_aggregate"], "split": "train"} +{"case_id": "synthetic_expanded_r011_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 1, "has_having_without_aggregate": 1, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 104}, "labels": ["having_without_aggregate"], "split": "train"} +{"case_id": "synthetic_expanded_r011_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 1, "has_having_without_aggregate": 1, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 112}, "labels": ["having_without_aggregate"], "split": "test"} +{"case_id": "synthetic_expanded_r011_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 1, "has_having_without_aggregate": 1, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 100}, "labels": ["having_without_aggregate"], "split": "train"} +{"case_id": "synthetic_expanded_r012_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 1, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 53}, "labels": ["constant_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r012_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 1, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 56}, "labels": ["constant_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r012_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 1, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 57}, "labels": ["constant_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r012_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 1, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 62}, "labels": ["constant_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r012_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 1, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 52}, "labels": ["constant_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r012_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 1, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 60}, "labels": ["constant_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r013_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 1, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 51}, "labels": ["length_empty_predicate"], "split": "test"} +{"case_id": "synthetic_expanded_r013_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 1, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 50}, "labels": ["length_empty_predicate"], "split": "test"} +{"case_id": "synthetic_expanded_r013_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 1, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 50}, "labels": ["length_empty_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r013_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 1, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 52}, "labels": ["length_empty_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r013_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 1, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 56}, "labels": ["length_empty_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r013_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 1, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 3, "query_length_chars": 55}, "labels": ["length_empty_predicate"], "split": "train"} +{"case_id": "synthetic_expanded_r014_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 1, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 43}, "labels": ["groupby_string_hash_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r014_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 1, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 51}, "labels": ["groupby_string_hash_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r014_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 1, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 49}, "labels": ["groupby_string_hash_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r014_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 1, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 51}, "labels": ["groupby_string_hash_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r014_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 1, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 46}, "labels": ["groupby_string_hash_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r014_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 1, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 51}, "labels": ["groupby_string_hash_candidate"], "split": "train"} +{"case_id": "synthetic_expanded_r015_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 92}, "labels": ["distinct_after_groupby"], "split": "train"} +{"case_id": "synthetic_expanded_r015_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 128}, "labels": ["distinct_after_groupby"], "split": "train"} +{"case_id": "synthetic_expanded_r015_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 139}, "labels": ["distinct_after_groupby"], "split": "train"} +{"case_id": "synthetic_expanded_r015_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 139}, "labels": ["distinct_after_groupby"], "split": "train"} +{"case_id": "synthetic_expanded_r015_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 139}, "labels": ["distinct_after_groupby"], "split": "test"} +{"case_id": "synthetic_expanded_r015_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 1, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 1, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 2, "where_clause_depth": 0, "query_length_chars": 150}, "labels": ["distinct_after_groupby"], "split": "test"} +{"case_id": "synthetic_expanded_r016_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 1, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 63}, "labels": ["orderby_without_limit_in_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r016_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 1, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 53}, "labels": ["orderby_without_limit_in_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r016_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 1, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 63}, "labels": ["orderby_without_limit_in_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r016_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 1, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 65}, "labels": ["orderby_without_limit_in_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r016_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 1, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 67}, "labels": ["orderby_without_limit_in_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r016_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 1, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 67}, "labels": ["orderby_without_limit_in_subquery"], "split": "train"} +{"case_id": "synthetic_expanded_r017_001", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 1, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 79}, "labels": ["subquery_filter_pushdown", "select_star_on_wide_table", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r017_002", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 1, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 77}, "labels": ["subquery_filter_pushdown", "select_star_on_wide_table", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r017_003", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 1, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 81}, "labels": ["subquery_filter_pushdown", "select_star_on_wide_table", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r017_004", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 1, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 83}, "labels": ["subquery_filter_pushdown", "select_star_on_wide_table", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r017_005", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 1, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 79}, "labels": ["subquery_filter_pushdown", "select_star_on_wide_table", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r017_006", "features": {"has_count_distinct": 0, "has_select_star": 1, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 1, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 1, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 2, "query_length_chars": 83}, "labels": ["subquery_filter_pushdown", "select_star_on_wide_table", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r018_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 1, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 2, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 69}, "labels": ["union_without_all"], "split": "train"} +{"case_id": "synthetic_expanded_r018_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 1, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 2, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 71}, "labels": ["union_without_all"], "split": "train"} +{"case_id": "synthetic_expanded_r018_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 1, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 2, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 80}, "labels": ["union_without_all"], "split": "train"} +{"case_id": "synthetic_expanded_r018_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 1, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 2, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 71}, "labels": ["union_without_all"], "split": "train"} +{"case_id": "synthetic_expanded_r018_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 1, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 2, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 75}, "labels": ["union_without_all"], "split": "train"} +{"case_id": "synthetic_expanded_r018_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 1, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 2, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 65}, "labels": ["union_without_all"], "split": "train"} +{"case_id": "synthetic_expanded_r019_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 93}, "labels": ["oversized_uint_type_narrowing"], "split": "train"} +{"case_id": "synthetic_expanded_r019_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 96}, "labels": ["oversized_uint_type_narrowing"], "split": "test"} +{"case_id": "synthetic_expanded_r019_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 82}, "labels": ["oversized_uint_type_narrowing"], "split": "test"} +{"case_id": "synthetic_expanded_r019_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 103}, "labels": ["oversized_uint_type_narrowing"], "split": "train"} +{"case_id": "synthetic_expanded_r019_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 95}, "labels": ["oversized_uint_type_narrowing"], "split": "train"} +{"case_id": "synthetic_expanded_r019_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 0, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 0, "has_cast_without_default": 0, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 0, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 0, "where_clause_depth": 0, "query_length_chars": 103}, "labels": ["oversized_uint_type_narrowing"], "split": "train"} +{"case_id": "synthetic_expanded_r020_001", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 44}, "labels": ["unsafe_cast_without_default", "redundant_cast_on_filter", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r020_002", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 37}, "labels": ["unsafe_cast_without_default", "redundant_cast_on_filter", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r020_003", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 47}, "labels": ["unsafe_cast_without_default", "redundant_cast_on_filter", "missing_limit_on_unbounded_result"], "split": "test"} +{"case_id": "synthetic_expanded_r020_004", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 43}, "labels": ["unsafe_cast_without_default", "redundant_cast_on_filter", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r020_005", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 45}, "labels": ["unsafe_cast_without_default", "redundant_cast_on_filter", "missing_limit_on_unbounded_result"], "split": "train"} +{"case_id": "synthetic_expanded_r020_006", "features": {"has_count_distinct": 0, "has_select_star": 0, "has_final_modifier": 0, "has_union": 0, "has_union_all": 0, "has_group_by": 0, "has_group_by_string_column": 0, "has_having": 0, "has_having_without_aggregate": 0, "has_function_on_filter_column": 1, "has_subquery": 0, "has_subquery_with_orderby_no_limit": 0, "has_nested_subquery_filter": 0, "has_or_chain_same_column": 0, "has_in_with_single_value": 0, "has_quantile_exact": 0, "has_cast": 1, "has_cast_without_default": 1, "has_async_insert_setting": 0, "has_async_insert_without_wait": 0, "has_limit": 0, "has_no_limit": 1, "has_constant_predicate": 0, "has_length_zero_check": 0, "table_count": 1, "column_count_in_select": 1, "where_clause_depth": 0, "query_length_chars": 39}, "labels": ["unsafe_cast_without_default", "redundant_cast_on_filter", "missing_limit_on_unbounded_result"], "split": "train"} diff --git a/docs/adr/ADR-009-technology-stack.md b/docs/adr/ADR-009-technology-stack.md index b6cfc38b..fc70ae34 100644 --- a/docs/adr/ADR-009-technology-stack.md +++ b/docs/adr/ADR-009-technology-stack.md @@ -12,9 +12,9 @@ MCP interface for AI-agent workflows. The stack must match what the codebase actually runs today: deterministic rules first, optional retrieval context, and no generative LLM execution in the MVP critical path. -The project still plans to add a classical ML problem classifier, but model -libraries should be introduced when the classifier code and experiments are -implemented, not kept as unused dependencies. +The project includes a classical ML problem classifier for evaluation +experiments. Heavy model libraries should stay tied to those scripts and +reports, not become an implied runtime requirement for the rule engine. ## Decision @@ -30,8 +30,9 @@ ClickAdvisor MVP standardizes on: - pytest, Hypothesis, Ruff, and mypy for development quality. The MVP intentionally does not depend on Anthropic, Ollama, vLLM, or other -generative LLM runtimes. It also does not keep CatBoost in runtime dependencies -until the classifier component is implemented and evaluated. +generative LLM runtimes. CatBoost is used only as a classifier ablation backend +when available in the evaluation environment; Logistic Regression and Random +Forest remain the portable classical baselines. ### Python 3.11+ @@ -76,9 +77,9 @@ The dependency graph becomes easier to defend: every heavy dependency has a current code path. The project no longer claims internal LLM modes that are not implemented. -When the ML classifier is added, its dependencies and SOTA/baseline rationale -must be introduced together with code, datasets, and evaluation reports. That -keeps future technical claims tied to reproducible experiments. +ML classifier dependencies and SOTA/baseline rationale must remain tied to code, +datasets, and evaluation reports. That keeps technical claims tied to +reproducible experiments. ## Alternatives Considered diff --git a/docs/adr/ADR-014-no-generative-llm-in-critical-path.md b/docs/adr/ADR-014-no-generative-llm-in-critical-path.md new file mode 100644 index 00000000..3a643c9b --- /dev/null +++ b/docs/adr/ADR-014-no-generative-llm-in-critical-path.md @@ -0,0 +1,39 @@ +# ADR-014: No Generative LLM in the Critical Path + +## Status + +Accepted + +## Context + +ClickAdvisor is evaluated as a local-first SQL advisor whose primary findings +come from deterministic rules, structured AST/SQL features, and retrieval over a +local knowledge base. Earlier product notes mentioned `--llm=none`, +`--llm=local`, and `--llm=remote`, but those modes are not part of the MVP +implementation. + +## Decision + +Generative LLMs are not allowed to select rules, change severities, rewrite +preconditions, or produce the trusted finding set in the MVP critical path. + +The production path is: + +```text +SQL -> parser -> deterministic rules -> optional ML evaluation surface + -> optional local retrieval -> report +``` + +AI agents may call ClickAdvisor through MCP, and Codex/Claude may assist +development, documentation, and manual review. Those tools are outside the +trusted analysis path. + +## Consequences + +The product story is narrower but easier to audit: each primary recommendation +has a rule id, tier, and reproducible matcher. Retrieval can add source context, +and ML classifiers can be evaluated as experiments, but neither replaces the +deterministic rule engine. + +Future generated explanations may be reconsidered only downstream of validated +findings and only if they cannot alter rule selection or confidence. diff --git a/docs/adr/README.md b/docs/adr/README.md index 7832fe99..767f5783 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -13,3 +13,4 @@ - [ADR-011: MCP Server as Secondary Interface](./ADR-011-mcp-server-interface.md) - Adds an MCP server as a thin second interface over the same core while keeping the CLI primary. - [ADR-012: Educational `explain` Mode](./ADR-012-explain-why-mode.md) - Introduces a separate explanation-first mode and requires explain templates in Tier 1 rule cards. - [ADR-013: Embedding Model Selection](./ADR-013-embedding-model-selection.md) - Keeps multilingual E5 as the default retrieval model while allowing MiniLM-L6 for English-only KB deployments. +- [ADR-014: No Generative LLM in the Critical Path](./ADR-014-no-generative-llm-in-critical-path.md) - Freezes the trusted runtime boundary around rules, ML evaluation, and local retrieval rather than generated recommendations. diff --git a/docs/evaluation.md b/docs/evaluation.md index a0ec292c..59455fa3 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -1,69 +1,14 @@ -# Evaluation Plan +# Evaluation -This document defines how ClickAdvisor quality is measured, compared, and -reported for v1.0 and later releases. +ClickAdvisor reports three separate quality surfaces. They should not be merged +into a single headline number because they measure different things. -ClickAdvisor has three evaluation surfaces: +## 1. Deterministic Rule Detection -1. deterministic rule quality -2. retrieval advisory quality -3. integration quality for CLI, EXPLAIN ESTIMATE, and MCP +The rule engine is fixed code, not a trained model. Its benchmark answers: +"Did the analyzer fire the expected implemented rules on this labeled case?" -## 1. Evaluation goals - -The evaluation framework must answer these questions: - -1. Does the system detect the right problem type? -2. Does it fire the right rule with the right tier semantics? -3. Does version filtering suppress rules that should not apply? -4. Does retrieval return documentation that helps explain the fired rules? -5. Does EXPLAIN ESTIMATE produce conservative before/after impact summaries? -6. Are CLI and MCP interfaces behaviorally consistent? -7. Is runtime latency acceptable for local and CI usage? - -## 2. Primary metrics - -### Per-rule precision and recall - -For each rule in scope: - -- precision = true positive matches / all matches emitted for the rule -- recall = true positive matches / all benchmark cases where the rule should - have fired - -This is the main unit for rule-catalog evolution because it reveals which rules -are trustworthy and which ones over-trigger or under-trigger. - -### Rule engine test pass rate (not an ML metric) - -The deterministic rule engine is regression-tested, not trained. Reporting it -as "F1" creates a false impression that a model could generalize incorrectly -on this surface. Instead this project reports it as **test pass rate**, -the same way unit-test coverage is reported for any deterministic codebase. - -Recommended reporting: - -- cases passed / cases total (overall) -- per-rule true positive / false positive / false negative counts -- explicit negative-case count (queries with zero expected findings) - -100% pass rate is the *expected* outcome for a correct deterministic matcher, -not a sign of overfitting. There is nothing to overfit: each rule is a fixed -AST/regex pattern, not a parameter learned from data. A failing case here means -a bug in the rule implementation or in the test itself, not a generalization -gap. - -Current hand-authored synthetic benchmark command: - -```bash -poetry run python scripts/eval/run_benchmark.py -``` - -The original curated 20-case synthetic benchmark is a smoke/regression suite for -implemented rules. It is intentionally small and should not be presented as a -generalization result. - -Expanded synthetic benchmark command: +Current expanded synthetic run: ```bash poetry run python scripts/eval/run_benchmark.py \ @@ -71,274 +16,74 @@ poetry run python scripts/eval/run_benchmark.py \ --mode strict ``` -`benchmark/cases/synthetic_expanded/` contains 162 generated cases with a fixed -train/test split in `benchmark/splits/synthetic_expanded_v1.yaml`. This metric -is still a rule-regression metric, not a trained-model result: F1 answers -"does the deterministic analyzer fire the expected implemented rules on -generated variations?", not "does ML generalize to unseen production queries?". -The ML classifier evaluation must report train/test metrics separately. +Result on 2026-06-30: -Be precise about what this split means here: it is useful metadata for -downstream ML experiments (see "ML classifier evaluation" below), but for the -rule engine itself it does not test generalization, because positive -variations within a rule template share the same structural AST pattern (only -literals, table names, and column names differ). This is documented explicitly -so the number is never mistaken for an ML accuracy claim. +| Dataset | Cases | Precision | Recall | F1 | Notes | +|---|---:|---:|---:|---:|---| +| `synthetic_expanded` | 180 | 1.000 | 1.000 | 1.000 | Generated rule-regression set | +| held-out split only | 36 | 1.000 | 1.000 | 1.000 | `synthetic_expanded_v1.yaml` test IDs | -**The only place F1 should be reported in this project is the ML classifier** -(see "ML classifier evaluation" below), where there is an actual learned -decision boundary and an actual risk of misclassifying an unseen case. +This is a regression result for deterministic matchers. It should not be +presented as ML generalization. -### Retrieval MRR@3 +## 2. ML Classifier Ablation -Retrieval quality is measured by mean reciprocal rank at 3 (`MRR@3`) over -synthetic benchmark cases. +The classifier is the only surface where F1 measures a learned decision +boundary. It uses AST/SQL features from `clickadvisor/ml/features.py` and +multi-label targets from `expected_rules_to_fire`. -Command: +Current command: ```bash -poetry run python scripts/eval/ablation_embeddings.py +poetry run python scripts/eval/ablation_classifiers.py --run-id classifier_ablation_current ``` -The ablation script: - -- compares `multilingual-e5-small`, `all-MiniLM-L6-v2`, and - `paraphrase-multilingual-MiniLM-L12-v2` -- indexes the first 2000 KB chunks for fast local comparison -- reports model size, MRR@3, elapsed time, and a recommendation -- removes temporary `.qdrant_ablation_*` directories after the run +Current results: -Latest documented 500-chunk result used for ADR-013: +| Model | Train F1 macro | Train F1 micro | Test F1 macro | Test F1 micro | Precision | Recall | +|---|---:|---:|---:|---:|---:|---:| +| logistic_regression | 0.908 | 0.868 | 0.678 | 0.870 | 0.667 | 0.704 | +| random_forest | 0.971 | 0.975 | 0.667 | 0.951 | 0.667 | 0.667 | +| catboost | 0.973 | 0.976 | 0.691 | 0.988 | 0.685 | 0.704 | -| Model | Size | MRR@3 | Notes | -|---|---:|---:|---| -| `multilingual-e5-small` | 117 MB | 0.38 | default, multilingual | -| `all-MiniLM-L6-v2` | 80 MB | 0.53 | best on English-heavy KB | -| `paraphrase-multilingual-MiniLM-L12-v2` | 420 MB | 0.30 | multilingual, larger | +Artifacts: -The default remains multilingual E5 because future KB sources and user queries -include Russian. See `docs/adr/ADR-013-embedding-model-selection.md`. +- `eval/results/classifier_ablation_current/metrics.json` +- `eval/results/classifier_ablation_current/metrics.csv` +- `docs/experiments/classifier_ablation.md` -### ML classifier evaluation (where F1 belongs) +## 3. Retrieval Ablation -This is the only evaluation surface in the project where a model is actually -trained on data and could genuinely overfit or fail to generalize. It is -implemented separately from the rule engine: `clickadvisor/ml/features.py` -extracts AST/SQL features, `clickadvisor/ml/classifier.py` trains and compares -candidate models on top of those features. +Retrieval quality is measured with `MRR@3` over explicit query-to-document gold +references. A result is relevant only when its URL/path or text matches the +gold reference for one of the expected rules. -Required reporting for every classifier experiment: - -- model name and hyperparameters -- train F1 (macro and micro) and test F1 (macro and micro), reported - side by side so a large train/test gap is visible at a glance, not hidden -- precision/recall per problem-type label -- exact train/test split used (must reuse - `benchmark/splits/synthetic_expanded_v1.yaml` train/test case IDs, never a - fresh random split per run, so results are comparable across experiments) -- baseline comparison: majority-class baseline, then increasingly complex - models (e.g. Logistic Regression as a simple baseline, Random Forest as a - mid-complexity baseline, CatBoost as the practical state-of-the-art choice - for tabular data) - -A test F1 close to 1.0 here would warrant the same scrutiny that was correctly -raised for the rule-engine number, because here it would actually be -plausible evidence of overfitting or of a test split too similar to train. -Unlike the rule engine, a perfect score on this surface is a signal to -investigate, not an expected baseline. - -Ablation script and results: see -`scripts/eval/ablation_classifiers.py` and `docs/experiments/classifier_ablation.md`. - -### EXPLAIN ESTIMATE impact quality - -For recommendations with rewrites, ClickAdvisor can attach planner-estimated -impact: - -- rows reduction factor -- marks reduction factor -- rows reduction percentage - -This is not a measured runtime speedup. It should be interpreted as a -planner-visible work estimate. Regression tests cover parser and comparator -behavior under `tests/explain/`. - -### MCP integration correctness - -The MCP server is tested as a protocol adaptation layer: - -- `analyze_query` returns Markdown containing rule findings -- `analyze_query_json` returns structured JSON findings -- `list_rules` returns grouped rule catalog text -- `detect_ch_version` returns either a version or a useful failure message -- `list_prompts` exposes `analyze` and `explain` - -Command: +Current command: ```bash -poetry run pytest tests/test_mcp_server.py -v -``` - -### Latency - -Measure end-to-end latency for: - -- parse + rule analysis -- retrieval advisory with existing `.qdrant_db` -- EXPLAIN ESTIMATE with reachable ClickHouse HTTP endpoint -- MCP tool call overhead - -Latency should be reported at: - -- p50 -- p95 -- max - -## 3. Datasets - -### Curated benchmark in `/benchmark/cases/` - -Primary benchmark target: generated regression coverage plus approximately 100 -hand-curated real-query cases. - -Current v1.0 synthetic subset: 20 validated cases under -`benchmark/cases/synthetic/`. - -Expanded generated subset: 162 validated cases under -`benchmark/cases/synthetic_expanded/`, including: - -- positive variations for each implemented rule/detector family -- explicit multi-label cases where rule overlaps are expected -- negative cases with no expected findings -- deterministic 80/20 split metadata for downstream ML experiments - -This dataset should contain: - -- positive and negative rule cases -- multi-issue queries -- degraded-context cases -- environment-sensitive cases -- false-friend cases where a naive rewrite should not trigger - -### Knowledge base chunks in `/data/kb/chunks/` - -Retrieval evaluation uses chunked documentation and KB material. The repository -currently contains roughly 8804 chunks. Fast ablation indexes 2000 chunks; full -index experiments are expected to improve MRR@3. - -### TPC-H, ClickBench, JOB Benchmark - -These remain secondary reference workloads for future broader validation: - -- TPC-H for recognizable analytical query shapes -- ClickBench for ClickHouse-style scans and aggregations -- JOB for join/subquery robustness after ClickHouse adaptation - -## 4. Baselines - -### Raw LLM without rule engine - -Use a baseline prompt that gives the same query and context to an LLM without -ClickAdvisor. This isolates the value of deterministic rules, version filtering, -and explicit tiers. - -### R-Bot / EverSQL - -Where legally and operationally permissible, compare against external advisors. -Record exact commit, endpoint, invocation template, and output normalization. - -Remote baselines must not receive private customer data. - -## 5. Reproducible procedure - -### Step 1: Freeze the code snapshot - -Before each benchmark run, record: - -- git commit SHA -- dirty/clean working tree state -- analyzer version -- dependency lock snapshot - -### Step 2: Freeze the benchmark selection - -Record: - -- case IDs included -- dataset source -- ClickHouse version used for metadata generation -- any environment overlays -- KB chunk count and embedding model for retrieval experiments - -### Step 3: Run ClickAdvisor - -For each case: - -- ingest SQL and optional context bundle -- run analysis in the target mode -- persist raw report JSON -- persist compact summary rows for aggregation - -### Step 4: Run retrieval ablations when relevant - -For each embedding model: - -- build a temporary Qdrant index -- query synthetic cases -- score MRR@3 -- persist model name, size, chunk count, and elapsed time - -### Step 5: Run baselines - -For each baseline: - -- execute with same or equivalent input bundle -- normalize outputs into comparable labels -- store raw and normalized outputs separately - -### Step 6: Score - -Compute: - -- per-rule precision/recall -- rule engine test pass rate -- retrieval MRR@3 -- EXPLAIN ESTIMATE impact summaries -- latency summaries - -### Step 7: Persist results - -All evaluation outputs should be saved under: - -```text -/eval/results// +poetry run python scripts/eval/ablation_embeddings.py ``` -Each run directory should contain: +Current results: -- `metadata.json` -- `system_metrics.json` -- `per_rule_metrics.json` -- `retrieval_metrics.json` when applicable -- `latency.json` -- `baseline_outputs/` -- `clickadvisor_outputs/` +| Model | Size | Queries | MRR@3 | Time (s) | +|---|---:|---:|---:|---:| +| multilingual-e5-small (current) | 117 MB | 20 | 0.458 | 17.7 | +| all-MiniLM-L6-v2 | 80 MB | 20 | 0.517 | 10.4 | +| paraphrase-multilingual-MiniLM-L12-v2 | 420 MB | 20 | 0.242 | 12.4 | -## 6. Interpretation guidance +Artifacts: -- A higher finding count is not inherently better. -- Tier confusion is a quality issue, not just a UI issue. -- Retrieval findings are documentation context, not proof of correctness. -- EXPLAIN ESTIMATE reductions are estimates, not runtime speedups. -- Latency should be interpreted together with retrieval and ClickHouse endpoint - availability. +- `eval/results/retrieval_ablation_20260630T124602Z/metrics.json` +- `docs/experiments/retrieval_ablation.md` -## 7. Minimum viable evaluation cadence +## Reproducibility Notes -Recommended cadence: +- Synthetic dataset: `benchmark/cases/synthetic_expanded` +- Split metadata: `benchmark/splits/synthetic_expanded_v1.yaml` +- Classifier script: `scripts/eval/ablation_classifiers.py` +- Retrieval script: `scripts/eval/ablation_embeddings.py` +- Benchmark runner: `scripts/eval/run_benchmark.py` -- on each major rule batch: run curated benchmark subset -- before release: run `pytest -k 'not test_detect_version'` -- before retrieval changes: run embedding ablation on the current KB slice -- before MCP changes: run `tests/test_mcp_server.py` -- before milestone demo: regenerate benchmark and ablation tables +Remote LLMs are not part of these metrics. They may assist development or call +ClickAdvisor through MCP, but they do not produce the trusted finding set. diff --git a/docs/experiments/classifier_ablation.md b/docs/experiments/classifier_ablation.md new file mode 100644 index 00000000..a87d1349 --- /dev/null +++ b/docs/experiments/classifier_ablation.md @@ -0,0 +1,34 @@ +# Classifier Ablation + +## Snapshot + +- Date: 2026-06-30 +- Dataset: `benchmark/cases/synthetic_expanded` (`180` cases) +- Split: `benchmark/splits/synthetic_expanded_v1.yaml` (`144` train, `36` test) +- Features: `clickadvisor/ml/features.py` +- Labels: multi-label `expected_rules_to_fire` +- Command: + +```bash +poetry run python scripts/eval/ablation_classifiers.py --run-id classifier_ablation_current +``` + +## Results + +| Model | Train F1 macro | Train F1 micro | Test F1 macro | Test F1 micro | Precision | Recall | +|---|---:|---:|---:|---:|---:|---:| +| logistic_regression | 0.908 | 0.868 | 0.678 | 0.870 | 0.667 | 0.704 | +| random_forest | 0.971 | 0.975 | 0.667 | 0.951 | 0.667 | 0.667 | +| catboost | 0.973 | 0.976 | 0.691 | 0.988 | 0.685 | 0.704 | + +Results were saved to `eval/results/classifier_ablation_current/`. + +## Interpretation + +Micro F1 is high because most frequent structural labels are easy to separate +from deterministic AST/text features. Macro F1 is lower because the held-out set +is small and several labels have only one or two positive examples in test. + +Random Forest is the strongest baseline in this run, but the train/test gap +shows why these numbers should be reported as synthetic classifier ablation +metrics, not as production generalization claims. diff --git a/docs/experiments/retrieval_ablation.md b/docs/experiments/retrieval_ablation.md new file mode 100644 index 00000000..1094ee68 --- /dev/null +++ b/docs/experiments/retrieval_ablation.md @@ -0,0 +1,35 @@ +# Retrieval Ablation + +## Snapshot + +- Date: 2026-06-30 +- Query set: `20` synthetic benchmark queries with expected rule labels +- KB sample: first `2000` chunks from `data/kb/chunks` out of `8804` +- Metric: `MRR@3` +- Scoring: explicit rule-to-doc gold URL fragments or keyword references +- Command: + +```bash +poetry run python scripts/eval/ablation_embeddings.py +``` + +## Results + +| Model | Size | Queries | MRR@3 | Time (s) | +|---|---:|---:|---:|---:| +| multilingual-e5-small (current) | 117 MB | 20 | 0.458 | 17.7 | +| all-MiniLM-L6-v2 | 80 MB | 20 | 0.517 | 10.4 | +| paraphrase-multilingual-MiniLM-L12-v2 | 420 MB | 20 | 0.242 | 12.4 | + +Results were saved to +`eval/results/retrieval_ablation_20260630T124602Z/`. + +## Interpretation + +The repaired metric no longer counts an arbitrary high-scoring ClickHouse chunk +as relevant. A retrieved chunk must match an explicit gold reference for one of +the expected rules. + +MiniLM-L6 is strongest on this English-heavy KB sample. The default can still +remain multilingual E5 when Russian queries and multilingual KB growth matter +more than this small English ablation. diff --git a/docs/rules/cards/D-015-optimize_table_final.yaml b/docs/rules/cards/D-015-optimize_table_final.yaml new file mode 100644 index 00000000..9e1c1d53 --- /dev/null +++ b/docs/rules/cards/D-015-optimize_table_final.yaml @@ -0,0 +1,40 @@ +id: D-015 +name: optimize_table_final +tier: detector +category: ddl_operations +status: proposed +statement: 'OPTIMIZE TABLE ... FINAL принудительно сливает все data parts в один, + игнорируя лимит max_bytes_to_merge_at_max_space_in_pool. На больших таблицах + ведёт к многочасовой I/O нагрузке, возможному OOM и созданию частей в сотни ГБ, + которые невозможно разбить обратно.' +preconditions: + syntactic: + - Запрос содержит OPTIMIZE TABLE ... FINAL + semantic: [] + data: [] +proof: + status: advisory + notes: 'Документировано в ClickHouse docs: OPTIMIZE FINAL игнорирует стандартные + лимиты слияния. Altinity KB: не рекомендуется для таблиц с более 10 млн строк.' +recommendation_template: 'OPTIMIZE TABLE ... FINAL обнаружен в SQL. Эта операция + перезаписывает все data parts в один, игнорируя max_bytes_to_merge_at_max_space_in_pool. + На больших таблицах это многочасовая I/O нагрузка и риск OOM. Используйте вместо + этого FINAL-квалификатор в SELECT: SELECT ... FROM t FINAL — он гарантирует + дедупликацию при чтении без принудительного слияния.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'OPTIMIZE TABLE events FINAL' +example_after: 'SELECT event_id, user_id FROM events FINAL WHERE dt >= today() - 7' +expected_speedup: + estimate: null + measurement_method: null +risks: +- После выполнения OPTIMIZE FINAL последствия необратимы без ручного разбиения части +- Создание части в сотни ГБ делает будущие мутации и ALTER крайне медленными +severity: high +opt_in: false +references: +- https://clickhouse.com/docs/best-practices/avoid-optimize-final +- https://kb.altinity.com/altinity-kb-queries-and-syntax/altinity-kb-optimize-vs-optimize-final/ diff --git a/docs/rules/cards/D-016-alter_table_mutation.yaml b/docs/rules/cards/D-016-alter_table_mutation.yaml new file mode 100644 index 00000000..d6b1d434 --- /dev/null +++ b/docs/rules/cards/D-016-alter_table_mutation.yaml @@ -0,0 +1,41 @@ +id: D-016 +name: alter_table_mutation +tier: detector +category: ddl_operations +status: proposed +statement: 'ALTER TABLE ... DELETE или ALTER TABLE ... UPDATE — мутации перезаписывают + весь data part, затронутый изменением. Даже изменение одной строки вызывает + перезапись сотен ГБ. Нельзя откатить после запуска.' +preconditions: + syntactic: + - Запрос начинается с ALTER TABLE и содержит DELETE или UPDATE + semantic: [] + data: [] +proof: + status: advisory + notes: 'Документировано в ClickHouse docs: мутации — асинхронные фоновые + процессы, перезаписывающие целые data parts. Не блокируют INSERT, но создают + огромную I/O нагрузку и продолжают выполняться после рестарта сервера.' +recommendation_template: 'ALTER TABLE ... DELETE/UPDATE (мутация) обнаружен в SQL. + Мутации в ClickHouse перезаписывают весь затронутый data part — даже при изменении + одной строки. Рассмотрите альтернативы: для удаления данных — lightweight DELETE + (CH >= 22.8) или DROP PARTITION; для уточнения данных — ReplacingMergeTree или + CollapsingMergeTree.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'ALTER TABLE events DELETE WHERE dt < ''2023-01-01''' +example_after: 'DELETE FROM events WHERE dt < ''2023-01-01'' -- lightweight delete, + CH >= 22.8' +expected_speedup: + estimate: null + measurement_method: null +risks: +- Мутацию нельзя откатить после запуска +- При сложных подзапросах (x IN (SELECT ...)) нагрузка на CPU и RAM многократно возрастает +severity: high +opt_in: false +references: +- https://clickhouse.com/docs/best-practices/avoid-mutations +- https://clickhouse.com/docs/optimize/avoid-mutations diff --git a/docs/rules/cards/D-017-nullable_column_in_ddl.yaml b/docs/rules/cards/D-017-nullable_column_in_ddl.yaml new file mode 100644 index 00000000..6af97e32 --- /dev/null +++ b/docs/rules/cards/D-017-nullable_column_in_ddl.yaml @@ -0,0 +1,42 @@ +id: D-017 +name: nullable_column_in_ddl +tier: detector +category: schema_types +status: proposed +statement: 'CREATE TABLE содержит колонку типа Nullable(T). Nullable создаёт + отдельный файл UInt8-маски для трекинга NULL-значений, что увеличивает хранение + и почти всегда снижает производительность.' +preconditions: + syntactic: + - DDL содержит CREATE TABLE + - В определении колонки присутствует тип Nullable(...) + semantic: [] + data: [] +proof: + status: advisory + notes: 'Документировано в ClickHouse docs: Nullable column consumes additional + storage space и almost always negatively affects performance. Nullable type + field cannot be included in table indexes.' +recommendation_template: 'Колонка {column} объявлена как Nullable({inner_type}). + Nullable создаёт дополнительный файл UInt8-маски. Если NULL не несёт семантической + нагрузки (отсутствие значения), рассмотрите замену на {inner_type} DEFAULT 0 + или {inner_type} DEFAULT ''''.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE events (user_id UInt64, score Nullable(Float64))\n\ + ENGINE = MergeTree ORDER BY user_id" +example_after: "CREATE TABLE events (user_id UInt64, score Float64 DEFAULT 0.0)\n\ + ENGINE = MergeTree ORDER BY user_id" +expected_speedup: + estimate: '5-15%' + measurement_method: 'Сравнение времени выполнения SELECT с агрегациями по Nullable + и не-Nullable колонкам.' +risks: +- Если NULL семантически значим (отличается от «нет данных» и нуля), замена нарушит логику +severity: medium +opt_in: false +references: +- https://clickhouse.com/docs/optimize/avoid-nullable-columns +- https://clickhouse.com/docs/sql-reference/data-types/nullable diff --git a/docs/rules/cards/D-018-deprecated_ngrambf_tokenbf_index.yaml b/docs/rules/cards/D-018-deprecated_ngrambf_tokenbf_index.yaml new file mode 100644 index 00000000..c5489cbd --- /dev/null +++ b/docs/rules/cards/D-018-deprecated_ngrambf_tokenbf_index.yaml @@ -0,0 +1,38 @@ +id: D-018 +name: deprecated_ngrambf_tokenbf_index +tier: detector +category: skip_index +status: proposed +statement: 'DDL содержит skip index типа ngrambf_v1 или tokenbf_v1. Оба типа + устарели начиная с ClickHouse 26.2 и заменены более эффективным text-индексом + (инвертированный индекс).' +preconditions: + syntactic: + - DDL содержит INDEX ... TYPE ngrambf_v1 или INDEX ... TYPE tokenbf_v1 + semantic: [] + data: [] +proof: + status: advisory + notes: 'Документировано в ClickHouse docs (use-data-skipping-indices): tokenbf_v1 + и ngrambf_v1 deprecated в CH >= 26.2 в пользу text indexes.' +recommendation_template: 'Skip index {index_name} использует тип {index_type}, + который устарел в CH >= 26.2. Замените на TEXT-индекс: INDEX {index_name} + {column} TYPE text GRANULARITY 1. Text-индекс детерминирован, поддерживает + многотерминный поиск и не требует настройки размера токенов.' +ch_version: + introduced: '0.720' + deprecated: '26.2' + last_validated: '25.3' +example_before: "INDEX idx_log message TYPE ngrambf_v1(4, 1024, 2, 0) GRANULARITY\ + \ 4" +example_after: "INDEX idx_log message TYPE text GRANULARITY 1" +expected_speedup: + estimate: null + measurement_method: null +risks: +- text-индекс требует CH >= 24.1; в старых версиях недоступен +severity: medium +opt_in: false +references: +- https://clickhouse.com/docs/best-practices/use-data-skipping-indices-where-appropriate +- https://clickhouse.com/docs/optimize/skipping-indexes diff --git a/docs/rules/cards/D-019-set_zero_unlimited_skip_index.yaml b/docs/rules/cards/D-019-set_zero_unlimited_skip_index.yaml new file mode 100644 index 00000000..3a3bfac9 --- /dev/null +++ b/docs/rules/cards/D-019-set_zero_unlimited_skip_index.yaml @@ -0,0 +1,40 @@ +id: D-019 +name: set_zero_unlimited_skip_index +tier: detector +category: skip_index +status: proposed +statement: 'DDL содержит skip index типа set(0). Значение 0 означает неограниченный + набор значений на блок — индекс будет накапливать все уникальные значения + в грануле без ограничения. На высококардинальных колонках это ведёт к огромному + потреблению памяти при слияниях и нулевому эффекту пропуска блоков.' +preconditions: + syntactic: + - DDL содержит INDEX ... TYPE set(0) + semantic: [] + data: [] +proof: + status: advisory + notes: 'Документировано в ClickHouse docs: set(N) tracks a set of values up + to N per block. N=0 означает неограниченный размер — эффективен только + для колонок с гарантированно малым числом уникальных значений в грануле.' +recommendation_template: 'Skip index {index_name} использует TYPE set(0) — + неограниченный размер набора. Если колонка имеет высокую кардинальность + в пределах гранулы, индекс не сможет пропускать блоки и только замедлит + слияния. Используйте set(N) с явным ограничением (например, set(100)) + или выберите другой тип индекса (minmax, bloom_filter).' +ch_version: + introduced: '0.720' + deprecated: null + last_validated: '25.3' +example_before: "INDEX idx_status status TYPE set(0) GRANULARITY 4" +example_after: "INDEX idx_status status TYPE set(100) GRANULARITY 4" +expected_speedup: + estimate: null + measurement_method: null +risks: +- Если колонка имеет низкую кардинальность, set(0) работает корректно +severity: low +opt_in: false +references: +- https://clickhouse.com/docs/best-practices/use-data-skipping-indices-where-appropriate +- https://clickhouse.com/docs/optimize/skipping-indexes diff --git a/docs/rules/cards/D-020-partition_by_non_date_column.yaml b/docs/rules/cards/D-020-partition_by_non_date_column.yaml new file mode 100644 index 00000000..addd169d --- /dev/null +++ b/docs/rules/cards/D-020-partition_by_non_date_column.yaml @@ -0,0 +1,46 @@ +id: D-020 +name: partition_by_non_date_column +tier: detector +category: schema_design +status: proposed +statement: 'PARTITION BY применён к необработанному столбцу (не к date-функции + вроде toYYYYMM/toStartOfMonth). Если столбец имеет высокую кардинальность, + каждое уникальное значение создаёт отдельную партицию — это ведёт к Part + Explosion (слишком много parts), деградации INSERT и MergeTree-мержей.' +preconditions: + syntactic: + - DDL содержит CREATE TABLE с PARTITION BY + - PARTITION BY не содержит явную date-функцию (toYYYYMM, toStartOfMonth, + toYear, toStartOfYear, toStartOfQuarter, toDate, toStartOfWeek) + semantic: [] + data: [] +proof: + status: advisory + notes: 'Документировано в ClickHouse docs: partitioning is primarily a data + management technique. High-cardinality PARTITION BY causes part explosion, + degrading insert performance and merge operations.' +recommendation_template: 'PARTITION BY {partition_expr} использует необработанный + столбец или выражение без date-функции. Если кардинальность {partition_expr} + высокая (тысячи уникальных значений), это приведёт к слишком большому числу + партиций. Рекомендуется использовать date-функцию (toYYYYMM, toStartOfMonth) + или убедиться, что кардинальность партиционирования не превышает нескольких + сотен уникальных значений.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE events (\n user_id UInt64,\n event_type String,\n\ + \ ts DateTime\n) ENGINE = MergeTree PARTITION BY user_id ORDER BY (user_id, ts)" +example_after: "CREATE TABLE events (\n user_id UInt64,\n event_type String,\n\ + \ ts DateTime\n) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY (user_id,\ + \ ts)" +expected_speedup: + estimate: null + measurement_method: null +risks: +- Низкокардинальные столбцы (тип, страна) могут быть корректными ключами партиционирования +severity: medium +opt_in: false +references: +- https://clickhouse.com/docs/best-practices/choosing-a-partitioning-key +- https://clickhouse.com/blog/ai-generated-clickhouse-schemas-mistakes-and-advice diff --git a/docs/rules/cards/D-021-select_star_in_mv_create.yaml b/docs/rules/cards/D-021-select_star_in_mv_create.yaml new file mode 100644 index 00000000..6f253599 --- /dev/null +++ b/docs/rules/cards/D-021-select_star_in_mv_create.yaml @@ -0,0 +1,41 @@ +id: D-021 +name: select_star_in_mv_create +tier: detector +category: materialized_views +status: proposed +statement: 'CREATE MATERIALIZED VIEW ... AS SELECT * FROM ... использует SELECT *, + что делает MV хрупким: при добавлении столбца в источник MV не получает его + автоматически; при удалении столбца MV сломается. Лучше явно перечислить нужные + столбцы.' +preconditions: + syntactic: + - Запрос содержит CREATE MATERIALIZED VIEW + - Подзапрос MV использует SELECT * + semantic: [] + data: [] +proof: + status: advisory + notes: 'SELECT * в MV фиксирует список столбцов на момент создания MV. + Добавление нового столбца в источник не приводит к его появлению в MV. + Удаление столбца может сломать MV при следующем INSERT.' +recommendation_template: 'CREATE MATERIALIZED VIEW использует SELECT * FROM {source}. + Это делает MV хрупким при изменении схемы источника. Замените SELECT * на + явный список колонок, которые нужны для агрегации или трансформации.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE MATERIALIZED VIEW events_mv\nTO events_agg\nAS SELECT *\ + \ FROM events" +example_after: "CREATE MATERIALIZED VIEW events_mv\nTO events_agg\nAS SELECT user_id,\ + \ event_type, toYYYYMM(ts) AS month, count() AS cnt FROM events GROUP BY 1, 2, 3" +expected_speedup: + estimate: null + measurement_method: null +risks: +- При смене схемы источника MV с SELECT * может начать возвращать unexpected columns +severity: medium +opt_in: false +references: +- https://clickhouse.com/docs/best-practices/use-materialized-views +- https://clickhouse.com/blog/ai-generated-clickhouse-schemas-mistakes-and-advice diff --git a/docs/rules/cards/D-022-delete_from_without_where.yaml b/docs/rules/cards/D-022-delete_from_without_where.yaml new file mode 100644 index 00000000..816617b9 --- /dev/null +++ b/docs/rules/cards/D-022-delete_from_without_where.yaml @@ -0,0 +1,39 @@ +id: D-022 +name: delete_from_without_where +tier: detector +category: ddl_operations +status: proposed +statement: 'DELETE FROM table без WHERE-условия удаляет ВСЕ строки таблицы. + Lightweight DELETE без WHERE — деструктивная операция, эквивалентная TRUNCATE, + но более медленная (помечает строки как удалённые вместо дропа партиции).' +preconditions: + syntactic: + - Запрос содержит DELETE FROM без WHERE + semantic: [] + data: [] +proof: + status: advisory + notes: 'DELETE FROM t без WHERE удаляет все строки. В ClickHouse lightweight + delete (CH >= 22.8) помечает строки deleted, что медленнее TRUNCATE. + Для удаления всех данных лучше использовать TRUNCATE TABLE.' +recommendation_template: 'DELETE FROM {table} без WHERE удаляет все строки таблицы. + Если цель — очистить таблицу полностью, используйте TRUNCATE TABLE {table} + — это намного быстрее (просто удаляет все parts метаданных). Если DELETE + задуман без условий случайно, добавьте WHERE-предикат.' +ch_version: + introduced: '22.8' + deprecated: null + last_validated: '25.3' +example_before: 'DELETE FROM events' +example_after: 'TRUNCATE TABLE events -- если нужно очистить таблицу целиком' +expected_speedup: + estimate: null + measurement_method: null +risks: +- TRUNCATE необратима; DELETE FROM без WHERE тоже необратима +- DELETE FROM помечает строки deleted; фактическое удаление происходит при merge +severity: high +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/statements/delete +- https://clickhouse.com/docs/sql-reference/statements/truncate diff --git a/docs/rules/cards/D-023-mv_without_to_clause.yaml b/docs/rules/cards/D-023-mv_without_to_clause.yaml new file mode 100644 index 00000000..b245a200 --- /dev/null +++ b/docs/rules/cards/D-023-mv_without_to_clause.yaml @@ -0,0 +1,41 @@ +id: D-023 +name: mv_without_to_clause +tier: detector +category: materialized_views +status: proposed +statement: 'CREATE MATERIALIZED VIEW без TO-клаузы создаёт неявную целевую таблицу + с именем `.inner.mvname`. Управление такой таблицей сложнее: нельзя изменить + её ENGINE, добавить индекс или легко ALTER схему без пересоздания MV.' +preconditions: + syntactic: + - Запрос содержит CREATE MATERIALIZED VIEW без ключевого слова TO + semantic: [] + data: [] +proof: + status: advisory + notes: 'Altinity KB best practice: использовать явный TO-синтаксис для более + простого управления схемой MV и target table. Без TO создаётся скрытая таблица + с именем .inner.mvname, управляемая только через MV.' +recommendation_template: 'CREATE MATERIALIZED VIEW без TO создаёт скрытую целевую + таблицу .inner.{mv_name}. Лучше создать target table явно и использовать TO + синтаксис: CREATE TABLE target ... ENGINE = ...; CREATE MATERIALIZED VIEW mv TO + target AS SELECT ...' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE MATERIALIZED VIEW events_mv\nAS SELECT user_id, count() AS\ + \ cnt\nFROM events GROUP BY user_id" +example_after: "CREATE TABLE events_mv_target (user_id UInt64, cnt UInt64) ENGINE\ + \ = SummingMergeTree ORDER BY user_id;\nCREATE MATERIALIZED VIEW events_mv TO events_mv_target\n\ + AS SELECT user_id, count() AS cnt FROM events GROUP BY user_id" +expected_speedup: + estimate: null + measurement_method: null +risks: +- Изменение ENGINE скрытой таблицы невозможно без пересоздания MV +severity: low +opt_in: false +references: +- https://kb.altinity.com/altinity-kb-schema-design/materialized-views/ +- https://clickhouse.com/docs/best-practices/use-materialized-views diff --git a/docs/rules/cards/D-024-mv_with_populate.yaml b/docs/rules/cards/D-024-mv_with_populate.yaml new file mode 100644 index 00000000..d5c591e7 --- /dev/null +++ b/docs/rules/cards/D-024-mv_with_populate.yaml @@ -0,0 +1,40 @@ +id: D-024 +name: mv_with_populate +tier: detector +category: materialized_views +status: proposed +statement: 'CREATE MATERIALIZED VIEW ... POPULATE читает данные из источника в фоне + и ОДНОВРЕМЕННО начинает триггериться на новые INSERT. Данные, вставленные ВО ВРЕМЯ + backfill-а, теряются: POPULATE не атомарен. Лучше делать backfill вручную.' +preconditions: + syntactic: + - Запрос содержит CREATE MATERIALIZED VIEW ... POPULATE + semantic: [] + data: [] +proof: + status: advisory + notes: 'Altinity KB: With POPULATE the data ingested to the source table during + MV populating will not appear in MV. POPULATE does not work with TO syntax.' +recommendation_template: 'CREATE MATERIALIZED VIEW ... POPULATE создаёт окно потери + данных: строки, вставленные ВО ВРЕМЯ backfill-а, не попадут в MV. Вместо POPULATE + создайте MV без POPULATE, затем вручную: INSERT INTO target SELECT ... FROM source.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE MATERIALIZED VIEW events_mv TO events_target\nAS SELECT user_id,\ + \ count() AS cnt\nFROM events GROUP BY user_id\nPOPULATE" +example_after: "-- 1. Создать MV без POPULATE\nCREATE MATERIALIZED VIEW events_mv\ + \ TO events_target\nAS SELECT user_id, count() AS cnt FROM events GROUP BY user_id;\n\ + -- 2. Вручную залить исторические данные\nINSERT INTO events_target SELECT user_id,\ + \ count() FROM events GROUP BY user_id" +expected_speedup: + estimate: null + measurement_method: null +risks: +- При ручном backfill возможна дублирование данных если одни строки были уже в target +severity: medium +opt_in: false +references: +- https://kb.altinity.com/altinity-kb-schema-design/materialized-views/ +- https://clickhouse.com/docs/best-practices/use-materialized-views diff --git a/docs/rules/cards/D-025-mv_with_join_in_select.yaml b/docs/rules/cards/D-025-mv_with_join_in_select.yaml new file mode 100644 index 00000000..19f120fc --- /dev/null +++ b/docs/rules/cards/D-025-mv_with_join_in_select.yaml @@ -0,0 +1,44 @@ +id: D-025 +name: mv_with_join_in_select +tier: detector +category: materialized_views +status: proposed +statement: 'CREATE MATERIALIZED VIEW ... AS SELECT ... JOIN ... MV с JOIN в SELECT + привязан к ЛЕВОЙ таблице. При каждом INSERT в левую таблицу выполняется JOIN + со ВСЕЙ правой таблицей — это критически бьёт по производительности вставки.' +preconditions: + syntactic: + - Запрос содержит CREATE MATERIALIZED VIEW + - SELECT тела MV содержит JOIN + semantic: [] + data: [] +proof: + status: advisory + notes: 'Altinity KB FAQ: Can I use JOINs in MV? It is possible but it is a very + bad idea for most use cases. The MV sees only freshly inserted rows of the left + table, but executes the JOIN with the WHOLE right table on every INSERT.' +recommendation_template: 'CREATE MATERIALIZED VIEW с JOIN в SELECT выполняет JOIN + со всей правой таблицей при каждом INSERT в источник. Рассмотрите использование + Dictionary, engine=Join или engine=Set для правой таблицы — они всегда в памяти + и значительно быстрее для lookup.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE MATERIALIZED VIEW enriched_mv TO enriched_target\nAS SELECT\ + \ e.user_id, u.country, count() AS cnt\nFROM events e\nLEFT JOIN users u ON e.user_id\ + \ = u.user_id\nGROUP BY e.user_id, u.country" +example_after: "-- Используйте Dictionary для lookup вместо JOIN:\nCREATE MATERIALIZED\ + \ VIEW enriched_mv TO enriched_target\nAS SELECT user_id, dictGet('users_dict',\ + \ 'country', user_id) AS country, count() AS cnt\nFROM events\nGROUP BY user_id,\ + \ country" +expected_speedup: + estimate: null + measurement_method: null +risks: +- Dictionary может содержать устаревшие данные; JOIN даёт актуальные данные +severity: medium +opt_in: false +references: +- https://kb.altinity.com/altinity-kb-schema-design/materialized-views/ +- https://clickhouse.com/docs/best-practices/use-materialized-views diff --git a/docs/rules/cards/E-012-join_use_nulls_overhead.yaml b/docs/rules/cards/E-012-join_use_nulls_overhead.yaml new file mode 100644 index 00000000..801dbe16 --- /dev/null +++ b/docs/rules/cards/E-012-join_use_nulls_overhead.yaml @@ -0,0 +1,46 @@ +id: E-012 +name: join_use_nulls_overhead +tier: env +category: join_settings +status: proposed +statement: 'join_use_nulls = 1 оборачивает результирующие колонки внешних + JOIN (LEFT/RIGHT/FULL) в Nullable, добавляя дополнительный UInt8-столбец маски + и overhead при обработке. Значение по умолчанию join_use_nulls = 0 использует + default-значения типа вместо NULL, что эффективнее.' +preconditions: + syntactic: [] + semantic: [] + data: [] +proof: '' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: '' +example_after: '' +expected_speedup: + estimate: null + measurement_method: 'N/A — environment rule, effect зависит от workload, hardware + и текущих settings.' +risks: +- При join_use_nulls=0 внешний JOIN возвращает default value (0/'') вместо NULL — могут нарушиться ожидания клиентов +- Изменение setting влияет на все запросы с LEFT/RIGHT/FULL JOIN в сессии +opt_in: false +references: +- https://clickhouse.com/docs/best-practices/minimize-optimize-joins +hardware_preconditions: [] +config_preconditions: +- join_use_nulls = 1 (включён в профиле или сессии) +workload_preconditions: +- Запросы активно используют LEFT/RIGHT/FULL JOIN +recommendation: + setting: join_use_nulls + current_value_check: 'SELECT value FROM system.settings WHERE name = ''join_use_nulls''' + recommended_formula: 'join_use_nulls = 0 (default). Использовать join_use_nulls=1 + только при необходимости соответствия ANSI NULL-семантике в outer join.' + example: 'SET join_use_nulls = 0;' +reasoning: 'join_use_nulls=1 оборачивает колонки результата в Nullable, + добавляя дополнительный файл маски null и overhead при ANY-операциях. + join_use_nulls=0 использует default-значение (0 для числовых, пустую строку + для String), что устраняет Nullable-overhead.' +condition: '' diff --git a/docs/rules/cards/E-013-async_insert_busy_timeout_tuning.yaml b/docs/rules/cards/E-013-async_insert_busy_timeout_tuning.yaml new file mode 100644 index 00000000..cbab1ee1 --- /dev/null +++ b/docs/rules/cards/E-013-async_insert_busy_timeout_tuning.yaml @@ -0,0 +1,47 @@ +id: E-013 +name: async_insert_busy_timeout_tuning +tier: env +category: insert_settings +status: proposed +statement: 'При использовании async_insert=1 параметр async_insert_busy_timeout_ms + (default 200ms на OSS, 1000ms на Cloud) определяет максимальное время ожидания + до flush буфера. Слишком низкое значение ведёт к частым мелким flush, + слишком высокое — к задержке видимости данных.' +preconditions: + syntactic: [] + semantic: [] + data: [] +proof: '' +ch_version: + introduced: '22.6' + deprecated: null + last_validated: '25.3' +example_before: '' +example_after: '' +expected_speedup: + estimate: null + measurement_method: 'N/A — environment rule, effect зависит от workload, hardware + и текущих settings.' +risks: +- Увеличение timeout задерживает видимость свежих данных для SELECT +- Уменьшение timeout увеличивает количество мелких parts и нагрузку на merge +opt_in: false +references: +- https://clickhouse.com/docs/optimize/asynchronous-inserts +- https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy +hardware_preconditions: [] +config_preconditions: +- async_insert = 1 включён +workload_preconditions: +- Ingestion-нагрузка из многих параллельных мелких INSERT-запросов +recommendation: + setting: async_insert_busy_timeout_ms + current_value_check: 'SELECT value FROM system.settings WHERE name = ''async_insert_busy_timeout_ms''' + recommended_formula: '200-1000ms для большинства workload. + При observability (logs/traces) — 1000ms для меньшего числа parts. + При OLTP-подобных workload — 200ms для свежести данных.' + example: 'SET async_insert_busy_timeout_ms = 1000;' +reasoning: 'Flush по таймауту объединяет мелкие INSERT в один part. + Баланс между свежестью данных и количеством parts: больший timeout = меньше + parts = меньше merge-нагрузки, но свежесть данных хуже.' +condition: 'async_insert = 1 активен' diff --git a/docs/rules/cards/E-014-merge_max_bytes_for_level.yaml b/docs/rules/cards/E-014-merge_max_bytes_for_level.yaml new file mode 100644 index 00000000..dd18ed3a --- /dev/null +++ b/docs/rules/cards/E-014-merge_max_bytes_for_level.yaml @@ -0,0 +1,47 @@ +id: E-014 +name: merge_max_bytes_for_level +tier: env +category: merge_settings +status: proposed +statement: 'max_bytes_to_merge_at_max_space_in_pool (default ~150GB) определяет + максимальный размер одного слияния при достаточном свободном месте. При + объектном хранилище (S3/GCS) это значение может быть ограничено пропускной + способностью сети; при локальных NVMe его можно увеличить.' +preconditions: + syntactic: [] + semantic: [] + data: [] +proof: '' +ch_version: + introduced: '10.1016' + deprecated: null + last_validated: '25.3' +example_before: '' +example_after: '' +expected_speedup: + estimate: null + measurement_method: 'N/A — environment rule, effect зависит от workload, hardware + и текущих settings.' +risks: +- Увеличение лимита ускоряет конвергенцию к меньшему числу parts, но требует больше RAM и CPU на слияние +- Не влияет на OPTIMIZE FINAL (который игнорирует этот лимит) +opt_in: false +references: +- https://clickhouse.com/docs/operations/settings/merge-tree-settings +hardware_preconditions: +- Объём свободного дискового пространства > 2x от max_bytes_to_merge_at_max_space_in_pool +config_preconditions: [] +workload_preconditions: +- Высокая скорость ingestion приводит к накоплению мелких parts +recommendation: + setting: max_bytes_to_merge_at_max_space_in_pool + current_value_check: 'SELECT value FROM system.merge_tree_settings WHERE name = + ''max_bytes_to_merge_at_max_space_in_pool''' + recommended_formula: '150GB (default) для большинства случаев. До 300-500GB + для локального NVMe при высоком write-throughput.' + example: "-- В SETTINGS CREATE TABLE или в merge_tree секции config.xml:\nmax_bytes_to_merge_at_max_space_in_pool\ + \ = 161061273600 -- 150 GiB default" +reasoning: 'Большие слияния сокращают number of parts быстрее, что улучшает + производительность SELECT. Ограничено пропускной способностью I/O: на SSD + лимит можно поднять, на S3 — лучше оставить стандартным.' +condition: 'system.parts_columns WHERE active = 1 AND bytes > 10GB' diff --git a/docs/rules/cards/E-015-optimize_aggregation_in_order.yaml b/docs/rules/cards/E-015-optimize_aggregation_in_order.yaml new file mode 100644 index 00000000..ab773fe2 --- /dev/null +++ b/docs/rules/cards/E-015-optimize_aggregation_in_order.yaml @@ -0,0 +1,45 @@ +id: E-015 +name: optimize_aggregation_in_order +tier: env +category: aggregation_settings +status: proposed +statement: 'optimize_aggregation_in_order = 1 позволяет ClickHouse использовать + порядок сортировки таблицы для потоковой агрегации без полного накопления + в памяти, если GROUP BY ключ — это префикс ORDER BY ключа таблицы. Снижает + memory footprint агрегации.' +preconditions: + syntactic: [] + semantic: [] + data: [] +proof: '' +ch_version: + introduced: '20.8' + deprecated: null + last_validated: '25.3' +example_before: '' +example_after: '' +expected_speedup: + estimate: null + measurement_method: 'N/A — environment rule, effect зависит от workload, hardware + и текущих settings.' +risks: +- В некоторых случаях (неоднородная кардинальность ключей) может замедлить запрос +opt_in: false +references: +- https://clickhouse.com/docs/operations/settings/settings#optimize_aggregation_in_order +- https://clickhouse.com/docs/sql-reference/statements/select/group-by +hardware_preconditions: [] +config_preconditions: +- GROUP BY содержит prefix ORDER BY ключа таблицы +workload_preconditions: +- Агрегационные запросы потребляют значительный объём RAM +recommendation: + setting: optimize_aggregation_in_order + current_value_check: 'SELECT value FROM system.settings WHERE name = ''optimize_aggregation_in_order''' + recommended_formula: 'optimize_aggregation_in_order = 1 когда GROUP BY ключи + совпадают с prefix ORDER BY таблицы. Default = 0.' + example: 'SET optimize_aggregation_in_order = 1;' +reasoning: 'Потоковая агрегация по ORDER BY-совместимому ключу позволяет + ClickHouse финализировать и отправить частичные результаты сразу, снижая + peak memory usage и улучшая latency.' +condition: 'GROUP BY key prefix == ORDER BY key prefix' diff --git a/docs/rules/cards/E-016-join_algorithm_for_large_tables.yaml b/docs/rules/cards/E-016-join_algorithm_for_large_tables.yaml new file mode 100644 index 00000000..015b187c --- /dev/null +++ b/docs/rules/cards/E-016-join_algorithm_for_large_tables.yaml @@ -0,0 +1,47 @@ +id: E-016 +name: join_algorithm_for_large_tables +tier: env +category: join_settings +status: proposed +statement: 'По умолчанию ClickHouse использует hash join, который требует загрузки + правой таблицы в RAM целиком. Для JOIN с большими правыми таблицами (> RAM) + следует использовать grace_hash или partial_merge алгоритм, который может + сбросить данные на диск.' +preconditions: + syntactic: [] + semantic: [] + data: [] +proof: '' +ch_version: + introduced: '22.6' + deprecated: null + last_validated: '25.3' +example_before: '' +example_after: '' +expected_speedup: + estimate: null + measurement_method: 'N/A — environment rule, effect зависит от workload, hardware + и текущих settings.' +risks: +- grace_hash и partial_merge используют диск — значительно медленнее in-memory hash join +- Эти алгоритмы следует использовать только когда JOIN не помещается в RAM +opt_in: false +references: +- https://clickhouse.com/docs/best-practices/minimize-optimize-joins +- https://clickhouse.com/docs/sql-reference/statements/select/join +hardware_preconditions: +- Правая таблица JOIN превышает max_memory_usage +config_preconditions: [] +workload_preconditions: +- JOIN с правой таблицей > нескольких ГБ в памяти +recommendation: + setting: join_algorithm + current_value_check: 'SELECT value FROM system.settings WHERE name = ''join_algorithm''' + recommended_formula: 'join_algorithm = ''grace_hash'' для JOIN с большими правыми + таблицами, которые не помещаются в RAM. join_algorithm = ''parallel_hash'' (default + в 24.12+) для большинства случаев.' + example: 'SET join_algorithm = ''grace_hash''; -- при OOM в JOIN' +reasoning: 'grace_hash JOIN может сбрасывать промежуточные состояния на диск, + избегая OOM при JOIN с большими таблицами. partial_merge аналогично, но + требует sort-merging и медленнее. Используйте как fallback при OOM.' +condition: 'OOM при JOIN операции, правая таблица > RAM' diff --git a/docs/rules/cards/E-017-distributed_aggregation_memory_efficient.yaml b/docs/rules/cards/E-017-distributed_aggregation_memory_efficient.yaml new file mode 100644 index 00000000..20fd5edb --- /dev/null +++ b/docs/rules/cards/E-017-distributed_aggregation_memory_efficient.yaml @@ -0,0 +1,48 @@ +id: E-017 +name: distributed_aggregation_memory_efficient +tier: env +category: aggregation_settings +status: proposed +statement: 'distributed_aggregation_memory_efficient = 1 снижает потребление RAM + при distributed GROUP BY запросах: вместо накопления всех partial results на + координаторе, результаты с шардов объединяются потоком. Особенно полезно + при использовании внешней агрегации (max_bytes_before_external_group_by).' +preconditions: + syntactic: [] + semantic: [] + data: [] +proof: '' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: '' +example_after: '' +expected_speedup: + estimate: null + measurement_method: 'N/A — environment rule, effect зависит от workload, hardware + и текущих settings.' +risks: +- Потоковое слияние может быть медленнее в некоторых сценариях (высокая кардинальность) +- Требует координации между шардами с возможной дополнительной задержкой +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/statements/select/group-by +- https://clickhouse.com/docs/operations/settings/settings#distributed_aggregation_memory_efficient +hardware_preconditions: +- Distributed deployment с несколькими шардами +config_preconditions: +- Запросы выполняются через distributed table +workload_preconditions: +- GROUP BY запросы с высокой кардинальностью ключей через distributed table +recommendation: + setting: distributed_aggregation_memory_efficient + current_value_check: 'SELECT value FROM system.settings WHERE name = ''distributed_aggregation_memory_efficient''' + recommended_formula: 'distributed_aggregation_memory_efficient = 1 для distributed + GROUP BY запросов с большим числом уникальных ключей. Default = 1 с CH 21.10.' + example: 'SET distributed_aggregation_memory_efficient = 1;' +reasoning: 'При distributed GROUP BY координатор обычно накапливает все partial + results от шардов в RAM перед финальным слиянием. С + distributed_aggregation_memory_efficient координатор сливает результаты + потоком (1/256 * threads от RAM), значительно снижая peak memory.' +condition: 'Distributed GROUP BY с > 1 million уникальных ключей' diff --git a/docs/rules/cards/E-018-input_format_for_bulk_inserts.yaml b/docs/rules/cards/E-018-input_format_for_bulk_inserts.yaml new file mode 100644 index 00000000..04061448 --- /dev/null +++ b/docs/rules/cards/E-018-input_format_for_bulk_inserts.yaml @@ -0,0 +1,46 @@ +id: E-018 +name: input_format_for_bulk_inserts +tier: env +category: insert_settings +status: proposed +statement: 'Формат вставки влияет на CPU-нагрузку при парсинге: JSONEachRow ~30с + user-CPU на 100M строк, Native ~2с. Для bulk-инжестии большого объёма данных + предпочтительны форматы RowBinary, Native или Parquet вместо JSONEachRow.' +preconditions: + syntactic: [] + semantic: [] + data: [] +proof: '' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: '' +example_after: '' +expected_speedup: + estimate: null + measurement_method: 'N/A — environment rule, effect зависит от workload, hardware + и текущих settings.' +risks: +- Native и RowBinary — бинарные форматы, требуют специальных клиентов/конвертации +- Parquet требует предварительного преобразования данных +opt_in: false +references: +- https://kb.altinity.com/altinity-kb-schema-design/ingestion-performance-and-formats/ +- https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy +hardware_preconditions: [] +config_preconditions: +- Использование JSONEachRow или CSV для высокочастотной вставки +workload_preconditions: +- Bulk ingestion > 10M строк в пакете +recommendation: + setting: input_format + current_value_check: '' + recommended_formula: 'Для bulk ingestion: Native > RowBinary > TSV > Parquet > CSV > JSONEachRow + (по убыванию скорости парсинга). JSONEachRow использует в 10-15x больше CPU + чем Native.' + example: 'clickhouse-client --query "INSERT INTO t FORMAT RowBinary" < data.bin' +reasoning: 'JSONEachRow требует парсинга каждой строки как JSON-объекта с проверкой + ключей, что создаёт значительную CPU нагрузку. RowBinary/Native — бинарные форматы + без такого overhead.' +condition: 'INSERT throughput > 100k rows/s с высоким user CPU в clickhouse-client' diff --git a/docs/rules/cards/E-019-prefer_localhost_replica.yaml b/docs/rules/cards/E-019-prefer_localhost_replica.yaml new file mode 100644 index 00000000..307a1281 --- /dev/null +++ b/docs/rules/cards/E-019-prefer_localhost_replica.yaml @@ -0,0 +1,45 @@ +id: E-019 +name: prefer_localhost_replica +tier: env +category: distributed_settings +status: proposed +statement: 'prefer_localhost_replica = 1 (default) заставляет ClickHouse читать + с локальной реплики, если она есть. Это снижает сетевой трафик и задержку. + Если отключено, ClickHouse может читать с удалённой реплики даже при наличии + локальной.' +preconditions: + syntactic: [] + semantic: [] + data: [] +proof: '' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: '' +example_after: '' +expected_speedup: + estimate: null + measurement_method: 'N/A — environment rule, effect зависит от workload, hardware + и текущих settings.' +risks: +- При prefer_localhost_replica=1 локальная реплика может быть более загружена +- Если локальная реплика отстаёт (lag), запросы могут читать стale данные +opt_in: false +references: +- https://clickhouse.com/docs/operations/settings/settings#prefer_localhost_replica +hardware_preconditions: +- Distributed/replicated deployment с несколькими репликами +config_preconditions: [] +workload_preconditions: +- SELECT запросы через Distributed table +recommendation: + setting: prefer_localhost_replica + current_value_check: 'SELECT value FROM system.settings WHERE name = ''prefer_localhost_replica''' + recommended_formula: 'prefer_localhost_replica = 1 (default). Отключать только + если нужно балансировать нагрузку между репликами или локальная реплика сильно + отстаёт.' + example: 'SET prefer_localhost_replica = 1; -- default, обычно не нужно менять' +reasoning: 'Чтение с локальной реплики устраняет сетевой round-trip, уменьшает + latency и снижает сетевой трафик между нодами кластера.' +condition: 'Distributed queries с несколькими репликами' diff --git a/docs/rules/cards/E-020-max_execution_time_for_protection.yaml b/docs/rules/cards/E-020-max_execution_time_for_protection.yaml new file mode 100644 index 00000000..480bfb73 --- /dev/null +++ b/docs/rules/cards/E-020-max_execution_time_for_protection.yaml @@ -0,0 +1,45 @@ +id: E-020 +name: max_execution_time_for_protection +tier: env +category: query_settings +status: proposed +statement: 'max_execution_time = 0 (default) означает неограниченное время выполнения + запроса. Без лимита один «тяжёлый» запрос может занять все ресурсы кластера + на неопределённое время. Рекомендуется установить разумный timeout для production.' +preconditions: + syntactic: [] + semantic: [] + data: [] +proof: '' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: '' +example_after: '' +expected_speedup: + estimate: null + measurement_method: 'N/A — environment rule, effect зависит от workload, hardware + и текущих settings.' +risks: +- Слишком короткий timeout может прерывать легитимные долгие запросы +- Отменённый запрос возвращает ошибку клиенту — нужна обработка в приложении +opt_in: false +references: +- https://clickhouse.com/docs/operations/settings/query-complexity#max-execution-time +hardware_preconditions: [] +config_preconditions: +- max_execution_time = 0 (default, unlimited) +workload_preconditions: +- Production-деплоймент с многопользовательским доступом +recommendation: + setting: max_execution_time + current_value_check: 'SELECT value FROM system.settings WHERE name = ''max_execution_time''' + recommended_formula: 'max_execution_time = 300 (5 min) для интерактивных запросов. + max_execution_time = 3600 (1h) для batch-запросов. 0 оставлять только для + доверенных batch-процессов.' + example: 'SET max_execution_time = 300;' +reasoning: 'Без ограничения на время одни запросы могут блокировать ресурсы кластера + бесконечно. Разумный timeout защищает от runaway queries и улучшает предсказуемость + производительности.' +condition: 'SELECT queries от ненадёжных пользователей или неоптимизированных приложений' diff --git a/docs/rules/cards/R-021-datetime64_zero_to_datetime.yaml b/docs/rules/cards/R-021-datetime64_zero_to_datetime.yaml new file mode 100644 index 00000000..3c584133 --- /dev/null +++ b/docs/rules/cards/R-021-datetime64_zero_to_datetime.yaml @@ -0,0 +1,43 @@ +id: R-021 +name: datetime64_zero_to_datetime +tier: 1A +category: schema_types +status: proposed +statement: 'Колонка объявлена как DateTime64(0), что семантически эквивалентно + DateTime — оба хранят время с точностью до секунды. DateTime занимает 4 байта + (UInt32 epoch), DateTime64(0) — 8 байт (Int64). Замена сокращает хранение вдвое + для этой колонки.' +preconditions: + syntactic: + - DDL содержит CREATE TABLE + - Колонка объявлена с типом DateTime64(0) или DateTime64(0, timezone) + semantic: [] + data: [] + user: [] +proof: 'DateTime64(0) хранит эпох-секунды как Int64 (8 байт). DateTime хранит + эпох-секунды как UInt32 (4 байта). При precision=0 семантика полностью совпадает + в диапазоне дат DateTime (1970-2106). Разница только в диапазоне: DateTime64 + поддерживает более широкий диапазон дат.' +explain_template: 'Колонка {column} объявлена как DateTime64(0), что идентично + DateTime по точности. DateTime занимает вдвое меньше памяти (4 байта vs 8 байт). + Замените тип на DateTime для экономии хранилища и улучшения производительности + GROUP BY и сортировки.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE events (\n ts DateTime64(0),\n user_id UInt64\n)\ + \ ENGINE = MergeTree ORDER BY ts" +example_after: "CREATE TABLE events (\n ts DateTime,\n user_id UInt64\n) ENGINE\ + \ = MergeTree ORDER BY ts" +expected_speedup: + estimate: '~50% compression для колонки timestamp' + measurement_method: 'Сравнение compressed_bytes по колонке в system.parts_columns + до и после замены типа.' +risks: +- DateTime поддерживает диапазон 1970-01-01 — 2106-02-07; DateTime64(0) шире +- При репликации между разными версиями CH возможна несовместимость типов +opt_in: false +references: +- https://clickhouse.com/docs/best-practices/select-data-types +- https://clickhouse.com/docs/sql-reference/data-types/datetime64 diff --git a/docs/rules/cards/R-022-float_monetary_column_to_decimal.yaml b/docs/rules/cards/R-022-float_monetary_column_to_decimal.yaml new file mode 100644 index 00000000..169b06af --- /dev/null +++ b/docs/rules/cards/R-022-float_monetary_column_to_decimal.yaml @@ -0,0 +1,45 @@ +id: R-022 +name: float_monetary_column_to_decimal +tier: 1B +category: schema_types +status: proposed +statement: 'Колонка объявлена как Float32 или Float64, но имя указывает на + монетарное значение (amount, price, cost, revenue, fee, total, balance, tax). + Float-арифметика не ассоциативна и даёт неточные результаты для денежных сумм. + Рекомендуется Decimal(p, s).' +preconditions: + syntactic: + - DDL содержит CREATE TABLE + - Колонка объявлена с типом Float32 или Float64 + - Имя колонки содержит паттерн монетарного домена + semantic: [] + data: [] + user: + - Проверить реальный диапазон и нужную точность перед заменой +proof: 'Float-арифметика не ассоциативна: (a + b) - a != b при больших разницах + в порядке. Для финансовых данных это недопустимо. Decimal(p, s) даёт точные + целочисленные вычисления со смещённой точкой.' +explain_template: 'Колонка {column} объявлена как {col_type}, но по имени + предположительно хранит денежные значения. Float не подходит для финансовых + расчётов из-за ошибок округления. Используйте Decimal32(2) для сумм до ~21M + или Decimal64(2) для больших значений.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE orders (\n order_id UInt64,\n total_amount Float64,\n\ + \ tax_amount Float64\n) ENGINE = MergeTree ORDER BY order_id" +example_after: "CREATE TABLE orders (\n order_id UInt64,\n total_amount Decimal64(2),\n\ + \ tax_amount Decimal64(2)\n) ENGINE = MergeTree ORDER BY order_id" +expected_speedup: + estimate: null + measurement_method: 'Сравнение точности агрегации SUM() по Float64 и Decimal64(2) + на реальных данных.' +risks: +- Decimal ограничен по диапазону (Decimal32 до ~21M, Decimal64 до ~9.2 * 10^16) +- Decimal не поддерживает NaN и Infinity +- Конвертация требует перезаливки данных +opt_in: true +references: +- https://kb.altinity.com/altinity-kb-schema-design/floats-vs-decimals/ +- https://clickhouse.com/docs/best-practices/select-data-types diff --git a/docs/rules/cards/R-023-string_datetime_column_to_date.yaml b/docs/rules/cards/R-023-string_datetime_column_to_date.yaml new file mode 100644 index 00000000..a79ab978 --- /dev/null +++ b/docs/rules/cards/R-023-string_datetime_column_to_date.yaml @@ -0,0 +1,43 @@ +id: R-023 +name: string_datetime_column_to_date +tier: 1B +category: schema_types +status: proposed +statement: 'Колонка объявлена как String, но имя указывает на дату или время + (date, datetime, timestamp, created_at, updated_at, event_time, event_date, + occurred_at). Хранение дат как String лишает возможности использовать + date-функции напрямую, индексацию по дате и сжатие Delta-кодеком.' +preconditions: + syntactic: + - DDL содержит CREATE TABLE + - Колонка объявлена с типом String + - Имя колонки содержит паттерн даты/времени + semantic: [] + data: [] + user: + - Проверить реальный формат дат в данных перед заменой +proof: 'String хранит дату как UTF-8 текст (~10-19 байт). Date хранит как UInt16 + (2 байта). DateTime хранит как UInt32 (4 байта). Числовые типы дат сжимаются + кодеком Delta намного лучше, чем строки.' +explain_template: 'Колонка {column} объявлена как String, но по имени предположительно + хранит дату или время. Это препятствует использованию date-функций без явного + toDate()/toDateTime(), исключает Delta-сжатие и замедляет диапазонные фильтры. + Используйте Date или DateTime.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE events (\n event_id UInt64,\n created_at String,\n\ + \ event_date String\n) ENGINE = MergeTree ORDER BY event_id" +example_after: "CREATE TABLE events (\n event_id UInt64,\n created_at DateTime,\n\ + \ event_date Date\n) ENGINE = MergeTree ORDER BY (event_date, event_id)" +expected_speedup: + estimate: '3-5x сжатие для колонки дат' + measurement_method: 'Сравнение data_compressed_bytes в system.parts_columns.' +risks: +- Требует валидации формата строк в данных перед конвертацией +- Строки вида '2024-01-01 00:00:00.000' требуют уточнения точности (DateTime vs DateTime64) +opt_in: true +references: +- https://clickhouse.com/docs/best-practices/select-data-types +- https://clickhouse.com/docs/sql-reference/data-types/datetime diff --git a/docs/rules/cards/R-024-string_ip_column_to_ipv4.yaml b/docs/rules/cards/R-024-string_ip_column_to_ipv4.yaml new file mode 100644 index 00000000..503295eb --- /dev/null +++ b/docs/rules/cards/R-024-string_ip_column_to_ipv4.yaml @@ -0,0 +1,44 @@ +id: R-024 +name: string_ip_column_to_ipv4 +tier: 1B +category: schema_types +status: proposed +statement: 'Колонка объявлена как String, но имя указывает на IP-адрес + (ip, ip_address, remote_addr, client_ip, server_ip, src_ip, dst_ip, remote_host). + Тип IPv4 хранит адрес в 4 байтах (UInt32) вместо ~15 байт строки и поддерживает + нативные IP-функции (IPv4CIDRToRange, toIPv4 и др.).' +preconditions: + syntactic: + - DDL содержит CREATE TABLE + - Колонка объявлена с типом String + - Имя колонки содержит паттерн IP-адреса + semantic: [] + data: [] + user: + - Проверить реальный формат IP-адресов (IPv4, IPv6 или смешанный) +proof: >- + IPv4 тип хранит адрес как UInt32 (4 байта). Строка '192.168.1.1' + занимает 11 байт + overhead. Экономия ~4x плюс нативная поддержка + CIDR-фильтрации. +explain_template: 'Колонка {column} объявлена как String, но по имени предположительно + хранит IP-адрес. Тип IPv4 экономит ~4x памяти, поддерживает нативные функции + IPv4NumToString/toIPv4/IPv4CIDRToRange и сортируется корректно для числовой + семантики адресов. Для IPv6 используйте тип IPv6.' +ch_version: + introduced: '116.253' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE access_log (\n request_id UInt64,\n client_ip String,\n\ + \ remote_addr String\n) ENGINE = MergeTree ORDER BY request_id" +example_after: "CREATE TABLE access_log (\n request_id UInt64,\n client_ip IPv4,\n\ + \ remote_addr IPv4\n) ENGINE = MergeTree ORDER BY request_id" +expected_speedup: + estimate: '~4x сжатие для колонки IP' + measurement_method: 'Сравнение data_compressed_bytes в system.parts_columns.' +risks: +- IPv6-адреса не поместятся в IPv4 (нужен IPv6 или String) +- Смешанные форматы (IPv4-mapped IPv6) требуют дополнительной обработки +opt_in: true +references: +- https://kb.altinity.com/altinity-kb-schema-design/how-to-store-ips/ +- https://clickhouse.com/docs/sql-reference/data-types/ipv4 diff --git a/docs/rules/cards/R-025-order_by_tuple_no_pk.yaml b/docs/rules/cards/R-025-order_by_tuple_no_pk.yaml new file mode 100644 index 00000000..fcd2ab89 --- /dev/null +++ b/docs/rules/cards/R-025-order_by_tuple_no_pk.yaml @@ -0,0 +1,42 @@ +id: R-025 +name: order_by_tuple_no_pk +tier: 1B +category: schema_design +status: proposed +statement: 'CREATE TABLE с явным ORDER BY tuple() означает отсутствие первичного + ключа. Без primary key каждый SELECT выполняет полный скан таблицы — sparse + index недоступен. Для аналитических запросов с фильтрацией это ведёт к + многократно избыточному чтению с диска.' +preconditions: + syntactic: + - DDL содержит CREATE TABLE + - DDL содержит ORDER BY tuple() + semantic: [] + data: [] + user: + - Требуется анализ паттернов запросов для выбора правильного ORDER BY +proof: 'ClickHouse хранит данные, отсортированные по ORDER BY-ключу, и строит + sparse primary index по этому ключу. ORDER BY tuple() отключает сортировку + и sparse index, гарантируя полный скан при каждом SELECT с WHERE.' +explain_template: 'Таблица создаётся без первичного ключа (ORDER BY tuple()). + Каждый SELECT будет сканировать все данные. Выберите ORDER BY по колонкам, + наиболее часто используемым в WHERE-условиях, например ORDER BY (user_id, + event_date).' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE events (\n user_id UInt64,\n event_type String,\n\ + \ ts DateTime\n) ENGINE = MergeTree ORDER BY tuple()" +example_after: "CREATE TABLE events (\n user_id UInt64,\n event_type String,\n\ + \ ts DateTime\n) ENGINE = MergeTree ORDER BY (user_id, ts)" +expected_speedup: + estimate: '10-1000x для selective queries' + measurement_method: 'Сравнение read_rows в system.query_log до и после добавления ORDER BY.' +risks: +- Изменение ORDER BY требует пересоздания и перезаливки таблицы +- Неправильный выбор ORDER BY может замедлить INSERT (сортировка данных при вставке) +opt_in: true +references: +- https://clickhouse.com/docs/best-practices/choosing-a-primary-key +- https://clickhouse.com/docs/blog/ai-generated-clickhouse-schemas-mistakes-and-advice diff --git a/docs/rules/cards/R-026-sum_case_to_countif.yaml b/docs/rules/cards/R-026-sum_case_to_countif.yaml new file mode 100644 index 00000000..962ccabb --- /dev/null +++ b/docs/rules/cards/R-026-sum_case_to_countif.yaml @@ -0,0 +1,35 @@ +id: R-026 +name: sum_case_to_countif +tier: 1A +category: aggregate_rewrite +status: proposed +statement: 'SUM(CASE WHEN cond THEN 1 ELSE 0 END) или COUNT(CASE WHEN cond THEN + 1 END) эквивалентны countIf(cond). countIf — встроенный комбинатор ClickHouse, + работает без материализации промежуточного CASE-выражения.' +preconditions: + syntactic: + - Запрос содержит SUM(CASE WHEN ... THEN 1 ELSE 0 END) или COUNT(CASE WHEN ... THEN 1 END) + semantic: [] + data: [] + user: [] +proof: 'countIf(cond) — синтаксический сахар поверх count() с суффиксом -If, + семантически идентичен COUNT(CASE WHEN cond THEN 1 END). ClickHouse выполняет + countIf без создания временного Int8-столбца результата CASE.' +explain_template: 'SUM(CASE WHEN ... THEN 1 ELSE 0 END) в позиции агрегации + заменяется более читаемым и эффективным countIf(cond). Код проще и не требует + материализации промежуточного CASE-результата.' +ch_version: + introduced: '1.23' + deprecated: null + last_validated: '25.3' +example_before: "SELECT\n SUM(CASE WHEN status = 'active' THEN 1 ELSE 0 END) AS\ + \ active_cnt,\n COUNT(CASE WHEN score > 90 THEN 1 END) AS high_scorers\nFROM users" +example_after: "SELECT\n countIf(status = 'active') AS active_cnt,\n countIf(score\ + \ > 90) AS high_scorers\nFROM users" +expected_speedup: + estimate: '5-10% на больших наборах данных' + measurement_method: 'EXPLAIN PIPELINE до и после; сравнение elapsed в system.query_log.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/aggregate-functions/combinators diff --git a/docs/rules/cards/R-027-sum_case_col_to_sumif.yaml b/docs/rules/cards/R-027-sum_case_col_to_sumif.yaml new file mode 100644 index 00000000..600e4839 --- /dev/null +++ b/docs/rules/cards/R-027-sum_case_col_to_sumif.yaml @@ -0,0 +1,36 @@ +id: R-027 +name: sum_case_col_to_sumif +tier: 1A +category: aggregate_rewrite +status: proposed +statement: 'SUM(CASE WHEN cond THEN col ELSE 0 END) семантически эквивалентно + sumIf(col, cond). sumIf — встроенный комбинатор ClickHouse, не требует + материализации CASE-выражения в промежуточный столбец.' +preconditions: + syntactic: + - Запрос содержит SUM(CASE WHEN ... THEN ELSE 0 END) + semantic: + - ELSE-ветка возвращает 0 (или эквивалент нуля) + data: [] + user: [] +proof: 'sumIf(col, cond) применяет сложение только к строкам, где cond=true, + эквивалентно SUM(CASE WHEN cond THEN col ELSE 0 END). При ELSE 0 результаты + математически идентичны, так как 0 нейтрален в сложении.' +explain_template: 'SUM(CASE WHEN {cond} THEN {col} ELSE 0 END) заменяется + sumIf({col}, {cond}). Это устраняет промежуточное CASE-вычисление и делает + код более читаемым.' +ch_version: + introduced: '1.23' + deprecated: null + last_validated: '25.3' +example_before: "SELECT\n SUM(CASE WHEN is_paid THEN amount ELSE 0 END) AS paid_total\n\ + FROM orders" +example_after: "SELECT\n sumIf(amount, is_paid) AS paid_total\nFROM orders" +expected_speedup: + estimate: '5-10% на больших наборах данных' + measurement_method: 'Сравнение elapsed в system.query_log.' +risks: +- Если ELSE-ветка не 0 (например, ELSE -1), замена нарушает семантику +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/aggregate-functions/combinators diff --git a/docs/rules/cards/R-028-coalesce_two_args_to_ifnull.yaml b/docs/rules/cards/R-028-coalesce_two_args_to_ifnull.yaml new file mode 100644 index 00000000..b5b34947 --- /dev/null +++ b/docs/rules/cards/R-028-coalesce_two_args_to_ifnull.yaml @@ -0,0 +1,32 @@ +id: R-028 +name: coalesce_two_args_to_ifnull +tier: 1A +category: function_rewrite +status: proposed +statement: 'COALESCE(x, y) с ровно двумя аргументами полностью эквивалентен + ifNull(x, y). ifNull — встроенная функция ClickHouse, которая компилируется + более эффективно и не требует перебора переменного числа аргументов.' +preconditions: + syntactic: + - Запрос содержит COALESCE( с ровно двумя аргументами + semantic: [] + data: [] + user: [] +proof: 'COALESCE(x, y) = первый не-NULL из x, y = ifNull(x, y) при двух + аргументах. Это математически идентично. ifNull — специализированная + функция без overhead перебора varargs.' +explain_template: 'COALESCE({x}, {y}) с двумя аргументами заменяется ifNull({x}, {y}). + Семантика идентична, ifNull компилируется без overhead varargs-перебора.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT COALESCE(user_name, 'anonymous') AS name FROM users" +example_after: "SELECT ifNull(user_name, 'anonymous') AS name FROM users" +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; основное преимущество — читаемость и совместимость с типом.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/functions-for-nulls diff --git a/docs/rules/cards/R-029-lower_col_like_to_ilike.yaml b/docs/rules/cards/R-029-lower_col_like_to_ilike.yaml new file mode 100644 index 00000000..99273572 --- /dev/null +++ b/docs/rules/cards/R-029-lower_col_like_to_ilike.yaml @@ -0,0 +1,36 @@ +id: R-029 +name: lower_col_like_to_ilike +tier: 1A +category: function_rewrite +status: proposed +statement: 'lower(col) LIKE pattern семантически эквивалентен col ILIKE pattern. + ILIKE — нативный оператор ClickHouse (с версии 22.6), который выполняет + регистронезависимое сравнение без явного вызова lower(), что упрощает + код и потенциально позволяет оптимизатору использовать skip-индексы.' +preconditions: + syntactic: + - Запрос содержит lower(col) LIKE pattern + semantic: [] + data: [] + user: [] +proof: 'ILIKE(col, pat) в ClickHouse применяет lower() к обоим операндам перед + сравнением, поэтому lower(col) LIKE lower_pat === col ILIKE pat для ASCII-символов. + Паттерн lower(col) LIKE требует двух операций (вызов lower + LIKE), ILIKE — одной.' +explain_template: 'lower({col}) LIKE ''{pattern}'' заменяется {col} ILIKE ''{pattern}''. + Оба выражения регистронезависимы. ILIKE — нативный оператор, не требующий + явной конвертации к lowercase.' +ch_version: + introduced: '22.6' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM logs WHERE lower(message) LIKE '%error%'" +example_after: "SELECT * FROM logs WHERE message ILIKE '%error%'" +expected_speedup: + estimate: null + measurement_method: 'Упрощение плана запроса; реальное ускорение зависит от наличия text-индекса.' +risks: +- ILIKE доступен только начиная с CH 22.6 +- Для non-ASCII символов поведение lower() и ILIKE может различаться в зависимости от locale +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/string-search-functions diff --git a/docs/rules/cards/R-030-not_in_singleton_to_not_equal.yaml b/docs/rules/cards/R-030-not_in_singleton_to_not_equal.yaml new file mode 100644 index 00000000..b032d46e --- /dev/null +++ b/docs/rules/cards/R-030-not_in_singleton_to_not_equal.yaml @@ -0,0 +1,35 @@ +id: R-030 +name: not_in_singleton_to_not_equal +tier: 1A +category: predicate_rewrite +status: proposed +statement: 'x NOT IN (single_value) семантически эквивалентен x != single_value + для не-Nullable колонок. NOT IN с одним элементом строит hash set из одного + значения, тогда как != — прямое скалярное сравнение без накладных расходов + на построение set.' +preconditions: + syntactic: + - WHERE содержит NOT IN с ровно одним литеральным значением + semantic: + - Колонка не Nullable (NULL NOT IN (x) возвращает NULL, а не TRUE) + data: [] + user: [] +proof: 'Для не-Nullable x: x NOT IN (c) ≡ x != c по стандарту SQL. + При одном элементе IN-оператор строит hash set O(1) вместо O(1) сравнения, + что излишне. ClickHouse optimizer не всегда упрощает это автоматически.' +explain_template: '{col} NOT IN ({val}) заменяется {col} != {val}. + Семантика идентична для не-Nullable колонок, запрос становится проще.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM events WHERE status NOT IN ('deleted')" +example_after: "SELECT * FROM events WHERE status != 'deleted'" +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость и простота плана запроса.' +risks: +- Для Nullable колонок семантика различается (NULL != x возвращает NULL, а не TRUE) +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/operators/in diff --git a/docs/rules/cards/R-031-string_uuid_column_to_uuid_type.yaml b/docs/rules/cards/R-031-string_uuid_column_to_uuid_type.yaml new file mode 100644 index 00000000..4ca9bf8f --- /dev/null +++ b/docs/rules/cards/R-031-string_uuid_column_to_uuid_type.yaml @@ -0,0 +1,44 @@ +id: R-031 +name: string_uuid_column_to_uuid_type +tier: 1B +category: schema_types +status: proposed +statement: 'Колонка объявлена как String, но имя указывает на UUID (uuid, guid, + trace_id, span_id, request_id). UUID-тип хранит идентификатор в 16 байтах + vs ~36 байт строки с дефисами, и поддерживает нативные UUID-функции.' +preconditions: + syntactic: + - DDL содержит CREATE TABLE + - Колонка объявлена с типом String + - Имя колонки указывает на UUID-домен + semantic: [] + data: [] + user: + - Проверить реальный формат значений (должны быть валидные UUIDs) +proof: >- + UUID хранит 128 бит (16 байт) в бинарном виде. String UUID + '61f0c404-5cb3-11e7-907b-a6006ad3dba0' занимает 36 байт. Экономия + ~2.25x плюс нативная поддержка generateUUIDv4(), UUIDStringToNum(), + toUUID(). +explain_template: 'Колонка {column} объявлена как String, но по имени предположительно + хранит UUID. Тип UUID занимает 16 байт вместо ~36, поддерживает нативные + UUID-функции и корректно сортируется. Примечание: UUIDv7 сортируется по + второй половине в ClickHouse, что может влиять на PRIMARY KEY.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE spans (\n trace_id String,\n span_id String,\n ts\ + \ DateTime\n) ENGINE = MergeTree ORDER BY ts" +example_after: "CREATE TABLE spans (\n trace_id UUID,\n span_id UUID,\n ts\ + \ DateTime\n) ENGINE = MergeTree ORDER BY ts" +expected_speedup: + estimate: '~2x сжатие для UUID-колонок' + measurement_method: 'Сравнение data_compressed_bytes в system.parts_columns.' +risks: +- UUIDv7 в PRIMARY KEY даёт неоптимальную сортировку из-за особенности хранения в CH +- Требует изменения INSERT-логики клиента (вставка UUID вместо строки) +opt_in: true +references: +- https://clickhouse.com/docs/sql-reference/data-types/uuid +- https://clickhouse.com/docs/best-practices/select-data-types diff --git a/docs/rules/cards/R-032-int8_boolean_column_to_bool.yaml b/docs/rules/cards/R-032-int8_boolean_column_to_bool.yaml new file mode 100644 index 00000000..d4a3dff4 --- /dev/null +++ b/docs/rules/cards/R-032-int8_boolean_column_to_bool.yaml @@ -0,0 +1,42 @@ +id: R-032 +name: int8_boolean_column_to_bool +tier: 1B +category: schema_types +status: proposed +statement: 'Колонка объявлена как UInt8 или Int8, но имя указывает на булево + значение (is_*, has_*, flag, enabled, active, deleted, published, visible). + Тип Bool хранится как UInt8 (те же 1 байт), но выводится как true/false, + обеспечивает самодокументирование схемы и проверку допустимых значений.' +preconditions: + syntactic: + - DDL содержит CREATE TABLE + - Колонка объявлена с типом UInt8 или Int8 + - Имя колонки указывает на булев домен + semantic: [] + data: [] + user: + - Убедиться, что значения только 0 и 1 (Bool отклоняет другие значения при строгом режиме) +proof: 'Bool тип в ClickHouse хранится как UInt8 (1 байт). Замена UInt8 на + Bool не меняет хранение, но добавляет семантическую ясность и совместимость + с клиентами, ожидающими boolean-тип.' +explain_template: 'Колонка {column} объявлена как {col_type}, но по имени + предположительно хранит булево значение. Замените на Bool для явного + документирования домена. Хранение идентично UInt8 (1 байт).' +ch_version: + introduced: '22.1' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE users (\n user_id UInt64,\n is_active UInt8,\n has_subscription\ + \ Int8\n) ENGINE = MergeTree ORDER BY user_id" +example_after: "CREATE TABLE users (\n user_id UInt64,\n is_active Bool,\n has_subscription\ + \ Bool\n) ENGINE = MergeTree ORDER BY user_id" +expected_speedup: + estimate: null + measurement_method: 'N/A — изменение семантики, не производительности.' +risks: +- Bool отклоняет значения кроме 0 и 1 при включённом strict_types +- Требует CH >= 22.1 +opt_in: true +references: +- https://clickhouse.com/docs/sql-reference/data-types/boolean +- https://clickhouse.com/docs/best-practices/select-data-types diff --git a/docs/rules/cards/R-033-max_case_to_maxif.yaml b/docs/rules/cards/R-033-max_case_to_maxif.yaml new file mode 100644 index 00000000..9b1f166a --- /dev/null +++ b/docs/rules/cards/R-033-max_case_to_maxif.yaml @@ -0,0 +1,33 @@ +id: R-033 +name: max_case_to_maxif +tier: 1A +category: aggregate_rewrite +status: proposed +statement: 'MAX(CASE WHEN cond THEN col END) семантически эквивалентен maxIf(col, cond) + — встроенному комбинатору ClickHouse. Когда cond = false, CASE возвращает NULL, + который MAX игнорирует — то же поведение, что maxIf.' +preconditions: + syntactic: + - Запрос содержит MAX(CASE WHEN ... THEN col END) без ELSE + semantic: [] + data: [] + user: [] +proof: 'MAX(CASE WHEN cond THEN col END) = MAX(col WHERE cond) = maxIf(col, cond). + NULL в CASE (когда cond=false) игнорируется MAX, аналогично поведению maxIf.' +explain_template: 'MAX(CASE WHEN {cond} THEN {col} END) заменяется maxIf({col}, {cond}). + Без материализации CASE-выражения. Результат идентичен.' +ch_version: + introduced: '1.23' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT MAX(CASE WHEN status = ''active'' THEN score END) AS max_active_score + FROM users' +example_after: 'SELECT maxIf(score, status = ''active'') AS max_active_score FROM users' +expected_speedup: + estimate: '5-10%' + measurement_method: 'Сравнение elapsed в system.query_log.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/aggregate-functions/combinators +- https://clickhouse.com/blog/aggregate-functions-combinators-in-clickhouse-for-arrays-maps-and-states diff --git a/docs/rules/cards/R-034-min_case_to_minif.yaml b/docs/rules/cards/R-034-min_case_to_minif.yaml new file mode 100644 index 00000000..c42d9797 --- /dev/null +++ b/docs/rules/cards/R-034-min_case_to_minif.yaml @@ -0,0 +1,31 @@ +id: R-034 +name: min_case_to_minif +tier: 1A +category: aggregate_rewrite +status: proposed +statement: 'MIN(CASE WHEN cond THEN col END) семантически эквивалентен minIf(col, cond) + — встроенному комбинатору ClickHouse. NULL в CASE (при cond=false) игнорируется MIN.' +preconditions: + syntactic: + - Запрос содержит MIN(CASE WHEN ... THEN col END) без ELSE + semantic: [] + data: [] + user: [] +proof: 'MIN(CASE WHEN cond THEN col END) = MIN(col WHERE cond) = minIf(col, cond). + NULL из CASE игнорируется MIN — аналогично поведению minIf.' +explain_template: 'MIN(CASE WHEN {cond} THEN {col} END) заменяется minIf({col}, {cond}). + Результат идентичен, без материализации CASE.' +ch_version: + introduced: '1.23' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT MIN(CASE WHEN is_paid THEN amount END) AS min_paid FROM orders' +example_after: 'SELECT minIf(amount, is_paid) AS min_paid FROM orders' +expected_speedup: + estimate: '5-10%' + measurement_method: 'Сравнение elapsed в system.query_log.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/aggregate-functions/combinators +- https://clickhouse.com/blog/aggregate-functions-combinators-in-clickhouse-for-arrays-maps-and-states diff --git a/docs/rules/cards/R-035-avg_case_to_avgif.yaml b/docs/rules/cards/R-035-avg_case_to_avgif.yaml new file mode 100644 index 00000000..fd71e5be --- /dev/null +++ b/docs/rules/cards/R-035-avg_case_to_avgif.yaml @@ -0,0 +1,33 @@ +id: R-035 +name: avg_case_to_avgif +tier: 1A +category: aggregate_rewrite +status: proposed +statement: 'AVG(CASE WHEN cond THEN col END) семантически эквивалентен avgIf(col, cond) + — встроенному комбинатору ClickHouse. NULL в CASE (при cond=false) исключается + из вычисления среднего — то же поведение avgIf.' +preconditions: + syntactic: + - Запрос содержит AVG(CASE WHEN ... THEN col END) без ELSE + semantic: [] + data: [] + user: [] +proof: 'AVG(CASE WHEN cond THEN col END) = SUM(col WHERE cond) / COUNT(col WHERE cond) + = avgIf(col, cond). AVG игнорирует NULL, аналогично avgIf.' +explain_template: 'AVG(CASE WHEN {cond} THEN {col} END) заменяется avgIf({col}, {cond}). + Результат идентичен, без материализации CASE.' +ch_version: + introduced: '1.23' + deprecated: null + last_validated: '25.3' +example_before: "SELECT AVG(CASE WHEN status = 'paid' THEN amount END) AS avg_paid\ + \ FROM orders" +example_after: "SELECT avgIf(amount, status = 'paid') AS avg_paid FROM orders" +expected_speedup: + estimate: '5-10%' + measurement_method: 'Сравнение elapsed в system.query_log.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/aggregate-functions/combinators +- https://clickhouse.com/blog/aggregate-functions-combinators-in-clickhouse-for-arrays-maps-and-states diff --git a/docs/rules/cards/R-036-nested_if_to_multiif.yaml b/docs/rules/cards/R-036-nested_if_to_multiif.yaml new file mode 100644 index 00000000..5003f2ec --- /dev/null +++ b/docs/rules/cards/R-036-nested_if_to_multiif.yaml @@ -0,0 +1,34 @@ +id: R-036 +name: nested_if_to_multiif +tier: 1A +category: function_rewrite +status: proposed +statement: 'IF(c1, v1, IF(c2, v2, v3)) — вложенный IF — эквивалентен + multiIf(c1, v1, c2, v2, v3). multiIf — нативная функция ClickHouse для + множественных условий, более читаемая и без накладных расходов вложенности.' +preconditions: + syntactic: + - Запрос содержит IF(..., ..., IF(..., ..., ...)) — один IF вложен в ELSE-ветку другого + semantic: [] + data: [] + user: [] +proof: 'multiIf(c1, v1, c2, v2, else_v) ≡ IF(c1, v1, IF(c2, v2, else_v)). + Проверяет условия последовательно, возвращает первое совпавшее значение. + Семантически идентичны.' +explain_template: 'Вложенный IF(c1, v1, IF(c2, v2, v3)) заменяется + multiIf(c1, v1, c2, v2, v3). Более читаемо и устраняет нежелательную вложенность.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT IF(score > 90, 'A', IF(score > 70, 'B', 'C')) AS grade FROM\ + \ students" +example_after: "SELECT multiIf(score > 90, 'A', score > 70, 'B', 'C') AS grade FROM\ + \ students" +expected_speedup: + estimate: null + measurement_method: 'Минимальное ускорение; основное преимущество — читаемость.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/conditional-functions diff --git a/docs/rules/cards/R-037-empty_string_eq_to_empty.yaml b/docs/rules/cards/R-037-empty_string_eq_to_empty.yaml new file mode 100644 index 00000000..d7953e65 --- /dev/null +++ b/docs/rules/cards/R-037-empty_string_eq_to_empty.yaml @@ -0,0 +1,32 @@ +id: R-037 +name: empty_string_eq_to_empty +tier: 1A +category: string_operations +status: proposed +statement: "col = '' для String-колонки семантически эквивалентно empty(col). + Функция empty() — специализированный предикат ClickHouse, не требующий + сравнения с пустым строковым литералом." +preconditions: + syntactic: + - Запрос содержит col = '' (col сравнивается с пустой строкой) + semantic: [] + data: [] + user: [] +proof: 'По определению ClickHouse: empty(s) истинно тогда и только тогда, + когда s == ''''. Для String типа это математически идентично.' +explain_template: "{col} = '' заменяется empty({col}). Более читаемо и\ + \ выражает намерение напрямую." +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM events WHERE message = ''" +example_after: 'SELECT * FROM events WHERE empty(message)' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: +- Для FixedString(N) empty() проверяет все N байт на нули, a = '' использует полное сравнение +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/string-functions diff --git a/docs/rules/cards/R-038-nonempty_string_neq_to_notempty.yaml b/docs/rules/cards/R-038-nonempty_string_neq_to_notempty.yaml new file mode 100644 index 00000000..6f21d887 --- /dev/null +++ b/docs/rules/cards/R-038-nonempty_string_neq_to_notempty.yaml @@ -0,0 +1,32 @@ +id: R-038 +name: nonempty_string_neq_to_notempty +tier: 1A +category: string_operations +status: proposed +statement: "col != '' или col <> '' для String-колонки семантически эквивалентно + notEmpty(col). Функция notEmpty() — специализированный предикат ClickHouse, + прямо выражающий намерение." +preconditions: + syntactic: + - "Запрос содержит col != '' или col <> '' (col не равно пустой строке)" + semantic: [] + data: [] + user: [] +proof: "notEmpty(s) истинно тогда и только тогда, когда s != ''. + Для String типа это математически идентично NOT empty(s)." +explain_template: "{col} != '' заменяется notEmpty({col}). Более читаемо и\ + \ выражает намерение напрямую." +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM events WHERE message != ''" +example_after: 'SELECT * FROM events WHERE notEmpty(message)' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: +- Для FixedString(N) notEmpty() и != '' могут иметь разное поведение для строк с нулевыми байтами +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/string-functions diff --git a/docs/rules/cards/R-039-length_gte_one_to_notempty.yaml b/docs/rules/cards/R-039-length_gte_one_to_notempty.yaml new file mode 100644 index 00000000..14823eea --- /dev/null +++ b/docs/rules/cards/R-039-length_gte_one_to_notempty.yaml @@ -0,0 +1,30 @@ +id: R-039 +name: length_gte_one_to_notempty +tier: 1A +category: string_operations +status: proposed +statement: 'length(x) >= 1 для String/Array/Map — эквивалентно notEmpty(x). + Аналогично R-013 (length(x) > 0), но использует оператор >= 1 вместо > 0.' +preconditions: + syntactic: + - Запрос содержит length(x) >= 1 + semantic: [] + data: [] + user: [] +proof: 'length(x) >= 1 ≡ length(x) > 0 ≡ notEmpty(x) для String, Array, Map. + length() всегда неотрицательна, поэтому >= 1 и > 0 математически эквивалентны.' +explain_template: 'length({col}) >= 1 заменяется notEmpty({col}). + notEmpty — специализированный предикат, более читаемый.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT * FROM events WHERE length(tags) >= 1' +example_after: 'SELECT * FROM events WHERE notEmpty(tags)' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/string-functions diff --git a/docs/rules/cards/R-040-todate_comparison_to_datetime.yaml b/docs/rules/cards/R-040-todate_comparison_to_datetime.yaml new file mode 100644 index 00000000..14ff23d8 --- /dev/null +++ b/docs/rules/cards/R-040-todate_comparison_to_datetime.yaml @@ -0,0 +1,38 @@ +id: R-040 +name: todate_comparison_to_datetime +tier: 1A +category: sargable +status: proposed +statement: 'toDate(datetime_col) >= date / > date / <= date / < date — применение + toDate() к колонке делает predicate не-sargable для DateTime primary key. + Прямое сравнение datetime_col с конвертированной DateTime-константой позволяет + использовать sparse primary index. Дополняет R-005 для операторов сравнения.' +preconditions: + syntactic: + - WHERE содержит toDate(col) OP literal где OP in (>=, >, <=, <) + semantic: + - Тип колонки — DateTime или DateTime64 + data: [] + user: [] +proof: 'toDate(dt) >= D ≡ dt >= toDateTime(D) (полночь D). + toDate(dt) > D ≡ dt >= toDateTime(D+1day). + toDate(dt) < D ≡ dt < toDateTime(D). + toDate(dt) <= D ≡ dt < toDateTime(D+1day). + Всё это арифметические тождества на временной оси.' +explain_template: 'toDate({col}) {op} ''{date}'' оборачивает колонку в функцию, + из-за чего ClickHouse не может использовать sparse primary index. + Замените на прямое сравнение {col} с DateTime-константой.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT * FROM events WHERE toDate(ts) >= ''2024-01-01''' +example_after: 'SELECT * FROM events WHERE ts >= ''2024-01-01 00:00:00''' +expected_speedup: + estimate: '10-1000x для selective queries' + measurement_method: 'Сравнение Granules processed в EXPLAIN indexes=1.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/date-time-functions +- https://clickhouse.com/docs/best-practices/choosing-a-primary-key diff --git a/docs/rules/cards/R-041-string_code_column_to_fixedstring.yaml b/docs/rules/cards/R-041-string_code_column_to_fixedstring.yaml new file mode 100644 index 00000000..ffe54413 --- /dev/null +++ b/docs/rules/cards/R-041-string_code_column_to_fixedstring.yaml @@ -0,0 +1,43 @@ +id: R-041 +name: string_code_column_to_fixedstring +tier: 1B +category: schema_types +status: proposed +statement: 'Колонка объявлена как String, но имя содержит суффикс _code или + _iso, указывающий на строку фиксированной длины (country_code, currency_code, + language_code, lang_iso). Тип FixedString(N) хранит строки точно N байт + без length-prefix overhead, что эффективнее String для значений фиксированной длины.' +preconditions: + syntactic: + - DDL содержит CREATE TABLE + - Колонка объявлена как String + - Имя колонки содержит _code или _iso суффикс + semantic: [] + data: [] + user: + - "Проверить фактическую длину значений перед заменой (country_code: 2, currency: 3)" +proof: 'FixedString(N) хранит ровно N байт без prefix длины. String хранит + с 1-4 байтной prefix длины. Для фиксированных кодов (ISO 2-3 символа) + FixedString(2/3) занимает меньше места и компрессируется лучше.' +explain_template: 'Колонка {column} объявлена как String, но по имени предположительно + хранит код фиксированной длины. Используйте FixedString(2) для ISO-кодов стран + (''US'', ''DE'') или FixedString(3) для кодов валют (''USD'', ''EUR'').' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "CREATE TABLE orders (\n order_id UInt64,\n country_code String,\n\ + \ currency_code String\n) ENGINE = MergeTree ORDER BY order_id" +example_after: "CREATE TABLE orders (\n order_id UInt64,\n country_code FixedString(2),\n\ + \ currency_code FixedString(3)\n) ENGINE = MergeTree ORDER BY order_id" +expected_speedup: + estimate: '~30% сжатие для колонки кодов' + measurement_method: 'Сравнение data_compressed_bytes в system.parts_columns.' +risks: +- FixedString(N) дополняет строки нулевыми байтами при вставке; сравнение + с != сложнее из-за trailing null bytes +- Требует точного знания длины всех возможных значений +opt_in: true +references: +- https://clickhouse.com/docs/sql-reference/data-types/fixedstring +- https://clickhouse.com/docs/best-practices/select-data-types diff --git a/docs/rules/cards/R-042-grouparray_without_limit.yaml b/docs/rules/cards/R-042-grouparray_without_limit.yaml new file mode 100644 index 00000000..ba93ff68 --- /dev/null +++ b/docs/rules/cards/R-042-grouparray_without_limit.yaml @@ -0,0 +1,36 @@ +id: R-042 +name: grouparray_without_limit +tier: 1B +category: aggregate_rewrite +status: proposed +statement: 'groupArray(col) без ограничения размера накапливает ВСЕ значения + в массив в памяти на агрегирующем узле. При большом числе строк в группе + это может исчерпать RAM. Рекомендуется явно задать лимит: groupArray(N)(col).' +preconditions: + syntactic: + - Запрос содержит groupArray(col) без аргумента N + semantic: [] + data: [] + user: + - Оценить максимальный размер группы перед добавлением лимита +proof: 'groupArray(col) накапливает все значения без ограничения. + groupArray(N)(col) останавливается после N значений. Без лимита на больших + данных результат может не помещаться в RAM.' +explain_template: 'groupArray({col}) без лимита накапливает все значения колонки + для каждой группы в RAM. При большом числе строк это может вызвать OOM. + Добавьте явный лимит: groupArray({N})({col}), где N — максимально нужное число.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT user_id, groupArray(event_type) AS events FROM log GROUP BY user_id' +example_after: 'SELECT user_id, groupArray(1000)(event_type) AS events FROM log GROUP BY user_id' +expected_speedup: + estimate: null + measurement_method: 'Контроль peak memory usage в system.query_log.' +risks: +- Лимит обрезает массив; если нужны все значения, лимит не подходит +severity: medium +opt_in: true +references: +- https://clickhouse.com/docs/sql-reference/aggregate-functions/reference/grouparray diff --git a/docs/rules/cards/R-043-having_count_gt_zero_removal.yaml b/docs/rules/cards/R-043-having_count_gt_zero_removal.yaml new file mode 100644 index 00000000..2319bdfa --- /dev/null +++ b/docs/rules/cards/R-043-having_count_gt_zero_removal.yaml @@ -0,0 +1,36 @@ +id: R-043 +name: having_count_gt_zero_removal +tier: 1A +category: predicate_rewrite +status: proposed +statement: 'GROUP BY x HAVING count() > 0 — избыточное условие: после GROUP BY + каждая группа по определению содержит хотя бы одну строку, поэтому count() + всегда >= 1. HAVING count() > 0 никогда не отфильтровывает строки.' +preconditions: + syntactic: + - Запрос содержит GROUP BY ... HAVING count() > 0 + semantic: + - Нет ссылок на NULL-возможные ключи которые могут создавать пустые группы + data: [] + user: [] +proof: 'По определению GROUP BY: каждая результирующая строка соответствует + хотя бы одной строке исходной таблицы. Следовательно, count() >= 1 всегда. + HAVING count() > 0 = HAVING 1 > 0 = HAVING true — тавтология.' +explain_template: 'HAVING count() > 0 после GROUP BY является тавтологией: + каждая группа содержит минимум одну строку. Удалите это условие — оно не + влияет на результат, но добавляет лишний шаг фильтрации.' +ch_version: + introduced: '0.5' + deprecated: null + last_validated: '25.3' +example_before: "SELECT user_id, count() FROM events GROUP BY user_id HAVING count()\ + \ > 0" +example_after: 'SELECT user_id, count() FROM events GROUP BY user_id' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — упрощение плана запроса.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/statements/select/group-by +- https://clickhouse.com/docs/sql-reference/statements/select/having diff --git a/docs/rules/cards/R-044-todatetime_todate_to_startofday.yaml b/docs/rules/cards/R-044-todatetime_todate_to_startofday.yaml new file mode 100644 index 00000000..2c661add --- /dev/null +++ b/docs/rules/cards/R-044-todatetime_todate_to_startofday.yaml @@ -0,0 +1,32 @@ +id: R-044 +name: todatetime_todate_to_startofday +tier: 1A +category: function_rewrite +status: proposed +statement: 'toDateTime(toDate(ts)) — двойное преобразование, вычисляющее начало + текущего дня (полночь) для DateTime/DateTime64 колонки. Нативная функция + toStartOfDay(ts) делает то же самое напрямую и более читаема.' +preconditions: + syntactic: + - Запрос содержит toDateTime(toDate(col)) + semantic: [] + data: [] + user: [] +proof: 'toDate(ts) = дата без времени. toDateTime(date) = date + 00:00:00. + Итог: toDateTime(toDate(ts)) = начало текущего дня для ts. + toStartOfDay(ts) возвращает то же самое.' +explain_template: 'toDateTime(toDate({col})) вычисляет начало дня через двойное + преобразование. toStartOfDay({col}) — нативная функция для той же операции.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT toDateTime(toDate(ts)) AS day_start FROM events' +example_after: 'SELECT toStartOfDay(ts) AS day_start FROM events' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/date-time-functions diff --git a/docs/rules/cards/R-045-like_without_wildcards_to_eq.yaml b/docs/rules/cards/R-045-like_without_wildcards_to_eq.yaml new file mode 100644 index 00000000..1f240c69 --- /dev/null +++ b/docs/rules/cards/R-045-like_without_wildcards_to_eq.yaml @@ -0,0 +1,32 @@ +id: R-045 +name: like_without_wildcards_to_eq +tier: 1A +category: predicate_rewrite +status: proposed +statement: "col LIKE 'pattern' где pattern не содержит символов-шаблонов (% и _) + семантически эквивалентен col = 'pattern'. LIKE без шаблонов компилируется + как регулярное выражение, тогда как = — прямое сравнение без накладных расходов." +preconditions: + syntactic: + - WHERE содержит col LIKE 'literal' где literal не содержит % или _ + semantic: [] + data: [] + user: [] +proof: "По определению SQL: LIKE 'pattern' без % и _ является exact match. + Это математически идентично = 'pattern'. + LIKE с шаблонами компилируется в regexp, без шаблонов — избыточный overhead." +explain_template: "{col} LIKE '{pattern}' без шаблонов заменяется {col} = '{pattern}'. + = выполняет прямое сравнение без regexp overhead." +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM events WHERE event_type LIKE 'click'" +example_after: "SELECT * FROM events WHERE event_type = 'click'" +expected_speedup: + estimate: '5-20% при высокой частоте выполнения' + measurement_method: 'Сравнение elapsed в system.query_log.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/string-search-functions diff --git a/docs/rules/cards/R-046-not_empty_to_notempty.yaml b/docs/rules/cards/R-046-not_empty_to_notempty.yaml new file mode 100644 index 00000000..e1695fb1 --- /dev/null +++ b/docs/rules/cards/R-046-not_empty_to_notempty.yaml @@ -0,0 +1,30 @@ +id: R-046 +name: not_empty_to_notempty +tier: 1A +category: string_operations +status: proposed +statement: 'NOT empty(x) семантически эквивалентно notEmpty(x). notEmpty — нативная + функция ClickHouse, избавляющая от оператора NOT и более читаемая.' +preconditions: + syntactic: + - Запрос содержит NOT empty(col) + semantic: [] + data: [] + user: [] +proof: 'По определению: notEmpty(x) = NOT empty(x). Это задокументированный + псевдоним в ClickHouse. Семантически идентично.' +explain_template: 'NOT empty({col}) заменяется notEmpty({col}). + Нативный предикат, более читаемый.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT * FROM events WHERE NOT empty(message)' +example_after: 'SELECT * FROM events WHERE notEmpty(message)' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/string-functions diff --git a/docs/rules/cards/R-047-position_gt_zero_to_like.yaml b/docs/rules/cards/R-047-position_gt_zero_to_like.yaml new file mode 100644 index 00000000..e9878d0a --- /dev/null +++ b/docs/rules/cards/R-047-position_gt_zero_to_like.yaml @@ -0,0 +1,30 @@ +id: R-047 +name: position_gt_zero_to_like +tier: 1A +category: function_rewrite +status: proposed +statement: 'position(col, ''pattern'') > 0 семантически эквивалентно col LIKE ''%pattern%''. + LIKE может использовать skip-индекс (ngrambf_v1, text), position() — нет.' +preconditions: + syntactic: + - Запрос содержит position(col, ''literal'') > 0 + semantic: [] + data: [] + user: [] +proof: 'position(haystack, needle) возвращает 1-indexed позицию или 0 если не найдено. + > 0 значит "найдено". LIKE ''%pattern%'' проверяет то же самое.' +explain_template: 'position({col}, ''{pattern}'') > 0 заменяется {col} LIKE ''%{pattern}%''. + LIKE может использовать skip-индекс для ускорения поиска.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM logs WHERE position(message, 'error') > 0" +example_after: "SELECT * FROM logs WHERE message LIKE '%error%'" +expected_speedup: + estimate: null + measurement_method: null +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/string-search-functions diff --git a/docs/rules/cards/R-048-positioncaseinsensitive_to_ilike.yaml b/docs/rules/cards/R-048-positioncaseinsensitive_to_ilike.yaml new file mode 100644 index 00000000..332a75fb --- /dev/null +++ b/docs/rules/cards/R-048-positioncaseinsensitive_to_ilike.yaml @@ -0,0 +1,32 @@ +id: R-048 +name: positioncaseinsensitive_to_ilike +tier: 1A +category: function_rewrite +status: proposed +statement: 'positionCaseInsensitive(col, ''pattern'') > 0 семантически эквивалентно + col ILIKE ''%pattern%''. ILIKE более читаем и может использовать case-insensitive + skip-индексы.' +preconditions: + syntactic: + - Запрос содержит positionCaseInsensitive(col, ''literal'') > 0 + semantic: [] + data: [] + user: [] +proof: 'positionCaseInsensitive(s, n) > 0 = "n есть в s без учёта регистра" = + s ILIKE ''%n%''.' +explain_template: 'positionCaseInsensitive({col}, ''{pattern}'') > 0 заменяется + {col} ILIKE ''%{pattern}%''.' +ch_version: + introduced: '22.6' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM logs WHERE positionCaseInsensitive(message, 'error') > 0" +example_after: "SELECT * FROM logs WHERE message ILIKE '%error%'" +expected_speedup: + estimate: null + measurement_method: null +risks: +- ILIKE требует CH >= 22.6 +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/string-search-functions diff --git a/docs/rules/cards/R-049-sumif_one_to_countif.yaml b/docs/rules/cards/R-049-sumif_one_to_countif.yaml new file mode 100644 index 00000000..d6a4ede6 --- /dev/null +++ b/docs/rules/cards/R-049-sumif_one_to_countif.yaml @@ -0,0 +1,29 @@ +id: R-049 +name: sumif_one_to_countif +tier: 1A +category: aggregate_rewrite +status: proposed +statement: 'sumIf(1, cond) — подсчёт строк через SUM — эквивалентен countIf(cond). + countIf — нативная функция ClickHouse для условного подсчёта строк.' +preconditions: + syntactic: + - Запрос содержит sumIf(1, condition) + semantic: [] + data: [] + user: [] +proof: 'sumIf(1, cond) = SUM of 1 for rows where cond = count of rows where cond + = countIf(cond). Математически идентично.' +explain_template: 'sumIf(1, {cond}) заменяется countIf({cond}).' +ch_version: + introduced: '1.23' + deprecated: null + last_validated: '25.3' +example_before: "SELECT sumIf(1, status = 'active') AS active_cnt FROM users" +example_after: "SELECT countIf(status = 'active') AS active_cnt FROM users" +expected_speedup: + estimate: null + measurement_method: null +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/aggregate-functions/combinators diff --git a/docs/rules/cards/R-050-toyyyymm_comparison_to_range.yaml b/docs/rules/cards/R-050-toyyyymm_comparison_to_range.yaml new file mode 100644 index 00000000..4381896f --- /dev/null +++ b/docs/rules/cards/R-050-toyyyymm_comparison_to_range.yaml @@ -0,0 +1,33 @@ +id: R-050 +name: toyyyymm_comparison_to_range +tier: 1A +category: sargable +status: proposed +statement: 'toYYYYMM(col) >= YYYYMM или toYYYYMM(col) > YYYYMM оборачивают колонку + в функцию, запрещая использование sparse primary index. Прямое сравнение col с + DateTime-константой первого числа месяца восстанавливает sargability. Дополняет + R-006 для операторов >= и >.' +preconditions: + syntactic: + - WHERE содержит toYYYYMM(col) OP YYYYMM_literal где OP in (>=, >, <=, <) + semantic: [] + data: [] + user: [] +proof: 'toYYYYMM(dt) >= YYYYMM ≡ dt >= toDateTime(''YYYY-MM-01 00:00:00''). + Арифметическое тождество на временной оси.' +explain_template: 'toYYYYMM({col}) {op} {yyyymm} оборачивает колонку в функцию. + Замените на прямое сравнение {col} с DateTime-константой.' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM events WHERE toYYYYMM(ts) >= 202401" +example_after: "SELECT * FROM events WHERE ts >= '2024-01-01 00:00:00'" +expected_speedup: + estimate: '10-1000x для selective queries' + measurement_method: 'EXPLAIN indexes=1 сравнение Granules processed.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/date-time-functions +- https://clickhouse.com/docs/best-practices/choosing-a-primary-key diff --git a/docs/rules/cards/R-051-date_trunc_to_native.yaml b/docs/rules/cards/R-051-date_trunc_to_native.yaml new file mode 100644 index 00000000..50a2faf3 --- /dev/null +++ b/docs/rules/cards/R-051-date_trunc_to_native.yaml @@ -0,0 +1,30 @@ +id: R-051 +name: date_trunc_to_native +tier: 1A +category: function_rewrite +status: proposed +statement: "DATE_TRUNC('unit', ts) — SQL-стандартная функция — имеет точные нативные + аналоги в ClickHouse: toStartOfDay, toStartOfHour, toStartOfMonth и т.д. Нативные + функции более явны и не требуют парсинга строки-единицы." +preconditions: + syntactic: + - Запрос содержит DATE_TRUNC('unit', col) где unit — 'day', 'hour', 'month', 'year' + semantic: [] + data: [] + user: [] +proof: "DATE_TRUNC('day', ts) = toStartOfDay(ts). DATE_TRUNC('month', ts) = toStartOfMonth(ts). + Документировано в ClickHouse как эквивалентные функции." +explain_template: "DATE_TRUNC('{unit}', {col}) заменяется toStartOf{Unit}({col})." +ch_version: + introduced: '21.4' + deprecated: null + last_validated: '25.3' +example_before: "SELECT DATE_TRUNC('day', ts) AS day FROM events" +example_after: 'SELECT toStartOfDay(ts) AS day FROM events' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость CH-кода.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/date-time-functions diff --git a/docs/rules/cards/R-052-formatdatetime_ymd_to_toString.yaml b/docs/rules/cards/R-052-formatdatetime_ymd_to_toString.yaml new file mode 100644 index 00000000..2504da50 --- /dev/null +++ b/docs/rules/cards/R-052-formatdatetime_ymd_to_toString.yaml @@ -0,0 +1,32 @@ +id: R-052 +name: formatdatetime_ymd_to_toString +tier: 1A +category: function_rewrite +status: proposed +statement: "formatDateTime(ts, '%Y-%m-%d') возвращает дату в формате YYYY-MM-DD. + Это идентично toString(toDate(ts)), но требует парсинга format-строки. toString(toDate(ts)) + — более прямой и эффективный путь." +preconditions: + syntactic: + - "Запрос содержит formatDateTime(col, '%Y-%m-%d')" + semantic: [] + data: [] + user: [] +proof: "formatDateTime(ts, '%Y-%m-%d') = 'YYYY-MM-DD' для любого DateTime. + toString(toDate(ts)) = 'YYYY-MM-DD' (формат по умолчанию для Date в ClickHouse). + Математически идентично." +explain_template: "formatDateTime({col}, '%Y-%m-%d') заменяется toString(toDate({col})). + toString(toDate) более прямой без overhead парсинга format-строки." +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT formatDateTime(ts, '%Y-%m-%d') AS day FROM events" +example_after: 'SELECT toString(toDate(ts)) AS day FROM events' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/date-time-functions diff --git a/docs/rules/cards/R-053-arrayreduce_sum_to_arraysum.yaml b/docs/rules/cards/R-053-arrayreduce_sum_to_arraysum.yaml new file mode 100644 index 00000000..a7cbc293 --- /dev/null +++ b/docs/rules/cards/R-053-arrayreduce_sum_to_arraysum.yaml @@ -0,0 +1,29 @@ +id: R-053 +name: arrayreduce_sum_to_arraysum +tier: 1A +category: function_rewrite +status: proposed +statement: "arrayReduce('sum', arr) эквивалентен arraySum(arr). arraySum — специализированная + нативная функция ClickHouse, более читаемая и не требующая парсинга строки." +preconditions: + syntactic: + - "Запрос содержит arrayReduce('sum', arr)" + semantic: [] + data: [] + user: [] +proof: "arrayReduce('sum', arr) применяет агрегацию sum ко всем элементам arr. + arraySum(arr) делает то же самое. Документировано как эквиваленты." +explain_template: "arrayReduce('sum', {arr}) заменяется arraySum({arr})." +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT arrayReduce('sum', amounts) AS total FROM orders" +example_after: 'SELECT arraySum(amounts) AS total FROM orders' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/array-functions diff --git a/docs/rules/cards/R-054-arrayreduce_max_to_arraymax.yaml b/docs/rules/cards/R-054-arrayreduce_max_to_arraymax.yaml new file mode 100644 index 00000000..854aba6a --- /dev/null +++ b/docs/rules/cards/R-054-arrayreduce_max_to_arraymax.yaml @@ -0,0 +1,28 @@ +id: R-054 +name: arrayreduce_max_to_arraymax +tier: 1A +category: function_rewrite +status: proposed +statement: "arrayReduce('max', arr) эквивалентен arrayMax(arr). arrayMax — специализированная + нативная функция ClickHouse." +preconditions: + syntactic: + - "Запрос содержит arrayReduce('max', arr)" + semantic: [] + data: [] + user: [] +proof: "arrayReduce('max', arr) = arrayMax(arr). Задокументированный псевдоним." +explain_template: "arrayReduce('max', {arr}) заменяется arrayMax({arr})." +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT arrayReduce('max', scores) AS top_score FROM results" +example_after: 'SELECT arrayMax(scores) AS top_score FROM results' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/array-functions diff --git a/docs/rules/cards/R-055-arrayreduce_min_to_arraymin.yaml b/docs/rules/cards/R-055-arrayreduce_min_to_arraymin.yaml new file mode 100644 index 00000000..cf8ec8d6 --- /dev/null +++ b/docs/rules/cards/R-055-arrayreduce_min_to_arraymin.yaml @@ -0,0 +1,28 @@ +id: R-055 +name: arrayreduce_min_to_arraymin +tier: 1A +category: function_rewrite +status: proposed +statement: "arrayReduce('min', arr) эквивалентен arrayMin(arr). arrayMin — специализированная + нативная функция ClickHouse." +preconditions: + syntactic: + - "Запрос содержит arrayReduce('min', arr)" + semantic: [] + data: [] + user: [] +proof: "arrayReduce('min', arr) = arrayMin(arr). Задокументированный псевдоним." +explain_template: "arrayReduce('min', {arr}) заменяется arrayMin({arr})." +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT arrayReduce('min', scores) AS min_score FROM results" +example_after: 'SELECT arrayMin(scores) AS min_score FROM results' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/array-functions diff --git a/docs/rules/cards/R-056-extract_year_to_toyear.yaml b/docs/rules/cards/R-056-extract_year_to_toyear.yaml new file mode 100644 index 00000000..2e915b1f --- /dev/null +++ b/docs/rules/cards/R-056-extract_year_to_toyear.yaml @@ -0,0 +1,29 @@ +id: R-056 +name: extract_year_to_toyear +tier: 1A +category: function_rewrite +status: proposed +statement: "EXTRACT(YEAR FROM ts) — SQL-стандартная форма — эквивалентна нативной + функции toYear(ts) в ClickHouse. Нативная функция более явна." +preconditions: + syntactic: + - Запрос содержит EXTRACT(YEAR FROM col) + semantic: [] + data: [] + user: [] +proof: "EXTRACT(YEAR FROM ts) = toYear(ts). Это документированная эквивалентность + в ClickHouse date-time functions." +explain_template: 'EXTRACT(YEAR FROM {col}) заменяется toYear({col}).' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT EXTRACT(YEAR FROM ts) AS yr FROM events' +example_after: 'SELECT toYear(ts) AS yr FROM events' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — CH-идиоматичность.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/date-time-functions diff --git a/docs/rules/cards/R-057-extract_month_to_tomonth.yaml b/docs/rules/cards/R-057-extract_month_to_tomonth.yaml new file mode 100644 index 00000000..2befaadd --- /dev/null +++ b/docs/rules/cards/R-057-extract_month_to_tomonth.yaml @@ -0,0 +1,27 @@ +id: R-057 +name: extract_month_to_tomonth +tier: 1A +category: function_rewrite +status: proposed +statement: "EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse." +preconditions: + syntactic: + - Запрос содержит EXTRACT(MONTH FROM col) + semantic: [] + data: [] + user: [] +proof: "EXTRACT(MONTH FROM ts) = toMonth(ts). Задокументированная эквивалентность." +explain_template: 'EXTRACT(MONTH FROM {col}) заменяется toMonth({col}).' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT EXTRACT(MONTH FROM ts) AS mon FROM events' +example_after: 'SELECT toMonth(ts) AS mon FROM events' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/date-time-functions diff --git a/docs/rules/cards/R-058-extract_day_to_todayofmonth.yaml b/docs/rules/cards/R-058-extract_day_to_todayofmonth.yaml new file mode 100644 index 00000000..8d3a4a64 --- /dev/null +++ b/docs/rules/cards/R-058-extract_day_to_todayofmonth.yaml @@ -0,0 +1,27 @@ +id: R-058 +name: extract_day_to_todayofmonth +tier: 1A +category: function_rewrite +status: proposed +statement: "EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse." +preconditions: + syntactic: + - Запрос содержит EXTRACT(DAY FROM col) + semantic: [] + data: [] + user: [] +proof: "EXTRACT(DAY FROM ts) = toDayOfMonth(ts). Задокументированная эквивалентность." +explain_template: 'EXTRACT(DAY FROM {col}) заменяется toDayOfMonth({col}).' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT EXTRACT(DAY FROM ts) AS day_num FROM events' +example_after: 'SELECT toDayOfMonth(ts) AS day_num FROM events' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/date-time-functions diff --git a/docs/rules/cards/R-059-extract_hour_to_tohour.yaml b/docs/rules/cards/R-059-extract_hour_to_tohour.yaml new file mode 100644 index 00000000..870da422 --- /dev/null +++ b/docs/rules/cards/R-059-extract_hour_to_tohour.yaml @@ -0,0 +1,27 @@ +id: R-059 +name: extract_hour_to_tohour +tier: 1A +category: function_rewrite +status: proposed +statement: "EXTRACT(HOUR FROM ts) эквивалентна toHour(ts) в ClickHouse." +preconditions: + syntactic: + - Запрос содержит EXTRACT(HOUR FROM col) + semantic: [] + data: [] + user: [] +proof: "EXTRACT(HOUR FROM ts) = toHour(ts). Задокументированная эквивалентность." +explain_template: 'EXTRACT(HOUR FROM {col}) заменяется toHour({col}).' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT EXTRACT(HOUR FROM ts) AS hr FROM events' +example_after: 'SELECT toHour(ts) AS hr FROM events' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/date-time-functions diff --git a/docs/rules/cards/R-060-has_and_has_to_hasall.yaml b/docs/rules/cards/R-060-has_and_has_to_hasall.yaml new file mode 100644 index 00000000..12dfa2b6 --- /dev/null +++ b/docs/rules/cards/R-060-has_and_has_to_hasall.yaml @@ -0,0 +1,30 @@ +id: R-060 +name: has_and_has_to_hasall +tier: 1A +category: function_rewrite +status: proposed +statement: 'has(arr, x) AND has(arr, y) — проверка содержания двух значений — + эквивалентна hasAll(arr, [x, y]). hasAll более читаем и допускает расширение + списка без добавления AND.' +preconditions: + syntactic: + - Запрос содержит has(arr, x) AND has(arr, y) для одного и того же arr + semantic: [] + data: [] + user: [] +proof: 'hasAll(arr, [x, y]) = arr содержит все элементы [x, y] = has(arr, x) AND + has(arr, y). Математически идентично по определению hasAll.' +explain_template: 'has({arr}, {x}) AND has({arr}, {y}) заменяется hasAll({arr}, [{x}, {y}]).' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: 'SELECT * FROM events WHERE has(tags, ''error'') AND has(tags, ''critical'')' +example_after: 'SELECT * FROM events WHERE hasAll(tags, [''error'', ''critical''])' +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/array-functions diff --git a/docs/rules/cards/R-061-has_or_has_to_hasany.yaml b/docs/rules/cards/R-061-has_or_has_to_hasany.yaml new file mode 100644 index 00000000..4476eeff --- /dev/null +++ b/docs/rules/cards/R-061-has_or_has_to_hasany.yaml @@ -0,0 +1,29 @@ +id: R-061 +name: has_or_has_to_hasany +tier: 1A +category: function_rewrite +status: proposed +statement: 'has(arr, x) OR has(arr, y) — проверка хотя бы одного из двух значений — + эквивалентна hasAny(arr, [x, y]). hasAny более читаем и расширяем.' +preconditions: + syntactic: + - Запрос содержит has(arr, x) OR has(arr, y) для одного и того же arr + semantic: [] + data: [] + user: [] +proof: 'hasAny(arr, [x, y]) = arr содержит хотя бы один из [x, y] = has(arr, x) + OR has(arr, y). Математически идентично по определению hasAny.' +explain_template: 'has({arr}, {x}) OR has({arr}, {y}) заменяется hasAny({arr}, [{x}, {y}]).' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM events WHERE has(tags, 'error') OR has(tags, 'warning')" +example_after: "SELECT * FROM events WHERE hasAny(tags, ['error', 'warning'])" +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; преимущество — читаемость.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/array-functions diff --git a/docs/rules/cards/R-062-arraycount_zero_to_not_has.yaml b/docs/rules/cards/R-062-arraycount_zero_to_not_has.yaml new file mode 100644 index 00000000..f7663c9d --- /dev/null +++ b/docs/rules/cards/R-062-arraycount_zero_to_not_has.yaml @@ -0,0 +1,30 @@ +id: R-062 +name: arraycount_zero_to_not_has +tier: 1A +category: function_rewrite +status: proposed +statement: 'arrayCount(x -> x = val, arr) = 0 семантически эквивалентно NOT has(arr, val). + has() — специализированная функция поиска элемента, более эффективная чем + полный перебор через arrayCount.' +preconditions: + syntactic: + - Запрос содержит arrayCount(x -> x = val, arr) = 0 + semantic: [] + data: [] + user: [] +proof: 'arrayCount(x -> x = val, arr) = 0 ≡ ни один элемент arr не равен val ≡ + NOT has(arr, val). has() использует более оптимальный поиск.' +explain_template: 'arrayCount(x -> x = {val}, {arr}) = 0 заменяется NOT has({arr}, {val}).' +ch_version: + introduced: '1.0' + deprecated: null + last_validated: '25.3' +example_before: "SELECT * FROM events WHERE arrayCount(x -> x = 'error', tags) = 0" +example_after: "SELECT * FROM events WHERE NOT has(tags, 'error')" +expected_speedup: + estimate: null + measurement_method: 'Минимальный эффект; has() может использовать оптимизации.' +risks: [] +opt_in: false +references: +- https://clickhouse.com/docs/sql-reference/functions/array-functions diff --git a/eval/results/classifier_ablation_current/metadata.json b/eval/results/classifier_ablation_current/metadata.json new file mode 100644 index 00000000..ce3e1e72 --- /dev/null +++ b/eval/results/classifier_ablation_current/metadata.json @@ -0,0 +1,7 @@ +{ + "cases_dir": "benchmark/cases/synthetic_expanded", + "created_at": "2026-06-30T13:09:41.929111+00:00", + "experiment": "classifier_ablation", + "random_state": 42, + "split_path": "benchmark/splits/synthetic_expanded_v1.yaml" +} \ No newline at end of file diff --git a/eval/results/classifier_ablation_current/metrics.csv b/eval/results/classifier_ablation_current/metrics.csv new file mode 100644 index 00000000..613dd971 --- /dev/null +++ b/eval/results/classifier_ablation_current/metrics.csv @@ -0,0 +1,4 @@ +model,train_f1_macro,train_f1_micro,test_f1_macro,test_f1_micro,test_precision_macro,test_recall_macro,test_precision_micro,test_recall_micro,train_cases,test_cases,labels +logistic_regression,0.9079413358157319,0.8677248677248677,0.6777777777777778,0.8695652173913043,0.6666666666666665,0.7037037037037037,0.7692307692307693,1.0,144,36,27 +random_forest,0.9705387205387205,0.9753086419753086,0.6666666666666666,0.9512195121951219,0.6666666666666666,0.6666666666666666,0.9285714285714286,0.975,144,36,27 +catboost,0.9732323232323232,0.975609756097561,0.6913580246913581,0.9876543209876543,0.6851851851851852,0.7037037037037037,0.975609756097561,1.0,144,36,27 diff --git a/eval/results/classifier_ablation_current/metrics.json b/eval/results/classifier_ablation_current/metrics.json new file mode 100644 index 00000000..8e681095 --- /dev/null +++ b/eval/results/classifier_ablation_current/metrics.json @@ -0,0 +1,44 @@ +[ + { + "labels": 27, + "model": "logistic_regression", + "test_cases": 36, + "test_f1_macro": 0.6777777777777778, + "test_f1_micro": 0.8695652173913043, + "test_precision_macro": 0.6666666666666665, + "test_precision_micro": 0.7692307692307693, + "test_recall_macro": 0.7037037037037037, + "test_recall_micro": 1.0, + "train_cases": 144, + "train_f1_macro": 0.9079413358157319, + "train_f1_micro": 0.8677248677248677 + }, + { + "labels": 27, + "model": "random_forest", + "test_cases": 36, + "test_f1_macro": 0.6666666666666666, + "test_f1_micro": 0.9512195121951219, + "test_precision_macro": 0.6666666666666666, + "test_precision_micro": 0.9285714285714286, + "test_recall_macro": 0.6666666666666666, + "test_recall_micro": 0.975, + "train_cases": 144, + "train_f1_macro": 0.9705387205387205, + "train_f1_micro": 0.9753086419753086 + }, + { + "labels": 27, + "model": "catboost", + "test_cases": 36, + "test_f1_macro": 0.6913580246913581, + "test_f1_micro": 0.9876543209876543, + "test_precision_macro": 0.6851851851851852, + "test_precision_micro": 0.975609756097561, + "test_recall_macro": 0.7037037037037037, + "test_recall_micro": 1.0, + "train_cases": 144, + "train_f1_macro": 0.9732323232323232, + "train_f1_micro": 0.975609756097561 + } +] \ No newline at end of file diff --git a/eval/results/classifier_ablation_current/summary.md b/eval/results/classifier_ablation_current/summary.md new file mode 100644 index 00000000..efafcdf2 --- /dev/null +++ b/eval/results/classifier_ablation_current/summary.md @@ -0,0 +1,7 @@ +# Classifier Ablation + +| Model | Train F1 macro | Train F1 micro | Test F1 macro | Test F1 micro | Precision | Recall | +|---|---:|---:|---:|---:|---:|---:| +| logistic_regression | 0.908 | 0.868 | 0.678 | 0.870 | 0.667 | 0.704 | +| random_forest | 0.971 | 0.975 | 0.667 | 0.951 | 0.667 | 0.667 | +| catboost | 0.973 | 0.976 | 0.691 | 0.988 | 0.685 | 0.704 | diff --git a/eval/results/retrieval_ablation_20260630T124602Z/metadata.json b/eval/results/retrieval_ablation_20260630T124602Z/metadata.json new file mode 100644 index 00000000..4d1f8de7 --- /dev/null +++ b/eval/results/retrieval_ablation_20260630T124602Z/metadata.json @@ -0,0 +1,8 @@ +{ + "cases_dir": "benchmark/cases/synthetic", + "created_at": "2026-06-30T12:46:02.147707+00:00", + "experiment": "retrieval_ablation", + "kb_chunks_dir": "data/kb/chunks", + "scoring": "explicit rule gold URL fragments or keywords", + "top_k": 3 +} \ No newline at end of file diff --git a/eval/results/retrieval_ablation_20260630T124602Z/metrics.json b/eval/results/retrieval_ablation_20260630T124602Z/metrics.json new file mode 100644 index 00000000..f1c82202 --- /dev/null +++ b/eval/results/retrieval_ablation_20260630T124602Z/metrics.json @@ -0,0 +1,32 @@ +[ + { + "elapsed_seconds": 17.74912425002549, + "max_chunks": 2000, + "model": "multilingual-e5-small (current)", + "model_name": "intfloat/multilingual-e5-small", + "mrr_at_3": 0.4583333333333333, + "query_count": 20, + "size": "117 MB", + "total_kb_chunks": 8804 + }, + { + "elapsed_seconds": 10.358752916974481, + "max_chunks": 2000, + "model": "all-MiniLM-L6-v2", + "model_name": "sentence-transformers/all-MiniLM-L6-v2", + "mrr_at_3": 0.5166666666666667, + "query_count": 20, + "size": "80 MB", + "total_kb_chunks": 8804 + }, + { + "elapsed_seconds": 12.36237687501125, + "max_chunks": 2000, + "model": "paraphrase-multilingual-MiniLM-L12-v2", + "model_name": "paraphrase-multilingual-MiniLM-L12-v2", + "mrr_at_3": 0.24166666666666664, + "query_count": 20, + "size": "420 MB", + "total_kb_chunks": 8804 + } +] \ No newline at end of file diff --git a/eval/results/retrieval_ablation_20260630T124602Z/summary.md b/eval/results/retrieval_ablation_20260630T124602Z/summary.md new file mode 100644 index 00000000..64adb4a5 --- /dev/null +++ b/eval/results/retrieval_ablation_20260630T124602Z/summary.md @@ -0,0 +1,7 @@ +# Retrieval Ablation + +| Model | Size | Queries | MRR@3 | Time (s) | +|---|---:|---:|---:|---:| +| multilingual-e5-small (current) | 117 MB | 20 | 0.458 | 17.7 | +| all-MiniLM-L6-v2 | 80 MB | 20 | 0.517 | 10.4 | +| paraphrase-multilingual-MiniLM-L12-v2 | 420 MB | 20 | 0.242 | 12.4 | diff --git a/pyproject.toml b/pyproject.toml index ef298905..766229b9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -58,6 +58,10 @@ strict = true module = ["yaml"] ignore_missing_imports = true +[[tool.mypy.overrides]] +module = ["catboost", "sklearn.*"] +ignore_missing_imports = true + [build-system] requires = ["poetry-core>=1.9.0"] build-backend = "poetry.core.masonry.api" diff --git a/scripts/benchmark/generate_synthetic_dataset.py b/scripts/benchmark/generate_synthetic_dataset.py index 8c1e6646..a5c21d8a 100644 --- a/scripts/benchmark/generate_synthetic_dataset.py +++ b/scripts/benchmark/generate_synthetic_dataset.py @@ -316,6 +316,51 @@ class RuleTemplate: "SELECT toUInt16(raw_code) FROM requests", ), ), + RuleTemplate( + label="r057", + rules=("R-057",), + severity="low", + issue_type="extract_month_to_tomonth", + description="EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse.", + sql_variants=( + "SELECT EXTRACT(MONTH FROM ts) AS mon FROM events LIMIT 100", + "SELECT EXTRACT(MONTH FROM created_at) AS mon FROM orders LIMIT 100", + "SELECT EXTRACT(MONTH FROM started_at) AS mon FROM sessions LIMIT 100", + "SELECT EXTRACT(MONTH FROM request_time) AS mon FROM requests LIMIT 100", + "SELECT EXTRACT(MONTH FROM processed_at) AS mon FROM payments LIMIT 100", + "SELECT EXTRACT(MONTH FROM event_time) AS mon FROM query_log LIMIT 100", + ), + ), + RuleTemplate( + label="r058", + rules=("R-058",), + severity="low", + issue_type="extract_day_to_todayofmonth", + description="EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse.", + sql_variants=( + "SELECT EXTRACT(DAY FROM ts) AS day_num FROM events LIMIT 100", + "SELECT EXTRACT(DAY FROM created_at) AS day_num FROM orders LIMIT 100", + "SELECT EXTRACT(DAY FROM started_at) AS day_num FROM sessions LIMIT 100", + "SELECT EXTRACT(DAY FROM request_time) AS day_num FROM requests LIMIT 100", + "SELECT EXTRACT(DAY FROM processed_at) AS day_num FROM payments LIMIT 100", + "SELECT EXTRACT(DAY FROM event_time) AS day_num FROM query_log LIMIT 100", + ), + ), + RuleTemplate( + label="r059", + rules=("R-059",), + severity="low", + issue_type="extract_hour_to_tohour", + description="EXTRACT(HOUR FROM ts) эквивалентна toHour(ts) в ClickHouse.", + sql_variants=( + "SELECT EXTRACT(HOUR FROM ts) AS hr FROM events LIMIT 100", + "SELECT EXTRACT(HOUR FROM created_at) AS hr FROM orders LIMIT 100", + "SELECT EXTRACT(HOUR FROM started_at) AS hr FROM sessions LIMIT 100", + "SELECT EXTRACT(HOUR FROM request_time) AS hr FROM requests LIMIT 100", + "SELECT EXTRACT(HOUR FROM processed_at) AS hr FROM payments LIMIT 100", + "SELECT EXTRACT(HOUR FROM event_time) AS hr FROM query_log LIMIT 100", + ), + ), RuleTemplate( label="d003", rules=("D-003",), @@ -399,6 +444,9 @@ class RuleTemplate: "R-018": ("union_without_all", "medium", "UNION выполняет дедупликацию; UNION ALL дешевле при непересекающихся источниках."), "R-019": ("oversized_uint_type_narrowing", "low", "Широкий Int64/UInt64 для статусных или типовых колонок может быть избыточен."), "R-020": ("unsafe_cast_without_default", "medium", "CAST/toType над сырой колонкой может падать на некорректных значениях."), + "R-057": ("extract_month_to_tomonth", "low", "EXTRACT(MONTH FROM ts) эквивалентна toMonth(ts) в ClickHouse."), + "R-058": ("extract_day_to_todayofmonth", "low", "EXTRACT(DAY FROM ts) эквивалентна toDayOfMonth(ts) в ClickHouse."), + "R-059": ("extract_hour_to_tohour", "low", "EXTRACT(HOUR FROM ts) эквивалентна toHour(ts) в ClickHouse."), "D-003": ("select_star_on_wide_table", "medium", "SELECT * читает все колонки; для column-store лучше перечислить нужные поля."), "D-004": ("missing_limit_on_unbounded_result", "medium", "SELECT без LIMIT и без top-level агрегата может вернуть неограниченное число строк."), "D-007": ("final_modifier_usage", "medium", "FINAL мержит parts на чтении и может быть дорогой операцией."), diff --git a/scripts/eval/ablation_classifiers.py b/scripts/eval/ablation_classifiers.py new file mode 100644 index 00000000..114865e7 --- /dev/null +++ b/scripts/eval/ablation_classifiers.py @@ -0,0 +1,150 @@ +from __future__ import annotations + +import argparse +import csv +import json +from datetime import UTC, datetime +from pathlib import Path + +from rich.console import Console +from rich.table import Table + +from clickadvisor.ml.classifier import ClassifierMetrics, evaluate_classifiers +from clickadvisor.ml.dataset import build_examples, load_benchmark_cases, load_split, split_examples + +DEFAULT_CASES_DIR = Path("benchmark/cases/synthetic_expanded") +DEFAULT_SPLIT_PATH = Path("benchmark/splits/synthetic_expanded_v1.yaml") +DEFAULT_RESULTS_DIR = Path("eval/results") + +console = Console() + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Run classifier ablation on synthetic benchmark features.") + parser.add_argument("--cases-dir", type=Path, default=DEFAULT_CASES_DIR) + parser.add_argument("--split-path", type=Path, default=DEFAULT_SPLIT_PATH) + parser.add_argument("--results-dir", type=Path, default=DEFAULT_RESULTS_DIR) + parser.add_argument("--run-id", default=None, help="Stable output directory name. Defaults to timestamp.") + parser.add_argument("--random-state", type=int, default=42) + parser.add_argument( + "--models", + nargs="*", + default=None, + help="Subset of models: logistic_regression random_forest catboost.", + ) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + cases = load_benchmark_cases(args.cases_dir) + examples = build_examples(cases) + train_examples, test_examples = split_examples(examples, load_split(args.split_path)) + metrics = evaluate_classifiers( + train_examples, + test_examples, + random_state=args.random_state, + model_names=args.models, + ) + + print_results(metrics) + output_dir = write_results( + metrics, + args.results_dir, + args.run_id, + cases_dir=args.cases_dir, + split_path=args.split_path, + random_state=args.random_state, + ) + console.print(f"Saved classifier ablation results to {output_dir}") + + +def print_results(metrics: list[ClassifierMetrics]) -> None: + table = Table(title="Classifier ablation") + table.add_column("Model") + table.add_column("Train F1 macro", justify="right") + table.add_column("Train F1 micro", justify="right") + table.add_column("Test F1 macro", justify="right") + table.add_column("Test F1 micro", justify="right") + table.add_column("Precision", justify="right") + table.add_column("Recall", justify="right") + + for item in metrics: + table.add_row( + item.model, + f"{item.train_f1_macro:.3f}", + f"{item.train_f1_micro:.3f}", + f"{item.test_f1_macro:.3f}", + f"{item.test_f1_micro:.3f}", + f"{item.test_precision_macro:.3f}", + f"{item.test_recall_macro:.3f}", + ) + console.print(table) + + +def write_results( + metrics: list[ClassifierMetrics], + results_dir: Path, + run_id: str | None, + *, + cases_dir: Path, + split_path: Path, + random_state: int, +) -> Path: + run_name = run_id or f"classifier_ablation_{datetime.now(UTC).strftime('%Y%m%dT%H%M%SZ')}" + output_dir = results_dir / run_name + output_dir.mkdir(parents=True, exist_ok=True) + + rows = [metric.as_row() for metric in metrics] + (output_dir / "metrics.json").write_text( + json.dumps(rows, indent=2, sort_keys=True), + encoding="utf-8", + ) + (output_dir / "metadata.json").write_text( + json.dumps( + { + "experiment": "classifier_ablation", + "cases_dir": str(cases_dir), + "split_path": str(split_path), + "random_state": random_state, + "created_at": datetime.now(UTC).isoformat(), + }, + indent=2, + sort_keys=True, + ), + encoding="utf-8", + ) + + with (output_dir / "metrics.csv").open("w", encoding="utf-8", newline="") as handle: + writer = csv.DictWriter(handle, fieldnames=list(rows[0]) if rows else ["model"]) + writer.writeheader() + writer.writerows(rows) + + (output_dir / "summary.md").write_text(markdown_summary(metrics), encoding="utf-8") + return output_dir + + +def markdown_summary(metrics: list[ClassifierMetrics]) -> str: + lines = [ + "# Classifier Ablation", + "", + "| Model | Train F1 macro | Train F1 micro | Test F1 macro | Test F1 micro | Precision | Recall |", + "|---|---:|---:|---:|---:|---:|---:|", + ] + for item in metrics: + lines.append( + "| " + f"{item.model} | " + f"{item.train_f1_macro:.3f} | " + f"{item.train_f1_micro:.3f} | " + f"{item.test_f1_macro:.3f} | " + f"{item.test_f1_micro:.3f} | " + f"{item.test_precision_macro:.3f} | " + f"{item.test_recall_macro:.3f} |" + ) + lines.append("") + return "\n".join(lines) + + +if __name__ == "__main__": + main() diff --git a/scripts/eval/ablation_embeddings.py b/scripts/eval/ablation_embeddings.py index 8a6f7a4a..c81b0bf9 100644 --- a/scripts/eval/ablation_embeddings.py +++ b/scripts/eval/ablation_embeddings.py @@ -1,9 +1,11 @@ from __future__ import annotations +import json import re import shutil import time from dataclasses import dataclass +from datetime import UTC, datetime from pathlib import Path from typing import Any @@ -22,6 +24,7 @@ MAX_CHUNKS = 2000 TOTAL_KB_CHUNKS = 8804 TOP_K = 3 +RESULTS_DIR = Path("eval/results") console = Console() @@ -45,9 +48,17 @@ class Chunk: class EvalResult: model: ModelSpec mrr_at_3: float + query_count: int elapsed_seconds: float +@dataclass(frozen=True) +class RuleGoldReference: + rule_id: str + url_fragments: tuple[str, ...] + keywords: tuple[str, ...] + + MODELS = [ ModelSpec( model_name="intfloat/multilingual-e5-small", @@ -68,27 +79,79 @@ class EvalResult: SQL_FUNCTION_PATTERN = re.compile(r"\b([A-Za-z_][A-Za-z0-9_]*)\s*\(") SQL_KEYWORD_FUNCTION_EXCLUSIONS = {"SELECT", "FROM", "WHERE", "GROUP", "ORDER", "LIMIT"} -RULE_KEYWORDS = { - "R-001": ["uniqExact", "COUNT(DISTINCT", "distinct", "count distinct"], - "R-002": ["uniq", "HyperLogLog", "approximate", "approx"], - "R-003": ["quantile", "quantileTDigest", "quantileExact"], - "R-004": ["uniqExact", "subquery", "DISTINCT"], - "R-005": ["toDate", "DateTime", "sargable", "range", "date range"], - "R-006": ["toYYYYMM", "toStartOfMonth", "date", "partition"], - "R-007": ["toStartOfHour", "toStartOfDay", "interval", "range"], - "R-008": ["CAST", "type conversion", "primary key", "index"], - "R-009": ["IN", "equality", "singleton"], - "R-010": ["OR", "IN", "disjunction", "chain"], - "R-011": ["HAVING", "WHERE", "aggregate", "GROUP BY", "pushdown"], - "R-012": ["WHERE TRUE", "constant", "predicate"], - "R-013": ["length", "empty", "notEmpty", "string"], - "R-014": ["GROUP BY", "cityHash64", "hash", "string"], - "R-015": ["DISTINCT", "GROUP BY", "redundant"], - "R-016": ["ORDER BY", "LIMIT", "subquery", "sort"], - "R-017": ["subquery", "filter", "pushdown", "WHERE"], - "R-018": ["UNION", "UNION ALL", "distinct"], - "D-003": ["SELECT *", "columnar", "columns", "star"], - "D-004": ["LIMIT", "unbounded", "result set"], +RULE_GOLD_REFERENCES = { + "R-001": RuleGoldReference( + "R-001", + ("uniqexact", "count-distinct", "aggregate-functions/reference/uniq"), + ("uniqExact", "COUNT(DISTINCT"), + ), + "R-002": RuleGoldReference( + "R-002", + ("uniq", "aggregate-functions/reference/uniq"), + ("uniq", "approximate"), + ), + "R-003": RuleGoldReference( + "R-003", + ("quantile", "quantiletdigest"), + ("quantileExact", "quantileTDigest"), + ), + "R-004": RuleGoldReference( + "R-004", + ("distinct", "uniqexact"), + ("SELECT DISTINCT", "uniqExact"), + ), + "R-005": RuleGoldReference( + "R-005", + ("todate", "datetime", "primary-key"), + ("toDate", "range"), + ), + "R-006": RuleGoldReference( + "R-006", + ("toyyyymm", "partition", "date-time-functions"), + ("toYYYYMM", "partition"), + ), + "R-007": RuleGoldReference( + "R-007", + ("tostartof", "date-time-functions"), + ("toStartOf", "range"), + ), + "R-008": RuleGoldReference( + "R-008", + ("cast", "type-conversion-functions"), + ("CAST", "type conversion"), + ), + "R-009": RuleGoldReference("R-009", ("in", "operators"), ("IN", "single value")), + "R-010": RuleGoldReference("R-010", ("in", "operators"), ("OR", "IN")), + "R-011": RuleGoldReference("R-011", ("having", "where"), ("HAVING", "WHERE")), + "R-012": RuleGoldReference("R-012", ("where", "operators"), ("constant", "predicate")), + "R-013": RuleGoldReference( + "R-013", + ("empty", "notempty", "string-functions"), + ("empty", "notEmpty"), + ), + "R-014": RuleGoldReference( + "R-014", + ("group-by", "cityhash64", "hash-functions"), + ("GROUP BY", "hash"), + ), + "R-015": RuleGoldReference("R-015", ("distinct", "group-by"), ("DISTINCT", "GROUP BY")), + "R-016": RuleGoldReference("R-016", ("order-by", "limit"), ("ORDER BY", "LIMIT")), + "R-017": RuleGoldReference("R-017", ("where", "subquery"), ("subquery", "filter")), + "R-018": RuleGoldReference("R-018", ("union", "union-all"), ("UNION ALL", "UNION")), + "R-019": RuleGoldReference("R-019", ("uint", "data-types", "integer"), ("UInt64", "Int64")), + "R-020": RuleGoldReference( + "R-020", + ("cast", "ordefault", "type-conversion-functions"), + ("OrDefault", "CAST"), + ), + "D-003": RuleGoldReference("D-003", ("select", "columnar"), ("SELECT *", "columns")), + "D-004": RuleGoldReference("D-004", ("limit", "result"), ("LIMIT", "unbounded")), + "D-007": RuleGoldReference("D-007", ("final", "replacingmergetree"), ("FINAL", "MergeTree")), + "D-014": RuleGoldReference( + "D-014", + ("async-insert", "wait_for_async_insert"), + ("async_insert", "wait_for_async_insert"), + ), } @@ -109,6 +172,8 @@ def main() -> None: results.append(evaluate_model(model, chunks, cases, db_path)) print_results(results) + output_dir = write_results(results) + console.print(f"Saved retrieval ablation results to {output_dir}") finally: cleanup(db_paths) @@ -165,12 +230,17 @@ def evaluate_model( vectors_config=VectorParams(size=vector_size, distance=Distance.COSINE), ) index_chunks(client, model, model_spec.model_name, chunks) - mrr_at_3 = compute_mrr_at_3(client, model, model_spec.model_name, cases) + mrr_at_3, query_count = compute_mrr_at_3(client, model, model_spec.model_name, cases) finally: client.close() elapsed_seconds = time.perf_counter() - start - return EvalResult(model=model_spec, mrr_at_3=mrr_at_3, elapsed_seconds=elapsed_seconds) + return EvalResult( + model=model_spec, + mrr_at_3=mrr_at_3, + query_count=query_count, + elapsed_seconds=elapsed_seconds, + ) def index_chunks( @@ -214,18 +284,22 @@ def compute_mrr_at_3( model: SentenceTransformer, model_name: str, cases: list[dict[str, Any]], -) -> float: +) -> tuple[float, int]: reciprocal_ranks = [] for case in cases: + gold_refs = gold_references_for_case(case) + if not gold_refs: + continue sql = str(case["sql"]) query = f"ClickHouse optimization: {sql[:200]}" query_vector = embed_query(model, model_name, query).tolist() results = search_top_k(client, query_vector, TOP_K) - rank = first_relevant_rank(results, case.get("expected_rules_to_fire", []), sql) + rank = first_relevant_rank(results, gold_refs) reciprocal_ranks.append(1 / rank if rank else 0.0) - return sum(reciprocal_ranks) / len(reciprocal_ranks) if reciprocal_ranks else 0.0 + mrr = sum(reciprocal_ranks) / len(reciprocal_ranks) if reciprocal_ranks else 0.0 + return mrr, len(reciprocal_ranks) def search_top_k(client: QdrantClient, query_vector: list[float], top_k: int) -> list[Any]: @@ -244,23 +318,37 @@ def search_top_k(client: QdrantClient, query_vector: list[float], top_k: int) -> return list(response.points) -def first_relevant_rank(results: list[Any], expected_rules: object, sql: str) -> int | None: - rule_ids = expected_rules if isinstance(expected_rules, list) else [] +def first_relevant_rank(results: list[Any], gold_refs: list[RuleGoldReference]) -> int | None: for rank, result in enumerate(results[:TOP_K], start=1): - if any(is_relevant(result, str(rule_id), sql) for rule_id in rule_ids): + if any(is_relevant(result, gold_ref) for gold_ref in gold_refs): return rank return None -def is_relevant(chunk: Any, rule_id: str, sql: str) -> bool: +def gold_references_for_case(case: dict[str, Any]) -> list[RuleGoldReference]: + expected_rules = case.get("expected_rules_to_fire", []) + if not isinstance(expected_rules, list): + return [] + return [ + RULE_GOLD_REFERENCES[rule_id] + for rule_id in expected_rules + if isinstance(rule_id, str) and rule_id in RULE_GOLD_REFERENCES + ] + + +def is_relevant(chunk: Any, gold_ref: RuleGoldReference) -> bool: payload = chunk.payload or {} - source = str(payload.get("source", "")).lower() - score = float(getattr(chunk, "score", 0.0)) - if ("clickhouse" in source or "altinity" in source) and score >= 0.75: + url = str(payload.get("url", "")).lower() + path = str(payload.get("file_path", "")).lower() + text = str(payload.get("text", "")).lower() + + if any( + fragment.lower() in url or fragment.lower() in path + for fragment in gold_ref.url_fragments + ): return True - text = str(payload.get("text", "")).lower() - return any(keyword.lower() in text for keyword in RULE_KEYWORDS.get(rule_id, [])) + return any(keyword.lower() in text for keyword in gold_ref.keywords) def extract_sql_functions(sql: str) -> list[str]: @@ -305,6 +393,7 @@ def print_results(results: list[EvalResult]) -> None: table = Table() table.add_column("Model") table.add_column("Size") + table.add_column("Queries", justify="right") table.add_column("MRR@3", justify="right") table.add_column("Time (s)", justify="right") @@ -312,6 +401,7 @@ def print_results(results: list[EvalResult]) -> None: table.add_row( result.model.label, result.model.size, + str(result.query_count), f"{result.mrr_at_3:.2f}", f"{result.elapsed_seconds:.1f}", ) @@ -319,7 +409,7 @@ def print_results(results: list[EvalResult]) -> None: console.print(table) console.print( f"* Evaluated on {MAX_CHUNKS} KB chunks (of {TOTAL_KB_CHUNKS} total). " - "Full index expected to improve MRR@3." + "MRR@3 uses explicit rule-to-doc gold URL/keyword references only." ) best = max(results, key=lambda result: result.mrr_at_3) console.print( @@ -328,6 +418,65 @@ def print_results(results: list[EvalResult]) -> None: ) +def write_results(results: list[EvalResult]) -> Path: + output_dir = RESULTS_DIR / f"retrieval_ablation_{datetime.now(UTC).strftime('%Y%m%dT%H%M%SZ')}" + output_dir.mkdir(parents=True, exist_ok=True) + rows = [ + { + "model": result.model.label, + "model_name": result.model.model_name, + "size": result.model.size, + "query_count": result.query_count, + "mrr_at_3": result.mrr_at_3, + "elapsed_seconds": result.elapsed_seconds, + "max_chunks": MAX_CHUNKS, + "total_kb_chunks": TOTAL_KB_CHUNKS, + } + for result in results + ] + (output_dir / "metrics.json").write_text( + json.dumps(rows, indent=2, sort_keys=True), + encoding="utf-8", + ) + (output_dir / "metadata.json").write_text( + json.dumps( + { + "experiment": "retrieval_ablation", + "cases_dir": str(SYNTHETIC_CASES_DIR), + "kb_chunks_dir": str(KB_CHUNKS_DIR), + "top_k": TOP_K, + "scoring": "explicit rule gold URL fragments or keywords", + "created_at": datetime.now(UTC).isoformat(), + }, + indent=2, + sort_keys=True, + ), + encoding="utf-8", + ) + (output_dir / "summary.md").write_text(markdown_summary(results), encoding="utf-8") + return output_dir + + +def markdown_summary(results: list[EvalResult]) -> str: + lines = [ + "# Retrieval Ablation", + "", + "| Model | Size | Queries | MRR@3 | Time (s) |", + "|---|---:|---:|---:|---:|", + ] + for result in results: + lines.append( + "| " + f"{result.model.label} | " + f"{result.model.size} | " + f"{result.query_count} | " + f"{result.mrr_at_3:.3f} | " + f"{result.elapsed_seconds:.1f} |" + ) + lines.append("") + return "\n".join(lines) + + def cleanup(db_paths: list[Path]) -> None: for db_path in db_paths: shutil.rmtree(db_path, ignore_errors=True) diff --git a/scripts/ml/__init__.py b/scripts/ml/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/scripts/ml/extract_dataset_features.py b/scripts/ml/extract_dataset_features.py new file mode 100644 index 00000000..339dbe4b --- /dev/null +++ b/scripts/ml/extract_dataset_features.py @@ -0,0 +1,107 @@ +"""Extract QueryFeatures from all synthetic_expanded benchmark cases and save to JSONL. + +Output format (one JSON object per line): +{ + "case_id": "synthetic_expanded_r001_001", + "features": {"has_count_distinct": 1, ...}, + "labels": ["exact_count_distinct_specialization", "approx_count_distinct_advisory"], + "split": "train" +} + +Labels come from known_issues[].type (semantic problem-type labels, not rule IDs). +Split assignment comes from benchmark/splits/synthetic_expanded_v1.yaml. +Negative cases have labels=[]. +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO_ROOT)) + +from clickadvisor.ml.features import FeatureExtractor # noqa: E402 + +CASES_DIR = REPO_ROOT / "benchmark" / "cases" / "synthetic_expanded" +SPLIT_PATH = REPO_ROOT / "benchmark" / "splits" / "synthetic_expanded_v1.yaml" +OUTPUT_PATH = REPO_ROOT / "data" / "ml" / "features_dataset.jsonl" + + +def load_split(split_path: Path) -> dict[str, str]: + """Return {case_id: "train"|"test"} mapping.""" + payload = yaml.safe_load(split_path.read_text(encoding="utf-8")) + mapping: dict[str, str] = {} + for case_id in payload.get("train_case_ids", []): + mapping[case_id] = "train" + for case_id in payload.get("test_case_ids", []): + mapping[case_id] = "test" + return mapping + + +def main() -> None: + extractor = FeatureExtractor() + split_map = load_split(SPLIT_PATH) + + case_files = sorted(CASES_DIR.glob("*.yaml")) + if not case_files: + print(f"No YAML files found in {CASES_DIR}", file=sys.stderr) + sys.exit(1) + + OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True) + + label_set: set[str] = set() + train_count = 0 + test_count = 0 + skipped = 0 + + with OUTPUT_PATH.open("w", encoding="utf-8") as out: + for path in case_files: + payload = yaml.safe_load(path.read_text(encoding="utf-8")) + if not isinstance(payload, dict): + skipped += 1 + continue + + case_id: str = payload.get("case_id", path.stem) + sql: str = payload.get("sql", "") + if not sql: + skipped += 1 + continue + + known_issues = payload.get("known_issues") or [] + labels: list[str] = [ + issue["type"] + for issue in known_issues + if isinstance(issue, dict) and "type" in issue + ] + label_set.update(labels) + + split = split_map.get(case_id, "unknown") + + features = extractor.extract(sql).to_vector() + + record = { + "case_id": case_id, + "features": features, + "labels": labels, + "split": split, + } + out.write(json.dumps(record, ensure_ascii=False) + "\n") + + if split == "train": + train_count += 1 + elif split == "test": + test_count += 1 + + total = train_count + test_count + skipped + print(f"Written {total} records to {OUTPUT_PATH}") + print(f" train: {train_count}, test: {test_count}, skipped/unknown: {skipped}") + print(f" unique labels ({len(label_set)}):") + for label in sorted(label_set): + print(f" {label}") + + +if __name__ == "__main__": + main() diff --git a/tests/ml/__init__.py b/tests/ml/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/ml/test_classifier.py b/tests/ml/test_classifier.py new file mode 100644 index 00000000..e08100d0 --- /dev/null +++ b/tests/ml/test_classifier.py @@ -0,0 +1,25 @@ +from __future__ import annotations + +from pathlib import Path + +from clickadvisor.ml.classifier import evaluate_classifiers +from clickadvisor.ml.dataset import build_examples, load_benchmark_cases, load_split, split_examples + + +def test_logistic_regression_classifier_ablation_smoke() -> None: + cases = load_benchmark_cases(Path("benchmark/cases/synthetic_expanded")) + examples = build_examples(cases) + train, test = split_examples( + examples, + load_split(Path("benchmark/splits/synthetic_expanded_v1.yaml")), + ) + + metrics = evaluate_classifiers(train, test, model_names=["logistic_regression"]) + + assert len(metrics) == 1 + result = metrics[0] + assert result.model == "logistic_regression" + assert result.train_cases == 144 + assert result.test_cases == 36 + assert result.labels >= 20 + assert 0.0 <= result.test_f1_macro <= 1.0 diff --git a/tests/ml/test_features.py b/tests/ml/test_features.py new file mode 100644 index 00000000..b2667d1b --- /dev/null +++ b/tests/ml/test_features.py @@ -0,0 +1,337 @@ +"""Tests for FeatureExtractor using real SQL from synthetic_expanded benchmark. + +Coverage strategy: +- 1 positive test per template (23 templates = 23 tests) +- 3 negative-case tests (no features should fire for clean SQL) +- 2 parse-error resilience tests +""" +from __future__ import annotations + +import logging + +from clickadvisor.ml.features import FeatureExtractor, QueryFeatures + +_fx = FeatureExtractor() + + +def extract(sql: str) -> QueryFeatures: + return _fx.extract(sql) + + +# --------------------------------------------------------------------------- +# R-001 / R-002: exact_count_distinct_specialization + approx_count_distinct +# --------------------------------------------------------------------------- +def test_r001_count_distinct(): + f = extract("SELECT COUNT(DISTINCT user_id) FROM events") + assert f.has_count_distinct is True + + +def test_r001_to_vector_bool_is_int(): + f = extract("SELECT COUNT(DISTINCT user_id) FROM events") + v = f.to_vector() + assert v["has_count_distinct"] == 1 + assert isinstance(v["has_count_distinct"], int) + + +# --------------------------------------------------------------------------- +# R-003: exact_quantile_candidate +# --------------------------------------------------------------------------- +def test_r003_quantile_exact(): + f = extract("SELECT quantileExact(0.95)(response_time) FROM requests") + assert f.has_quantile_exact is True + + +# --------------------------------------------------------------------------- +# R-004: count_star_distinct_subquery +# --------------------------------------------------------------------------- +def test_r004_count_star_distinct_subquery(): + f = extract("SELECT COUNT(*) FROM (SELECT DISTINCT user_id FROM events)") + assert f.has_subquery is True + # COUNT(*) over subquery – no count_distinct at the outer level + assert f.has_count_distinct is False + + +# --------------------------------------------------------------------------- +# R-005: function_on_datetime_filter (toDate equality) +# --------------------------------------------------------------------------- +def test_r005_todate_equality(): + f = extract("SELECT count() FROM events WHERE toDate(created_at) = '2024-01-15'") + assert f.has_function_on_filter_column is True + + +# --------------------------------------------------------------------------- +# R-006: date_part_filter_to_range (toYYYYMM equality) +# --------------------------------------------------------------------------- +def test_r006_toyyyymm_equality(): + f = extract("SELECT count() FROM events WHERE toYYYYMM(event_date) = 202401") + assert f.has_function_on_filter_column is True + + +# --------------------------------------------------------------------------- +# R-007: interval_start_filter_to_range (toStartOfHour equality) +# --------------------------------------------------------------------------- +def test_r007_tostartofhour_equality(): + f = extract("SELECT count() FROM logs WHERE toStartOfHour(ts) = '2024-01-15 14:00:00'") + assert f.has_function_on_filter_column is True + + +# --------------------------------------------------------------------------- +# R-008: redundant_cast_on_filter +# --------------------------------------------------------------------------- +def test_r008_redundant_cast(): + f = extract("SELECT count() FROM users WHERE CAST(user_id AS UInt64) = 12345") + assert f.has_cast is True + assert f.has_function_on_filter_column is True + + +# --------------------------------------------------------------------------- +# R-009: singleton_in_predicate +# --------------------------------------------------------------------------- +def test_r009_singleton_in(): + f = extract("SELECT count() FROM events WHERE country IN ('RU')") + assert f.has_in_with_single_value is True + + +# --------------------------------------------------------------------------- +# R-010: disjunction_chain_to_in +# --------------------------------------------------------------------------- +def test_r010_or_chain(): + f = extract( + "SELECT count() FROM events WHERE country = 'RU' OR country = 'BY' OR country = 'KZ'" + ) + assert f.has_or_chain_same_column is True + + +# --------------------------------------------------------------------------- +# R-011: having_without_aggregate +# --------------------------------------------------------------------------- +def test_r011_having_without_aggregate(): + f = extract( + "SELECT user_id, COUNT(*) FROM events GROUP BY user_id " + "HAVING country = 'RU' AND COUNT(*) > 100" + ) + assert f.has_having is True + assert f.has_having_without_aggregate is True + assert f.has_group_by is True + + +# --------------------------------------------------------------------------- +# R-012: constant_predicate (WHERE TRUE) +# --------------------------------------------------------------------------- +def test_r012_constant_predicate(): + f = extract("SELECT count() FROM events WHERE TRUE AND user_id = 5") + assert f.has_constant_predicate is True + + +# --------------------------------------------------------------------------- +# R-013: length_empty_predicate +# --------------------------------------------------------------------------- +def test_r013_length_zero_check(): + f = extract("SELECT count() FROM comments WHERE length(body) > 0") + assert f.has_length_zero_check is True + + +# --------------------------------------------------------------------------- +# R-014: groupby_string_hash_candidate +# --------------------------------------------------------------------------- +def test_r014_groupby_string_column(): + f = extract("SELECT url, COUNT(*) FROM logs GROUP BY url") + assert f.has_group_by is True + assert f.has_group_by_string_column is True + + +# --------------------------------------------------------------------------- +# R-015: distinct_after_groupby +# --------------------------------------------------------------------------- +def test_r015_distinct_after_groupby(): + f = extract( + "SELECT DISTINCT a, b FROM (SELECT a, b, COUNT(*) AS cnt FROM events GROUP BY a, b) LIMIT 100" + ) + assert f.has_subquery is True + assert f.has_group_by is True + assert f.has_limit is True + + +# --------------------------------------------------------------------------- +# R-016: orderby_without_limit_in_subquery +# --------------------------------------------------------------------------- +def test_r016_orderby_no_limit_subquery(): + f = extract("SELECT COUNT(*) FROM (SELECT * FROM events ORDER BY created_at)") + assert f.has_subquery_with_orderby_no_limit is True + assert f.has_subquery is True + + +# --------------------------------------------------------------------------- +# R-017: subquery_filter_pushdown +# --------------------------------------------------------------------------- +def test_r017_subquery_filter_pushdown(): + f = extract( + "SELECT * FROM (SELECT * FROM events WHERE status = 'active') WHERE user_id = 42" + ) + assert f.has_nested_subquery_filter is True + assert f.has_subquery is True + + +# --------------------------------------------------------------------------- +# R-018: union_without_all +# --------------------------------------------------------------------------- +def test_r018_union_not_all(): + f = extract("SELECT user_id FROM events_2023 UNION SELECT user_id FROM events_2024") + assert f.has_union is True + assert f.has_union_all is False + + +def test_r018_union_all_no_flag(): + f = extract("SELECT user_id FROM events_2023 UNION ALL SELECT user_id FROM events_2024") + assert f.has_union is True + assert f.has_union_all is True + + +# --------------------------------------------------------------------------- +# R-019: oversized_uint_type_narrowing (CREATE TABLE context) +# --------------------------------------------------------------------------- +def test_r019_create_table(): + f = extract( + "CREATE TABLE events (event_type UInt64, user_id UInt64) ENGINE = MergeTree() ORDER BY event_type" + ) + # No SELECT-level features should fire + assert f.has_count_distinct is False + assert f.has_select_star is False + assert f.has_no_limit is False + assert f.query_length_chars > 0 + + +# --------------------------------------------------------------------------- +# R-020: unsafe_cast_without_default +# --------------------------------------------------------------------------- +def test_r020_throwing_cast(): + f = extract("SELECT CAST(raw_value AS UInt32) FROM events") + assert f.has_cast is True + assert f.has_cast_without_default is True + + +def test_r020_safe_cast_ordefault(): + f = extract("SELECT toUInt32OrDefault(raw_value) FROM events") + assert f.has_cast is True + assert f.has_cast_without_default is False + + +# --------------------------------------------------------------------------- +# D-003: select_star_on_wide_table +# --------------------------------------------------------------------------- +def test_d003_select_star(): + f = extract("SELECT * FROM events LIMIT 10") + assert f.has_select_star is True + assert f.has_limit is True + assert f.has_no_limit is False + + +# --------------------------------------------------------------------------- +# D-004: missing_limit_on_unbounded_result +# --------------------------------------------------------------------------- +def test_d004_missing_limit(): + f = extract("SELECT user_id, event_type, created_at FROM events WHERE status = 'active'") + assert f.has_no_limit is True + assert f.has_limit is False + + +# --------------------------------------------------------------------------- +# D-007: final_modifier_usage +# --------------------------------------------------------------------------- +def test_d007_final_modifier(): + f = extract("SELECT count() FROM events FINAL WHERE user_id = 1") + assert f.has_final_modifier is True + + +# --------------------------------------------------------------------------- +# D-014: async_insert_without_wait_flag +# --------------------------------------------------------------------------- +def test_d014_async_insert_no_wait(): + f = extract("INSERT INTO events SETTINGS async_insert=1 VALUES (1, 'click')") + assert f.has_async_insert_setting is True + assert f.has_async_insert_without_wait is True + + +def test_d014_async_insert_with_wait_ok(): + f = extract( + "INSERT INTO events SETTINGS async_insert=1, wait_for_async_insert=1 VALUES (1, 'click')" + ) + assert f.has_async_insert_setting is True + assert f.has_async_insert_without_wait is False + + +# --------------------------------------------------------------------------- +# Negative cases: clean queries should produce all-False features +# --------------------------------------------------------------------------- +def test_negative_simple_count(): + """Simple timestamp-range query – no problematic patterns.""" + sql = ( + "SELECT count() FROM events " + "WHERE created_at >= '2024-01-15 00:00:00' AND created_at < '2024-01-16 00:00:00'" + ) + f = extract(sql) + assert f.has_count_distinct is False + assert f.has_select_star is False + assert f.has_final_modifier is False + assert f.has_function_on_filter_column is False + assert f.has_quantile_exact is False + assert f.has_async_insert_without_wait is False + + +def test_negative_aggregate_no_limit(): + """Aggregate query – missing_limit rule should NOT fire.""" + f = extract("SELECT COUNT(*) FROM events WHERE event_type = 'click'") + assert f.has_no_limit is False + + +def test_negative_union_all_clean(): + """UNION ALL is fine – union_without_all should be False.""" + f = extract("SELECT id FROM a UNION ALL SELECT id FROM b LIMIT 100") + assert f.has_union is True + assert f.has_union_all is True + + +# --------------------------------------------------------------------------- +# Parse error resilience +# --------------------------------------------------------------------------- +def test_parse_error_returns_features(caplog): + """Badly malformed SQL must not raise – fallback path returns QueryFeatures.""" + bad_sql = "SELECT @@@ BROKEN SQL @@@ FROM" + with caplog.at_level(logging.WARNING, logger="clickadvisor.ml.features"): + f = extract(bad_sql) + assert isinstance(f, QueryFeatures) + assert f.query_length_chars == len(bad_sql) + + +def test_parse_error_logs_warning(caplog): + """A warning must be emitted when AST parsing fails.""" + with caplog.at_level(logging.WARNING): + _fx.extract("SELECT FROM WHERE ??? !!!") + assert any("AST parse failed" in r.message for r in caplog.records) + + +# --------------------------------------------------------------------------- +# Numeric fields +# --------------------------------------------------------------------------- +def test_numeric_fields_basic(): + f = extract("SELECT user_id, event_type FROM events WHERE status = 'active'") + assert f.table_count >= 1 + assert f.column_count_in_select == 2 + assert f.query_length_chars > 0 + + +def test_where_clause_depth_increases_with_nesting(): + simple = extract("SELECT count() FROM events WHERE x = 1") + nested = extract( + "SELECT count() FROM events WHERE (a = 1 AND (b = 2 OR (c = 3 AND d = 4)))" + ) + assert nested.where_clause_depth > simple.where_clause_depth + + +def test_to_vector_keys_match_fields(): + """to_vector() must return exactly the same set of field names as the dataclass.""" + import dataclasses + f = extract("SELECT count() FROM events") + vector = f.to_vector() + field_names = {field.name for field in dataclasses.fields(QueryFeatures)} + assert set(vector.keys()) == field_names diff --git a/tests/rules/test_d015.py b/tests/rules/test_d015.py new file mode 100644 index 00000000..ee47c4e6 --- /dev/null +++ b/tests/rules/test_d015.py @@ -0,0 +1,19 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D015_optimize_table_final import D015OptimizeTableFinal + + +def test_d015_triggers_on_optimize_final() -> None: + rule = D015OptimizeTableFinal() + ctx = QueryContext(sql="OPTIMIZE TABLE events FINAL") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-015" + assert finding.severity == "high" + assert "FINAL" in finding.description + + +def test_d015_no_trigger_on_optimize_without_final() -> None: + rule = D015OptimizeTableFinal() + ctx = QueryContext(sql="OPTIMIZE TABLE events") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d016.py b/tests/rules/test_d016.py new file mode 100644 index 00000000..a7fa4cac --- /dev/null +++ b/tests/rules/test_d016.py @@ -0,0 +1,19 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D016_alter_table_mutation import D016AlterTableMutation + + +def test_d016_triggers_on_alter_delete() -> None: + rule = D016AlterTableMutation() + ctx = QueryContext(sql="ALTER TABLE events DELETE WHERE dt < '2023-01-01'") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-016" + assert finding.severity == "high" + assert "DELETE" in finding.description + + +def test_d016_no_trigger_on_select() -> None: + rule = D016AlterTableMutation() + ctx = QueryContext(sql="SELECT * FROM events WHERE dt > '2023-01-01'") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d017.py b/tests/rules/test_d017.py new file mode 100644 index 00000000..9edaccd9 --- /dev/null +++ b/tests/rules/test_d017.py @@ -0,0 +1,23 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D017_nullable_column_in_ddl import D017NullableColumnInDDL + + +def test_d017_triggers_on_nullable_in_ddl() -> None: + rule = D017NullableColumnInDDL() + ctx = QueryContext( + sql="CREATE TABLE t (id UInt64, score Nullable(Float64)) ENGINE = MergeTree ORDER BY id" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-017" + assert finding.severity == "medium" + assert "Nullable" in finding.description + + +def test_d017_no_trigger_on_non_nullable_ddl() -> None: + rule = D017NullableColumnInDDL() + ctx = QueryContext( + sql="CREATE TABLE t (id UInt64, score Float64) ENGINE = MergeTree ORDER BY id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d018.py b/tests/rules/test_d018.py new file mode 100644 index 00000000..66394d7d --- /dev/null +++ b/tests/rules/test_d018.py @@ -0,0 +1,25 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D018_deprecated_ngrambf_tokenbf_index import ( + D018DeprecatedNgramBFIndex, +) + + +def test_d018_triggers_on_ngrambf_v1() -> None: + rule = D018DeprecatedNgramBFIndex() + ctx = QueryContext( + sql="ALTER TABLE logs ADD INDEX idx_msg message TYPE ngrambf_v1(4, 1024, 2, 0) GRANULARITY 4" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-018" + assert finding.severity == "medium" + assert "ngrambf_v1" in finding.description + + +def test_d018_no_trigger_on_text_index() -> None: + rule = D018DeprecatedNgramBFIndex() + ctx = QueryContext( + sql="ALTER TABLE logs ADD INDEX idx_msg message TYPE text GRANULARITY 1" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d019.py b/tests/rules/test_d019.py new file mode 100644 index 00000000..04ac556d --- /dev/null +++ b/tests/rules/test_d019.py @@ -0,0 +1,25 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D019_set_zero_unlimited_skip_index import ( + D019SetZeroUnlimitedSkipIndex, +) + + +def test_d019_triggers_on_set_zero() -> None: + rule = D019SetZeroUnlimitedSkipIndex() + ctx = QueryContext( + sql="CREATE TABLE t (status String, INDEX idx_status status TYPE set(0) GRANULARITY 4) ENGINE = MergeTree ORDER BY tuple()" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-019" + assert finding.severity == "low" + assert "set(0)" in finding.description + + +def test_d019_no_trigger_on_set_with_limit() -> None: + rule = D019SetZeroUnlimitedSkipIndex() + ctx = QueryContext( + sql="CREATE TABLE t (status String, INDEX idx_status status TYPE set(100) GRANULARITY 4) ENGINE = MergeTree ORDER BY tuple()" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d020.py b/tests/rules/test_d020.py new file mode 100644 index 00000000..c4c4c078 --- /dev/null +++ b/tests/rules/test_d020.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D020_partition_by_non_date import D020PartitionByNonDate + + +def test_d020_triggers_on_non_date_partition() -> None: + rule = D020PartitionByNonDate() + ctx = QueryContext( + sql="CREATE TABLE events (user_id UInt64, ts DateTime) ENGINE = MergeTree PARTITION BY user_id ORDER BY ts" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-020" + assert finding.severity == "medium" + + +def test_d020_no_trigger_on_date_function_partition() -> None: + rule = D020PartitionByNonDate() + ctx = QueryContext( + sql="CREATE TABLE events (user_id UInt64, ts DateTime) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY ts" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d021.py b/tests/rules/test_d021.py new file mode 100644 index 00000000..3c83ae31 --- /dev/null +++ b/tests/rules/test_d021.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D021_select_star_in_mv import D021SelectStarInMV + + +def test_d021_triggers_on_select_star_in_mv() -> None: + rule = D021SelectStarInMV() + ctx = QueryContext( + sql="CREATE MATERIALIZED VIEW events_mv TO events_agg AS SELECT * FROM events" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-021" + assert finding.severity == "medium" + + +def test_d021_no_trigger_on_explicit_columns() -> None: + rule = D021SelectStarInMV() + ctx = QueryContext( + sql="CREATE MATERIALIZED VIEW events_mv TO events_agg AS SELECT user_id, count() FROM events GROUP BY user_id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d022.py b/tests/rules/test_d022.py new file mode 100644 index 00000000..136d8e10 --- /dev/null +++ b/tests/rules/test_d022.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D022_delete_without_where import D022DeleteWithoutWhere + + +def test_d022_triggers_on_delete_without_where() -> None: + rule = D022DeleteWithoutWhere() + ctx = QueryContext(sql="DELETE FROM events") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-022" + assert finding.severity == "high" + + +def test_d022_no_trigger_on_delete_with_where() -> None: + rule = D022DeleteWithoutWhere() + ctx = QueryContext(sql="DELETE FROM events WHERE dt < '2023-01-01'") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d023.py b/tests/rules/test_d023.py new file mode 100644 index 00000000..89068a24 --- /dev/null +++ b/tests/rules/test_d023.py @@ -0,0 +1,21 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D023_mv_without_to import D023MVWithoutTo + + +def test_d023_triggers_on_mv_without_to() -> None: + rule = D023MVWithoutTo() + ctx = QueryContext( + sql="CREATE MATERIALIZED VIEW events_mv AS SELECT user_id, count() FROM events GROUP BY user_id" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-023" + + +def test_d023_no_trigger_on_mv_with_to() -> None: + rule = D023MVWithoutTo() + ctx = QueryContext( + sql="CREATE MATERIALIZED VIEW events_mv TO events_target AS SELECT user_id, count() FROM events GROUP BY user_id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d024.py b/tests/rules/test_d024.py new file mode 100644 index 00000000..ec359cdf --- /dev/null +++ b/tests/rules/test_d024.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D024_mv_with_populate import D024MVWithPopulate + + +def test_d024_triggers_on_populate() -> None: + rule = D024MVWithPopulate() + ctx = QueryContext( + sql="CREATE MATERIALIZED VIEW mv TO target AS SELECT user_id FROM events POPULATE" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-024" + assert finding.severity == "medium" + + +def test_d024_no_trigger_without_populate() -> None: + rule = D024MVWithPopulate() + ctx = QueryContext( + sql="CREATE MATERIALIZED VIEW mv TO target AS SELECT user_id FROM events" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_d025.py b/tests/rules/test_d025.py new file mode 100644 index 00000000..ff8f0c33 --- /dev/null +++ b/tests/rules/test_d025.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.detectors.D025_mv_with_join import D025MVWithJoin + + +def test_d025_triggers_on_mv_with_join() -> None: + rule = D025MVWithJoin() + ctx = QueryContext( + sql="CREATE MATERIALIZED VIEW mv TO t AS SELECT e.uid, u.name FROM events e LEFT JOIN users u ON e.uid = u.uid" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "D-025" + assert finding.severity == "medium" + + +def test_d025_no_trigger_on_mv_without_join() -> None: + rule = D025MVWithJoin() + ctx = QueryContext( + sql="CREATE MATERIALIZED VIEW mv TO t AS SELECT user_id, count() FROM events GROUP BY user_id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r021.py b/tests/rules/test_r021.py new file mode 100644 index 00000000..bc1f7567 --- /dev/null +++ b/tests/rules/test_r021.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R021_datetime64_zero import R021DateTime64ZeroToDateTime + + +def test_r021_triggers_on_datetime64_zero() -> None: + rule = R021DateTime64ZeroToDateTime() + ctx = QueryContext( + sql="CREATE TABLE events (ts DateTime64(0), user_id UInt64) ENGINE = MergeTree ORDER BY ts" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-021" + assert "DateTime64(0)" in finding.description + + +def test_r021_no_trigger_on_datetime64_with_precision() -> None: + rule = R021DateTime64ZeroToDateTime() + ctx = QueryContext( + sql="CREATE TABLE events (ts DateTime64(3), user_id UInt64) ENGINE = MergeTree ORDER BY ts" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r022.py b/tests/rules/test_r022.py new file mode 100644 index 00000000..491344d9 --- /dev/null +++ b/tests/rules/test_r022.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R022_float_monetary import R022FloatMonetary + + +def test_r022_triggers_on_float_amount_column() -> None: + rule = R022FloatMonetary() + ctx = QueryContext( + sql="CREATE TABLE orders (order_id UInt64, total_amount Float64) ENGINE = MergeTree ORDER BY order_id" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-022" + assert "total_amount" in finding.description + + +def test_r022_no_trigger_on_float_non_monetary() -> None: + rule = R022FloatMonetary() + ctx = QueryContext( + sql="CREATE TABLE metrics (ts DateTime, cpu_usage Float64) ENGINE = MergeTree ORDER BY ts" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r023.py b/tests/rules/test_r023.py new file mode 100644 index 00000000..fd880994 --- /dev/null +++ b/tests/rules/test_r023.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R023_string_datetime_column import R023StringDatetimeColumn + + +def test_r023_triggers_on_string_created_at() -> None: + rule = R023StringDatetimeColumn() + ctx = QueryContext( + sql="CREATE TABLE events (user_id UInt64, created_at String) ENGINE = MergeTree ORDER BY user_id" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-023" + assert "created_at" in finding.description + + +def test_r023_no_trigger_on_string_name() -> None: + rule = R023StringDatetimeColumn() + ctx = QueryContext( + sql="CREATE TABLE users (user_id UInt64, username String) ENGINE = MergeTree ORDER BY user_id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r024.py b/tests/rules/test_r024.py new file mode 100644 index 00000000..f3373e98 --- /dev/null +++ b/tests/rules/test_r024.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R024_string_ip_column import R024StringIPColumn + + +def test_r024_triggers_on_string_client_ip() -> None: + rule = R024StringIPColumn() + ctx = QueryContext( + sql="CREATE TABLE access_log (request_id UInt64, client_ip String) ENGINE = MergeTree ORDER BY request_id" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-024" + assert "client_ip" in finding.description + + +def test_r024_no_trigger_on_string_username() -> None: + rule = R024StringIPColumn() + ctx = QueryContext( + sql="CREATE TABLE users (user_id UInt64, username String) ENGINE = MergeTree ORDER BY user_id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r025.py b/tests/rules/test_r025.py new file mode 100644 index 00000000..f6318f5a --- /dev/null +++ b/tests/rules/test_r025.py @@ -0,0 +1,23 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R025_order_by_tuple import R025OrderByTupleNoPK + + +def test_r025_triggers_on_order_by_tuple() -> None: + rule = R025OrderByTupleNoPK() + ctx = QueryContext( + sql="CREATE TABLE events (user_id UInt64, ts DateTime) ENGINE = MergeTree ORDER BY tuple()" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-025" + assert finding.severity == "medium" + assert "primary key" in finding.description.lower() or "tuple()" in finding.description + + +def test_r025_no_trigger_on_proper_order_by() -> None: + rule = R025OrderByTupleNoPK() + ctx = QueryContext( + sql="CREATE TABLE events (user_id UInt64, ts DateTime) ENGINE = MergeTree ORDER BY (user_id, ts)" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r026.py b/tests/rules/test_r026.py new file mode 100644 index 00000000..2894e6e2 --- /dev/null +++ b/tests/rules/test_r026.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R026_sum_case_to_countif import R026SumCaseToCountIf + + +def test_r026_triggers_on_sum_case_one() -> None: + rule = R026SumCaseToCountIf() + ctx = QueryContext( + sql="SELECT SUM(CASE WHEN status = 'active' THEN 1 ELSE 0 END) AS active_cnt FROM users" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-026" + assert "countIf" in finding.suggestion + + +def test_r026_no_trigger_on_sum_col() -> None: + rule = R026SumCaseToCountIf() + ctx = QueryContext( + sql="SELECT SUM(CASE WHEN is_paid THEN amount ELSE 0 END) AS total FROM orders" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r027.py b/tests/rules/test_r027.py new file mode 100644 index 00000000..42cf6fc8 --- /dev/null +++ b/tests/rules/test_r027.py @@ -0,0 +1,20 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R027_sum_case_col_to_sumif import R027SumCaseColToSumIf + + +def test_r027_triggers_on_sum_case_col() -> None: + rule = R027SumCaseColToSumIf() + ctx = QueryContext( + sql="SELECT SUM(CASE WHEN is_paid THEN amount ELSE 0 END) AS paid_total FROM orders" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-027" + assert "sumIf" in finding.suggestion + + +def test_r027_no_trigger_on_sum_without_case() -> None: + rule = R027SumCaseColToSumIf() + ctx = QueryContext(sql="SELECT SUM(amount) FROM orders") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r028.py b/tests/rules/test_r028.py new file mode 100644 index 00000000..0b27c41a --- /dev/null +++ b/tests/rules/test_r028.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R028_coalesce_to_ifnull import R028CoalesceToIfNull + + +def test_r028_triggers_on_coalesce_two_args() -> None: + rule = R028CoalesceToIfNull() + ctx = QueryContext( + sql="SELECT COALESCE(user_name, 'anonymous') AS name FROM users" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-028" + assert "ifNull" in finding.suggestion + + +def test_r028_no_trigger_on_coalesce_three_args() -> None: + rule = R028CoalesceToIfNull() + ctx = QueryContext( + sql="SELECT COALESCE(a, b, c) FROM t" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r029.py b/tests/rules/test_r029.py new file mode 100644 index 00000000..baf53dbc --- /dev/null +++ b/tests/rules/test_r029.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R029_lower_like_to_ilike import R029LowerLikeToILike + + +def test_r029_triggers_on_lower_like() -> None: + rule = R029LowerLikeToILike() + ctx = QueryContext( + sql="SELECT * FROM logs WHERE lower(message) LIKE '%error%'" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-029" + assert "ILIKE" in finding.suggestion + + +def test_r029_no_trigger_on_plain_like() -> None: + rule = R029LowerLikeToILike() + ctx = QueryContext( + sql="SELECT * FROM logs WHERE message LIKE '%error%'" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r030.py b/tests/rules/test_r030.py new file mode 100644 index 00000000..620e816e --- /dev/null +++ b/tests/rules/test_r030.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R030_not_in_singleton import R030NotInSingleton + + +def test_r030_triggers_on_not_in_single_value() -> None: + rule = R030NotInSingleton() + ctx = QueryContext( + sql="SELECT * FROM events WHERE status NOT IN ('deleted')" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-030" + assert "!=" in finding.suggestion + + +def test_r030_no_trigger_on_not_in_multiple() -> None: + rule = R030NotInSingleton() + ctx = QueryContext( + sql="SELECT * FROM events WHERE status NOT IN ('deleted', 'archived')" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r031.py b/tests/rules/test_r031.py new file mode 100644 index 00000000..b150b9f2 --- /dev/null +++ b/tests/rules/test_r031.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R031_string_uuid_column import R031StringUUIDColumn + + +def test_r031_triggers_on_string_trace_id() -> None: + rule = R031StringUUIDColumn() + ctx = QueryContext( + sql="CREATE TABLE spans (trace_id String, span_id String, ts DateTime) ENGINE = MergeTree ORDER BY ts" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-031" + assert "UUID" in finding.suggestion + + +def test_r031_no_trigger_on_string_name() -> None: + rule = R031StringUUIDColumn() + ctx = QueryContext( + sql="CREATE TABLE users (user_id UInt64, full_name String) ENGINE = MergeTree ORDER BY user_id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r032.py b/tests/rules/test_r032.py new file mode 100644 index 00000000..768b5d1b --- /dev/null +++ b/tests/rules/test_r032.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R032_int8_boolean_column import R032Int8BooleanColumn + + +def test_r032_triggers_on_uint8_is_active() -> None: + rule = R032Int8BooleanColumn() + ctx = QueryContext( + sql="CREATE TABLE users (user_id UInt64, is_active UInt8) ENGINE = MergeTree ORDER BY user_id" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-032" + assert "Bool" in finding.suggestion + + +def test_r032_no_trigger_on_uint8_non_boolean_name() -> None: + rule = R032Int8BooleanColumn() + ctx = QueryContext( + sql="CREATE TABLE events (user_id UInt64, event_count UInt8) ENGINE = MergeTree ORDER BY user_id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r033.py b/tests/rules/test_r033.py new file mode 100644 index 00000000..55db771e --- /dev/null +++ b/tests/rules/test_r033.py @@ -0,0 +1,20 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R033_max_case_to_maxif import R033MaxCaseToMaxIf + + +def test_r033_triggers_on_max_case() -> None: + rule = R033MaxCaseToMaxIf() + ctx = QueryContext( + sql="SELECT MAX(CASE WHEN status = 'active' THEN score END) AS max_score FROM users" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-033" + assert "maxIf" in finding.suggestion + + +def test_r033_no_trigger_on_plain_max() -> None: + rule = R033MaxCaseToMaxIf() + ctx = QueryContext(sql="SELECT MAX(score) FROM users") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r034.py b/tests/rules/test_r034.py new file mode 100644 index 00000000..03542f53 --- /dev/null +++ b/tests/rules/test_r034.py @@ -0,0 +1,20 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R034_min_case_to_minif import R034MinCaseToMinIf + + +def test_r034_triggers_on_min_case() -> None: + rule = R034MinCaseToMinIf() + ctx = QueryContext( + sql="SELECT MIN(CASE WHEN is_paid THEN amount END) AS min_paid FROM orders" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-034" + assert "minIf" in finding.suggestion + + +def test_r034_no_trigger_on_min_without_case() -> None: + rule = R034MinCaseToMinIf() + ctx = QueryContext(sql="SELECT MIN(amount) FROM orders WHERE is_paid") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r035.py b/tests/rules/test_r035.py new file mode 100644 index 00000000..78bd71ec --- /dev/null +++ b/tests/rules/test_r035.py @@ -0,0 +1,20 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R035_avg_case_to_avgif import R035AvgCaseToAvgIf + + +def test_r035_triggers_on_avg_case() -> None: + rule = R035AvgCaseToAvgIf() + ctx = QueryContext( + sql="SELECT AVG(CASE WHEN status = 'paid' THEN amount END) AS avg_paid FROM orders" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-035" + assert "avgIf" in finding.suggestion + + +def test_r035_no_trigger_on_avg_without_case() -> None: + rule = R035AvgCaseToAvgIf() + ctx = QueryContext(sql="SELECT AVG(amount) FROM orders") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r036.py b/tests/rules/test_r036.py new file mode 100644 index 00000000..6dc5fc12 --- /dev/null +++ b/tests/rules/test_r036.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R036_nested_if_to_multiif import R036NestedIfToMultiIf + + +def test_r036_triggers_on_nested_if() -> None: + rule = R036NestedIfToMultiIf() + ctx = QueryContext( + sql="SELECT IF(score > 90, 'A', IF(score > 70, 'B', 'C')) AS grade FROM students" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-036" + assert "multiIf" in finding.suggestion + + +def test_r036_no_trigger_on_simple_if() -> None: + rule = R036NestedIfToMultiIf() + ctx = QueryContext( + sql="SELECT IF(score > 90, 'A', 'B') AS grade FROM students" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r037.py b/tests/rules/test_r037.py new file mode 100644 index 00000000..94e3c803 --- /dev/null +++ b/tests/rules/test_r037.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R037_empty_string_eq_to_empty import R037EmptyStringEqToEmpty + + +def test_r037_triggers_on_eq_empty_string() -> None: + rule = R037EmptyStringEqToEmpty() + ctx = QueryContext(sql="SELECT * FROM events WHERE message = ''") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-037" + assert "empty" in finding.suggestion + + +def test_r037_no_trigger_on_eq_nonempty_string() -> None: + rule = R037EmptyStringEqToEmpty() + ctx = QueryContext(sql="SELECT * FROM events WHERE message = 'hello'") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r038.py b/tests/rules/test_r038.py new file mode 100644 index 00000000..cffe2bb1 --- /dev/null +++ b/tests/rules/test_r038.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R038_nonempty_string_neq import R038NonEmptyStringNeqToNotEmpty + + +def test_r038_triggers_on_neq_empty_string() -> None: + rule = R038NonEmptyStringNeqToNotEmpty() + ctx = QueryContext(sql="SELECT * FROM events WHERE message != ''") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-038" + assert "notEmpty" in finding.suggestion + + +def test_r038_no_trigger_on_neq_nonempty_string() -> None: + rule = R038NonEmptyStringNeqToNotEmpty() + ctx = QueryContext(sql="SELECT * FROM events WHERE message != 'error'") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r039.py b/tests/rules/test_r039.py new file mode 100644 index 00000000..625c3414 --- /dev/null +++ b/tests/rules/test_r039.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R039_length_gte_one_to_notempty import R039LengthGteOneToNotEmpty + + +def test_r039_triggers_on_length_gte_one() -> None: + rule = R039LengthGteOneToNotEmpty() + ctx = QueryContext(sql="SELECT * FROM events WHERE length(tags) >= 1") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-039" + assert "notEmpty" in finding.suggestion + + +def test_r039_no_trigger_on_length_gt_zero() -> None: + rule = R039LengthGteOneToNotEmpty() + ctx = QueryContext(sql="SELECT * FROM events WHERE length(tags) > 0") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r040.py b/tests/rules/test_r040.py new file mode 100644 index 00000000..3e0d6dff --- /dev/null +++ b/tests/rules/test_r040.py @@ -0,0 +1,23 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R040_todate_comparison import R040TodateComparisonToDatetime + + +def test_r040_triggers_on_todate_gte_date() -> None: + rule = R040TodateComparisonToDatetime() + ctx = QueryContext( + sql="SELECT * FROM events WHERE toDate(ts) >= '2024-01-01'" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-040" + assert finding.severity == "high" + assert "ts >=" in finding.suggestion + + +def test_r040_no_trigger_on_direct_datetime_comparison() -> None: + rule = R040TodateComparisonToDatetime() + ctx = QueryContext( + sql="SELECT * FROM events WHERE ts >= '2024-01-01 00:00:00'" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r041.py b/tests/rules/test_r041.py new file mode 100644 index 00000000..73727f41 --- /dev/null +++ b/tests/rules/test_r041.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R041_string_code_column import R041StringCodeColumn + + +def test_r041_triggers_on_string_country_code() -> None: + rule = R041StringCodeColumn() + ctx = QueryContext( + sql="CREATE TABLE orders (order_id UInt64, country_code String) ENGINE = MergeTree ORDER BY order_id" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-041" + assert "FixedString" in finding.suggestion + + +def test_r041_no_trigger_on_string_description() -> None: + rule = R041StringCodeColumn() + ctx = QueryContext( + sql="CREATE TABLE products (product_id UInt64, description String) ENGINE = MergeTree ORDER BY product_id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r042.py b/tests/rules/test_r042.py new file mode 100644 index 00000000..22f27c96 --- /dev/null +++ b/tests/rules/test_r042.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R042_grouparray_no_limit import R042GroupArrayNoLimit + + +def test_r042_triggers_on_grouparray_without_limit() -> None: + rule = R042GroupArrayNoLimit() + ctx = QueryContext( + sql="SELECT user_id, groupArray(event_type) AS events FROM log GROUP BY user_id" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-042" + assert "1000" in finding.suggestion + + +def test_r042_no_trigger_on_grouparray_with_limit() -> None: + rule = R042GroupArrayNoLimit() + ctx = QueryContext( + sql="SELECT user_id, groupArray(1000)(event_type) AS events FROM log GROUP BY user_id" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r043.py b/tests/rules/test_r043.py new file mode 100644 index 00000000..d4d683e3 --- /dev/null +++ b/tests/rules/test_r043.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R043_having_count_gt_zero import R043HavingCountGtZero + + +def test_r043_triggers_on_having_count_gt_zero() -> None: + rule = R043HavingCountGtZero() + ctx = QueryContext( + sql="SELECT user_id, count() FROM events GROUP BY user_id HAVING count() > 0" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-043" + assert "тавтологи" in finding.description + + +def test_r043_no_trigger_on_having_count_gt_one() -> None: + rule = R043HavingCountGtZero() + ctx = QueryContext( + sql="SELECT user_id, count() FROM events GROUP BY user_id HAVING count() > 1" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r044.py b/tests/rules/test_r044.py new file mode 100644 index 00000000..02ad1fa2 --- /dev/null +++ b/tests/rules/test_r044.py @@ -0,0 +1,20 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R044_todatetime_todate import R044ToDateTimeToDateToStartOfDay + + +def test_r044_triggers_on_todatetime_todate() -> None: + rule = R044ToDateTimeToDateToStartOfDay() + ctx = QueryContext( + sql="SELECT toDateTime(toDate(ts)) AS day_start FROM events" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-044" + assert "toStartOfDay" in finding.suggestion + + +def test_r044_no_trigger_on_plain_todate() -> None: + rule = R044ToDateTimeToDateToStartOfDay() + ctx = QueryContext(sql="SELECT toDate(ts) AS day FROM events") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r045.py b/tests/rules/test_r045.py new file mode 100644 index 00000000..7840ae06 --- /dev/null +++ b/tests/rules/test_r045.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R045_like_without_wildcards import R045LikeWithoutWildcardsToEq + + +def test_r045_triggers_on_like_without_wildcards() -> None: + rule = R045LikeWithoutWildcardsToEq() + ctx = QueryContext( + sql="SELECT * FROM events WHERE event_type LIKE 'click'" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-045" + assert "= 'click'" in finding.suggestion + + +def test_r045_no_trigger_on_like_with_wildcards() -> None: + rule = R045LikeWithoutWildcardsToEq() + ctx = QueryContext( + sql="SELECT * FROM events WHERE event_type LIKE '%click%'" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r046.py b/tests/rules/test_r046.py new file mode 100644 index 00000000..a726d38a --- /dev/null +++ b/tests/rules/test_r046.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R046_not_empty_to_notempty import R046NotEmptyToNotEmpty + + +def test_r046_triggers_on_not_empty() -> None: + rule = R046NotEmptyToNotEmpty() + ctx = QueryContext(sql="SELECT * FROM events WHERE NOT empty(message)") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-046" + assert "notEmpty" in finding.suggestion + + +def test_r046_no_trigger_on_notempty() -> None: + rule = R046NotEmptyToNotEmpty() + ctx = QueryContext(sql="SELECT * FROM events WHERE notEmpty(message)") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r047.py b/tests/rules/test_r047.py new file mode 100644 index 00000000..630dc51f --- /dev/null +++ b/tests/rules/test_r047.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R047_position_to_like import R047PositionToLike + + +def test_r047_triggers_on_position_gt_zero() -> None: + rule = R047PositionToLike() + ctx = QueryContext(sql="SELECT * FROM logs WHERE position(message, 'error') > 0") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-047" + assert "LIKE" in finding.suggestion + + +def test_r047_no_trigger_on_position_eq_zero() -> None: + rule = R047PositionToLike() + ctx = QueryContext(sql="SELECT * FROM logs WHERE position(message, 'error') = 0") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r048.py b/tests/rules/test_r048.py new file mode 100644 index 00000000..2d0efe7b --- /dev/null +++ b/tests/rules/test_r048.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R048_positionci_to_ilike import R048PositionCIToILike + + +def test_r048_triggers_on_positionci_gt_zero() -> None: + rule = R048PositionCIToILike() + ctx = QueryContext(sql="SELECT * FROM logs WHERE positionCaseInsensitive(message, 'error') > 0") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-048" + assert "ILIKE" in finding.suggestion + + +def test_r048_no_trigger_on_positionci_eq_zero() -> None: + rule = R048PositionCIToILike() + ctx = QueryContext(sql="SELECT * FROM logs WHERE positionCaseInsensitive(message, 'error') = 0") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r049.py b/tests/rules/test_r049.py new file mode 100644 index 00000000..947f6103 --- /dev/null +++ b/tests/rules/test_r049.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R049_sumif_one_to_countif import R049SumIfOneToCountIf + + +def test_r049_triggers_on_sumif_one() -> None: + rule = R049SumIfOneToCountIf() + ctx = QueryContext(sql="SELECT sumIf(1, status = 'active') AS active_cnt FROM users") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-049" + assert "countIf" in finding.suggestion + + +def test_r049_no_trigger_on_sumif_col() -> None: + rule = R049SumIfOneToCountIf() + ctx = QueryContext(sql="SELECT sumIf(amount, is_paid) AS total FROM orders") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r050.py b/tests/rules/test_r050.py new file mode 100644 index 00000000..c0de0ff5 --- /dev/null +++ b/tests/rules/test_r050.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R050_toyyyymm_comparison import R050ToYYYYMMComparison + + +def test_r050_triggers_on_toyyyymm_gte() -> None: + rule = R050ToYYYYMMComparison() + ctx = QueryContext(sql="SELECT * FROM events WHERE toYYYYMM(ts) >= 202401") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-050" + assert finding.severity == "high" + + +def test_r050_no_trigger_on_direct_datetime() -> None: + rule = R050ToYYYYMMComparison() + ctx = QueryContext(sql="SELECT * FROM events WHERE ts >= '2024-01-01 00:00:00'") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r051.py b/tests/rules/test_r051.py new file mode 100644 index 00000000..27f5db79 --- /dev/null +++ b/tests/rules/test_r051.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R051_date_trunc_to_native import R051DateTruncToNative + + +def test_r051_triggers_on_date_trunc_day() -> None: + rule = R051DateTruncToNative() + ctx = QueryContext(sql="SELECT DATE_TRUNC('day', ts) AS day FROM events") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-051" + assert "toStartOfDay" in finding.suggestion + + +def test_r051_no_trigger_on_native_function() -> None: + rule = R051DateTruncToNative() + ctx = QueryContext(sql="SELECT toStartOfDay(ts) AS day FROM events") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r052.py b/tests/rules/test_r052.py new file mode 100644 index 00000000..e8e09f3c --- /dev/null +++ b/tests/rules/test_r052.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R052_formatdatetime_ymd import R052FormatDateTimeYMD + + +def test_r052_triggers_on_formatdatetime_ymd() -> None: + rule = R052FormatDateTimeYMD() + ctx = QueryContext(sql="SELECT formatDateTime(ts, '%Y-%m-%d') AS day FROM events") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-052" + assert "toDate" in finding.suggestion + + +def test_r052_no_trigger_on_other_format() -> None: + rule = R052FormatDateTimeYMD() + ctx = QueryContext(sql="SELECT formatDateTime(ts, '%Y-%m') AS month FROM events") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r053.py b/tests/rules/test_r053.py new file mode 100644 index 00000000..d0d9aed0 --- /dev/null +++ b/tests/rules/test_r053.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R053_arrayreduce_sum import R053ArrayReduceSum + + +def test_r053_triggers_on_arrayreduce_sum() -> None: + rule = R053ArrayReduceSum() + ctx = QueryContext(sql="SELECT arrayReduce('sum', amounts) FROM orders") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-053" + assert "arraySum" in finding.suggestion + + +def test_r053_no_trigger_on_arraysum() -> None: + rule = R053ArrayReduceSum() + ctx = QueryContext(sql="SELECT arraySum(amounts) FROM orders") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r054.py b/tests/rules/test_r054.py new file mode 100644 index 00000000..3354027c --- /dev/null +++ b/tests/rules/test_r054.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R054_arrayreduce_max import R054ArrayReduceMax + + +def test_r054_triggers_on_arrayreduce_max() -> None: + rule = R054ArrayReduceMax() + ctx = QueryContext(sql="SELECT arrayReduce('max', scores) FROM results") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-054" + assert "arrayMax" in finding.suggestion + + +def test_r054_no_trigger_on_arraymax() -> None: + rule = R054ArrayReduceMax() + ctx = QueryContext(sql="SELECT arrayMax(scores) FROM results") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r055.py b/tests/rules/test_r055.py new file mode 100644 index 00000000..b99c886f --- /dev/null +++ b/tests/rules/test_r055.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R055_arrayreduce_min import R055ArrayReduceMin + + +def test_r055_triggers_on_arrayreduce_min() -> None: + rule = R055ArrayReduceMin() + ctx = QueryContext(sql="SELECT arrayReduce('min', scores) FROM results") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-055" + assert "arrayMin" in finding.suggestion + + +def test_r055_no_trigger_on_arraymin() -> None: + rule = R055ArrayReduceMin() + ctx = QueryContext(sql="SELECT arrayMin(scores) FROM results") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r056.py b/tests/rules/test_r056.py new file mode 100644 index 00000000..428b3aac --- /dev/null +++ b/tests/rules/test_r056.py @@ -0,0 +1,25 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R056_extract_year import R056ExtractToNative + + +def test_r056_triggers_on_extract_year() -> None: + rule = R056ExtractToNative() + ctx = QueryContext(sql="SELECT EXTRACT(YEAR FROM ts) AS yr FROM events") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-056" + assert "toYear" in finding.suggestion + + +def test_r056_does_not_trigger_on_extract_month() -> None: + rule = R056ExtractToNative() + ctx = QueryContext(sql="SELECT EXTRACT(MONTH FROM ts) AS mon FROM events") + finding = rule.check(ctx) + assert finding is None + + +def test_r056_no_trigger_on_toyear() -> None: + rule = R056ExtractToNative() + ctx = QueryContext(sql="SELECT toYear(ts) AS yr FROM events") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r057.py b/tests/rules/test_r057.py new file mode 100644 index 00000000..b5f21f50 --- /dev/null +++ b/tests/rules/test_r057.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R057_extract_month import R057ExtractMonthToMonth + + +def test_r057_triggers_on_extract_month() -> None: + rule = R057ExtractMonthToMonth() + ctx = QueryContext(sql="SELECT EXTRACT(MONTH FROM ts) AS mon FROM events") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-057" + assert "toMonth" in finding.suggestion + + +def test_r057_no_trigger_on_tomonth() -> None: + rule = R057ExtractMonthToMonth() + ctx = QueryContext(sql="SELECT toMonth(ts) AS mon FROM events") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r058.py b/tests/rules/test_r058.py new file mode 100644 index 00000000..4bb7b0dc --- /dev/null +++ b/tests/rules/test_r058.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R058_extract_day import R058ExtractDayToDayOfMonth + + +def test_r058_triggers_on_extract_day() -> None: + rule = R058ExtractDayToDayOfMonth() + ctx = QueryContext(sql="SELECT EXTRACT(DAY FROM ts) AS day_num FROM events") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-058" + assert "toDayOfMonth" in finding.suggestion + + +def test_r058_no_trigger_on_todayofmonth() -> None: + rule = R058ExtractDayToDayOfMonth() + ctx = QueryContext(sql="SELECT toDayOfMonth(ts) AS day_num FROM events") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r059.py b/tests/rules/test_r059.py new file mode 100644 index 00000000..1bc1b60a --- /dev/null +++ b/tests/rules/test_r059.py @@ -0,0 +1,18 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R059_extract_hour import R059ExtractHourToHour + + +def test_r059_triggers_on_extract_hour() -> None: + rule = R059ExtractHourToHour() + ctx = QueryContext(sql="SELECT EXTRACT(HOUR FROM ts) AS hr FROM events") + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-059" + assert "toHour" in finding.suggestion + + +def test_r059_no_trigger_on_tohour() -> None: + rule = R059ExtractHourToHour() + ctx = QueryContext(sql="SELECT toHour(ts) AS hr FROM events") + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r060.py b/tests/rules/test_r060.py new file mode 100644 index 00000000..2465c133 --- /dev/null +++ b/tests/rules/test_r060.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R060_has_and_has_to_hasall import R060HasAndHasToHasAll + + +def test_r060_triggers_on_has_and_has() -> None: + rule = R060HasAndHasToHasAll() + ctx = QueryContext( + sql="SELECT * FROM events WHERE has(tags, 'error') AND has(tags, 'critical')" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-060" + assert "hasAll" in finding.suggestion + + +def test_r060_no_trigger_on_different_arrays() -> None: + rule = R060HasAndHasToHasAll() + ctx = QueryContext( + sql="SELECT * FROM events WHERE has(tags, 'error') AND has(labels, 'critical')" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r061.py b/tests/rules/test_r061.py new file mode 100644 index 00000000..eb78c57f --- /dev/null +++ b/tests/rules/test_r061.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R061_has_or_has_to_hasany import R061HasOrHasToHasAny + + +def test_r061_triggers_on_has_or_has() -> None: + rule = R061HasOrHasToHasAny() + ctx = QueryContext( + sql="SELECT * FROM events WHERE has(tags, 'error') OR has(tags, 'warning')" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-061" + assert "hasAny" in finding.suggestion + + +def test_r061_no_trigger_on_different_arrays() -> None: + rule = R061HasOrHasToHasAny() + ctx = QueryContext( + sql="SELECT * FROM events WHERE has(tags, 'error') OR has(labels, 'warning')" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/rules/test_r062.py b/tests/rules/test_r062.py new file mode 100644 index 00000000..995dd75b --- /dev/null +++ b/tests/rules/test_r062.py @@ -0,0 +1,22 @@ +from clickadvisor.core.models import QueryContext +from clickadvisor.rules.tier1.R062_arraycount_zero_to_not_has import R062ArrayCountZeroToNotHas + + +def test_r062_triggers_on_arraycount_eq_zero() -> None: + rule = R062ArrayCountZeroToNotHas() + ctx = QueryContext( + sql="SELECT * FROM events WHERE arrayCount(x -> x = 'error', tags) = 0" + ) + finding = rule.check(ctx) + assert finding is not None + assert finding.rule_id == "R-062" + assert "NOT has" in finding.suggestion + + +def test_r062_no_trigger_on_arraycount_gt_zero() -> None: + rule = R062ArrayCountZeroToNotHas() + ctx = QueryContext( + sql="SELECT * FROM events WHERE arrayCount(x -> x = 'error', tags) > 0" + ) + finding = rule.check(ctx) + assert finding is None diff --git a/tests/test_generate_synthetic_dataset.py b/tests/test_generate_synthetic_dataset.py index bea33ae3..4b20b75a 100644 --- a/tests/test_generate_synthetic_dataset.py +++ b/tests/test_generate_synthetic_dataset.py @@ -4,8 +4,10 @@ def test_generated_synthetic_dataset_has_expected_size_and_negatives() -> None: cases = build_cases() negative_cases = [case for case in cases if not case["expected_rules_to_fire"]] - assert len(cases) >= 150 + labels = {rule_id for case in cases for rule_id in case["expected_rules_to_fire"]} + assert len(cases) == 180 assert len(negative_cases) >= 20 + assert {"R-057", "R-058", "R-059"}.issubset(labels) assert all(case["status"] == "validated" for case in cases)