diff --git a/README.md b/README.md index 2c8881c0..b4cf1213 100644 --- a/README.md +++ b/README.md @@ -37,8 +37,9 @@ ClickHouse, могут опираться на устаревшие рекоме - CLI-команда `chadvisor analyze`; - MCP-сервер `chadvisor mcp-server`; -- 62 rewrite/advisory-правил `R-*`; -- 15 детекторов антипаттернов `D-*`; +- 74 rewrite/advisory-правила `R-*`; +- 25 детекторов антипаттернов `D-*`; +- 20 environment-правил `E-*`; - version detection через ClickHouse HTTP API; - console / JSON / Markdown отчёты; - режим объяснений `--mode explain`; @@ -47,8 +48,8 @@ ClickHouse, могут опираться на устаревшие рекоме - synthetic benchmark для проверки срабатывания правил. В каталоге `/docs/rules/cards/` описаны 119 валидируемых карточек правил: -77 из них уже имеют реализацию и тесты, 42 остаются backlog-карточками для -следующих итераций. +119 имеют зарегистрированную реализацию и прямое тестовое покрытие, backlog +по карточкам закрыт. ## Быстрый старт @@ -171,19 +172,20 @@ poetry run chadvisor analyze --sql query.sql \ пользовательские данные не читаются. Используется только оценка планировщика ClickHouse (`rows`, `marks`). -### Schema и EXPLAIN как дополнительные входы +### Schema, EXPLAIN и environment как дополнительные входы CLI уже принимает опциональные файлы: ```bash poetry run chadvisor analyze --sql query.sql \ --schema schema.sql \ + --environment environment.json \ --explain explain.json ``` -На текущем этапе основная часть реализованных правил работает по SQL AST. -Схема и EXPLAIN зарезервированы для правил, которым нужен дополнительный -контекст. +`environment.json` передаёт настройки, hardware, cluster/workload facts и +system metrics для `E-*` и части Tier 2 advisory rules. Если файл окружения не +передан, environment-правила не срабатывают и SQL-only анализ остаётся прежним. ## Knowledge base и retrieval advisory @@ -258,15 +260,13 @@ poetry run chadvisor mcp-server | Surface | Карточки | Реализовано и протестировано | Backlog | |---|---:|---:|---:| -| `R-*` rewrite/advisory rules | 74 | 62 | 12 | -| `D-*` detectors | 25 | 15 | 10 | -| `E-*` environment cards | 20 | 0 | 20 | -| Всего | 119 | 77 | 42 | +| `R-*` rewrite/advisory rules | 74 | 74 | 0 | +| `D-*` detectors | 25 | 25 | 0 | +| `E-*` environment cards | 20 | 20 | 0 | +| Всего | 119 | 119 | 0 | -Реализованные диапазоны: `R-001`…`R-062`, -`D-003`, `D-004`, `D-007`, `D-014`…`D-025`. Карточки `R-101`…`R-112`, -`D-001`, `D-002`, `D-005`…`D-013` и `E-001`…`E-020` -пока описывают backlog. +Реализованные диапазоны: `R-001`…`R-062`, `R-101`…`R-112`, +`D-001`…`D-025` и `E-001`…`E-020`. Полный список карточек хранится в [`docs/rules/cards/`](docs/rules/cards/), а фактическая регистрация правил — в @@ -278,13 +278,14 @@ poetry run chadvisor mcp-server - `1B` — приближённые или opt-in рекомендации; - `1C` — условные рекомендации, зависящие от схемы или контекста; - `detector` — диагностика антипаттерна без автоматического rewrite. +- `env` — environment-aware настройки по hardware, settings, cluster и workload. ## Метрики качества Текущие воспроизводимые метрики на 2026-06-30: -- rule detection on synthetic held-out: `36/36` cases, strict precision - `1.000`, recall `1.000`, F1 `1.000`; +- rule detection on expanded synthetic benchmark: `222/222` cases, strict + precision `1.000`, recall `1.000`, F1 `1.000`; - classifier on `synthetic_expanded_v1` test split: best test macro F1 `0.691`, best test micro F1 `0.988`; - retrieval MRR@3: `0.517` on `20` explicit query-doc pairs with MiniLM-L6 @@ -322,7 +323,7 @@ poetry run python scripts/eval/ablation_embeddings.py ## Архитектура ```text -SQL + optional Schema / EXPLAIN / CH version +SQL + optional Schema / EXPLAIN / Environment / CH version ↓ SQL Parser (sqlglot, ClickHouse dialect) ↓ @@ -380,7 +381,6 @@ CLI/MCP формируются rule engine, ML evaluation surface и локал - автоматический анализ `query_log`; - автоматические DDL-изменения; - выполнение `ANALYZE` или реальный replay запросов на данных; -- полноценные environment rules по hardware/config. +- автоматическое применение Tier 2 design/storage рекомендаций без проверки DBA. -Эти направления описаны в ADR и backlog, но README отражает именно то, что -поддерживает текущий код репозитория. +README отражает именно то, что поддерживает текущий код репозитория. diff --git a/benchmark/README.md b/benchmark/README.md index 87fb99ba..a2e56dcc 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -26,14 +26,17 @@ poetry run python scripts/eval/run_benchmark.py The repository also includes an expanded generated synthetic dataset: -- `benchmark/cases/synthetic_expanded/`: 180 deterministic cases +- `benchmark/cases/synthetic_expanded/`: 222 deterministic cases, including + SQL-only, schema-aware, and env-aware rule probes - `benchmark/splits/synthetic_expanded_v1.yaml`: fixed 80/20 train/test split + for the original 180-case ML dataset - `scripts/benchmark/generate_synthetic_dataset.py`: reproducible generator The expanded dataset exists to avoid presenting a suspicious metric on only 20 hand-authored probes. It contains positive variations for implemented rules, -multi-label overlaps where the current analyzer really emits several findings, -and negative queries where no implemented rule should fire. +targeted backlog-closure probes for `D-*`, `E-*`, and `R-101+`, multi-label +overlaps where the current analyzer really emits several findings, and +negative queries where no implemented rule should fire. Run the expanded benchmark: @@ -101,6 +104,9 @@ Core fields: - `source` - `sql` - `schema_files` +- `schema_ddl` for inline schema-aware cases +- `environment` for env-aware cases +- `context_type`: `sql-only`, `schema-aware`, `env-aware`, or `explain-aware` - `known_issues` - `expected_rules_to_fire` - `expected_improvement` @@ -113,8 +119,9 @@ Core fields: - `cases/clickbench/`: ClickBench query seeds - `cases/job/`: selected JOIN-heavy JOB seeds - `cases/synthetic/`: 20 validated targeted rule cases -- `cases/synthetic_expanded/`: 180 generated rule-regression cases with - negative examples and a fixed split +- `cases/synthetic_expanded/`: 222 rule-regression cases with negative examples, + fixed split metadata for the original generated subset, and targeted + schema/env-aware probes for newly implemented rules - `cases/github-issues/`: placeholder for manually curated issue-derived cases ## Validation diff --git a/benchmark/SCHEMA.yaml b/benchmark/SCHEMA.yaml index e2ec6741..8b398b12 100644 --- a/benchmark/SCHEMA.yaml +++ b/benchmark/SCHEMA.yaml @@ -29,6 +29,13 @@ properties: type: array items: type: string + schema_ddl: + type: ["string", "null"] + environment: + type: ["object", "null"] + context_type: + type: string + enum: ["sql-only", "schema-aware", "env-aware", "explain-aware"] known_issues: type: array items: diff --git a/benchmark/TEMPLATE.yaml b/benchmark/TEMPLATE.yaml index 249d4bf1..0291f4e4 100644 --- a/benchmark/TEMPLATE.yaml +++ b/benchmark/TEMPLATE.yaml @@ -8,6 +8,9 @@ source: "TODO: tpch | clickbench | job | synthetic | github-issues" sql: | -- TODO: paste SQL here schema_files: [] # TODO: add relative schema file paths if relevant +schema_ddl: null # TODO: inline DDL for schema-aware cases when no schema file is needed +environment: null # TODO: structured environment context for env-aware cases +context_type: sql-only # sql-only | schema-aware | env-aware | explain-aware known_issues: [] # TODO: list of {issue_id, type, detected_by_rule, severity, description} expected_rules_to_fire: [] # TODO: expert-only labeling with DBA review expected_findings_count: 0 # TODO: expected number of findings in the final report diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d001_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d001_001.yaml new file mode 100644 index 00000000..79bfa80d --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d001_001.yaml @@ -0,0 +1,19 @@ +status: validated +case_id: synthetic_expanded_d001_001 +source: synthetic +sql: SELECT count() FROM events WHERE user_id = 42 +schema_files: [] +schema_ddl: CREATE TABLE events (ts DateTime, user_id UInt64) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) PRIMARY KEY user_id ORDER BY user_id +context_type: schema-aware +known_issues: +- issue_id: I-1 + type: full_scan_on_partitioned_table + detected_by_rule: D-001 + severity: high + description: Partitioned table query does not filter the partition key. +expected_rules_to_fire: +- D-001 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-001. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d002_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d002_001.yaml new file mode 100644 index 00000000..8e6f230f --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d002_001.yaml @@ -0,0 +1,18 @@ +status: validated +case_id: synthetic_expanded_d002_001 +source: synthetic +sql: SELECT count() FROM events CROSS JOIN users +schema_files: [] +context_type: sql-only +known_issues: +- issue_id: I-1 + type: cross_join_risk + detected_by_rule: D-002 + severity: high + description: CROSS JOIN can create a large Cartesian product. +expected_rules_to_fire: +- D-002 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-002. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d005_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d005_001.yaml new file mode 100644 index 00000000..bd3dc7fb --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d005_001.yaml @@ -0,0 +1,24 @@ +status: validated +case_id: synthetic_expanded_d005_001 +source: synthetic +sql: SELECT count() FROM logs WHERE message LIKE '%timeout%' +schema_files: [] +context_type: sql-only +known_issues: +- issue_id: I-1 + type: leading_wildcard_like + detected_by_rule: D-005 + severity: medium + description: Leading wildcard LIKE cannot use ordered pruning efficiently. +- issue_id: I-2 + type: skip_index_recommendation + detected_by_rule: R-102 + severity: medium + description: Substring search may need token or ngram skip indexes. +expected_rules_to_fire: +- D-005 +- R-102 +expected_findings_count: 2 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-005 with the related R-102 advisory. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d006_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d006_001.yaml new file mode 100644 index 00000000..4044e76e --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d006_001.yaml @@ -0,0 +1,18 @@ +status: validated +case_id: synthetic_expanded_d006_001 +source: synthetic +sql: SELECT arrayJoin(tags) AS tag FROM events WHERE user_id = 42 LIMIT 10 +schema_files: [] +context_type: sql-only +known_issues: +- issue_id: I-1 + type: arrayjoin_before_filter + detected_by_rule: D-006 + severity: high + description: arrayJoin appears before WHERE and may expand rows too early. +expected_rules_to_fire: +- D-006 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-006. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d008_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d008_001.yaml new file mode 100644 index 00000000..2ff9426a --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d008_001.yaml @@ -0,0 +1,19 @@ +status: validated +case_id: synthetic_expanded_d008_001 +source: synthetic +sql: SELECT count() FROM events SAMPLE 0.1 +schema_files: [] +schema_ddl: CREATE TABLE events (id UInt64) ENGINE = MergeTree ORDER BY id +context_type: schema-aware +known_issues: +- issue_id: I-1 + type: sample_without_sample_by + detected_by_rule: D-008 + severity: low + description: Query uses SAMPLE but table DDL has no SAMPLE BY expression. +expected_rules_to_fire: +- D-008 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-008. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d009_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d009_001.yaml new file mode 100644 index 00000000..85efdf01 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d009_001.yaml @@ -0,0 +1,22 @@ +status: validated +case_id: synthetic_expanded_d009_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +schema_ddl: CREATE TABLE events (status Nullable(String)) ENGINE = MergeTree ORDER BY status +environment: + null_counts: + status: 0 +context_type: schema-aware +known_issues: +- issue_id: I-1 + type: nullable_without_need + detected_by_rule: D-009 + severity: low + description: Nullable column has no observed NULL values in supplied context. +expected_rules_to_fire: +- D-009 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-009. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d010_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d010_001.yaml new file mode 100644 index 00000000..f6c8239c --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d010_001.yaml @@ -0,0 +1,18 @@ +status: validated +case_id: synthetic_expanded_d010_001 +source: synthetic +sql: SELECT id FROM (SELECT id, payload FROM events) LIMIT 10 +schema_files: [] +context_type: sql-only +known_issues: +- issue_id: I-1 + type: unused_columns_in_select + detected_by_rule: D-010 + severity: low + description: Inner subquery projects a column unused by the outer query. +expected_rules_to_fire: +- D-010 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-010. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d011_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d011_001.yaml new file mode 100644 index 00000000..e98c55f7 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d011_001.yaml @@ -0,0 +1,30 @@ +status: validated +case_id: synthetic_expanded_d011_001 +source: synthetic +sql: SELECT count() FROM events e JOIN users u ON toUInt64(e.user_id) = u.id +schema_files: [] +context_type: sql-only +known_issues: +- issue_id: I-1 + type: implicit_type_coercion_in_join + detected_by_rule: D-011 + severity: high + description: JOIN key comparison contains an explicit cast. +- issue_id: I-2 + type: redundant_cast_on_filter_removal + detected_by_rule: R-008 + severity: medium + description: Cast around a column can block index use. +- issue_id: I-3 + type: throwing_cast_to_or_default_cast + detected_by_rule: R-020 + severity: medium + description: Throwing toType conversion on a column can interrupt the query. +expected_rules_to_fire: +- D-011 +- R-008 +- R-020 +expected_findings_count: 3 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-011; explicit JOIN cast also exercises cast advisories. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d012_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d012_001.yaml new file mode 100644 index 00000000..4883f229 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d012_001.yaml @@ -0,0 +1,18 @@ +status: validated +case_id: synthetic_expanded_d012_001 +source: synthetic +sql: SELECT row_number() OVER (ORDER BY ts) FROM events LIMIT 10 +schema_files: [] +context_type: sql-only +known_issues: +- issue_id: I-1 + type: window_function_without_partition + detected_by_rule: D-012 + severity: medium + description: Window function has no PARTITION BY clause. +expected_rules_to_fire: +- D-012 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-012. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_d013_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_d013_001.yaml new file mode 100644 index 00000000..bd569ee1 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_d013_001.yaml @@ -0,0 +1,18 @@ +status: validated +case_id: synthetic_expanded_d013_001 +source: synthetic +sql: SELECT id FROM (SELECT id FROM (SELECT id FROM (SELECT id FROM (SELECT id FROM events)))) LIMIT 10 +schema_files: [] +context_type: sql-only +known_issues: +- issue_id: I-1 + type: deeply_nested_subqueries + detected_by_rule: D-013 + severity: low + description: Query contains deeply nested subqueries. +expected_rules_to_fire: +- D-013 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for D-013. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e001_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e001_001.yaml new file mode 100644 index 00000000..a2280d9a --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e001_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e001_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + settings: + max_threads: 64 + hardware: + cpu_cores: 8 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: max_threads_vs_cpu_cores + detected_by_rule: E-001 + severity: medium + description: max_threads is high relative to CPU cores. +expected_rules_to_fire: +- E-001 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-001. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e002_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e002_001.yaml new file mode 100644 index 00000000..3e530856 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e002_001.yaml @@ -0,0 +1,24 @@ +status: validated +case_id: synthetic_expanded_e002_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + settings: + max_memory_usage: 90 + max_bytes_before_external_group_by: 60 + hardware: + ram_bytes: 100 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: max_memory_usage_vs_total_ram + detected_by_rule: E-002 + severity: high + description: max_memory_usage leaves too little RAM headroom. +expected_rules_to_fire: +- E-002 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-002. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e003_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e003_001.yaml new file mode 100644 index 00000000..0376216e --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e003_001.yaml @@ -0,0 +1,21 @@ +status: validated +case_id: synthetic_expanded_e003_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + settings: + max_memory_usage: 1000 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: external_groupby_threshold + detected_by_rule: E-003 + severity: medium + description: External aggregation threshold is absent or too high. +expected_rules_to_fire: +- E-003 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-003. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e004_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e004_001.yaml new file mode 100644 index 00000000..627d2069 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e004_001.yaml @@ -0,0 +1,22 @@ +status: validated +case_id: synthetic_expanded_e004_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + caches: + mark_cache_size: 100 + active_marks_bytes: 1000 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: mark_cache_sizing + detected_by_rule: E-004 + severity: medium + description: Mark cache is small relative to active marks. +expected_rules_to_fire: +- E-004 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-004. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e005_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e005_001.yaml new file mode 100644 index 00000000..67808620 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e005_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e005_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + workload: + repeated_reads: true + caches: + use_uncompressed_cache: false +context_type: env-aware +known_issues: +- issue_id: I-1 + type: uncompressed_cache_sizing + detected_by_rule: E-005 + severity: low + description: Repeated reads run without uncompressed cache. +expected_rules_to_fire: +- E-005 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-005. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e006_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e006_001.yaml new file mode 100644 index 00000000..446c53e7 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e006_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e006_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + hardware: + disk_type: hdd + settings: + background_pool_size: 32 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: merge_settings_for_disk_type + detected_by_rule: E-006 + severity: medium + description: Merge concurrency is high for HDD storage. +expected_rules_to_fire: +- E-006 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-006. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e007_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e007_001.yaml new file mode 100644 index 00000000..f5ec0736 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e007_001.yaml @@ -0,0 +1,21 @@ +status: validated +case_id: synthetic_expanded_e007_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + storage: + has_hot_cold_data: true +context_type: env-aware +known_issues: +- issue_id: I-1 + type: storage_policy_recommendation + detected_by_rule: E-007 + severity: low + description: Hot/cold data is present without a storage policy. +expected_rules_to_fire: +- E-007 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-007. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e008_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e008_001.yaml new file mode 100644 index 00000000..3682038b --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e008_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e008_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + workload: + complex_expressions: true + settings: + compile_expressions: false +context_type: env-aware +known_issues: +- issue_id: I-1 + type: jit_compilation_toggle + detected_by_rule: E-008 + severity: low + description: Complex expressions run without JIT compilation. +expected_rules_to_fire: +- E-008 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-008. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e009_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e009_001.yaml new file mode 100644 index 00000000..bfb7d751 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e009_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e009_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + workload: + repeated_queries: true + settings: + use_query_cache: false +context_type: env-aware +known_issues: +- issue_id: I-1 + type: query_cache_for_repeated_workload + detected_by_rule: E-009 + severity: low + description: Repeated queries run without query cache. +expected_rules_to_fire: +- E-009 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-009. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e010_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e010_001.yaml new file mode 100644 index 00000000..772aad40 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e010_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e010_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + system_metrics: + active_parts: 900 + settings: + parts_to_throw_insert: 1000 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: parts_to_throw_insert_tuning + detected_by_rule: E-010 + severity: high + description: Active part count is close to parts_to_throw_insert. +expected_rules_to_fire: +- E-010 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-010. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e011_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e011_001.yaml new file mode 100644 index 00000000..083ffa7b --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e011_001.yaml @@ -0,0 +1,24 @@ +status: validated +case_id: synthetic_expanded_e011_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + cluster: + shards: 4 + replicas: 2 + settings: + max_distributed_connections: 4 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: distributed_connections_sizing + detected_by_rule: E-011 + severity: medium + description: max_distributed_connections is below cluster fan-out. +expected_rules_to_fire: +- E-011 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-011. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e012_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e012_001.yaml new file mode 100644 index 00000000..8f836af4 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e012_001.yaml @@ -0,0 +1,21 @@ +status: validated +case_id: synthetic_expanded_e012_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + settings: + join_use_nulls: true +context_type: env-aware +known_issues: +- issue_id: I-1 + type: join_use_nulls_overhead + detected_by_rule: E-012 + severity: medium + description: join_use_nulls adds Nullable overhead to JOIN output. +expected_rules_to_fire: +- E-012 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-012. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e013_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e013_001.yaml new file mode 100644 index 00000000..f0b04508 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e013_001.yaml @@ -0,0 +1,22 @@ +status: validated +case_id: synthetic_expanded_e013_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + settings: + async_insert: true + async_insert_busy_timeout_ms: 10 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: async_insert_busy_timeout_tuning + detected_by_rule: E-013 + severity: low + description: async_insert busy timeout is too low for batching. +expected_rules_to_fire: +- E-013 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-013. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e014_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e014_001.yaml new file mode 100644 index 00000000..38807c78 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e014_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e014_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + workload: + large_parts: true + settings: + max_bytes_to_merge_at_max_space_in_pool: 0 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: merge_max_bytes_for_level + detected_by_rule: E-014 + severity: medium + description: Large-parts workload lacks explicit merge max bytes tuning. +expected_rules_to_fire: +- E-014 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-014. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e015_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e015_001.yaml new file mode 100644 index 00000000..b4b09576 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e015_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e015_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + workload: + group_by_order_key_prefix: true + settings: + optimize_aggregation_in_order: false +context_type: env-aware +known_issues: +- issue_id: I-1 + type: optimize_aggregation_in_order + detected_by_rule: E-015 + severity: medium + description: Workload can use aggregation in order but the setting is disabled. +expected_rules_to_fire: +- E-015 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-015. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e016_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e016_001.yaml new file mode 100644 index 00000000..bac86076 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e016_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e016_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + workload: + large_join: true + settings: + join_algorithm: hash +context_type: env-aware +known_issues: +- issue_id: I-1 + type: join_algorithm_for_large_tables + detected_by_rule: E-016 + severity: high + description: Large JOIN workload uses a non-adaptive join algorithm. +expected_rules_to_fire: +- E-016 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-016. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e017_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e017_001.yaml new file mode 100644 index 00000000..1fb0d345 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e017_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e017_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + workload: + distributed_aggregation: true + settings: + distributed_aggregation_memory_efficient: false +context_type: env-aware +known_issues: +- issue_id: I-1 + type: distributed_aggregation_memory_efficient + detected_by_rule: E-017 + severity: medium + description: Distributed aggregation runs without memory efficient mode. +expected_rules_to_fire: +- E-017 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-017. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e018_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e018_001.yaml new file mode 100644 index 00000000..5b37a645 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e018_001.yaml @@ -0,0 +1,22 @@ +status: validated +case_id: synthetic_expanded_e018_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + workload: + bulk_inserts: true + insert_format: JSONEachRow +context_type: env-aware +known_issues: +- issue_id: I-1 + type: input_format_for_bulk_inserts + detected_by_rule: E-018 + severity: medium + description: Bulk insert workload uses a less efficient text format. +expected_rules_to_fire: +- E-018 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-018. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e019_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e019_001.yaml new file mode 100644 index 00000000..3b502a9c --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e019_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e019_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + cluster: + has_local_replica: true + settings: + prefer_localhost_replica: false +context_type: env-aware +known_issues: +- issue_id: I-1 + type: prefer_localhost_replica + detected_by_rule: E-019 + severity: low + description: Local replica exists but prefer_localhost_replica is disabled. +expected_rules_to_fire: +- E-019 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-019. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_e020_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_e020_001.yaml new file mode 100644 index 00000000..0e5f29f9 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_e020_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_e020_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + workload: + interactive_queries: true + settings: + max_execution_time: 0 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: max_execution_time_for_protection + detected_by_rule: E-020 + severity: medium + description: Interactive workload has no max_execution_time guardrail. +expected_rules_to_fire: +- E-020 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for E-020. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r101_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r101_001.yaml new file mode 100644 index 00000000..36309b06 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r101_001.yaml @@ -0,0 +1,19 @@ +status: validated +case_id: synthetic_expanded_r101_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +schema_ddl: CREATE TABLE events (id UInt64) ENGINE = MergeTree ORDER BY tuple() +context_type: schema-aware +known_issues: +- issue_id: I-1 + type: order_by_key_redesign + detected_by_rule: R-101 + severity: high + description: MergeTree table uses ORDER BY tuple(). +expected_rules_to_fire: +- R-101 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-101. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r102_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r102_001.yaml new file mode 100644 index 00000000..eaf9b887 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r102_001.yaml @@ -0,0 +1,24 @@ +status: validated +case_id: synthetic_expanded_r102_001 +source: synthetic +sql: SELECT count() FROM logs WHERE message LIKE '%timeout%' +schema_files: [] +context_type: sql-only +known_issues: +- issue_id: I-1 + type: leading_wildcard_like + detected_by_rule: D-005 + severity: medium + description: Leading wildcard LIKE cannot use ordered pruning efficiently. +- issue_id: I-2 + type: skip_index_recommendation + detected_by_rule: R-102 + severity: medium + description: Substring search may need token or ngram skip indexes. +expected_rules_to_fire: +- D-005 +- R-102 +expected_findings_count: 2 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-102. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r103_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r103_001.yaml new file mode 100644 index 00000000..44bb4c21 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r103_001.yaml @@ -0,0 +1,21 @@ +status: validated +case_id: synthetic_expanded_r103_001 +source: synthetic +sql: SELECT user_id, count() FROM events GROUP BY user_id +schema_files: [] +environment: + workload: + repeated_group_by: true +context_type: env-aware +known_issues: +- issue_id: I-1 + type: projection_recommendation + detected_by_rule: R-103 + severity: medium + description: Repeated GROUP BY query shape may benefit from a projection. +expected_rules_to_fire: +- R-103 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-103. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r104_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r104_001.yaml new file mode 100644 index 00000000..221ff1e0 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r104_001.yaml @@ -0,0 +1,19 @@ +status: validated +case_id: synthetic_expanded_r104_001 +source: synthetic +sql: SELECT count() FROM events WHERE user_id = 42 +schema_files: [] +schema_ddl: CREATE TABLE events (user_id UInt64, ts DateTime) ENGINE = MergeTree ORDER BY (user_id, ts) +context_type: schema-aware +known_issues: +- issue_id: I-1 + type: primary_key_refinement + detected_by_rule: R-104 + severity: medium + description: Table has ORDER BY but no explicit PRIMARY KEY for selective filters. +expected_rules_to_fire: +- R-104 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-104. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r105_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r105_001.yaml new file mode 100644 index 00000000..e52e49d7 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r105_001.yaml @@ -0,0 +1,22 @@ +status: validated +case_id: synthetic_expanded_r105_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +environment: + system_metrics: + active_parts: 1000 + active_partitions: 5 +context_type: env-aware +known_issues: +- issue_id: I-1 + type: partition_strategy_adjustment + detected_by_rule: R-105 + severity: high + description: Table has too many active parts per partition. +expected_rules_to_fire: +- R-105 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-105. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r106_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r106_001.yaml new file mode 100644 index 00000000..84395b34 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r106_001.yaml @@ -0,0 +1,19 @@ +status: validated +case_id: synthetic_expanded_r106_001 +source: synthetic +sql: SELECT country FROM events LIMIT 10 +schema_files: [] +schema_ddl: CREATE TABLE events (country String, id UInt32) ENGINE = MergeTree ORDER BY id +context_type: schema-aware +known_issues: +- issue_id: I-1 + type: lowcardinality_conversion + detected_by_rule: R-106 + severity: medium + description: Low-cardinality String column may benefit from LowCardinality. +expected_rules_to_fire: +- R-106 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-106. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r107_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r107_001.yaml new file mode 100644 index 00000000..ab6d33ad --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r107_001.yaml @@ -0,0 +1,22 @@ +status: validated +case_id: synthetic_expanded_r107_001 +source: synthetic +sql: SELECT count() FROM events +schema_files: [] +schema_ddl: CREATE TABLE events (ts DateTime, value UInt64) ENGINE = MergeTree ORDER BY ts +environment: + workload: + compression_sensitive: true +context_type: schema-aware +known_issues: +- issue_id: I-1 + type: codec_recommendation + detected_by_rule: R-107 + severity: low + description: Compressible columns have no explicit CODEC. +expected_rules_to_fire: +- R-107 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-107. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r108_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r108_001.yaml new file mode 100644 index 00000000..0ea0e3d9 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r108_001.yaml @@ -0,0 +1,27 @@ +status: validated +case_id: synthetic_expanded_r108_001 +source: synthetic +sql: SELECT * FROM events WHERE user_id = 42 LIMIT 10 +schema_files: [] +environment: + workload: + wide_table: true +context_type: sql-only +known_issues: +- issue_id: I-1 + type: select_star_on_wide_table + detected_by_rule: D-003 + severity: medium + description: SELECT * reads all columns. +- issue_id: I-2 + type: prewhere_injection + detected_by_rule: R-108 + severity: medium + description: Wide table query filters in WHERE only. +expected_rules_to_fire: +- D-003 +- R-108 +expected_findings_count: 2 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-108 with existing D-003 overlap. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r109_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r109_001.yaml new file mode 100644 index 00000000..1bbfe446 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r109_001.yaml @@ -0,0 +1,21 @@ +status: validated +case_id: synthetic_expanded_r109_001 +source: synthetic +sql: SELECT count() FROM small s JOIN large l ON s.id = l.id +schema_files: [] +environment: + workload: + right_table_larger_than_left: true +context_type: env-aware +known_issues: +- issue_id: I-1 + type: join_reorder_small_on_right + detected_by_rule: R-109 + severity: medium + description: JOIN appears to put the larger table on the right side. +expected_rules_to_fire: +- R-109 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-109. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r110_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r110_001.yaml new file mode 100644 index 00000000..fbc7b51b --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r110_001.yaml @@ -0,0 +1,23 @@ +status: validated +case_id: synthetic_expanded_r110_001 +source: synthetic +sql: SELECT count() FROM events e JOIN users u ON e.user_id = u.id +schema_files: [] +environment: + workload: + large_join: true + settings: + join_algorithm: auto +context_type: env-aware +known_issues: +- issue_id: I-1 + type: join_algorithm_choice + detected_by_rule: R-110 + severity: medium + description: Large JOIN needs explicit algorithm review. +expected_rules_to_fire: +- R-110 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-110. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r111_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r111_001.yaml new file mode 100644 index 00000000..6e7aaf39 --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r111_001.yaml @@ -0,0 +1,21 @@ +status: validated +case_id: synthetic_expanded_r111_001 +source: synthetic +sql: SELECT user_id, count() FROM events GROUP BY user_id +schema_files: [] +environment: + workload: + repeated_expensive_query: true +context_type: env-aware +known_issues: +- issue_id: I-1 + type: materialized_view_recommendation + detected_by_rule: R-111 + severity: medium + description: Repeated expensive aggregation may benefit from a materialized view. +expected_rules_to_fire: +- R-111 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-111. diff --git a/benchmark/cases/synthetic_expanded/synthetic_expanded_r112_001.yaml b/benchmark/cases/synthetic_expanded/synthetic_expanded_r112_001.yaml new file mode 100644 index 00000000..a818b90e --- /dev/null +++ b/benchmark/cases/synthetic_expanded/synthetic_expanded_r112_001.yaml @@ -0,0 +1,21 @@ +status: validated +case_id: synthetic_expanded_r112_001 +source: synthetic +sql: SELECT user_id, count() FROM events GROUP BY user_id +schema_files: [] +environment: + workload: + high_cardinality_group_by: true +context_type: env-aware +known_issues: +- issue_id: I-1 + type: groupby_multipass_split + detected_by_rule: R-112 + severity: medium + description: High-cardinality GROUP BY may need staged aggregation. +expected_rules_to_fire: +- R-112 +expected_findings_count: 1 +expected_improvement: null +synthetic_explain_path: null +notes: Synthetic positive case for R-112. diff --git a/clickadvisor/cli/main.py b/clickadvisor/cli/main.py index 2fb6ba01..685eaa2b 100644 --- a/clickadvisor/cli/main.py +++ b/clickadvisor/cli/main.py @@ -1,5 +1,6 @@ from __future__ import annotations +import json from pathlib import Path from typing import Annotated @@ -32,6 +33,18 @@ def _read_optional(path: Path | None) -> str | None: return path.read_text(encoding="utf-8") +def _read_environment(path: Path | None) -> dict[str, object] | None: + if path is None: + return None + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as error: + raise typer.BadParameter(f"invalid environment JSON: {error}") from error + if not isinstance(payload, dict): + raise typer.BadParameter("environment JSON must be an object") + return payload + + @app.command() def version() -> None: console.print(f"ClickAdvisor {__version__}") @@ -50,6 +63,7 @@ def analyze( sql: Annotated[Path, typer.Option(help="SQL файл")], explain: Annotated[Path | None, typer.Option()] = None, schema: Annotated[Path | None, typer.Option()] = None, + environment: Annotated[Path | None, typer.Option(help="Environment JSON file")] = None, connect: Annotated[str | None, typer.Option(help="clickhouse://host:9000")] = None, ch_version: Annotated[str | None, typer.Option(help="24.3")] = None, ch_user: Annotated[str, typer.Option(help="ClickHouse user")] = "default", @@ -82,6 +96,7 @@ def analyze( sql_text = sql.read_text(encoding="utf-8") explain_text = _read_optional(explain) schema_text = _read_optional(schema) + environment_data = _read_environment(environment) version = ch_version if connect and not ch_version: @@ -92,6 +107,7 @@ def analyze( explain_output=explain_text, schema_ddl=schema_text, ch_version=version, + environment=environment_data, ) rules = get_all_rules() diff --git a/clickadvisor/core/models.py b/clickadvisor/core/models.py index 33a5f638..f07f486f 100644 --- a/clickadvisor/core/models.py +++ b/clickadvisor/core/models.py @@ -1,7 +1,7 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import Optional +from typing import Any, Optional @dataclass(slots=True) @@ -10,6 +10,7 @@ class QueryContext: explain_output: Optional[str] = None schema_ddl: Optional[str] = None ch_version: Optional[str] = None + environment: dict[str, Any] | None = None @dataclass(slots=True) diff --git a/clickadvisor/rules/detectors/__init__.py b/clickadvisor/rules/detectors/__init__.py index f92ccf42..840b546b 100644 --- a/clickadvisor/rules/detectors/__init__.py +++ b/clickadvisor/rules/detectors/__init__.py @@ -1,3 +1,15 @@ +from clickadvisor.rules.detectors.backlog import ( + D001FullScanOnPartitionedTable, + D002CrossJoinRisk, + D005LeadingWildcardLike, + D006ArrayJoinBeforeFilter, + D008SampleWithoutSampleBy, + D009NullableWithoutNeed, + D010UnusedColumnsInSelect, + D011ImplicitTypeCoercionInJoin, + D012WindowFunctionWithoutPartition, + D013DeeplyNestedSubqueries, +) from clickadvisor.rules.detectors.D003_select_star import D003SelectStar from clickadvisor.rules.detectors.D004_missing_limit import D004MissingLimit from clickadvisor.rules.detectors.D007_final_modifier import D007FinalModifier @@ -20,6 +32,16 @@ __all__ = [ "D003SelectStar", + "D001FullScanOnPartitionedTable", + "D002CrossJoinRisk", + "D005LeadingWildcardLike", + "D006ArrayJoinBeforeFilter", + "D008SampleWithoutSampleBy", + "D009NullableWithoutNeed", + "D010UnusedColumnsInSelect", + "D011ImplicitTypeCoercionInJoin", + "D012WindowFunctionWithoutPartition", + "D013DeeplyNestedSubqueries", "D004MissingLimit", "D007FinalModifier", "D014AsyncInsertNoWait", diff --git a/clickadvisor/rules/detectors/backlog.py b/clickadvisor/rules/detectors/backlog.py new file mode 100644 index 00000000..01426935 --- /dev/null +++ b/clickadvisor/rules/detectors/backlog.py @@ -0,0 +1,246 @@ +from __future__ import annotations + +import re + +from clickadvisor.core.models import Finding, QueryContext +from clickadvisor.rules.base import Rule + + +def _finding( + rule: Rule, + severity: str, + description: str, + suggestion: str, + *, + confidence: str = "advisory", +) -> Finding: + return Finding( + rule_id=rule.rule_id, + rule_name=rule.name, + tier=rule.tier, + severity=severity, + description=description, + suggestion=suggestion, + confidence=confidence, + ch_version_introduced=rule.ch_version_introduced, + ) + + +def _schema_text(context: QueryContext) -> str: + return context.schema_ddl or context.sql + + +class D001FullScanOnPartitionedTable(Rule): + rule_id = "D-001" + name = "full_scan_on_partitioned_table" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + schema = context.schema_ddl + if not schema or "PARTITION BY" not in schema.upper(): + return None + match = re.search(r"\bPARTITION\s+BY\s+(.+?)(?:\s+ORDER\s+BY|\s+PRIMARY\s+KEY|\s+SETTINGS|$)", schema, re.I | re.S) + if not match: + return None + partition_expr = match.group(1).strip() + identifiers = {item.lower() for item in re.findall(r"[A-Za-z_][A-Za-z0-9_]*", partition_expr)} + functions = {"toyyyymm", "todate", "tostartofmonth", "tostartofday", "tostartofyear"} + partition_cols = identifiers - functions + where_match = re.search(r"\bWHERE\b(.+?)(?:\bGROUP\b|\bORDER\b|\bLIMIT\b|$)", context.sql, re.I | re.S) + where_text = where_match.group(1).lower() if where_match else "" + if partition_cols and any(col in where_text for col in partition_cols): + return None + return _finding( + self, + "high", + f"Таблица партиционирована по {partition_expr}, но запрос не фильтрует partition key.", + "Добавьте фильтр по partition-колонке или временной функции партиционирования.", + ) + + +class D002CrossJoinRisk(Rule): + rule_id = "D-002" + name = "cross_join_risk" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + if re.search(r"\bCROSS\s+JOIN\b", sql, re.I): + return _finding(self, "high", "Запрос содержит CROSS JOIN.", "Проверьте, что декартово произведение намеренное.") + for match in re.finditer(r"\bJOIN\b(?P
.*?)(?=\bJOIN\b|\bWHERE\b|\bGROUP\b|\bORDER\b|\bLIMIT\b|$)", sql, re.I | re.S): + body = match.group("body") + on_match = re.search(r"\bON\b(.+)", body, re.I | re.S) + if not on_match or "=" not in on_match.group(1): + return _finding(self, "high", "JOIN не содержит equality-условия.", "Добавьте ON с equality-ключами соединения.") + return None + + +class D005LeadingWildcardLike(Rule): + rule_id = "D-005" + name = "leading_wildcard_like" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + if not re.search(r"\bLIKE\s+['\"]%", context.sql, re.I): + return None + return _finding( + self, + "medium", + "LIKE с leading wildcard не может эффективно использовать индекс.", + "Рассмотрите ngram/token skip-index или другой способ поиска по подстроке.", + ) + + +class D006ArrayJoinBeforeFilter(Rule): + rule_id = "D-006" + name = "arrayjoin_before_filter" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + sql = context.sql + array_pos = sql.lower().find("arrayjoin") + where_pos = sql.lower().find("where") + if array_pos < 0 or where_pos < 0 or array_pos > where_pos: + return None + return _finding( + self, + "high", + "arrayJoin встречается до WHERE и может резко увеличить промежуточную кардинальность.", + "Отфильтруйте строки до arrayJoin или вынесите arrayJoin в более поздний этап.", + ) + + +class D008SampleWithoutSampleBy(Rule): + rule_id = "D-008" + name = "sample_without_sample_by" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + if not re.search(r"\bSAMPLE\b", context.sql, re.I): + return None + if context.schema_ddl and re.search(r"\bSAMPLE\s+BY\b", context.schema_ddl, re.I): + return None + return _finding( + self, + "low", + "Запрос использует SAMPLE, но schema DDL не содержит SAMPLE BY.", + "Добавьте SAMPLE BY в MergeTree DDL или используйте явную sampling-логику.", + ) + + +class D009NullableWithoutNeed(Rule): + rule_id = "D-009" + name = "nullable_without_need" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + text = _schema_text(context) + match = re.search(r"\b([A-Za-z_][A-Za-z0-9_]*)\s+Nullable\s*\(", text, re.I) + if not match: + return None + env = context.environment or {} + null_counts = env.get("null_counts") + if isinstance(null_counts, dict) and null_counts.get(match.group(1), 1) != 0: + return None + return _finding( + self, + "low", + f"Колонка {match.group(1)} объявлена Nullable; это добавляет NULL bitmap.", + "Если NULL-значений нет, рассмотрите non-nullable тип.", + ) + + +class D010UnusedColumnsInSelect(Rule): + rule_id = "D-010" + name = "unused_columns_in_select" + tier = "detector" + ch_version_introduced = "1.0" + + def check(self, context: QueryContext) -> Finding | None: + match = re.search( + r"SELECT\s+(?P