Skip to content

feat(sql): extract table-to-table lineage from INSERT/CTAS/MERGE/UPDATE (#1572)#1777

Open
rithyKabir wants to merge 1 commit into
Graphify-Labs:v8from
rithyKabir:feat/1572-sql-dml-lineage
Open

feat(sql): extract table-to-table lineage from INSERT/CTAS/MERGE/UPDATE (#1572)#1777
rithyKabir wants to merge 1 commit into
Graphify-Labs:v8from
rithyKabir:feat/1572-sql-dml-lineage

Conversation

@rithyKabir

Copy link
Copy Markdown
Contributor

Closes #1572.

Problem

Dataflow edges were only walked inside create_view / create_function / create_procedure, so ETL corpora — mostly INSERT INTO … SELECT bodies — produced schema structure but essentially zero source→target lineage (the issue measured ~16% of 700 files yielding any node and no lineage edges).

What this adds

extract_sql now emits <target> --reads_from--> <source> edges, reusing the existing _walk_from_refs traversal, for all four requested statement types:

  • INSERT INTO tgt … SELECT … FROM/JOIN src — the target node is materialized (so plain VALUES inserts now at least produce a node, per the issue's "no target node" complaint), and sources come from the nested from/join walk. VALUES inserts have no from node, so they can't create false lineage.
  • CREATE TABLE tgt AS SELECT … (CTAS) — the SELECT lives in a create_query child of create_table; walked exactly like create_view already is.
  • MERGE INTO tgt USING src — tree-sitter-sql has no dedicated merge node: the keywords sit directly under statement, so this is handled at the statement loop. Only direct children are scanned for the target/source object_references (nested ON-clause fields never surface there), and USING (SELECT … FROM src) subqueries are covered by the nested from walk.
  • UPDATE tgt SET … FROM src — the target is the relation directly under update, so _walk_from_refs (which only looks under from/join) never sees the target itself.

_walk_from_refs now skips self-loops: INSERT INTO t SELECT … FROM t (dedup/backfill patterns) reads its own target and previously would have emitted a t → t edge.

Also includes the issue's nice-to-have: .ddl is registered as a SQL extension — dispatch table, CODE_EXTENSIONS, the case-insensitive-identifier set, and the #1745 missing-extra warning map.

Tests

8 new tests over a realistic ETL fixture (tests/fixtures/sample_dml_lineage.sql) covering each statement type plus the two negative cases (no false lineage from VALUES, no self-loop from self-referential backfill) and a no-dangling-edges check. tests/test_multilang.py 56/56; test_extract.py + test_pg_introspect.py green; ruff clean. Full-suite failures on my machine (4× test_ollama_retry_cap, 1 order-flaky labeling test) reproduce identically on clean v8 — pre-existing and unrelated.

The remaining nice-to-have from the issue (normalizing templated identifiers like {project}.{dataset}.tbl) is not included — it needs a design decision about placeholder handling and feels like its own change.

🤖 Generated with Claude Code

…TE (Graphify-Labs#1572)

Dataflow edges were only walked inside create_view/create_function/
create_procedure, so ETL corpora (mostly INSERT INTO ... SELECT bodies)
produced schema structure but essentially zero source->target lineage.

extract_sql now emits <target> --reads_from--> <source> edges, reusing
the existing _walk_from_refs traversal, for:

- INSERT INTO tgt ... SELECT ... FROM/JOIN src  (the target node is
  materialized, so plain VALUES inserts now at least produce a node)
- CREATE TABLE tgt AS SELECT ... (CTAS; the SELECT lives in a
  create_query child)
- MERGE INTO tgt USING src (tree-sitter-sql has no merge node - the
  keywords sit directly under statement, so it is handled there; USING
  (SELECT ...) subqueries are covered via the nested from walk)
- UPDATE tgt SET ... FROM src

_walk_from_refs now skips self-loops (INSERT INTO t SELECT ... FROM t
backfill patterns read their own target).

Also registers .ddl as a SQL extension (dispatch, CODE_EXTENSIONS,
case-insensitive set, and the Graphify-Labs#1745 extras map), per the issue's
nice-to-have.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: extract table-to-table dependencies from INSERT/CTAS/MERGE, not just CREATE VIEW

1 participant