JSON AST with properties

[peter-leonov-ch]

AST as JSON draft. Covers basic cases.

Contributing

Ask LLM to load context from AST.md in the repo root and move from there. The rest is already set up by the upstream repo.

Design choices

• the node type aware serializer is a downcast switch-case chain with a default fallback (as opposed to implementing overload methods): this allows to cover non-intrusively all the core nodes and leave the AST code alone for now
• the AST itself is object bases (as opposed to positional s-expressions): this is for readability and semantic meaning per-se (no need to introduce wrapper nodes for if/else blocks etc), the main downside is that the traverser has to be aware of all the "children" slots to be able to iterate on the tree

Format

The top-level value is a versioned document: { "version": 1, "ast": { ... } }. version (AST_JSON_FORMAT_VERSION) is bumped on any backwards-incompatible change so external consumers (e.g. clickhouse-js-parser) can pin fixtures and detect breaks.

Each node has a type; sub-nodes appear under named slots; children survives only on genuinely homogeneous lists (ExpressionList, TablesInSelectQuery). Absent optional slots are omitted (no nulls). 64-bit integer literal values (and settings values) are emitted as JSON strings — values above 2^53 are corrupted by JavaScript's JSON.parse; value_type says how to read them. See AST.md for the full schema, the enum value sets, the fallback stringification, and known source-fidelity divergences.

Examples

Run with:

clickhouse local --format TSVRaw -q "EXPLAIN AST json = 1 <query>"

--format TSVRaw is what gives the clean multi-line JSON (otherwise the result is a single TSV cell with newlines escaped as \n). The query only needs to parse — referenced tables/columns need not exist.

A few patterns visible across all of them: operators are ordinary functions (>= → greaterOrEquals, = → equals, is_operator: true); absent clauses are simply omitted; integer literal values are strings; and children survives only on TablesInSelectQuery here — everything else is a named slot.

SELECT * FROM foo WHERE x = 1

{
  "version": 1,
  "ast": {
    "type": "SelectWithUnionQuery",
    "selects": [
      {
        "type": "SelectQuery",
        "select": [
          { "type": "Asterisk" }
        ],
        "from": {
          "type": "TablesInSelectQuery",
          "children": [
            {
              "type": "TablesInSelectQueryElement",
              "table_expression": {
                "type": "TableExpression",
                "database_and_table_name": { "type": "TableIdentifier", "name": "foo" }
              }
            }
          ]
        },
        "where": {
          "type": "Function",
          "name": "equals",
          "is_operator": true,
          "arguments": [
            { "type": "Identifier", "name": "x" },
            { "type": "Literal", "value_type": "UInt64", "value": "1" }
          ]
        }
      }
    ]
  }
}

SELECT name, count() AS c FROM users WHERE age >= 18 GROUP BY name ORDER BY c DESC LIMIT 10

Every clause is a named slot on SelectQuery; the aggregate carries its alias.

{
  "version": 1,
  "ast": {
    "type": "SelectWithUnionQuery",
    "selects": [
      {
        "type": "SelectQuery",
        "select": [
          { "type": "Identifier", "name": "name" },
          { "type": "Function", "alias": "c", "name": "count", "arguments": [] }
        ],
        "from": {
          "type": "TablesInSelectQuery",
          "children": [
            {
              "type": "TablesInSelectQueryElement",
              "table_expression": {
                "type": "TableExpression",
                "database_and_table_name": { "type": "TableIdentifier", "name": "users" }
              }
            }
          ]
        },
        "where": {
          "type": "Function",
          "name": "greaterOrEquals",
          "is_operator": true,
          "arguments": [
            { "type": "Identifier", "name": "age" },
            { "type": "Literal", "value_type": "UInt64", "value": "18" }
          ]
        },
        "group_by": [
          { "type": "Identifier", "name": "name" }
        ],
        "order_by": [
          {
            "type": "OrderByElement",
            "expression": { "type": "Identifier", "name": "c" },
            "direction": "DESC"
          }
        ],
        "limit": { "type": "Literal", "value_type": "UInt64", "value": "10" }
      }
    ]
  }
}

SELECT u.name, o.total FROM users AS u INNER JOIN orders AS o ON u.id = o.user_id

The from (TablesInSelectQuery) list holds the base table plus a JOIN element carrying both table_join and table_expression; compound identifiers expose name_parts, and table aliases ride on the identifier as alias.

{
  "version": 1,
  "ast": {
    "type": "SelectWithUnionQuery",
    "selects": [
      {
        "type": "SelectQuery",
        "select": [
          { "type": "Identifier", "name": "u.name", "name_parts": ["u", "name"] },
          { "type": "Identifier", "name": "o.total", "name_parts": ["o", "total"] }
        ],
        "from": {
          "type": "TablesInSelectQuery",
          "children": [
            {
              "type": "TablesInSelectQueryElement",
              "table_expression": {
                "type": "TableExpression",
                "database_and_table_name": { "type": "TableIdentifier", "alias": "u", "name": "users" }
              }
            },
            {
              "type": "TablesInSelectQueryElement",
              "table_join": {
                "type": "TableJoin",
                "kind": "INNER",
                "on": {
                  "type": "Function",
                  "name": "equals",
                  "is_operator": true,
                  "arguments": [
                    { "type": "Identifier", "name": "u.id", "name_parts": ["u", "id"] },
                    { "type": "Identifier", "name": "o.user_id", "name_parts": ["o", "user_id"] }
                  ]
                }
              },
              "table_expression": {
                "type": "TableExpression",
                "database_and_table_name": { "type": "TableIdentifier", "alias": "o", "name": "orders" }
              }
            }
          ]
        }
      }
    ]
  }
}

INSERT INTO events (id, ts) VALUES (1, now())

The inline row data is not part of the parsed AST; the column list is inlined and the format is exposed.

{
  "version": 1,
  "ast": {
    "type": "InsertQuery",
    "table": { "type": "Identifier", "name": "events" },
    "columns": [
      { "type": "Identifier", "name": "id" },
      { "type": "Identifier", "name": "ts" }
    ],
    "format": "Values"
  }
}

CREATE TABLE hits (id UInt64, url String) ENGINE = MergeTree ORDER BY id

Each ColumnDeclaration exposes its data_type (a DataType with the real type name), and the engine clause becomes storage.

{
  "version": 1,
  "ast": {
    "type": "CreateQuery",
    "table": { "type": "Identifier", "name": "hits" },
    "columns_list": {
      "type": "Columns",
      "columns": [
        { "type": "ColumnDeclaration", "name": "id", "data_type": { "type": "DataType", "name": "UInt64" } },
        { "type": "ColumnDeclaration", "name": "url", "data_type": { "type": "DataType", "name": "String" } }
      ]
    },
    "storage": {
      "type": "Storage",
      "engine": { "type": "Function", "name": "MergeTree", "kind": "TABLE_ENGINE", "arguments": [] },
      "order_by": { "type": "Identifier", "name": "id" }
    }
  }
}

Changelog category

• New Feature

Changelog entry

Added EXPLAIN AST json = 1 which serializes the parsed AST of a query to JSON as a versioned document { "version": 1, "ast": ... }. Each node exposes its sub-nodes through named slots (e.g. arguments, where, table_expression) instead of a positional children array, which is kept only for homogeneous lists. 64-bit integer values are emitted as strings to stay safe for JSON consumers.

[Copilot]

Pull request overview

Adds a new EXPLAIN AST json = 1 mode that serializes the parsed ClickHouse AST into a structured JSON format (named slots for semantic children, children kept for homogeneous lists), along with documentation, stateless golden tests, and a standalone fixture corpus for external parser/serializer validation.

Changes:

• Implement formatASTAsJSON() (JSONBuilder-based AST serializer) and integrate it into EXPLAIN AST via a new json option.
• Add stateless tests covering core AST node families (functions, ORDER BY, SELECT clauses, wrappers, UNION/INTERSECT/EXCEPT, leaf state).
• Add AST.md spec plus a separate tests/ast_json_fixtures/ corpus + generator script.

Reviewed changes

Copilot reviewed 104 out of 104 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
AST.md	Documents the JSON AST contract, schema, and extension/testing guidance.
src/Interpreters/InterpreterExplainQuery.cpp	Adds json option to EXPLAIN AST and routes output through formatASTAsJSON().
src/Parsers/DumpASTNode.h	Declares formatASTAsJSON() and documents JSON AST conventions.
src/Parsers/DumpASTNode.cpp	Implements JSON AST formatting and per-node enrichment logic.
tests/queries/0_stateless/03917_explain_ast_json_function.sh	Stateless test coverage for Function node slots.
tests/queries/0_stateless/03917_explain_ast_json_function.reference	Golden output for function JSON AST extraction.
tests/queries/0_stateless/03918_explain_ast_json_order_by.sh	Stateless test coverage for OrderByElement slots.
tests/queries/0_stateless/03918_explain_ast_json_order_by.reference	Golden output for ORDER BY JSON AST extraction.
tests/queries/0_stateless/03919_explain_ast_json_select_query.sh	Stateless test coverage for SelectQuery clause slots and ordering.
tests/queries/0_stateless/03919_explain_ast_json_select_query.reference	Golden output for SELECT clause slot behavior.
tests/queries/0_stateless/03920_explain_ast_json_expressions.sh	Stateless test coverage for literals/operators/lambda/cast/nulls/identifiers.
tests/queries/0_stateless/03920_explain_ast_json_expressions.reference	Golden output for expression JSON AST extraction.
tests/queries/0_stateless/03921_explain_ast_json_union.sh	Stateless test coverage for UNION modes and INTERSECT/EXCEPT operator+operands.
tests/queries/0_stateless/03921_explain_ast_json_union.reference	Golden output for set-ops JSON AST extraction.
tests/queries/0_stateless/03922_explain_ast_json_showcase.sh	Full AST JSON “showcase” query (no filtering).
tests/queries/0_stateless/03922_explain_ast_json_showcase.reference	Golden pretty-printed JSON AST for showcase query.
tests/queries/0_stateless/03923_explain_ast_json_wrappers.sh	Stateless test coverage for structural wrapper nodes (JOIN, subquery, window, interpolate, etc.).
tests/queries/0_stateless/03923_explain_ast_json_wrappers.reference	Golden output for wrapper-node JSON AST extraction.
tests/queries/0_stateless/03924_explain_ast_json_leaf_state.sh	Stateless test coverage for leaf-ish node state (SETTINGS, SAMPLE, asterisks, COLUMNS).
tests/queries/0_stateless/03924_explain_ast_json_leaf_state.reference	Golden output for leaf-state JSON AST extraction.
tests/ast_json_fixtures/README.md	Describes the standalone SQL→JSON-AST fixture corpus and regeneration/verification workflow.
tests/ast_json_fixtures/generate.sh	Script to regenerate the fixture corpus via EXPLAIN AST json = 1.
tests/ast_json_fixtures/cases/01_literals.sql	Fixture input: literals.
tests/ast_json_fixtures/cases/01_literals.json	Fixture output: literals AST JSON.
tests/ast_json_fixtures/cases/02_identifiers.sql	Fixture input: identifiers.
tests/ast_json_fixtures/cases/02_identifiers.json	Fixture output: identifiers AST JSON.
tests/ast_json_fixtures/cases/03_function_call.sql	Fixture input: function call.
tests/ast_json_fixtures/cases/03_function_call.json	Fixture output: function call AST JSON.
tests/ast_json_fixtures/cases/04_no_args.sql	Fixture input: zero-arg function.
tests/ast_json_fixtures/cases/04_no_args.json	Fixture output: zero-arg function AST JSON.
tests/ast_json_fixtures/cases/05_operators.sql	Fixture input: operators.
tests/ast_json_fixtures/cases/05_operators.json	Fixture output: operators normalized to functions.
tests/ast_json_fixtures/cases/06_subscript.sql	Fixture input: subscripts/tuple element access.
tests/ast_json_fixtures/cases/06_subscript.json	Fixture output: subscript AST JSON.
tests/ast_json_fixtures/cases/07_cast.sql	Fixture input: CAST forms.
tests/ast_json_fixtures/cases/07_cast.json	Fixture output: CAST AST JSON.
tests/ast_json_fixtures/cases/08_lambda.sql	Fixture input: lambda expression.
tests/ast_json_fixtures/cases/08_lambda.json	Fixture output: lambda AST JSON.
tests/ast_json_fixtures/cases/09_parametric_agg.sql	Fixture input: parametric aggregate.
tests/ast_json_fixtures/cases/09_parametric_agg.json	Fixture output: parametric aggregate AST JSON.
tests/ast_json_fixtures/cases/10_nulls_action.sql	Fixture input: NULLS action.
tests/ast_json_fixtures/cases/10_nulls_action.json	Fixture output: NULLS action AST JSON.
tests/ast_json_fixtures/cases/11_window_named.sql	Fixture input: named window.
tests/ast_json_fixtures/cases/11_window_named.json	Fixture output: named window AST JSON.
tests/ast_json_fixtures/cases/12_window_inline.sql	Fixture input: inline window definition + frame.
tests/ast_json_fixtures/cases/12_window_inline.json	Fixture output: inline window AST JSON.
tests/ast_json_fixtures/cases/13_select_star.sql	Fixture input: SELECT * with WHERE.
tests/ast_json_fixtures/cases/13_select_star.json	Fixture output: SELECT * AST JSON.
tests/ast_json_fixtures/cases/14_distinct.sql	Fixture input: DISTINCT.
tests/ast_json_fixtures/cases/14_distinct.json	Fixture output: DISTINCT AST JSON.
tests/ast_json_fixtures/cases/15_group_by_rollup.sql	Fixture input: GROUP BY ROLLUP + HAVING.
tests/ast_json_fixtures/cases/15_group_by_rollup.json	Fixture output: GROUP BY ROLLUP AST JSON.
tests/ast_json_fixtures/cases/16_grouping_sets.sql	Fixture input: GROUPING SETS.
tests/ast_json_fixtures/cases/16_grouping_sets.json	Fixture output: GROUPING SETS AST JSON.
tests/ast_json_fixtures/cases/17_qualify.sql	Fixture input: QUALIFY.
tests/ast_json_fixtures/cases/17_qualify.json	Fixture output: QUALIFY AST JSON.
tests/ast_json_fixtures/cases/18_order_by_fill.sql	Fixture input: ORDER BY WITH FILL + INTERPOLATE.
tests/ast_json_fixtures/cases/18_order_by_fill.json	Fixture output: ORDER BY fill/interpolate AST JSON.
tests/ast_json_fixtures/cases/19_limits.sql	Fixture input: LIMIT BY + LIMIT/OFFSET.
tests/ast_json_fixtures/cases/19_limits.json	Fixture output: LIMITs AST JSON.
tests/ast_json_fixtures/cases/20_settings.sql	Fixture input: SETTINGS clause.
tests/ast_json_fixtures/cases/20_settings.json	Fixture output: SETTINGS AST JSON.
tests/ast_json_fixtures/cases/21_join_on.sql	Fixture input: JOIN ... ON.
tests/ast_json_fixtures/cases/21_join_on.json	Fixture output: JOIN ON AST JSON.
tests/ast_json_fixtures/cases/22_join_using.sql	Fixture input: JOIN ... USING.
tests/ast_json_fixtures/cases/22_join_using.json	Fixture output: JOIN USING AST JSON.
tests/ast_json_fixtures/cases/23_join_global_any.sql	Fixture input: GLOBAL ANY join.
tests/ast_json_fixtures/cases/23_join_global_any.json	Fixture output: GLOBAL ANY join AST JSON.
tests/ast_json_fixtures/cases/24_cross_comma.sql	Fixture input: CROSS + COMMA joins.
tests/ast_json_fixtures/cases/24_cross_comma.json	Fixture output: CROSS/COMMA AST JSON.
tests/ast_json_fixtures/cases/25_array_join.sql	Fixture input: ARRAY JOIN.
tests/ast_json_fixtures/cases/25_array_join.json	Fixture output: ARRAY JOIN AST JSON.
tests/ast_json_fixtures/cases/26_table_function.sql	Fixture input: table function.
tests/ast_json_fixtures/cases/26_table_function.json	Fixture output: table function AST JSON.
tests/ast_json_fixtures/cases/27_subquery_from.sql	Fixture input: subquery in FROM.
tests/ast_json_fixtures/cases/27_subquery_from.json	Fixture output: subquery FROM AST JSON.
tests/ast_json_fixtures/cases/28_sample_final.sql	Fixture input: FINAL + SAMPLE.
tests/ast_json_fixtures/cases/28_sample_final.json	Fixture output: FINAL/SAMPLE AST JSON.
tests/ast_json_fixtures/cases/29_cte.sql	Fixture input: CTE.
tests/ast_json_fixtures/cases/29_cte.json	Fixture output: CTE AST JSON.
tests/ast_json_fixtures/cases/30_scalar_with.sql	Fixture input: scalar WITH binding.
tests/ast_json_fixtures/cases/30_scalar_with.json	Fixture output: scalar WITH AST JSON.
tests/ast_json_fixtures/cases/31_union_all.sql	Fixture input: UNION ALL.
tests/ast_json_fixtures/cases/31_union_all.json	Fixture output: UNION ALL AST JSON.
tests/ast_json_fixtures/cases/32_union_distinct.sql	Fixture input: UNION DISTINCT.
tests/ast_json_fixtures/cases/32_union_distinct.json	Fixture output: UNION DISTINCT AST JSON.
tests/ast_json_fixtures/cases/33_intersect.sql	Fixture input: INTERSECT.
tests/ast_json_fixtures/cases/33_intersect.json	Fixture output: INTERSECT AST JSON.
tests/ast_json_fixtures/cases/34_except.sql	Fixture input: EXCEPT.
tests/ast_json_fixtures/cases/34_except.json	Fixture output: EXCEPT AST JSON.
tests/ast_json_fixtures/cases/35_qualified_star.sql	Fixture input: qualified asterisk.
tests/ast_json_fixtures/cases/35_qualified_star.json	Fixture output: qualified asterisk AST JSON.
tests/ast_json_fixtures/cases/36_except_transform.sql	Fixture input: * EXCEPT.
tests/ast_json_fixtures/cases/36_except_transform.json	Fixture output: * EXCEPT AST JSON.
tests/ast_json_fixtures/cases/37_apply_transform.sql	Fixture input: * APPLY.
tests/ast_json_fixtures/cases/37_apply_transform.json	Fixture output: * APPLY AST JSON.
tests/ast_json_fixtures/cases/38_replace_transform.sql	Fixture input: * REPLACE.
tests/ast_json_fixtures/cases/38_replace_transform.json	Fixture output: * REPLACE AST JSON.
tests/ast_json_fixtures/cases/39_columns_regexp.sql	Fixture input: COLUMNS('regexp').
tests/ast_json_fixtures/cases/39_columns_regexp.json	Fixture output: COLUMNS('regexp') AST JSON.
tests/ast_json_fixtures/cases/40_columns_list.sql	Fixture input: COLUMNS(a, b).
tests/ast_json_fixtures/cases/40_columns_list.json	Fixture output: COLUMNS(a, b) AST JSON.
tests/ast_json_fixtures/cases/41_showcase.sql	Fixture input: maximal showcase query.
tests/ast_json_fixtures/cases/41_showcase.json	Fixture output: maximal showcase AST JSON.