[FEATURE]: nodespec — the single unified dataflow spec (replaces standard/flow/materialized_view)

[haillew]

READ FIRST: nodespec is NOT a fourth spec type

nodespec is the single, unified dataflow spec that replaces standard, flow, and materialized_view going forward.

This is not a fourth option to pick from. It is the one and only spec every new pipeline should use. The three legacy formats are kept working only for backward compatibility during migration and are on a path to be retired. If you are writing a new pipeline: write nodespec.

---

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

The framework currently offers several distinct dataflow spec formats (standard, flow, and materialized view), and each one has its own structure and rules. This creates avoidable friction:

• Multiple formats to learn instead of one. A user must first decide which spec type fits their case, then learn a different field layout for each. Knowledge gained writing one format does not transfer to the others.
• The structure leaks framework internals. The flow format asks authors to think in flow groups, view registration, and staging tables, which are concepts that serve the engine rather than the person describing a pipeline.
• Deep nesting is hard to write and hard to read. Settings live at different depths depending on the format and feature. Answering "what feeds this table, and what happens to the data on the way in?" can mean tracing keys across several levels of nested objects.
• Topology is implicit. Connections between steps are inferred (from SQL, key names, or position) rather than stated, so the pipeline graph is not visible from the spec.
• Streaming tables and materialized views must live in separate specs. Because each output type has its own format, a pipeline that mixes streaming tables and materialized views has to be split across multiple specs. When those tables feed each other (for example a streaming table chained into a materialized view) the split is artificial and inconvenient: one logical pipeline ends up fragmented across files for no reason other than the spec format.

This cost is paid most by the two groups this framework exists to help. The first is newcomers writing their first pipeline, who should not have to learn the engine's internals to get started. The second is large organizations that want to standardize and to easily read, understand, and edit each other's pipelines across many people. Making both of those easier is the goal of the framework.

Proposed Solution

Introduce a single, unified dataflow spec — nodespec — and make it the way pipelines are described going forward (not an additional option alongside the others, but the replacement for all of them).

A nodespec is a graph of nodes with three node types that compose by chaining:

source  ->  transformation  ->  target

• source: where data comes from (table, files, stream)
• transformation: how data is reshaped (SQL or Python)
• target: where data lands, carrying its own table-level settings (CDC, data quality, quarantine, clustering, and so on)

Nodes are wired together in two complementary ways. A target node declares what feeds it through an explicit input list. Source and transformation nodes are connected by explicit reference inside their own definition: a transformation names its upstream view directly in its SQL or Python (for example FROM STREAM(live.v_source_customer)), and a source names the table, path, or stream it reads. The input list is therefore a target-node construct, while the intermediate source-to-transformation wiring lives where the logic that uses it lives. This one uniform model replaces all the legacy formats. The node graph lowers into the framework's existing flow-spec representation, so no engine changes are required and all current capabilities remain available. Legacy formats continue to work, and a migration script converts existing specs.

Because output type is just a property of a target node, a single nodespec can declare both streaming-table and materialized-view targets at once, including chains where one feeds the other. A logical pipeline that mixes the two no longer has to be split across separate specs by output type.

Flow names are controllable: by default they are derived from the graph, and a target's input item can be written as { "view": "...", "flow": "..." } to define the SDP flow name. This matters because renaming a flow forces a full refresh in SDP, so defining it keeps the name stable across edits.

To steer adoption toward the chaining model:

• Inline SQL/Python sources are discouraged (warning, non-breaking) in both legacy and nodespec formats. A source whose type is sql or python (or an append_sql flow) folds transformation logic into the source definition. The recommended replacement is a plain source chained into a dedicated transformation node.
• Inline source_view on materialized view targets is removed (breaking). Authors chain a source node into the MV target via input, like every other node. This pattern is not in use, so it is removed rather than deprecated.

Additional Context

A design decision record is included in the repo at docs/decisions/0008-unified-nodespec-dataflow-spec.md, with before/after examples and the rationale.