[BUG]: Operational metadata columns added to target schema when features.operationalMetadataEnabled is false

[haillew]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When a dataflow spec sets features.operationalMetadataEnabled: false, the target table is still created with the operational metadata columns appended to its schema. The source/MV read paths correctly skip adding those columns to the DataFrame, so the table schema and the data written to it no longer match — resulting in null-valued metadata columns or schema-mismatch write failures.

There are three ways to control operational metadata today, but the target schema-init step only honors two of them:

Control	Where it's set	Honored at target schema init?	Honored at data read / write?
TargetConfigFlags.DISABLE_OPERATIONAL_METADATA (per target)	target.configFlags	Yes	Yes
Pipeline layer not set (no schema loaded)	pipeline_layer empty	Yes	Yes
features.operationalMetadataEnabled = false (per dataflow spec)	dataflow_spec.features	No	Yes

The per-spec features flag is not visible to the target at schema-init time because features is constructed later, when the dataflow is assembled — by then the target schema has already been augmented.

CDC snapshot tables are not uniquely affected — they only hit this bug when the feature is disabled at the spec level.

Expected Behavior

When features.operationalMetadataEnabled is false, the target table should be created without operational metadata columns, matching what the source/MV read paths produce. All three controls should behave consistently across schema init and data read/write.

Steps To Reproduce

1. Configure a pipeline where pipeline_layer is set (e.g. silver) so the framework loads an operational metadata schema.
2. In a dataflow spec, set features.operationalMetadataEnabled: false.
3. Define a target with a schemaPath (any target type — streaming table, MV, staging table, or CDC snapshot).
4. Run the pipeline and inspect the created target table schema vs. the written data.

Channel

CURRENT

Relevant log output

Suggested fix

Defer the operational-metadata column augmentation out of the target's schema-init step and into create_table(...), where the features object is (or can be made) available. This mirrors the gating already used by the materialized view target.