Use a typed sentinel for dynamic/batch dimensions instead of raw `null` in `TensorType.dims`

[khatchad]

Summary

After #544 closed the ragged half with a typed RaggedDim sentinel, per-element null in TensorType.dims still encodes "dynamic/batch/placeholder" dimensions at the following sites. The Iterable<Dimension<?>> contract is still violated per-element.

Current State

Raw null is emitted into final shape lists at:

• Input.java:137 (batchSizes.add(null) for unknown batch size)
• Input.java:154 (newShape.add(null) for dynamic batch dim)
• FlowFromDirectoryGenerator.java:75 (labelShape.add(null))
• TensorGenerator.java:2683 (ret.add(null))

These dims are semantically distinct from ragged: uniform within a tensor instance but unknown statically (e.g., tf.keras.Input(shape=(None, 4))).

Test fixtures TENSOR_*_NONE_* (e.g., TENSOR_NONE_4_FLOAT32, TENSOR_2_NONE_INT32, TENSOR_NONE_NONE_STRING) encode this representation directly and will need migration alongside the source change.

Proposed Fix

Two options:

1. Mirror Use a typed RaggedDim sentinel for ragged dimensions instead of raw null in TensorType.dims #544 with a DynamicDim extends Dimension<Void> (cheap, follows the just-landed precedent).
2. Unify with RaggedDim under a single UnknownDim(kind) carrying a tag (RAGGED|DYNAMIC|PLACEHOLDER)—the latter fully discharges the per-element-non-null contract claim originally made in Use a typed RaggedDim sentinel for ragged dimensions instead of raw null in TensorType.dims #544 in one move, at the cost of revisiting that fix.

Hidden coupling to update

TensorShapeUtil.areBroadcastable and TensorShapeUtil.getBroadcastedShapes already special-case null. Whichever encoding is chosen, broadcast must remain "compatible+propagate" for dynamic/batch dims.

Background

Spinoff from #544. The Iterable<Dimension<?>> non-null-element contract claim in that issue is only partially resolved by introducing RaggedDim; this issue tracks the remainder.

[khatchad]

Self-review after filing—flagging weaknesses so the next scoper isn't misled:

• Lower urgency than Use a typed RaggedDim sentinel for ragged dimensions instead of raw null in TensorType.dims #544. That issue had Hybridize hitting an actual precision loss; this one is hygiene with no concrete consumer mishandling the dynamic-batch nulls today. Worth doing but not parity-level priority.
• The two-option presentation punts the design. Option 1 (DynamicDim parallel to RaggedDim) and Option 2 (unify under UnknownDim(kind)) differ wildly in scope—Option 2 means re-doing Use a typed RaggedDim sentinel for ragged dimensions instead of raw null in TensorType.dims #544. I'd recommend Option 1 unless someone wants to argue Option 2 explicitly.
• "Fully discharges the per-element-non-null contract" is overpromised for Option 2—List<Dimension<?>> still permits null at the type level regardless of encoding; the contract is held by caller discipline either way.
• Confirmed sites; not necessarily exhaustive. The 4-site enumeration is from a one-shot grep. Helper-collection null adds (e.g. possibleRowDims.add(null) in the ragged generators) and possibly other shape-emit paths weren't audited.
• Hidden coupling: also reshape, not just broadcast. TensorTypeAnalysis.java:363 reads symbolicDims()/concreteSize() with implicit null behavior. A DynamicDim.symbolicDims()=1 choice would shift reshape just like RaggedDim did.
• Test fixture cost is more concrete than the body lets on. Per the Use a typed RaggedDim sentinel for ragged dimensions instead of raw null in TensorType.dims #544 migration: ~5 SHARED TENSOR_*_NONE_* constants, ~50 fixture-site updates, plus parallel-naming decisions where ragged-vs-dynamic semantics differ.

[khatchad]

Two pieces of context for the next scoper:

1. External anchor. ponder-lab/Hybridize-Functions-Refactoring is about to extend its precision audit to fixtures using tf.keras.Input(shape=(None, ...)) (batch-3 of the audit, follow-up to Extend precision audit to fixtures 60-66 (RaggedTensor.from_nested_row_splits) ponder-lab/Hybridize-Functions-Refactoring#525). Those tests will TODO-anchor against this issue, so it does have a concrete downstream documentary use beyond pure hygiene—closing it would break the audit anchor.
2. Onboarding fit. With Use a typed RaggedDim sentinel for ragged dimensions instead of raw null in TensorType.dims #544 having just established the typed-sentinel precedent, Option 1 (DynamicDim parallel to RaggedDim) is a tractable first-PR task. Suggest a good first issue label.

[khatchad]

Two pieces of context for the next scoper:

1. External anchor. ponder-lab/Hybridize-Functions-Refactoring is about to extend its precision audit to fixtures using tf.keras.Input(shape=(None, ...)) (batch-3 of the audit, follow-up to Extend precision audit to fixtures 60-66 (RaggedTensor.from_nested_row_splits) ponder-lab/Hybridize-Functions-Refactoring#525). Those tests will TODO-anchor against this issue, so it does have a concrete downstream documentary use beyond pure hygiene—closing it would break the audit anchor.
2. Onboarding fit. With Use a typed RaggedDim sentinel for ragged dimensions instead of raw null in TensorType.dims #544 having just established the typed-sentinel precedent, Option 1 (DynamicDim parallel to RaggedDim) is a tractable first-PR task. Suggest a good first issue label.

Both fair—my "lower urgency" framing was wrong with the audit anchor in play. Adding good first issue now.

[khatchad]

One more reframing: strictly in Ariadne's own output, this isn't a precision loss. After #544 closed the ragged conflation, null in TensorType.dims is unambiguous (dynamic/placeholder), and the analysis emits the same information it always did.

The "precision gap" label fits from the downstream-dispatch perspective the audit operates in: a consumer can write dim instanceof RaggedDim after #544 but not dim instanceof DynamicDim, so it falls back to dim == null. Asymmetric, and null carries no symbolicDims()/concreteSize() API, so any interrogation has to special-case it instead of calling methods uniformly.

So accurate framing: dispatch/type-safety gap in Ariadne, which surfaces as a precision gap for downstream tooling. The issue body's "non-null-element contract" wording captures the Ariadne side correctly.