From d222ac327544c3da7f3460e11749367a5e69a65d Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Thu, 16 Jul 2026 21:53:17 +0000 Subject: [PATCH 1/6] modular.md: document checkpoint variants as separate blocks assemblies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Contributors keep handling checkpoint variants (distilled/turbo) with a ConfigSpec flag and an if-branch inside a shared block — the standard-pipeline mindset ported into modular. The doc never said what to do instead: nothing routed on 'behavior differs per checkpoint', and flux2-klein was only cited as a flatness example, not as the variant pattern it embodies. Adds: - a per-checkpoint branch in the block types decision tree - 'Key pattern: Checkpoint variants' — separate assembly when the variant changes the contract (inputs/components/blocks), ConfigSpec only when it changes a value; flux2-klein as reference; the config-flag anti-pattern - gotcha #9 for the if-components.config. smell Co-Authored-By: Claude Fable 5 --- .ai/modular.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/.ai/modular.md b/.ai/modular.md index 11dedf821fef..f27cbce649c1 100644 --- a/.ai/modular.md +++ b/.ai/modular.md @@ -58,6 +58,11 @@ Does it choose ONE block based on which input is present? Is the selection 1:1 with trigger inputs? YES -> AutoPipelineBlocks (simple trigger mapping) NO -> ConditionalPipelineBlocks (custom select_block method) + +Does behavior differ per CHECKPOINT (distilled / turbo / a variant with its own schedule), +not per call? + YES -> a separate blocks assembly per variant -- NOT a config flag branching + inside a shared block (see Key pattern: Checkpoint variants) ``` ## Build order (easiest first) @@ -166,6 +171,33 @@ class AutoDenoise(ConditionalPipelineBlocks): default_block_name = "text2video" ``` +## Key pattern: Checkpoint variants + +Workflow selection (above) handles behavior that varies **per call** — which inputs the user passed. Behavior that varies **per checkpoint** — a distilled / turbo variant, a schedule baked into the weights — is a different situation with a different answer: + +- **The variant changes the contract** (which inputs users may pass, which components exist, which blocks run): give it **its own blocks assembly** and pipeline class. Flux2 Klein (guidance-distilled) is the reference: `modular_blocks_flux2_klein.py` composes the *same* base leaf blocks (`Flux2TextInputStep`, `Flux2PrepareLatentsStep`, `Flux2SetTimestepsStep`, …) and swaps in only what actually differs — a text encoder that doesn't declare `negative_prompt`, a denoise step without a guider. No duplicated code: the variant is a different *composition*, not a different copy. Each variant gets a `ModularPipeline` subclass carrying its `default_blocks_name` (e.g. `Flux2KleinModularPipeline`), and checkpoints resolve to the right class via `_class_name` in `modular_model_index.json` or a config-keyed map fn in `MODULAR_PIPELINE_MAPPING` (see `_flux2_klein_map_fn`). +- **The variant only changes a value** consumed by the same code path (a fixed sigma list, a default prompt-template flag): a `ConfigSpec` on the block is enough — the checkpoint ships the value in its `modular_model_index.json`. + +The litmus test: would the variant's documentation list different inputs, components, or blocks? Then it's a new assembly. If everything is identical except a number the checkpoint ships, it's config. + +**Anti-pattern** — a boolean variant flag selecting between behaviors inside a shared block: + +```python +# don't do this — two checkpoints' behaviors hidden inside one block +@property +def expected_configs(self): + return [ConfigSpec(name="is_distilled", default=False)] + +def __call__(self, components, state): + ... + if components.config.is_distilled: + mu = 1.15 + else: + mu = calculate_shift(...) +``` + +This is the standard-pipeline mindset (`if self.config.x:` inside `__call__`) ported into modular. The branch never stays contained: the same flag soon wants to gate the text encoder (does the distilled variant accept `negative_prompt`?) and the denoise step (does it run a guider?), until every block hides two behaviors, each checkpoint carries the other's dead branch, and the distilled checkpoint silently accepts inputs it ignores. Split the assembly instead — only the steps that truly differ need new block classes; everything else is reused by composition. + ## Key pattern: Standalone block reusability One of the core reason a pipeline is split into blocks at all: each block (text encoder, VAE encoder, prepare-latents, denoise, decoder) must be runnable on its own, and its output must be reusable as the input to a different downstream chain. @@ -249,6 +281,8 @@ ComponentSpec( 8. **No-op skip logic inside an optional block.** If a step is conditional (e.g. an optional prompt enhancer), don't have the block check a flag at the top of `__call__` and `return` early. Wrap it in an `AutoPipelineBlocks` with `block_trigger_inputs = ["use_xxx"]` so the block is only assembled into the pipeline when the trigger input is provided. The block's own `__call__` should always assume its components and inputs are present. +9. **A checkpoint-variant flag branching inside a shared block.** `ConfigSpec(name="is_distilled")` plus `if components.config.is_distilled:` in a block body means one assembly is serving two checkpoints with different behavior. Give the variant its own blocks assembly (see Key pattern: Checkpoint variants). Config values are for per-checkpoint *values* the same code path consumes — not for selecting between code paths. + ## Conversion checklist - [ ] Read original pipeline's `__call__` end-to-end, map stages From 0ae1687d14559fdeb84f02d5fe7413c814444fa9 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Fri, 17 Jul 2026 00:12:50 +0000 Subject: [PATCH 2/6] Clarify when the map fn resolves the variant (standard model_index repos) Co-Authored-By: Claude Fable 5 --- .ai/modular.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.ai/modular.md b/.ai/modular.md index f27cbce649c1..8f38ebee79fb 100644 --- a/.ai/modular.md +++ b/.ai/modular.md @@ -175,7 +175,7 @@ class AutoDenoise(ConditionalPipelineBlocks): Workflow selection (above) handles behavior that varies **per call** — which inputs the user passed. Behavior that varies **per checkpoint** — a distilled / turbo variant, a schedule baked into the weights — is a different situation with a different answer: -- **The variant changes the contract** (which inputs users may pass, which components exist, which blocks run): give it **its own blocks assembly** and pipeline class. Flux2 Klein (guidance-distilled) is the reference: `modular_blocks_flux2_klein.py` composes the *same* base leaf blocks (`Flux2TextInputStep`, `Flux2PrepareLatentsStep`, `Flux2SetTimestepsStep`, …) and swaps in only what actually differs — a text encoder that doesn't declare `negative_prompt`, a denoise step without a guider. No duplicated code: the variant is a different *composition*, not a different copy. Each variant gets a `ModularPipeline` subclass carrying its `default_blocks_name` (e.g. `Flux2KleinModularPipeline`), and checkpoints resolve to the right class via `_class_name` in `modular_model_index.json` or a config-keyed map fn in `MODULAR_PIPELINE_MAPPING` (see `_flux2_klein_map_fn`). +- **The variant changes the contract** (which inputs users may pass, which components exist, which blocks run): give it **its own blocks assembly** and pipeline class. Flux2 Klein (guidance-distilled) is the reference: `modular_blocks_flux2_klein.py` composes the *same* base leaf blocks (`Flux2TextInputStep`, `Flux2PrepareLatentsStep`, `Flux2SetTimestepsStep`, …) and swaps in only what actually differs — a text encoder that doesn't declare `negative_prompt`, a denoise step without a guider. No duplicated code: the variant is a different *composition*, not a different copy. Each variant gets a `ModularPipeline` subclass carrying its `default_blocks_name` (e.g. `Flux2KleinModularPipeline`), so `ModularPipeline.from_pretrained` routes each checkpoint to the right variant automatically: a modular repo names its class directly via `_class_name` in `modular_model_index.json`; a repo that only ships a standard `model_index.json` has no `_class_name` to resolve against, so a config-keyed map fn in `MODULAR_PIPELINE_MAPPING` picks the class from the checkpoint's config instead (see `_flux2_klein_map_fn`, which keys on `is_distilled`). - **The variant only changes a value** consumed by the same code path (a fixed sigma list, a default prompt-template flag): a `ConfigSpec` on the block is enough — the checkpoint ships the value in its `modular_model_index.json`. The litmus test: would the variant's documentation list different inputs, components, or blocks? Then it's a new assembly. If everything is identical except a number the checkpoint ships, it's config. From 92e13f0164bec3ecb32797081f3af3466fc85c4d Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Fri, 17 Jul 2026 00:15:47 +0000 Subject: [PATCH 3/6] Sharpen variant rule: inputs are the hard line (repo can override components/config, never inputs) Co-Authored-By: Claude Fable 5 --- .ai/modular.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.ai/modular.md b/.ai/modular.md index 8f38ebee79fb..c166afa26f3f 100644 --- a/.ai/modular.md +++ b/.ai/modular.md @@ -178,7 +178,7 @@ Workflow selection (above) handles behavior that varies **per call** — which i - **The variant changes the contract** (which inputs users may pass, which components exist, which blocks run): give it **its own blocks assembly** and pipeline class. Flux2 Klein (guidance-distilled) is the reference: `modular_blocks_flux2_klein.py` composes the *same* base leaf blocks (`Flux2TextInputStep`, `Flux2PrepareLatentsStep`, `Flux2SetTimestepsStep`, …) and swaps in only what actually differs — a text encoder that doesn't declare `negative_prompt`, a denoise step without a guider. No duplicated code: the variant is a different *composition*, not a different copy. Each variant gets a `ModularPipeline` subclass carrying its `default_blocks_name` (e.g. `Flux2KleinModularPipeline`), so `ModularPipeline.from_pretrained` routes each checkpoint to the right variant automatically: a modular repo names its class directly via `_class_name` in `modular_model_index.json`; a repo that only ships a standard `model_index.json` has no `_class_name` to resolve against, so a config-keyed map fn in `MODULAR_PIPELINE_MAPPING` picks the class from the checkpoint's config instead (see `_flux2_klein_map_fn`, which keys on `is_distilled`). - **The variant only changes a value** consumed by the same code path (a fixed sigma list, a default prompt-template flag): a `ConfigSpec` on the block is enough — the checkpoint ships the value in its `modular_model_index.json`. -The litmus test: would the variant's documentation list different inputs, components, or blocks? Then it's a new assembly. If everything is identical except a number the checkpoint ships, it's config. +The hard line is the **inputs**. A checkpoint's repo can override *components* and *config values* per checkpoint through `modular_model_index.json` — but it can never change which inputs the blocks declare; the input surface is baked into the assembly. So the moment a variant should accept different inputs (a guidance-distilled checkpoint must not expose `negative_prompt`), a separate assembly is the only mechanism that can express it. If everything is identical except a value the checkpoint ships, it's config. **Anti-pattern** — a boolean variant flag selecting between behaviors inside a shared block: @@ -196,7 +196,7 @@ def __call__(self, components, state): mu = calculate_shift(...) ``` -This is the standard-pipeline mindset (`if self.config.x:` inside `__call__`) ported into modular. The branch never stays contained: the same flag soon wants to gate the text encoder (does the distilled variant accept `negative_prompt`?) and the denoise step (does it run a guider?), until every block hides two behaviors, each checkpoint carries the other's dead branch, and the distilled checkpoint silently accepts inputs it ignores. Split the assembly instead — only the steps that truly differ need new block classes; everything else is reused by composition. +An internal value branch like the `mu` above is the mild end — tolerable, though shipping the value as checkpoint config would remove the branch entirely. The real damage is what the flag *cannot* fix: with one shared assembly, the distilled checkpoint still declares `negative_prompt` and silently accepts an input it ignores, and no repo config can undeclare it. Once the flag exists it also tends to spread (soon the denoise step branches on it too), which is the standard-pipeline `if self.config.x:` mindset ported into modular. Split the assembly instead — only the steps that truly differ need new block classes; everything else is reused by composition. ## Key pattern: Standalone block reusability From c3139a3963b4f9186259a1431ac4389c0013860d Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Fri, 17 Jul 2026 00:20:09 +0000 Subject: [PATCH 4/6] Reframe checkpoint variants around the advantage, not the prohibition The config-flag branch is what standard pipelines are forced into; modular's pitch is that the loaded pipeline describes exactly its checkpoint. Lead with the payoff (clean per-variant contract, automatic routing, input-surface expressiveness) and present the flag branch as tolerable-but-lossy rather than forbidden. Co-Authored-By: Claude Fable 5 --- .ai/modular.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/.ai/modular.md b/.ai/modular.md index c166afa26f3f..9309e0f0c668 100644 --- a/.ai/modular.md +++ b/.ai/modular.md @@ -61,8 +61,8 @@ Does it choose ONE block based on which input is present? Does behavior differ per CHECKPOINT (distilled / turbo / a variant with its own schedule), not per call? - YES -> a separate blocks assembly per variant -- NOT a config flag branching - inside a shared block (see Key pattern: Checkpoint variants) + YES -> prefer a separate blocks assembly per variant over a config flag + branching inside a shared block (see Key pattern: Checkpoint variants) ``` ## Build order (easiest first) @@ -173,17 +173,19 @@ class AutoDenoise(ConditionalPipelineBlocks): ## Key pattern: Checkpoint variants -Workflow selection (above) handles behavior that varies **per call** — which inputs the user passed. Behavior that varies **per checkpoint** — a distilled / turbo variant, a schedule baked into the weights — is a different situation with a different answer: +Workflow selection (above) handles behavior that varies **per call** — which inputs the user passed. Behavior that varies **per checkpoint** — a distilled / turbo variant, a schedule baked into the weights — is where modular has a structural advantage over standard pipelines. A standard pipeline has one class per task, so a variant has no choice but to live inside it as `if self.config.is_distilled:` branches. Modular lifts that constraint: give each variant its own blocks assembly, and what a user loads describes exactly the checkpoint in front of them — nothing bundled in from a sibling variant. -- **The variant changes the contract** (which inputs users may pass, which components exist, which blocks run): give it **its own blocks assembly** and pipeline class. Flux2 Klein (guidance-distilled) is the reference: `modular_blocks_flux2_klein.py` composes the *same* base leaf blocks (`Flux2TextInputStep`, `Flux2PrepareLatentsStep`, `Flux2SetTimestepsStep`, …) and swaps in only what actually differs — a text encoder that doesn't declare `negative_prompt`, a denoise step without a guider. No duplicated code: the variant is a different *composition*, not a different copy. Each variant gets a `ModularPipeline` subclass carrying its `default_blocks_name` (e.g. `Flux2KleinModularPipeline`), so `ModularPipeline.from_pretrained` routes each checkpoint to the right variant automatically: a modular repo names its class directly via `_class_name` in `modular_model_index.json`; a repo that only ships a standard `model_index.json` has no `_class_name` to resolve against, so a config-keyed map fn in `MODULAR_PIPELINE_MAPPING` picks the class from the checkpoint's config instead (see `_flux2_klein_map_fn`, which keys on `is_distilled`). -- **The variant only changes a value** consumed by the same code path (a fixed sigma list, a default prompt-template flag): a `ConfigSpec` on the block is enough — the checkpoint ships the value in its `modular_model_index.json`. +Flux2 Klein (guidance-distilled) is the reference: `modular_blocks_flux2_klein.py` composes the *same* base leaf blocks (`Flux2TextInputStep`, `Flux2PrepareLatentsStep`, `Flux2SetTimestepsStep`, ...) and swaps in only what actually differs — a text encoder that doesn't declare `negative_prompt`, a denoise step without a guider. No duplicated code: the variant is a different *composition*, not a different copy. -The hard line is the **inputs**. A checkpoint's repo can override *components* and *config values* per checkpoint through `modular_model_index.json` — but it can never change which inputs the blocks declare; the input surface is baked into the assembly. So the moment a variant should accept different inputs (a guidance-distilled checkpoint must not expose `negative_prompt`), a separate assembly is the only mechanism that can express it. If everything is identical except a value the checkpoint ships, it's config. +The payoff, concretely: -**Anti-pattern** — a boolean variant flag selecting between behaviors inside a shared block: +- `ModularPipeline.from_pretrained("")` returns a pipeline whose declared inputs, components, workflow map, and auto-generated docs *are* the distilled contract — no `negative_prompt` input, no guider component, no docstring caveats like "ignored for distilled checkpoints". The base checkpoint's pipeline keeps its CFG surface. Neither carries the other's baggage. +- Routing is automatic — users never pick the assembly by hand. Each variant gets a `ModularPipeline` subclass carrying its `default_blocks_name` (e.g. `Flux2KleinModularPipeline`); a modular repo names its class directly via `_class_name` in `modular_model_index.json`, and a repo that only ships a standard `model_index.json` resolves through a config-keyed map fn in `MODULAR_PIPELINE_MAPPING` (see `_flux2_klein_map_fn`, which keys on `is_distilled`). +- Some differences *only* an assembly can express. A checkpoint's repo can override components and config values per checkpoint through `modular_model_index.json`, but it can never change which inputs the blocks declare — the input surface is baked into the assembly. When the variant should accept different inputs (a guidance-distilled checkpoint shouldn't expose `negative_prompt` at all), the split isn't just cleaner, it's the only mechanism there is. + +You *can* still handle a variant the standard-pipeline way — a `ConfigSpec` flag and a branch inside a shared block: ```python -# don't do this — two checkpoints' behaviors hidden inside one block @property def expected_configs(self): return [ConfigSpec(name="is_distilled", default=False)] @@ -196,7 +198,7 @@ def __call__(self, components, state): mu = calculate_shift(...) ``` -An internal value branch like the `mu` above is the mild end — tolerable, though shipping the value as checkpoint config would remove the branch entirely. The real damage is what the flag *cannot* fix: with one shared assembly, the distilled checkpoint still declares `negative_prompt` and silently accepts an input it ignores, and no repo config can undeclare it. Once the flag exists it also tends to spread (soon the denoise step branches on it too), which is the standard-pipeline `if self.config.x:` mindset ported into modular. Split the assembly instead — only the steps that truly differ need new block classes; everything else is reused by composition. +It works, and for a single internal value like the `mu` here it's tolerable (though shipping the value as checkpoint config would remove the branch entirely). But every such branch gives back a piece of the advantage: the declared surface and docs stop describing any one checkpoint, each checkpoint carries the other's code path, and anything the flag can't reach — the input surface — leaks silently (the distilled checkpoint still accepts `negative_prompt` and ignores it). When a variant is a different checkpoint, prefer a different assembly; only the steps that truly differ need new block classes. ## Key pattern: Standalone block reusability @@ -281,7 +283,7 @@ ComponentSpec( 8. **No-op skip logic inside an optional block.** If a step is conditional (e.g. an optional prompt enhancer), don't have the block check a flag at the top of `__call__` and `return` early. Wrap it in an `AutoPipelineBlocks` with `block_trigger_inputs = ["use_xxx"]` so the block is only assembled into the pipeline when the trigger input is provided. The block's own `__call__` should always assume its components and inputs are present. -9. **A checkpoint-variant flag branching inside a shared block.** `ConfigSpec(name="is_distilled")` plus `if components.config.is_distilled:` in a block body means one assembly is serving two checkpoints with different behavior. Give the variant its own blocks assembly (see Key pattern: Checkpoint variants). Config values are for per-checkpoint *values* the same code path consumes — not for selecting between code paths. +9. **Serving a checkpoint variant through a config flag in a shared block.** `ConfigSpec(name="is_distilled")` plus `if components.config.is_distilled:` works, but it gives up modular's main advantage — a loaded pipeline that describes exactly its checkpoint — and it cannot express input differences at all (a distilled variant would still accept `negative_prompt`). Suggest a separate blocks assembly per variant instead (see Key pattern: Checkpoint variants); config values are best kept for per-checkpoint *values* the same code path consumes. ## Conversion checklist From f696c3411e82a0b11bb45169ca05da0f4a1c6542 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Fri, 17 Jul 2026 02:20:16 +0000 Subject: [PATCH 5/6] Simplify checkpoint-variants section: different checkpoint -> own blockset unless literally the same Co-Authored-By: Claude Fable 5 --- .ai/modular.md | 36 +++++++----------------------------- 1 file changed, 7 insertions(+), 29 deletions(-) diff --git a/.ai/modular.md b/.ai/modular.md index 9309e0f0c668..88284e961a10 100644 --- a/.ai/modular.md +++ b/.ai/modular.md @@ -59,10 +59,9 @@ Does it choose ONE block based on which input is present? YES -> AutoPipelineBlocks (simple trigger mapping) NO -> ConditionalPipelineBlocks (custom select_block method) -Does behavior differ per CHECKPOINT (distilled / turbo / a variant with its own schedule), -not per call? - YES -> prefer a separate blocks assembly per variant over a config flag - branching inside a shared block (see Key pattern: Checkpoint variants) +Is it a different CHECKPOINT (distilled / turbo / a variant with its own schedule)? + YES -> its own blocks assembly, unless it behaves literally the same + (see Key pattern: Checkpoint variants) ``` ## Build order (easiest first) @@ -173,32 +172,11 @@ class AutoDenoise(ConditionalPipelineBlocks): ## Key pattern: Checkpoint variants -Workflow selection (above) handles behavior that varies **per call** — which inputs the user passed. Behavior that varies **per checkpoint** — a distilled / turbo variant, a schedule baked into the weights — is where modular has a structural advantage over standard pipelines. A standard pipeline has one class per task, so a variant has no choice but to live inside it as `if self.config.is_distilled:` branches. Modular lifts that constraint: give each variant its own blocks assembly, and what a user loads describes exactly the checkpoint in front of them — nothing bundled in from a sibling variant. +A different checkpoint (distilled / turbo / a variant with its own schedule) can have its own blocks assembly mapped to it: give the variant a `ModularPipeline` subclass carrying its `default_blocks_name`, and checkpoints route to it automatically — via `_class_name` in `modular_model_index.json`, or, for repos that only ship a standard `model_index.json`, a config-keyed map fn in `MODULAR_PIPELINE_MAPPING` (see `_flux2_klein_map_fn`). -Flux2 Klein (guidance-distilled) is the reference: `modular_blocks_flux2_klein.py` composes the *same* base leaf blocks (`Flux2TextInputStep`, `Flux2PrepareLatentsStep`, `Flux2SetTimestepsStep`, ...) and swaps in only what actually differs — a text encoder that doesn't declare `negative_prompt`, a denoise step without a guider. No duplicated code: the variant is a different *composition*, not a different copy. +Default to taking that option. The only reason not to is when the variant behaves literally the same; if the split buys anything at all — the distilled variant doesn't have to declare `negative_prompt`, doesn't carry a guider, its docs describe exactly what the checkpoint does — make the separate assembly. It costs almost nothing: assemblies compose the same shared leaf blocks, and only the steps that truly differ need new block classes. See `modular_blocks_flux2_klein.py`, which reuses the base flux2 leaf blocks and swaps in just a `negative_prompt`-free text encoder and a guider-free denoise step. -The payoff, concretely: - -- `ModularPipeline.from_pretrained("")` returns a pipeline whose declared inputs, components, workflow map, and auto-generated docs *are* the distilled contract — no `negative_prompt` input, no guider component, no docstring caveats like "ignored for distilled checkpoints". The base checkpoint's pipeline keeps its CFG surface. Neither carries the other's baggage. -- Routing is automatic — users never pick the assembly by hand. Each variant gets a `ModularPipeline` subclass carrying its `default_blocks_name` (e.g. `Flux2KleinModularPipeline`); a modular repo names its class directly via `_class_name` in `modular_model_index.json`, and a repo that only ships a standard `model_index.json` resolves through a config-keyed map fn in `MODULAR_PIPELINE_MAPPING` (see `_flux2_klein_map_fn`, which keys on `is_distilled`). -- Some differences *only* an assembly can express. A checkpoint's repo can override components and config values per checkpoint through `modular_model_index.json`, but it can never change which inputs the blocks declare — the input surface is baked into the assembly. When the variant should accept different inputs (a guidance-distilled checkpoint shouldn't expose `negative_prompt` at all), the split isn't just cleaner, it's the only mechanism there is. - -You *can* still handle a variant the standard-pipeline way — a `ConfigSpec` flag and a branch inside a shared block: - -```python -@property -def expected_configs(self): - return [ConfigSpec(name="is_distilled", default=False)] - -def __call__(self, components, state): - ... - if components.config.is_distilled: - mu = 1.15 - else: - mu = calculate_shift(...) -``` - -It works, and for a single internal value like the `mu` here it's tolerable (though shipping the value as checkpoint config would remove the branch entirely). But every such branch gives back a piece of the advantage: the declared surface and docs stop describing any one checkpoint, each checkpoint carries the other's code path, and anything the flag can't reach — the input surface — leaks silently (the distilled checkpoint still accepts `negative_prompt` and ignores it). When a variant is a different checkpoint, prefer a different assembly; only the steps that truly differ need new block classes. +Don't fall back to the standard-pipeline habit of a config flag branching inside a shared block (`ConfigSpec(name="is_distilled")` + `if components.config.is_distilled:`). That keeps both variants' behavior bundled in one blockset — and the input surface is the one thing it can never fix: a repo can override components and config values per checkpoint, but never which inputs the blocks declare, so the distilled checkpoint would still accept `negative_prompt` and silently ignore it. ## Key pattern: Standalone block reusability @@ -283,7 +261,7 @@ ComponentSpec( 8. **No-op skip logic inside an optional block.** If a step is conditional (e.g. an optional prompt enhancer), don't have the block check a flag at the top of `__call__` and `return` early. Wrap it in an `AutoPipelineBlocks` with `block_trigger_inputs = ["use_xxx"]` so the block is only assembled into the pipeline when the trigger input is provided. The block's own `__call__` should always assume its components and inputs are present. -9. **Serving a checkpoint variant through a config flag in a shared block.** `ConfigSpec(name="is_distilled")` plus `if components.config.is_distilled:` works, but it gives up modular's main advantage — a loaded pipeline that describes exactly its checkpoint — and it cannot express input differences at all (a distilled variant would still accept `negative_prompt`). Suggest a separate blocks assembly per variant instead (see Key pattern: Checkpoint variants); config values are best kept for per-checkpoint *values* the same code path consumes. +9. **Serving a checkpoint variant through a config flag in a shared block.** `ConfigSpec(name="is_distilled")` plus `if components.config.is_distilled:` bundles two checkpoints' behavior into one blockset — and it can't change the input surface at all (the distilled variant would still accept `negative_prompt`). Suggest a separate blocks assembly for the variant instead (see Key pattern: Checkpoint variants). ## Conversion checklist From 641214e9ed6050e2564e66c76e72bdc85e6c0831 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Fri, 17 Jul 2026 02:33:50 +0000 Subject: [PATCH 6/6] Add growth rule: one workflow at a time, new blocks over edits, generalize only by removing branches/duplicates Co-Authored-By: Claude Fable 5 --- .ai/modular.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.ai/modular.md b/.ai/modular.md index 88284e961a10..407ac8bea805 100644 --- a/.ai/modular.md +++ b/.ai/modular.md @@ -71,6 +71,17 @@ Is it a different CHECKPOINT (distilled / turbo / a variant with its own schedul 3. `before_denoise.py` -- Timesteps, latent prep, noise setup. Each logical operation = one block 4. `denoise.py` -- The hardest. Convert guidance to guider abstraction +## Growing a pipeline: one workflow at a time + +Build one workflow end-to-end first (e.g. t2v), then add the next workflow — and later the next blocks assembly / checkpoint variant — one at a time. Each addition should **introduce new blocks rather than modify existing ones**: existing blocks are already wired into working workflows, and a new leaf block plus a new assembly entry can't break them. + +The one good reason to touch an existing block is to make it strictly more **general**. Be honest about which direction the edit goes: + +- **Adding a branch is not generalizing — it's specializing.** `if components.config.foo:` or `if block_state.image is not None:` inside an existing block means the block now does two things. Add a new block for the new case instead, and let workflow selection or a variant assembly pick between them. +- **Collapsing duplicates is generalizing.** If two blocks are identical except for which conditioning inputs they pass to the denoiser, don't keep both — rework the one block to take `kwargs_type="denoiser_input_fields"` (see the `kwargs_type` pattern below) so the same block serves every workflow, as the Cosmos3 denoise step does. + +Rule of thumb: a generalizing edit *removes* an if/else or a duplicate block. If your edit *adds* an if/else, it's a new block trying to get out. + ## Key pattern: Guider abstraction Original pipeline has guidance baked in: