Skip to content

feat(#188): warn-by-default guardrail for product templates without variants#202

Merged
bosd merged 4 commits into
masterfrom
feat/variant-guardrail
Jun 11, 2026
Merged

feat(#188): warn-by-default guardrail for product templates without variants#202
bosd merged 4 commits into
masterfrom
feat/variant-guardrail

Conversation

@bosd

@bosd bosd commented Jun 11, 2026

Copy link
Copy Markdown
Member

Follow-up to #188 / #201. Implements the guardrail discussed on #201: the issue's real pain was silent discovery in production, which a manual command alone doesn't prevent.

What it does

After every product.template import, fluvo automatically checks the just-imported templates (scoped via the id_map — one search_count) and warns if any ended up with no variant:

WARNING  3 imported product template(s) have NO variants and are unusable in
         sales/purchase orders and BoMs ... Fix with 'fluvo workflow
         create-missing-variants', or re-run with --fix-missing-variants.

--fix-missing-variants on the import command creates them inline instead of only warning.

Design (per the #201 discussion)

  • Warn by default — no surprising writes during an import; the warning surfaces the problem (it could be a signal that the source data was corrupt, which you'd want to know, not have silently patched).
  • --fix-missing-variants — opt-in inline cure.
  • The separate workflow create-missing-variants command stays as the reusable cure (any tool/migration). Command = cure, guardrail = smoke detector.
  • The check is scoped to the import's records and skipped for every other model, so zero overhead on non-product imports. Connection errors in the check are swallowed — it never breaks an otherwise-successful import.

Changes

  • variant_manager.check_missing_variants_after_import(config, model, id_map, fix)
  • run_import: new fix_missing_variants param; calls the guardrail before returning.
  • import CLI: --fix-missing-variants flag.
  • 6 unit tests (skip-other-models, empty id_map, no-orphans, warn-only, fix, swallow-connection-error); docs section.

After every product.template import, automatically check the just-imported
templates and warn if any have no variant (Odoo's load() doesn't auto-create
them) — so the problem surfaces at import time, not in production. The check is
scoped to the import's records and skipped for other models.

- variant_manager.check_missing_variants_after_import(config, model, id_map, fix).
- run_import: new fix_missing_variants param; calls the guardrail before returning.
- import CLI: --fix-missing-variants flag (create inline instead of only warning).
- 6 unit tests + docs.

Follow-up to #188 / PR #201.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a post-import guardrail for product.template imports to warn users or automatically create missing default variants via the new --fix-missing-variants flag, addressing an issue where Odoo's load() does not auto-create them. Feedback suggests running this guardrail even during partial import successes (whenever id_map is populated) rather than only on full success, and batching the database search in chunks of 2000 to prevent extremely large RPC payloads during large-scale imports.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +124 to +133
db_ids = list(id_map.values())
try:
if isinstance(config, dict):
connection: Any = conf_lib.get_connection_from_dict(config)
else:
connection = conf_lib.get_connection_from_config(config_file=config)
template_obj = connection.get_model("product.template")
orphan_ids = template_obj.search(
[("id", "in", db_ids), ("product_variant_count", "=", 0)]
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For large-scale imports, id_map can contain tens of thousands of records. Passing all of them at once in an in query can lead to extremely large RPC payloads (which might exceed web server limits like Nginx's client_max_body_size) or cause slow database queries in Odoo. Batching the search query in chunks of 2000 ensures robust execution even for very large imports.

Suggested change
db_ids = list(id_map.values())
try:
if isinstance(config, dict):
connection: Any = conf_lib.get_connection_from_dict(config)
else:
connection = conf_lib.get_connection_from_config(config_file=config)
template_obj = connection.get_model("product.template")
orphan_ids = template_obj.search(
[("id", "in", db_ids), ("product_variant_count", "=", 0)]
)
db_ids = list(id_map.values())
try:
if isinstance(config, dict):
connection: Any = conf_lib.get_connection_from_dict(config)
else:
connection = conf_lib.get_connection_from_config(config_file=config)
template_obj = connection.get_model("product.template")
# Batch the search to prevent huge RPC payloads and SQL limits on large imports
orphan_ids = []
for i in range(0, len(db_ids), 2000):
batch_ids = db_ids[i : i + 2000]
batch_orphans = template_obj.search(
[("id", "in", batch_ids), ("product_variant_count", "=", 0)]
)
orphan_ids.extend(batch_orphans)

Comment thread src/fluvo/importer.py Outdated
# Guardrail (#188): Odoo's load() does not auto-create default variants, so a
# product.template import can silently leave templates unusable. Warn (or fix
# with --fix-missing-variants) for the just-imported templates.
if is_truly_successful:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Currently, the guardrail is only executed if the import is fully successful (is_truly_successful). However, if an import is partially successful (e.g., 9,999 records succeed and 1 fails), the successfully imported templates are still written to the database and might still lack variants. Restricting this check to is_truly_successful means the guardrail is skipped for these successfully imported records. Changing the condition to if id_map: ensures the guardrail runs for any successfully imported templates, even in the presence of partial failures.

Suggested change
if is_truly_successful:
if id_map:

- Batch the orphan-template search in chunks of 2000 ids to avoid oversized RPC
  payloads / slow SQL on large imports (the issue had 22k templates). + test.
- Run the guardrail whenever records were imported (if id_map), not only on a
  fully-successful import — partially-successful imports still write templates
  that may lack variants.
@bosd

bosd commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

Both applied — good scalability/correctness catches:

  • Batched search: the orphan check now searches in chunks of 2000 ids, so a large import (the issue had ~22k templates) won't build an oversized in RPC payload or slow SQL. Added a test asserting the chunking (4500 ids -> 3 calls).
  • Partial imports: changed the guard from is_truly_successful to if id_map: — a partially-successful import still writes templates that may lack variants, so the check should run on them too.

@bosd

bosd commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an automatic guardrail during product.template imports to detect and optionally fix imported templates that end up without any variants. It adds a --fix-missing-variants CLI option, integrates the check into the import orchestration, and provides comprehensive unit tests. Feedback on the changes highlights that while the initial search for orphan templates is batched, the subsequent call to run_create_missing_variants with the entire list of orphan IDs is not, which could lead to oversized RPC payloads or timeouts for large imports. It is recommended to batch these calls as well.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +150 to +155
if fix:
log.warning(
f"{len(orphan_ids)} imported product template(s) have no variants; "
"creating the missing default variants (--fix-missing-variants)."
)
run_create_missing_variants(config, domain=[("id", "in", orphan_ids)])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

While the initial search for orphan_ids is carefully batched in chunks of 2000 to avoid oversized RPC payloads and slow SQL queries, passing the entire orphan_ids list directly to run_create_missing_variants via domain=[('id', 'in', orphan_ids)] defeats this safeguard.\n\nInside run_create_missing_variants, Odoo is queried again using the provided domain without any batching. For large imports with thousands of templates lacking variants, this unbatched search can trigger oversized RPC payloads or database timeouts.\n\nTo preserve the batching safeguard, consider chunking the calls to run_create_missing_variants when fixing the missing variants.

    if fix:\n        log.warning(\n            f'{len(orphan_ids)} imported product template(s) have no variants; '\n            'creating the missing default variants (--fix-missing-variants).'\n        )\n        for i in range(0, len(orphan_ids), 2000):\n            run_create_missing_variants(config, domain=[('id', 'in', orphan_ids[i : i + 2000])])

The fix path passed all orphan_ids to run_create_missing_variants in one domain,
undoing the batched-search safeguard. Chunk the fix calls in 2000s too.
@bosd

bosd commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

Good catch — the fix path now chunks too. --fix-missing-variants calls run_create_missing_variants per 2000-id slice, so the no-oversized-payload safeguard the search applies isn't undone by an unbatched create domain.

@bosd

bosd commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a post-import guardrail for product.template imports to warn or automatically fix missing variants, including a new --fix-missing-variants CLI option, corresponding documentation, and comprehensive unit tests. Feedback on the changes suggests optimizing the guardrail check by restricting its execution in importer.py to only run when the model is product.template to avoid unnecessary dynamic imports. Additionally, it is recommended to deduplicate database IDs using a set to prevent redundant queries, and to avoid chunking the call to run_create_missing_variants in a loop, as it already handles batching internally and doing so repeatedly causes inefficient connection overhead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/fluvo/importer.py Outdated
Comment on lines +545 to +550
if id_map:
from .lib.actions.variant_manager import check_missing_variants_after_import

check_missing_variants_after_import(
config, model, id_map, fix=fix_missing_variants
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To avoid unnecessary dynamic imports and function calls for non-product template imports, we can restrict this guardrail check to only run when the model is indeed product.template.

Suggested change
if id_map:
from .lib.actions.variant_manager import check_missing_variants_after_import
check_missing_variants_after_import(
config, model, id_map, fix=fix_missing_variants
)
if model == "product.template" and id_map:
from .lib.actions.variant_manager import check_missing_variants_after_import
check_missing_variants_after_import(
config, model, id_map, fix=fix_missing_variants
)

if model != "product.template" or not id_map:
return 0

db_ids = list(id_map.values())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If the id_map contains duplicate database IDs (which can happen if multiple external IDs map to the same Odoo record), converting the values directly to a list will result in redundant queries. Converting to a set first ensures each database ID is only queried once.

Suggested change
db_ids = list(id_map.values())
db_ids = list(set(id_map.values()))

Comment on lines +155 to +160
# Chunk the fix calls too, so the no-oversized-payload safeguard that the
# search above applies isn't undone by an unbatched create domain.
for i in range(0, len(orphan_ids), 2000):
run_create_missing_variants(
config, domain=[("id", "in", orphan_ids[i : i + 2000])]
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Calling run_create_missing_variants in a loop chunked by 2000 is inefficient because it establishes a new Odoo connection and performs a redundant search on every invocation. Since run_create_missing_variants already internally batches the creation of variants in chunks of batch_size (default 200), there is no risk of oversized payloads during the creation phase. Calling it once with the entire list of orphan_ids avoids multiple connection roundtrips and significantly improves performance.

Suggested change
# Chunk the fix calls too, so the no-oversized-payload safeguard that the
# search above applies isn't undone by an unbatched create domain.
for i in range(0, len(orphan_ids), 2000):
run_create_missing_variants(
config, domain=[("id", "in", orphan_ids[i : i + 2000])]
)
run_create_missing_variants(config, domain=[("id", "in", orphan_ids)])

…plate (review)

- Extract _create_default_variants (batched create + Odoo<14 fallback), shared by
  run_create_missing_variants and the guardrail.
- Guardrail fix path now creates directly from the orphan_ids it already found —
  no redundant re-search (and no oversized id-domain), resolving the tension
  between the two review rounds.
- Dedup id_map values (list(set(...))) so duplicate db ids aren't queried twice.
- Gate the guardrail call on model == 'product.template' to skip the import/call
  entirely for other models.
@bosd

bosd commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

All three addressed:

  • Gate on product.template: the guardrail call is now if model == "product.template" and id_map, so non-product imports skip the dynamic import + call entirely.
  • Dedup ids: db_ids = list(set(id_map.values())).
  • The fix-path batching: this one conflicted with the previous round (which asked me to chunk the fix calls to avoid an oversized search domain). Rather than re-search at all, the guardrail now creates directly from the orphan_ids it already found, via a shared _create_default_variants helper that batches the creates internally. That removes the redundant re-search/re-connect and avoids the oversized id-domain — satisfying both rounds.

@bosd bosd merged commit 7ef13b2 into master Jun 11, 2026
24 checks passed
@bosd bosd deleted the feat/variant-guardrail branch June 11, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant