fix: derive the extends merge policy from each key's validator (#58)

* fix: derive the extends merge policy from each key's validator

resolve_extends runs ahead of validate(), so a merge that normalizes a value
decides what the gate then sees. The policy was a hand-maintained pair of key
sets, and it had drifted from the gate in both directions.

It laundered invalid input into valid: ulimits has no list form and the gate
refuses one, but _merge_map coerced `ulimits: ["nofile=2"]` into
{"nofile": "2"} before the gate ran. Same for every list-only key given a bare
scalar -- cap_add, devices, security_opt, group_add, volumes, secrets, configs
all refuse a scalar standalone and all accepted one via extends.

And it refused valid input: extra_hosts has a list form, accepted standalone,
but rejected on merge.

One rule now: a merge may normalize a value only through a form that key
actually has. Registry keys run spec.validate on both sides before spec.merge,
so the merge's accepted forms ARE the gate's, by construction. Structural
normalizers are narrowed to the forms Compose defines -- scalar only for
tmpfs/env_file, list form only for environment/extra_hosts/depends_on.

extra_hosts gets a real normalizer: its list form is colon-separated, so
pairs_to_mapping (which splits on '=') would have produced a single
{'host:ip': None} key. Splits on the first colon, so IPv6 survives.

A parametrized test pins the invariant for every mergeable key: a form the gate
refuses standalone must be refused through extends too. Verified to go red
without the fix.

* fix: attribute merge errors to the right service; make the invariant test real

The base-side spec.validate reported the base's malformed value under the
extending service's name. _merge now takes base_name and attributes each side's
failure to the service the value belongs to.

The parametrized invariant test was 10/11 vacuous: it paired the list-only keys
with a dict, which the old code already refused, instead of the bare scalar that
actually laundered. It now uses the laundering shape, covers all 17 mergeable
keys, and asserts its own completeness so a new key cannot be added without a
case. Against the pre-fix code it goes red on nine keys; before, only one.

* docs: correct the normalization list and attribution claim; pin structural attribution

The Extends section claimed 'nothing else is coerced' while omitting list-form
labels/annotations, which are legitimately normalized through their own
validator-approved list form.

The 'error always names the owning service' claim had one reachable exception:
spec.merge only receives the extending service's name, so a null ulimits in the
base is reported against the child. Stated rather than overclaimed.

Adds tests pinning base-side attribution on both structural paths (_as_mapping,
_as_concat_list), which were correct but unpinned.

Author: lesnik512

architecture/supported-subset.md

@@ -234,26 +234,42 @@ slot (`architecture/glossary.md`).
     `entrypoint` (argv replaced wholesale, never concatenated) — also
     covers unknown keys, which `validate()` then rejects downstream exactly
     as it would without `extends`.
-  - **Normalization before merge:** list-form `environment` and list-form
-    `depends_on` (a bare service-name list) are normalized to mappings
-    before the mapping-merge; scalar-form `tmpfs`/`env_file` are normalized
-    to a one-element list before the concatenation.
+  - **Normalization before merge — a merge never widens the gate.** A merged
+    side is normalized only through a form that key *actually has*: list-form
+    `environment`, `labels`, `annotations`, `extra_hosts` and `depends_on`
+    become mappings before the mapping-merge, and scalar-form `tmpfs`/`env_file`
+    become a one-element list before the concatenation. Nothing else is
+    coerced. This matters because `resolve_extends` runs **ahead of**
+    `validate()`: normalizing a form a key does not have would hand the gate a
+    document it would have
+    refused standalone. So `ulimits` and `healthcheck` (no list form) refuse a
+    list, and every list-only key (`cap_add`, `cap_drop`, `security_opt`,
+    `devices`, `group_add`, `volumes`, `secrets`, `configs`) refuses a bare
+    scalar — exactly as each does outside `extends`. For a **registry** key the
+    accepted forms are read from the key's own validator (`spec.validate` runs
+    on both sides before `spec.merge`), so those cannot drift from the gate by
+    construction. The **structural** keys have no `KeySpec`, so their accepted
+    forms are declared in `extends.py` (`_STRUCTURAL_*`, `_SCALAR_FORM_KEYS`)
+    and kept honest by a test asserting, for every mergeable key, that a form
+    the gate refuses standalone is refused through `extends` too.
+    `extra_hosts`' list form is `host:ip` — colon-separated, split on the
+    *first* colon so an IPv6 address survives (`myhost:::1` →
+    `{myhost: ::1}`).
 - **Refused loudly:** cross-file `extends: {file: ..., service: ...}`; a
   bare-string (or any other non-mapping) `extends`; an unrecognized key
   under `extends` other than `service`; a non-string `service`; a `service`
-  naming one that doesn't exist; and a merge across incompatible forms (a
-  mapping-merge key that is neither a mapping nor list-form
-  `environment`/`depends_on`; a sequence-concatenate key that is neither a
-  list nor a scalar string).
-- **Divergences from Compose:** `environment`/`depends_on` accept list form
-  directly in `extends.py`'s own merge (`_as_mapping`); `extra_hosts`/
-  `healthcheck` do not — list form on a merged side is refused for those two.
-  `labels`/`annotations`/`ulimits` instead route through their `SERVICE_KEYS`
-  merge policy (`_merge_map`), which *does* coerce list form, via the same
-  `pairs_to_mapping` normalizer `environment` uses — both paths reject a
-  non-string list element identically, since it's the one function they
-  share. Short-form `volumes` are concatenated rather than merged by target
-  path; podman resolves duplicate mounts at run time. Referenced resources
+  naming one that doesn't exist; and a merge across incompatible forms — a
+  side whose form that key does not have. A registry key raises with its own
+  validator's message (`'cap_add' must be a list`), the same message the value
+  produces standalone; a structural key raises `cannot merge '<key>' across
+  incompatible forms`. A malformed *form* is reported against the service the
+  value belongs to — the base, when it is the base's value at fault, not the
+  service extending it. (One exception: an explicit `null` on a mergeable key
+  is valid standalone but refused on merge, and a null in the base is reported
+  against the extending service.)
+- **Divergences from Compose:** short-form `volumes` are concatenated rather
+  than merged by target path; podman resolves duplicate mounts at run time.
+  Referenced resources
   (top-level `volumes`, `networks`, `secrets`, `configs`) are not
   auto-imported — as in Compose, the extending service must declare what it
   needs.

compose2pod/extends.py

@@ -3,16 +3,23 @@
 from typing import Any
 
 from compose2pod.exceptions import UnsupportedComposeError
-from compose2pod.keys import SERVICE_KEYS, concat_list, pairs_to_mapping
+from compose2pod.keys import SERVICE_KEYS, as_list, pairs_to_mapping
 
 
 # Merge policy for keys with a SERVICE_KEYS KeySpec comes from spec.merge (see
 # _merge below); these two sets cover only the remaining structural keys, which
-# have no KeySpec. Unifying structural-key merge policy is deferred — see
-# decisions/2026-07-12-reject-structural-key-registry.md's revisit trigger.
+# have no KeySpec. Both categories obey one rule: a merge may normalize a value
+# only through a form that key actually has, so it can never accept a shape the
+# gate would reject standalone.
 _STRUCTURAL_MERGE_KEYS = {"environment", "extra_hosts", "healthcheck", "depends_on"}
 _STRUCTURAL_CONCAT_KEYS = {"secrets", "configs", "volumes", "tmpfs", "env_file"}
 
+# The only concat keys Compose gives a bare-string form. The gate accepts a
+# scalar for exactly these two and requires a list for the rest, so only these
+# two may be normalized scalar -> [scalar] on a merged side; normalizing any
+# other would accept a shape the gate refuses standalone.
+_SCALAR_FORM_KEYS = {"tmpfs", "env_file"}
+
 
 def _extends_target(name: str, ext: Any) -> str:  # noqa: ANN401 - Compose values are untyped
     """Return the referenced service name, after refusing cross-file and malformed forms."""
@@ -39,19 +46,54 @@ def _extends_target(name: str, ext: Any) -> str:  # noqa: ANN401 - Compose value
     return service
 
 
+def _extra_hosts_to_mapping(name: str, value: list[Any]) -> dict[str, Any]:
+    """Normalize list-form `extra_hosts` ('host:ip') to a mapping.
+
+    Separated by a colon, not '=', so `pairs_to_mapping` would mangle the whole
+    entry into a single `{'host:ip': None}` key. Split on the *first* colon only:
+    an IPv6 address is itself full of them (`myhost:::1` -> `{'myhost': '::1'}`).
+    """
+    result: dict[str, Any] = {}
+    for item in value:
+        if not isinstance(item, str) or ":" not in item:
+            msg = f"service {name!r}: extra_hosts entries must be 'host:ip' strings"
+            raise UnsupportedComposeError(msg)
+        host, _sep, address = item.partition(":")
+        result[host] = address
+    return result
+
+
+def _as_concat_list(name: str, key: str, value: Any) -> list[Any]:  # noqa: ANN401 - Compose values are untyped
+    """Normalize a structural concat key's value to a list, without widening the gate.
+
+    `keys.as_list` turns a bare string into a one-element list, which is right
+    for `tmpfs`/`env_file` (Compose gives them a scalar form and the gate accepts
+    one) and wrong for `volumes`/`secrets`/`configs`, where the gate requires a
+    list -- normalizing there would let `extends` accept a scalar the same
+    document would be refused for standalone.
+    """
+    if isinstance(value, str) and key not in _SCALAR_FORM_KEYS:
+        msg = f"service {name!r}: '{key}' must be a list"
+        raise UnsupportedComposeError(msg)
+    return as_list(name, key, value)
+
+
 def _as_mapping(name: str, key: str, value: Any) -> dict[str, Any]:  # noqa: ANN401 - Compose values are untyped
     """Normalize a structural mapping-merge key's value to a mapping.
 
-    Deliberately stricter than `keys.pairs_to_mapping`: list form is accepted
-    only for `environment` and `depends_on`, the two keys Compose actually
-    defines a list form for. `extra_hosts`/`healthcheck` in list form on a
-    merged side are refused as an incompatible form rather than coerced.
+    List form is accepted for exactly the keys Compose defines one for --
+    `environment`, `extra_hosts`, `depends_on` -- each through the normalizer
+    that key's own list form actually needs. `healthcheck` has no list form, so
+    a list is refused rather than coerced: a merge must not accept a shape the
+    gate would reject standalone (see `_merge`).
     """
     if isinstance(value, dict):
         return value
     if isinstance(value, list):
         if key == "environment":
             return pairs_to_mapping(name, key, value)
+        if key == "extra_hosts":
+            return _extra_hosts_to_mapping(name, value)
         if key == "depends_on":
             for dep in value:
                 if not isinstance(dep, str):
@@ -62,17 +104,32 @@ def _as_mapping(name: str, key: str, value: Any) -> dict[str, Any]:  # noqa: ANN
     raise UnsupportedComposeError(msg)
 
 
-def _merge(base: dict[str, Any], local: dict[str, Any], name: str) -> dict[str, Any]:
-    """Merge `local` onto `base` per key category: mapping-merge, sequence-concat, else override."""
+def _merge(base: dict[str, Any], local: dict[str, Any], name: str, base_name: str) -> dict[str, Any]:
+    """Merge `local` onto `base` per key category: mapping-merge, sequence-concat, else override.
+
+    `name` is the extending service, `base_name` the one it extends: each side's
+    failures are reported against the service the offending value actually
+    belongs to.
+    """
     merged: dict[str, Any] = dict(base)
     for key, local_val in local.items():
         spec = SERVICE_KEYS.get(key)
         if key in base and spec is not None and spec.merge is not None:
+            # A merge must never *widen* what the gate accepts. `resolve_extends`
+            # runs ahead of `validate()`, so a normalizing merge (list -> mapping)
+            # can launder a form the key does not have into one it does:
+            # `ulimits: ["nofile=2"]` is refused standalone, but coercing it here
+            # would produce a valid-looking `{"nofile": "2"}` that then sails
+            # through the gate. Checking each side with the key's own validator
+            # first derives the merge's accepted forms from the gate's, so the
+            # two cannot drift apart.
+            spec.validate(base_name, key, base[key])
+            spec.validate(name, key, local_val)
             merged[key] = spec.merge(name, key, base[key], local_val)
         elif key in base and key in _STRUCTURAL_MERGE_KEYS:
-            merged[key] = {**_as_mapping(name, key, base[key]), **_as_mapping(name, key, local_val)}
+            merged[key] = {**_as_mapping(base_name, key, base[key]), **_as_mapping(name, key, local_val)}
         elif key in base and key in _STRUCTURAL_CONCAT_KEYS:
-            merged[key] = concat_list(name, key, base[key], local_val)
+            merged[key] = _as_concat_list(base_name, key, base[key]) + _as_concat_list(name, key, local_val)
         else:
             merged[key] = local_val
     return merged
@@ -115,7 +172,7 @@ def resolve(name: str) -> Any:  # noqa: ANN401 - Compose values are untyped
         if not isinstance(base, dict):
             resolved[name] = local
             return local
-        merged = _merge(base, local, name)
+        merged = _merge(base, local, name, base_name)
         resolved[name] = merged
         return merged
 

planning/changes/2026-07-14.05-extends-merge-policy.md

@@ -0,0 +1,126 @@
+---
+summary: An extends merge may normalize a value only through a form that key actually has, derived from the key's own validator, so a merge can no longer launder a shape the gate refuses standalone (ulimits/cap_add/volumes scalars and lists) nor refuse one it accepts (list-form extra_hosts).
+---
+
+# Design: Derive the extends merge policy from each key's validator
+
+## Summary
+
+`resolve_extends` runs **ahead of** `validate()`. A merge that *normalizes* a
+value (list → mapping, scalar → list) therefore decides what the gate will
+subsequently see — and if it normalizes a form the key does not actually have,
+it turns an invalid document into a valid one. That is what it was doing.
+
+One rule now governs every mergeable key: **a merge may normalize a value only
+through a form that key actually has.** For registry keys the accepted forms are
+derived from the key's own validator, so those cannot drift from the gate by
+construction. Structural keys have no `KeySpec`, so theirs stay declared in
+`extends.py` — kept honest by a test that asserts the invariant for *every*
+mergeable key and fails if a new one is added without a case.
+
+## Motivation
+
+The merge policy had drifted out of step with the gate in *both* directions.
+
+**It laundered invalid input into valid** (the serious half). `ulimits` has no
+list form in Compose, and the gate refuses one:
+
+```
+$ validate({"app": {"ulimits": ["nofile=2"]}})     -> 'ulimits' must be a mapping
+```
+
+But arriving through `extends`, `_merge_map` coerced that list into
+`{"nofile": "2"}` *before* the gate ran, and the document sailed through. The
+same held for every list-only key given a bare scalar — `cap_add`, `devices`,
+`security_opt`, `group_add`, `volumes`, `secrets`, `configs` all refuse a scalar
+standalone, and all silently accepted one via `extends`.
+
+**And it refused valid input.** `extra_hosts` *does* have a list form
+(`- "host:1.2.3.4"`), accepted standalone — but a merge rejected it as an
+"incompatible form".
+
+The asymmetry was not cosmetic: it was two bugs pointing in opposite directions,
+both caused by the policy being an enumeration maintained by hand instead of
+being derived from what each key actually accepts.
+
+## Design
+
+**Registry keys** (`SERVICE_KEYS`): `_merge` runs `spec.validate` on *both* sides
+before `spec.merge`. The merge's accepted forms are therefore exactly the gate's,
+by construction. `ulimits: [...]` and `cap_add: "SCALAR"` now raise — with the
+validator's own message (`'cap_add' must be a list`), which is also the message
+the same value produces standalone.
+
+**Structural keys** (no `KeySpec`): the two normalizers are narrowed to the forms
+Compose actually defines.
+
+- `_as_concat_list` normalizes `scalar → [scalar]` only for `tmpfs` and
+  `env_file` — the only concat keys with a bare-string form, and the only two the
+  gate accepts a scalar for. `volumes`/`secrets`/`configs` must stay lists.
+- `_as_mapping` accepts list form for `environment`, `extra_hosts`, and
+  `depends_on` — each through the normalizer that key's list form actually needs.
+  `healthcheck` has no list form, so a list is still refused.
+
+`extra_hosts` gets a real normalizer: its list form is `host:ip`, **colon**-
+separated, so routing it through `pairs_to_mapping` (which splits on `=`) would
+have silently produced a single `{'host:ip': None}` key. It splits on the *first*
+colon only, so an IPv6 address survives (`myhost:::1` → `{'myhost': '::1'}`).
+
+No structural-key registry — `decisions/2026-07-12` stands. This derives policy
+from validators that already exist; it does not build a dispatch table.
+
+## The invariant, pinned
+
+A parametrized test asserts, for **every** mergeable key, that a form the gate
+refuses standalone is also refused through `extends`, using the shape that
+actually laundered (a bare scalar for the list-only keys, a list for the
+mapping-only ones) rather than merely any invalid value. A companion test
+asserts the table covers every mergeable key, so a new key cannot be added
+without a case — the drift this change exists to end.
+
+Run against the pre-fix `extends.py` it goes red on nine keys (`cap_add`,
+`cap_drop`, `configs`, `devices`, `group_add`, `secrets`, `security_opt`,
+`ulimits`, `volumes`), which is the laundering it now prevents.
+
+## Non-goals
+
+- Not validating the whole service pre-merge (the heavier structural option).
+  Deriving per-key from the validator closes the class with a much smaller
+  change, and the invariant test guards the seam.
+- Not changing what the gate itself accepts. Every shape valid standalone before
+  this change is still valid; every shape invalid standalone is now *also*
+  invalid through `extends`.
+
+## Behavior change
+
+Three previously-accepted merges now raise, all of which produced documents the
+gate would have refused standalone:
+
+- `ulimits` in list form on a merged side
+- any list-only key (`cap_add`, `devices`, `security_opt`, `group_add`,
+  `volumes`, `secrets`, `configs`) given a bare scalar on a merged side
+- (unchanged, still refused) `healthcheck` in list form
+
+One previously-refused merge now works: list-form `extra_hosts`.
+
+Registry-key merge errors now come from the key's own validator (`'cap_add'
+must be a list`) rather than the vaguer `cannot merge 'cap_add' across
+incompatible forms`, matching what the same value yields standalone. Structural
+keys keep the `cannot merge ...` message.
+
+Merge errors also now name the service the offending value belongs to: a
+malformed value in the *base* is reported against the base, not against the
+service extending it.
+
+## Testing
+
+`just test-ci` at 100%. The invariant test above is the centrepiece; new tests
+cover the `extra_hosts` list form (including IPv6 colon-safety and a
+missing-colon refusal) and the scalar-form narrowing.
+
+## Risk
+
+- **A user relying on the laundering** — e.g. a base with `cap_add: "NET_ADMIN"`
+  (scalar) — now gets an error. This is the intended correction: that document
+  was already invalid without `extends`, and the tool was accepting it only by
+  accident of merge order. The error names the key and the required form.