feat(render): switch to crossplane internal render via crossplane/cli#326
Open
jcogilvie wants to merge 14 commits into
Open
feat(render): switch to crossplane internal render via crossplane/cli#326jcogilvie wants to merge 14 commits into
jcogilvie wants to merge 14 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates crossplane-diff from the deprecated in-process render.Render() path to the Crossplane CLI engine that drives crossplane internal render, aiming to make diffs reflect the real composite reconciler behavior seen in a live cluster.
Changes:
- Introduces an engine-backed render implementation (
EngineRenderFn) with lifecycle management (setup, function runtimes, render, cleanup) and partial-output-on-fatal handling. - Updates requirements iteration to use the new flat
RequiredResources []*ResourceSelectorcontract and adapts processors/tests accordingly. - Bumps Crossplane-related dependencies and updates E2E/integration fixtures (schemas, CRDs, expected outputs) to match reconciler-faithful render output.
Reviewed changes
Copilot reviewed 83 out of 84 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/e2e/xr_basic_test.go | Switches condition helpers to Crossplane apis/v2 types. |
| test/e2e/xr_advanced_test.go | Switches condition helpers to Crossplane apis/v2 types. |
| test/e2e/main_test.go | Updates pkg API import path to crossplane/apis/v2. |
| test/e2e/comp_test.go | Updates condition helpers and imports to apis/v2. |
| test/e2e/claim_test.go | Updates condition helpers and imports to apis/v2. |
| go.mod | Bumps Crossplane/CLI deps and related transitive modules. |
| cmd/diff/xr.go | Swaps crank loader import to crossplane/cli and removes global render mutex wiring. |
| cmd/diff/types/types.go | Updates apiextensions import path to crossplane/apis/v2. |
| cmd/diff/testutils/mocks.go | Updates imports and extends mock DefinitionClient / render output types. |
| cmd/diff/testutils/mock_builder.go | Updates imports and SecretReference type to core/v2. |
| cmd/diff/testdata/diff/crds/xparentresources.ns.nested.example.org.yaml | Fixture schema updates for spec.crossplane.* and observedGeneration. |
| cmd/diff/testdata/diff/crds/xparentnopclaims.claimnested.diff.example.org.yaml | Adds observedGeneration to conditions schema. |
| cmd/diff/testdata/diff/crds/xnopresource-v2withv1paths-crd.yaml | Adds resourceRefs + observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xnopresource-ns-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xnopresource-legacycluster-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xnopresource-cluster-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xenvresource-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xdownstreamresource-ns-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xdownstreamresource-legacycluster-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xdownstreamresource-cluster-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xdownstreamenvresource-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xdefaultresource-ns-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xconcurrenttest-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/xchildresources.ns.nested.example.org.yaml | Fixture schema updates for spec.crossplane.* and observedGeneration. |
| cmd/diff/testdata/diff/crds/xchildnopclaims.claimnested.diff.example.org.yaml | Adds observedGeneration to conditions schema. |
| cmd/diff/testdata/diff/crds/xapimigrateresource-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/diff/crds/parentnopclaims.claimnested.diff.example.org.yaml | Adds observedGeneration to conditions schema. |
| cmd/diff/testdata/diff/crds/nopclaim-crd.yaml | Adds observedGeneration to conditions schema. |
| cmd/diff/testdata/diff/crds/clusternopresources.nop.crossplane.io.yaml | Adds observedGeneration to conditions schema. |
| cmd/diff/testdata/comp/crds/xparentresource-ns-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/comp/crds/xnopresource-ns-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/comp/crds/xnopresource-legacycluster-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/comp/crds/xnopresource-cluster-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/comp/crds/xenvresource-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/comp/crds/xdownstreamresource-ns-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/comp/crds/xdownstreamresource-legacycluster-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/comp/crds/xdownstreamresource-cluster-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/comp/crds/xdownstreamenvresource-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/testdata/comp/crds/xdefaultresource-ns-crd.yaml | Adds observedGeneration to fixture CRD schema. |
| cmd/diff/serial/serial.go | Removes old global-mutex-based render serialization helper (deleted). |
| cmd/diff/serial/serial_test.go | Removes tests for deleted serial wrapper (deleted). |
| cmd/diff/diffprocessor/schema_validator.go | Switches validate imports to CLI and changes XR validation to strip spec.crossplane on a deep copy. |
| cmd/diff/diffprocessor/resource_manager.go | Updates crank imports, fixes ownerRef UID overwrite behavior, and dedupes duplicate ownerRefs. |
| cmd/diff/diffprocessor/resource_manager_test.go | Updates crank resource import path to CLI. |
| cmd/diff/diffprocessor/requirements_provider.go | Reworks requirements resolution to flat selectors (ResolveSelectors) and updates SDK proto import. |
| cmd/diff/diffprocessor/requirements_provider_test.go | Updates tests for ResolveSelectors and new selector contract. |
| cmd/diff/diffprocessor/render_engine.go | Adds engine-backed render implementation and stderr-capturing local engine. |
| cmd/diff/diffprocessor/render_engine_test.go | Adds unit tests for engine lifecycle, cleanup idempotence, and serialization. |
| cmd/diff/diffprocessor/processor_config.go | Retypes RenderFunc to new RenderFn and removes render mutex option. |
| cmd/diff/diffprocessor/function_provider.go | Updates imports to Crossplane apis/v2. |
| cmd/diff/diffprocessor/function_provider_test.go | Updates imports to Crossplane apis/v2. |
| cmd/diff/diffprocessor/diff_processor.go | Uses engine-backed default render fn, adds engine cleanup, updates claim backing XR logic, schema plumbing, and requirements iteration. |
| cmd/diff/diffprocessor/diff_processor_test.go | Updates render mocks and tests to new render types and requirements shape; adds schema plumbing assertion test. |
| cmd/diff/diffprocessor/diff_calculator.go | Updates render types, strips ownerReferences from SSA dry-run input to avoid multi-controller SSA failure modes. |
| cmd/diff/diffprocessor/diff_calculator_test.go | Updates tests to use render.CompositionOutputs. |
| cmd/diff/diffprocessor/comp_processor.go | Updates imports and avoids dead render fn defaulting in comp diff processor. |
| cmd/diff/diffprocessor/comp_processor_test.go | Updates render mocks to new render types. |
| cmd/diff/diff_test.go | Updates loader imports to CLI and Crossplane apis/v2. |
| cmd/diff/diff_it_utils_test.go | Adds helper to locate local crossplane binary and set render env var for integration tests. |
| cmd/diff/diff_integration_test.go | Uses local crossplane binary for internal render and updates expected patterns + skipped test annotations. |
| cmd/diff/comp.go | Swaps crank loader import to crossplane/cli and removes global render mutex wiring. |
| cmd/diff/cmd_utils.go | Removes now-unused global render mutex and updates loader import to CLI. |
| cmd/diff/client/kubernetes/schema_client.go | Updates apiextensions imports to Crossplane apis/v2. |
| cmd/diff/client/crossplane/resource_tree_client.go | Swaps crank resource/xrm imports to CLI equivalents. |
| cmd/diff/client/crossplane/function_client.go | Updates imports to Crossplane apis/v2. |
| cmd/diff/client/crossplane/function_client_test.go | Updates imports to Crossplane apis/v2. |
| cmd/diff/client/crossplane/definition_client.go | Adds GetCompositeSchema API and implements schema detection via XRD spec.scope. |
| cmd/diff/client/crossplane/definition_client_test.go | Adds unit coverage for GetCompositeSchema. |
| cmd/diff/client/crossplane/credential_client.go | Updates apiextensions imports to Crossplane apis/v2. |
| cmd/diff/client/crossplane/credential_client_test.go | Updates apiextensions imports to Crossplane apis/v2. |
| cmd/diff/client/crossplane/composition_revision_client.go | Updates apiextensions imports to Crossplane apis/v2. |
| cmd/diff/client/crossplane/composition_revision_client_test.go | Updates apiextensions imports to Crossplane apis/v2. |
| cmd/diff/client/crossplane/composition_client.go | Updates apiextensions imports to Crossplane apis/v2. |
| cmd/diff/client/crossplane/composition_client_test.go | Updates apiextensions imports to Crossplane apis/v2. |
| cmd/diff/client/core/core.go | Swaps crank xrm import to CLI and updates pkg import to apis/v2. |
| .serena/project.yml | Adds additional_workspace_folders config entry. |
| .requirements/20260527T173616Z_scope_aware_composite_render/REQUIREMENTS.md | Adds working requirements doc for scope-aware schema handling. |
| .requirements/20260504T230404Z_render_engine_migration/REQUIREMENTS.md | Adds working requirements doc for render engine migration. |
| .requirements/20260504T230404Z_render_engine_migration/upstream-issue-*.md | Adds upstream issue drafts / notes supporting the migration. |
Comments suppressed due to low confidence (2)
cmd/diff/diffprocessor/requirements_provider.go:48
- The resourceCache comment claims the cache key is apiVersion+kind+name, but getCachedResource uses dt.MakeDiffKey(apiVersion, kind, namespace, name). This comment should include namespace to reflect the actual (and important) behavior.
cmd/diff/diffprocessor/requirements_provider_test.go:159 - This test file ends with a dangling comment block describing TestRequirementsProvider_NamespaceCollision, but the actual test was removed. Either reintroduce the test for the updated ResolveSelectors API or remove the stale comment to avoid confusion.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jcogilvie
commented
Jun 5, 2026
jcogilvie
added a commit
that referenced
this pull request
Jun 5, 2026
Mechanical fixes from the PR review pass: - diffprocessor/requirements_provider.go: drop the orphaned function-stub comments left over from ProvideRequirements, drop the unused `stepName` arg threaded through processSelector / processNameSelector / processLabelSelector / resolveNamespace (callers now always passed ""), and update the resourceCache doc to mention namespace (which has been part of the cache key since dt.MakeDiffKey took on namespace). - diffprocessor/resource_manager.go: dedupe owner refs via dt.MakeDiffKey instead of an ad-hoc "|"-joined string, matching the rest of the diff layer's keying convention. - diffprocessor/diff_processor.go: switch the err==nil/else block in RenderToStableState to a switch (project style), tighten the generateName comment to make clear "placeholder" is purely a render-side RFC1123-valid filler and the (generated) suffix the user sees is produced independently by diff_formatter.go, and update the schema re-stamp comment to point at crossplane/crossplane#7452 (merged on main; v2.3.1 still defaults the binary to SchemaModern). - diffprocessor/diff_processor_test.go: same #7452 update on the schema- plumbing test's documentation. - diffprocessor/render_engine_test.go: add a header comment justifying why the three lifecycle tests (HappyPath / CleanupIdempotent / Serialization) are procedural rather than table-driven, and switch to t.Context(). - diffprocessor/schema_validator.go: rewrite the spec.crossplane stripping comment so it accurately states that real cluster-derived CRDs declare the subtree (Crossplane's CRD generator emits it) — the gap is in our hand-rolled integration-test CRD fixtures, not in production. - client/crossplane/definition_client.go: bring the GetCompositeSchema doc comments (interface + impl) into sync with the actual implementation, which reads spec.scope rather than the XRD's apiVersion (apiserver conversion can rewrite the apiVersion stamp; spec.scope is preserved). - client/crossplane/definition_client_test.go: drop a duplicated comment line; clarify the "ModernClaim" reference (the claim kind in the test is just the fixture's "ClaimedResource" string — there's no upstream type called ModernClaim). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
jcogilvie
added a commit
to jcogilvie/cli
that referenced
this pull request
Jun 8, 2026
…oth engines Both render engines previously dropped information that downstream tools need: - localRenderEngine piped stderr straight to os.Stderr, so callers couldn't programmatically inspect FATAL pipeline messages. - dockerRenderEngine discarded stderr at the engine layer (the RunContainer return value was assigned to `_`); only on container error did stderr surface, folded into a fmt.Errorf string. - Neither engine implemented the partial-output-on-fatal contract from crossplane/crossplane#7455: when `crossplane internal render` exits with code 3 it populates stdout with a partial RenderResponse so callers can recover output.RequiredResources and iterate, but both engines wrapped the *exec.ExitError / *ContainerExitError and dropped stdout. This change: - Adds ExitCodePipelineFatal (= 3) in cmd/crossplane/render/render.go, documenting the contract. - Adds *ContainerExitError to internal/docker (carries ExitCode + captured stderr) so callers can errors.As against it. RunContainer returns it on non-zero exit instead of an opaque fmt.Errorf. - Captures stderr in localRenderEngine.Render via bytes.Buffer instead of os.Stderr, and surfaces it in the returned error. - Uses RunContainer's stderr return in dockerRenderEngine.Render and surfaces it in the returned error. - On exit-3 with non-empty stdout, both engines parse the partial *RenderResponse and return (rsp, err) — both non-nil — so callers can recover the partial output. - Adds engine_local_test.go (helper-process pattern via TestMain re-exec) and engine_docker_test.go (injectable runContainer fn) covering success, fatal-with-partial, fatal-no-partial, and hard-fail paths. Motivating downstream use case: crossplane-contrib/crossplane-diff currently wraps localRenderEngine to add this behaviour. See crossplane-contrib/crossplane-diff#326 — once this lands the wrapper can be deleted and crossplane-diff can drop its local-binary fallback entirely. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Merged
5 tasks
jcogilvie
added a commit
to jcogilvie/cli
that referenced
this pull request
Jun 8, 2026
The runContainer field on dockerRenderEngine and the env-var-driven TestMain dispatch in engine_local_test.go are both standard Go test patterns, but they're not obvious at a glance — and reviewer questions on the downstream PR (crossplane-contrib/crossplane-diff#326) showed both warrant inline justification. - Expand the doc comment on dockerRenderEngine.runContainer to explain why it exists (hermetic failure-mode coverage without a real Docker daemon) and why a per-instance unexported field is preferable to a package-level var (the latter would race in parallel tests). - Expand the doc comment on engine_local_test.go's envHelperMode + TestMain to explain the helper-process / re-exec idiom: why we re-use the test binary as a stand-in for the `crossplane` binary (no shell scripts, no testdata helper binary, deterministic protobuf payloads), how dispatch works (parent t.Setenv -> child inherits -> TestMain sees the var and exits before m.Run), and what the trade-off is (extra TestMain plumbing in exchange for a self-contained, cross-platform fake binary). No functional changes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
adamwg
pushed a commit
to crossplane/cli
that referenced
this pull request
Jun 9, 2026
…oth engines Both render engines previously dropped information that downstream tools need: - localRenderEngine piped stderr straight to os.Stderr, so callers couldn't programmatically inspect FATAL pipeline messages. - dockerRenderEngine discarded stderr at the engine layer (the RunContainer return value was assigned to `_`); only on container error did stderr surface, folded into a fmt.Errorf string. - Neither engine implemented the partial-output-on-fatal contract from crossplane/crossplane#7455: when `crossplane internal render` exits with code 3 it populates stdout with a partial RenderResponse so callers can recover output.RequiredResources and iterate, but both engines wrapped the *exec.ExitError / *ContainerExitError and dropped stdout. This change: - Adds ExitCodePipelineFatal (= 3) in cmd/crossplane/render/render.go, documenting the contract. - Adds *ContainerExitError to internal/docker (carries ExitCode + captured stderr) so callers can errors.As against it. RunContainer returns it on non-zero exit instead of an opaque fmt.Errorf. - Captures stderr in localRenderEngine.Render via bytes.Buffer instead of os.Stderr, and surfaces it in the returned error. - Uses RunContainer's stderr return in dockerRenderEngine.Render and surfaces it in the returned error. - On exit-3 with non-empty stdout, both engines parse the partial *RenderResponse and return (rsp, err) — both non-nil — so callers can recover the partial output. - Adds engine_local_test.go (helper-process pattern via TestMain re-exec) and engine_docker_test.go (injectable runContainer fn) covering success, fatal-with-partial, fatal-no-partial, and hard-fail paths. Motivating downstream use case: crossplane-contrib/crossplane-diff currently wraps localRenderEngine to add this behaviour. See crossplane-contrib/crossplane-diff#326 — once this lands the wrapper can be deleted and crossplane-diff can drop its local-binary fallback entirely. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com> (cherry picked from commit 1843d3d)
adamwg
pushed a commit
to crossplane/cli
that referenced
this pull request
Jun 9, 2026
The runContainer field on dockerRenderEngine and the env-var-driven TestMain dispatch in engine_local_test.go are both standard Go test patterns, but they're not obvious at a glance — and reviewer questions on the downstream PR (crossplane-contrib/crossplane-diff#326) showed both warrant inline justification. - Expand the doc comment on dockerRenderEngine.runContainer to explain why it exists (hermetic failure-mode coverage without a real Docker daemon) and why a per-instance unexported field is preferable to a package-level var (the latter would race in parallel tests). - Expand the doc comment on engine_local_test.go's envHelperMode + TestMain to explain the helper-process / re-exec idiom: why we re-use the test binary as a stand-in for the `crossplane` binary (no shell scripts, no testdata helper binary, deterministic protobuf payloads), how dispatch works (parent t.Setenv -> child inherits -> TestMain sees the var and exits before m.Run), and what the trade-off is (extra TestMain plumbing in exchange for a self-contained, cross-platform fake binary). No functional changes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com> (cherry picked from commit b196c89)
Replace the in-process `render.Render()` call (Crossplane v2.2-era) with `crossplane internal render` driven through `github.com/crossplane/cli/v2`'s Engine interface. The binary runs the real composite reconciler, so crossplane-diff now produces output that matches what users would see if they reconciled the same XR in a live cluster. Dependency bump: - crossplane/v2, crossplane-runtime/v2, apis/v2 → v2.3.1 - crossplane/cli/v2 → main @ b88f8a1 (pinned via Go pseudo-version pending the next cli release; carries the `composite_resource_definition` proto field and `CompositionInputs.XRD` Go wire-up that the new render contract requires) - function-sdk-go promoted to a direct dep (replaces crossplane/v2/proto/fn/v1) - crank/* imports moved to crossplane/cli/v2/cmd/crossplane/* Render contract adaptations: - Pass the input XR's XRD through `CompositionInputs.XRD` so the render binary can select the right composite.Schema (Legacy vs Modern). New `DefinitionClient.GetCompositeSchema` reads `spec.scope` instead of apiVersion since v1 XRDs round-tripped through the apiserver come back as v2-form objects with scope=LegacyCluster preserved. - Honour the v2.4 partial-output-on-fatal contract: when a pipeline step FATALs, the binary now exits 3 with stdout populated, surfacing recorded RequiredResources. Our `stderrCapturingLocalEngine` unmarshals the partial output, and `RenderToStableState` keeps iterating on the resolved selectors until the pipeline stabilises. - New `EngineRenderFn` owns the render-engine lifecycle (Setup, function runtimes, Render, Cleanup), replacing `cmd/diff/serial`. Default rendering image flips from `:main` (frozen at xpkg.crossplane.io since the nix migration) to cli's `:stable`, which tracks current releases. Schema/fixture updates driven by the reconciler-faithful output: - Add `status.conditions[].observedGeneration` to ~22 test CRDs (the reconciler now writes it on every condition). - Add `spec.crossplane.*` subtree to nested-XR test fixture CRDs that previously declared only user fields. - Strip `spec.crossplane` from dry-run apply input only when needed (composed resources whose CRD doesn't declare the subtree); keep it for root XRs so revision upgrades surface in the diff. - Synthesize a backing XR for existing claims that lack `spec.resourceRef` so the reconciler's GVK compatibility check accepts the input. - Fix `NewXRWithGenerateName` to emit a placeholder name that passes RFC 1123 validation; display layer continues to render `(generated)`. Skipped tests, with TODOs: - `TestGetRestConfig/EmptyKubeconfigEnvVar`: pre-existing skip, now annotated with how to make it run on developer machines too. - `TestCompDiffIntegration/CrossNamespaceResourceCollision`: blocked on crossplane-contrib/function-extra-resources#106 — the function emits Selector{Namespace:""} for `Reference`-typed extras that omit `ref.namespace`, and the v2.3+ binary fetcher takes it verbatim. Re-enable once a function release defaults to the XR's namespace. Working notes for two upstream issues we drove this round live under .requirements/20260504T230404Z_render_engine_migration/, including the filed cli/proto change (crossplane/crossplane#7452, merged) and the filed function-extra-resources issue. Test result: 609 PASS / 0 FAIL / 2 SKIP. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
The test was dropped in da2f192 when ProvideRequirements (step-keyed map) was rewritten as ResolveSelectors (flat selector slice). The underlying behavior — resolveNamespace defaulting empty selector namespace to xrNamespace, and namespace-aware cache keys — is still in the code, but nothing was asserting it. Pairs with the (currently skipped) E2E TestCompDiffIntegration/CrossNamespaceResourceCollision, which is blocked on function-extra-resources#106. The unit test exercises the same defaulting + cache-keying path without depending on the function, so regressions are caught even while the E2E remains skipped. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
…v-var race Previously, integration tests pointed the render engine at a local crossplane binary by mutating the process-global CROSSPLANE_RENDER_BINARY env var inside requireCrossplaneBinary. Both TestDiffIntegration and TestCompDiffIntegration call t.Parallel(), so whichever finished first would restore/unset the var while the other was still running, racing on engine selection. Replace with explicit path threading: - ProcessorConfig.CrossplaneRenderBinary + WithCrossplaneRenderBinary option. - NewEngineRenderFn now takes the path as a positional arg; empty string means "use the upstream docker engine" (production default). - New hidden CLI flag --crossplane-render-binary plumbs the path through defaultProcessorOptions so each kong invocation gets its own copy. - requireCrossplaneBinary now returns the absolute path; runIntegrationTest threads it into args as --crossplane-render-binary=<path>. No env mutation. The local-binary path itself remains a temporary test affordance — documented in render_engine.go as something we can delete once crossplane/cli's localRenderEngine adopts equivalent stderr-capture and exit-code-3 (PR #7455 partial-output-on-fatal) semantics. Upstream PR to follow. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Mechanical fixes from the PR review pass: - diffprocessor/requirements_provider.go: drop the orphaned function-stub comments left over from ProvideRequirements, drop the unused `stepName` arg threaded through processSelector / processNameSelector / processLabelSelector / resolveNamespace (callers now always passed ""), and update the resourceCache doc to mention namespace (which has been part of the cache key since dt.MakeDiffKey took on namespace). - diffprocessor/resource_manager.go: dedupe owner refs via dt.MakeDiffKey instead of an ad-hoc "|"-joined string, matching the rest of the diff layer's keying convention. - diffprocessor/diff_processor.go: switch the err==nil/else block in RenderToStableState to a switch (project style), tighten the generateName comment to make clear "placeholder" is purely a render-side RFC1123-valid filler and the (generated) suffix the user sees is produced independently by diff_formatter.go, and update the schema re-stamp comment to point at crossplane/crossplane#7452 (merged on main; v2.3.1 still defaults the binary to SchemaModern). - diffprocessor/diff_processor_test.go: same #7452 update on the schema- plumbing test's documentation. - diffprocessor/render_engine_test.go: add a header comment justifying why the three lifecycle tests (HappyPath / CleanupIdempotent / Serialization) are procedural rather than table-driven, and switch to t.Context(). - diffprocessor/schema_validator.go: rewrite the spec.crossplane stripping comment so it accurately states that real cluster-derived CRDs declare the subtree (Crossplane's CRD generator emits it) — the gap is in our hand-rolled integration-test CRD fixtures, not in production. - client/crossplane/definition_client.go: bring the GetCompositeSchema doc comments (interface + impl) into sync with the actual implementation, which reads spec.scope rather than the XRD's apiVersion (apiserver conversion can rewrite the apiVersion stamp; spec.scope is preserved). - client/crossplane/definition_client_test.go: drop a duplicated comment line; clarify the "ModernClaim" reference (the claim kind in the test is just the fixture's "ClaimedResource" string — there's no upstream type called ModernClaim). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
… wrapper crossplane/cli#91 (in flight) lands stderr capture and exit-code-3 partial-output handling in both the local and docker render engines. That makes our stderrCapturingLocalEngine wrapper redundant — both upstream engines now capture stderr and return (rsp, err) pairs on exit 3, which is the contract our EngineRenderFn.Render already expects. This commit: - Pins github.com/crossplane/cli/v2 via a `replace` directive at the upstream PR's branch SHA (jcogilvie/cli@f9700864). Will swap to a real upstream version once #91 merges. - Deletes stderrCapturingLocalEngine (and its stand-alone Setup / Render / CheckContextSupport plumbing) entirely. NewEngineRenderFn now constructs the engine via render.NewEngineFromFlags with EngineFlags.CrossplaneBinary set from the threaded binaryPath — no conditional, no wrapper, just upstream. - Drops the now-unused bytes / os/exec / proto / renderv1alpha1 imports (~80 lines of code gone). EngineRenderFn.Render's caller-level handling (the rsp != nil && err != nil → "partial output after pipeline fatal" branch) is unchanged — it was already engine-agnostic and matches the upstream contract verbatim. The WithCrossplaneRenderBinary option, the hidden --crossplane-render-binary CLI flag, and requireCrossplaneBinary test helper all stay: they're how integration tests thread a path into EngineFlags.CrossplaneBinary without an env var. Their purpose is now "expose upstream's first-class flag through our config" rather than "drive our own custom wrapper." Verified: full unit tests + integration smoke (XR + comp) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Track the latest revision of the upstream PR (crossplane/cli#91), which addresses adamwg's review feedback: - ContainerExitError.Error() no longer embeds stderr (kept in .Stderr field for callers). - Both engines collapsed their post-Run error tails to a single errors.Wrapf("...returned error with output: %s", stderr). - Cosmetic shortening of the containerRunner doc comment. No diff-side change required — our caller-level handling in EngineRenderFn.Render is engine-agnostic (rsp != nil + err != nil => partial output) and the new upstream behaviour matches that contract unchanged. Verified: full unit + integration tests still pass against the bumped replace. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
crossplane/cli#91 merged as dc7a1fa on origin/main. Drop the temporary `replace` directive that pointed at the user-fork branch and pin github.com/crossplane/cli/v2 to the upstream pseudo-version v2.4.0-rc.0.0.20260609191853-dc7a1fa2788a. No code changes — the upstream contract is identical to what we were already testing against on the fork branch; the bump just shifts the source of truth back to crossplane/cli. Verified: full unit + integration tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
All four crossplane modules cut v2.3.2 today. Pin to those releases: - crossplane/cli/v2: dc7a1fa pseudo-version → v2.3.2 - crossplane/crossplane-runtime/v2: v2.4.0-rc.0 → v2.3.2 - crossplane/crossplane/apis/v2: v2.4.0-rc.0 → v2.3.2 - crossplane/crossplane/v2: v2.3.1 → v2.3.2 cli v2.3.2 includes our PR #91 backported via #95 (crossplane/cli@95), so the partial-output-on-fatal contract and stderr capture are part of a real release tag — no more pre-release pseudo-versions in our go.mod. Verified: full unit + integration tests pass against the pinned set. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
EngineRenderFn previously called engine.Setup + StartFunctionRuntimes exactly once on the first render and reused that single batch's addresses for every subsequent render. When a single `xr` invocation processed multiple XRs that resolved to different Compositions with overlapping-but-not-identical function pipelines (e.g. diffing a GitOps directory), the second composition's new functions never got upstream's runtime-docker-network annotation — their containers landed on the default Docker bridge and were unreachable from the render container. Self-contained workaround on the diff side (no upstream changes required against cli v2.3.2): - Track started functions in a startedNames set, dedup'd by fn name. - Accumulate every *FunctionAddresses returned by startRuntimes in fnAddrsList; merge their Addresses() maps into a single addrs map. - On the first render, capture upstream's auto-stamped network name off any first-batch function via render.AnnotationKeyRuntimeDockerNetwork. - On subsequent renders, manually apply that captured annotation to any new function (preserving any caller-set value), then call startRuntimes for new functions only. - BuildCompositeRequest receives FunctionAddrs filtered to exactly this render's function set. - Cleanup iterates fnAddrsList and stops every entry, then runs the network cleanup once. The workaround couples to internal state of dockerRenderEngine via an annotation side-channel. Tracked for unwind in #338 once a clean upstream API lands (crossplane/cli#96 — either idempotent Setup or a new Engine.AnnotateFunctions method). Tests: - Existing TestEngineRenderFn_HappyPath / CleanupIdempotent / Serialization continue to pass (single-comp regression coverage). minimalRenderInputs() now includes one Function so the "first call → start runtimes" assertion is exercised meaningfully. - New TestEngineRenderFn_MultiCompositionFunctionSet covers the staleness scenario: render with [F1, F2], then [F1, F3], then [F1, F2] again. Asserts startRuntimes is called exactly twice (F1+F2, then F3 only), every started fn carries the captured network annotation, and already-running fns are not restarted. - New TestEngineRenderFn_PreservesExistingNetworkAnnotation asserts caller-supplied runtime-docker-network values are not overwritten. - New TestEngineRenderFn_CleanupStopsAllFunctionAddresses asserts Cleanup invokes stopRuntimes for every *FunctionAddresses ever returned, not just the most recent. Includes the per-task REQUIREMENTS.md (.requirements/20260609T220505Z_multi_composition_render/) capturing the As-Is / To-Be / requirements / acceptance criteria / testing / implementation plan. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
jcogilvie
commented
Jun 9, 2026
jcogilvie
commented
Jun 9, 2026
- diff_processor.go: extract resolveSchemaAndXRDForRender + resolveFunctionCredentials helpers from RenderToStableState (gocognit was 35 > 30 threshold). update v2.3.1 -> v2.3.2 in the schema-restamp comment now that crossplane/crossplane#7452 is shipped. - definition_client.go: //nolint:interfacebloat with reason. The 6 methods are cohesively about XRD lookup; splitting just to satisfy the linter would create surface without value. - render_engine.go: guard against (nil rsp, nil err) from a misbehaving Engine implementation; previously rsp.GetComposite() would have panicked. - schema_validator.go: rewrite the SchemaValidation defaults-propagation comment to accurately describe the shared-map mechanism (composed.Unstructured embeds unstructured.Unstructured; UnstructuredContent returns Object by reference; defaults DO propagate via the shared map). - render_engine_test.go: drop the structpb dummy import + var (no test referenced it). - diff_it_utils_test.go: rename requireCrossplaneBinary to localCrossplaneBinary; return path-or-empty instead of skipping. CI's go-test target doesn't build _output/bin/crossplane, so the previous skip silently turned IT into a no-op in CI. Now empty path falls through to the docker render engine (slower but real). Local devs can still build the binary for fast iteration; path is overridable via CROSSPLANE_DIFF_RENDER_BINARY env var. - diff_integration_test.go: only add --crossplane-render-binary when a path is present (empty would parse as 'use docker engine' anyway, but explicit absence reads cleaner). - renderer/types/types.go + renderer/diff_formatter.go + diff_processor.go: introduce shared GenerateNamePlaceholder constant ('xrgenplace0000') replacing the prior 'placeholder' string. Formatter now substitutes the sentinel back to '(generated)' for both the displayed resource name and the diff-body metadata, fixing TestDiffNewClaim/CanDiffNewClaim e2e (ANSI golden) and TestDiffIntegration/NewXRWithGenerateName. Extracted stripSyntheticName + normalizedGenerateName helpers to keep cleanupForDiff under the gocognit threshold. - main.go + diff_integration_test.go: lint --fix realignment of struct tags + wsl_v5 whitespace nudges. Verified: full unit + integration suite green; lint clean on touched packages (only pre-existing revive var-naming on cmd/diff/renderer/types remains). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
…xture instead
stripCrossplaneManagedFields existed solely to paper over hand-rolled CRD test fixtures that didn't declare the spec.crossplane subtree the composite reconciler populates at runtime. Real cluster CRDs derived from XRDs declare it (Crossplane's CRD generator emits it), so the strip was production code coding around a test-data limitation.
Investigation found exactly one fixture actually needed updating: testdata/{diff,comp}/crds/xdefaultresource-ns-crd.yaml — the kind=XTestDefaultResource v2 XR used by TestDiffIntegration/XRDDefaultsAppliedBeforeRendering. Every other v2 XR fixture either already declares spec.crossplane.* or is a v1 (legacy) fixture that puts compositionRef et al. at the top level of spec.properties (no spec.crossplane needed). Claim and managed-resource fixtures don't have spec.crossplane at all and aren't affected.
Removed:
- DefaultSchemaValidator.stripCrossplaneManagedFields helper
- the call site in ValidateResources
- the now-stale comment block referencing the strip from the SchemaValidation defaults-propagation explanation
Added spec.crossplane.* (canonical Crossplane shape) to both copies of xdefaultresource-ns-crd.yaml. Full unit + integration suite remains green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
google.golang.org/protobuf has no direct usage in our code anymore (the dummy 'var _ = structpb.NewNullValue' was removed in 90fa8b8). 'earthly +generate' moved it to the indirect block accordingly. Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Per Copilot review on PR #326: ResolveSelectors fans out through processSelector to either processNameSelector (MatchName -> client.GetResource) or processLabelSelector (MatchLabels -> client.GetResourcesByLabel), but the unit test table only exercised MatchName. Adds two cases covering the label path: - MatchLabels: tier=cache selector returns both labelled ConfigMaps via GetResourcesByLabel; verifies the resources flow through ResolveSelectors unchanged. - MatchLabelsFetchError: error path parallel to the existing FetchError case for MatchName, covering propagation from the label-list call site. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 82 out of 83 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
cmd/diff/diffprocessor/requirements_provider.go:60
- RequirementsProvider stores a renderFn field (and NewRequirementsProvider takes a renderFn parameter) but nothing ever uses it. With golangci-lint's
unusedlinter enabled, this will be reported as an unused field/identifier and can break CI.
1. requirements_provider.go: drop the dead RequirementsProvider.renderFn field. RequirementsProvider stored a renderFn but never used it after the switch from ProvideRequirements (which needed it for env-config FATAL recovery render passes) to ResolveSelectors (which doesn't). The field, the constructor parameter, the ComponentFactories.RequirementsProvider signature, the WithRequirementsProviderFactory option signature, and the call sites in diff_processor.go + tests all drop the renderFn arg. No behaviour change. 2. diff_processor.go + definition_client.go: fold the duplicate XRD lookup in resolveSchemaAndXRDForRender into a single call. GetCompositeSchema internally calls GetXRDForXR / GetXRDForClaim, then resolveSchemaAndXRDForRender called the same lookups again to forward the XRD to the render binary - two cache hits per render iteration. Extracts the spec.scope -> Schema rule into an exported xp.SchemaFromXRD helper that both GetCompositeSchema (when called directly) and resolveSchemaAndXRDForRender (which already has the XRD) can call. Net: one lookup per iteration. 3. renderer/diff_formatter.go: handle composed-resource names that crossplane's reconciler synthesized from a generateName template. Pre-migration, render.Render() left composed resources with generateName + no name. Post-migration, crossplane internal render's reconciler stamps a deterministic name (e.g. test-claim-b0348ce08462) on resources whose template only had generateName. The diff formatter's existing logic only recognised three name shapes - the legacy '(generated)' suffix, our GenerateNamePlaceholder sentinel, and 'no name + generateName set' - so it kept the generated name in the diff body and the test framework's masking turned it into XXXXX. Adds isGeneratedFromGenerateName(desired) which returns true when generateName is non-empty and name starts with generateName (i.e. the suffix is generated). Display path substitutes <generateName>(generated); strip path drops metadata.name and normalises generateName. Both treat the new shape identically to the existing two. Fixes TestDiffNewClaim/CanDiffNewClaim e2e regression. Updates TestDefaultDiffProcessor_RenderToStableState_SchemaPlumbing to mock GetXRDForXR (matching the new single-lookup path) instead of GetCompositeSchema. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Replace the in-process
render.Render()call (Crossplane v2.2-era) withcrossplane internal renderdriven throughgithub.com/crossplane/cli/v2's Engine interface. The binary runs the real composite reconciler, so crossplane-diff now produces output that matches what users would see if they reconciled the same XR in a live cluster.Dependency bump
All four crossplane modules pinned at v2.3.2:
crossplane/cli/v2 v2.3.2(includes the merged-and-released exit-code-3 + stderr-capture work from feat(render): capture stderr and handle pipeline-fatal exit code in both engines crossplane/cli#91 backported via e2es spend too long in setup/teardown #95)crossplane/crossplane-runtime/v2 v2.3.2crossplane/crossplane/apis/v2 v2.3.2crossplane/crossplane/v2 v2.3.2function-sdk-gowas promoted to a direct dep (replacescrossplane/v2/proto/fn/v1);crank/*imports moved tocrossplane/cli/v2/cmd/crossplane/*.Render contract adaptations
CompositionInputs.XRDso the render binary can select the rightcomposite.Schema(Legacy vs Modern). NewDefinitionClient.GetCompositeSchemareadsspec.scopeinstead of apiVersion, since v1 XRDs round-tripped through the apiserver come back as v2-form objects withscope=LegacyClusterpreserved.crossplane internal renderexits with code 3 and a populated stdout.EngineRenderFnrecognises the(rsp != nil, err != nil)shape upstream's wrapped engines now return on that path, andRenderToStableStatekeeps iterating on the resolved selectors until the pipeline stabilises.EngineRenderFnowns the render-engine lifecycle (Setup, function runtimes, Render, Cleanup), replacingcmd/diff/serial. Default rendering image flips from:main(frozen at xpkg.crossplane.io since the nix migration) to cli's:stable, which tracks current releases.xrinvocation processes XRs from different compositions with overlapping-but-not-identical function pipelines (e.g. diffing a GitOps directory). On the first render, capture the engine's auto-stamped network name from upstream'sSetup; on subsequent renders, manually apply that annotation to any newly-discovered functions before starting their runtimes. Self-contained workaround tracked for unwind in render: unwind multi-composition workaround in EngineRenderFn once upstream Engine API supports it #338 once a clean upstream API lands (crossplane/cli#96).Schema/fixture updates driven by the reconciler-faithful output
status.conditions[].observedGenerationto ~22 test CRDs (the reconciler now writes it on every condition).spec.crossplane.*subtree to nested-XR test fixture CRDs that previously declared only user fields.spec.crossplanefrom dry-run apply input only when needed (composed resources whose CRD doesn't declare the subtree); keep it for root XRs so revision upgrades surface in the diff.spec.resourceRefso the reconciler's GVK compatibility check accepts the input.NewXRWithGenerateNameto emit a placeholder name that passes RFC 1123 validation; display layer continues to render(generated).Skipped tests (with TODOs)
TestGetRestConfig/EmptyKubeconfigEnvVar: pre-existing skip, now annotated with how to make it run on developer machines too.TestCompDiffIntegration/CrossNamespaceResourceCollision: blocked on Reference-typed extra resources fail for namespaced kinds when ref.namespace is omitted function-extra-resources#106 — the function emitsSelector{Namespace:""}forReference-typed extras that omitref.namespace, and the v2.3+ binary fetcher takes it verbatim. Re-enable once a function release defaults to the XR's namespace.Upstream work driven during this round (all merged or filed)
internal rendercrossplane/crossplane#7452 — render: honor input XR's schema based on supplied XRDs. Merged, shipped in v2.3.2.Working notes live under
.requirements/20260504T230404Z_render_engine_migration/and.requirements/20260609T220505Z_multi_composition_render/.Test result: full unit + integration suite green. (E2E unchanged; suite gated on cli releases that publish the rendered image.)
Fixes #
I have:
earthly -P +reviewableto ensure this PR is ready for review.Added or updated e2e tests— existing E2E suite is unchanged; the migration is internal. The render-binary version under E2E test is gated on the cli release that pulls in the new field.Followed the API promotion workflow— no API changes.🤖 Generated with Claude Code