feat(gcp): refactor GCP modules into composer + submodules (full)#233
Draft
micheledaddetta-databricks wants to merge 19 commits into
Draft
feat(gcp): refactor GCP modules into composer + submodules (full)#233micheledaddetta-databricks wants to merge 19 commits into
micheledaddetta-databricks wants to merge 19 commits into
Conversation
Captures the brainstormed design for refactoring GCP-related Terraform examples and modules: a single composer module under modules/gcp/ that conditionally instantiates submodules (network, private-connectivity, account, dns), with each example becoming a thin caller that varies only scenario inputs. Co-authored-by: Isaac
31 tasks across 6 PRs implementing the design spec. Each task has explicit file paths, code blocks, validation commands, and commit messages. Self-review checks complete. Co-authored-by: Isaac
Adds modules/gcp/Makefile mirroring the modules/ pattern (discover sub-projects via */README.md) and updates modules/Makefile to recurse into the gcp/ subdir for terraform-docs generation. Co-authored-by: Isaac
Adds modules/gcp/network module supporting three VPC provenance modes: - vpc_source="create": Terraform creates spoke VPC + subnet + Cloud Router + NAT - vpc_source="existing": data-source lookup for pre-existing VPC + subnet - create_hub=true: adds hub VPC + subnet + bidirectional peering + optional shared-VPC host/service binding Outputs cover all spoke and hub identifiers. Test fixtures in tests/ cover create, existing, and create-with-hub scenarios. Co-authored-by: Isaac
Adds modules/gcp/private-connectivity with: - Dedicated PSC subnet in spoke VPC - Backend (SCC) PSC endpoint gated on enable_backend - Frontend PSC endpoint (spoke) gated on enable_frontend - Hub-side frontend PSC endpoint when hub present + frontend enabled - Regional PSC service-attachment maps for 14 GCP regions - Egress firewall stack gated on restrict_egress: deny-egress (priority 1100), allow Google APIs, allow Databricks control plane (to PSC IPs), optional allow managed Hive, hub ingress from spoke CIDR Test fixtures: full-isolated and no-egress. Co-authored-by: Isaac
Adds modules/gcp/account housing all databricks_mws_* resources: - databricks_mws_workspaces (always emitted) - databricks_mws_networks (when vpc_source != databricks_managed), with dynamic vpc_endpoints block populated when both PSC endpoint IDs are present - databricks_mws_vpc_endpoint frontend/backend/transit, each gated on its enable_* flag plus presence of the matching forwarding-rule name from private-connectivity - databricks_mws_private_access_settings (when private_access_only) Test fixtures: databricks-managed, byovpc, psc-with-pas. Co-authored-by: Isaac
Adds modules/gcp/dns for restricted-egress private DNS: - Hub zones: gcp.databricks.com, gcr.io, googleapis.com, pkg.dev, with workspace/psc-auth/dp records pointing at hub PSC IP - Spoke zone: gcp.databricks.com with workspace/dp records pointing at spoke frontend PSC IP, and tunnel record at backend PSC IP - workspace_dns_id extracted via regex from workspace_url Split into its own submodule (rather than colocated with PSC) because DNS depends on workspace_url which only exists after account creates the workspace; this keeps the composer's dependency graph linear. Test fixture: hub-and-spoke. Co-authored-by: Isaac
Top-level composer that orchestrates network, private-connectivity, account, and dns submodules through orthogonal feature flags: - vpc_source: databricks_managed | create | existing - private_link_frontend, private_link_backend, private_access_only, restricted_egress (each defaults false) - hub/spoke project + CIDR vars required only when restricted_egress Submodules instantiated conditionally: - network when vpc_source != databricks_managed - private-connectivity when any PrivateLink flag is true - account always - dns when restricted_egress is true Cross-variable preconditions (null_resource.preconditions) enforce six rules from the design spec, including: restricted_egress requires vpc_source=create + at least one PrivateLink flag + hub/CIDR vars; databricks_managed forbids any PrivateLink or restricted_egress. Random suffix declared once in the composer, passed to each submodule. Test fixtures: 4 positive (basic, byovpc, existing-vpc, psc-isolated) and 4 negative (each violating one precondition rule). Co-authored-by: Isaac
git mv only; no functional changes. Old path has a deprecation README pointing to the new location. Makefile updated for new depth. Co-authored-by: Isaac
git mv only; no functional changes. Old path has a deprecation README. Co-authored-by: Isaac
Generated README content for network, private-connectivity, account, dns, databricks-workspace, service-account, and unity-catalog via `make -C modules/gcp docs`. Includes an added README placeholder for unity-catalog (the original modules/gcp-unity-catalog had no README). Co-authored-by: Isaac
Migrates all four existing GCP examples to call modules/gcp/databricks-workspace: - gcp-basic: vpc_source="databricks_managed" - gcp-byovpc: vpc_source="create" with spoke_vpc_cidr + subnet_cidr - gcp-with-psc-exfiltration-protection: vpc_source="create" + all 4 PrivateLink/egress flags + Unity Catalog wired via modules/gcp/unity-catalog - gcp-sa-provisioning: source repointed from github URL to ../../modules/gcp/service-account Adds a NEW example examples/gcp-existing-vpc demonstrating vpc_source="existing" (data-source lookup of a pre-existing VPC + subnet). Variable name changes documented in each example's README migration table: subnet_ip_cidr_range -> subnet_cidr, pod/svc renames, subnet/router/nat_name dropped (composer derives), delegate_from dropped (SA-provisioning concern). State from old applies does NOT migrate cleanly because resource addresses differ. Re-apply on clean state. Co-authored-by: Isaac
Deletes:
- modules/gcp-workspace-basic, modules/gcp-workspace-byovpc,
modules/gcp-with-psc-exfiltration-protection (replaced by the composer
+ submodules under modules/gcp/)
- modules/gcp-sa-provisioning, modules/gcp-unity-catalog deprecation stubs
(relocated to modules/gcp/{service-account,unity-catalog} in PR 1's
foundation commit)
- examples/gcp-sa-provisionning (typo dir, only had a Makefile)
- examples/gcp-test-modules (only contained terraform.tfstate files)
All examples now point at modules/gcp/*. Stray terraform.tfstate* files
inside remaining example dirs are gitignored and remain on disk untouched.
Co-authored-by: Isaac
Examples section now lists all four migrated examples (gcp-basic, gcp-byovpc, gcp-with-psc-exfiltration-protection) plus the new gcp-existing-vpc and the corrected-spelling gcp-sa-provisioning. Modules section now lists the modules/gcp/ tree: the composer (databricks-workspace) and its five submodules (network, private-connectivity, account, dns, plus service-account and unity-catalog). Co-authored-by: Isaac
micheledaddetta-databricks
changed the title
Plans a polish pass on the GCP modules and examples landed in PR #233: - fill ~70 missing variable descriptions - split large main.tf files by concern - standardize versions.tf placement - expand composer outputs, drop redundancy, rename misnomers - add critical-only region validations - add README usage examples Co-authored-by: Isaac
8 tasks corresponding to the 8 commits planned in the spec. Each task is self-contained with exact file paths, code blocks, validation commands, and commit messages. Co-authored-by: Isaac
… outputs Reorganizes network, account, dns, and databricks-workspace modules so each .tf file has one clear responsibility, and fixes the *_psc_fr_id misnomer for outputs/inputs that hold forwarding-rule names (not IDs). File splits (no behavioral change): - network/main.tf -> vpc.tf, subnets.tf, nat.tf, peering.tf, shared-vpc.tf, data.tf, locals.tf - account/main.tf -> workspace.tf, networks.tf, locals.tf - dns/hub.tf: workspace_dns_id local extracted to dns/locals.tf - databricks-workspace/main.tf -> main.tf (module blocks only), locals.tf, preconditions.tf, random.tf Renames (coordinated across outputs, inputs, wiring, and fixture): - private-connectivity outputs: frontend_psc_fr_id -> frontend_forwarding_rule_name, backend_psc_fr_id -> backend_forwarding_rule_name, hub_frontend_psc_fr_id -> hub_frontend_forwarding_rule_name - account inputs renamed to match - composer wiring and psc-with-pas fixture updated accordingly Co-authored-by: Isaac
…ion validations Three layered improvements to module surface area and documentation: Composer outputs (drop redundancy, add downstream-friendly outputs): - Drop vpc_id (was an alias of spoke_vpc_id) - Add 14 outputs: private_access_settings_id, frontend/backend/transit _endpoint_id, spoke_vpc_self_link, spoke_subnet_id/self_link, hub_vpc_self_link, nat_id, frontend/backend_psc_ip_spoke, frontend_psc_ip_hub, google_region - Replace try(module.network[0].*) with explicit local.databricks_managed / var.restricted_egress ternaries (same behavior, intent visible) - Add private_access_settings_id output on the account module - Update gcp-byovpc and gcp-with-psc examples to consume spoke_vpc_id instead of the removed vpc_id Variable descriptions (~70 added): - Every variable in private-connectivity, account, dns, and the databricks-workspace composer now has a description attribute describing purpose, format, nullability semantics, and required-when context - Gap-fill on service-account / unity-catalog where missing Region validation (clearer error messages): - private-connectivity validates google_region unconditionally against the 14-region list backing the regional PSC service-attachment maps - composer adds the same validation as a precondition, but gated on any private_link_* flag or restricted_egress being true; bad regions now fail plan with a clear message instead of a map-lookup error Co-authored-by: Isaac
File organization (versions.tf everywhere, no provider configs in modules):
- service-account module: init.tf -> versions.tf + data.tf
(no provider "google" {} block — caller's responsibility)
- unity-catalog module: terraform.tf -> versions.tf
- examples gcp-basic, gcp-byovpc, gcp-existing-vpc, gcp-sa-provisioning:
init.tf -> versions.tf + providers.tf (+ data.tf where the original
carried google_client_openid_userinfo / google_client_config)
- examples gcp-with-psc-exfiltration-protection: terraform.tf -> versions.tf
(providers.tf already existed)
README usage sections (terraform-google-modules convention):
- Each module README gets a ## Usage HCL block above the terraform-docs
marker showing a minimal calling example
- Submodule READMEs note "Typically called by modules/gcp/databricks-
workspace (the composer)"
- The composer README points to the four scenario examples
terraform-docs regen:
- Auto-generated README sections refreshed to reflect Task-2 renames,
Task-3 new outputs, Task-4 ~70 variable descriptions, and Task-5
region validation block
Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full GCP modules refactor described in
docs/superpowers/specs/2026-05-14-gcp-modules-refactor-design.md.The repo previously shipped three duplicated GCP workspace modules. Each example wrapped its own dedicated module. A change to a shared piece — e.g. a new GCP region added to the regional PSC service-attachment map, or a workspace argument added by the provider — needed to land in 2–3 places.
This PR replaces that structure with a unified composer at
modules/gcp/databricks-workspacethat takes scenario flags as inputs and conditionally instantiates focused submodules:modules/gcp/network— VPC + subnet + router + NAT + peering + shared-VPC (create or existing)modules/gcp/private-connectivity— PSC subnet + endpoints + egress firewall stackmodules/gcp/account— alldatabricks_mws_*resources (workspaces, networks, vpc_endpoint, private_access_settings)modules/gcp/dns— private DNS zones for hub + spoke (gcp.databricks.com, gcr.io, googleapis.com, pkg.dev)Existing
modules/gcp-sa-provisioningandmodules/gcp-unity-catalogare relocated undermodules/gcp/(no functional changes).Architecture
Dependency graph is linear:
network → private-connectivity → account → dns. Alldatabricks_mws_*resources colocated inaccount;databricks_mws_vpc_endpointreferences PSC forwarding rules fromprivate-connectivity;dnsconsumesaccount.workspace_url.Composer enforces 6 cross-variable preconditions via
null_resource.preconditions:restricted_egress⇒vpc_source="create"restricted_egress⇒ at least one PrivateLink flagrestricted_egress⇒hub_vpc_google_project+hub_vpc_cidr+psc_subnet_cidrsetvpc_source="create"⇒spoke_vpc_cidr+subnet_cidrsetvpc_source="existing"⇒existing_vpc_name+existing_subnet_namesetvpc_source="databricks_managed"⇒ all PrivateLink/egress flags falseExample migrations
examples/gcp-basicmodules/gcp-workspace-basicvpc_source="databricks_managed"examples/gcp-byovpcmodules/gcp-workspace-byovpcvpc_source="create"+ CIDRsexamples/gcp-with-psc-exfiltration-protectionmodules/gcp-with-psc-exfiltration-protection+modules/gcp-unity-catalogvpc_source="create"+ 4 connectivity flags + relocated UCexamples/gcp-existing-vpc(NEW)vpc_source="existing"examples/gcp-sa-provisioning../../modules/gcp/service-accountState from old applies does NOT migrate cleanly to the new composer because resource addresses differ. Each example's README documents this. Re-apply on clean state.
Commits
docs: add design spec for GCP modules refactordocs: add implementation plan for GCP modules refactorbuild: add Makefile recursion for modules/gcp/ submodulesfeat(gcp/network): VPC create/existing/hub + fixturesfeat(gcp/private-connectivity): PSC + firewall + fixturesfeat(gcp/account): mws_* resources + fixturesfeat(gcp/dns): hub + spoke private zones + fixturefeat(gcp/databricks-workspace): composer + preconditions + fixturesrefactor(gcp/service-account): relocate from gcp-sa-provisioningrefactor(gcp/unity-catalog): relocate from gcp-unity-catalogdocs(gcp): terraform-docs READMEs for new submodulesrefactor(examples/gcp): migrate to new composer + add existing-vpcchore(gcp): remove deprecated modules and junk directoriesdocs: refresh top-level README for new GCP layoutTest plan
terraform validatepasses for every submodule and the composermodules/gcp/databricks-workspace/tests/validate (basic, byovpc, existing-vpc, psc-isolated)terraform planwith the expected precondition errortests/<scenario>/) under each submodule plan with expected resource countsmake -C modules/gcp docsregenerated all module READMEs; no drift outsidemodules/gcp/terraform validateterraform applyagainst the migrated PSC example (medium-risk, manual verification before merge)gcp-basicandgcp-byovpc(smoke)Spec + plan
docs/superpowers/specs/2026-05-14-gcp-modules-refactor-design.mddocs/superpowers/plans/2026-05-14-gcp-modules-refactor.md