feat: add adb-service-direct-private-endpoint module + example

README.md

@@ -46,6 +46,7 @@ The folder `examples` contains the following Terraform implementation examples :
 | Azure | ~~adb-external-hive-metastore~~ **REMOVED** | This example was removed in February 2026. External Hive metastore has been superseded by [Unity Catalog](https://docs.databricks.com/en/data-governance/unity-catalog/index.html). Use [adb-unity-catalog-basic-demo](examples/adb-unity-catalog-basic-demo/) instead. |
 | Azure | [adb-kafka](examples/adb-kafka/)                                                   | ADB - single node kafka template                                                                                                                                                                     |
 | Azure | [adb-private-links](examples/adb-private-links/)                                   | Azure Databricks Private Links                                                                                                                                                                       |
+| Azure | [adb-service-direct-private-endpoint](examples/adb-service-direct-private-endpoint/) | Inbound "service-direct" Private Link to performance-intensive services (Zerobus Ingest, Lakebase Autoscaling) — Public Preview                                                                      |
 | Azure | [adb-squid-proxy](examples/adb-squid-proxy/)                                       | ADB clusters with HTTP proxy                                                                                                                                                                         |
 | Azure | [adb-teradata](examples/adb-teradata/)                                             | ADB with single VM Teradata integration                                                                                                                                                              |
 | Azure | [adb-uc](examples/adb-uc/)                                                         | ADB Unity Catalog Process                                                                                                                                                                            |
@@ -77,6 +78,7 @@ The folder `modules` contains the following Terraform modules :
 | Azure | [adb-with-private-link-standard](modules/adb-with-private-link-standard/)                                 | Provisioning Databricks on Azure with Private Link - Standard deployment                                                                                                                  |
 | Azure | [adb-exfiltration-protection](modules/adb-exfiltration-protection/)                                       | A sample implementation of [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html)                             |
 | Azure | [adb-with-private-links-exfiltration-protection](modules/adb-with-private-links-exfiltration-protection/) | Provisioning Databricks on Azure with Private Link and [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
+| Azure | [adb-service-direct-private-endpoint](modules/adb-service-direct-private-endpoint/)                       | Inbound "service-direct" Private Link to performance-intensive services (Zerobus Ingest, Lakebase Autoscaling) — Public Preview                                                            |
 | Azure | [adb-overwatch-regional-config](modules/adb-overwatch-regional-config/)                                   | Overwatch regional configuration on Azure                                                                                                                                                 |
 | Azure | [adb-overwatch-mws-config](modules/adb-overwatch-mws-config/)                                             | Overwatch multi-workspace deployment on Azure                                                                                                                                             |
 | Azure | [adb-overwatch-main-ws](modules/adb-overwatch-main-ws/)                                                   | Main Overwatch workspace deployment                                                                                                                                                       |

examples/adb-service-direct-private-endpoint/Makefile

@@ -0,0 +1,7 @@
+.PHONY: docs test_docs
+
+docs:
+	terraform-docs -c ../../.terraform-docs.yml .
+
+test_docs:
+	terraform-docs -c ../../.terraform-docs.yml --output-check .

examples/adb-service-direct-private-endpoint/README.md

@@ -0,0 +1,97 @@
+# Example — inbound "service-direct" Private Link (performance-intensive services)
+
+Deploys the [`adb-service-direct-private-endpoint`](../../modules/adb-service-direct-private-endpoint)
+module: an Azure private endpoint to the Databricks per-region Private Link
+Service for performance-intensive services (Zerobus Ingest, Lakebase
+Autoscaling), the `privatelink.azuredatabricks.net` DNS A record
+(`<region>.service-direct`), and the account-side `databricks_endpoint`
+registration that drives it from `PENDING` to `APPROVED`.
+
+> **Note**
+> This feature and the `databricks_endpoint` resource are both in **Public
+> Preview**. Run `terraform plan` and inspect carefully before applying.
+
+## Prerequisites
+
+* A **Premium-tier** Databricks account with the **"Private connectivity for
+  performance-intensive services"** Public Preview feature enabled in the
+  account console.
+* An existing VNet + a dedicated subnet for the private endpoint (PE network
+  policies disabled — the Azure default).
+* The per-region PLS resource ID from the
+  [Microsoft Learn region table](https://learn.microsoft.com/en-us/azure/databricks/resources/ip-domain-region#service-direct-resource-ids).
+
+## How to use
+
+1. Copy `terraform.tfvars` and fill in your values.
+2. `terraform init`
+3. `terraform plan`
+4. `terraform apply`
+5. Confirm `endpoint_state` is `APPROVED` and `endpoint_use_case` is
+   `SERVICE_DIRECT` in the outputs.
+
+> **Authentication — account provider**
+> The `databricks_endpoint` registration uses an **account-level** provider
+> (`host` + `account_id`). All Azure Databricks accounts share the host
+> `accounts.azuredatabricks.net`, so if you have more than one account profile
+> in `~/.databrickscfg`, the CLI auth resolver cannot pick one and `apply`
+> fails with `... match https://accounts.azuredatabricks.net ... Use --profile`.
+> Disambiguate by exporting `DATABRICKS_CONFIG_PROFILE=<your-account-profile>`
+> (or add `profile = "<name>"` to the `databricks.accounts` provider block).
+
+> **Approval is asynchronous**
+> After `apply`, `endpoint_state` is typically `PENDING` — Databricks approves
+> the cross-tenant connection out-of-band, usually within a few minutes. This
+> is expected, not a failure. Run `terraform refresh` (or re-`plan`) after a
+> few minutes to see `APPROVED`; the Azure private endpoint connection flips
+> from `Pending` to `Approved` at the same time.
+
+<!-- BEGIN_TF_DOCS -->
+## Requirements
+
+| Name | Version |
+| ---- | ------- |
+| <a name="requirement_azapi"></a> [azapi](#requirement\_azapi) | 2.0.1 |
+| <a name="requirement_azurerm"></a> [azurerm](#requirement\_azurerm) | >=4.31.0 |
+| <a name="requirement_databricks"></a> [databricks](#requirement\_databricks) | >=1.107.0 |
+| <a name="requirement_time"></a> [time](#requirement\_time) | >=0.9.0 |
+
+## Providers
+
+No providers.
+
+## Modules
+
+| Name | Source | Version |
+| ---- | ------ | ------- |
+| <a name="module_adb-service-direct-private-endpoint"></a> [adb-service-direct-private-endpoint](#module\_adb-service-direct-private-endpoint) | ../../modules/adb-service-direct-private-endpoint | n/a |
+
+## Resources
+
+No resources.
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+| ---- | ----------- | ---- | ------- | :------: |
+| <a name="input_azure_region"></a> [azure\_region](#input\_azure\_region) | Azure region short name (e.g. australiaeast). Must match your workspace region. | `string` | n/a | yes |
+| <a name="input_azure_subscription_id"></a> [azure\_subscription\_id](#input\_azure\_subscription\_id) | Azure subscription ID to deploy into. | `string` | n/a | yes |
+| <a name="input_databricks_account_id"></a> [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks account ID (UUID). | `string` | n/a | yes |
+| <a name="input_databricks_pls_resource_id"></a> [databricks\_pls\_resource\_id](#input\_databricks\_pls\_resource\_id) | Databricks per-region PLS resource ID for performance-intensive services (from the MS Learn region table). | `string` | n/a | yes |
+| <a name="input_private_endpoint_subnet_id"></a> [private\_endpoint\_subnet\_id](#input\_private\_endpoint\_subnet\_id) | Resource ID of an existing subnet to host the private endpoint (PE network policies disabled). | `string` | n/a | yes |
+| <a name="input_create_private_dns_zone"></a> [create\_private\_dns\_zone](#input\_create\_private\_dns\_zone) | Create privatelink.azuredatabricks.net here, or reuse an existing zone. | `bool` | `true` | no |
+| <a name="input_databricks_host"></a> [databricks\_host](#input\_databricks\_host) | Databricks account console host. | `string` | `"https://accounts.azuredatabricks.net"` | no |
+| <a name="input_rg_name"></a> [rg\_name](#input\_rg\_name) | Name of the resource group to create for the private endpoint and DNS zone. | `string` | `"rg-service-direct-pe"` | no |
+| <a name="input_tags"></a> [tags](#input\_tags) | Tags applied to created resources. | `map(string)` | `{}` | no |
+| <a name="input_vnet_ids_to_link"></a> [vnet\_ids\_to\_link](#input\_vnet\_ids\_to\_link) | VNet IDs to link to the DNS zone (used only when create\_private\_dns\_zone = true). | `list(string)` | `[]` | no |
+
+## Outputs
+
+| Name | Description |
+| ---- | ----------- |
+| <a name="output_dns_fqdn"></a> [dns\_fqdn](#output\_dns\_fqdn) | Resolvable FQDN for service-direct (<region>.service-direct.privatelink.azuredatabricks.net). |
+| <a name="output_endpoint_state"></a> [endpoint\_state](#output\_endpoint\_state) | Account-side endpoint state. Must be APPROVED to be usable. |
+| <a name="output_endpoint_use_case"></a> [endpoint\_use\_case](#output\_endpoint\_use\_case) | Endpoint use\_case — expected SERVICE\_DIRECT. |
+| <a name="output_private_endpoint_name"></a> [private\_endpoint\_name](#output\_private\_endpoint\_name) | Name of the Azure private endpoint. |
+| <a name="output_private_ip_address"></a> [private\_ip\_address](#output\_private\_ip\_address) | Private IP assigned to the private endpoint. |
+<!-- END_TF_DOCS -->

examples/adb-service-direct-private-endpoint/main.tf

@@ -0,0 +1,16 @@
+module "adb-service-direct-private-endpoint" {
+  source = "../../modules/adb-service-direct-private-endpoint"
+
+  azure_subscription_id      = var.azure_subscription_id
+  azure_region               = var.azure_region
+  rg_name                    = var.rg_name
+  databricks_host            = var.databricks_host
+  databricks_account_id      = var.databricks_account_id
+  private_endpoint_subnet_id = var.private_endpoint_subnet_id
+  databricks_pls_resource_id = var.databricks_pls_resource_id
+
+  create_private_dns_zone = var.create_private_dns_zone
+  vnet_ids_to_link        = var.vnet_ids_to_link
+
+  tags = var.tags
+}

examples/adb-service-direct-private-endpoint/outputs.tf

@@ -0,0 +1,24 @@
+output "private_endpoint_name" {
+  description = "Name of the Azure private endpoint."
+  value       = module.adb-service-direct-private-endpoint.private_endpoint_name
+}
+
+output "private_ip_address" {
+  description = "Private IP assigned to the private endpoint."
+  value       = module.adb-service-direct-private-endpoint.private_ip_address
+}
+
+output "dns_fqdn" {
+  description = "Resolvable FQDN for service-direct (<region>.service-direct.privatelink.azuredatabricks.net)."
+  value       = module.adb-service-direct-private-endpoint.dns_fqdn
+}
+
+output "endpoint_state" {
+  description = "Account-side endpoint state. Must be APPROVED to be usable."
+  value       = module.adb-service-direct-private-endpoint.endpoint_state
+}
+
+output "endpoint_use_case" {
+  description = "Endpoint use_case — expected SERVICE_DIRECT."
+  value       = module.adb-service-direct-private-endpoint.endpoint_use_case
+}

examples/adb-service-direct-private-endpoint/providers.tf

@@ -0,0 +1,38 @@
+# We strongly recommend using the required_providers block to set the
+# provider sources and versions being used
+terraform {
+  required_providers {
+    azurerm = {
+      source  = "hashicorp/azurerm"
+      version = ">=4.31.0"
+    }
+    databricks = {
+      source  = "databricks/databricks"
+      version = ">=1.107.0"
+    }
+    azapi = {
+      source  = "Azure/azapi"
+      version = "2.0.1"
+    }
+    time = {
+      source  = "hashicorp/time"
+      version = ">=0.9.0"
+    }
+  }
+}
+
+provider "azurerm" {
+  subscription_id = var.azure_subscription_id
+  features {}
+}
+
+# Account-level provider — required by databricks_endpoint.
+provider "databricks" {
+  alias      = "accounts"
+  host       = var.databricks_host
+  account_id = var.databricks_account_id
+}
+
+provider "azapi" {
+  subscription_id = var.azure_subscription_id
+}

examples/adb-service-direct-private-endpoint/terraform.tfvars

@@ -0,0 +1,27 @@
+azure_subscription_id = "00000000-0000-0000-0000-000000000000"
+azure_region          = "australiaeast"
+rg_name               = "rg-service-direct-pe"
+
+databricks_host       = "https://accounts.azuredatabricks.net"
+databricks_account_id = "00000000-0000-0000-0000-000000000000"
+
+# Existing subnet for the private endpoint (PE network policies disabled).
+private_endpoint_subnet_id = "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/virtualNetworks/<vnet>/subnets/<pe-subnet>"
+
+# Databricks per-region PLS resource ID for performance-intensive services.
+# Pull the current value from the MS Learn region table:
+# https://learn.microsoft.com/en-us/azure/databricks/resources/ip-domain-region#service-direct-resource-ids
+# NOTE: this is a Microsoft.Databricks/workspaces resource ID (the Databricks-owned
+# per-region ingress workspace), NOT a Microsoft.Network/privateLinkServices path.
+# The PE connects to it by resource ID with subresource service_direct.
+databricks_pls_resource_id = "/subscriptions/<databricks-sub>/resourceGroups/regional_ingress_<region>_resource_group/providers/Microsoft.Databricks/workspaces/regional_ingress_<region>_workspace"
+
+# Create privatelink.azuredatabricks.net here and link the VNet hosting the PE.
+# Set false (and pre-create the zone) if the workspace already uses inbound PL.
+create_private_dns_zone = true
+vnet_ids_to_link        = ["/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/virtualNetworks/<vnet>"]
+
+tags = {
+  Environment = "dev"
+  Workload    = "service-direct-privatelink"
+}

examples/adb-service-direct-private-endpoint/variables.tf

@@ -0,0 +1,54 @@
+variable "azure_subscription_id" {
+  type        = string
+  description = "Azure subscription ID to deploy into."
+}
+
+variable "azure_region" {
+  type        = string
+  description = "Azure region short name (e.g. australiaeast). Must match your workspace region."
+}
+
+variable "rg_name" {
+  type        = string
+  description = "Name of the resource group to create for the private endpoint and DNS zone."
+  default     = "rg-service-direct-pe"
+}
+
+variable "databricks_host" {
+  type        = string
+  description = "Databricks account console host."
+  default     = "https://accounts.azuredatabricks.net"
+}
+
+variable "databricks_account_id" {
+  type        = string
+  description = "Databricks account ID (UUID)."
+}
+
+variable "private_endpoint_subnet_id" {
+  type        = string
+  description = "Resource ID of an existing subnet to host the private endpoint (PE network policies disabled)."
+}
+
+variable "databricks_pls_resource_id" {
+  type        = string
+  description = "Databricks per-region PLS resource ID for performance-intensive services (from the MS Learn region table)."
+}
+
+variable "create_private_dns_zone" {
+  type        = bool
+  description = "Create privatelink.azuredatabricks.net here, or reuse an existing zone."
+  default     = true
+}
+
+variable "vnet_ids_to_link" {
+  type        = list(string)
+  description = "VNet IDs to link to the DNS zone (used only when create_private_dns_zone = true)."
+  default     = []
+}
+
+variable "tags" {
+  type        = map(string)
+  description = "Tags applied to created resources."
+  default     = {}
+}

modules/adb-service-direct-private-endpoint/Makefile

@@ -0,0 +1,7 @@
+.PHONY: docs test_docs
+
+docs:
+	terraform-docs -c ../../.terraform-docs.yml .
+
+test_docs:
+	terraform-docs -c ../../.terraform-docs.yml --output-check .