feat(codecs, databricks_zerobus sink): Unity Catalog VARIANT support

[yorickvanzweeden]

Summary

Hi Vector team,

Databricks released another fix for their Zerobus Ingestion SDK. This PR:

• Adds support for Unity Catalog VARIANT columns in the databricks_zerobus sink.
• The implementation converts JSON values into Databricks’ Parquet VARIANT binary representation and recursively supports VARIANT fields nested in structs, lists, fixed-size lists, large lists, and maps.
• This also upgrades the Zerobus SDK to 2.4.0, which includes the Arrow schema support required for VARIANT columns.

Please let me know if you require any changes, or have any questions.

Vector configuration

The sink can be configured as follows:

 [sinks.databricks]
 type = "databricks_zerobus"
 inputs = ["events"]
 ingestion_endpoint = "https://<workspace-id>.zerobus.<region>.cloud.databricks.com"
 unity_catalog_endpoint = "https://<workspace>.cloud.databricks.com"
 table_name = "<catalog>.<schema>.<table>"
 
 [sinks.databricks.auth]
 strategy = "oauth"
 client_id = "${DATABRICKS_CLIENT_ID}"
 client_secret = "${DATABRICKS_CLIENT_SECRET}"

When using ${...} environment variables, Vector environment interpolation must be explicitly enabled or replaced with an appropriate secret-injection mechanism.

How did you test this PR?

• Ran cargo fmt --all -- --check.
• Ran focused VARIANT codec tests: 3 passed.
• Ran the full Arrow codec test suite: 16 passed.
• Ran Databricks Zerobus sink tests: 39 passed.
• Ran Clippy with -D warnings for the codec and Zerobus sink.
• Built Vector with databricks-zerobus-ingest-sdk 2.4.0.
• Tested top-level VARIANT object encoding.
• Tested nested VARIANT values in structs, lists, and maps.
• Tested invalid VARIANT JSON handling.
• Tested preservation of VARIANT child nullability.
• Regenerated third-party license metadata after the dependency updates.

Additionally, I ran an internal test against our Databricks instance.

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Related: [Rust] Fix variant Arrow schema databricks/zerobus-sdk#525

[thomasqueirozb]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 938623cd33

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Preserve string scalars when encoding VARIANT values

When a VARIANT column receives a normal Vector string value, this branch treats that string as already-serialized JSON instead of as the JSON string scalar it is. In practice inputs such as "hello" are rejected as invalid JSON, while strings like "42" or "true" are silently stored as numeric/boolean VARIANTs rather than strings; this corrupts or drops rows for VARIANT columns containing scalar string data. Since non-string serde_json::Values are serialized below, string values need a path that preserves them as strings unless the user has explicitly provided pre-serialized JSON.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Avoid treating ordinary binary structs as VARIANT

Because this generic Arrow/Parquet encoder now infers VARIANT solely from the child field shape, any user schema with a normal Struct<metadata: LargeBinary, value: LargeBinary> is rewritten as a Parquet VARIANT instead of preserving the two binary fields. This affects non-Databricks users of the same serializer, such as user-provided Arrow schemas for ClickHouse or Parquet output, where an event object containing metadata and value would be serialized as a new VARIANT payload and the original binary fields are lost. Please gate this conversion on an explicit VARIANT marker/configuration rather than the plain struct shape.

Useful? React with 👍 / 👎.

[flaviofcruz]

Duck typing here might be a bit risky in case a user defines a struct type in a table with exactly the same fields. It probably would make sense to attach an annotation that the Field is variant before checking it but I think it would require changes in the zerobus server and SDK.