feat(connectors): add SurrealDB sink connector by countradooku · Pull Request #3453 · apache/iggy

countradooku · 2026-06-11T08:48:26Z

Summary

Adds a SurrealDB sink connector for writing Iggy messages into SurrealDB over the HTTP API via reqwest.

The connector supports deterministic record IDs, bulk INSERT IGNORE writes for idempotent replay, configurable batch sizing, root/namespace/database/no-auth modes, optional table and offset-index definition, payload modes (auto, json, text, base64/binary), metadata/header/checksum/origin timestamp fields, retry/backoff handling, and runtime metrics logging.

This PR does not depend on the SurrealDB Rust SDK. The v3.1.4 reference is only the Docker image tag used by the integration-test fixture, so the earlier SurrealDB SDK BUSL licensing concern is not applicable to this connector.

It also wires the connector into workspace membership, connector docs, binary artifact builds, edge-release output, version bump scripts, and Docker-backed integration test scaffolding.

Tests

cargo fmt --all
cargo test -p iggy_connector_surrealdb_sink
cargo clippy -p iggy_connector_surrealdb_sink --all-targets -- -D warnings
cargo test -p integration --no-run
cargo build -p iggy_connector_surrealdb_sink
cargo test -p integration -- connectors::surrealdb::surrealdb_sink
cargo sort --no-format --workspace --check
./scripts/ci/taplo.sh --check
git diff --check

./scripts/ci/markdownlint.sh could not be run locally because markdownlint is not installed in this shell.

SurrealDB is a document database target for Iggy connector users, so the sink writes batches with deterministic record ids and bulk INSERT IGNORE to keep runtime redelivery idempotent without per-message round trips. Constraint: User explicitly requested the latest SurrealDB Rust SDK and chose to keep it despite BUSL-1.1 license-validation warnings for SurrealDB crates. Constraint: Local Docker daemon was unavailable, so real-container integration execution could not run here. Rejected: Per-message SDK writes | too many round trips and weaker batching throughput. Rejected: Using the testcontainers SurrealDB module | module source hardcodes an older SurrealDB image. Confidence: medium Scope-risk: moderate Directive: Keep record ids deterministic across releases; changing build_record_id breaks replay idempotency. Tested: cargo fmt --all; cargo sort --no-format --workspace; cargo clippy --all-features --all-targets -- -D warnings; cargo check --all --all-features; cargo test -p iggy_connector_surrealdb_sink; cargo test -p integration --no-run connectors::surrealdb; cargo test --locked --doc; cargo doc --no-deps --all-features --quiet; taplo/license/shellcheck/version/diff/binary checks; prek install Not-tested: Docker-backed SurrealDB integration execution, because Docker daemon was not running locally.

codecov · 2026-06-11T09:20:14Z

Codecov Report

❌ Patch coverage is 84.46215% with 195 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.11%. Comparing base (8c1fefb) to head (e9cd7d4).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
core/connectors/sinks/surrealdb_sink/src/lib.rs	84.46%	150 Missing and 45 partials ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3453       +/-   ##
=============================================
- Coverage     74.02%   45.11%   -28.91%     
  Complexity      937      937               
=============================================
  Files          1247     1245        -2     
  Lines        127567   112592    -14975     
  Branches     103436    88506    -14930     
=============================================
- Hits          94427    50795    -43632     
- Misses        30104    59053    +28949     
+ Partials       3036     2744      -292

Components	Coverage Δ
Rust Core	`37.61% <84.46%> (-37.06%)`	⬇️
Java SDK	`62.44% <ø> (ø)`
C# SDK	`71.41% <ø> (-0.70%)`	⬇️
Python SDK	`88.88% <ø> (ø)`
PHP SDK	`84.29% <ø> (ø)`
Node SDK	`91.22% <ø> (ø)`
Go SDK	`40.14% <ø> (ø)`

Files with missing lines	Coverage Δ
core/connectors/sinks/surrealdb_sink/src/lib.rs	`84.46% <84.46%> (ø)`

... and 375 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ryerraguntla · 2026-06-14T16:42:48Z

/author

ryerraguntla · 2026-06-14T16:43:11Z

Please check the pre-checks failure

countradooku · 2026-06-14T16:44:00Z

Sure

HawkEye maps Rust files to the double-slash license style, so the block comments in the new SurrealDB connector files were treated as missing headers by CI. Constraint: CI runs the updated HawkEye-based license check with strict header matching. Confidence: high Scope-risk: narrow Tested: PATH=/opt/homebrew/bin:/Users/radudiaconu/.vite-plus/bin:/Users/radudiaconu/.codex/tmp/arg0/codex-arg0uTrL1r:/Users/radudiaconu/Library/pnpm/bin:/Users/radudiaconu/.opencode/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/4.0.0/bin:/Users/radudiaconu/.local/bin:/Users/radudiaconu/Library/Application Support/Herd/bin/:/Users/radudiaconu/.bun/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/opt/homebrew/bin:/opt/zerobrew/bin:/Users/radudiaconu/.zerobrew/bin:/Users/radudiaconu/.cargo/bin:/Users/radudiaconu/Library/Application Support/JetBrains/Toolbox/scripts:/Users/radudiaconu/Library/Android/sdk/platform-tools:/Applications/Codex.app/Contents/Resources ./scripts/ci/license-headers.sh --check; cargo fmt --all --check; git diff --check

The Rust pre-merge machete job reported that the SurrealDB sink crate declared toml without using it. Removing the dev-dependency is simpler than adding an ignore entry. Constraint: CI runs cargo machete --with-metadata and fails on unused dependencies. Rejected: Add cargo-machete metadata ignore | the dependency is genuinely unused. Confidence: high Scope-risk: narrow Tested: cargo sort --no-format --workspace; cargo test -p iggy_connector_surrealdb_sink; PATH=/opt/homebrew/bin:/Users/radudiaconu/.vite-plus/bin:/Users/radudiaconu/.codex/tmp/arg0/codex-arg0uTrL1r:/Users/radudiaconu/Library/pnpm/bin:/Users/radudiaconu/.opencode/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/4.0.0/bin:/Users/radudiaconu/.local/bin:/Users/radudiaconu/Library/Application Support/Herd/bin/:/Users/radudiaconu/.bun/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/opt/homebrew/bin:/opt/zerobrew/bin:/Users/radudiaconu/.zerobrew/bin:/Users/radudiaconu/.cargo/bin:/Users/radudiaconu/Library/Application Support/JetBrains/Toolbox/scripts:/Users/radudiaconu/Library/Android/sdk/platform-tools:/Applications/Codex.app/Contents/Resources ./scripts/ci/license-headers.sh --check; cargo fmt --all --check; git diff --check; cargo metadata confirms toml is absent from iggy_connector_surrealdb_sink

countradooku · 2026-06-15T09:08:00Z

/ready

ryerraguntla · 2026-06-17T13:06:08Z

/author - could you please check the pre-checks failures

countradooku · 2026-06-23T15:31:35Z

/ready

ryerraguntla · 2026-06-24T10:22:03Z

/author

hubcio · 2026-06-24T15:24:45Z

how much time do the newly added tests take, on your machine?

countradooku · 2026-06-24T15:28:47Z

On my machine, the focused SurrealDB integration test filter:
cargo test -p integration -- connectors::surrealdb::surrealdb_sink
runs 4 tests in 1.94s test time, 11.57s wall-clock including incremental compile.
The SurrealDB sink unit tests:
cargo test -p iggy_connector_surrealdb_sink
run 35 tests in 0.02s test time, 0.61s wall-clock.

countradooku · 2026-06-24T16:08:21Z

/ready

hubcio

a few non-line notes:

the description says this uses the surrealdb rust sdk 3.1.4 and flags BUSL-1.1 license failures from surrealdb sdk crates, but the code uses raw reqwest and the lockfile adds no surrealdb crate. 3.1.4 is the test-container image tag, not a dep, so the BUSL concern is moot. please fix the description and drop that item.
the jwt default crypto-provider install (under core/server/src/http/jwt + the a2a_jwt tests) and the cpp-bazel cargo registry retries look unrelated to the surrealdb sink. if the jwt change is a rustls feature-unification fix triggered by linking the sink into the all-crates build, please say so; otherwise split these out.
no adversarial round-trip test for the query builder (quotes/brackets in payloads and identifiers). it's safe by construction - identifiers are validated and bodies go through serde_json escaping - but a test would lock that in.

countradooku · 2026-06-26T07:25:53Z

/ready

ryankert01

Some comments

ryankert01 · 2026-06-27T19:38:36Z

+        let mut outcome = self.insert_records_with_retry(records).await;
+        outcome.error_count += record_error_count;
+        let db_error = outcome.error.take();
+        outcome.error = last_record_error.or(db_error);


Option::or keeps the receiver when Some, so a per-record build error always beats the DB result. If a payload fails to decode and SurrealDB is down, the retryable db_error is dropped and the log/returned error show InvalidRecordValue instead of the outage. db_error.or(last_record_error) would surface a real write failure when one occurred. (The runtime ignores consume()'s return, so this is a log/metric issue, not redelivery.)

ryankert01 · 2026-06-27T19:38:36Z

+    let message = error.to_string().to_ascii_lowercase();
+    message.contains("timeout")
+        || message.contains("timed out")
+        || message.contains("temporarily unavailable")
+        || message.contains("service unavailable")


This fallback substring-matches the full error Display. For HttpStatus, Display includes the response body, so a permanent 400/422 whose body contains "timeout"/"service unavailable" is retried (same for a non-transient Query). The typed arms above already catch the real transient cases, and the SDK's is_transient_status (retry.rs:190) covers the status set. Drop the string fallback and match on the typed reqwest::Error/StatusCode.

ryankert01 · 2026-06-27T19:38:36Z

+        if self.define_indexes {
+            let offset_index = format!("{table}_iggy_offset_idx");
+            validate_identifier("index", &offset_index)?;
+            query.push_str(&format!(
+                " DEFINE INDEX IF NOT EXISTS {offset_index} ON TABLE {table} FIELDS iggy_stream, iggy_topic, iggy_partition_id, iggy_offset;"
+            ));


Index fields iggy_stream/iggy_topic/iggy_partition_id/iggy_offset are only written when include_metadata=true. With define_indexes=true, include_metadata=false the index covers fields no record ever sets. Reject define_indexes && !include_metadata in open(), or document the dependency.

ryankert01 · 2026-06-27T19:39:35Z

/author

countradooku marked this pull request as ready for review June 11, 2026 08:49

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jun 11, 2026

countradooku force-pushed the feat/surrealdb-sink-connector branch from 7b8305a to 48c3a9d Compare June 11, 2026 08:58

countradooku added 2 commits June 12, 2026 11:06

Merge branch 'master' into feat/surrealdb-sink-connector

1a2b917

Merge branch 'master' into feat/surrealdb-sink-connector

5373ee4

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 14, 2026

countradooku added 2 commits June 14, 2026 20:13