[fix][io] Upgrade ClickHouse jdbc client version to 0.9.8 by jknetl · Pull Request #33 · apache/pulsar-connectors

jknetl · 2026-06-08T06:38:42Z

Updates ClickHouse jdbc driver to the latest version.

fixes #32

david-streamlio · 2026-06-11T14:16:15Z

Thanks for the fix, @jknetl — the driver bump looks correct for #32.

One thing to flag before merge: CI hasn't actually run on this PR yet. The "Pulsar Connectors CI" workflow is sitting in the action_required state (pending maintainer approval for a first-time contributor), so neither the build nor the unit test job has executed. A maintainer will need to click "Approve and run workflows" so we get a green run before merging.

A couple of caveats worth noting even once CI passes:

The jdbc/clickhouse module has no src/test, and there are no ClickHouse integration tests in the repo. The CI matrix only runs ./gradlew test (no container-based integration job), so a green build won't actually exercise this driver against ClickHouse.
The bug in [bug] Clickhouse JDBC sink connector fails with recent ClickHouse versions #32 (Not able to find table: pulsar_messages) is a runtime driver-compatibility issue that unit tests won't catch. Could you confirm you've manually verified the sink against a recent ClickHouse version with this 0.9.8 driver?
This is a large jump (0.4.6 → 0.9.8). The ClickHouse Java client reorganized its artifacts/packages across the 0.5–0.7 line, so it'd be good to confirm the com.clickhouse:clickhouse-jdbc coordinates and classes still resolve cleanly at build time.

Once CI is approved/green and you can confirm a manual ClickHouse check, this is good to go from my side. 👍

Adds a Testcontainers-based integration test that verifies the ClickHouse JDBC sink can discover an existing table and write a record against a real ClickHouse server. This guards the driver upgrade in this PR: the test fails on the old driver (0.4.6) with "Not able to find table: pulsar_messages" at JdbcUtils.getTableId (issue apache#32) and passes on 0.9.8. The table is created and results are verified over ClickHouse's HTTP interface, so the JDBC driver under test is exercised only by the sink. - Add testcontainers-clickhouse to the version catalog. - Add test dependencies to the clickhouse module build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

david-streamlio · 2026-06-11T15:29:13Z

I went ahead and added an integration test for this change (pushed as a commit on the branch) so the driver upgrade is actually validated against a real ClickHouse server.

ClickHouseJdbcSinkIntegrationTest spins up ClickHouse via Testcontainers, creates the pulsar_messages table, opens the sink, writes a record, and verifies the row landed. It lives in jdbc/clickhouse/src/test, so it runs in the existing CI test job alongside the other container-based connector tests (mongo, elasticsearch, cassandra) — no ci.yaml changes needed.

The test is a genuine regression guard for #32 — red on the old driver, green on the new one:

✅ Driver 0.9.8 (this PR): testOpenDiscoversTableAndWrites passes.

🔴 Driver 0.4.6 (before): fails at exactly the [bug] Clickhouse JDBC sink connector fails with recent ClickHouse versions #32 location:

java.lang.Exception: Not able to find table: pulsar_messages
    at org.apache.pulsar.io.jdbc.JdbcUtils.getTableId(JdbcUtils.java:112)
    at org.apache.pulsar.io.jdbc.JdbcAbstractSink.open(JdbcAbstractSink.java:116)

One deliberate design choice: the table is created and the result is read back over ClickHouse's HTTP interface rather than JDBC, so the driver under test is exercised only by the sink. That's what makes the failure surface precisely at getTableId instead of in the test harness.

Changes in the added commit:

testcontainers-clickhouse added to the version catalog
test dependencies added to jdbc/clickhouse/build.gradle.kts
the new integration test

Verified locally on JDK 17 with Docker; full :jdbc:pulsar-io-jdbc-clickhouse:build (spotless + test + nar) is green. Once CI is approved, this should run automatically.

A couple of notes for whoever merges:

The test pins clickhouse/clickhouse-server:24.3 — happy to bump that to whatever version the project prefers to validate against.
This needs the CI workflow approved (still action_required) for it to run in the pipeline.

Copilot

Pull request overview

This PR upgrades the ClickHouse JDBC driver used by the Pulsar IO JDBC ClickHouse sink to address a metadata/table-discovery incompatibility with newer ClickHouse server versions (issue #32), and adds an integration test that reproduces the original failure mode against a real ClickHouse instance via Testcontainers.

Changes:

Bump com.clickhouse:clickhouse-jdbc from 0.4.6 to 0.9.8.
Add a Testcontainers-based integration test that verifies open() can discover an existing table and that a write succeeds.
Add the ClickHouse Testcontainers module and required Pulsar/Avro test dependencies for the new test.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
jdbc/clickhouse/src/test/java/org/apache/pulsar/io/jdbc/ClickHouseJdbcSinkIntegrationTest.java	Adds an integration test to guard against the table-discovery regression from issue #32.
jdbc/clickhouse/build.gradle.kts	Adds test dependencies needed to run the new ClickHouse integration test.
gradle/libs.versions.toml	Updates ClickHouse JDBC driver version and adds the Testcontainers ClickHouse library alias.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    public void tearDown() throws Exception {
+        if (jdbcSink != null) {
+            jdbcSink.close();
+        }
+        if (clickhouse != null) {
+            clickhouse.stop();
+        }
+    }


+    private String httpQuery(String sql) throws Exception {
+        URL url = new URL("http://" + clickhouse.getHost() + ":"
+                + clickhouse.getMappedPort(8123) + "/");
+        HttpURLConnection conn = (HttpURLConnection) url.openConnection();
+        conn.setRequestMethod("POST");
+        conn.setDoOutput(true);
+        conn.setRequestProperty("X-ClickHouse-User", clickhouse.getUsername());
+        conn.setRequestProperty("X-ClickHouse-Key", clickhouse.getPassword());
+        try (OutputStream os = conn.getOutputStream()) {
+            os.write(sql.getBytes(StandardCharsets.UTF_8));
+        }
+        int code = conn.getResponseCode();
+        try (InputStream is = code < 400 ? conn.getInputStream() : conn.getErrorStream()) {
+            String body = new String(is.readAllBytes(), StandardCharsets.UTF_8);
+            if (code >= 400) {
+                throw new RuntimeException("ClickHouse HTTP query failed (" + code + "): " + body);
+            }
+            return body;
+        }
+    }


jknetl · 2026-06-12T07:17:26Z

Thanks @david-streamlio for adding the test. I can confirm I succesfully tested it end to end against the ClickHouse server version 26.2.1.

[fix][io] Upgrade ClickHouse jdbc client version

6fa7892

david-streamlio requested review from Copilot and lhotari June 11, 2026 15:47

Copilot started reviewing on behalf of david-streamlio June 11, 2026 15:48 View session

Copilot AI reviewed Jun 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix][io] Upgrade ClickHouse jdbc client version to 0.9.8#33

[fix][io] Upgrade ClickHouse jdbc client version to 0.9.8#33
jknetl wants to merge 2 commits into
apache:masterfrom
jknetl:fix/clickhouse-jdbc-version

jknetl commented Jun 8, 2026 •

edited

Loading

Uh oh!

david-streamlio commented Jun 11, 2026

Uh oh!

david-streamlio commented Jun 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jknetl commented Jun 12, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jknetl commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

david-streamlio commented Jun 11, 2026

Uh oh!

david-streamlio commented Jun 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jknetl commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jknetl commented Jun 8, 2026 •

edited

Loading

jknetl commented Jun 12, 2026 •

edited

Loading