-
Notifications
You must be signed in to change notification settings - Fork 3.8k
[regression-test](streaming-job) add cdc cases for source/jdbc timezone and TIMESTAMP/timestamptz pk #63543
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
JNSimba
wants to merge
1
commit into
apache:master
Choose a base branch
from
JNSimba:feat/streaming-mysql-source-timezone-case
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
[regression-test](streaming-job) add cdc cases for source/jdbc timezone and TIMESTAMP/timestamptz pk #63543
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
14 changes: 14 additions & 0 deletions
14
...ssion-test/data/job_p0/streaming_job/cdc/test_streaming_mysql_job_jdbc_servertimezone.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !desc -- | ||
| id int No true \N | ||
| tag varchar(96) Yes false \N NONE | ||
| ts0 datetime Yes false \N NONE | ||
| dt0 datetime Yes false \N NONE | ||
|
|
||
| -- !select_snapshot -- | ||
| 1 snapshot_plus01 2024-06-15T18:00 2024-06-15T11:00 | ||
|
|
||
| -- !select_binlog -- | ||
| 1 snapshot_plus01 2024-06-15T18:00 2024-06-15T11:00 | ||
| 2 binlog_plus01 2024-06-15T18:00 2024-06-15T11:00 | ||
|
|
29 changes: 29 additions & 0 deletions
29
regression-test/data/job_p0/streaming_job/cdc/test_streaming_mysql_job_source_timezone.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !desc -- | ||
| id int No true \N | ||
| tag varchar(96) Yes false \N NONE | ||
| ts0 datetime Yes false \N NONE | ||
| ts3 datetime(3) Yes false \N NONE | ||
| ts6 datetime(6) Yes false \N NONE | ||
| dt0 datetime Yes false \N NONE | ||
| dt6 datetime(6) Yes false \N NONE | ||
| d date Yes false \N NONE | ||
|
|
||
| -- !select_snapshot -- | ||
| 1 snapshot_plus08 2024-06-15T12:00 2024-06-15T12:00:00.123 2024-06-15T12:00:00.123456 2024-06-15T20:00 2024-06-15T20:00:00.123456 2024-06-15 | ||
| 2 snapshot_minus05 2024-06-16T01:00 2024-06-16T01:00:00.123 2024-06-16T01:00:00.123456 2024-06-15T20:00 2024-06-15T20:00:00.123456 2024-06-15 | ||
| 3 snapshot_utc 2024-06-15T20:00 2024-06-15T20:00:00.123 2024-06-15T20:00:00.123456 2024-06-15T20:00 2024-06-15T20:00:00.123456 2024-06-15 | ||
| 4 snapshot_null \N \N \N \N \N \N | ||
| 5 snapshot_epoch_plus08 1970-01-01T00:00:01 1970-01-01T00:00:01.123 1970-01-01T00:00:01.123456 1970-01-01T08:00:01 1970-01-01T08:00:01.123456 1970-01-01 | ||
|
|
||
| -- !select_binlog -- | ||
| 1 snapshot_plus08 2024-06-15T14:00 2024-06-15T12:00:00.123 2024-06-15T12:00:00.123456 2024-06-15T20:00 2024-06-15T20:00:00.123456 2024-06-15 | ||
| 2 snapshot_minus05 2024-06-16T01:00 2024-06-16T01:00:00.123 2024-06-16T01:00:00.123456 2024-06-15T20:00 2024-06-15T20:00:00.123456 2024-06-15 | ||
| 3 snapshot_utc 2024-06-15T20:00 2024-06-15T20:00:00.123 2024-06-15T20:00:00.123456 2024-06-15T20:00 2024-06-15T20:00:00.123456 2024-06-15 | ||
| 4 snapshot_null \N \N \N \N \N \N | ||
| 5 snapshot_epoch_plus08 1970-01-01T00:00:01 1970-01-01T00:00:01.123 1970-01-01T00:00:01.123456 1970-01-01T08:00:01 1970-01-01T08:00:01.123456 1970-01-01 | ||
| 101 binlog_plus08 2024-06-15T12:00 2024-06-15T12:00:00.123 2024-06-15T12:00:00.123456 2024-06-15T20:00 2024-06-15T20:00:00.123456 2024-06-15 | ||
| 102 binlog_minus05 2024-06-16T01:00 2024-06-16T01:00:00.123 2024-06-16T01:00:00.123456 2024-06-15T20:00 2024-06-15T20:00:00.123456 2024-06-15 | ||
| 103 binlog_utc 2024-06-15T20:00 2024-06-15T20:00:00.123 2024-06-15T20:00:00.123456 2024-06-15T20:00 2024-06-15T20:00:00.123456 2024-06-15 | ||
| 105 binlog_epoch_plus08 1970-01-01T00:00:01 1970-01-01T00:00:01.123 1970-01-01T00:00:01.123456 1970-01-01T08:00:01 1970-01-01T08:00:01.123456 1970-01-01 | ||
|
|
28 changes: 28 additions & 0 deletions
28
regression-test/data/job_p0/streaming_job/cdc/test_streaming_mysql_job_timestamp_pk.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !select_snapshot_timestamp_pk -- | ||
| 2024-01-01T00:00 A1 | ||
| 2024-06-15T12:00:00.123456 B1 | ||
| 2025-01-01T00:00 C1 | ||
| 2025-06-15T12:34:56.999999 D1 | ||
| 2026-01-01T00:00 E1 | ||
|
|
||
| -- !select_snapshot_composite_pk -- | ||
| 2024-02-01T00:00 1 A2 | ||
| 2024-02-01T00:00 2 B2 | ||
| 2024-02-02T12:00:00.500 3 C2 | ||
| 2024-02-03T23:59:59.999999 4 D2 | ||
| 2024-02-04T00:00 5 E2 | ||
|
|
||
| -- !select_after_incr_timestamp_pk -- | ||
| 2024-01-01T00:00 A1 | ||
| 2024-06-15T12:00:00.123456 B2_upd | ||
| 2025-01-01T00:00 C1 | ||
| 2026-01-01T00:00 E1 | ||
| 2026-06-01T00:00 F2 | ||
|
|
||
| -- !select_after_incr_composite_pk -- | ||
| 2024-02-01T00:00 1 A2 | ||
| 2024-02-01T00:00 2 B2 | ||
| 2024-02-02T12:00:00.500 3 C3_upd | ||
| 2024-02-04T00:00 5 E2 | ||
| 2024-02-05T00:00 6 F3 |
14 changes: 14 additions & 0 deletions
14
...on-test/data/job_p0/streaming_job/cdc/test_streaming_postgres_job_jdbc_servertimezone.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !desc -- | ||
| id int No true \N | ||
| tag text Yes false \N NONE | ||
| ts datetime(6) Yes false \N NONE | ||
| tstz datetime(6) Yes false \N NONE | ||
|
|
||
| -- !select_snapshot -- | ||
| 1 snapshot_plus01 2024-06-15T11:00 2024-06-15T18:00 | ||
|
|
||
| -- !select_binlog -- | ||
| 1 snapshot_plus01 2024-06-15T11:00 2024-06-15T18:00 | ||
| 2 binlog_plus01 2024-06-15T11:00 2024-06-15T18:00 | ||
|
|
29 changes: 29 additions & 0 deletions
29
...ession-test/data/job_p0/streaming_job/cdc/test_streaming_postgres_job_source_timezone.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !desc -- | ||
| id int No true \N | ||
| tag text Yes false \N NONE | ||
| ts datetime(6) Yes false \N NONE | ||
| tstz0 datetime Yes false \N NONE | ||
| tstz3 datetime(3) Yes false \N NONE | ||
| tstz6 datetime(6) Yes false \N NONE | ||
| ttz text Yes false \N NONE | ||
| d date Yes false \N NONE | ||
|
|
||
| -- !select_snapshot -- | ||
| 1 snapshot_plus08 2024-06-15T20:00 2024-06-15T12:00 2024-06-15T12:00:00.123 2024-06-15T12:00:00.123456 20:00:00.123456+08 2024-06-15 | ||
| 2 snapshot_minus05 2024-06-15T20:00 2024-06-16T01:00 2024-06-16T01:00:00.123 2024-06-16T01:00:00.123456 20:00:00.123456-05 2024-06-15 | ||
| 3 snapshot_utc 2024-06-15T20:00 2024-06-15T20:00 2024-06-15T20:00:00.123 2024-06-15T20:00:00.123456 20:00:00.123456+00 2024-06-15 | ||
| 4 snapshot_null \N \N \N \N \N \N | ||
| 5 snapshot_epoch_plus08 1970-01-01T08:00:01 1970-01-01T00:00:01 1970-01-01T00:00:01.123 1970-01-01T00:00:01.123456 08:00:01.123456+08 1970-01-01 | ||
|
|
||
| -- !select_binlog -- | ||
| 1 snapshot_plus08 2024-06-15T20:00 2024-06-15T14:00 2024-06-15T12:00:00.123 2024-06-15T12:00:00.123456 20:00:00.123456+08 2024-06-15 | ||
| 2 snapshot_minus05 2024-06-15T20:00 2024-06-16T01:00 2024-06-16T01:00:00.123 2024-06-16T01:00:00.123456 20:00:00.123456-05 2024-06-15 | ||
| 3 snapshot_utc 2024-06-15T20:00 2024-06-15T20:00 2024-06-15T20:00:00.123 2024-06-15T20:00:00.123456 20:00:00.123456+00 2024-06-15 | ||
| 4 snapshot_null \N \N \N \N \N \N | ||
| 5 snapshot_epoch_plus08 1970-01-01T08:00:01 1970-01-01T00:00:01 1970-01-01T00:00:01.123 1970-01-01T00:00:01.123456 08:00:01.123456+08 1970-01-01 | ||
| 101 binlog_plus08 2024-06-15T20:00 2024-06-15T12:00 2024-06-15T12:00:00.123 2024-06-15T12:00:00.123456 20:00:00.123456+08 2024-06-15 | ||
| 102 binlog_minus05 2024-06-15T20:00 2024-06-16T01:00 2024-06-16T01:00:00.123 2024-06-16T01:00:00.123456 20:00:00.123456-05 2024-06-15 | ||
| 103 binlog_utc 2024-06-15T20:00 2024-06-15T20:00 2024-06-15T20:00:00.123 2024-06-15T20:00:00.123456 20:00:00.123456+00 2024-06-15 | ||
| 105 binlog_epoch_plus08 1970-01-01T08:00:01 1970-01-01T00:00:01 1970-01-01T00:00:01.123 1970-01-01T00:00:01.123456 08:00:01.123456+08 1970-01-01 | ||
|
|
42 changes: 42 additions & 0 deletions
42
regression-test/data/job_p0/streaming_job/cdc/test_streaming_postgres_job_timestamp_pk.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !select_snapshot_timestamp_pk -- | ||
| 2024-01-01T00:00 A1 | ||
| 2024-06-15T12:00:00.123456 B1 | ||
| 2025-01-01T00:00 C1 | ||
| 2025-06-15T12:34:56.999999 D1 | ||
| 2026-01-01T00:00 E1 | ||
|
|
||
| -- !select_snapshot_timestamptz_pk -- | ||
| 2024-01-01T00:00 A1 | ||
| 2024-06-15T12:00:00.123456 B1 | ||
| 2025-01-01T00:00 C1 | ||
| 2025-06-15T12:34:56.999999 D1 | ||
| 2026-01-01T00:00 E1 | ||
|
|
||
| -- !select_snapshot_composite_pk -- | ||
| 2024-02-01T00:00 1 A2 | ||
| 2024-02-01T00:00 2 B2 | ||
| 2024-02-02T12:00:00.500 3 C2 | ||
| 2024-02-03T23:59:59.999999 4 D2 | ||
| 2024-02-04T00:00 5 E2 | ||
|
|
||
| -- !select_after_incr_timestamp_pk -- | ||
| 2024-01-01T00:00 A1 | ||
| 2024-06-15T12:00:00.123456 B2_upd | ||
| 2025-01-01T00:00 C1 | ||
| 2026-01-01T00:00 E1 | ||
| 2026-06-01T00:00 F2 | ||
|
|
||
| -- !select_after_incr_timestamptz_pk -- | ||
| 2024-01-01T00:00 A1 | ||
| 2024-06-15T12:00:00.123456 B2_upd | ||
| 2025-01-01T00:00 C1 | ||
| 2026-01-01T00:00 E1 | ||
| 2026-06-01T00:00 F2 | ||
|
|
||
| -- !select_after_incr_composite_pk -- | ||
| 2024-02-01T00:00 1 A2 | ||
| 2024-02-01T00:00 2 B2 | ||
| 2024-02-02T12:00:00.500 3 C3_upd | ||
| 2024-02-04T00:00 5 E2 | ||
| 2024-02-05T00:00 6 F3 |
137 changes: 137 additions & 0 deletions
137
...-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_jdbc_servertimezone.groovy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
|
|
||
| import org.awaitility.Awaitility | ||
|
|
||
| import static java.util.concurrent.TimeUnit.SECONDS | ||
|
|
||
| // Recommended end-to-end tz configuration: align jdbc_url's serverTimezone | ||
| // with Doris session time_zone, so Doris users see TIMESTAMP columns as | ||
| // wall-clock in the cluster's local tz. | ||
| // | ||
| // jdbc_url is built from the Doris session tz at runtime, so the case works | ||
| // on clusters configured with different default tz values without code | ||
| // changes. | ||
| // | ||
| // Setup: | ||
| // source SET SESSION time_zone='+01:00', INSERT '2024-06-15 11:00:00' | ||
| // ts0 (TIMESTAMP) -> source-internal UTC instant 2024-06-15 10:00:00Z | ||
| // dt0 (DATETIME) -> literal '2024-06-15 11:00:00' | ||
| // jdbc_url serverTimezone=<Doris session tz> | ||
| // | ||
| // Expectations at Doris (.out is pre-filled for the standard Doris default | ||
| // session time_zone '+08:00'): | ||
| // ts0 -> '2024-06-15T18:00' (UTC 10:00Z + 8h = 18:00 in +08) | ||
| // dt0 -> '2024-06-15T11:00' (DATETIME has no tz semantics, stored verbatim) | ||
| suite("test_streaming_mysql_job_jdbc_servertimezone", "p0,external,mysql,external_docker,external_docker_mysql,nondatalake") { | ||
| def jobName = "test_streaming_mysql_job_jdbc_servertimezone_name" | ||
| def currentDb = (sql "select database()")[0][0] | ||
| def table1 = "streaming_mysql_jdbc_servertimezone" | ||
| def mysqlDb = "test_cdc_db" | ||
|
|
||
| sql """DROP JOB IF EXISTS where jobname = '${jobName}'""" | ||
| sql """drop table if exists ${currentDb}.${table1} force""" | ||
|
|
||
| String enabled = context.config.otherConfigs.get("enableJdbcTest") | ||
| if (enabled != null && enabled.equalsIgnoreCase("true")) { | ||
| String mysql_port = context.config.otherConfigs.get("mysql_57_port"); | ||
| String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") | ||
| String s3_endpoint = getS3Endpoint() | ||
| String bucket = getS3BucketName() | ||
| String driver_url = "https://${bucket}.${s3_endpoint}/regression/jdbc_driver/mysql-connector-j-8.4.0.jar" | ||
|
|
||
| // Read Doris session tz so the cdc job aligns with it. | ||
| def dorisTz = (sql "select @@time_zone")[0][0] | ||
| log.info("Doris session time_zone = ${dorisTz}; jdbc_url serverTimezone will use the same.") | ||
|
|
||
| connect("root", "123456", "jdbc:mysql://${externalEnvIp}:${mysql_port}") { | ||
| sql """CREATE DATABASE IF NOT EXISTS ${mysqlDb}""" | ||
| sql """DROP TABLE IF EXISTS ${mysqlDb}.${table1}""" | ||
| sql """ | ||
| create table ${mysqlDb}.${table1} ( | ||
| id int primary key, | ||
| tag varchar(32), | ||
| ts0 timestamp null, | ||
| dt0 datetime null | ||
| ) engine=innodb charset=utf8; | ||
| """ | ||
|
|
||
| sql """SET SESSION time_zone = '+01:00'""" | ||
| sql """INSERT INTO ${mysqlDb}.${table1} VALUES (1, 'snapshot_plus01', | ||
| '2024-06-15 11:00:00', '2024-06-15 11:00:00')""" | ||
| } | ||
|
|
||
| sql """CREATE JOB ${jobName} | ||
| ON STREAMING | ||
| FROM MYSQL ( | ||
| "jdbc_url" = "jdbc:mysql://${externalEnvIp}:${mysql_port}?serverTimezone=${dorisTz}", | ||
| "driver_url" = "${driver_url}", | ||
| "driver_class" = "com.mysql.cj.jdbc.Driver", | ||
| "user" = "root", | ||
| "password" = "123456", | ||
| "database" = "${mysqlDb}", | ||
| "include_tables" = "${table1}", | ||
| "offset" = "initial" | ||
| ) | ||
| TO DATABASE ${currentDb} ( | ||
| "table.create.properties.replication_num" = "1" | ||
| ) | ||
| """ | ||
|
|
||
| try { | ||
| Awaitility.await().atMost(300, SECONDS) | ||
| .pollInterval(2, SECONDS).until( | ||
| { | ||
| def cnt = sql """select count(1) from ${currentDb}.${table1}""" | ||
| log.info("snapshot row count: " + cnt) | ||
| cnt.get(0).get(0) == 1 | ||
| } | ||
| ) | ||
| } catch (Exception ex) { | ||
| def showjob = sql """select * from jobs("type"="insert") where Name='${jobName}'""" | ||
| def showtask = sql """select * from tasks("type"="insert") where JobName='${jobName}'""" | ||
| log.info("show job: " + showjob) | ||
| log.info("show task: " + showtask) | ||
| throw ex | ||
| } | ||
|
|
||
| qt_desc """desc ${currentDb}.${table1};""" | ||
| qt_select_snapshot """select * from ${currentDb}.${table1} order by id;""" | ||
|
|
||
| connect("root", "123456", "jdbc:mysql://${externalEnvIp}:${mysql_port}") { | ||
| sql """SET SESSION time_zone = '+01:00'""" | ||
| sql """INSERT INTO ${mysqlDb}.${table1} VALUES (2, 'binlog_plus01', | ||
| '2024-06-15 11:00:00', '2024-06-15 11:00:00')""" | ||
| } | ||
|
|
||
| Awaitility.await().atMost(180, SECONDS) | ||
| .pollInterval(2, SECONDS).until( | ||
| { | ||
| def cnt = sql """select count(1) from ${currentDb}.${table1}""" | ||
| cnt.get(0).get(0) == 2 | ||
| } | ||
| ) | ||
|
|
||
| qt_select_binlog """select * from ${currentDb}.${table1} order by id;""" | ||
|
|
||
| sql """DROP JOB IF EXISTS where jobname = '${jobName}'""" | ||
|
|
||
| def jobCountRsp = sql """select count(1) from jobs("type"="insert") where Name = '${jobName}'""" | ||
| assert jobCountRsp.get(0).get(0) == 0 | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test reads
@@time_zoneat runtime and uses it asserverTimezone, but the committed.outis fixed for Doris+08:00(ts0is expected as2024-06-15T18:00). On any runner whose Doris session timezone is not+08:00, the CDC job will correctly render a different wall clock and this regression will fail. The PostgreSQL jdbc_servertimezone case has the same pattern. Please make the suite deterministic, for example by setting the Doris session timezone to the value used by the.outbefore reading it, or by using a fixed timezone in both the URL and expected output.