DBZ-6387 Addding new isostring, microseconds, nanoseconds TemporalPrecisionModes by ismailsimsek · Pull Request #6002 · debezium/debezium

ismailsimsek · 2024-11-20T18:52:56Z

Adding 3 new non adaptive temporal modes:

isostring: converts to UTC ISO 8601
microseconds: converts to int64, always with microseconds precision mode. not adaptive
nanoseconds: converts to int64, always with nanoseconds precision mode. not adaptive

remaining todos

update code comments
add it to postgresql array types as well

Requires debezium/debezium-connector-vitess#219

github-actions · 2024-11-20T18:53:09Z

Hi @ismailsimsek. Thank you for your valuable contribution.
Please author your commit(s) using an email linked to your GitHub account.

github-actions · 2024-11-20T18:56:47Z

Hi @ismailsimsek. Thank you for your valuable contribution.
Please author your commit(s) using an email linked to your GitHub account.

...nnector-postgres/src/test/java/io/debezium/connector/postgresql/RecordsStreamProducerIT.java

...onnector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresValueConverter.java

debezium-core/src/main/java/io/debezium/jdbc/JdbcValueConverters.java

ismailsimsek · 2024-11-22T21:11:34Z

JDBC test failing with following error. i think, because its using old version of debezium (test-container) and trying to validate the new enum value.

2024-11-22T18:09:08.0937176Z Caused by: java.lang.IllegalStateException: Unexpected response: Response{protocol=http/1.1, code=400, message=Bad Request, url=http://localhost:32779/connectors/} ; Response Body: {"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nThe 'time.precision.mode' value is invalid: Value must be one of adaptive, adaptive_time_microseconds, connect\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}

ismailsimsek · 2024-11-28T08:43:38Z

@jpechane its ready for review. JDBC test failing, i think because its using old code with container.

jpechane

@ismailsimsek I pushed some refactorings related to the schema building and I hope they work as intended. Also few more comments were added so please take a look. Thsi PR will be merged at the beginning of January into Debezium 3.1.

debezium-core/src/main/java/io/debezium/jdbc/JdbcValueConverters.java

debezium-core/src/test/java/io/debezium/jdbc/JdbcValueConvertersTemporalPrecisionTest.java

...gres/src/test/java/io/debezium/connector/postgresql/PostgresTemporalPrecisionHandlingIT.java

jpechane · 2024-12-11T05:51:17Z

...gres/src/test/java/io/debezium/connector/postgresql/PostgresTemporalPrecisionHandlingIT.java

+
+        assertEquals(after.get("c_id"), 2);
+        // '2017-09-15'::DATE
+        // assertEquals(after.get("c_date"), "2017-09-15Z");


Makes sure no asserts are comented out.

Added FIx for date conversion and activated this line.

debezium-core/src/main/java/io/debezium/jdbc/JdbcValueConverters.java

debezium-core/src/main/java/io/debezium/time/IsoDate.java

jpechane · 2024-12-11T08:59:51Z

/packit test --labels oracle

…es, apply code review suggestions and fix date conversion

ismailsimsek · 2024-12-11T12:30:27Z

debezium-core/src/main/java/io/debezium/jdbc/JdbcValueConverters.java

+            case NANOSECONDS:
+            case MICROSECONDS:


this was missing and catched by test. added them.

NANOSECONDS and MICROSECONDS will generate EpochDays for DATE values.

ismailsimsek · 2024-12-11T12:31:32Z

@ismailsimsek I pushed some refactorings related to the schema building and I hope they work as intended. Also few more comments were added so please take a look. Thsi PR will be merged at the beginning of January into Debezium 3.1.

Thank you @jpechane applied suggestions and added a fix for DATE values.

jpechane · 2025-01-14T07:15:20Z

@ismailsimsek Applied, thanks! Could you please send a follw-up PR with doc updates? Thanks a lot

ismailsimsek force-pushed the isostring-temporal-type branch from e0028b1 to e7418ee Compare November 20, 2024 18:56

ismailsimsek force-pushed the isostring-temporal-type branch 2 times, most recently from 1621d23 to 77bee5c Compare November 20, 2024 19:19

jpechane reviewed Nov 21, 2024

View reviewed changes

...nnector-postgres/src/test/java/io/debezium/connector/postgresql/RecordsStreamProducerIT.java Show resolved Hide resolved

ismailsimsek force-pushed the isostring-temporal-type branch 3 times, most recently from a2725c0 to 17a0802 Compare November 21, 2024 19:12

ismailsimsek commented Nov 21, 2024

View reviewed changes

...onnector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresValueConverter.java Outdated Show resolved Hide resolved

ismailsimsek force-pushed the isostring-temporal-type branch 2 times, most recently from 4a8988b to 607db6d Compare November 21, 2024 20:18

ismailsimsek commented Nov 21, 2024

View reviewed changes

debezium-core/src/main/java/io/debezium/jdbc/JdbcValueConverters.java Show resolved Hide resolved

debezium-core/src/main/java/io/debezium/jdbc/JdbcValueConverters.java Show resolved Hide resolved

ismailsimsek force-pushed the isostring-temporal-type branch 3 times, most recently from 733e48f to 7bb4259 Compare November 22, 2024 17:53

ismailsimsek force-pushed the isostring-temporal-type branch from 9f07312 to b6e6ba3 Compare November 24, 2024 12:24

ismailsimsek requested a review from jpechane November 25, 2024 14:36

ismailsimsek changed the title ~~DBZ-6387 Add isostring TemporalPrecisionMode~~ DBZ-6387 Addding new isostring,microseconds,nanoseconds TemporalPrecisionModes Nov 25, 2024

ismailsimsek changed the title ~~DBZ-6387 Addding new isostring,microseconds,nanoseconds TemporalPrecisionModes~~ DBZ-6387 Addding new isostring,microseconds,nanoseconds TemporalPrecisionModes Nov 25, 2024

ismailsimsek changed the title ~~DBZ-6387 Addding new isostring,microseconds,nanoseconds TemporalPrecisionModes~~ DBZ-6387 Addding new isostring, microseconds, nanoseconds TemporalPrecisionModes Nov 25, 2024

ismailsimsek force-pushed the isostring-temporal-type branch 2 times, most recently from 5bc2b7c to 69563df Compare November 25, 2024 17:57

DBZ-6387 Add isostring, nanosecond, milliseconds TemporalPrecisionModes

e0b7da6

ismailsimsek force-pushed the isostring-temporal-type branch from 69563df to e0b7da6 Compare November 25, 2024 20:15

jpechane added the stream/3.1 label Dec 10, 2024

DBZ-6387 Schema creatio refactoring

a96aab1

jpechane mentioned this pull request Dec 11, 2024

DBZ-6387 Reflect core refactoring debezium/debezium-connector-vitess#219

Merged

jpechane requested changes Dec 11, 2024

View reviewed changes

jpechane reviewed Dec 11, 2024

View reviewed changes

debezium-core/src/main/java/io/debezium/time/IsoDate.java Outdated Show resolved Hide resolved

DBZ-6387 Add isostring, nanosecond, milliseconds TemporalPrecisionMod…

62c2235

…es, apply code review suggestions and fix date conversion

ismailsimsek commented Dec 11, 2024

View reviewed changes

jpechane merged commit 36419da into debezium:main Jan 14, 2025

jpechane mentioned this pull request Jan 15, 2025

DBZ-8564 Switch percona server to quay.io #6098

Merged

Uh oh!

Conversation

ismailsimsek commented Nov 20, 2024 • edited by jpechane Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 20, 2024

Uh oh!

github-actions bot commented Nov 20, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ismailsimsek commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ismailsimsek commented Nov 28, 2024

Uh oh!

jpechane left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jpechane Dec 11, 2024

Choose a reason for hiding this comment

Uh oh!

ismailsimsek Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jpechane commented Dec 11, 2024

Uh oh!

ismailsimsek Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ismailsimsek commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jpechane commented Jan 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ismailsimsek commented Nov 20, 2024 •

edited by jpechane

Loading

ismailsimsek commented Nov 22, 2024 •

edited

Loading

ismailsimsek Dec 11, 2024 •

edited

Loading

ismailsimsek Dec 11, 2024 •

edited

Loading

ismailsimsek commented Dec 11, 2024 •

edited

Loading