DBZ-8319 - Debezium readonly connection by lgazire · Pull Request #5949 · debezium/debezium

lgazire · 2024-10-23T20:21:10Z

/packit test --labels oracle

github-actions · 2024-10-23T20:21:24Z

Hi @lgazire, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

lgazire · 2024-10-23T20:24:25Z

@jpechane what am i missing here? Is there something wrong with the message that i set?

Naros · 2024-10-24T01:38:09Z

Hi @lgazire please prefix the commit with DBZ-8319 and that will fix the failure.

Naros · 2024-10-24T01:38:30Z

/packit test --labels oracle

Naros

Hi @lgazire, I haven't had a chance to look at the test failures yet, but I wanted to give some quick feedback on the implementation.

Also, a small nit-pick, can we use the notion of ReadOnly rather than Readonly in various places? To me the subtle capitalization just makes more logical sense from a naming perspective.

Finally, I noticed that the Jira issue specifically refers to non-CDB. Is there a reason why this needs to be scoped to just non-CDB instead of supporting both CDB and non-CDB? Oracle enforces CDB as the default since Oracle 19 and in fact they're predominately moving toward CDB being the only installation method, so not having a CDB solution to me is concerning.

...connector-oracle/src/main/java/io/debezium/connector/oracle/DualOracleConnectionFactory.java

Naros · 2024-10-24T05:33:54Z

debezium-connector-oracle/src/main/java/io/debezium/connector/oracle/OracleConnectorConfig.java

+    public static final Field LOG_MINING_PATH_DICTIONARY = Field.create("log.mining.path.dictionary")
+            .withDisplayName("Defines the dictionary path for the mining session")
+            .withType(Type.STRING)
+            .withWidth(Width.LONG)
+            .withImportance(Importance.LOW)
+            .withValidation(OracleConnectorConfig::validateDictionaryFromFile)
+            .withDescription("Path to LogMiner dictionary file. This is required when the database is in" +
+                    " a different host than the Debezium connector");


I don't believe the description here is technically accurate. The dictionary file is required because the connector is capturing changes from a read-only replica where the data dictionary details may not be available and where LogMiner is unable to write to the target database. Perhaps it makes more sense to simply say "this is required when using the connector against a read-only database replica."

Naros · 2024-10-24T05:37:18Z

debezium-connector-oracle/src/main/java/io/debezium/connector/oracle/OracleConnectorConfig.java

+    public static final Field LOG_MINING_READONLY_HOSTNAME = Field.create("log.mining.readonly.hostname")
+            .withDisplayName("Read-only hostname for the Oracle RAC cluster.")
+            .withType(Type.STRING)
+            .withWidth(Width.MEDIUM)
+            .withImportance(Importance.LOW)
+            .withDescription("The readonly hostname for the Oracle RAC cluster. This is used to determine the primary node in the RAC cluster.");


Lets avoid the use of Oracle RAC here, that's misleading and unnecessary. I'd also rephrase the description simply "The hostname the connector will use to connect and perform read-only operations for the the replica" or something like this.

debezium-connector-oracle/src/main/java/io/debezium/connector/oracle/OracleConnectorConfig.java

Naros · 2024-10-24T05:49:26Z

...um-connector-oracle/src/main/java/io/debezium/connector/oracle/ReadonlyOracleConnection.java

+    @Override
+    public JdbcConnection prepareUpdate(String stmt, StatementPreparer preparer) {
+        throw new UnsupportedOperationException("Updates are not allowed for read-only connections");
+    }
+
+    @Override
+    public JdbcConnection executeWithoutCommitting(String... statements) {
+        throw new UnsupportedOperationException("Updates are not allowed for read-only connections");
+    }


It seems odd to avoid the use of executeWithoutCommitting versus allowing execute. Both are capable of making modifications to the underlying database in their own ways, no?

Yes, you are correct, i will make the changes.

debezium-connector-oracle/src/main/java/io/debezium/connector/oracle/logminer/SqlUtils.java

github-actions · 2024-10-27T13:58:39Z

Hi @lgazire, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

github-actions · 2024-11-08T13:11:57Z

Hi @lgazire, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

lgazire · 2024-11-08T13:13:42Z

After a while (sorry, busy with work) i changed the commit message, should i do something else?

Naros · 2024-11-20T13:24:24Z

/packit test --labels oracle

lgazire · 2025-05-29T13:53:13Z

@Naros can you help me with what is missing here with the testing-farm? I can’t simulate it.

jpechane · 2025-05-30T04:44:04Z

/packit test --labels oracle

jpechane · 2025-05-30T04:44:56Z

@lgazire Thanks for coming back. I re-triggered the tests so we can fresh results.
Do you please plan to rebase the PR on the latest main?

DBZ-7051 Use DebeziumException for null input validations. Replaced IllegalArgumentException with DebeziumException in tests to align with the framework's exception handling standard. Refactor import order in NamingStyleUtilsTest.

This update clarifies and enhances the documentation for column naming strategies by introducing new properties, such as `column.naming.style.enabled`, `column.naming.style`, `column.naming.prefix`, and `column.naming.suffix`.

Replaced direct property injection with explicit configuration methods in column and collection naming strategies, improving clarity and maintainability. Enhanced debug logging to provide better visibility into the naming transformation process and handled null cases for test scenarios in collection naming. Added fallback mechanisms and warnings for deprecated or invalid configurations.

… Redis

relates to https://issues.redhat.com/browse/DBZ-7996

+ added docs for `-Dsource.connectors` argument to README.md + added shortcut "all" for `-Dsource.connector=all` to enable all source connectors (MySQL, Postgres, SQL Server and Oracle DB) for end-to-end tests relates to https://issues.redhat.com/browse/DBZ-7996

- remove docs for ConvertCloudEventToSaveableForm SMT relates to https://issues.redhat.com/browse/DBZ-7996

* address PR comments: ** rename `jdbc.type.Type` to `jdbc.type.JdbcType` ** fix kafkaCoordinates not always included in `allFields` but in a separate `kafkaFields` Map ** KafkaDebeziumSinkRecord not extend Kafka's Values class but use an inner class instead ** minor related fixes relates to https://issues.redhat.com/browse/DBZ-7996

Thee config optiones were already released, thus we should keep them, just make them deprecated. Also add more validations.

Now it's correctly handled by deprecation field option.

Handle nested, complex and empty arrays and documents; unify array schema for array encoding

… tests

…formEvents

…is sink

Address comments

Naros · 2025-05-30T12:02:32Z

Hi @lgazire can we just rebase the few commits you originally had on top of main rather than importing hundreds of commits here?

lgazire · 2025-05-30T18:01:46Z

Hi @Naros, @jpechane ! I did a git fetch and a rebase, after that i did a git push --force-with-lease.
I don’t know what i’m missing that is causing the list of commits that are not mine. Can you guys light me up?

jpechane · 2025-06-02T05:59:19Z

@lgazire Hi, given current state of affairs I'd recommend you to reset --hard your branch on the latest main and then cherry-pick your commits on top of it and then force push to GitHub. This should ensure that the PR will again show only your new commits.

The other option is that you'll jsut close this PR. Create a new branch off the latest main, cherry pick commits and open a new PR.

Naros requested changes Oct 24, 2024

View reviewed changes

lgazire force-pushed the DBZ-8319 branch from 3f0f2fd to 778245e Compare November 8, 2024 13:11

lgazire force-pushed the DBZ-8319 branch from 778245e to 50a70fc Compare November 8, 2024 14:11

lgazire requested a review from Naros November 8, 2024 14:28

Naros and others added 15 commits May 30, 2025 07:44

DBZ-8874 Fix test to skip based on adapter rule

63e81ad

DBZ-8921 Remove extra filter check for non-hybrid strategies

2b435e9

DBZ-8921 Change log level

b41bf3a

DBZ-7051 Pass connector configuration to Column/Table naming strategies

c2e5b5d

DBZ-7051 Pass connector configuration to Column/Table naming strategies

e001086

DBZ-7051 Pass connector configuration to Column/Table naming strategies

169857c

DBZ-7051 Pass connector configuration to Column/Table naming strategies

f4e5e8b

DBZ-7051 Only rename fields in before/after

8dba10d

DBZ-7051 Apply suggested changes

b2155cf

DBZ-7051 Document transforms

355ea26

DBZ-8927 Add MariaDB connector link to Installation Guide

bb2b351

DBZ-8911 docs: Add configuration option to skip heartbeat messages in…

2bcafe8

… Redis

vjuranek and others added 24 commits May 30, 2025 08:21

DBZ-9063 Allow some volatility in embeddings comparison

0b12578

DBZ-7996 Part 1 moving to a shared sink connector framework

10f86a3

relates to https://issues.redhat.com/browse/DBZ-7996

DBZ-7996 Part 1 moving to a shared sink connector framework

9a07b0b

- remove docs for ConvertCloudEventToSaveableForm SMT relates to https://issues.redhat.com/browse/DBZ-7996

DBZ-7996 Fix Db2 test failure

f941d61

DBZ-9064 Set proper validator when setting up deprecated allias

c55d9aa

DBZ-9056 Remove embeddgins prefix from embeddgins SMT config

9c02c89

DBZ-9056 Preserve existing config options

0c6fbd8

Thee config optiones were already released, thus we should keep them, just make them deprecated. Also add more validations.

DBZ-9056 Remove deprecated field validation

a15641f

Now it's correctly handled by deprecation field option.

DBZ-9077 enforce CI to build all dependent modules

96a989c

DBZ-5920 Refactor MongoDB ExtractNewDocumentState SMT

8eeea61

Handle nested, complex and empty arrays and documents; unify array schema for array encoding

DBZ-5920 Add suggestions from code review, use multi-line strings for…

997202a

… tests

DBZ-5920 Update Mongodb outbox transform schema building logic

bfd3ff1

DBZ-5920 Revert changes for ExtractNewDocumentStateTestIT#shouldTrans…

49921b0

…formEvents

DBZ-9075 move packages from io.quarkus.* to io.debezium.*

3737c0c

DBZ-9025 New release process for Maven Central

20a2da0

DBZ-9086 Fix Postgres reselectColumns for serial primary key

e988b16

DBZ-9085 Exclude TRUNCATE and REPLACE statements from schema history

63ebc49

DBZ-9082 Add option to specify custom keystore and truststore for Red…

0e4bd3e

…is sink

DBZ-9082 Add documentation for keystore/truststore properties

c6d820e

DBZ-9082 Refactor creation of RedisConnection

fa6cbe3

DBZ-9070 Execute storage settings check in case storage pre-exists

9e43933

Address comments

[release] Changelog for 3.2.0.Beta1

7ce6464

lgazire force-pushed the DBZ-8319 branch from b458933 to 7ce6464 Compare May 30, 2025 12:00

lgazire closed this Jun 3, 2025

lgazire deleted the DBZ-8319 branch June 3, 2025 14:31

Uh oh!

Conversation

lgazire commented Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

lgazire commented Oct 23, 2024

Uh oh!

Naros commented Oct 24, 2024

Uh oh!

Naros commented Oct 24, 2024

Uh oh!

Naros left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Naros Oct 24, 2024

Choose a reason for hiding this comment

Uh oh!

Naros Oct 24, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Naros Oct 24, 2024

Choose a reason for hiding this comment

Uh oh!

lgazire Oct 27, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 27, 2024

Uh oh!

github-actions bot commented Nov 8, 2024

Uh oh!

lgazire commented Nov 8, 2024

Uh oh!

Naros commented Nov 20, 2024

Uh oh!

lgazire commented May 29, 2025

Uh oh!

jpechane commented May 30, 2025

Uh oh!

jpechane commented May 30, 2025

Uh oh!

Naros commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lgazire commented May 30, 2025

Uh oh!

jpechane commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

lgazire commented Oct 23, 2024 •

edited

Loading

Naros left a comment •

edited

Loading

Naros commented May 30, 2025 •

edited

Loading

jpechane commented Jun 2, 2025 •

edited

Loading