DBZ-4903 Allow schema retrieval when schema is missing during incremental snapshot#5962
DBZ-4903 Allow schema retrieval when schema is missing during incremental snapshot#5962mfvitale merged 1 commit intodebezium:mainfrom nathan-smit-1:DBZ-4903
Conversation
|
The test failures for the JDBC storage, JDBC sink connector, and Spanner are related to fixes in #5963 and debezium/debezium-connector-spanner#98 |
Thanks Chris. Struggling to check all the error messages for the ones that failed on my side. Some seem like they can't be related to my change. Maybe the sql server one. |
|
@nathan-smit-1 Please wait couple of hourse. @mfvitale is working on additonal failures in non-core repos like debezium/debezium-connector-spanner#99 When all are merged the please rebase upon the latest main and you should get back to reasonable results. |
Okay cool thanks @jpechane sounds good. |
Looks a lot better now, but still two failures. Will try and build those locally and see where it's failing as they failed on the last one as well. |
|
Okay, so it's just the DB2 check that I'm failing now but seems to be same failure across different pull requests so I don't think related to my change. Was just thinking though that all the tests I'm passing would mainly be the negative testing to make sure invalid values don't crash the connector. Not sure if testing should be added to actually see if the snapshot works on a new table? Would have to be something like create table ---> add new table to config ---> restart connector ---> send signal ---> check output? |
|
@nathan-smit-1 Yes, we definitely need happy path test. So yes, it would be exactly as you've described. |
"happy path test" is a great description for it. I'll work on that later today. |
|
Hi @nathan-smit-1, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key. |
@jpechane I added my happy path test last night. I see I have two failing checks but I don't think these are related to my change? |
|
I can seemingly never pass the SQL Server tests. Even locally I'm eventually getting this: I did pass all the incremental snapshot tests though: Test set: io.debezium.connector.sqlserver.IncrementalSnapshotIT Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1913 s -- in io.debezium.connector.sqlserver.IncrementalSnapshotIT |
|
@Naros just wanted to check if there is anything more I need to do here for now or if I must just wait for a reviewer? |
|
@mfvitale Could you please review and merge this one please? Also I am quite surprised that the new test works with all exisiting connectors. |
|
@jpechane yeah, might be worth double-checking but I basically just re-used an existing test but with different input parameters which might be why. |
|
@nathan-smit-1 I'll take a look tomorrow. |
...ezium/pipeline/source/snapshot/incremental/AbstractIncrementalSnapshotChangeEventSource.java
Outdated
Show resolved
Hide resolved
|
/packit test --label oracle |
The problem here is that in your test you are using the |
|
@nathan-smit-1 I started the Oracle tests, let's see the result. I just added some minor comment for now but I'm continuing the review. |
|
/packit test --identifier oracle-19-xs |
|
@mfvitale I fixed that initial test for sql server for IncrementalSnapshotCollationSortOrderMismatchIT but then I also failed locally on a later test so need to do a few more checks there. I'm not sure how to interpret the Oracle results as I see some passed and some failed Edit: Looks like all tests passed for the latest commit so it's just the Oracle stuff then |
...ezium/pipeline/source/snapshot/incremental/AbstractIncrementalSnapshotChangeEventSource.java
Outdated
Show resolved
Hide resolved
|
/packit test --labels oracle |
|
/packit test --identifier oracle-23 |
|
@mfvitale latest checks done. Failure seems unrelated I think |
|
/packit test -labels oracle |
|
/packit test --identifier oracle-21 |
|
/packit test --identifier oracle-21-xs |
|
@jpechane I think that test failures are not related and the code LGTM. Do your want to have the last look? |
|
@mfvitale I am fine with your judgement |
|
@nathan-smit-1 tests seems to be unrelated. LGTM. |
|
Applied! Thanks! |
4 participants
https://issues.redhat.com/browse/DBZ-4903
Hi there! I wanted to take a stab at this, so I resurrected my extremely old ticket, hopefully that's okay? I believe there are other tickets as well that deal with a similar idea.
What I'm trying to do is retrieve the schema when currentTable == null in the isTableInvalid. My thinking was that it could be similar to the process to create the schema when transactions are done on a table whose schema hasn't been captured yet.
I tested in mysql and oracle and seems to do the job, but definitely need someone to look over it and let me know if I'm on completely the wrong track or not. I could only find Oracle examples where the DDL is included in the schemaChangeEvent