Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature](cdc) add DB2 database sync #316

Merged
merged 42 commits into from
Jul 26, 2024
Merged

Conversation

vinlee19
Copy link
Contributor

@vinlee19 vinlee19 commented Feb 18, 2024

Proposed changes

In this PR, it will be supported to synchronize the snapshot data and incremental data from Db2 to Apache Doris.

<FLINK_HOME>/bin/flink run \
     -Dexecution.checkpointing.interval=10s \
     -Dparallelism.default=1 \
     -c org.apache.doris.flink.tools.cdc.CdcTools \
     ./lib/flink-doris-connector-1.17-1.6.2-SNAPSHOT.jar \
     db2-sync-database \
     --database db2_test\
     --sqlserver-conf hostname=127.0.0.1 \
     --sqlserver-conf port=50000 \
     --sqlserver-conf username=db2inst1 \
     --sqlserver-conf password="doris123456" \
     --sqlserver-conf database-name=testdb \
     --sqlserver-conf schema-name=DB2INST1 \
     --including-tables "tbl1|tbl2" \
     --sink-conf fenodes=127.0.0.1:8030 \
     --sink-conf username=root \
     --sink-conf password=\
     --sink-conf jdbc-url=jdbc:mysql://127.0.0.1:9030 \
     --sink-conf sink.label-prefix=label \
     --table-conf replication_num=1

It should be noted that when you configure --db2-conf scan.incremental.snapshot.enabled=true in shell, please use Flink CDC version 3.1 or above.
Issue Number: close #xxx

Problem Summary:

Describe the overview of changes.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@vinlee19 vinlee19 changed the title [Draft](cdc) add DB2 database sync [wip](cdc) add DB2 database sync Feb 23, 2024
vinlee19 added 4 commits July 23, 2024 16:34
# Conflicts:
#	flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/JsonDebeziumSchemaChangeImplV2.java
@vinlee19 vinlee19 marked this pull request as ready for review July 23, 2024 11:42
@vinlee19 vinlee19 changed the title [wip](cdc) add DB2 database sync [Feature](cdc) add DB2 database sync Jul 23, 2024
@JNSimba
Copy link
Member

JNSimba commented Jul 24, 2024

Please rebase master

# Conflicts:
#	flink-doris-connector/pom.xml
#	flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/postgres/PostgresDatabaseSync.java
@JNSimba
Copy link
Member

JNSimba commented Jul 25, 2024

There are some conflicts in the code @vinlee19 PTAL

vinlee19 added 2 commits July 25, 2024 10:46
# Conflicts:
#	flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/JsonDebeziumChangeUtils.java
@vinlee19
Copy link
Contributor Author

There are some conflicts in the code @vinlee19 PTAL

done

vinlee19 added 3 commits July 25, 2024 18:33
# Conflicts:
#	flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/CdcTools.java
#	flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/SourceConnector.java
Copy link
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JNSimba JNSimba merged commit f559ea2 into apache:master Jul 26, 2024
6 checks passed
@JNSimba JNSimba mentioned this pull request Aug 27, 2024
PeatBoy pushed a commit to hyperpaas/doris-flink-connector that referenced this pull request Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants