Skip to content

Commit

Permalink
[ISSUE #523 & OSPP2023] Feature: support archetype to create connecto…
Browse files Browse the repository at this point in the history 
…rs (#525)

* initial commit

* commit

* rename

* finish source connector

* finish sink connector

* add license

* Update README.md

* Update connect-standalone.conf

* support archetype

* Delete README.md
  • Loading branch information
@joeCarf
joeCarf authored Oct 17, 2023
1 parent cf49208 commit 8dcf2b3
Show file tree
Hide file tree
Showing 19 changed files with 1,220 additions and 0 deletions.
36 changes: 36 additions & 0 deletions connectors/rocketmq-connect-connectors-archetype/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
## How to Use Connnector-Archetype

1. 进入脚手架文件夹

```shell
cd rocketmq-connect-connectors-archetype/
```

2. 将脚手架安装到本地

```shell
mvn -e clean install
```

3. 创建connector模版工程

```shell
cd connectors/
mvn archetype:generate \
-DarchetypeGroupId=org.apache.rocketmq \
-DarchetypeArtifactId=rocketmq-connect-connectors-archetype \
-DarchetypeVersion=1.0-SNAPSHOT \
-DdatabaseName=<databasename>
```

例：创建Clickhouse-Connector

```shell
mvn archetype:generate \
-DarchetypeGroupId=org.apache.rocketmq \
-DarchetypeArtifactId=rocketmq-connect-connectors-archetype \
-DarchetypeVersion=1.0-SNAPSHOT \
-DdatabaseName=clickhouse
```

4. 如上指令将创建一个connector的框架，开发者主要关心`helper/xxxHelperClient`以及`xxxxSourceTask`,`xxxSinkTask`的实现即可，剩余配置可以按需修改。
37 changes: 37 additions & 0 deletions connectors/rocketmq-connect-connectors-archetype/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-connect-connectors-archetype</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>maven-archetype</packaging>

<name>rocketmq-connect-connectors-archetype</name>

<build>
<extensions>
<extension>
<groupId>org.apache.maven.archetype</groupId>
<artifactId>archetype-packaging</artifactId>
<version>3.2.1</version>
</extension>
</extensions>

<pluginManagement>
<plugins>
<plugin>
<artifactId>maven-archetype-plugin</artifactId>
<version>3.2.1</version>
</plugin>
</plugins>
</pluginManagement>
</build>

<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
</license>
</licenses>
</project>
66 changes: 66 additions & 0 deletions ...tmq-connect-connectors-archetype/src/main/resources/META-INF/maven/archetype-metadata.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
<?xml version="1.0" encoding="UTF-8"?>
<archetype-descriptor xsi:schemaLocation="https://maven.apache.org/plugins/maven-archetype-plugin/archetype-descriptor/1.1.0 http://maven.apache.org/xsd/archetype-descriptor-1.1.0.xsd" name="rocketmq-connect-connectors-archetype"
xmlns="https://maven.apache.org/plugins/maven-archetype-plugin/archetype-descriptor/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<requiredProperties>
<!--必填属性-->
<requiredProperty key="groupId">
<defaultValue>org.apache.rocketmq</defaultValue>
</requiredProperty>
<requiredProperty key="artifactId">
<defaultValue>rocketmq-connect-${dbNameToLowerCase}</defaultValue>
</requiredProperty>
<requiredProperty key="version">
<defaultValue>1.0.0-SNAPSHOT</defaultValue>
</requiredProperty>
<requiredProperty key="package">
<defaultValue>org.apache.rocketmq.connect.${dbNameToLowerCase}</defaultValue>
</requiredProperty>

<!-- The database to connect -->
<requiredProperty key="databaseName"/>

<requiredProperty key="dbNameToUpperCase" >
<defaultValue>${databaseName.toUpperCase()}</defaultValue>
</requiredProperty>

<requiredProperty key="dbNameToCamel" >
<defaultValue>${databaseName.toLowerCase().substring(0,1).toUpperCase()}${databaseName.toLowerCase().substring(1)}</defaultValue>
</requiredProperty>

<requiredProperty key="dbNameToLowerCase" >
<defaultValue>${databaseName.toLowerCase()}</defaultValue>
</requiredProperty>
</requiredProperties>

<fileSets>
<fileSet filtered="true" encoding="UTF-8">
<directory/>
<includes>
<include>.reviewboardrc</include>
<include>README.md</include>
</includes>
</fileSet>
<fileSet encoding="UTF-8">
<directory/>
<includes>
<include>.gitignore</include>
<include>TODO.md</include>
</includes>
</fileSet>
<fileSet encoding="UTF-8">
<directory/>
<includes>
<include>.gitignore</include>
</includes>
</fileSet>
<fileSet filtered="true" packaged="true" encoding="UTF-8">
<directory>src/main/java</directory>
</fileSet>
<fileSet filtered="true" packaged="true" encoding="UTF-8">
<directory>src/test/java</directory>
</fileSet>
</fileSets>

</archetype-descriptor>
55 changes: 55 additions & 0 deletions ...q-connect-connectors-archetype/src/main/resources/archetype-resources/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
## FIXME: fix document

##### ${dbNameToCamel}SourceConnector fully-qualified name

org.apache.rocketmq.connect.${dbNameToLowerCase}.source.${dbNameToCamel}SourceConnector

**${dbNameToLowerCase}-source-connector** start

```
POST http://${runtime-ip}:${runtime-port}/connectors/${dbNameToLowerCase}SourceConnector
{
"connector.class":"org.apache.rocketmq.connect.${dbNameToLowerCase}.source.${dbNameToCamel}SourceConnector",
"${dbNameToLowerCase}host":"localhost",
"${dbNameToLowerCase}port":8123,
"database":"default",
"username":"default",
"password":"123456",
"table":"tableName",
"topic":"test${dbNameToCamel}Topic",
"value.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter",
"key.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter"
}
```

##### ${dbNameToCamel}SinkConnector fully-qualified name

org.apache.rocketmq.connect.${dbNameToLowerCase}.sink.${dbNameToCamel}SinkConnector

**${dbNameToLowerCase}-sink-connector** start

```
POST http://${runtime-ip}:${runtime-port}/connectors/${dbNameToLowerCase}SinkConnector
{
"connector.class":"org.apache.rocketmq.connect.${dbNameToLowerCase}.sink.${dbNameToCamel}SinkConnector",
"${dbNameToLowerCase}host":"localhost",
"${dbNameToLowerCase}port":8123,
"database":"clickhouse",
"username":"default",
"password":"123456",
"connect.topicnames":"test${dbNameToCamel}Topic",
"value.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter",
"key.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter"
}
```

##### parameter configuration

| parameter | effect | required | default |
|--------------------------|---------------------------------------------------|-------------------|---------|
| ${dbNameToLowerCase}host | The Host of the ${dbNameToCamel} server | yes | null |
| ${dbNameToLowerCase}port | The Port of the ${dbNameToCamel} server | yes | null |
| database | The database to read or write | yes | null |
| table | The source table to read | yes (source only) | null |
| topic | RocketMQ topic for source connector to write into | yes (source only) | null |
| connect.topicnames | RocketMQ topic for sink connector to read from | yes (sink only) | null |
188 changes: 188 additions & 0 deletions ...tors/rocketmq-connect-connectors-archetype/src/main/resources/archetype-resources/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for additional
information regarding copyright ownership. The ASF licenses this file to
You under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of
the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
by applicable law or agreed to in writing, software distributed under the
License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
OF ANY KIND, either express or implied. See the License for the specific
language governing permissions and limitations under the License. --><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>${groupId}</groupId>
<artifactId>${artifactId}</artifactId>
<version>${version}</version>

<name>connect-${dbNameToLowerCase}</name>

<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
</license>
</licenses>

<issueManagement>
<system>jira</system>
<url>https://issues.apache.org/jira/browse/RocketMQ</url>
</issueManagement>

<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>versions-maven-plugin</artifactId>
<version>2.3</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>clirr-maven-plugin</artifactId>
<version>2.7</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<compilerVersion>${maven.compiler.source}</compilerVersion>
<showDeprecation>true</showDeprecation>
<showWarnings>true</showWarnings>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<argLine>-Xms512m -Xmx1024m</argLine>
<forkMode>always</forkMode>
<includes>
<include>**/*Test.java</include>
</includes>
</configuration>
</plugin>
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.6</version>
<configuration>
<locales>en_US</locales>
<outputEncoding>UTF-8</outputEncoding>
<inputEncoding>UTF-8</inputEncoding>
</configuration>
</plugin>
<plugin>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.4</version>
<configuration>
<charset>UTF-8</charset>
<locale>en_US</locale>
<excludePackageNames>io.openmessaging.internal</excludePackageNames>
</configuration>
<executions>
<execution>
<id>aggregate</id>
<goals>
<goal>aggregate</goal>
</goals>
<phase>site</phase>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<encoding>${project.build.sourceEncoding}</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>3.0.4</version>
</plugin>
<plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<version>0.12</version>
<configuration>
<excludes>
<exclude>README.md</exclude>
<exclude>README-CN.md</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>io.openmessaging</groupId>
<artifactId>openmessaging-connector</artifactId>
<version>0.1.4</version>
<scope>compile</scope>
</dependency>

<!-- FIXME: Write dependencies here-->

<dependency>
<groupId>org.lz4</groupId>
<artifactId>lz4-java</artifactId>
<version>1.8.0</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.83</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>RELEASE</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
</dependencies>

</project>
Loading

0 comments on commit 8dcf2b3

Please sign in to comment.