Skip to content

Commit

Permalink
Release to Bintray
Browse files Browse the repository at this point in the history
  • Loading branch information
Jordan Moore committed Nov 6, 2019
1 parent cf5d375 commit b76f3a6
Show file tree
Hide file tree
Showing 4 changed files with 35 additions and 13 deletions.
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
Schema Registry Transfer SMT
Schema Registry Transfer SMT
============================
[ ![Download](https://api.bintray.com/packages/cricket007/maven-releases/schema-registry-transfer-smt/images/download.svg) ](https://bintray.com/cricket007/maven-releases/schema-registry-transfer-smt/_latestVersion) 
[![CircleCI](https://circleci.com/gh/cricket007/schema-registry-transfer-smt.svg?style=svg)](https://circleci.com/gh/cricket007/schema-registry-transfer-smt)

A [Kafka Connect Single Message Transformation (SMT)][smt] that reads the serialized [wire format header][wire-format] of Confluent's `KafkaAvroSerializer`, performs a lookup against a source [Confluent Schema Registry][schema-registry] for the ID in the message, and registers that schema into a destination Registry for that topic/subject under a new ID.

A [Kafka Connect Single Message Transformation (SMT)][smt] that reads the serialized [wire format header][wire-format] of Confluent's `KafkaAvroSerializer`, performs a lookup against a source [Confluent Schema Registry][schema-registry] for the ID in the message, and registers that schema into a destination Registry for that topic/subject under a new ID.

To be used where it is not feasible to make the destination Schema Registry as a follower to the source Registry, or when migrating topics to a new cluster.
To be used where it is not feasible to make the destination Schema Registry as a follower to the source Registry, or when migrating topics to a new cluster.

> _Requires that the Kafka Connect tasks can reach both Schema Registries._
This transform doesn't mirror the contents of the `_schemas` topic, so therefore each registry can be completely isolated from one another. As a side-effect of this, the subject configurations that might be applied to the `/config` endpoint in the source registry are not copied to the destination. In other words, you might get schema registration errors if using differing compatibility levels on the registries. Just a heads-up.
This transform doesn't mirror the contents of the `_schemas` topic, so therefore each registry can be completely isolated from one another. As a side-effect of this, the subject configurations that might be applied to the `/config` endpoint in the source registry are not copied to the destination. In other words, you might get schema registration errors if using differing compatibility levels on the registries. Just a heads-up.

Example Kafka Connectors where this could be applied.

Expand Down
20 changes: 16 additions & 4 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
<name>Jordan Moore</name>
<url>https://github.com/cricket007</url>
<roles>
<role>Committer</role>
<role>Owner</role>
</roles>
</developer>
</developers>
Expand All @@ -39,6 +39,13 @@
<system>github</system>
<url>https://github.com/cricket007/schema-registry-transfer-smt/issues</url>
</issueManagement>
<distributionManagement>
<repository>
<id>bintray</id>
<name>bintray-releases</name>
<url>https://api.bintray.com/maven/${env.BINTRAY_REPO_OWNER}/${env.BINTRAY_REPO}/${project.artifactId}/;publish=1</url>
</repository>
</distributionManagement>

<repositories>
<repository>
Expand All @@ -60,6 +67,7 @@
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>

<slf4j.version>1.7.25</slf4j.version>
<kafka.version>2.1.0</kafka.version>
<confluent.version>5.1.0</confluent.version>
<confluent.patch.version>-cp1</confluent.patch.version>
Expand All @@ -77,7 +85,7 @@
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.25</version>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>

Expand Down Expand Up @@ -212,7 +220,7 @@
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
<version>1.7.25</version>
<version>${slf4j.version}</version>
<scope>test</scope>
</dependency>

Expand Down Expand Up @@ -242,6 +250,10 @@
</plugins>
</pluginManagement>
<plugins>
<plugin>
<artifactId>maven-release-plugin</artifactId>
<version>2.5.3</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
Expand Down Expand Up @@ -339,4 +351,4 @@
</plugins>
</build>

</project>
</project>
11 changes: 11 additions & 0 deletions settings.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<?xml version='1.0' encoding='UTF-8'?>
<settings xsi:schemaLocation='http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd'
xmlns='http://maven.apache.org/SETTINGS/1.0.0' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'>
<servers>
<server>
<id>bintray</id>
<username>${env.BINTRAY_USER}</username>
<password>${env.BINTRAY_API_KEY}</password>
</server>
</servers>
</settings>
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
package cricket.jmoore.kafka.connect.transforms;

import static cricket.jmoore.kafka.connect.transforms.SchemaRegistryTransfer.ConfigName;
import static org.apache.avro.Schema.Type.BOOLEAN;
import static org.apache.avro.Schema.Type.INT;
import static org.apache.avro.Schema.Type.STRING;
import static org.junit.jupiter.api.Assertions.*;

import java.io.ByteArrayOutputStream;
Expand Down Expand Up @@ -38,10 +41,6 @@
import io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException;
import io.confluent.kafka.serializers.NonRecordContainer;

import static org.apache.avro.Schema.Type.INT;
import static org.apache.avro.Schema.Type.BOOLEAN;
import static org.apache.avro.Schema.Type.STRING;

@SuppressWarnings("unchecked")
public class TransformTest {

Expand Down

0 comments on commit b76f3a6

Please sign in to comment.