README.md

Overview

This Confluent Cloud demo showcases a hybrid Kafka deployment: one cluster is a self-managed cluster running locally, the other is a |ccloud| cluster. Data streams into topics both a local cluster and a cluster in Confluent Cloud, and Replicator copies the on-prem data to Cloud so that stream processing can happen in the Cloud.

Note: if you are looking for code examples of producers writing to and consumers reading from Confluent Cloud, or producers and consumers using Avro with Confluent Schema Registry, checkout the clients subdirectory of examples. It provides client examples written in various programming languages.

This automated demo is an expansion of the KSQL Tutorial. Instead of the Kafka cluster backing the KSQL stream processing running on your local install, it runs on your Confluent Cloud cluster. There are also additional Confluent Platform components including Confluent Control Center and Confluent Replicator.

End-to-end Solution

This demo showcases:

• Confluent Cloud
• Confluent Cloud Schema Registry
• KSQL
• Confluent Replicator
• Confluent Control Center
• Kafka Connect
• kafka-connect-datagen connectors

Automated Configurations

It also includes a script that reads the Confluent Cloud configuration in $HOME/.ccloud/config and writes delta configuration files into ./delta_configs for Confluent Platform components and clients connecting to Confluent Cloud. See the template examples for examples of lines to add to each component or client.

• Confluent Platform Components:
  • Confluent Schema Registry
  • KSQL Data Generator
  • KSQL server
  • Confluent Replicator (standalone binary)
  • Confluent Control Center
  • Kafka Connect
• Kafka Clients:
  • Java (Producer/Consumer)
  • Java (Streams)
  • Python
  • .NET
  • Go
  • Node.js (https://github.com/Blizzard/node-rdkafka)
  • C++

Prerequisites

Local

As with the other demos in this repo, you may run the entire demo end-to-end with ./start.sh, and it runs on your local Confluent Platform install. This requires the following:

• Common demo prerequisites
• Confluent Platform 5.2
• Confluent Cloud CLI
• An initialized Confluent Cloud cluster used for development only
• Maven to compile the data generator, i.e. the KafkaMusicExampleDriver class
• jq installed on your machine

To run the local setup:

$ ./start.sh

Docker

• Docker version 17.06.1-ce
• Docker Compose version 1.14.0 with Docker Compose file format 2.1
• Confluent Cloud CLI
• An initialized Confluent Cloud cluster used for development only
• jq installed on your machine

To run the Docker setup:

$ ./start-docker.sh

NOTE: Do not run this demo against your production Confluent Cloud cluster. Run this demo only in a development cluster.

Documentation

You can find the documentation for running this demo, and its accompanying playbook, at https://docs.confluent.io/current/tutorials/examples/ccloud/docs/index.html