1. Prerequisites

💡	Please check if you have all required tools installed on your computer

• JDK 8

• Scala 2.11.12

• SBT

• Docker

• Docker compose

2. Run unit tests

sbt test

3. Run BigData solution

💡	I strongly encourage you to install docker plugin for IntelliJ

• Run docker containers with infrastructure

  docker-compose up -d

• Create Scylla schema

  docker-compose exec cassandra cqlsh -f /schema.cql

  ❗	Give the Scylla DB some time to run before you start creating the database schema

• Run Spark application

  sbt run

• Send example messages to the Kafka

  send_messages_to_kafka.sh
  # For Windows
  send_messages_to_kafka.ps1

• Go to http://localhost:4040 in order to see how Spark run the computations

• Check Scylla tables

  docker-compose exec cassandra cqlsh -e 'select * from trap.sums'
  docker-compose exec cassandra cqlsh -e 'select * from trap.sums_counter'

Inspired by

🔥	Paul Leclercq https://github.com/polomarcus/Spark-Structured-Streaming-Examples

Credits

🔥	Mariusz Dubielecki [email protected] http://cranberrysoft.com