movizclub_ETL_Pipeline

An end-to-end ETL Pipeline for Building User Behavior Metric Data Warehouse and Visualize Data by Metabase

Table of Contents

• Architecture Diagram
• References
• License

Architecture Diagram

ETL Flow

• Loading movie reviews into Amazon s3
• Classification process on movie reviews with Apache Spark
• Loading the classified movie reviews into the data warehouse
• Extract user purchase data from an OLTP database and load it into the data warehouse
• Joining the classified movie review data and user purchase data to get user behavior metric data

Reference

Inspired by following codes, articles and videos:

• Pyspark documentation
• Psycopg documentation
• Apache Airflow Documentation

License

Distributed under the MIT License. for more information see Lisence