Overview

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Versatile Data Kit enables Data Engineers to develop, deploy, run and manage Data Jobs. A Data Job is a data processing workload and can be written in Python, SQL, or both at the same time. A Data Job enables Data Engineers to implement automated pull ingestion (E in ELT) and batch data transformation (T in ELT) into a database or any type of data storage.

Versatile Data Kit consists of two main components:

• A Data SDK provides all tools for the automation of data extraction, transformation, and loading, as well as a plugin framework that allows users to extend the framework according to their specific requirements.
• A Control Service allows users to create, deploy, manage and execute Data Jobs in Kubernetes runtime environment.

To help solve common data engineering problems Versatile Data Kit:

• allows ingestion of data from different sources, including CSV files, JSON objects, data provided by REST API services, etc.;
• ensures data applications are packaged, versioned, and deployed correctly while dealing with credentials, retries, reconnects, etc.;
• provides built-in monitoring and smart notification capabilities;
• tracks both code and data modifications and the relations between them, enabling engineers to troubleshoot faster and providing an easy revert to a stable version.

Data Journey and where VDK fits in

Installation and Getting Started

Install Versatile Data Kit SDK

pip install -U pip setuptools wheel
pip install quickstart-vdk

Note that Versatile Data Kit requires Python 3.7+.

See the Installation page for more details.

Use

# see Help to see what you can do
vdk --help

Check out the Getting Started page to create and run your first Data Job.

Documentation

Official documentation for Versatile Data Kit can be found here.

Contributing

If you are interested in contributing as a developer, visit CONTRIBUTING.md.

Contacts

Feedback is very welcome via the GitHub site as issues or pull requests

• Join our dedicated Slack channel on the CNCF Slack workspace:
  • First Join CNCF Slack workspace
  • Then search for #versatile-data-kit or click here to join the channel

• Follow us on twitter.

• Subscribe to the Versatile Data Kit YouTube Channel.

• Join our mailing list

How to use Versatile Data Kit?

• Video Data Ingestion with Versatile Data Kit
• Video Data Transformation with Versatile Data Kit
• Blog Post A complete example using the Versatile Data Kit and Trino DB

For the full list of resources go to Community and Resources

Code of Conduct

Everyone involved in working on the project's source code, or engaging in any issue trackers, Slack channels and mailing lists is expected to follow the Code of Conduct.