In this course you will learn the fundamentals of dlt alongside some of the most important topics in the world of Pythonic data engineering.
Lesson 1: Quick Start
Discover what dlt is, run your first pipeline with toy data, and explore it like a pro using DuckDB, sql_client, and dlt datasets!
Lesson 2: dlt Resources and Sources
Learn to run pipelines with diverse data sources (dataframes, databases, and REST APIs),
master dlt.resource, dlt.source, and dlt.transformer, and create your first REST API pipeline!
Since it is never a good idea to publicly put your API keys into your code, different environments have different methods to set and access these secret keys. dlt is no different.
Master pagination and authentication for REST APIs, explore dlt's RESTClient and manage secrets and configs.
Now that you took a data source and loaded it into a duckdb destination, it is time to look into what other possibilities dlt offers.
In this notebook we will take a look at pre-built verified sources and destinations and how to use them.
Learn to control data behavior with dlt write dispositions (Append, Replace, Merge), master incremental loading, and efficiently update and deduplicate your datasets.
Lesson 6: How dlt works
Discover the magic behind dlt! Learn its three main steps — Extract, Normalize, Load — along with default behaviors and supported file formats.
Lesson 7: Inspecting & Adjusting Schema
dlt creates and manages the schema automatically, but what if you want to control it yourself? Explore the schema and customize it to your needs easily with dlt!
Lesson 8: Understanding Pipeline Metadata
After having learnt about pipelines and how to move data from one place to another. We now learn about information about the pipeline itself. Or, metadata of a pipeline that can be accessed and edited through dlt.
This notebook explores dlt states, what it collected and where this extra information is stored. It also expands a bit more on what the load info and trace in dlt is capable of.
You finished the dlt Fundamentals course! As a summary of everything we covered in this course, there’s a homework quiz for you to go through. Hopefully it’ll refresh your memory on everything we covered.
Here you go: https://forms.gle/LRohLeRYKsfCfQaTA
Submit your application before January 17th, 2025, score at least 70% of the points, and get certified!
We read all your feedback, and trying to apply your suggestions, so feel free to share your thoughts!
Move your comments and ideas there: https://forms.gle/yVpNBvqRpPUmjeSU6
If you have any questions or need further assistance, don’t hesitate to reach out—we’re here to help.