Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Week 3 Project Submission #53

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Prev Previous commit
Next Next commit
Add static partitioned config for docker
  • Loading branch information
ianyoung committed Sep 4, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 9e4757604f445d65db0a736a8d8512bd0e64e5d0
12 changes: 10 additions & 2 deletions week_3/project/week_3.py
Original file line number Diff line number Diff line change
@@ -92,8 +92,16 @@ def week_3_pipeline():
}


def docker_config():
pass
# Create a fixed number of partitions (1-10)
partition_keys = [str(i) for i in range(1, 11)]

@static_partitioned_config(partition_keys=partition_keys)
def docker_config(partition_key: str):
key = f'prefix/stock_{partition_key}.csv'
return {
"resources": {**docker["resources"]},
"ops": {"get_s3_data": {"config": {"s3_key": key }}}
}


local_week_3_pipeline = week_3_pipeline.to_job(