Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

week 3 project #90

Open
wants to merge 25 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
cf83146
first commit.
frankcholula Nov 23, 2022
b2dec37
added input and output.
frankcholula Nov 23, 2022
b0fc875
finihed challenge.
frankcholula Nov 23, 2022
fa965c8
fixed type hints.
frankcholula Nov 23, 2022
72502ae
fix type hint.
frankcholula Nov 23, 2022
f552435
finished week2 resources.
frankcholula Nov 28, 2022
d077dd2
implemented challenge need to ask OH proper success failure implement…
frankcholula Nov 29, 2022
17f2929
modify dockerfile for dbt run to finish.
frankcholula Dec 2, 2022
5e272d3
get challenge working with success and failure hooks.
frankcholula Dec 2, 2022
7989c59
added tags.
frankcholula Dec 2, 2022
fb79a50
removed unused type.
frankcholula Dec 2, 2022
1a678e2
fixed outs in dbt test ops.
frankcholula Dec 2, 2022
245448b
remove legacy log file.
frankcholula Dec 2, 2022
3f980fd
recover dagster user.
frankcholula Dec 2, 2022
f652265
rerecover dagster user.
frankcholula Dec 2, 2022
01c3861
remove committed user yml file.
frankcholula Dec 2, 2022
0207361
Merge branch 'dehume:master' into master
frankcholula Dec 2, 2022
a5dd786
need to fix sensor eval context.
frankcholula Dec 5, 2022
5dfd4a5
one more docker validation test.
frankcholula Dec 5, 2022
ba6679f
finished week3. still need to do challenge.
frankcholula Dec 7, 2022
bb5b9f0
remove unused imports.
frankcholula Dec 7, 2022
f7b1af2
change endpoint url.
frankcholula Dec 8, 2022
758da6d
fixed typo.
frankcholula Dec 8, 2022
17daf5a
typo fix.
frankcholula Dec 8, 2022
1dca73b
attempt at custom io manager.
frankcholula Dec 8, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
first commit.
  • Loading branch information
frankcholula committed Nov 23, 2022
commit cf8314603a2efb6094f439429b0e0e68fef428f3
23 changes: 15 additions & 8 deletions week_1/project/week_1.py
Original file line number Diff line number Diff line change
@@ -41,21 +41,28 @@ def csv_helper(file_name: str) -> Iterator[Stock]:
yield Stock.from_list(row)


@op
def get_s3_data():
pass
@op (
config_schema= {
"s3_key": String
}
)
def get_s3_data(context) -> List[Stock]:
s3_key = context.op_config["s3_key"]
return list(csv_helper(s3_key))


@op
def process_data():
pass

def process_data(context, stocks: List[Stock]) -> Aggregation:
highest_stock = max(stocks, key= lambda stock: stock.high)
return Aggregation(date=highest_stock.date,high=highest_stock.high)

@op
def put_redis_data():
def put_redis_data(context, aggs: Aggregation):
pass


@job
def week_1_pipeline():
pass
s3_data = get_s3_data()
processed = process_data(s3_data)
put_redis_data(processed)