Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lightMLFlow 0.7.0 #25

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

lightMLFlow 0.7.0 #25

wants to merge 13 commits into from

Conversation

mrkaye97
Copy link
Contributor

@mrkaye97 mrkaye97 commented Jul 8, 2022

  • Migrates away from aws.s3 in favor of paws.storage
  • Adds functionality for using S3 Select

@tonyelhabr now if you hypothetically had a 10GB CSV file in S3 with CF scores, you can slice and dice it. Unfortunately, I don't think S3 Select let's you group by so I think you'd need to load the data into memory regardless :/

Given the inability to group by and stuff, I may not merge this in. But leaving it here at least to come back to later in case the paws migration seems like a particularly great idea one day.

@mrkaye97 mrkaye97 requested a review from tonyelhabr July 8, 2022 21:58

bucket <- str_extract(
without_s3_prefix,
".+?(?=/)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regex go brr

@@ -0,0 +1,346 @@
#' @importFrom stringr str_extract
generate_s3_key_bucket_ext <- function(artifact_name, run_id = get_active_run_id(), client = mlflow_client()) {
Copy link
Contributor

@tonyelhabr tonyelhabr Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does _ext signify extension? maybe _info is better given the list that's returned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants