Stream metadata from S3, filter, and compress #92
Conversation
Instead of downloading the complete metadata files from S3, uncompressing them to disk, filtering the uncompressed file to the subset of columns we want into an uncompressed file, and deleting the full metadata, this commit proposes streaming the original metadata from S3 through the filter step and writing the subset of metadata to a compressed file. This change replaces the vendored `download-from-s3` script with `aws s3 cp` since the latter can stream to stdout while the former cannot because it also writes log messages to stdout. Testing with the "open" ncov dataset, this change avoids storing an 11GB full metadata file locally and produces a 71MB compressed subset metadata file in roughly the same amount of processing time.
|
@joverlee521 This is just a suggestion PR after I tried and failed to run the ingest workflow on my work laptop because I didn't have enough disk space to even process the open data. If this seems like a terrible idea for other reasons I'm naive about, don't hesitate to close this. |
There was a problem hiding this comment.
Thanks @huddlej for digging into this!
The changes look good to me, feel free to merge if the trial run completes successfully.
I do wonder if this subsetting of metadata is useful to add to the shared ingest/download-from-s3, but as of right now I can't think of any other workflow that would use this feature.
|
Thanks, @joverlee521! The trial run looks like it ran as expected, so I will merge this. I don't see a notification in the |
Ah yeah, I used the #scratch channel when I triggered the trial run so they wouldn't mix in with the "real" notifications. |
Description of proposed changes
Instead of downloading the complete metadata files from S3, uncompressing them to disk, filtering the uncompressed file to the subset of columns we want into an uncompressed file, and deleting the full metadata, this commit proposes streaming the original metadata from S3 through the filter step and writing the subset of metadata to a compressed file.
This change replaces the vendored
download-from-s3script withaws s3 cpsince the latter can stream to stdout while the former cannot because it also writes log messages to stdout.Testing with the "open" ncov dataset, this change avoids storing an 11GB full metadata file locally and produces a 71MB compressed subset metadata file in roughly the same amount of processing time.
Related issue
Closes #80
Checklist