FR: Be able to upload a file/directory from any url

[barnabasbusa]

Background & motivation

It would be great if we would be able to natively upload files into files artifacts/pvc from any arbitrary url.

This feature will be heavily used for shadowfork, where the previous state of a public testnet will need to be fetched from an s3 bucket (or something similar).

Desired behaviour

I should be able to use an s3 endpoint as url, and the file should be saved into an artifact/pvc that can be mounted to a service later.

How important is this to you?

Critical; Kurtosis is unusable for me without it.

What area of the product does this pertain to?

CLI: the Command Line Interface

[barnabasbusa]

An additional feature request would be to support basic authentication methods. Such as passing custom headers, or api tokens, or username/password.

[galenmarchetti]

thank you @barnabasbusa ! we have an area of work around making files artifacts a lot more accessible here, tracking this along with that work

[barnabasbusa]

Are files artifacts meant to store TBs of data?

Ideally we would want to have parallel sync from s3 buckets into pvcs. Maybe we shouldn't even bother with storing anything in files artifacts ?

[mieubrisse]

Very interesting idea @barnabasbusa ; gets me thinking if we could treat files artifacts as references rather than something actually gets downloaded. Or, at minimum, when we pull in things from the web they could be (so that we do the expansion exactly like you're talking about). Ultimately, someone does have to download the data into the PVC, but maybe we should be streaming it (rather than doing a download-to-APIC-then-unpack).

[barnabasbusa]

when it's Terrabytes of data, you probably want to make sure you download the whole state before you proceed to the next step. Pull on request will work for small files, so will streaming.

My use case would be for files between 16GB-2.5TB

[mieubrisse]

Related to #2215 and #2216 , in that all of them could benefit from being able to use "implicit packages" - Github repos that don't have a kurtosis.yml file.