Feature request: serve presigned S3 URLs

[skogsbrus]

In order to browser hyperdataset entries, users must currently provide their own IAM access key in order to authenticate against S3. These are stored in the user's local storage.

A more secure approach, as to avoid the spread of access keys, would be for the server implementation to serve a presigned URLs of the S3 assets using a IAM access key or role-based auth on the server side. This way, the frontend doesn't need any authentication logic and users aren't required to have access to access keys.

[ainoam]

Thanks for suggesting @skogsbrus.

Though Hyper-Datasets are not part of ClearML OSS (and hence, are not addressed in this repo), the underlying mechanism also applies to other UI served resources such as debug samples.

The current implementation is an intentional security design choice: Storage access credentials never leave the user's domain (they are actually stored in your browser local store) so that while any team member can access the "science" (i.e. ClearML experiment results), any restricted data access is reserved to those authorized (e.g. even the server maintainer is not exposed to such credentials unless explicitly provided).

Hope this clarifies things :)

[skogsbrus]

Hi @ainoam , thanks for the response!

The implementation makes sense from that POV, but it would be nice to be able to choose. The current design requires all users to have access to a potentially high-value secret, whereas the solution with presigned URLs only requires one team/person to enroll the secret once. Access is then provided to all authorized users without having to distribute the secret. There's a much smaller risk of credentials leakage that way.

LMK if I should x-post this feature request elsewhere if this was a bad place!

[ainoam]

@skogsbrus

LMK if I should x-post this feature request elsewhere if this was a bad place!

Hyper-Datasets being a ClearML Enterprise feature, should be discussed in commercial support channels.