Added implementation for Google cloud storage #120
Conversation
Animagne
requested review from
amikoliunas and
robertas-kerpys
as code owners
storage/google/README.md
Outdated
|
|
||
| ## About this package | ||
|
|
||
| This package contains implementations for object storage interfaces exposed by `@itwin/object-storage-core` which allow to consume Azure Blob Storage service. Communication with the Blob Service is implemented using using [Azure Storage Blob client library](https://www.npmjs.com/package/@azure/storage-blob). |
storage/google/package.json
Outdated
| { | ||
| "name": "@itwin/object-storage-google", | ||
| "version": "2.2.5", | ||
| "description": "Object storage implementation using Azure Blob Storage", |
| FrontendUrlDownloadInput, | ||
| } from "@itwin/object-storage-core/lib/frontend"; | ||
|
|
||
| export const uploadFileSizeLimit = 5_000_000_000; // 5GB |
There was a problem hiding this comment.
Is this limit set somewhere?
There was a problem hiding this comment.
It's not used. I've removed it and this file and merged the changes into original frontend Helpers file.
| 100 | ||
| )) { | ||
| for (const object of objectPage) { | ||
| await this.deleteObject(object); |
There was a problem hiding this comment.
If an error were to occur during this deletion, is there error handling above to recover?
Also, this waits for each object to be deleted one at a time, it might be more efficient to gather all of the promises into an array and await all simultaneously
There was a problem hiding this comment.
If some files failed to be deleted, you could retry it again and delete rest of the files.
We will see how the performance is in real use cases, because right now we don't have any performance metrics. Because this has to conform to the same deleteBaseDirectory method signature as other clients, we can't expose page size and deletion in parallel could be more taxing on the storage.
| }); | ||
| const client = await googleAuth.getClient(); | ||
| const downscopedClient = new DownscopedClient(client, cab); | ||
| const { token, expirationTime } = await downscopedClient.getAccessToken(); |
There was a problem hiding this comment.
Are there any tests that validate the access tokens returned by this function/the roles that token possesses? Curious to see how they look, I know that google uses its own JWT format
There was a problem hiding this comment.
We have integration tests shared across all of the packages, for example this describe would test with these tokens. The tests can be ran through VSCode and you can use just default application credentials if you're logged in with your account.
| scopes: "https://www.googleapis.com/auth/cloud-platform", | ||
| projectId: this._config.projectId, | ||
| }); | ||
| const client = await googleAuth.getClient(); |
There was a problem hiding this comment.
getClient() uses default access creds, which can be obtained from gcloud, but in a service setting we would need to use env.GOOGLE_APPLICATION_CREDENTIALS and provide a service key correct? Have we decided on the best way to obtain a service account service key in our pipelines?
There was a problem hiding this comment.
I don't know if it's necessarily the best long term solution, but right now we just store those credentials as secret files in Azure DevOps.
Implementation for google cloud storage. The biggest gap between this and our other implementations is that metadata setting for objects is not supported in some of the workflows, because it is made in a separate request.