Skip to content

Commit

Permalink
feat(shared): support multiple credential locations (#3407)
Browse files Browse the repository at this point in the history
### Motivation

We have a number of configuration files to allow reading and writing of
S3 buckets across accounts, sometimes it's useful to join a collection
of access lists together to provide a more comprehensive list of
credentials for the calling service.

### Modifications

Allow credentials to be loaded from multiple sources using either a
comma separated list or a JSON list, this behavior is the same as the
workflows inside of argo

https://github.com/linz/argo-tasks/blob/master/src/fs.register.ts#L132

### Verification

Logic is taken from argo tasks, and will be tested in argo.
  • Loading branch information
blacha authored Feb 21, 2025
1 parent 5e25afa commit 92de7c4
Showing 1 changed file with 17 additions and 1 deletion.
18 changes: 17 additions & 1 deletion packages/shared/src/file.system.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,24 @@ credentials.onFileSystemFound = (acc: AwsCredentialConfig, fs?: FsAwsS3, path?:
fsa.register(acc.prefix, fs);
};

/**
* Split JSON or comma separated lists into individual components
*
* Allowing for credentials to be loaded from multiple sources eg
*
* @example comma separated list
* ```typescript
* "s3://foo/bar.json,s3://foo/baz.json"
* ```
*/
function splitConfig(x: string): string[] {
if (x.startsWith('[')) return JSON.parse(x) as string[];
return x.split(',');
}
const credentialPath = Env.get(Env.AwsRoleConfigPath);
if (credentialPath) credentials.registerConfig(fsa.toUrl(credentialPath), s3Fs);
if (credentialPath) {
for (const loc of splitConfig(credentialPath)) credentials.registerConfig(fsa.toUrl(loc), s3Fs);
}

s3Fs.credentials = credentials;

Expand Down

0 comments on commit 92de7c4

Please sign in to comment.