AWS EKS IRSA for Zot #2853

huynguyen-anduin · 2024-12-26T09:12:03Z

huynguyen-anduin
Dec 26, 2024

I'm using S3 for storageDriver and DynamoDb for cacheDriver. Based on this documentation, env AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and credentials file are supported.

I'm trying to deploy Zot with IAM Roles for Service Account on my EKS cluster. But failed to create Pod with error:

{"level":"error","error":"operation error DynamoDB: CreateTable, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 404, RequestID: ***, api error UnknownError: UnknownError","tableName":"MyZotBlobTable","goroutine":1,"caller":"zotregistry.dev/zot/pkg/storage/cache/dynamodb.go:98","time":"2024-12-26T09:05:15.053656375Z","message":"failed to create table for cache"}

{"level":"error","error":"operation error DynamoDB: CreateTable, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 404, RequestID: ***, api error UnknownError: UnknownError","goroutine":1,"caller":"zotregistry.dev/zot/pkg/cli/server/root.go:76","time":"2024-12-26T09:05:15.053700839Z","message":"failed to init controller"}

Zot does not support IRSA for now or I'm missing something?

Confirmed that my IRSA has enough permission.

huynguyen-anduin · 2024-12-27T06:56:54Z

huynguyen-anduin
Dec 27, 2024
Author

Seems zot is not supported yet.

https://github.com/project-zot/zot/blob/main/pkg/storage/cache/dynamodb.go#L86

0 replies

rchincha · 2024-12-31T06:47:24Z

rchincha
Dec 31, 2024
Maintainer

@huynguyen-anduin

#2650

0 replies

huynguyen-anduin · 2025-01-03T09:34:41Z

huynguyen-anduin
Jan 3, 2025
Author

@rchincha Could you give more details? I checked the discussion but not get your point.

0 replies

rchincha · 2025-01-07T16:45:04Z

rchincha
Jan 7, 2025
Maintainer

@tamilhce

Since you are working on a PR in a related area, your feedback here?

1 reply

tamilhce Jan 9, 2025

In the referenced file dynamodb.go, you can see that the import statement uses the Go SDK github.com/aws/aws-sdk-go-v2/config. This SDK relies on aws-sdk-go-v2 version v1.32.7, which supports IAM roles for service accounts. For more details, you can refer to the AWS documentation: IAM roles for service accounts.
Let me know if you encounter any issues.

xakraz · 2025-01-16T13:39:30Z

xakraz
Jan 16, 2025

We face the same issue

zot-7546f948fd-btvth zot {"level":"error","error":"operation error DynamoDB: CreateTable, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post \"/\": unsupported protocol scheme \"\"","goroutine":1,"caller":"zotregistry.dev/zot/pkg/cli/server/root.go:76","time":"2025-01-16T13:34:04.896603284Z","message":"failed to init controller"}
zot-7546f948fd-btvth zot Error: operation error DynamoDB: CreateTable, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post "/": unsupported protocol scheme ""

FYI:

We are deploying Zot on Kubernetes (AWS EKS)
We are using IAM IRSA
We have no issue with the S3 remote storage access

0 replies

rchincha · 2025-01-17T06:58:57Z

rchincha
Jan 17, 2025
Maintainer

@huynguyen-anduin can you pls post your zot config (remove secrets before you do)

1 reply

xakraz Jan 17, 2025

@rchincha

Here is my config

Zot deployment

AWS Policy

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<MY-BUCKET>-k7zw422"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<MY-BUCKET>-k7zw422/*"
            ]
        },
        {
            "Action": [
                "dynamodb:CreateTable",
                "dynamodb:GetItem",
                "dynamodb:UpdateItem",
                "dynamodb:DeleteItem"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:dynamodb:*:<MY-ACCOUNT-ID>:table/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Which is attached to a role, mapped to EKS serviceAccount (I get the link and everything in the AWS Console)

Zot config

    {
      "storage": {
        "rootDirectory": "/var/lib/registry",
        "storageDriver": {
          "name": "s3",
          "rootdirectory": "/zot",
          "region": "eu-west-3",
          "bucket": "<MY-BUCKET>-k7zw422",
          "forcepathstyle": true,
          "secure": true,
          "skipverify": false
        },
        "dedupe": true,
        "remoteCache": true,
        "cacheDriver": {
          "name": "dynamodb",
          "endpoint": "https://<MY-ACCOUNT-ID>.ddb.eu-west-3.amazonaws.com",
          "region": "eu-west-3",
          "cacheTablename": "<PREFIX>-ZotBlobTable",
          "imageMetaTablename": "<PREFIX>-ZotImageMetaTable",
          "manifestDataTablename": "<PREFIX>-ZotManifestDataTable",
          "repoBlobsInfoTablename": "<PREFIX>-ZotRepoBlobsInfoTable",
          "repoMetaTablename": "<PREFIX>-ZotRepoMetadataTable",
          "userDataTablename": "<PREFIX>-ZotUserDataTable",
          "versionTablename": "<PREFIX>-ZotVersion"
        }
      },
...
}

Note

I tested this successfully with the IAM Policy Simulator

Results

But I get the same error in Zot logs at startup

zot-7d747599d-lghvj zot Error: operation error DynamoDB: CreateTable, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 404, RequestID: SEUES5VECKJNQ73ECP6HLGS4MBVV4KQNSO5AEMVJF66Q9ASUAAJG, api error UnknownError: UnknownError

zot-7d747599d-lghvj zot {"level":"error","error":"operation error DynamoDB: CreateTable, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 404, RequestID: SEUES5VECKJNQ73ECP6HLGS4MBVV4KQNSO5AEMVJF66Q9ASUAAJG, api error UnknownError: UnknownError","tableName":"<PREFIX>-ZotBlobTable","goroutine":1,"caller":"zotregistry.dev/zot/pkg/storage/cache/dynamodb.go:98","time":"2025-01-17T15:31:20.269968184Z","message":"failed to create table for cache"}

zot-7d747599d-lghvj zot {"level":"error","error":"operation error DynamoDB: CreateTable, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 404, RequestID: SEUES5VECKJNQ73ECP6HLGS4MBVV4KQNSO5AEMVJF66Q9ASUAAJG, api error UnknownError: UnknownError","goroutine":1,"caller":"zotregistry.dev/zot/pkg/cli/server/root.go:76","time":"2025-01-17T15:31:20.270011549Z","message":"failed to init controller"}

Notes

I troubleshooted with a side-car container:

Pod specs

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
  name: zot
  namespace: zot
spec:
  ...
    spec:
      containers:
        - name: aws-cli
          image: amazon/aws-cli:latest
          imagePullPolicy: IfNotPresent
          command: ["/bin/bash"]
          args: ["-c", "while true; do sleep 30; done;"]
          resources: {}
          securityContext: {}
          volumeMounts:
            - mountPath: /etc/zot
              name: zot-config
            - mountPath: /secret
              name: zot-secret

        - name: zot
          image: ghcr.io/project-zot/zot-linux-amd64:v2.1.1

Get a shell in the pod

kubectl -n zot exec -it svc/zot -- /bin/bash

Check the IRSA integration

printenv | grep -i aws

AWS_ROLE_ARN=arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME>
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_DEFAULT_REGION=eu-west-3
PWD=/aws
AWS_REGION=eu-west-3
AWS_STS_REGIONAL_ENDPOINTS=regional

Check AWS DynamoDB capabilities

$ aws dynamodb create-table \
    --table-name Music \
    --attribute-definitions \
        AttributeName=Artist,AttributeType=S \
        AttributeName=SongTitle,AttributeType=S \
    --key-schema \
        AttributeName=Artist,KeyType=HASH \
        AttributeName=SongTitle,KeyType=RANGE \
    --provisioned-throughput \
        ReadCapacityUnits=5,WriteCapacityUnits=5 \
    --table-class STANDARD

->

Results

CLI output

{
    "TableDescription": {
        "AttributeDefinitions": [
            {
                "AttributeName": "Artist",
                "AttributeType": "S"
            },
            {
                "AttributeName": "SongTitle",
                "AttributeType": "S"
            }
        ],
        "TableName": "Music",
        "KeySchema": [
            {
                "AttributeName": "Artist",
                "KeyType": "HASH"
            },
            {
                "AttributeName": "SongTitle",
                "KeyType": "RANGE"
            }
        ],
        "TableStatus": "CREATING",
        "CreationDateTime": "2025-01-17T15:51:32.311000+00:00",
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 5,
            "WriteCapacityUnits": 5
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:eu-west-3:539242259304:table/Music",
        "TableId": "c02e0cab-557a-4001-b520-7c51b9fef4c6",
        "TableClassSummary": {
            "TableClass": "STANDARD"
        },
        "DeletionProtectionEnabled": false
    }
}

-> ✅ It works, so that means the issue is indeed in Zot code 🤔

❓ What can I do to help?

rchincha · 2025-01-17T22:44:54Z

rchincha
Jan 17, 2025
Maintainer

@eusebiu-constantin-petu-dbk can you pls take a look?

1 reply

xakraz Jan 29, 2025

For the record, here is an opened PR related to this issue: #2888

tamilhce · 2025-01-23T16:48:34Z

tamilhce
Jan 23, 2025

Thanks @xakraz for sharing the configurations, I will take a look.

1 reply

tamilhce Jan 28, 2025

First, I will attempt to reproduce this issue on my local setup. Since I worked on #2650, I should be able to work on the fix.

tamilhce · 2025-01-28T14:14:12Z

tamilhce
Jan 28, 2025

@xakraz
I reviewed your config and the error message


{
  "level": "error",
  "error": "operation error DynamoDB: CreateTable, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 404, RequestID: ***, api error UnknownError: UnknownError",
  "tableName": "MyZotBlobTable",
  "goroutine": 1,
  "caller": "zotregistry.dev/zot/pkg/storage/cache/dynamodb.go:98",
  "time": "2024-12-26T09:05:15.053656375Z",
  "message": "failed to create table for cache"
}

I suspect this issue is due to the account endpoint specified in your configuration. Instead, try using the region-specific endpoint for dynamob eg: endpoint: https://dynamodb.eu-west-3.amazonaws.com in the cache driver.

FYI:
We are currently using aws-sdk-go-v2 version v1.33.0. The account-endpoints feature is supported starting from version v1.35.0. If switching to the region-specific endpoint works, I can work on upgrading the Go SDK to v1.35.0. Since I don’t currently have a DynamoDB setup, please test this and let me know.
https://docs.aws.amazon.com/sdkref/latest/guide/feature-account-endpoints.html

5 replies

xakraz Jan 28, 2025

I will test it tomorrow.
Thanks for investigating 🙏

xakraz Jan 29, 2025

Hi @tamilhce ,

I just tried with the same version of Zot (v2.1.1) and with the suggested endpoint for DynamoDB but unfortunately, I got the same error 😕

eusebiu-constantin-petu-dbk Jan 29, 2025
Collaborator

Can you also try with my patch?#2888

xakraz Jan 29, 2025

Yes work in progress 😇

xakraz Jan 29, 2025

I have replied on the PR: #2888 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS EKS IRSA for Zot #2853

{{title}}

Replies: 9 comments 9 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

AWS EKS IRSA for Zot #2853

Replies: 9 comments · 9 replies

huynguyen-anduin Dec 27, 2024 Author

rchincha Dec 31, 2024 Maintainer

huynguyen-anduin Jan 3, 2025 Author

rchincha Jan 7, 2025 Maintainer

rchincha Jan 17, 2025 Maintainer

Zot deployment

AWS Policy

Zot config

Note

Results

Notes

Pod specs

Get a shell in the pod

Check the IRSA integration

Check AWS DynamoDB capabilities

Results

rchincha Jan 17, 2025 Maintainer

eusebiu-constantin-petu-dbk Jan 29, 2025 Collaborator

Replies: 9 comments 9 replies

huynguyen-anduin
Dec 27, 2024
Author

rchincha
Dec 31, 2024
Maintainer

huynguyen-anduin
Jan 3, 2025
Author

rchincha
Jan 7, 2025
Maintainer

rchincha
Jan 17, 2025
Maintainer

rchincha
Jan 17, 2025
Maintainer

eusebiu-constantin-petu-dbk Jan 29, 2025
Collaborator