[Bug]: zot fails to restart in case of scylladb (dynamod compatible) usage for cache

[shcherbak]

zot version

{"level":"info","distribution-spec":"1.1.0","commit":"v2.1.2-rc4-0-g75c41ad","binary-type":"-imagetrust-lint-metrics-mgmt-profile-scrub-search-sync-ui-userprefs","go version":"go1.23.4","time":"2025-01-24T18:31:23Z","message":"version"}

Describe the bug

We are trying to implement clustered on-premise Zot setup with Scylladb Alternator as a replacement for Dynamodb and Openstack Swift backend with s3 compatibility

two zot instances can not share one scylla (dynamodb) because tables are created on startup

zot config:

cat /etc/default/zot 
AWS_ACCESS_KEY_ID="None"
AWS_SECRET_ACCESS_KEY="None"

cat /etc/zot/config.json
...
        "cacheDriver": {
            "name": "dynamodb",
            "endpoint": "http://10.12.18.52:8000",
            "region": "None",
            "cacheTablename": "ZotBlobTable",
            "repoMetaTablename": "ZotRepoMetadataTable",
            "imageMetaTablename": "ZotImageMetaTable",
            "repoBlobsInfoTablename": "ZotRepoBlobsInfoTable",
            "userDataTablename": "ZotUserDataTable",
            "versionTablename": "ZotVersion",
            "apiKeyTablename": "ZotApikeyTable"
        },
...

Zot is starting and creating a tables, next writing to them, all ok
See tables are created:

cqlsh
Connected to  at 127.0.0.1:9042
[cqlsh 6.0.23.dev9+gb09bc79 | Scylla 6.2.2-0.20241128.c6ef055e9c3b | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> desc keyspaces;                       

"alternator_ZotApikeyTable"         "alternator_ZotVersion"        system_traces
"alternator_ZotBlobTable"           system                       
"alternator_ZotImageMetaTable"      system_auth                  
"alternator_ZotRepoBlobsInfoTable"  system_distributed           
"alternator_ZotRepoMetadataTable"   system_distributed_everywhere
"alternator_ZotUserDataTable"       system_schema                

cqlsh>

but process fails to restart after first db initialization:

{"level":"info","cpus":16,"max. open files":500000,"listen backlog":"8192","max. inotify watches":"8192","goroutine":1,"caller":"zotregistry.dev/zot/pkg/api/controller.go:130","time":"2025-01-23T22:50:09.570588995Z","message":"runtime params"}
{"level":"error","error":"operation error DynamoDB: CreateTable, 
https response error StatusCode: 400, RequestID: , 
ResourceInUseException: Table ZotVersion already exists","goroutine":1,"caller":"zotregistry.dev/zot/pkg/cli/server/root.go:76",
"time":"2025-01-23T22:50:09.572067041Z","message":"failed to init controller"}

To reproduce

1.     "cacheDriver": {
        "name": "dynamodb",
        "endpoint": "http://10.12.18.52:8000",
        "region": "None",
        "cacheTablename": "ZotBlobTable",
        "repoMetaTablename": "ZotRepoMetadataTable",
        "imageMetaTablename": "ZotImageMetaTable",
        "repoBlobsInfoTablename": "ZotRepoBlobsInfoTable",
        "userDataTablename": "ZotUserDataTable",
        "versionTablename": "ZotVersion",
        "apiKeyTablename": "ZotApikeyTable"
    },
   

AWS_ACCESS_KEY_ID="None"
AWS_SECRET_ACCESS_KEY="None"
config for scylladb

3.{"level":"info","cpus":16,"max. open files":500000,"listen backlog":"8192","max. inotify watches":"8192","goroutine":1,"caller":"zotregistry.dev/zot/pkg/api/controller.go:130","time":"2025-01-23T22:50:09.570588995Z","message":"runtime params"}
{"level":"error","error":"operation error DynamoDB: CreateTable,
https response error StatusCode: 400, RequestID: ,
ResourceInUseException: Table ZotVersion already exists","goroutine":1,"caller":"zotregistry.dev/zot/pkg/cli/server/root.go:76",
"time":"2025-01-23T22:50:09.572067041Z","message":"failed to init controller"}

Expected behavior

two zot instances are using the same dynamodb or one zot instance is able to restart without manaly deleting tables from scylladb

Screenshots

No response

Additional context

suppose checking error message for scylla should be with another string

zot/pkg/meta/dynamodb/dynamodb.go

Line 2136 in ececc9c

if err != nil && !strings.Contains(err.Error(), "Table already exists") {

[andaaron]

@shcherbak this should fix it: #2909
Can you please try this patch?

[shcherbak]

@andaaron on it

[shcherbak]

@andaaron it works as a clock!
excellent, thanks a lot for that fix!
hope to see the commit in next release soon

[rchincha]

@shcherbak v2.1.3-rc1 is now released with this fix.
Pls close this issue if resolved.

[shcherbak]

now zot has scylladb support

[rchincha]

@shcherbak pls give us your feedback once you have the setup up and running!

[shcherbak]

@rchincha We have setup of Zot registry with Openstack Swift backend, it works fine for a month. Now we have a clustered Zot with nginx load balancing and Scylladb as a cache. The registry is running with docker compatibility for migration reason. Previously we used Distrubution Registry (vanilla one) with the same backend storage for a 5 years. Zot is great of its features, that vanilla registry does not have