Local Dev Environment not scanning containers

[chrisbsmith]

I've set up a local dev cluster following the make local-dev-up-with-quay commands in the documentation. When I push an image to Quay, the image is never scanned. The status sits in Queued forever.

I've deployed this from both the main branch as well as the PR #1193 branch. Both result in the same thing, though the PR branch does resolve the issue with the notifier crashing on startup.

I've noticed some 404 and 304 errors in the traefik image logs:

172.18.0.9 - - [08/Mar/2021:21:51:27 +0000] "GET /indexer/api/v1/index_state HTTP/1.1" 400 72 "-" "-" 4903 "indexer@docker" "http://172.18.0.9:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:51:28 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4904 "matcher@docker" "http://172.18.0.10:6000" 0ms
69.143.63.250 - - [08/Mar/2021:21:51:31 +0000] "GET /api/overview HTTP/1.1" 200 435 "-" "-" 4905 "api@internal" "-" 0ms
69.143.63.250 - - [08/Mar/2021:21:51:36 +0000] "GET /api/overview HTTP/1.1" 200 435 "-" "-" 4906 "api@internal" "-" 0ms
69.143.63.250 - - [08/Mar/2021:21:51:41 +0000] "GET /api/overview HTTP/1.1" 200 435 "-" "-" 4907 "api@internal" "-" 0ms
172.18.0.11 - - [08/Mar/2021:21:51:43 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4908 "matcher@docker" "http://172.18.0.10:6000" 0ms
69.143.63.250 - - [08/Mar/2021:21:51:46 +0000] "GET /api/overview HTTP/1.1" 200 435 "-" "-" 4909 "api@internal" "-" 0ms
172.18.0.9 - - [08/Mar/2021:21:51:57 +0000] "GET /indexer/api/v1/index_state HTTP/1.1" 400 72 "-" "-" 4910 "indexer@docker" "http://172.18.0.9:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:51:58 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4911 "matcher@docker" "http://172.18.0.10:6000" 1ms
172.18.0.11 - - [08/Mar/2021:21:52:13 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4912 "matcher@docker" "http://172.18.0.10:6000" 0ms
172.18.0.9 - - [08/Mar/2021:21:52:27 +0000] "GET /indexer/api/v1/index_state HTTP/1.1" 400 72 "-" "-" 4913 "indexer@docker" "http://172.18.0.9:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:52:28 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4914 "matcher@docker" "http://172.18.0.10:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:52:43 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4915 "matcher@docker" "http://172.18.0.10:6000" 1ms
172.18.0.9 - - [08/Mar/2021:21:52:57 +0000] "GET /indexer/api/v1/index_state HTTP/1.1" 400 72 "-" "-" 4916 "indexer@docker" "http://172.18.0.9:6000" 2ms
172.18.0.11 - - [08/Mar/2021:21:52:58 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4917 "matcher@docker" "http://172.18.0.10:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:53:13 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4918 "matcher@docker" "http://172.18.0.10:6000" 0ms
172.18.0.9 - - [08/Mar/2021:21:53:27 +0000] "GET /indexer/api/v1/index_state HTTP/1.1" 400 72 "-" "-" 4919 "indexer@docker" "http://172.18.0.9:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:53:28 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4920 "matcher@docker" "http://172.18.0.10:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:53:43 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4921 "matcher@docker" "http://172.18.0.10:6000" 0ms
172.18.0.9 - - [08/Mar/2021:21:53:57 +0000] "GET /indexer/api/v1/index_state HTTP/1.1" 400 72 "-" "-" 4922 "indexer@docker" "http://172.18.0.9:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:53:58 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4923 "matcher@docker" "http://172.18.0.10:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:54:13 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4924 "matcher@docker" "http://172.18.0.10:6000" 1ms
172.18.0.9 - - [08/Mar/2021:21:54:27 +0000] "GET /indexer/api/v1/index_state HTTP/1.1" 400 72 "-" "-" 4925 "indexer@docker" "http://172.18.0.9:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:54:28 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4926 "matcher@docker" "http://172.18.0.10:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:54:43 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4927 "matcher@docker" "http://172.18.0.10:6000" 0ms
172.18.0.9 - - [08/Mar/2021:21:54:57 +0000] "GET /indexer/api/v1/index_state HTTP/1.1" 400 72 "-" "-" 4928 "indexer@docker" "http://172.18.0.9:6000" 0ms
172.18.0.11 - - [08/Mar/2021:21:54:58 +0000] "GET /matcher/api/v1/internal/update_operation/ HTTP/1.1" 304 0 "-" "-" 4929 "matcher@docker" "http://172.18.0.10:6000" 0ms

I'm very new to Quay and Clair and an just looking to compare different image scanners and was hoping to get a chance to use this one. Any tips or pointers to where to start debugging would be greatly appreciated.

I've tried this on an Amazon Linux2 and Ubuntu EC2, both with the same results.

Current Environment

• AWS EC2: Ubuntu: 20.04.2 LTS (Focal Fossa)
• go: go1.16 linux/amd64
• docker: 19.03.8
• docker-compose version 1.25.0

[hdonnay]

The 304 returns are expected, and not errors.

I don't know how those 400 statuses are returned, we'd need to see application logs, not just traefik logs.

I will note that quay is a bit of a beast in terms of resources, so make sure there's plenty of RAM and the workers aren't crashlooping.

[chrisbsmith]

So I pulled the latest code from the PR and ran make local-dev-restart-quay to restart Quay. The errors in Quay are gone, but I'm not getting this error on the indexer:

6:10PM DBG updater tick component=rhel/repo2cpe/updater/LocalUpdaterJob.Start version=1.0
6:10PM DBG attempting fetch of repo2cpe mapping file component=rhel/repo2cpe/updater/LocalUpdaterJob.do url=https://www.redhat.com/security/data/metrics/repository-to-cpe.json version=1.0
6:10PM ERR received error updating mapping file error="Get \"https://www.redhat.com/security/data/metrics/repository-to-cpe.json\": dial tcp: lookup www.redhat.com on 127.0.0.11:53: read udp 127.0.0.1:41011->127.0.0.11:53: read: connection refused" component=rhel/repo2cpe/updater/LocalUpdaterJob.Start version=1.0

I've opened up the security groups to allow all traffic in bound/out bound of the EC2, but I don't think that's the actual issue. Images scans are still queued.

Any other thoughts?

[ldelossa]

Chris, that error above looks like your host cannot reach a DNS server to resolve www.redhat.com, noticing the failure to dial post 53 (DNS).

However, that should not stop Quay from scanning. Let me check on my env.

[chrisbsmith]

That indeed is the case from the indexer image. Sorry to just throw these errors out. I really don't know the Quay + Clair stack well enough to debug much more. This may be a docker problem on my ubuntu machine. I'll look into possible issues at that level.

edited:

I was able to fix the DNS issue in Docker. Running containers can now resolve DNS correctly.

[chrisbsmith]

@ldelossa That may have done it. I am now seeing scan results in Quay. Thanks for working through this with me!

[ldelossa]

@chrisbsmith glad to have helped. thanks for helping us track down that bug!