x/pkgsite/devtools/cmd/seeddb: -keep_going through version list errors

[jeanbza]

What is the URL of the page with the issue?

https://artifacts.netflix.net/api/go/goproxy (private jfrog artifactory go module proxy)

What is your user agent?

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36

Screenshot

No response

What did you do?

GO_MODULE_PROXY_URL=https://artifacts.netflix.net/api/go/goproxy go run devtools/cmd/seeddb/main.go -keep_going=true

What did you see happen?

Blows up at https://cs.opensource.google/go/x/pkgsite/+/master:devtools/cmd/seeddb/main.go;l=95

What did you expect to see?

keep_going should keep going when versions results in an error.

---

Explanation: jFrog Artifactory version listing for version-less repos

For repos with no tagged versions, jFrog Artifactory's go module proxy returns 404 for version list. ex https://github.netflix.net/example/foo has no releases, https://artifacts.netflix.net/api/go/goproxy/github.netflix.net/example/foo/@v/list returns {"errors":[{"status":404,"message":"Could not find resource"}]}.

This is in contrast to proxy.golang.org, which would return a blank 200. See https://github.com/jeanbza/sim-hashing which has no versions, https://proxy.golang.org/github.com/jeanbza/sim-hashing/@v/list returns 200:

$ curl https://proxy.golang.org/github.com/jeanbza/sim-hashing/@v/list -i
HTTP/2 200

This makes using seeddb a pain: you have to first prune your seed.txt of version-less repos before seeding. We could just amend -keep_going to keep going through version errors instead.

[jeanbza]

Fix: https://go-review.googlesource.com/c/pkgsite/+/640615

[gabyhelp]

Related Issues

• x/pkgsite: versions list with spaces #71141

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dmitshur]

Note that the module proxy protocol specification (https://go.dev/ref/mod#goproxy-protocol) is clear about expected HTTP status codes:

Successful HTTP responses must have the status code 200 (OK). Redirects (3xx) are followed. Responses with status codes 4xx and 5xx are treated as errors. The error codes 404 (Not Found) and 410 (Gone) indicate that the requested module or version is not available on the proxy, but it may be found elsewhere.

Returning 404 is used to indicate a module doesn't exist. When a module exists but has an empty list of known versions (perhaps because they're all pseudo-versions), a successful response, quoting, "must have the status code 200 (OK)".

Separately from x/pkgsite, is it possible to address the bug in the module proxy protocol implementation you're using?

[jba]

@dmitshur I agree that jFrog should change their status code. But I bet they won't, because too many of their customers depend on it.

Meanwhile, this change is to a program used only for pkgsite testing, so I think it is harmless.