Unable to download templates reliably

[gaby]

Nuclei version: v2.5.8

Current Behavior:

As part of our CI/CD process we add nuclei and nuclei templates as part of a toolkit Docker image. Consistently when trying to run -update-templates I think cloudflare or something is blocking us in https://version-check.nuclei.sh/versions?v=2.5.8 which makes nuclei to fail and kill our whole CI process.

I understand having nuclei.sh protected by cloudflare, but wouldn't it make more sense to store the results file we get back from that URL on each release of https://github.com/projectdiscovery/nuclei-templates on Github?

All the query is returning is this:

{"nuclei":"2.5.8","templates":"8.8.3","ignore-hash":"7e404e94b241604ce419690b689c5b6d"}

It would make more sense, that on each tagged release of nuclei-templates that file is added on the root of the git repo, and nuclei to query that instead of trying to reach nuclei.sh.

From what I remember back in 2020 nuclei used to get templates from Github.

Expected Behavior:

Being able to reliably download templates

Steps To Reproduce:

nuclei -update-templates -vv

Related issues: #1145

[ehsandeep]

@gaby have a look at this issue #895 for the underying problem when we were using GitHub endpoint directly.

We can explore further to know what causing the block and what change we can (if required) to fix it.

[gaby]

@ehsandeep I'm basically having the same problem but with "nuclei.sh", specially if the host running nuclei is behind a VPN. No matter the provider we get throttle by Cloudflare or it triggers a captcha via browser, which makes nuclei to fail.

We never had this issue with Github perse.

[ehsandeep]

@gaby

specially if the host running nuclei is behind a VPN

Could you give the details of the setup / environment to replicate this bhevhaior?