HTTP 503 when running --cache #283
Comments
|
Those are in fact issues with the NVD API, mostly on there side, but also due to the parameters used with the current fetch. The NVD API is fairly fragile (for more then some days), but it becomes even more faulty when the requested fetch uses a huge page size, big range and so on. I had the very same issues, I could never complete a full fetch since weeks using the current client. That is why implemented #279 Still, e.g. today, you will need to run the new implementation for several hours. It will slowly crawl up the years on and on until it looks like this: So no matter what you do, the API is 503ing all over the place. Key is to take smaller results and save them, and then retry |
|
@EugenMayer thanks for your reply! I'll try to use your image now and see if i can build a successful cache now. |
Hello,
since a few months, i can't run a full cache update with vulnz. We usually run this in a docker container but for sake of simplicity i tried to run it on my machine directly using the following parameters:
java -jar .\vulnz-7.2.2.jar cve --cache --directory .\data(yes, without an API-Key)
This results in this stacktrace:
` java -jar .\vulnz-7.2.2.jar cve --cache --directory .\data
_/
_/ _/ / / / /// ////
_/ _/ _/ _/ _/ _/ _/ _/
_/ _/ _/ _/ _/ _/ _/ _/
/ /// _/ / / ////
Version: 7.2.2
Open Vulnerability Project
? Sponsor: https://github.com/sponsors/jeremylong
NVD_API_KEY not found. Supply an API key for more generous rate limits
NVD API request failures are occurring; retrying request for the 5th time
NVD API request failures are occurring; retrying request for the 6th time
NVD API request failures are occurring; retrying request for the 7th time
NVD API request failures are occurring; retrying request for the 8th time
NVD API request failures are occurring; retrying request for the 9th time
NVD API request failures are occurring; retrying request for the 10th time
NVD API request failures are occurring; retrying request for the 11th time
Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=18000 : 3rd time
Recoverable I/O exception (org.apache.hc.core5.http.ConnectionClosedException) caught when processing request to {s}->https://services.nvd.nist.gov:443
Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5229624b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@17a47605[Wrapped task = org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$ScheduledRequestExecution@3e74cae6]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@797cf65c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5229624b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@17a47605[Wrapped task = org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$ScheduledRequestExecution@3e74cae6]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@797cf65c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2081)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:705)
at java.base/java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:754)
at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.executeScheduled(InternalAbstractHttpAsyncClient.java:387)
at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$1.scheduleExecution(InternalAbstractHttpAsyncClient.java:137)
at org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec$1.failed(AsyncHttpRequestRetryExec.java:187)
at org.apache.hc.client5.http.impl.async.AsyncProtocolExec$1.failed(AsyncProtocolExec.java:295)
at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.failed(HttpAsyncMainClientExec.java:131)
at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.failed(ClientHttp1StreamHandler.java:285)
at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.disconnected(ClientHttp1StreamDuplexer.java:220)
at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onDisconnect(AbstractHttp1StreamDuplexer.java:409)
at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.disconnected(AbstractHttp1IOEventHandler.java:95)
at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.disconnected(ClientHttp1IOEventHandler.java:41)
at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.disconnected(SSLIOSession.java:247)
at org.apache.hc.core5.reactor.InternalDataChannel.disconnected(InternalDataChannel.java:204)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.processClosedSessions(SingleCoreIOReactor.java:231)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.doTerminate(SingleCoreIOReactor.java:106)
at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:93)
at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
at java.base/java.lang.Thread.run(Thread.java:1583)
Unable to complete NVD cache update due to error: NVD Returned Status Code: 503
io.github.jeremylong.vulnz.cli.cache.CacheException: Unable to complete NVD cache update due to error: NVD Returned Status Code: 503
at io.github.jeremylong.vulnz.cli.commands.CveCommand.downloadAllUpdates(CveCommand.java:385)
at io.github.jeremylong.vulnz.cli.commands.CveCommand.processRequest(CveCommand.java:274)
at io.github.jeremylong.vulnz.cli.commands.CveCommand.timedCall(CveCommand.java:246)
at io.github.jeremylong.vulnz.cli.commands.TimedCommand.call(TimedCommand.java:36)
at io.github.jeremylong.vulnz.cli.commands.TimedCommand.call(TimedCommand.java:25)
at picocli.CommandLine.executeUserObject(CommandLine.java:2045)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
at picocli.CommandLine.execute(CommandLine.java:2174)
at io.github.jeremylong.vulnz.cli.Application.run(Application.java:85)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:765)
at org.springframework.boot.SpringApplication.lambda$callRunners$2(SpringApplication.java:749)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1300)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1289)
at io.github.jeremylong.vulnz.cli.Application.main(Application.java:70)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Completed in 299 seconds`
If i run it with an API-Key, it's just stuck for hours on the line after sponsor and does not anything.
I've read in a previous issue, that there were problems at NVD's site in November but for me it's hard to believe that these issues still persist in February, especially because the "Old" Nist-data-mirror still allows me to pull CVE's daily.
Can you please elaborate.
The text was updated successfully, but these errors were encountered: