Setting nvdApiResultsPerPage to something other than 2000 results in downloading more than 100%

[deennyy]

Describe the bug
Hello, when I set nvdApiResultsPerPage to something other than 2000, say 1000, I get messages reporting that I have downloaded more than 100%. I need to set it to ~1000, because when it's at the default 2000 my requests get a "connection reset" sometimes.

Version of dependency-check used
The problem occurs using version 10.0.3 of the maven plugin.

Log file
gist

To Reproduce
Steps to reproduce the behavior:

1. Set nvdApiResultsPerPage to something other than 2000
2. Run mvn -DnvdApiResultsPerPage=1000 -DnvdApiKey=... verify
3. Wait and see logs about more than 100% being downloaded

Expected behavior
It would stop downloading at 100% and continue with the checking.

[deennyy]

Hey, I appreciate the quick reaction, one more thing, as far as I see this is just a visual bug (in the line being printed to the logs), the download has been stuck for a good few hours at 198% at this point, would you have an idea as to why?

Here's some logs:
gist

[jeremylong]

Did it ever complete?

[deennyy]

Well it was running for 3-4 hours and I stopped it, I’ve started a new one today that also got stuck at 98-99%, I haven’t stopped it yet

[jeremylong]

If it doesn't complete - can you make sure to run it with debug (I know -X spews a ton of noise). But it might help narrow down what the problem is.

[deennyy]

The gist I’ve sent 20 hours ago is with -X

[deennyy]

Hey, I attached JMC to the stuck dependency-check process, here are the only 2 threads I could find that had calls to org.owasp.* or io.github.* in their call stack

main:
java.lang.Thread.sleep line: not available [native method]
io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.getCompletedFuture line: 481
io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient._next line: 345
io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next line: 331
org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi line: 352
org.owasp.dependencycheck.data.update.NvdApiDataSource.update line: 116
org.owasp.dependencycheck.Engine.doUpdates line: 906
org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase line: 711
org.owasp.dependencycheck.Engine.analyzeDependencies line: 637
org.owasp.dependencycheck.App.runScan line: 262
org.owasp.dependencycheck.App.run line: 194
org.owasp.dependencycheck.App.main line: 89

pool-4-thread-1:
java.lang.Object.wait line: not available [native method]
java.lang.Object.wait line: 338
org.apache.hc.core5.concurrent.BasicFuture.get line: 83
io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient.delayedExecute line: 202
io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient.lambda$execute$0 line: 173
io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient$$Lambda$217/0x00007fb7b42af528.call line: not available
java.util.concurrent.FutureTask.run line: 264
java.util.concurrent.ThreadPoolExecutor.runWorker line: 1136
java.util.concurrent.ThreadPoolExecutor$Worker.run line: 635
java.lang.Thread.run line: 840

[deennyy]

Also seems to ignore nvdMaxRetryCount, I tried setting it to 100 to see if maybe it would go through after many attempts, but after "DEBUG - Retrying request 8 time" nothing is ever printed to the log until shutdown.

[jeremylong]

@deennyy I'm reopening this - even though this should really be a different issue at this point. I've add some additional debug logic here: jeremylong/open-vulnerability-cli#200

The only thing I can think of is that somehow a massive delay is either requested by the NVD or it is being incorrectly calculated. Is there any chance you increased the delay?

[deennyy]

Hi, yes I’ve tried to increase the delay to no avail, one thing I noticed - it always gets stuck when startIndex gets to 246000, and thats on every run. Maybe its an issue with NVD?

[jeremylong]

I know myself - its likely a dumb mistake in my code ;) (I hope not... but I'm trying to reproduce this to validate)

[jeremylong]

Are you using an API Key? I just retested, successfully, that the maven plugin worked with <nvdApiResultsPerPage>1000</nvdApiResultsPerPage> - my test used an API key.

[deennyy]

Yea, im using an api key, It might be some weird network issue honestly, im going to try on a different network later today and report results

[deennyy]

Hey, I just tried on another PC and network and everything worked fine, I guess it's some weird network issue. Closing.