fix: improve memory usage

[jeremylong]

Improves memory usage when updating the NVD data cache.

@EugenMayer can you take a look at this one? I haven't run it through a profiler yet, but it should be much better on updates unless the NVD does a mass update again.

[EugenMayer]

tried 7.2.1 and it hovers around 3GB with a max records per page of 200 - so IMHO this is still way more then it used to be, i was running with 2000 beforehand.

I mean it finishes - most probably. It takes more then 60 minutes, rather close to 2 hours to complete - i assume this is rather due to NVDs rate limits / response timings.

Did not want to necro this issue, just to give feedback.

[EugenMayer]

Well, crashed after 15 minutes with

Requested At: 08:00:46; URI: /rest/json/cves/2.0?resultsPerPage=200&startIndex=164200

Content-Type Received: application/json
2025-01-27T08:00:46.826401378Z
Exception in thread "httpclient-dispatch-2" java.lang.OutOfMemoryError: Java heap space
	at org.apache.hc.core5.util.ByteArrayBuffer.expand(ByteArrayBuffer.java:58)
	at org.apache.hc.core5.util.ByteArrayBuffer.append(ByteArrayBuffer.java:88)
	at org.apache.hc.client5.http.async.methods.SimpleAsyncEntityConsumer.data(SimpleAsyncEntityConsumer.java:62)
	at org.apache.hc.core5.http.nio.entity.AbstractBinDataConsumer.consume(AbstractBinDataConsumer.java:75)

i was running with crgroups on

          resources:
            limits:
              memory: 5Gi

Happened to notice that my helm chart was using the wrong image, i was running on 7.2.0 here. Accidently we might just crated a baseline then. Running with 7.2.1 and reporting back.

[EugenMayer]

So with the proper 7.2.1 release it looks similar

   8     1 mirror   S    3446m  44%   0   0% java -XX:InitialRAMPercentage=50.0 -jar /usr/local/bin/vulnz cve --delay=1000 --debug ---recordsPerPage=200 --cache --directory /usr/local/apache2/htdocs

so we have more then 3GB usage with 200 records per page. I upped the cgroups limit to 7GB, so let's see if this actually finishes at all.

[EugenMayer]

Well, crashed again:

Application run failed
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.lang.StringUTF16.newBytesFor(StringUTF16.java:53)
	at java.base/java.lang.String.<init>(String.java:568)
	at java.base/java.lang.String.<init>(String.java:1387)
	at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient._next(NvdCveClient.java:380)
	at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:357)
	at io.github.jeremylong.vulnz.cli.commands.CveCommand.downloadAllUpdates(CveCommand.java:376)
	at io.github.jeremylong.vulnz.cli.commands.CveCommand.processRequest(CveCommand.java:274)
	at io.github.jeremylong.vulnz.cli.commands.CveCommand.timedCall(CveCommand.java:246)
	at io.github.jeremylong.vulnz.cli.commands.TimedCommand.call(TimedCommand.java:36)
	at io.github.jeremylong.vulnz.cli.commands.TimedCommand.call(TimedCommand.java:25)