fix: per year bug fixes

[jeremylong]

• ensure the correct lastUpdated is captured per year
• cleanup code duplication
• ensure CVE-1988 - CVE-2001 are stored in year 2002

[jeremylong]

Existing cache from yesterday - ran the update and it took 2632 seconds (~43 minutes).

[jeremylong]

Fixed one minor bug and re-ran the update. 1943 seconds (32 minutes) later and the cache is updated (see the log). It is taking 149 requests to the NVD to update the cache when in reality if I updated the cache an hour ago - it should be a single API call to get all the data.

[EugenMayer]

nice, just docs and this is a good addition for sure

[EugenMayer]

We should either name the variables in a way it is obvious or document why both exist, or the next time, it will be not implemented the right way again :)

[EugenMayer]

We should add docs why this is needed and also reference the NVD docs you found on that. In the end, we fetch 1988-2002 as a special case, and store them in the year 2002 alltogether

[EugenMayer]

Fixed one minor bug and re-ran the update. 1943 seconds (32 minutes) later and the cache is updated (see the log). It is taking 149 requests to the NVD to update the cache when in reality if I updated the cache an hour ago - it should be a single API call to get all the data.

Are you able to compare this with the old implementation? (not memory, rather a real test). Asking because i cannot do it myself, it does never finish, neither on my k8s cluster nor locally (either ooms or fails due to API errors). It would be nice to
have a comparison.

Current:
23 yeast * 4 requests (120 days slices) = 132 requests.

We could use an entire separated fetch-logic for the modified file, which might be an easy fix and easy to implement. Idea

1. Update years

• if the cache file for a year already
  • exists and size > 0: update it only if the c-time is older then 8 days. Update is year based, 4 requests per year
  • not exist:Fetch for all years, year by year, 4 requests per year (same as above)

2. Update modified file

• if the cache
  • exists: get the c-time and and use it as $start + pre-load the existing file into the result
  • not exist: $start = today-8, nothing to preload
• Make a single requests, fetching all changedSinceStart=$start and changedSinceTo=today, no year limitations - fetch them all at once.
  • merge the result with the preloaded cache
• write file

So we split both steps entirely, they fetching all years first or last will not matter, "year fetching" and "modify" fetching are separated entirely. With the current code structure, this can already be done fairly DRY.

So what we do is, we split the efforts and using different strategies. This makes sense since 1. will load a huge amount of items (a magnitude more then 2. and thus we should rather focus on relyability, memory usage and consistency.

2. loads and expected low amount of items, thus preferring speed to keep daily updates "fast" might be viable - thus using one request to load all items at once. I'am expected NVD to close this kind of request down with limits sooner or later, but as long as it lasts, it saves time.

// rant on
Main reason for the entire slowdown is the speed the NVD api answers with. This is hands down the slowest API i have ever used (beside some hacked PoCs). Considering they update the data once a night, thus can hard-cache the persistence layer to read-cluster, easily scale the application since there is zero relation between requests. I have to say how it is, it is silly to offer this as an API for anybody to use, let allow offering it with this global usage scope.
// rant off

Still, if it is more reliable, i do not care it takes 30 mins ones a night.