Implement Bulk Metadata download

[Mithnar]

Playnite has the ability to download Metadata for all games in the library.

The Plugin should support this.

The issues to solve:

• The VNDB API has rate limits for the usage, those need to be handled

• To avoid running into the rate limits, there'll need to be a strategy to avoid doing unnecessary calls against the API, or the bulk import will take too long for bigger libraries)

[Snap47]

Hey there, I just came across a browser extension for VNDB which also had to deal with the rate limits.
https://github.com/marcussacana/VNDB-Extender-2020
Perhaps you could glimpse in there for a solution?

[darklinkpower]

@Snap47 it seems they parse the page instead of using the API like PlayniteVndb does, so the extension would need a heavy rework.
https://github.com/marcussacana/VNDB-Extender-2020/blob/master/scripts/vnAssetsController.js#L64

Issues a download request for the page for the vn with the given ID. Loading the page is better than using the API, because the data usage for just the HTML is minimal and there is no request limit for this method."

[EazyCheeze1978]

Hello! It's been a while since the last update - any progress on this issue? Am experiencing the efficiency loss associated with having to edit metadata for each of my 50+ visual novels individually; have put that on hold for the time being to keep my sanity :)

Looking forward to some positive progress! Take care :)

[darklinkpower]

A possible rate limiter for this when someone looks into this https://github.com/David-Desmaisons/RateLimiter

[Nikey646]

VNDB has great rate limits. You would need to issue effectively 3 commands a second before you even hit the rate limit, or "1 second of SQL time per minute" which is incredibly variable, because it'll change based off of server load.

You can definitely merge some requests, EG: When requesting data by id you can grab up to 25 visual novels in 1 request (Might hit that SQL time limit that way, so probably go for 3-5 per request instead). Definitely for the release data, grabbing in bulk would be the way to go.

[Mithnar]

Hitting 200 Requests in 10 minutes is incredibly easy, when you have a big library in playnite.

Requesting multiple games is not easy in the playnite context:

• I do get a single game object, and have to return the metadata for that game.
• I have no control over the order of those games
• I don't know if the game will provide the id, or if that game even is on VNDB

So a import with a game library of 400 games will take at least 20 Minutes, when none of those games will have a match on VNDB, longer if there are matches (and I have to grab the Publisher/Developer data aswell)

A Library of 1200 Games would take at least an hour.

I'm currently building a solution by caching relevant data, but that will still take some time.