Improve resiliency of performance calculation requests

[tsunyoku]

Server has had some instability recently and made me realise that this isn't even really trying to be resilient.. I've added tenacity retry logic onto the performance usecases and opted to use raise_for_status instead of returning zeros on a failure - this seemed like odd and unintuitive behavior to me, even with the logging

[cmyui]

hmm, i think this (& the previous code) lacks consideration that the osu client retries score submission, which is a part of the resiliency system.. i think setting pp=0 without a strategy to calculate these is worse than hard failing tbh

[cmyui]

i think i agree with moving to response.raise_for_status() to hard fail requests rather than setting 0

[tsunyoku]

hmm, i think this (& the previous code) lacks consideration that the osu client retries score submission, which is a part of the resiliency system.. i think setting pp=0 without a strategy to calculate these is worse than hard failing tbh

my concern with this is that it has the implication that if performance-service never comes back up that the score will never submit. i think it would be better to have a score submitted with zero pp than no score

[cmyui]

hmm, i think this (& the previous code) lacks consideration that the osu client retries score submission, which is a part of the resiliency system.. i think setting pp=0 without a strategy to calculate these is worse than hard failing tbh

my concern with this is that it has the implication that if performance-service never comes back up that the score will never submit. i think it would be better to have a score submitted with zero pp than no score

Yeah, ultimately a deadletter queue for performance calculations would be a nice solution here. Does the processor on performance service already support us building such a queue?

[tsunyoku]

i would hope that logging the score id on failure would make it easier to create a strategy to recalculate scores in this case

but having an asynchronous score processing model like i've discussed before would solve all of these concerns i think, since we take separate ownership of calculating performance outside of the hot path (the actual score submission) so we can retry as much as we like (without relying on the client to retry) solong as we have the initial score data. lazer does this and i think it works well

[tsunyoku]

Yeah, ultimately a deadletter queue for performance calculations would be a nice solution here. Does the processor on performance service already support us building such a queue?

not really, performance-service is intentionally detached from a concept of "this score has xpp" and rather taking in statistics and spitting out a value. i don't really know if i want to change that relationship either

[cmyui]

Yea I think I agree that keeping it on score-service would be nice.

having an asynchronous score processing model like i've discussed before would solve all of these concerns i think, since we take separate ownership of calculating performance outside of the hot path (the actual score submission) so we can retry as much as we like

Yea imo this would likely look like a main perf calculation queue + a deadletter queue for failed requests (so that the main queue is not blocked by a single broken score); both on score service that depend on perf-service and the db

[tsunyoku]

yep i agree, i think this is a "long term consideration" though