Invalid GVL Backoff

[bretg]

We’re getting hit with a different invalid version of the GVL file, which causes a burn of extra network fetches and error processing,.

I'd like to propose implementing a backoff strategy for invalid GVL versions. Minimally, each server host can try once per hour per version.

See below for the 'exponential backoff' algorithm actually implemented in PBS-Java.

[Net-burst]

@And1sS please chime in on this one as you had some ideas about the implementation.
Plus I propose one change right off the bat: instead of using static wait time, do an exponential backoff with a configurable min and max (or only max) time limit. This should lessen the impact of sporadic network hiccups while still achieving our goal of not doing an awful lot of lookups.

[SerhiiNahornyi]

The Java part will be resolved with
prebid/prebid-server-java#2919
@And1sS, Please provide this issue with properties to control the behavior of the feature

[bretg]

Done in PBS-Java 2.9

[And1sS]

So, on Java side we implemented per-GVL version exponential-backoff/fixed-interval delays (PBS host can configure this).

On a high level, this means that for each version of GVL we save the time server tried to download json last time, and if someone requests same version of GVL json and enough time since last download attempt (delay) has passed (or we haven't fetched it yet) - download json.

On a lower level It looks like this:

1. Someone requests json for some version.

2. If we have this file - just return.

3. If not and we haven't tried to fetch it yet - start downloading.

4. If we already tried to fetch it and not enough time have passed since last try - return an error.

5. If enough time have passed since last attempt - start downloading, compute next delay using specified algorithm, store it along with attempt time (in case downloading process fails).

6. If we successfully downloaded file - save it and remove timestamp/delay records for saving memory.

Exponential backoff:
formula: newDelay = min(delay * factor * (1 + random[from 0 to jitter] ), maxDelay),
where:

• factor - growth multiplicator, should be > 1
• jitter - "randomness" factor
• maxDelay - cap value for delay

So, basically, with this approach delay time will grow exponentially (with small randomness) up to some specified value.

Fixed interval delay:
I think the name is self-explanatory.

PBS-Java configuration:
Sample yaml configuration (but can be configured via dot notation as any other PBS-Java configuration):

gdpr:
  vendorlist:
    [v2/v3]: # can be configured per v2/v3 versions
      retry-policy:
        exponential-backoff:
          delay-millis: 60000 # initial delay
          max-delay-millis: 120000 # max delay
          factor: 1.1
          jitter: 0.2
        fixed-interval:
          delay: 60000

PBS host has to choose either exponential backoff or fixed interval retry policy. Specifying both (as in provided yaml) will result in startup error.