Load bundle from backup service

[rmetcalf9]

We are deploying opa across multiple microservices. Each will have a number of agents and these agents are configured to download the bundle file from a s3 bucket.

I have been told to implement redundancy in this process. If one s3 bucket in one region goes down I need the opa agent to get the bundle from a second region.

I would like to be able to make a configuration something like this:

services:
  s3:
    url: https://my-bundle-bucket-a.s3.eu-west-1.amazonaws.com
    credentials:
      s3_signing:
        metadata_credentials:
          aws_region: eu-west-1
  s3b:
    url: https://my-bundle-bucket-b.s3.eu-west-1.amazonaws.com
    credentials:
      s3_signing:
        metadata_credentials:
          aws_region: eu-west-1

bundles:
  authz:
    service: s3, s3b
    resource: live-policy-bundle.tar.gz
    persist: true
    polling:
      min_delay_seconds: 30
      max_delay_seconds: 60

Then the bundle will use service s3 to poll for the bundle and if that fails it will go to service s3b and use a different bucket.
I have checked and the etags on both buckets match so there won't be an issue there.

I don't think a config like the above can work. I have also tried having multiple authz bundles but this didn't work either.
What do you recommend as the best strategy for having agents access redundant bundle locations?

Robert

[anderseknert]

Hi Robert 👋 Great question!

The remote bundle model has some level of redundancy built-in, given how OPA will keep serving the latest bundle it downloaded even when the remote goes down. The persist option (as you've included in your example) additionally allows OPA to even be restarted with the remote being unreachable. So from an "uptime" point of view, OPA should have you fairly well covered. One problem this model doesn't address is of course that of data getting outdated. If the remote is down for long enough, you might not see downtime, but you will start to see OPA working on data that might no longer be up-to-date. Someone even suggested shutting down entirely under such conditions, but introducing a failover mechanism could definitely be a less dramatic alternative. I wonder though, would it not be possible to put a loadbalancer (or whatever the correct term for such a proxy would be) in front of two geographically separate S3 buckets, and have traffic directed to the second one only if the first one should fail?

[rmetcalf9]

We are hosting our opa-agents in AWS fargate and the tasks can scale up as well as down.
The problem is going to be that if the task has to scale up due to load while the S3 bucket is down then the methods you described won't help because there will be new agent container instances launched.
I guess I need to look into the loadbalancer options you mentioned, but it would be nice if OPA could use a fallback.

[anderseknert]

Indeed, it doesn’t cover all scenarios. I think you have a valid case for a failover, so I’d suggest creating a feature request in the OPA tracker, and we can see if others have ideas around that too, or would like to work on it.

[rmetcalf9]

I have raised open-policy-agent/opa#4397