kube-mgmt does not retry adding policies to OPA

[kechengcao-db]

Hi team,

Recently we have discovered a weird issue. We are using kube-mgmt to help us load some Kafka Kubernetes resources and polices stored in configmap into OPA. Sometimes the OPA container can fail on image pulling at the beginning, so the kube-mgmt starts first and tries to load Kafka resources and configmap polices into OPA. Due to the OPA not being ready, kube-mgmt will fail with the error below.

For Kafka resources:
time="2023-05-03T22:01:26Z" level=error msg="Sync for kafka.strimzi.io/v1beta1/kafkatopics failed, trying again in 52.020401977s. Reason: Put \"http://localhost:8181/v1/data/kubernetes/kafkatopics\": dial tcp [::1]:8181: connect: connection refused"

For configmap policies:
time="2023-05-03T21:58:09Z" level=info msg="Added policy kafka-general-0/opa-policy-allow-upgrade/main, err=Put \"http://localhost:8181/v1/policies/kafka-general-0/opa-policy-allow-upgrade/main\": dial tcp [::1]:8181: connect: connection refused"

You can clearly see that for the Kafka resources, kube-mgmt retruns an error and retry to load the resources again in the future. However, for the configmap policies, it returns with an INFO log and stop retrying to load the data again even when it failed. Finally, when the OPA container is ready, the Kafka Resources are successfully loaded due to retry. However, the policies in the configmap are missing.

We are defining our kube-mgmt container with the following code:


Policies are defined in configmap and marked with
"openpolicyagent.org/policy": "rego", label.
Kafka resoruces are loaded in through --replicate=

Do you have any idea on how we could make the kube-mgmt also retry to load the policies again when failing?

[eshepelyuk]

Hello
The resolution can be quite similar to #189
Closing the issue in favor of it.