issue with http.send

[karthik-phl]

Hi,

I'm running a policy bundle within an OPA sidecar that processes allow decisions sent by the Istio Envoy. The policy bundle tries to call the Azure AD tenant metadata endpoint to retrieve the JWKS URL. I can see that the policy is trying to call the endpoint, but http.send(...) doesn't seem to return - can't see any failures in the log even in "debug" mode. The policy query evaluates to "null", so I'm guessing some exception happened?

The OPA sidecar container image is pulled from "openpolicyagent/opa:latest-istio"

The metadataRequest function is:

metadataRequest(url) := rsp {
  print("META REQ *** ", url)
  rsp := http.send({
    "url": url,
    "method": "GET",
    "force_cache": true,
    "force_cache_duration_seconds": 86400, # Cache response for 24 hours
    "timeout": "10s",
  }).body
  print("META RSP ***", rsp)
}

This function and the policy query works fine when I run it via opa eval

Would appreciate some help with this. Thanks :)

[anderseknert]

Hi @karthik-phl 👋 That's weird! Could you try and add "raise_error": false to the options provided to see if that changes the result?

[karthik-phl]

Hi @anderseknert, I accidentally posted rather than replying...

[karthik-phl]

Hi @anderseknert,

Thanks for getting back to me. I tried that but, unfortunately, it didn't work.

I have modified the ext-authz EnvoyFilter to include a timeout (found this while trawling some old posts in GitHub)

############################################################
# Envoy External Authorization filter that will query OPA.
############################################################
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz
  namespace: istio-system
spec:
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.ext_authz
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
            transport_api_version: V3
            status_on_error:
              code: 401
            with_request_body:
              max_request_bytes: 8192
              allow_partial_message: true
            grpc_service:
              # Added timeout (seems like default timeout is 250ms)
              timeout: 5s
              # NOTE(tsandall): when this was tested with the envoy_grpc client the gRPC
              # server was receiving check requests over HTTP 1.1. The gRPC server in
              # OPA-Istio would immediately close the connection and log that a bogus
              # preamble was sent by the client (it expected HTTP 2). Switching to the
              # google_grpc client resolved this issue.
              google_grpc:
                target_uri: 127.0.0.1:9191
                stat_prefix: "ext_authz"

and modified the metadata function to have a 1s timeout.

This seems to work now - I'm guessing that the grpc timeout was closing the connection between Istio and OPA sidecars causing the channel responsible for http.Send(...) to be discarded?

[anderseknert]

Ah, nice find!