HTTPRoute applied on Gateway and cross-cluster traffic

[NikitaCloudRuntime]

Hi there :)

sorry for bringing up a new question but sadly after hours of debugging I was unable to solve the issue :(

I have two clusters - linkerd-a and linkerd-b
Both running 1.16.10 version of Linkerd control plane. Also multicluster and failover extension are installed to both. Clusters use shared CA authority.

In cluster linkerd-b I have installed echo service and mirrored it to the cluster linkerd-a. Connection is established over the gateway, not flat network.
I used the following yaml manifests to install this part

apiVersion: v1
kind: Namespace
metadata:
  name: echoserver
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
  namespace: echoserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
      annotations:
        linkerd.io/inject: enabled
    spec:
      containers:
      - image: ealen/echo-server:latest
        imagePullPolicy: IfNotPresent
        name: echoserver
        ports:
        - containerPort: 80
        env:
        - name: PORT
          value: "80"
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
  namespace: echoserver
  labels:
    mirror.linkerd.io/exported: "true"
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: ClusterIP
  selector:
    app: echoserver

After manifests from above are applied, I can see mirrored service in cluster-a

 kubectl get services -n echoserver --context $A 
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
echoserver-linkerd-b   ClusterIP   10.100.160.156   <none>        80/TCP    3h7m

and I can curl it

kubectl exec -it nginx-netshoot-5f6cd695b6-h66kj -c netshoot -- curl echoserver-linkerd-b.echoserver   | jq -r '.environment.HOSTNAME'
echoserver-5b88cccdf6-h5wns

The responding HOSTNAME is correct and matching pod name running in cluster linkerd-b

As the next step I'm trying to test HttpRoute resource and route traffic coming to gateway in cluster linkerd-b to either echoserver-headers service if coming request contains x-request-id: alternative header or to the echoserver service otherwise. For that I'm applying the following manifests

apiVersion: policy.linkerd.io/v1beta2
kind: HTTPRoute
metadata:
  name: echo-server-headers
  namespace: echoserver
spec:
  parentRefs:
    - name: echoserver
      kind: Service
      group: core
      namespace: echoserver
      port: 80
  rules:
    - matches:
      - headers:
        - name: "x-request-id"
          value: "alternative"
      backendRefs:
        - name: "echoserver-headers"
          namespace: echoserver
          port: 80
    - backendRefs:
      - name: "echoserver"
        namespace: echoserver
        port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver-headers
  namespace: echoserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echoserver-headers
  template:
    metadata:
      labels:
        app: echoserver-headers
      annotations:
        linkerd.io/inject: enabled
    spec:
      containers:
      - image: ealen/echo-server:latest
        imagePullPolicy: IfNotPresent
        name: echoserver
        ports:
        - containerPort: 80
        env:
        - name: PORT
          value: "80"
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver-headers
  namespace: echoserver
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  type: ClusterIP
  selector:
    app: echoserver-headers

Once it's applied, I'm trying to connect to the same endpoint(echoserver-linkerd-b.echoserver) from cluster linkerd-a and use configured HTTP header, but my requests get always landed to the echoserver deployment :(

kubectl exec -it --context $A nginx-netshoot-5f6cd695b6-h66kj -c netshoot -- curl -H 'x-request-id: alternative' echoserver-linkerd-b.echoserver   | jq -r '.environment.HOSTNAME'
echoserver-5b88cccdf6-h5wns

However, if calling from within the local linkerd-b cluster, everything works as expected

kubectl exec --context $B nginx-netshoot-6fdd4fcc5b-x9lzq -c netshoot -- curl  -H 'x-request-id: alternative' echoserver.echoserver | jq -r '.environment.HOSTNAME'
echoserver-headers-788f697db5-9hmg7

I'm almost sure this must be working since previously I tested out how Argo Rollouts works with similar installation

• creating echoserver-canary and echoserver-stable svc/deployments
• creating argo rollout and httpRoute
  Once the rollout starts, controller changes weights of the backend routes and I saw how traffic from cluster linkerd-a was balancer across two services in cluster linkerd-b.

Does anyone has any idea why it might be not working and what I'm missing or where is my mistake/misunderstanding?

Thank you

[alpeb]

Thank you so much @NikitaCloudRuntime for the very detailed report; this allowed me to actually reproduce the problem. So for now I'm gonna move this into an issue for better tracking.

[kflynn]

Just realized that it's likely worth copying my answer from the issue to here for any future readers:

I think that this is actually expected behavior if the HTTPRoute is installed in linkerd-b. The simplest way to view this is that routing decisions are made by the Linkerd at the Pod that originates the request: if that Pod is in linkerd-a, it can't see any HTTPRoutes in linkerd-b, so it won't use them.

If you mirror both echoserver and echoserver-headers, you can use an HTTPRoute in linkerd-a to canary between those two mirrors, and that should work fine.