From 56664d8a60b7ae00133d68453a46c4b89ac5a1cb Mon Sep 17 00:00:00 2001 From: Alex Leong Date: Wed, 4 Dec 2024 09:55:16 -0800 Subject: [PATCH] Add federated services docs (#1874) * Add federated services docs Signed-off-by: Alex Leong * feedback Signed-off-by: Alex Leong * servce Signed-off-by: Alex Leong * feedback Signed-off-by: Alex Leong * Feedback Signed-off-by: Alex Leong * Update linkerd.io/content/2-edge/features/multicluster.md * Update linkerd.io/content/2-edge/tasks/federated-services.md Co-authored-by: Alejandro Pedraza --------- Signed-off-by: Alex Leong Co-authored-by: Flynn Co-authored-by: Alejandro Pedraza Signed-off-by: Ivan Porta --- .../content/2-edge/features/multicluster.md | 32 +- .../content/2-edge/reference/multicluster.md | 35 +- .../2-edge/tasks/federated-services.md | 415 ++++++++++++++++++ 3 files changed, 466 insertions(+), 16 deletions(-) create mode 100644 linkerd.io/content/2-edge/tasks/federated-services.md diff --git a/linkerd.io/content/2-edge/features/multicluster.md b/linkerd.io/content/2-edge/features/multicluster.md index 6b7281dc34..71d532a4ac 100644 --- a/linkerd.io/content/2-edge/features/multicluster.md +++ b/linkerd.io/content/2-edge/features/multicluster.md @@ -39,8 +39,8 @@ splitting](../traffic-split/) or [dynamic request routing](../request-routing/) to allow local services to access the *Foo* service as if it were on the local cluster. -Linkerd supports two basic forms of multi-cluster communication: hierarchical -and flat. +Linkerd supports three basic forms of multi-cluster communication: hierarchical, +flat, and federated. ![Architectural diagram comparing hierarchical and flat network modes](/docs/images/multicluster/flat-network.png) @@ -65,11 +65,18 @@ several advantages: * Better multi-cluster authorization policies, as workload identity is preserved across cluster boundaries. -Hierarchical (gateway-based) and flat (direct pod-to-pod) modes can be combined, -and pod-to-pod mode can be enabled for specific services by using the -`remote-discovery` value for the label selector used to export services to other -clusters. See the [pod-to-pod multicluster -communication](../../tasks/pod-to-pod-multicluster/) guide and the +### Federated services + +A federated service is a union of services with the same name and namespace +in multiple different clusters. Meshed clients that send traffic to a federated +service will have that traffic distributed across all replicas of services in +the federated service across clusters. Federated services use the *flat +networking* model and do not use a gateway intermediary. + +These modes can be combined, with each specific service selecting the mode that +is most appropriate for that service. See the +[pod-to-pod multicluster communication](../../tasks/pod-to-pod-multicluster/) +guide, the [federated services](../../tasks/federated-services/) guide, and the [multi-cluster reference](../../reference/multicluster/) for more. ## Headless services @@ -80,8 +87,8 @@ communication](../../tasks/pod-to-pod-multicluster/) guide and the By default, Linkerd will mirror all exported services as Kubernetes `clusterIP` services. This also extends to [headless services][headless-svc]; an exported headless service will be mirrored as `clusterIP` and have an IP address -assigned to it. In general, headless services _should not have an IP address_, -they are used when a workloads needs a stable network identifier or to +assigned to it. In general, headless services *should not have an IP address*; +they are used when a workload needs a stable network identifier or to facilitate service discovery without being tied to Kubernetes' native implementation. This allows clients to either implement their own load balancing or to address a pod directly through its DNS name. In certain @@ -110,14 +117,17 @@ exported as a headless service, the hosts backing the service need to be named Deployment would not be supported, since they do not allow for arbitrary hostnames in the pod spec). +Note that headless services can *not* be part of a federated service. + Ready to get started? See the [getting started with multi-cluster guide](../../tasks/multicluster/) for a walkthrough. ## Further reading -* [Multi-cluster installation instructions](../../tasks/installing-multicluster/). +* [Multi-cluster installation instructions](../../tasks/installing-multicluster/) * [Pod-to-pod multicluster communication](../../tasks/pod-to-pod-multicluster/) -* [Multi-cluster communication with StatefulSets](../../tasks/multicluster-using-statefulsets/). +* [Multi-cluster communication with StatefulSets](../../tasks/multicluster-using-statefulsets/) +* [Federated services](../../tasks/federated-services/) * [Architecting for multi-cluster Kubernetes](/2020/02/17/architecting-for-multicluster-kubernetes/), a blog post explaining some of the design rationale behind Linkerd's multi-cluster diff --git a/linkerd.io/content/2-edge/reference/multicluster.md b/linkerd.io/content/2-edge/reference/multicluster.md index 1ab60ee1c2..7934291375 100644 --- a/linkerd.io/content/2-edge/reference/multicluster.md +++ b/linkerd.io/content/2-edge/reference/multicluster.md @@ -5,14 +5,16 @@ description: Multi-cluster communication Linkerd's [multi-cluster functionality](../../features/multicluster/) allows pods to connect to Kubernetes services across cluster boundaries in a way that -is secure and fully transparent to the application. As of Linkerd 2.14, this -feature supports two modes: hierarchical (using an gateway) and flat (without a -gateway): +is secure and fully transparent to the application. This feature supports three +modes: hierarchical (using a gateway), flat (without a gateway), and federated. -* **Flat mode** requires that all pods on the source cluster be able to directly - connect to pods on the destination cluster. * **Hierarchical mode** only requires that the gateway IP of the destination cluster be reachable by pods on the source cluster. +* **Flat mode** requires that all pods on the source cluster be able to directly + connect to pods on the destination cluster. +* **Federated mode** has the same requirements as flat mode but allows a service + deployed to multiple clusters to be treated as a single cluster agnostic + service. These modes can be mixed and matched. @@ -66,3 +68,26 @@ together, a Kubernetes `Secret` is created in the control plane's namespace with a kubeconfig file that allows an API client to be configured. The kubeconfig file uses RBAC to provide the "principle of least privilege", ensuring the *destination service* may only access only the resources it needs. + +## Federated Services + +Federated services take this a step farther by allowing a service which is +deployed to multiple clusters to be joined into a single unified service. + +The service mirror controller will look for all services in all linked clusters +which match a label selector (`mirror.linkerd.io/federated=member` by default) +and create a federated service called `-federated` which will act as +a union of all those services with that name. For example, all traffic sent to +the `store-web-federated` federated service will be load balanced over all +replicas of all services named `store-web` in all linked clusters. + +The concept of "namespace sameness" applies, which means that the federated +service will be created in the same namespace as the individual services and +services can only join a federated service in the same namespace. + +Since Linkerd's *destination service* uses "remote-discovery" to discover the +endpoints of a federated service, all of the requirements for flat mode also +apply to federated services: the clusters must be on a flat network where pods +in one cluster can connect to pods in the others, the clusters must have the +same trust root, and any clients connecting to the federated service must be +meshed. diff --git a/linkerd.io/content/2-edge/tasks/federated-services.md b/linkerd.io/content/2-edge/tasks/federated-services.md new file mode 100644 index 0000000000..4605f3f3fc --- /dev/null +++ b/linkerd.io/content/2-edge/tasks/federated-services.md @@ -0,0 +1,415 @@ +--- +title: Multi-cluster Federated Services +description: Using multi-cluster federated services +--- + +Linkerd's [multicluster extension](../multicluster/) can create federated +services which act as a union of multiple services in different clusters with +the same name and namespace. By sending traffic to the federated service, that +traffic will be load balanced among all endpoints of that service in all linked +clusters. This allows the client to be cluster agnostic, balance traffic across +multiple clusters, and be resiliant to the failure of any individual cluster. + +Federated services send traffic directly to the pods of the member services +rahter than through a gateway. Therefore, federated services have the same +requirements as *pod-to-pod* multicluster services: + +* The clusters must be on a *flat network*. In other words, pods from one + cluster must be able to address and connect to pods in the other cluster. +* The clusters must have the same trust root. +* Any clients connecting to the federated service must be meshed. + +This guide will walk you through creating a federated service to load balance +traffic to a service which exists in multiple clusters. A federated service can +include services from any number of clusters, but in this guide we'll create +a federated service for a service that spans 3 clusters. + +## Prerequisites + +* Three clusters. We will refer to them as `west`, `east`, and `north` in this + guide. +* The clusters must be on a *flat network*. In other words, pods from one + cluster must be able to address and connect to pods in the other cluster. +* Each of these clusters should be configured as `kubectl` + [contexts](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/). + We'd recommend you use the names `west`, `east`, and `north` so that you can + follow along with this guide. It is easy to + [rename contexts](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-rename-context-em-) + with `kubectl`, so don't feel like you need to keep them all named this way + forever. + +## Step 1: Installing Linkerd and Linkerd-Viz + +First, install Linkerd and Linkerd-Viz into all three clusters, as described in +the [multicluster guide](../multicluster/#install-linkerd-and-linkerd-viz). +Make sure to take care that all clusters share a common trust anchor. + +## Step 2: Installing Linkerd-Multicluster + +We will install the multicluster extension into all three clusters. We can +install without the gateway because federated services use direct pod-to-pod +communication. + +```console +> linkerd --context west multicluster install --gateway=false | kubectl --context west apply -f - +> linkerd --context west check + +> linkerd --context east multicluster install --gateway=false | kubectl --context east apply -f - +> linkerd --context east check + +> linkerd --context north multicluster install --gateway=false | kubectl --context north apply -f - +> linkerd --context north check +``` + +## Step 3: Linking the Clusters + +We use the `linkerd multicluster link` command to link the `east` and `north` +cluster to the `west` cluster. This is exactly the same as in the regular +[Multicluster guide](../multicluster/#linking-the-clusters) except that we pass +the `--gateway=false` flag to create a Link which doesn't require a gateway. + +```console +> linkerd --context east multicluster link --cluster-name=east --gateway=false | kubectl --context west apply -f - +> linkerd --context north multicluster link --cluster-name=north --gateway=false | kubectl --context west apply -f - +> linkerd --context west check +``` + +## Step 4: Deploy a Service + +For our guide, we'll deploy the [bb](https://github.com/BuoyantIO/bb) service, +which is a simple server that just returns a static response. We deploy it +into all three clusters but configure each one with a different response string +so that we can tell the responses apart: + +```bash +> cat < cat < cat < kubectl --context east -n mc-demo label svc/bb mirror.linkerd.io/federated=member +> kubectl --context north -n mc-demo label svc/bb mirror.linkerd.io/federated=member +> kubectl --context west -n mc-demo label svc/bb mirror.linkerd.io/federated=member +``` + +You should immediately see a federated service created in the `west` cluster: + +```console +> kubectl --context west -n mc-demo get svc +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +bb-federated ClusterIP 10.43.56.245 8080/TCP 114s +``` + +We can also check the `status` subresource of each of the Link resources to see +which services have joined federated services or if there are any errors. + +```console +> kubectl --context west -n linkerd-multicluster get link/east -ojsonpath='{.status.federatedServices}' | jq . +[ + { + "conditions": [ + { + "lastTransitionTime": "2024-11-07T19:53:01Z", + "localRef": { + "group": "", + "kind": "Service", + "name": "bb-federated", + "namespace": "mc-demo" + }, + "message": "", + "reason": "Mirrored", + "status": "True", + "type": "Mirrored" + } + ], + "controllerName": "linkerd.io/service-mirror", + "remoteRef": { + "group": "", + "kind": "Service", + "name": "bb", + "namespace": "mc-demo" + } + } +] +> kubectl --context west -n linkerd-multicluster get link/north -ojsonpath='{.status.federatedService +s}' | jq . +[ + { + "conditions": [ + { + "lastTransitionTime": "2024-11-07T19:53:06Z", + "localRef": { + "group": "", + "kind": "Service", + "name": "bb-federated", + "namespace": "mc-demo" + }, + "message": "", + "reason": "Mirrored", + "status": "True", + "type": "Mirrored" + } + ], + "controllerName": "linkerd.io/service-mirror", + "remoteRef": { + "group": "", + "kind": "Service", + "name": "bb", + "namespace": "mc-demo" + } + } +] +``` + +## Step 6: Send some traffic! + +We'll create a deployment that uses `curl` to generate traffic to the +`bb-federated` service. + +```bash +> cat < kubectl --context west -n mc-demo logs deploy/traffic -c traffic +{"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-407945949","payload":"hello from east\n"} +{"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-420928530","payload":"hello from west\n"} +{"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-433442439","payload":"hello from north\n"} +{"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-445418175","payload":"hello from west\n"} +{"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-457469540","payload":"hello from west\n"} +{"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-469729132","payload":"hello from west\n"} +{"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-481971153","payload":"hello from west\n"} +{"requestUID":"in:http-sid:terminus-grpc:-1-h1:8080-496032705","payload":"hello from east\n"} +... +``` + +## Next Steps + +We now have a federated service that balances traffic accross services in three +clusters. Additional clusters can be added simply by linking the new cluster +and adding the `mirror.linkerd.io/federated=member` label to the services that +you wish to add to the federated service. Similarly, services can be removed +from the federated service at any time by removing the label. + +You may notice that the `bb-federated` federated service exists only in the +`west` cluster and not in the `east` or `north` clusters. This is because Links +are directional and to keep this guide simple, we only linked north and east to +west, and not the other way around. If we were to create links in both +directions between all three clusters, we would get a `bb-federated` service in +all three clusters. + +## Troubleshooting + +* The first step of troubleshooting should be to run the `linkerd check` command + in each of the clusters. In particular, look for the `linkerd-multicluster` + checks and ensure that all linked clusters are listed: + +```console +linkerd-multicluster +-------------------- +√ Link CRD exists +√ Link resources are valid + * east + * north +√ remote cluster access credentials are valid + * east + * north +√ clusters share trust anchors + * east + * north +√ service mirror controller has required permissions + * east + * north +√ service mirror controllers are running + * east + * north +``` + +* Check the `status` subresource of the Link resource. If any services failed to + join the federated service, they will appear as an error here. +* If a service that should join a federated service is not present in the Link + `status`, ensure that the service matches the federated service label selector + (`mirror.linkerd.io/federated=memeber` by default). +* Use the `linkerd diagnostics endpoints` command to see all of the endpoints + in a federated service: + +```console +> linkerd --context west diagnostics endpoints bb-federated.mc-demo.svc.cluster.local:8080 +NAMESPACE IP PORT POD SERVICE +mc-demo 10.42.0.108 8080 bb-85f9bbc898-j7fbq bb.mc-demo +mc-demo 10.23.1.43 8080 bb-7d9f44c6fd-9s848 bb.mc-demo +mc-demo 10.23.0.42 8080 bb-74c6c64948-j5drn bb.mc-demo +```