Multi-cluster with kubebuilder

[HamzaZo]

Hello,

I'd like to develop an operator multi-cluster ( I mean by that, the operator will manage kubernetes resources in both clusters ), I was wondering how we can achieve something like this with kubebuilder ?

How the operator can communicate with the second cluster ? by specifying a kubeconfig ?

Thanks!

[FillZpp]

A kubeconfig of the second cluster, and then create a controller-runtime cluster with it, which provides client and cache for the second cluster. But it is still a little laborious if you expect to run those controllers for multiple clusters.

[HamzaZo]

Can you elaborate more on how I can create a controller-runtime using a specified kubeconfig ? Is it something we can do with a kubebuilder CLI ? Can you share links or docs on how to achieve something like this ?

[camilamacedo86]

@HamzaZo you scaffold the project with kubebuilder but the example above is a code implementation that you will need to do/add on the project

[HamzaZo]

@camilamacedo86 Thanks for your reply, If I understand correctly I need to add the whole cluster.go file on the project?
My question is how things are going to work together (how the operator will use the cluster.go in order to communicate with the second cluster), sorry I still didn't get the idea.

do you have any links or docs on how things work for this use case?

[camilamacedo86]

HI @HamzaZo,

What I understand that you need a configure the client to use the context (kubeconfig) for cluster A and then for cluster B. The rest would be the same. See: #2729 (comment)

Then, when you configure the controller you will pass the client for cluster A or B: https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/cmd/main.go#L105-L111. If the controller should run in both then you would start twice the same controller where it would receive in the first start the client A and in the second the client B for example. If you want
a controller only in cluster A and not in B then you will define it just once.

That would be how you would configure that without any further method/help from the controller runtime. However, @FillZpp shared that controller-runtime has an option to help you with see: https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/cluster/cluster.go#L146

You will not copy and paste this cluster.go. That is from controller-runtime implementation. Instead of that you should check the docs and see how can you use the funcs to configure the manager and controllers to make it simple.

I hope that helps you out.

[camilamacedo86]

Hi @HamzaZo,

I think you might able to achieve this by setting up a combined kubeconfig with contexts for each of the clusters such as follows:

• Combine kubeconfigs:

First, ensure that your kubeconfig has contexts set up for both the clusters. This usually involves merging multiple kubeconfig files or manually adding cluster, user, and context entries.

• Initialize Controller Manager:
  Set up a single controller-runtime Manager.

var mgr, err = ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
   Scheme:             scheme,
   MetricsBindAddress: metricsAddr,
   Port:               9443,
   // other necessary configurations
})

• Context-Specific Clients:
  When you want your reconciler to communicate with a specific cluster, you should set up its client using the correct context:

cluster1Config := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
   clientcmd.NewDefaultClientConfigLoadingRules(),
   &clientcmd.ConfigOverrides{CurrentContext: "context-of-cluster-1"})
client1, err := client.New(cluster1Config, client.Options{Scheme: scheme})

cluster2Config := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
   clientcmd.NewDefaultClientConfigLoadingRules(),
   &clientcmd.ConfigOverrides{CurrentContext: "context-of-cluster-2"})
client2, err := client.New(cluster2Config, client.Options{Scheme: scheme})

Now, within your reconciler, you can decide which client (client1 or client2) to use based on which cluster you want to interact with.

• Reconciler Logic:

Your reconciler logic will have the intelligence to determine which client to use for each resource reconciliation. Typically, this can be based on certain metadata or spec attributes of the resource.

**However, that is a really a good question.
I never try to do something like that

Did you try something like the above approach?
Did you progressed on this one?**

[camilamacedo86]

I am closing this one as answered since we no longer interact here.
however, please feel free to raise new topics if you you need to.