Feature Request: Support exec plugin in Sveltos

[kahirokunn]

Overview

Currently in Sveltos, authentication information is provided using kubeconfig to access clusters. However, in Kubernetes as a service like GKE, tokens have short expiration periods (e.g., 48 hours) and require regular updates. Furthermore, when using managed Kubernetes services, it is common to dynamically obtain tokens provided by vendor-specific authentication mechanisms.
In such cases, kubectl natively supports "exec plugin" and its use is recommended as one of the best practices. Therefore, we would like to add exec plugin support to Sveltos to enable automatic authentication token retrieval from managed services.

Background / Motivation

• Taking GKE as an example, access tokens expire in 48 hours, necessitating an automatic renewal mechanism.
• Other Kubernetes managed services like EKS also use vendor-specific authentication infrastructure for token retrieval. The challenge is how to handle the required CLI and libraries for using this infrastructure in Sveltos.
• If "exec plugin" can be used like kubectl, users can continue using their familiar kubeconfig settings, and operations can dynamically obtain and update tokens using managed service-specific CLIs.

Example: kubeconfig with exec plugin when using EKS

As shown below, exec is specified in the users section, executing aws eks get-token to obtain the token.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: yyyy
    server: https://xxxx.gr7.ap-northeast-1.eks.amazonaws.com
  name: arn:aws:eks:ap-northeast-1:ooooo:cluster/sveltos-mgmt
contexts:
- context:
    cluster: arn:aws:eks:ap-northeast-1:ooooo:cluster/sveltos-mgmt
    user: arn:aws:eks:ap-northeast-1:ooooo:cluster/sveltos-mgmt
  name: arn:aws:eks:ap-northeast-1:ooooo:cluster/sveltos-mgmt
current-context: arn:aws:eks:ap-northeast-1:ooooo:cluster/sveltos-mgmt
kind: Config
preferences: {}
users:
- name: arn:aws:eks:ap-northeast-1:ooooo:cluster/sveltos-mgmt
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - ap-northeast-1
      - eks
      - get-token
      - --cluster-name
      - sveltos-mgmt
      - --output
      - json
      command: aws
      interactiveMode: IfAvailable
      provideClusterInfo: false

Expected Approaches

1. Enable all controllers to use AWS CLI and other necessary vendor CLIs.

   • Prepare images with pre-installed CLIs for deployment in each container and enable execution across all controllers.
   • When kubeconfig uses the exec plugin, automatically obtain tokens using that CLI.
   • For CLI injection mechanisms, content utilizing Nix might be helpful as reference.

2. Set up a dedicated controller like Token Resolver Controller.

   • Inject CLIs only into this controller as sidecars or init containers to obtain necessary tokens.
   • Create a mechanism for this controller to redistribute and manage updated kubeconfig within Sveltos.
   • Through centralized management, token updates can be controlled while minimizing CLI installations.

Expected Benefits

• Reduced operational overhead in managing authentication token updates for managed services.
• Enables configuration aligned with managed Kubernetes service best practices, allowing users to directly utilize their existing kubeconfig mechanisms.
• By clarifying CLI installation locations and update mechanisms, security and operational management (version upgrades, installation oversight, etc.) can be handled more reliably.

Suggestions/Requests

• Please add native support for exec plugin in Sveltos, enabling the use of CLI commands defined in the users section of kubeconfig files.
• Please provide either a mechanism for executing CLIs on the Sveltos controller side or a dedicated mechanism like Token Resolver Controller.
• It would be convenient if users could define options and extension points for automatic token updates.

Thank you for your consideration.

[gianlucam76]

Thank @kahirokunn

Currently Sveltos does not use kubectl (and so extension). Once Sveltos has access to a cluster, Sveltos has a token renewal mechanism. User can instruct Sveltos to renew token every X hours (so before token expiration) using a ServiceAccount (also specified by the user) in the managed cluster which has enough permissions to do what Sveltos needs to deploy in such cluster.

I know doing the way you are suggesting will open more opportunities, but at the same time having kubectl (and extension) in a pod running in the management cluster might also open to more CVEs.

what do you think?

[kahirokunn]

The exec plugin is described in the official documentation as supporting not only kubectl but also client-go.
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

Therefore, we believe that kubectl management is unnecessary.

[gianlucam76]

Thank you. Not familiar with that. I will have to go through doc first to understand what's available.

[kahirokunn]

Since Sveltos is using client-go, I believe the exec plugin should work without any additional configuration in Sveltos itself. However, I suspect it won't actually function because the commands we want to execute may not be present in the runtime environment.
That said, since Kubernetes operators ideally should operate by responding to custom resources, I believe the exec plugin would become unnecessary once the following PR progresses and supports multiple CAP authentication methods:#440
However, this is just my perspective, and I created this issue because I would like to hear your thoughts on this matter as well.

[gianlucam76]

Exactly. I was referring to executable. Those are not in any of Sveltos pods.

[kahirokunn]

Yes, I'm actually concerned about that as well.
While there are actually many implementation methods available.
For example, here are some options:

1. One option is to inject commands via initContainer or sidecar and cache them in PVC.
2. Another option is to tune it like Crossplane to execute commands in a separate pod.
3. Lazy loading, which downloads commands when needed if they don't exist, is also an option.

I'm finding it challenging to determine which method best combines security with practicality and convenience.