First-class support for external destination clusters in ArgoCD

[NiklasRosenstein]

Nyl requires access to the Kubernetes API for certain features (see https://niklasrosenstein.github.io/nyl/reference/cluster-connectivity/). As an ArgoCD plugin, Nyl runs in the argocd-repo-server Pod.

Status-quo

Currently, Nyl is injected as a sidecar container into the argocd-repo-server Pod. In order to enable e.g. the ${{ lookup(...) }} feature, it requires access to the destination Kubernetes cluster. This is enabled by granting the argocd-repo-server ServiceAccount full access to the cluster.

# argocd-values.yaml
repoServer:
  extraContainers:
    - name: nyl-v1
      image: ghcr.io/niklasrosenstein/nyl/argocd-cmp:0.0.7
      # ...

  # HACK: Allow argocd-repo-server to read all Kubernetes resources. This is used by Helm and Nyl lookups.
  #       However, this only works in-cluster and not with other clusters.
  clusterRoleRules:
    enabled: true

But this means a lookup() won't be performing a lookup against the destination cluster, if the destination cluster is not the same as the local cluster that ArgoCD is installed to.

Proposed solution

Nyl can lookup the ArgoCD AppProject and Application that it is being invoked for in the local Kubernetes cluster (see ArgoCD Build Environment). It can then lookup the corresponding destination cluster from the project and initialize its Kubernetes API client using the credentials stored in the corresponding Kubernetes secret.

ArgoCD already stores Kubernetes cluster credentials in a secret. This requires no special configuration on the user or administrator end, but just needs to be implemented in nyl template.