Feature Request: Support for Multi-Cluster ClusterProfiles

[kahirokunn]

Hello,

Thank you for your work on Sveltos!

I wanted to ask if there are any plans to support multi-cluster ClusterProfiles in Sveltos. For example, aligning with the ClusterProfile API as described in KEP-4322.

[gianlucam76]

Thank you @kahirokunn

Not familiar with that. From what I read (quickly) the goal seems to be to have a unique representation for clusters (instead of using the SveltosCluster or ClusterAPI concept).

Is your question related to that? If not would you mind expanding a bit? Thank you

[kahirokunn]

Hello! Thank you so much for taking the time to look into my question. Yes, it is indeed related to unifying how clusters are represented, similar to what’s described in KEP-4322. Specifically, I’d love to know if there are any plans or discussions around supporting a multi-cluster ClusterProfile in Sveltos—one that doesn’t rely solely on SveltosCluster or Cluster API. Thank you!

[gianlucam76]

Thank you @kahirokunn.

The short answer is yes. If there is a new generic way of representing clusters, I will definitely have Sveltos support that. It should not be difficult to integrate.

At the end of the day Sveltos just needs to:

1. Have a resource in the management cluster representing a managed cluster where labels can be added
2. A kubeconfig to access the managed cluster

[kahirokunn]

@gianlucam76,

Thank you for your response.

I have been using ClusterAPI to create clusters, but I also wanted to manage existing on-premises clusters simultaneously. In doing so, I encountered challenges that ClusterAPI couldn't resolve, and I found the multi-cluster support of ClusterProfile to be extremely useful.

If Sveltos could also support multi-cluster ClusterProfile, it would allow for managing clusters in a more standardized way. Additionally, compared to SveltosCluster, it would be great that Sveltos itself could natively integrate with many multi-cluster tools.

I look forward to the continued development of Sveltos.

Thank you very much!

[gianlucam76]

Thank you. It makes sense. Currently you can register on prem clusters (or any other existing cluster) with Sveltos and then Sveltos can manage add-ons and applications.
But I do see your point.

I will read more about it. But if you are already familiar, can you share more about ClusterProfile? The questions I have are following

1. I assume there is a management/control cluster where there is an instance for every managed cluster
2. Where/how can I get the kubeconfig to access those clusters

Assuming # 2 is something that can be found in the management cluster, having Sveltos support it would be really easy.

Thank you again for suggesting this integration.

[kahirokunn]

The ClusterProfile API provides a standardized way to represent clusters in a multi-cluster environment. While there are three possible approaches for cluster access, let me focus on the most straightforward implementation path:

1. Regarding management/control cluster setup:
   The ClusterProfile API runs on a management/hub cluster, with each managed cluster represented by a ClusterProfile object:

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
  name: onprem-cluster-east
  namespace: production-clusters
  labels:
    x-k8s.io/cluster-manager: sveltos
spec:
  displayName: "East DC Production Cluster"
  clusterManager:
    name: sveltos
status:
  version:
    kubernetes: 1.28.0
  properties:
    - name: location
      value: east-datacenter
    - name: environment
      value: production
  conditions:
    - type: ControlPlaneHealthy
      status: "True"
      lastTransitionTime: "2024-01-16T10:00:00Z"
    - type: Joined
      status: "True"
      lastTransitionTime: "2024-01-16T09:55:00Z"

2. For cluster access:
   While the KEP describes multiple approaches, I recommend starting with the simplest implementation using secrets (Push Model via Credentials in Secret). While not the most sophisticated approach, it's the most straightforward to implement and understand and usability:

apiVersion: v1
kind: Secret
metadata:
  name: cluster-east-credentials
  namespace: sveltos-system
  labels:
    x-k8s.io/cluster-inventory-consumer: sveltos
    x-k8s.io/cluster-profile: onprem-cluster-east
type: Opaque
data:
  config: <base64-encoded kubeconfig>

For Sveltos integration, the initial implementation would:

1. Implement the ClusterProfile API as a cluster manager
2. Create ClusterProfile objects for each managed cluster
3. Store cluster access credentials in secrets following the prescribed format
4. Update the ClusterProfile status to reflect current cluster state

This approach allows you to:

• Get started quickly with a simple, well-understood mechanism
• Provide a standardized cluster inventory
• Enable other tools to discover and work with Sveltos-managed clusters
• Leave room for future improvements (such as moving to identity federation when the standards are more established)

[kahirokunn]

After addressing some SIG-Multicluster content, it would be good to add Sveltos to the implementation status page at:

https://multicluster.sigs.k8s.io/guides/#implementation-status

[gianlucam76]

Thanks @kahirokunn

Sveltos though does not manage the creation of clusters. Once you have a Cluster, it can be registered with Sveltos and Sveltos can deploy add-ons and applications. In other words, if you have a cluster you created on your own (either on prem or somewhere else GKE, EKS, Civo, ...), you can register with Sveltos and from that point on Sveltos can deploy applications and add-ons there.

So instead of Sveltos creating a multicluster.x-k8s.io/v1alpha1 ClusterProfile, what if:

1. user creates ClusterProfile;
2. we make Sveltos aware of multicluster.x-k8s.io/v1alpha1 ClusterProfile => as soon as user creates one Sveltos understands there is a new Cluster where add-ons and applications can be deployed

This should be easy to achieve. Using Sveltos event framework, whenever a ClusterProfile is created/deleted a SveltosCluster can be created/deleted. User manages the ClusterProfile and does not care about SveltosClusters.

Doing so, Sveltos can be used immediately to deploy add-ons and applications on those clusters.

The reason I am suggesting this is because again Sveltos is not in the business of creating/deleting clusters. Sveltos comes once a cluster is been created.

Of course we could also do the opposite, so for every SveltosCluster create a ClusterProfile. But then Sveltos, except version, would not know how to set the other properties.

Let me know what you think.

[kahirokunn]

Thank you for the detailed explanation. I completely agree with your approach regarding Sveltos' role in cluster management. Your suggestion to have Sveltos recognize multicluster.x-k8s.io/v1alpha1 ClusterProfile and create corresponding SveltosClusters makes perfect sense, as it maintains a clear separation of concerns while enabling Sveltos to focus on its core strength of managing add-ons and applications.
By the way, I recently had the opportunity to try out the Sveltos dashboard, and I was really impressed with its functionality and user experience. While I understand this is separate from the operator discussion, I thought it might be worth mentioning that having the ability to register clusters through the dashboard (when proper permissions are in place) could be a nice quality-of-life enhancement for users. Of course, this is just a UI consideration and completely independent from the core architectural decisions we're discussing.
Thanks again for sharing your thoughts on this!

[kahirokunn]

In addition, is there any plan to provide a plugin for Backstage, the recently popular Idp on Dashboard?

[gianlucam76]

Thanks. Will prepare something tomorrow for ClusterProfile => SveltosCluster and share.

Thanks for the feedback on UI. Registering clusters via UI is something I would love to do. While I maintain all the backend for Sveltos, I am not a frontend engineer though. So I am looking for frontend engineers who want to contribute.

Regarding Backstage integration, would you mind filing an enhancement request here so we don't loose track of it? Thank you

[kahirokunn]

Looking forward to it.

Ok, I posted about Backstage below.
https://github.com/orgs/projectsveltos/discussions/958

[gianlucam76]

@kahirokunn here is the necessary configuration. If you just post this (and label the SveltosCluster in the mgmt namespace with env: management), a SveltosCluster will get created for every clusterprofiles.multicluster.x-k8s.io (of course updating/deleting clusterprofiles.multicluster.x-k8s.io corresponding SveltosCluster gets also updated/deleted)

# Detect all ClusterProfile resources
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
  name: detect-multicluster-resources
spec:
  collectResources: true
  resourceSelectors:
  - group: "multicluster.x-k8s.io"
    version: "v1alpha1"
    kind: "ClusterProfile"
---
# For every ClusterProfile resource, creates a SveltosCluster
# All ClusterProfile labels are copied
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: register-cluster
spec:
  sourceClusterSelector:
    matchLabels:
      env: management
  eventSourceName: detect-multicluster-resources
  oneForEvent: true
  policyRefs:
  - name: sveltoscluster-metadata
    namespace: default
    kind: ConfigMap
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sveltoscluster-metadata
  namespace: default
  annotations:
    projectsveltos.io/instantiate: ok
data:
  sveltos-cluster.yaml: |
    apiVersion: lib.projectsveltos.io/v1beta1
    kind: SveltosCluster
    metadata:
      name: {{ .Resource.metadata.name }}
      namespace: {{ .Resource.metadata.namespace }}
      labels:
        {{ range $key, $value := .Resource.metadata.labels }}
          {{ $key }}: {{ $value }}
        {{end}}
---
# Detect all Secrets with labels x-k8s.io/cluster-profile
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
  name: detect-config-secret
spec:
  collectResources: true
  resourceSelectors:
  - group: ""
    version: "v1"
    kind: "Secret"
    evaluate: |
      function evaluate()
        hs = {}
        hs.matching = false
        if obj.metadata.labels ~= nil then
          if obj.metadata.labels["x-k8s.io/cluster-profile"] ~= nil then
            hs.matching = true
          end
        end
        return hs
      end
---
# For every Secret containing Kubeconfig for a ClusterProfile,
# corresponding SveltosCluster's Secret is created
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: update-sveltoscluster
spec:
  sourceClusterSelector:
    matchLabels:
      env: management
  eventSourceName: detect-config-secret
  oneForEvent: true
  policyRefs:
  - name: sveltoscluster-spec
    namespace: default
    kind: ConfigMap
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sveltoscluster-spec
  namespace: default
  annotations:
    projectsveltos.io/instantiate: ok
data: 
  sveltos-cluster.yaml: |
    {{ $value := (index .Resource.metadata.labels `x-k8s.io/cluster-profile`) }}
    apiVersion: v1
    kind: Secret
    metadata:
      name: {{ $value }}-sveltos-kubeconfig
      namespace: {{ .Resource.metadata.namespace }}
    data:
        {{ range $key, $value := .Resource.data }}
          {{ $key }}: {{ $value }}
        {{end}}

Assuming Sveltos is deployed in the management cluster and the SveltosCluter mgmt in the mgmt namespace has label env: management, then

1. Anytime a multicluster.x-k8s.io/v1alpha1 ClusterProfile is created, Sveltos Event Framework creates a SveltosCluster. SveltosCluster namespace/name and labels are same as ClusterProfile
2. Anytime a Secret containing a CLusterProfile Kubeconfig is created, Sveltos Event Framework creates the corresponding Secret needed for the SveltosCluster

In my test I created a ClusterProfile and a Secret and automatically SveltosCluster was created and Sveltos started managing add-ons and applications in such a cluster.

➜ addon-controller git:(deps) kubectl get clusterprofiles.multicluster.x-k8s.io -A NAMESPACE NAME AGE production-clusters onprem-cluster-east 20m ➜ addon-controller git:(deps) kubectl get sveltoscluster -n production-clusters NAME READY VERSION onprem-cluster-east true v1.28.7+k3s1

kubectl get secret -n production-clusters NAME TYPE DATA AGE cluster-east-credentials Opaque 1 13m onprem-cluster-east-sveltos-kubeconfig Opaque 1 7m54s

Please let me know if this is sufficient. No extra pods are necessary.

[kahirokunn]

Thank you, I will try to get the code to work for me! That code is really great!
I found a point where it could be done better, if you'd like to take a look.

ClusterProfile is used in conjunction with the ClusterProperty custom resource.

https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/2149-clusterid#implementing-the-clusterproperty-crd-and-its-admission-controllers

Its contents are written in .status.properties.

https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/4322-cluster-inventory#api-example

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
 name: generated-cluster-name
 labels:
   x-k8s.io/cluster-manager: some-cluster-manager
spec:
  displayName: cluster-us-east
  clusterManager:
    name: some-cluster-manager
status:
  version:
    kubernetes: 1.28.0
  properties:
   - name: clusterset.k8s.io
     value: some-clusterset
   - name: location
     value: apac
  conditions:
   - type: ControlPlaneHealthy
     status: True
     lastTransitionTime: "2023-05-08T07:56:55Z"
     message: ""
   - type: Joined
     status: True
     lastTransitionTime: "2023-05-08T07:58:55Z"
     message: ""

I thought it would be better if I could manipulate the SveltosCluster spec by referencing the contents of .status.version.kubernetes and .status.properties .

[gianlucam76]

Thanks @kahirokunn

Sveltos gets Kubernetes version by querying directly the APIServer of the managed cluster. And mark it ready once it can connect to it.

So I think we can ignore this (unless I misunderstood your point)

[bsctl]

Found this code not working properly when .Resource.metadata.labels are missing or null

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sveltoscluster-metadata
  namespace: default
  annotations:
    projectsveltos.io/instantiate: ok
data:
  sveltos-cluster.yaml: |
    apiVersion: lib.projectsveltos.io/v1beta1
    kind: SveltosCluster
    metadata:
      name: {{ .Resource.metadata.name }}
      namespace: {{ .Resource.metadata.namespace }}
      labels:
        {{ range $key, $value := .Resource.metadata.labels }}
          {{ $key }}: {{ $value }}
        {{end}}

Trying this (it seems almost correct to me) but still not working in case of .Resource.metadata.labels not set.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sveltoscluster-metadata
  namespace: default
  annotations:
    projectsveltos.io/instantiate: ok
data:
  sveltos-cluster.yaml: |
    apiVersion: lib.projectsveltos.io/v1beta1
    kind: SveltosCluster
    metadata:
      name: {{ .Resource.metadata.name }}
      namespace: {{ .Resource.metadata.namespace }}
      {{ if .Resource.metadata.labels }}
      labels:
        {{ range $key, $value := .Resource.metadata.labels }}
        {{ $key }}: {{ $value }}
        {{ end }}
      {{ end }}

Any suggestion for improvement?

[kahirokunn]

Thanks @kahirokunn

Sveltos gets Kubernetes version by querying directly the APIServer of the managed cluster. And mark it ready once it can connect to it.

So I think we can ignore this (unless I misunderstood your point)

Oh my gosh, that's great!
Then what about items other than the version?
I'm sure there are all sorts of great features like pause, maintenance window, etc.

[gianlucam76]

Thanks @bsctl good catch. This works even if there are no labels

apiVersion: v1
kind: ConfigMap
metadata:
  name: sveltoscluster-metadata
  namespace: default
  annotations:
    projectsveltos.io/instantiate: ok
data:
  sveltos-cluster.yaml: |
    apiVersion: lib.projectsveltos.io/v1beta1
    kind: SveltosCluster
    metadata:
      name: {{ .Resource.metadata.name }}
      namespace: {{ .Resource.metadata.namespace }}
      {{- if (index .Resource.metadata `labels`) -}}
      labels:
        {{ range $key, $value := .Resource.metadata.labels }}
        {{ $key }}: {{ $value }}
        {{ end }}
      {{- end -}}

[gianlucam76]

Thanks @kahirokunn
Sveltos gets Kubernetes version by querying directly the APIServer of the managed cluster. And mark it ready once it can connect to it.
So I think we can ignore this (unless I misunderstood your point)

Oh my gosh, that's great! Then what about items other than the version? I'm sure there are all sorts of great features like pause, maintenance window, etc.

Hi @kahirokunn yes SveltosCluster can be paused (so no updates are sent to cluster). Just set the Spec.Paused field.

        // +optional
        Paused bool `json:"paused,omitempty"`

and you can also set an active window when updates can be sent

[kahirokunn]

@gianlucam76

When I try testing the if statement by giving a resource that doesn't have the key “labels”, I get an error saying “map has no entry for key ‘labels’”. Do you know anything about this? 👀

#430 (comment)

event-manager-86889b7b59-7g9bk manager I0203 08:07:43.927814       1 eventreport_collection.go:397] "failed to update ClusterProfile for EventTrigger hogehoge: template: mgmt-mgmt-hogehoge:1:168: executing \"mgmt-mgmt-hogehoge\" at <.Resource.metadata.labels>: map has no entry for key \"labels\"" cluster="mgmt/mgmt"

[kahirokunn]

I got the same error with both “with” and “if”, so it didn't work.

[kahirokunn]

I tried the following code, but the Secret gets written into the ConfigMap in projectsveltos's namespace:

#430 (comment)

When a Secret is written into a ConfigMap, the Secret's contents get output to the Kubernetes audit log. Additionally, from an RBAC perspective, granting ConfigMap permissions effectively means granting Secret permissions as well, which poses a high risk.

Therefore, I tried the following verification, but received an error saying "failed to update ClusterProfile for EventTrigger transform-tls-secret: unsupported secret type":

---
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
  name: detect-tls-secret
spec:
  collectResources: true
  resourceSelectors:
    - group: ""
      version: "v1"
      kind: "Secret"
      namespace: mgmt
---
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: transform-tls-secret
spec:
  sourceClusterSelector:
    matchLabels:
      role: mgmt
  destinationClusterSelector:
    matchLabels:
      role: worker
  eventSourceName: detect-tls-secret
  syncMode: ContinuousWithDriftDetection
  policyRefs:
    - name: transform-tls-secret
      namespace: mgmt
      kind: Secret
---
apiVersion: v1
kind: Secret
metadata:
  name: transform-tls-secret
  namespace: mgmt
  annotations:
    projectsveltos.io/instantiate: ok
type: Opaque
stringData:
  tls-secret.yaml: |
    apiVersion: v1
    kind: Secret
    metadata:
      name: {{ .Resource.metadata.name }}
      namespace: {{ .Resource.metadata.namespace }}
    type: {{ .Resource.type }}
    data:
      {{ range $key, $value := .Resource.data }}
        {{ $key }}: {{ $value }}
      {{end}}

If we must choose between ConfigMap and Secret, using Secret is generally better from both audit logging and RBAC perspectives.

[gianlucam76]

@kahirokunn I tried again (with version v0.46.0, last time I tried with version v0.45.0) this and it worked for me

This worked both when matching Resource has no label, has one label and has more than one label

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    projectsveltos.io/instantiate: ok
  name: sveltoscluster-metadata
  namespace: default
data:
  sveltosc-cluster.yaml: |
    apiVersion: lib.projectsveltos.io/v1beta1
    kind: SveltosCluster
    metadata:
      name: {{ .Resource.metadata.name }}
      namespace: {{ .Resource.metadata.namespace }}
      {{- if (index .Resource.metadata `labels`) -}}
      labels:
        {{ range $key, $value := .Resource.metadata.labels }}
        {{ $key }}: {{ $value }}
        {{ end }}
      {{- end -}}

which version of Sveltos are you running?

[gianlucam76]

@kahirokunn if you reference a ConfigMap, then a ConfigMap is created after instantiation.

If you reference a Secret (as in your example) a Secret is created after instantiation.

The problem with your example is that Secret must be of type type=addons.projectsveltos.io/cluster-profile documentation

In other words, Secret referenced by Sveltos custom resources, must be of type=addons.projectsveltos.io/cluster-profile

[kahirokunn]

I use the main tag.
This is because I am verifying it with make quickstart.

[kahirokunn]

{{- if (index .Resource.metadata `labels`) -}}

This format worked well!
If you use it like helm

{{- if .Resource.metadata.labels -}},

it doesn't work. (This is used as a matter of course in helm, so it may be a pitfall.

[kahirokunn]

#430 (comment)
I have already read the documentation and actually tried the following format:

---
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
  name: detect-tls-secret
spec:
  collectResources: true
  resourceSelectors:
    - group: ""
      version: "v1"
      kind: "Secret"
      namespace: mgmt
---
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: transform-tls-secret
spec:
  sourceClusterSelector:
    matchLabels:
      role: mgmt
  destinationClusterSelector:
    matchLabels:
      role: worker
  eventSourceName: detect-tls-secret
  syncMode: ContinuousWithDriftDetection
  policyRefs:
    - name: transform-tls-secret
      namespace: mgmt
      kind: Secret
---
apiVersion: v1
kind: Secret
metadata:
  name: transform-tls-secret
  namespace: mgmt
  annotations:
    projectsveltos.io/instantiate: ok
type: addons.projectsveltos.io/cluster-profile
stringData:
  tls-secret.yaml: |
    apiVersion: v1
    kind: Secret
    metadata:
      name: {{ .Resource.metadata.name }}
      namespace: {{ .Resource.metadata.namespace }}
    type: {{ .Resource.type }}
    data:
      {{ range $key, $value := .Resource.data }}
        {{ $key }}: {{ $value }}
      {{end}}

However, I get an error like executing "mgmt-mgmt-transform-tls-secret" at <.Resource.metadata.name>: can't evaluate field Resource:

event-manager-86889b7b59-bqkv9 manager I0203 10:11:53.885312       1 eventtrigger_deployer.go:1440] "failed to execute content: template: mgmt-mgmt-transform-tls-secret:1:81: executing \"mgmt-mgmt-transform-tls-secret\" at <.Resource.metadata.name>: can't evaluate field Resource in type *controllers.currentObjects" cluster="mgmt/mgmt" referencedResource="&TypeMeta{Kind:Secret,APIVersion:v1,}:mgmt/transform-tls-secret"
event-manager-86889b7b59-bqkv9 manager I0203 10:11:53.885339       1 eventtrigger_deployer.go:1602] "failed to instantiated referenced resource content: template: mgmt-mgmt-transform-tls-secret:1:81: executing \"mgmt-mgmt-transform-tls-secret\" at <.Resource.metadata.name>: can't evaluate field Resource in type *controllers.currentObjects" cluster="mgmt/mgmt" referencedResource="&TypeMeta{Kind:Secret,APIVersion:v1,}:mgmt/transform-tls-secret"
event-manager-86889b7b59-bqkv9 manager I0203 10:11:53.885347       1 eventtrigger_deployer.go:910] "failed to create one clusterProfile instance per matching resource: template: mgmt-mgmt-transform-tls-secret:1:81: executing \"mgmt-mgmt-transform-tls-secret\" at <.Resource.metadata.name>: can't evaluate field Resource in type *controllers.currentObjects" cluster="mgmt/mgmt"
event-manager-86889b7b59-bqkv9 manager I0203 10:11:53.885352       1 eventreport_collection.go:397] "failed to update ClusterProfile for EventTrigger transform-tls-secret: template: mgmt-mgmt-transform-tls-secret:1:81: executing \"mgmt-mgmt-transform-tls-secret\" at <.Resource.metadata.name>: can't evaluate field Resource in type *controllers.currentObjects" cluster="mgmt/mgmt"

[kahirokunn]

I think this is probably an implementation that has been omitted (because, although I searched through all the source code, I couldn't find any examples of goTemplate or lua being written in Secret).

As an implementation, I think it would be enough to just paste the processing done in ConfigMap into Secret as well.

[kahirokunn]

If Secret works, I believe that ClusterProfile support will be production-grade.

#430 (comment)

[gianlucam76]

Thanks @kahirokunn I believe your problem is your are not setting OneForEvent: true in the EventTrigger.
When you set it, Sveltos will create one ClusterProfile for event (and Resource is available). When it is set to false, Sveltos will create one ClusterProfile for all events (and Resources is available, which is essentially a slice of each event).

So in your configuration just add this oneForEvent: true

apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: transform-tls-secret
spec:
  oneForEvent: true # add this 
  sourceClusterSelector:
    matchLabels:
      role: mgmt
  destinationClusterSelector:
    matchLabels:
      role: worker
  eventSourceName: detect-tls-secret
  syncMode: ContinuousWithDriftDetection
  policyRefs:
    - name: transform-tls-secret
      namespace: mgmt
      kind: Secret

I tested it and it works

[kahirokunn]

Indeed it worked.
Thank you very much!