Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use hive clusterdeployment for creating spoke clusters #472

Merged
merged 6 commits into from
May 30, 2024
Merged

feat: use hive clusterdeployment for creating spoke clusters #472

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f2215f7
feat: add support for hive clusterdeployments creating spokes
TomerFi Apr 16, 2024
f9bf1f7
test: regenerated tests after clusterdeployment commit
TomerFi Apr 16, 2024
5b4e903
test: updated test-cased and regeneated expectations
TomerFi Apr 17, 2024
a073b8c
chore: added annotations controling gitops and fail for missing meta …
TomerFi Apr 23, 2024
c6ffd0e
chore: removed managedclusterset spec
TomerFi Apr 26, 2024
6cd4e85
fix: when using clusterdeployments, secrets should exist in the clust…
TomerFi May 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions acm/templates/provision/clusterdeployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
{{- range .Values.clusterGroup.managedClusterGroups }}
{{- $group := . }}

{{- range $group.clusterDeployments}}
{{ $cluster := . }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing that we could add here (because I am an idiot an I got bitten by it) is the following:

--- a/common/acm/templates/provision/clusterdeployment.yaml
+++ b/common/acm/templates/provision/clusterdeployment.yaml
@@ -3,6 +3,12 @@

 {{- range $group.clusterDeployments}}
 {{ $cluster := . }}
+{{- if (eq $cluster.name nil) }}
+{{- fail (printf "managedClusterGroup clusterDeployment cluster name is empty: %s" $cluster) }}
+{{- end }}
+{{- if (eq $group.name nil) }}
+{{- fail (printf "managedClusterGroup clusterDeployment group name is empty: %s" $cluster) }}
+{{- end }}
 {{- $deploymentName := print $cluster.name "-" $group.name }}

 {{- $cloud := "None" }}

Reason was that I did not add the name attribute under the group and when running make preview-acm I saw a bunch of <nil> in the object names and it surprised me a bit ;)

We can also do it on top later, just writing it here so I don't forget

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohh good catch! we'll add it here.

Copy link
Member Author

@TomerFi TomerFi Apr 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in a073b8c.

Should we do the same for clusterpool.yaml?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah let's do later it in another PR maybe

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#524


{{- if (eq $cluster.name nil) }}
{{- fail (printf "managedClusterGroup clusterDeployment cluster name is empty: %s" $cluster) }}
{{- end }}
{{- if (eq $group.name nil) }}
{{- fail (printf "managedClusterGroup clusterDeployment group name is empty: %s" $cluster) }}
{{- end }}

{{- $deploymentName := print $cluster.name "-" $group.name }}

{{- $cloud := "None" }}
{{- $region := "None" }}

{{- if $cluster.platform.aws }}
{{- $cloud = "aws" }}
{{- $region = $cluster.platform.aws.region }}
{{- else if $cluster.platform.azure }}
{{- $cloud = "azure" }}
{{- $region = $cluster.platform.azure.region }}
{{- end }}

---
apiVersion: v1
kind: Namespace
metadata:
name: {{ $deploymentName }}

---
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
name: {{ $deploymentName }}
namespace: {{ $deploymentName }}
labels:
vendor: OpenShift
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add the following here:

annotations:
  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

because if you do a deployment from scratch (i.e. one where ACM gets installed at the same time), the ACM app might be stuck because the multiclusterhub (which provides these CRDs) has not been installed yet and Argo will be stuck with:

The Kubernetes API could not find hive.openshift.io/ClusterDeployment for requested resource first-spoke-deployments/first-spoke-deployments. Make sure the "ClusterDeployment" CRD is installed on the destination cluster.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to know - thanks!

Copy link
Member Author

@TomerFi TomerFi Apr 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in a073b8c.

annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
baseDomain: {{ $cluster.baseDomain }}
clusterName: {{ $deploymentName }}
installAttemptsLimit: 1
platform:
{{ $cloud }}:
credentialsSecretRef:
name: {{ $deploymentName }}-creds
region: {{ $region }}
provisioning:
installConfigSecretRef:
name: {{ $deploymentName }}-install-config
sshPrivateKeySecretRef:
name: {{ $deploymentName }}-ssh-private-key
imageSetRef:
name: img{{ $cluster.openshiftVersion }}-multi-appsub
pullSecretRef:
name: {{ $deploymentName }}-pull-secret

---
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above with ClusterDeployment. We should probably add the annotation:

annotations:
  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

otherwise during an install from scratch we get:

The Kubernetes API could not find cluster.open-cluster-management.io/ManagedCluster for requested resource open-cluster-management/first-spoke-deployments. Make sure the "ManagedCluster" CRD is installed on the
 destination cluster.

Copy link
Member Author

@TomerFi TomerFi Apr 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in a073b8c.

labels:
cluster.open-cluster-management.io/clusterset: {{ $group.name }}
{{- if (not $group.acmlabels) }}
clusterGroup: {{ $group.name }}
{{- else if eq (len $group.acmlabels) 0 }}
clusterGroup: {{ $group.name }}
{{- else }}
{{- range $group.acmlabels }}
{{ .name }}: {{ .value }}
{{- end }}
{{- end }}
name: {{ $deploymentName }}
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
hubAcceptsClient: true
{{- end }}{{- /* range $group.clusterDeployments */}}
{{- end }}{{- /* range .Values.clusterGroup.managedClusterGroups */}}
15 changes: 1 addition & 14 deletions acm/templates/provision/clusterpool.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,5 @@
{{- range .Values.clusterGroup.managedClusterGroups }}
{{- $group := . }}
{{- if .clusterPools }}{{- /* We only create ManagedClusterSets if there are clusterPools defined */}}
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: ManagedClusterSet
metadata:
annotations:
cluster.open-cluster-management.io/submariner-broker-ns: {{ .name }}-broker
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
name: {{ .name }}
spec:
clusterSelector:
selectorType: LegacyClusterSetLabel
---
{{- range .clusterPools }}

{{- $pool := . }}
Expand Down Expand Up @@ -54,7 +42,7 @@ spec:
runningCount: {{ $numClusters }}
baseDomain: {{ .baseDomain }}
installConfigSecretTemplateRef:
name: {{ $poolName }}-install-config
name: {{ $poolName }}-install-config
imageSetRef:
name: img{{ .openshiftVersion }}-multi-appsub
pullSecretRef:
Expand Down Expand Up @@ -91,5 +79,4 @@ spec:
---
{{- end }}{{- /* range .range clusters */}}
{{- end }}{{- /* range .clusterPools */}}
{{- end }}{{- /* if .clusterPools) */}}
{{- end }}{{- /* range .Values.clusterGroup.managedClusterGroups */}}
13 changes: 13 additions & 0 deletions acm/templates/provision/managedclusterset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{{- range .Values.clusterGroup.managedClusterGroups }}
{{- if or .clusterPools .clusterDeployments }}{{- /* We only create ManagedClusterSets if there are clusterPools or clusterDeployments defined */}}
---
apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
annotations:
cluster.open-cluster-management.io/submariner-broker-ns: {{ .name }}-broker
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
name: {{ .name }}

{{- end }}{{- /* if .clusterPools) */}}
{{- end }}{{- /* range .Values.clusterGroup.managedClusterGroups */}}
71 changes: 49 additions & 22 deletions acm/templates/provision/secrets-aws.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,58 +3,88 @@
{{- range .clusterPools }}
{{- $poolName := print .name "-" $group.name }}
{{- if .platform.aws }}
---
{{- template "externalsecret.aws.creds" (dict "name" $poolName "context" . "secretStore" $.Values.secretStore) }}
---
{{- template "externalsecret.aws.infra-creds" (dict "name" $poolName "context" . "secretStore" $.Values.secretStore) }}

{{- end }}{{- /* if .platform.aws */}}
{{- end }}{{- /* range .clusterPools */}}

{{- range .clusterDeployments }}
{{- $deploymentName := print .name "-" $group.name }}
{{- if .platform.aws }}
---
{{- template "externalsecret.aws.creds" (dict "name" $deploymentName "context" . "secretStore" $.Values.secretStore "namespaced" true) }}
---
{{- template "externalsecret.aws.infra-creds" (dict "name" $deploymentName "context" . "secretStore" $.Values.secretStore "namespaced" true) }}

{{- end }}{{- /* if .platform.aws */}}
{{- end }}{{- /* range .clusterDeployments */}}

{{- end }}{{- /* range .Values.clusterGroup.managedClusterGroups */}}

{{- define "externalsecret.aws.creds" }}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: {{ $poolName }}-creds
name: {{ .name }}-creds
{{- if .namespaced }}
namespace: {{ .name }}
{{- end }}
spec:
dataFrom:
- extract:
# Expects entries called: aws_access_key_id and aws_secret_access_key
key: {{ default "secret/data/hub/aws" .awsKeyPath }}
key: {{ default "secret/data/hub/aws" .context.awsKeyPath }}
refreshInterval: 24h0m0s
secretStoreRef:
name: {{ $.Values.secretStore.name }}
kind: {{ $.Values.secretStore.kind }}
name: {{ .secretStore.name }}
kind: {{ .secretStore.kind }}
target:
name: {{ $poolName }}-creds
name: {{ .name }}-creds
creationPolicy: Owner
template:
type: Opaque
---
{{- end}}

{{- define "externalsecret.aws.infra-creds"}}
# For use when manually creating clusters with ACM
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: {{ $poolName }}-infra-creds
spec:
name: {{ .name }}-infra-creds
{{- if .namespaced }}
namespace: {{ .name }}
{{- end }}
spec:
data:
- secretKey: openshiftPullSecret
remoteRef:
key: {{ default "secret/data/hub/openshiftPullSecret" .pullSecretKeyPath }}
key: {{ default "secret/data/hub/openshiftPullSecret" .context.pullSecretKeyPath }}
property: content
- secretKey: awsKeyId
remoteRef:
key: {{ default "secret/data/hub/aws" .awsKeyPath }}
key: {{ default "secret/data/hub/aws" .context.awsKeyPath }}
property: aws_access_key_id
- secretKey: awsAccessKey
remoteRef:
key: {{ default "secret/data/hub/aws" .awsKeyPath }}
key: {{ default "secret/data/hub/aws" .context.awsKeyPath }}
property: aws_secret_access_key
- secretKey: sshPublicKey
remoteRef:
key: {{ default "secret/data/hub/publickey" .sshPublicKeyPath }}
key: {{ default "secret/data/hub/publickey" .context.sshPublicKeyPath }}
property: content
- secretKey: sshPrivateKey
remoteRef:
key: {{ default "secret/data/hub/privatekey" .sshPrivateKeyPath }}
key: {{ default "secret/data/hub/privatekey" .context.sshPrivateKeyPath }}
property: content
refreshInterval: 24h0m0s
secretStoreRef:
name: {{ $.Values.secretStore.name }}
kind: {{ $.Values.secretStore.kind }}
secretStoreRef:
name: {{ .secretStore.name }}
kind: {{ .secretStore.kind }}
target:
name: {{ $poolName }}-infra-creds
name: {{ .name }}-infra-creds
creationPolicy: Owner
template:
type: Opaque
Expand All @@ -63,7 +93,7 @@ spec:
cluster.open-cluster-management.io/credentials: ""
cluster.open-cluster-management.io/type: aws
data:
baseDomain: "{{ .baseDomain }}"
baseDomain: "{{ .context.baseDomain }}"
pullSecret: |-
{{ "{{ .openshiftPullSecret | toString }}" }}
aws_access_key_id: |-
Expand All @@ -78,7 +108,4 @@ spec:
httpsProxy: ""
noProxy: ""
additionalTrustBundle: ""
---
{{- end }}
{{- end }}
{{- end }}
{{- end}}
71 changes: 50 additions & 21 deletions acm/templates/provision/secrets-azure.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,58 +3,90 @@
{{- range .clusterPools }}
{{- $poolName := print .name "-" $group.name }}
{{- if .platform.azure }}
---
{{- template "externalsecret.azure.creds" (dict "name" $poolName "context" . "secretStore" $.Values.secretStore) }}
---
{{- template "externalsecret.azure.infra-creds" (dict "name" $poolName "context" . "secretStore" $.Values.secretStore) }}

---
{{- end }}{{- /* if .platform.azure */}}
{{- end }}{{- /* range .clusterPools */}}

{{- range .clusterDeployments }}
{{- $deploymentName := print .name "-" $group.name }}
{{- if .platform.azure }}
---
{{- template "externalsecret.azure.creds" (dict "name" $deploymentName "context" . "secretStore" $.Values.secretStore "namespaced" true) }}
---
{{- template "externalsecret.azure.infra-creds" (dict "name" $deploymentName "context" . "secretStore" $.Values.secretStore "namespaced" true) }}


{{- end }}{{- /* if .platform.azure */}}
{{- end }}{{- /* range .clusterPools */}}

{{- end }}{{- /* range .Values.clusterGroup.managedClusterGroups */}}

{{- define "externalsecret.azure.creds" }}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: {{ $poolName }}-creds
name: {{ .name }}-creds
{{- if .namespaced }}
namespace: {{ .name }}
{{- end }}
spec:
data:
- secretKey: azureOsServicePrincipal
remoteRef:
key: {{ default "secret/data/hub/azureOsServicePrincipal" .azureKeyPath }}
key: {{ default "secret/data/hub/azureOsServicePrincipal" .context.azureKeyPath }}
property: content
refreshInterval: 24h0m0s
secretStoreRef:
name: {{ $.Values.secretStore.name }}
kind: {{ $.Values.secretStore.kind }}
name: {{ .secretStore.name }}
kind: {{ .secretStore.kind }}
target:
name: {{ $poolName }}-creds
name: {{ .name }}-creds
creationPolicy: Owner
template:
type: Opaque
data:
osServicePrincipal.json: |-
{{ "{{ .azureOsServicePrincipal | toString }}" }}
---
{{- end }}

{{- define "externalsecret.azure.infra-creds"}}
# For use when manually creating clusters with ACM
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: {{ $poolName }}-infra-creds
spec:
name: {{ .name }}-infra-creds
{{- if .namespaced }}
namespace: {{ .name }}
{{- end }}
spec:
data:
- secretKey: openshiftPullSecret
remoteRef:
key: {{ default "secret/data/hub/openshiftPullSecret" .pullSecretKeyPath }}
key: {{ default "secret/data/hub/openshiftPullSecret" .context.pullSecretKeyPath }}
property: content
- secretKey: sshPublicKey
remoteRef:
key: {{ default "secret/data/hub/publickey" .sshPublicKeyPath }}
key: {{ default "secret/data/hub/publickey" .context.sshPublicKeyPath }}
property: content
- secretKey: sshPrivateKey
remoteRef:
key: {{ default "secret/data/hub/privatekey" .sshPrivateKeyPath }}
key: {{ default "secret/data/hub/privatekey" .context.sshPrivateKeyPath }}
property: content
- secretKey: azureOsServicePrincipal
remoteRef:
key: {{ default "secret/data/hub/azureOsServicePrincipal" .azureKeyPath }}
key: {{ default "secret/data/hub/azureOsServicePrincipal" .context.azureKeyPath }}
property: content
refreshInterval: 24h0m0s
secretStoreRef:
name: {{ $.Values.secretStore.name }}
kind: {{ $.Values.secretStore.kind }}
secretStoreRef:
name: {{ .secretStore.name }}
kind: {{ .secretStore.kind }}
target:
name: {{ $poolName }}-infra-creds
name: {{ .name }}-infra-creds
creationPolicy: Owner
template:
type: Opaque
Expand All @@ -66,8 +98,8 @@ spec:
cloudName: AzurePublicCloud
osServicePrincipal.json: |-
{{ "{{ .azureOsServicePrincipal | toString }}" }}
baseDomain: "{{ .baseDomain }}"
baseDomainResourceGroupName: "{{ .platform.azure.baseDomainResourceGroupName | toString }}"
baseDomain: "{{ .context.baseDomain }}"
baseDomainResourceGroupName: "{{ .context.platform.azure.baseDomainResourceGroupName | toString }}"
pullSecret: |-
{{ "{{ .openshiftPullSecret | toString }}" }}
ssh-privatekey: |-
Expand All @@ -78,7 +110,4 @@ spec:
httpsProxy: ""
noProxy: ""
additionalTrustBundle: ""
---
{{- end }}
{{- end }}
{{- end }}
Loading
Loading