Support additional OIDC configuration with shoot-oidc-service extension

[pbochynski]

Description
Enable the option to trust the additional identity provider compliant with OpenID. The provider can be registered in the Kyma cluster and kubernetes API server will authenticate tokens that match the provider issuer.
The complete solution should allow to establish the trust during the provisioning so the cluster can be accessed from fully automated processes (without user presence). To accomplish that the following changes are required:

• enable shoot-oidc-service extension for shoot clusters
• add possibility to pass the OpenIDConnect resource in the shoot cluster as Kyma instance parameter (KEB)
• grant permissions to accounts authenticated by new provider by some role and role binding (or use the existing feature of extending administrator list)

Acceptance Criteria

• Ensure Isolation between user facing and operator-facing OIDC configs. User should not be able to override Opereator facing OIDC
  • Configure Operator facing OIDC
  • Extend Kyma Provisioning Parameters to allow multiple OIDC definitions kyma-environment-broker#423

Reasons
Many users request a possibility to deploy software to freshly creaated Kyma clusters in automated way. Changing the default IDP for the cluster is the only solution available for now, but then IDP has to support both human users and service accounts what is usually challenging. With additional OIDC provider it can be used only for system to system authorization and will be much easier to set up.

Links

• Provide tooling for automated Kyma lifecycle and subscription management #18198
• https://github.com/gardener/oidc-webhook-authenticator/blob/master/README.md
• https://blogs.sap.com/2022/09/23/using-github-actions-openid-connect-in-kubernetes/
• https://github.com/gardener/gardener-extension-shoot-oidc-service/blob/master/docs/usage/openidconnects.md

[maximilianbraun]

Would love to test it at earliest convinience.

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

[kwiatekus]

The goal would be to allow kyma users to deploy apps (and run tests ) on the provisioned kyma runtimes in the CI jobs .

Execution can be divided into several stages:

• [POC] Authenticate towards gardener's shoot cluster using shoot-oidc-service extension and GH token  #18519
• Handle shoot-oidc-service extension when provisioning kyma [KIM/feature] infrastructure-manager#381 - Q2 2024 (see comment)
• Extend Kyma Provisioning Parameters to allow multiple OIDC definitions kyma-environment-broker#423

[kwiatekus]

OIDC configuration will be included in the gardener provisioning in q2
kyma-project/infrastructure-manager#381

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[kwiatekus]

Kyma Infrastructure Manager (KIM) should apply Operator-Facing OIDC separately from User-facing OIDC(s)

In order to allow this we need to adjust the interfaces:

1. User -> Kyma Environment Broker (KEB)

• User should be able to define multiple OIDCs
• User should be able to define requiedClaims (key-value pairs)

2. KEB -> KIM

• If user provides OIDC(s) KEB populates them as additionalOidcConfigs
• If user provides no OIDC config KIM defaults the additionalOidcConfigs[0] with the default User-facing OIDC config
• KIM defaults oidcConfig with Operator-facing OIDC

3. KIM -> Gardener

• KIM enables shoot-oidc-service extension in shoot CR
• KIM defines Operator-facing OIDC explicitely in shoot CR

4. KIM -> Kyma

• KIM reconciles authentication.gardener.cloud/v1alpha1/OpenIDConnect objects in kyma based on additionalOidcConfigs
• KIM reconciles the clusterrolebindings based on administrators it got from KEB
• KIM ensures clusterrole binding for operators

to avoid unwanted impersonations, we should:

• ensure prefixes in Operator-facing OIDC (for username and groups)
• document recommendation about prefixes in user facing documentation

[Disper]

Regarding migration from existing shoots to Runtime CRs.

• oidcConfig <- Populate withoidcConfig
• additionalOidcConfigs <- Populate as the first entry withoidcConfig

[burkardendres]

Would be great to have this available soon, as a dependency we want to use requires that and also to bring our kya into production it would be kind of a prerequisite.

[tobiscr]

To unblock customers who are waiting for the enablement of the OIDC extension in GArdener, we updated the Provisioner last week to activate this extension per default for all new created clusters. So, any new created SKR cluster will have the OIDC extension now enabled and customers can configure their own OIDC provider by creating the corresponding CR in their SKR clusters.

It is planned to start the replacement of the Provisioner with KIM (Kyma Infrasturcture Manager) by end of this month. KIM will also per default enable the OIDC extension for all managed clusters.

[s-u-b-h-a-k-a-r]

it would be very helpful to have this available soon, as a dependency we want to use requires that and also to bring our kyma into production it would be kind of a prerequisite and this will help us to manage BTP Kyma Workload via Cloud Orchestrator/Crossplane without the need for manual task

[vrenjith]

It will be super helpful if we have this capability so that we can bootstrap ArgoCD in the provisioned Kyma cluster so that ArgoCD to take over the rest of the deployments that are coming in from the service pipelines.

[ptesny]

It will be super helpful if we have this capability so that we can bootstrap ArgoCD in the provisioned Kyma cluster so that ArgoCD to take over the rest of the deployments that are coming in from the service pipelines.

That's sth which is already done via a terraform based automation module: https://github.com/quovadis-btp/btp-automation/tree/main/btp-context/runtime-context
At the end of kyma env provisioning an existing ArgoCD instance is being bootstrapped with the newly created kyma cluster. I also create a dedicated ArgoCD project on-the-fly

I'm trying hard to finalize a public root module which will then allow you to use the above child module. Will keep you posted. Thx

#18666

[abhijith-darshan]

This is great news that shoot-oidc-service extension is available in all newly created kyma clusters. What is the timeline to roll this out to already existing SAP Kyma clusters?

[tobiscr]

Sorry for the late reply @abhijith-darshan. We were facing and fixing several unexpected challenges in the new Infrastructure Manager during the last weeks which were again causing delay in our rollout.

Unfortunaltey, we were not able to migrate all productive clusters to the new Kyma Infrastructure Manager yet which is a pre-requisite for releasing the OIDC feature. We are currently finalising the testing phase of KIM and plan to migrate all productive clusters beginning of next year.

Our SRE team will, after KIM was released, migrate all existing clusters to support the new OIDC configuration. @ebensom can provide more details when this migration will happen.

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[PK85]

24.01.2025 - Meeting with Benjamin about OIDC migration and release on production (exposed by KEB):

Execution Plan. Repeat per DEV, STAGE and PROD

0.1 ) Backup TASK_SRE/FROGS
0.2 ) Right now KEB uses only mainOIDC in the RuntimeCR.
1 ) KEB must add two entries(during create and update operations) into RuntimeCR, main and additional OIDC, both the same one entry TASK_GOPHERS
2 ) Run script to copy mainOIDC into additionalOIDC. Script contains sanity check for all RuntimeCR after first part. TASK_SRE
3 ) KIM uses operator-OIDC as default (requires config update in KIM!) for new SKRs TASK_SRE
4 ) KEB use only additionalOIDC TASK_GOPHERS
5 ) SRE run migration which means: TASK_SRE
5.1 ) remove mainOIDC and
5.2 ) add kyma operator OIDC into main OIDC
6 ) SRE remove backdoor solution

When dev, stage, prod done then:

1 ) Gophers: AT this point KEB can expose additionalOIDC parameter for users to handle list of oidcs. TASK_GOPHERS
2 ) Busola improvements with selecting user-context from kubeconfig TASK_BUSOLA
3 ) Busola after user login: "OIDCconfiguration" resource could be displayed differenetly when label from KIM is added TASK_BUSOLA

Questions:

1. Who creates a script copying customer OIDC from main OIDC into additionalOIDC? We need that before doing next steps @tobiscr @ebensom
2. Does RuntimeCR support empty mainOIDC? @tobiscr

[tobiscr]

Thanks for your request, here our feedback:

1. We can support here, but we were actually thinking that the migration from oidcConfig to additionalOIDC will become part of the rollout of the Kyma Operator OIDC configuration (the one for our on-call guys) managed by SRE. Please let us know if you would need support from us.
2. Right now, if the field oidcConfig is not provided, KIM will use the default values configured in his configuration (see https://github.com/kyma-project/infrastructure-manager/blob/d8458fca36ee3bf06034f1f531f6c839c348cdef/pkg/gardener/shoot/extender/oidc.go#L13-L34). But I assume this behaviour is not colliding with your goals: just configure the operator-OIDC in KIM and KEB won't have to provide this field anymore after we are done with the migration.

JFYI - @Disper ⏫

[pbochynski]

Please consider if we need the OIDC provider for operator access enabled all the time. Some certification programs require that customers grant operator access. Therefore we should consider having the operator OIDC provider configured on demand or at least role binding should be created on demand by the customer that wants the operator to troubleshoot the cluster.

[tobiscr]

@ebensom / @PK85 : as @pbochynski mentioned, I fear we have to revisit our decision how we deal with operator access to clusters. Our last agreement to have always access would, at least for PCI relevant runtimes, introduce several disadvantages. the PCI auditor recommended that our ops-team should not be able to access any machines. The customers should grant us access when needed (avoids that our engineers would be able to access CHD related data).

@pbochynski - do you think we should also revisit our general approach how we manage PCI relevant systems in KLM/KIM? Maybe we should for such systems disable the reconciliation and customers have to enable it actively to receive latest updates.

[abhijith-darshan]

Since Kubernetes 1.29 structured authentication gate is enabled by default and Gardener will disable OIDC config from >=1.32 maybe OpenIDConnect can be optional setting by default.

It is anyway recommended by Gardener to use structured authentication if the need is less than 64 issuers. Plus there is no need of APIServer restart (but Gardener needs shoot reconciliation for the updates to be propagated)

Any plans of allowing multiple issuers for Kyma in BTP cockpit?

[tobiscr]

Hi @abhijith-darshan , thanks for your request! We are going to support multiple OIDC providers soon. The implementation is mostly done, but we have still several migration tasks ahead and our SRE team is working on them.

It's also planned to add support for multiple OIDC providers to the Kyma Dashboard.

This feature will still use Gardeners OIDC extension. We move internally to structured authentication, but this is approach is at the moment not used for OIDC configurations provided by customers.