ResourceSet dynamic inputs from 3rd-party orchestrators

[stefanprodan]

With the introduction of ResourceSet input providers, Flux Operator can dynamically deploy/uninstall applications based on data provided by Git services e.g. open/close Pull Request or push/delete Git branch.

While this feature is akin to GitOps, it limits the decision-making to the operations performed by developers on the Git service (e.g. GitHub, GitLab) and to the data provided by these services (e.g. Pull Request ID, labels, branch name, commit SHA).

For complex scenarios when the decision-making is based on a mix of data from Git services and from other systems (e.g. testing, monitoring, ticketing), the Flux Operator could delegate the decision-making to a 3rd-party orchestrator that could provide the necessary data for the ResourceSet controller to perform the desired actions (e.g. deploy app version X to the canary production cluster when the release manager has signed off on it).

In the case of a 3rd-party orchestrator, the desired state is not fully defined in Git, instead it is dynamically generated from multiple sources:

• Git Repository - where the ResourceSet and ResourceSetInputProvider manifests are defined.
• Container Registry - where the application images, Helm charts and Flux artifacts are stored.
• Orchestrator API - where the decision-making logic is implemented and from where the Flux Operator
  fetches the input data for the ResourceSet templates.

While we could consolidate the desired state in the Container Registry by packaging the Flux manifests from Git and pushing them to the registry, the inputs for the ResourceSet templates would still need to be fetched from the orchestrator API at runtime.

To close the feedback loop, the orchestrator could expose an API endpoint where Flux will send the status of the deployments managed by a ResourceSet. This is already implemented in Flux, where events are sent to external services via notification-controller. The Flux events contain the information a 3rd-party orchestrator would need: the app version deployed, the target cluster, the deployment status, etc.

Proposal

To enable the Flux Operator to dynamically deploy applications based on data provided by 3rd-party orchestrators, the ResourceSetInputProvider should be extended with a new type ExternalService that would allow the operator to fetch arbitrary data from an external service via HTTP/S.

Example

Example of a ResourceSetInputProvider that fetches input values from an orchestrator API:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
  name: app1-feature-flags
  namespace: apps
  annotations:
    fluxcd.controlplane.io/reconcile: "enabled"
    fluxcd.controlplane.io/reconcileEvery: "5m"
spec:
  type: ExternalService
  url: https://my-orchestrator.com/api/v1/flux/inputs?env=prod&app=app1
  secretRef:
    name: my-orchestrator-basic-auth
  certSecretRef:
    name: my-orchestrator-client-certs

The orchestrator API should be able to authorise the Flux Operator API calls using basic auth, bearer token and/or mTLS.

The orchestrator API should return a JSON object that contains an array of input values for the ResourceSet templates.

Example of a response that contains input values for two tenants of app1 with different feature flags:

{
  "inputs": [
    {
      "tenant": "tenant1",
      "version": "2.0.0-rc.1",
      "featureFlags": [
        "feature1",
        "feature2",
        "feature3"
      ]
    },
    {
      "tenant": "tenant2",
      "version": "1.9.0",
      "featureFlags": [
        "feature1",
        "feature2"
      ]
    }
  ]
}

Example of a ResourceSet that uses the inputs fetched from the orchestrator API:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: app1
  namespace: apps
spec:
  inputsFrom:
    - kind: ResourceSetInputProvider
      name: app1-feature-flags
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1beta2
      kind: OCIRepository
      metadata:
        name: app1-<< inputs.tenant >>
        namespace: apps
      spec:
        interval: 1h
        url: oci://ghcr.io/org/charts/app1
        ref:
          semver: << inputs.version | quote >>
    - apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: app1-<< inputs.tenant >>
        namespace: apps
        annotations:
          event.toolkit.fluxcd.io/tenant: << inputs.tenant | quote >>
          event.toolkit.fluxcd.io/env: "production"
          event.toolkit.fluxcd.io/app: "app1"
      spec:
        interval: 1h
        chartRef:
          kind: OCIRepository
          name: app1-<< inputs.tenant >>
        values:
          features: << inputs.featureFlags | join "," >>
          ingress:
            hosts:
              - host: app1-<< inputs.tenant >>.example.com

Given the above configuration, the orchestrator can instruct Flux Operator to perform the following actions:

• Onboard a new tenant by adding a new input set to the response. The Flux Operator will deploy an instance of app1 for the new tenant and expose it outside the cluster using a dedicated hostname.
• Offboard a tenant by removing the input set from the response. The Flux Operator will uninstall the app1 instance dedicated to the tenant.
• Toggle features for a tenant by updating the input set in the response. The Flux Operator will update the HelmRelease values and trigger a Helm upgrade.
• Change the app version for a tenant by updating the input set in the response. The Flux Operator will update the OCIRepository ref and trigger a Helm upgrade for the new Chart version.

The orchestrator can use the Flux events to track the status of the deployed instances by exposing an API endpoint that Flux notifications-controller can call.

Example of Flux Alert that sends events to the orchestrator API:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: orchestrator-webhook
  namespace: apps
spec:
  type: generic
  address: https://my-orchestrator.com/api/v1/flux/events
  secretRef:
    name: my-orchestrator-basic-auth
  certSecretRef:
    name: my-orchestrator-client-certs
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: orchestrator
  namespace: apps
spec:
  providerRef:
    name: orchestrator-webhook
  eventSources:
    - kind: HelmRelease
      name: '*'

Every time a HelmRelease changes its status, the orchestrator will receive a HTTP POST request, for example:

{
  "involvedObject": {
    "apiVersion": "helm.toolkit.fluxcd.io/v2",
    "kind": "HelmRelease",
    "name": "app1-tenant1",
    "namespace": "apps",
    "uid": "7d0cdc51-ddcf-4743-b223-83ca5c699632"
  },
  "metadata": {
    "helm.toolkit.fluxcd.io/app-version": "2.0.0-rc.1",
    "event.toolkit.fluxcd.io/tenant": "tenant1",
    "event.toolkit.fluxcd.io/env": "production",
    "event.toolkit.fluxcd.io/app": "app1"
  },
  "severity":"info",
  "reason": "UpgradeSucceeded",
  "message":"Helm upgrade succeeded for release apps/app1-tenant1.v2 with chart [email protected]+0cc9a8446c95",
  "reportingController":"helm-controller",
  "timestamp":"2025-01-18T07:26:19Z"
}

The Flux events can contain arbitrary data set in the ResourceSet using the event.toolkit.fluxcd.io/tenant annotations. The custom metadata can be used by the orchestrator to identify the deployment and update its internal state.