Provider produced inconsistent result after apply

[mailbox171]

Hello, we had an issue updating a resource.
The resource was actually updated, but we got several error messages and I am not sure of the terraform state we have now.

Thanks a lot.

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.5.1
Kubernetes provider version: v2.21.1
Kubernetes version: 1.27.1

Affected Resource(s)

resource "kubernetes_manifest" (argocd Application)

Terraform Configuration Files

resource "kubernetes_manifest" "xxxxxxxxx" {

  computed_fields = [ "spec.source.targetRevision" ]

  field_manager {
    force_conflicts = true
  }

  manifest = yamldecode(templatefile(var.argocd_application_file, {
    "app_path" = var.app_path
  })) 
}


#### Manifest

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sdp-apps
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-options: Delete=false
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  project: default
  source:     
    path: ${app_path}
    repoURL: git@--repo----.git
    targetRevision: prod
  syncPolicy:
    automated:
      selfHeal: true
      # prune: true
    syncOptions:
    - ApplyOutOfSyncOnly=true
    - PruneLast=true
    - CreateNamespace=true

Debug Output

Panic Output

Steps to Reproduce

We have the resource

apiVersion: argoproj.io/v1alpha1
kind: Application

and we want to update just one field:
spec.source.targetRevision

Expected Behavior

Resource updated
TF state updated

Actual Behavior

Resource updated
Error messages
TF state.. we do not know

---

Run terraform apply -input=false 9ef16c8acdfa418eff36bd96026dc74481b6ff2c.tfplan
kubernetes_manifest.buildbuddy: Modifying...
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.buildbuddy, provider
│ "provider["registry.terraform.io/hashicorp/kubernetes"]" produced an
│ unexpected new value: .object.operation.initiatedBy.automated: was null,
│ but now cty.True.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.buildbuddy, provider
│ "provider["registry.terraform.io/hashicorp/kubernetes"]" produced an
│ unexpected new value: .object.operation.retry.limit: was null, but now
│ cty.NumberIntVal(5).
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.buildbuddy, provider
│ "provider["registry.terraform.io/hashicorp/kubernetes"]" produced an
│ unexpected new value: .object.operation.sync.resources: was null, but now
│ cty.ListVal([]cty.Valuecty.ObjectVal(map[string]cty.Value"group":cty.StringVal("argoproj.io"),
│ "kind":cty.StringVal("Application"), "name":cty.StringVal("zuul-ci"),
│ "namespace":cty.NullVal(cty.String)),
│ cty.ObjectVal(map[string]cty.Value"group":cty.StringVal("argoproj.io"),
│ "kind":cty.StringVal("AppProject"), "name":cty.StringVal("jenkins"),
│ "namespace":cty.NullVal(cty.String))).
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.buildbuddy, provider
│ "provider["registry.terraform.io/hashicorp/kubernetes"]" produced an
│ unexpected new value: .object.operation.sync.revision: was null, but now
│ cty.StringVal("d7207271230f926f89aa02f8000ccd9c92827135").
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.buildbuddy, provider
│ "provider["registry.terraform.io/hashicorp/kubernetes"]" produced an
│ unexpected new value: .object.operation.sync.syncOptions: was null, but now
│ cty.ListVal([]cty.Valuecty.StringVal("ApplyOutOfSyncOnly=true"),
│ cty.StringVal("PruneLast=true"), cty.StringVal("CreateNamespace=true")).
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.buildbuddy, provider
│ "provider["registry.terraform.io/hashicorp/kubernetes"]" produced an
│ unexpected new value: .object.metadata.annotations: new element
│ "kubectl.kubernetes.io/last-applied-configuration" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
Error: Process completed with exit code 1.

---

Important Factoids

References

• Missing secretNamespace on azure_file of persistent_volume_source #1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[akumar-99]

I am facing the exact same issue with a different manifest. I am using ECK CRDs to setup elastic, kibana etc. First apply goes smoothly and resources get created but following terraform apply gives me similar error even if I don't make any changes to manifest.

Is there anything missing in my configuration or is it really a bug?

Terraform

resource "kubernetes_manifest" "eck-kibana" {
  manifest = yamldecode(templatefile("../scripts/eck/eck-kibana.yaml", {
    namespace = kubernetes_namespace.observability.metadata[0].name
  }))
}

Manifest

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: ${namespace}
spec:
  version: 8.8.2
  count: 1
  elasticsearchRef:
    name: elasticsearch
    namespace: ${namespace}
  enterpriseSearchRef:
    name: enterprise-search
    namespace: ${namespace}
  config:
    server:
      publicBaseUrl: http://kibana.ai.internal
  podTemplate:
    spec:
      nodeSelector:
        workload-type: observability

Command

terraform apply -target=kubernetes_manifest.eck-kibana

First apply output

kubernetes_manifest.eck-kibana: Creating...
kubernetes_manifest.eck-kibana: Creation complete after 1s

Second apply output with no changes anywhere

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # kubernetes_manifest.eck-kibana will be updated in-place
  ~ resource "kubernetes_manifest" "eck-kibana" {
      ~ object   = {
          ~ spec       = {
              ~ podTemplate          = {
                  - metadata = {
                      - creationTimestamp = null
                    }
                  ~ spec     = {
                      - containers   = null
                        # (1 unchanged attribute hidden)
                    }
                }
                # (11 unchanged attributes hidden)
            }
            # (3 unchanged attributes hidden)
        }
        # (1 unchanged attribute hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

kubernetes_manifest.eck-kibana: Modifying...

╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to kubernetes_manifest.eck-kibana, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value:
│ .object: wrong final value type: incorrect object attributes.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

[mailbox171]

Hello, thanks for you help.

I think the urgent question is.. since I saw the updates went through, and also the new state file looks plausible to me (long shot, I know), how scared should I be?
I haven't run the scripts again, but sooner or later I will have to, I am afraid.
How dangerous it is?

It is a PRODUCTION environment.

Thank you very much.

[mcallaghan-geotab]

+1 - there has been a regression

[mailbox171]

Is it positive there was a regression.
Which version should we use then?
Can anybody recommend a safe version to use now?

[mcallaghan-geotab]

pinned to previous version as a workaround and issue was avoided

[hassanzhd]

@mcallaghan-geotab i am using version = "2.21.1" and still gives the same error, all computed fields are explicitly defined

[roman-parkhunovskyi]

@mcallaghan-geotab i am using version = "2.21.1" and still gives the same error, all computed fields are explicitly defined

Confirm. Have just faced this. v2.21.0 is affected as well.

[hassanzhd]

We provided all computed_fields with default values part of the manifest key and that resolved the error, this can be a workaround as long as provider isnt fixed

[centime-bhavyasree]

Hi,

We are using hashicorp/kubernetes v2.16.1 and wanted to apply this manifest:
apiVersion: v1
kind: Secret
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
kubernetes.io/service-account.name: ingress-nginx-admission
type: kubernetes.io/service-account-token
on eks of version 1.27, tried with kubectl version 1.19.0 and 1.28.1 got the same issue with both :
│ Error: Provider produced inconsistent result after apply

│

│ When applying changes to

│ module.eks-fs-cluster.kubectl_manifest.ingress_controller_secret["/api/v1/namespaces/ingress-nginx/secrets/ingress-nginx"],

│ provider

│ "module.eks-fs-cluster.provider["registry.terraform.io/gavinbunney/kubectl"]"

│ produced an unexpected new value: Root resource was present, but now

│ absent.

│

│ This is a bug in the provider, which should be reported in the provider's

│ own issue tracker.

Please look into this issue

[Kostanos]

Hi, having the same issue here. My configuration to apply

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api.mysite.com
  namespace: ${namespace}
spec:
  replicas: 1
  selector:
    matchLabels:
      App: "api.mysite.com"
  template:
    metadata:
      creationTimestamp: null
      labels:
        App: "api.mysite.com"
    spec:
      # imagePullSecrets:
      #   - name: ${ecr_registry_secret_name}
      containers:
        - name: api-mysite-com-container
          image: hasura/graphql-engine:v2.31.0
          imagePullPolicy: IfNotPresent
          ports:
          - name: http
            containerPort: 8080
          envFrom:
          - secretRef:
              name: api-mysite-com-env
          livenessProbe:
            httpGet:
              path: "/healthz"
              port: http
          readinessProbe:
            httpGet:
              path: "/healthz"
              port: http
          resources:
            requests:
              memory: "1.5Gi"
              cpu: "1"
            limits:
              memory: "1.7Gi"
              cpu: "1"

Error getting:

╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to kubernetes_manifest.api_mysite_com_resources["./yamls/api-mysite-com/api-mysite-com-deployment.tpl.yaml"], provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value:
│ .object.spec.template.spec.containers[0].resources.limits["memory"]: was cty.StringVal("1.7Gi"), but now cty.StringVal("1825361100800m").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to kubernetes_manifest.api_mysite_com_resources["./yamls/api-mysite-com/api-mysite-com-deployment.tpl.yaml"], provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value:
│ .object.spec.template.spec.containers[0].resources.requests["memory"]: was cty.StringVal("1.5Gi"), but now cty.StringVal("1536Mi").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

[jasonhuling]

@mcallaghan-geotab i am using version = "2.21.1" and still gives the same error, all computed fields are explicitly defined

FWIW I was encountering this error on 2.23.0, rolling back to 2.21.1 resolved it for me. I am not explicitly defining the computed fields.

[rigman24]

We provided all computed_fields with default values part of the manifest key and that resolved the error, this can be a workaround as long as provider isnt fixed

I wouldn't hold your breath. This issue (the inconsistent final result after apply) has been around an extremely long time and the only response I've seen really are dismissive comments from maintainers about how this type of thing isn't a bug and/or something you need fixed.

Sometimes adding the problematic field to computed_fields fixes it, often not.

The gavinbunny/kubectl provider works as you'd expect it to. No idea why this repository doesn't do a similar thing.

[Torvalds9]

I encountered the same problem while trying to update: istio_request_authentication manifest using hashicorp k8s provider.

The problem was while i was trying to remove match_labels part of the manifest, removing "app.kubernetes.io/component" = "api" label.

Apply was returning:

╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to
│ module.auth.module.istio["namespace"].kubernetes_manifest.istio_authorization_policy_jwt["xxx-api"],
│ provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]"
│ produced an unexpected new value: .object.spec.selector.matchLabels: new
│ element "app.kubernetes.io/component" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵

Apply was unsuccessful, I'm sure that there was nothing modifing that field of label outside of Terraform, confirmed that doing manual kubectl edit on this manifest, and after that the state in Terraform was matching as expected.
Another way to fix that was to taint and re-create the resource.

This seems to me like a bug either with applying the only label change in that resource or some problem with the state refresh and the change is getting rolled back after the inconsistent result of the current state.

I tried rolling down the provider version to 2.20 and exerienced the same issue.