Error listing K8s resources using data_kubernetes_resources

[rafed]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v1.5.5
Kubernetes provider version: 1.22.0
Kubernetes version: v1.24.15

Affected Resource(s)

• pvc
• pv
• namespace

Issue appears to affect multiple resources, it may be an issue with Terraform's core.

Terraform Configuration Files

data "kubernetes_resources" "pvc" {
  api_version    = "v1"
  kind           = "PersistentVolumeClaim"
  namespace      = "mongodb"
  field_selector = "metadata.namespace=mongodb"
}

Debug Output

https://gist.github.com/rafed/742a2f107c92f81f2b0a0ba37eeffc3a

Steps to Reproduce

1. terraform plan

Expected Behavior

Should list the pvcs in the mongodb namespace

Actual Behavior

Throws error

Important Factoids

Running in EKS

References

#2214

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[arybolovlev]

Hi @rafed,

I am not able to reproduce this issue. Is there any other facts we need to know to reproduce the issue? Have you tried data source kubernetes_persistent_volume_claim_v1 to gather the data you need?

Thank you.

[rafed]

Hey @arybolovlev. Thanks for looking into this!

I think I understand what's causing this on my end.

When I tried listing PVCs on all other namespaces in my cluster, I am getting them without any issue.

However, the issue occurs in my mongodb namespace. And it's probably because of one non ready PVC. The following is a screenshot of my mongodb PVCs.

If I use limit=2 in the parameter, the two running PVCs data are returned without any issue. However, by omitting limit=2 an error is thrown and I see the following in my console.

`│Error: Failed to save resource state

with data.kubernetes_resources.pvc,
on main.tf line 71, in data "kubernetes_resources" "pvc":
71: data "kubernetes_resources" "pvc" {

AttributeName("metadata").AttributeName("managedFields"): AttributeName("metadata").AttributeName("managedFields")] incompatible tuple types`

Any thoughts on this? :/

[arybolovlev]

Hi @rafed,

Thank you for this clarification. Now I am able to reproduce this issue and get the same error message.

A quick investigation brought me to the idea that it might be related to the status of the objects.

My example:

$ kubectl -n this get pvc

NAME             STATUS    VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
task-pv-claim    Bound     task-pv-volume   10Gi       RWO            manual         28m
task-pv-claim2   Pending                                              manual         28m


$ kubectl -n this get pvc -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  ...
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    phase: Bound

- apiVersion: v1
  kind: PersistentVolumeClaim
  ...
  status:
    phase: Pending

As you can see, PVC with status Bound has 3 attributes under status: accessModes, capacity, and phase. PVC with status Pending has 1 attribute under status: phase. This status mismatch leads to this condition where attr has length 3 and as the result it ends up with an error.

We definitely need more time to investigate and address this issue.

[bradhandy]

I get the same error when searching for Services in Kubernetes with kubernetes_resources.

data "kubernetes_resources" "services" {
  api_version = "v1"
  kind = "Service"
  namespace = "[namepsace]"
  label_selector = "customer-label=value"
}

EDIT: I'm using version 2.23.0 of the Kubernetes Provider. This issue says 1.22.0. I'm not sure if that's a mistype or if I should create another ticket.

[jeremymcgee73]

@bradhandy any luck with this? I'm running into it as well. I'm trying to get a podname.

data "kubernetes_resources" "gitlab_toolbox" {
  api_version = "v1"
  kind        = "Pod"
  namespace   = var.namespace

  label_selector = "app=toolbox"
}

 AttributeName("metadata").AttributeName("managedFields"):
│ [AttributeName("metadata").AttributeName("managedFields")] incompatible
│ tuple types

[bradhandy]

What I ended up doing was creating a list of the services I expected to be in the namespace, then using for_each on a kubernetes_service_v1 data source to get all of the information I needed. Then I ended up not needing the logic anyway so I ditched it.

I experimented with using an external data source to call a script. The script would run kubectl to output the results in JSON, then use jq to massage the results. The final output of the script would be a JSON object with a single key/value pair where the value was the massaged JSON encoded as a string. Then I could use the jsondecode method to access the data. Definitely not ideal, but it seemed to work locally. I haven't tried it in project code yet.

[JBOClara]

Same error with v2.24.0

data "kubernetes_resources" "namespaces" {
  api_version    = "v1"
  kind           = "Namespace"
  field_selector = "metadata.name!=kube-system"
}
output "namespaces" {
  value = data.kubernetes_resources.namespaces.objects
}

│
│ AttributeName("metadata").AttributeName("managedFields"): [AttributeName("metadata").AttributeName("managedFields")] incompatible tuple types
╵

[cunningr]

so, even the example in the docs doesn't work?

[ldelgadop44]

Same error with v2.25.2

data "kubernetes_resources" "example" {
  provider       = kubernetes.local
  api_version    = "apps/v1"
  kind           = "Deployment"
  namespace      = "linkerd"
}
output "deployments" {
  value = data.kubernetes_resources.namespaces.objects
}

Error: AttributeName("metadata").AttributeName("managedFields"): │ [AttributeName("metadata").AttributeName("managedFields")] incompatible tuple types

[macleanj]

Still issues on 2.27.0

[clone206]

Still issues on 2.27.0

Same

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.27.0"
    }
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.97.1"
    }
  }
  required_version = ">= 1.1.0"
}
...
data "kubernetes_resources" "my_namespaces" {
  api_version = "v1"
  kind        = "Namespace"
}

output "my_objs" {
  value = data.kubernetes_resources.my_namespaces.objects
}

...
data.kubernetes_resources.my_namespaces: Reading...
╷
│ Error: Failed to save resource state
│
│   with data.kubernetes_resources.my_namespaces,
│   on main.tf line 67, in data "kubernetes_resources" "my_namespaces":
│   67: data "kubernetes_resources" "my_namespaces" {
│
│ AttributeName("metadata").AttributeName("managedFields"): [AttributeName("metadata").AttributeName("managedFields")] incompatible tuple types