Cluster health yellow and data-node-0 getting excluded from shard assignments

[hollowdew]

Hello! I'm using the Helm OpenSearch-Operator and Helm OpenSearch-Cluster.
My cluster is showing a yellow status due to unassigned replica shards.
I ran GET _cluster/settings and received the following output:

{
  "persistent" : {
    "plugins" : {
      "index_state_management" : {
        "template_migration" : {
          "control" : "-1"
        }
      }
    }
  },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all",
          "exclude" : {
            "_name" : "opensearch-cluster-data-nodes-0"
          }
        }
      }
    }
  }
}

And the health of my cluster: GET _cluster/health?pretty

{
  "cluster_name" : "opensearch-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 6,
  "number_of_data_nodes" : 3,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 171,
  "active_shards" : 335,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 108,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 75.62076749435666
}

I attempted to remove data-nodes-0 from the exclusion list using a PUT request, but it automatically gets added back to the list after a few seconds.
Here are some details about my setup:

helm list
NAME                    NAMESPACE       REVISION        UPDATED                                         STATUS          CHART                           APP VERSION
opensearch-cluster      opensearch      5               2024-04-22 13:53:25.886739305 +0200 CEST        deployed        opensearch-cluster-2.5.1        2.5.1
opensearch-operator     opensearch      4               2024-04-22 14:13:59.756698617 +0200 CEST        deployed        opensearch-operator-2.5.1       2.5.1

kubectl get OpenSearchCluster
NAME                 HEALTH   NODES   VERSION   PHASE     AGE
opensearch-cluster   yellow   6       2.8.0     RUNNING   66d

kubectl describe OpenSearchCluster opensearch-cluster
Name:         opensearch-cluster
Namespace:    opensearch
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: opensearch-cluster
              meta.helm.sh/release-namespace: opensearch
API Version:  opensearch.opster.io/v1
Kind:         OpenSearchCluster
Metadata:
  Creation Timestamp:  2024-02-15T15:44:05Z
  Finalizers:
    Opster
  Generation:        5
  Resource Version:  86865705
  UID:               73cea515-fdf2-4d78-a841-ec91a889a1a3
Spec:
  Bootstrap:
    Resources:
  Conf Mgmt:
  Dashboards:
    Enable:  true
    Opensearch Credentials Secret:
      Name:    admin-credentials-secret
    Replicas:  1
    Resources:
      Limits:
        Cpu:     600m
        Memory:  4096Mi
      Requests:
        Cpu:     300m
        Memory:  4096Mi
    Service:
      Type:   ClusterIP
    Version:  2.3.0
  General:
    Drain Data Nodes:  true
    Http Port:         9200
    Monitoring:
    Service Name:          opensearch-cluster
    Set VM Max Map Count:  true
    Vendor:                opensearch
    Version:               2.8.0
  Init Helper:
    Resources:
  Node Pools:
    Component:  cluster-managers
    Disk Size:  100Gi
    Replicas:   3
    Resources:
      Limits:
        Cpu:     1600m
        Memory:  4096Mi
      Requests:
        Cpu:     500m
        Memory:  2048Mi
    Roles:
      cluster_manager
    Component:  data-nodes
    Disk Size:  4096Gi
    Replicas:   3
    Resources:
      Limits:
        Cpu:     2000m
        Memory:  24576Mi
      Requests:
        Cpu:     1000m
        Memory:  24576Mi
    Roles:
      data
  Security:
    Config:
      Admin Credentials Secret:
        Name:  admin-credentials-secret
      Admin Secret:
      Security Config Secret:
        Name:  securityconfig-secret
    Tls:
      Http:
        Ca Secret:
        Generate:  true
        Secret:
      Transport:
        Ca Secret:
        Generate:  true
        Secret:
Status:
  Available Nodes:  6
  Components Status:
    Component:  Restarter
    Status:     Finished
  Health:       yellow
  Initialized:  true
  Phase:        RUNNING
  Version:      2.8.0
Events:
  Type    Reason          Age                 From                     Message
  ----    ------          ----                ----                     -------
  Normal  RollingRestart  53m (x68 over 63m)  containerset-controller  Starting to rolling restart

Here are my Helm values for the OpenSearch cluster:

opensearchCluster:
  enabled: true
  general:
    httpPort: "9200"
    version: 2.8.0
    serviceName: "opensearch-cluster"
    drainDataNodes: true
    setVMMaxMapCount: true
  dashboards:
    enable: true
    replicas: 1
    tls:
      enable: false
      generate: false
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    resources:
      limits:
        memory: "4096Mi"
        cpu: "600m"
      requests:
        memory: "4096Mi"
        cpu: "300m"
  nodePools:
    - component: cluster-managers
      diskSize: "100Gi"
      replicas: 3
      roles:
        - "cluster_manager"
      resources:
        limits:
          memory: "4096Mi"
          cpu: "1600m"
        requests:
          memory: "2048Mi"
          cpu: "500m"
    - component: data-nodes
      diskSize: "4096Gi"
      replicas: 3
      roles:
        - "data"
      resources:
        limits:
          memory: "24576Mi"
          cpu: "2000m"
        requests:
          cpu: "1000m"
          memory: "24576Mi"
  security:
    config:
      adminCredentialsSecret:
        name: admin-credentials-secret
      securityConfigSecret:
        name: securityconfig-secret
    tls:
      transport:
        generate: true
      http:
        generate: true

And here are the Helm values for the operator:

COMPUTED VALUES:
fullnameOverride: ""
installCRDs: true
kubeRbacProxy:
  enable: true
  image:
    repository: gcr.io/kubebuilder/kube-rbac-proxy
    tag: v0.15.0
  livenessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 10443
      scheme: HTTPS
    initialDelaySeconds: 10
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
  readinessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 10443
      scheme: HTTPS
    initialDelaySeconds: 10
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
  resources:
    limits:
      cpu: 50m
      memory: 50Mi
    requests:
      cpu: 25m
      memory: 25Mi
  securityContext: null
manager:
  dnsBase: cluster.local
  extraEnv: []
  image:
    pullPolicy: Always
    repository: opensearchproject/opensearch-operator
    tag: ""
  imagePullSecrets: []
  livenessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 8081
    initialDelaySeconds: 10
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
  loglevel: info
  parallelRecoveryEnabled: true
  readinessProbe:
    failureThreshold: 3
    httpGet:
      path: /readyz
      port: 8081
    initialDelaySeconds: 10
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
  resources:
    limits:
      cpu: 350m
      memory: 8192Mi
    requests:
      cpu: 100m
      memory: 1024Mi
  securityContext:
    allowPrivilegeEscalation: false
  watchNamespace: null
nameOverride: ""
nodeSelector: {}
securityContext:
  runAsNonRoot: true
serviceAccount:
  create: true
  name: ""
tolerations: []

Here are some log entries from my operator that repeat every few seconds:

{"level":"info","ts":"2024-04-22T13:00:10.876Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"opensearch"},"namespace":"opensearch","name":"opensearch-cluster","reconcileID":"153edb1f-786a-426c-9e88-23569d99696b","cluster":{"name":"opensearch-cluster","namespace":"opensearch"}}
{"level":"info","ts":"2024-04-22T13:00:10.892Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"opensearch"},"namespace":"opensearch","name":"opensearch-cluster","reconcileID":"153edb1f-786a-426c-9e88-23569d99696b","interface":"transport"}
{"level":"info","ts":"2024-04-22T13:00:10.892Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"opensearch"},"namespace":"opensearch","name":"opensearch-cluster","reconcileID":"153edb1f-786a-426c-9e88-23569d99696b","interface":"http"}
{"level":"info","ts":"2024-04-22T13:00:10.894Z","msg":"ServiceMonitor crd not found, skipping deletion","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"opensearch"},"namespace":"opensearch","name":"opensearch-cluster","reconcileID":"153edb1f-786a-426c-9e88-23569d99696b"}

I haven't found any errors in the logs for the manager-node or data-node.
After I remove data-node-0 from the exclusion list, the unassigned shards begin to get processed slowly.
However, data-node-0 is then re-added to the exclusion list.
Could anyone suggest how to resolve this issue? Any help would be greatly appreciated.
Thank you!