You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The rancher vcluster died for an unknown reaso, in the past it died due to resource constraints and needed to have the memory limit bumped up. This issue may have been a result of the PVC becoming over used? After checking I found the PVC to be 100% used and I bumped it up to 20 Gigs instead of 5. However the K3s cluster (and by extension Rancher) can no longer seem to start. I have posted what I believe are the relevant logs below. The cluster never finds its way back to a working condition.
Rancher
2024/06/12 16:15:03 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
2024/06/12 16:15:15 [ERROR] error syncing 'c-m-pc2tc9xz': handler cluster-deploy: Get "https://10.53.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:15:15 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
2024/06/12 16:15:19 [ERROR] error syncing 'c-m-lf7m5cm9': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:15:34 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
W0612 16:15:40.369594 33 warnings.go:80] cluster.x-k8s.io/v1alpha3 Machine is deprecated; use cluster.x-k8s.io/v1beta1 Machine
W0612 16:15:40.369647 33 reflector.go:533] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
E0612 16:15:40.369666 33 reflector.go:148] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
2024/06/12 16:15:45 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
2024/06/12 16:16:01 [ERROR] error syncing 'c-m-pc2tc9xz': handler cluster-deploy: Get "https://10.53.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:16:03 [ERROR] error syncing 'c-m-lf7m5cm9': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:16:04 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
2024/06/12 16:16:10 [INFO] Stopping cluster agent for c-m-pc2tc9xz
2024/06/12 16:16:10 [ERROR] failed to start cluster controllers c-m-pc2tc9xz: context canceled
2024/06/12 16:16:13 [INFO] Stopping cluster agent for c-m-lf7m5cm9
2024/06/12 16:16:13 [ERROR] failed to start cluster controllers c-m-lf7m5cm9: context canceled
2024/06/12 16:16:16 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
W0612 16:16:17.695919 33 warnings.go:80] cluster.x-k8s.io/v1alpha3 Machine is deprecated; use cluster.x-k8s.io/v1beta1 Machine
W0612 16:16:17.695961 33 reflector.go:533] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
E0612 16:16:17.695977 33 reflector.go:148] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
2024/06/12 16:16:35 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
2024/06/12 16:16:44 [ERROR] error syncing 'c-m-lf7m5cm9': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:16:46 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
2024/06/12 16:16:46 [ERROR] error syncing 'c-m-pc2tc9xz': handler cluster-deploy: Get "https://10.53.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:17:05 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
W0612 16:17:15.541263 33 warnings.go:80] cluster.x-k8s.io/v1alpha3 Machine is deprecated; use cluster.x-k8s.io/v1beta1 Machine
W0612 16:17:15.541322 33 reflector.go:533] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
E0612 16:17:15.541345 33 reflector.go:148] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
2024/06/12 16:17:16 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
2024/06/12 16:17:26 [ERROR] error syncing 'c-m-lf7m5cm9': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:17:31 [ERROR] error syncing 'c-m-pc2tc9xz': handler cluster-deploy: Get "https://10.53.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:17:36 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
2024/06/12 16:17:46 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
W0612 16:17:56.489718 33 warnings.go:80] cluster.x-k8s.io/v1alpha3 Machine is deprecated; use cluster.x-k8s.io/v1beta1 Machine
W0612 16:17:56.489905 33 reflector.go:533] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
E0612 16:17:56.490001 33 reflector.go:148] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
2024/06/12 16:18:07 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
2024/06/12 16:18:08 [ERROR] error syncing 'c-m-lf7m5cm9': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:18:10 [INFO] Stopping cluster agent for c-m-lf7m5cm9
2024/06/12 16:18:10 [ERROR] failed to start cluster controllers c-m-lf7m5cm9: context canceled
2024/06/12 16:18:13 [ERROR] error syncing 'c-m-pc2tc9xz': handler cluster-deploy: Get "https://10.53.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:18:13 [INFO] Stopping cluster agent for c-m-pc2tc9xz
2024/06/12 16:18:13 [ERROR] failed to start cluster controllers c-m-pc2tc9xz: context canceled
2024/06/12 16:18:17 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
2024/06/12 16:18:37 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
2024/06/12 16:18:47 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
2024/06/12 16:18:55 [ERROR] error syncing 'c-m-lf7m5cm9': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
W0612 16:18:56.386634 33 warnings.go:80] cluster.x-k8s.io/v1alpha3 Machine is deprecated; use cluster.x-k8s.io/v1beta1 Machine
W0612 16:18:56.386795 33 reflector.go:533] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
E0612 16:18:56.386861 33 reflector.go:148] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
2024/06/12 16:18:59 [ERROR] error syncing 'c-m-pc2tc9xz': handler cluster-deploy: Get "https://10.53.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:19:08 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
2024/06/12 16:19:17 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
2024/06/12 16:19:38 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- d635f289c9bd9610ada071b36d19d6ad0b38863c error: exit status 128, detail: error: Server does not allow request for unadvertised object d635f289c9bd9610ada071b36d19d6ad0b38863c
, requeuing
2024/06/12 16:19:40 [ERROR] error syncing 'c-m-lf7m5cm9': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:19:46 [ERROR] error syncing 'c-m-pc2tc9xz': handler cluster-deploy: Get "https://10.53.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing
2024/06/12 16:19:47 [ERROR] error syncing 'harvester': handler feature-handler: Internal error occurred: failed calling webhook "rancher.cattle.io.nodedrivers.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s": service "rancher-webhook" not found, requeuing
W0612 16:19:49.808545 33 warnings.go:80] cluster.x-k8s.io/v1alpha3 Machine is deprecated; use cluster.x-k8s.io/v1beta1 Machine
W0612 16:19:49.808703 33 reflector.go:533] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
E0612 16:19:49.808759 33 reflector.go:148] pkg/mod/github.com/rancher/[email protected]/tools/cache/reflector.go:231: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Machine failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": service "capi-webhook-service" not found
2024/06/12 16:19:53 [INFO] Stopping cluster agent for c-m-lf7m5cm9
2024/06/12 16:19:53 [ERROR] failed to start cluster controllers c-m-lf7m5cm9: context canceled
Rancher vcluster
syncer 2024-06-12 16:10:41 ERROR filters/authentication.go:73 Unable to authenticate the request {"component": "vcluster", "error": "Post \"https://127.0.0.1:6443/apis/authentication.k8s.io/v1/tokenreviews\": context canceled"}
syncer 2024-06-12 16:10:41 ERROR filters/authentication.go:73 Unable to authenticate the request {"component": "vcluster", "error": "Post \"https://127.0.0.1:6443/apis/authentication.k8s.io/v1/tokenreviews\": context canceled"}
syncer 2024-06-12 16:10:41 ERROR filters/authentication.go:73 Unable to authenticate the request {"component": "vcluster", "error": "Post \"https://127.0.0.1:6443/apis/authentication.k8s.io/v1/tokenreviews\": context canceled"}
syncer 2024-06-12 16:10:41 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:10:41 ERROR filters/authentication.go:73 Unable to authenticate the request {"component": "vcluster", "error": "Post \"https://127.0.0.1:6443/apis/authentication.k8s.io/v1/tokenreviews\": context canceled"}
syncer 2024-06-12 16:10:41 INFO event.capi-controller-manager-79ddf9b9d8-4g26s.17d84da2d0916d9e syncer/syncer.go:134 update virtual event cattle-provisioning-capi-system/capi-controller-manager-79ddf9b9d8-4g26s.17d84da2d0916d9e {"component": "vcluster"}
syncer 2024-06-12 16:10:41 INFO pod.capi-controller-manager-79ddf9b9d8-4g26s syncer/syncer.go:134 update virtual pod cattle-provisioning-capi-system/capi-controller-manager-79ddf9b9d8-4g26s, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:42 INFO event.fleet-controller-6fc8c65685-fnfrf.17d84d2dc74155e5 syncer/syncer.go:134 update virtual event cattle-fleet-system/fleet-controller-6fc8c65685-fnfrf.17d84d2dc74155e5 {"component": "vcluster"}
syncer 2024-06-12 16:10:42 INFO pod.fleet-controller-6fc8c65685-fnfrf syncer/syncer.go:134 update virtual pod cattle-fleet-system/fleet-controller-6fc8c65685-fnfrf, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:42 INFO event.fleet-controller-6fc8c65685-fnfrf.17d84d2dc8dd53a7 syncer/syncer.go:134 update virtual event cattle-fleet-system/fleet-controller-6fc8c65685-fnfrf.17d84d2dc8dd53a7 {"component": "vcluster"}
syncer 2024-06-12 16:10:42 INFO event.fleet-controller-6fc8c65685-fnfrf.17d84d2dd0d98df0 syncer/syncer.go:134 update virtual event cattle-fleet-system/fleet-controller-6fc8c65685-fnfrf.17d84d2dd0d98df0 {"component": "vcluster"}
syncer 2024-06-12 16:10:43 INFO pod.fleet-controller-6fc8c65685-fnfrf syncer/syncer.go:134 update virtual pod cattle-fleet-system/fleet-controller-6fc8c65685-fnfrf, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:44 INFO event.cert-manager-cainjector-d66ccc667-tjff8.17d84da2a8ec43ba syncer/syncer.go:134 update virtual event cert-manager/cert-manager-cainjector-d66ccc667-tjff8.17d84da2a8ec43ba {"component": "vcluster"}
syncer 2024-06-12 16:10:44 INFO pod.cert-manager-cainjector-d66ccc667-tjff8 syncer/syncer.go:134 update virtual pod cert-manager/cert-manager-cainjector-d66ccc667-tjff8, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:44 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:10:44 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 84.44µs, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:10:49 INFO event.ui-plugin-operator-9dc69b9d7-z6lpj.17d84d2d0859d65b syncer/syncer.go:134 update virtual event cattle-ui-plugin-system/ui-plugin-operator-9dc69b9d7-z6lpj.17d84d2d0859d65b {"component": "vcluster"}
syncer 2024-06-12 16:10:49 INFO pod.ui-plugin-operator-9dc69b9d7-z6lpj syncer/syncer.go:134 update virtual pod cattle-ui-plugin-system/ui-plugin-operator-9dc69b9d7-z6lpj, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:50 INFO event.ui-plugin-operator-9dc69b9d7-z6lpj.17d84da9f4a60b20 syncer/syncer.go:146 create virtual event cattle-ui-plugin-system/ui-plugin-operator-9dc69b9d7-z6lpj.17d84da9f4a60b20 {"component": "vcluster"}
syncer 2024-06-12 16:10:50 INFO event.ui-plugin-operator-9dc69b9d7-z6lpj.17d84d2d38144067 syncer/syncer.go:134 update virtual event cattle-ui-plugin-system/ui-plugin-operator-9dc69b9d7-z6lpj.17d84d2d38144067 {"component": "vcluster"}
syncer 2024-06-12 16:10:50 INFO event.fleet-agent-5bdc49b496-bz5b4.17d84d2cf6460799 syncer/syncer.go:134 update virtual event cattle-fleet-local-system/fleet-agent-5bdc49b496-bz5b4.17d84d2cf6460799 {"component": "vcluster"}
syncer 2024-06-12 16:10:50 INFO pod.fleet-agent-5bdc49b496-bz5b4 syncer/syncer.go:134 update virtual pod cattle-fleet-local-system/fleet-agent-5bdc49b496-bz5b4, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:50 INFO event.fleet-agent-5bdc49b496-bz5b4.17d84d2cf8454fee syncer/syncer.go:134 update virtual event cattle-fleet-local-system/fleet-agent-5bdc49b496-bz5b4.17d84d2cf8454fee {"component": "vcluster"}
syncer 2024-06-12 16:10:50 INFO event.gitjob-6c8875d84d-hxxt4.17d84d2ce9a0d30f syncer/syncer.go:134 update virtual event cattle-fleet-system/gitjob-6c8875d84d-hxxt4.17d84d2ce9a0d30f {"component": "vcluster"}
syncer 2024-06-12 16:10:50 INFO pod.gitjob-6c8875d84d-hxxt4 syncer/syncer.go:134 update virtual pod cattle-fleet-system/gitjob-6c8875d84d-hxxt4, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:50 INFO event.gitjob-6c8875d84d-hxxt4.17d84d2cea7d3408 syncer/syncer.go:134 update virtual event cattle-fleet-system/gitjob-6c8875d84d-hxxt4.17d84d2cea7d3408 {"component": "vcluster"}
syncer 2024-06-12 16:10:51 INFO event.gitjob-6c8875d84d-hxxt4.17d84d2cede0e394 syncer/syncer.go:134 update virtual event cattle-fleet-system/gitjob-6c8875d84d-hxxt4.17d84d2cede0e394 {"component": "vcluster"}
syncer 2024-06-12 16:10:52 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:10:52 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 106.876µs, GET "/apis/management.cattle.io/v3/users" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:10:52 INFO event.fleet-agent-5bdc49b496-bz5b4.17d84d2db4b46d91 syncer/syncer.go:134 update virtual event cattle-fleet-local-system/fleet-agent-5bdc49b496-bz5b4.17d84d2db4b46d91 {"component": "vcluster"}
syncer 2024-06-12 16:10:52 INFO event.ui-plugin-operator-9dc69b9d7-z6lpj.17d84d2d81af8d27 syncer/syncer.go:134 update virtual event cattle-ui-plugin-system/ui-plugin-operator-9dc69b9d7-z6lpj.17d84d2d81af8d27 {"component": "vcluster"}
syncer 2024-06-12 16:10:52 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:10:52 ERROR filters/wrap.go:54 timeout or abort while handling: method=GET URI="/readyz" audit-ID="84013688-35ec-4b40-a216-d8b4886a2662" {"component": "vcluster"}
syncer 2024-06-12 16:10:52 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 599ns, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:10:53 INFO pod.ui-plugin-operator-9dc69b9d7-z6lpj syncer/syncer.go:134 update virtual pod cattle-ui-plugin-system/ui-plugin-operator-9dc69b9d7-z6lpj, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:53 INFO event.rancher-7bcd45c474-nn962.17d84d3408e2c60b syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84d3408e2c60b {"component": "vcluster"}
syncer 2024-06-12 16:10:54 INFO pod.fleet-agent-5bdc49b496-bz5b4 syncer/syncer.go:134 update virtual pod cattle-fleet-local-system/fleet-agent-5bdc49b496-bz5b4, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:54 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:10:54 ERROR filters/wrap.go:54 timeout or abort while handling: method=GET URI="/readyz" audit-ID="037c3827-ff29-4251-a5bc-3e8667d25808" {"component": "vcluster"}
syncer 2024-06-12 16:10:54 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 738ns, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:10:54 INFO pod.rancher-7bcd45c474-nn962 syncer/syncer.go:134 update virtual pod cattle-system/rancher-7bcd45c474-nn962, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:56 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:10:56 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 804ns, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:10:57 INFO pod.cert-manager-cainjector-d66ccc667-tjff8 syncer/syncer.go:134 update virtual pod cert-manager/cert-manager-cainjector-d66ccc667-tjff8, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:57 INFO pod.capi-controller-manager-79ddf9b9d8-4g26s syncer/syncer.go:134 update virtual pod cattle-provisioning-capi-system/capi-controller-manager-79ddf9b9d8-4g26s, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:58 INFO event.rancher-7bcd45c474-nn962.17d84daac744d017 syncer/syncer.go:146 create virtual event cattle-system/rancher-7bcd45c474-nn962.17d84daac744d017 {"component": "vcluster"}
syncer 2024-06-12 16:10:58 INFO pod.gitjob-6c8875d84d-hxxt4 syncer/syncer.go:134 update virtual pod cattle-fleet-system/gitjob-6c8875d84d-hxxt4, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:10:58 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:10:58 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 47.106µs, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:11:00 INFO event.rancher-7bcd45c474-nn962.17d84d2dc8f9aa4b syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84d2dc8f9aa4b {"component": "vcluster"}
syncer 2024-06-12 16:11:00 INFO event.rancher-7bcd45c474-nn962.17d84d2dcd8aca99 syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84d2dcd8aca99 {"component": "vcluster"}
syncer 2024-06-12 16:11:00 INFO event.capi-controller-manager-79ddf9b9d8-4g26s.17d84d2d170b6d71 syncer/syncer.go:134 update virtual event cattle-provisioning-capi-system/capi-controller-manager-79ddf9b9d8-4g26s.17d84d2d170b6d71 {"component": "vcluster"}
syncer 2024-06-12 16:11:00 INFO event.capi-controller-manager-79ddf9b9d8-4g26s.17d84d2d19999932 syncer/syncer.go:134 update virtual event cattle-provisioning-capi-system/capi-controller-manager-79ddf9b9d8-4g26s.17d84d2d19999932 {"component": "vcluster"}
syncer 2024-06-12 16:11:00 INFO event.rancher-7bcd45c474-nn962.17d84d2de4ee2b5a syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84d2de4ee2b5a {"component": "vcluster"}
syncer 2024-06-12 16:11:00 INFO event.cert-manager-cainjector-d66ccc667-tjff8.17d84d2d024c9059 syncer/syncer.go:134 update virtual event cert-manager/cert-manager-cainjector-d66ccc667-tjff8.17d84d2d024c9059 {"component": "vcluster"}
syncer 2024-06-12 16:11:00 INFO event.cert-manager-cainjector-d66ccc667-tjff8.17d84d2d03876e0d syncer/syncer.go:134 update virtual event cert-manager/cert-manager-cainjector-d66ccc667-tjff8.17d84d2d03876e0d {"component": "vcluster"}
syncer 2024-06-12 16:11:00 INFO event.cert-manager-cainjector-d66ccc667-tjff8.17d84d2d06f61df4 syncer/syncer.go:134 update virtual event cert-manager/cert-manager-cainjector-d66ccc667-tjff8.17d84d2d06f61df4 {"component": "vcluster"}
syncer 2024-06-12 16:11:01 INFO event.capi-controller-manager-79ddf9b9d8-4g26s.17d84d2d8774857b syncer/syncer.go:134 update virtual event cattle-provisioning-capi-system/capi-controller-manager-79ddf9b9d8-4g26s.17d84d2d8774857b {"component": "vcluster"}
syncer 2024-06-12 16:11:01 INFO event.capi-controller-manager-79ddf9b9d8-4g26s.17d84d2da7f19898 syncer/syncer.go:134 update virtual event cattle-provisioning-capi-system/capi-controller-manager-79ddf9b9d8-4g26s.17d84d2da7f19898 {"component": "vcluster"}
syncer 2024-06-12 16:11:01 INFO pod.capi-controller-manager-79ddf9b9d8-4g26s syncer/syncer.go:134 update virtual pod cattle-provisioning-capi-system/capi-controller-manager-79ddf9b9d8-4g26s, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:11:02 INFO pod.rancher-7bcd45c474-nn962 syncer/syncer.go:134 update virtual pod cattle-system/rancher-7bcd45c474-nn962, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:11:02 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:11:02 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 38.692µs, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:11:06 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:11:06 ERROR filters/wrap.go:54 timeout or abort while handling: method=GET URI="/readyz" audit-ID="34d748b4-21bd-49b7-92a0-6251462cd828" {"component": "vcluster"}
syncer 2024-06-12 16:11:06 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 699ns, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:11:23 INFO event.rancher-7bcd45c474-nn962.17d84d3408e2c60b syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84d3408e2c60b {"component": "vcluster"}
syncer 2024-06-12 16:11:24 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:11:24 ERROR filters/wrap.go:54 timeout or abort while handling: method=GET URI="/readyz" audit-ID="7d7cf496-494a-4d2a-9acc-a23dc62d9663" {"component": "vcluster"}
syncer 2024-06-12 16:11:24 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 605ns, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:11:34 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:11:34 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 125.84µs, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:11:41 ERROR filters/wrap.go:54 timeout or abort while handling: method=GET URI="/v1-k3s/connect" audit-ID="65c4bc8c-ebf2-45df-b48d-19c2dc0255ae" {"component": "vcluster"}
syncer 2024-06-12 16:11:41 INFO event.rancher-7bcd45c474-nn962.17d84d3408e2c60b syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84d3408e2c60b {"component": "vcluster"}
syncer 2024-06-12 16:11:53 INFO event.rancher-7bcd45c474-nn962.17d84d3408e2c60b syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84d3408e2c60b {"component": "vcluster"}
syncer 2024-06-12 16:12:23 INFO event.rancher-7bcd45c474-nn962.17d84daac744d017 syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84daac744d017 {"component": "vcluster"}
syncer 2024-06-12 16:12:23 INFO event.rancher-7bcd45c474-nn962.17d84d3408e2c60b syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84d3408e2c60b {"component": "vcluster"}
syncer 2024-06-12 16:12:25 INFO event.rancher-7bcd45c474-nn962.17d84d3408e2c60b syncer/syncer.go:134 update virtual event cattle-system/rancher-7bcd45c474-nn962.17d84d3408e2c60b {"component": "vcluster"}
syncer 2024-06-12 16:12:53 INFO pod.rancher-7bcd45c474-nn962 syncer/syncer.go:134 update virtual pod cattle-system/rancher-7bcd45c474-nn962, because status has changed {"component": "vcluster"}
syncer 2024-06-12 16:13:06 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:13:06 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 1.051µs, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:13:10 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:13:10 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 84.974µs, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:13:20 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:13:20 ERROR filters/wrap.go:54 timeout or abort while handling: method=GET URI="/readyz" audit-ID="c2a4e875-32ad-413d-afa4-1a6a64b05a43" {"component": "vcluster"}
syncer 2024-06-12 16:13:20 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 650ns, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:13:22 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:13:22 ERROR filters/wrap.go:54 timeout or abort while handling: method=GET URI="/readyz" audit-ID="59c9b904-70ed-4be8-9305-0513c385dc95" {"component": "vcluster"}
syncer 2024-06-12 16:13:22 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 605ns, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:13:38 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:13:38 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 89.378µs, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:13:40 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:13:40 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 77.134µs, GET "/readyz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:13:40 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:13:40 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 427.339µs, GET "/healthz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:13:44 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:13:44 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 1.339µs, GET "/healthz" result: <nil> {"component": "vcluster"}
syncer 2024-06-12 16:15:28 ERROR handler/handler.go:49 Error while proxying request: context canceled {"component": "vcluster"}
syncer 2024-06-12 16:15:28 ERROR filters/timeout.go:142 post-timeout activity - time-elapsed: 32.065µs, GET "/readyz" result: <nil> {"component": "vcluster"}
Stream closed EOF for rancher-vcluster/rancher-vcluster-0 (vcluster)
The text was updated successfully, but these errors were encountered:
Summary
The rancher vcluster died for an unknown reaso, in the past it died due to resource constraints and needed to have the memory limit bumped up. This issue may have been a result of the PVC becoming over used? After checking I found the PVC to be 100% used and I bumped it up to 20 Gigs instead of 5. However the K3s cluster (and by extension Rancher) can no longer seem to start. I have posted what I believe are the relevant logs below. The cluster never finds its way back to a working condition.
Rancher
Rancher vcluster
The text was updated successfully, but these errors were encountered: