Mayastor is removing volumes with set persistentVolumeReclaimPolicy Retain

[fara-tode]

Describe the bug
when setting pv with Retain and delete this pv afterwards, mayastor is removing completely this volume.
This is not happening on my other installations of mayastor, also was not occuring on this installation but one day it just started.

To Reproduce

create pvc
kubectl patch pv pv-123213  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
kubectl delete pvc ...
kubetcl delete pv pv-123213

It may be hard to reproduce, as it just started one day to occur on one of my mayastor deployment.

Expected behavior
When PV with retain is deleted, data is not be deleted from mayastor.

Screenshots
If applicable, add screenshots to help explain your problem.

** OS info (please complete the following information):**
Distro: Ubuntu 24.04.1 LTS
Kernel version: 6.8.0-47-generic
MayaStor revision or container image: openebs.io/version=2.7.0

Additional context
adding dump from working and nonworking installation.

mayastor-2024-12-18--13-39-19-UTC-notwork.tar.gz
mayastor-2024-12-18--13-42-41-UTC-work.tar.gz

[tiagolobocastro]

If you delete the PV then yes we are also deleting the corresponding mayastor volume IIRC.

Is there any reason why you'd want to keep the mayastor volume when PV is deleted? We could provide a way of disabling this behavior.

[fara-tode]

Yes, i want to change node affinity on pv definition. So i'd like to remove it, then recreate it, but when is set to retain I expect the data will not be deleted from maya.

And that is pretty weird that it was not removed before, and it is not on my other clusters, but now it is on one.

[tiagolobocastro]

Looks like it runs on a 10 minute timer by default, so it might explain why you haven't seen it before.
This was mainly used as a work around for a K8s bug which I believe has been fixed now so I suggest we make this disabled by default and configurable via helm just in case.
Maybe we can get this in for v2.7.2

[fara-tode]

I don't think that 10min timer is a case as the volume from mayastor is removed within second every time.

[fara-tode]

is there any workaround for this ? I did a test on both clusters:

create pvc
kubectl patch pv pv-123213  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
kubectl delete pvc ...
kubetcl delete pv pv-123213
-- check if pv in mayastor
kubectl mayastor get volume-replica-topologies -n openebs | grep ..

on the one where is the issue, pv is immediately removed within 1sec from mayastor.
one the 2nd where no issue pv is retained. Volume is deleted after exactly 10 minutes.

Is there any bug that I encounter on one of my cluster?

[tiagolobocastro]

I'm able to reproduce.
So the 10 minutes is for a separate scenario: #1269

The issue you're seeing here is because we're explicitly listening on change events and deleting volumes: openebs/mayastor-control-plane#283
I think maybe we should not be doing this at all, and should instead run it on-demand.

There's no work-around for this without code changes.
I think we should probably release a new patch to address this, maybe disable this logic and in future minor add an on-demand GC? @abhilashshetty04

[fara-tode]

that's still weird as I'm doing exact same scenario on both. So immediate deletion is correct behavior according to current implementation for this scenario and 10mins not?

[fara-tode]

according to k8s doc : https://kubernetes.io/docs/concepts/storage/persistent-volumes/
If i read this correctly when the PV with Retain policy is deleted the data should persist and in this case be not deleted from mayastor.

Retain:

Delete the PersistentVolume. The associated storage asset in external infrastructure still exists after the PV is deleted.
Manually clean up the data on the associated storage asset accordingly.
Manually delete the associated storage asset.
If you want to reuse the same storage asset, create a new PersistentVolume with the same storage asset definition.

[tiagolobocastro]

Indeed, I've raised a PR to disable it on 2.7
We'll release a patch with this and another bug we've found

[abhilashshetty04]

@fara-tode , Current fix disables the garbage collection completely. We will have a helm arg in future release if you want to enable it explicitly. Also we will expose a delete volume command in plugin in case you want to manually cleanup such volumes (when garbage collection is not enabled).

[fara-tode]

any plans to release openebs chart 4.1.3 or should I just use 4.1.2 updating mayastor dependency to 2.7.3 ?

[avishnu]

any plans to release openebs chart 4.1.3 or should I just use 4.1.2 updating mayastor dependency to 2.7.3 ?

Next week is the plan.