Migration between two clusters that also have shared storage

[djjudas21]

I have a bare metal Kubernetes cluster with 2 CSI storage classes - hyperconverged Ceph, and off-cluster TrueNAS. The cluster is also configured for Velero backup to S3.

I now want to migrate to a new cluster using Velero to do its thing and recreate the new volumes. But, the new cluster is also connected to the same TrueNAS. For the volumes on TrueNAS, it will be way faster to create the existing PVs on the new cluster and magically have access to the data, rather than migrate them to and from and the same NAS via S3. I've already verified with the author of the TrueNAS CSI driver that it's possible to access my TrueNAS RWX PVs from two clusters.

I've made a basic diagram to show what I'm talking about.

Is it possible to do this? Obviously I can get Velero to make backups and exclude the TrueNAS PVs, but when I come to restore the workloads I guess I will need to manually create the PVs to point at my existing TrueNAS volumes, then do the restore, which will reinstate the PVCs and allow the workloads to mount the volumes.

[blackpiglet]

It's possible, but Velero only does that in a special case.

velero/pkg/restore/restore.go

Lines 1241 to 1247 in 804d73c

	} else {
	obj, err = ctx.handleSkippedPVHasRetainPolicy(obj, restoreLogger)
	if err != nil {
	errs.Add(namespace, err)
	return warnings, errs, itemExists
	}
	}

This is the logic that Velero can restore the PV with the volume as it was backed up.
It requires the PV's ReclaimPolicy set to Retain, and the PVs are not backed up by filesystem backup or snapshot backup.

This can be done by CLI velero backup create --default-volumes-to-fs-backup=false --snapshot-volumes=false