Enable config map update retry mechanism during NodeStageVolume

[siyanshen]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change

/kind bug

/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
This PR implements a retry mechanism in NodeStageVolume- Instead of failing the NodeStageVolume operation after a single attempt to update the ConfigMap, a retry logic should be introduced. This would ensure that the operation continues to attempt updates until it succeeds, reducing the chance of unnecessary failures.

Why we need this? Before this change, there is a case where the CSI driver attempts to mount multiple Filestore-based volumes. During this process, the CSI driver stores lock info in a ConfigMap, which tracks the mounted Filestore information for each node. However, this can lead to conflicts when multiple NodeStageVolume calls attempt to update the ConfigMap simultaneously. In such cases, the update may fail because the key-value pairs have already been modified by another process. The resulting error message may look like this:

rpc error: code = Internal desc = failed to store lock info after NodeStageVolume succeeded on volume xxx to path xxx: Operation cannot be fulfilled on configmaps "fscsi-gke-xxx": the object has been modified; please apply your changes to the latest version and try again

This is not an unrecoverable error, as the operation will self-resolve during a subsequent NodeStageVolume call and eventually succeed within milliseconds.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: siyanshen
Once this PR has been reviewed and has the lgtm label, please assign tyuchn for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @siyanshen. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tyuchn]

/ok-to-test

[tyuchn]

I think this is the only place (UPDATE configmap) we need to have a retry mechanism, rather than the entire node stage volume function.

[k8s-ci-robot]

@siyanshen: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gcp-filestore-csi-driver-e2e	af55b9b	link	true	/test pull-gcp-filestore-csi-driver-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.