Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Volume delete failed with static provisioning when persistentVolumeReclaimPolicy is Delete #1207

Closed
aparker-cityworks opened this issue Dec 5, 2023 · 6 comments · Fixed by #1507
Closed

Volume delete failed with static provisioning when persistentVolumeReclaimPolicy is Delete #1207

aparker-cityworks opened this issue Dec 5, 2023 · 6 comments · Fixed by #1507
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@aparker-cityworks
Copy link

/kind bug

What happened?
Persistent Volume deletion failed with error:

I1204 23:24:58.375475       1 controller.go:1502] delete "tenant-shared-efs-pv": started
E1204 23:24:58.376974       1 controller.go:1512] delete "tenant-shared-efs-pv": volume deletion failed: rpc error: code = NotFound desc = Failed to find access point for volume: fs-04653df1fc5fe9f42:/__shared-data
W1204 23:24:58.377321       1 controller.go:989] Retrying syncing volume "tenant-shared-efs-pv", failure 0
I1204 23:24:58.377688       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"tenant-shared-efs-pv", UID:"9ea5799b-f6fe-4084-aa22-26a985f6465d", APIVersion:"v1", ResourceVersion:"6113468", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' rpc error: code = NotFound desc = Failed to find access point for volume: fs-04653df1fc5fe9f42:/__shared-data
E1204 23:24:58.377733       1 controller.go:1007] error syncing volume "tenant-shared-efs-pv": rpc error: code = NotFound desc = Failed to find access point for volume: fs-04653df1fc5fe9f42:/__shared-data

What you expected to happen?
The PV should have been deleted successfully with the persistentVolumeReclaimPolicy set to Delete.

How to reproduce it (as minimally and precisely as possible)?
Create PV and PVC similar to the static provisioning/volume_path example, but set the persistentVolumeReclaimPolicy on the PV to Delete. When the PVC is deleted the controller will run the delete volume function and will fail with the above message.

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com

---
# PV with persistentVolumeReclaimPolicy: Delete
apiVersion: v1
kind: PersistentVolume
metadata:
    name: tenant-shared-efs-pv
spec:
    capacity:
        storage: 5Gi
    volumeMode: Filesystem
    accessModes:
        - ReadWriteMany
    storageClassName: efs-sc
    persistentVolumeReclaimPolicy: Delete
    csi:
        driver: efs.csi.aws.com
        volumeHandle: fs-04653df1fc5fe9f42:/__shared-data

---
# PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: tenant-shared-efs-claim
spec:
    accessModes:
        - ReadWriteMany
    storageClassName: efs-sc
    volumeName: tenant-shared-efs-pv
    resources:
        requests:
            storage: 5Gi

Environment

  • Kubernetes version (use kubectl version):
    Client Version: v1.28.2
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.28.3-eks-4f4795d

  • Driver version:
    1.7.1
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Dec 5, 2023
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 4, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 3, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

@k8s-ci-robot k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale May 3, 2024
@k8s-ci-robot
Copy link
Contributor

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mugdha-adhav
Copy link

We are facing the same issue where the volume deletion fails with below error -

controller.go:1558] "Volume deletion failed" err="rpc error: code = NotFound desc = Failed to find access point for volume: fs-foo" PV="bar"

But we are not using access points while creating static volumes.

If I understand correctly, the error is thrown from here: https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/pkg/driver/controller.go#L478

@mskanth972 mskanth972 reopened this Nov 19, 2024
@mskanth972
Copy link
Contributor

I am able to reproduce this, looks like a bug in the code, When we set persistentVolumeReclaimPolicy: Delete , the code is looking for Access point to delete which should happen only in case of Dynamic Provisioning.

@dankova22 dankova22 mentioned this issue Nov 21, 2024
Merged
seanzatzdev-amazon added a commit that referenced this issue Nov 26, 2024
@seanzatzdev-amazon
Merge pull request #1507 from dankova22/fix/1207-volume-delete-failed
35eb086 
Fix: Volume delete failed with static provisioning (#1207)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix: Volume delete failed with static provisioning (#1207) dankova22/aws-efs-csi-driver
5 participants
@k8s-ci-robot @aparker-cityworks @mugdha-adhav @k8s-triage-robot @mskanth972