Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubelet Unable to attach or mount volumes #94

Open
cnzf1 opened this issue Jan 29, 2024 · 4 comments
Open

kubelet Unable to attach or mount volumes #94

cnzf1 opened this issue Jan 29, 2024 · 4 comments

Comments

@cnzf1
Copy link

cnzf1 commented Jan 29, 2024 •
edited
Loading

how should i do ,if I want to use static pvc with backend minio in kubernetes v1.22?

Now , I meet an error:
Warning FailedMount 2m6s (x6 over 13m) kubelet Unable to attach or mount volumes: unmounted volumes=[webroot], unattached volumes=[webroot kube-api-access-ttwx2]: timed out waiting for the condition
Warning FailedAttachVolume 52s (x7 over 13m) attachdetach-controller AttachVolume.Attach failed for volume "mypv" : Attach timeout for volume data

# pvc.yaml:
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ch.ctrox.csi-s3-driver 
    volumeHandle: data
    volumeAttributes:
      endpoint: "http://127.0.0.1:9000"
      bucket: "data"
      accessKey: "5CJABSyn48fLAucMEUmT"
      secretKey: "4tWFfPpKKK2gmEpcig5bkDUlvRZJqm6D0fjBP7Iu"
---
# pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-s3
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: ""
  volumeName: mypv
@cnzf1 cnzf1 changed the title how should i do ,if I want to use static pvc with backend minio in kubernetes v1.22? kubelet Unable to attach or mount volumes Feb 2, 2024
@manju-rn
Copy link

manju-rn commented Feb 22, 2024 •
edited
Loading

@cnzf1
The problem is with the attacher - in the attacher.yaml - update the image tag to new one
image: quay.io/k8scsi/csi-attacher:v3.1.0

@msaustral
Copy link

msaustral commented Mar 13, 2024 •
edited
Loading

Hi we have same issue,
attacher, provisioner, csi-s3, secret and storage class are on namespace kube-system

PVC is on another namespace

on the attacher we have this error:

I0313 23:17:16.924292       1 controller.go:198] Started VA processing "csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19"
I0313 23:17:16.924323       1 trivial_handler.go:53] Trivial sync[csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19] started
I0313 23:17:16.924328       1 util.go:37] Marking as attached "csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19"
W0313 23:17:16.927907       1 trivial_handler.go:57] Error saving VolumeAttachment csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19 as attached: volumeattachments.storage.k8s.io "csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19" is forbidden: User "system:serviceaccount:kube-system:csi-attacher-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io" at the cluster scope

on the pod we get this error:

AttachVolume.Attach failed for volume "pvc-9b8dba18-c486-43fd-8240-8f1f9dba8438" : timed out waiting for external-attacher of ch.ctrox.csi.s3-driver CSI driver to attach volume backup-k8n/pvc-9b8dba18-c486-43fd-8240-8f1f9dba8438

we delete the attacher, change

image: quay.io/k8scsi/csi-attacher:v3.1.0
image: quay.io/k8scsi/csi-node-driver-registrar:v2.1.0
image: quay.io/k8scsi/csi-provisioner:v2.1.2

and re-create all

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: csi-s3-secret
stringData:
  accessKeyID: xxxxxxxxxxxxxx
  secretAccessKey: xxxxxxxxxxxxxxxxxx
  endpoint: https://sfo3.digitaloceanspaces.com
  region: ""
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: csi-s3
provisioner: ch.ctrox.csi.s3-driver
parameters:
  # specify which mounter to use
  # can be set to rclone, s3fs, goofys or s3backer
  mounter: s3fs
  bucket: backup-k8n
  csi.storage.k8s.io/provisioner-secret-name: csi-s3-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
  csi.storage.k8s.io/node-stage-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-stage-secret-namespace: kube-system
  csi.storage.k8s.io/node-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-s3-pvc
  namespace: xxx-prod
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 256Gi
  storageClassName: csi-s3

the PVC is Bound and created on the S3 platform, the only issue is to attached it to the container on creation

any idea?

@msaustral
Copy link

Hi, we solve the issue by changing the ClusterRole of the attacher from

- apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update", "patch"]

to

- apiGroups: ["storage.k8s.io"]
    resources: ["*"]
    verbs: ["get", "list", "watch", "update", "patch"]

@msaustral msaustral mentioned this issue Mar 13, 2024
Open
@Alestrix
Copy link

Alestrix commented Oct 6, 2024

See Issue #80, the solution is described here: #80 (comment)
Worked also for me (I deployed csi-s3, including the secret, in a dedicated namespace and it all works after updating the references)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Alestrix @cnzf1 @manju-rn @msaustral