After device is attached, devicepath of FC and iSCSI plugin are empty #54108
Description
Is this a BUG REPORT or FEATURE REQUEST?: BUG
/kind bug
What happened:
FC and iSCSI plugins have attacher interface, these plugins need to run attach operations at kubelet node. Therefore attach operations are called during volumeAttacher.WaitForAttach() call currently.
WaitForAttach() is called during GenerateMountVolumeFunc(), and in this method even if plugin attaches a volume to kubelet node, kubelet doesn't update devicePath into actualStateOfWorld by using MarkVolumeAsAttached().
As a result, we hit following errors during GenerateUnmountDeviceFunc() call.
E1017 19:44:49.165870 30726 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/fc/[50060e801049cfd1]:0\"" failed. No retries permitted until 2017-10-17 19:44:49.665850506 -0400 EDT (durationBeforeRetry 500ms). Error: "UnmountDevice.DeviceOpened failed for volume \"file-pv001\" (UniqueName: \"kubernetes.io/fc/[50060e801049cfd1]:0\") on node \"127.0.0.1\" : PathIsDevice failed for path \"\": stat : no such file or directory"
I1017 19:44:49.726788 30726 reconciler.go:283] operationExecutor.UnmountDevice started for volume "file-pv001" (UniqueName: "kubernetes.io/fc/[50060e801049cfd1]:0") on node "127.0.0.1"
W1017 19:44:49.727215 30726 mount.go:165] could not determine device for path: "/var/lib/kubelet/plugins/kubernetes.io/fc/50060e801049cfd1-lun-0"
W1017 19:44:49.727433 30726 util.go:94] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/fc/50060e801049cfd1-lun-0
E1017 19:44:49.727469 30726 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/fc/[50060e801049cfd1]:0\"" failed. No retries permitted until 2017-10-17 19:44:50.727452685 -0400 EDT (durationBeforeRetry 1s). Error: "UnmountDevice failed for volume \"file-pv001\" (UniqueName: \"kubernetes.io/fc/[50060e801049cfd1]:0\") on node \"127.0.0.1\" : fc: failed to detach disk: \nError: fc detach disk: invalid device name: "
Also this problem affects FC plugin's block volume support.
#51493
At the block volume support, we will have GenerateUnmapDeviceFunc() which has similar functionality to the GenerateUnmountDeviceFunc().
In this method, we need devicePath at step 2 & 3.
- Check global map path if symbolic links exist
- Remove loopback device which takes file descriptor lock for block device(/dev/sdX)
- Check DeviceOpened()
What you expected to happen:
Plugins which attach volume during WaitForAttach() can store devicePath into actualStateOfWorld properly.
How to reproduce it (as minimally and precisely as possible):
- Create FC PV/PVC then create pod with a PVC
- Delete the pod
Anything else we need to know?:
/cc @msau42 @jingxu97 @rootfs
Similar problem was reported from @rootfs in previous and then solution was proposed, but this wasn't merged.
#35433
Environment:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.1.1281+e3e2e24cc59161-dirty", GitCommit:"e3e2e24cc59161736be459abc2581f2a10b9252d", GitTreeState:"dirty", BuildDate:"2017-10-17T23:19:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration**:
- OS (e.g. from /etc/os-release): Fedora 25
- Kernel (e.g.
uname -a): 4.11.3-200.fc25.x86_64 - Install tools: ./hack/local-up-cluster.sh
- Others:
/sig storage
cc @kubernetes/sig-storage-bugs