fix azure disk mount failure on coreos and some other distros #54334

andyzhangx · 2017-10-21T03:42:29Z

What this PR does / why we need it:
azure disk mount failure on coreos due to /dev/disk/azure not populated on coreos, and also on some other distros.
After a long time back and forth discussion, I have decided to fix this issue on k8s side. Customers are leaving azure because of this udev issue, you could read details from #50150 . I have verifed the code works well on current coreos on azure.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged):
fixes #50150: azure_dd: managed disks don't pass "FormatAndMount"
In coreos on azure and some old distros, one udev rule is not applied, which means /dev/disk/azure not populated, while current disk detection code logic relies on /dev/disk/azure, so it would lead to azure disk mount failure, for more details, pls refer to #50150

Special notes for your reviewer:
In v1.6, there is no such issue becuase it's not using /dev/azure/disk to detect the LUN of new disk, so I refer the code logic of v1.6:
https://github.com/kubernetes/kubernetes/blob/release-1.6/pkg/volume/azure_dd/vhd_util.go#L59-L66

while from v1.7, it's using /dev/azure/disk, the code logic does not apply to distro which does not have udev rules(/dev/azure/disk), see:
https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/volume/azure_dd/azure_common.go#L231-L236

Release note:

fix azure disk mount failure on coreos and some other distro due to udev rule not applied

/sig azure
@rootfs @brendanburns @colemickens
@tlyng @karataliu @szarkos

andyzhangx · 2017-10-21T04:25:16Z

/test pull-kubernetes-unit

andyzhangx · 2017-10-21T11:41:31Z

/test pull-kubernetes-kubemark-e2e-gce

andyzhangx · 2017-10-21T11:41:44Z

/test pull-kubernetes-unit

rootfs · 2017-10-21T13:42:56Z

pkg/volume/azure_dd/azure_common_linux.go

+					glog.Errorf("failed to parse target from %v (%v), err %v", arr[0], name, err)
+					continue
+				}
+				// as observed, targets 0-3 are used by OS disks. Skip them


I used to use this heuristics but it is not reliable: on certain instances (last time I used F instance and found this test was different than D instance), this test fails.

which test are you refering to? You mean ""/sys/bus/scsi/devices"" on different VM instance is different?

hi @rootfs I have tried coreos on Standard F1s VM, it works well. Do you have any other instance suggestion?

Paste all the info and logs on F instance, I attached two disks and try to detect Lun 1 disk:

azureuser@andy-coreosf /tmp $ ./detect-new-lun1 Azure sys disks paths: []there is nothing under /dev/disk/azure/ there is nothing under /dev/disk/azure/ there is nothing under /dev/disk/azure/ there is nothing under /dev/disk/azure/ /dev/sdd <nil> azureuser@andy-coreosf /tmp $ ls /sys/bus/scsi/devices -lt total 0 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 target0:0:0 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/00000000-0000-8899-0000-000000000000/host0/target0:0:0 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 target3:0:1 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/00000000-0001-8899-0000-000000000000/host3/target3:0:1 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 target5:0:0 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/f8b3781b-1e82-4818-a1c3-63d806ec15bb/host5/target5:0:0 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 5:0:0:0 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/f8b3781b-1e82-4818-a1c3-63d806ec15bb/host5/target5:0:0/5:0:0:0 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 5:0:0:1 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/f8b3781b-1e82-4818-a1c3-63d806ec15bb/host5/target5:0:0/5:0:0:1 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 host5 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/f8b3781b-1e82-4818-a1c3-63d806ec15bb/host5 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 3:0:1:0 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/00000000-0001-8899-0000-000000000000/host3/target3:0:1/3:0:1:0 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 host3 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/00000000-0001-8899-0000-000000000000/host3 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 host4 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/f8b3781a-1e82-4818-a1c3-63d806ec15bb/host4 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 0:0:0:0 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/00000000-0000-8899-0000-000000000000/host0/target0:0:0/0:0:0:0 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 host0 -> ../../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/00000000-0000-8899-0000-000000000000/host0 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 host1 -> ../../../devices/pci0000:00/0000:00:07.1/ata1/host1 lrwxrwxrwx. 1 root root 0 Oct 23 02:52 host2 -> ../../../devices/pci0000:00/0000:00:07.1/ata2/host2

as observed, targets 0-3 are used by OS disks, there are similar observations too

However, this observation failed when i was using an F8s instance running RHEL. And that gave rise to this fix

Hi @rootfs , thanks for the check.
The latest code already contains #44295, while current coreos issue is: since /dev/disk/azure/ is not populated in coreos, which means there would be no filtering, see: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_dd/azure_common_linux.go#L119-L124. In the end, findDiskByLun would return a wrong dev name which is actually os disk. So my fix is add back the skip logic if azureDisks is empty.
I have also tested on RHEL F instance, my PR works well. Let me know if there is any other issue, thx.

@andyzhangx thanks, please glog a message here.

@rootfs done

rootfs · 2017-10-21T13:44:19Z

setting consistent udev rules on coreos is a more robust fix

rootfs · 2017-10-21T13:47:46Z

WALinuxAgent should populate the same udev rules
https://raw.githubusercontent.com/Azure/WALinuxAgent/master/config/66-azure-storage.rules

andyzhangx · 2017-10-21T15:14:43Z

Hi @rootfs , I agree WALinuxAgent should create same udev rule, while current coreos image in azure does not have this udev rule, so lots of customers are complaining about azure disk mount failure of coreos. In this situation, I would suggest we should provide k8s side fix for this issue. You could refer to #50150 for more details, lots of customers complaining in the thread. add @jdumars for your comments, thx.

tlyng · 2017-10-23T09:10:57Z

The CoreOS I am using have this udev rules file already populated, does not fix anything in my case at least.

andyzhangx · 2017-10-23T09:22:33Z

@tlyng then it's maybe other issue, could you pls paste out the pod failure info into #50150 ? It's important for diagnostics. Thx.
Anyway I still would like to fix this issue in k8s side.

tlyng · 2017-10-24T11:38:58Z

@andyzhangx I don't know what is causing it, many times it works flawlessly. Sometimes when disks get detached it looks like the VM get stuck in

{
  "...": "...",
  "provisioningState" : "updating"
}

Why this happend is beyond my knowledge, perhaps the agent service? It renders the node useless at least. I currently have one node in this updating state (due to chart upgrade which moved the PVC from one node to another, source node still in "updating" state). Hopefully this times out, but I've seen it last for over 24h. In my current situation it has been waiting for 45 minutes - which is unacceptable.

andyzhangx · 2017-10-24T11:51:23Z

@tlyng could you paste your pod info into issue#50150 ?
kubectl describe YOUR-POD-NAME

This is a PR thread, would you discuss your issue in #50150 Or you may open a new issue, thx.

add log message

rootfs · 2017-10-25T18:55:20Z

/lgtm

k8s-github-robot · 2017-10-25T18:56:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx, rootfs

Associated issue: 50150

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~pkg/volume/azure_dd/OWNERS~~ [rootfs]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-10-25T22:16:10Z

Automatic merge from submit-queue (batch tested with PRs 54336, 54470, 54334, 54175). If you want to cherry-pick this change to another branch, please follow the instructions here.

…4334-upstream-release-1.8 Automatic merge from submit-queue. Automated cherry pick of #54334 Cherry pick of #54334 on release-1.8. #54334: fix#50150: azure disk mount failure on coreos

…4334-upstream-release-1.7 Automatic merge from submit-queue. Automated cherry pick of #54334 Cherry pick of #54334 on release-1.7. #54334: fix#50150: azure disk mount failure on coreos

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/azure size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 21, 2017

k8s-github-robot assigned saad-ali and thockin Oct 21, 2017

andyzhangx changed the title ~~fix#50150: azure disk mount failure on coreos and some other distros~~ fix: azure disk mount failure on coreos and some other distros Oct 21, 2017

andyzhangx changed the title ~~fix: azure disk mount failure on coreos and some other distros~~ fix azure disk mount failure on coreos and some other distros Oct 21, 2017

rootfs reviewed Oct 21, 2017

View reviewed changes

fix#50150: azure disk mount failure on coreos

db3c5a7

add log message

andyzhangx force-pushed the coreos-azure-fix branch from 6442102 to db3c5a7 Compare October 25, 2017 05:20

k8s-ci-robot assigned rootfs Oct 25, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 25, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 25, 2017

k8s-github-robot merged commit 59ad01f into kubernetes:master Oct 25, 2017

This was referenced Oct 26, 2017

Automated cherry pick of #54334 #54598

Merged

Automated cherry pick of #54334 #54599

Merged

andyzhangx mentioned this pull request Dec 28, 2017

unable to use azure disk in StatefulSet since /dev/sd* changed after detach/attach disk #57444

Closed

derekperkins mentioned this pull request Feb 3, 2018

Error using xfs filesystem Azure/AKS#162

Closed

andyzhangx deleted the coreos-azure-fix branch May 8, 2018 07:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix azure disk mount failure on coreos and some other distros #54334

fix azure disk mount failure on coreos and some other distros #54334

andyzhangx commented Oct 21, 2017 •

edited

Loading

andyzhangx commented Oct 21, 2017

andyzhangx commented Oct 21, 2017

andyzhangx commented Oct 21, 2017

rootfs Oct 21, 2017

andyzhangx Oct 21, 2017

andyzhangx Oct 23, 2017

andyzhangx Oct 23, 2017

rootfs Oct 23, 2017

andyzhangx Oct 24, 2017

rootfs Oct 24, 2017

andyzhangx Oct 25, 2017

rootfs commented Oct 21, 2017

rootfs commented Oct 21, 2017

andyzhangx commented Oct 21, 2017 •

edited

Loading

tlyng commented Oct 23, 2017

andyzhangx commented Oct 23, 2017

tlyng commented Oct 24, 2017

andyzhangx commented Oct 24, 2017

rootfs commented Oct 25, 2017

k8s-github-robot commented Oct 25, 2017

k8s-github-robot commented Oct 25, 2017

fix azure disk mount failure on coreos and some other distros #54334

fix azure disk mount failure on coreos and some other distros #54334

Conversation

andyzhangx commented Oct 21, 2017 • edited Loading

andyzhangx commented Oct 21, 2017

andyzhangx commented Oct 21, 2017

andyzhangx commented Oct 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rootfs commented Oct 21, 2017

rootfs commented Oct 21, 2017

andyzhangx commented Oct 21, 2017 • edited Loading

tlyng commented Oct 23, 2017

andyzhangx commented Oct 23, 2017

tlyng commented Oct 24, 2017

andyzhangx commented Oct 24, 2017

rootfs commented Oct 25, 2017

k8s-github-robot commented Oct 25, 2017

k8s-github-robot commented Oct 25, 2017

andyzhangx commented Oct 21, 2017 •

edited

Loading

andyzhangx commented Oct 21, 2017 •

edited

Loading