/test pull-kubernetes-cross

@andyzhangx is this something we need for 1.14? can you talk a little bit about how a user could hit this issue (under what conditions)?
cc: @craiglpeters

hi @michmike I could easily repro this issue by "kubectl create -f https://raw.githubusercontent.com/andyzhangx/demo/master/windows/azurefile/aspnet-azurefile-multiple-replicas.yaml", the error is like following:

Events:
  Type     Reason       Age    From                 Message
  ----     ------       ----   ----                 -------
  Normal   Scheduled    2m28s  default-scheduler    Successfully assigned default/deployment-azurefile-7c8599c8c8-4fbbn to 1426k8s001
  Warning  FailedMount  2m17s  kubelet, 1426k8s001  MountVolume.SetUp failed for volume "pvc-b731c131-46d2-11e9-ba1f-000d3aa28f1b" : Remove-SmbGlobalMapping failed: exit status 1, output: "Remove-SmbGlobalMapping : No MSFT_SmbGlobalMapping objects found with property 'RemotePath' equal to \r\n'\\\\fa33e7109465b11e98xxxx.file.core.windows.net\\andy-win1134-dynamic-pvc-b731c131-46d2-11e9-ba1f-000d3aa28f1b'.  \r\nVerify the value of the property and retry.\r\nAt line:1 char:1\r\n+ Remove-SmbGlobalMapping -RemotePath $Env:smbremotepath -Force\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : ObjectNotFound: (\\\\fa33e7109465b...1f-000d3xxx:String) [Remove-SmbGlobalMapping], Ci \r\n   mJobException\r\n    + FullyQualifiedErrorId : CmdletizationQuery_NotFound_RemotePath,Remove-SmbGlobalMapping\r\n \r\n"
  Warning  FailedMount  2m12s  kubelet, 1426k8s001  MountVolume.SetUp failed for volume "pvc-b731c131-46d2-11e9-ba1f-000d3aa28f1b" : Remove-SmbGlobalMapping failed: exit status 1, output: "Remove-SmbGlobalMapping : This network connection does not exist. \r\nAt line:1 char:1\r\n+ Remove-SmbGlobalMapping -RemotePath $Env:smbremotepath -Force\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : NotSpecified: (MSFT_SmbGlobalM...le.core.win...):ROOT/Microsoft/...mbGlobalMapping) [Rem \r\n   ove-SmbGlobalMapping], CimException\r\n    + FullyQualifiedErrorId : Windows System Error 2250,Remove-SmbGlobalMapping\r\n \r\n"
  Normal   Pulling      96s    kubelet, 1426k8s001  pulling image "mcr.microsoft.com/dotnet/framework/aspnet:4.7.2-windowsservercore-ltsc2019"

And I have verified this PR has fixed this issue. It's better make it into v1.14 if possible.
/test pull-kubernetes-cross

cc @harshaperera532

/test pull-kubernetes-cross

i pinged sig-storage
cc: @kubernetes/sig-storage-pr-reviews

I can't say anything about Windows, but the code looks OK.
/approve

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx, jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

/milestone v1.14

I have a few questions

1. Why we hit this issue in 1.13? Do we have the same issue before?

2. So in windows, there will be race condition if user tries to mount with the same target? But according to your example, isn't the mount target different under different pod volume directory?

3. How often you hit this problem, I tried the similar deployment, but could not reproduce it.

Why we hit this issue in 1.13? Do we have the same issue before?

I introduced a fix for remount same smb path issue on windows: #73661, that fix introduces this issue, it's not a regression btw since original remount on same smb path does not work.

So in windows, there will be race condition if user tries to mount with the same target? But according to your example, isn't the mount target different under different pod volume directory?

I used 6 replicas in my example, though those pod has different mount dirs, while they point to same smb path, that will cause race condition.

How often you hit this problem, I tried the similar deployment, but could not reproduce it.

If you use same smb path with multiple pods in parallel, I think it could reproduce easily.

@jingxu97 Let me explain a little more here. On Linux, every smb mount is independent even mount with same smb path, so there won't be race condition, while on Windows, we are using SMB Global Mapping, same smb path shares the smb mount, so we need to add lock for this entry.

@andyzhangx Is this PR still planned to be merged in time for 1.14? I'd like to remind that Code Thaw is starting tomorrow (Tuesday PST EOD), so it would be nice to merge this PR by then, otherwise it'll compete with herd of incoming PRs post-thaw.

@xmudrii yes, it should be merged into v1.14
could someone lgtm, thanks. @jingxu97 @msau42

@neolit123 can you please take a look as well?

/lgtm
/hold
Before removing the hold I'd like to hear back from @jingxu97 to understand if their questions have been sufficiently answered

/lgtm

/hold cancel

thank you all for a quick review and approval!

/lgtm