Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

defunct processes, the possible explanation #74

Open
sterchelen opened this issue Jun 21, 2022 · 0 comments · May be fixed by #85
Open

defunct processes, the possible explanation #74

sterchelen opened this issue Jun 21, 2022 · 0 comments · May be fixed by #85

Comments

@sterchelen
Copy link

sterchelen commented Jun 21, 2022 •
edited
Loading

introduction

The following explanation is focused on using csi-s3 with goofys as a backend. All the components are in their latest version.
The issue I stumbled upon is the number of goofys Zombie processes.
The number doesn't have any importance in the understanding.

explanation

I looked in the csi-s3 code and more importantly at the FuseUnmount function and then at waitForProcess

csi-s3/pkg/mounter/mounter.go

Lines 133 to 156 in ddbd6fd

func waitForProcess(p *os.Process, backoff int) error {
if backoff == 20 {
return fmt.Errorf("Timeout waiting for PID %v to end", p.Pid)
}
cmdLine, err := getCmdLine(p.Pid)
if err != nil {
glog.Warningf("Error checking cmdline of PID %v, assuming it is dead: %s", p.Pid, err)
return nil
}
if cmdLine == "" {
// ignore defunct processes
// TODO: debug why this happens in the first place
// seems to only happen on k8s, not on local docker
glog.Warning("Fuse process seems dead, returning")
return nil
}
if err := p.Signal(syscall.Signal(0)); err != nil {
glog.Warningf("Fuse process does not seem active or we are unprivileged: %s", err)
return nil
}
glog.Infof("Fuse process with PID %v still active, waiting...", p.Pid)
time.Sleep(time.Duration(backoff*100) * time.Millisecond)
return waitForProcess(p, backoff+1)
}

Due to the name of the function I was expected to see a wait4 syscall to consume the child process, in our case goofys.
If we look at the below outputs:

  • we have a goofys Zombie process with pid=32767
$ ps aux | grep goofys
root     32767  0.0  0.0      0     0 ?        Zs   Jun14   0:00 [goofys] <defunct>
  • its parent process the s3driver
$ pstree -s 32767
systemd───containerd-shim───s3driver───goofys

As s3driver launches goofys backend (I guess it is the case for the other backends 🤷🏼‍♂️), s3driver is the parent process. Then as a good parent 😃 it should wait4 its child to know what was its status.

In other words, there is a leak on child termination. The fix should be trivial; in the waitForProcess when the cmdLine is empty, we have to syscall.wait4 on the given pid.

csi-s3/pkg/mounter/mounter.go

Lines 142 to 148 in ddbd6fd

if cmdLine == "" {
// ignore defunct processes
// TODO: debug why this happens in the first place
// seems to only happen on k8s, not on local docker
glog.Warning("Fuse process seems dead, returning")
return nil
}

wdyt @ctrox?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for custom rclone options and fix defuct processes ppacher/csi-s3
1 participant
@sterchelen