openssh bugs - Apr 2012 - [Bug 2000] New: when using ssh with ControlMaster/ControlPersist, one may get zombie processes

https://bugzilla.mindrot.org/show_bug.cgi?id=2000

             Bug #: 2000
           Summary: when using ssh with ControlMaster/ControlPersist, one
                    may get zombie processes
    Classification: Unclassified
           Product: Portable OpenSSH
           Version: 5.9p1
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: major
          Priority: P2
         Component: ssh
        AssignedTo: unassigned-bugs at mindrot.org
        ReportedBy: calestyo at scientia.net


Hi.

This is basically from:
https://dev.icinga.org/issues/2546
http://tracker.nagios.org/view.php?id=321

It was suggested there, that the actual problem may be in ssh, so I
open a bug here, too.

I basically asked before already at:
https://lists.mindrot.org/pipermail/openssh-unix-dev/2012-April/030379.html
but no one knew any advise.


What I do is using Icinga/Nagios and having checks on remote hosts
executed via ssh.
In order to dramatically speed checks up (from about 0,300 ms to 0,010
ms) I use ControlMaster = auto, which also makes the mux process
spawned on the first check.
As checks are typically sequentially scheduled I want the mux process
to persist but it should also go away automatically after some days if
not re-used (e.g. when I don't check a host anymore).
So I have something like ControlPersist 2d.

Now I stumbled across the following problem (and I'm actually not sure
whether it's a ssh issue or Icinga/Nagios):
The first time the check is done (which is when the mux process is
spawned) it times out.
The mux process keeps running and everything works on subsequent
checks.

The timeout is one enforced by Icinga/Nagios (60s), when it thinks the
command doesn't return.

I made some checks and the following turns out to happen on the FIRST
connection:
- executing the command on the remote side is actually done
- on the local side, the ssh process (or a wrapper shell script around)
becomes a zombie as soon as the remote command was executed
- after 60s, when Icinga/Nagios enforces it's timeout, the zombie goes
away
- the (local) mux process continues to run

The Zombie process is a child of icinga/nagios, while the mux process
(which is not a zombie) is a child of init.

Any ideas why this could happen? Is there perhaps something that lets
the parent processes notice that there is still a running child (i.e.
the mux process)?

I'll happily try out anything needed :)

Thanks,
Chris.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2000

Tomas Mraz <t8m at centrum.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |t8m at centrum.cz

--- Comment #1 from Tomas Mraz <t8m at centrum.cz> 2012-04-26 00:16:00 EST
---
Congrats, you won the B2K contest! :)

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2000

--- Comment #2 from Christoph Anton Mitterer <calestyo at scientia.net>
2012-04-26 00:40:08 EST ---
*G* What did I win?! The next release is named in my honours?! ;-P
(Actually I'd prefer help on that issue much more ;) )

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2000

--- Comment #3 from Tomas Mraz <t8m at centrum.cz> 2012-04-26 01:04:59 EST
---
I'd say that the primary issue is in nagios and/or icinga in how it
handles the child process execution and collection of the SIGCHLDs and
wait-ing for the exit statuses. However there might also play a role
stray stderr file descriptor which is left open on the mux process.

You can try to redirect the stderr to /dev/null when you're running the
ssh from the wrapper script.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2000

Christoph Anton Mitterer <calestyo at scientia.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID

--- Comment #4 from Christoph Anton Mitterer <calestyo at scientia.net>
2012-05-16 05:33:04 EST ---
I'd suspect you're right.

In the meantime I've made a small test prog, that just forks and does
nothing else special (which ssh could potentially do) and the problem
happens, too.

So I'll close the bug for now as invalid and reopen it in case anything
should be needed from ssh side.

Thanks,
Chris.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.