openssh bugs - Jul 2004 - [Bug 897] scp doesn't clean up forked children when processing multiple files

http://bugzilla.mindrot.org/show_bug.cgi?id=897

           Summary: scp doesn't clean up forked children when processing
                    multiple files
           Product: Portable OpenSSH
           Version: 3.8p1
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: scp
        AssignedTo: openssh-bugs at mindrot.org
        ReportedBy: s.riehm at hse24.de


scp forks one ssh process for each file to be copied, but doesn't wait for
the processes until all files
have been processed.

Problem: if you are copying large numbers of files scp collects one zombie per
file copied. If the
number of files to be copied in one scp command excedes the number of processes
a user may have,
the user gets the message: "fork: resource temporarily unavailable"

See patch (very simple - just moved waitpid call into the respective for loops.
My Imperical tests worked
fine (Mac OS-X 10.3.4)



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=897





------- Additional Comments From s.riehm at hse24.de  2004-07-20 23:33 -------
Created an attachment (id=686)
 --> (http://bugzilla.mindrot.org/attachment.cgi?id=686&action=view)
patch to clean up forked processes before starting new ones




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=897

s.riehm at hse24.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #686 is|0                           |1
           obsolete|                            |



------- Additional Comments From s.riehm at hse24.de  2004-07-21 00:19 -------
Created an attachment (id=687)
 --> (http://bugzilla.mindrot.org/attachment.cgi?id=687&action=view)
problem only effects tolocal(). Previous patch broke on toremote copies

The previous patch changed both the tolocal and toremote routines. toremote
doesn't suffer the same problem (because there's only one target?) and
the
previous patch actually broke remote copied.
This is the minimal patch required to clean up child processes when copying to
the local host - all other operations are uneffected.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=897

s.riehm at hse24.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #687 is|0                           |1
           obsolete|                            |



------- Additional Comments From s.riehm at hse24.de  2004-07-21 01:47 -------
Created an attachment (id=688)
 --> (http://bugzilla.mindrot.org/attachment.cgi?id=688&action=view)
3rd attempt at a bug-free patch :-(

After testing with production scripts, another slight bug. I think I've got
it
now.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=897





------- Additional Comments From mouring at eviladmin.org  2004-07-21 10:18
-------
I don't see the behavior you claim on OS/X.  Nor have I seen this behavior
on any of the platforms I have
around me.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=897





------- Additional Comments From djm at mindrot.org  2004-07-21 11:20 -------
Could you please redo the patch as a unified diff ("diff -u")? Context
diffs are
unreadable :)



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=897

s.riehm at hse24.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #688 is|0                           |1
           obsolete|                            |



------- Additional Comments From s.riehm at hse24.de  2004-07-22 16:35 -------
Created an attachment (id=695)
 --> (http://bugzilla.mindrot.org/attachment.cgi?id=695&action=view)
...and now for something completely diff. The same again for -u




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=897





------- Additional Comments From s.riehm at hse24.de  2004-07-22 16:55 -------
Hi Ben,
you will only see the behaviour I mention if you copy multiple files from a
remote directory to a local
one. You should also make sure the files are large enough to keep scp busy for a
while, as soon as scp
has finished the OS cleans up and there's no record that anything was amiss.

Try something like this:

scp user at host:largefile1 user at host:largefile2 user at host:largefile3 user
at host:largefile4 user at host:
thebiggestfileyouvegot ~/tmp

then watch your processes with ps -u from another terminal - by the time scp is
copying
thebiggestfileyouvegot you will see 4 "(scp)" processes. These are
terminated child processes (they
finished their act properly and have left the building) which are waiting for
the parent (scp) to pick them
up at the backstage door (and check their exit code etc etc). These things are
called zombies and are
nothing more than an entry in the process table. The problem for me was that
OS-X (panther) only
allows 100 processes per user by default - copy about 80 files in a single scp
command and you'll get
the error: "fork: resource temporarily not available".
I hit this because I'm syncing an image library of around 150,000 images...
The patch waits for each ssh child before starting the next one, thus preventing
the accumulation of
zombies.

If you still don't see this behaviour, I'd be most interested in your
exact environment. This handling of
child processes is perfectly normal unix behaviour.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=897





------- Additional Comments From s.riehm at hse24.de  2004-08-03 20:02 -------
um... hello? The first responses were extremely quick. Now that I've
uploaded the patch the way you
like it and provided a detailed explanation of my motives, everything has gone
veeewwwyy quiet.

Did I step on someone's toes?



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.