For a couple of years people have been reporting intermittent problems with rsync over ssh freezing during a large transfer (typically several GB). I have now gotten to the bottom of these problems, and have written a small test program which demonstrates a deadlock in sshd when doing large bi-directional transfers. Luckily the problem is easily solved by modifying sshd to use socketpair() instead of pipe(). Just remove "#define USE_PIPES 1" from near the bottom of includes.h does the trick. The test program, along with instructions for use and an explanation of some of the pipe() bugs in various OSes that make this change necessary is available from ftp://samba.org/pub/tridge/misc/sshtest.c In past releases of rsync I added various buffering tricks in rsync which reduced the likelyhood of this problem occuring, but didn't eliminate it completely. I have removed those tricks in rsync 2.4.0 so you may find an increase in the number of bug reports regarding rsync over ssh. I removed the tricks as they caused excessive memory usage (to the point of memory starvation) in some cases and are completely unnecessary once sshd is modified to use socketpair(). I think the combination of pipe/select bugs on various OSes means that you can't work around this problem by playing with non-blocking pipes or the size of writes (except by dropping the size by 1). I'm also amazed at the number of pipe/select bugs in the various OSes I tested - Linux 2.2 was particularly bad (fixed in 2.3). I really expected pipes to be pretty well defined and bug-free by now! Cheers, Tridge PS: I am not on this list, I'm just using this address as noted in the README.
[Removed tridge at linuxcare.com, and thanked him for a bit of help with it on NeXT] Has anyone tried this with OpenSSH? I know I ran it for about 5 - 10 minutes and it never deadlocked. Wondering if anyone else had any problems. On Sat, 29 Jan 2000 tridge at linuxcare.com wrote:> For a couple of years people have been reporting intermittent problems > with rsync over ssh freezing during a large transfer (typically > several GB). I have now gotten to the bottom of these problems, and > have written a small test program which demonstrates a deadlock in > sshd when doing large bi-directional transfers. Luckily the problem is > easily solved by modifying sshd to use socketpair() instead of > pipe(). Just remove "#define USE_PIPES 1" from near the bottom of > includes.h does the trick. > > The test program, along with instructions for use and an explanation > of some of the pipe() bugs in various OSes that make this change > necessary is available from ftp://samba.org/pub/tridge/misc/sshtest.c > > In past releases of rsync I added various buffering tricks in rsync > which reduced the likelyhood of this problem occuring, but didn't > eliminate it completely. I have removed those tricks in rsync 2.4.0 so > you may find an increase in the number of bug reports regarding rsync > over ssh. I removed the tricks as they caused excessive memory usage > (to the point of memory starvation) in some cases and are completely > unnecessary once sshd is modified to use socketpair(). > > I think the combination of pipe/select bugs on various OSes means that > you can't work around this problem by playing with non-blocking pipes > or the size of writes (except by dropping the size by 1). > > I'm also amazed at the number of pipe/select bugs in the various OSes > I tested - Linux 2.2 was particularly bad (fixed in 2.3). I really > expected pipes to be pretty well defined and bug-free by now! > > Cheers, Tridge > > PS: I am not on this list, I'm just using this address as noted in the > README. >
did you test pipe() or socketpair() ? On Sat, Jan 29, 2000 at 07:53:53PM -0600, Ben Lindstrom wrote:> [Removed tridge at linuxcare.com, and thanked him for a bit of help with > it on NeXT] > > Has anyone tried this with OpenSSH? I know I ran it for about 5 - 10 > minutes and it never deadlocked. Wondering if anyone else had any > problems.
The default which is pipe(). I never tried it with socketpair() at this point. On Sun, 30 Jan 2000, Markus Friedl wrote:> > did you test pipe() or socketpair() ? > > On Sat, Jan 29, 2000 at 07:53:53PM -0600, Ben Lindstrom wrote: > > [Removed tridge at linuxcare.com, and thanked him for a bit of help with > > it on NeXT] > > > > Has anyone tried this with OpenSSH? I know I ran it for about 5 - 10 > > minutes and it never deadlocked. Wondering if anyone else had any > > problems. >
On Sat, 29 Jan 2000 tridge at linuxcare.com wrote:> For a couple of years people have been reporting intermittent problems > with rsync over ssh freezing during a large transfer (typically > several GB). I have now gotten to the bottom of these problems, and > have written a small test program which demonstrates a deadlock in > sshd when doing large bi-directional transfers. Luckily the problem is > easily solved by modifying sshd to use socketpair() instead of > pipe(). Just remove "#define USE_PIPES 1" from near the bottom of > includes.h does the trick.Behaviour (and fix) verified under Linux 2.2.12. OpenBSD 2.6 + errata seems not be affected. Can anyone detect any problems when using socket pairs? If not, I shall make it the default. Regards, Damien -- | "Bombay is 250ms from New York in the new world order" - Alan Cox | Damien Miller - http://www.mindrot.org/ | Email: djm at mindrot.org (home) -or- djm at ibs.com.au (work)