thr3ads.net - rsync - rsync 2.6.2 hang (was rsync 2.6.2 crash) [Sep 2004]

If this information is useful, please help other people find it:
Share via:

jim

2004-Sep-28 22:18 UTC

rsync 2.6.2 hang (was rsync 2.6.2 crash)

OK, I set up a new PC with a fresh XP load to be my rsync server. Rsync by
itself seems
to work fine (thank God. I'm going nuts.)

I am still having problems when I throw ssh into the mix. At least now, things
don't
crash, but they do hang.  The file list gets build, the transfer starts and it
gets through
maybe a few hundred files, but then everything stops. All the processes are
still there,
but all logging stops and CPU utilization by the rsync and ssh processes go to
zero.

Any thoughts?



   -------Original Message-------
   > From: jim <jim@thegallaghers.biz>
   > Subject: Re: rsync 2.6.2 crash
   > Sent: 24 Sep 2004 16:19:02
   >
   >  I switched both ends back to 2.6.2 and started rsync in daemon mode by
hand (no ssh). The sync is still in progress, but it looks like it is working.
There were three small problems, however:
   >  
   >  2004/09/24 14:25:34 [1896] opendir "/17B - CONTROL/17Bxxxxx -
xxxxxxx xxxx, xxx
   >  xxxxx xxx-297 A24A-53A/Drawing Sheet 17Bxxxxxx -(44).cal" (in
MPDM) failed: Not
   >  a directory
   >  2004/09/24 14:25:51 [1896] file has vanished: "/17B -
CONTROL/17Bxxxxxx - xxxxxx
   >  x xxxx, xxx xxxxx xxx-297 A24A-53A/Drawing Sheet 17Bxxxxxx
-(89).cal" (in MPDM)
   >  2004/09/24 14:29:28 [1896] opendir "/17B - CONTROL/17Bxxxxxx -
xxxx, xxxxxXxxx-x
   >  xxxxxxxxxx/Drawing Sheet 17B1N5001 B(2.2).cal" (in MPDM) failed:
Not a directory
   >  
   >

Chuck Wolber

2004-Sep-29 04:13 UTC

head link

rsync 2.6.2 hang (was rsync 2.6.2 crash)

On Tue, 28 Sep 2004, jim wrote:
> OK, I set up a new PC with a fresh XP load to be my rsync server. Rsync 
> by itself seems to work fine (thank God. I'm going nuts.)
> 
> I am still having problems when I throw ssh into the mix. At least now, 
> things don't crash, but they do hang.  The file list gets build, the 
> transfer starts and it gets through maybe a few hundred files, but then 
> everything stops. All the processes are still there, but all logging 
> stops and CPU utilization by the rsync and ssh processes go to zero.
> 
> Any thoughts?
Install Linux instead of XP? Seriously, I gotta wonder why you are serving 
rsync in an environment that it's really not meant to be used in?

FWIW; We use rsync (under Linux) to back up folders on Windows machines 
quite a bit. Works great.

-Chuck


-- 
http://www.quantumlinux.com 
 Quantum Linux Laboratories, LLC.
 ACCELERATING Business with Open Technology

 "The measure of the restoration lies in the extent to which we apply 
  social values more noble than mere monetary profit." - FDR

jim

2004-Sep-29 05:24 UTC

head link

rsync 2.6.2 hang (was rsync 2.6.2 crash)

Hey, that's pretty funny. Actually, I would much prefer a Linux environment,
but I don't
always get what I want. We are primarily a Windows and HP-UX shop, so it's
not so
easy get the powers that be to buy into a Linux box.  In addition, I'm using
several
sets of in-house libraries for this project that have not been built on Linux. I
could
easily burn another couple of months setting up build environments and testing
the Linux builds. Not going to happen at this point. 

Our low-end server environment is W2K3, so that's what I'm stuck with. 


   -------Original Message-------
   > From: Chuck Wolber <chuckw@quantumlinux.com>
   > Subject: Re: rsync 2.6.2 hang (was rsync 2.6.2 crash)
   > Sent: 28 Sep 2004 21:13:47
   >
   >  On Tue, 28 Sep 2004, jim wrote:
   >  
   >  > OK, I set up a new PC with a fresh XP load to be my rsync server.
Rsync
   >  > by itself seems to work fine (thank God. I'm going nuts.)
   >  >
   >  > I am still having problems when I throw ssh into the mix. At least
now,
   >  > things don't crash, but they do hang.  The file list gets
build, the
   >  > transfer starts and it gets through maybe a few hundred files, but
then
   >  > everything stops. All the processes are still there, but all
logging
   >  > stops and CPU utilization by the rsync and ssh processes go to
zero.
   >  >
   >  > Any thoughts?
   >  
   >  Install Linux instead of XP? Seriously, I gotta wonder why you are
serving
   >  rsync in an environment that it's really not meant to be used in?
   >  
   >  FWIW; We use rsync (under Linux) to back up folders on Windows machines
   >  quite a bit. Works great.
   >  
   >  -Chuck
   >  
   >  
   >  --
   >  http://www.quantumlinux.com
   >  Quantum Linux Laboratories, LLC.
   >  ACCELERATING Business with Open Technology
   >  
   >  "The measure of the restoration lies in the extent to which we
apply
   >    social values more noble than mere monetary profit." - FDR
   -------Original Message-------

Wayne Davison

2004-Sep-29 08:12 UTC

head link

rsync 2.6.2 hang (was rsync 2.6.2 crash)

On Tue, Sep 28, 2004 at 03:19:43PM -0700, jim wrote:> I am still having problems when I throw ssh into the mix. At least
> now, things don't crash, but they do hang.
Yes, others have experienced hangs when using rsync over ssh under
cygwin.  If possible, stick with the rsync daemon protocol for cygwin
transfers and it should be fine.

We haven't had any diagnostic reports yet that would help us to narrow
down the hang, though, so feel free to attach a debugger to the hanging
process(es) and let us know what they're up to at the time of the hang
(if you'd like).

..wayne..

jim

2004-Sep-29 17:07 UTC

head link

rsync 2.6.2 hang (was rsync 2.6.2 crash)

Do you have a gut feeling as to where I should look first, or what the 
issue might be? Do you suspect rsync or ssh as the likely culprit?
>Yes, others have experienced hangs when using rsync over ssh under
>cygwin.  If possible, stick with the rsync daemon protocol for cygwin
>transfers and it should be fine.
>
>We haven't had any diagnostic reports yet that would help us to narrow
>down the hang, though, so feel free to attach a debugger to the hanging
>process(es) and let us know what they're up to at the time of the hang
>(if you'd like).
>
>..wayne..

jim

2004-Sep-29 18:27 UTC

head link

rsync 2.6.2 hang (was rsync 2.6.2 crash)

OK, I'll try it. 

One more question.  Does rsync intentionally behave differently if it's
talking
over ssh vs directly over a socket? 



   -------Original Message-------
   > From: Wayne Davison <wayned@samba.org>
   > Subject: Re: rsync 2.6.2 hang (was rsync 2.6.2 crash)
   > Sent: 29 Sep 2004 10:20:11
   >
   >  On Wed, Sep 29, 2004 at 10:08:16AM -0700, jim wrote:
   >  > Do you have a gut feeling as to where I should look first, or what
the
   >  > issue might be? Do you suspect rsync or ssh as the likely culprit?
   >  
   >  It's hard to know, but I'd suggest starting by assuming
it's rsync's
   >  fault and going from there.  The easiest thing to do is to compile
rsync
   >  with debug symbols, run that until it hangs, and then attach either
   >  strace and/or gdb to each running process to see what it is doing.  (In
   >  gdb you can get a "bt" backtrace, possibly go "up"
a stack frame or two
   >  into the rsync code and see what exactly is happening.)  If it is
   >  rsync's fault, rsync is probably trying to output some data down a
pipe
   >  that is full, while ignoring some data on an input pipe that it needs
to
   >  read to avoid deadlock.  These situations have been sorted out in the
   >  past, but there my still be something that hasn't yet been solved.
   >  
   >  ..wayne..
   -------Original Message-------

jim

2004-Sep-30 18:17 UTC

head link

rsync 2.6.2 hang (was rsync 2.6.2 crash)

OK, I installed 2.6.3pre2 on both the client and server. The server is run
via a ssh command option in authorized keys: command="rsync --server
--daemon ."
The client command is: rsync -Pvvvtr --rsh="ssh -l mpdm -i
ADMIN/rsa-mpdm01" pcls046147::"MPDM/subdir" .

For this test, there were 2 client rsync processes and 1 server process at the
time of the hang.
Here are the 3 sets of traces, server, client, and client child:

$ gdb /usr/bin/rsync 2808
GNU gdb 2003-09-20-cvs (cygwin-special)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i686-pc-cygwin"...
Attaching to program `/usr/bin/rsync.exe', process 3288
[Switching to thread 3288.0xe40]
(gdb) info threads
* 4 thread 3288.0xe40  0x7c901231 in ntdll!DbgUiConnectToDbg ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
  3 thread 3288.0xd0  0x7c90eb94 in ntdll!LdrAccessResource ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
  2 thread 3288.0xa04  0x7c90eb94 in ntdll!LdrAccessResource ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
  1 thread 3288.0xb24  0x7c90eb94 in ntdll!LdrAccessResource ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
(gdb) bt
#0  0x7c901231 in ntdll!DbgUiConnectToDbg ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#1  0x7c9507a8 in ntdll!KiIntSystemCall ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#2  0x00000005 in ?? ()
#3  0x00000004 in ?? ()
#4  0x00000001 in ?? ()
#5  0x00c2ffd0 in ?? ()
#6  0xf8dc31e8 in ?? ()
#7  0xffffffff in ?? ()
#8  0x7c90ee18 in strchr () from /cygdrive/c/WINDOWS/system32/ntdll.dll
#9  0x7c9507c8 in ntdll!KiIntSystemCall ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
(gdb) thread 3
[Switching to thread 3 (thread 3288.0xd0)]#0  0x7c90eb94 in ntdll!LdrAccessResou
rce () from /cygdrive/c/WINDOWS/system32/ntdll.dll
(gdb) bt
#0  0x7c90eb94 in ntdll!LdrAccessResource ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#1  0x7c90d85c in ntdll!ZwDelayExecution ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#2  0x7c8023ed in SleepEx () from /cygdrive/c/WINDOWS/system32/kernel32.dll
(gdb) thread 2
[Switching to thread 2 (thread 3288.0xa04)]#0  0x7c90eb94 in ntdll!LdrAccessReso
urce () from /cygdrive/c/WINDOWS/system32/ntdll.dll
(gdb) bt
#0  0x7c90eb94 in ntdll!LdrAccessResource ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#1  0x7c90e288 in ntdll!ZwReadFile ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#2  0x7c801875 in ReadFile () from /cygdrive/c/WINDOWS/system32/kernel32.dll
#3  0x00000748 in ?? ()
(gdb) thread 1
[Switching to thread 1 (thread 3288.0xb24)]#0  0x7c90eb94 in ntdll!LdrAccessReso
urce () from /cygdrive/c/WINDOWS/system32/ntdll.dll
(gdb) bt
#0  0x7c90eb94 in ntdll!LdrAccessResource ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#1  0x7c90e9ab in ntdll!ZwWaitForMultipleObjects ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#2  0x7c8094f2 in KERNEL32!CreateFileMappingA ()
   from /cygdrive/c/WINDOWS/system32/kernel32.dll
#3  0x00000002 in ?? ()
#4  0x0022d668 in ?? ()
#5  0x00000001 in ?? ()
(gdb)

-----------------------------------------------------------

$ gdb /usr/bin/rsync 2492
GNU gdb 2003-09-20-cvs (cygwin-special)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i686-pc-cygwin"...
Attaching to program `/usr/bin/rsync.exe', process 2496
[Switching to thread 2496.0x980]
(gdb) info threads
* 5 thread 2496.0x980  0x77f813b2 in ntdll!DbgUiConnectToDbg () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
  4 thread 2496.0x9d0  0x77f82870 in ntdll!ZwWaitForSingleObject () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
  3 thread 2496.0x8c0  0x77f8287e in ntdll!ZwWaitForMultipleObjects () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
  2 thread 2496.0x9c4  0x77f82926 in ntdll!ZwReadFile () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
  1 thread 2496.0x6c4  0x77f8287e in ntdll!ZwWaitForMultipleObjects () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
(gdb) bt
#0  0x77f813b2 in ntdll!DbgUiConnectToDbg () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
#1  0x7c57180b in KERNEL32!DebugActiveProcess () from
/cygdrive/c/WINNT/system32/KERNEL32.DLL
#2  0x61003e84 in getprogname () from /usr/bin/cygwin1.dll
#3  0x00f8f0d0 in ?? ()
(gdb) thread 4
[Switching to thread 4 (thread 2496.0x9d0)]#0  0x77f82870 in
ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/WINNT/system32/NTDLL.DLL
(gdb) bt
#0  0x77f82870 in ntdll!ZwWaitForSingleObject () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
#1  0x74fdbcd2 in WSPSetSockOpt () from /cygdrive/c/WINNT/system32/msafd.dll
#2  0x74fd7ee6 in WSPSetSockOpt () from /cygdrive/c/WINNT/system32/msafd.dll
#3  0x75031da9 in WSACreateEvent () from /cygdrive/c/WINNT/system32/ws2_32.dll
#4  0x61089f34 in cygwin1!aclcheck () from /usr/bin/cygwin1.dll
#5  0x61003186 in getprogname () from /usr/bin/cygwin1.dll
#6  0xffffffff in ?? ()
(gdb) thread 3
[Switching to thread 3 (thread 2496.0x8c0)]#0  0x77f8287e in
ntdll!ZwWaitForMultipleObjects ()
   from /cygdrive/c/WINNT/system32/NTDLL.DLL
(gdb) bt
#0  0x77f8287e in ntdll!ZwWaitForMultipleObjects () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
#1  0x7c573c23 in WaitForMultipleObjectsEx () from
/cygdrive/c/WINNT/system32/KERNEL32.DLL
#2  0x7c578f0d in WaitForMultipleObjects () from
/cygdrive/c/WINNT/system32/KERNEL32.DLL
#3  0x00d2ef48 in ?? ()
#4  0x00000001 in ?? ()
(gdb) thread 2
[Switching to thread 2 (thread 2496.0x9c4)]#0  0x77f82926 in ntdll!ZwReadFile ()
   from /cygdrive/c/WINNT/system32/NTDLL.DLL
(gdb) bt
#0  0x77f82926 in ntdll!ZwReadFile () from /cygdrive/c/WINNT/system32/NTDLL.DLL
#1  0x7c5785d1 in ReadFile () from /cygdrive/c/WINNT/system32/KERNEL32.DLL
#2  0x61093bd6 in sigfillset () from /usr/bin/cygwin1.dll
#3  0x61003186 in getprogname () from /usr/bin/cygwin1.dll
(gdb) thread 1
[Switching to thread 1 (thread 2496.0x6c4)]#0  0x77f8287e in
ntdll!ZwWaitForMultipleObjects ()
   from /cygdrive/c/WINNT/system32/NTDLL.DLL
(gdb) bt
#0  0x77f8287e in ntdll!ZwWaitForMultipleObjects () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
#1  0x7c573c23 in WaitForMultipleObjectsEx () from
/cygdrive/c/WINNT/system32/KERNEL32.DLL
#2  0x7c578f0d in WaitForMultipleObjects () from
/cygdrive/c/WINNT/system32/KERNEL32.DLL
#3  0x0022e6e8 in ?? ()
#4  0x00000001 in ?? ()
(gdb)

-----------------------------------------------------------
 
$ gdb /usr/bin/rsync 1620
GNU gdb 2003-09-20-cvs (cygwin-special)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i686-pc-cygwin"...
Attaching to program `/usr/bin/rsync.exe', process 1620
[Switching to thread 1620.0x934]
(gdb) info threads
* 4 thread 1620.0x934  0x77f813b2 in ntdll!DbgUiConnectToDbg () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
  3 thread 1620.0x92c  0x77f82870 in ntdll!ZwWaitForSingleObject () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
  2 thread 1620.0x928  0x77f82926 in ntdll!ZwReadFile () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
  1 thread 1620.0x2c0  0x77f8287e in ntdll!ZwWaitForMultipleObjects () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
(gdb) bt
#0  0x77f813b2 in ntdll!DbgUiConnectToDbg () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
#1  0x7c57180b in KERNEL32!DebugActiveProcess () from
/cygdrive/c/WINNT/system32/KERNEL32.DLL
#2  0x61003e84 in getprogname () from /usr/bin/cygwin1.dll
#3  0x00dcf0d0 in ?? ()
(gdb) thread 3
[Switching to thread 3 (thread 1620.0x92c)]#0  0x77f82870 in
ntdll!ZwWaitForSingleObject ()
   from /cygdrive/c/WINNT/system32/NTDLL.DLL
(gdb) bt
#0  0x77f82870 in ntdll!ZwWaitForSingleObject () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
#1  0x74fdbcd2 in WSPSetSockOpt () from /cygdrive/c/WINNT/system32/msafd.dll
#2  0x74fd7ee6 in WSPSetSockOpt () from /cygdrive/c/WINNT/system32/msafd.dll
#3  0x75031da9 in WSACreateEvent () from /cygdrive/c/WINNT/system32/ws2_32.dll
#4  0x61089f34 in cygwin1!aclcheck () from /usr/bin/cygwin1.dll
#5  0x61003186 in getprogname () from /usr/bin/cygwin1.dll
#6  0xffffffff in ?? ()
(gdb) thread 2
[Switching to thread 2 (thread 1620.0x928)]#0  0x77f82926 in ntdll!ZwReadFile ()
   from /cygdrive/c/WINNT/system32/NTDLL.DLL
(gdb) bt
#0  0x77f82926 in ntdll!ZwReadFile () from /cygdrive/c/WINNT/system32/NTDLL.DLL
#1  0x7c5785d1 in ReadFile () from /cygdrive/c/WINNT/system32/KERNEL32.DLL
#2  0x61093bd6 in sigfillset () from /usr/bin/cygwin1.dll
#3  0x61003186 in getprogname () from /usr/bin/cygwin1.dll
(gdb) thread 1
[Switching to thread 1 (thread 1620.0x2c0)]#0  0x77f8287e in
ntdll!ZwWaitForMultipleObjects ()
   from /cygdrive/c/WINNT/system32/NTDLL.DLL
(gdb) bt
#0  0x77f8287e in ntdll!ZwWaitForMultipleObjects () from
/cygdrive/c/WINNT/system32/NTDLL.DLL
#1  0x7c573c23 in WaitForMultipleObjectsEx () from
/cygdrive/c/WINNT/system32/KERNEL32.DLL
#2  0x7c578f0d in WaitForMultipleObjects () from
/cygdrive/c/WINNT/system32/KERNEL32.DLL
#3  0x0022e1c8 in ?? ()
#4  0x00000001 in ?? ()
(gdb)
 
I still have the processes in the debuggers, so let me know if there is
something you
want me to check.

Jim

jim

2004-Oct-01 06:47 UTC

head link

rsync 2.6.2 hang (was rsync 2.6.2 crash)

Wayne,

Do you think an earlier version might be missing this bug? There's really
not anything fancy about
the directories I'm syncing- no links, no multi-byte char filenames, I
don't care about permissions,
etc., just timestamps, names and contents. I guess the only thing out of the
ordinary is that they
are big, about 30GB and 200,000 files.

Thanks,
Jim

Possibly Parallel Threads

Search for more apparently analagous threads

rsync - Sep 2004 - rsync 2.6.2 hang (was rsync 2.6.2 crash)

rsync 2.6.2 hang (was rsync 2.6.2 crash)

rsync 2.6.2 hang (was rsync 2.6.2 crash)

rsync 2.6.2 hang (was rsync 2.6.2 crash)

rsync 2.6.2 hang (was rsync 2.6.2 crash)

rsync 2.6.2 hang (was rsync 2.6.2 crash)

rsync 2.6.2 hang (was rsync 2.6.2 crash)

rsync 2.6.2 hang (was rsync 2.6.2 crash)

rsync 2.6.2 hang (was rsync 2.6.2 crash)

Possibly Parallel Threads