Hi all,
I'm having problems with an rsync via ssh process hanging during transfer.
The basic problem, (gory details to follow):
I've set up a machine on my LAN to act as the receiver and have two
(soon to be more) remote servers transferring files via a script.
It seems as if certain files are causing the transfer to hang. These
are different files on each of the sending machines, and there
doesn't seem to be any correlation between them. The plan is to back
up several directories (/usr /var /home /etc) so I thought at first
there could be some sort of memory issue as the transfers stalled
partway through. Then I realized that even if I only transferred one
directory the process would still hang on this particular file. In
one case, the file was a 57kb executable and in another it was a
238kb JPEG. Everything works fine up to that point and then bam! It
just sits there on that file and never finishes. I've turned on some
of the extended reporting options, and here's some output from one of
the transfers from a /usr/bin directory:
---------
1205 files to consider
appletproxy
4867 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1185/1205)
apropos
2378 100% 2.27MB/s 0:00:00 (xfer#2, to-check=1184/1205)
ar
35480 100% 2.82MB/s 0:00:00 (xfer#3, to-check=1183/1205)
ark
4831 100% 314.52kB/s 0:00:00 (xfer#4, to-check=1182/1205)
artsbuilder
344201 100% 517.13kB/s 0:00:00 (xfer#5, to-check=1181/1205)
artscat
Killed by signal 2. 48.71kB/s 0:00:00
---------
In the case above the 'artscat' file hangs at precisely 55% each
time, at exactly 32768 bytes. Hmmmm..... A clue, perhaps?
One of the sending systems is a RedHat box, the other is FC2. The
Fedora system is running rsync 2.6.2, the RH box is now running 2.6.8
after hoping that an upgrade from 2.5.5 would cure the problem. No
change.
That leaves the receiving system as the common link, so I suspect the
problem may lie there or with the HD. It's a G4 Mac running OS X
10.4.6 with the standard install of rsync 2.6.3. It's a fresh install
of the OS and the machine will be used solely as a backup server. I'm
backing up to a LaCie 250GB Ethernet disk, formatted with the Ext3
filesystem. I don't have enough free space on the receiver's internal
HD to test the entire backup, but the backup of just the /usr/bin
directory completes successfully if I back up to the internal HD
instead of the Ethernet disk. As you can see above, the transfer
hangs only six files into a 1205 file run when backing up to the
LaCie. However it works fine on other large directories.
I'm out of ideas on this one, except for possibly installing a larger
internal HD and temporarily forgetting about the LaCie.
Any ideas? I'd be grateful for any insights you can provide.
Gawain
Flames invitied if I'm wrong, but I think you're looking at the last file successfully transferred as opposed to the first file unsuccessfully transferred. I think I saw a /var in there You get something ungodly long trying to rsync the file that is logging the rsync. something like rsync -av --exclude "log/" --exclude "work/" $name /sdb1/2006-04-01-hostname/ may be in order (the above is just a "local" copy to a USB drive. You probably have some timing issues as to whether or not rsyncing the log file actually works or is an infinite loop. I'd put watching the grass grow at a higher priority than trying to "fix" the "problem".> -----Original Message----- > From: rsync-bounces+tony=servacorp.com@lists.samba.org > [mailto:rsync-bounces+tony=servacorp.com@lists.samba.org]On Behalf Of > Gawain > Sent: Tuesday, May 09, 2006 8:53 PM > To: rsync@lists.samba.org > Subject: Rsync via ssh hangs on same file repeatedly > > > Hi all, > > I'm having problems with an rsync via ssh process hanging during transfer. > > The basic problem, (gory details to follow): > > I've set up a machine on my LAN to act as the receiver and have two > (soon to be more) remote servers transferring files via a script. > > It seems as if certain files are causing the transfer to hang. These > are different files on each of the sending machines, and there > doesn't seem to be any correlation between them. The plan is to back > up several directories (/usr /var /home /etc) so I thought at first > there could be some sort of memory issue as the transfers stalled > partway through. Then I realized that even if I only transferred one > directory the process would still hang on this particular file. In > one case, the file was a 57kb executable and in another it was a > 238kb JPEG. Everything works fine up to that point and then bam! It > just sits there on that file and never finishes. I've turned on some > of the extended reporting options, and here's some output from one of > the transfers from a /usr/bin directory: > > --------- > > 1205 files to consider > appletproxy > 4867 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1185/1205) > apropos > 2378 100% 2.27MB/s 0:00:00 (xfer#2, to-check=1184/1205) > ar > 35480 100% 2.82MB/s 0:00:00 (xfer#3, to-check=1183/1205) > ark > 4831 100% 314.52kB/s 0:00:00 (xfer#4, to-check=1182/1205) > artsbuilder > 344201 100% 517.13kB/s 0:00:00 (xfer#5, to-check=1181/1205) > artscat > Killed by signal 2. 48.71kB/s 0:00:00 > > --------- > > In the case above the 'artscat' file hangs at precisely 55% each > time, at exactly 32768 bytes. Hmmmm..... A clue, perhaps? > > One of the sending systems is a RedHat box, the other is FC2. The > Fedora system is running rsync 2.6.2, the RH box is now running 2.6.8 > after hoping that an upgrade from 2.5.5 would cure the problem. No > change. > > That leaves the receiving system as the common link, so I suspect the > problem may lie there or with the HD. It's a G4 Mac running OS X > 10.4.6 with the standard install of rsync 2.6.3. It's a fresh install > of the OS and the machine will be used solely as a backup server. I'm > backing up to a LaCie 250GB Ethernet disk, formatted with the Ext3 > filesystem. I don't have enough free space on the receiver's internal > HD to test the entire backup, but the backup of just the /usr/bin > directory completes successfully if I back up to the internal HD > instead of the Ethernet disk. As you can see above, the transfer > hangs only six files into a 1205 file run when backing up to the > LaCie. However it works fine on other large directories. > > I'm out of ideas on this one, except for possibly installing a larger > internal HD and temporarily forgetting about the LaCie. > > Any ideas? I'd be grateful for any insights you can provide. > > Gawain > -- > To unsubscribe or change options:https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
On Tue, May 09, 2006 at 08:52:57PM -0500, Gawain wrote:> I'm having problems with an rsync via ssh process hanging during > transfer.The issues and debugging page on the rsync website gives some advice on how to investigate a hang. It mentions running netstat on each system to check on the state of the Send-Q and Recv-Q for the socket, and also using a tool such as strace or truss to see what system calls each process gets stuck on. If you can provide this information, it will help us to diagnose the problem. ..wayne..
At 9:26 PM -0500 on 5/9/06, Tony Abernethy wrote:>Flames invitied if I'm wrong, but I think you're looking at >the last file successfully transferred as opposed to the >first file unsuccessfully transferred. > >I think I saw a /var in there >You get something ungodly long trying to rsync the file that is logging the >rsync. >something like >rsync -av --exclude "log/" --exclude "work/" $name >/sdb1/2006-04-01-hostname/ >may be in order (the above is just a "local" copy to a USB drive. > >You probably have some timing issues as to whether or not >rsyncing the log file actually works or is an infinite loop. >I'd put watching the grass grow at a higher priority than trying to "fix" >the "problem".Thanks for your reply. I needed some time to continue testing. It turned out that the LaCie "Ethernet disk" was the culprit. Something was causing a timeout during the data write and the process was hanging up. I've since installed another internal drive in the machine and we've been happily backing up servers ever since.