CRUISE,DAVID (A-FtCollins,ex1)
2001-Oct-31 10:40 UTC
Problem using -v within rsync transfer.
Hello, We have used rsync on a mirroring application successfully for just over a year. After our upgrade to 2.4.6, however, the process started failing about 50% of the time. The nature of our failure was that the rsync processes on the remote system would die with no error message, leaving the local system processes hung. I have netstat and trace output for available if needed, but I see that others have sent this information in. Other details of the transfer include: our platforms are HPUX 10.20 (each with 1.5GB Ram), the total data size is around 20GB, a typical transfer is around 300MB (around 40,000 files). After reading several other postings, we tried removing the -v from our rsync command and our success rate improved dramatically (12 successes in a row now). What other platforms have this issue with -v? Is there a known fix, or is a fix in progress? I ask because part of our process has always included a transfer size and performance report. Without some verbosity, -v, this information is not obviously available. Thank you for any workarounds or updates, Dave Dave Cruise |4380 Ziegler Rd, MS 72 Agilent Technologies |Ft. Collins, CO 80525 IT Engineer |(970) 288-2895 GIS Americas |david_cruise@agilent.com
Dave: I suspect what you need will be provided by adding --stats to your
commandline.
That'll give you something like this at the end of the transfer:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Number of files: 1
Number of files transferred: 1
Total file size: 43 bytes
Total transferred file size: 43 bytes
Literal data: 43 bytes
Matched data: 0 bytes
File list size: 30
Total bytes written: 125
Total bytes read: 36
wrote 125 bytes read 36 bytes 322.00 bytes/sec
total size is 43 speedup is 0.27
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
It still doesn't solve the problem with verbosity, and the process STILL
may hang, but less. I've had to break my transfers down into units of
about 60000 files each. Ugly, but it works. Unfortunately, it means
--delete doesn't often help.
Tim Conway
tim.conway@philips.com
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, n9hmg on AIM
perl -e 'print pack(nnnnnnnnnnnn,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
".\n" '
"There are some who call me.... Tim?"
"CRUISE,DAVID (A-FtCollins,ex1)" <david_cruise@agilent.com>
Sent by: rsync-admin@lists.samba.org
10/30/2001 04:40 PM
To: "'rsync@lists.samba.org'"
<rsync@lists.samba.org>
cc: (bcc: Tim Conway/LMT/SC/PHILIPS)
Subject: Problem using -v within rsync transfer.
Classification:
Hello,
We have used rsync on a mirroring application successfully for just over a
year. After our upgrade to 2.4.6, however, the process started failing
about 50% of the time. The nature of our failure was that the rsync
processes on the remote system would die with no error message, leaving
the
local system processes hung. I have netstat and trace output for
available
if needed, but I see that others have sent this information in.
Other details of the transfer include: our platforms are HPUX 10.20 (each
with 1.5GB Ram), the total data size is around 20GB, a typical transfer is
around 300MB (around 40,000 files).
After reading several other postings, we tried removing the -v from our
rsync command and our success rate improved dramatically (12 successes in
a
row now).
What other platforms have this issue with -v? Is there a known fix, or is
a
fix in progress?
I ask because part of our process has always included a transfer size and
performance report. Without some verbosity, -v, this information is not
obviously available.
Thank you for any workarounds or updates,
Dave
Dave Cruise |4380 Ziegler Rd, MS 72
Agilent Technologies |Ft. Collins, CO 80525
IT Engineer |(970) 288-2895
GIS Americas |david_cruise@agilent.com
-------------- next part --------------
HTML attachment scrubbed and removed
On Tue, Oct 30, 2001 at 04:40:26PM -0700, CRUISE,DAVID (A-FtCollins,ex1) wrote: ...> After reading several other postings, we tried removing the -v from our > rsync command and our success rate improved dramatically (12 successes in a > row now). > > What other platforms have this issue with -v? Is there a known fix, or is a > fix in progress?I don't think the answer to that is clear from the reports sent to the mailing list. There are also two versions of hang fixes at http://www.clari.net/~wayne/rsync-nohang2.patch at http://www.clari.net/~wayne/rsync-nohang2.patch Please try them out and see if they fix the problem with -v. You can also try the 2.4.7pre1 snapshot or the current CVS. Somebody else asked recently how to get those so I will give that info in a separate message. I had been under the impression that Martin had put one of the nohang patches into the next version already, but I see now that he hasn't. I don't recall any other patches related to hanging that might already be in the pre-release, but it would still be good to try it. - Dave Dykstra