Hi,
I have some files size 400MB+ that my rsync ends on, error message is :
"Write failed: Broken pipe".
Without this "big files" everything works smoothe.
I searched to find anything about this problem but so far without any luck.
Source machine : Solaris 8, rsync-2.4.6
Destination machine : FreeBSD 4.1.1, rsync 2.4.6
Command run(from source machine):
rsync -vaRH --timeout=2400 --bwlimit=200 --numeric-ids --delete \
-e 'ssh -i /root/.ssh/identity.machine -l root -C -x' \
. copy:machine >> /var/tmp/logg 2>&1
Help apreciated.
Regards
Lasse
Easy enough... look at your parameters. You're saying that you must not
transfer more than 480000kb in a session... you actual mileage will be slightly
smaller, because of overhead. You have limited yourself to a bandwidth of
200kbps for a duration of
2400 seconds.
A single 400Mb file would take a minimum of 2048 seconds at 200kbps, all by
itself.
Tim Conway
tim.conway@philips.com
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips
Available as n9hmg on AIM
perl -e 'print pack(nnnnnnnnnnnn,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
".\n" '
"There are some who call me.... Tim?"
Lasse Andersson <lasse@netcraft.se>@lists.samba.org on 09/17/2001 12:03:24
PM
Sent by: rsync-admin@lists.samba.org
To: <rsync@lists.samba.org>
cc: (bcc: Tim Conway/LMT/SC/PHILIPS)
Subject: Problem with transfering large files.
Classification:
Hi,
I have some files size 400MB+ that my rsync ends on, error message is :
"Write failed: Broken pipe".
Without this "big files" everything works smoothe.
I searched to find anything about this problem but so far without any luck.
Source machine : Solaris 8, rsync-2.4.6
Destination machine : FreeBSD 4.1.1, rsync 2.4.6
Command run(from source machine):
rsync -vaRH --timeout=2400 --bwlimit=200 --numeric-ids --delete \
-e 'ssh -i /root/.ssh/identity.machine -l root -C -x' \
. copy:machine >> /var/tmp/logg 2>&1
Help apreciated.
Regards
Lasse
i meant to send this to the whole list... maybe someone else has seen it and can
figure out how to fix it.
Tim Conway
tim.conway@philips.com
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips
Available as n9hmg on AIM
perl -e 'print pack(nnnnnnnnnnnn,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
".\n" '
"There are some who call me.... Tim?"
---------------------- Forwarded by Tim Conway/LMT/SC/PHILIPS on 09/20/2001
08:37 AM ---------------------------
Tim Conway
09/20/2001 08:36 AM
To: Scott Howard <scott@doc.net.au> @SMTP
cc:
Subject: Re: Problem with transfering large files. (Document link: Tim Conway)
Classification: Unclassified
I know that's what the docs say, but I have found that i have to set the
timeout to be at least as long as the time it takes to send the largest file.
If it doesn't handle any protocol data, only data data, for the timeout
time, it drops. I can't trace
the execution well enough to figure out why, or to fix it. I'm pretty sure
that's what's going on, because if you leave timeout=0, it sets
select_timeout to 60. When i ran with what should be no timeout at all, it
choked on any significant file.
Tim Conway
tim.conway@philips.com
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips
Available as n9hmg on AIM
perl -e 'print pack(nnnnnnnnnnnn,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
".\n" '
"There are some who call me.... Tim?"
Scott Howard <scott@doc.net.au> on 09/20/2001 07:32:04 AM
To: Tim Conway/LMT/SC/PHILIPS@AMEC
cc: lasse@netcraft.se
rsync@lists.samba.org
Subject: Re: Problem with transfering large files.
Classification:
On Mon, Sep 17, 2001 at 12:15:12PM -0600, tim.conway@philips.com
wrote:> Easy enough... look at your parameters. You're saying that you must
not transfer more than 480000kb in a session... you actual mileage will be
slightly smaller, because of overhead. You have limited yourself to a bandwidth
of 200kbps for a duration of
> 2400 seconds.
> A single 400Mb file would take a minimum of 2048 seconds at 200kbps, all by
itself.
Not so.
--timeout=TIMEOUT
This option allows you to set a maximum IO timeout in
seconds. If no data is transferred for the specified
time then rsync will exit. The default is 0, which
means no timeout.
The 2400 second timeout means that the connection must be _idle_ for 2400
seconds. It does not mean that the session can not last for more than 2400
seconds total.
Scott.
I'm not Dave Dykstra [dwd@bell-labs.com] writes:> On September 9 Tridge submitted a fix to CVS for that problem. See > revision 1.25 at > http://pserver.samba.org/cgi-bin/cvsweb/rsync/generator.cI'm not sure that fixes the use of the timeout for the overall process. See a recent answer by me to this list in the "Feedback on 2.4.7pre1" thread, which included an older patch from last year that I've been using since then. The overall timeout problem is due to the parent process doing a read_int on the child process to wait for final completion, which is subject to the same timeout setting as the child processes is using on individual I/O. But the parent won't hear from the child until it's fully done. -- David /-----------------------------------------------------------------------\ \ David Bolen \ E-mail: db3l@fitlinxx.com / | FitLinxx, Inc. \ Phone: (203) 708-5192 | / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \-----------------------------------------------------------------------/