Hi,
I tried to sync two files (50 MB and 100 MB) with my webdav folder
using rsync 3.0.5 with Mac OS X (10.4.11) Terminal and X11. With the
Terminal application the sync always failed with both files. With X11
I once was successful synchronizing the 50 MB file. The error message
is always like this (also when I set the timeout to eg. 7200 seconds):
io timeout after 1003 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(200)
[sender=3.0.5]
the sync command I used:
/usr/local/bin/rsync --size-only --human-readable --progre
ss --recursive --delete --timeout=1000 --bwlimit=70 --times --
stats /Users/
Helge/temp /Volumes/helgekraak/ >> /users/Helge/Pictures/
rsync_Pictures_` dat
e +%Y_%m_%d__%H_%M`.log
The log file (shortened):
sending incremental file list
temp/
temp/.DS_Store
6.15K 100% 0.00kB/s 0:00:00
6.15K 100% 0.00kB/s 0:00:00 (xfer#1, to-check=2/4)
temp/Layout_BM.psd
32.77K 0% 0.00kB/s 0:00:00
98.30K 0% 59.87kB/s 0:14:25
163.84K 0% 58.58kB/s 0:14:42
229.38K 0% 59.10kB/s 0:14:34
.
.
.
49.64M 95% 53.93kB/s 0:00:41
51.71M 99% 57.12kB/s 0:00:03
51.81M 99% 57.97kB/s 0:00:01
51.87M 99% 58.44kB/s 0:00:00
51.89M 100% 53.96kB/s 0:15:38 (xfer#2, to-check=1/4)
temp/Thumb64.data
Thumb64.data is the 100 MB file which doesn't start to upload after
the Layout_BM.psd file.
I also tried two other webdav services with the same result. When I
use the Finder to copy the files it works perfectly. Also when I use
the cp command in the Terminal respectively with X11.
Do you think this is a bug of rsync? Should I file an issue?
Thanks in advance!
Helge
On Sat, 2009-01-10 at 18:16 +0100, Kraak Helge wrote:> I tried to sync two files (50 MB and 100 MB) with my webdav folder > using rsync 3.0.5 with Mac OS X (10.4.11) Terminal and X11. With the > Terminal application the sync always failed with both files. With X11 > I once was successful synchronizing the 50 MB file. The error message > is always like this (also when I set the timeout to eg. 7200 seconds): > > io timeout after 1003 seconds -- exiting > rsync error: timeout in data send/receive (code 30) at io.c(200) > [sender=3.0.5] > > the sync command I used: > > /usr/local/bin/rsync --size-only --human-readable --progre > ss --recursive --delete --timeout=1000 --bwlimit=70 --times -- > stats /Users/ > Helge/temp /Volumes/helgekraak/ >> /users/Helge/Pictures/ > rsync_Pictures_` dat > e +%Y_%m_%d__%H_%M`.log> I also tried two other webdav services with the same result. When I > use the Finder to copy the files it works perfectly. Also when I use > the cp command in the Terminal respectively with X11. > > Do you think this is a bug of rsync? Should I file an issue?Rsync tends to run into inadequacies in network filesystems more often than cp since it is a more sophisticated tool. Please obtain a system call trace of all three rsync processes using ktrace so I can see how rsync is hanging and determine whether the fault is with rsync or the WebDAV program. -- Matt
Helge, please CC rsync@lists.samba.org in your replies so your messages are archived for others' future benefit. On Wed, 2009-01-21 at 19:48 +0100, Kraak Helge wrote:> Thanks for your reply Matt! > > I now tested two different commands with two different versions of > rsync. Once I used the timeout option and once I didn't. > > When I don't use the timeout option I don't get any error and the > upload to the webdav server works for all files (with 2.6.9 and with > 3.0.5), but there's a long break (I guess about 60-90 minutes) > between the large files with size of 50 Megabytes and 100 Megabyte.I see. Then the underlying problem is the slowness of the operation; obviously, if you set a timeout, rsync will give up after being stalled for that amount of time. We can still try to determine whether the slowness is the fault of rsync or the WebDAV client.> I uploaded all system call traces to this address as I don't know > which parts of the files are relevant for you: http://kraak.info/rsync/rsync_system_call_trace_3.0.5.txt is the case I care about (recent version of rsync, no timeout). Could you obtain a new trace that shows the time (either relative or absolute) of each line? Without that, it's impossible to tell where the 60-minute delay is. Incidentally, I wonder if the WebDAV client is performing the final rename in a silly way, such as by downloading and re-uploading the file instead of using the WebDAV MOVE method. If that's the problem, you could work around it by passing --inplace to rsync. Or it could be that the WebDAV client buffers all the data rsync writes to a destination file and does not actually upload anything until rsync closes the file (like AFS); in that case, rsync would see a long delay and there's really nothing that can be done about it.> Without timeout: > /usr/local/bin/rsync --size-only --human-readable --progress -- > recursive --delete --bwlimit=50 --times --stats /Users/Helge/ > temp /Volumes/webdav.macbay.de/backup/temp >> /users/Helge/Pictures/ > rsync_Pictures_` date +%Y_%m_%d__%H_%M`.log > > As said: No error messages when I don't use a timeout but it takes > several hours to upload the files.-- Matt