Dear rsync people (particularly Wayne), I am finding that the CVS version of rsync hangs during most, but not all, local copies. The problem goes away if I pass --protocol=29, so I'm guessing it is related to the incremental file-list sending. The hanging seems to be timing-dependent. Increasing the verbosity level seems to increase the chance of a hang. If I run rsync repeatedly with the same source directory, the chance of a hang seems to gradually decrease, possibly because the source directory goes into cache. I ran the following command line and then killed the rsync processes when they printed nothing for several seconds: rm -rf * && strace -ff -o rsync-incremental.strace ~/rsync/rsync/rsync -a -ivvv --progress ~/rsync/ dest2/ 2>&1 | tee rsync-incremental.log The resulting rsync-incremental* files are available at: http://www.kepreon.com/~matt/private/rsync-hanging/ Matt
On Sun, Jan 14, 2007 at 10:30:56PM -0500, Matt McCutchen wrote:> I am finding that the CVS version of rsync hangs during most, but not > all, local copies. The problem goes away if I pass --protocol=29, soI was only able to reproduce this with at least -vvv output, and the hang also affected protocol 29. The hang is happening when the generator is trying to either forward a message from the receiver down the socket, or it is creating a message while reading file-list data from the receiver (e.g. when protocol 30 is reading the file-list info from the receiver). The problem occurs because an active read of the receiver->generator pipe disables further reads on the pipe, and thus if a write to the socket stalls, we risk deadlock should the receiver also be stalled trying to write data to us. Older rsyncs used a different flushing method in the receiver when writing into the pipe which could not deadlock, but the CVS version currently has this changed due to protocol 30's need to flush file-list info without buffering. I have checked-in a fix that ensures that we defer outgoing messages in the generator whenever we disable reading from the receiver->generator pipe. I'm going to investigate if it would be possible to avoid disabling the reading on the pipe altogether these days (I think it should be easy, but haven't tried it yet). Attached is the current fix that I checked in to CVS. Thanks! ..wayne.. -------------- next part -------------- A non-text attachment was scrubbed... Name: defer.patch Type: text/x-diff Size: 925 bytes Desc: not available Url : http://lists.samba.org/archive/rsync/attachments/20070120/a00b9597/defer.bin
On 1/20/07, Wayne Davison <wayned@samba.org> wrote:> Attached is the current fix that I checked in to CVS. Thanks!Whatever you changed fixed the hanging for my original command. However, I am still getting hanging with the command below, which is a modified version of the command I use via rsnapshot to back up my computer. Here an external hard disk with a relatively slow USB 1 connection is mounted at /mnt/external-disk , and the current directory is /mnt/external-disk/rsync-test . rm -rf dest rsync-incremental* && strace -ff -o rsync-incremental.strace ~/rsync/rsync/rsync -axxx --del --numeric-ids --delete-excluded --relative --progress --filter=P_/home/ --filter=P_/mnt/Matt-common/ --exclude=media/external-disk/snapshots --link-dest=/media/external-disk/snapshots/occasional.0/mattlaptop/ /home/matt/rsync/ dest/ 2>&1 | tee rsync-incremental.log The new log and straces are at: http://www.kepreon.com/~matt/private/rsync-hanging/ Matt
On Sat, Jan 20, 2007 at 07:24:39PM -0500, Matt McCutchen wrote:> The new log and straces are at: > http://www.kepreon.com/~matt/private/rsync-hanging/What would be helpful would be to attach a debugger to the generator when it is stalled (and perhaps the receiver too) and ask for a function backtrace. ..wayne..