I have been using BackupPC via rsync daemons on the target machines and all has been working well. At least until about the time rsync went to version 3.0.6. Since then backups have been failing for some targets. The failure is a TCP ZeroWindow issue. It looks like the server rsync process cannot empty it's buffer soon enough and eventually the process fails. I really do not know if it is rsync version related, because I never looked at the packet stream until I started having failures. BackupPC was running on a VM host and the failing machine was one of the guest VMs. I tried upgrading rsync to 3.0.9 and that didn't help. So I moved the BackupPC set up to a different server (no VM). Not all backups are working. though I still see some ZeroWindow packets along with a few retransmision packets. But the backup completes on all machines, including the VM that was failing before. I've googled rsync zerowindow and found some references, but none seemed to apply to my situation. Anybody else having a similar experience? Emmett
On Tue, Sep 25, 2012 at 10:33 PM, Emmett Culley <emmett at webengineer.com> wrote:> I have been using BackupPC via rsync daemons on the target machines and all has been working well. At least until about the time rsync went to version 3.0.6. > > Since then backups have been failing for some targets. The failure is a TCP ZeroWindow issue. It looks like the server rsync process cannot empty it's buffer soon enough and eventually the process fails. > > I really do not know if it is rsync version related, because I never looked at the packet stream until I started having failures. > > BackupPC was running on a VM host and the failing machine was one of the guest VMs. I tried upgrading rsync to 3.0.9 and that didn't help. > > So I moved the BackupPC set up to a different server (no VM). Not all backups are working. though I still see some ZeroWindow packets along with a few retransmision packets. But the backup completes on all machines, including the VM that was failing before. > > I've googled rsync zerowindow and found some references, but none seemed to apply to my situation. > > Anybody else having a similar experience?I found an old mention of turning off TCP window scaling solving a problem with rsync (not involving backuppc). Seems unlikely but might be worth a try: sysctl -w net.ipv4.tcp_window_scaling=0. -- Les Mikesell lesmikesell at gmail.com
On 09/25/2012 08:33 PM, Emmett Culley wrote:> Since then backups have been failing for some targets. The failure > is a TCP ZeroWindow issue. It looks like the server rsync process > cannot empty it's buffer soon enough and eventually the process > fails.If the rsync server sends a TCP ZeroWindow packet to a client, that probably indicates that the server is either hung or looped. Either way, it has stopped reading data from the socket, and the kernel sends ZeroWindow to the client's host to indicate that the client should stop sending data. You may have found a bug in rsync that the developers need to resolve. Have you tried attaching strace to the rsync server process when it hangs?