rsync - Nov 2012 - [Bug 9407] New: rsync transfers cause zero window packets

https://bugzilla.samba.org/show_bug.cgi?id=9407

           Summary: rsync transfers cause zero window packets
           Product: rsync
           Version: 3.0.9
          Platform: All
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: emmett at webengineer.com
         QAContact: rsync-qa at samba.org


I am using BackupPC with rsync daemons running on the target machines.  Some
targets are virtual and some are on their own hardware.

A couple of months ago I noticed that one of the VM targets always failed to
get a backup, and upon investigation I found that the packet stream contained
lots of zero window and keep alive packets.

The failure turned out to be caused by a wrong VM network driver on that
particular VM and after resolving that all backups completed.  However that
caused me to look into it further when I discovered that fixing the driver did
not stop the zero window packets from happening and upon further investigation
I discovered that ALL backup sessions are seeing lots of zero window packets,
then further investigation shows that backups are taking way too long; as long
as ten hours for a 35GB backup.

I've googled for zero window errors with rsync and nothing I've found
points to
anything that could resolve my issue.  But how can that be?  Am I the only one
seeing this issue?  Or is it that no one is looking?

I am running CentOS 6 and Fedora 17, all of which have rsync 3.0.9-1, except
one, which has 3.0.6-9.  I have a Gigibit LAN and iperf shows that there is a
solid 900+Mbs throughput in both directions to/from all target machines and the
backup server.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

On Sat, Nov 17, 2012 at 2:24 AM,  <samba-bugs at samba.org>
wrote:
> https://bugzilla.samba.org/show_bug.cgi?id=9407
>
>            Summary: rsync transfers cause zero window packets
>            Product: rsync
>            Version: 3.0.9
>           Platform: All
>         OS/Version: Linux
>             Status: NEW
>           Severity: major
>           Priority: P5
>          Component: core
>         AssignedTo: wayned at samba.org
>         ReportedBy: emmett at webengineer.com
>          QAContact: rsync-qa at samba.org
>
>
> I am using BackupPC with rsync daemons running on the target machines. 
Some
> targets are virtual and some are on their own hardware.
>
> A couple of months ago I noticed that one of the VM targets always failed
to
> get a backup, and upon investigation I found that the packet stream
contained
> lots of zero window and keep alive packets.
>
> The failure turned out to be caused by a wrong VM network driver on that
> particular VM and after resolving that all backups completed.  However that
> caused me to look into it further when I discovered that fixing the driver
did
> not stop the zero window packets from happening and upon further
investigation
> I discovered that ALL backup sessions are seeing lots of zero window
packets,
> then further investigation shows that backups are taking way too long; as
long
> as ten hours for a 35GB backup.
>
> I've googled for zero window errors with rsync and nothing I've
found points to
> anything that could resolve my issue.  But how can that be?  Am I the only
one
> seeing this issue?  Or is it that no one is looking?
There have been so many interesting bugs fixed in the last year in the
network stack, that I have no idea where to start.
> I am running CentOS 6 and Fedora 17, all of which have rsync 3.0.9-1,
except
> one, which has 3.0.6-9.  I have a Gigibit LAN and iperf shows that there is
a
> solid 900+Mbs throughput in both directions to/from all target machines and
the
> backup server.
This strikes me as the wrong list for this problem.  I would argue in
favor of putting a packet capture up, and telling people on
linux-netdev about it.



-- 
Dave T?ht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

https://bugzilla.samba.org/show_bug.cgi?id=9407

--- Comment #1 from Emmett Culley <emmett at webengineer.com> 2012-11-17
14:03:34 UTC ---
Created attachment 8212
  --> https://bugzilla.samba.org/attachment.cgi?id=8212
Wireshark packet stream capture (binary tcpdump)

Please use tcp.port==873 as a filter to see an example of results of this bug.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.