rsync - May 2009 - DO NOT REPLY [Bug 6378] New: "rsync -v -rsync -v --inplace --progress --rsh=ssh -a" reports erroneous and completely unrealistic transferred size

https://bugzilla.samba.org/show_bug.cgi?id=6378

           Summary: "rsync -v -rsync -v --inplace --progress --rsh=ssh
-a"
                    reports erroneous and completely unrealistic transferred
                    size
           Product: rsync
           Version: 3.0.6
          Platform: x86
               URL: http://www.shlomifish.org/Files/files/code/rsync/jing-
                    20081028-1mdv2010.0.src.rpm
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P3
         Component: core
        AssignedTo: wayned@samba.org
        ReportedBy: shlomif@iglu.org.il
         QAContact: rsync-qa@samba.org


When doing:

{{{
rsync -v --inplace --progress --rsh=ssh -a jing-20081028-1mdv2010.0.src.rpm
shlomifish.org@ragnar.eonspace.net:
}}}

The transferred size almost immediately jumps to 2129920 bytes which is
completely unrealistic given that I have an ADSL connection with an upstream of
at most 25 KBytes/second. Afterwards it hangs after transferring 262,144 bytes,
and I constantly need to restart it. But please report the actual bytes
transferred over the connection rather than some unrealistic value.

I'm using --inplace so I can resume interrupted connections. (My ISP
connection
has become flaky lately).

I'm using rsync-3.0.6-1mdv2010.0 on Mandriva Cooker.

Regards,

      Shlomi Fish


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6378


matt@mattmccutchen.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID




------- Comment #1 from matt@mattmccutchen.net  2009-05-19 21:50 CST -------
As stated in the man page description of --progress, the size shown is the
amount of the file reconstructed by the receiver so far.  I'm guessing the
receiver already had the correct first 2129920 bytes of the file from a
previous run, so the delta-transfer algorithm verifies quickly that those
2129920 bytes match the beginning of the source file and the size jumps to
2129920.  From that point on, literal data has to be sent, so progress is
slower.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6378


shlomif@iglu.org.il changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|INVALID                     |




------- Comment #2 from shlomif@iglu.org.il  2009-05-20 03:08 CST -------
Hi!

(In reply to comment #1)
> As stated in the man page description of --progress, the size shown is the
> amount of the file reconstructed by the receiver so far.  I'm guessing
the
> receiver already had the correct first 2129920 bytes of the file from a
> previous run,
That's not true - it's a brand-new transfer. And ls -l on the file in
the
remote end does not show nearly as much (as expected from my connection).
> so the delta-transfer algorithm verifies quickly that those
> 2129920 bytes match the beginning of the source file and the size jumps to
> 2129920.  
Well, it's not what happens in this case.
> From that point on, literal data has to be sent, so progress is
> slower.
I'm re-opening this bug, because your explanation was not true. Sorry for
not
stating earlier that this was a brand new transfer.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6378





------- Comment #3 from matt@mattmccutchen.net  2009-05-28 09:25 CST -------
I see what is going on here.  In the current rsync implementation, progress is
always computed by the client, which in this case is the sender.  The local ssh
process has a 2 MB buffer for outgoing data (see CHAN_SES_WINDOW_DEFAULT in
http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/channels.h?rev=1.98;content-type=text%2Fx-cvsweb-markup
).  The rsync sender fills up this buffer with data and counts that as 2 MB of
progress.

The obvious approach to fixing this would be to always have the receiver
compute progress, which would involve a protocol change.  The problem is that,
since rsync is pipelined, while the receiver is reporting progress for one
file, the sender may have moved on to another file.  Thus, to get consistent
output, the sender would have to wait to output a filename until it gets
acknowledgment that the receiver has started to receive the file.  And that has
a minor security ramification: a user who realizes that the source directory
contains a file the remote server shouldn't see and interrupts rsync can no
longer assume that the information has not been compromised if the name of the
secret file was not in the output.  Thoughts?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.