rsync - Jul 2004 - [Bug 1529] 32bit rollover problem rsyncing files greater than 4GB in size

https://bugzilla.samba.org/show_bug.cgi?id=1529


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED




------- Additional Comments From wayned@samba.org  2004-07-14 09:55 -------
Your bug report made the problem easy to find.  I've checked in a fix to
CVS,
but you can fix your current source by changing the "size_t" to
"OFF_T" in the
generate_and_send_sums() function in generator.c.  It should look like this:

static void generate_and_send_sums(struct map_struct *buf, OFF_T len, int f_out)

Then, recompile, install, and the problem should be gone.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=1529





------- Additional Comments From lnds@hotmail.com  2004-07-31 03:48 -------
I was getting bitten by the same bug, im my case a 4.1GB file rolling over in
100MBs.  I've applied the patch and the problem in generate_and_send_sums is
indeed fixed, but it appears to have just moved onto another much graver bug
which causes nothing to get transfered:

rsync
rsync.planetmirror.com::fedora/linux/core/test/2.90/i386/iso/FC3-test1-i386-DVD.iso
-azvvv .

opening tcp connection to rsync.planetmirror.com port 873

  Welcome to PlanetMirror's rsync service.


  You can find a web front end to this archive at:

       http://planetmirror.com

  You can also access this service via FTP at

        ftp://ftp.planetmirror.com


  If you are a regular PM user, please consider supporting PM by
  subscribing to a Premium or PremiumDownload account via the
  web front end.


receiving file list ...
recv_file_name(FC3-test1-i386-DVD.iso)
received 1 names
done
recv_file_list done
get_local_name count=1 .
recv_files(1) starting
generator starting pid=2243 count=1
delta transmission enabled
recv_generator(FC3-test1-i386-DVD.iso,0)
generating and sending sums for 0
count=66400 rem=4729 blength=66384 s2length=4 flength=4407835945
generate_files phase=1
recv_files(FC3-test1-i386-DVD.iso)
FC3-test1-i386-DVD.iso
recv mapped FC3-test1-i386-DVD.iso of size 4407835945
rsync: connection unexpectedly closed (89 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(359)
_exit_cleanup(code=12, file=io.c, line=359): about to call exit(12)
rsync: connection unexpectedly closed (69 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(359)
_exit_cleanup(code=12, file=io.c, line=359): about to call exit(12)




(In reply to comment #1)
> Your bug report made the problem easy to find.  I've checked in a fix
to CVS,
> but you can fix your current source by changing the "size_t" to
"OFF_T" in the
> generate_and_send_sums() function in generator.c.  It should look like
this:
> 
> static void generate_and_send_sums(struct map_struct *buf, OFF_T len, int
f_out)
> 
> Then, recompile, install, and the problem should be gone.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=1529





------- Additional Comments From eharris@puremagic.com  2004-07-31 09:43 -------
(In reply to comment #2)
> I was getting bitten by the same bug, im my case a 4.1GB file rolling over
in
> 100MBs.  I've applied the patch and the problem in
generate_and_send_sums is
> indeed fixed, but it appears to have just moved onto another much graver
bug
> which causes nothing to get transfered:
Keep in mind, BOTH ends of the rsync connection need to apply this fix in order
to work correctly, since both ends call generate_and_send_sums() to decide what
needs to be transferred.

So unless you've also gotten planetmirror.com to apply this fix to their
rsync
server, you will still have related rollover problems.  This is probably what is
causing your problem.

Evan

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=1529


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |lnds@hotmail.com




------- Additional Comments From wayned@samba.org  2004-07-31 12:18
-------
> Keep in mind, BOTH ends of the rsync connection need to apply
> this fix in order to work correctly
Only the receiving side calls generate_and_send_sums(), so only the receiver is
affected by this fix.  As long as Ludi is just pulling files, it shouldn't
matter if their rsync is patched or not.

I'm going to do some testing and determine if I need to reopen the bug.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=1529





------- Additional Comments From eharris@puremagic.com  2004-07-31 12:42 -------
I was under the impression that both sides needed it, because after I
patched one end, it still didn't work for me, but after I patched both ends,
it did.  But maybe I screwed up something in the test.   

Evan

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=1529





------- Additional Comments From wayned@samba.org  2004-08-01 13:45 -------
(In reply to comment #2)
> I've applied the patch and the problem in generate_and_send_sums is
> indeed fixed, but it appears to have just moved onto another much graver
bug
> which causes nothing to get transfered:
I too saw an error trying to copy that file from planetmirror.com when I have a
4GB basis file on the receiving system, but it is the rsync daemon on the
planetmirror.com server that is going away, so the only way to see what is wrong
is to ask someone at that site to look in the rsync log and see if there are any
errors.  I tried a similar copy from two local rsync daemons (one running 2.6.2
and one running CVS), and neither one exhibited this failure, so it may be some
kind of a resource problem on the planetmirror server (e.g. perhaps the daemon
exceeded the ulimit memory limit).

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=1529





------- Additional Comments From lnds@hotmail.com  2004-08-04 05:52 -------
OK, so assuming the fault was with rsync.planetmirror.com I retried this time
using rsync.kernel.org, but seem to have hit another problem:

After taking an eternity the file was downloaded, size was ultimately correct,
upon which it barfed, deleted the tempfile and restarted.  Because the file was
automatically deleted I cannot confirm or deny if it actually had a bad
checksum.

...
chunk[66398] of size 66384 at 4407764832 offset=4407675447
chunk[66399] of size 4729 at 4407831216 offset=4407741831
got file_sum
WARNING: FC3-test1-i386-DVD.iso failed verification -- update discarded (will
try again).
recv_generator(FC3-test1-i386-DVD.iso,0)
recv_files phase=1
gen mapped FC3-test1-i386-DVD.iso of size 4407835945
generating and sending sums for 0
count=66400 rem=4729 blength=66384 s2length=16 flength=4407835945
chunk[0] offset=0 len=66384 sum1=48387eea
chunk[1] offset=66384 len=66384 sum1=f402c97e




(In reply to comment #6)
> I too saw an error trying to copy that file from planetmirror.com when I
have a
> 4GB basis file on the receiving system, but it is the rsync daemon on the
> planetmirror.com server that is going away, so the only way to see what is
wrong
> is to ask someone at that site to look in the rsync log and see if there
are any
> errors.  I tried a similar copy from two local rsync daemons (one running
2.6.2
> and one running CVS), and neither one exhibited this failure, so it may be
some
> kind of a resource problem on the planetmirror server (e.g. perhaps the
daemon
> exceeded the ulimit memory limit).
-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.