rsync - Oct 2017 - rsync does hours of "fake-work" after failure

Hi,

I just stepped on a strange and very annoying bug in rsync-3.1.0 as
shipped with SuSE Linux Enterprise 12, but verified the bug also
with rsync-HEAD-20170123.

I tried to copy some of my movie collection to a usb disk that our
TV could read, so it was formatted with vfat. I forgot that vfat can't
handle files > 4 GB, and some of the movies were larger.

rsync worked for 3 hours copying hundreds of GB, but after it had
finished the last file it complained

rsync: write failed on "/media/disk/some_movie.mpg": File too large
(27)
rsync error: error in file IO (code 11) at receiver.c(389) [receiver=3.1.0]

This file had been the third in the list of files to copy, and when
I looked at the usb disk I saw that the two files before and 4 GB of
some_movie.mpg had been copied. But the 400 GB of the remaining files
had not! rsync had claimed to copy each of it, and as I use "-avP" 
I had indeed been watching the progress. The speed and the MB/s
were the usual values for copying to the USB disk.

So rsync doesn't stop and fail at the point it sees the first file
too large for vfat, it just goes on and "fakes" the rest of the
process :-) And because it took some hours, this was a real bad
surprise at the end. 

Below is the output of a little test that can easily be reprocuded.
Is this a known bug? I couldn't find something similar in bugzilla 
or the mailinglist archives.

cu,
Frank

galois [15:42] test 4940) ~/tmp/rsync-HEAD-20170123-0003GMT/rsync -avP .
/media/disk/test/
sending incremental file list
created directory /media/disk/test
./
a
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=6/8)
test1.dd
  4,533,766,144 100%  110.55MB/s    0:00:39 (xfr#2, to-chk=5/8)
u
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=4/8)
v
              0 100%    0.00kB/s    0:00:00 (xfr#4, to-chk=3/8)
w
              0 100%    0.00kB/s    0:00:00 (xfr#5, to-chk=2/8)
x
              0 100%    0.00kB/s    0:00:00 (xfr#6, to-chk=1/8)
y
              0 100%    0.00kB/s    0:00:00 (xfr#7, to-chk=0/8)
rsync: write failed on "/media/disk/test/test1.dd": File too large
(27)
rsync error: error in file IO (code 11) at receiver.c(389) [receiver=3.1.0]


galois [15:43] test 4942) ls -la /media/disk/test/
total 4194368
drwx------ 2 fst mmh      32768 Oct  5 15:43 .
drwx------ 5 fst mmh      32768 Oct  5 15:42 ..
-rw------- 1 fst mmh          0 Oct  5 15:41 a
-rw------- 1 fst mmh 4294967295 Jan  1  1970 test1.dd
galois [15:43] test 4943) 



-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

> On 06 Oct 2017, at 12:24, Frank Steiner via rsync <rsync at
lists.samba.org> wrote:
> 
> Hi,
> 
> I just stepped on a strange and very annoying bug in rsync-3.1.0 as
> shipped with SuSE Linux Enterprise 12, but verified the bug also
> with rsync-HEAD-20170123.
> 
> I tried to copy some of my movie collection to a usb disk that our
> TV could read, so it was formatted with vfat. I forgot that vfat can't
> handle files > 4 GB, and some of the movies were larger.
> 
> rsync worked for 3 hours copying hundreds of GB, but after it had
> finished the last file it complained
> 
> rsync: write failed on "/media/disk/some_movie.mpg": File too
large (27)
> rsync error: error in file IO (code 11) at receiver.c(389) [receiver=3.1.0]
> 
> This file had been the third in the list of files to copy, and when
> I looked at the usb disk I saw that the two files before and 4 GB of
> some_movie.mpg had been copied. But the 400 GB of the remaining files
> had not! rsync had claimed to copy each of it, and as I use
"-avP"
> I had indeed been watching the progress. The speed and the MB/s
> were the usual values for copying to the USB disk.
> 
> So rsync doesn't stop and fail at the point it sees the first file
> too large for vfat, it just goes on and "fakes" the rest of the
> process :-) And because it took some hours, this was a real bad
> surprise at the end. 
> 
> Below is the output of a little test that can easily be reprocuded.
> Is this a known bug? I couldn't find something similar in bugzilla 
> or the mailinglist archives.
Hi,

I encountered same issue and proposed the following patch :
https://bugzilla.samba.org/show_bug.cgi?id=12525

Perhaps you should give it a try ?

Ben

> On 06 Oct 2017, at 13:18, Frank Steiner <fsteiner-mail1 at
bio.ifi.lmu.de> wrote:
> 
> Ben RUBSON wrote
>> 
>> I encountered same issue and proposed the following patch :
>> https://bugzilla.samba.org/show_bug.cgi?id=12525
> 
> Sorry, I really missed that as my search keywords were completely
> different :-)
> 
>> Perhaps you should give it a try ?
> 
> Yes, that works fine! Thanks a lot! I'll use my own patched rsync
> until these patches are accepted (given how old they are, is rsync
> unmaintained at the moment?).
You're welcome !

Wayne is committing regularly :
https://git.samba.org/rsync.git/?p=rsync.git;a=summary
I've written some patches and bug fixes I hope to see in 3.1.3 :)

Ben