rsync - May 2004 - rsync creates wrong sized files

I'm having a pretty serious rsync bug, which I've submitted to the
Debian bug system.  But as the rsync maintainer there seems to be a bit
slow in fixing problems, I thought perhaps I should report it here
as well.

I'm using rsync 2.6.2 on a Debian woody system, with libc 2.2.5.
I have rsync running daily to mirror the Debian archives, mainly for
i386 files.  The command I use is this:

rsync -r -R -l -t --delete --exclude=/Incoming --exclude=/UploadQueue \
 --exclude=*_alpha.deb --exclude=*_alpha.udeb --exclude=*-alpha \
 --exclude=*_arm.deb --exclude=*_arm.udeb --exclude=*-arm \
 --exclude=*_hppa.deb --exclude=*_hppa.udeb --exclude=*-hppa \
 --exclude=*_ia64.deb --exclude=*_ia64.udeb --exclude=*-ia64 \
 --exclude=*_m68k.deb --exclude=*_m68k.udeb --exclude=*-m68k \
 --exclude=*_mips.deb --exclude=*_mips.udeb --exclude=*-mips \
 --exclude=*_mipsel.deb --exclude=*_mipsel.udeb --exclude=*-mipsel \
 --exclude=*_powerpc.deb --exclude=*_powerpc.udeb --exclude=*-powerpc \
 --exclude=*_s390.deb --exclude=*_s390.udeb --exclude=*-s390 \
 --exclude=*_sparc.deb --exclude=*_sparc.udeb --exclude=*-sparc \
 --exclude=*-alpha/* --exclude=*-arm/* --exclude=*-hppa/* --exclude=*-ia64/* \
 --exclude=*-m68k/* --exclude=*-mips/* --exclude=*-mipsel/* \
 --exclude=*-powerpc/* --exclude=*-s390/* --exclude=*-sparc/* \
 --password-file /ftp/mirror/rsync.d/ftp.uk.debian.org \
 leblanc@ftp.uk.debian.org::debian/* /pub/linux/distributions/Debian

Every day this command leaves some wrong files in the tree; here is
an ls -l of /pub/linux/distributions/Debian/dists/sarge:

-rw-r--r--    1 zlsiial  root      6559910 May  5 20:22 Contents-alpha.gz
-rw-r--r--    1 zlsiial  sys       6353912 May  8 20:18 Contents-arm.gz
-rw-r--r--    1 zlsiial  sys       6322515 May  9 20:20 Contents-hppa.gz
-rw-r--r--    1 zlsiial  sys       7315705 May  9 20:23 Contents-i386.gz
-rw-r--r--    1 zlsiial  root      6409582 May 11 20:18 Contents-ia64.gz
-rw-r--r--    1 zlsiial  root      6276314 May  2 20:18 Contents-m68k.gz
-rw-r--r--    1 zlsiial  root      6222678 May  2 20:20 Contents-mips.gz
-rw-r--r--    1 zlsiial  root      6247077 May  3 20:19 Contents-mipsel.gz
-rw-r--r--    1 zlsiial  root      6512485 May  3 20:22 Contents-powerpc.gz
-rw-r--r--    1 zlsiial  root      6328820 May  4 20:22 Contents-s390.gz
-rw-r--r--    1 zlsiial  root      6332818 May  4 20:25 Contents-sparc.gz
-rw-r--r--    1 zlsiial  root      6353912 May 12 20:23 Release
-rw-r--r--    1 zlsiial  root      6322515 May 12 20:23 Release.gpg
drwxr-xr-x    4 zlsiial  root         2048 Jul 19  2002 contrib
drwxr-xr-x    6 zlsiial  root         2048 May  3 13:38 main

Note that the Release and Release.gpg files are the same size as
Contents-arm.gz and Contents-hppa.gz respectively.  I have checked,
and these files do _not_ have the same inode numbers, though their
contents are in fact the same.  To correct this problem, I run
(currently manually) a different rsync job on the same machine:

rsync -r -R -l -t --delete --exclude=/Incoming --exclude=/UploadQueue \
 --exclude=*_alpha.deb --exclude=*_alpha.udeb --exclude=*-alpha \
 --exclude=*_arm.deb --exclude=*_arm.udeb --exclude=*-arm \
 --exclude=*_hppa.deb --exclude=*_hppa.udeb --exclude=*-hppa \
 --exclude=*_ia64.deb --exclude=*_ia64.udeb --exclude=*-ia64 \
 --exclude=*_m68k.deb --exclude=*_m68k.udeb --exclude=*-m68k \
 --exclude=*_mips.deb --exclude=*_mips.udeb --exclude=*-mips \
 --exclude=*_mipsel.deb --exclude=*_mipsel.udeb --exclude=*-mipsel \
 --exclude=*_powerpc.deb --exclude=*_powerpc.udeb --exclude=*-powerpc \
 --exclude=*_s390.deb --exclude=*_s390.udeb --exclude=*-s390 \
 --exclude=*_sparc.deb --exclude=*_sparc.udeb --exclude=*-sparc \
 --exclude=*-alpha/* --exclude=*-arm/* --exclude=*-hppa/* --exclude=*-ia64/* \
 --exclude=*-m68k/* --exclude=*-mips/* --exclude=*-mipsel/* \
 --exclude=*-powerpc/* --exclude=*-s390/* --exclude=*-sparc/* \
 --password-file /ftp/mirror/rsync.d/ftp.uk.debian.org \
 leblanc@ftp.uk.debian.org::debian/dists/* /pub/linux/distributions/Debian

This is exactly the same as the first job with one exception: on the
last line '::debian/*' is changed to '::debian/dists/*'.  After
running
this job, the command 'ls -l
/pub/linux/distributions/Debian/dists/sarge'
produces this output:

-rw-r--r--    1 zlsiial  root      6559910 May  5 20:22 Contents-alpha.gz
-rw-r--r--    1 zlsiial  sys       6353912 May  8 20:18 Contents-arm.gz
-rw-r--r--    1 zlsiial  sys       6322515 May  9 20:20 Contents-hppa.gz
-rw-r--r--    1 zlsiial  sys       7315705 May  9 20:23 Contents-i386.gz
-rw-r--r--    1 zlsiial  root      6409582 May 11 20:18 Contents-ia64.gz
-rw-r--r--    1 zlsiial  root      6276314 May  2 20:18 Contents-m68k.gz
-rw-r--r--    1 zlsiial  root      6222678 May  2 20:20 Contents-mips.gz
-rw-r--r--    1 zlsiial  root      6247077 May  3 20:19 Contents-mipsel.gz
-rw-r--r--    1 zlsiial  root      6512485 May  3 20:22 Contents-powerpc.gz
-rw-r--r--    1 zlsiial  root      6328820 May  4 20:22 Contents-s390.gz
-rw-r--r--    1 zlsiial  root      6332818 May  4 20:25 Contents-sparc.gz
-rw-r--r--    1 zlsiial  root        22828 May 12 20:23 Release
-rw-r--r--    1 zlsiial  root          315 May 12 20:23 Release.gpg
drwxr-xr-x    4 zlsiial  root         2048 Jul 19  2002 contrib
drwxr-xr-x    6 zlsiial  root         2048 May  3 13:38 main

Here you can see that the Release and Release.gpg files are the correct
sizes.  I have noticed this problem almost daily in the dists directories,
but I have no idea whether there are files in the pool directories which
are the wrong size as well.  All of our servers are running woody, and we
run rsync version 2.6.2-1, which I compiled from the (patched) source
using debuild.

A lot of the log of the faulty rsync run is taken up with error messages
like those below.  The directories exist and are readable; for example,

ll /pub/linux/distributions/Debian/pool/main/m/mdcfg
total 228
-rw-r--r--    1 zlsiial  root        26262 May  5 04:02
mdcfg-utils_0.17_all.udeb
-rw-r--r--    1 zlsiial  root        35822 May 19 13:32
mdcfg-utils_0.18_all.udeb
-rw-r--r--    1 zlsiial  root          570 May  5 04:02 mdcfg_0.17.dsc
-rw-r--r--    1 zlsiial  root        76964 May  5 04:02 mdcfg_0.17.tar.gz
-rw-r--r--    1 zlsiial  root         3138 May  5 04:02 mdcfg_0.17_all.udeb
-rw-r--r--    1 zlsiial  root          570 May 19 13:32 mdcfg_0.18.dsc
-rw-r--r--    1 zlsiial  root        82425 May 19 13:32 mdcfg_0.18.tar.gz
-rw-r--r--    1 zlsiial  root         3182 May 19 13:32 mdcfg_0.18_all.udeb

I would be grateful for any help or advice with this problem.

     -- Owen
     LeBlanc@mcc.ac.uk

read errors mapping "pool/main/m/mdcfg" (in debian): (21) Is a
directory
read errors mapping "pool/main/n/neon0.23" (in debian): (21) Is a
directory
read errors mapping "pool/main/o/openggsn" (in debian): (21) Is a
directory
read errors mapping "pool/main/s/snd" (in debian): (21) Is a directory

On Thu, May 20, 2004 at 08:21:56AM +0100, Dr A V Le Blanc
wrote:
> Every day this command leaves some wrong files in the tree
Sounds like the rsync daemon you are contacting is running 2.6.1.  If
so, just remove the -R option and all should be well (it's not needed by
the command you're using).  Also, since you're using --delete, get rid
of the trailing '*' in the source path -- it is inhibiting deletions in
the base dir of the transfer (leave a trailing '/').  After that, advise
the folks at ftp.uk.debian.org that they should update their rsync to
2.6.2.

..wayne..

On Thu, May 20, 2004 at 11:17:58AM -0700, Wayne Davison
wrote:
> Sounds like the rsync daemon you are contacting is running 2.6.1.  If
> so, just remove the -R option and all should be well (it's not needed
by
> the command you're using).  Also, since you're using --delete, get
rid
> of the trailing '*' in the source path -- it is inhibiting
deletions in
> the base dir of the transfer (leave a trailing '/').  After that,
advise
> the folks at ftp.uk.debian.org that they should update their rsync to
> 2.6.2.
Many thanks to Wayne Davison for his advice in this.  I have made
the changes he suggested, and the mirror has worked correctly last
night.  I shall also advise the maintainer of ftp.uk.debian.org
that he needs to update his rsync.

This bug can now be considered closed.

     -- Owen
     LeBlanc@mcc.ac.uk