rsync - Oct 2008 - assert problem

rsync -rltpcvxH --progress --delete /mnt/sdf1/dvd .
....
  lots of stuff works, but then....
....
dvd/setdir/www/web_admin-1.0-noarch-001.tgz
    1306806 100%    3.34MB/s    0:00:00 (xfer#927, to-check=1004/2734)
dvd/setdir/xlibs/
deleting dvd/setdir/xlibs/gstreamer-0.10.21-i586-001.tgz
deleting dvd/setdir/xlibs/gst_plugins_base-0.10.21-i586-001.tgz
dvd/setdir/xlibs/gst_plugins_base-0.10.20-i586-001.tgz
    1243619 100%   96.23MB/s    0:00:00 (xfer#928, to-check=1030/3129)
dvd/setdir/xlibs/gstreamer-0.10.20-i586-001.tgz
    2053355 100%   48.96MB/s    0:00:00 (xfer#929, to-check=1029/3129)
rsync: hlink.c:273: check_prior: Assertion `node->data != ((void *)0)'
failed.
rsync: writefd_unbuffered failed to write 212 bytes [sender]: Broken pipe (32)
deleting dvd/source/base/e2fsprogs/e2fsprogs-1.41.3.tar.gz.asc
deleting dvd/source/base/e2fsprogs/e2fsprogs-1.41.3.tar.gz
deleting dvd/source/development/cmake/cmake-2.6.2.tar.gz
rsync: connection unexpectedly closed (18268 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at
io.c(595) [sender=3.0.5pre1]

/mnt/sdf1 is a USB thumb drive, 4GB.  If I use tar, the files are fine when
I copy them.  File system is ext2.  Mounted rw,nosuid,nodev,noatime.

. is ext3 on a LUKS encrypted LVM2 on top of software RAID 5.  Mounted rw.

Everything execpt rsync is fine (for about 6 months) on the machine,
the thumb drive, and the LUKS/LVM2/RAID5 drive.

I reported this once before and you suggested it was an out-of-memory
issue, but if it is a memory issue, why doesn't rsync say 'out of
memory'
instead of this cryptic assert?

I am testing rsync-2.0.5pre1.
Kernel is 2.6.26.6

I run this as root, and here is the output of 'ulimit -a' on bash:
BLS #ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 32768
max locked memory       (kbytes, -l) 4096
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 512
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

BLS #cat /proc/meminfo
MemTotal:      3625756 kB
MemFree:        132820 kB
Buffers:         20420 kB
Cached:        3322020 kB
SwapCached:         12 kB
Active:         185264 kB
Inactive:      3253484 kB
HighTotal:     2751360 kB
HighFree:         7316 kB
LowTotal:       874396 kB
LowFree:        125504 kB
SwapTotal:     8393952 kB
SwapFree:      8393876 kB
Dirty:             284 kB
Writeback:           0 kB
AnonPages:       96308 kB
Mapped:          38556 kB
Slab:            36624 kB
SReclaimable:    27568 kB
SUnreclaim:       9056 kB
PageTables:       1956 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:  10206828 kB
Committed_AS:   329040 kB
VmallocTotal:   114680 kB
VmallocUsed:      6788 kB
VmallocChunk:   107652 kB

BLS #free
            total       used       free     shared    buffers     cached
Mem:       3625756    3493040     132716          0      20440    3322072
-/+ buffers/cache:     150528    3475228
Swap:      8393952         76    8393876

BLS #du -sh /mnt/sdf1/dvd
4.2G    /mnt/sdf1/dvd

BLS #du -sh dvd
4.2G    dvd

With all of that RAM and SWAP, do you really think this is a memory problem?
Every time I have rsync crash, it is on a hard link.  rsync 3.0.3 and
3.0.4 also have the same problem for me.

On Tue, Oct 14, 2008 at 08:24:31AM +0700, BuraphaLinux Server
wrote:
> rsync: hlink.c:273: check_prior: Assertion `node->data != ((void
*)0)' failed.
Everywhere in the hlink.c code that sets a node, sets a data value and
ensures that it is non-zero, so there's no code path in the logic that
can allow that assert to trigger.  Thus, it would appear that there has
to be some kind of a corruption bug somewhere.  It would help to have a
reproducible test case so I can look to see what exactly is going on.
Failing that, it could help if you ran the 3.1.0dev version (see the
nightly tar files or git repository) as this would allow you to run with
the "--debug=hlink5,hash" option, and that might provide some insight
into what is going on in the hashtable and hard-link code.  I should
also look into a patch that would dump the hashtable when it discovers
it to be invalid, and that might help to figure out what has happened.

..wayne..