Stuart Anderson
2002-Feb-01 10:26 UTC
error in rsync protocol data stream (code 12) at token.c(288)
I am getting the following error when mirroring part of the RedHat
distribution tree over a slow connection (~T1 speed). When running
over a faster network (100BaseT) the problem does not appear. Note,
the problem file a large 600MB ISO image, whereas other small files
appear to be fine.
rsync: open connection using /path/ssh remote.host /path/rsync --server
-vlHogDtprRz --timeout=600 --delete --force . /
rsync: building file list...
rsync: 90108 files to consider.
export/mirror/linux/redhat/7.1/en/iso/i386/
/export/mirror/linux/redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso
deflate on token returned 0 (16384 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(288)
rsync finished
The file has in fact been mirrored correctly (md5sum is identical),
however, the file is left with today's date rather synchronizing the
file times as specified.
A repeat of the rsync command with the file already on the destination
machines results in the same error.
This is running between two Sun servers running Solaris 8 and using:
rsync version 2.5.2 protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no
IPv6,
64-bit system inums, 64-bit internal inums
--
Stuart Anderson sba@srl.caltech.edu http://www.srl.caltech.edu/personnel/sba
Dave Dykstra
2002-Feb-02 01:38 UTC
error in rsync protocol data stream (code 12) at token.c(288)
On Thu, Jan 31, 2002 at 03:26:16PM -0800, Stuart Anderson wrote:> I am getting the following error when mirroring part of the RedHat > distribution tree over a slow connection (~T1 speed). When running > over a faster network (100BaseT) the problem does not appear. Note, > the problem file a large 600MB ISO image, whereas other small files > appear to be fine. > > rsync: open connection using /path/ssh remote.host /path/rsync --server -vlHogDtprRz --timeout=600 --delete --force . / > rsync: building file list... > rsync: 90108 files to consider. > export/mirror/linux/redhat/7.1/en/iso/i386/ > /export/mirror/linux/redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso > deflate on token returned 0 (16384 bytes left) > rsync error: error in rsync protocol data stream (code 12) at token.c(288) > rsync finished > > > The file has in fact been mirrored correctly (md5sum is identical), > however, the file is left with today's date rather synchronizing the > file times as specified. > > A repeat of the rsync command with the file already on the destination > machines results in the same error. > > > This is running between two Sun servers running Solaris 8 and using: > > rsync version 2.5.2 protocol version 26 > Copyright (C) 1996-2002 by Andrew Tridgell and others > <http://rsync.samba.org/> > Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6, > 64-bit system inums, 64-bit internal inumsThat's a different symptom than what we've seen before, but have you applied the following patch that I posted on Tuesday? Rsync 2.5.2 is badly broken without it. - Dave Dykstra --- match.c.O Tue Jan 29 15:31:37 2002 +++ match.c Tue Jan 29 15:31:54 2002 @@ -246,7 +246,7 @@ match. The 3 reads are caused by the running match, the checksum update and the literal send. */ - if (offset-last_match >= CHUNK_SIZE+s->n && + if (offset-last_match >= CHUNK_SIZE+(int)s->n && (end-offset > CHUNK_SIZE)) { matched(f,s,buf,offset - s->n, -2); }
Stuart Anderson
2002-Feb-02 04:09 UTC
error in rsync protocol data stream (code 12) at token.c(288)
According to sba:> According to Dave Dykstra: > > On Thu, Jan 31, 2002 at 03:26:16PM -0800, Stuart Anderson wrote: > > > I am getting the following error when mirroring part of the RedHat > > > distribution tree over a slow connection (~T1 speed). When running > > > over a faster network (100BaseT) the problem does not appear. Note, > > > the problem file a large 600MB ISO image, whereas other small files > > > appear to be fine. > > > > > > rsync: open connection using /path/ssh remote.host /path/rsync --server -vlHogDtprRz --timeout=600 --delete --force . / > > > rsync: building file list... > > > rsync: 90108 files to consider. > > > export/mirror/linux/redhat/7.1/en/iso/i386/ > > > /export/mirror/linux/redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso > > > deflate on token returned 0 (16384 bytes left) > > > rsync error: error in rsync protocol data stream (code 12) at token.c(288) > > > rsync finished > > > > > > > > > The file has in fact been mirrored correctly (md5sum is identical), > > > however, the file is left with today's date rather synchronizing the > > > file times as specified. > > > > > > A repeat of the rsync command with the file already on the destination > > > machines results in the same error. > > > > > > > > > This is running between two Sun servers running Solaris 8 and using: > > > > > > rsync version 2.5.2 protocol version 26 > > > Copyright (C) 1996-2002 by Andrew Tridgell and others > > > <http://rsync.samba.org/> > > > Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6, > > > 64-bit system inums, 64-bit internal inums > > > > > > > > That's a different symptom than what we've seen before, but have you > > applied the following patch that I posted on Tuesday? Rsync 2.5.2 is badly > > broken without it. > > > > - Dave Dykstra > > > > > > --- match.c.O Tue Jan 29 15:31:37 2002 > > +++ match.c Tue Jan 29 15:31:54 2002 > > @@ -246,7 +246,7 @@ > > match. The 3 reads are caused by the > > running match, the checksum update and the > > literal send. */ > > - if (offset-last_match >= CHUNK_SIZE+s->n && > > + if (offset-last_match >= CHUNK_SIZE+(int)s->n && > > (end-offset > CHUNK_SIZE)) { > > matched(f,s,buf,offset - s->n, -2); > > } > > > No I did not apply the patch, but I verified the same problem with > rsync-2.4.8, is that sufficient? > > I also determined that my comment above about working/not-working on > fast/slow network was misleading. The important discriminant is whether > compression was turned on or not: works without out it, fails with it > (my driver script was automatically selecting compression for slow > networks). > > I also found on deja-news reports that this is a known and fixed problem > with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3? >I have verified that the (int) typecast fix to match.c does not solve this problem. I have also determined that a good way to reproduce this problem is to touch (change the timestamp) on some large iso files and then attempt to re-rsync them with compression turned on, i.e., no need to re-push the actual file over the network. -- Stuart Anderson sba@srl.caltech.edu http://www.srl.caltech.edu/personnel/sba
Stuart Anderson
2002-Feb-02 06:48 UTC
error in rsync protocol data stream (code 12) at token.c(288)
According to Albert Chin:> On Fri, Feb 01, 2002 at 08:51:17AM -0800, Stuart Anderson wrote: > > I also found on deja-news reports that this is a known and fixed problem > > with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3? > > Jos Backus posted a patch to do this a few days ago. Try applying his > patch and reporting back to the list if it solves your problem. >Unfortunately, this does not solve my problem, i.e., the following fix in zlib-1.1.3 is not the one that rsync is triggering: - fix "an inflate input buffer bug that shows up on rare but persistent occasions" (Mark) At this point the workaround is to turn off compression. -- Stuart Anderson sba@srl.caltech.edu http://www.srl.caltech.edu/personnel/sba
Stuart Anderson
2002-Apr-03 03:10 UTC
error in rsync protocol data stream (code 12) at token.c(288)
While this does appear to work in 2.5.4, I suspect that is because of the 2.5.4 whole-file bug. With 2.5.5 it fails again, with the following error message: deflate on token returned 0 (16384 bytes left) rsync error: error in rsync protocol data stream (code 12) at token.c(288) According to sba:> I did one quick test with an unmodified rsync-2.5.4, and it seems to have fixed > this problem. However, someone else should verify this. > > According to Dave Dykstra: > > I did some playing with this. It takes a while because the file is so > > big. I don't really know what's going on, but I did find that deflate() is > > returning at line 493 of zlib/deflate.c just after a comment that says > > "this is not an error situation". So if I change like 285 of token.c to > > delete the "|| tx_strm.avail_in != 0" it all seems to work properly. I'm > > not sure if that's an OK thing to do or not, though. Somebody who knows > > something about zlib would have to look at it. > > > > - Dave Dykstra > > > > On Wed, Feb 06, 2002 at 04:21:47PM -0800, Stuart Anderson wrote: > > > The newer zlib-1.1.3 patch still has the problem. > > > > > > According to Dave Dykstra: > > > > I was able to reproduce this between a Solaris 2.7 and 2.8 machine with and > > > > without ssh, although I don't have any time to debug it today. Can > > > > somebody try applying Jos Backus latest zlib 1.1.3 upgrade patch to see > > > > if that helps? > > > > > > > > - Dave > > > > > > > > On Fri, Feb 01, 2002 at 02:01:03PM -0800, Stuart Anderson wrote: > > > > > According to Dave Dykstra: > > > > > > On Fri, Feb 01, 2002 at 09:09:18AM -0800, Stuart Anderson wrote: > > > > > > > According to sba: > > > > > > > > > > > > > > No I did not apply the patch, but I verified the same problem with > > > > > > > > rsync-2.4.8, is that sufficient? > > > > > > > > > > > > No, because rsync-2.4.8 include the same security patch that went into > > > > > > 2.5.2 last weekend. If you could try 2.4.6 or 2.5.1 that would tell us if > > > > > > it was some other problem introduced last weekend or not. > > > > > > > > > > The problem was introduced in version 2.4.4 and exists in all later > > > > > versions up to and including 2.5.2. Whereas version 2.4.3 and 2.4.0 > > > > > are both fine (the only earlier versions checked). > > > > > > > > > > The problem only exists for compression and large files. > > > > > > > > > > I encountered and tested the various rsync versions with the file, > > > > > redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso > > > > > gotten from your nearest RedHat mirror, but it is more general > > > > > than just this file. > > > > > > > > > > However, the problem is not reproducible with any of the 1018 "small" > > > > > files from redhat/7.1/en/os/i386/RedHat/RPMS which range in size from > > > > > 1810 Byte to 21541821 Byte. > > > > > > > > > > > > > > > To reproduce: > > > > > > > > > > 1) rsync --archive a large file (ISO size?) to a remote machine > > > > > 2) /bin/touch it on the destination machine > > > > > 3) rsync --archive --compress again > > > > > 4) Check for error message: "deflate on token returned 0 (16384 bytes left)" > > > > > > > > > > I don't know if it matters but I am running rsync between two Solaris 8 > > > > > machines, and using ssh as the transport: > > > > > > > > > > > ssh -V > > > > > OpenSSH_2.9p2, SSH protocols 1.5/2.0, OpenSSL 0x0090601f > > > > > > > > > > -- > > > > > Stuart Anderson sba@srl.caltech.edu http://www.srl.caltech.edu/personnel/sba > > > > > > -- > > > Stuart Anderson sba@srl.caltech.edu http://www.srl.caltech.edu/personnel/sba > > -- > Stuart Anderson sba@srl.caltech.edu http://www.srl.caltech.edu/personnel/sba-- Stuart Anderson sba@srl.caltech.edu http://www.srl.caltech.edu/personnel/sba