rsync - Nov 2009 - DO NOT REPLY [Bug 6916] New: Avoid bundling a modified zlib

https://bugzilla.samba.org/show_bug.cgi?id=6916

           Summary: Avoid bundling a modified zlib
           Product: rsync
           Version: 3.1.0
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P3
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: matt at mattmccutchen.net
         QAContact: rsync-qa at samba.org


Fedora wants us not to bundle a modified version of zlib because it makes it
harder to ensure that security updates are applied everywhere they are needed. 
See:

https://bugzilla.redhat.com/show_bug.cgi?id=495310

Wayne, I don't know what (if anything) you want to do about this, but I
didn't
want the issue to be forgotten.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6916





------- Comment #1 from wayned at samba.org  2009-11-25 19:54 CST -------
The only thing for rsync to potentially do is to have a build option for using
a compatible, shared zlib.  The redhat bug has links to rsync's needed
changes.
 If fedora patched their version of zlib with the Z_INSERT_ONLY support, it
should (presumably) be usable by normal zlib apps, and rsync as well.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6916





------- Comment #2 from matt at mattmccutchen.net  2009-11-27 23:57 CST -------
(In reply to comment #1)
> The only thing for rsync to potentially do is to have a build option for
using
> a compatible, shared zlib.
What happened to, "Tridge has mentioned to me that he thinks the slight
savings in compression wasn't really worth all the hassle, so some
future version of rsync may get rid of this optimization and start
to use the system's zlib"?

http://lists.samba.org/archive/rsync/2005-December/014180.html

I'm not saying that is definitely the right decision, just that it deserves
fair consideration.
> If fedora patched their version of zlib with the Z_INSERT_ONLY support, it
> should (presumably) be usable by normal zlib apps, and rsync as well.
The Fedora folks don't wish to be maintain the patched zlib themselves:

https://bugzilla.redhat.com/show_bug.cgi?id=495310#c7


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6916


jzeleny at redhat.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jzeleny at redhat.com




------- Comment #3 from jzeleny at redhat.com  2009-12-10 05:49 CST -------
Hi, I'm currently the maintainer of rsync in Fedora. As I was informed, zlib
upstream is willing to make some changes, so rsync doesn't need its bundled
version. But some discussion about this should take place first. To quote Mark
Adler, zlib's developer: "I would change how to decide when to throw
away the
history so that it would not happen so often as the rsync patch would do it,
but would retain the desired rsyncability of the compressed streams.  If I can
constrain the impact to the compression efficiency, I might even make it the
default behavior."


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6916





------- Comment #4 from matt at mattmccutchen.net  2009-12-10 10:19 CST -------
The gzip enhancement bundled with rsync is Z_INSERT_ONLY, which lets rsync
enter matched blocks in the gzip history without transmitting them in the
compressed stream.  The quote in comment #3 seems to be about something
completely different: the --rsyncable option that makes compressed files more
amenable to delta-transfer.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6916





------- Comment #5 from jzeleny at redhat.com  2009-12-17 02:23 CST -------
Well, it's possible. I'm not rsync maintainer for long, so I don't
have that
much knowledge about the situation. But my point was that zlib's upstream is
willing to consider some changes in order to reach compromise. Could you please
contact them and try to discuss the situation, so it isn't necessary to
bundle
modified zlib? Thanks


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6916





------- Comment #6 from matt at mattmccutchen.net  2010-02-07 19:32 CST -------
Some technical details: Z_INSERT_ONLY is used on the sender side to enter
matched blocks in the gzip history without transmitting them in the compressed
stream.  The receiver-side counterpart is the code in see_deflate_token in
token.c which inserts into the input stream a synthesized zlib-format block
containing the matched data.

The receiver-side behavior was possible to implement based on knowledge of the
zlib format without actually modifying zlib.  Implementing the sender-side
behavior in the same way would require excising the matched data from zlib's
output and flushing in order to make the excision possible.  I guess that was
determined to be too hard, though it's not clear to me what is so hard about
it.

Wayne said previously that the receiver-side implementation is a hack and
should be moved into zlib if Z_INSERT_ONLY is upstreamed:

http://lists.samba.org/archive/rsync/2005-December/014180.html

I agree.  But note that after doing that, we would still have a dependency on
the zlib format in send_deflated_token, which fiddles with the "0, 0, ff,
ff"
bytes generated by flushing.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=6916

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #7 from Wayne Davison <wayned at samba.org> 2011-12-24
21:00:51 UTC ---
3.1.0 has a configure option --with-included-popt=no that can be used to get
rsync to use the system's popt library.  I changed the rsync code to be able
to
support this, and in my limited testing it appears to work fine.

There is going to be some new support for adding dictionary data during the
transfer in the official zlib code, but I haven't yet tested to see if using
this new support is identical to the rsync kluges or not.  If it is, the rsync
code will eventually be updated to use the official support.

As for the "0, 0, ff, ff" bytes, Mark Adler mentions that these are
documented,
so it should be fine to tweak things based on that.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=6916

--- Comment #8 from Matt McCutchen <matt at mattmccutchen.net> 2011-12-26
01:43:02 UTC ---
(In reply to comment #7)
> 3.1.0 has a configure option --with-included-popt=no that can be used to
get
> rsync to use the system's popt library.
That should say "zlib", not "popt".

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.