https://bugzilla.samba.org/show_bug.cgi?id=14338
Bug ID: 14338
Summary: ZSTD support
Product: rsync
Version: 3.1.3
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P5
Component: core
Assignee: wayne at opencoder.net
Reporter: Rsync at ml.breakpoint.cc
QA Contact: rsync-qa at samba.org
Created attachment 15898
--> https://bugzilla.samba.org/attachment.cgi?id=15898&action=edit
zstd support
zstd compression was announced as "good compression with high
throughput" so I gave it a try. With zlib, on high speed links the CPU
is usually the bottle neck. With zstd I'm able to fill a 200Mbit link :)
zstd detection happens automatically via pkg-config. No zstd header means
no error about missing zstd. So that should be okay. However, pkg-config
is now kind of required?
I duplicated the zlib code and replaced it with zstd hooks once I
understood what was going on. I made a few local tests with and without
tokens and it seems to work. The compression can be selected with `-Z'
option. By default `0' is used as the compression level which is a
special default (it currently maps to 3). The compression level can be
specified by the same option as for zlib.
The compressor feeds data into zstd and starts sending data once the
outgoing buffer is full or when a flush is requested. That flush will
close the current compression block and create a new one for the
following transfer (saving the internal compression / history state).
The decompressor allocates space for two blocks. Should one block
contain more data, then it will loop more often.
--
You are receiving this mail because:
You are the QA Contact for the bug.
https://bugzilla.samba.org/show_bug.cgi?id=14338
Wayne Davison <wayne at opencoder.net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
--- Comment #1 from Wayne Davison <wayne at opencoder.net> ---
Thanks for the patch! I have transformed it into an official
"zstd.diff" file
in the rsync-patches git repo using the new negotiated compression code. I
imagine it will be included in master soon.
--
You are receiving this mail because:
You are the QA Contact for the bug.
https://bugzilla.samba.org/show_bug.cgi?id=14338 --- Comment #2 from Wayne Davison <wayne at opencoder.net> --- I'll note that I had to install an upgraded zstd lib to get this to work. I first tried 1.3.8 (since it was mentioned in the configure check) but it didn't have the right exported functions. Then then tried 1.4.5, which worked fine. -- You are receiving this mail because: You are the QA Contact for the bug.
https://bugzilla.samba.org/show_bug.cgi?id=14338
Wayne Davison <wayne at opencoder.net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |RESOLVED
Resolution|--- |FIXED
--- Comment #3 from Wayne Davison <wayne at opencoder.net> ---
Actually, I decided it would be better to just go ahead and add this and the
lz4 code into master and put them lower in the negotiation list, which will
make them easier to test (and they're easy to disable via configure, as
needed).
--
You are receiving this mail because:
You are the QA Contact for the bug.
https://bugzilla.samba.org/show_bug.cgi?id=14338 --- Comment #4 from Sebastian A. Siewior <Rsync at ml.breakpoint.cc> --- Thanks for the merge! Sorry for the version. According zstd's git v1.3.8 contains ZSTD_compressStream2(). Debian Buster ships v1.3.8 and it does not find the function. Buh. Looking closer it is marked as experimental and only avaible for static linking. As per header file history, since v1.4.0 it is available for linking. I'm going to send a pull for move the function for ZSTD_compressStream2() which hopefully should cover it. Regarding the compression preferences. If I understand it correctly: - "zlibx" is the external zlib. - "zlib" is the internal zlib which feeds blocks into the the zlib history/dictonary without sending it over the wire. Could those two be swapped? That zlibX sounds like zlib-eXtended. Or does it stand for zlib-eXternal? Can we move zlib* at the end of the supported algorihms in terms of preferences? My motivation was the lower for zstd was to use decent compression with _low_ CPU usage. The rsync.yo file says that this "--compress-choice=zlibx" can be used since version 3.1.1. Is this working for negotiation and would support say "zstd,lz4,zlib,zlibx" with zlibx with 3.1.1 and zstd on master or did misunderstand it? Sebastian -- You are receiving this mail because: You are the QA Contact for the bug.
https://bugzilla.samba.org/show_bug.cgi?id=14338 --- Comment #5 from Randall S. Becker <rsbecker at nexbridge.com> --- Wondering about git 1.3.8, which is many years old. (Git platform maintainer here) -- You are receiving this mail because: You are the QA Contact for the bug.
https://bugzilla.samba.org/show_bug.cgi?id=14338 --- Comment #6 from Wayne Davison <wayne at opencoder.net> --- The name "zlibx" is both for external and excluding of unsent data. It's listed first because it should be the safer choice with the weirdness in external zlib going on. The new compress items are listed afterwards because they're untested. As for what you would like the order to be, please refer to the manpage, especially the RSYNC_COMPRESS_LIST environment variable. Feel free to set it to whatever order you like. -- You are receiving this mail because: You are the QA Contact for the bug.
Possibly Parallel Threads
- [Bug 14390] New: Feature request: don't fail if using "-z" transferring to rsync complied with --with-included-zlib=no
- [PATCH 0/1] ZSTD compression support for OpenSSH
- [PATCH 1/1] Add support for ZSTD compression
- ZSTD compression support for OpenSSH
- [PATCH 0/5] ZSTD compression support for OpenSSH