bugzilla-daemon at bugzilla.mindrot.org
2018-Jul-30 18:19 UTC
[Bug 2888] New: Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Bug ID: 2888
Summary: Consider adding other compression schemes (lz4, zstd)
Product: Portable OpenSSH
Version: 7.7p1
Hardware: All
OS: All
Status: NEW
Severity: enhancement
Priority: P5
Component: Miscellaneous
Assignee: unassigned-bugs at mindrot.org
Reporter: fabio.pedretti at unibs.it
I did some test and when enabling compression my CPU usage goes at 100%
at about 70-150Mb/s, depending on file compressibility. So it is not
very useful with a Gigabit Ethernet link.
To improve this I would propose to add other compression schemes, such
as lz4 ( https://github.com/lz4/lz4 ) or zstd (
https://github.com/facebook/zstd ).
They are order of magnitudes faster than current zlib-6 compression and
easily able to exceed Gigabit Ethernet speed.
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2018-Jul-30 18:20 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888 --- Comment #1 from Fabio <fabio.pedretti at unibs.it> --- See also bug #2887. -- You are receiving this mail because: You are watching the assignee of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2019-Sep-23 17:43 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Scott Baker <scott at perturb.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |scott at perturb.org
--- Comment #2 from Scott Baker <scott at perturb.org> ---
I came here to request alternate compression methods, and found someone
had beat me to it. Consider this my +1
Currently OpenSSH only supports GZIP compression which has speed
constraints on modern CPUs. Enabling compression on file transfers over
a gigabit link actually increases the amount of time taken to transfer
large files due to the limitations of GZIP.
Please consider adding support for alternate compression methods. LZ4
and ZSTD would be excellent choices.
Even better would be support for external compression methods like
`tar` does with `-I`. That way in the future if new/better compression
algorithms are found SSH will automatically be able to utilize them.
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2019-Sep-23 18:03 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888 --- Comment #3 from Scott Baker <scott at perturb.org> --- Transferring a 1.8GB VM image on a gigabit network: ``` root at green(~) :time scp test.bin yellow:/dev/null test.bin 100% 1821MB 108.0MB/s 00:16 # Enabling compression is significantly slower root at green(~) :time scp -C test.bin yellow:/dev/null test.bin 100% 1821MB 20.0MB/s 01:30 ``` Alternately using this "hack" with `cat`, `ssh`, and ZSTD I was able to decrease transfer time by ~40%. ``` root at green(~) :time cat test.bin | zstd | ssh yellow 'zstd - -d > /dev/null' real 0m9.691s user 0m12.178s sys 0m4.458s ``` For transferring large files, and/or groups of large files via scp/rsync having a modern compression method would be a **huge** win. -- You are receiving this mail because: You are watching the assignee of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2019-Dec-12 22:22 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Luke <lukenbutters at gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |lukenbutters at gmail.com
--- Comment #4 from Luke <lukenbutters at gmail.com> ---
please add support for this.
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2020-Apr-06 19:33 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Sebastian A. Siewior <openssh at ml.breakpoint.cc> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |openssh at ml.breakpoint.cc
--- Comment #5 from Sebastian A. Siewior <openssh at ml.breakpoint.cc> ---
Created attachment 3373
--> https://bugzilla.mindrot.org/attachment.cgi?id=3373&action=edit
Support for zstd against V_8_2_P1
This archive contains four patches against openssh v8.2.p1.
Would love some feedback if the expectations regarding scp
performance/CPU load are met.
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2020-Jul-18 17:55 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888 --- Comment #6 from Fabio <fabio.pedretti at unibs.it> --- @Sebastian Thanks for your patches. I'd suggest sending them on the OpenSSH Developers list: https://www.openssh.com/list.html and eventually as a Pull request on github: https://github.com/openssh/openssh-portable/pulls -- You are receiving this mail because: You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2023-Sep-14 11:42 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Celeste Liu <CoelacanthusHex at gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |CoelacanthusHex at gmail.com
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2025-Feb-24 09:07 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Paul Szabo <psz at maths.usyd.edu.au> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |psz at maths.usyd.edu.au
Attachment #3862| |ok+
Flags| |
--- Comment #7 from Paul Szabo <psz at maths.usyd.edu.au> ---
Created attachment 3862
--> https://bugzilla.mindrot.org/attachment.cgi?id=3862&action=edit
Patch for zstd support on 9,2p
Dear OpenSSH developers,
PLEASE add support of ZSTD compression!
ZSTD seems "better" than ZLIB: it achieves a better compression ratio
for X11 traffic at a lower CPU usage, allowing use of "ssh -X" when
ZLIB
would "fail" (being unresponsive). ZSTD seems to be available for most
Linux distributions; I do not know about Windows or Mac or other
operating systems.
The patches attached (against version 9.2p as distributed by Debian)
are
essentially a copy those of Sebastian Andrzej Siewior in
https://lists.mindrot.org/pipermail/openssh-unix-dev/2023-February/040592.html
https://lists.mindrot.org/pipermail/openssh-unix-dev/2023-February/040593.html
These patches have been in active (heavy) use on my machines since
end of 2023, without any issues.
These patches are fully backwards compatible with un-patched remote
ends. When both client and server are patched, one can use "ssh -C" or
"ssh -o Compression=yes" to get ZSTD, picked as best by negotiation;
or
use "ssh -o Compression=zlib" or "ssh -o Compression=zstd"
to get that.
When either the client or server are un-patched, using "ssh -C" or
"ssh
-o Compression=yes" gets ZLIB, as it always did.
Thanks, Paul
--
Paul Szabo psz at maths.usyd.edu.au
www.maths.usyd.edu.au/u/psz
School of Mathematics and Statistics University of Sydney
Australia
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2025-Feb-24 09:09 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Paul Szabo <psz at maths.usyd.edu.au> changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|7.7p1 |9.2p1
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2025-Feb-24 09:12 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888 --- Comment #8 from Paul Szabo <psz at maths.usyd.edu.au> --- For what it is worth... I had tried LZ4 also: implemented that as an option for OpenSSH, alongside ZSTD. This gave a lower CPU usage, but the compression was insufficient so network bandwidth was limiting and "ssh -X" became quite "unresponsive". Cheers, Paul -- You are receiving this mail because: You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2025-Feb-24 09:25 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Paul Szabo <psz at maths.usyd.edu.au> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #3862|Patch for zstd support on |Patch for zstd support on
description|9,2p |9.2p
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2025-Mar-02 20:48 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Sebastian A. Siewior <openssh at ml.breakpoint.cc> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #3373|0 |1
is obsolete| |
--- Comment #9 from Sebastian A. Siewior <openssh at ml.breakpoint.cc> ---
Created attachment 3865
--> https://bugzilla.mindrot.org/attachment.cgi?id=3865&action=edit
zstd support
Update to V_9_7_P1
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2025-Aug-07 22:33 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
Paul Szabo <psz at maths.usyd.edu.au> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #3895| |ok+
Flags| |
--- Comment #10 from Paul Szabo <psz at maths.usyd.edu.au> ---
Created attachment 3895
--> https://bugzilla.mindrot.org/attachment.cgi?id=3895&action=edit
zstd for 10.0p1
Patch for zstd support on version 10.0p1 (as distributed by Debian)
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2025-Aug-08 14:09 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888
--- Comment #11 from Fabio <fabio.pedretti at unibs.it> ---
Thanks. Maybe you could send it to HPN-SSH ("high performance soft fork
of OpenSSH") https://github.com/rapier1/hpn-ssh they may be interested.
--
You are receiving this mail because:
You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2025-Aug-08 20:55 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888 --- Comment #12 from Paul Szabo <psz at maths.usyd.edu.au> --- Do not HPN-SSH track all OpenSSH bugs, including this one? They say this is "their" bugzilla. Can you encourage them to be accepted into Debian? -- You are receiving this mail because: You are watching the assignee of the bug.
bugzilla-daemon at mindrot.org
2025-Aug-19 09:25 UTC
[Bug 2888] Consider adding other compression schemes (lz4, zstd)
https://bugzilla.mindrot.org/show_bug.cgi?id=2888 --- Comment #13 from Fabio <fabio.pedretti at unibs.it> --- (In reply to Paul Szabo from comment #12)> Do not HPN-SSH track all OpenSSH bugs, including this one? > They say this is "their" bugzilla.No idea, sorry.> Can you encourage them to be accepted into Debian?No, but I found these, maybe someone can eventually help: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1006282 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1006287 -- You are receiving this mail because: You are watching the assignee of the bug.