openssh bugs - Jul 2018 - [Bug 2888] New: Consider adding other compression schemes (lz4, zstd)

https://bugzilla.mindrot.org/show_bug.cgi?id=2888

            Bug ID: 2888
           Summary: Consider adding other compression schemes (lz4, zstd)
           Product: Portable OpenSSH
           Version: 7.7p1
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: Miscellaneous
          Assignee: unassigned-bugs at mindrot.org
          Reporter: fabio.pedretti at unibs.it

I did some test and when enabling compression my CPU usage goes at 100%
at about 70-150Mb/s, depending on file compressibility. So it is not
very useful with a Gigabit Ethernet link.

To improve this I would propose to add other compression schemes, such
as lz4 ( https://github.com/lz4/lz4 ) or zstd (
https://github.com/facebook/zstd ).

They are order of magnitudes faster than current zlib-6 compression and
easily able to exceed Gigabit Ethernet speed.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2888

--- Comment #1 from Fabio <fabio.pedretti at unibs.it> ---
See also bug #2887.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2888

Scott Baker <scott at perturb.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |scott at perturb.org

--- Comment #2 from Scott Baker <scott at perturb.org> ---
I came here to request alternate compression methods, and found someone
had beat me to it. Consider this my +1

Currently OpenSSH only supports GZIP compression which has speed
constraints on modern CPUs. Enabling compression on file transfers over
a gigabit link actually increases the amount of time taken to transfer
large files due to the limitations of GZIP.

Please consider adding support for alternate compression methods. LZ4
and ZSTD would be excellent choices.

Even better would be support for external compression methods like
`tar` does with `-I`. That way in the future if new/better compression
algorithms are found SSH will automatically be able to utilize them.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2888

--- Comment #3 from Scott Baker <scott at perturb.org> ---
Transferring a 1.8GB VM image on a gigabit network:

```
root at green(~)
:time scp test.bin yellow:/dev/null
test.bin                               100% 1821MB 108.0MB/s   00:16    

# Enabling compression is significantly slower
root at green(~)
:time scp -C test.bin yellow:/dev/null
test.bin                               100% 1821MB  20.0MB/s   01:30    
```

Alternately using this "hack" with `cat`, `ssh`, and ZSTD I was able
to
decrease transfer time by ~40%.

```
root at green(~)
:time cat test.bin | zstd | ssh yellow 'zstd - -d > /dev/null'

real    0m9.691s
user    0m12.178s
sys     0m4.458s
```

For transferring large files, and/or groups of large files via
scp/rsync having a modern compression method would be a **huge** win.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2888

Luke <lukenbutters at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |lukenbutters at gmail.com

--- Comment #4 from Luke <lukenbutters at gmail.com> ---
please add support for this.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2888

Sebastian A. Siewior <openssh at ml.breakpoint.cc> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |openssh at ml.breakpoint.cc

--- Comment #5 from Sebastian A. Siewior <openssh at ml.breakpoint.cc> ---
Created attachment 3373
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3373&action=edit
Support for zstd against V_8_2_P1

This archive contains four patches against openssh v8.2.p1.
Would love some feedback if the expectations regarding scp
performance/CPU load are met.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2888

--- Comment #6 from Fabio <fabio.pedretti at unibs.it> ---
@Sebastian
Thanks for your patches. I'd suggest sending them on the OpenSSH
Developers list:
https://www.openssh.com/list.html

and eventually as a Pull request on github:
https://github.com/openssh/openssh-portable/pulls

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2888

Celeste Liu <CoelacanthusHex at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |CoelacanthusHex at gmail.com

-- 
You are receiving this mail because:
You are watching the assignee of the bug.