openssh unix dev - Jun 2019 - About rsync over SSH and compression

Thanks; I did not think of that. I have just run a few basic tests with
both rsync and OpenSSH in their default settings, when it comes to
compression. SSH compression seems to have a very slight edge. However, I
surmise that, given the number of knobs available on both sides (OpenSSH,
in particular) one can tinker with settings almost endlessly in either
side, probably being able to end up with specific situations where one or
the other would be advantageous. At first sight, however, the overall
difference seems to be small.

On Tue, Jun 25, 2019 at 10:33 AM Tim Broberg <Tim.Broberg at
servicenow.com>
wrote:
> Those are all good points.
>
> A few more to consider:
> 1 - You may want to exert more effort compressing your rsync traffic than
> you would want to apply across the board in ssh.
> 2 - You *know* rsync supports compression on both sides. You don't
> necessarily control what algorithms are turned on on both ends in ssh.
>
> I think I would lean towards doing it in the application myself.
>     - Tim.
>
> ?On 6/25/19, 9:11 AM, "openssh-unix-dev on behalf of JCA"
> <openssh-unix-dev-bounces+tim.broberg=servicenow.com at mindrot.org on
> behalf of 1.41421 at gmail.com> wrote:
>
>     [External Email]
>
>
>     Rsync supports the capability of compressing data before sending. So
> does
>     OpenSSH. It would be probably be a waste of resources and time to
> enable
>     both compression capabilities at the same time, but it is not clear to
> me
>     whether, in general, it makes better sense to enable rsync compression
> or
>     SSH compression.
>
>     My first thought would be that SSH compression might yield better
> results,
>     on the ground that SSH will try to cram as much data as possible in a
>     channel data message, within the limitations imposed by the maximum SSH
>     message size and the current window size. On the other hand, rsync
> might
>     (and 'might' is the keyword here) resort to individual
>     SSH2_MSG_CHANNEL_DATA messages for particular deltas, thus giving the
>     compression code smaller amounts of data to play with every time.
>     Additionally, SSH compression will be able to compress the rsync
> protocol
>     control data, which rsync will not be able to compress (right?)
>
>     Feedback on the issues above will be much appreciated.
>     _______________________________________________
>     openssh-unix-dev mailing list
>     openssh-unix-dev at mindrot.org
>
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mindrot.org_mailman_listinfo_openssh-2Dunix-2Ddev&d=DwICAg&c=Zok6nrOF6Fe0JtVEqKh3FEeUbToa1PtNBZf6G01cvEQ&r=WxtmI2HcpDF2j1UPw-tBSatMtcAHcEc-gP6FGr3XijQ&m=Ny2-77klF-EsTfwgTrMCrfxneJvRMHiwko0T5N7UXJc&s=X9vAz3bjoD5UIvomN5ynX7n3ojNTcG-BfDYD7OzX6Ew&e>
>
>

On Wed, 26 Jun 2019 at 02:50, JCA <1.41421 at gmail.com>
wrote:
> Thanks; I did not think of that. I have just run a few basic tests with
> both rsync and OpenSSH in their default settings, when it comes to
> compression. SSH compression seems to have a very slight edge.
OpenSSH does not allow you to set the zlib compression level for SSH2
(the protocol spec does not have a mechanism for doing so, although
it'd be possible to add it as a non-standard hack) and OpenSSH
hardcodes it to 6.  In rsync you can crank it up to --compress-level=9
which I'd expect to make it beat any theoretical advantage SSH might
have on compressible data, plus it can be smarter about skipping
already-compressed file types.

That said, I'll second what Stuart said that what matters is what
works best for your data set and environment.

-- 
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA (new)
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

On Wed 2019-06-26 08:04:44 +1000, Darren Tucker wrote:
> On Wed, 26 Jun 2019 at 02:50, JCA <1.41421 at gmail.com> wrote:
>> Thanks; I did not think of that. I have just run a few basic tests with
>> both rsync and OpenSSH in their default settings, when it comes to
>> compression. SSH compression seems to have a very slight edge.
>
> OpenSSH does not allow you to set the zlib compression level for SSH2
> (the protocol spec does not have a mechanism for doing so, although
> it'd be possible to add it as a non-standard hack) and OpenSSH
> hardcodes it to 6.  In rsync you can crank it up to --compress-level=9
> which I'd expect to make it beat any theoretical advantage SSH might
> have on compressible data, plus it can be smarter about skipping
> already-compressed file types.
I note that the original question mentioned a concern about a "waste of
resources and time" -- this discussion seems to assume that bandwidth is
the only expensive or latency-inducing resource.

But of course compression and decompression can be expensive and
latency-inducing in terms of CPU and RAM (especially at the more extreme
ends of the compression dial), even if you might save on bandwidth.
This is going to really depend on which resources are more valuable to
you in a given context.

        --dkg