Rsync supports the capability of compressing data before sending. So does OpenSSH. It would be probably be a waste of resources and time to enable both compression capabilities at the same time, but it is not clear to me whether, in general, it makes better sense to enable rsync compression or SSH compression. My first thought would be that SSH compression might yield better results, on the ground that SSH will try to cram as much data as possible in a channel data message, within the limitations imposed by the maximum SSH message size and the current window size. On the other hand, rsync might (and 'might' is the keyword here) resort to individual SSH2_MSG_CHANNEL_DATA messages for particular deltas, thus giving the compression code smaller amounts of data to play with every time. Additionally, SSH compression will be able to compress the rsync protocol control data, which rsync will not be able to compress (right?) Feedback on the issues above will be much appreciated.
Those are all good points.
A few more to consider:
1 - You may want to exert more effort compressing your rsync traffic than you
would want to apply across the board in ssh.
2 - You *know* rsync supports compression on both sides. You don't
necessarily control what algorithms are turned on on both ends in ssh.
I think I would lean towards doing it in the application myself.
- Tim.
?On 6/25/19, 9:11 AM, "openssh-unix-dev on behalf of JCA"
<openssh-unix-dev-bounces+tim.broberg=servicenow.com at mindrot.org on behalf
of 1.41421 at gmail.com> wrote:
[External Email]
Rsync supports the capability of compressing data before sending. So does
OpenSSH. It would be probably be a waste of resources and time to enable
both compression capabilities at the same time, but it is not clear to me
whether, in general, it makes better sense to enable rsync compression or
SSH compression.
My first thought would be that SSH compression might yield better results,
on the ground that SSH will try to cram as much data as possible in a
channel data message, within the limitations imposed by the maximum SSH
message size and the current window size. On the other hand, rsync might
(and 'might' is the keyword here) resort to individual
SSH2_MSG_CHANNEL_DATA messages for particular deltas, thus giving the
compression code smaller amounts of data to play with every time.
Additionally, SSH compression will be able to compress the rsync protocol
control data, which rsync will not be able to compress (right?)
Feedback on the issues above will be much appreciated.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev at mindrot.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mindrot.org_mailman_listinfo_openssh-2Dunix-2Ddev&d=DwICAg&c=Zok6nrOF6Fe0JtVEqKh3FEeUbToa1PtNBZf6G01cvEQ&r=WxtmI2HcpDF2j1UPw-tBSatMtcAHcEc-gP6FGr3XijQ&m=Ny2-77klF-EsTfwgTrMCrfxneJvRMHiwko0T5N7UXJc&s=X9vAz3bjoD5UIvomN5ynX7n3ojNTcG-BfDYD7OzX6Ew&e=
Thanks; I did not think of that. I have just run a few basic tests with both rsync and OpenSSH in their default settings, when it comes to compression. SSH compression seems to have a very slight edge. However, I surmise that, given the number of knobs available on both sides (OpenSSH, in particular) one can tinker with settings almost endlessly in either side, probably being able to end up with specific situations where one or the other would be advantageous. At first sight, however, the overall difference seems to be small. On Tue, Jun 25, 2019 at 10:33 AM Tim Broberg <Tim.Broberg at servicenow.com> wrote:> Those are all good points. > > A few more to consider: > 1 - You may want to exert more effort compressing your rsync traffic than > you would want to apply across the board in ssh. > 2 - You *know* rsync supports compression on both sides. You don't > necessarily control what algorithms are turned on on both ends in ssh. > > I think I would lean towards doing it in the application myself. > - Tim. > > ?On 6/25/19, 9:11 AM, "openssh-unix-dev on behalf of JCA" > <openssh-unix-dev-bounces+tim.broberg=servicenow.com at mindrot.org on > behalf of 1.41421 at gmail.com> wrote: > > [External Email] > > > Rsync supports the capability of compressing data before sending. So > does > OpenSSH. It would be probably be a waste of resources and time to > enable > both compression capabilities at the same time, but it is not clear to > me > whether, in general, it makes better sense to enable rsync compression > or > SSH compression. > > My first thought would be that SSH compression might yield better > results, > on the ground that SSH will try to cram as much data as possible in a > channel data message, within the limitations imposed by the maximum SSH > message size and the current window size. On the other hand, rsync > might > (and 'might' is the keyword here) resort to individual > SSH2_MSG_CHANNEL_DATA messages for particular deltas, thus giving the > compression code smaller amounts of data to play with every time. > Additionally, SSH compression will be able to compress the rsync > protocol > control data, which rsync will not be able to compress (right?) > > Feedback on the issues above will be much appreciated. > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mindrot.org_mailman_listinfo_openssh-2Dunix-2Ddev&d=DwICAg&c=Zok6nrOF6Fe0JtVEqKh3FEeUbToa1PtNBZf6G01cvEQ&r=WxtmI2HcpDF2j1UPw-tBSatMtcAHcEc-gP6FGr3XijQ&m=Ny2-77klF-EsTfwgTrMCrfxneJvRMHiwko0T5N7UXJc&s=X9vAz3bjoD5UIvomN5ynX7n3ojNTcG-BfDYD7OzX6Ew&e> > >
On 2019/06/25 10:03, JCA wrote:> Rsync supports the capability of compressing data before sending. So does > OpenSSH. It would be probably be a waste of resources and time to enable > both compression capabilities at the same time, but it is not clear to me > whether, in general, it makes better sense to enable rsync compression or > SSH compression. > > My first thought would be that SSH compression might yield better results, > on the ground that SSH will try to cram as much data as possible in a > channel data message, within the limitations imposed by the maximum SSH > message size and the current window size. On the other hand, rsync might > (and 'might' is the keyword here) resort to individual > SSH2_MSG_CHANNEL_DATA messages for particular deltas, thus giving the > compression code smaller amounts of data to play with every time. > Additionally, SSH compression will be able to compress the rsync protocol > control data, which rsync will not be able to compress (right?) > > Feedback on the issues above will be much appreciated.If you're updating changed files, then rsync has more information available about them available than ssh possibly can (see the rsync manual about --compress). If you want to save bandwidth over all else then you probably want to test for yourself with your usual set of files and typical changes, but in that case it might be worth using both to get control data as well (especially if you have large numbers of files).
I've been doing a lot of testing of different options for both rsync and ssh. For our environment, turning compression off in rsync and on in ssh has given us the best results. Turning compression on in ssh gave us a small improvement compared to the large slowdown that compression gave us in rsync. For us, we also made sure that the checksum option in rsync was off. It caused a major slowdown for us. It was doing a checksum on every file before deciding if needed to transfer the file. That was killing us. We are transferring a wide variety of sizeed files, from small KB sized to GB sized files. We are also transferring 10's of thousands of files between states. I hope this helps some. Brent L. Bates On Tue, Jun 25, 2019, 12:06 PM JCA <1.41421 at gmail.com> wrote:> Rsync supports the capability of compressing data before sending. So does > OpenSSH. It would be probably be a waste of resources and time to enable > both compression capabilities at the same time, but it is not clear to me > whether, in general, it makes better sense to enable rsync compression or > SSH compression. > > My first thought would be that SSH compression might yield better results, > on the ground that SSH will try to cram as much data as possible in a > channel data message, within the limitations imposed by the maximum SSH > message size and the current window size. On the other hand, rsync might > (and 'might' is the keyword here) resort to individual > SSH2_MSG_CHANNEL_DATA messages for particular deltas, thus giving the > compression code smaller amounts of data to play with every time. > Additionally, SSH compression will be able to compress the rsync protocol > control data, which rsync will not be able to compress (right?) > > Feedback on the issues above will be much appreciated. > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev >
Thanks, that's very useful feedback for me, for my circumstances are likely to be similar to yours. On Wed, Jun 26, 2019 at 4:41 AM Brent L. Bates <blbates1701 at gmail.com> wrote:> I've been doing a lot of testing of different options for both rsync > and ssh. For our environment, turning compression off in rsync and on in > ssh has given us the best results. Turning compression on in ssh gave us a > small improvement compared to the large slowdown that compression gave us > in rsync. For us, we also made sure that the checksum option in rsync was > off. It caused a major slowdown for us. It was doing a checksum on every > file before deciding if needed to transfer the file. That was killing us. > We are transferring a wide variety of sizeed files, from small KB sized to > GB sized files. We are also transferring 10's of thousands of files > between states. I hope this helps some. > > > Brent L. Bates > > > On Tue, Jun 25, 2019, 12:06 PM JCA <1.41421 at gmail.com> wrote: > >> Rsync supports the capability of compressing data before sending. So does >> OpenSSH. It would be probably be a waste of resources and time to enable >> both compression capabilities at the same time, but it is not clear to me >> whether, in general, it makes better sense to enable rsync compression or >> SSH compression. >> >> My first thought would be that SSH compression might yield better results, >> on the ground that SSH will try to cram as much data as possible in a >> channel data message, within the limitations imposed by the maximum SSH >> message size and the current window size. On the other hand, rsync might >> (and 'might' is the keyword here) resort to individual >> SSH2_MSG_CHANNEL_DATA messages for particular deltas, thus giving the >> compression code smaller amounts of data to play with every time. >> Additionally, SSH compression will be able to compress the rsync protocol >> control data, which rsync will not be able to compress (right?) >> >> Feedback on the issues above will be much appreciated. >> _______________________________________________ >> openssh-unix-dev mailing list >> openssh-unix-dev at mindrot.org >> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev >> >