??????? Original Message ??????? On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync <rsync at lists.samba.org> wrote:> In my experience backing up multiple hosts to a single host can speed > things up. Especially if using rsync over ssh with multiple CPUs on the > single host. You would need to do some experimentation to determine the > best number for your hardware and network. Also, if you exceed that > number you can put more stress on the backup machine lessening the > impact to the hosts being backed up. > > On 8/24/21 3:58 PM, hancooper via rsync wrote: > > > It makes sense to we to run multiple rsync commands to speed up a backup. > > At work, some have argued that if I sync the data all to the same host, > > there is no advantage > > in parallelization. ? Whether you sync 3x 1G in parallel or 1x 3G makes > > no noticeable difference. > > Speed in general does not increase by increasing the number of sync > > processes. > > However, if one want to sync to multiple hosts, the effort may be > > worthwhile. > > What is the current advice about speeding up data transfer ? > > -- > Kevin Korb Phone: (407) 252-6853Hi Kevin. What about a single host to another with multiple CPU's ?
??????? Original Message ??????? On Tuesday, August 24, 2021 9:12 PM, hancooper via rsync <rsync at lists.samba.org> wrote:> ??????? Original Message ??????? > On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync rsync at lists.samba.org wrote: > > > In my experience backing up multiple hosts to a single host can speed > > things up. Especially if using rsync over ssh with multiple CPUs on the > > single host. You would need to do some experimentation to determine the > > best number for your hardware and network. Also, if you exceed that > > number you can put more stress on the backup machine lessening the > > impact to the hosts being backed up. > > On 8/24/21 3:58 PM, hancooper via rsync wrote: > > > > > It makes sense to we to run multiple rsync commands to speed up a backup. > > > At work, some have argued that if I sync the data all to the same host, > > > there is no advantage > > > in parallelization. ? Whether you sync 3x 1G in parallel or 1x 3G makes > > > no noticeable difference. > > > Speed in general does not increase by increasing the number of sync > > > processes. > > > However, if one want to sync to multiple hosts, the effort may be > > > worthwhile. > > > What is the current advice about speeding up data transfer ? > > > > -- > > Kevin Korb Phone: (407) 252-6853 > > Hi Kevin. What about a single host to another with multiple CPU's ?rsync really needs a lot of CPU power. It is not so much disk-bound (as many conclude against using multiple processes), but cpu-bound.
Most of the speedup is due to ssh being bound to a single CPU. You will get a little more speed out of a single CPU system but the speedup is unlikely to go past 2 in parallel. Even with that you may end up adding more latency via disk seeks to accomplish much. On 8/24/21 5:12 PM, hancooper wrote:> ??????? Original Message ??????? > On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync <rsync at lists.samba.org> wrote: > >> In my experience backing up multiple hosts to a single host can speed >> things up. Especially if using rsync over ssh with multiple CPUs on the >> single host. You would need to do some experimentation to determine the >> best number for your hardware and network. Also, if you exceed that >> number you can put more stress on the backup machine lessening the >> impact to the hosts being backed up. >> >> On 8/24/21 3:58 PM, hancooper via rsync wrote: >> >>> It makes sense to we to run multiple rsync commands to speed up a backup. >>> At work, some have argued that if I sync the data all to the same host, >>> there is no advantage >>> in parallelization. ? Whether you sync 3x 1G in parallel or 1x 3G makes >>> no noticeable difference. >>> Speed in general does not increase by increasing the number of sync >>> processes. >>> However, if one want to sync to multiple hosts, the effort may be >>> worthwhile. >>> What is the current advice about speeding up data transfer ? >> >> -- >> Kevin Korb Phone: (407) 252-6853 > > > Hi Kevin. What about a single host to another with multiple CPU's ? >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,