CentOS - May 2017 - Best practices for copying lots of files machine-to-machine

Vanhorn, Mike wrote:
> On 5/17/17, 12:03 PM, "CentOS on behalf of ken"
<centos-bounces at centos.org
> on behalf of gebser at mousecar.com> wrote:
>
>>An entire filesystem (~180g) needs to be copied from one local linux
>>machine to another.  Since both systems are on the same local subnet,
>>there's no need for encryption.
>>
>>I've done this sort of thing before a few times in the past in
different
>>ways, but wanted to get input from others on what's worked best for
>> them.
>
> If shutting the machines down is feasible, I?d put the source hard drive
> into the destination machine and use rsync to copy it from one drive to
> the other (rather than using rsync to copy from one machine to the other
> over the network).
Why? I just rsync'd 159G in less than one workday from one server to
another. Admittedly, we allegedly have a 1G network, but....

       mark

On 17 May 2017 at 22:27, <m.roth at 5-cent.us> wrote:
> Vanhorn, Mike wrote:
> > On 5/17/17, 12:03 PM, "CentOS on behalf of ken" <
> centos-bounces at centos.org
> > on behalf of gebser at mousecar.com> wrote:
> >
> >>An entire filesystem (~180g) needs to be copied from one local
linux
> >>machine to another.  Since both systems are on the same local
subnet,
> >>there's no need for encryption.
> >>
> >>I've done this sort of thing before a few times in the past in
different
> >>ways, but wanted to get input from others on what's worked best
for
> >> them.
> >
> > If shutting the machines down is feasible, I?d put the source hard
drive
> > into the destination machine and use rsync to copy it from one drive
to
> > the other (rather than using rsync to copy from one machine to the
other
> > over the network).
>
> Why? I just rsync'd 159G in less than one workday from one server to
> another. Admittedly, we allegedly have a 1G network, but....
>
>        mark
>
> Hi,
you can parallelize rsync with xargs's -P (max-procs) option (man xargs).

rsync -a -f"+ */" -f"- *" source/ server:/destination/ 
#sync directory
first
cd source/; find . -type f | *xargs* -n1 -*P0* -I% rsync -az %
server:/destination/% # 0 to let xargs deal with the num of procs

Julius

On 5/17/17, 5:27 PM, "CentOS on behalf of m.roth at 5-cent.us"
<centos-bounces at centos.org on behalf of m.roth at 5-cent.us> wrote:
>Why? I just rsync'd 159G in less than one workday from one server to
>another. Admittedly, we allegedly have a 1G network, but....
Well, I?ve don?t recall ever having to rsync more than 100G (although I am doing
multiple rsyncs of about 86G as we speak), and I?ve never been able to do it
with machines on their own, isolated switch (so my rsync?s are competing with
everything else on the network), and it?s been a while since I?ve actually tried
it multiple ways and measured it, but in my experience I?ve never see the
network outperform the system bus.

---
Mike VanHorn
Senior Computer Systems Administrator
College of Engineering and Computer Science
Wright State University
265 Russ Engineering Center
937-775-5157
michael.vanhorn at wright.edu

Vanhorn, Mike wrote:
>
> On 5/17/17, 5:27 PM, "CentOS on behalf of m.roth at 5-cent.us"
> <centos-bounces at centos.org on behalf of m.roth at 5-cent.us>
wrote:
>
>>Why? I just rsync'd 159G in less than one workday from one server to
>>another. Admittedly, we allegedly have a 1G network, but....
>
> Well, I?ve don?t recall ever having to rsync more than 100G (although I am
> doing multiple rsyncs of about 86G as we speak), and I?ve never been able
> to do it with machines on their own, isolated switch (so my rsync?s are
> competing with everything else on the network), and it?s been a while
> since I?ve actually tried it multiple ways and measured it, but in my
> experience I?ve never see the network outperform the system bus.
I wasn't saying the network outperformed the system bus. Most of the time,
though, I don't have that as a possibility. Usually, all the drive bays
are full and in use.

When we get to terabytes, that's at least overnight; but a few hundred gig
I can do in a day, if I start early.

    mark