rsync - Oct 2018 - rsync of big image files from virtual hosts takes lot of time

----- Am 20. Okt 2018 um 18:56 schrieb rsync ML rsync at lists.samba.org:
> First, add --stats to find out how much is being transferred.  Second,
> if these are fixed size or sparse image files then --sparse will be a
> big help.  Finally, if they are not fixed size or sparse then --inplace
> is probably making things worse.
> 
> --no-whole-file is the default btw unless you are doing a local copy.
> If you are doing a local copy then just use cp as it is much faster at
that.
> 
This is what i just got rsyncing a 300GB file:

sending incremental file list

mausdb.raw

sent 1,195,156,775 bytes  received 19,873,319 bytes  67,721.77 bytes/sec
total size is 289,408,024,064  speedup is 238.19

real    299m1.693s
user    68m12.737s
sys     7m56.389s
(i put time before the command).

The files are fixed size, no sparse files.
What do you mean with a "local copy" ? Source and Destination on the
same disk ?
In my case the source reside on a SAN attached via FC, the destination is a cifs
server.

Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: NN
Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler,
Dr. rer. nat. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671

I don't see any --stats output.  It will tell you exactly how much data
was involved.

A local copy means that rsync isn't networking.  Rsync speeds things up
by running on 2 separate computers and communicating using the rsync
protocol.  If you are running rsync on a single computer with the source
being a mounted SAN volume and the target being a mounted cifs share
then as far as rsync is concerned that is a local copy.

So, either get the cifs server to speak rsync (either over ssh or with
rsyncd) or just use cp.  Only copying the changes isn't all that helpful
when rsync has to read the whole target file over the network just to
find out what is different about it.  That is why --whole-file is
implied when rsync isn't networking.

On 10/20/2018 02:00 PM, Lentes, Bernd via rsync wrote:
> 
> 
> ----- Am 20. Okt 2018 um 18:56 schrieb rsync ML rsync at lists.samba.org:
> 
>> First, add --stats to find out how much is being transferred.  Second,
>> if these are fixed size or sparse image files then --sparse will be a
>> big help.  Finally, if they are not fixed size or sparse then --inplace
>> is probably making things worse.
>>
>> --no-whole-file is the default btw unless you are doing a local copy.
>> If you are doing a local copy then just use cp as it is much faster at
that.
>>
> 
> This is what i just got rsyncing a 300GB file:
> 
> sending incremental file list
> 
> mausdb.raw
> 
> sent 1,195,156,775 bytes  received 19,873,319 bytes  67,721.77 bytes/sec
> total size is 289,408,024,064  speedup is 238.19
> 
> real    299m1.693s
> user    68m12.737s
> sys     7m56.389s
> (i put time before the command).
> 
> The files are fixed size, no sparse files.
> What do you mean with a "local copy" ? Source and Destination on
the same disk ?
> In my case the source reside on a SAN attached via FC, the destination is a
cifs server.
> 
> Bernd
>  
> 
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de
> Aufsichtsratsvorsitzende: NN
> Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
> Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich
Bassler, Dr. rer. nat. Alfons Enhsen
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
> 
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			https://sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20181020/5ba54806/signature.sig>

----- Am 20. Okt 2018 um 20:05 schrieb rsync ML rsync at lists.samba.org:
> I don't see any --stats output.  It will tell you exactly how much data
> was involved.
> 
> A local copy means that rsync isn't networking.  Rsync speeds things up
> by running on 2 separate computers and communicating using the rsync
> protocol.  If you are running rsync on a single computer with the source
> being a mounted SAN volume and the target being a mounted cifs share
> then as far as rsync is concerned that is a local copy.
> 
> So, either get the cifs server to speak rsync (either over ssh or with
> rsyncd) or just use cp.  Only copying the changes isn't all that
helpful
> when rsync has to read the whole target file over the network just to
> find out what is different about it.  That is why --whole-file is
> implied when rsync isn't networking.
> 
> On 10/20/2018 02:00 PM, Lentes, Bernd via rsync wrote:
This is the output with --stats for a 50GB image.
Appended also the output from time.
You see it's horrible slow.
I will try cp.

Literal data: 187,432,960 bytes
Matched data: 53,499,658,240 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 189,111,658
Total bytes received: 3,277,035

sent 189,111,658 bytes  received 3,277,035 bytes  6,493.15 bytes/sec
total size is 53,687,091,200  speedup is 279.06

real    493m49.055s
user    15m28.933s
sys     26m13.445s

Bernd

 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: NN
Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler,
Dr. rer. nat. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671