rsync - Oct 2018 - rsync of big image files from virtual hosts takes lot of time

Hi,

i'd like to rsync some image files from virtual hosts each night. The image
files are between 50GB and 400GB big.
I thought rsync would be the appropriate solution because it just transfers the
differences, not the whole file.
But nevertheless rsync takes hours.
The images aren't much modified over the day, i would expect just some tens
of megabyte needed to be transfered.

I have the following command line:
rsync -av --no-whole-file --inplace source destination.

Is that ok ? Any suggestions ?

I skipped the check with the hash values before the transfer because creating a
hash value from a 400GB file takes loooong.
But the man pages mentione that nevertheless AFTER the transfer a hash value is
created on both sides to check that the files are equal.
Is it possible to switch the after-transfer check off ?

I have SLES 12 SP3 and rsync 3.1.0

Thanks.

Bernd

-- 

Bernd Lentes 
Systemadministration 
Institut für Entwicklungsgenetik 
Gebäude 35.34 - Raum 208 
HelmholtzZentrum münchen 
[ mailto:bernd.lentes at helmholtz-muenchen.de | bernd.lentes at
helmholtz-muenchen.de ]
phone: +49 89 3187 1241 
fax: +49 89 3187 2294 
[ http://www.helmholtz-muenchen.de/idg | http://www.helmholtz-muenchen.de/idg ] 

wer Fehler macht kann etwas lernen 
wer nichts macht kann auch nichts lernen
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: NN
Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler,
Dr. rer. nat. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671

First, add --stats to find out how much is being transferred.  Second,
if these are fixed size or sparse image files then --sparse will be a
big help.  Finally, if they are not fixed size or sparse then --inplace
is probably making things worse.

--no-whole-file is the default btw unless you are doing a local copy.
If you are doing a local copy then just use cp as it is much faster at that.

On 10/20/2018 12:36 PM, Lentes, Bernd via rsync wrote:
> Hi,
> 
> i'd like to rsync some image files from virtual hosts each night. The
image files are between 50GB and 400GB big.
> I thought rsync would be the appropriate solution because it just transfers
the differences, not the whole file.
> But nevertheless rsync takes hours.
> The images aren't much modified over the day, i would expect just some
tens of megabyte needed to be transfered.
> 
> I have the following command line:
> rsync -av --no-whole-file --inplace source destination.
> 
> Is that ok ? Any suggestions ?
> 
> I skipped the check with the hash values before the transfer because
creating a hash value from a 400GB file takes loooong.
> But the man pages mentione that nevertheless AFTER the transfer a hash
value is created on both sides to check that the files are equal.
> Is it possible to switch the after-transfer check off ?
> 
> I have SLES 12 SP3 and rsync 3.1.0
> 
> Thanks.
> 
> Bernd
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			https://sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20181020/5f5a536b/signature.sig>

----- Am 20. Okt 2018 um 18:56 schrieb rsync ML rsync at lists.samba.org:
> First, add --stats to find out how much is being transferred.  Second,
> if these are fixed size or sparse image files then --sparse will be a
> big help.  Finally, if they are not fixed size or sparse then --inplace
> is probably making things worse.
> 
> --no-whole-file is the default btw unless you are doing a local copy.
> If you are doing a local copy then just use cp as it is much faster at
that.
> 
This is what i just got rsyncing a 300GB file:

sending incremental file list

mausdb.raw

sent 1,195,156,775 bytes  received 19,873,319 bytes  67,721.77 bytes/sec
total size is 289,408,024,064  speedup is 238.19

real    299m1.693s
user    68m12.737s
sys     7m56.389s
(i put time before the command).

The files are fixed size, no sparse files.
What do you mean with a "local copy" ? Source and Destination on the
same disk ?
In my case the source reside on a SAN attached via FC, the destination is a cifs
server.

Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: NN
Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler,
Dr. rer. nat. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671