Hi,
how does rsync work if it compares two very huge files on two distinct
hosts (rsync uses a networked connection, via SSH)?
Some observation seems to indicate, that rsync first reads (and
checksums?) the remote (destination) side, then, if finished, it reads
(and checksums?) the local (source) side and then starts transferring
the delta if done with the comparison of these two files on both sides.
The major problem (with respect to runtime) seems to be the missing
parellism in read (and checksumming) the source and the destination.
1) Is this observation right?
1a) Does it depend on the rsync version?
2) Is there a way to get read (and checksumming) of both sides
in parallel?
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de ---------------------------- internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --------------- key ID: F69376CE -
! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20180314/f1dac4bb/signature.sig>
Your observation would be right if you are using --checksum which you shouldn't be. Otherwise, unless you are using --whole-file rsync will use its differential algorithm to compare the files. If you are using --progress you will see it step through the file at a faster speed than the network is capable of. If there is no difference then rsync will go very fast to 100% then do any metadata changes. If it hits a difference and you aren't using --inplace then rsync will copy the correct beginning of the file to a new temp file then begin transferring the differences. On huge files this can cause a significant pause while it does that local copy to a temp file. On 03/14/2018 09:25 AM, Heiko Schlittermann via rsync wrote:> Hi, > > > how does rsync work if it compares two very huge files on two distinct > hosts (rsync uses a networked connection, via SSH)? > > Some observation seems to indicate, that rsync first reads (and > checksums?) the remote (destination) side, then, if finished, it reads > (and checksums?) the local (source) side and then starts transferring > the delta if done with the comparison of these two files on both sides. > > The major problem (with respect to runtime) seems to be the missing > parellism in read (and checksumming) the source and the destination. > > 1) Is this observation right? > 1a) Does it depend on the rsync version? > > 2) Is there a way to get read (and checksumming) of both sides > in parallel? > > > Best regards from Dresden/Germany > Viele Grüße aus Dresden > Heiko Schlittermann > > >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 224 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20180314/10729e7c/signature.sig>
Kevin Korb via rsync <rsync at lists.samba.org> (Mi 14 Mär 2018 14:52:55 CET):> Your observation would be right if you are using --checksum which you > shouldn't be. Otherwise, unless you are using --whole-file rsync will > use its differential algorithm to compare the files. If you are using > --progress you will see it step through the file at a faster speed thanOk, Thank you. I'll try to find the options they're using. But, anyway, even with --checksum, why can't it run and checksum the file on both sides in parallel? If I understand your answer, then it does it in sequence, doesn't it? -- Heiko -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: <http://lists.samba.org/pipermail/rsync/attachments/20180314/e1fd3256/signature.sig>