A block level tool won't do a differential back up like rsync does will it?
On Tue, Sep 13, 2011 at 11:04, Kevin Korb <kmk at sanitarium.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> The rsync is doing a whole lot more than transferring 600MB of data. It
> is transferring 600MB of differences. It is going through 27.4 million
> files on one end and possibly that many on the other. That is a 402MB
> listing of files plus as many as 54.8 million calls to stat().
>
> Rsync is not about minimizing transfer time. It is about minimizing
> network IO at the expense of CPU and disk IO. Rsync reduced your
> network IO from 185.39GB to 600MB only 200 of which was actually
> differences in your data.
>
> IOW rsync did its job.
>
> The unfortunate truth is that no file based tool will perform anywhere
> near wire speeds when it has millions of files to deal with. If you
> want something wire speed then you should look for a block level tool.
>
> On 09/12/11 21:54, David Somers-Harris wrote:
> > No, I don't have checksum based checking enabled.
> >
> > Sorry for neglecting to mention the options I'm using, here they
are:
> >
> > *rsync -vah --exclude=".*" --delete --stats*
> >
> > Would anything in there cause it to be unnecessarily slow? It's
taking 3
> > hours but shouldn't take more than 5 minutes...
> >
> >
> > On Tue, Sep 13, 2011 at 00:02, Sandon Van Ness <sandon at
van-ness.com
> > <mailto:sandon at van-ness.com>> wrote:
> >
> > It could definitely be normal if say you have checksum based
> > checking enabled...
> >
> >
> > On 09/12/2011 01:56 AM, David Somers-Harris wrote:
> >
> >
> > Is it normal for rsync to take 3 hours on this transfer?
> >
> > Number of files: 27419348
> > Number of files transferred: 19501
> > Total file size: 185.39G bytes
> > Total transferred file size: 195.92M bytes
> > Literal data: 195.68M bytes
> > Matched data: 241.09K bytes
> > File list size: 402.01M
> > File list generation time: 0.561 seconds
> > File list transfer time: 0.000 seconds
> > Total bytes sent: 600.61M
> > Total bytes received: 2.44M
> >
> > sent 600.61M bytes received 2.44M bytes 54.42K bytes/sec
> > total size is 185.39G speedup is 307.42
> >
> >
> > A normal transfer of 600 MB for me between these two servers
> > only takes a few seconds.... Is there some way I can speed
this
> up?
> >
> >
> >
>
> - --
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> Kevin Korb Phone: (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. Kevin at FutureQuest.net (work)
> Orlando, Florida kmk at sanitarium.net (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk5uuhYACgkQVKC1jlbQAQcV1wCeLLN+f45ydJre5UHmrdIj77FK
> OakAoOMdRTPLvdnSFSEX9407LiWuEMH3
> =O2vB
> -----END PGP SIGNATURE-----
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20110913/d8601e58/attachment.html>