search for: heavycomputing

https://bugzilla.samba.org/show_bug.cgi?id=3099 --- Comment #8 from Chip Schweiss <chip at innovates.com> --- I would argue that optionally all directory scanning should be made parallel. Modern file systems perform best when request queues are kept full. The current mode of rsync scanning directories does nothing to take advantage of this. I currently use scripts to split a couple

...gt; >Thanks. >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

...; >Please use reply-all for most replies to avoid omitting the mailing list. > >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > >-- >Ken Chase - ken at heavycomputing.ca skype:kenchase23 Toronto Canada >Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W. > >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: https://lists.samba.org/mailman/list...

...PGP SIGNATURE----- >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - ken att heavycomputing.ca Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

> If your goal is to reduce storage, and scanning inodes doesnt matter, > use --link-dest for targets. However, that'll keep a backup for every > time that you run it, by link-desting yesterday's copy. The goal was not to reduce storage, it was to reduce work. A full rsync takes more than the whole night, and the destination server is almost unusable for anything else when it

https://bugzilla.samba.org/show_bug.cgi?id=11378 Bug ID: 11378 Summary: Please add a '--line-buffered' option to rsync to make logging/output more friendly with pipes/syslog/CI systems/etc. Product: rsync Version: 3.1.1 Hardware: All OS: All Status: NEW

Hi all, Suppose I have a file on the remote rsync server: rsync://path/to/myfile And I want to only retrieve a part of the file based a ranges of bytes to my local host, say, 0-499, means only transfer the first 500 bytes of that file. Is this possible with rsync client? Regards -- .: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

...> QWARTA > >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

...ng files and directories that are older than one week on >the source side is a waste of time and effort, as the rsync is done >every day, so they can safely be assumed to be in sync already. > >Dirk van Deun >-- >Ceterum censeo Redmond delendum -- Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

> What is taking time, scanning inodes on the destination, or recopying the entire > backup because of either source read speed, target write speed or a slow interconnect > between them? It takes hours to traverse all these directories with loads of small files on the backup server. That is the limiting factor. Not even copying: just checking the timestamp and size of the old copies.

...tionally, I dont know if linux (or freebsd or any unix) can be told to cache metadata more aggressively than data - not much point for the latter on a backup server. The former would be great. I dont know how big metadata is in ram either for typical OS's, per inode. /kc -- Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

On Mon, 13 Jul 2015 17:38:35 -0400, Selva Nair wrote: > As with any dedup solution, performance does take a hit and its often > not worth it unless you have a lot of duplication in the data. This is so only in some volumes in our case, but it appears that zfs permits this to be enabled/disabled on a per-volume basis. That would work for us. Is there a way to save cycles by offering zfs

...> > > >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

...4 14:16 . drwxr-xr-x 3 user user 4096 Jan 24 14:17 .. drwxr-xr-x 2 user user 4096 Jan 24 14:16 a /tmp$ ls -la bar/foo/a total 8 drwxr-xr-x 2 user user 4096 Jan 24 14:16 . drwxr-xr-x 3 user user 4096 Jan 24 14:16 .. where's .baz in the copy? how does * match .baz? /kc -- Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

I investigated the rsync code and found the reason why. For every file in the source, it searches the entire filter-list looking to see if that filename is on the exclude/include list. Most aren't, so it compares (350K - 72K) * 72K names (the non-listed files) plus (72K * 72K/2) names (the ones that are listed), for a total of about 22,608,000,000 strcmp's. That's 22 BILLION

...PGP SIGNATURE----- >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - ken at heavycomputing.ca

...eo Redmond delendum >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

...; >Paul > >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

...slower than pax again). I have detailed an example of this scenario at http://unix.stackexchange.com/questions/193308/rsyncs-link-dest-option-does-not-link-identical-files-if-an-old-file-exists which also indicates --delete-before and --whole-file do not help at all. /kc -- Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.

...ase use reply-all for most replies to avoid omitting the mailing > list. > >To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > >Before posting, read: > http://www.catb.org/~esr/faqs/smart-questions.html > > -- > Ken Chase - ken att heavycomputing.ca Toronto Canada > Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 > Front St. W. > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Be...