Hello, I am going to use Rsync over millions of files, with --link-dest option. So huge read activity at the beginning of each backup, and huge hard links creation activity. Of course I will use a RAID array on receiver side, dedicated to Rsync. Do you have any recommandation over RAID type, number of disks and chunk size to use ? Thank you very much ! Ben
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 With millions of files I suggest ditching --link-dest and using ZFS with subvolume snapshots. You will have nothing but pain trying to deal with many millions of links. On 07/12/13 12:48, AZ 9901 wrote:> Hello, > > I am going to use Rsync over millions of files, with --link-dest > option. So huge read activity at the beginning of each backup, and > huge hard links creation activity. > > Of course I will use a RAID array on receiver side, dedicated to > Rsync. > > Do you have any recommandation over RAID type, number of disks and > chunk size to use ? > > Thank you very much ! > > Ben >- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHgPQ4ACgkQVKC1jlbQAQcrhQCfXBThu9qveub+SlVZFfy/WXQQ 9i0AnjA2blfkJe1MgHn/2kwE/ayJfnjO =wFIA -----END PGP SIGNATURE-----
Le 4 ao?t 2013 ? 17:25, K S Braunsdorf a ?crit :> In message <B9CF2765-0BB7-4B0F-B758-1D5CF0219713 at gmail.com>, AZ 9901 writes: >> Due to --link-dest, I have a huge read activity at the beginning of each >> backup, and huge hard links creation activity. >> I also plan to have several backups at the same time, coming from dozens >> of servers. > > Then the network will be the bottle-neck. I would stagger the backups > with some active management (not just stagger the start times). > The easy thing to help would be --bwlimit=KBPS. > > The disk tunes should be anything that is an even factor of the transfer > size. So 1 transfer (256k) might update 8 blocks, or 64. This helps > keep the VM system cache in check, otherwise you end up flushing blocks > you will reference again to often. > > Setting the transfer size to a large prime number is about the worst > thing you can do, then set disk block size to a larger prime and > see how bad "Bad" can be. :-(KSB, thanks for your answer ! So I will test with chunks of 256K, 512K and 1024K. Should I leave FS blocks to 4K, or increase them to let's say 256K ? What about RAID type ? I was thinking about the "common" RAID5. Thx ! Ben