At our lab we have storage with many small files. For example a directory can contain over 15,000 files and each file averages about 75k. I would like to sync this to another filesystem on a different server but I am not sure if there is a rsync tuning flag I can use for such a intensive job. I am using rsync 3.0.3. Also, I would like to use little memory as possible. Any thoughts? TIA
Tue, 23 Sep 2008 06:49:33 -0400, magawake wrote:> At our lab we have storage with many small files. For example a > directory can contain over 15,000 files and each file averages about > 75k. I would like to sync this to another filesystem on a different > server but I am not sure if there is a rsync tuning flag I can use for > such a intensive job. I am using rsync 3.0.3. Also, I would like to > use little memory as possible.Synchronizing many files should be no problem with rsync versions >= 3 due to new incremental feature; I do this regularly with around 15 million mostly small files. But: The initial copy might take some days. If so, you might look around for a somewhat faster unconditional copy method. I prefer cpio here, but cloning tools might be better suited. Sven -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://lists.samba.org/archive/rsync/attachments/20080923/0589446d/attachment.bin
On Tue 23 Sep 2008, Mag Gam wrote:> At our lab we have storage with many small files. For example a > directory can contain over 15,000 files and each file averages about > 75k. I would like to sync this to another filesystem on a different > server but I am not sure if there is a rsync tuning flag I can use for > such a intensive job. I am using rsync 3.0.3. Also, I would like to > use little memory as possible.You may also want to look at csync2 http://oss.linbit.com/csync2/ for keeping the copies synchronised, it's perhaps better suited than rsync where you have such a large collection of files where there are not that many updates going on (i.e. the searching for updates takes significantly longer than transferring those updates). Paul Slootman