rsync - Jul 2010 - deep vs. shallow directory structure

Hi all,

I am using rsync to sync a directory tree of ~2 million files. I am using
rsyncrypto with filename encryption to encrypt the files before syncing.

When encrypting filenames, rsyncrypto assigned each file a filename which is
made of hex digits. It also has the option to put the files in directories,
where the location of the file is determined by the file's name and a
parameter indicating the depth of the directory structure. i.e. if the
filename is 2EB2FE7F11258E27AF21BF60D2F4B06B , and the depth is 3, the file
will be in:
2/2E/2EB/2EB2FE7F11258E27AF21BF60D2F4B06B
for a depth of 1 the file will be in:
2/2EB2FE7F11258E27AF21BF60D2F4B06B
etc...

My question (finally): how to optimize rsync's speed? - how deep should the
directory structure be? What would be faster to rsync - 1 directory with 2M
files, or 16 directories with 125K files each, or 256 directories with 8K
files each, ....

Shai
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20100703/f501eacd/attachment.html>

On Sat 03 Jul 2010, Shai Ayal wrote:
> My question (finally): how to optimize rsync's speed? - how deep should
the
> directory structure be? What would be faster to rsync - 1 directory with 2M
> files, or 16 directories with 125K files each, or 256 directories with 8K
> files each, ....
Smaller directories will enable rsync's incremental transfer mode to
work more efficiently. I'd go for 256 directories in this scenario.
Rsync can then scan de next directory while transferring the current
one.
More directories probably won't help much.


Paul