dovecot - Aug 2013 - The docs a re a bit weird on "Directory hashing"

In squid we use a double layer of hashed directories on the FS to allow
storage of millions of files.
I was reading the "Directory hashing" section but never understood
it..
since it's written.. in a way I could not understand.
I am using this line:
mail_location = maildir:/home/vmail/%d/%n/Maildir/

and I want to migrate to a hash based directory scheme.
While trying to understand how that hash will work I stumbled a old
thread at:
http://www.dovecot.org/list/dovecot/2010-June/049695.html
there they used:
mail_location=maildir:/buzones/us.es/%1Hu/%2.1u/%n

so I assume it should be used like that:
mail_location=maildir:/home/vmail/%H/%2.256Hn/%d_%n/Maildir/
or:
mail_location=maildir:/home/vmail/%1Mu/%2.1Mu/%d_%n/Maildir/

It's a bit hard to think alone so I hope you can assist me.

let say I want to follow the model of squid cache_dir which has:
cache_dir aufs /usr/local/squid/var/cache/squid 40000 16 256

And means a two layers cache of max 16 directories on the first layer
and 256 directories on the second layer.
The above allows millions of files storage and can benefit from all ext4
lower kernel levels of compatibly rather then do stuff on the user-land..
Since I am not 100% sure that the scheme I understood is indeed what I
think I assume the above will need a small correction.

Eliezer

On Thu, Aug 08, 2013 at 01:42:43AM +0300, Eliezer Croitoru
wrote:
> 
> And means a two layers cache of max 16 directories on the first layer
> and 256 directories on the second layer.
> The above allows millions of files storage and can benefit from all ext4
> lower kernel levels of compatibly rather then do stuff on the user-land..
> Since I am not 100% sure that the scheme I understood is indeed what I
> think I assume the above will need a small correction.
I use:

	mail_home = /srv/mailstore/%256LRHu/%Ld/%Ln

which gives me 256 buckets containing domainname/username/, and the
buckets are a hash of Lowercase Reverse usernames. To get the same
layout as squid, I would try:

	mail_home = /srv/mailstore/%16LRHu/%256LRHu/%Lu

Ref: http://wiki2.dovecot.org/Variables for variables and modifiers.

BTW: I'm lowercasing everything, because I once got bitten by a variable
not being lowercased in one version, and suddenly this changing in
another version. It's probably redundant here -- but it was painful to
fix when it happened..


  -jf

BTW. If you're using v2.2.3+ %N hash works better than the old %H hash. I
updated http://wiki2.dovecot.org/Variables for it also.

On 8.8.2013, at 1.42, Eliezer Croitoru <eliezer at ngtech.co.il> wrote:
> In squid we use a double layer of hashed directories on the FS to allow
> storage of millions of files.
> I was reading the "Directory hashing" section but never
understood it..
> since it's written.. in a way I could not understand.
> I am using this line:
> mail_location = maildir:/home/vmail/%d/%n/Maildir/
> 
> and I want to migrate to a hash based directory scheme.
> While trying to understand how that hash will work I stumbled a old
> thread at:
> http://www.dovecot.org/list/dovecot/2010-June/049695.html
> there they used:
> mail_location=maildir:/buzones/us.es/%1Hu/%2.1u/%n
> 
> so I assume it should be used like that:
> mail_location=maildir:/home/vmail/%H/%2.256Hn/%d_%n/Maildir/
> or:
> mail_location=maildir:/home/vmail/%1Mu/%2.1Mu/%d_%n/Maildir/
> 
> It's a bit hard to think alone so I hope you can assist me.
> 
> let say I want to follow the model of squid cache_dir which has:
> cache_dir aufs /usr/local/squid/var/cache/squid 40000 16 256
> 
> And means a two layers cache of max 16 directories on the first layer
> and 256 directories on the second layer.
> The above allows millions of files storage and can benefit from all ext4
> lower kernel levels of compatibly rather then do stuff on the user-land..
> Since I am not 100% sure that the scheme I understood is indeed what I
> think I assume the above will need a small correction.
> 
> Eliezer
>