dovecot - Oct 2010 - minimize mbox mdbox fragmentation

Hi Timo,

Any chance the mbox/mdbox writer code could be modified to do physical
preallocation on files to help avoid file(system) fragmentation?
Constantly appending a file is the prime recipe for causing
fragmentation, and mbox is notorious for this--not a fault of Dovecot
but the nature of the mbox beast.  Obviously maildir doesn't have such a
problem, but some (many?) of us still prefer mbox for many reasons, fast
full body search being one, portability being another.  mdbox file
fragmentation could benefit from such a change as well.

I was having a discussion on the XFS list about this, trying to tweak
XFS mount options to mitigate the fragmentation effects.  Alas, there is
no way to do this purely at the filesystem level.  From Dave Chinner,
one of the lead XFS devs:

"What you want is _physical_ preallocation, not speculative
preallocation. i.e. look up XFS_IOC_RESVSP or FIEMAP so your
application does _permanent_ preallocate past EOF. Alternatively, the
filesystem will avoid the truncation on close() if the file has the
APPEND attribute set and the application is writing via O_APPEND...

The filesystem cannot do everything for you. Sometimes the
application has to help...."

How great would the effort need to be to implement something like this?
 Would the return on investment be sufficient to justify doing this, in
your eyes?

-- 
Stan

On Tue, 2010-10-19 at 21:55 -0500, Stan Hoeppner wrote:
> Any chance the mbox/mdbox writer code could be modified to do physical
> preallocation on files to help avoid file(system) fragmentation?
I've been thinking about that before.
> "What you want is _physical_ preallocation, not speculative
> preallocation. i.e. look up XFS_IOC_RESVSP or FIEMAP so your
> application does _permanent_ preallocate past EOF. 
Oh, interesting. I didn't know that was possible. And even better: Linux
has fallocate() that can do it for other filesystems than just XFS. Or
looks like it's only XFS and ext4 (ext3 doesn't support it). I don't
know if other OSes support this. Maybe in future I could make mdbox
support writing to files whose size has been preallocated by actually
writing NUL bytes, but that requires some extra code.

http://hg.dovecot.org/dovecot-2.0/rev/22c81f884032
http://hg.dovecot.org/dovecot-2.0/rev/b884441a713f