dovecot - Mar 2013 - Maildir or Mdbox and expunge messages.

Hello.

I tested dovecot's work with expunge messages on Mdbox over strace (see
at tail of message). As I can see  dovecot process opens old storage m.*
file, reads it content, opens new temporary file, writes in this one
content and rename this new one to m.(*+1).

How fast this algorithm works on system with about 10000 users and 5Tb
data? I will use mdbox_rotate_interval for delayed expunge but I think
that simple delete file in Maildir must be faster than expunge from Mdbox.

Please, tell me about real experience work with Mdbox on big loaded systems.


++++++++++++++++

    0.000017
open("/var/vmail/example.org/user/storage/.temp.1362474108.P7152Q0M909647.mailsys",
O_RDWR|O_CREAT|O_TRUNC, 0666) = 11
     0.000063 umask(077)                = 066
     0.000015 fallocate(11, 01, 0, 10485760) = 0
     0.000054 fstat(11, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
     0.000043 pread(10, "\1\2N          0000000000000285\nFC"...,
8192,
16) = 1465
     0.000025 write(11, "2 M1e C5135b47c\n\1\2N          000"..., 691)
= 691
     0.000041
stat("/var/vmail/example.org/user/indexes/storage/dovecot.map.index.log.2",
0x7fff235b34c0) = -1 ENOENT (No such file or directory)
     0.000026 alarm(180)                = 0
     0.000018 fcntl(9, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET,
start=0, len=0}) = 0
     0.000020 alarm(0)                  = 180
     0.000018
stat("/var/vmail/example.org/user/indexes/storage/dovecot.map.index.log",
{st_mode=S_IFREG|0600,
st_size=3656, ...}) = 0
     0.000031 fstat(9, {st_mode=S_IFREG|0600, st_size=3656, ...}) = 0
     0.000041 write(11, "\n\1\3\nR5135ad97\nV29b\nG5840870697ad"...,
62)
= 62
     0.000023 fdatasync(11)             = 0
     0.079275 stat("/var/vmail/example.org/user/storage/m.8",
0x7fff235b3640) = -1 ENOENT (No such file or directory)
     0.000061
rename("/var/vmail/example.org/user/storage/.temp.1362474108.P7152Q0M909647.mailsys",
"/var/vmail/example.org/user/storage/m.8") = 0
     0.000164 write(9,
"\200\200\200\203\0\0\10\20\210\0\0\0\200\200\200\207@\0\0\20\0\0\0\0\0\0\0\0\10\0\0\0"...,
152) = 152
     0.000094 fcntl(9, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0,
len=0}) = 0
     0.000042 unlink("/var/vmail/example.org/user/storage/m.7") = 0
     0.000059
stat("/var/vmail/example.org/user/indexes/storage/dovecot.map.index.log",
{st_mode=S_IFREG|0600,
st_size=3808, ...}) = 0
     0.000055 fstat(9, {st_mode=S_IFREG|0600, st_size=3808, ...}) = 0

On 5.3.2013, at 11.55, Evgeny Basov <ya.mwork at yandex.ru> wrote:
> I tested dovecot's work with expunge messages on Mdbox over strace (see
> at tail of message). As I can see  dovecot process opens old storage m.*
> file, reads it content, opens new temporary file, writes in this one
> content and rename this new one to m.(*+1).
You're talking about doveadm purge, right? Not expunge.
> How fast this algorithm works on system with about 10000 users and 5Tb
> data? I will use mdbox_rotate_interval for delayed expunge but I think
> that simple delete file in Maildir must be faster than expunge from Mdbox.
Yes, Maildir/sdbox is faster when freeing up disk space for mails. Read and save
operations should be faster in mdbox though. And if the purge is run at a time
when there is less disk IO, it shouldn't be that bad.