Hello I am starting testing with dovecot 2.0.9. In my //etc/dovecot/dovecot.conf/ I have the following line: /mail_location = mdbox:/mydomain.org/%2.26Hn/%2.200Hn/%n:INDEX=/mydomain.org/indexes/%2.26Hn/%2.200Hn/%n/ These are my questions (a long mail): *1? How should I restore from backup with mdbox & separate indexes?* I have read in dovecot list archive about restoring the full user mailbox to a temp location and then run something like: /doveadm import mdbox:/tmp/restored_mailbox/ "" savedsince 2011-01-01 / In my schema indexes and mail stores are in separated paths. What (and where!) should I copy to the temp location and how do I restore it? *2? Disaster recovery: indexes & mail store mismatch* When creating backups, indexes and mail store are in different paths. My backup takes several hours, so indexes are copied BEFORE mail stores. Is the backup consistent? What would happens if I lose indexes? Can I still recover the messages? Reading the doc it seems I would lose all the message flags and all the messages would be in INBOX, is this right? *3? Performance tuning with mdbox* I have read in dovecot archive about using mdbox_rotate_size & mdbox_rotate_interval to balance between a lot of smaill files (like Maildir) or really big files (a "variant" of mbox). I have tested with a big vaule of mdbox_rotate_size (100 MB) and is a complete disaster for backup:if a user delete a message, then a nightly cronjob purges the mailbox (refcount=0) a new mdbox .m file is created!. So deleting a 1 k message from the mailbox of an user is a new 99 MB file to backup. Any hint about this? I am also using ext4 (rw,noatime,nodiratime,commit=15,data=writeback) and I have read about mdbox_preallocate_space. I will try it, any hint about this setup? This is not still in production, so I have no idea about IOPs and user concurrency, but I have several thousand users. *4? Compression with mdbox* I am reading about compression in mdbox (zlib/bzlib). I am starting with zlib, has anyone experience with bzlib? How much load is bzlib against zlib and how much delay adds to user experience (and space savings vs zlib???) ? *5? Recover a message with refcount=0* Is there any way to recover a message with refcount=0 without using a backup? I am thinking about the following plan: /1? a user gets her mail via pop3 (or delete a message via imap and empties trash) 2? the message is still stored in my mail server, but has refcount=0 3? nightly backup is performed 4? nightly cronjob purges the mailbox (deleting messaged with refcount=0)/ Using this schema the backup is bigger, but If I can recover files with refcount=0 from backup we can give our users a safeguard against accidentaly deleted pop3 mail. Thank you very much for your responsed and sorry for my (awful) english! Maria
Hello everybody I have found that with bzip compression level 6 I have a better balance between compression and decompression speed with dovecot+mdbox, but still too slow compared with zlib. Does anybody have any other hint about my original questions (see below) Regards Maria Arrea ----- Original Message ----- From: Maria Arrea Sent: 02/15/11 09:16 PM To: dovecot at dovecot.org Subject: [Dovecot] Some questions about mdbox Hello I am starting testing with dovecot 2.0.9. In my //etc/dovecot/dovecot.conf/ I have the following line: /mail_location = mdbox:/mydomain.org/%2.26Hn/%2.200Hn/%n:INDEX=/mydomain.org/indexes/%2.26Hn/%2.200Hn/%n/ These are my questions (a long mail): *1? How should I restore from backup with mdbox & separate indexes?* I have read in dovecot list archive about restoring the full user mailbox to a temp location and then run something like: /doveadm import mdbox:/tmp/restored_mailbox/ "" savedsince 2011-01-01 / In my schema indexes and mail stores are in separated paths. What (and where!) should I copy to the temp location and how do I restore it? *2? Disaster recovery: indexes & mail store mismatch* When creating backups, indexes and mail store are in different paths. My backup takes several hours, so indexes are copied BEFORE mail stores. Is the backup consistent? What would happens if I lose indexes? Can I still recover the messages? Reading the doc it seems I would lose all the message flags and all the messages would be in INBOX, is this right? *3? Performance tuning with mdbox* I have read in dovecot archive about using mdbox_rotate_size & mdbox_rotate_interval to balance between a lot of smaill files (like Maildir) or really big files (a "variant" of mbox). I have tested with a big vaule of mdbox_rotate_size (100 MB) and is a complete disaster for backup:if a user delete a message, then a nightly cronjob purges the mailbox (refcount=0) a new mdbox .m file is created!. So deleting a 1 k message from the mailbox of an user is a new 99 MB file to backup. Any hint about this? I am also using ext4 (rw,noatime,nodiratime,commit=15,data=writeback) and I have read about mdbox_preallocate_space. I will try it, any hint about this setup? This is not still in production, so I have no idea about IOPs and user concurrency, but I have several thousand users. *4? Compression with mdbox* I am reading about compression in mdbox (zlib/bzlib). I am starting with zlib, has anyone experience with bzlib? How much load is bzlib against zlib and how much delay adds to user experience (and space savings vs zlib???) ? *5? Recover a message with refcount=0* Is there any way to recover a message with refcount=0 without using a backup? I am thinking about the following plan: /1? a user gets her mail via pop3 (or delete a message via imap and empties trash) 2? the message is still stored in my mail server, but has refcount=0 3? nightly backup is performed 4? nightly cronjob purges the mailbox (deleting messaged with refcount=0)/ Using this schema the backup is bigger, but If I can recover files with refcount=0 from backup we can give our users a safeguard against accidentaly deleted pop3 mail. Thank you very much for your responsed and sorry for my (awful) english! Maria
On 15.2.2011, at 22.16, Maria Arrea wrote:> *1? How should I restore from backup with mdbox & separate indexes?* > > I have read in dovecot list archive about restoring the full user mailbox to a temp location and then run something like: > > /doveadm import mdbox:/tmp/restored_mailbox/ "" savedsince 2011-01-01 / In my schema indexes and mail stores are in separated paths. What (and where!) should I copy to the temp location and how do I restore it?The import parameter works exactly the same as mail_location. So if you for example restore user's files to: /tmp/backup/mdbox/storage/m.* /tmp/backup/indexes/dovecot.index* You can use "doveadm import mdbox:/tmp/backup/mdbox:INDEX=/tmp/backup/indexes".> *2? Disaster recovery: indexes & mail store mismatch* > > When creating backups, indexes and mail store are in different paths. My backup takes several hours, so indexes are copied BEFORE mail stores. Is the backup consistent?Well, this isn't ideal.. Better would probably be if indexes were backed up after. Now it's possible that when restoring a mail there are some mails that aren't in index yet. You could run "doveadm force-resync" to make sure they're all added.> What would happens if I lose indexes? Can I still recover the messages? Reading the doc it seems I would lose all the message flags and all the messages would be in INBOX, is this right?doveadm force-resync makes all messages visible, even if all flags are lost. The messages will show up where they were saved to the first time (INBOX and Sent mainly, also if Sieve fileinto was used then those too). So mainly if user/client copied/moved messages around, those changes would be lost. Anyway, since you have backups and Dovecot also generates dovecot.index.backup files, you should always have some versions of them available. And mdbox index rebuilding tries really hard to use those files to avoid losing changes.> *3? Performance tuning with mdbox* > > I have read in dovecot archive about using mdbox_rotate_size & mdbox_rotate_interval to balance between a lot of smaill files (like Maildir) or really big files (a "variant" of mbox). I have tested with a big vaule of mdbox_rotate_size (100 MB) and is a complete disaster for backup:if a user delete a message, then a nightly cronjob purges the mailbox (refcount=0) a new mdbox .m file is created!. So deleting a 1 k message from the mailbox of an user is a new 99 MB file to backup. Any hint about this?Well, that's how it is intended to work.. Typically users would delete only recent mails, so if you rotate files daily, then deletions would happen only from the file that hasn't yet even been backed up. Of course, with daily rotations you rarely get even close to 100 MB files.> *5? Recover a message with refcount=0* > > Is there any way to recover a message with refcount=0 without using a backup? I am thinking about the following plan: > > /1? a user gets her mail via pop3 (or delete a message via imap and empties trash) > 2? the message is still stored in my mail server, but has refcount=0 > 3? nightly backup is performed > 4? nightly cronjob purges the mailbox (deleting messaged with refcount=0)/ > > Using this schema the backup is bigger, but If I can recover files with refcount=0 from backup we can give our users a safeguard against accidentaly deleted pop3 mail.There is no clean way to do this currently. An ugly way would be to delete dovecot.map.index* files and rebuild index. The rebuilding sees that there are some mails that don't exist in any mailbox indexes, and assigns them to the mailbox where they initially were saved to. No (flag) changes are lost by doing this.