hey, i am considering changing my mailbox format from maildir to mdbox. the reason for this is mainly, b/c i have (a) multiple large mailboxes with tens of thousands of mail files, and (b) most of these mails files have a size significantly smaller than the sector size of the disk. so, since the emails themselves are only about nGB large, the disk space used is at least twice as much, if not even three times. i know, hard disk space is 'cheap', but still... but then i read at https://doc.dovecot.org/admin_manual/mailbox_formats/dbox/ the following: [...] you must not lose the dbox index files, as they can?t be regenerated without data loss. so, raid is mandatory, which is already the case, but what about backup? how can i achieve a backup/snapshot of both, the mdbox (nfs share) and the index files (local raid) and assure they are consistent? greetings...
> > i am considering changing my mailbox format from maildir to mdbox. the > reason for this is mainly, b/c i have (a) multiple large mailboxes with > tens of thousands of mail files, and (b) most of these mails files have > a > size significantly smaller than the sector size of the disk. > > so, since the emails themselves are only about nGB large, the disk space > used is at least twice as much, if not even three times. > > i know, hard disk space is 'cheap', but still...Also think about iops. rsyncing 10GB of small files takes a lot longer than 10 1GB files.> > but then i read at > > https://doc.dovecot.org/admin_manual/mailbox_formats/dbox/ > > the following: > > [...] you must not lose the dbox index files, as they can?t be > regenerated without data loss.I have read this also, and was also worried about this, but when I look at the flat m.988 file, I still have quite a lot of useful data there. Received: from xxxxx (localhost [127.0.0.1]) by xxxxxxx (8.15.2/8.15.2) with ESMTPS id 29IAeeZw2293734 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT) for <xxxxxxxxxx>; Tue, 18 Oct 2022 12:40:40 +0200 X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.5 at xxxxxxx Received: (from xxxxxx at localhost) by xxxxxxxxxx (8.15.2/8.15.2/Submit) id 29IAeeoF2293733 for xxxxxxx; Tue, 18 Oct 2022 12:40:40 +0200 From: xxxxx <xxxxxxxxxxxxx> Message-ID: <202210181040.29IAeeoF2293733 at xxxxxxxxxxxxx> Date: Tue, 18 Oct 2022 12:40:40 +0200 To: <xxxxxxx> Subject: test User-Agent: Heirloom mailx 12.5 7/5/10 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Received-SPF: neutral (spf not configured) test R634e82ab V7e3 G18424b37ab824e63be3a0000fe361dd3 BINBOX
On 10/18/22 18:17, Michael wrote:> what about backup? how can i achieve a backup/snapshot of both, the mdbox (nfs share) and the index?files?(local?raid)?and?assure?they?are?consistent?If you do your backups using doveadm backup, then the result should be consistent, at least in the sense that it would be usable. Your destination can also be set up similarly with separate storage for indexes. However I'm pretty sure the consistency would be per mailbox ("folder"), so e.g. if a user moved a message from one mailbox to another, you could potentially end up with the message appearing in both mailboxes in the backup.
On 18/10/2022 12:17, Michael wrote: > > [...]> so, raid is mandatory, which is already the case, but what about backup? > how can i achieve a backup/snapshot of both, the mdbox (nfs share) and > the index files (local raid) and assure they are consistent?You can use doveadm to backup the mailboxes, which should work correctly even in a live system. My backup "strategy" (hopefully it deserves that name) is to weekly run something like: for MAILBOX in $USERS; do doveadm expunge -u "$MAILBOX" mailbox Trash savedbefore 7d doveadm expunge -u "$MAILBOX" mailbox Spam savedbefore 30d doveadm purge -u "$MAILBOX" LOCATION2="mdbox:/srv/snap_mail/$MAILBOX/mdbox" doveadm -v backup -u "$MAILBOX" -P "$LOCATION2" done which makes a replica of the mailbox (including dovecot.list.index, but e.g. dovecot-uidvalidity is not there, I don't know if this is bad or not). Once you have this "snapshot" of the mailbox(es), you can rsync them to wherever you like (so you avoid rsync'ing a changing system). I've never had to restore from backup (rsync back, and either doveadm backup back in the other direction), but I'd tend to assume this should work fine. BTW I use mdbox_rotate_interval = 0 and mdbox_rotate_size = 64M, so the mdbox'es have a relatively big size (but not too big), which is nice for rsync. Cheers, Bernardo