I've been running continous dsync backups of our Maildirs for a few
weeks now, with the destination dsync server using mdbox and SIS. The
idea was that the destination server would act as a warm copy of
all our active users data.
The active servers are using Maildir, and has:
$ df -h /usr/local/atmail/users/
Filesystem Size Used Avail Use% Mounted on
/dev/atmailusers 14T 12T 2.2T 85% /usr/local/atmail/users
$ df -hi /usr/local/atmail/users/
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/atmailusers 145M 113M 33M 78% /usr/local/atmail/users
very little of this is compressed (zlib plugin enabled during christmas).
I'm surprised that the destination server is so large, was expecting zlib
and
mdbox and SIS would compress it down to much less than what we're seeing
(12TB -> 5TB):
$ df -h /srv/mailbackup
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/mailbackupvg-mailbackuplv
5.7T 4.8T 882G 85% /srv/mailbackup
Lots and lots of the attachement storage is duplicated into identical files,
instead of hard linked.
When running "doveadm purge -u $user", we're seeing lots of
Error:
unlink(/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab)
failed: No such file or directory
"/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab"
is
missing, but there are 205 other copies of this file named
/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-*
with
identical sha1sum.
Also we see corrupted indexes during the purge. This makes me quite uncertain
if dsync is a workable backup solution.. or if we can trust mdboxes.
Also on the source side, during dsync, we see too many problems. Some samples:
Error: Mailboxes don't have unique GUIDs: 08b46439069d3d4db0490000e671bf84
is shared by INBOX and INBOX
Error: command BOX-LIST failed
Error: Worker server's mailbox iteration failed
Error: read() from worker server failed: EOF
Error: Failed to sync mailbox INBOX.ferie 2006.: Invalid mailbox name
Error: read() from proxy client failed: EOF
Error: Unexpected finish reply: 1 596fec275888dbd89f6d1f5356c22db6 3720
0 \dsync-expunged 0
Error: Unexpected reply from server: 1 12200572a70726fca946da6f9378dc03
3721 0 \dsync-expunged 0
Error: Failed to sync mailbox INBOX.INBOX.Gerda: Mailbox doesn't exist:
INBOX/Gerda
Error: command BOX-LIST failed
Error: read() failed: Broken pipe
Panic: file dsync-worker-local.c: line 1678 (local_worker_save_msg_continue):
assertion failed: (ret == -1)
Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0 [0x367703c680] ->
/usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x35) [0x367703c765]
-> /usr/lib64/dovecot/libdovecot.so.0 [0x367703bb93] -> /usr/bin/dsync
[0x40f48d] -> /usr/bin/dsync [0x40f589] ->
/usr/bin/dsync(dsync_worker_msg_save+0x8e) [0x40eb3e] -> /usr/bin/dsync
[0x40d71a] -> /usr/bin/dsync [0x40cdbf] -> /usr/bin/dsync [0x40d105] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x48) [0x3677047278] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0xd5) [0x36770485c5]
-> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x2d) [0x367704720d] ->
/usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x3677035a83] ->
/usr/bin/dsync(main+0x71e) [0x406c4e] ->
/lib64/libc.so.6(__libc_start_main+0xf4) [0x3e3941d994] -> /usr/bin/dsync
[0x406369]
Do you have any idea for what our problems might be? Should we:
avoid SIS ?
avoid doing Maildir on one side and mdbox on the other?
try other dovecot version for dsync?
anything else?
-jf
------------- destination server, running dovecot v2.0.14 --------
mail_attachment_dir = /srv/mailbackup/attachments
mail_location = mdbox:~/mdbox
mail_plugins = zlib
mdbox_rotate_size = 5 M
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
passdb {
driver = static
}
plugin {
zlib_save = gz
zlib_save_level = 9
}
protocols =
service auth-worker {
user = $default_internal_user
}
service auth {
unix_listener auth-userdb {
mode = 0600
user = mailbackup
}
}
ssl = no
userdb {
args = home=/srv/mailbackup/%256Hu/%d/%n
driver = static
}
-------------/destination server --------
-jf
Il 01/02/2012 13:29, Jan-Frode Myklebust ha scritto:> I've been running continous dsync backups of our Maildirs for a few > weeks now, with the destination dsync server using mdbox and SIS. The > idea was that the destination server would act as a warm copy of > all our active users data.How many users there are in this installation?> The active servers are using Maildir, and has: > > $ df -h /usr/local/atmail/users/ > Filesystem Size Used Avail Use% Mounted on > /dev/atmailusers 14T 12T 2.2T 85% /usr/local/atmail/users > $ df -hi /usr/local/atmail/users/ > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/atmailusers 145M 113M 33M 78% /usr/local/atmail/users > > very little of this is compressed (zlib plugin enabled during christmas).This is the old storage in Maildir format?> I'm surprised that the destination server is so large, was expecting zlib and > mdbox and SIS would compress it down to much less than what we're seeing > (12TB -> 5TB): > > $ df -h /srv/mailbackup > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/mailbackupvg-mailbackuplv > 5.7T 4.8T 882G 85% /srv/mailbackupThis is the new storage in mdbox format? What size you would expect? -- Alessio Cecchi is: @ ILS -> http://www.linux.it/~alessice/ on LinkedIn -> http://www.linkedin.com/in/alessice Assistenza Sistemi GNU/Linux -> http://www.cecchi.biz/ @ PLUG -> ex-Presidente, adesso senatore a vita, http://www.prato.linux.it @ LOLUG -> Socio http://www.lolug.net
On Wed, 2012-02-01 at 13:29 +0100, Jan-Frode Myklebust wrote:> I'm surprised that the destination server is so large, was expecting zlib and > mdbox and SIS would compress it down to much less than what we're seeing > (12TB -> 5TB):Note that with SIS the attachments aren't compressed.> Lots and lots of the attachement storage is duplicated into identical files, > instead of hard linked.Something's wrong then.> When running "doveadm purge -u $user", we're seeing lots of > > Error: unlink(/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab) failed: No such file or directorySomething's wrong.> "/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab" is > missing, but there are 205 other copies of this file named > /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-* with > identical sha1sum.All of them have a link count of 2, with the other link being in hashes/ directory?> Also on the source side, during dsync, we see too many problems.That is most likely related to your troubles. If the dsync runs crash, the result could leave extra files lying around etc..> Some samples: > > Error: Mailboxes don't have unique GUIDs: 08b46439069d3d4db0490000e671bf84 is shared by INBOX and INBOXThis is a little bit strange. What is the doveconf -n output of the source server?> Error: Failed to sync mailbox INBOX.ferie 2006.: Invalid mailbox nameIs this a namespace prefix? It shouldn't be trying to sync a mailbox named this (there's an extra "." suffix).> Error: read() from proxy client failed: EOFI guess the remote dsync crashes or otherwise aborted.> Error: Failed to sync mailbox INBOX.INBOX.Gerda: Mailbox doesn't exist: INBOX/GerdaI guess some kind of mismatch related to namespace configuration.> Error: read() failed: Broken pipe > Panic: file dsync-worker-local.c: line 1678 (local_worker_save_msg_continue): assertion failed: (ret == -1)Probably can't handle properly when remote dsync dies. Of course it still shouldn't crash. There seems to be some bugs left when dsyncing to a remote host (instead of locally). It would help if I could reproduce the errors that you're seeing. Can you easily reproduce them with some accounts? If so, if you can give enough details for me to reproduce the problems I can fix them. (Except for the "file not found" issues, since that problems occurred earlier already. I should probably somehow make Dovecot fix those missing files though..)