dovecot - Feb 2011 - Great time savings backing a mdbox versus Maildir

Hello

     I am writing to this mailing list to thanks Timo for dovecot 2 & 
mdbox. We have almost 30.000 active users and our life was sad with 
Maildir & backup: 24 hours for a full backup  with bacula (zlib enabled 
maildirs, 1.4 TB). After switching to mdbox, the backup time is under 12 
hours ! Instead of backing 17 millions files, with mdbox our backup is 
only of 1 million files, and that speeds up a lot the backup operation.


     Timo, here you have detailed info about the bacula backup jobs, you 
can use them in the wiki if you desire. If you need aditional info 
(hardware specs, dovecot config, etc) I can share it.

*Maildir:*

|//Job:                    Backup_Linux_buzon_us.2011-01-21_19.03.26_38
   Backup Level:           Full
   Client:                 "buzon_us" 2.0.3 (06Mar07)
x86_64-redhat-linux-gnu,redhat,Enterprise release
   FileSet:                "Full Buzon" 2011-01-21 19:03:26
   Pool:                   "Pool_Linux_Buzones_US" (From Job resource)
   Catalog:                "MyCatalog" (From Client resource)
   Storage:                "File" (From command line)
   Scheduled time:         21-ene-2011 19:03:20
   Start time:             21-ene-2011 19:03:29
   End time:               22-ene-2011 19:46:45
   Elapsed time:*1 day 43 mins 16 secs*
   Priority:               10
   FD Files Written:       16,903,801
   SD Files Written:*16,903,801*
   FD Bytes Written:       1,445,943,227,706 (1.445 TB)
   SD Bytes Written:       1,448,983,971,450 (1.448 TB)
   Rate:*16247.3 KB/s*
   Software Compression:   None
   VSS:                    no
   Encryption:             no
   Accurate:               no
   Volume name(s):         Buzones_US_2
   Volume Session Id:      26
   Volume Session Time:    1295511704
   Last Volume Bytes:      1,450,628,892,676 (*1.450 TB*)
   Non-fatal FD errors:    0
   SD Errors:              0
   FD termination status:  OK
   SD termination status:  OK
   Termination:            Backup OK//|







*mdbox with mdbox_rotate_size=10m:*


|Build OS:               x86_64-redhat-linux-gnu redhat Enterprise release
   JobId:                  3587
   Job:                    Backup_Linux_buzon_us.2011-02-07_08.13.52_53
   Backup Level:           Full (upgraded from Incremental)
   Client:                 "buzon_us" 2.0.3 (06Mar07)
x86_64-redhat-linux-gnu,redhat,Enterprise release
   FileSet:                "Full Buzon" 2011-01-21 19:03:26
   Pool:                   "Pool_Linux_Buzones_US" (From Job resource)
   Catalog:                "MyCatalog" (From Client resource)
   Storage:                "File" (From command line)
   Scheduled time:         07-feb-2011 08:13:44
   Start time:             07-feb-2011 08:13:54
   End time:               07-feb-2011 19:43:50
   Elapsed time:*11 hours 29 mins 56 secs*
   Priority:               10
   FD Files Written:*1,148,780*
   SD Files Written:       1,148,780
   FD Bytes Written:*1,537,062,152,773 (1.537 TB)*
   SD Bytes Written:       1,537,218,147,402 (1.537 TB)
   Rate:*37130.7 KB/s*
   Software Compression:   None
   VSS:                    no
   Encryption:             no
   Accurate:               yes
   Volume name(s):         Buzones_US_4|Buzones_US_5
   Volume Session Id:      101
   Volume Session Time:    1296724657
   Last Volume Bytes:      438,873,898,586 (438.8 GB)
   Non-fatal FD errors:    0
   SD Errors:              0
   FD termination status:  OK
   SD termination status:  OK
   Termination:            Backup OK|



     Regards

     Javier de Miguel
     University of Seville

On Tue, Feb 08, 2011 at 08:42:40AM +0100, Javier Miguel Rodr?guez
wrote:
> 
>     I am writing to this mailing list to thanks Timo for dovecot 2 &
> mdbox. We have almost 30.000 active users and our life was sad with
> Maildir & backup: 24 hours for a full backup  with bacula (zlib
> enabled maildirs, 1.4 TB). After switching to mdbox, the backup time
> is under 12 hours ! Instead of backing 17 millions files, with mdbox
> our backup is only of 1 million files, and that speeds up a lot the
> backup operation.
Oh.. I envy you. Will probably need to do the same at some point, but
I'm having problems understanding how we will ever be able to make the
transition. Too many files -- too many users..

How long did it take to convert from maildir to mdbox, how much downtime ?

Do you have a clustered setup, or single node? I'm wondering how safe
mdbox will be on a clusterfs (GPFS), as we've had a bit of trouble with
the index-files when they're accessed from multiple nodes at the same
time (but that was with v1.0.15 -- so we should maybe trust that such
problems has since been fixed :-)


  -jf

On 8.2.2011, at 9.42, Javier Miguel Rodr?guez wrote:
>    I am writing to this mailing list to thanks Timo for dovecot 2 &
mdbox. We have almost 30.000 active users and our life was sad with Maildir
& backup: 24 hours for a full backup  with bacula (zlib enabled maildirs,
1.4 TB). After switching to mdbox, the backup time is under 12 hours ! Instead
of backing 17 millions files, with mdbox our backup is only of 1 million files,
and that speeds up a lot the backup operation.
Hmm. I guess if you were doing backups 24h/day, then you can't really say
how much faster mdbox performs than maildir (outside backups)?