Hi All, We are soon to migrate our mail server from one piece of hardware to another and we would like to take this opportunity to optimize things. As a result, we would like to replace "uw-imapd" and "qpopper" with "dovecot". The version we will be installing is 1.1.13-2, as this is what is available through the latest Debian stable backports. We will also be using exim to deliver mail (through dovecot's deliver mechanism, of course). So... We are currently using the mbox format with uw-imapd, and would like to migrate to the fastest solution possible with dovecot on the new hardware. My understanding is that "multi-dbox" is not an option in this version anyway, maildir is OK, but not great, and "single-dbox" is therefore going to be the highest performing solution. Is single-dbox the fastest way to go, considering we're going to be using email in the following ways: - IMAP connections with all email in the Inbox (Gmail-style). - IMAP connections with email split into many IMAP folders. - POP3 connections with no email left on the server. - POP3 connections with *all* email left on the server. All connections check for new mail every 5 minutes (on average) and there are 50-60 users). Also, we are not able to change user behaviour in this instance, unfortunately. Can anyone see any problems with the above proposal? Hopefully not... One problem that may arise is the fact that when we migrate, all msg UIDs will be lost. If i'm not mistaken, this means that all emails will be treated by the mail client as brand new, and if through IMAP, will all go bold, and if through POP3, will all be downloaded again (if still on the server) and therefore duplicated in the mail client. If this is the case, is there anything we can do to stop this happening? Does the "Convert" plugin does this job well? Finally, I have a rough draft of our migration plan - is there anything horribly wrong in it that's going to cause lots of problems, that anyone can spot by any chance? 1. Install Debian with exim, mysqld (for Horde/IMP) and mailman. 2. Run an online update. 3. Rsync homedirs and inboxes onto new server, ready for initial exim configuration. 4. Configure exim as per existing mail server and test that mailing lists and normal email works. You should now have the existing mail delivery solution on the brand new hardware. 5. Once mail delivery is sorted, add "deb http://www.backports.org/debian lenny-backports main contrib non-free" into "/etc/apt/sources.list" and run "aptitude update && aptitude install debian-backports-keyring && aptitude update". 6. Install dovecot (at the time of writing, this was version 1.1.13-2) and configure to use existing mbox files (inboxes in /var/spool/mail/ and IMAP folders in /home/user/mail/) 7. Setup exim to use dovecot's "deliver" mechanism for interacting with the inboxes (which are still in mbox format). 8. Configure the "convert" plugin to begin converting the mail to dbox format. 9. Run something manually (if possible) to convert mailboxes before people connect, so the task is already done by the time the outage is over. 10. Give staff access to new speedy mail server! Thanks in advance, people - any help is greatly appreciated! :-) Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.hobbs at crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999 Mobile: +44 7811 803377 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3306 bytes Desc: S/MIME Cryptographic Signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20090511/c36273fa/attachment-0002.bin>
Hello, Apologies for replying to my own email so soon, but I've had other thoughts as well... Our server is going to have 4 hard disks. These can be configured into 2 RAID 1 (mirror) arrays, a single RAID 5 array, or a single RAID 0+1 array. Previously, we thought that two RAID 1 arrays would be best because the inboxes can sit on one set of disks, and the IMAP folders on another. This is so both types of user don't annoy each other. However, with no knowledge of how the highest performing mailbox format works with dovecot, perhaps this isn't the best option. Basically, i suppose i'm asking, with the highest performing mailbox option, is dovecot going to run faster with 2 individual arrays each made from 2 disks, or a single 4-disk array (in which case we'd go RAID 0+1)? Also... would it be useful to turn off "atime" when we mount the volume(s) or does dovecot rely on this? Thanks again, people... :-) Richard. Richard Hobbs wrote:> Hi All, > > We are soon to migrate our mail server from one piece of hardware to > another and we would like to take this opportunity to optimize things. > As a result, we would like to replace "uw-imapd" and "qpopper" with > "dovecot". The version we will be installing is 1.1.13-2, as this is > what is available through the latest Debian stable backports. We will > also be using exim to deliver mail (through dovecot's deliver mechanism, > of course). > > So... We are currently using the mbox format with uw-imapd, and would > like to migrate to the fastest solution possible with dovecot on the new > hardware. My understanding is that "multi-dbox" is not an option in this > version anyway, maildir is OK, but not great, and "single-dbox" is > therefore going to be the highest performing solution. Is single-dbox > the fastest way to go, considering we're going to be using email in the > following ways: > > - IMAP connections with all email in the Inbox (Gmail-style). > - IMAP connections with email split into many IMAP folders. > - POP3 connections with no email left on the server. > - POP3 connections with *all* email left on the server. > > All connections check for new mail every 5 minutes (on average) and > there are 50-60 users). Also, we are not able to change user behaviour > in this instance, unfortunately. > > Can anyone see any problems with the above proposal? Hopefully not... > > One problem that may arise is the fact that when we migrate, all msg > UIDs will be lost. If i'm not mistaken, this means that all emails will > be treated by the mail client as brand new, and if through IMAP, will > all go bold, and if through POP3, will all be downloaded again (if still > on the server) and therefore duplicated in the mail client. If this is > the case, is there anything we can do to stop this happening? Does the > "Convert" plugin does this job well? > > Finally, I have a rough draft of our migration plan - is there anything > horribly wrong in it that's going to cause lots of problems, that anyone > can spot by any chance? > > 1. Install Debian with exim, mysqld (for Horde/IMP) and mailman. > > 2. Run an online update. > > 3. Rsync homedirs and inboxes onto new server, ready for initial exim > configuration. > > 4. Configure exim as per existing mail server and test that mailing > lists and normal email works. > You should now have the existing mail delivery solution on the brand > new hardware. > > 5. Once mail delivery is sorted, add "deb > http://www.backports.org/debian lenny-backports main contrib non-free" > into "/etc/apt/sources.list" and run "aptitude update && aptitude > install debian-backports-keyring && aptitude update". > > 6. Install dovecot (at the time of writing, this was version 1.1.13-2) > and configure to use existing mbox files (inboxes in /var/spool/mail/ > and IMAP folders in /home/user/mail/) > > 7. Setup exim to use dovecot's "deliver" mechanism for interacting with > the inboxes (which are still in mbox format). > > 8. Configure the "convert" plugin to begin converting the mail to dbox > format. > > 9. Run something manually (if possible) to convert mailboxes before > people connect, so the task is already done by the time the outage is over. > > 10. Give staff access to new speedy mail server! > > Thanks in advance, people - any help is greatly appreciated! :-) > > Richard. >-- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.hobbs at crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999 Mobile: +44 7811 803377 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3306 bytes Desc: S/MIME Cryptographic Signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20090511/ac4668cb/attachment-0002.bin>
On Mon, 2009-05-11 at 15:05 +0000, Richard Hobbs wrote:> Hi All, > > We are soon to migrate our mail server from one piece of hardware to > another and we would like to take this opportunity to optimize things. > As a result, we would like to replace "uw-imapd" and "qpopper" with > "dovecot". The version we will be installing is 1.1.13-2, as this is > what is available through the latest Debian stable backports. We will > also be using exim to deliver mail (through dovecot's deliver mechanism, > of course). > > So... We are currently using the mbox format with uw-imapd, and would > like to migrate to the fastest solution possible with dovecot on the new > hardware. My understanding is that "multi-dbox" is not an option in this > version anyway, maildir is OK, but not great, and "single-dbox" is > therefore going to be the highest performing solution. Is single-dbox > the fastest way to go, considering we're going to be using email in the > following ways:Single-dbox is the highest performing, but note that it's not as much tested as mbox and Maildir code. I think it should work ok, but I'm not aware of any larger installations using dbox currently. So in case you find a problem, you might have to upgrade/patch Dovecot to get it fixed and that would require compiling from sources.> One problem that may arise is the fact that when we migrate, all msg > UIDs will be lost. If i'm not mistaken, this means that all emails will > be treated by the mail client as brand new, and if through IMAP, will > all go bold, and if through POP3, will all be downloaded again (if still > on the server) and therefore duplicated in the mail client. If this is > the case, is there anything we can do to stop this happening? Does the > "Convert" plugin does this job well?mbox -> Maildir conversion can preserve both IMAP and POP3 UIDLs using an external script. Maildir -> dbox conversion can also preserve both, but that causes Dovecot to use this "hybrid Maildir-dbox format", which is slower than the full native dbox.> 8. Configure the "convert" plugin to begin converting the mail to dbox > format. > > 9. Run something manually (if possible) to convert mailboxes before > people connect, so the task is already done by the time the outage is over.There's convert-tool that you could use. I don't know if Debian packages it.> Basically, i suppose i'm asking, with the highest performing mailbox > option, is dovecot going to run faster with 2 individual arrays each > made from 2 disks, or a single 4-disk array (in which case we'd go > RAID > 0+1)?My guess is that two RAID-1s would be faster, but I haven't really done any benchmarking. Anyway index files are 10-30% of the mailbox size, so the index-disks would be using a lot less disk space.> Also... would it be useful to turn off "atime" when we mount the > volume(s) or does dovecot rely on this?Dovecot doesn't rely on atime updates, so turn them off. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20090511/7bf8071e/attachment-0002.bin>
Richard Hobbs wrote:> Hi All, > > We are soon to migrate our mail server from one piece of hardware to > another and we would like to take this opportunity to optimize things. >Can I recommend you add virtualisation to your todo list. I use linux-vserver, but there are plenty other ideas out there. It's just superb that later on you can migrate services between physical hardware with MUCH less hassle. You can easily test upgrades in a sandbox first. etc I personally split all the tasks into different virtual servers. Right now I actually still have quite a few mail related services in a single vserver, but ideally you would split everything up and then later if you needed to upgrade a single service or move it to a new machine it would have minimal effect. Your general process sounds about right though - I think it may be possible for you to preserve pop uidls, but see the wiki for more notes on that Good luck Ed W
Hello again, people,
I have been reading all of the emails sent back and forth recently, and
reading many web pages, and I have a new draft of my migration plan. If
anyone has the time and generosity to read through this latest plan for
me and let me know if you spot any potential problems with it, i'd
really appreciate it.
FYI, we're moving from a Debian 3.1/exim/uw-imapd/qpopper/mbox setup to
a Debian 5.0/exim/dovecot/maildir setup, and the final version of this
document can hopefully become a useful guide for lots of people in
future too, if I get it right :-)
Anyway... here's the plan...
1. Install Debian with exim, spamassassin, mysqld (for Horde/IMP) and
mailman.
Use ReiserFS for the indexes and data. XFS would be second choice
for filesystem, and this is something of a religious argument, but
Dovecot's own page states that ReiserFS is a better choice, and loads of
pages on the Internet say that one or the other is better - there's no
clear winner, really, so ReiserFS (the FS we already know and trust)
wins the battle.
2. Run an online update.
3. Setup IPMI and monitor it from Nagios.
4. Install HotSaNIC to monitor historical system stats.
5. Install and configure Dell OpenManage Server Administrator (or
whatever it's called) to monitor hardware, and make sure we will be
emailed if a hard drive fails.
6. Rsync homedirs and inboxes onto new server, ready for initial exim
configuration.
7. Configure exim as per existing mail server and test that mailing
lists and normal email works.
You should now have the existing mail delivery solution on the
brand new hardware.
8. Configure Spamassassin as per current configuration and test.
9. Configure mailman as per current configuration and test.
10. Once mail delivery is sorted, add "deb
http://www.backports.org/debian lenny-backports main contrib non-free"
into "/etc/apt/sources.list" and run "aptitude update &&
aptitude
install debian-backports-keyring && aptitude update".
11. Install dovecot (at the time of writing, this was version 1.1.13-2).
12. Run the conversion utility (use "mb2md.py" from
http://wiki.dovecot.org/Migration/MailFormat because "convert-tool"
doesn't preserve msg UIDs) to convert all existing "mbox"
mailboxes into
"maildir" mailboxes, for use with Dovecot.
13. Setup exim to use Dovecot's delivery agent ("deliver") so
the
indexes are updated on mail delivery instead of when the mailbox is
requested and also to use the new maildir format.
14. Setup dovecot for IMAP and POP3 and test.
*****THIS IS WHERE THE REPEATABLE SECTION BEGINS*****
15. Stop exim and dovecot services.
16. Rsync homedirs and inboxes onto new server again.
17. Run the conversion utility (use "mb2md.py" from
http://wiki.dovecot.org/Migration/MailFormat because "convert-tool"
doesn't preserve msg UIDs) to convert all existing "mbox"
mailboxes into
"maildir" mailboxes.
18. Start exim and dovecot services and test SMTP, Spamassassin,
mailman, IMAP and POP3.
*****THIS IS WHERE THE REPEATABLE SECTION ENDS*****
19. Once everything is working perfectly, send an email to the entire
company instructing them what to do after the outage and arrange an
outage and do the following steps as soon as the outage begins:
a. Unplug DMZ switch from firewall to make delivered mail wait at
the sender.
b. Stop exim, IMAP, POP3 and dovecot services on both mail servers.
c. Rsync homedirs and inboxes onto new server again.
d. Run the conversion utility (use "mb2md.py" from
http://wiki.dovecot.org/Migration/MailFormat because "convert-tool"
doesn't preserve msg UIDs) to convert all existing "mbox"
mailboxes into
"maildir" mailboxes.
e. Start exim and dovecot services and test SMTP, Spamassassin,
mailman, IMAP and POP3.
f. Reconfigure my own mail client to point to the new mail server
(192.168.2.3).
g. Reconfigure firewall to think the mail server is now on
192.168.2.3 instead of 192.168.2.2.
h. Prepare an email in webmail somewhere in the world, ready to send
to a user on our mail server.
i. Plug DMZ switch back into firewall and begin monitoring exim logs
to check that mail is being delivered.
j. Send the email from webmail and check that is goes through exim
perfectly and is sitting in the users Inbox.
NOTE: We now have a working mail delivery solution!
k. Start dovecot IMAP and POP3 and test both.
l. Once exim and dovecot are both working against live mailboxes,
tell everyone the outage is over.
20. Install and configure Horde & IMP to access email via IMAP.
21. Reconfigure backup script to make sure new server is backed up.
22. Modify Nagios config to monitor new server.
23. DONE!
Thanks in advance, everyone! :-)
Richard.
Brandon Lamb wrote:> On Fri, May 15, 2009 at 5:02 AM, Richard Hobbs
> <richard.hobbs at crl.toshiba.co.uk> wrote:
>> Seth Mattinen wrote:
>>> Richard Hobbs wrote:
>>>> Seth Mattinen wrote:
>>>>> Phillip Macey wrote:
>>>>>> On 14/05/2009 5:11 PM, Steffen Kaiser wrote:
>>>>>>> On Wed, 13 May 2009, Richard Hobbs wrote:
>>>>>>>> The main complaint we have from users is that
their IMAP Inbox, with
>>>>>>>> 5000 emails in it takes ages to appear, and no
amount of coaxing will
>>>>>>>> convince them to split their inbox into
multiple folders.
>>>>>>> Oh, we serve Maildir via Dovecot IMAP and 5000
messages per folder are
>>>>>>> a wimp. Problems start if the user:
>>>>>> We are having some performancec issues on our server at
the moment - all
>>>>>> I can put it down to is the large size of some
maildirs. Eg. `ls -ld
>>>>>> Maildir/cur` might show a directory >20Mb in size
for some of our users
>>>>>> (20-30k emails).
>>>>>> (Performance issues == everything is running ok then
all of a sudden
>>>>>> load avg goes through the roof, system cpu time goes
crazy. Reading mail
>>>>>> grinds to a halt. Then everything recovers just as
suddenly and the load
>>>>>> avg gradually returns to normal levels)
>>>>> Are you using ext3 by chance? Vanilla ext3 without
directory indexing
>>>>> (or whatever it's called) *hates* directories with a
lot of files - like
>>>>> maildir. Personally, I use XFS, which doesn't suffer
from this problem
>>>>> since it uses b-trees instead of a table(!) like ext3 does.
>>>> This raises another question for me actually...
>>>>
>>>> We will have one volume for indexes and another volume for data
(using
>>>> maildir). We will be using the latest stable Debian Linux
distro.
>>>>
>>>> Any opinions on the best filesystem to use? We would normally
go
>>>> ReiserFS for large volumes, and ext3 for small volumes because
of the
>>>> unlimited inodes in reiserfs, but i understand that support for
that is
>>>> beginning to disappear given that Hans Reiser got into a bit of
trouble
>>>> a few years ago.
>>>>
>>>> Anyway... that would leave ext3, but is there a better choice i
could
>>>> make in this instance? We do want performance, of course, but
also
>>>> complete reliability and resilience when it comes to system
crashes
>>>> etc... we do *not* want data corruption, and ext3 we know has a
very
>>>> good journalling and data recovery methods. Well... they're
very mature,
>>>> anyway.
>>>>
>>> I used to use ext3, ran into its horrible performance even with
>>> directory indexing enabled, went to XFS and never looked back. All
of my
>>> systems are Debian stable. Reiser3 is part of the kernel so I
wouldn't
>>> worry about that; Namesys considered it complete and stopped work
on it
>>> long before the whole murder thing. Both Reiser3 and XFS have worse
>>> reputations than ext3, however, I've seen ext3 filesystems
hosed beyond
>>> repair, too. All my XFS filesystems have battery-backed cache
>>> controllers, so it hasn't happened to me yet, hopefully never.
;) One
>>> catch with XFS (such as with LVM) to keep in mind is you can't
ever
>>> shrink it, only grow.
>> Trouble is... i've been googling this as well, just now, and loads
of
>> people say XFS has the better performance, but loads of other people
say
>> ReiserFS has the better performance.
>>
>> We have battery backed up RAID controllers too, in this new system, and
>> the systems are UPSd, so on that basis i'm still none the wiser!
lol
>>
>> I appreciate your experience with XFS is a positive one, but even the
>> dovecot web site says XFS might now be a good choice...
>>
>> http://wiki.dovecot.org/MailboxFormat/Maildir
>>
>> What a tough decision! I know it probably won't make too much
difference
>> in my situation, but i want this to be a very long-term solution, so
>> want to do it right first time!
>>
>> Any other opinions on XFS vs Reiserfs with Dovecot maildir?
>>
>> Thanks again!
>>
>> Richard.
>>
>>
>>> ext3 is mature but IMHO completely unsuitable for a busy mail
server or
>>> any situation where you'll have a bajillion of files in one
directory.
>>> The exact point at which ext3 will screw you over obviously depends
on
>>> many factors. But when it happens it'll probably be painful to
reformat
>>> to something better.
>>>
>>> ~Seth
>>>
>>>
______________________________________________________________________
>>> This email has been scanned by the MessageLabs Email Security
System.
>>> For more information please visit http://www.messagelabs.com/email
>>>
______________________________________________________________________
>>>
>>>
>> --
>> Richard Hobbs (IT Specialist)
>> Toshiba Research Europe Ltd. - Cambridge Research Laboratory
>> Email: richard.hobbs at crl.toshiba.co.uk
>> Web: http://www.toshiba-europe.com/research/
>> Tel: +44 1223 436999 Mobile: +44 7811 803377
>
> We have been using XFS on our raids for the last few years. We have
> fairly new hardware, and using maildir. Delivering over nfs using
> exim+dovecot deliver. pop3/imap is near instant load on our webmail.
> We have about 16,000 email accounts with ~175 imap+pop3 connections
> open all the time. 20 drive sata2 raid 10. amd quad core 9850 with 8
> gigs ram. load sits about 0.30 - 0.45 all day/night. Last I checked
> backups were 30-45 minutes, 413 gigs. 6,403,942 files.
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
>
--
Richard Hobbs (IT Specialist)
Toshiba Research Europe Ltd. - Cambridge Research Laboratory
Email: richard.hobbs at crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 436999 Mobile: +44 7811 803377
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3306 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20090520/aa1e401a/attachment-0002.bin>
Richard Hobbs <richard.hobbs at crl.toshiba.co.uk> writes:> 19. Once everything is working perfectly, send an email to the entire > company instructing them what to do after the outage and arrange an > outage and do the following steps as soon as the outage begins: > > a. Unplug DMZ switch from firewall to make delivered mail wait at > the sender.[...]> i. Plug DMZ switch back into firewall and begin monitoring exim logs > to check that mail is being delivered.If I'm not misunderstanding the steps between 19.a -- 19.i are going to be done while not network connected? I'd be slightly concerned that these steps may involve anything some that needs to do DNS lookups or the like at which point they may hit long(ish) timeouts or just fail completely.