dovecot - Mar 2007 - Best way to migrate from Qpopper to Dovecot

Hi List,

what do you think is the best way to migrate (to a new maschine) round 
about 30000 mboxes (Qpopper) with an amount of 43Gigs data, to maildir 
format (Dovecot).
I think there are two ways:

1.
- stop services (smtp and pop3) on the old maschine
- copy the mboxes to the new maschine
- run a conversion script (for excample: "Perfect_maildir" 
http://perfectmaildir.home-dn.net/)
- start the services on the new maschine (smtp and Dovecot)

2.
- tell the users to use the new server (from the day X) with the new 
feature (imap)
- stop the smtp on the old maschine on day X
- leave the old maschine active for a month or so (let users get their 
"old" mail)

With the first alternative we have a high downtime, because of copying 
the mboxes and converting the stuff.

What do you think?


Thanks
Torsten

Torsten wrote:
> - stop services (smtp and pop3) on the old maschine
> - copy the mboxes to the new maschine
> - run a conversion script (for excample: "Perfect_maildir" 
> http://perfectmaildir.home-dn.net/)
> - start the services on the new maschine (smtp and Dovecot)
Hi,

So, parts of these processes can be parallelised:

* Copy mailbox data from old machine to the new one.  Leave production services
   running.
* Shut-down services on old machine.
* Update the mailbox data on the new machine using rsync.
* Convert mailboxes to Maildir format, eg using Perfect_maildir or mb2md.
* Start services on new server, update DNS aliases, etc.

Now, ideally, you'd do each of the above _per-user_ -- disabling just *one* 
account at a time whilst it was being updated.  This will require a bit more 
work and infrastructure support (eg a front-end POP/IMAP proxy).

Oh, and in principle, I see no reason why you can't have both SMTP services 
running at the same time..

Cheers,
David
-- 
David McBride <dwm at tastycake.net>

Hi Torsten,

You can try installing an IMAP daemon on the old box and use a tool like 
imapsync to copy existing e-mails to the new box. This will allow you to 
reduce downtime to the user whose mailbox is being copied at that point 
in time.

While this goes on, you may need a POP3/IMAP proxy like Perdition to proxy 
POP3/IMAP4 requests to the correct box that contains their mailstore.

Hope this helps in some way.

Cheers.

On Mon, 19 Mar 2007, Torsten wrote:
> Hi List,
>
> what do you think is the best way to migrate (to a new maschine) round
about
> 30000 mboxes (Qpopper) with an amount of 43Gigs data, to maildir format 
> (Dovecot).
> I think there are two ways:
>
> 1.
> - stop services (smtp and pop3) on the old maschine
> - copy the mboxes to the new maschine
> - run a conversion script (for excample: "Perfect_maildir" 
> http://perfectmaildir.home-dn.net/)
> - start the services on the new maschine (smtp and Dovecot)
>
> 2.
> - tell the users to use the new server (from the day X) with the new
feature
> (imap)
> - stop the smtp on the old maschine on day X
> - leave the old maschine active for a month or so (let users get their
"old"
> mail)
>
> With the first alternative we have a high downtime, because of copying the 
> mboxes and converting the stuff.
>
> What do you think?
>
>
> Thanks
> Torsten
>
>
>

Hi Tan (or is that your surname),

thanks very much.
This is an interesting tool, which I have to test.

Regards
Torsten


On Tue, 20 Mar 2007 08:36:46 +0800 (SGT), Tan Shao Yi <tansy at
pacific.net.sg> wrote:
> 
> Hi Torsten,
> 
> You can try installing an IMAP daemon on the old box and use a tool like
> imapsync to copy existing e-mails to the new box. This will allow you to
> reduce downtime to the user whose mailbox is being copied at that point
> in time.
> 
> While this goes on, you may need a POP3/IMAP proxy like Perdition to proxy
> POP3/IMAP4 requests to the correct box that contains their mailstore.
> 
> Hope this helps in some way.
> 
> Cheers.
> 
> On Mon, 19 Mar 2007, Torsten wrote:
>

> Hi List,
>
> what do you think is the best way to migrate (to a new maschine) round
about
> 30000 mboxes (Qpopper) with an amount of 43Gigs data, to maildir format
> (Dovecot).
> I think there are two ways:
>
> 1.
> - stop services (smtp and pop3) on the old maschine
> - copy the mboxes to the new maschine
> - run a conversion script (for excample: "Perfect_maildir"
> http://perfectmaildir.home-dn.net/)
> - start the services on the new maschine (smtp and Dovecot)
>
> 2.
> - tell the users to use the new server (from the day X) with the new
feature
> (imap)
> - stop the smtp on the old maschine on day X
> - leave the old maschine active for a month or so (let users get their
"old"
> mail)
>
> With the first alternative we have a high downtime, because of copying the
> mboxes and converting the stuff.
>
> What do you think?
Hello Torsten,

We switched our 3500+ users from Qpopper to Dovecot last week-end. We
had around 27Gb of data to migrate. We had a maintenance window in
which we could stop the POP services, so it was quite easy to do. Even
if we changed hardware, we kept the old name and IP so that we had no
DNS changes to do. Clean and simple.

Here's how we've done things on FreeBSD 6.2-RELEASE-p2

-- Shutdown qpopper POP services on the old machine.
-- Create a tar and transfered it to the new machine. Acutally, this
was a single step in which we used a pipe to send that tar STDOUT into
ssh's STDIN.
-- Import /etc/master.passwd and /etc/group from the old system into
the new machine.
-- Run a simple shell script to change from mbox format to Maildir
using mb2md and set permissions.
-- Exchange FQDN and IP between to old machine and the new one in
order to prevent any DNS changes.
-- Start Dovecot and your MTA on the new machine.

Concerning your desire to move your user's Maildir into
/var/mail/first_letter_of_username/username. It should be trivial to
do using a simple /bin/sh script.

Have fun !

David
-- 
David Robillard
UNIX systems administrator & Oracle DBA
CISSP, RHCE & Sun Certified Security Administrator
Montreal: +1 514 966 0122

> we did nearly the same procedure the last time we switched from an old
> Qpopper maschine to a new Qpopper maschine.
> The last time we had not so much users (maybe 16000), not so much data
> (round about 20Gigs) and there was no need of conversion.
> This time we have new DNS-Names, new environment (Dovecot) and so on.
That should not pose any real problems :)
> So David, may I ask you how long did it take to convert the data?
We used a cross-link cable between the two machines to transfer the
data over their gigabit link. Both systems had three 10k rpm SCSI
disks in hardware RAID 5 which pushed data at around 20 mb/s. That
took around 20 minutes.

Now for the migration script, that took about 1:30 hours to run on a
single Intel Xeon 2.80GHz.

Cheers,

David
-- 
David Robillard
UNIX systems administrator & Oracle DBA
CISSP, RHCE & Sun Certified Security Administrator
Montreal: +1 514 966 0122

> That should not pose any real problems :)
You`re right. ;-)
> We used a cross-link cable between the two machines to transfer the
> data over their gigabit link. Both systems had three 10k rpm SCSI
> disks in hardware RAID 5 which pushed data at around 20 mb/s. That
> took around 20 minutes.
>
> Now for the migration script, that took about 1:30 hours to run on a
> single Intel Xeon 2.80GHz.
That sounds really good.
I?ve to test some things and I think I`ll migrate the old mboxes.

Thanks very much.


Cheers
Torsten