dovecot - Feb 2012 - Questions still about a uw-imap -> dovecot conversion

The more I read about all of this, the more I'm thinking about moving to 
maildir format. My switchover this weekend is full of holes due to the 
way user's imap folders were laid out. Some had folders in their home 
directory and others might have folders in their /home/mail directory. 
Some how, Horde/Imp seemed to keep track of it. Dovecot is not doing so 
well.

I've been reading about the conversion technique to convert to maildir, 
but I'm wondering if I can do this based on the way our users are set 
up. Most are pop accounts. Our webmail is configured to read the inbox 
from /var/spool/mail and their imap folders from /home or /home/mail. 
Some have imap accounts on their desktop defined through their mail client.

RH/Centos seems to want me to switch to postfix since most of the 
documentation I find doesn't mention Sendmail. My filesystem is laid out 
to handle Inboxes in /var/spool/mail and imap folders in /home(/mail) 
and these aren't logical volumes but true partitions, so I've got to 
consider this as I make any changes.

I also run pop using dovecot, so I'll need to figure all this out.

I've read the technique linked to on the dovecot site that points to 
"Replacing UW-IMAP with Dovecot on RHEL 3" written back in 2004, and 
that's about the only real pointers I'm finding.

I don't want to have to go around to each machine and change their 
clients to IMAP..

Anyone care to comment on anything similar they've done and how many 
pitfalls they ran into and what they were?

steve campbell

On 2012-02-20 1:10 PM, Steve Campbell <campbell at cnpapers.com>
wrote:
> Our webmail is configured to read the inbox from /var/spool/mail and
> their imap folders from /home or /home/mail.
I can't help with your specific problem, but I do know that having mail 
stored directly in /home will cause problems - you will need to fix that.

-- 

Best regards,

Charles

On 2/20/12 1:10 PM, Steve Campbell wrote:
> The more I read about all of this, the more I'm thinking about moving 
> to maildir format. My switchover this weekend is full of holes due to 
> the way user's imap folders were laid out. Some had folders in their 
> home directory and others might have folders in their /home/mail 
> directory. Some how, Horde/Imp seemed to keep track of it. Dovecot is 
> not doing so well.

Not sure what problem maildir will solve for you.  Large mbox mailboxes 
under UW-IMAP are an awful bottleneck and will bring the most powerful 
storage subsystems I've used to their knees, but add Dovecot indexing on 
top and you will be surprised how much faster everything is.  We're 
still using mbox today and if we were to change it would be to mdbox, 
not maildir.
> I've been reading about the conversion technique to convert to 
> maildir, but I'm wondering if I can do this based on the way our users 
> are set up. Most are pop accounts. Our webmail is configured to read 
> the inbox from /var/spool/mail and their imap folders from /home or 
> /home/mail. Some have imap accounts on their desktop defined through 
> their mail client.

When we did our conversion from uw-imap to dovecot (about 30k accounts) 
we had to do a lot of clean-up work.  Mail folders in the homedir needed 
to be moved to ~/mail.  This can be scripted: if you're using mbox 
format,  look for the first line of the file starting with "From ".  
(note the space)

We use the uw-imap backwards compatibility option documented at 
http://wiki.dovecot.org/Namespaces#Backwards_Compatibility:_UW-IMAP 
which allows users who have specified "mail/" as their IMAP namespace 
prefix to see the same message folders as people who don't.  This works 
most of the time, but we have seen some clients (old versions of 
Horde/IMP) which still try to create a mail/ folder, which ends up 
creating ~/mail/mail, and that becomes inaccessible.  Fortunately, since 
moving to Horde 4/IMP 5 it hasn't been a problem.  You might want to 
keep an eye out for this if you go the backwards compatibility route.

We also kept (and still keep) inboxes in /var/spool/mail (which is 
actually a link tree, pointing to other filesystems.)  We had enough 
users that we needed to do "shard" directories to prevent directory
lock
contention from bringing everything to a crawl, e.g. jsmith's mail is in 
/var/spool/mail/j/s/jsmith.  Dovecot nicely supports this as a config 
option.

>
> RH/Centos seems to want me to switch to postfix since most of the 
> documentation I find doesn't mention Sendmail. My filesystem is laid 
> out to handle Inboxes in /var/spool/mail and imap folders in 
> /home(/mail) and these aren't logical volumes but true partitions, so 
> I've got to consider this as I make any changes.

We still use sendmail here, which calls procmail for delivery, mostly 
for historical reasons - many users have complex .procmailrc files.  If 
I had to start over I'd certainly give postfix a shot.
>
> I also run pop using dovecot, so I'll need to figure all this out.

Make sure you've read http://wiki.dovecot.org/Migration/UW , 
particularly for the pop3_uidl_format option, so clients do not 
re-download everything when you switch from UW -> Dovecot.

Cheers,
Jim Lawson

Steve Campbell wrote:
> The more I read about all of this, the more I'm thinking about moving
to
> maildir format. My switchover this weekend is full of holes due to the 
> way user's imap folders were laid out. Some had folders in their home 
> directory and others might have folders in their /home/mail directory.
I had the same problem, and handled it more or less the same way as
the later poster (Jim Lawson <jtl+dovecot at uvm.edu>) handled his site.
I retained use of mbox format as well.

I planned the migration like this

 	- scan user home directories for mailboxes (especially "Trash",
 	"Drafts", "Sent", etc.) looking for "From " as
the first 5 bytes.

 	This was piped into a script that Email'd users about the changes
 	that was going to happen, what they would expect to see, and a
 	FAQ on how to set up a mail client correctly.

 	- during the cutover,

 		- mailboxes left on the home directory were moved to
 		~/mail, or renamed (e.g. "Sent" -> "00Sent" to avoid
 		name collision for users that had a mixture of correct
 		and blank prefixes.

 		(I think I deleted "Trash", "Junk", etc. anyways).

 		- Namespace aliasing was used so that prefixes "",
"mail",
 		"~/mail", and whatever darn fool settings my users used,
 		would map to the same directory.

 		- .subscription files were moved into the mail folder
 		(don't have to edit prefixes since the aliasing
 		will take care of that).

 		- depending on what POP3 client you used (I used
 		qpopper), you may need to configure

 			pop3_reuse_xuidl = yes

 		to avoid a massive re-downloading from POP3 clients
 		after cutover.

 		- the mail clients I control centrally (e.g. webmail,
 		public server mail clients, etc). shouldn't need updating
 		since they ought to have been set up properly in the first
 		place.

 	- after cutover, a second notification was sent for users that
 	didn't move their mailboxes the first time around, and was it
 	done for them during migration.

And of course, test like crazy and watch the logs like a hawk.

Other gotcha's:

 	- your setup is fairly close to mine, so you may also run into
 	the problem of user having mailboxes with group ownership that
 	users are not part of (for example, group "mail" for INBOX set
 	by your LDA or personal mailboxes with groups the user is no
 	longer a member of) *and* with group permission not mode=0.

 	You'll have to treat these (set mode=0, or change the group to
 	something the user is part of), or the dovecot index creation
 	will fail and they won't be able to access their mailboxes.

I think my migration went pretty smoothly.  Less than a handful of wazzup'
Email problem reports.

Joseph Tam <jtam.home at gmail.com>

On 2/20/2012 4:58 PM, Jim Lawson wrote:
> On 2/20/12 3:36 PM, Steve Campbell wrote:
>>
>> Thanks for that input. I still think I'm missing something since I 
>> too used the compatibility link that you pointed to. Only thing is 
>> that proceeding those namespaces, I used the first example of:
>>
>> namespace {
>>   type = private
>>   separator = /
>>   prefix = "#mbox/"
>>   location = mbox:~/mail:INBOX=/var/mail/%u
>>   inbox = yes
>>   hidden = yes
>>   list = no
>> }
>> namespace {
>>   type = private
>>   separator = /
>>   prefix >>   location = maildir:~/mail
>> }
>>
>> This seemed to get the ball rolling so that users could access 
>> anything at all. I still have the problem of client imap folders 
>> being different from webmail imap folders. That's pretty much why
I'm
>> thinking of using mbox as the INBOX and all other imap folders in the 
>> ~/mail directory in maildir format.
> You ought to be able to get webmail to see the same folders as the 
> imap clients.
>
>>
>> Should I remove the first two namespaces, in your opinion? Right now, 
>> the whole thing is kind of fragile.
>>
>
> From your added namespaces, it looks like you are trying to support 
> mbox and maildir and ~/mail at the same time.  Do you already have 
> maildir folders to support?  If not, I would try to get things working 
> well with mbox first before I started a conversion to maildir.  I 
> would also ask why you're thinking of moving to maildir.  Yes, there 
> are caching benefits, but when you add the Dovecot indexing on top of 
> mbox, it's pretty much a wash.  If you are using file-level backup, 
> rather than some sort of snapshot technology, maildir will be much, 
> much slower to back up.  Your system will spend all its time walking 
> directories, opening and closing files. If you don't have many users 
> to worry about it might be OK, but make sure it's worth it.  A lot of 
> sites went to maildir in the 1998-2004 era and have regretted the 
> decision as their systems get overloaded with files and they can't 
> back them up.
No, I've got all mbox formats. Previous posts probably suggested that I 
wanted to move to maildir, but all of the replies I've received have 
convinced me that I do not want to do that. The folders in ~ and ~/mail 
are mboxes, so I need to see what damage I've done with a maildir 
namespace. The maildir reference could be part of the problems I'm seeing.

I'm still not sure whether I should be seeing .subscriptions or 
mail/subscriptions anywhere and whether Dovecot will use the 
.mailboxlist that exists. The wiki suggests that I need to recompile 
Dovecot to continue using .mailboxlist. This is something I don't want 
to do.
>
>> Horde/Imp updates are probably out of the question until I can get a 
>> server to install the upgrade on.
> For Dovecot and IMP both, you should set up an alternate server to 
> test out your config changes on before you put them into production.  
> If you are running on a bare metal single server, set up another 
> Dovecot instance on a different set of ports (I commonly use 20143 
> (imap), 20993 (imaps), 20110 (pop3)) which you can fiddle with 
> freely.  Once you are satisfied with the result in your various 
> clients, put it into production.  You can do the same with Horde/IMP 
> by putting an installation in a different location on your webserver.
>
> Jim
I've got a second server that is totally independent of the one I'm 
messing with. I've done the horde/imp alternate port/config. It all went 
well. The downside is that I didn't realize those secondary folders in 
~/mail were being hidden. Kinda late to switch back, but the secondary 
server supports another domain, so I have it to test with.

Again, the damage I've caused to those secondary imap folders needs to 
be determined to get this fixed properly. Most users are seing the 
secondary folders and can use them. It's just those users who have 
folders in ~ that are seeing problems as far as I can tell.

Thanks

steve
>
>

Steve Campbell <campbell at cnpapers.com> writes:
> >     This was piped into a script that Email'd users about the
changes
> >     that was going to happen, what they would expect to see, and a
> >     FAQ on how to set up a mail client correctly.
> 
> Here, I'm not sure what should be done. The users with the secondary 
> folders that are not in ~/mail can't seem to get the client configured.
"Great artists steal".  Google around for a good set of instructions
on how to set up IMAP prefix's for various mail clients, and refer
your users to that.

I also referred them to our webmail (with the correct mail prefix) as a
reference to a list of mailboxes they ought to see.  If they don't match,
then that's a symptom of a bad prefix.
> I've tried this and modified the .mailboxlist, but I'm thinking
Dovecot
> is ignoring this and I'm not sure what it's looking at to determine
the
> imap folders.
>
> >         - .subscription files were moved into the mail folder
> >         (don't have to edit prefixes since the aliasing
> >         will take care of that).
> 
> And when would these .subscription files be created? The first time the 
> folders would be accessed, or when? This seems to be part of my fix that 
> I'm getting lost on.
Sorry, as Timo intimated, dovecot uses .subscription, which can
be adjusted back to .mailboxlist, which uw-imapd uses.  You shouldn't
have to edit it if you set up namespace aliasing.  Just move it from
the top-level into your user's mail directory during cutover:

 	(if you use dovecot's default subscription filename)
 	mv $USER/.mailboxlist $USER/mail/.subscription

If your IMAP client is still ignoring subscriptions, it usually
means the client was configured that way.
> >     - your setup is fairly close to mine, so you may also run into
> >     the problem of user having mailboxes with group ownership that
> >     users are not part of (for example, group "mail" for
INBOX set
> >     by your LDA or personal mailboxes with groups the user is no
> >     longer a member of) *and* with group permission not mode=0.
> 
> Sendmail sort of requires the "mail" group, does it not? I'll
take a
> look and see if all users are part of this or not. A crazy solution 
> would be in order here?
I think sendmail works fine if you set your mail spool mode=1777 (like
/tmp), but have each inbox mode=0600.  This setting will persist.

It's sendmail LDA mail.local that's the problem: it autocreates new
inboxes with permissions user:mail/660 and unless the user is in group
"mail", chaos will ensue.

You can maybe do a cron script or some other hack, or better yet,
use dovecot LDA, which sets up the perms correctly (and keeps indexes
updated).  A one time "chmod go-rwx" during cutover is all that is
required then.

The log entry that points this problem out looks like

 	dovecot: imap(smith): Error: chown(/dc-cache/smith/.imap/xx, group=6(mail))
failed: Operation not permitted (egid=5678(goodgroup), group based on
/home/smith/mail/xx - see http://wiki2.dovecot.org/Errors/ChgrpNoPerm)
 	dovecot: imap(smith): Error: mkdir(/dc-cache/smith/.imap/xx) failed: Not owner

Joseph Tam <jtam.home at gmail.com>