dovecot - Mar 2006 - OS X-centricities? Too many files, temp, indexes, etc...

I'm just writing to see if anyone else is seeing the following issues, 
and particularly if they are running OS X (especially 10.4), and/or have 
compatible solutions (I've been reading the mailing list quite a bit, 
and every now and then, I find a solution that just "doesn't work"
because of the differences in OS:

1) Too many files open  - no plimit command on OS X, so I can't figure 
out how to increase the max number of open files... Also, are there any 
alternative means of CLOSING the files? stopping/restarting dovecot 
didn't seem to help;

2) I was running 1.0beta1 last week, and am now on 1.0beta3, but I'm 
still not seeing corrupt indexes/other dovecot files rebuilding 
themselves. I really can't stick with this solution if I'll always have 
to worry about manually fixing broken caches/indexes.

3) Any advice on fcntl/flock on OS X, particularly? They both appear to 
work, but I'd prefer to use the fastest/least resource intensive options 
for everything if possible.

Thanks!

-deano

On Tue, Mar 21, 2006 at 02:26:00PM -0800, Dean Blackburn
wrote:
> I'm just writing to see if anyone else is seeing the following issues, 
> and particularly if they are running OS X (especially 10.4), and/or have 
> compatible solutions (I've been reading the mailing list quite a bit, 
> and every now and then, I find a solution that just "doesn't
work"
> because of the differences in OS:
> 
> 1) Too many files open  - no plimit command on OS X, so I can't figure 
> out how to increase the max number of open files... Also, are there any 
> alternative means of CLOSING the files? stopping/restarting dovecot 
> didn't seem to help;
You can use limit/ulimit - depending on how you start dovecot.  (If
you use launchd, take a look at 'launchctl limit maxfiles'.)

If you need to change it for the system, you can use sysctl - though
at least on my machine the default seem more than adequate:

[p7:1004] ~%sysctl -a | grep files                                       6:01PM
kern.maxfiles = 12288
kern.maxfilesperproc = 10240

-- 
Nicholas Riley <njriley at uiuc.edu> |
<http://www.uiuc.edu/ph/www/njriley>

On Tue, 2006-03-21 at 14:26 -0800, Dean Blackburn wrote:
> I'm just writing to see if anyone else is seeing the following issues, 
> and particularly if they are running OS X (especially 10.4), and/or have 
> compatible solutions (I've been reading the mailing list quite a bit, 
> and every now and then, I find a solution that just "doesn't
work"
> because of the differences in OS:
> 
> 1) Too many files open  - no plimit command on OS X, so I can't figure 
> out how to increase the max number of open files... Also, are there any 
> alternative means of CLOSING the files? stopping/restarting dovecot 
> didn't seem to help;
Which process is running out of them? There are multiple Dovecot
processes, and each could have a different reason of running out of file
handles.
> 2) I was running 1.0beta1 last week, and am now on 1.0beta3, but I'm 
> still not seeing corrupt indexes/other dovecot files rebuilding 
> themselves. I really can't stick with this solution if I'll always
have
> to worry about manually fixing broken caches/indexes.
First of all they shouldn't anymore be corrupting themselves at least
easily. What exactly are the error messages you're seeing?

Also as far as I know they're rebuilding themselves automatically in
most situations. Only if Dovecot crashes because of the corruption it
doesn't get rebuilt. If it's not crashing with you, could it be just
that they keep re-breaking all the time?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20060325/5e2c6666/attachment.bin>

We're an ISP with approximately 9000 email customers.  Up until early  
in January our primary mail servers ran on OS X, with exim as the MTA  
and courier-imap as the pop/imap server.

We had continual problems with courier on OS X.  The load it  
generates on the machine is horrible.  You'll get absolutely no help  
from the courier mailing list or developers with OS X issues.

When we started evaluating Dovecot, we first attempted it on OS X,  
however we were unable to get it running in an NFS environement on OS  
X in any sort of stable manner.

I can't recommend highly enough getting off of OS X if you can for  
mail services.  We went to Red Hat machines, and they have been  
amazingly stable and reliable.  I love OS X, and use it for other  
things, but it makes a bad mail server at least for the volume of  
mail we process.

--
Roger J. Weeks
Systems & Network Administrator
Mendocino Community Network

On Mar 26, 2006, at 12:24 AM, dovecot-request at dovecot.org wrote:
> Message: 2
> Date: Sat, 25 Mar 2006 09:22:18 -0800
> From: Dean Blackburn <dean.blackburn at viz.com>
> Subject: Re: [Dovecot] OS X-centricities? Too many files, temp,
> 	indexes, etc...
> To: dovecot at dovecot.org
> Message-ID: <44257C4A.6030801 at viz.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Timo Sirainen wrote:
>>> 1) Too many files open  - no plimit command on OS X, so I can't
>>> figure
>>> out how to increase the max number of open files... Also, are  
>>> there any
>>> alternative means of CLOSING the files? stopping/restarting dovecot
>>> didn't seem to help;
>>>
>>
>> Which process is running out of them? There are multiple Dovecot
>> processes, and each could have a different reason of running out  
>> of file
>> handles.
>>
>>
> dovecot: Mar 24 11:35:51 Error: imap(yukimurashige): pipe() failed:  
> Too
> many open files
> dovecot: Mar 24 11:35:51 Error: child 21465 (imap) returned error 89
>
>>> 2) I was running 1.0beta1 last week, and am now on 1.0beta3, but
I'm
>>> still not seeing corrupt indexes/other dovecot files rebuilding
>>> themselves. I really can't stick with this solution if I'll
>>> always have
>>> to worry about manually fixing broken caches/indexes.
>>>
>>
>> First of all they shouldn't anymore be corrupting themselves at
least
>> easily. What exactly are the error messages you're seeing?
>>
>>
> Lots of this:
>
> dovecot: Mar 24 12:18:45 Error: imap(mikitanaka): Corrupted index file
> /Users/mikitanaka/Maildir/dovecot.index: uid_validity = 0, next_uid  
> = 1366
>
> Some of this:
>
> dovecot: Mar 24 10:55:45 Error: imap(kristinegivas): imap(19936)  
> malloc:
> *** error for object 0x602200: incorrect checksum for freed object -
> object was probably modified after being freed, break at  
> szone_error to
> debug
>
> And occasionally this:
>
> dovecot: Mar 24 11:00:14 Error: imap(deanblackburn): Maildir
> /Users/deanblackburn/Maildir sync: UID inserted in the middle of  
> mailbox
> (1042 > 1041, file = 1143071249.Ve000008I8e7f3b40.mailserv.local:2,Sk)
>
> That's just yesterday's errors, we've seen some others as well.
But
> they
> all go away if one deletes dovecot* from the affected directory, and
> they don't go away on their own. Also, we'll see a huge number of  
> temp.*
> files build up within the maildir once these files "go bad"...
>
>  From the client perspective, you either get a "connection timed
out"
> error, or on some less robust clients, "connection refused by IMAP  
> server".
>> Also as far as I know they're rebuilding themselves automatically
in
>> most situations. Only if Dovecot crashes because of the corruption it
>> doesn't get rebuilt. If it's not crashing with you, could it be
just
>> that they keep re-breaking all the time?
>>
> Definitely possible. ;) How can we tell?
>
> If we can solve these issues, we'll be very happy with Dovecot...
> Courier seems a lot more solid, in retrospect, but required so much  
> disk
> access it was completely unusable by our staff. In contrast,  
> Dovecot is
> fast, light on resources, but will break around 5-10 individuals'
> mailboxes every day. So far, we just can't win!
>
> -deano