dovecot - Sep 2008 - No buffer space available

Hi,

I'm running Dovecot 1.1.3 on FreeBSD 7.0.    In the past couple days, 
I've been seeing errors like this:

Sep  9 08:43:51 sysvol dovecot: pop3-login: socketpair() failed: No 
buffer space available
Sep  9 08:43:51 sysvol dovecot: pop3-login: socket(default) failed: No 
buffer space available
Sep  9 08:43:51 sysvol dovecot: pop3-login: Can't connect to auth server 
at default: No buffer space available
Sep  9 08:25:15 sysvol dovecot: pop3-login: socketpair() failed: No 
buffer space available
 pop3-login: Can't connect to auth server at default: No buffer space 
available

What ends up happening is that the port doesn't accept any new 
connections until there is enough buffer space.

Yesterday, I saw it a LOT with POP3 and changed my config login processes:
login_process_per_connection: no
login_process_size: 256
login_processes_count: 6
This seemed to clear up the issues, but now it's occurring again this 
morning.

I haven't had any of these since I made that change either.. So there's 
some progress... :
Sep  8 12:53:40 sysvol dovecot: Temporary failure in creating login 
processes, slowing down for now


TOP shows:
Mem: 267M Active, 5558M Inact, 1637M Wired, 217M Cache, 214M Buf, 87M Free
Swap: 4096M Total, 4096M Free

At this time, there are 120 imap processes, 10 POP and 1 dovecot and 1 
dovecot-auth
pop and imap both have 6 login processes.  
It doesn't seem like a load issue, but obviously something is set too 
low or unavailable... 

It's fixed for a period of time when I restart dovecot.  I did a netstat 
-m prior to restart, and after restart, and the only main difference was 
this line:
Before:
11/414/425/12800 4k (page size) jumbo clusters in use 
(current/cache/total/max)
After:
0/425/425/12800 4k (page size) jumbo clusters in use 
(current/cache/total/max)
(After about 5 minutes 'current' went to 3)

This machine also runs NFSd and MySQL.  

Any ideas?

Thanks,
Rick

I forgot to add that I also upgraded from 1.1.1 to 1.1.3 yesterday in  
an attempt to fix the problem.


On Sep 9, 2008, at 8:58 AM, Rick Romero <rick at havokmon.com> wrote:
> Hi,
>
> I'm running Dovecot 1.1.3 on FreeBSD 7.0.    In the past couple  
> days, I've been seeing errors like this:
>
> Sep  9 08:43:51 sysvol dovecot: pop3-login: socketpair() failed: No  
> buffer space available
> Sep  9 08:43:51 sysvol dovecot: pop3-login: socket(default) failed:  
> No buffer space available
> Sep  9 08:43:51 sysvol dovecot: pop3-login: Can't connect to auth  
> server at default: No buffer space available
> Sep  9 08:25:15 sysvol dovecot: pop3-login: socketpair() failed: No  
> buffer space available
> pop3-login: Can't connect to auth server at default: No buffer space  
> available
>
> What ends up happening is that the port doesn't accept any new  
> connections until there is enough buffer space.
>
> Yesterday, I saw it a LOT with POP3 and changed my config login  
> processes:
> login_process_per_connection: no
> login_process_size: 256
> login_processes_count: 6
> This seemed to clear up the issues, but now it's occurring again  
> this morning.
>
> I haven't had any of these since I made that change either.. So  
> there's some progress... :
> Sep  8 12:53:40 sysvol dovecot: Temporary failure in creating login  
> processes, slowing down for now
>
>
> TOP shows:
> Mem: 267M Active, 5558M Inact, 1637M Wired, 217M Cache, 214M Buf,  
> 87M Free
> Swap: 4096M Total, 4096M Free
>
> At this time, there are 120 imap processes, 10 POP and 1 dovecot and  
> 1 dovecot-auth
> pop and imap both have 6 login processes.  It doesn't seem like a  
> load issue, but obviously something is set too low or unavailable...
> It's fixed for a period of time when I restart dovecot.  I did a  
> netstat -m prior to restart, and after restart, and the only main  
> difference was this line:
> Before:
> 11/414/425/12800 4k (page size) jumbo clusters in use (current/cache/ 
> total/max)
> After:
> 0/425/425/12800 4k (page size) jumbo clusters in use (current/cache/ 
> total/max)
> (After about 5 minutes 'current' went to 3)
>
> This machine also runs NFSd and MySQL.
> Any ideas?
>
> Thanks,
> Rick

On Tue, 2008-09-09 at 08:58 -0500, Rick Romero wrote:
> Hi,
> 
> I'm running Dovecot 1.1.3 on FreeBSD 7.0.    In the past couple days, 
> I've been seeing errors like this:
> 
> Sep  9 08:43:51 sysvol dovecot: pop3-login: socketpair() failed: No 
> buffer space available
Only thing I can suggest is to google for "ENOBUFS socketpair freebsd"
or to ask in FreeBSD lists. I've no idea why it would return ENOBUFS.
The manual page doesn't say ENOBUFS is a possible error for
socketpair().

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080909/f131cd8b/attachment-0002.bin>

Timo Sirainen wrote:
> On Tue, 2008-09-09 at 08:58 -0500, Rick Romero wrote:
>   
>> Hi,
>>
>> I'm running Dovecot 1.1.3 on FreeBSD 7.0.    In the past couple
days,
>> I've been seeing errors like this:
>>
>> Sep  9 08:43:51 sysvol dovecot: pop3-login: socketpair() failed: No 
>> buffer space available
>>     
>
> Only thing I can suggest is to google for "ENOBUFS socketpair
freebsd"
> or to ask in FreeBSD lists. I've no idea why it would return ENOBUFS.
> The manual page doesn't say ENOBUFS is a possible error for
> socketpair().
>
>   
Got it.  Thanks Timo, your search terms were better than mine.

sysvol# vmstat -z | grep sock
ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  
FAILURES
socket:                   696,   204800,    12362,      128, 
82179541,    60873

My old kern.ipc.maxsockets: 12328      I was running out of Sockets.  
The error was popping up multiple times per minute, now I haven't seen 
one in 5 minutes.
I also found a great FreeBSD connection optimization page:
http://rerepi.wordpress.com/2008/04/19/tuning-freebsd-sysoev-rit/

Rick