dovecot - Aug 2015 - multiple messages per second to a single mailbox

On Aug 12, 2015, at 11:58 AM, Daniel Tr?der <troeder at univention.de>
wrote:
> On 08/12/2015 17:19, Chad M Stewart wrote:
>> What I'm seeing is very high load on the system (40) and queues
building on the Postfix side.
> High load means, that there are a lot of processes waiting to run. The
> most likely cause for this is not CPU consumption, but I/O wait.
> 
> Please run vmstat and iostat and post their output.
I was watching iostat and avg service times, and maybe once every 30-45 seconds
I'd see times of 10ms, but otherwise it was below that.   I achieved the
biggest impact by limiting the number of outbound connections from Postfix to
Dovecot.  I limited Postfix to 5 connections, which means a total of 10 inbound
LMTP to Dovecot.  Then I saw near 500 msgs per LMTP connection.

I suspect the problem was a locking issue on the mailbox in question.  Too many
simultaneous delivery attempts via too many LMTP sessions.

The backlog has cleared so I'm done troubleshooting for now.  If this
happens again I'll resume looking into it more.  These are new servers so
I'm tuning for the load, etc..


-Chad

The problem happened again this morning.  Removing fsync calls helped, but
I'm not sure about leaving that enabled long term.

I still believe the problem is multiple dovecot processes trying to write to a
single folder at the same time.  (If I could run dtrace I might be able to
cobble together a script to prove it.)

I tried writing a sieve script to direct the messages to a set of folders, but
I'm not able to make the logic work. I was thinking something like: generate
random # ($N) between say 1-10, then file message into folder mail$N.  But I
didn't find a method to do that within sieve.

My next thought was to try parsing the message-id header.  If the first
character is [0-5] then fileinto mail1, etc.  Then I could go so far as having
36 subfolders which the messages could be written too.  This mailbox only keeps
messages for a rolling 1d window.  Right now for example it has 260,186 messages
in the INBOX.

The sieve script I tried (with only about 4 hours of sleep) was

require ["fileinto","regex"];

if header :regex "message-id" "^1" {
   fileinto "mail1";
} else {
   keep;
}


If anyone has some suggestions on how I might spread the messages out over
multiple folders I'd like to hear your thoughts.  Again the servers are
configured using maildir, so each folder should have its own index and thus file
locking contention should be less, at least so goes the theory in my head.


Thank you,
Chad

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
 
On 2015-08-14 7:52 AM, Chad M Stewart wrote:
>
> The problem happened again this morning.  Removing fsync calls helped,
but I'm not sure about leaving that enabled long
term.
>
> I still believe the problem is multiple dovecot processes trying to
write to a single folder at the same time.  (If I could run dtrace I
might be able to cobble together a script to prove it.)
>
> I tried writing a sieve script to direct the messages to a set of
folders, but I'm not able to make the logic work. I was thinking
something like: generate random # ($N) between say 1-10, then file
message into folder mail$N.  But I didn't find a method to do that
within sieve.  
>
> My next thought was to try parsing the message-id header.  If the
first character is [0-5] then fileinto mail1, etc.  Then I could go so
far as having 36 subfolders which the messages could be written too. 
This mailbox only keeps messages for a rolling 1d window.  Right now for
example it has 260,186 messages in the INBOX.
>
> The sieve script I tried (with only about 4 hours of sleep) was
>
> require ["fileinto","regex"];
>
> if header :regex "message-id" "^1" {
>    fileinto "mail1";
> } else {
>    keep;
> }
>
>
> If anyone has some suggestions on how I might spread the messages out
over multiple folders I'd like to hear your thoughts.  Again the servers
are configured using maildir, so each folder should have its own index
and thus file locking contention should be less, at least so goes the
theory in my head.
>
>
> Thank you,
> Chad
>
With that many messages, it may be the index updates that are slowing
things down. Depending on your needs or how you use this folder (it
sounds automated to me), you may not need the indexes on disk at all. If
you set the indexes to RAM, it may help.

mail_location = maildir:~/Maildir:INDEX=MEMORY

as per http://wiki2.dovecot.org/MailLocation#Index_files

If indexes in RAM isn't an option, maybe just put them on a separate
spindle / SSD would help.

Please note I'm shooting from the hip with this advice, I don't have a
system that pushes dovecot that hard.

Greg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
 
iQIcBAEBCAAGBQJV0e/oAAoJECD7Htp+IT9eUXsQALgtSocaTUXWjMzp1oIEZTsT
IOBzc25q4LZtNWeQpa/czmc9TBQfmqcRzXspyqu/dTxdesJ05WL3PiD/c66Bsmb8
0652a+6l7rLgMroIgsYCQZNBaoU+7FgIakWQlyHA/Ods7DTFNNRvAVNhZ6e6NcvR
pkIM5EtLaHzc7318ahR2mC4tRHbmlSxZT/fnWuCu185VyuG6GiHum0piyplEIRqw
ngbHkMROgD/OJTKScngRP73umLznxOfReVrM50tIlgpONY+ngTLvD6rnkbLO1t1Y
KdVAc49iEQPTRuRVDDS+kf2haUx9Rh8Zz0R4VRdoiBC9bcRmeuQDNU4mg4fBCgQF
JlkIkuYBvZe/y7TcxJ5rJXNtEn6g4Uew7mIt+sMnGx7Ro55kl0X61Qg/IWu+F8m5
aO9Y7jMjptsjm/04YWlaeCNUPolgUQrFLDjiO5uycEZh1NReNZEH5AFM0cYWMhBK
PWzDRXwv4XEypNA+y+uDLdN3mX/KcWMUv90ipJT5PdrLPjvgjTbyU/Wr391H01Kf
85ShLVvBsV2NGvXY0ZRAxuF5Ndq04TNrRYDvVgzb4XlCH++l2WeZJ2CZ8b1KL5AC
BydVyp5tWoP5TIzelG6NcMcqNjD05rtmcH2QCT1BLMksY9+7YkbU6IX+W5Dc5sOX
krjSPNAsLOFJJCxBIZ/S
=AYpS
-----END PGP SIGNATURE-----