dovecot - Apr 2012 - Outlook (2010) -> Dovecot (IMAP) >10x slower with high network load and many folders

Hi,

I am seeing a >10x as slow performance when trying to complete a
"send/receive" from an Outlook 2010 client to Dovecot via IMAP, but
only when the LAN is fully loaded with other traffic, EG file copying.
It seems the problem is when outlook is trying to identify folders
that have changed since last "send/receive" thus traversing the
hierachy.

Observations:
- Apple's Mail.app does not have problems when exposed to the same
environment.
- Eliminating Outlook is not an option.
- The tests have been performed during the easter vacation, thus
almost no other clients are using the network/mailserver. Under normal
load there are 2-300 users using the network.


Description of the environment:
Server sw: Mac OS X Server 10.6.8 running dovecot: 1.1.20apple0.5 (OS
= fully updated 10.6.x)
Server hw: Xserve Quadcore intel Xeon 2.26 GHz 12 GB RAM
- Never having a cpu load > 20%, mail is stored on a Promise vtrack
RAID connected via Fibre Channel

Client: Windows 7, Outlook 2010
The client have maybe 50 folders (and each folder about a handfull of
subfolders) the size of the mailbox is around 3-5 GB.


Test results:
CLIENT-1 is having the problems when CLIENT-2 is using all the
(100Mbps) bandwidth eg. copying files to MAIL-SRV.
If I move CLIENT-1 to CLIENT-3 then almost all the delay is gone.
NB.: I have not (yet) tested if the problem also exists when CLIENT-2
generates traffic to MAIL-SRV as opposed to OTHER-SRV (but I am
expecting the same problems).


When dumping the traffic on CLIENT-1 (with Wireshark) one thing is
catching my eyes for the 'bad' case:
There is a very long delay after each "REQUEST IDLE" until the next
"REQUEST DONE" ca. 0,3 seconds.
Comparing to the 'good' setup the pause at the same place in the
communication is less than 1 ms !

So why this delay?
Where/how shall I continue my debugging?
- Run dtruss/dtrace scripts on the server?
- Get dovecot to output more debug info (I guess it's doing it's best
so no problems will be seen there?)?

Are there any dovecot configurations that can be altered to workaround
[what I think is a limitation in Outlook's IMAP implementation]?
The current dovecot configuration is Apple's defaults (+POP3 disabled))

I am seing one warning from dovecotd -n though:
- But I see the same warning on a clean installed not-yet-configured
OS X Server so I guess it's not 'that' bad:
Warning: fd limit 256 is lower than what Dovecot can use under full
load (more than 456). Either grow the limit or change
login_max_processes_count and max_mail_processes settings



Physical setup:
+---------------+ ?+------------------------------+ ? ?+-----------------+
| CLIENT-1 ? ? ?| ?| CLIENT-2 ? ? ? ? ? ? ? ? ? ? | ? ?| CLIENT-3 ? ? ? ?|
| Outlook "bad" | ?| Traffic generator, eg. Samba | ? ?| Outlook
?"good" |
+---------------+ ?+------------------------------+ ? ?+-----------------+
? ? ? ? | ? ? ? ? ? ? ? ? ? ? ? | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?|
+-------------------------------------------------+ ? ? ? ? ? ? |
| ? ? ? ? ? 100Mbps switch (clients) ? ? ? ? ? ? ?| ? ? ? ? ? ? |
+-------------------------------------------------+ ? ? ? ? ? ? |
? ? ? ? ? ? ? ? ? ? ?| ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? |
+------------------------------------------------------------------------+
| ? ? ? ? ? ? ? ? ? ? 1Gbps switch (for servers) ? ? ? ? ? ? ? ? ? ? ? ? |
+------------------------------------------------------------------------+
? ? ? ? ? ? ? | ? ? ? ? ? ? ? ? ? ? ? ? |
+-----------------------------+ ? ?+-----------+
| MAIL-SRV: Dovecot, AFP, SMB | ? ?| OTHER-SRV |
+-----------------------------+ ? ?+-----------+




Wireshark dump - IMAP communication:
|Time ? ? | 10.211.55.3 ? ? ? ? ? ? ? ? ? ? ? ? ? |
| ? ? ? ? | ? ? ? ? ? ? ? ? ? | 10.0.0.10 ? ? ? ? |
|9.851 ? ?| ? ? ? ? Response: gatq OK L ? ? ? ? ? |IMAP: Response:
gatq OK Logged in.
| ? ? ? ? |(49433) ?<------------------ ?(143) ? ?|
|9.852 ? ?| ? ? ? ? Request: o47u SELEC ? ? ? ? ? |IMAP: Request: o47u
SELECT "1_GROUPS"
| ? ? ? ? |(49433) ?------------------> ?(143) ? ?|
|9.853 ? ?| ? ? ? ? Response: * FLAGS ( ? ? ? ? ? |IMAP: Response: *
FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
| ? ? ? ? |(49433) ?<------------------ ?(143) ? ?|
|9.854 ? ?| ? ? ? ? Request: 3y4b NOOP ? ? ? ? ? ?|IMAP: Request: 3y4b NOOP
| ? ? ? ? |(49433) ?------------------> ?(143) ? ?|
|9.854 ? ?| ? ? ? ? Response: 3y4b OK N ? ? ? ? ? |IMAP: Response:
3y4b OK NOOP completed.
| ? ? ? ? |(49433) ?<------------------ ?(143) ? ?|
|9.855 ? ?| ? ? ? ? Request: 4vlj IDLE ? ? ? ? ? ?|IMAP: Request: 4vlj IDLE
| ? ? ? ? |(49433) ?------------------> ?(143) ? ?|
|9.856 ? ?| ? ? ? ? Response: + idling ? ? ? ? ? ?|IMAP: Response: + idling
| ? ? ? ? |(49433) ?<------------------ ?(143) ? ?|
|10.108 ? | ? ? ? ? Request: DONE ? ? ? ? ? ? ? ? |IMAP: Request: DONE
| ? ? ? ? |(49433) ?------------------> ?(143) ? ?|
|10.108 ? | ? ? ? ? Response: 4vlj OK I ? ? ? ? ? |IMAP: Response:
4vlj OK Idle completed.
| ? ? ? ? |(49433) ?<------------------ ?(143) ? ?|
|10.108 ? | ? ? ? ? Request: wh89 SELEC ? ? ? ? ? |IMAP: Request: wh89
SELECT "1_GROUPS.Adm"


BR Thomas von Eyben

It seems that you have packets lost in the network. MAC and Windows have 
different network stacks so this may count for the different behavior.

-- 
Best regards,
Adrian Minta

On 4/6/2012 3:52 AM, Thomas von Eyben wrote:
> Test results:
> CLIENT-1 is having the problems when CLIENT-2 is using all the
> (100Mbps) bandwidth eg. copying files to MAIL-SRV.
> If I move CLIENT-1 to CLIENT-3 then almost all the delay is gone.
> NB.: I have not (yet) tested if the problem also exists when CLIENT-2
> generates traffic to MAIL-SRV as opposed to OTHER-SRV (but I am
> expecting the same problems).
So the link between your 100 Mbps switch and the 1 Gbps switch is 
saturated by CLIENT-2, so CLIENT-1 is just getting the leftovers?

Since CLIENT-3 doesn't go through that 100 Mbps switch, it obviously 
doesn't see that issue.

On 2012-04-06, Thomas von Eyben <thomasvoneyben at gmail.com>
wrote:
> I am seeing a >10x as slow performance when trying to complete a
> "send/receive" from an Outlook 2010 client to Dovecot via IMAP,
but
> only when the LAN is fully loaded with other traffic, EG file copying.
> It seems the problem is when outlook is trying to identify folders
> that have changed since last "send/receive" thus traversing the
> hierachy.
Not sure why it would only affect Outlook clients, but if your
switches are managed, you might like to check if flow control is
enabled and, if so, try disabling it.