So after several days of more troubleshooting, I have some things to
report to the list.
First and foremost, I have discovered that the issue has nothing to do
with SSL/TLS, which was my earlier suspicion because after doing some
PCAPs I discovered that the transactions were negotiating TLS 1.2 on the
new server, as opposed to 1.0 on the old.
Also thank you for the rawlog suggestion: that helped a lot in
determining what was happening on the IMAP level.
That all said, this is what I've discovered:
There is a very curious and reproducible four-second delay during the
negotiation between server and client which is not present in Dovecot
2.1. This is what our customer is complaining about using Outlook 2010.
During a plaintext TCP stream, I'm seeing this:
1. Client connects (SYN) to server.
2. Server ACKs and throws back CAPABILITIES.
3. User attempts to auth with DIGEST-MD5.
4. Server says, "no thanks." (Not sure why, but I don't believe
this is
relevant.)
5. User attempts to auth with plaintext.
6. Server says, "Yup. You are you. You're logged in."
7. Client sends the following: ID ("name" "Microsoft
Outlook" "version"
"14.0")
8. Server sends an ACK
... and then there's this very curious four-second delay.
9. Server then sends out new CAPABILITIES, and everything proceeds
thereafter as normal and zippy and fast.
Does this shed any light on the subject?
On 10/13/16 11:21 AM, Bryan Holloway wrote:> On 10/13/16 11:01 AM, Aki Tuomi wrote:
>>
>>> On October 13, 2016 at 6:52 PM Konstantin Khomoutov
>>> <flatworm at users.sourceforge.net> wrote:
>>>
>>>
>>> On Thu, 13 Oct 2016 10:35:14 -0500
>>> Bryan Holloway <bryan at shout.net> wrote:
>>>
>>>>> [...]
>>>>>> Is there a way to see the IMAP commands coming from the
client?
>>>>>> I've tried looking at PCAPs, but of course
they're encrypted so I
>>>>>> can't see the actual dialog going on between the
server and
>>>>>> client. I didn't see an obvious way to do this in
the docs.
>>>>>
>>>>> If you have access to the SSL/TLS key (IOW, the private
part of the
>>>>> cert) the server uses to secure IMAP connections you can
dump the
>>>>> IMAP traffic using the `ssldump` utility (which builds on
>>>>> `tcpdump`).
>>>>
>>>> I do, but the client is using a DH key exchange so I only have
the
>>>> server-side private key.
>>>>
>>>> Tried that using Wireshark's decoder features and ran into
this
>>>> problem. I'm assuming I'd run into the same using
ssldump, but I'll
>>>> give it a shot!
>>>
>>> I think DH is not the culprit: just to be able to actually decode
SSL
>>> traffic, you must have the server private key when you're
decoding the
>>> SSL handshake phase -- to be able to recover the session keys,
which
>>> you then use to decode the actual tunneled data.
>>
>> You can also enable only non DH algorithms in ssl settings if rawlog
>> isn't working for you.
>>
>> Aki
>>
>
> Ah -- interesting tip. I hadn't thought of that. Thank you! I'll
report
> my findings to the list.
In case anyone is interested, we finally found the problem: The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled along with "plain". This is what was causing the four-second delay, but only with Outlook clients. Everything is working great now across the board. Thanks again to everyone's suggestions. - bryan On 10/27/16 7:09 PM, Bryan Holloway wrote:> So after several days of more troubleshooting, I have some things to > report to the list. > > First and foremost, I have discovered that the issue has nothing to do > with SSL/TLS, which was my earlier suspicion because after doing some > PCAPs I discovered that the transactions were negotiating TLS 1.2 on the > new server, as opposed to 1.0 on the old. > > Also thank you for the rawlog suggestion: that helped a lot in > determining what was happening on the IMAP level. > > That all said, this is what I've discovered: > > There is a very curious and reproducible four-second delay during the > negotiation between server and client which is not present in Dovecot > 2.1. This is what our customer is complaining about using Outlook 2010. > > During a plaintext TCP stream, I'm seeing this: > > 1. Client connects (SYN) to server. > > 2. Server ACKs and throws back CAPABILITIES. > > 3. User attempts to auth with DIGEST-MD5. > > 4. Server says, "no thanks." (Not sure why, but I don't believe this is > relevant.) > > 5. User attempts to auth with plaintext. > > 6. Server says, "Yup. You are you. You're logged in." > > 7. Client sends the following: ID ("name" "Microsoft Outlook" "version" > "14.0") > > 8. Server sends an ACK > > ... and then there's this very curious four-second delay. > > 9. Server then sends out new CAPABILITIES, and everything proceeds > thereafter as normal and zippy and fast. > > Does this shed any light on the subject? > > > On 10/13/16 11:21 AM, Bryan Holloway wrote: >> On 10/13/16 11:01 AM, Aki Tuomi wrote: >>> >>>> On October 13, 2016 at 6:52 PM Konstantin Khomoutov >>>> <flatworm at users.sourceforge.net> wrote: >>>> >>>> >>>> On Thu, 13 Oct 2016 10:35:14 -0500 >>>> Bryan Holloway <bryan at shout.net> wrote: >>>> >>>>>> [...] >>>>>>> Is there a way to see the IMAP commands coming from the client? >>>>>>> I've tried looking at PCAPs, but of course they're encrypted so I >>>>>>> can't see the actual dialog going on between the server and >>>>>>> client. I didn't see an obvious way to do this in the docs. >>>>>> >>>>>> If you have access to the SSL/TLS key (IOW, the private part of the >>>>>> cert) the server uses to secure IMAP connections you can dump the >>>>>> IMAP traffic using the `ssldump` utility (which builds on >>>>>> `tcpdump`). >>>>> >>>>> I do, but the client is using a DH key exchange so I only have the >>>>> server-side private key. >>>>> >>>>> Tried that using Wireshark's decoder features and ran into this >>>>> problem. I'm assuming I'd run into the same using ssldump, but I'll >>>>> give it a shot! >>>> >>>> I think DH is not the culprit: just to be able to actually decode SSL >>>> traffic, you must have the server private key when you're decoding the >>>> SSL handshake phase -- to be able to recover the session keys, which >>>> you then use to decode the actual tunneled data. >>> >>> You can also enable only non DH algorithms in ssl settings if rawlog >>> isn't working for you. >>> >>> Aki >>> >> >> Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report >> my findings to the list.
On 11/1/16 6:20 PM, Bryan Holloway wrote:> In case anyone is interested, we finally found the problem: > > The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled along > with "plain". This is what was causing the four-second delay, but only > with Outlook clients. > > Everything is working great now across the board. > > Thanks again to everyone's suggestions. > > - bryan >Sorry -- that wasn't very clear, was it: Removing "digest-md5" fixed the issue.> On 10/27/16 7:09 PM, Bryan Holloway wrote: >> So after several days of more troubleshooting, I have some things to >> report to the list. >> >> First and foremost, I have discovered that the issue has nothing to do >> with SSL/TLS, which was my earlier suspicion because after doing some >> PCAPs I discovered that the transactions were negotiating TLS 1.2 on the >> new server, as opposed to 1.0 on the old. >> >> Also thank you for the rawlog suggestion: that helped a lot in >> determining what was happening on the IMAP level. >> >> That all said, this is what I've discovered: >> >> There is a very curious and reproducible four-second delay during the >> negotiation between server and client which is not present in Dovecot >> 2.1. This is what our customer is complaining about using Outlook 2010. >> >> During a plaintext TCP stream, I'm seeing this: >> >> 1. Client connects (SYN) to server. >> >> 2. Server ACKs and throws back CAPABILITIES. >> >> 3. User attempts to auth with DIGEST-MD5. >> >> 4. Server says, "no thanks." (Not sure why, but I don't believe this is >> relevant.) >> >> 5. User attempts to auth with plaintext. >> >> 6. Server says, "Yup. You are you. You're logged in." >> >> 7. Client sends the following: ID ("name" "Microsoft Outlook" "version" >> "14.0") >> >> 8. Server sends an ACK >> >> ... and then there's this very curious four-second delay. >> >> 9. Server then sends out new CAPABILITIES, and everything proceeds >> thereafter as normal and zippy and fast. >> >> Does this shed any light on the subject? >> >> >> On 10/13/16 11:21 AM, Bryan Holloway wrote: >>> On 10/13/16 11:01 AM, Aki Tuomi wrote: >>>> >>>>> On October 13, 2016 at 6:52 PM Konstantin Khomoutov >>>>> <flatworm at users.sourceforge.net> wrote: >>>>> >>>>> >>>>> On Thu, 13 Oct 2016 10:35:14 -0500 >>>>> Bryan Holloway <bryan at shout.net> wrote: >>>>> >>>>>>> [...] >>>>>>>> Is there a way to see the IMAP commands coming from the client? >>>>>>>> I've tried looking at PCAPs, but of course they're encrypted so I >>>>>>>> can't see the actual dialog going on between the server and >>>>>>>> client. I didn't see an obvious way to do this in the docs. >>>>>>> >>>>>>> If you have access to the SSL/TLS key (IOW, the private part of the >>>>>>> cert) the server uses to secure IMAP connections you can dump the >>>>>>> IMAP traffic using the `ssldump` utility (which builds on >>>>>>> `tcpdump`). >>>>>> >>>>>> I do, but the client is using a DH key exchange so I only have the >>>>>> server-side private key. >>>>>> >>>>>> Tried that using Wireshark's decoder features and ran into this >>>>>> problem. I'm assuming I'd run into the same using ssldump, but I'll >>>>>> give it a shot! >>>>> >>>>> I think DH is not the culprit: just to be able to actually decode SSL >>>>> traffic, you must have the server private key when you're decoding the >>>>> SSL handshake phase -- to be able to recover the session keys, which >>>>> you then use to decode the actual tunneled data. >>>> >>>> You can also enable only non DH algorithms in ssl settings if rawlog >>>> isn't working for you. >>>> >>>> Aki >>>> >>> >>> Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report >>> my findings to the list.
On Tue, 1 Nov 2016 18:20:14 -0500 Bryan Holloway <bryan at shout.net> wrote:> In case anyone is interested, we finally found the problem: > > The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled > along with "plain". This is what was causing the four-second delay, > but only with Outlook clients. > > Everything is working great now across the board. > > Thanks again to everyone's suggestions.Thanks for sharing. It's pretty amazing how far removed the problem cause can be from that problem's manifestation ;-)