dovecot - Apr 2016 - Slow reading of large dovecot-uidlist files

On 04/10/2016 10:27 AM, Adrian Minta wrote:
> It will be nice if the "SELECT MyMailbox" command will be
"SELECT
> mymailbox LIMIT 100" or something, to get the first files since only
the
> last messages are shown to the user.
SELECTing a mailbox has nothing to do with FETCHing messages, so above
does not make much sense.

I don't know dovecot's code, but I suppose it uses uidlist file to get
mailbox statistics that it returns as EXISTS, RECENT, UNSEEN, UIDNEXT,
UIDVALIDITY, etc, which are required by IMAP standard. I don't know,
maybe they could be stored in more optimized way, but I think in most
cases this data is also needed for SORT/THREAD/FETCH which is sent after
SELECT in many cases - so it will be needed anyway. There are cases
(e.g. mailbox synchronization) when you indeed do only SELECT.

-- 
Aleksander 'A.L.E.C' Machniak
Kolab Groupware Developer        [http://kolab.org]
Roundcube Webmail Developer  [http://roundcube.net]
---------------------------------------------------
PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl

On 12 April 2016 at 10:23, A.L.E.C <alec at alec.pl> wrote:
> I don't know dovecot's code, but I suppose it uses uidlist file to
get
> mailbox statistics that it returns as EXISTS, RECENT, UNSEEN, UIDNEXT,
> UIDVALIDITY, etc, which are required by IMAP standard. I don't know,
> maybe they could be stored in more optimized way, but I think in most
> cases this data is also needed for SORT/THREAD/FETCH which is sent after
> SELECT in many cases - so it will be needed anyway. There are cases
> (e.g. mailbox synchronization) when you indeed do only SELECT.
>
(Oh, Alec, hi :)

Can someone (who is more intimate with Dovecot's internals) comment on what
dovecot is actually doing, where this time is being spent? That would be
awesome:)

My uidfile is 4MB "large" and reading it takes about 3ms on my system,
so
there is still ~160ms (98%) of dovecot's delay that is unaccounted for.

BTW: I would have thought that the whole process goes like this:
- user logs in, dovecot imap process starts
- "SELECT MyMailbox" is issued for the first time
- dovecot reads uidlist file for that mailbox and caches it along with
timestamp metadata
- on subsequent requests, uidlist file is stat-ed to detect any changes
(changed by i.e. another process that handles same user's connection, from
another device) and trigger rereading when needed, otherwise cache is used
- I would imagine having certain (fairly low) TTL on the cache to prevent
too much memory consumption would be nice, too.

Would that make sense?

(I am just trying to figure out the way how to optimize response time to
SELECT and thus make rendering mailbox listing in Roundcube faster, or,
better, acceptable.)


Best,
b.


>
> --
> Aleksander 'A.L.E.C' Machniak
> Kolab Groupware Developer        [http://kolab.org]
> Roundcube Webmail Developer  [http://roundcube.net]
> ---------------------------------------------------
> PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 12 Apr 2016, Bostjan Skufca wrote:
> On 12 April 2016 at 10:23, A.L.E.C <alec at alec.pl> wrote:
>
>> I don't know dovecot's code, but I suppose it uses uidlist file
to get
>> mailbox statistics that it returns as EXISTS, RECENT, UNSEEN, UIDNEXT,
>> UIDVALIDITY, etc, which are required by IMAP standard. I don't
know,
>> maybe they could be stored in more optimized way, but I think in most
>> cases this data is also needed for SORT/THREAD/FETCH which is sent
after
>> SELECT in many cases - so it will be needed anyway. There are cases
>> (e.g. mailbox synchronization) when you indeed do only SELECT.
>>
>
> (Oh, Alec, hi :)
>
> Can someone (who is more intimate with Dovecot's internals) comment on
what
> dovecot is actually doing, where this time is being spent? That would be
> awesome:)
>
> My uidfile is 4MB "large" and reading it takes about 3ms on my
system, so
> there is still ~160ms (98%) of dovecot's delay that is unaccounted for.
You could try to trace Dovecot and see if there are a lot of syscalls when 
a new connection starts up, e.g. with strace or dtruss.
> BTW: I would have thought that the whole process goes like this:
> - user logs in, dovecot imap process starts
> - "SELECT MyMailbox" is issued for the first time
> - dovecot reads uidlist file for that mailbox and caches it along with
> timestamp metadata
does your mailboxes change without Dovecot? Do you have some script or 
something like that that causes the mtime of the directory change? If 
Dovecot thinks an external program changed the mailbox, it validates all 
messages in cache and from store.
> - on subsequent requests, uidlist file is stat-ed to detect any changes
> (changed by i.e. another process that handles same user's connection,
from
> another device) and trigger rereading when needed, otherwise cache is used
> - I would imagine having certain (fairly low) TTL on the cache to prevent
> too much memory consumption would be nice, too.
>
> Would that make sense?
>
> (I am just trying to figure out the way how to optimize response time to
> SELECT and thus make rendering mailbox listing in Roundcube faster, or,
> better, acceptable.)
>
>
> Best,
> b.
>
>
>
>>
>> --
>> Aleksander 'A.L.E.C' Machniak
>> Kolab Groupware Developer        [http://kolab.org]
>> Roundcube Webmail Developer  [http://roundcube.net]
>> ---------------------------------------------------
>> PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
>>
>
- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEVAwUBVwzkWnz1H7kL/d9rAQLxzgf/VYvyXO2AhhpMO96EWHekgUZAFx8f+Wn+
UKJcVcbRVUvabo+8CDHyqClLbBugGDJfYDVMTOLplIJ3ZrExRqbh3sENmoRYnk8q
nlWoKuwoskUYYM30Ax9yYuVxDzhGOdrI7TDrojsnJgyV1w8u8jMW6iCi1ziYvcIc
u34DvOBLL16biP/rqziCouTbOZcsy9wL/T7RgBE/27ph651I7A8kQ3udfKWMW0gn
+EIaDWquVnHmgozP93Ln0TpN5tMi9GW50zNmT8tWUnr07aYc3E5vaS3uYLvwxVx9
P5WIVgEObF4+BrZVCFfVRpkp51hxngk2k70RdmjgzD1dOSGGLELZkw==j6AD
-----END PGP SIGNATURE-----