Hi there, (context: I was optimizing Roundcube mailbox list server response, and in that 300-400ms response time, around 170ms is spent on single fgets() call which is waiting IMAP repsonse to "SELECT MyMailbox" command) I straced dovecot and of the whole request/response process, around 30ms is spent for everything else, and overwhelming majority of time (150-170ms) is spent for reading dovecot-uidlist file for given mailbox. I skimmed over src/lib-storage/index/maildir/maildir-uidlist.c and src/src/lib/istream.c, but I am out of ideas on how to optimize mentioned file reading elegantly to make it faster. Is there any way to cache parsed content of uidlist file(s) or some other obvious quicktrick I am missing to speed this process up? Does anyone have any other ideas, how to speed this up? b.
It will be nice if the "SELECT MyMailbox" command will be "SELECT mymailbox LIMIT 100" or something, to get the first files since only the last messages are shown to the user. I don't know if "Limit" clause is available over IMAP. On 04/09/2016 09:04 PM, Bostjan Skufca wrote:> Hi there, > > (context: I was optimizing Roundcube mailbox list server response, and in > that 300-400ms response time, around 170ms is spent on single fgets() call > which is waiting IMAP repsonse to "SELECT MyMailbox" command) > > I straced dovecot and of the whole request/response process, around 30ms is > spent for everything else, and overwhelming majority of time (150-170ms) is > spent for reading dovecot-uidlist file for given mailbox. > > I skimmed over src/lib-storage/index/maildir/maildir-uidlist.c > and src/src/lib/istream.c, but I am out of ideas on how to optimize > mentioned file reading elegantly to make it faster. > > Is there any way to cache parsed content of uidlist file(s) or some other > obvious quicktrick I am missing to speed this process up? > > Does anyone have any other ideas, how to speed this up? > > > b.-- Best regards, Adrian Minta
I tested this by timestamping tcpdump output while issuing this command directly to dovecot (limit 100 works, but does nothing) and there is no difference, it keeps taking ~160ms to respond. b. On 10 April 2016 at 10:27, Adrian Minta <adrian.minta at gmail.com> wrote:> It will be nice if the "SELECT MyMailbox" command will be "SELECT > mymailbox LIMIT 100" or something, to get the first files since only the > last messages are shown to the user. > I don't know if "Limit" clause is available over IMAP. > > > > > On 04/09/2016 09:04 PM, Bostjan Skufca wrote: > >> Hi there, >> >> (context: I was optimizing Roundcube mailbox list server response, and in >> that 300-400ms response time, around 170ms is spent on single fgets() call >> which is waiting IMAP repsonse to "SELECT MyMailbox" command) >> >> I straced dovecot and of the whole request/response process, around 30ms >> is >> spent for everything else, and overwhelming majority of time (150-170ms) >> is >> spent for reading dovecot-uidlist file for given mailbox. >> >> I skimmed over src/lib-storage/index/maildir/maildir-uidlist.c >> and src/src/lib/istream.c, but I am out of ideas on how to optimize >> mentioned file reading elegantly to make it faster. >> >> Is there any way to cache parsed content of uidlist file(s) or some other >> obvious quicktrick I am missing to speed this process up? >> >> Does anyone have any other ideas, how to speed this up? >> >> >> b. >> > > -- > Best regards, > Adrian Minta >
On 04/10/2016 10:27 AM, Adrian Minta wrote:> It will be nice if the "SELECT MyMailbox" command will be "SELECT > mymailbox LIMIT 100" or something, to get the first files since only the > last messages are shown to the user.SELECTing a mailbox has nothing to do with FETCHing messages, so above does not make much sense. I don't know dovecot's code, but I suppose it uses uidlist file to get mailbox statistics that it returns as EXISTS, RECENT, UNSEEN, UIDNEXT, UIDVALIDITY, etc, which are required by IMAP standard. I don't know, maybe they could be stored in more optimized way, but I think in most cases this data is also needed for SORT/THREAD/FETCH which is sent after SELECT in many cases - so it will be needed anyway. There are cases (e.g. mailbox synchronization) when you indeed do only SELECT. -- Aleksander 'A.L.E.C' Machniak Kolab Groupware Developer [http://kolab.org] Roundcube Webmail Developer [http://roundcube.net] --------------------------------------------------- PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
On 09 Apr 2016, at 21:04, Bostjan Skufca <bostjan at a2o.si> wrote:> > Hi there, > > (context: I was optimizing Roundcube mailbox list server response, and in > that 300-400ms response time, around 170ms is spent on single fgets() call > which is waiting IMAP repsonse to "SELECT MyMailbox" command) > > I straced dovecot and of the whole request/response process, around 30ms is > spent for everything else, and overwhelming majority of time (150-170ms) is > spent for reading dovecot-uidlist file for given mailbox. > > I skimmed over src/lib-storage/index/maildir/maildir-uidlist.c > and src/src/lib/istream.c, but I am out of ideas on how to optimize > mentioned file reading elegantly to make it faster. > > Is there any way to cache parsed content of uidlist file(s) or some other > obvious quicktrick I am missing to speed this process up? > > Does anyone have any other ideas, how to speed this up?Switch to mdbox or sdbox format to get better performance. I'm sure there are ways to optimize Maildir too, but it doesn't seem worth the effort since it'll always be slower than mdbox/sdbox.
On 12 April 2016 at 23:16, Timo Sirainen <tss at iki.fi> wrote:> > On 09 Apr 2016, at 21:04, Bostjan Skufca <bostjan at a2o.si> wrote: > > (context: I was optimizing Roundcube mailbox list server response, and in > > that 300-400ms response time, around 170ms is spent on single fgets() call > > which is waiting IMAP repsonse to "SELECT MyMailbox" command) > > > > Does anyone have any other ideas, how to speed this up? > > Switch to mdbox or sdbox format to get better performance. I'm sure there are ways to optimize Maildir too, but it doesn't seem worth the effort since it'll always be slower than mdbox/sdbox.Now we are talking! Response time cut almost in half. Thanks Timo, b.