Hi, On 05/04/15 10:59, Timo Sirainen wrote:> On 28 Apr 2015, at 23:49, Nagy, Attila <bra at fsn.hu> wrote: >> Hi, >> >> imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines. >> >> I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop? > Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails. >I'm aware of that, but it doesn't, or at least not always. For example Thunderbird issues this: 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) Dovecot does this to the IMAP backend 18 UID FETCH 333574 (BODY.PEEK[]) 19 UID FETCH 333575 (BODY.PEEK[]) 20 UID FETCH 333576 (BODY.PEEK[]) 21 UID FETCH 333577 (BODY.PEEK[]) And so on. I have mail_prefetch_count set to 100: # doveconf -n | grep prefet mail_prefetch_count = 100
On 04 May 2015, at 16:59, Nagy, Attila <bra at fsn.hu> wrote:> > Hi, > > On 05/04/15 10:59, Timo Sirainen wrote: >> On 28 Apr 2015, at 23:49, Nagy, Attila <bra at fsn.hu> wrote: >>> Hi, >>> >>> imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines. >>> >>> I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop? >> Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails. >> > I'm aware of that, but it doesn't, or at least not always. > For example Thunderbird issues this: > 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) > Dovecot does this to the IMAP backend > 18 UID FETCH 333574 (BODY.PEEK[])Oh, there were several bugs related to that. Fixed: http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2 http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b
On 04 May 2015, at 18:45, Timo Sirainen <tss at iki.fi> wrote:> > On 04 May 2015, at 16:59, Nagy, Attila <bra at fsn.hu> wrote: >> >> Hi, >> >> On 05/04/15 10:59, Timo Sirainen wrote: >>> On 28 Apr 2015, at 23:49, Nagy, Attila <bra at fsn.hu> wrote: >>>> Hi, >>>> >>>> imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines. >>>> >>>> I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop? >>> Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails. >>> >> I'm aware of that, but it doesn't, or at least not always. >> For example Thunderbird issues this: >> 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) >> Dovecot does this to the IMAP backend >> 18 UID FETCH 333574 (BODY.PEEK[]) > > Oh, there were several bugs related to that. Fixed: > > http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc > http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2 > http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789bAlso if you want imapc to use the full set of features with latest Dovecot, use: imapc_features=rfc822.size fetch-headers search After that it shouldn't be fetching BODY.PEEK[] anymore unless the body was actually wanted to be fetched.
On 05/04/15 17:45, Timo Sirainen wrote>> I'm aware of that, but it doesn't, or at least not always. >> For example Thunderbird issues this: >> 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) >> Dovecot does this to the IMAP backend >> 18 UID FETCH 333574 (BODY.PEEK[]) > Oh, there were several bugs related to that. Fixed: > > http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc > http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2 > http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789bThank you very much for the quick fix(es), I will try them.
On 05/04/15 17:45, Timo Sirainen wrote:> On 04 May 2015, at 16:59, Nagy, Attila <bra at fsn.hu> wrote: >> Hi, >> >> On 05/04/15 10:59, Timo Sirainen wrote: >>> On 28 Apr 2015, at 23:49, Nagy, Attila <bra at fsn.hu> wrote: >>>> Hi, >>>> >>>> imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it works even with the dumbest IMAP server, altough it really kills performance, especially on high latency lines. >>>> >>>> I wonder: if IMAP servers can effectively handle boundless fetches (like a list with all wanted UIDs, or simply 1:* if all are needed), do you see this as a good addition to develop? >>> Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. The higher the value, the more memory/disk space is used for storing the received mails. >>> >> I'm aware of that, but it doesn't, or at least not always. >> For example Thunderbird issues this: >> 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]) >> Dovecot does this to the IMAP backend >> 18 UID FETCH 333574 (BODY.PEEK[]) > Oh, there were several bugs related to that. Fixed: > > http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc > http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2 > http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789bI've just tried 2.2.18, which has the above modifications, but the effect is the same. I have these in the config: # doveconf -n | egrep 'prefe|imapc' imapc_features = rfc822.size fetch-headers imapc_host = 10.3.34.12 mail_location = imapc:~/imapc mail_prefetch_count = 100 With an empty imapc directory a POP3 login yields these on the backend IMAP server: 2 LOGIN "user" "pass" 1 LIST "" "" 3 SELECT "INBOX" 4 NOOP 5 UID FETCH 1:* (FLAGS) 6 UID FETCH 2 (RFC822.SIZE) 7 UID FETCH 3 (RFC822.SIZE) 8 UID FETCH 4 (RFC822.SIZE) 9 UID FETCH 5 (RFC822.SIZE) 10 UID FETCH 6 (RFC822.SIZE) 11 UID FETCH 7 (RFC822.SIZE) 12 UID FETCH 8 (RFC822.SIZE) [...] so on, for every message... The response times for each of the FETCHes are around 13-16 ms, so logging into dovecot for the first time takes more than a minute with 5000 mails in the INBOX and a half day with around a million... It would be good to have a: 6 UID FETCH 1:* (RFC822.SIZE) or even a: 5 UID FETCH 1:* (FLAGS RFC822.SIZE) if needed (dovecot first fills up its cache, or other cases when the full list is needed) If dovecot would issue the latter for this 5000-mail inbox, the first fetch (and POP login) would take only 500 ms instead of the 75 seconds that it takes now... Could you please tweak this some more? :) Thanks,