dovecot - Dec 2021 - Searching 30 GB mailbox

Hi,

We're migrating our mailboxes from Zimbra to dovecot, running on FreeBSD 
12.2. We're using mdbox and solr for fts. dovecot version 2.3.17.

One of our mailboxes gets a bcc of every outgoing mail from our system 
and it currently holds ~4.5 million messages taking 30 GB.

One of the ways we use the mailbox is that when outgoing mail is sent, 
we store the Message-ID in our database, and in our admin interface we 
can open the mail and it's fetched from IMAP by doing a 'SEARCH HEADER 
Message-ID <MSG_ID>'.

In Zimbra this was reasonably fast, but after migrating the mailbox to 
dovecot it's 30 seconds which I think we can reduce.

After turning on mail_debug I see this in our log 
https://pastebin.com/kP9H7gSB

Looks like solr finishes searching instantly, then there's ~30 seconds 
of silence in the logs, and then more log lines about fetching data from 
mail cache.

The same thing happens if I use telnet and just issue a 'SEARCH 
HEADER...' command.

How can I find out what dovecot is doing in those 30 seconds?

.einar

> On 1. Dec 2021, at 0.30, Einar Bjarni Halld?rsson <einar at isnic.is>
wrote:
> 
> Hi,
> 
> We're migrating our mailboxes from Zimbra to dovecot, running on
FreeBSD 12.2. We're using mdbox and solr for fts. dovecot version 2.3.17.
> 
> One of our mailboxes gets a bcc of every outgoing mail from our system and
it currently holds ~4.5 million messages taking 30 GB.
> 
> One of the ways we use the mailbox is that when outgoing mail is sent, we
store the Message-ID in our database, and in our admin interface we can open the
mail and it's fetched from IMAP by doing a 'SEARCH HEADER Message-ID
<MSG_ID>'.
> 
> In Zimbra this was reasonably fast, but after migrating the mailbox to
dovecot it's 30 seconds which I think we can reduce.
> 
> After turning on mail_debug I see this in our log
https://pastebin.com/kP9H7gSB
> 
> Looks like solr finishes searching instantly, then there's ~30 seconds
of silence in the logs, and then more log lines about fetching data from mail
cache.
> 
> The same thing happens if I use telnet and just issue a 'SEARCH
HEADER...' command.
> 
> How can I find out what dovecot is doing in those 30 seconds?

Did you have Messsage-ID listed in mail_always_cache_fields when you imported
the mailbox?
Do you have it now? 

If not, please add it and then use doveadm fetch to populate that in cache.

I think the correct setting is

mail_always_cache_fields = hdr.message-id

But you also might want to add some other fields

Sami