dovecot - Oct 2012 - When are search indexes updated?

I noticed that occasionally searching in my huge archive mailbox can be
really slow, so I tried doveadm index on it and it seemed to do a lot of
work, which seemed strange given, for example, that dovecot-lda says it
keeps Dovecot index files up-to-date.  Then I thought, "maybe these are
different files than the search indices."  If so, that's not entirely
clear from the docs and Wiki.  So, questions:

* When are search indexes updated?
* Are they updated incrementally?
* If not, why not?
* If so, why would a mailbox's index drift out-of-date, as mine had?

BTW, I'm using the clucene search backend.

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost

On 10/27/2012 3:00 PM, David Abrahams wrote:
> 
> I noticed that occasionally searching in my huge archive mailbox can be
> really slow, so I tried doveadm index on it and it seemed to do a lot of
> work, which seemed strange given, for example, that dovecot-lda says it
> keeps Dovecot index files up-to-date.  Then I thought, "maybe these
are
> different files than the search indices."  If so, that's not
entirely
> clear from the docs and Wiki.  So, questions:
Mailbox and search indexes are separate.  Look in your mailbox directory
and you'll see them, such as on 1.2.x with mbox:

$ la /home/stan/mail/.imap/1-Dovecot
total 3.4M
drwx------  2 stan stan  135 Oct 25 21:39 .
drwx------ 51 stan stan 4.0K Apr 13  2012 ..
-rw-------  1 stan stan  44K Oct 27 13:28 dovecot.index
-rw-------  1 stan stan 1.2M Oct 27 21:23 dovecot.index.cache
-rw-------  1 stan stan  18K Oct 27 21:23 dovecot.index.log
-rw-------  1 stan stan 1.1M May 20 06:32 dovecot.index.search
-rw-------  1 stan stan 1.1M May 20 06:32 dovecot.index.search.uids

I've not full text searched this folder for quite some time, thus the
search indexes are not current, and the next FTS of this mail folder
will take much more time than if the FTS indexes were current.
> * When are search indexes updated?
When the index is stale.
> * Are they updated incrementally?
> * If not, why not?
> * If so, why would a mailbox's index drift out-of-date, as mine had?
When a sufficient number of messages are added to an IMAP folder the FTS
index becomes stale.  This index is not updated in real time.  This is
why Timo and others recommend cron'ing a script to index folders
regularly that are searched regularly.  This keeps the indexes up to
date and keeps searches fast.  If you don't do this or search often,
your indexes become stale.  Then each time you do an FTS search the
first thing that happens is an FTS re-indexing of the mail folder.  Only
then does it display the search results.
> BTW, I'm using the clucene search backend.
I've not used Lucene, but I believe the default behavior is similar to
the Dovecot 1.2.x FTS indexer.

-- 
Stan