dovecot - Oct 2015 - Dovecot frequently full-text reindexes the whole maildir

Hi!

I'm using Dovecot 2.2.13 (Debian Jessie package 1:2.2.13-12~deb8u1) on 
my personal mail server (the address I'm writing from is on this 
server).

I use Maildirs, I have fts + fts_squat enabled, and I have a problem 
with it for a long time - dovecot seems to not update the index always 
"incrementally".

Yesterday I've finally made a test by telneting to imap port and issuing 
a search request. Dovecot started to reindex the mailbox. Full 
reindexing has taken around 15-20 minutes.

After reindexing the fulltext searches were fast, just as they should 
be. And they were fast yesterday even after some new emails arrived.

But I've tried to do a new search today and Dovecot started to reindex 
the whole maildir again!

Why is it reindexing the whole maildir again? Could it be related to my 
anti-spam scripts removing messages from SPAM imap folder? And if yes, 
how to remove them correctly?

-- 
With best regards,
   Vitaliy Filippov

Hi,

I'm not an expert, but am doing something similar (moving messages and
training the spam filter).

To detect the existence of new messages looking directly at the maildir can be a
quick and dirty solution to get a trigger event.

But to move or delete I think you should use  'doveadm' to search for
all messages in your spam directory and get a list of all uids an guids. Then
you can call doveadm again to move vor delete the messages by id.

Just look for "doveadm search/move/purge" in the dovecot wiki.
This should ensure that the indexes stay up to date.

Am 27. Oktober 2015 11:24:39 MEZ, schrieb vitalif at
yourcmc.ru:
>Hi!
>
>I'm using Dovecot 2.2.13 (Debian Jessie package 1:2.2.13-12~deb8u1) on 
>my personal mail server (the address I'm writing from is on this 
>server).
>
>I use Maildirs, I have fts + fts_squat enabled, and I have a problem 
>with it for a long time - dovecot seems to not update the index always 
>"incrementally".
>
>Yesterday I've finally made a test by telneting to imap port and
>issuing 
>a search request. Dovecot started to reindex the mailbox. Full 
>reindexing has taken around 15-20 minutes.
>
>After reindexing the fulltext searches were fast, just as they should 
>be. And they were fast yesterday even after some new emails arrived.
>
>But I've tried to do a new search today and Dovecot started to reindex 
>the whole maildir again!
>
>Why is it reindexing the whole maildir again? Could it be related to my
>
>anti-spam scripts removing messages from SPAM imap folder? And if yes, 
>how to remove them correctly?
>
>-- 
>With best regards,
>   Vitaliy Filippov
-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Thanks, after rewriting anti-spam scripts to doveadm and changing the 
search engine to fts-lucene fulltext indexing started to work reliably 
(fts-squat has constantly corrupted its index files, but lucene is 
working OK).

Now I have another small question: 'doveadm expunge' (even with -v) does
not print any errors, but always returns exit code 68 in my case. What 
does it mean? I would like to get zero exit code in case of success...
> Hi,
> 
> I'm not an expert, but am doing something similar (moving messages and
> training the spam filter).
> 
> To detect the existence of new messages looking directly at the
> maildir can be a quick and dirty solution to get a trigger event.
> 
> But to move or delete I think you should use  'doveadm' to search
for
> all messages in your spam directory and get a list of all uids an
> guids. Then you can call doveadm again to move vor delete the messages
> by id.
> 
> Just look for "doveadm search/move/purge" in the dovecot wiki.
> This should ensure that the indexes stay up to date.
> 
> Am 27. Oktober 2015 11:24:39 MEZ, schrieb vitalif at yourcmc.ru:
>> Hi!
>> 
>> I'm using Dovecot 2.2.13 (Debian Jessie package 1:2.2.13-12~deb8u1)
on
>> my personal mail server (the address I'm writing from is on this
>> server).
>> 
>> I use Maildirs, I have fts + fts_squat enabled, and I have a problem
>> with it for a long time - dovecot seems to not update the index always
>> "incrementally".
>> 
>> Yesterday I've finally made a test by telneting to imap port and
>> issuing
>> a search request. Dovecot started to reindex the mailbox. Full
>> reindexing has taken around 15-20 minutes.
>> 
>> After reindexing the fulltext searches were fast, just as they should
>> be. And they were fast yesterday even after some new emails arrived.
>> 
>> But I've tried to do a new search today and Dovecot started to
reindex
>> the whole maildir again!
>> 
>> Why is it reindexing the whole maildir again? Could it be related to 
>> my
>> 
>> anti-spam scripts removing messages from SPAM imap folder? And if yes,
>> how to remove them correctly?
>> 
-- 
With best regards,
   Vitaliy Filippov