dovecot - May 2014 - Incorrect IMAP search results when FTS/Squat indexes are present with 2.1.7

We recently upgraded from Dovecot 1.2.15 (on Debian squeeze) to Dovecot
2.1.7 (on Debian wheezy/stable).

For now, we continue to use FTS with Squat indexes. We'd like to
eventually switch to Lucene but it's not yet supported in Debian stable.

Since upgrading to 2.1.7, we've seen some problems with IMAP search
results when an FTS Squat index is present (i.e., when the
"dovecot.index.search" and "dovecot.index.search.uids" files
are
present). If it's present, even header searches sometimes give incorrect
results (such as searches on FROM or SUBJECT headers).

I'm wondering if this is a known issue with 2.1.7 and whether upgrading
to a later version of Dovecot with Squat is likely to fix this (due to,
say, <http://www.dovecot.org/list/dovecot-cvs/2012-June/020881.html>),
or if Squat in all versions is a lost cause and I need to switch to Lucene.

I can demonstrate the problem by showing the results of three searches
on my mailbox. First, here are the correct results for header FTS
searches when no "dovecot.index.search[.uids]" file is present:

----
. UID SEARCH FROM "Nick"
* SEARCH 9 66 100 116 179 212 274 338 347 386 494 519 579 603 643 766
781 788 827 1107 1280 1626 1641 1667 1688 1702 1703 2184 2187 3202 3519
3716 3798 3885 3886 4024 4166 4345 4400 4401 4436 4491 4561 4595 4618
4676 4711 4712 4746 4788 4883 5217 5253 5577 5602 5722
. OK Search completed (0.059 secs).

. UID SEARCH ALL FROM "Nick"
* SEARCH 9 66 100 116 179 212 274 338 347 386 494 519 579 603 643 766
781 788 827 1107 1280 1626 1641 1667 1688 1702 1703 2184 2187 3202 3519
3716 3798 3885 3886 4024 4166 4345 4400 4401 4436 4491 4561 4595 4618
4676 4711 4712 4746 4788 4883 5217 5253 5577 5602 5722
. OK Search completed (0.058 secs).

. UID SEARCH SUBJECT "Nick"
* SEARCH 3466 3467 3507
. OK Search completed (0.054 secs).
----

These results are consistent (and correct) whether "mail_plugins = fts
fts_squat" is enabled or not, as long as no
"dovecot.index.search[.uids]" file is present.

Now I generate an FTS index:

----
. UID SEARCH TEXT "Nick"
* OK Indexed 98% of the mailbox, ETA 0:00
* OK Mailbox indexing finished
* SEARCH 9 32 66 100 111 112 113 116 179 199 212 228 229 240 274 303 338
347 386 393 448 478 491 494 519 579 591 603 614 618 624 643 649 742 766
781 788 827 837 839 841 1095 1106 1107 1174 1188 1225 1227 1228 1240
1254 1261 1278 1280 1452 1482 1519 1520 1595 1626 1641 1656 1667 1688
1702 1703 1724 1758 2184 2187 2798 2883 2932 3034 3198 3202 3206 3230
3371 3432 3454 3457 3466 3467 3507 3519 3631 3637 3685 3702 3703 3716
3765 3798 3799 3803 3853 3854 3855 3857 3873 3885 3886 3962 3969 4024
4033 4050 4051 4058 4161 4166 4173 4176 4224 4273 4345 4400 4401 4436
4461 4490 4491 4498 4561 4595 4618 4651 4676 4711 4712 4713 4729 4731
4736 4737 4738 4739 4746 4786 4787 4788 4813 4816 4822 4847 4883 4885
4933 4941 4942 4957 4958 5029 5052 5097 5135 5136 5182 5217 5253 5271
5357 5543 5551 5565 5577 5589 5602 5612 5643 5692 5722
. OK Search completed (11.078 secs).
----

Now I run the same original three searches, but they return completely
different results(!):

----
. UID SEARCH FROM "Nick"
* SEARCH 9 66 100 116 179 212 228 240 274 338 347 386 448 494 519 579
603 643 766 781 788 827 1095 1107 1174 1240 1254 1261 1280 1595 1626
1641 1667 1688 1702 1703 1724 2184 2187 2883 2932 3202 3371 3432 3457
3466 3467 3507 3519 3631 3637 3685 3716 3765 3798 3803 3853 3854 3855
3885 3886 3969 4024 4050 4166 4176 4224 4345 4400 4401 4436 4461 4490
4491 4498 4561 4595 4618 4676 4711 4712 4729 4731 4746 4788 4883 5029
5135 5136 5182 5217 5253 5577 5602 5722
. OK Search completed (0.001 secs).

. UID SEARCH ALL FROM "Nick"
* SEARCH
. OK Search completed (0.001 secs).

. UID SEARCH SUBJECT "Nick"
* SEARCH 9 66 100 116 179 212 228 240 274 338 347 386 448 494 519 579
603 643 766 781 788 827 1095 1107 1174 1240 1254 1261 1280 1595 1626
1641 1667 1688 1702 1703 1724 2184 2187 2883 2932 3202 3371 3432 3457
3466 3467 3507 3519 3631 3637 3685 3716 3765 3798 3803 3853 3854 3855
3885 3886 3969 4024 4050 4166 4176 4224 4345 4400 4401 4436 4461 4490
4491 4498 4561 4595 4618 4676 4711 4712 4729 4731 4746 4788 4883 5029
5135 5136 5182 5217 5253 5577 5602 5722
. OK Search completed (0.001 secs).
----

The "ALL" one returns nothing at all, even though an "AND"
search on
"ALL" should make no difference, and the "FROM" and
"SUBJECT" ones now
return identical incorrect results, many of which do not actually match
the header requested when examined -- it's returning messages with
"Nick" in *any* header. For example, message 9 does not contain
"Nick"
in the subject, despite the results shown above:

----
. FETCH 9 UID
* 9 FETCH (UID 9)
. OK Fetch completed.

FETCH 9 BODY.PEEK[HEADER.FIELDS (SUBJECT)]
* 9 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {47}
Subject: Re: Finishing Gallery support page

)
. OK Fetch completed.
----

If I delete the "dovecot.index.search" and
"dovecot.index.search.uids"
files, or if I disable fts in the configuration entirely, the results
immediately become correct again.

As I mentioned above, I'm wondering if anyone knows the cause of (and
fix for) this.

Here's my doveconf -n:

# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4
auth_master_user_separator = *
auth_mechanisms = plain login
auth_username_format default_vsz_limit = 1 G
disable_plaintext_auth = no
lda_mailbox_autosubscribe = yes
listen = *
log_timestamp = "%Y-%m-%d %H:%M:%S "
login_log_format_elements = pid=[%p] user=<%u> method=%m rip=%r lip=%l %c
mail_fsync = never
mail_location = maildir:~/
mail_max_userip_connections = 100
mail_plugins = fts fts_squat mail_log notify
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex
imap4flags copy include variables body enotify environment mailbox date
ihave spamtest spamtestplus
namespace {
  inbox = yes
  location   prefix = INBOX.
  separator = .
  type = private
}
namespace {
  hidden = yes
  inbox = no
  list = no
  location   prefix   separator = .
  type = private
}
passdb {
  args = /etc/dovecot/dovecot.users
  driver = passwd-file
}
passdb {
  args = /etc/dovecot/master-passwords
  driver = passwd-file
  master = yes
  pass = yes
}
plugin {
  fts = squat
  mail_log_events = copy expunge
  mail_log_fields = box msgid size from
  sieve = %h/mailbox.sieve
  sieve_after = %h/../../domain-after.sieve
  sieve_before = %h/../../domain-before.sieve
  sieve_dir = %h/sieve
  sieve_extensions = -reject +spamtest +spamtestplus
  sieve_global_dir = %h/../../sieve-global-include-scripts
  sieve_spamtest_max_value = 7
  sieve_spamtest_status_header = X-Spam-Level
  sieve_spamtest_status_type = strlen
}
protocols = pop3 imap lmtp sieve
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0660
    user = postfix
  }
}
service imap-login {
  process_min_avail = 8
  service_count = 0
}
service imap {
  process_limit = 2000
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0660
    user = postfix
  }
}
service pop3-login {
  process_min_avail = 8
  service_count = 0
}
service pop3 {
  process_limit = 2000
}
ssl = no
userdb {
  args = /etc/dovecot/dovecot.users
  driver = passwd-file
}
verbose_proctitle = yes
protocol imap {
  imap_client_workarounds = delay-newmail
  imap_logout_format = %i bytes in, %o bytes out
}
protocol pop3 {
  pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
  pop3_logout_format = ; %i bytes in, %o bytes out, %t top (%p bytes),
%r retr (%b bytes), %d of %m deleted, mailbox size %s bytes
  pop3_uidl_format = UID%u-%v
}
protocol lda {
  mail_fsync = optimized
  mail_plugins = sieve
  postmaster_address = postmaster at tigertech.net
  sendmail_path = /usr/local/bin/dovecot-sendmail-wrapper
}
protocol lmtp {
  mail_fsync = optimized
  mail_plugins = sieve
  postmaster_address = postmaster at tigertech.net
  sendmail_path = /usr/local/bin/dovecot-sendmail-wrapper
}

-- 
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/