dovecot - Dec 2011 - Dovecot 2.1rc1 + 2.0.16 woes regarding fts_squat

I can confirm the report posted in
http://dovecot.org/list/dovecot/2011-November/062263.html that fts_squat no
longer seems to be used after moving from 2.0.16->2.1 rc 1.  I don't see
crash reports in the logs, just "0 messages indexed". My search test
tool just does a normal IMAP SEARCH for a long non-existent string. If
there's another way to trigger re-indexing in 2.1, I don't see anything
in the documentation for it.

I've enabled mail_debug, but no log entries that shed light on the problem
are available.  Has the configuration for fts_squat changed?

If anyone has a working fts_squat setup with Dovecot 2.1rc1, I'd appreciate
hearing how you have it setup and working.

During a large mail import with 2.0.16 today, I ran across a worrying message in
the logs during an fts_squat reindex: out of memory. The plugin doesn't obey
the mmap_disable configuration directive, which I've confirmed in the plugin
source.

The mailbox in question has only 17GB (mdbox style), with about 90,000 emails in
it.  Its "index" (for the purposes of normal IMAP retrieval as opposed
to IMAP TEXT/BODY searching) is fine and uncorrupted. I freshly import these
mailboxes between test iterations and any version changes anyway, so if
there's corruption, it's happening within dovecot only.  I'm using
use Mail::IMAPClient to create + append mail over localhost, not any direct
mdbox conversion trickery.

In looking through the code, I see that mmap() is called for the *ENTIRE FILE*,
which is guaranteed to fail on large indexes.  I assume this was done out of
expedience, but it's a "risky" sort of thing to do in a server
process, even if 8GB RAM systems do seem to grow on trees.  I intend to put this
to work in a large installation (>10K users), so this IS of some concern for
me in the long-term.

Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: mremap_anon(188084224)
failed: Cannot allocate memory
Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error:
read(.../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids) failed:
Cannot allocate memory
Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: mremap_anon(188280832)
failed: Cannot allocate memory
Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error:
read(.../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids) failed:
Cannot allocate memory
Dec 12 22:50:47 linuxcode dovecot: imap(user1001): Error: Corrupted squat
uidlist file .../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids:
uidlist not found

===============
dovecot -n output:

# 2.0.16: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.35.7-smp i686 Slackware 13.1.0 
auth_mechanisms = plain cram-md5 digest-md5 apop
default_vsz_limit = 192 M
disable_plaintext_auth = no
first_valid_gid = 100
hostname = linuxcode
info_log_path = /tmp/dovecot.log
last_valid_gid = 60000
last_valid_uid = 60000
listen = *
mail_location = mdbox:~/mdbox
mail_plugins = " zlib acl"
mdbox_preallocate_space = yes
mdbox_rotate_interval = 1 days
mmap_disable = yes
passdb {
  args = scheme=plain /etc/cram-md5.pwd
  driver = passwd-file
}
plugin {
  acl = vfile
}
postmaster_address = postmaster at linuxcode
quota_full_tempfail = yes
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_min_avail = 0
  vsz_limit = 64 M
}
service imap {
  vsz_limit = 512 M
}
service lmtp {
  unix_listener lmtp {
    mode = 0666
  }
}
ssl = no
userdb {
  args = blocking=no
  driver = passwd
}
protocol lmtp {
  mail_plugins = " zlib acl"
}
protocol imap {
  mail_plugins = " zlib acl fts fts_squat imap_acl imap_zlib"
  plugin {
    fts = squat
    fts_squat = partial=4 full=10
  }
}
===============
CONFIGURE
===============LIBS=-lnsl CFLAGS='-O2 -march=core2 -mtune=core2
-fstack-protector -fomit-frame-pointer' \
CXXFLAGS='-O2 -march=core2 -mtune=core2 -fstack-protector
-fomit-frame-pointer' \
LDFLAGS=-s ./configure --prefix=/usr --sysconfdir=/etc \
--with-mysql --with-sqlite --with-pgsql --without-pam --with-sql \
--with-libwrap --with-libcap -with-ssl=openssl --with-solr \
--with-mem-align=16 --with-bzlib --with-zlib --localstatedir=/var
===============
OS: Slackware 13.1 (32-bit, 2GB physical RAM, kernel setup for 2G/2G split)
fully patched up

=R=

On Tue, 2011-12-13 at 15:50 -0800, Robin wrote:
> I can confirm the report posted in
> http://dovecot.org/list/dovecot/2011-November/062263.html that
> fts_squat no longer seems to be used after moving from 2.0.16->2.1 rc
> 1.  I don't see crash reports in the logs, just "0 messages
indexed".
> My search test tool just does a normal IMAP SEARCH for a long
> non-existent string. If there's another way to trigger re-indexing in
> 2.1, I don't see anything in the documentation for it.
Maybe it was buggy in v2.1.rc1 .. at least in my tests now it works with
latest v2.1 hg. I should release rc2 soon I guess.
> During a large mail import with 2.0.16 today, I ran across a worrying
> message in the logs during an fts_squat reindex: out of memory.
I'm not planning on doing any changes to fts-squat anymore. With v2.1
you can use fts-lucene instead.