Aki Tuomi
2018-May-21 10:38 UTC
Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
can you try turning on pluign { fts_enforced = yes } and repeat your test? Aki On 21.05.2018 13:31, kadafax at gmail.com wrote:> Le 21/05/2018 ? 11:46, Aki Tuomi a ?crit?: >> That message is coming from the temporary raw user, not from the actual >> user. >> >> How did you verify that the FTS indexes are not created or updated >> automatically? > I send an email containing a specific strings ('thahB8ea' in that case) > and search agains it: > # doveadm search -u username mailbox INBOX body thahB8ea > --> No result > (same with integrated search on roundcube) > > Then I force a rescan, followed by the same search: > # doveadm? fts rescan -u username > # doveadm search -u username mailbox INBOX body thahB8ea > (Long waiting time) > --> e09cce0283e8695ab760000002deed92 29048 > > So I conclude that autoindexing is not working. > > Thanks, > kfx > >> Aki >> >> >> On 04.05.2018 12:47, kadafax at gmail.com wrote: >>> ?Bump. Can someone from dovecot dev team please respond something to my >>> problem ? I've provided all the possible information, and if not please >>> tell me. >>> Or do you only provide support for your own FTS solution bundled with >>> the paid version of dovecot-pro ? If so, you should clearly state it in >>> the wiki. >>> >>> Thanks >>> >>> Le 19/04/2018 ? 10:30, kadafax at gmail.com a ?crit?: >>>> I think I'm progressing: if I create a brand new user, send her an >>>> email, it will not be indexed right away. I need to doveadm rescan / >>>> index the Inbox before. >>>> At the end of this email, a dovecot's debug log snipset (grep'ed on this >>>> user). In it I can read: "Debug: fts: Indexes disabled for namespace '' >>>> Can this be a problem ? >>>> >>>> Thanks >>>> >>>> doveconf -n output: >>>> # 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf >>>> # Pigeonhole version 0.4.22 (22940fb7) >>>> # OS: Linux 3.10.0-693.17.1.el7.x86_64 x86_64 CentOS Linux release >>>> 7.4.1708 (Core)? ext4 >>>> # Hostname: mailServer.myDomain >>>> auth_master_user_separator = * >>>> auth_socket_path = /var/run/dovecot/auth-userdb >>>> auth_username_chars >>>> abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@* >>>> auth_verbose = yes >>>> base_dir = /var/run/dovecot/ >>>> debug_log_path = /var/log/dovecot-debug.log >>>> default_client_limit = 5120 >>>> default_process_limit = 1024 >>>> default_vsz_limit = 2 G >>>> hostname = myDomain >>>> info_log_path = /var/log/dovecot.log >>>> lda_mailbox_autocreate = yes >>>> lda_mailbox_autosubscribe = yes >>>> listen = * >>>> log_path = /var/log/dovecot-error.log >>>> login_greeting = Welcome >>>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c >>>> mail_gid = vmail >>>> mail_location = maildir:/var/vmail/%Ln >>>> mail_plugins = " quota stats fts fts_solr" >>>> mail_uid = vmail >>>> managesieve_notify_capability = mailto >>>> managesieve_sieve_capability = fileinto reject envelope >>>> encoded-character vacation subaddress comparator-i;ascii-numeric >>>> relational regex imap4flags copy include variables body enotify >>>> environment mailbox date index ihave duplicate mime foreverypart extracttext >>>> namespace { >>>> ? inbox = yes >>>> ? location = maildir:/var/vmail/%Ln >>>> ? mailbox Drafts { >>>> ??? auto = subscribe >>>> ??? special_use = \Drafts >>>> ? } >>>> ? mailbox Sent { >>>> ??? auto = subscribe >>>> ??? special_use = \Sent >>>> ? } >>>> ? mailbox "Sent Messages" { >>>> ??? special_use = \Sent >>>> ? } >>>> ? mailbox Trash { >>>> ??? auto = subscribe >>>> ??? special_use = \Trash >>>> ? } >>>> ? prefix >>>> ? type = private >>>> } >>>> passdb { >>>> ? args = /usr/local/etc/dovecot/deny-users >>>> ? deny = yes >>>> ? driver = passwd-file >>>> } >>>> passdb { >>>> ? args = /usr/local/etc/dovecot/master-users >>>> ? driver = passwd-file >>>> ? master = yes >>>> } >>>> passdb { >>>> ? args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext >>>> ? driver = ldap >>>> } >>>> plugin { >>>> ? fts = solr >>>> ? fts_autoindex = yes >>>> ? fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search >>>> ? quota = maildir:User quota >>>> ? quota_rule = *:storage=20GB >>>> ? quota_rule2 = Trash:storage=+200M >>>> ? quota_warning = storage=95%% quota-warning 95 %u >>>> ? quota_warning2 = storage=80%% quota-warning 80 %u >>>> ? sieve = /var/sieve-scripts/%Ln.sieve >>>> ? sieve_dir = /var/vmail/%Ln/home/sieve >>>> ? stats_command_min_time = 1 mins >>>> ? stats_domain_min_time = 12 hours >>>> ? stats_ip_min_time = 12 hours >>>> ? stats_memory_limit = 16 M >>>> ? stats_refresh = 30 secs >>>> ? stats_session_min_time = 15 mins >>>> ? stats_track_cmds = yes >>>> ? stats_user_min_time = 1 hours >>>> } >>>> postmaster_address = postmaster at myDomain >>>> protocols = imap pop3 sieve sieve >>>> service auth-worker { >>>> ? user = root >>>> } >>>> service auth { >>>> ? unix_listener /var/spool/postfix/private/auth { >>>> ??? mode = 0777 >>>> ??? user = vmail >>>> ? } >>>> } >>>> service imap-login { >>>> ? inet_listener imaps { >>>> ??? port = 993 >>>> ??? ssl = yes >>>> ? } >>>> ? process_min_avail = 10 >>>> } >>>> service imap { >>>> ? process_limit = 2128 >>>> } >>>> service managesieve-login { >>>> ? inet_listener sieve { >>>> ??? port = 4190 >>>> ? } >>>> } >>>> service pop3-login { >>>> ? inet_listener pop3s { >>>> ??? port = 995 >>>> ??? ssl = yes >>>> ? } >>>> } >>>> service quota-warning { >>>> ? executable = script /usr/local/bin/quota-warning.sh >>>> ? user = nobody >>>> } >>>> service stats { >>>> ? fifo_listener stats-mail { >>>> ??? mode = 0600 >>>> ??? user = vmail >>>> ? } >>>> } >>>> ssl_cert = </etc/letsencrypt/live/mailServer.myDomain/fullchain.pem >>>> ssl_key =? # hidden, use -P to show it >>>> userdb { >>>> ? args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext >>>> ? driver = ldap >>>> ? override_fields = uid=vmail gid=vmail home=/var/vmail/%Ln/home >>>> } >>>> verbose_proctitle = yes >>>> protocol lda { >>>> ? mail_plugins = " quota stats fts fts_solr sieve virtual" >>>> } >>>> protocol imap { >>>> ? mail_max_userip_connections = 20 >>>> ? mail_plugins = " quota stats fts fts_solr imap_quota virtual imap_stats" >>>> } >>>> protocol sieve { >>>> ? mail_max_userip_connections = 20 >>>> } >>>> >>>> DEBUG LOG: >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Loading modules from >>>> directory: /usr/local/lib/dovecot >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_sieve_plugin.so >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: auth USER input: >>>> NewUser at myDomain uid=992 gid=989 home=/var/vmail/NewUser/home >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Effective uid=992, >>>> gid=989, home=/var/vmail/NewUser/home >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota root: name=User >>>> quota backend=maildir args>>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota rule: root=User >>>> quota mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota rule: root=User >>>> quota mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota warning: >>>> bytes=20401094656 (95%) messages=0 reverse=no command=quota-warning 95 >>>> NewUser at myDomain >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota warning: >>>> bytes=17179869184 (80%) messages=0 reverse=no command=quota-warning 80 >>>> NewUser at myDomain >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota grace: root=User >>>> quota bytes=2147483648 (10%) >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Namespace : type=private, >>>> prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: quota: quota_over_flag >>>> check: quota_over_script unset - skipping >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota root: name=User >>>> quota backend=maildir args>>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota rule: root=User >>>> quota mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota rule: root=User >>>> quota mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota warning: >>>> bytes=20401094656 (95%) messages=0 reverse=no command=quota-warning 95 >>>> raw mail user >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota warning: >>>> bytes=17179869184 (80%) messages=0 reverse=no command=quota-warning 80 >>>> raw mail user >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Quota grace: root=User >>>> quota bytes=2147483648 (10%) >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: none: root=, index=, >>>> indexpvt=, control=, inbox=, alt>>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: fts: Indexes disabled for >>>> namespace '' >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: quota: quota_over_flag >>>> check: quota_over_script unset - skipping >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Destination address: >>>> NewUser at myDomain (source: -a parameter) >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: sieve: Pigeonhole version >>>> 0.4.22 (22940fb7) initializing >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: sieve: include: >>>> sieve_global is not set; it is currently not possible to include >>>> `:global' scripts. >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: sieve: file storage: >>>> Storage path `/var/vmail/NewUser/home/sieve' not found >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: sieve: file storage: Using >>>> Sieve script path: /var/sieve-scripts/NewUser.sieve >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: sieve: file storage: >>>> Storage path `/var/sieve-scripts/NewUser.sieve' not found >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: sieve: storage: No default >>>> script location configured >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: sieve: User has no >>>> personal script >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: sieve: No scripts to >>>> execute: reverting to default delivery. >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: INBOX: Mailbox opened >>>> because: lib-lda delivery >>>> Apr 19 10:23:26 lda(NewUser at myDomain): Debug: Mailbox stdin: Opened mail >>>> UID=1 because: copying >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: auth USER >>>> input: NewUser at myDomain uid=992 gid=989 home=/var/vmail/NewUser/home >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Effective >>>> uid=992, gid=989, home=/var/vmail/NewUser/home >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Quota root: >>>> name=User quota backend=maildir args>>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Quota rule: >>>> root=User quota mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Quota rule: >>>> root=User quota mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Quota warning: >>>> bytes=20401094656 (95%) messages=0 reverse=no command=quota-warning 95 >>>> NewUser at myDomain >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Quota warning: >>>> bytes=17179869184 (80%) messages=0 reverse=no command=quota-warning 80 >>>> NewUser at myDomain >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Quota grace: >>>> root=User quota bytes=2147483648 (10%) >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Namespace : >>>> type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, >>>> subscriptions=yes location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: quota: >>>> quota_over_flag check: quota_over_script unset - skipping >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: INBOX: Mailbox >>>> opened because: indexing >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Mailbox INBOX: >>>> Opened mail UID=6 because: prefetch >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Mailbox INBOX: >>>> Opened mail UID=6 because: access >>>> Apr 19 10:23:26 indexer-worker(NewUser at myDomain): Debug: Mailbox INBOX: >>>> Opened mail UID=6 because: fts indexing >>>> Apr 19 10:23:50 imap(NewUser): Debug: Loading modules from directory: >>>> /usr/local/lib/dovecot >>>> Apr 19 10:23:50 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:23:50 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib11_imap_quota_plugin.so >>>> Apr 19 10:23:50 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:23:50 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:23:50 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:23:50 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:23:50 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib95_imap_stats_plugin.so >>>> Apr 19 10:23:50 imap(NewUser): Debug: Effective uid=992, gid=989, >>>> home=/var/vmail/NewUser/home >>>> Apr 19 10:23:50 imap(NewUser): Debug: Quota root: name=User quota >>>> backend=maildir args>>>> Apr 19 10:23:50 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:50 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:50 imap(NewUser): Debug: Quota warning: bytes=20401094656 >>>> (95%) messages=0 reverse=no command=quota-warning 95 NewUser >>>> Apr 19 10:23:50 imap(NewUser): Debug: Quota warning: bytes=17179869184 >>>> (80%) messages=0 reverse=no command=quota-warning 80 NewUser >>>> Apr 19 10:23:50 imap(NewUser): Debug: Quota grace: root=User quota >>>> bytes=2147483648 (10%) >>>> Apr 19 10:23:50 imap(NewUser): Debug: Namespace : type=private, prefix=, >>>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:50 imap(NewUser): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:50 imap(NewUser): Debug: quota: quota_over_flag check: >>>> quota_over_script unset - skipping >>>> Apr 19 10:23:50 imap(NewUser): Debug: INBOX: Mailbox opened because: SELECT >>>> Apr 19 10:23:52 imap(NewUser): Debug: Loading modules from directory: >>>> /usr/local/lib/dovecot >>>> Apr 19 10:23:52 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:23:52 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib11_imap_quota_plugin.so >>>> Apr 19 10:23:52 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:23:52 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:23:52 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:23:52 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:23:52 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib95_imap_stats_plugin.so >>>> Apr 19 10:23:52 imap(NewUser): Debug: Effective uid=992, gid=989, >>>> home=/var/vmail/NewUser/home >>>> Apr 19 10:23:52 imap(NewUser): Debug: Quota root: name=User quota >>>> backend=maildir args>>>> Apr 19 10:23:52 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:52 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:52 imap(NewUser): Debug: Quota warning: bytes=20401094656 >>>> (95%) messages=0 reverse=no command=quota-warning 95 NewUser >>>> Apr 19 10:23:52 imap(NewUser): Debug: Quota warning: bytes=17179869184 >>>> (80%) messages=0 reverse=no command=quota-warning 80 NewUser >>>> Apr 19 10:23:52 imap(NewUser): Debug: Quota grace: root=User quota >>>> bytes=2147483648 (10%) >>>> Apr 19 10:23:52 imap(NewUser): Debug: Namespace : type=private, prefix=, >>>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:52 imap(NewUser): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:52 imap(NewUser): Debug: quota: quota_over_flag check: >>>> quota_over_script unset - skipping >>>> Apr 19 10:23:52 imap(NewUser): Debug: INBOX: Mailbox opened because: STATUS >>>> Apr 19 10:23:52 imap(NewUser): Debug: INBOX: Mailbox opened because: SELECT >>>> Apr 19 10:23:53 imap(NewUser): Debug: Loading modules from directory: >>>> /usr/local/lib/dovecot >>>> Apr 19 10:23:53 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:23:53 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib11_imap_quota_plugin.so >>>> Apr 19 10:23:53 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:23:53 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:23:53 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:23:53 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:23:53 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib95_imap_stats_plugin.so >>>> Apr 19 10:23:53 imap(NewUser): Debug: Effective uid=992, gid=989, >>>> home=/var/vmail/NewUser/home >>>> Apr 19 10:23:53 imap(NewUser): Debug: Quota root: name=User quota >>>> backend=maildir args>>>> Apr 19 10:23:53 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:53 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:53 imap(NewUser): Debug: Quota warning: bytes=20401094656 >>>> (95%) messages=0 reverse=no command=quota-warning 95 NewUser >>>> Apr 19 10:23:53 imap(NewUser): Debug: Quota warning: bytes=17179869184 >>>> (80%) messages=0 reverse=no command=quota-warning 80 NewUser >>>> Apr 19 10:23:53 imap(NewUser): Debug: Quota grace: root=User quota >>>> bytes=2147483648 (10%) >>>> Apr 19 10:23:53 imap(NewUser): Debug: Namespace : type=private, prefix=, >>>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:53 imap(NewUser): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:53 imap(NewUser): Debug: quota: quota_over_flag check: >>>> quota_over_script unset - skipping >>>> Apr 19 10:23:53 imap(NewUser): Debug: INBOX: Mailbox opened because: SELECT >>>> Apr 19 10:23:53 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=3 >>>> because: prefetch >>>> Apr 19 10:23:53 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=3 >>>> because: access >>>> Apr 19 10:23:53 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=3 >>>> because: MIME part >>>> Apr 19 10:23:54 imap(NewUser): Debug: Loading modules from directory: >>>> /usr/local/lib/dovecot >>>> Apr 19 10:23:54 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:23:54 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib11_imap_quota_plugin.so >>>> Apr 19 10:23:54 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:23:54 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:23:54 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:23:54 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:23:54 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib95_imap_stats_plugin.so >>>> Apr 19 10:23:54 imap(NewUser): Debug: Effective uid=992, gid=989, >>>> home=/var/vmail/NewUser/home >>>> Apr 19 10:23:54 imap(NewUser): Debug: Quota root: name=User quota >>>> backend=maildir args>>>> Apr 19 10:23:54 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:54 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:54 imap(NewUser): Debug: Quota warning: bytes=20401094656 >>>> (95%) messages=0 reverse=no command=quota-warning 95 NewUser >>>> Apr 19 10:23:54 imap(NewUser): Debug: Quota warning: bytes=17179869184 >>>> (80%) messages=0 reverse=no command=quota-warning 80 NewUser >>>> Apr 19 10:23:54 imap(NewUser): Debug: Quota grace: root=User quota >>>> bytes=2147483648 (10%) >>>> Apr 19 10:23:54 imap(NewUser): Debug: Namespace : type=private, prefix=, >>>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:54 imap(NewUser): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:54 imap(NewUser): Debug: quota: quota_over_flag check: >>>> quota_over_script unset - skipping >>>> Apr 19 10:23:54 imap(NewUser): Debug: INBOX: Mailbox opened because: SELECT >>>> Apr 19 10:23:54 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=4 >>>> because: prefetch >>>> Apr 19 10:23:54 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=4 >>>> because: access >>>> Apr 19 10:23:54 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=4 >>>> because: MIME part >>>> Apr 19 10:23:55 imap(NewUser): Debug: Loading modules from directory: >>>> /usr/local/lib/dovecot >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib11_imap_quota_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib95_imap_stats_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Effective uid=992, gid=989, >>>> home=/var/vmail/NewUser/home >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota root: name=User quota >>>> backend=maildir args>>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota warning: bytes=20401094656 >>>> (95%) messages=0 reverse=no command=quota-warning 95 NewUser >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota warning: bytes=17179869184 >>>> (80%) messages=0 reverse=no command=quota-warning 80 NewUser >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota grace: root=User quota >>>> bytes=2147483648 (10%) >>>> Apr 19 10:23:55 imap(NewUser): Debug: Namespace : type=private, prefix=, >>>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:55 imap(NewUser): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:55 imap(NewUser): Debug: quota: quota_over_flag check: >>>> quota_over_script unset - skipping >>>> Apr 19 10:23:55 imap(NewUser): Debug: INBOX: Mailbox opened because: SELECT >>>> Apr 19 10:23:55 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=5 >>>> because: prefetch >>>> Apr 19 10:23:55 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=5 >>>> because: access >>>> Apr 19 10:23:55 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=5 >>>> because: MIME part >>>> Apr 19 10:23:55 imap(NewUser): Debug: Loading modules from directory: >>>> /usr/local/lib/dovecot >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib11_imap_quota_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib95_imap_stats_plugin.so >>>> Apr 19 10:23:55 imap(NewUser): Debug: Effective uid=992, gid=989, >>>> home=/var/vmail/NewUser/home >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota root: name=User quota >>>> backend=maildir args>>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota warning: bytes=20401094656 >>>> (95%) messages=0 reverse=no command=quota-warning 95 NewUser >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota warning: bytes=17179869184 >>>> (80%) messages=0 reverse=no command=quota-warning 80 NewUser >>>> Apr 19 10:23:55 imap(NewUser): Debug: Quota grace: root=User quota >>>> bytes=2147483648 (10%) >>>> Apr 19 10:23:55 imap(NewUser): Debug: Namespace : type=private, prefix=, >>>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:55 imap(NewUser): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:55 imap(NewUser): Debug: quota: quota_over_flag check: >>>> quota_over_script unset - skipping >>>> Apr 19 10:23:55 imap(NewUser): Debug: INBOX: Mailbox opened because: SELECT >>>> Apr 19 10:23:55 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=6 >>>> because: prefetch >>>> Apr 19 10:23:55 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=6 >>>> because: access >>>> Apr 19 10:23:55 imap(NewUser): Debug: Mailbox INBOX: Opened mail UID=6 >>>> because: MIME part >>>> Apr 19 10:23:57 imap(NewUser): Debug: Loading modules from directory: >>>> /usr/local/lib/dovecot >>>> Apr 19 10:23:57 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:23:57 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib11_imap_quota_plugin.so >>>> Apr 19 10:23:57 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:23:57 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:23:57 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:23:57 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:23:57 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib95_imap_stats_plugin.so >>>> Apr 19 10:23:57 imap(NewUser): Debug: Effective uid=992, gid=989, >>>> home=/var/vmail/NewUser/home >>>> Apr 19 10:23:57 imap(NewUser): Debug: Quota root: name=User quota >>>> backend=maildir args>>>> Apr 19 10:23:57 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:57 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:57 imap(NewUser): Debug: Quota warning: bytes=20401094656 >>>> (95%) messages=0 reverse=no command=quota-warning 95 NewUser >>>> Apr 19 10:23:57 imap(NewUser): Debug: Quota warning: bytes=17179869184 >>>> (80%) messages=0 reverse=no command=quota-warning 80 NewUser >>>> Apr 19 10:23:57 imap(NewUser): Debug: Quota grace: root=User quota >>>> bytes=2147483648 (10%) >>>> Apr 19 10:23:57 imap(NewUser): Debug: Namespace : type=private, prefix=, >>>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:57 imap(NewUser): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:57 imap(NewUser): Debug: quota: quota_over_flag check: >>>> quota_over_script unset - skipping >>>> Apr 19 10:23:57 imap(NewUser): Debug: INBOX: Mailbox opened because: STATUS >>>> Apr 19 10:23:57 imap(NewUser): Debug: INBOX: Mailbox opened because: SELECT >>>> Apr 19 10:23:59 imap(NewUser): Debug: Loading modules from directory: >>>> /usr/local/lib/dovecot >>>> Apr 19 10:23:59 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:23:59 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib11_imap_quota_plugin.so >>>> Apr 19 10:23:59 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:23:59 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:23:59 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:23:59 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:23:59 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib95_imap_stats_plugin.so >>>> Apr 19 10:23:59 imap(NewUser): Debug: Effective uid=992, gid=989, >>>> home=/var/vmail/NewUser/home >>>> Apr 19 10:23:59 imap(NewUser): Debug: Quota root: name=User quota >>>> backend=maildir args>>>> Apr 19 10:23:59 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:23:59 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:23:59 imap(NewUser): Debug: Quota warning: bytes=20401094656 >>>> (95%) messages=0 reverse=no command=quota-warning 95 NewUser >>>> Apr 19 10:23:59 imap(NewUser): Debug: Quota warning: bytes=17179869184 >>>> (80%) messages=0 reverse=no command=quota-warning 80 NewUser >>>> Apr 19 10:23:59 imap(NewUser): Debug: Quota grace: root=User quota >>>> bytes=2147483648 (10%) >>>> Apr 19 10:23:59 imap(NewUser): Debug: Namespace : type=private, prefix=, >>>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:23:59 imap(NewUser): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:23:59 imap(NewUser): Debug: quota: quota_over_flag check: >>>> quota_over_script unset - skipping >>>> Apr 19 10:23:59 imap(NewUser): Debug: INBOX: Mailbox opened because: SELECT >>>> Apr 19 10:24:07 imap(NewUser): Debug: Loading modules from directory: >>>> /usr/local/lib/dovecot >>>> Apr 19 10:24:07 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib10_quota_plugin.so >>>> Apr 19 10:24:07 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib11_imap_quota_plugin.so >>>> Apr 19 10:24:07 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_fts_plugin.so >>>> Apr 19 10:24:07 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib20_virtual_plugin.so >>>> Apr 19 10:24:07 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so >>>> Apr 19 10:24:07 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib90_stats_plugin.so >>>> Apr 19 10:24:07 imap(NewUser): Debug: Module loaded: >>>> /usr/local/lib/dovecot/lib95_imap_stats_plugin.so >>>> Apr 19 10:24:07 imap(NewUser): Debug: Effective uid=992, gid=989, >>>> home=/var/vmail/NewUser/home >>>> Apr 19 10:24:07 imap(NewUser): Debug: Quota root: name=User quota >>>> backend=maildir args>>>> Apr 19 10:24:07 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=* bytes=21474836480 messages=0 >>>> Apr 19 10:24:07 imap(NewUser): Debug: Quota rule: root=User quota >>>> mailbox=Trash bytes=+209715200 messages=0 >>>> Apr 19 10:24:07 imap(NewUser): Debug: Quota warning: bytes=20401094656 >>>> (95%) messages=0 reverse=no command=quota-warning 95 NewUser >>>> Apr 19 10:24:07 imap(NewUser): Debug: Quota warning: bytes=17179869184 >>>> (80%) messages=0 reverse=no command=quota-warning 80 NewUser >>>> Apr 19 10:24:07 imap(NewUser): Debug: Quota grace: root=User quota >>>> bytes=2147483648 (10%) >>>> Apr 19 10:24:07 imap(NewUser): Debug: Namespace : type=private, prefix=, >>>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>> location=maildir:/var/vmail/NewUser >>>> Apr 19 10:24:07 imap(NewUser): Debug: maildir++: >>>> root=/var/vmail/NewUser, index=, indexpvt=, control=, >>>> inbox=/var/vmail/NewUser, alt>>>> Apr 19 10:24:07 imap(NewUser): Debug: quota: quota_over_flag check: >>>> quota_over_script unset - skipping >>>> Apr 19 10:24:07 imap(NewUser): Debug: INBOX: Mailbox opened because: STATUS >>>> Apr 19 10:24:07 imap(NewUser): Debug: INBOX: Mailbox opened because: SELECT >>>> >>>> >
kadafax at gmail.com
2018-May-21 11:11 UTC
Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
Le 21/05/2018 ? 12:38, Aki Tuomi a ?crit?:> can you try turning on pluign { fts_enforced = yes } and repeat your test?Same (wrong) result: 1. Send an email with "too6Ouka" in the body 2. Search against "too6Ouka": # doveadm search -u username mailbox INBOX body too6Ouka --> No result 3. Force re-index: # doveadm fts rescan -u username 4. Search again against "too6Ouka": # doveadm search -u username mailbox INBOX body too6Ouka --> e09cce0283e8695ab760000002deed92 29055 Don't know if relevant, but on a side note, if I send a second message with "too6Ouka" in the body, followed by: # doveadm -v index -u username Inbox --> doveadm(username): Info: INBOX: Cache is already up to date And a search against the pattern immediately return only one result instead of two. # doveconf -n # 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.22 (22940fb7) # OS: Linux 3.10.0-862.2.3.el7.x86_64 x86_64 CentOS Linux release 7.5.1804 (Core)? ext4 # Hostname: mailserver.domain auth_master_user_separator = * auth_socket_path = /var/run/dovecot/auth-userdb auth_username_chars abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@* auth_verbose = yes base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot-debug.log default_client_limit = 5120 default_process_limit = 1024 default_vsz_limit = 2 G hostname = domain info_log_path = /var/log/dovecot.log lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = * log_path = /var/log/dovecot-error.log login_greeting = Welcome to dovecot on mydomain login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c mail_gid = vmail mail_location = maildir:/var/vmail/%Ln mail_plugins = " quota stats fts fts_solr" mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { ? inbox = yes ? location = maildir:/var/vmail/%Ln ? mailbox Drafts { ??? auto = subscribe ??? special_use = \Drafts ? } ? mailbox Sent { ??? auto = subscribe ??? special_use = \Sent ? } ? mailbox "Sent Messages" { ??? special_use = \Sent ? } ? mailbox Trash { ??? auto = subscribe ??? special_use = \Trash ? } ? prefix ? type = private } passdb { ? args = /usr/local/etc/dovecot/deny-users ? deny = yes ? driver = passwd-file } passdb { ? args = /usr/local/etc/dovecot/master-users ? driver = passwd-file ? master = yes } passdb { ? args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext ? driver = ldap } plugin { ? fts = solr ? fts_autoindex = yes ? fts_enforced = yes ? fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search ? quota = maildir:User quota ? quota_rule = *:storage=20GB ? quota_rule2 = Trash:storage=+200M ? quota_warning = storage=95%% quota-warning 95 %u ? quota_warning2 = storage=80%% quota-warning 80 %u ? sieve = /var/sieve-scripts/%Ln.sieve ? sieve_dir = /var/vmail/%Ln/home/sieve ? stats_command_min_time = 1 mins ? stats_domain_min_time = 12 hours ? stats_ip_min_time = 12 hours ? stats_memory_limit = 16 M ? stats_refresh = 30 secs ? stats_session_min_time = 15 mins ? stats_track_cmds = yes ? stats_user_min_time = 1 hours } postmaster_address = postmaster at domain protocols = imap pop3 sieve sieve service auth-worker { ? user = root } service auth { ? unix_listener /var/spool/postfix/private/auth { ??? mode = 0777 ??? user = vmail ? } } service imap-login { ? inet_listener imaps { ??? port = 993 ??? ssl = yes ? } ? process_min_avail = 10 } service imap { ? process_limit = 2128 } service managesieve-login { ? inet_listener sieve { ??? port = 4190 ? } } service pop3-login { ? inet_listener pop3s { ??? port = 995 ??? ssl = yes ? } } service quota-warning { ? executable = script /usr/local/bin/quota-warning.sh ? user = nobody } service stats { ? fifo_listener stats-mail { ??? mode = 0600 ??? user = vmail ? } } ssl_cert = </etc/letsencrypt/live/mailserver.domain/fullchain.pem ssl_key =? # hidden, use -P to show it userdb { ? args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext ? driver = ldap ? override_fields = uid=vmail gid=vmail home=/var/vmail/%Ln/home } verbose_proctitle = yes protocol lda { ? mail_plugins = " quota stats fts fts_solr sieve virtual" } protocol imap { ? mail_max_userip_connections = 20 ? mail_plugins = " quota stats fts fts_solr imap_quota virtual imap_stats" } protocol sieve { ? mail_max_userip_connections = 20 }> > Aki > > > On 21.05.2018 13:31, kadafax at gmail.com wrote: >> Le 21/05/2018 ? 11:46, Aki Tuomi a ?crit?: >>> That message is coming from the temporary raw user, not from the actual >>> user. >>> >>> How did you verify that the FTS indexes are not created or updated >>> automatically? >> I send an email containing a specific strings ('thahB8ea' in that case) >> and search agains it: >> # doveadm search -u username mailbox INBOX body thahB8ea >> --> No result >> (same with integrated search on roundcube) >> >> Then I force a rescan, followed by the same search: >> # doveadm? fts rescan -u username >> # doveadm search -u username mailbox INBOX body thahB8ea >> (Long waiting time) >> --> e09cce0283e8695ab760000002deed92 29048 >> >> So I conclude that autoindexing is not working. >> >> Thanks, >> kfx >>
kfx
2018-May-24 12:22 UTC
Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
After seeing that dovecot 2.2.36 was out and that:> - fts: Fixes to indexing mails via virtual mailboxes. > - fts: If mails contained NUL characters, the text around it wasn't > indexed.I've upgraded but the problem still persist :( On 21/05/2018 13:11, kadafax at gmail.com wrote:> Le 21/05/2018 ? 12:38, Aki Tuomi a ?crit?: >> can you try turning on pluign { fts_enforced = yes } and repeat your test? > > Same (wrong) result: > 1. Send an email with "too6Ouka" in the body > > 2. Search against "too6Ouka": > # doveadm search -u username mailbox INBOX body too6Ouka > --> No result > > 3. Force re-index: > # doveadm fts rescan -u username > > 4. Search again against "too6Ouka": > # doveadm search -u username mailbox INBOX body too6Ouka > --> e09cce0283e8695ab760000002deed92 29055 > > Don't know if relevant, but on a side note, if I send a second message > with "too6Ouka" in the body, followed by: > # doveadm -v index -u username Inbox > --> doveadm(username): Info: INBOX: Cache is already up to date > And a search against the pattern immediately return only one result > instead of two. > > # doveconf -n > # 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf > # Pigeonhole version 0.4.22 (22940fb7) > # OS: Linux 3.10.0-862.2.3.el7.x86_64 x86_64 CentOS Linux release > 7.5.1804 (Core)? ext4 > # Hostname: mailserver.domain > auth_master_user_separator = * > auth_socket_path = /var/run/dovecot/auth-userdb > auth_username_chars > abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@* > auth_verbose = yes > base_dir = /var/run/dovecot/ > debug_log_path = /var/log/dovecot-debug.log > default_client_limit = 5120 > default_process_limit = 1024 > default_vsz_limit = 2 G > hostname = domain > info_log_path = /var/log/dovecot.log > lda_mailbox_autocreate = yes > lda_mailbox_autosubscribe = yes > listen = * > log_path = /var/log/dovecot-error.log > login_greeting = Welcome to dovecot on mydomain > login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c > mail_gid = vmail > mail_location = maildir:/var/vmail/%Ln > mail_plugins = " quota stats fts fts_solr" > mail_uid = vmail > managesieve_notify_capability = mailto > managesieve_sieve_capability = fileinto reject envelope > encoded-character vacation subaddress comparator-i;ascii-numeric > relational regex imap4flags copy include variables body enotify > environment mailbox date index ihave duplicate mime foreverypart extracttext > namespace { > ? inbox = yes > ? location = maildir:/var/vmail/%Ln > ? mailbox Drafts { > ??? auto = subscribe > ??? special_use = \Drafts > ? } > ? mailbox Sent { > ??? auto = subscribe > ??? special_use = \Sent > ? } > ? mailbox "Sent Messages" { > ??? special_use = \Sent > ? } > ? mailbox Trash { > ??? auto = subscribe > ??? special_use = \Trash > ? } > ? prefix > ? type = private > } > passdb { > ? args = /usr/local/etc/dovecot/deny-users > ? deny = yes > ? driver = passwd-file > } > passdb { > ? args = /usr/local/etc/dovecot/master-users > ? driver = passwd-file > ? master = yes > } > passdb { > ? args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext > ? driver = ldap > } > plugin { > ? fts = solr > ? fts_autoindex = yes > ? fts_enforced = yes > ? fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search > ? quota = maildir:User quota > ? quota_rule = *:storage=20GB > ? quota_rule2 = Trash:storage=+200M > ? quota_warning = storage=95%% quota-warning 95 %u > ? quota_warning2 = storage=80%% quota-warning 80 %u > ? sieve = /var/sieve-scripts/%Ln.sieve > ? sieve_dir = /var/vmail/%Ln/home/sieve > ? stats_command_min_time = 1 mins > ? stats_domain_min_time = 12 hours > ? stats_ip_min_time = 12 hours > ? stats_memory_limit = 16 M > ? stats_refresh = 30 secs > ? stats_session_min_time = 15 mins > ? stats_track_cmds = yes > ? stats_user_min_time = 1 hours > } > postmaster_address = postmaster at domain > protocols = imap pop3 sieve sieve > service auth-worker { > ? user = root > } > service auth { > ? unix_listener /var/spool/postfix/private/auth { > ??? mode = 0777 > ??? user = vmail > ? } > } > service imap-login { > ? inet_listener imaps { > ??? port = 993 > ??? ssl = yes > ? } > ? process_min_avail = 10 > } > service imap { > ? process_limit = 2128 > } > service managesieve-login { > ? inet_listener sieve { > ??? port = 4190 > ? } > } > service pop3-login { > ? inet_listener pop3s { > ??? port = 995 > ??? ssl = yes > ? } > } > service quota-warning { > ? executable = script /usr/local/bin/quota-warning.sh > ? user = nobody > } > service stats { > ? fifo_listener stats-mail { > ??? mode = 0600 > ??? user = vmail > ? } > } > ssl_cert = </etc/letsencrypt/live/mailserver.domain/fullchain.pem > ssl_key =? # hidden, use -P to show it > userdb { > ? args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext > ? driver = ldap > ? override_fields = uid=vmail gid=vmail home=/var/vmail/%Ln/home > } > verbose_proctitle = yes > protocol lda { > ? mail_plugins = " quota stats fts fts_solr sieve virtual" > } > protocol imap { > ? mail_max_userip_connections = 20 > ? mail_plugins = " quota stats fts fts_solr imap_quota virtual imap_stats" > } > protocol sieve { > ? mail_max_userip_connections = 20 > } > > > >> >> Aki >> >> >> On 21.05.2018 13:31, kadafax at gmail.com wrote: >>> Le 21/05/2018 ? 11:46, Aki Tuomi a ?crit?: >>>> That message is coming from the temporary raw user, not from the actual >>>> user. >>>> >>>> How did you verify that the FTS indexes are not created or updated >>>> automatically? >>> I send an email containing a specific strings ('thahB8ea' in that case) >>> and search agains it: >>> # doveadm search -u username mailbox INBOX body thahB8ea >>> --> No result >>> (same with integrated search on roundcube) >>> >>> Then I force a rescan, followed by the same search: >>> # doveadm? fts rescan -u username >>> # doveadm search -u username mailbox INBOX body thahB8ea >>> (Long waiting time) >>> --> e09cce0283e8695ab760000002deed92 29048 >>> >>> So I conclude that autoindexing is not working. >>> >>> Thanks, >>> kfx >>>
Timo Sirainen
2018-May-24 13:33 UTC
Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
On 21 May 2018, at 14.11, kadafax at gmail.com wrote:> > Le 21/05/2018 ? 12:38, Aki Tuomi a ?crit : >> can you try turning on pluign { fts_enforced = yes } and repeat your test? > > Same (wrong) result: > 1. Send an email with "too6Ouka" in the body > > 2. Search against "too6Ouka": > # doveadm search -u username mailbox INBOX body too6Ouka > --> No result > > 3. Force re-index: > # doveadm fts rescan -u username > > 4. Search again against "too6Ouka": > # doveadm search -u username mailbox INBOX body too6Ouka > --> e09cce0283e8695ab760000002deed92 29055 > > Don't know if relevant, but on a side note, if I send a second message > with "too6Ouka" in the body, followed by: > # doveadm -v index -u username Inbox > --> doveadm(username): Info: INBOX: Cache is already up to date > And a search against the pattern immediately return only one result > instead of two.So it looks like after "doveadm fts rescan" you can do initial indexing successfully. Afterwards the index isn't updated at all, unless you rebuild it entirely. fts_autoindex=yes appears to work as expected, because after fts rescan you did a search which automatically updated the index. So this doesn't have anything to do with fts_autoindex, just that your index updates don't seem to work at all. What does it show in the header if you do: doveadm dump /var/vmail/username/dovecot.index Especially what is in the "fts" header vs. "next uid" header? Does the UID in "fts" header keep changing every time you save a new mail? I suppose it will. You could also monitor (e.g. tcpdump/wireshark) the network traffic between Dovecot <-> Solr what happens when a new mail arrives. I suspect Dovecot sends it to Solr, which for whatever reason just ignores the update.
Seemingly Similar Threads
- Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
- recover missing messages - files still present in storage
- Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
- Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
- Flags in public folders disappear when more than 25 flags are used