dovecot - Apr 2016 - fts-lucene with virtual folders

Hi,

It looks like I've found something strange. It looks like dovecot
updates fts-lucene index every time I'm opening virtual folder that
contains FTS query:

indexer-worker(dion): Warning: fts-lucene: Settings have changed, rebuilding
index for mailbox

dovecot-virtual is pretty simple:

	archive/INBOX
		BODY "test"

first of all, I'm trying to perform FTS in archive/INBOX itself, then
I'm
trying to open virtual folder.

both default namespace and 'archive' are private namespaces with mdbox
storage.

plugin {
        fts = lucene
        fts_lucene = whitespace_chars=@.
        fts_autoindex = no
}

Any suggestions?

-- 
WBR, Dmitry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20160412/37dea114/attachment-0001.sig>

On Tue, Apr 12, 2016 at 11:05:08AM +0300, Dmitry Nezhevenko
wrote:
> Hi,
> 
> It looks like I've found something strange. It looks like dovecot
> updates fts-lucene index every time I'm opening virtual folder that
> contains FTS query:
> 
> indexer-worker(dion): Warning: fts-lucene: Settings have changed,
rebuilding index for mailbox
> 
Ok. It seems that it's unrelated to virtual at all. It's enough to just
use any folder from non-inbox private namespace and perform FTS multiple
times. 

I've added a few debug prints around fts_lucene_settings_checksum,
fts_index_have_compatible_settings and fts_index_set_header and fount
that fts_lucene_settings_checksum is always same.

The root issue is that fts_lucene_settings_checksum reads checksum from
mailbox with empty name (probably namespace root mbox). At the same time
fts_index_set_header is called for all mailboxes in namespace except this
'root' mbox. 

That's actually why I'm always getting 'Settings have changed'
warning.

I've tried to create 'archive' mailbox (same as namespace name) and
got
call to fts_index_set_header() for it during scan, but with zero
settings_checksum.

Any suggestions how to fix it?

-- 
WBR, Dmitry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20160412/976aaa66/attachment.sig>

On Tue, Apr 12, 2016 at 11:26:05AM +0300, Dmitry Nezhevenko
wrote:
> On Tue, Apr 12, 2016 at 11:05:08AM +0300, Dmitry Nezhevenko wrote:
> > indexer-worker(dion): Warning: fts-lucene: Settings have changed,
rebuilding index for mailbox
> The root issue is that fts_lucene_settings_checksum reads checksum from
> mailbox with empty name (probably namespace root mbox). At the same time
> fts_index_set_header is called for all mailboxes in namespace except this
> 'root' mbox. 
This proof-of-concept patch fixes issue for me. I don't think that this is
right way to fix it. I've copied vname calculation code from
fts_index_have_compatible_settings. Maybe it's better to create
something like fts_index_write_settings_checksum() in fts-api.

Index: dovecot-2.2.22/src/plugins/fts-lucene/lucene-wrapper.cc
==================================================================---
dovecot-2.2.22.orig/src/plugins/fts-lucene/lucene-wrapper.cc
+++ dovecot-2.2.22/src/plugins/fts-lucene/lucene-wrapper.cc
@@ -832,6 +832,11 @@ static void rescan_clear_unseen_mailboxe
 	struct mailbox_metadata metadata;
 	struct fts_index_header hdr;
 
+	struct mail_namespace *ns;
+	const char* vname;
+	struct fts_index_header hdr_root;
+	unsigned int len;
+
 	memset(&hdr, 0, sizeof(hdr));
 	hdr.settings_checksum = fts_lucene_settings_checksum(&index->set);
 
@@ -852,6 +857,26 @@ static void rescan_clear_unseen_mailboxe
 		mailbox_free(&box);
 	}
 	(void)mailbox_list_iter_deinit(&iter);
+
+	// Make sure we've stored settings checksum for non-INBOX namespaces
+	ns = mailbox_list_get_namespace(index->list);
+	if ((ns->flags & NAMESPACE_FLAG_INBOX_USER) == 0) {
+		len = strlen(ns->prefix);
+		if (len > 0 && ns->prefix[len-1] == mail_namespace_get_sep(ns))
+			len--;
+		vname = t_strndup(ns->prefix, len);
+
+		box = mailbox_alloc(index->list, vname,
+				    (enum mailbox_flags)0);
+		if (mailbox_open(box) == 0 &&
+		    fts_index_get_header(box, &hdr_root)) {
+			if (hdr_root.settings_checksum != hdr.settings_checksum) {
+				hdr_root.settings_checksum = hdr.settings_checksum;
+				(void)fts_index_set_header(box, &hdr_root);
+			}
+		}
+		mailbox_free(&box);
+	}
 }
 
 int lucene_index_rescan(struct lucene_index *index)
 
-- 
WBR, Dmitry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20160412/a00a621a/attachment.sig>