dovecot - Nov 2009 - Virtual mailboxes not always up to date?

Hi,

I have been experimenting with the Virtual mailboxes plugin [1] recently.

This is my setup: the following dovecot-virtual file exists in a virtual
mailbox called "todo":

------ dovecot-virtual ------
*
-Trash
-Trash/*
-Spam
-Spam/*
  OR KEYWORD $TODO KEYWORD todo
-----------------------------

so if any mail in any mailbox except Trash+Spam has the keyword
"$TODO"
(that's what KMail sets when you mark a message as "action item")
or "todo"
(that's what Thunderbird sets when you put the Todo tag on a message), it
should be shown in that 'todo' virtual mailbox.

This works for the first time after I edit the dovecot-virtual file and
then access the virtual mailbox (all mails I expect to be shown in the todo
mailbox are there) - but further accesses always show the same content,
even though I removed the keyword from one of the mails (and I can confirm
that the keyword has actually been removed by dovecot, since the filename
doesn't contain the letter corresponding to the keyword anymore).

It seems that changing the dovecot-virtual file triggers the execution of
the "search program" - but shouldn't simply accessing the virtual
mailbox
also trigger it? Or is there a (undocumented?) timeout between search
program executions?

Patrick.

[1]: http://wiki.dovecot.org/Plugins/Virtual

# dovecot -n
# 1.2.6: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.22-vs2.2.0.7-gentoo x86_64 Gentoo Base System release
2.0.1
protocols: imaps managesieve
listen: x.x.x.x
ssl: required
ssl_ca_file: /etc/ssl/dovecot/sub.class1.server.ca.crt
ssl_cert_file: /etc/ssl/dovecot/ssl.crt
ssl_key_file: /etc/ssl/dovecot/ssl.key
login_dir: /var/run/dovecot/login
login_executable(default): /usr/libexec/dovecot/imap-login
login_executable(imap): /usr/libexec/dovecot/imap-login
login_executable(managesieve): /usr/libexec/dovecot/managesieve-login
mail_max_userip_connections(default): 20
mail_max_userip_connections(imap): 20
mail_max_userip_connections(managesieve): 10
mail_location: maildir:~/.maildir
mail_executable(default): /usr/libexec/dovecot/imap
mail_executable(imap): /usr/libexec/dovecot/imap
mail_executable(managesieve): /usr/libexec/dovecot/managesieve
mail_process_size: 2048
mail_plugins(default): virtual fts fts_squat
mail_plugins(imap): virtual fts fts_squat
mail_plugins(managesieve):
mail_plugin_dir(default): /usr/lib64/dovecot/imap
mail_plugin_dir(imap): /usr/lib64/dovecot/imap
mail_plugin_dir(managesieve): /usr/lib64/dovecot/managesieve
imap_client_workarounds(default): tb-extra-mailbox-sep
imap_client_workarounds(imap): tb-extra-mailbox-sep
imap_client_workarounds(managesieve):
namespace:
  type: private
  separator: .
  inbox: yes
  list: yes
  subscriptions: yes
namespace:
  type: private
  separator: .
  prefix: virtual.
  location: virtual:~/.maildir/virtual
  list: yes
  subscriptions: yes
lda:
  postmaster_address: postmaster at x
  mail_plugins: sieve virtual
auth default:
  passdb:
    driver: pam
    args: *
  userdb:
    driver: passwd
  socket:
    type: listen
    client:
      path: /var/spool/postfix/private/auth
      mode: 432
      user: postfix
      group: postfix
plugin:
  fts: squat
  fts_squat: partial=4 full=10

-- 
STAR Software (Shanghai) Co., Ltd.              http://www.star-group.net/
Phone:    +86 (21) 3462 7688 x 826               Fax:   +86 (21) 3462 7779

PGP key:  E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc
Fingerprint:             E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005

On Mon, 09 Nov 2009 08:03:59 +0800
Patrick Nagel <patrick.nagel at star-group.net> wrote:
> Hi,
> 
> I have been experimenting with the Virtual mailboxes plugin [1] recently.
> 
> This is my setup: the following dovecot-virtual file exists in a virtual
> mailbox called "todo":
> 
> ------ dovecot-virtual ------
> *
> -Trash
> -Trash/*
> -Spam
> -Spam/*
>   OR KEYWORD $TODO KEYWORD todo
> -----------------------------
> 
> so if any mail in any mailbox except Trash+Spam has the keyword
"$TODO"
> (that's what KMail sets when you mark a message as "action
item") or "todo"
> (that's what Thunderbird sets when you put the Todo tag on a message),
it
> should be shown in that 'todo' virtual mailbox.
> 
> This works for the first time after I edit the dovecot-virtual file and
> then access the virtual mailbox (all mails I expect to be shown in the todo
> mailbox are there) - but further accesses always show the same content,
> even though I removed the keyword from one of the mails (and I can confirm
> that the keyword has actually been removed by dovecot, since the filename
> doesn't contain the letter corresponding to the keyword anymore).
> 
> It seems that changing the dovecot-virtual file triggers the execution of
> the "search program" - but shouldn't simply accessing the
virtual mailbox
> also trigger it? Or is there a (undocumented?) timeout between search
> program executions?
> namespace:
>   type: private
>   separator: .
>   prefix: virtual.
>   location: virtual:~/.maildir/virtual
>   list: yes
>   subscriptions: yes
Try to add :INDEX=MEMORY to location setting.

On Mon, 2009-11-09 at 08:03 +0800, Patrick Nagel wrote:
> This works for the first time after I edit the dovecot-virtual file and
> then access the virtual mailbox (all mails I expect to be shown in the todo
> mailbox are there) - but further accesses always show the same content,
> even though I removed the keyword from one of the mails (and I can confirm
> that the keyword has actually been removed by dovecot, since the filename
> doesn't contain the letter corresponding to the keyword anymore).
What about if you get client to issue EXPUNGE command? New messages
should show up in mailbox automatically, but existing ones aren't
removed unless EXPUNGE is given or mailbox is re-selected.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20091109/0c92fa1d/attachment-0002.bin>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Timo,

On 2009-11-10 01:56, Timo Sirainen wrote:
> On Mon, 2009-11-09 at 08:03 +0800, Patrick Nagel wrote:
>> This works for the first time after I edit the dovecot-virtual file and
>> then access the virtual mailbox (all mails I expect to be shown in the
todo
>> mailbox are there) - but further accesses always show the same content,
>> even though I removed the keyword from one of the mails (and I can
confirm
>> that the keyword has actually been removed by dovecot, since the
filename
>> doesn't contain the letter corresponding to the keyword anymore).
> 
> What about if you get client to issue EXPUNGE command? New messages
> should show up in mailbox automatically, but existing ones aren't
> removed unless EXPUNGE is given or mailbox is re-selected.
Hmm... I did some more testing. Since the Thunderbird 3.0 beta 4 I'm using
right now seems to choose '$label4' as IMAP keyword for the 'To
Do' tag, I
added it to the search command, which is now

  OR (OR KEYWORD $TODO KEYWORD todo) KEYWORD $label4

(Off topic: what a mess, why do they change the keywords all the time? I also
see KMAILTODO, maybe from another version of KMail?)

Anyway, I did the following:
- - Set the '$label4' keyword on two mails in my inbox via Thunderbird
- - Checked the virtual.todo mailbox in Thunderbird - both mails were shown
- - Unset the '$label4' keyword on one of the two mails in my inbox via
  Thunderbird
- - Closed Thunderbird and opened it again (this should definitely re-select the
  virtual.todo inbox, right?)
- - Checked the virtual.todo mailbox in Thunderbird - still both mails were
shown

And then I had the following talk with my dovecot server:

$ /usr/sbin/dovecot --exec-mail imap
* PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT
THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE
UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES
WITHIN CONTEXT=SEARCH] Logged in as patrick
1 select virtual.todo
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $TODO KMAILFORWARDED
KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $WATCHED $IGNORED $label4)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $TODO
KMAILFORWARDED KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $WATCHED $IGNORED
$label4 \*)] Flags permitted.
* 2 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1257723001] UIDs valid
* OK [UIDNEXT 4] Predicted next UID
* OK [HIGHESTMODSEQ 4] Highest
1 OK [READ-WRITE] Select completed.
1 fetch 1:2 flags
* 1 FETCH (FLAGS (\Seen))
* 2 FETCH (FLAGS (\Seen $label4))
1 OK Fetch completed.
1 expunge
1 OK Expunge completed.
1 fetch 1:2 flags
* 1 FETCH (FLAGS (\Seen))
* 2 FETCH (FLAGS (\Seen $label4))
1 OK Fetch completed.
1 select virtual.todo
* OK [CLOSED] Previous mailbox closed.
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $TODO KMAILFORWARDED
KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $WATCHED $IGNORED $label4)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $TODO
KMAILFORWARDED KMAILTODO KMAILWATCHED KMAILIGNORED $FORWARDED $WATCHED $IGNORED
$label4 \*)] Flags permitted.
* 2 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1257723001] UIDs valid
* OK [UIDNEXT 4] Predicted next UID
* OK [HIGHESTMODSEQ 4] Highest
1 OK [READ-WRITE] Select completed.
1 fetch 1:2 flags
* 1 FETCH (FLAGS (\Seen))
* 2 FETCH (FLAGS (\Seen $label4))
1 OK Fetch completed.

So it seems that neither EXPUNGE nor re-SELECT seems to have any
"refreshing"
effect.

Nikita Koshikov told me to "Try to add :INDEX=MEMORY to location
setting".

After my tests above I tried that, and suddenly everything works as expected.
It seems logical to have the index of a virtual folder in memory only, since it
is generated dynamically anyway, and is outdated before it's even written to
disk (at least without some configurable "virtual search result max
age"
parameter). So maybe the easiest solution is, to imply ':INDEX=MEMORY'
when a
namespace location is a virtual folder?

Patrick.

- -- 
STAR Software (Shanghai) Co., Ltd.              http://www.star-group.net/
Phone:    +86 (21) 3462 7688 x 826               Fax:   +86 (21) 3462 7779

PGP key:  E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc
Fingerprint:             E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkr5FGQACgkQ7yMg/OiDoAXY/wCgqJEw6yy3YH3//JBGDDBB+6Pj
FCcAnREwazHZAEqqFyc7iGXIQiCz+mnP
=u6rT
-----END PGP SIGNATURE-----