dovecot - Feb 2017 - Replacement for antispam plugin

> On February 10, 2017 at 12:13 PM Ralph Seichter wrote:
> 
> On 10.02.17 18:34, Michael Slusarz wrote:
> > > Can we add an exception for the Trash folder?
> > This is handled in the sieve script. E.g.:
> > 
> > require "environment";
> > if environment "imap.mailbox" "Trash" {
> >  stop;
> > }
> 
> This does not work for me, and I don't really expect it to work either.
> https://tools.ietf.org/html/rfc6785#section-4.4 states:
> 
>  The implementation MUST set the Environment [RFC5183] item
"imap.mailbox"
>  to the name of the mailbox that the affected message is in, in the
>  case of existing messages, or is targeted to be stored into, in the
>  case of new messages.
> 
> The message already exists in the Spam folder, hence imap.mailbox should
> be "Spam" instead of "Trash", correct?
Incorrect.

When you move a message to a new mailbox, that is a "new message"
event (a new UID in the target mailbox is created; the message count
increases).? So imap.mailbox is set to the name of the *target* mailbox.
> Is there perhaps another way to ensure that manually deleted spam is not
> erroneously learned as ham?
> 
> -Ralph

Hello,

Chiming in on this with a question, and will be getting to it over the
weekend or later this evening time permitting.

Does retraining a message as either spam or ham alter message headers
for example x-spam or the spamassassin-modified subject header?

If not is it possible to do so after processing? For example, I have a
message inadvertently tagged as spam, from Spamassassin it gets an
x-spam header added as well as a modified subject. Retraining that
message as ham moving it to say any other folder but spam i'd like for
that x-spam header to be set to as it is not spam, and the
spamassassin subject to be removed. Is this doable?

Thanks.
Dave.


On 2/10/17, Michael Slusarz <michael.slusarz at dovecot.fi>
wrote:
>
>> On February 10, 2017 at 12:13 PM Ralph Seichter wrote:
>>
>> On 10.02.17 18:34, Michael Slusarz wrote:
>> > > Can we add an exception for the Trash folder?
>> > This is handled in the sieve script. E.g.:
>> >
>> > require "environment";
>> > if environment "imap.mailbox" "Trash" {
>> >  stop;
>> > }
>>
>> This does not work for me, and I don't really expect it to work
either.
>> https://tools.ietf.org/html/rfc6785#section-4.4 states:
>>
>>  The implementation MUST set the Environment [RFC5183] item
>> "imap.mailbox"
>>  to the name of the mailbox that the affected message is in, in the
>>  case of existing messages, or is targeted to be stored into, in the
>>  case of new messages.
>>
>> The message already exists in the Spam folder, hence imap.mailbox
should
>> be "Spam" instead of "Trash", correct?
>
> Incorrect.
>
> When you move a message to a new mailbox, that is a "new message"
event (a
> new UID in the target mailbox is created; the message count increases).? So
> imap.mailbox is set to the name of the *target* mailbox.
>
>> Is there perhaps another way to ensure that manually deleted spam is
not
>> erroneously learned as ham?
>>
>> -Ralph
>

On 10.02.17 20:34, Michael Slusarz wrote:
> When you move a message to a new mailbox, that is a "new message"
> event (a new UID in the target mailbox is created; the message count
> increases). So imap.mailbox is set to the name of the *target* mailbox.
My tests seem to indicate otherwise. Deleting a message currently
located in the Junk folder causes the report-ham.sieve script to be
invoked, which in my case contains the following:

  require ["vnd.dovecot.pipe", "copy",
"environment"];
  if environment "imap.mailbox" "Trash" {
    stop;
  } elsif environment "imap.mailbox" "Junk" {
    pipe :copy "debug-junk";
  } else {
    pipe :copy "learn-ham";
  }

I can see that "learn-ham" is always invoked when a message is deleted
from or moved out of the Junk folder, so my guess is that imap.mailbox
is neither "Trash" nor "Junk" ("debug-junk" is
never called, according
to the Dovecot logs). Unfortunately I don't know how to debug this
further.

I don't want "learn-ham" to be run when a message located in the
Junk
folder is manually deleted or moved to Trash, and right now I don't know
how to accomplish this.

-Ralph

On Fri, Feb 10, 2017 at 9:59 PM, Ralph Seichter <dovecot-ml at
seichter.de>
wrote:
> On 10.02.17 20:34, Michael Slusarz wrote:
>
> > When you move a message to a new mailbox, that is a "new
message"
> > event (a new UID in the target mailbox is created; the message count
> > increases). So imap.mailbox is set to the name of the *target*
mailbox.
>
> My tests seem to indicate otherwise. Deleting a message currently
> located in the Junk folder causes the report-ham.sieve script to be
> invoked, which in my case contains the following:
>
>   require ["vnd.dovecot.pipe", "copy",
"environment"];
>   if environment "imap.mailbox" "Trash" {
>     stop;
>   } elsif environment "imap.mailbox" "Junk" {
>     pipe :copy "debug-junk";
>   } else {
>     pipe :copy "learn-ham";
>   }
>
> I can see that "learn-ham" is always invoked when a message is
deleted
> from or moved out of the Junk folder, so my guess is that imap.mailbox
> is neither "Trash" nor "Junk" ("debug-junk"
is never called, according
> to the Dovecot logs). Unfortunately I don't know how to debug this
> further.
>
> I don't want "learn-ham" to be run when a message located in
the Junk
> folder is manually deleted or moved to Trash, and right now I don't
know
> how to accomplish this.
>
> -Ralph
>
Same problem here. As a workaround I tried the following:

# From Spam folder to Inbox
  imapsieve_mailbox2_name = Inbox
  imapsieve_mailbox2_from = Spam
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/usr/lib/dovecot/sieve/report-ham.sieve

In theory this should trigger the  report-ham script, only if a mail is
moved from Spam to Inbox. However, it does not seem to work either.
-- 
George Kontostanos
---

Op 2/10/2017 om 8:46 PM schreef David Mehler:
> Hello,
>
> Chiming in on this with a question, and will be getting to it over the
> weekend or later this evening time permitting.
>
> Does retraining a message as either spam or ham alter message headers
> for example x-spam or the spamassassin-modified subject header?
>
> If not is it possible to do so after processing? For example, I have a
> message inadvertently tagged as spam, from Spamassassin it gets an
> x-spam header added as well as a modified subject. Retraining that
> message as ham moving it to say any other folder but spam i'd like for
> that x-spam header to be set to as it is not spam, and the
> spamassassin subject to be removed. Is this doable?
I think you could use the "editheader" extension in the
report-ham.sieve
script to do that. You'll need "variables" as well to modify the
subject.

Regards,

Stephan.

Op 2/10/2017 om 8:59 PM schreef Ralph Seichter:
> On 10.02.17 20:34, Michael Slusarz wrote:
>
>> When you move a message to a new mailbox, that is a "new
message"
>> event (a new UID in the target mailbox is created; the message count
>> increases). So imap.mailbox is set to the name of the *target* mailbox.
> My tests seem to indicate otherwise. Deleting a message currently
> located in the Junk folder causes the report-ham.sieve script to be
> invoked, which in my case contains the following:
>
>   require ["vnd.dovecot.pipe", "copy",
"environment"];
>   if environment "imap.mailbox" "Trash" {
>     stop;
>   } elsif environment "imap.mailbox" "Junk" {
>     pipe :copy "debug-junk";
>   } else {
>     pipe :copy "learn-ham";
>   }
>
> I can see that "learn-ham" is always invoked when a message is
deleted
> from or moved out of the Junk folder, so my guess is that imap.mailbox
> is neither "Trash" nor "Junk" ("debug-junk"
is never called, according
> to the Dovecot logs). Unfortunately I don't know how to debug this
> further.
>
> I don't want "learn-ham" to be run when a message located in
the Junk
> folder is manually deleted or moved to Trash, and right now I don't
know
> how to accomplish this.
The "imap.mailbox" environment is the empty string in this case. Why?
Well, the Sieve interpreter does not know about it, since the
"imapsieve" extension is not activated in the require line.

You could debug this with the non-standard "vnd.dovecot.debug"
extension. I debugged it like this:

require ["imapsieve", "environment", "variables",
"vnd.dovecot.debug"];

if environment :matches "imap.mailbox" "*" {
        set "mailbox" "${1}";
}

debug_log "imap.mailbox = ${mailbox}";

if string "${mailbox}" "Trash" {
        stop;
}

debug_log "REPORT HAM";

This will log one or two info messages, depending on what the target
mailbox is.

I tested this with the example configuration and it works fine. I've
verified this by invoking the imap service directly from the command
line, so that it is immediately logged-in and spews al debug messages
directly to stderr:

$ sudo /usr/lib/dovecot/imap -u harrie
imap(harrie)<>: Debug: Loading modules from directory:
/usr/lib/dovecot/modules
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib15_notify_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_fts_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_mail_log_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_replication_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_virtual_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so
imap(harrie)<>: Debug: auth USER input: harrie uid=1000 gid=124
home=/ext/test/home/test/harrie
Debug: Effective uid=1000, gid=124, home=/ext/test/home/test/harrie
Debug: replication: No mail_replica setting - replication disabled
Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes,
hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir
Debug: maildir++: root=/ext/test/home/test/harrie/Maildir, index=,
indexpvt=, control=, inbox=/ext/test/home/test/harrie/Maildir, altDebug:
Namespace : type=private, prefix=virtual/, sep=/, inbox=no,
hidden=no, list=yes, subscriptions=yes location=virtual:~/Maildir/virtual
Debug: fs: root=/ext/test/home/test/harrie/Maildir/virtual, index=,
indexpvt=, control=, inbox=, alt* PREAUTH [CAPABILITY IMAP4rev1 SASL-IR
LOGIN-REFERRALS ID ENABLE IDLE
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES
WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE LITERAL+ SEARCH=FUZZY
NOTIFY URLAUTH URLAUTH=BINARY METADATA SPECIAL-USE] Logged in as harrie
23423 SELECT Spam
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: Namespace : Using
permissions from /ext/test/home/test/harrie/Maildir: mode=0700 gid=default
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
Flags permitted.
* 1 EXISTS
* 1 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1485122806] UIDs valid
* OK [UIDNEXT 2] Predicted next UID
23423 OK [READ-WRITE] Select completed (0.000 + 0.000 secs).

This is what happens for moving from Spam to a non-Trash mailbox:

2343 MOVE 1 "Frop"
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: imapsieve: mailbox Frop:
MOVE event
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Pigeonhole version
0.5.devel (462a535) initializing
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: include:
sieve_global is not set; it is currently not possible to include
`:global' scripts.
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Pigeonhole Sieve PGP
Encrypt plugin version 0.2.devel loaded
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Sieve imapsieve
plugin for Pigeonhole version 0.5.devel (462a535) loaded
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: imapsieve: Static mailbox
rule [1]: mailbox=`Spam' from=`*' causes=(COPY) =>
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: imapsieve: Static mailbox
rule [2]: mailbox=`*' from=`Spam' causes=(COPY) =>
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: imapsieve: Matched static
mailbox rule [2]
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage:
Performing auto-detection
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage: Root
exists (/ext/test/home/test/harrie)
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage:
Storage path `/ext/test/home/test/harrie/sieve' not found
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage: Active
script path is unconfigured; using default (path=~/.dovecot.sieve)
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage: Using
Sieve script path: /ext/test/home/test/harrie/.dovecot.sieve
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage:
Storage path `/ext/test/home/test/harrie/.dovecot.sieve' not found
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: storage: No default
script location configured
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage: Using
Sieve script path: /usr/lib/dovecot/sieve/report-ham.sieve
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file script: Opened
script `report-ham' from `/usr/lib/dovecot/sieve/report-ham.sieve'
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Opening script 1 of
1 from `/usr/lib/dovecot/sieve/report-ham.sieve'
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Loading script
/usr/lib/dovecot/sieve/report-ham.sieve
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Script binary
/usr/lib/dovecot/sieve/report-ham.svbin successfully loaded
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: binary save: not
saving binary /usr/lib/dovecot/sieve/report-ham.svbin, because it is
already stored
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Executing script
from `/usr/lib/dovecot/sieve/report-ham.svbin'
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Info: sieve: report-ham: line 11:
DEBUG: imap.mailbox = Frop
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Info: sieve: report-ham: line 17:
DEBUG: REPORT HAM
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Info: copy from Spam: box=Frop,
uid=2, msgid=<201701181731.v0IHVjvo001751 at localhost.localdomain>,
flags=()
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Info: expunge: box=Spam, uid=1,
msgid=<201701181731.v0IHVjvo001751 at localhost.localdomain>,
flags=(\Recent)
* OK [COPYUID 1485122805 1 2] Moved UIDs.
* 1 EXPUNGE
* 0 RECENT
2343 OK Move completed (0.089 + 0.000 + 0.088 secs).

Alternatively, towards Trash:

2343 MOVE 1 Trash
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug:
/ext/test/home/test/harrie/Maildir/.Trash/dovecot.index.cache:
Compressed, file_seq changed 0 -> 1486901076, size=32, max_uid=0
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: imapsieve: mailbox Trash:
MOVE event
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: Mailbox Trash: Opened mail
UID=0 because: header Message-ID (Cache file is empty, reset_id=1486901076)
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: imapsieve: Matched static
mailbox rule [2]
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage:
Performing auto-detection
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage: Root
exists (/ext/test/home/test/harrie)
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage:
Storage path `/ext/test/home/test/harrie/sieve' not found
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage: Active
script path is unconfigured; using default (path=~/.dovecot.sieve)
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage: Using
Sieve script path: /ext/test/home/test/harrie/.dovecot.sieve
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage:
Storage path `/ext/test/home/test/harrie/.dovecot.sieve' not found
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: storage: No default
script location configured
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file storage: Using
Sieve script path: /usr/lib/dovecot/sieve/report-ham.sieve
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: file script: Opened
script `report-ham' from `/usr/lib/dovecot/sieve/report-ham.sieve'
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Opening script 1 of
1 from `/usr/lib/dovecot/sieve/report-ham.sieve'
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Loading script
/usr/lib/dovecot/sieve/report-ham.sieve
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Script binary
/usr/lib/dovecot/sieve/report-ham.svbin successfully loaded
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: binary save: not
saving binary /usr/lib/dovecot/sieve/report-ham.svbin, because it is
already stored
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Debug: sieve: Executing script
from `/usr/lib/dovecot/sieve/report-ham.svbin'
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Info: sieve: report-ham: line 11:
DEBUG: imap.mailbox = Trash
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Info: copy from Spam: box=Trash,
uid=1, msgid=<201701181731.v0IHVjvo001751 at localhost.localdomain>,
flags=()
imap(harrie)</fX4JwpPoFhq2AAAmoV78g>: Info: expunge: box=Spam, uid=2,
msgid=<201701181731.v0IHVjvo001751 at localhost.localdomain>,
flags=(\Recent)
* OK [COPYUID 1485122804 2 1] Moved UIDs.
* 1 EXPUNGE
* 0 RECENT
2343 OK Move completed (0.090 + 0.000 + 0.089 secs).

If you want to debug this in even more detail, you can enable trace
debugging for the targeted user (or the whole server if you really want
to). The imapsieve plugins should obey this as well:

https://github.com/dovecot/pigeonhole/blob/master/INSTALL#L551

This is apparently not currently documented in the wiki.

Regards,

Stephan.