thr3ads.net - dovecot - antispam plugin pipe script seems not to be called when it "should be" [Aug 2016]

If this information is useful, please help other people find it:
Share via:

Ben Johnson

2016-Aug-19 02:26 UTC

antispam plugin pipe script seems not to be called when it "should be"

Hello,

(Full "doveconf -n" output is at the end of this message.)

I'm attempting to recreate a "recipe" I had used successfully for
quite
some time (in Ubuntu 12.04 LTS), which allowed me to train
SpamAssassin's Bayes database when someone dragged an email message from
one sub-folder of the account to another.

I later tried to re-implement it (in Ubuntu 14.04 LTS), and core-dumps
resulted. I posted to this list about it several times, but nobody was
ever able to help me solve the problem.

I just tried the same (in Ubuntu 16.04 LTS), hoping that the source had
evolved since then, and while I don't see a core-dump, the pipe script
doesn't seem to be called.

I've booby-trapped the pipe script with quite a bit of debugging output,
which should be written to a file, but I don't see any indication that
the pipe script is ever executed.

When logged-in as an IMAP user, and I drag a message from the user's
Inbox into the "Spam" folder, I do see the following activity in the
syslog:

Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(Junk): 0
Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(INBOX): 0
Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(Junk): 0
Aug 18 22:12:15 example.com imap: antispam: mail copy: from trash: 0, to
trash: 0
Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(INBOX): 0
Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(Junk): 1
Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(INBOX): 0
Aug 18 22:12:15 example.com imap: antispam: mail copy: src spam: 0, dst
spam: 1, src unsure: 0
Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
program /bin/bash
Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
program /bin/bash
Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
program parameter 1 /usr/local/bin/sa-learn-pipe.sh
Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
program parameter 2 --spam

But the piped message is never delivered to the target mailbox.
The mail log at /var/log/mail.log does not register any activity when I
move messages (and maybe that's expected).

Here are the contents of the /usr/local/bin/sa-learn-pipe.sh script:

http://pastebin.com/7vm74jmk

The permissions and mode on the script seem appropriate:

$ stat /usr/local/bin/sa-learn-pipe.sh
  File: '/usr/local/bin/sa-learn-pipe.sh'
  Size: 1765            Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 669345      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 5000/   vmail)   Gid: ( 5000/   vmail)
Access: 2016-08-18 19:09:47.307392919 -0400
Modify: 2016-08-18 19:09:31.231391749 -0400
Change: 2016-08-18 19:09:31.231391749 -0400
 Birth: -

What might I be doing incorrectly here?

Please let me know if I can provide any other information.

Thank you in advance,

-Ben



Full "doveconf -n" output:

$ doveconf -n
# 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.13 (7b14904)
# OS: Linux 4.4.0-34-generic x86_64 Ubuntu 16.04.1 LTS
auth_mechanisms = plain login
disable_plaintext_auth = no
listen = *,[::]
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_max_userip_connections = 100
mail_plugins = " quota"
mail_privileged_group = vmail
passdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}
plugin {
  antispam_backend = pipe
  antispam_debug_target = syslog
  antispam_pipe_program = /bin/bash
  antispam_pipe_program_args = /usr/local/bin/sa-learn-pipe.sh
  antispam_pipe_program_notspam_arg = --ham
  antispam_pipe_program_spam_arg = --spam
  antispam_pipe_tmpdir = /tmp
  antispam_spam_pattern_ignorecase = SPAM;JUNK
  antispam_trash_pattern_ignorecase = trash;Deleted *
  antispam_verbose_debug = 1
  quota = dict:user::file:/var/vmail/%d/%n/.quotausage
  quota_rule2 = Trash:storage=+100M
  quota_rule3 = Junk:ignore
  quota_rule4 = INBOX:storage=+100M
  quota_warning = storage=100%% quota-reached 100 %u %d
  quota_warning2 = storage=95%% quota-warning 95 %u %d
  quota_warning3 = storage=80%% quota-warning 80 %u %d
  quota_warning4 = -storage=100%% quota-below below %u %d
  sieve = /var/vmail/%d/%n/.sieve
  sieve_max_redirects = 25
}
postmaster_address = postmaster at example.com
protocols = imap pop3
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0660
    user = postfix
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0600
    user = vmail
  }
  user = root
}
service imap-login {
  client_limit = 1000
  process_limit = 512
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}
service quota-below {
  executable = script /usr/local/bin/quota-below.sh
  unix_listener quota-below {
    group = vmail
    mode = 0666
    user = vmail
  }
  user = vmail
}
service quota-reached {
  executable = script /usr/local/bin/quota-reached.sh
  unix_listener quota-reached {
    group = vmail
    mode = 0666
    user = vmail
  }
  user = vmail
}
service quota-warning {
  executable = script /usr/local/bin/quota-warning.sh
  unix_listener quota-warning {
    group = vmail
    mode = 0666
    user = vmail
  }
  user = vmail
}
ssl_cert = </etc/postfix/smtpd.cert
ssl_key = </etc/postfix/smtpd.key
ssl_protocols = !SSLv2 !SSLv3
userdb {
  driver = prefetch
}
userdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}
protocol imap {
  mail_plugins = quota imap_quota antispam
}
protocol pop3 {
  mail_plugins = quota
  pop3_uidl_format = %08Xu%08Xv
}
protocol lda {
  mail_plugins = sieve quota
}
protocol lmtp {
  mail_plugins = quota sieve
  postmaster_address = webmaster at localhost
}

Ben Johnson

2016-Aug-19 02:43 UTC

head link

antispam plugin pipe script seems not to be called when it "should be"

On 8/18/2016 10:26 PM, Ben Johnson wrote:> Hello,
> 
> (Full "doveconf -n" output is at the end of this message.)
> 
> I'm attempting to recreate a "recipe" I had used successfully
for quite
> some time (in Ubuntu 12.04 LTS), which allowed me to train
> SpamAssassin's Bayes database when someone dragged an email message
from
> one sub-folder of the account to another.
> 
> I later tried to re-implement it (in Ubuntu 14.04 LTS), and core-dumps
> resulted. I posted to this list about it several times, but nobody was
> ever able to help me solve the problem.
> 
> I just tried the same (in Ubuntu 16.04 LTS), hoping that the source had
> evolved since then, and while I don't see a core-dump, the pipe script
> doesn't seem to be called.
> 
> I've booby-trapped the pipe script with quite a bit of debugging
output,
> which should be written to a file, but I don't see any indication that
> the pipe script is ever executed.
> 
> When logged-in as an IMAP user, and I drag a message from the user's
> Inbox into the "Spam" folder, I do see the following activity in
the syslog:
> 
> Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(Junk): 0
> Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(INBOX): 0
> Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(Junk): 0
> Aug 18 22:12:15 example.com imap: antispam: mail copy: from trash: 0, to
> trash: 0
> Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(INBOX): 0
> Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(Junk): 1
> Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(INBOX): 0
> Aug 18 22:12:15 example.com imap: antispam: mail copy: src spam: 0, dst
> spam: 1, src unsure: 0
> Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> program /bin/bash
> Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> program /bin/bash
> Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> program parameter 1 /usr/local/bin/sa-learn-pipe.sh
> Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> program parameter 2 --spam
> 
> But the piped message is never delivered to the target mailbox.
> The mail log at /var/log/mail.log does not register any activity when I
> move messages (and maybe that's expected).
> 
> Here are the contents of the /usr/local/bin/sa-learn-pipe.sh script:
> 
> http://pastebin.com/7vm74jmk
> 
> The permissions and mode on the script seem appropriate:
> 
> $ stat /usr/local/bin/sa-learn-pipe.sh
>   File: '/usr/local/bin/sa-learn-pipe.sh'
>   Size: 1765            Blocks: 8          IO Block: 4096   regular file
> Device: fd01h/64769d    Inode: 669345      Links: 1
> Access: (0755/-rwxr-xr-x)  Uid: ( 5000/   vmail)   Gid: ( 5000/   vmail)
> Access: 2016-08-18 19:09:47.307392919 -0400
> Modify: 2016-08-18 19:09:31.231391749 -0400
> Change: 2016-08-18 19:09:31.231391749 -0400
>  Birth: -
> 
> What might I be doing incorrectly here?
> 
> Please let me know if I can provide any other information.
> 
> Thank you in advance,
> 
> -Ben
> 
> 
> 
> Full "doveconf -n" output:
> 
> $ doveconf -n
> # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.13 (7b14904)
> # OS: Linux 4.4.0-34-generic x86_64 Ubuntu 16.04.1 LTS
> auth_mechanisms = plain login
> disable_plaintext_auth = no
> listen = *,[::]
> log_timestamp = "%Y-%m-%d %H:%M:%S "
> mail_max_userip_connections = 100
> mail_plugins = " quota"
> mail_privileged_group = vmail
> passdb {
>   args = /etc/dovecot/dovecot-sql.conf
>   driver = sql
> }
> plugin {
>   antispam_backend = pipe
>   antispam_debug_target = syslog
>   antispam_pipe_program = /bin/bash
>   antispam_pipe_program_args = /usr/local/bin/sa-learn-pipe.sh
>   antispam_pipe_program_notspam_arg = --ham
>   antispam_pipe_program_spam_arg = --spam
>   antispam_pipe_tmpdir = /tmp
>   antispam_spam_pattern_ignorecase = SPAM;JUNK
>   antispam_trash_pattern_ignorecase = trash;Deleted *
>   antispam_verbose_debug = 1
>   quota = dict:user::file:/var/vmail/%d/%n/.quotausage
>   quota_rule2 = Trash:storage=+100M
>   quota_rule3 = Junk:ignore
>   quota_rule4 = INBOX:storage=+100M
>   quota_warning = storage=100%% quota-reached 100 %u %d
>   quota_warning2 = storage=95%% quota-warning 95 %u %d
>   quota_warning3 = storage=80%% quota-warning 80 %u %d
>   quota_warning4 = -storage=100%% quota-below below %u %d
>   sieve = /var/vmail/%d/%n/.sieve
>   sieve_max_redirects = 25
> }
> postmaster_address = postmaster at example.com
> protocols = imap pop3
> service auth {
>   unix_listener /var/spool/postfix/private/auth {
>     group = postfix
>     mode = 0660
>     user = postfix
>   }
>   unix_listener auth-userdb {
>     group = vmail
>     mode = 0600
>     user = vmail
>   }
>   user = root
> }
> service imap-login {
>   client_limit = 1000
>   process_limit = 512
> }
> service lmtp {
>   unix_listener /var/spool/postfix/private/dovecot-lmtp {
>     group = postfix
>     mode = 0600
>     user = postfix
>   }
> }
> service quota-below {
>   executable = script /usr/local/bin/quota-below.sh
>   unix_listener quota-below {
>     group = vmail
>     mode = 0666
>     user = vmail
>   }
>   user = vmail
> }
> service quota-reached {
>   executable = script /usr/local/bin/quota-reached.sh
>   unix_listener quota-reached {
>     group = vmail
>     mode = 0666
>     user = vmail
>   }
>   user = vmail
> }
> service quota-warning {
>   executable = script /usr/local/bin/quota-warning.sh
>   unix_listener quota-warning {
>     group = vmail
>     mode = 0666
>     user = vmail
>   }
>   user = vmail
> }
> ssl_cert = </etc/postfix/smtpd.cert
> ssl_key = </etc/postfix/smtpd.key
> ssl_protocols = !SSLv2 !SSLv3
> userdb {
>   driver = prefetch
> }
> userdb {
>   args = /etc/dovecot/dovecot-sql.conf
>   driver = sql
> }
> protocol imap {
>   mail_plugins = quota imap_quota antispam
> }
> protocol pop3 {
>   mail_plugins = quota
>   pop3_uidl_format = %08Xu%08Xv
> }
> protocol lda {
>   mail_plugins = sieve quota
> }
> protocol lmtp {
>   mail_plugins = quota sieve
>   postmaster_address = webmaster at localhost
> }
> 
If it's helpful, I've tried replacing the pipe script's contents
with
the following:

#####################################################################
#!/bin/bash

/usr/lib/dovecot/deliver -d "sa-training at example.com" -m
"Training.$mode" -p /var/vmail/gtube.txt

# Exit with "success" status code.
exit 0
#####################################################################

And still, the message never arrives when I drag a message from
"Inbox"
to "Spam" in the IMAP client, despite the messages logged to syslog,
which "all seem to be correct".

Yet, if I do this using the above content for the script, the "gtube"
message is delivered successfully:

# su vmail
$ /usr/local/bin/sa-learn-pipe-test.sh

The message arrives in the Inbox, presumably because -m
"Training.$mode"
does not "resolve" to anything, and per the documentation, the
delivery
destination will default to the Inbox in such cases.

What's wrong with my the custom pipe script to which I posted a link in
my previous message, I wonder?

Thanks again for any assistance,

-Ben

Edgar Pettijohn

2016-Aug-19 02:58 UTC

head link

antispam plugin pipe script seems not to be called when it "should be"

On 16-08-18 22:43:32, Ben Johnson wrote:> On 8/18/2016 10:26 PM, Ben Johnson wrote:
> > Hello,
> > 
> > (Full "doveconf -n" output is at the end of this message.)
> > 
> > I'm attempting to recreate a "recipe" I had used
successfully for quite
> > some time (in Ubuntu 12.04 LTS), which allowed me to train
> > SpamAssassin's Bayes database when someone dragged an email
message from
> > one sub-folder of the account to another.
> > 
> > I later tried to re-implement it (in Ubuntu 14.04 LTS), and core-dumps
> > resulted. I posted to this list about it several times, but nobody was
> > ever able to help me solve the problem.
> > 
> > I just tried the same (in Ubuntu 16.04 LTS), hoping that the source
had
> > evolved since then, and while I don't see a core-dump, the pipe
script
> > doesn't seem to be called.
> > 
> > I've booby-trapped the pipe script with quite a bit of debugging
output,
> > which should be written to a file, but I don't see any indication
that
> > the pipe script is ever executed.
> > 
> > When logged-in as an IMAP user, and I drag a message from the
user's
> > Inbox into the "Spam" folder, I do see the following
activity in the syslog:
> > 
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(Junk): 0
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(INBOX): 0
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(Junk): 0
> > Aug 18 22:12:15 example.com imap: antispam: mail copy: from trash: 0,
to
> > trash: 0
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(INBOX): 0
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(Junk): 1
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(INBOX):
0
> > Aug 18 22:12:15 example.com imap: antispam: mail copy: src spam: 0,
dst
> > spam: 1, src unsure: 0
> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> > program /bin/bash
> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> > program /bin/bash
> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> > program parameter 1 /usr/local/bin/sa-learn-pipe.sh
> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> > program parameter 2 --spam
> > 
> > But the piped message is never delivered to the target mailbox.
> > The mail log at /var/log/mail.log does not register any activity when
I
> > move messages (and maybe that's expected).
> > 
> > Here are the contents of the /usr/local/bin/sa-learn-pipe.sh script:
> > 
> > http://pastebin.com/7vm74jmk
> > 
> > The permissions and mode on the script seem appropriate:
> > 
> > $ stat /usr/local/bin/sa-learn-pipe.sh
> >   File: '/usr/local/bin/sa-learn-pipe.sh'
> >   Size: 1765            Blocks: 8          IO Block: 4096   regular
file
> > Device: fd01h/64769d    Inode: 669345      Links: 1
> > Access: (0755/-rwxr-xr-x)  Uid: ( 5000/   vmail)   Gid: ( 5000/  
vmail)
> > Access: 2016-08-18 19:09:47.307392919 -0400
> > Modify: 2016-08-18 19:09:31.231391749 -0400
> > Change: 2016-08-18 19:09:31.231391749 -0400
> >  Birth: -
> > 
> > What might I be doing incorrectly here?
> > 
> > Please let me know if I can provide any other information.
> > 
> > Thank you in advance,
> > 
> > -Ben
> > 
> > 
> > 
> > Full "doveconf -n" output:
> > 
> > $ doveconf -n
> > # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
> > # Pigeonhole version 0.4.13 (7b14904)
> > # OS: Linux 4.4.0-34-generic x86_64 Ubuntu 16.04.1 LTS
> > auth_mechanisms = plain login
> > disable_plaintext_auth = no
> > listen = *,[::]
> > log_timestamp = "%Y-%m-%d %H:%M:%S "
> > mail_max_userip_connections = 100
> > mail_plugins = " quota"
> > mail_privileged_group = vmail
> > passdb {
> >   args = /etc/dovecot/dovecot-sql.conf
> >   driver = sql
> > }
> > plugin {
> >   antispam_backend = pipe
> >   antispam_debug_target = syslog
> >   antispam_pipe_program = /bin/bash
> >   antispam_pipe_program_args = /usr/local/bin/sa-learn-pipe.sh
> >   antispam_pipe_program_notspam_arg = --ham
> >   antispam_pipe_program_spam_arg = --spam
> >   antispam_pipe_tmpdir = /tmp
> >   antispam_spam_pattern_ignorecase = SPAM;JUNK
> >   antispam_trash_pattern_ignorecase = trash;Deleted *
> >   antispam_verbose_debug = 1
> >   quota = dict:user::file:/var/vmail/%d/%n/.quotausage
> >   quota_rule2 = Trash:storage=+100M
> >   quota_rule3 = Junk:ignore
> >   quota_rule4 = INBOX:storage=+100M
> >   quota_warning = storage=100%% quota-reached 100 %u %d
> >   quota_warning2 = storage=95%% quota-warning 95 %u %d
> >   quota_warning3 = storage=80%% quota-warning 80 %u %d
> >   quota_warning4 = -storage=100%% quota-below below %u %d
> >   sieve = /var/vmail/%d/%n/.sieve
> >   sieve_max_redirects = 25
> > }
> > postmaster_address = postmaster at example.com
> > protocols = imap pop3
> > service auth {
> >   unix_listener /var/spool/postfix/private/auth {
> >     group = postfix
> >     mode = 0660
> >     user = postfix
> >   }
> >   unix_listener auth-userdb {
> >     group = vmail
> >     mode = 0600
> >     user = vmail
> >   }
> >   user = root
> > }
> > service imap-login {
> >   client_limit = 1000
> >   process_limit = 512
> > }
> > service lmtp {
> >   unix_listener /var/spool/postfix/private/dovecot-lmtp {
> >     group = postfix
> >     mode = 0600
> >     user = postfix
> >   }
> > }
> > service quota-below {
> >   executable = script /usr/local/bin/quota-below.sh
> >   unix_listener quota-below {
> >     group = vmail
> >     mode = 0666
> >     user = vmail
> >   }
> >   user = vmail
> > }
> > service quota-reached {
> >   executable = script /usr/local/bin/quota-reached.sh
> >   unix_listener quota-reached {
> >     group = vmail
> >     mode = 0666
> >     user = vmail
> >   }
> >   user = vmail
> > }
> > service quota-warning {
> >   executable = script /usr/local/bin/quota-warning.sh
> >   unix_listener quota-warning {
> >     group = vmail
> >     mode = 0666
> >     user = vmail
> >   }
> >   user = vmail
> > }
> > ssl_cert = </etc/postfix/smtpd.cert
> > ssl_key = </etc/postfix/smtpd.key
> > ssl_protocols = !SSLv2 !SSLv3
> > userdb {
> >   driver = prefetch
> > }
> > userdb {
> >   args = /etc/dovecot/dovecot-sql.conf
> >   driver = sql
> > }
> > protocol imap {
> >   mail_plugins = quota imap_quota antispam
> > }
> > protocol pop3 {
> >   mail_plugins = quota
> >   pop3_uidl_format = %08Xu%08Xv
> > }
> > protocol lda {
> >   mail_plugins = sieve quota
> > }
> > protocol lmtp {
> >   mail_plugins = quota sieve
> >   postmaster_address = webmaster at localhost
> > }
> > 
> 
> If it's helpful, I've tried replacing the pipe script's
contents with
> the following:
> 
> #####################################################################
> #!/bin/bash
> 
> /usr/lib/dovecot/deliver -d "sa-training at example.com" -m
> "Training.$mode" -p /var/vmail/gtube.txt
> 
> # Exit with "success" status code.
> exit 0
> #####################################################################
> 
> And still, the message never arrives when I drag a message from
"Inbox"
> to "Spam" in the IMAP client, despite the messages logged to
syslog,
> which "all seem to be correct".
> 
> Yet, if I do this using the above content for the script, the
"gtube"
> message is delivered successfully:
> 
> # su vmail
> $ /usr/local/bin/sa-learn-pipe-test.sh
> 
> The message arrives in the Inbox, presumably because -m
"Training.$mode"
> does not "resolve" to anything, and per the documentation, the
delivery
> destination will default to the Inbox in such cases.
> 
> What's wrong with my the custom pipe script to which I posted a link in
> my previous message, I wonder?
> 
> Thanks again for any assistance,
> 
> -Ben
I don't know how to solve your problem, but I have an alternate method.

Create a Junk folder and a ham folder then add the following cron jobs:

30      5       *       *       *       /usr/local/bin/sa-learn --spam
/var/vmail/Maildir/.Junk
30      6       *       *       6       /usr/local/bin/sa-learn --backup >
/etc/mail/spamassassin/spam.db
30      7       *       *       *       /usr/local/bin/sa-learn --ham
/var/vmail/Maildir/.Ham

-- 
Edgar Pettijohn

dovecot - Aug 2016 - antispam plugin pipe script seems not to be called when it "should be"

antispam plugin pipe script seems not to be called when it "should be"

antispam plugin pipe script seems not to be called when it "should be"

antispam plugin pipe script seems not to be called when it "should be"