dovecot - Aug 2016 - dovecot-lda core-dumps when antispam pipe script calls it

On 16-08-18 22:43:32, Ben Johnson wrote:
> On 8/18/2016 10:26 PM, Ben Johnson wrote:
> > Hello,
> > 
> > (Full "doveconf -n" output is at the end of this message.)
> > 
> > I'm attempting to recreate a "recipe" I had used
successfully for quite
> > some time (in Ubuntu 12.04 LTS), which allowed me to train
> > SpamAssassin's Bayes database when someone dragged an email
message from
> > one sub-folder of the account to another.
> > 
> > I later tried to re-implement it (in Ubuntu 14.04 LTS), and core-dumps
> > resulted. I posted to this list about it several times, but nobody was
> > ever able to help me solve the problem.
> > 
> > I just tried the same (in Ubuntu 16.04 LTS), hoping that the source
had
> > evolved since then, and while I don't see a core-dump, the pipe
script
> > doesn't seem to be called.
> > 
> > I've booby-trapped the pipe script with quite a bit of debugging
output,
> > which should be written to a file, but I don't see any indication
that
> > the pipe script is ever executed.
> > 
> > When logged-in as an IMAP user, and I drag a message from the
user's
> > Inbox into the "Spam" folder, I do see the following
activity in the syslog:
> > 
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(Junk): 0
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(INBOX): 0
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_trash(Junk): 0
> > Aug 18 22:12:15 example.com imap: antispam: mail copy: from trash: 0,
to
> > trash: 0
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(INBOX): 0
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(Junk): 1
> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_unsure(INBOX):
0
> > Aug 18 22:12:15 example.com imap: antispam: mail copy: src spam: 0,
dst
> > spam: 1, src unsure: 0
> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> > program /bin/bash
> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> > program /bin/bash
> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> > program parameter 1 /usr/local/bin/sa-learn-pipe.sh
> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain backend
> > program parameter 2 --spam
> > 
> > But the piped message is never delivered to the target mailbox.
> > The mail log at /var/log/mail.log does not register any activity when
I
> > move messages (and maybe that's expected).
> > 
> > Here are the contents of the /usr/local/bin/sa-learn-pipe.sh script:
> > 
> > http://pastebin.com/7vm74jmk
> > 
> > The permissions and mode on the script seem appropriate:
> > 
> > $ stat /usr/local/bin/sa-learn-pipe.sh
> >   File: '/usr/local/bin/sa-learn-pipe.sh'
> >   Size: 1765            Blocks: 8          IO Block: 4096   regular
file
> > Device: fd01h/64769d    Inode: 669345      Links: 1
> > Access: (0755/-rwxr-xr-x)  Uid: ( 5000/   vmail)   Gid: ( 5000/  
vmail)
> > Access: 2016-08-18 19:09:47.307392919 -0400
> > Modify: 2016-08-18 19:09:31.231391749 -0400
> > Change: 2016-08-18 19:09:31.231391749 -0400
> >  Birth: -
> > 
> > What might I be doing incorrectly here?
> > 
> > Please let me know if I can provide any other information.
> > 
> > Thank you in advance,
> > 
> > -Ben
> > 
> > 
> > 
> > Full "doveconf -n" output:
> > 
> > $ doveconf -n
> > # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
> > # Pigeonhole version 0.4.13 (7b14904)
> > # OS: Linux 4.4.0-34-generic x86_64 Ubuntu 16.04.1 LTS
> > auth_mechanisms = plain login
> > disable_plaintext_auth = no
> > listen = *,[::]
> > log_timestamp = "%Y-%m-%d %H:%M:%S "
> > mail_max_userip_connections = 100
> > mail_plugins = " quota"
> > mail_privileged_group = vmail
> > passdb {
> >   args = /etc/dovecot/dovecot-sql.conf
> >   driver = sql
> > }
> > plugin {
> >   antispam_backend = pipe
> >   antispam_debug_target = syslog
> >   antispam_pipe_program = /bin/bash
> >   antispam_pipe_program_args = /usr/local/bin/sa-learn-pipe.sh
> >   antispam_pipe_program_notspam_arg = --ham
> >   antispam_pipe_program_spam_arg = --spam
> >   antispam_pipe_tmpdir = /tmp
> >   antispam_spam_pattern_ignorecase = SPAM;JUNK
> >   antispam_trash_pattern_ignorecase = trash;Deleted *
> >   antispam_verbose_debug = 1
> >   quota = dict:user::file:/var/vmail/%d/%n/.quotausage
> >   quota_rule2 = Trash:storage=+100M
> >   quota_rule3 = Junk:ignore
> >   quota_rule4 = INBOX:storage=+100M
> >   quota_warning = storage=100%% quota-reached 100 %u %d
> >   quota_warning2 = storage=95%% quota-warning 95 %u %d
> >   quota_warning3 = storage=80%% quota-warning 80 %u %d
> >   quota_warning4 = -storage=100%% quota-below below %u %d
> >   sieve = /var/vmail/%d/%n/.sieve
> >   sieve_max_redirects = 25
> > }
> > postmaster_address = postmaster at example.com
> > protocols = imap pop3
> > service auth {
> >   unix_listener /var/spool/postfix/private/auth {
> >     group = postfix
> >     mode = 0660
> >     user = postfix
> >   }
> >   unix_listener auth-userdb {
> >     group = vmail
> >     mode = 0600
> >     user = vmail
> >   }
> >   user = root
> > }
> > service imap-login {
> >   client_limit = 1000
> >   process_limit = 512
> > }
> > service lmtp {
> >   unix_listener /var/spool/postfix/private/dovecot-lmtp {
> >     group = postfix
> >     mode = 0600
> >     user = postfix
> >   }
> > }
> > service quota-below {
> >   executable = script /usr/local/bin/quota-below.sh
> >   unix_listener quota-below {
> >     group = vmail
> >     mode = 0666
> >     user = vmail
> >   }
> >   user = vmail
> > }
> > service quota-reached {
> >   executable = script /usr/local/bin/quota-reached.sh
> >   unix_listener quota-reached {
> >     group = vmail
> >     mode = 0666
> >     user = vmail
> >   }
> >   user = vmail
> > }
> > service quota-warning {
> >   executable = script /usr/local/bin/quota-warning.sh
> >   unix_listener quota-warning {
> >     group = vmail
> >     mode = 0666
> >     user = vmail
> >   }
> >   user = vmail
> > }
> > ssl_cert = </etc/postfix/smtpd.cert
> > ssl_key = </etc/postfix/smtpd.key
> > ssl_protocols = !SSLv2 !SSLv3
> > userdb {
> >   driver = prefetch
> > }
> > userdb {
> >   args = /etc/dovecot/dovecot-sql.conf
> >   driver = sql
> > }
> > protocol imap {
> >   mail_plugins = quota imap_quota antispam
> > }
> > protocol pop3 {
> >   mail_plugins = quota
> >   pop3_uidl_format = %08Xu%08Xv
> > }
> > protocol lda {
> >   mail_plugins = sieve quota
> > }
> > protocol lmtp {
> >   mail_plugins = quota sieve
> >   postmaster_address = webmaster at localhost
> > }
> > 
> 
> If it's helpful, I've tried replacing the pipe script's
contents with
> the following:
> 
> #####################################################################
> #!/bin/bash
> 
> /usr/lib/dovecot/deliver -d "sa-training at example.com" -m
> "Training.$mode" -p /var/vmail/gtube.txt
> 
> # Exit with "success" status code.
> exit 0
> #####################################################################
> 
> And still, the message never arrives when I drag a message from
"Inbox"
> to "Spam" in the IMAP client, despite the messages logged to
syslog,
> which "all seem to be correct".
> 
> Yet, if I do this using the above content for the script, the
"gtube"
> message is delivered successfully:
> 
> # su vmail
> $ /usr/local/bin/sa-learn-pipe-test.sh
> 
> The message arrives in the Inbox, presumably because -m
"Training.$mode"
> does not "resolve" to anything, and per the documentation, the
delivery
> destination will default to the Inbox in such cases.
> 
> What's wrong with my the custom pipe script to which I posted a link in
> my previous message, I wonder?
> 
> Thanks again for any assistance,
> 
> -Ben
I don't know how to solve your problem, but I have an alternate method.

Create a Junk folder and a ham folder then add the following cron jobs:

30      5       *       *       *       /usr/local/bin/sa-learn --spam
/var/vmail/Maildir/.Junk
30      6       *       *       6       /usr/local/bin/sa-learn --backup >
/etc/mail/spamassassin/spam.db
30      7       *       *       *       /usr/local/bin/sa-learn --ham
/var/vmail/Maildir/.Ham

-- 
Edgar Pettijohn

On 2016-08-18 22:58, Edgar Pettijohn wrote:
> On 16-08-18 22:43:32, Ben Johnson wrote:
>> On 8/18/2016 10:26 PM, Ben Johnson wrote:
>> > Hello,
>> >
>> > (Full "doveconf -n" output is at the end of this
message.)
>> >
>> > I'm attempting to recreate a "recipe" I had used
successfully for quite
>> > some time (in Ubuntu 12.04 LTS), which allowed me to train
>> > SpamAssassin's Bayes database when someone dragged an email
message from
>> > one sub-folder of the account to another.
>> >
>> > I later tried to re-implement it (in Ubuntu 14.04 LTS), and
core-dumps
>> > resulted. I posted to this list about it several times, but nobody
was
>> > ever able to help me solve the problem.
>> >
>> > I just tried the same (in Ubuntu 16.04 LTS), hoping that the
source had
>> > evolved since then, and while I don't see a core-dump, the
pipe script
>> > doesn't seem to be called.
>> >
>> > I've booby-trapped the pipe script with quite a bit of
debugging output,
>> > which should be written to a file, but I don't see any
indication that
>> > the pipe script is ever executed.
>> >
>> > When logged-in as an IMAP user, and I drag a message from the
user's
>> > Inbox into the "Spam" folder, I do see the following
activity in the syslog:
>> >
>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_unsure(Junk): 0
>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_trash(INBOX): 0
>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_trash(Junk): 0
>> > Aug 18 22:12:15 example.com imap: antispam: mail copy: from trash:
0, to
>> > trash: 0
>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_spam(INBOX): 0
>> > Aug 18 22:12:15 example.com imap: antispam: mailbox_is_spam(Junk):
1
>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_unsure(INBOX): 0
>> > Aug 18 22:12:15 example.com imap: antispam: mail copy: src spam:
0, dst
>> > spam: 1, src unsure: 0
>> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain
backend
>> > program /bin/bash
>> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain
backend
>> > program /bin/bash
>> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain
backend
>> > program parameter 1 /usr/local/bin/sa-learn-pipe.sh
>> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain
backend
>> > program parameter 2 --spam
>> >
>> > But the piped message is never delivered to the target mailbox.
>> > The mail log at /var/log/mail.log does not register any activity
when I
>> > move messages (and maybe that's expected).
>> >
>> > Here are the contents of the /usr/local/bin/sa-learn-pipe.sh
script:
>> >
>> > http://pastebin.com/7vm74jmk
>> >
>> > The permissions and mode on the script seem appropriate:
>> >
>> > $ stat /usr/local/bin/sa-learn-pipe.sh
>> >   File: '/usr/local/bin/sa-learn-pipe.sh'
>> >   Size: 1765            Blocks: 8          IO Block: 4096  
regular file
>> > Device: fd01h/64769d    Inode: 669345      Links: 1
>> > Access: (0755/-rwxr-xr-x)  Uid: ( 5000/   vmail)   Gid: ( 5000/  
vmail)
>> > Access: 2016-08-18 19:09:47.307392919 -0400
>> > Modify: 2016-08-18 19:09:31.231391749 -0400
>> > Change: 2016-08-18 19:09:31.231391749 -0400
>> >  Birth: -
>> >
>> > What might I be doing incorrectly here?
>> >
>> > Please let me know if I can provide any other information.
>> >
>> > Thank you in advance,
>> >
>> > -Ben
>> >
>> >
>> >
>> > Full "doveconf -n" output:
>> >
>> > $ doveconf -n
>> > # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
>> > # Pigeonhole version 0.4.13 (7b14904)
>> > # OS: Linux 4.4.0-34-generic x86_64 Ubuntu 16.04.1 LTS
>> > auth_mechanisms = plain login
>> > disable_plaintext_auth = no
>> > listen = *,[::]
>> > log_timestamp = "%Y-%m-%d %H:%M:%S "
>> > mail_max_userip_connections = 100
>> > mail_plugins = " quota"
>> > mail_privileged_group = vmail
>> > passdb {
>> >   args = /etc/dovecot/dovecot-sql.conf
>> >   driver = sql
>> > }
>> > plugin {
>> >   antispam_backend = pipe
>> >   antispam_debug_target = syslog
>> >   antispam_pipe_program = /bin/bash
>> >   antispam_pipe_program_args = /usr/local/bin/sa-learn-pipe.sh
>> >   antispam_pipe_program_notspam_arg = --ham
>> >   antispam_pipe_program_spam_arg = --spam
>> >   antispam_pipe_tmpdir = /tmp
>> >   antispam_spam_pattern_ignorecase = SPAM;JUNK
>> >   antispam_trash_pattern_ignorecase = trash;Deleted *
>> >   antispam_verbose_debug = 1
>> >   quota = dict:user::file:/var/vmail/%d/%n/.quotausage
>> >   quota_rule2 = Trash:storage=+100M
>> >   quota_rule3 = Junk:ignore
>> >   quota_rule4 = INBOX:storage=+100M
>> >   quota_warning = storage=100%% quota-reached 100 %u %d
>> >   quota_warning2 = storage=95%% quota-warning 95 %u %d
>> >   quota_warning3 = storage=80%% quota-warning 80 %u %d
>> >   quota_warning4 = -storage=100%% quota-below below %u %d
>> >   sieve = /var/vmail/%d/%n/.sieve
>> >   sieve_max_redirects = 25
>> > }
>> > postmaster_address = postmaster at example.com
>> > protocols = imap pop3
>> > service auth {
>> >   unix_listener /var/spool/postfix/private/auth {
>> >     group = postfix
>> >     mode = 0660
>> >     user = postfix
>> >   }
>> >   unix_listener auth-userdb {
>> >     group = vmail
>> >     mode = 0600
>> >     user = vmail
>> >   }
>> >   user = root
>> > }
>> > service imap-login {
>> >   client_limit = 1000
>> >   process_limit = 512
>> > }
>> > service lmtp {
>> >   unix_listener /var/spool/postfix/private/dovecot-lmtp {
>> >     group = postfix
>> >     mode = 0600
>> >     user = postfix
>> >   }
>> > }
>> > service quota-below {
>> >   executable = script /usr/local/bin/quota-below.sh
>> >   unix_listener quota-below {
>> >     group = vmail
>> >     mode = 0666
>> >     user = vmail
>> >   }
>> >   user = vmail
>> > }
>> > service quota-reached {
>> >   executable = script /usr/local/bin/quota-reached.sh
>> >   unix_listener quota-reached {
>> >     group = vmail
>> >     mode = 0666
>> >     user = vmail
>> >   }
>> >   user = vmail
>> > }
>> > service quota-warning {
>> >   executable = script /usr/local/bin/quota-warning.sh
>> >   unix_listener quota-warning {
>> >     group = vmail
>> >     mode = 0666
>> >     user = vmail
>> >   }
>> >   user = vmail
>> > }
>> > ssl_cert = </etc/postfix/smtpd.cert
>> > ssl_key = </etc/postfix/smtpd.key
>> > ssl_protocols = !SSLv2 !SSLv3
>> > userdb {
>> >   driver = prefetch
>> > }
>> > userdb {
>> >   args = /etc/dovecot/dovecot-sql.conf
>> >   driver = sql
>> > }
>> > protocol imap {
>> >   mail_plugins = quota imap_quota antispam
>> > }
>> > protocol pop3 {
>> >   mail_plugins = quota
>> >   pop3_uidl_format = %08Xu%08Xv
>> > }
>> > protocol lda {
>> >   mail_plugins = sieve quota
>> > }
>> > protocol lmtp {
>> >   mail_plugins = quota sieve
>> >   postmaster_address = webmaster at localhost
>> > }
>> >
>> 
>> If it's helpful, I've tried replacing the pipe script's
contents with
>> the following:
>> 
>> #####################################################################
>> #!/bin/bash
>> 
>> /usr/lib/dovecot/deliver -d "sa-training at example.com" -m
>> "Training.$mode" -p /var/vmail/gtube.txt
>> 
>> # Exit with "success" status code.
>> exit 0
>> #####################################################################
>> 
>> And still, the message never arrives when I drag a message from 
>> "Inbox"
>> to "Spam" in the IMAP client, despite the messages logged to
syslog,
>> which "all seem to be correct".
>> 
>> Yet, if I do this using the above content for the script, the
"gtube"
>> message is delivered successfully:
>> 
>> # su vmail
>> $ /usr/local/bin/sa-learn-pipe-test.sh
>> 
>> The message arrives in the Inbox, presumably because -m 
>> "Training.$mode"
>> does not "resolve" to anything, and per the documentation,
the
>> delivery
>> destination will default to the Inbox in such cases.
>> 
>> What's wrong with my the custom pipe script to which I posted a
link
>> in
>> my previous message, I wonder?
>> 
>> Thanks again for any assistance,
>> 
>> -Ben
> 
> I don't know how to solve your problem, but I have an alternate method.
> 
> Create a Junk folder and a ham folder then add the following cron jobs:
> 
> 30      5       *       *       *       /usr/local/bin/sa-learn --spam
> /var/vmail/Maildir/.Junk
> 30      6       *       *       6       /usr/local/bin/sa-learn
> --backup > /etc/mail/spamassassin/spam.db
> 30      7       *       *       *       /usr/local/bin/sa-learn --ham
> /var/vmail/Maildir/.Ham
Thank you for taking a look, Edgar!

I already have the very setup that you describe.

The challenge at hand, however, is with regard to *how* messages end-up 
in the "ham" and "spam" folders in the first place.

This is a multi-user system and each user must be able to contribute to 
ham/spam training. To make this as easy as possible, and transparent to 
the end-users, I need for the simple act of dragging a message from 
Inbox -> Spam (or vice versa) to submit the sample on the user's behalf, 
automatically. This has all worked swimmingly in the past.

In any case, I've made some progress on this go-around.

I tested my backend pipe script by doing this, as root, and it works:

# /bin/bash /usr/local/bin/sa-learn-pipe.sh --spam < 
/var/vmail/gtube.txt

The message is delivered to the appropriate mailbox and all is well in 
the world.

Of course, in practice, Dovecot does not run as root. So, let's try the 
same as the vmail user:

# su vmail
$ /bin/bash /usr/local/bin/sa-learn-pipe.sh --spam < 
/var/vmail/gtube.txt
/usr/local/bin/sa-learn-pipe.sh: line 8: /tmp/sa-learn-pipe.log: 
Permission denied
/usr/local/bin/sa-learn-pipe.sh: line 10: /tmp/sa-learn-pipe.log: 
Permission denied
/usr/local/bin/sa-learn-pipe.sh: line 12: /tmp/sa-learn-pipe.log: 
Permission denied
/usr/local/bin/sa-learn-pipe.sh: line 16: /tmp/sendmail-parms.txt: 
Permission denied
/usr/local/bin/sa-learn-pipe.sh: line 22: /tmp/sa-learn-pipe.log: 
Permission denied
/usr/local/bin/sa-learn-pipe.sh: line 37: /tmp/sa-learn-pipe.log: 
Permission denied
/usr/local/bin/sa-learn-pipe.sh: line 40: /tmp/sa-learn-pipe.log: 
Permission denied
/usr/local/bin/sa-learn-pipe.sh: line 43: /tmp/strace.txt: Permission 
denied
/usr/local/bin/sa-learn-pipe.sh: line 47: /tmp/sa-learn-pipe.log: 
Permission denied
/usr/local/bin/sa-learn-pipe.sh: line 54: /tmp/sa-learn-pipe.log: 
Permission denied

Aha! Clearly, the vmail user cannot read from nor write to /tmp. (Why 
that is, I have no idea, as the /tmp directory's permissions certainly 
allow for both; maybe Dovecot implements this as a security measure.)

This prompted me to change all references to /tmp in the pipe script to 
~/tmp, and create this directory:

$ whoami
vmail
$ mkdir ~/tmp && chmod 770 ~/tmp
$ /bin/bash /usr/local/bin/sa-learn-pipe.sh --ham < /var/vmail/gtube.txt

No errors this time (at least not on the console).

But I do get this in /var/log/mail.err:

Aug 19 12:04:24 example.com dovecot: lda(sa-training at example.com): 
Fatal: Can't open delivery mail as raw: Permission denied

I'm not sure how to interpret this message. Where is permission being 
denied? More importantly, what's the fix?

Thanks for any hints!

-Ben

Sent from my iPhone
> On Aug 19, 2016, at 11:17 AM, ben at indietorrent.org wrote:
> 
>> On 2016-08-18 22:58, Edgar Pettijohn wrote:
>>> On 16-08-18 22:43:32, Ben Johnson wrote:
>>> On 8/18/2016 10:26 PM, Ben Johnson wrote:
>>> > Hello,
>>> >
>>> > (Full "doveconf -n" output is at the end of this
message.)
>>> >
>>> > I'm attempting to recreate a "recipe" I had used
successfully for quite
>>> > some time (in Ubuntu 12.04 LTS), which allowed me to train
>>> > SpamAssassin's Bayes database when someone dragged an
email message from
>>> > one sub-folder of the account to another.
>>> >
>>> > I later tried to re-implement it (in Ubuntu 14.04 LTS), and
core-dumps
>>> > resulted. I posted to this list about it several times, but
nobody was
>>> > ever able to help me solve the problem.
>>> >
>>> > I just tried the same (in Ubuntu 16.04 LTS), hoping that the
source had
>>> > evolved since then, and while I don't see a core-dump, the
pipe script
>>> > doesn't seem to be called.
>>> >
>>> > I've booby-trapped the pipe script with quite a bit of
debugging output,
>>> > which should be written to a file, but I don't see any
indication that
>>> > the pipe script is ever executed.
>>> >
>>> > When logged-in as an IMAP user, and I drag a message from the
user's
>>> > Inbox into the "Spam" folder, I do see the following
activity in the syslog:
>>> >
>>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_unsure(Junk): 0
>>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_trash(INBOX): 0
>>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_trash(Junk): 0
>>> > Aug 18 22:12:15 example.com imap: antispam: mail copy: from
trash: 0, to
>>> > trash: 0
>>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_spam(INBOX): 0
>>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_spam(Junk): 1
>>> > Aug 18 22:12:15 example.com imap: antispam:
mailbox_is_unsure(INBOX): 0
>>> > Aug 18 22:12:15 example.com imap: antispam: mail copy: src
spam: 0, dst
>>> > spam: 1, src unsure: 0
>>> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain
backend
>>> > program /bin/bash
>>> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain
backend
>>> > program /bin/bash
>>> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain
backend
>>> > program parameter 1 /usr/local/bin/sa-learn-pipe.sh
>>> > Aug 18 22:12:15 example.com imap: antispam: running mailtrain
backend
>>> > program parameter 2 --spam
>>> >
>>> > But the piped message is never delivered to the target
mailbox.
>>> > The mail log at /var/log/mail.log does not register any
activity when I
>>> > move messages (and maybe that's expected).
>>> >
>>> > Here are the contents of the /usr/local/bin/sa-learn-pipe.sh
script:
>>> >
>>> > http://pastebin.com/7vm74jmk
>>> >
>>> > The permissions and mode on the script seem appropriate:
>>> >
>>> > $ stat /usr/local/bin/sa-learn-pipe.sh
>>> >   File: '/usr/local/bin/sa-learn-pipe.sh'
>>> >   Size: 1765            Blocks: 8          IO Block: 4096  
regular file
>>> > Device: fd01h/64769d    Inode: 669345      Links: 1
>>> > Access: (0755/-rwxr-xr-x)  Uid: ( 5000/   vmail)   Gid: (
5000/   vmail)
>>> > Access: 2016-08-18 19:09:47.307392919 -0400
>>> > Modify: 2016-08-18 19:09:31.231391749 -0400
>>> > Change: 2016-08-18 19:09:31.231391749 -0400
>>> >  Birth: -
>>> >
>>> > What might I be doing incorrectly here?
>>> >
>>> > Please let me know if I can provide any other information.
>>> >
>>> > Thank you in advance,
>>> >
>>> > -Ben
>>> >
>>> >
>>> >
>>> > Full "doveconf -n" output:
>>> >
>>> > $ doveconf -n
>>> > # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
>>> > # Pigeonhole version 0.4.13 (7b14904)
>>> > # OS: Linux 4.4.0-34-generic x86_64 Ubuntu 16.04.1 LTS
>>> > auth_mechanisms = plain login
>>> > disable_plaintext_auth = no
>>> > listen = *,[::]
>>> > log_timestamp = "%Y-%m-%d %H:%M:%S "
>>> > mail_max_userip_connections = 100
>>> > mail_plugins = " quota"
>>> > mail_privileged_group = vmail
>>> > passdb {
>>> >   args = /etc/dovecot/dovecot-sql.conf
>>> >   driver = sql
>>> > }
>>> > plugin {
>>> >   antispam_backend = pipe
>>> >   antispam_debug_target = syslog
>>> >   antispam_pipe_program = /bin/bash
>>> >   antispam_pipe_program_args = /usr/local/bin/sa-learn-pipe.sh
>>> >   antispam_pipe_program_notspam_arg = --ham
>>> >   antispam_pipe_program_spam_arg = --spam
>>> >   antispam_pipe_tmpdir = /tmp
>>> >   antispam_spam_pattern_ignorecase = SPAM;JUNK
>>> >   antispam_trash_pattern_ignorecase = trash;Deleted *
>>> >   antispam_verbose_debug = 1
>>> >   quota = dict:user::file:/var/vmail/%d/%n/.quotausage
>>> >   quota_rule2 = Trash:storage=+100M
>>> >   quota_rule3 = Junk:ignore
>>> >   quota_rule4 = INBOX:storage=+100M
>>> >   quota_warning = storage=100%% quota-reached 100 %u %d
>>> >   quota_warning2 = storage=95%% quota-warning 95 %u %d
>>> >   quota_warning3 = storage=80%% quota-warning 80 %u %d
>>> >   quota_warning4 = -storage=100%% quota-below below %u %d
>>> >   sieve = /var/vmail/%d/%n/.sieve
>>> >   sieve_max_redirects = 25
>>> > }
>>> > postmaster_address = postmaster at example.com
>>> > protocols = imap pop3
>>> > service auth {
>>> >   unix_listener /var/spool/postfix/private/auth {
>>> >     group = postfix
>>> >     mode = 0660
>>> >     user = postfix
>>> >   }
>>> >   unix_listener auth-userdb {
>>> >     group = vmail
>>> >     mode = 0600
>>> >     user = vmail
>>> >   }
>>> >   user = root
>>> > }
>>> > service imap-login {
>>> >   client_limit = 1000
>>> >   process_limit = 512
>>> > }
>>> > service lmtp {
>>> >   unix_listener /var/spool/postfix/private/dovecot-lmtp {
>>> >     group = postfix
>>> >     mode = 0600
>>> >     user = postfix
>>> >   }
>>> > }
>>> > service quota-below {
>>> >   executable = script /usr/local/bin/quota-below.sh
>>> >   unix_listener quota-below {
>>> >     group = vmail
>>> >     mode = 0666
>>> >     user = vmail
>>> >   }
>>> >   user = vmail
>>> > }
>>> > service quota-reached {
>>> >   executable = script /usr/local/bin/quota-reached.sh
>>> >   unix_listener quota-reached {
>>> >     group = vmail
>>> >     mode = 0666
>>> >     user = vmail
>>> >   }
>>> >   user = vmail
>>> > }
>>> > service quota-warning {
>>> >   executable = script /usr/local/bin/quota-warning.sh
>>> >   unix_listener quota-warning {
>>> >     group = vmail
>>> >     mode = 0666
>>> >     user = vmail
>>> >   }
>>> >   user = vmail
>>> > }
>>> > ssl_cert = </etc/postfix/smtpd.cert
>>> > ssl_key = </etc/postfix/smtpd.key
>>> > ssl_protocols = !SSLv2 !SSLv3
>>> > userdb {
>>> >   driver = prefetch
>>> > }
>>> > userdb {
>>> >   args = /etc/dovecot/dovecot-sql.conf
>>> >   driver = sql
>>> > }
>>> > protocol imap {
>>> >   mail_plugins = quota imap_quota antispam
>>> > }
>>> > protocol pop3 {
>>> >   mail_plugins = quota
>>> >   pop3_uidl_format = %08Xu%08Xv
>>> > }
>>> > protocol lda {
>>> >   mail_plugins = sieve quota
>>> > }
>>> > protocol lmtp {
>>> >   mail_plugins = quota sieve
>>> >   postmaster_address = webmaster at localhost
>>> > }
>>> >
>>> If it's helpful, I've tried replacing the pipe script's
contents with
>>> the following:
>>>
#####################################################################
>>> #!/bin/bash
>>> /usr/lib/dovecot/deliver -d "sa-training at example.com"
-m
>>> "Training.$mode" -p /var/vmail/gtube.txt
>>> # Exit with "success" status code.
>>> exit 0
>>>
#####################################################################
>>> And still, the message never arrives when I drag a message from
"Inbox"
>>> to "Spam" in the IMAP client, despite the messages logged
to syslog,
>>> which "all seem to be correct".
>>> Yet, if I do this using the above content for the script, the
"gtube"
>>> message is delivered successfully:
>>> # su vmail
>>> $ /usr/local/bin/sa-learn-pipe-test.sh
>>> The message arrives in the Inbox, presumably because -m
"Training.$mode"
>>> does not "resolve" to anything, and per the
documentation, the delivery
>>> destination will default to the Inbox in such cases.
>>> What's wrong with my the custom pipe script to which I posted a
link in
>>> my previous message, I wonder?
>>> Thanks again for any assistance,
>>> -Ben
>> I don't know how to solve your problem, but I have an alternate
method.
>> Create a Junk folder and a ham folder then add the following cron jobs:
>> 30      5       *       *       *       /usr/local/bin/sa-learn --spam
>> /var/vmail/Maildir/.Junk
>> 30      6       *       *       6       /usr/local/bin/sa-learn
>> --backup > /etc/mail/spamassassin/spam.db
>> 30      7       *       *       *       /usr/local/bin/sa-learn --ham
>> /var/vmail/Maildir/.Ham
> 
> Thank you for taking a look, Edgar!
> 
> I already have the very setup that you describe.
> 
> The challenge at hand, however, is with regard to *how* messages end-up in
the "ham" and "spam" folders in the first place.
> 
> This is a multi-user system and each user must be able to contribute to
ham/spam training. To make this as easy as possible, and transparent to the
end-users, I need for the simple act of dragging a message from Inbox -> Spam
(or vice versa) to submit the sample on the user's behalf, automatically.
This has all worked swimmingly in the past.
> 
> In any case, I've made some progress on this go-around.
> 
> I tested my backend pipe script by doing this, as root, and it works:
> 
> # /bin/bash /usr/local/bin/sa-learn-pipe.sh --spam <
/var/vmail/gtube.txt
> 
> The message is delivered to the appropriate mailbox and all is well in the
world.
> 
> Of course, in practice, Dovecot does not run as root. So, let's try the
same as the vmail user:
> 
> # su vmail
> $ /bin/bash /usr/local/bin/sa-learn-pipe.sh --spam <
/var/vmail/gtube.txt
> /usr/local/bin/sa-learn-pipe.sh: line 8: /tmp/sa-learn-pipe.log: Permission
denied
> /usr/local/bin/sa-learn-pipe.sh: line 10: /tmp/sa-learn-pipe.log:
Permission denied
> /usr/local/bin/sa-learn-pipe.sh: line 12: /tmp/sa-learn-pipe.log:
Permission denied
> /usr/local/bin/sa-learn-pipe.sh: line 16: /tmp/sendmail-parms.txt:
Permission denied
> /usr/local/bin/sa-learn-pipe.sh: line 22: /tmp/sa-learn-pipe.log:
Permission denied
> /usr/local/bin/sa-learn-pipe.sh: line 37: /tmp/sa-learn-pipe.log:
Permission denied
> /usr/local/bin/sa-learn-pipe.sh: line 40: /tmp/sa-learn-pipe.log:
Permission denied
> /usr/local/bin/sa-learn-pipe.sh: line 43: /tmp/strace.txt: Permission
denied
> /usr/local/bin/sa-learn-pipe.sh: line 47: /tmp/sa-learn-pipe.log:
Permission denied
> /usr/local/bin/sa-learn-pipe.sh: line 54: /tmp/sa-learn-pipe.log:
Permission denied
> 
> Aha! Clearly, the vmail user cannot read from nor write to /tmp. (Why that
is, I have no idea, as the /tmp directory's permissions certainly allow for
both; maybe Dovecot implements this as a security measure.)
> 
If in a chroot /tmp may not exist. That's all I can think of there.
> This prompted me to change all references to /tmp in the pipe script to
~/tmp, and create this directory:
> 
> $ whoami
> vmail
> $ mkdir ~/tmp && chmod 770 ~/tmp
> $ /bin/bash /usr/local/bin/sa-learn-pipe.sh --ham < /var/vmail/gtube.txt
> 
> No errors this time (at least not on the console).
> 
> But I do get this in /var/log/mail.err:
> 
> Aug 19 12:04:24 example.com dovecot: lda(sa-training at example.com):
Fatal: Can't open delivery mail as raw: Permission denied
> 
That's a new one for me.
> I'm not sure how to interpret this message. Where is permission being
denied? More importantly, what's the fix?
> 
> Thanks for any hints!
> 
> -Ben

On 2016-08-19 12:17, ben at indietorrent.org wrote:
> Aha! Clearly, the vmail user cannot read from nor write to /tmp. (Why
> that is, I have no idea, as the /tmp directory's permissions certainly
> allow for both; maybe Dovecot implements this as a security measure.)
> 
> This prompted me to change all references to /tmp in the pipe script
> to ~/tmp, and create this directory:
> 
> $ whoami
> vmail
> $ mkdir ~/tmp && chmod 770 ~/tmp
> $ /bin/bash /usr/local/bin/sa-learn-pipe.sh --ham < 
> /var/vmail/gtube.txt
> 
> No errors this time (at least not on the console).
> 
> But I do get this in /var/log/mail.err:
> 
> Aug 19 12:04:24 example.com dovecot: lda(sa-training at example.com):
> Fatal: Can't open delivery mail as raw: Permission denied
> 
> I'm not sure how to interpret this message. Where is permission being
> denied? More importantly, what's the fix?
> 
> Thanks for any hints!
> 
> -Ben
Apologies for the rapid-fire replies here.

The strace output that I'm capturing in the pipe script pinpointed the 
problem:

open("/root/~/tmp/sendmail-msg-26272.txt", O_RDONLY) = -1 EACCES 
(Permission denied)

There seems to be some expansion occurring that assumes the root user, 
despite executing the pipe script as the vmail user, so I changed all 
references  to ~/tmp in the pipe script to /var/vmail/tmp and permission 
is no longer denied.

But, now dovecot-lda is core-dumping. Here is the strace output:

http://pastebin.com/RrKmFhzC

So, I'm back to where I was with this problem two years ago.

At that time, I gave-up, because I couldn't invest the time required to 
compile the latest versions of Dovecot and all plugins from scratch in 
an effort to prove that the bug exists in the latest source.

"Dovecot always logs a detailed error message if something goes wrong. 
If it doesn't, it's considered a bug and will be fixed." - 
http://wiki2.dovecot.org/Logging

I'm happy to help identify the root-cause, but I need some guidance 
here.

Thank you,

-Ben