Dean Carpenter
2013-Nov-08 22:27 UTC
[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin
This might be a fairly long message, but I wanted to be sure to include
as much information as possible. I'm having an issue with the
dovecot-antispam plugin in that it seems to be unable to successfully
run anything from the pipe backend. To qualify that, they run, but they
fail ...
Running /usr/bin/sa-learn directly always returns with an error code of
1, and the bayes DB isn't actually updated. Running the
/usr/local/bin/sa-learn-pipe.sh script from the example will run
sa-learn successfully, but sa-learn fails partway through, right on a
sql DB access. I have both bayes and FuzzyOcr data stored in mysql, and
the error occurs on accessing either one. That is, I've tested with
FuzzyOcr enabled, and it fails on the db access to the FuzzyOcr DB, and
I've tried it with FuzzyOcr disabled. In that case, it fails on the
access to the bayes DB.
The error line right there is :
libgcc_s.so.1 must be installed for pthread_cancel to work
Since sa-learn terminated prematurely, the bayes DB isn't updated at all.
Running /usr/bin/sa-learn or the script /usr/local/bin/sa-learn-pipe.sh
directly from cmdline works fine though - sa-learn completes correctly
and updates the bayes DB properly, I can see the new tokens.
$ sa-learn-pipe.sh --spam <
/usr/share/doc/spamassassin/examples/sample-spam.txt
$ sa-learn --spam
/usr/share/doc/spamassassin/examples/sample-spam.txt
I think it would be great to have a dedicated logfile for antispam, one
that would capture both stdout as well as stderr.
These tests are being done on a clean Ubuntu 13.04 system. I'm
rebuilding the VM repeatedly as I configure the various pieces, and can
test any changes or whatever you might suggest very easily. I've fully
scripted the install and configuration of Exim4, Dovecot, Spamassassin,
Clamav, Percona/Mysql, Roundcube, and can build a full VM under
Virtualbox or any VPS in less than 10 minutes.
Ubuntu 13.04
Dovecot 2.1.7
dovecot-antispam 2.0+20120225-3
Anyone else using the antispam plugin with spamassassin 3.3.2 ?
Anything I can check to see what's causing this problem ?
--------------------------------------------------------
$ cat /usr/local/bin/sa-learn-pipe.sh
#!/bin/bash
echo /usr/bin/sa-learn $* /tmp/sendmail-msg-$$.txt
echo "$$-start ($*)" >> /var/log/sa-learn-pipe.log
cat<&0 >> /tmp/sendmail-msg-$$.txt
/usr/bin/sa-learn -D $* /tmp/sendmail-msg-$$.txt >>
/tmp/sa-learn-pipe.$$.log 2>&1
echo $$ sa-learn rc=$? id=$(id) HOME=$HOME >> /var/log/sa-learn-pipe.log
while read line; do
echo $$-sa-learn "$line" >> /var/log/sa-learn-pipe.log
done < /tmp/sa-learn-pipe.$$.log
rm -f /tmp/sendmail-msg-$$.txt
echo "$$-end" >> /var/log/sa-learn-pipe.log
exit 0
--------------------------------------------------------
With FuzzyOcr DISabled, fails right after Bayes DB access
$ cat /var/log/sa-learn-pipe.log
4505-start (--spam)
4505 sa-learn rc=134 id=uid=108(Debian-exim) gid=113(Debian-exim)
groups=113(Debian-exim) HOME4505-sa-learn Nov 8 09:05:26.134 [4507] dbg:
logger: adding facilities: all
4505-sa-learn Nov 8 09:05:26.134 [4507] dbg: logger: logging level is DBG
4505-sa-learn Nov 8 09:05:26.134 [4507] dbg: generic: SpamAssassin
version 3.3.2
4505-sa-learn Nov 8 09:05:26.134 [4507] dbg: generic: Perl 5.014002,
PREFIX=/usr, DEF_RULES_DIR=/usr/share/spamassassin,
LOCAL_RULES_DIR=/etc/spamassassin, LOCAL_STATE_DIR=/var/lib/spamassassin
:
:
4505-sa-learn Nov 8 09:05:27.131 [4507] dbg: replacetags: done
replacing tags
4505-sa-learn Nov 8 09:05:27.131 [4507] dbg: FreeMail: loaded
freemail_domains entries: 2470 normal, 29 wildcard
4505-sa-learn Nov 8 09:05:27.132 [4507] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0x3879e88) implements
'learner_new', priority 0
4505-sa-learn Nov 8 09:05:27.132 [4507] dbg: bayes: learner_new
self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x3879e88),
bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL
4505-sa-learn Nov 8 09:05:27.151 [4507] dbg: bayes: using username:
debian-spamd
4505-sa-learn Nov 8 09:05:27.151 [4507] dbg: bayes: learner_new: got
store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3ec0388)
4505-sa-learn Nov 8 09:05:27.152 [4507] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0x3879e88) implements
'learner_is_scan_available', priority 0
4505-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work
4505-end
7235-start ()
--------------------------------------------------------
With FuzzyOcr ENabled, fails right after FuzzyOcr DB access
$ cat /var/log/sa-learn-pipe.log
:
:
7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
ocrad-invert: /usr/bin/ocrad -s5 -i $input
7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
ocrad-decolorize-invert: /usr/bin/ocrad -s5 -i $input
7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
ocrad-decolorize: /usr/bin/ocrad -s5 $input
7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
gocr: /usr/bin/gocr -i $input
7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
gocr-180: /usr/bin/gocr -l 180 -d 2 -i $input
7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
tesseract: /usr/bin/tesseract $input $output
7804-sa-learn Nov 8 09:37:09.612 [7806] info: FuzzyOcr: Importing for
MLDBM databases not available (dependencies missing)
7804-sa-learn Nov 8 09:37:09.612 [7806] dbg: FuzzyOcr: Connecting to:
dbi:mysql:database=FuzzyOcr;mysql_socket=/var/run/mysqld/mysqld.sock
7804-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work
7804-end
--------------------------------------------------------
syslog entries from running with sa-learn-pipe.sh script
antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh
Nov 8 09:36:41 newgalah1 imap: antispam: plugin initialising (2.0-notgit)
Nov 8 09:36:41 newgalah1 imap: antispam: verbose debug enabled
Nov 8 09:36:41 newgalah1 imap: antispam: "Junk" is exact match spam
folder
Nov 8 09:36:41 newgalah1 imap: antispam: "Spam" is exact match spam
folder
Nov 8 09:36:41 newgalah1 imap: antispam: no unsure folders
Nov 8 09:36:41 newgalah1 imap: antispam: "Trash" is wildcard match
trash folder
Nov 8 09:36:41 newgalah1 imap: antispam: "Deleted *" is wildcard
match
trash folder
Nov 8 09:36:41 newgalah1 imap: antispam: pipe backend spam argument --spam
Nov 8 09:36:41 newgalah1 imap: antispam: pipe backend not-spam argument
= --ham
Nov 8 09:36:41 newgalah1 imap: antispam: pipe backend program
/usr/local/bin/sa-learn-pipe.sh
Nov 8 09:36:41 newgalah1 imap: antispam: pipe backend tmpdir /tmpNov 8
09:37:08 newgalah1 imap: antispam: mailbox_is_unsure(Junk): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_trash(INBOX): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_trash(Junk): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mail copy: from trash: 0, to
trash: 0
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_spam(INBOX): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_spam(Junk): 1
Nov 8 09:37:08 newgalah1 imap: antispam: mailbox_is_unsure(INBOX): 0
Nov 8 09:37:08 newgalah1 imap: antispam: mail copy: src spam: 0, dst
spam: 1, src unsure: 0
Nov 8 09:37:08 newgalah1 imap: antispam: running mailtrain backend
program /usr/local/bin/sa-learn-pipe.sh
Nov 8 09:37:08 newgalah1 imap: antispam: running mailtrain backend
program /usr/local/bin/sa-learn-pipe.sh
Nov 8 09:37:08 newgalah1 imap: antispam: running mailtrain backend
program parameter 1 --spam
--------------------------------------------------------
syslog entries from running with sa-learn directly
antispam_pipe_program = /usr/bin/sa-learn
Nov 8 09:44:57 newgalah1 imap: antispam: plugin initialising (2.0-notgit)
Nov 8 09:44:57 newgalah1 imap: antispam: verbose debug
enabled
Nov 8 09:44:57 newgalah1 imap: antispam: "Junk" is exact match spam
folder
Nov 8 09:44:57 newgalah1 imap: antispam: "Spam" is exact match spam
folder
Nov 8 09:44:57 newgalah1 imap: antispam: no unsure
folders
Nov 8 09:44:57 newgalah1 imap: antispam: "Trash" is wildcard match
trash folder
Nov 8 09:44:57 newgalah1 imap: antispam: "Deleted *" is wildcard
match
trash folder
Nov 8 09:44:57 newgalah1 imap: antispam: pipe backend spam argument --spam
Nov 8 09:44:57 newgalah1 imap: antispam: pipe backend not-spam argument
= --ham
Nov 8 09:44:57 newgalah1 imap: antispam: pipe backend program /usr/bin/sa-learn
Nov 8 09:44:57 newgalah1 imap: antispam: pipe backend tmpdir
/tmp
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_unsure(Junk):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_trash(INBOX):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_trash(Junk):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mail copy: from trash: 0, to
trash: 0
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_spam(INBOX):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_spam(Junk):
1
Nov 8 09:44:57 newgalah1 imap: antispam: mailbox_is_unsure(INBOX):
0
Nov 8 09:44:57 newgalah1 imap: antispam: mail copy: src spam: 0, dst
spam: 1, src unsure: 0
Nov 8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
program /usr/bin/sa-learn
Nov 8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
program /usr/bin/sa-learn
Nov 8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
program parameter 1 --spam
Nov 8 09:44:58 newgalah1 imap: antispam: run program failed with exit
code -1
--------------------------------------------------------
$ cat 90-plugin.conf
plugin {
antispam_debug_target = syslog
antispam_verbose_debug = 1
antispam_backend = pipe
antispam_trash_pattern = Trash;Deleted *
antispam_spam = Junk;Spam
antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh
# antispam_pipe_program_args = --progress
antispam_pipe_program_spam_arg = --spam
antispam_pipe_program_notspam_arg = --ham
antispam_pipe_tmpdir = /tmp
}
--------------------------------------------------------
$ dovecot -n
# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 3.8.0-32-generic x86_64 Ubuntu 13.04
auth_mechanisms = cram-md5 login plain
first_valid_uid = 108
mail_location = maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;as
cii-numeric relational regex imap4flags copy include variables body
enotify environment mailbox date ihave
namespace inbox {
inbox = yes
location mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Spam {
auto = subscribe
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix }
passdb {
args = scheme=plain username_format=%u /var/mail/%d/PASSWD
driver = passwd-file
}
plugin {
antispam_backend = pipe
antispam_debug_target = syslog
antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh
antispam_pipe_program_notspam_arg = --ham
antispam_pipe_program_spam_arg = --spam
antispam_pipe_tmpdir = /tmp
antispam_spam = Junk;Spam
antispam_trash_pattern = Trash;Deleted *
antispam_verbose_debug = 1
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
}
protocols = " imap lmtp sieve"
service auth {
unix_listener auth-client {
mode = 0666
}
}
ssl = required
ssl_cert = </etc/ssl/certs/dovecot1.crt
ssl_key = </etc/ssl/private/dovecot1.pkcs8.key
userdb {
args = username_format=%u /var/mail/%d/PASSWD
default_fields = uid=Debian-exim gid=Debian-exim home=/var/mail/%d/%n
driver = static
}
protocol lmtp {
mail_plugins = " sieve"
}
protocol lda {
mail_plugins = " sieve"
}
protocol imap {
mail_plugins = " antispam"
}
Steffen Kaiser
2013-Nov-11 07:31 UTC
[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 8 Nov 2013, Dean Carpenter wrote:> Running /usr/bin/sa-learn directly always returns with an error code of > 1, and the bayes DB isn't actually updated. Running the > /usr/local/bin/sa-learn-pipe.sh script from the example will run > sa-learn successfully, but sa-learn fails partway through, right on a > sql DB access. I have both bayes and FuzzyOcr data stored in mysql, and > the error occurs on accessing either one. That is, I've tested with > FuzzyOcr enabled, and it fails on the db access to the FuzzyOcr DB, and > I've tried it with FuzzyOcr disabled. In that case, it fails on the > access to the bayes DB. > > The error line right there is : > > libgcc_s.so.1 must be installed for pthread_cancel to work > > Since sa-learn terminated prematurely, the bayes DB isn't updated at all.But this is not a Dovecot problem, isn't it? Antispam calls sa-learn, but that program fails. Do you have the system library libgcc_s installed as the error message says? Search the net for that error message, see: https://lists.ubuntu.com/archives/foundations-bugs/2012-March/070050.html If you are on a 64bit system, maybe sa-learn is compiled 32bit only, then the library must be installed as 32bit version as well.> 4505 sa-learn rc=134 id=uid=108(Debian-exim) gid=113(Debian-exim) > groups=113(Debian-exim) HOME> 4505-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work > 4505-end> Nov 8 09:44:57 newgalah1 imap: antispam: running mailtrain backend > program parameter 1 --spam > Nov 8 09:44:58 newgalah1 imap: antispam: run program failed with exit > code -1- -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBUoCH0F3r2wJMiz2NAQKypwf+M63gGswfoTkNqB0VB/TgK47hb8GQfwNj S6WpypdiSUbVU9NkHezdKmQNmSEyGP/34UChacNCtUbezRV77j/vc1kb/Uap+Z76 cETR5m0zFt5Svl84lm14027ZU9Ph5O0ddr2NTi1NJqmos4V8zBw2C6OfD85vFN1T HDPz6e1h8mIFKhoisudbHt/2KcQkkPA9NinGnSgtpyyGwX7pVghQ6Hb4XD5xD/7F HTXqOit52NvzDn2tflIWxdyOIbAxN5wU4q5tWvA0ZckCbAqpc9o+zbAYZm8nOqmo lm+oaaBpKNbPyScQTFTn7ccmm2GDsxxUEQ6ubX7GxPXOnQvuHjMefg==Xew9 -----END PGP SIGNATURE-----
Joseph Tam
2013-Nov-21 01:21 UTC
[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin
> LD_PRELOAD=/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/bin/sa-learn -D > --progress $* /tmp/sendmail-msg-$$.txt >> /tmp/sa-learn-pipe.$$.log 2>&1 > > So far to no avail. I'm going to try the spool2dir backend with > incron. Cumbersome, but it should work ...Maybe try rebuilding whatever is using libgcc_s.so with GCC compiler option -static-libgcc? Joseph Tam <jtam.home at gmail.com>