thr3ads.net - dovecot - [Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin [Nov 2013]

If this information is useful, please help other people find it:
Share via:

Dean Carpenter

2013-Nov-08 22:27 UTC

[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin

This might be a fairly long message, but I wanted to be sure to include
as much information as possible.  I'm having an issue with the
dovecot-antispam plugin in that it seems to be unable to successfully
run anything from the pipe backend.  To qualify that, they run, but they
fail ...

Running /usr/bin/sa-learn directly always returns with an error code of
1, and the bayes DB isn't actually updated.  Running the
/usr/local/bin/sa-learn-pipe.sh script from the example will run
sa-learn successfully, but sa-learn fails partway through, right on a
sql DB access.  I have both bayes and FuzzyOcr data stored in mysql, and
the error occurs on accessing either one.  That is, I've tested with
FuzzyOcr enabled, and it fails on the db access to the FuzzyOcr DB, and
I've tried it with FuzzyOcr disabled.  In that case, it fails on the
access to the bayes DB.

The error line right there is :

        libgcc_s.so.1 must be installed for pthread_cancel to work

Since sa-learn terminated prematurely, the bayes DB isn't updated at all.

Running /usr/bin/sa-learn or the script /usr/local/bin/sa-learn-pipe.sh
directly from cmdline works fine though - sa-learn completes correctly
and updates the bayes DB properly, I can see the new tokens.

        $ sa-learn-pipe.sh --spam <
/usr/share/doc/spamassassin/examples/sample-spam.txt
        $ sa-learn --spam
/usr/share/doc/spamassassin/examples/sample-spam.txt

I think it would be great to have a dedicated logfile for antispam, one
that would capture both stdout as well as stderr.

These tests are being done on a clean Ubuntu 13.04 system.  I'm
rebuilding the VM repeatedly as I configure the various pieces, and can
test any changes or whatever you might suggest very easily.  I've fully
scripted the install and configuration of Exim4, Dovecot, Spamassassin,
Clamav, Percona/Mysql, Roundcube, and can build a full VM under
Virtualbox or any VPS in less than 10 minutes.

Ubuntu 13.04
Dovecot 2.1.7
dovecot-antispam    2.0+20120225-3

Anyone else using the antispam plugin with spamassassin 3.3.2 ? 
Anything I can check to see what's causing this problem ?

--------------------------------------------------------
$ cat /usr/local/bin/sa-learn-pipe.sh
#!/bin/bash

echo /usr/bin/sa-learn $* /tmp/sendmail-msg-$$.txt
echo "$$-start ($*)" >> /var/log/sa-learn-pipe.log
cat<&0 >> /tmp/sendmail-msg-$$.txt
/usr/bin/sa-learn -D $* /tmp/sendmail-msg-$$.txt >>
/tmp/sa-learn-pipe.$$.log 2>&1
echo $$ sa-learn rc=$? id=$(id) HOME=$HOME >> /var/log/sa-learn-pipe.log

while read line; do
  echo $$-sa-learn "$line" >> /var/log/sa-learn-pipe.log
done < /tmp/sa-learn-pipe.$$.log

rm -f /tmp/sendmail-msg-$$.txt
echo "$$-end" >> /var/log/sa-learn-pipe.log
exit 0

--------------------------------------------------------
With FuzzyOcr DISabled, fails right after Bayes DB access

$ cat /var/log/sa-learn-pipe.log
4505-start (--spam)
4505 sa-learn rc=134 id=uid=108(Debian-exim) gid=113(Debian-exim)
groups=113(Debian-exim) HOME4505-sa-learn Nov  8 09:05:26.134 [4507] dbg:
logger: adding facilities: all
4505-sa-learn Nov  8 09:05:26.134 [4507] dbg: logger: logging level is DBG
4505-sa-learn Nov  8 09:05:26.134 [4507] dbg: generic: SpamAssassin
version 3.3.2
4505-sa-learn Nov  8 09:05:26.134 [4507] dbg: generic: Perl 5.014002,
PREFIX=/usr, DEF_RULES_DIR=/usr/share/spamassassin,
LOCAL_RULES_DIR=/etc/spamassassin, LOCAL_STATE_DIR=/var/lib/spamassassin
 :
 :
4505-sa-learn Nov  8 09:05:27.131 [4507] dbg: replacetags: done
replacing tags
4505-sa-learn Nov  8 09:05:27.131 [4507] dbg: FreeMail: loaded
freemail_domains entries: 2470 normal, 29 wildcard
4505-sa-learn Nov  8 09:05:27.132 [4507] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0x3879e88) implements
'learner_new', priority 0
4505-sa-learn Nov  8 09:05:27.132 [4507] dbg: bayes: learner_new
self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x3879e88),
bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL
4505-sa-learn Nov  8 09:05:27.151 [4507] dbg: bayes: using username:
debian-spamd
4505-sa-learn Nov  8 09:05:27.151 [4507] dbg: bayes: learner_new: got
store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3ec0388)
4505-sa-learn Nov  8 09:05:27.152 [4507] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0x3879e88) implements
'learner_is_scan_available', priority 0
4505-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work
4505-end
7235-start ()

--------------------------------------------------------
With FuzzyOcr ENabled, fails right after FuzzyOcr DB access

$ cat /var/log/sa-learn-pipe.log
 :
 :
7804-sa-learn Nov  8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
ocrad-invert: /usr/bin/ocrad -s5 -i $input
7804-sa-learn Nov  8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
ocrad-decolorize-invert: /usr/bin/ocrad -s5 -i $input
7804-sa-learn Nov  8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
ocrad-decolorize: /usr/bin/ocrad -s5 $input
7804-sa-learn Nov  8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
gocr: /usr/bin/gocr -i $input
7804-sa-learn Nov  8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
gocr-180: /usr/bin/gocr -l 180 -d 2 -i $input
7804-sa-learn Nov  8 09:37:09.612 [7806] info: FuzzyOcr: Using scan
tesseract: /usr/bin/tesseract $input $output
7804-sa-learn Nov  8 09:37:09.612 [7806] info: FuzzyOcr: Importing for
MLDBM databases not available (dependencies missing)
7804-sa-learn Nov  8 09:37:09.612 [7806] dbg: FuzzyOcr: Connecting to:
dbi:mysql:database=FuzzyOcr;mysql_socket=/var/run/mysqld/mysqld.sock
7804-sa-learn libgcc_s.so.1 must be installed for pthread_cancel to work
7804-end

--------------------------------------------------------
syslog entries from running with sa-learn-pipe.sh script
antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh

Nov  8 09:36:41 newgalah1 imap: antispam: plugin initialising (2.0-notgit)
Nov  8 09:36:41 newgalah1 imap: antispam: verbose debug enabled
Nov  8 09:36:41 newgalah1 imap: antispam: "Junk" is exact match spam
folder
Nov  8 09:36:41 newgalah1 imap: antispam: "Spam" is exact match spam
folder
Nov  8 09:36:41 newgalah1 imap: antispam: no unsure folders
Nov  8 09:36:41 newgalah1 imap: antispam: "Trash" is wildcard match
trash folder
Nov  8 09:36:41 newgalah1 imap: antispam: "Deleted *" is wildcard
match
trash folder
Nov  8 09:36:41 newgalah1 imap: antispam: pipe backend spam argument --spam
Nov  8 09:36:41 newgalah1 imap: antispam: pipe backend not-spam argument
= --ham
Nov  8 09:36:41 newgalah1 imap: antispam: pipe backend program
/usr/local/bin/sa-learn-pipe.sh
Nov  8 09:36:41 newgalah1 imap: antispam: pipe backend tmpdir /tmpNov  8
09:37:08 newgalah1 imap: antispam: mailbox_is_unsure(Junk): 0
Nov  8 09:37:08 newgalah1 imap: antispam: mailbox_is_trash(INBOX): 0
Nov  8 09:37:08 newgalah1 imap: antispam: mailbox_is_trash(Junk): 0
Nov  8 09:37:08 newgalah1 imap: antispam: mail copy: from trash: 0, to
trash: 0
Nov  8 09:37:08 newgalah1 imap: antispam: mailbox_is_spam(INBOX): 0
Nov  8 09:37:08 newgalah1 imap: antispam: mailbox_is_spam(Junk): 1
Nov  8 09:37:08 newgalah1 imap: antispam: mailbox_is_unsure(INBOX): 0
Nov  8 09:37:08 newgalah1 imap: antispam: mail copy: src spam: 0, dst
spam: 1, src unsure: 0                 
Nov  8 09:37:08 newgalah1 imap: antispam: running mailtrain backend
program /usr/local/bin/sa-learn-pipe.sh
Nov  8 09:37:08 newgalah1 imap: antispam: running mailtrain backend
program /usr/local/bin/sa-learn-pipe.sh
Nov  8 09:37:08 newgalah1 imap: antispam: running mailtrain backend
program parameter 1 --spam

--------------------------------------------------------
syslog entries from running with sa-learn directly
antispam_pipe_program = /usr/bin/sa-learn

Nov  8 09:44:57 newgalah1 imap: antispam: plugin initialising (2.0-notgit)
Nov  8 09:44:57 newgalah1 imap: antispam: verbose debug
enabled                                              
Nov  8 09:44:57 newgalah1 imap: antispam: "Junk" is exact match spam
folder                                  
Nov  8 09:44:57 newgalah1 imap: antispam: "Spam" is exact match spam
folder                                  
Nov  8 09:44:57 newgalah1 imap: antispam: no unsure
folders                                                  
Nov  8 09:44:57 newgalah1 imap: antispam: "Trash" is wildcard match
trash folder                             
Nov  8 09:44:57 newgalah1 imap: antispam: "Deleted *" is wildcard
match
trash folder                         
Nov  8 09:44:57 newgalah1 imap: antispam: pipe backend spam argument --spam
Nov  8 09:44:57 newgalah1 imap: antispam: pipe backend not-spam argument
= --ham                             
Nov  8 09:44:57 newgalah1 imap: antispam: pipe backend program /usr/bin/sa-learn
Nov  8 09:44:57 newgalah1 imap: antispam: pipe backend tmpdir
/tmp                                           
Nov  8 09:44:57 newgalah1 imap: antispam: mailbox_is_unsure(Junk):
0                                         
Nov  8 09:44:57 newgalah1 imap: antispam: mailbox_is_trash(INBOX):
0                                         
Nov  8 09:44:57 newgalah1 imap: antispam: mailbox_is_trash(Junk):
0                                          
Nov  8 09:44:57 newgalah1 imap: antispam: mail copy: from trash: 0, to
trash: 0
Nov  8 09:44:57 newgalah1 imap: antispam: mailbox_is_spam(INBOX):
0                                          
Nov  8 09:44:57 newgalah1 imap: antispam: mailbox_is_spam(Junk):
1                                           
Nov  8 09:44:57 newgalah1 imap: antispam: mailbox_is_unsure(INBOX):
0                                        
Nov  8 09:44:57 newgalah1 imap: antispam: mail copy: src spam: 0, dst
spam: 1, src unsure: 0                 
Nov  8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
program /usr/bin/sa-learn
Nov  8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
program /usr/bin/sa-learn                
Nov  8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
program parameter 1 --spam               
Nov  8 09:44:58 newgalah1 imap: antispam: run program failed with exit
code -1

--------------------------------------------------------
$ cat 90-plugin.conf
plugin {
  antispam_debug_target = syslog
  antispam_verbose_debug = 1
  antispam_backend = pipe
  antispam_trash_pattern = Trash;Deleted *
  antispam_spam = Junk;Spam
  antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh
  # antispam_pipe_program_args = --progress
  antispam_pipe_program_spam_arg = --spam
  antispam_pipe_program_notspam_arg = --ham
  antispam_pipe_tmpdir = /tmp
}

--------------------------------------------------------
$ dovecot -n
# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 3.8.0-32-generic x86_64 Ubuntu 13.04
auth_mechanisms = cram-md5 login plain
first_valid_uid = 108
mail_location = maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;as
cii-numeric relational regex imap4flags copy include variables body
enotify environment mailbox date ihave
namespace inbox {
  inbox = yes
  location   mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Junk {
    auto = subscribe
    special_use = \Junk
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Spam {
    auto = subscribe
    special_use = \Junk
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix }
passdb {
  args = scheme=plain username_format=%u /var/mail/%d/PASSWD
  driver = passwd-file
}
plugin {
  antispam_backend = pipe
  antispam_debug_target = syslog
  antispam_pipe_program = /usr/local/bin/sa-learn-pipe.sh
  antispam_pipe_program_notspam_arg = --ham
  antispam_pipe_program_spam_arg = --spam
  antispam_pipe_tmpdir = /tmp
  antispam_spam = Junk;Spam
  antispam_trash_pattern = Trash;Deleted *
  antispam_verbose_debug = 1
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
}
protocols = " imap lmtp sieve"
service auth {
  unix_listener auth-client {
    mode = 0666
  }
}
ssl = required
ssl_cert = </etc/ssl/certs/dovecot1.crt
ssl_key = </etc/ssl/private/dovecot1.pkcs8.key
userdb {
  args = username_format=%u /var/mail/%d/PASSWD
  default_fields = uid=Debian-exim gid=Debian-exim home=/var/mail/%d/%n
  driver = static
}
protocol lmtp {
  mail_plugins = " sieve"
}
protocol lda {
  mail_plugins = " sieve"
}
protocol imap {
  mail_plugins = " antispam"
}

Steffen Kaiser

2013-Nov-11 07:31 UTC

head link

[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 8 Nov 2013, Dean Carpenter wrote:
> Running /usr/bin/sa-learn directly always returns with an error code of
> 1, and the bayes DB isn't actually updated.  Running the
> /usr/local/bin/sa-learn-pipe.sh script from the example will run
> sa-learn successfully, but sa-learn fails partway through, right on a
> sql DB access.  I have both bayes and FuzzyOcr data stored in mysql, and
> the error occurs on accessing either one.  That is, I've tested with
> FuzzyOcr enabled, and it fails on the db access to the FuzzyOcr DB, and
> I've tried it with FuzzyOcr disabled.  In that case, it fails on the
> access to the bayes DB.
>
> The error line right there is :
>
>        libgcc_s.so.1 must be installed for pthread_cancel to work
>
> Since sa-learn terminated prematurely, the bayes DB isn't updated at
all.
But this is not a Dovecot problem, isn't it? Antispam calls sa-learn, but 
that program fails.

Do you have the system library libgcc_s installed as the error message 
says?

Search the net for that error message, see: 
https://lists.ubuntu.com/archives/foundations-bugs/2012-March/070050.html

If you are on a 64bit system, maybe sa-learn is compiled 32bit only, then 
the library must be installed as 32bit version as well.
> 4505 sa-learn rc=134 id=uid=108(Debian-exim) gid=113(Debian-exim)
> groups=113(Debian-exim) HOME> 4505-sa-learn libgcc_s.so.1 must be
installed for pthread_cancel to work
> 4505-end
> Nov  8 09:44:57 newgalah1 imap: antispam: running mailtrain backend
> program parameter 1 --spam
> Nov  8 09:44:58 newgalah1 imap: antispam: run program failed with exit
> code -1
- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBUoCH0F3r2wJMiz2NAQKypwf+M63gGswfoTkNqB0VB/TgK47hb8GQfwNj
S6WpypdiSUbVU9NkHezdKmQNmSEyGP/34UChacNCtUbezRV77j/vc1kb/Uap+Z76
cETR5m0zFt5Svl84lm14027ZU9Ph5O0ddr2NTi1NJqmos4V8zBw2C6OfD85vFN1T
HDPz6e1h8mIFKhoisudbHt/2KcQkkPA9NinGnSgtpyyGwX7pVghQ6Hb4XD5xD/7F
HTXqOit52NvzDn2tflIWxdyOIbAxN5wU4q5tWvA0ZckCbAqpc9o+zbAYZm8nOqmo
lm+oaaBpKNbPyScQTFTn7ccmm2GDsxxUEQ6ubX7GxPXOnQvuHjMefg==Xew9
-----END PGP SIGNATURE-----

Joseph Tam

2013-Nov-21 01:21 UTC

head link

[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin

>    LD_PRELOAD=/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/bin/sa-learn -D
> --progress $* /tmp/sendmail-msg-$$.txt >> /tmp/sa-learn-pipe.$$.log
2>&1
>
> So far to no avail.  I'm going to try the spool2dir backend with
> incron.  Cumbersome, but it should work ...
Maybe try rebuilding whatever is using libgcc_s.so with GCC compiler
option -static-libgcc?

Joseph Tam <jtam.home at gmail.com>

Reasonably Related Threads

Search for more reasonably related threads

dovecot - Nov 2013 - Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin

[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin

[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin

[Dovecot] Problems with dovecot 2.1.7, spamassassin 3.3.2 and antispam plugin

Reasonably Related Threads