thr3ads.net - dovecot - Sieve filter extracting wrong header value on certain mail messages [Aug 2014]

If this information is useful, please help other people find it:
Share via:

gowen (dovecot)

2014-Aug-06 15:56 UTC

Sieve filter extracting wrong header value on certain mail messages

# 2.2.12: /etc/dovecot/dovecot.conf
# OS: Linux 3.15.4-x86_64-linode45 x86_64 openSUSE 12.3 (x86_64)

Short version:

1) I filter out spam by using a dovecot sieve filter to act based on a 
header inserted by bogofilter
2) In some cases, sieve fails to correctly file based on this header
3) Testing with sieve-test indicates that it's testing the wrong header
4) Suggestions?  Am I doing something wrong?  Is there a problem with 
these headers that bends test-sieve out of shape?
5) Test files from below are also available via 
http://www.swynwyr.com/sieve-test/

Any help is appreciated!!!


Long version with full details:

Here is the simple sieve filter I'm using to test, and two different 
variants of an email which triggers this problem.  I basically deleted 
things one by one from an original email (1) to get minimal testcases 
that triggered (2) or did not trigger (3) the problem.

The filter:

=== begin test.sieve ==require ["fileinto", "body",
"regex"];

if header :contains "X-Bogosity" "Spam," {
     fileinto "SPAM.SPAM";
     stop;
} elsif header :contains "X-Bogosity" "Unsure," {
     fileinto "SPAM.UNSURE";
     stop;
}

keep;
stop;
=== end test.sieve ==
Here is an example email that will cause this filter to not function as 
expected:

=== begin message.2 ==Return-Path: <alora at promosforusio.org>
Subject:   =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?Subject:          
?HS Kit?hen ?ak??ov??r
Giv??away
Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?Subject:
Subject:                AHS Kitchen Makeover Giveaway
X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3


Home Warranty Summer Giveaway
=== end message.2 ==
If I run test-sieve on this, we can clearly see in the trace log that 
when it's supposed to test for header "X-Bogosity" it in fact
looks at
one of the multiple "Subject" headers.  Note that the result is
storing
in INBOX rather than SPAM.SPAM, and that the "matching value" for 
"X-Bogosity headers" is actually the contents of the final
"Subject"
header in the test on line 3.  In the second test, on line 6, it uses a 
different and correct value for the same header!  But too late to help 
as the first test was the one it should have had a match on.

=== begin session ==$ sieve-test -t trace2.log -Tlevel=matching test.sieve
message.2

Performed actions:

  * store message in folder: INBOX

Implicit keep:

   (none)

sieve-test(gowen): Info: final result: success
$ cat trace2.log

       ## Started executing script 'test'
    3: header test
    3:   starting `:contains' match with `i;ascii-casemap' comparator:
    3:   extracting `X-Bogosity' headers from message
    3:   matching value `AHS Kitchen Makeover Giveaway'
    3:     with key `Spam,' => 0
    3:   finishing match with result: not matched
    3: jump if result is false
    3:   jumping to line 6
    6: header test
    6:   starting `:contains' match with `i;ascii-casemap' comparator:
    6:   extracting `X-Bogosity' headers from message
    6:   matching value `Spam, tests=bogofilter, spamicity=1.000000, 
version=1.2.3'
    6:     with key `Unsure,' => 0
    6:   finishing match with result: not matched
    6: jump if result is false
    6:   jumping to line 11
   11: keep action; store message in default mailbox
   12: stop command; end all script execution
       ## Finished executing script 'test'

$
=== end session ==
Now lets remove the "Return-Path" from that message.  (Of the original
26 headers, all but the 7 shown in message.2 could be removed and still 
have this message break the filter).  This alters thing enough that 
sieve starts working as expected (in fact, removing any one of the 6 
non-X-Bogosity headers will cause this to start working).

=== begin message.3 ==Subject:  
=?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?Subject:                     
?HS Kit?hen ?ak??ov??r
Giv??away
Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?Subject:
Subject:                AHS Kitchen Makeover Giveaway
X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3


Home Warranty Summer Giveaway
=== end message.3 ==
Now when we run test-sieve it works as expected, filing into SPAM.SPAM 
and showing the correct matching value for line 3:

=== begin session ==$ sieve-test -t trace3.log -Tlevel=matching test.sieve
message.3

Performed actions:

  * store message in folder: SPAM.SPAM

Implicit keep:

   (none)

sieve-test(gowen): Info: final result: success
$ cat trace3.log

       ## Started executing script 'test'
    3: header test
    3:   starting `:contains' match with `i;ascii-casemap' comparator:
    3:   extracting `X-Bogosity' headers from message
    3:   matching value `Spam, tests=bogofilter, spamicity=1.000000, 
version=1.2.3'
    3:     with key `Spam,' => 1
    3:   finishing match with result: matched
    3: jump if result is false
    3:   not jumping
    4: fileinto action
    4:   store message in mailbox `SPAM.SPAM'
    5: stop command; end all script execution
       ## Finished executing script 'test'

$
=== end session ===

Stephan Bosch

2014-Aug-09 01:43 UTC

head link

Sieve filter extracting wrong header value on certain mail messages

On 8/6/2014 5:56 PM, gowen (dovecot) wrote:>
> # 2.2.12: /etc/dovecot/dovecot.conf
> # OS: Linux 3.15.4-x86_64-linode45 x86_64 openSUSE 12.3 (x86_64)
>
> Short version:
>
> 1) I filter out spam by using a dovecot sieve filter to act based on a
> header inserted by bogofilter
> 2) In some cases, sieve fails to correctly file based on this header
> 3) Testing with sieve-test indicates that it's testing the wrong header
> 4) Suggestions?  Am I doing something wrong?  Is there a problem with
> these headers that bends test-sieve out of shape?
> 5) Test files from below are also available via
> http://www.swynwyr.com/sieve-test/
This looks like a Dovecot bug. We're working on this one.

Regards,

Stephan.

Timo Sirainen

2014-Oct-09 13:43 UTC

head link

Sieve filter extracting wrong header value on certain mail messages

Finally fixed: http://hg.dovecot.org/dovecot-2.2/rev/0e1a3c909a13

On 06 Aug 2014, at 18:56, gowen (dovecot) <gowen-dovecot at swynwyr.com>
wrote:
> 
> # 2.2.12: /etc/dovecot/dovecot.conf
> # OS: Linux 3.15.4-x86_64-linode45 x86_64 openSUSE 12.3 (x86_64)
> 
> Short version:
> 
> 1) I filter out spam by using a dovecot sieve filter to act based on a
header inserted by bogofilter
> 2) In some cases, sieve fails to correctly file based on this header
> 3) Testing with sieve-test indicates that it's testing the wrong header
> 4) Suggestions?  Am I doing something wrong?  Is there a problem with these
headers that bends test-sieve out of shape?
> 5) Test files from below are also available via
http://www.swynwyr.com/sieve-test/
> 
> Any help is appreciated!!!
> 
> 
> Long version with full details:
> 
> Here is the simple sieve filter I'm using to test, and two different
variants of an email which triggers this problem.  I basically deleted things
one by one from an original email (1) to get minimal testcases that triggered
(2) or did not trigger (3) the problem.
> 
> The filter:
> 
> === begin test.sieve ==> require ["fileinto",
"body", "regex"];
> 
> if header :contains "X-Bogosity" "Spam," {
>    fileinto "SPAM.SPAM";
>    stop;
> } elsif header :contains "X-Bogosity" "Unsure," {
>    fileinto "SPAM.UNSURE";
>    stop;
> }
> 
> keep;
> stop;
> === end test.sieve ==> 
> Here is an example email that will cause this filter to not function as
expected:
> 
> === begin message.2 ==> Return-Path: <alora at promosforusio.org>
> Subject:   =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?> Subject:
?HS Kit?hen ?ak??ov??r Giv??away
> Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?>
Subject:
> Subject:                AHS Kitchen Makeover Giveaway
> X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3
> 
> 
> Home Warranty Summer Giveaway
> === end message.2 ==> 
> If I run test-sieve on this, we can clearly see in the trace log that when
it's supposed to test for header "X-Bogosity" it in fact looks at
one of the multiple "Subject" headers.  Note that the result is
storing in INBOX rather than SPAM.SPAM, and that the "matching value"
for "X-Bogosity headers" is actually the contents of the final
"Subject" header in the test on line 3.  In the second test, on line
6, it uses a different and correct value for the same header!  But too late to
help as the first test was the one it should have had a match on.
> 
> === begin session ==> $ sieve-test -t trace2.log -Tlevel=matching
test.sieve message.2
> 
> Performed actions:
> 
> * store message in folder: INBOX
> 
> Implicit keep:
> 
>  (none)
> 
> sieve-test(gowen): Info: final result: success
> $ cat trace2.log
> 
>      ## Started executing script 'test'
>   3: header test
>   3:   starting `:contains' match with `i;ascii-casemap'
comparator:
>   3:   extracting `X-Bogosity' headers from message
>   3:   matching value `AHS Kitchen Makeover Giveaway'
>   3:     with key `Spam,' => 0
>   3:   finishing match with result: not matched
>   3: jump if result is false
>   3:   jumping to line 6
>   6: header test
>   6:   starting `:contains' match with `i;ascii-casemap'
comparator:
>   6:   extracting `X-Bogosity' headers from message
>   6:   matching value `Spam, tests=bogofilter, spamicity=1.000000,
version=1.2.3'
>   6:     with key `Unsure,' => 0
>   6:   finishing match with result: not matched
>   6: jump if result is false
>   6:   jumping to line 11
>  11: keep action; store message in default mailbox
>  12: stop command; end all script execution
>      ## Finished executing script 'test'
> 
> $
> === end session ==> 
> Now lets remove the "Return-Path" from that message.  (Of the
original 26 headers, all but the 7 shown in message.2 could be removed and still
have this message break the filter).  This alters thing enough that sieve starts
working as expected (in fact, removing any one of the 6 non-X-Bogosity headers
will cause this to start working).
> 
> === begin message.3 ==> Subject:  
=?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?> Subject:                
?HS Kit?hen ?ak??ov??r Giv??away
> Subject:      =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?>
Subject:
> Subject:                AHS Kitchen Makeover Giveaway
> X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3
> 
> 
> Home Warranty Summer Giveaway
> === end message.3 ==> 
> Now when we run test-sieve it works as expected, filing into SPAM.SPAM and
showing the correct matching value for line 3:
> 
> === begin session ==> $ sieve-test -t trace3.log -Tlevel=matching
test.sieve message.3
> 
> Performed actions:
> 
> * store message in folder: SPAM.SPAM
> 
> Implicit keep:
> 
>  (none)
> 
> sieve-test(gowen): Info: final result: success
> $ cat trace3.log
> 
>      ## Started executing script 'test'
>   3: header test
>   3:   starting `:contains' match with `i;ascii-casemap'
comparator:
>   3:   extracting `X-Bogosity' headers from message
>   3:   matching value `Spam, tests=bogofilter, spamicity=1.000000,
version=1.2.3'
>   3:     with key `Spam,' => 1
>   3:   finishing match with result: matched
>   3: jump if result is false
>   3:   not jumping
>   4: fileinto action
>   4:   store message in mailbox `SPAM.SPAM'
>   5: stop command; end all script execution
>      ## Finished executing script 'test'
> 
> $
> === end session ===

Maybe Matching Threads

Search for more seemingly similar threads

dovecot - Aug 2014 - Sieve filter extracting wrong header value on certain mail messages

Sieve filter extracting wrong header value on certain mail messages

Sieve filter extracting wrong header value on certain mail messages

Sieve filter extracting wrong header value on certain mail messages

Maybe Matching Threads