dovecot - May 2014 - When the subject portion of an e-mail contains a control character, dovecot.sieve terminates unexpectatedly.

Hello.

We have currently set dovecot.sieve to insert the text "[SPAM]" at the
beginning of an e-mail's subject when it's X-Spam-Score is above 80%.
After we set our system as stated the following errors occur:

1) When an e-mail's subject contains control characters like
[Ctrl+V|^V], dovecot.sieve terminates with an error and an e-mail is not
able to be sent. When a MIME encoded Subject like [????^V????] is sent
we're not able to edit the subject and dovecot ends with an error.

This is a sample of the data that was used in testing.
Subject: =?ISO-2022-JP?B?GyRCI1QjRSNTI1QbKEIWGyRCI00jQSNJI0wbKEI=?X-Spam-Score:
100.00%
?
Subject: =?ISO-2022-JP?B?GyRCI1QjRSNTI1QbKEIWGyRCI00jQSNJI0wbKEI=?X-Spam-Score:
100.00%

This is the log generated by the data above.
----------------------------------------------------------------------
sieve: info: started log at May 02 10:46:22.
main script: line 14: error: addheader action: specified value `[SPAM]
???????' is invalid.
----------------------------------------------------------------------

Aside from [Ctrl + V] the following control charcters also cause errors:
backspace
Ctrl + A
Ctrl + C
Ctrl + [
Ctrl + X
Ctrl + Y

2) When an e-mail's subject line contains a "\0" character,
everything
following the null character is deleted.
When a MIME encoded Subject is sent, for example "????\0????", the
subject becomes "[SPAM]????", the characters after the NULL character
are deleted and dovecot.sieve ends normally.

This is a sample of the data that we collected.
Subject: =?ISO-2022-JP?B?GyRCI1QjRSNTI1QbKEIAGyRCI00jQSNJI0wbKEI=?X-Spam-Score:
100.00%
?
Subject: [SPAM] =?utf-8?b?77y077yl77yz77y0?X-Spam-Score: 100.00%


-Environment
?OS
CentOS release 6.5 (Final) x86_64

?Software Version
dovecot-2.2.12
dovecot-2.2-pigeonhole-0.4.2

?Results from the "dovecot -n" command.
----------------------------------------------------------------------
# 2.2.12: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-431.5.1.el6.x86_64 x86_64 CentOS release 6.5 (Final)
disable_plaintext_auth = no
lock_method = dotlock
mail_fsync = always
mail_gid = vmail
mail_location = maildir:~/Maildir
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = quota
mail_temp_dir = /var/tmp
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress

comparator-i;ascii-numeric relational regex imap4flags copy include
variables body enotify

environment mailbox date ihave spamtest spamtestplus editheader
mbox_write_locks = dotlock
mmap_disable = yes
passdb {
  driver = passwd
}
plugin {
  quota = maildir:User quota
  sieve = ~/Maildir/dovecot.sieve
  sieve_dir = ~/Maildir/sieve
  sieve_editheader_max_header_size = 1k
  sieve_editheader_protected = X-Verified
  sieve_extensions = +editheader +spamtest +spamtestplus
  sieve_max_actions = 250
  sieve_max_redirects = 100
  sieve_spamtest_max_value = 100.00
  sieve_spamtest_status_header = X-Spam-Score: ([[:digit:]]+\.[[:digit:]]+)%
  sieve_spamtest_status_type = score
  sieve_user_log = /var/log/dovecot.sieve/%d/%u.log
}
postmaster_address = postmaster at example.jp
protocols = pop3 imap
service auth {
  unix_listener auth-userdb {
    group = vmail
    mode = 0600
    user = vmail
  }
}
ssl_ca = </etc/pki/tls/certs/CA.crt
ssl_cert = </etc/pki/tls/certs/example.jp.crt
ssl_key = </etc/pki/tls/private/example.jp.key
submission_host = 127.0.0.1:20025
userdb {
  args = /etc/dovecot/ldap/userdb.conf.ext
  driver = ldap
}
protocol lda {
  mail_plugins = quota sieve
}
protocol imap {
  mail_plugins = quota imap_quota
  passdb {
    args = /etc/dovecot/ldap/passdb.conf.ext
    driver = ldap
  }
}
protocol pop3 {
  passdb {
    args = /etc/dovecot/ldap/passdb.conf.ext
    driver = ldap
  }
}
----------------------------------------------------------------------

-Contents of the "dovecot.sieve" file
user/Maildir/dovecot.sieve
----------------------------------------------------------------------
require "fileinto";
require "mailbox";
require "envelope";
require "variables";
require "editheader";
require "spamtestplus";
require "relational";
require "enotify";
require "comparator-i;ascii-numeric";

if spamtest :percent :value "ge" :comparator
"i;ascii-numeric" "80" {
  if exists "subject" {
    deleteheader :matches "subject" "*";
    addheader "Subject" "[SPAM] ${1}";
  } else {
    addheader "Subject" "[SPAM]";
  }
}
----------------------------------------------------------------------

Please reply.
Thank you.

-- 
----------------------------------------
Atsuko Tanaka
DesigNET Inc.
e-mail?tanaka at designet.co.jp
----------------------------------------

On 2.5.2014, at 11.27, Atsuko Tanaka <tanaka at designet.co.jp> wrote:
> We have currently set dovecot.sieve to insert the text "[SPAM]"
at the
> beginning of an e-mail's subject when it's X-Spam-Score is above
80%.
> After we set our system as stated the following errors occur:
> 
> 1) When an e-mail's subject contains control characters like
> [Ctrl+V|^V], dovecot.sieve terminates with an error and an e-mail is not
> able to be sent. When a MIME encoded Subject like [????^V????] is sent
> we're not able to edit the subject and dovecot ends with an error.
..
> Aside from [Ctrl + V] the following control charcters also cause errors:
> backspace
> Ctrl + A
> Ctrl + C
> Ctrl + [
> Ctrl + X
> Ctrl + Y
> 
> 2) When an e-mail's subject line contains a "\0" character,
everything
> following the null character is deleted.
Is there a reason why mails contain these kind of control characters? Is it
commonly used? And is it OK if Dovecot translates them to UTF-8 for the
rewritten header, or would the subject have to stay ISO-2022-JP encoded?

We were able to solve the problem,
"When a control character is included in the subject of an e-mail,
dovecot exits with a fatal error" using the following information:


1. We added the following field to the sieve setting file:

/etc/dovecot/conf.d/90-sieve.conf
---------------------------------------------------------------------
sieve_editheader_rfc2822_check = yes
---------------------------------------------------------------------
This is a switch for whether or not to check if a file is RFC2822 or not.
When it's set to yes, it works the exact same as before changes were made.
The default is yes.

2. When the sieve_editheader_rfc2822_check setting is set to no, we make
sure not to check for RFC2822.


Version: dovecot-2.2-pigeonhole-0.4.2

The patch is listed as below:
---------------------------------------------------------------------
diff -Nur
dovecot-2.2-pigeonhole-0.4.2_org/src/lib-sieve/plugins/editheader/cmd-addheader.c
dovecot-2.2-pigeonhole-0.4.2/src/lib-sieve/plugins/editheader/cmd-addheader.c
---
dovecot-2.2-pigeonhole-0.4.2_org/src/lib-sieve/plugins/editheader/cmd-addheader.c
2014-05-14 15:17:52.586774630 +0900
+++
dovecot-2.2-pigeonhole-0.4.2/src/lib-sieve/plugins/editheader/cmd-addheader.c
2014-05-14 15:22:12.536780572 +0900
@@ -281,7 +281,8 @@
 		return SIEVE_EXEC_OK;
 	}
 
-	if ( !rfc2822_header_field_body_verify
+	if ( this_ext->svinst->chk_rfc2822 == TRUE &&
+             !rfc2822_header_field_body_verify
 		(str_c(value), str_len(value), TRUE, TRUE) ) {
 		sieve_runtime_error(renv, NULL, "addheader action: "
 			"specified value `%s' is invalid",
diff -Nur dovecot-2.2-pigeonhole-0.4.2_org/src/lib-sieve/sieve-common.h
dovecot-2.2-pigeonhole-0.4.2/src/lib-sieve/sieve-common.h
--- dovecot-2.2-pigeonhole-0.4.2_org/src/lib-sieve/sieve-common.h	2014-05-14
15:17:52.593774606 +0900
+++ dovecot-2.2-pigeonhole-0.4.2/src/lib-sieve/sieve-common.h	2014-05-14
15:23:57.584775286 +0900
@@ -189,6 +189,9 @@
 	size_t max_script_size;
 	unsigned int max_actions;
 	unsigned int max_redirects;
+
+        /* Check */
+        bool chk_rfc2822;
 };
 
 #endif /* __SIEVE_COMMON_H */
diff -Nur dovecot-2.2-pigeonhole-0.4.2_org/src/lib-sieve/sieve.c
dovecot-2.2-pigeonhole-0.4.2/src/lib-sieve/sieve.c
--- dovecot-2.2-pigeonhole-0.4.2_org/src/lib-sieve/sieve.c	2014-05-14
15:17:52.583774672 +0900
+++ dovecot-2.2-pigeonhole-0.4.2/src/lib-sieve/sieve.c	2014-05-14
15:19:39.628771207 +0900
@@ -51,6 +51,7 @@
 	size_t size_setting;
 	const char  *domain;
 	pool_t pool;
+        bool bool_setting;
 
 	/* Create Sieve engine instance */
 	pool = pool_alloconly_create("sieve", 8192);
@@ -118,6 +119,15 @@
 		svinst->max_redirects = (unsigned int) uint_setting;
 	}
 
+	/* Check RFC2822 from configuration */
+
+        svinst->chk_rfc2822 = TRUE;
+
+        if (sieve_setting_get_bool_value
+		(svinst, "sieve_editheader_rfc2822_check", &bool_setting) ) {
+                svinst->chk_rfc2822 = bool_setting;
+        }
+
 	/* Initialize extensions */
 	if ( !sieve_extensions_init(svinst) ) {
 		sieve_deinit(&svinst);
---------------------------------------------------------------------

> 1) When an e-mail's subject contains control characters like
> [Ctrl+V|^V], dovecot.sieve terminates with an error and an e-mail is not
> able to be sent. When a MIME encoded Subject like [????^V????] is sent
> we're not able to edit the subject and dovecot ends with an error.
> 
> This is a sample of the data that was used in testing.
> Subject: =?ISO-2022-JP?B?GyRCI1QjRSNTI1QbKEIWGyRCI00jQSNJI0wbKEI=?>
X-Spam-Score: 100.00%
> ?
> Subject: =?ISO-2022-JP?B?GyRCI1QjRSNTI1QbKEIWGyRCI00jQSNJI0wbKEI=?>
X-Spam-Score: 100.00%
> 
> This is the log generated by the data above.
> ----------------------------------------------------------------------
> sieve: info: started log at May 02 10:46:22.
> main script: line 14: error: addheader action: specified value `[SPAM]
> ???????' is invalid.
> ----------------------------------------------------------------------
> 
> Aside from [Ctrl + V] the following control charcters also cause errors:
> backspace
> Ctrl + A
> Ctrl + C
> Ctrl + [
> Ctrl + X
> Ctrl + Y

Atsuko Tanaka