thr3ads.net - dovecot - SMTPUTF8 support [Apr 2019]

If this information is useful, please help other people find it:
Share via:

David Bürgin

2019-Apr-05 07:57 UTC

SMTPUTF8 support

Andr?, are you quite sure you have it working?

In this thread someone from Open-Xchange stated that no, Dovecot doesn?t
have SMTPUTF8 support implemented, and the same response was given by
another Dovecot developer last September (it ?is being considered? was
the answer then, see
https://dovecot.org/pipermail/dovecot/2018-September/112887.html).

I am using LMTP to deliver mail to Dovecot from Postfix, and delivery
fails with Postfix reporting: ?SMTPUTF8 is required, but was not offered
by host mail.my.org[private/dovecot-lmtp]?. I doubt that this can work
without a change in Dovecot LMTP.

Anyway, I am also interested in SMTPUTF8 support, so +1 from me!


-- 
David

André Rodier

2019-Apr-05 19:47 UTC

head link

SMTPUTF8 support

On 2019-04-05 08:57, David B?rgin via dovecot wrote:> Andr?, are you quite sure you have it working?
> 
> In this thread someone from Open-Xchange stated that no, Dovecot 
> doesn?t
> have SMTPUTF8 support implemented, and the same response was given by
> another Dovecot developer last September (it ?is being considered? was
> the answer then, see
> https://dovecot.org/pipermail/dovecot/2018-September/112887.html).
> 
> I am using LMTP to deliver mail to Dovecot from Postfix, and delivery
> fails with Postfix reporting: ?SMTPUTF8 is required, but was not 
> offered
> by host mail.my.org[private/dovecot-lmtp]?. I doubt that this can work
> without a change in Dovecot LMTP.
> 
> Anyway, I am also interested in SMTPUTF8 support, so +1 from me!
Hello David,

Well, it seems to work from one Homebox to another, for instance:

Here the server logs:
----------------------------------> Apr 05 19:20:46 portal postfix/qmgr[21470]: 6AC3240807: 
> from=<andre at rodier.me>, size=3066, nrcpt=1 (queue active)
> Apr 05 19:20:46 portal clamsmtpd[14542]: 10004D: from=andre at rodier.me, 
> to=andr?@homebox.space, status=CLEAN
> Apr 05 19:20:46 portal postfix/smtp[30761]: 1FCA540467: 
> to=<andr?@homebox.space>, relay=127.0.0.1[127.0.0.1]:34001,
delay=1.4,
> delays=0.73/0.02/0.07/0.59, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued 
> as 6AC3240807)
> Apr 05 19:20:46 portal postfix/smtpd[30763]: disconnect from 
> localhost[127.0.0.1] ehlo=1 xforward=2 mail=1 rcpt=1 data=1 quit=1 
> commands=7
> Apr 05 19:20:46 portal postfix/qmgr[21470]: 1FCA540467: removed
> Apr 05 19:20:46 portal dovecot[12813]: lmtp(30766): Connect from local
> Apr 05 19:20:46 portal dovecot[12813]: lmtp(andre): 
> WGTdAo6qp1wueAAAeNB+QA: sieve: 
> msgid=<7d53-5ca7aa80-1-51005a80 at 31476855>: stored mail into
mailbox
> 'INBOX'
> Apr 05 19:20:46 portal postfix/lmtp[30765]: 6AC3240807: 
> to=<andre at homebox.space>, orig_to=<andr?@homebox.space>, 
> relay=portal.homebox.space[private/dovecot-lmtp], delay=0.62, 
> delays=0.59/0.01/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 
> <andre at homebox.space> WGTdAo6qp1wueAAAeNB+QA Saved)
> Apr 05 19:20:46 portal dovecot[12813]: lmtp(30766): Disconnect from 
> local: Successful quit
> Apr 05 19:20:46 portal postfix/qmgr[21470]: 6AC3240807: removed----------------------------------

And the message source:
----------------------------------> Return-Path: <andre at rodier.me>
> Delivered-To: andre at homebox.space
> Received: from portal.homebox.space
> 	by portal (Dovecot) with LMTP id WGTdAo6qp1wueAAAeNB+QA
> 	for <andre at homebox.space>; Fri, 05 Apr 2019 19:20:46 +0000
> Received: from portal.homebox.space (localhost [127.0.0.1])
> 	by portal.homebox.space (Postfix) with ESMTP id 6AC3240807
> 	for <andr??@homebox.space>; Fri,  5 Apr 2019 19:20:45 +0000 (UTC)
> Authentication-Results: portal.homebox.space;
> 	dkim=pass (4096-bit key; unprotected) header.d=rodier.me 
> header.i=@rodier.me header.b="bNaIhJcQ";
> 	dkim-atps=neutral
> Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=92.19.253.42; 
> helo=osaka.rodier.me; envelope-from=andre at rodier.me;
receiver=<UNKNOWN>
> Received: from osaka.rodier.me (host-92-19-253-42.static.as13285.net 
> [92.19.253.42])
> 	by portal.homebox.space (Postfix) with ESMTPS id 1FCA540467
> 	for <andr??@homebox.space>; Fri,  5 Apr 2019 19:20:44 +0000 (UTC)
> Received: from osaka.rodier.me (localhost [127.0.0.1])
> 	by osaka.rodier.me (Postfix) with ESMTP id 68658201DE
> 	for <andr??@homebox.space>; Fri,  5 Apr 2019 20:20:43 +0100 (BST)
> Received: from localhost (localhost [127.0.0.1])
> 	by osaka.rodier.me (Postfix) with ESMTPA id 32C1A200CC
> 	for <andr??@homebox.space>; Fri,  5 Apr 2019 20:20:43 +0100 (BST)
> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=rodier.me; 
> s=main;
> 	t=1554492043; bh=ojnzW/eF3u3/ihaSAhQfvUWEX7Blx4fCTTVCC9mfgZE=;
> 	h=From:Date:To:Subject:From;
> 	b=bNaIhJcQtD7/IbjI/VGUIWpOk1QyYioKsgsCva8sL040L7VDMmLgfbQ5TWFwO/wub
> 	 +8THBgOTWj/39FyoKQJYk0BahQ7Z4LkUrH20Jcq5yDEOC7q/xfwIwUV/sUaj6C3wLZ
> 	 VGUp8Fg2oVDD+ciGEnfa2OuEC/LajvmXoDqUHHtSwTFZvfkh1nRDqxcNd/FSyvYQe8
> 	 frgBySFgzI+xTWVG7Oa07CQPG+HenqwrPyaZD0SNpHnrvH5ICdAriv2UMKgQnsWO80
> 	 aSz4EDySdwMah17kfh4u06GlXd3duQCN8hyxBBDjcI879N/+CQM3Uz1NTbIDe82Lf7
> 	 bSUJzvMczCdZFm5J84hvLd+j85pY7T9052JTcUANUQPnTc1Q3yrQFC3QuKD8Amb8VO
> 	 qHR0q0OP6SYqoQ0g/moMF2FhWaQg3pCRpIEOS73Ncnnf+fJGQJnPi8G8YcLVLZCDCx
> 	 RlSauA/YuF5tiIBHFLdjtlgVgwk38tT8ck+D8kS3BAB4nXBVlreCiK0vMK8HyoCpsC
> 	 LMJ0e5KhBzCDXOPvSLiohn2zcmYpa9DywmWFVTZNx3INzTH7nsa40AYGFl+k+G+pe+
> 	 ej+MRDwmdOePfl3DGGEgpn0JMNiAmECe6gHA+/N1TTk5SbbiKLNh8KXzrZECWVrvUx
> 	 55kqY/LPuQEwxCnA/VZsQbDk> Content-Type: multipart/alternative; 
>
boundary="----=_=-_OpenGroupware_org_NGMime-32083-1554492042.981625-0------"
>  From: =?utf-8?q?Andr=C3=A9_Rodier?= <andre at rodier.me>
> X-Forward: 127.0.0.1
> Date: Fri, 05 Apr 2019 20:20:42 +0100
> To: =?utf-8?q?andr=E9=40homebox=2Espace?=@rodier.me
> MIME-Version: 1.0
> Message-ID: <7d53-5ca7aa80-1-51005a80 at 31476855>
> Disposition-Notification-To: andre at rodier.me
> Subject: test 2020
> User-Agent: SOGoMail
> X-AV-Checked: ClamSMTP (Client IP: 127.0.0.1)
> Authentication-Results: portal.homebox.space;
> 	spf=temperror (portal.homebox.space: error in processing during lookup 
> of andre at rodier.me: DNS error) smtp.mailfrom=andre at rodier.me
> X-Spam-Status: Yes, score=6.61
> X-AV-Checked: ClamSMTP (Client IP: 127.0.0.1)
> X-Spam: Yes
> 
> ------=_=-_OpenGroupware_org_NGMime-32083-1554492042.981625-0------
> Content-Type: text/plain; charset=utf-8
> Content-Transfer-Encoding: 7bit
> Content-Length: 15
> 
> 
> This is a test
> 
> ------=_=-_OpenGroupware_org_NGMime-32083-1554492042.981625-0------
> Content-Type: text/html; charset=utf-8
> Content-Transfer-Encoding: 7bit
> Content-Length: 27
> 
> <html>This is a test</html>
> ------=_=-_OpenGroupware_org_NGMime-32083-1554492042.981625-0------------------------------------------

However, it seems that mails from GMails are not received, and I have an error
message from the postfix mailter:
> Apr  5 10:16:39 portal postfix/smtpd[2566]: Anonymous TLS connection
established from mail-lf1-f51.google.com[209.85.167.51]: TLSv1.2 with cipher
ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
> Apr  5 10:16:39 portal policyd-spf[2574]: prepend Received-SPF: Pass
(mailfrom) identity=mailfrom; client-ip=209.85.167.51;
helo=mail-lf1-f51.google.com; envelope-from=andre.rodier at gmail.com;
receiver=<UNKNOWN>
> Apr  5 10:16:39 portal postfix/smtpd[2566]: EB38240467:
client=mail-lf1-f51.google.com[209.85.167.51]
> Apr  5 10:16:40 portal postfix/cleanup[2576]: EB38240467:
message-id=<CAKr9_Nv1pgDAC31cFF4YPE-mnYxSusmKsWHOCxWPSeJ2Adj8cQ at
mail.gmail.com>
> Apr  5 10:16:40 portal opendkim[15477]: EB38240467: mail-lf1-f51.google.com
[209.85.167.51] not internal
> Apr  5 10:16:40 portal opendkim[15477]: EB38240467: not authenticated
> Apr  5 10:16:40 portal opendkim[15477]: EB38240467: DKIM verification
successful
> Apr  5 10:16:40 portal opendkim[15477]: EB38240467: s=20161025 d=gmail.com
SSL
> Apr  5 10:16:40 portal opendmarc[16548]: implicit authentication service:
portal.homebox.space
> Apr  5 10:16:40 portal opendmarc[16548]: EB38240467: gmail.com pass
> Apr  5 10:16:40 portal postfix/cleanup[2576]: EB38240467: milter-reject:
END-OF-MESSAGE from mail-lf1-f51.google.com[209.85.167.51]: 4.7.1 Try again
later; from=<andre.rodier at gmail.com> to=<andr?@homebox.space>
proto=ESMTP helo=<mail-lf1-f51.google.com>
> Apr  5 10:16:40 portal postfix/smtpd[2566]: disconnect from
mail-lf1-f51.google.com[209.85.167.51] ehlo=2 starttls=1 mail=1 rcpt=1 data=0/1
quit=1 commands=6/7
Thanks to you, I have another bug to chase with gmail and postfix
cleanup. I could ask Google to use HomeBox, but I am not sure they
would agree ;-)

Regarding your issue, there might be many causes.

I remember one, with the dovecot, for the Quota plugin that failed to lookup the
user, I had to add "quota_status_nouser = DUNNO",
so it does not fail when the user is not found (if I understood well).

Another note, I am using LDAP lookups, _and_ a specific schema for unicode email
addresses. Otherwise, this would not work:

root at portal:/etc/postfix# cat ldap-aliases.cf 
# Basic LDAP settings for postfix
server_host = ldap://ldap.homebox.space/
start_tls = yes

search_base = dc=homebox,dc=space
search_scope = sub

# Allow advanced LDAP search with the recipient delimiter
query_filter = (|(mail=%s)(intlMailAddr=%s))
result_attribute = mail

# Force support for UTF8
version = 3

Thanks to this, ldap lookup is unicode compliant:
> root at portal:/etc/postfix# postmap -q andr?@homebox.space
ldap:/etc/postfix/ldap-aliases.cf
> andre at homebox.space
>
The LDAP schema for internationalised email addresses:
> # The attribute type 'intlMailAddr' is defined for storing SMTPUTF8
> # compliant addresses [RFC6530]
> # https://tools.ietf.org/html/draft-stroeder-mailboxrelatedobject-07
> 
> dn: cn=intlMailAddr,cn=schema,cn=config
> objectClass: olcSchemaConfig
> cn: intlMailAddr
> olcAttributeTypes: ( 1.3.6.1.4.1.5427.1.389.4.18
>   NAME 'intlMailAddr'
>   DESC 'Internationalized Email Address'
>   EQUALITY caseIgnoreMatch
>   SUBSTR caseIgnoreSubstringsMatch
>   SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )
> olcObjectClasses: ( 1.3.6.1.4.1.5427.1.389.6.9
>   NAME 'mailboxRelatedObject'
>   DESC 'Associated RFC 5321 mailbox for any entry'
>   AUXILIARY
>   MAY ( displayName $ mail $ intlMailAddr ) )
> 
As I wrote earlier, All the software in the chain should be UTF-8 ready

-- 
Andr? Rodier
HomeBox: https://github.com/progmaticltd/homebox

David Bürgin

2019-Apr-05 20:18 UTC

head link

SMTPUTF8 support

Well, what a puzzle. I will have to investigate. I also tried it with
Gmail, and the failure was definitely between Postfix and Dovecot LMTP,
with Dovecot failing to advertise SMTPUTF8 support. My setup is much
more minimal, I use file-based virtual mailboxes and aliases, and only
OpenDKIM and SpamAssassin milters.

Thank you for your reply, Andr?!

David Bürgin

2019-Apr-05 20:36 UTC

head link

SMTPUTF8 support

I?m afraid there is a possibility that your setup only works by chance ?
I don?t think Dovecot LMTP is SMTPUTF8-capable according to RFC 6531. It
needs to advertise the SMTPUTF8 extension but doesn?t.

# nc -U /var/spool/postfix/private/dovecot-lmtp
220 mail.my.org Dovecot ready.
LHLO mail.my.org
250-mail.my.org
250-STARTTLS
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 PIPELINING


-- 
David

André Rodier

2019-Apr-06 08:07 UTC

head link

SMTPUTF8 support

On 05/04/2019 08:57, David B?rgin via dovecot wrote:> Andr?, are you quite sure you have it working?
> 
> In this thread someone from Open-Xchange stated that no, Dovecot doesn?t
> have SMTPUTF8 support implemented, and the same response was given by
> another Dovecot developer last September (it ?is being considered? was
> the answer then, see
> https://dovecot.org/pipermail/dovecot/2018-September/112887.html).
> 
> I am using LMTP to deliver mail to Dovecot from Postfix, and delivery
> fails with Postfix reporting: ?SMTPUTF8 is required, but was not offered
> by host mail.my.org[private/dovecot-lmtp]?. I doubt that this can work
> without a change in Dovecot LMTP.
> 
> Anyway, I am also interested in SMTPUTF8 support, so +1 from me!
> 
> 
Hello David et al,

I have it working with my set up, and I confirm it is not easy.

If I limit the stack to OpenLDAP, Postfix and Dovecot to their simplest 
configuration, it is relatively easy, even with slightly outdated 
software in Debian Stretch.

The trick I am using is to include the internationalised email address 
aside the main one, and to use Postfix LDAP lookup.

For instance, I am using this LDAP schema:
https://tools.ietf.org/html/draft-stroeder-mailboxrelatedobject-07

So, my LDAP lookup table for Postfix looks like this:

     # Basic LDAP settings for postfix
     server_host = ldap://ldap.rodier.me/
     start_tls = yes

     search_base = dc=rodier,dc=me
     search_scope = sub

     # Allow advanced LDAP search with the recipient delimiter
     query_filter = (|(mail=%s)(intlMailAddr=%s))
     result_attribute = mail

     # Force support for UTF8
     version = 3

Therefore, postmap query works as expected:

     postmap -q andr?@rodier.me ldap:/etc/postfix/ldap-aliases.cf
     andre at rodier.me

That said, if only one of the milter reject your message, it is breaking 
the whole chain.

For instance, the quota plugin for Dovecot, I had to use DUNNO when the 
user is unknown instead of reject.

I have it working perfectly from Homebox to Homebox, so perhaps Postfix 
to Postfix, but I reckon I still have some bugs.

Some providers are working, others not. Outlook does not seems to work , 
but I am on it. Yahoo does not let you type an accent in the email 
address, only Google allows you to.

However, I made a test yesterday, and realised that the emails from 
GMail are rejected by one of the plugin, and I need to investigate which 
one. The error message is:
> Apr  5 10:16:39 portal postfix/smtpd[2566]: Anonymous TLS connection
established from mail-lf1-f51.google.com[209.85.167.51]: TLSv1.2 with cipher
ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
> Apr  5 10:16:39 portal policyd-spf[2574]: prepend Received-SPF: Pass
(mailfrom) identity=mailfrom; client-ip=209.85.167.51;
helo=mail-lf1-f51.google.com; envelope-from=andre.rodier at gmail.com;
receiver=<UNKNOWN>
> Apr  5 10:16:39 portal postfix/smtpd[2566]: EB38240467:
client=mail-lf1-f51.google.com[209.85.167.51]
> Apr  5 10:16:40 portal postfix/cleanup[2576]: EB38240467:
message-id=<CAKr9_Nv1pgDAC31cFF4YPE-mnYxSusmKsWHOCxWPSeJ2Adj8cQ at
mail.gmail.com>
> Apr  5 10:16:40 portal opendkim[15477]: EB38240467: mail-lf1-f51.google.com
[209.85.167.51] not internal
> Apr  5 10:16:40 portal opendkim[15477]: EB38240467: not authenticated
> Apr  5 10:16:40 portal opendkim[15477]: EB38240467: DKIM verification
successful
> Apr  5 10:16:40 portal opendkim[15477]: EB38240467: s=20161025 d=gmail.com
SSL
> Apr  5 10:16:40 portal opendmarc[16548]: implicit authentication service:
portal.homebox.space
> Apr  5 10:16:40 portal opendmarc[16548]: EB38240467: gmail.com pass
> Apr  5 10:16:40 portal postfix/cleanup[2576]: EB38240467: milter-reject:
END-OF-MESSAGE from mail-lf1-f51.google.com[209.85.167.51]: 4.7.1 Try again
later; from=<andre.rodier at gmail.com> to=<andr?@homebox.space>
proto=ESMTP helo=<mail-lf1-f51.google.com>
> Apr  5 10:16:40 portal postfix/smtpd[2566]: disconnect from
mail-lf1-f51.google.com[209.85.167.51] ehlo=2 starttls=1 mail=1 rcpt=1 data=0/1
quit=1 commands=6/7
So perhaps it is one of the milter. I will investigate today.

Also, it seems the client is important. I have no problem with SOGo or 
Evolution. Roundcube and Thunderbird, in Debian Stretch, don't support 
this yet.

With Debian Buster coming soon, I am sure this will work better. I will 
create a branch to test this. My preliminary investigations don't show 
major problems in the upgrade.

At one point, I was using the Dovecot packages from backports, but I 
reverted this as there has been an incompatibility.

Happy to answer more questions if you have.

-- 
Andr? Rodier

John Fawcett

2019-Apr-06 09:19 UTC

head link

SMTPUTF8 support

On 06/04/2019 10:07, Andr? Rodier via dovecot wrote:> On 05/04/2019 08:57, David B?rgin via dovecot wrote:
>> Andr?, are you quite sure you have it working?
>>
>> In this thread someone from Open-Xchange stated that no, Dovecot
doesn?t
>> have SMTPUTF8 support implemented, and the same response was given by
>> another Dovecot developer last September (it ?is being considered? was
>> the answer then, see
>> https://dovecot.org/pipermail/dovecot/2018-September/112887.html).
>>
>> I am using LMTP to deliver mail to Dovecot from Postfix, and delivery
>> fails with Postfix reporting: ?SMTPUTF8 is required, but was not
offered
>> by host mail.my.org[private/dovecot-lmtp]?. I doubt that this can work
>> without a change in Dovecot LMTP.
>>
>> Anyway, I am also interested in SMTPUTF8 support, so +1 from me!
>>
>>
>
> Hello David et al,
>
> I have it working with my set up, and I confirm it is not easy.
>
> If I limit the stack to OpenLDAP, Postfix and Dovecot to their
> simplest configuration, it is relatively easy, even with slightly
> outdated software in Debian Stretch. 
I think you are confusing acceptance of UTF8 in the localpart of the
email address (which if I remember correctly worked in the 2.2 branch)
with SMTPUTF8 support. As mentioned in the thread above Dovecot does not
as yet support SMTPUTF8. Support for SMTPUTF8 would mean among other
things changing the source code of Dovecot so that it announces the
SMTPUTF8 key word to the upstream MTA.

John

@lbutlr

2019-Apr-06 12:53 UTC

head link

SMTPUTF8 support

On 5 Apr 2019, at 13:47, Andr? Rodier via dovecot <dovecot at dovecot.org>
wrote:> 
>> root at portal:/etc/postfix# postmap -q andr?@homebox.space
ldap:/etc/postfix/ldap-aliases.cf
>> andre at homebox.space
You have a solution that properly maps UTF to a non-UTF namespace. As I
understand it, SMTPUTF8 uses a UTF8 namespace.

So, this would indicate SMTPUTF8

# postmap -q andr?@homebox.space ldap:/etc/postfix/ldap-aliases.cf 
andr?@homebox.space

It's a hard problem to solve because you have to, as you've already
discovered, replace the entire chain with SMTPUTF8 compatible tools all at once.


-- 
Is it my imagination, or do buffalo wings taste like chicken?

Possibly Parallel Threads

Search for more maybe matching threads

dovecot - Apr 2019 - SMTPUTF8 support

SMTPUTF8 support

SMTPUTF8 support

SMTPUTF8 support

SMTPUTF8 support

SMTPUTF8 support

SMTPUTF8 support

SMTPUTF8 support

Possibly Parallel Threads