dovecot - Sep 2018 - Dovecot submission announces CHUNKING but backend Postfix does not support it

On 09/20/2018 12:56 PM, Robert Schetterer wrote:
> Am 20.09.2018 um 11:04 schrieb Michael Goth:
>> Hello,
>>
>> I'm setting up Dovecot 2.3.2.1 as a submission proxy to a Postfix 
>> backend server. Dovecot announces CHUNKING, but the Postfix backend 
>> does not support (or announce) it.
>>
>> HELO from Postfix:
>>
>> ?? 220 backend.mydomain.com ESMTP Postfix (Ubuntu)
>> ?? ehlo test
>> ?? 250-backend.mydomain.com
>> ?? 250-PIPELINING
>> ?? 250-SIZE 104857600
>> ?? 250-VRFY
>> ?? 250-ETRN
>> ?? 250-STARTTLS
>> ?? 250-ENHANCEDSTATUSCODES
>> ?? 250-8BITMIME
>> ?? 250 DSN
>>
>>
>> HELO from Dovecot proxy:
>>
>> ?? 220 proxy.mydomain.com Dovecot ready.
>> ?? ehlo test
>> ?? 250-proxy.mydomain.com
>> ?? 250-8BITMIME
>> ?? 250-AUTH PLAIN LOGIN
>> ?? 250-BURL imap
>> ?? 250-CHUNKING
>> ?? 250-ENHANCEDSTATUSCODES
>> ?? 250-SIZE
>> ?? 250-STARTTLS
>> ?? 250 PIPELINING
>>
>>
>> When a client sends BDAT, Postfix closes the connection. Here's a
log
>> snippet from Postfix:
>>
>> ?? < MAIL FROM:<user at mydomain.com>
>> ?? > 250 2.1.0 Ok
>> ?? < RCPT TO:<some at recipient.com>
>> ?? > 250 2.1.5 Ok
>> ?? < BDAT 114098 LAST
>> ?? > 502 5.5.2 Error: command not recognized
>> ?? < Content-Type: multipart/alternative; 
>> boundary=Apple-Mail-55D35F74-2EB7-4B3B-A607-E421DD71C07A
>> ?? > 221 2.7.0 Error: I can break rules, too. Goodbye.
>>
>>
>> Apple Mail seems to ignore the fact that Postfix does not understand 
>> BDAT and Postfix does not like that :-)
>>
>>
>> ?From a quick glance at the code it seems CHUNKING is always added to 
>> the capabilities and that's not configurable. Is this a bug or am I
>> doing something wrong?
>>
>>
>> Thanks,
>>
>> ??Michael
> 
> that "may" version depend
> 
> http://www.postfix.org/features.html
> ...
> Postfix 3.4???? SMTP server support for RFC 3030 CHUNKING (without 
> BINARYMIME).
> ...
> 
As far as I understand, Dovecot always adds certain capabilities. This 
is the responsible code (I think):

https://github.com/dovecot/core/blob/c8d03c3cab68328947a5afb47f48aef5b5a1e4ab/src/submission/submission-client.c#L95

> On 20 September 2018 at 14:39 Michael Goth <mg at webflow.de> wrote:
> 
> 
> On 09/20/2018 12:56 PM, Robert Schetterer wrote:
> > Am 20.09.2018 um 11:04 schrieb Michael Goth:
> >> Hello,
> >>
> >> I'm setting up Dovecot 2.3.2.1 as a submission proxy to a
Postfix
> >> backend server. Dovecot announces CHUNKING, but the Postfix
backend
> >> does not support (or announce) it.
> >>
> >> HELO from Postfix:
> >>
> >> ?? 220 backend.mydomain.com ESMTP Postfix (Ubuntu)
> >> ?? ehlo test
> >> ?? 250-backend.mydomain.com
> >> ?? 250-PIPELINING
> >> ?? 250-SIZE 104857600
> >> ?? 250-VRFY
> >> ?? 250-ETRN
> >> ?? 250-STARTTLS
> >> ?? 250-ENHANCEDSTATUSCODES
> >> ?? 250-8BITMIME
> >> ?? 250 DSN
> >>
> >>
> >> HELO from Dovecot proxy:
> >>
> >> ?? 220 proxy.mydomain.com Dovecot ready.
> >> ?? ehlo test
> >> ?? 250-proxy.mydomain.com
> >> ?? 250-8BITMIME
> >> ?? 250-AUTH PLAIN LOGIN
> >> ?? 250-BURL imap
> >> ?? 250-CHUNKING
> >> ?? 250-ENHANCEDSTATUSCODES
> >> ?? 250-SIZE
> >> ?? 250-STARTTLS
> >> ?? 250 PIPELINING
> >>
> >>
> >> When a client sends BDAT, Postfix closes the connection.
Here's a log
> >> snippet from Postfix:
> >>
> >> ?? < MAIL FROM:<user at mydomain.com>
> >> ?? > 250 2.1.0 Ok
> >> ?? < RCPT TO:<some at recipient.com>
> >> ?? > 250 2.1.5 Ok
> >> ?? < BDAT 114098 LAST
> >> ?? > 502 5.5.2 Error: command not recognized
> >> ?? < Content-Type: multipart/alternative; 
> >> boundary=Apple-Mail-55D35F74-2EB7-4B3B-A607-E421DD71C07A
> >> ?? > 221 2.7.0 Error: I can break rules, too. Goodbye.
> >>
> >>
> >> Apple Mail seems to ignore the fact that Postfix does not
understand
> >> BDAT and Postfix does not like that :-)
> >>
> >>
> >> ?From a quick glance at the code it seems CHUNKING is always added
to
> >> the capabilities and that's not configurable. Is this a bug or
am I
> >> doing something wrong?
> >>
> >>
> >> Thanks,
> >>
> >> ??Michael
> > 
> > that "may" version depend
> > 
> > http://www.postfix.org/features.html
> > ...
> > Postfix 3.4???? SMTP server support for RFC 3030 CHUNKING (without 
> > BINARYMIME).
> > ...
> > 
> 
> As far as I understand, Dovecot always adds certain capabilities. This 
> is the responsible code (I think):
> 
>
https://github.com/dovecot/core/blob/c8d03c3cab68328947a5afb47f48aef5b5a1e4ab/src/submission/submission-client.c#L95
Hi!

Thank you for reporting this bug, we'll look into it! 

Aki

Op 20/09/2018 om 13:39 schreef Michael Goth:
> On 09/20/2018 12:56 PM, Robert Schetterer wrote:
>> Am 20.09.2018 um 11:04 schrieb Michael Goth:
>>> Hello,
>>>
>>> I'm setting up Dovecot 2.3.2.1 as a submission proxy to a
Postfix
>>> backend server. Dovecot announces CHUNKING, but the Postfix backend
>>> does not support (or announce) it.
>>>
>>> HELO from Postfix:
>>>
>>> ?? 220 backend.mydomain.com ESMTP Postfix (Ubuntu)
>>> ?? ehlo test
>>> ?? 250-backend.mydomain.com
>>> ?? 250-PIPELINING
>>> ?? 250-SIZE 104857600
>>> ?? 250-VRFY
>>> ?? 250-ETRN
>>> ?? 250-STARTTLS
>>> ?? 250-ENHANCEDSTATUSCODES
>>> ?? 250-8BITMIME
>>> ?? 250 DSN
>>>
>>>
>>> HELO from Dovecot proxy:
>>>
>>> ?? 220 proxy.mydomain.com Dovecot ready.
>>> ?? ehlo test
>>> ?? 250-proxy.mydomain.com
>>> ?? 250-8BITMIME
>>> ?? 250-AUTH PLAIN LOGIN
>>> ?? 250-BURL imap
>>> ?? 250-CHUNKING
>>> ?? 250-ENHANCEDSTATUSCODES
>>> ?? 250-SIZE
>>> ?? 250-STARTTLS
>>> ?? 250 PIPELINING
>>>
>>>
>>> When a client sends BDAT, Postfix closes the connection. Here's
a
>>> log snippet from Postfix:
>>>
>>> ?? < MAIL FROM:<user at mydomain.com>
>>> ?? > 250 2.1.0 Ok
>>> ?? < RCPT TO:<some at recipient.com>
>>> ?? > 250 2.1.5 Ok
>>> ?? < BDAT 114098 LAST
>>> ?? > 502 5.5.2 Error: command not recognized
>>> ?? < Content-Type: multipart/alternative; 
>>> boundary=Apple-Mail-55D35F74-2EB7-4B3B-A607-E421DD71C07A
>>> ?? > 221 2.7.0 Error: I can break rules, too. Goodbye.
>>>
>>>
>>> Apple Mail seems to ignore the fact that Postfix does not
understand
>>> BDAT and Postfix does not like that :-)
>>>
>>>
>>> ?From a quick glance at the code it seems CHUNKING is always added 
>>> to the capabilities and that's not configurable. Is this a bug
or am
>>> I doing something wrong?
>>>
>>>
>>> Thanks,
>>>
>>> ??Michael
>>
>> that "may" version depend
>>
>> http://www.postfix.org/features.html
>> ...
>> Postfix 3.4???? SMTP server support for RFC 3030 CHUNKING (without 
>> BINARYMIME).
>> ...
>>
>
> As far as I understand, Dovecot always adds certain capabilities. This 
> is the responsible code (I think):
>
>
https://github.com/dovecot/core/blob/c8d03c3cab68328947a5afb47f48aef5b5a1e4ab/src/submission/submission-client.c#L95
>
Dovecot adds the capability to the server side (facing the client), but 
not on the proxy client side (facing Postfix). If Postfix doesn't 
support CHUNKING, Dovecot will translate between BDAT from client and 
DATA towards Postfix. That's the way it is supposed to work anyway. I 
cannot reproduce your problem here with a quick test.

Can you enable mail_debug=yes (e.g. for that particular user) and show 
me the debug log of the proxy activity?

The output from `dovecot -n` would also be helpful.

Regards,

Stephan.

On 09/20/2018 10:24 PM, Stephan Bosch wrote:
> Op 20/09/2018 om 13:39 schreef Michael Goth:
>> On 09/20/2018 12:56 PM, Robert Schetterer wrote:
>>> Am 20.09.2018 um 11:04 schrieb Michael Goth:
>>>> Hello,
>>>>
>>>> I'm setting up Dovecot 2.3.2.1 as a submission proxy to a
Postfix
>>>> backend server. Dovecot announces CHUNKING, but the Postfix
backend
>>>> does not support (or announce) it.
>>>>
>>>> HELO from Postfix:
>>>>
>>>> ?? 220 backend.mydomain.com ESMTP Postfix (Ubuntu)
>>>> ?? ehlo test
>>>> ?? 250-backend.mydomain.com
>>>> ?? 250-PIPELINING
>>>> ?? 250-SIZE 104857600
>>>> ?? 250-VRFY
>>>> ?? 250-ETRN
>>>> ?? 250-STARTTLS
>>>> ?? 250-ENHANCEDSTATUSCODES
>>>> ?? 250-8BITMIME
>>>> ?? 250 DSN
>>>>
>>>>
>>>> HELO from Dovecot proxy:
>>>>
>>>> ?? 220 proxy.mydomain.com Dovecot ready.
>>>> ?? ehlo test
>>>> ?? 250-proxy.mydomain.com
>>>> ?? 250-8BITMIME
>>>> ?? 250-AUTH PLAIN LOGIN
>>>> ?? 250-BURL imap
>>>> ?? 250-CHUNKING
>>>> ?? 250-ENHANCEDSTATUSCODES
>>>> ?? 250-SIZE
>>>> ?? 250-STARTTLS
>>>> ?? 250 PIPELINING
>>>>
>>>>
>>>> When a client sends BDAT, Postfix closes the connection.
Here's a
>>>> log snippet from Postfix:
>>>>
>>>> ?? < MAIL FROM:<user at mydomain.com>
>>>> ?? > 250 2.1.0 Ok
>>>> ?? < RCPT TO:<some at recipient.com>
>>>> ?? > 250 2.1.5 Ok
>>>> ?? < BDAT 114098 LAST
>>>> ?? > 502 5.5.2 Error: command not recognized
>>>> ?? < Content-Type: multipart/alternative; 
>>>> boundary=Apple-Mail-55D35F74-2EB7-4B3B-A607-E421DD71C07A
>>>> ?? > 221 2.7.0 Error: I can break rules, too. Goodbye.
>>>>
>>>>
>>>> Apple Mail seems to ignore the fact that Postfix does not
understand
>>>> BDAT and Postfix does not like that :-)
>>>>
>>>>
>>>> ?From a quick glance at the code it seems CHUNKING is always
added
>>>> to the capabilities and that's not configurable. Is this a
bug or am
>>>> I doing something wrong?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> ??Michael
>>>
>>> that "may" version depend
>>>
>>> http://www.postfix.org/features.html
>>> ...
>>> Postfix 3.4???? SMTP server support for RFC 3030 CHUNKING (without 
>>> BINARYMIME).
>>> ...
>>>
>>
>> As far as I understand, Dovecot always adds certain capabilities. This 
>> is the responsible code (I think):
>>
>>
https://github.com/dovecot/core/blob/c8d03c3cab68328947a5afb47f48aef5b5a1e4ab/src/submission/submission-client.c#L95
>>
> 
> Dovecot adds the capability to the server side (facing the client), but 
> not on the proxy client side (facing Postfix). If Postfix doesn't 
> support CHUNKING, Dovecot will translate between BDAT from client and 
> DATA towards Postfix. That's the way it is supposed to work anyway. I 
> cannot reproduce your problem here with a quick test.
I just did a test with telnet and got the same results as before:

mg at mymachine:~$ telnet proxy.mydomain.com 25
Trying 195.201.36.46...
Connected to proxy.mydomain.com.
Escape character is '^]'.
220 proxy.mydomain.com Dovecot ready.
EHLO test
250-mail2.mydomain.com
250-8BITMIME
250-AUTH PLAIN LOGIN
250-BURL imap
250-CHUNKING
250-ENHANCEDSTATUSCODES
250-SIZE
250-STARTTLS
250 PIPELINING
AUTH PLAIN AHRlc3RAbXlkb21haW4uY29tAHBhc3N3b3Jk
235 2.7.0 Authentication successful
MAIL FROM:<test at mydomain.com>
250 2.1.0 Ok
RCPT TO:<mg at webflow.de>
250 2.1.5 Ok
BDAT 114098 LAST
502 5.5.2 Error: command not recognized
Content-Type: multipart/alternative; 
boundary=Apple-Mail-55D35F74-2EB7-4B3B-A607-E421DD71C07A
221 2.7.0 Error: I can break rules, too. Goodbye.
Connection closed by foreign host.

> Can you enable mail_debug=yes (e.g. for that particular user) and show 
> me the debug log of the proxy activity?
I did enable mail_debug=yes and restarted Dovecot, but I don't see any 
debug messages in /var/log/mail.log ?

'doveadm log find' tells me debug logging goes to /var/log/mail.log
> The output from `dovecot -n` would also be helpful.
Please see the attached dovecot.conf, I'm running 2.3.2.1 installed from 
repo.dovecot.org


Regards,
   Michael

-------------- next part --------------
# 2.3.2.1 (0719df592): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.2 (5d6d7c92)
# OS: Linux 4.15.0-33-generic x86_64 Ubuntu 18.04.1 LTS
# Hostname: proxy.mydomain.com
auth_mechanisms = plain login
auth_verbose = yes
disable_plaintext_auth = no
listen = *
mail_debug = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extracttext
passdb {
  args = /dev/null
  driver = passwd-file
}
protocols = imap pop3 submission sieve
service imap-login {
  inet_listener imap_10143 {
    address = *
    port = 10143
    ssl = yes
  }
}
service pop3-login {
  inet_listener imap_10110 {
    address = *
    port = 10110
    ssl = yes
  }
}
service submission-login {
  inet_listener smtp {
    address = *
    port = 25
  }
  inet_listener smtp_10025 {
    address = *
    port = 10025
    ssl = yes
  }
  inet_listener smtps {
    address = *
    port = 465
    ssl = yes
  }
  inet_listener submission {
    address = *
    port = 587
  }
}
ssl_ca = </etc/ssl/certs/ca-certificates.crt
ssl_cert = </etc/dovecot/private/wildcard.mydomain.com.pem
ssl_dh =  # hidden, use -P to show it
ssl_key =  # hidden, use -P to show it
ssl_require_crl = no
userdb {
  args = static uid=65534 gid=65534 home=/dev/null
  driver = static
}
protocol imap {
  passdb {
    args = /etc/dovecot/dovecot-sql.conf
    driver = sql
    name   }
  passdb {
    args = proxy=y nopassword=y host=legacy-proxy.mydomain.com port=993 ssl=y
    driver = static
    name   }
}
protocol pop3 {
  passdb {
    args = /etc/dovecot/dovecot-sql.conf
    driver = sql
    name   }
  passdb {
    args = proxy=y nopassword=y host=legacy-proxy.mydomain.com port=995 ssl=y
    driver = static
    name   }
}
protocol submission {
  passdb {
    args = /etc/dovecot/dovecot-sql.conf
    driver = sql
    name   }
  passdb {
    args = proxy=y nopassword=y host=legacy-proxy.mydomain.com port=465 ssl=y
    driver = static
    name   }
}
protocol sieve {
  passdb {
    args = /etc/dovecot/dovecot-sql.conf
    driver = sql
    name   }
}

On 09/20/2018 10:24 PM, Stephan Bosch wrote:
> 
> 
> Op 20/09/2018 om 13:39 schreef Michael Goth:
>> On 09/20/2018 12:56 PM, Robert Schetterer wrote:
>>> Am 20.09.2018 um 11:04 schrieb Michael Goth:
>>>> Hello,
>>>>
>>>> I'm setting up Dovecot 2.3.2.1 as a submission proxy to a
Postfix
>>>> backend server. Dovecot announces CHUNKING, but the Postfix
backend
>>>> does not support (or announce) it.
>>>>
>>>> HELO from Postfix:
>>>>
>>>> ?? 220 backend.mydomain.com ESMTP Postfix (Ubuntu)
>>>> ?? ehlo test
>>>> ?? 250-backend.mydomain.com
>>>> ?? 250-PIPELINING
>>>> ?? 250-SIZE 104857600
>>>> ?? 250-VRFY
>>>> ?? 250-ETRN
>>>> ?? 250-STARTTLS
>>>> ?? 250-ENHANCEDSTATUSCODES
>>>> ?? 250-8BITMIME
>>>> ?? 250 DSN
>>>>
>>>>
>>>> HELO from Dovecot proxy:
>>>>
>>>> ?? 220 proxy.mydomain.com Dovecot ready.
>>>> ?? ehlo test
>>>> ?? 250-proxy.mydomain.com
>>>> ?? 250-8BITMIME
>>>> ?? 250-AUTH PLAIN LOGIN
>>>> ?? 250-BURL imap
>>>> ?? 250-CHUNKING
>>>> ?? 250-ENHANCEDSTATUSCODES
>>>> ?? 250-SIZE
>>>> ?? 250-STARTTLS
>>>> ?? 250 PIPELINING
>>>>
>>>>
>>>> When a client sends BDAT, Postfix closes the connection.
Here's a
>>>> log snippet from Postfix:
>>>>
>>>> ?? < MAIL FROM:<user at mydomain.com>
>>>> ?? > 250 2.1.0 Ok
>>>> ?? < RCPT TO:<some at recipient.com>
>>>> ?? > 250 2.1.5 Ok
>>>> ?? < BDAT 114098 LAST
>>>> ?? > 502 5.5.2 Error: command not recognized
>>>> ?? < Content-Type: multipart/alternative; 
>>>> boundary=Apple-Mail-55D35F74-2EB7-4B3B-A607-E421DD71C07A
>>>> ?? > 221 2.7.0 Error: I can break rules, too. Goodbye.
>>>>
>>>>
>>>> Apple Mail seems to ignore the fact that Postfix does not
understand
>>>> BDAT and Postfix does not like that :-)
>>>>
>>>>
>>>> ?From a quick glance at the code it seems CHUNKING is always
added
>>>> to the capabilities and that's not configurable. Is this a
bug or am
>>>> I doing something wrong?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> ??Michael
>>>
>>> that "may" version depend
>>>
>>> http://www.postfix.org/features.html
>>> ...
>>> Postfix 3.4???? SMTP server support for RFC 3030 CHUNKING (without 
>>> BINARYMIME).
>>> ...
>>>
>>
>> As far as I understand, Dovecot always adds certain capabilities. This 
>> is the responsible code (I think):
>>
>>
https://github.com/dovecot/core/blob/c8d03c3cab68328947a5afb47f48aef5b5a1e4ab/src/submission/submission-client.c#L95
>>
> 
> Dovecot adds the capability to the server side (facing the client), but 
> not on the proxy client side (facing Postfix). If Postfix doesn't 
> support CHUNKING, Dovecot will translate between BDAT from client and 
> DATA towards Postfix. That's the way it is supposed to work anyway. I 
> cannot reproduce your problem here with a quick test.
> 
> Can you enable mail_debug=yes (e.g. for that particular user) and show 
> me the debug log of the proxy activity?
> 
> The output from `dovecot -n` would also be helpful.
> 
> Regards,
> 
> Stephan.
> 
I think I found the problem:

I am not using the submission_* config directives to configure the relay 
SMTP server.
Instead, I'm using Dovecot's proxy feature for selecting a backend 
server based on the user's authentication. The submission_* directives 
are all unset.

When using the submission_* directives, Dovecot translates from BDAT to 
DATA. With my configuration, it seems to just relay all the commands to 
the backend server.

Now that I think about it, I'm not sure what dovecot can do in this 
case. It doesn't really know the supported capabilities until the user 
authenticates themself.


Regards,
   Michael