dovecot - Apr 2015 - Dovecot LDA/LMTP vs postfix virtual delivery agent and the x-original-to header

On 2014-01-08 14:32, Charles Marcus wrote:
> On 2012-04-09 8:53 AM, Timo Sirainen <tss at iki.fi> wrote:
>> On 9.4.2012, at 15.50, Charles Marcus wrote:
>>>> LMTP adds a new Delivered-To:<rcpt-to at address>  header
when there is
>>>> a single RCPT TO. You can force a single RCPT TO from Postfix
side by
>>>> setting lmtp_destination_recipient_limit=1. LMTP doesn't
>>>> add/remove/change X-Original-To: header.
>
>>> Ok, thanks Timo... but...
>>>
>>> Are you saying that this 'Delivered-To:' header can somehow
be
>>> leveraged to provide the same info as the x-original-to header?
>
>> I guess X-Original-To is the same address as what Postfix sees as the 
>> original RCPT TO address before alias expansion and such? In that 
>> case, see my today's mail in Postfix list..
>
> Hi Timo,
>
> I just tried to find your email from that day, but don't see it in the 
> archives...
>
> Was this ever resolved (getting x-original-to support in LMTP, like it 
> is for the LDA)?
>
> If not, since it seemed like it wasn't going to be much work, any 
> chance you can revisit it soon?
>
> Thanks,
>
Hello,

I have tried to keep tabs on the various discussions going on related to 
the X-Original-To header when using Dovecot LMTP. Until now I have not 
needed a solution, but I am now finally about to migrate my old server.

Old setup: Postfix + SpamAssassin (after-queue content filter via pipe) 
+ virtual transport, and Courier-IMAP.
New setup: Postfix + amavisd-new (after-queue content filter via smtp, 
with ClamAV and SpamAssassin) + Dovecot LMTP, and Dovecot for IMAP.

Charles, have you found a way that works for you?

I was experimenting some with my test server and came up with a way that 
utilizes some additional internal smtp content filter forwarding, which 
produces some overhead. It should be light compared with the load from 
ClamAV and SpamAssassin, however.

I'm not yet sure how amavisd-new funneling would handle multiple local 
recipients with different settings without passing the mail through 
multiple time, at least once per local user, let alone without first 
performing address mapping in postfix (for alias expansion). I have 
configured per-user SpamAssassin bayes filtering, and may introduce a 
whitelist based on address book entries (Roundcube.)


This solution I'm currently testing will pass each message through 
amavisd-new one time each per local and remote recipient, and will only 
add the X-Original-To header to the specific local recipient each 
envelope is intended for. No external users will receive the header, and 
no local user will see which other local users (e.g. via BCC) have 
potentially received the same message.

Flow:
all mail in (both external and tls-authenticated internal) -> smtp (1) 
-> smtp-split (2) -> smtp-to-me (3a) | smtp-to-external (3b) -> 
smtp-amavis (4) -> dovecot-lmtp (5)

1) I rely on default_destination_recipient_limit=1 in main.cf to split 
each incoming mail into one stream per recipient.
2) smtp-split will receive one stream per recipient. Default 
content_filter=smtp-to-me, followed by option 
"smtpd_recipient_restrictions=permit_auth_destination,check_recipient_access,pcre:/usr/local/etc/postfix/filter-to-external.pcre,permit_mynetworks,reject"
means I stop processing restrictions if my server is the destination. If 
my server is not the destination, the FILTER in check_recipient_access 
will override the preceding smtp-to-me filter.

Both 1) and 2) smtpd instances include option 
receive_override_options=no_address_mappings, to wait with mapping to 
internal recipient until we can add X-Original-To header for my server's 
users only.

3a) For mail to my server, smtp-to-me will add X-Original-To using a 
pcre script, in a similar fashion to step 2's filter. This step also 
expands the address mapping (by not specifying any 
receive_override_options).
   -o 
smtpd_recipient_restrictions=check_recipient_access,pcre:/usr/local/etc/postfix/recipient_access_x-orig.pcre,permit_mynetworks,reject

3b) For mail leaving my server, smtp-to-external will not add any 
processing besides implied expanding of the address mapping.

4) Mail is funneled through amavisd-new, once per final recipient. Mails 
leaving the server (sent from smtp-to-external) will be scanned by ClamV 
only. Mails with my server as the destination (sent from smtp-to-me) 
will go through ClamV, and SpamAssassin (together with per-user bayes 
filtering).

5) Nothing special is done here. The final destination address is sent 
to LMTP for delivery.

Contents of /usr/local/etc/postfix/recipient_access_x-orig.pcre:
/(.+)/        prepend X-Original-To: <$1>

Contents of /usr/local/etc/postfix/filter-to-external.pcre:
/^/        FILTER smtp-to-external:[127.0.0.1]:<port>


Room for improvement:
Postfix seem to know the orig_to even after processing in amavisd-new, 
however I have yet to find a way to use this option.
I can move the amavisd-new filter to before the X-Original-To header 
addition, however for amavisd-new to utilize per-user bayes, I currently 
need to do the address mapping in postfix before sending the content to 
amavisd-new. This may be possible to circumvent either with alias lookup 
in amavisd-new, or if I can find another way to use the 
postfix-available "orig_to" to populate X-Original-To header after 
scanning in amavisd-new.

I've tried to split the mails into one per recipient after address 
mapping and amavisd-new, instead of before as my solution above, without 
default_destination_recipient_limit=1 in main.cf. Instead I tried the 
options as part of smtp (and even smtpd) services in master.cf, 
unfortunately without success. I keep ending up with multiple 
X-Original-To with all local recipients (including BCC) in all internal 
copies delivered.


Have anyone successfully tackled this conundrum with other solutions?

Regards,
Tobias

On 4/28/2015 1:40 PM, Tobias Franz?n <lists.zxinn at otaking.se>
wrote:
> On 2014-01-08 14:32, Charles Marcus wrote:
>> On 2012-04-09 8:53 AM, Timo Sirainen <tss at iki.fi> wrote:
>>> On 9.4.2012, at 15.50, Charles Marcus wrote:
>>>>> LMTP adds a new Delivered-To:<rcpt-to at address> 
header when there is
>>>>> a single RCPT TO. You can force a single RCPT TO from
Postfix side by
>>>>> setting lmtp_destination_recipient_limit=1. LMTP
doesn't
>>>>> add/remove/change X-Original-To: header.
>>>> Ok, thanks Timo... but...
>>>>
>>>> Are you saying that this 'Delivered-To:' header can
somehow be
>>>> leveraged to provide the same info as the x-original-to header?
>>> I guess X-Original-To is the same address as what Postfix sees as
the
>>> original RCPT TO address before alias expansion and such? In that 
>>> case, see my today's mail in Postfix list..
>> Hi Timo,
>>
>> I just tried to find your email from that day, but don't see it in
the
>> archives...
>>
>> Was this ever resolved (getting x-original-to support in LMTP, like it 
>> is for the LDA)?
>>
>> If not, since it seemed like it wasn't going to be much work, any 
>> chance you can revisit it soon?
> Hello,
>
> I have tried to keep tabs on the various discussions going on related to 
> the X-Original-To header when using Dovecot LMTP. Until now I have not 
> needed a solution, but I am now finally about to migrate my old server.
>
> Old setup: Postfix + SpamAssassin (after-queue content filter via pipe) 
> + virtual transport, and Courier-IMAP.
> New setup: Postfix + amavisd-new (after-queue content filter via smtp, 
> with ClamAV and SpamAssassin) + Dovecot LMTP, and Dovecot for IMAP.
>
> Charles, have you found a way that works for you?
No, and I simply haven't switched to LMTP yet, for this and one other
reason (political, not technical)...

As for the rest below... wow... all I can say is, it sure would be nice
if Timo/Wietse could just add the few lines of code that Timo said would
be needed to properly support it natively.
> I was experimenting some with my test server and came up with a way that 
> utilizes some additional internal smtp content filter forwarding, which 
> produces some overhead. It should be light compared with the load from 
> ClamAV and SpamAssassin, however.
>
> I'm not yet sure how amavisd-new funneling would handle multiple local 
> recipients with different settings without passing the mail through 
> multiple time, at least once per local user, let alone without first 
> performing address mapping in postfix (for alias expansion). I have 
> configured per-user SpamAssassin bayes filtering, and may introduce a 
> whitelist based on address book entries (Roundcube.)
>
>
> This solution I'm currently testing will pass each message through 
> amavisd-new one time each per local and remote recipient, and will only 
> add the X-Original-To header to the specific local recipient each 
> envelope is intended for. No external users will receive the header, and 
> no local user will see which other local users (e.g. via BCC) have 
> potentially received the same message.
>
> Flow:
> all mail in (both external and tls-authenticated internal) -> smtp (1) 
> -> smtp-split (2) -> smtp-to-me (3a) | smtp-to-external (3b) -> 
> smtp-amavis (4) -> dovecot-lmtp (5)
>
> 1) I rely on default_destination_recipient_limit=1 in main.cf to split 
> each incoming mail into one stream per recipient.
> 2) smtp-split will receive one stream per recipient. Default 
> content_filter=smtp-to-me, followed by option 
>
"smtpd_recipient_restrictions=permit_auth_destination,check_recipient_access,pcre:/usr/local/etc/postfix/filter-to-external.pcre,permit_mynetworks,reject"
> means I stop processing restrictions if my server is the destination. If 
> my server is not the destination, the FILTER in check_recipient_access 
> will override the preceding smtp-to-me filter.
>
> Both 1) and 2) smtpd instances include option 
> receive_override_options=no_address_mappings, to wait with mapping to 
> internal recipient until we can add X-Original-To header for my
server's
> users only.
>
> 3a) For mail to my server, smtp-to-me will add X-Original-To using a 
> pcre script, in a similar fashion to step 2's filter. This step also 
> expands the address mapping (by not specifying any 
> receive_override_options).
>    -o 
>
smtpd_recipient_restrictions=check_recipient_access,pcre:/usr/local/etc/postfix/recipient_access_x-orig.pcre,permit_mynetworks,reject
>
> 3b) For mail leaving my server, smtp-to-external will not add any 
> processing besides implied expanding of the address mapping.
>
> 4) Mail is funneled through amavisd-new, once per final recipient. Mails 
> leaving the server (sent from smtp-to-external) will be scanned by ClamV 
> only. Mails with my server as the destination (sent from smtp-to-me) 
> will go through ClamV, and SpamAssassin (together with per-user bayes 
> filtering).
>
> 5) Nothing special is done here. The final destination address is sent 
> to LMTP for delivery.
>
> Contents of /usr/local/etc/postfix/recipient_access_x-orig.pcre:
> /(.+)/        prepend X-Original-To: <$1>
>
> Contents of /usr/local/etc/postfix/filter-to-external.pcre:
> /^/        FILTER smtp-to-external:[127.0.0.1]:<port>
>
>
> Room for improvement:
> Postfix seem to know the orig_to even after processing in amavisd-new, 
> however I have yet to find a way to use this option.
> I can move the amavisd-new filter to before the X-Original-To header 
> addition, however for amavisd-new to utilize per-user bayes, I currently 
> need to do the address mapping in postfix before sending the content to 
> amavisd-new. This may be possible to circumvent either with alias lookup 
> in amavisd-new, or if I can find another way to use the 
> postfix-available "orig_to" to populate X-Original-To header
after
> scanning in amavisd-new.
>
> I've tried to split the mails into one per recipient after address 
> mapping and amavisd-new, instead of before as my solution above, without 
> default_destination_recipient_limit=1 in main.cf. Instead I tried the 
> options as part of smtp (and even smtpd) services in master.cf, 
> unfortunately without success. I keep ending up with multiple 
> X-Original-To with all local recipients (including BCC) in all internal 
> copies delivered.
>
>
> Have anyone successfully tackled this conundrum with other solutions?
>
> Regards,
> Tobias
>

I'm resurrecting this again because I'm getting pretty close to possibly
being ready to install a brand new dovecot server (finally), but I still
need for dovecots LMTP to add the x-original-to header.

So... was this completed quietly, or is support for it still not there?

Thanks,

Charles

On Tue Apr 28 2015 15:27:07 GMT-0400 (Eastern Standard Time), Charles
Marcus <CMarcus at Media-Brokers.com> wrote:
> On 4/28/2015 1:40 PM, Tobias Franz?n <lists.zxinn at otaking.se>
wrote:
>> On 2014-01-08 14:32, Charles Marcus wrote:
>>> On 2012-04-09 8:53 AM, Timo Sirainen <tss at iki.fi> wrote:
>>>> On 9.4.2012, at 15.50, Charles Marcus wrote:
>>>>>> LMTP adds a new Delivered-To:<rcpt-to at address>
header when there is
>>>>>> a single RCPT TO. You can force a single RCPT TO from
Postfix side by
>>>>>> setting lmtp_destination_recipient_limit=1. LMTP
doesn't
>>>>>> add/remove/change X-Original-To: header.
>>>>> Ok, thanks Timo... but...
>>>>>
>>>>> Are you saying that this 'Delivered-To:' header can
somehow be
>>>>> leveraged to provide the same info as the x-original-to
header?
>>>> I guess X-Original-To is the same address as what Postfix sees
as the
>>>> original RCPT TO address before alias expansion and such? In
that
>>>> case, see my today's mail in Postfix list..
>>> Hi Timo,
>>>
>>> I just tried to find your email from that day, but don't see it
in the
>>> archives...
>>>
>>> Was this ever resolved (getting x-original-to support in LMTP, like
it
>>> is for the LDA)?
>>>
>>> If not, since it seemed like it wasn't going to be much work,
any
>>> chance you can revisit it soon?
>> Hello,
>>
>> I have tried to keep tabs on the various discussions going on related
to
>> the X-Original-To header when using Dovecot LMTP. Until now I have not 
>> needed a solution, but I am now finally about to migrate my old server.
>>
>> Old setup: Postfix + SpamAssassin (after-queue content filter via pipe)
>> + virtual transport, and Courier-IMAP.
>> New setup: Postfix + amavisd-new (after-queue content filter via smtp, 
>> with ClamAV and SpamAssassin) + Dovecot LMTP, and Dovecot for IMAP.
>>
>> Charles, have you found a way that works for you?
> 
> No, and I simply haven't switched to LMTP yet, for this and one other
> reason (political, not technical)...
> 
> As for the rest below... wow... all I can say is, it sure would be nice
> if Timo/Wietse could just add the few lines of code that Timo said would
> be needed to properly support it natively.
> 
>> I was experimenting some with my test server and came up with a way
that
>> utilizes some additional internal smtp content filter forwarding, which
>> produces some overhead. It should be light compared with the load from 
>> ClamAV and SpamAssassin, however.
>>
>> I'm not yet sure how amavisd-new funneling would handle multiple
local
>> recipients with different settings without passing the mail through 
>> multiple time, at least once per local user, let alone without first 
>> performing address mapping in postfix (for alias expansion). I have 
>> configured per-user SpamAssassin bayes filtering, and may introduce a 
>> whitelist based on address book entries (Roundcube.)
>>
>>
>> This solution I'm currently testing will pass each message through 
>> amavisd-new one time each per local and remote recipient, and will only
>> add the X-Original-To header to the specific local recipient each 
>> envelope is intended for. No external users will receive the header,
and
>> no local user will see which other local users (e.g. via BCC) have 
>> potentially received the same message.
>>
>> Flow:
>> all mail in (both external and tls-authenticated internal) -> smtp
(1)
>> -> smtp-split (2) -> smtp-to-me (3a) | smtp-to-external (3b)
->
>> smtp-amavis (4) -> dovecot-lmtp (5)
>>
>> 1) I rely on default_destination_recipient_limit=1 in main.cf to split 
>> each incoming mail into one stream per recipient.
>> 2) smtp-split will receive one stream per recipient. Default 
>> content_filter=smtp-to-me, followed by option 
>>
"smtpd_recipient_restrictions=permit_auth_destination,check_recipient_access,pcre:/usr/local/etc/postfix/filter-to-external.pcre,permit_mynetworks,reject"
>> means I stop processing restrictions if my server is the destination.
If
>> my server is not the destination, the FILTER in check_recipient_access 
>> will override the preceding smtp-to-me filter.
>>
>> Both 1) and 2) smtpd instances include option 
>> receive_override_options=no_address_mappings, to wait with mapping to 
>> internal recipient until we can add X-Original-To header for my
server's
>> users only.
>>
>> 3a) For mail to my server, smtp-to-me will add X-Original-To using a 
>> pcre script, in a similar fashion to step 2's filter. This step
also
>> expands the address mapping (by not specifying any 
>> receive_override_options).
>>    -o 
>>
smtpd_recipient_restrictions=check_recipient_access,pcre:/usr/local/etc/postfix/recipient_access_x-orig.pcre,permit_mynetworks,reject
>>
>> 3b) For mail leaving my server, smtp-to-external will not add any 
>> processing besides implied expanding of the address mapping.
>>
>> 4) Mail is funneled through amavisd-new, once per final recipient.
Mails
>> leaving the server (sent from smtp-to-external) will be scanned by
ClamV
>> only. Mails with my server as the destination (sent from smtp-to-me) 
>> will go through ClamV, and SpamAssassin (together with per-user bayes 
>> filtering).
>>
>> 5) Nothing special is done here. The final destination address is sent 
>> to LMTP for delivery.
>>
>> Contents of /usr/local/etc/postfix/recipient_access_x-orig.pcre:
>> /(.+)/        prepend X-Original-To: <$1>
>>
>> Contents of /usr/local/etc/postfix/filter-to-external.pcre:
>> /^/        FILTER smtp-to-external:[127.0.0.1]:<port>
>>
>>
>> Room for improvement:
>> Postfix seem to know the orig_to even after processing in amavisd-new, 
>> however I have yet to find a way to use this option.
>> I can move the amavisd-new filter to before the X-Original-To header 
>> addition, however for amavisd-new to utilize per-user bayes, I
currently
>> need to do the address mapping in postfix before sending the content to
>> amavisd-new. This may be possible to circumvent either with alias
lookup
>> in amavisd-new, or if I can find another way to use the 
>> postfix-available "orig_to" to populate X-Original-To header
after
>> scanning in amavisd-new.
>>
>> I've tried to split the mails into one per recipient after address 
>> mapping and amavisd-new, instead of before as my solution above,
without
>> default_destination_recipient_limit=1 in main.cf. Instead I tried the 
>> options as part of smtp (and even smtpd) services in master.cf, 
>> unfortunately without success. I keep ending up with multiple 
>> X-Original-To with all local recipients (including BCC) in all internal
>> copies delivered.
>>
>>
>> Have anyone successfully tackled this conundrum with other solutions?
>>
>> Regards,
>> Tobias
>>
>