dovecot - Oct 2022 - Pigeonhole redirect is adding a message-id header when it already exists

Hi there,   I can confirm this behavior. A few months ago I introduced a milter
which is checking for multiple headers when the RFC says that there just should
be one of them For example "Message-Id".   I found the described
problem in an email coming from Alibaba, which had an invalid
"Message-Id" header. It didn't contain an "@" sign or
similar. It was RFC-invalid.   This email was sent from Alibaba to a German
email provider. There was a redirect at that email provider, pointing to my
mailserver.   My server rejected the email because there were 2
"Message-Id" headers: The original invalid "Message-Id"
header from Alibaba, and a new "Message-Id" header from the German
provider, which seems to have been added during the redirect. There were
"Dovecot-sieve" headers in that mail, so my guess was that it happened
because of Dovecot-sieve/pigeonhole implementation.   I contacted the email
provider, asking for help. Asking if it really is a bug in pigeonhole (or maybe
some other system at that provider, who knows). And I contacted Alibaba, so they
fix the invalid "Message-Id". I got responses from both, but until
now, as far as I can see, it has not been fixed.   The best fix would be (if it
really is a bug in pigeonhole), if pigeonhole fixes the problem, then it's
fixed for all users of Dovecot. I guess Alibaba is not the only sender with an
invalid "Message-ID" header, but that's the only one I saw.  
Michael

Am 01-Oct-2022 14:00:45 +0200 schrieb sriccio at swisscenter.com:
>You wrote in the original email the message was rejected. Sorry I don't
have login access to my gmail test account anymore since the google @#$%@#$%
wanted to have me add a phone number.
In my original post I said that gmail was rejecting the forwards because 
of duplicate headers, and that the duplicate header seems to be a 
Message-ID added by pigeonhole when it's "not happy" with the
original
mail Message-ID.

I probably failed to explain the issue clearly and sorry for that.

Thank you anyway for trying to help :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20221001/250773ab/attachment.htm>

Hi there,

I can confirm this behavior. A few months ago I introduced a milter which is
checking for multiple headers when the RFC says that there just should be one of
them For example "Message-Id".

I found the described problem in an email coming from Alibaba, which had an
invalid "Message-Id" header. It didn't contain an "@"
sign or similar. It was RFC-invalid.

This email was sent from Alibaba to a German email provider. There was a
redirect at that email provider, pointing to my mailserver.

My server rejected the email because there were 2 "Message-Id"
headers: The original invalid "Message-Id" header from Alibaba, and a
new "Message-Id" header from the German provider, which seems to have
been added during the redirect. There were "Dovecot-sieve" headers in
that mail, so my guess was that it happened because of Dovecot-sieve/pigeonhole
implementation.

I contacted the email provider, asking for help. Asking if it really is a bug in
pigeonhole (or maybe some other system at that provider, who knows). And I
contacted Alibaba, so they fix the invalid "Message-Id". I got
responses from both, but until now, as far as I can see, it has not been fixed.

The best fix would be (if it really is a bug in pigeonhole), if pigeonhole fixes
the problem, then it's fixed for all users of Dovecot. I guess Alibaba is
not the only sender with an invalid "Message-ID" header, but
that's the only one I saw.

Michael


Hello Michael,

Thanks a lot for your feedback. It comfort me that I'm maybe on the right
track about this issue.

The mail I used to troubleshoot the issue was a mail from aliexpress containing
a funky Message-ID. Probably the same people behind Aliexpress/Alibaba.
But for sure there are probably other mail senders that generate RFC-invalide
Message-ID.

If Pigeonhole consider the Message-ID is bad and add it's own, it should
maybe at least remove the original one... ?

Kind regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20221001/6562914f/attachment-0001.htm>

Hi,

After reading a bit the code and trying to understand it, here is what I think
happens here:

Given a bogus Message-ID, for example (notice it's missing angle brackets
< >:

Message-ID:
1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7

Dovecot is parsing Message-ID headers and is looking for the angle bracket as
the begining of the Message-ID:
https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55c/src/lib-mail/message-id.c#L75

As none is found it will act as if there was no Message-ID header in the mail
(even that the header is present).

Then, pigeonhole's redirect function  is told to generate a new Message-ID
if none was previously detected:
https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac86e15f99/src/lib-sieve/cmd-redirect.c#L569

The result is the mail being forwarded, in this case, is now having dual
Message-ID and is not RFC 5322 compliant anymore and can be rejected for this
reason (hi, gmail?)
https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.html

Some thoughts:

- First, to be honest, I'm not sure gmail would accept the original mail
with the bogus Message-ID sent directly to their servers, but if it was refused,
I would assume that these senders would have fixed the issue on their side so
their message are delivered (unless there is some whitelisting going on?)

- What options could we have to resolve this?

a) Having dovecot core to remove the Message-ID header line from the mail if it
is not going to consider it valid ? (So there is no dupe headers when pigeonhole
adds one?)
b) Having pigeonhole check, when adding a new valid Message-ID, if there is
already one existing, and remove the bogus one ?

For now, to workaround this, I'm trying to find a way in the mail flow on
our servers to keep only the top most Message-ID when more than one exists.
Maybe using: https://www.postfix.org/postconf.5.html#smtp_header_checks but
I'm not sure how to achieve it yet or even if it's possible.

Kind regards

S?bastien RICCIO
SYSTEM ADMINISTRATOR
P  +41 840 888 888
F  +41 840 888 000
M sriccio at swisscenter.com<mailto:sriccio at swisscenter.com>



------ Message d'origine ------
De "michael.zork at feierfighter.de<mailto:michael.zork at
feierfighter.de>" <michael.zork at
feierfighter.de<mailto:michael.zork at feierfighter.de>>
? "dovecot at dovecot.org<mailto:dovecot at dovecot.org>"
<dovecot at dovecot.org<mailto:dovecot at dovecot.org>>
Date 01.10.2022 14:49:13
Objet Re: Re[6]: Pigeonhole redirect is adding a message-id header when it
already exists

Hi there,

I can confirm this behavior. A few months ago I introduced a milter which is
checking for multiple headers when the RFC says that there just should be one of
them For example "Message-Id".

I found the described problem in an email coming from Alibaba, which had an
invalid "Message-Id" header. It didn't contain an "@"
sign or similar. It was RFC-invalid.

This email was sent from Alibaba to a German email provider. There was a
redirect at that email provider, pointing to my mailserver.

My server rejected the email because there were 2 "Message-Id"
headers: The original invalid "Message-Id" header from Alibaba, and a
new "Message-Id" header from the German provider, which seems to have
been added during the redirect. There were "Dovecot-sieve" headers in
that mail, so my guess was that it happened because of Dovecot-sieve/pigeonhole
implementation.

I contacted the email provider, asking for help. Asking if it really is a bug in
pigeonhole (or maybe some other system at that provider, who knows). And I
contacted Alibaba, so they fix the invalid "Message-Id". I got
responses from both, but until now, as far as I can see, it has not been fixed.

The best fix would be (if it really is a bug in pigeonhole), if pigeonhole fixes
the problem, then it's fixed for all users of Dovecot. I guess Alibaba is
not the only sender with an invalid "Message-ID" header, but
that's the only one I saw.

Michael



Am 01-Oct-2022 14:00:45 +0200 schrieb sriccio at
swisscenter.com<mailto:sriccio at
swisscenter.com>:
>You wrote in the original email the message was rejected. Sorry I don't
have login access to my gmail test account anymore since the google @#$%@#$%
wanted to have me add a phone number.
In my original post I said that gmail was rejecting the forwards because
of duplicate headers, and that the duplicate header seems to be a
Message-ID added by pigeonhole when it's "not happy" with the
original
mail Message-ID.

I probably failed to explain the issue clearly and sorry for that.

Thank you anyway for trying to help :)

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20221002/66826ba7/attachment.htm>