freedesktop - Feb 2019 - [fdo] PSA: Mailman changes, From addresses no longer accurate

Hi all,
We have hit another step change in aggressive anti-spam techniques
from major mail providers. Over the past few days, we saw a huge spike
in the number of mails we were failing to deliver to GMail and
outlook.com in particular.

It looks like it is now no longer acceptable for us to break
DMARC/DKIM/SPF. These are DNS-based extensions to SMTP, which allow
domains to publish policies as to who should be allowed to send email
on their behalf. SPF provides source filtering, so e.g.
freedesktop.org could specify that no-one should accept mail with a
From: *@freedesktop.org unless it came from gabe.freedesktop.org.
Mailman completely breaks this: if I specified a filter only allowing
Google to send mail for @fooishbar.org, then anyone enforcing SPF
would reject receipt of this mail, as it would arrive from fd.o with
my From address.

DKIM allows domains to publish a public key in DNS, inserting a header
into mails sent from their domain cryptographically signing the value
of named headers. Mailman breaks this too: changing the Sender header
(such that bounces get processed by Mailman, rather than sending a
deluge of out-of-office and mailbox-over-quota messages to whoever
posts to the list) can break most DKIM signatures. Mailman adding the
unsubscribe footer also breaks this; we could make it not add the
footer, but in order to do so we'd have to convince ourselves that we
were not decreasing our GDPR compliance.

DMARC ties the two together, allowing domains to specify whether or
not DKIM/SPF should be mandatory, and if they fail, what action should
be taken. Despite some domains specifying a fail action of 'none'
(receiving MTA to send an advisory report to a named email address,
but still allow the email), it seems that popular services still
interpret 'none' as 'reject'.

As a result, Google in particular is dropping some number of our mails
on the floor. This does _not_ just apply to mails which fail
DMARC/DKIM/SPF: every mail we send that fails these filters (which is
a lot - e.g. Intel and NVIDIA both have SPF) decreases our sender
reputation with GMail and causes it to reject legitimate mails.

I've reached out to Google through a couple of channels to see what we
can do to increase our delivery rate to them. In the meantime, if your
mail is hosted by Google, or Outlook, and you think you're missing
mails - you probably are.

Mailman has also now been reconfigured such that if it spots a
potential DMARC violation, it rewrites the From address to be
@lists.freedesktop.org, keeping the original author in Reply-To. It
also strips DKIM headers. This seems to be about the best we can do,
unless and until the major mail service providers offer us some
alternate way to send mail. If you are replying privately to someone,
you should check very carefully that you are replying to them and not
to the list.

Unfortunately we don't have a good answer in the long run. The latest
published RFC at https://tools.ietf.org/html/rfc6377 suggests that
there are no good solutions. If anyone is blessed with an abundance of
time and familiar with the current email landscape, I would love to
talk to you and get your help to work on this. Unfortunately we don't
have the manpower to actually properly monitor email; it can often
take a collapse in successful-delivery rates for us to notice.

Ultimately, I suspect the best solution is to move most of our
discussions to dedicated fora like GitLab issues, or something like
Discourse. Fundamentally, the thing we're trying to do (send email to
thousands of people at a time using a fake From address) is ... kind
of the opposite of what the 2019 Internet wants us to do. Every few
months the major providers drop more of our mail as they become more
aggressive with spam, and every few months their userbase increases by
a non-trivial amount.

We've done a lot of work on our email infrastructure, and are doing
our best to be a responsible citizen within the constraint of having
to launder mail and forge identity on an industrial scale, but it's
coming to the point where it just may not be possible to run such a
service at such a scale anymore.

This is before even considering our other issues with Mailman 2.x: no
centralised identity management (mailing your passwords out every
month ... ?!), difficulty of GDPR compliance (editing archives
requires hand-editing every single HTML index, as there is no
non-destructive archive rebuild), the flat-out bugs (e.g. the mesa-dev
archives are usually missing half the messages), and the fact it's
been abandoned upstream in favour of Mailman 3.x, which is not
obviously better, nor is there a clear upgrade path to.

Of course we do not have any plans to stop providing email any time
soon, but it might be worth thinking about what you can do to reduce
your dependency on email lists. At the current rate of degradation, it
might be non-viable quicker than you'd think. Maybe this is unduly
gloomy, but the entire internet's direction of travel has been away
from services like Mailman, and its velocity is only increasing.

Cheers,
Daniel

Hi all,
Unfortunately, freedesktop.org's job of sending mail to a huge number
of people whilst pretending to be other people, has just got even
harder than it was.

fd.o can no longer (at least for the moment) send mail with the From
addresses of DMARC/DKIM/SPF-protected sender domains. When we try to
do so, large providers reject the mail, despite DMARC records
explicitly specifying that the mail should be accepted. Worse than
that, not only does the immediate delivery attempt fail, but it
punishes our sender reputation enough that _other_ mail delivery
fails: after we hit a DMARC-related failure, large providers often
refuse delivery attempts for mail from non-DMARC-protected domains.

As a result, if the sender domain has a DMARC record, we rewrite
the
>From address to be the list's address, preserving the original author
in Reply-To.

I'm chasing this up through a few channels, but in the meantime,
please be aware that the From address on mails may no longer be
accurate. If you are attempting to apply patches with git-am, please
check that the commit author is not 'Person Name via listname-devel
<listname-devel at lists.freedesktop.org>'.

If you are replying privately to a list mail, please be _very_ careful
that you are replying to the original sender (in Reply-To) and not the
list itself.

Cheers,
Daniel

---------- Forwarded message ---------
From: Daniel Stone <daniel at fooishbar.org>
Date: Mon, 11 Feb 2019 at 23:38
Subject: PSA: Mailman changes, From addresses no longer accurate
To: <freedesktop at lists.freedesktop.org>, <sitewranglers at
lists.freedesktop.org>


Hi all,
We have hit another step change in aggressive anti-spam techniques
from major mail providers. Over the past few days, we saw a huge spike
in the number of mails we were failing to deliver to GMail and
outlook.com in particular.

It looks like it is now no longer acceptable for us to break
DMARC/DKIM/SPF. These are DNS-based extensions to SMTP, which allow
domains to publish policies as to who should be allowed to send email
on their behalf. SPF provides source filtering, so e.g.
freedesktop.org could specify that no-one should accept mail with a
From: *@freedesktop.org unless it came from gabe.freedesktop.org.
Mailman completely breaks this: if I specified a filter only allowing
Google to send mail for @fooishbar.org, then anyone enforcing SPF
would reject receipt of this mail, as it would arrive from fd.o with
my From address.

DKIM allows domains to publish a public key in DNS, inserting a header
into mails sent from their domain cryptographically signing the value
of named headers. Mailman breaks this too: changing the Sender header
(such that bounces get processed by Mailman, rather than sending a
deluge of out-of-office and mailbox-over-quota messages to whoever
posts to the list) can break most DKIM signatures. Mailman adding the
unsubscribe footer also breaks this; we could make it not add the
footer, but in order to do so we'd have to convince ourselves that we
were not decreasing our GDPR compliance.

DMARC ties the two together, allowing domains to specify whether or
not DKIM/SPF should be mandatory, and if they fail, what action should
be taken. Despite some domains specifying a fail action of 'none'
(receiving MTA to send an advisory report to a named email address,
but still allow the email), it seems that popular services still
interpret 'none' as 'reject'.

As a result, Google in particular is dropping some number of our mails
on the floor. This does _not_ just apply to mails which fail
DMARC/DKIM/SPF: every mail we send that fails these filters (which is
a lot - e.g. Intel and NVIDIA both have SPF) decreases our sender
reputation with GMail and causes it to reject legitimate mails.

I've reached out to Google through a couple of channels to see what we
can do to increase our delivery rate to them. In the meantime, if your
mail is hosted by Google, or Outlook, and you think you're missing
mails - you probably are.

Mailman has also now been reconfigured such that if it spots a
potential DMARC violation, it rewrites the From address to be
@lists.freedesktop.org, keeping the original author in Reply-To. It
also strips DKIM headers. This seems to be about the best we can do,
unless and until the major mail service providers offer us some
alternate way to send mail. If you are replying privately to someone,
you should check very carefully that you are replying to them and not
to the list.

Unfortunately we don't have a good answer in the long run. The latest
published RFC at https://tools.ietf.org/html/rfc6377 suggests that
there are no good solutions. If anyone is blessed with an abundance of
time and familiar with the current email landscape, I would love to
talk to you and get your help to work on this. Unfortunately we don't
have the manpower to actually properly monitor email; it can often
take a collapse in successful-delivery rates for us to notice.

Ultimately, I suspect the best solution is to move most of our
discussions to dedicated fora like GitLab issues, or something like
Discourse. Fundamentally, the thing we're trying to do (send email to
thousands of people at a time using a fake From address) is ... kind
of the opposite of what the 2019 Internet wants us to do. Every few
months the major providers drop more of our mail as they become more
aggressive with spam, and every few months their userbase increases by
a non-trivial amount.

We've done a lot of work on our email infrastructure, and are doing
our best to be a responsible citizen within the constraint of having
to launder mail and forge identity on an industrial scale, but it's
coming to the point where it just may not be possible to run such a
service at such a scale anymore.

This is before even considering our other issues with Mailman 2.x: no
centralised identity management (mailing your passwords out every
month ... ?!), difficulty of GDPR compliance (editing archives
requires hand-editing every single HTML index, as there is no
non-destructive archive rebuild), the flat-out bugs (e.g. the mesa-dev
archives are usually missing half the messages), and the fact it's
been abandoned upstream in favour of Mailman 3.x, which is not
obviously better, nor is there a clear upgrade path to.

Of course we do not have any plans to stop providing email any time
soon, but it might be worth thinking about what you can do to reduce
your dependency on email lists. At the current rate of degradation, it
might be non-viable quicker than you'd think. Maybe this is unduly
gloomy, but the entire internet's direction of travel has been away
from services like Mailman, and its velocity is only increasing.

Cheers,
Daniel

Hi,

On Mon, 11 Feb 2019 at 23:38, Daniel Stone <daniel at fooishbar.org>
wrote:
> We have hit another step change in aggressive anti-spam techniques
> from major mail providers. Over the past few days, we saw a huge spike
> in the number of mails we were failing to deliver to GMail and
> outlook.com in particular.
>
> [...]
>
> Mailman has also now been reconfigured such that if it spots a
> potential DMARC violation, it rewrites the From address to be
> @lists.freedesktop.org, keeping the original author in Reply-To. It
> also strips DKIM headers. This seems to be about the best we can do,
> unless and until the major mail service providers offer us some
> alternate way to send mail. If you are replying privately to someone,
> you should check very carefully that you are replying to them and not
> to the list.
Thanks a lot to everyone who has offered help with our woes. After
some configuration changes, we are reverting back to always sending
email with the From: address of the individual sender.

I've done this on a Sunday afternoon as the lists are quiet: I'm
intending to monitor them through the evening and Monday to see what
our failed-delivery rates look like, and if we can continue to send
messages. In the longer term, we're also looking at enabling things
like ARC.

Cheers,
Daniel

Hi,

On Sun, 17 Feb 2019 at 15:29, Daniel Stone <daniel at fooishbar.org>
wrote:
> On Mon, 11 Feb 2019 at 23:38, Daniel Stone <daniel at fooishbar.org>
wrote:
> > Mailman has also now been reconfigured such that if it spots a
> > potential DMARC violation, it rewrites the From address to be
> > @lists.freedesktop.org, keeping the original author in Reply-To. It
> > also strips DKIM headers. This seems to be about the best we can do,
> > unless and until the major mail service providers offer us some
> > alternate way to send mail. If you are replying privately to someone,
> > you should check very carefully that you are replying to them and not
> > to the list.
>
> Thanks a lot to everyone who has offered help with our woes. After
> some configuration changes, we are reverting back to always sending
> email with the From: address of the individual sender.
>
> I've done this on a Sunday afternoon as the lists are quiet: I'm
> intending to monitor them through the evening and Monday to see what
> our failed-delivery rates look like, and if we can continue to send
> messages. In the longer term, we're also looking at enabling things
> like ARC.
At the moment, things are not looking too bad. For whatever reason
Google seems to reject most of our list digests as spam, but at least
normal delivery is mostly working.

Cheers,
Daniel