Xapian discuss - Sep 2015 - Administrivia: Problems caused by Yahoo addresses

A few hours ago 20 addresses got unsubscribed from this list due to
bounces caused by Yahoo's DMARC p=reject policy:

https://help.yahoo.com/kb/SLN24050.html

This breaks mailing lists.  Yahoo are well aware of this, but don't seem
to care:

http://www.pcworld.com/article/2141120/yahoo-email-antispoofing-policy-breaks-mailing-lists.html

The nearest Yahoo seem to come to useful advice is suggesting lists
should rewrite the "From:" header to be from the mailing list, which
breaks the way people expect mailing lists to work.

This change actually happened last year, but we've not hit issues so far
as we don't have many posts from Yahoo addresses.  Recently there have
been quite a few, so there's been a corresponding number of bounced
messages.

I've resubscribed the affected addresses - I'm fairly sure all were due
to this, but if anyone actually meant to unsubscribe, just remove
yourself again (and sorry for the inconvenience).

I've also disabled the footer that mailman was configured to add to
each message, as that breaks the DKIM signature on the message - my
understanding is that if that signature is good them the message
shouldn't bounce, so we'll see if that helps.  You can find links
for unsubscribing, etc in the message headers anyway.

Yahoo's DKIM signature also includes the "Subject:" header, so
messages
which get the subject altered due to the addition of
"[Xapian-discuss]"
probably also have a broken signature.  I'm not a fan of such tags
personally (and there are better headers to filter list email on), but I
know they are fairly popular so they're less of a no-brainer to just
turn off.

The other option is to ban posting to the list from Yahoo addresses (and
similarly for any other email providers who implement policies that
cause collateral damage such as this).  You could probably still
subscribe from such addresses, but it would be read-only.

Any thoughts?

Cheers,
    Olly

Olly Betts <olly at survex.com> wrote:
> A few hours ago 20 addresses got unsubscribed from this list due to
> bounces caused by Yahoo's DMARC p=reject policy:
> 
> https://help.yahoo.com/kb/SLN24050.html
> 
> This breaks mailing lists.  Yahoo are well aware of this, but don't
seem
> to care:
Fwiw, mailman has a 'cleanse-dkim' handler which strips DKIM-related
headers entirely.  Maybe that's enough to get mails through...

I haven't used Mailman myself, but I took the hint from Mailman and
strip those same headers in the public-inbox.org MDA code[1] before it
(optionally) gets sent through mlmmj.

But then again there are no active Yahoo posters on lists I run...
There have not been any bounces from Yahoo lurkers, though.
> The other option is to ban posting to the list from Yahoo addresses (and
> similarly for any other email providers who implement policies that
> cause collateral damage such as this).  You could probably still
> subscribe from such addresses, but it would be read-only.
> 
> Any thoughts?
Unfortunately, I suspect things like this Yahoo change are sign of
things to come with big mail providers continuing to make life harder
for independent admins.

(Trying not hijack and promote my own project, here, but then again
 it is the reason I started using Xapian :)

  Things like this Yahoo change is one of the reasons I chose to take an
  "archives first" approach to mailing lists with public-inbox.  Email
  gets dumped into a git repository whic can get mirrored and synced
  efficiently.  Readers can subscribe via git://, Atom feed or follow
  along on a Xapian-backed website.  I'll hopefully have an NNTP
  server up in a week or so.

  Having these options will hopefully avoid problems of push-based SMTP
  delivery.  Unfortunately most mailing list participants are still used
  to getting email pushed to them.

  I also encourage reply-to-all with public-inbox since it makes the
  list server less of a single-point-of-failure and encourages drive-by
  posters.


[1] git clone git://80x24.org/public-inbox
    (still terribly undocumented at the moment)
</end promotion>

Hi Eric,

Fwiw, mailman has a 'cleanse-dkim' handler which strips
DKIM-related
> headers entirely.  Maybe that's enough to get mails through...
>
This is insufficient. You need to rewrite the From. Looks like Mailman has
this feature now:

http://wiki.list.org/DEV/DMARC

Olly, you'll need to upgrade to  2.1.16 to be able to turn this on.


Unfortunately, I suspect things like this Yahoo change are sign
of
> things to come with big mail providers continuing to make life harder
> for independent admins.
>
Yahoo's change was harsh, and made everybody's list management
difficult,
even for Google Groups.

It was relatively easy for us to rewrite the header so that's what we did.
AOL has similar problems, btw. We currently rewrite emails from the
following domains: google.com ebay.com paypal.com yahoo.com aol.com. FYI,
the code that does this is here:

https://github.com/biviosoftware/perl-Bivio/blob/master/Mail/Outgoing.pm#L387

(Trying not hijack and promote my own project, here, but then
again
>  it is the reason I started using Xapian :)
>
>   Things like this Yahoo change is one of the reasons I chose to take an
>   "archives first" approach to mailing lists with public-inbox. 
Email
>   gets dumped into a git repository whic can get mirrored and synced
>   efficiently.  Readers can subscribe via git://, Atom feed or follow
>   along on a Xapian-backed website.  I'll hopefully have an NNTP
>   server up in a week or so.
>
Neat idea. Thanks for sharing. :)

>
>   Having these options will hopefully avoid problems of push-based SMTP
>   delivery.  Unfortunately most mailing list participants are still used
>   to getting email pushed to them.
>
IMHO: Email is still the killer app. :) What did you use to read and reply
to this? Good archival software is helpful especially with Xapian, but
we've found that the more you can do via a user's MUA, the more
they'll
like your software. :)

Cheers,
Rob