Gluster users - Sep 2017 - lists.gluster.org issues this weekend

Hello folks,

We have discovered that for the last few weeks our mailman server was used
for a spam attack. The attacker would make use of the + feature offered by
gmail and hotmail. If you send an email to example at hotmail.com,
example+foo at hotmail.com, example+bar at hotmail.com, it goes to the same
inbox. We were constantly hit with requests to subscribe to a few inboxes.
These requests overloaded our mail server so much that it gave up. We
detected this failure because a postmortem email to
gluster-infra at gluster.org bounced. Any emails sent to our mailman server
may have been on hold for the last 24 hours or so. They should be processed
now as your email provider re-attempts.

For the moment, we've banned subscribing with an email address with a + in
the name. If you are already subscribed to the lists with a + in your email
address, you will continue to be able to use the lists.

We're looking at banning the spam IP addresses from being able to hit the
web interface at all. When we have a working alternative, we will look at
removing the current ban of using + in address.

Apologies for the outage and a big shout out to Michael for taking time out
of his weekend to debug and fix the issue.

-- 
nigelb
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170916/511e68ab/attachment.html>

Le samedi 16 septembre 2017 ? 20:48 +0530, Nigel Babu a
?crit?:
> Hello folks,
> 
> We have discovered that for the last few weeks our mailman server was
> used
> for a spam attack. The attacker would make use of the + feature
> offered by
> gmail and hotmail. If you send an email to example at hotmail.com,
> example+foo at hotmail.com, example+bar at hotmail.com, it goes to the same
> inbox. We were constantly hit with requests to subscribe to a few
> inboxes.
> These requests overloaded our mail server so much that it gave up. We
> detected this failure because a postmortem email to
> gluster-infra at gluster.org bounced. Any emails sent to our mailman
> server
> may have been on hold for the last 24 hours or so. They should be
> processed
> now as your email provider re-attempts.
> 
> For the moment, we've banned subscribing with an email address with a
> + in
> the name. If you are already subscribed to the lists with a + in your
> email
> address, you will continue to be able to use the lists.
> 
> We're looking at banning the spam IP addresses from being able to hit
> the
> web interface at all. When we have a working alternative, we will
> look at
> removing the current ban of using + in address.
So we have a alternative in place, I pushed a blacklist using
mod_security and a few DNS blacklist:
https://github.com/gluster/gluster.org_ansible_configuration/commit/2f4
c1b8feeae16e1d0b7d6073822a6786ed21dde



> Apologies for the outage and a big shout out to Michael for taking
> time out
> of his weekend to debug and fix the issue.
Well, you can thanks the airport in Prague for being less interesting
than a spammer attacking us.

-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170919/533a2129/attachment.sig>