Gluster users - Sep 2017 - [Gluster-infra] lists.gluster.org issues this weekend

Le samedi 16 septembre 2017 ? 20:48 +0530, Nigel Babu a
?crit?:
> Hello folks,
> 
> We have discovered that for the last few weeks our mailman server was
> used
> for a spam attack. The attacker would make use of the + feature
> offered by
> gmail and hotmail. If you send an email to example at hotmail.com,
> example+foo at hotmail.com, example+bar at hotmail.com, it goes to the same
> inbox. We were constantly hit with requests to subscribe to a few
> inboxes.
> These requests overloaded our mail server so much that it gave up. We
> detected this failure because a postmortem email to
> gluster-infra at gluster.org bounced. Any emails sent to our mailman
> server
> may have been on hold for the last 24 hours or so. They should be
> processed
> now as your email provider re-attempts.
> 
> For the moment, we've banned subscribing with an email address with a
> + in
> the name. If you are already subscribed to the lists with a + in your
> email
> address, you will continue to be able to use the lists.
> 
> We're looking at banning the spam IP addresses from being able to hit
> the
> web interface at all. When we have a working alternative, we will
> look at
> removing the current ban of using + in address.
So we have a alternative in place, I pushed a blacklist using
mod_security and a few DNS blacklist:
https://github.com/gluster/gluster.org_ansible_configuration/commit/2f4
c1b8feeae16e1d0b7d6073822a6786ed21dde



> Apologies for the outage and a big shout out to Michael for taking
> time out
> of his weekend to debug and fix the issue.
Well, you can thanks the airport in Prague for being less interesting
than a spammer attacking us.

-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170919/533a2129/attachment.sig>

Hello,
Are our servers still facing the overload issue? My replies to 
gluster-users ML are not getting delivered to the list.
Regards,
Ravi

On 09/19/2017 10:03 PM, Michael Scherer wrote:
> Le samedi 16 septembre 2017 ? 20:48 +0530, Nigel Babu a ?crit?:
>> Hello folks,
>>
>> We have discovered that for the last few weeks our mailman server was
>> used
>> for a spam attack. The attacker would make use of the + feature
>> offered by
>> gmail and hotmail. If you send an email to example at hotmail.com,
>> example+foo at hotmail.com, example+bar at hotmail.com, it goes to the
same
>> inbox. We were constantly hit with requests to subscribe to a few
>> inboxes.
>> These requests overloaded our mail server so much that it gave up. We
>> detected this failure because a postmortem email to
>> gluster-infra at gluster.org bounced. Any emails sent to our mailman
>> server
>> may have been on hold for the last 24 hours or so. They should be
>> processed
>> now as your email provider re-attempts.
>>
>> For the moment, we've banned subscribing with an email address with
a
>> + in
>> the name. If you are already subscribed to the lists with a + in your
>> email
>> address, you will continue to be able to use the lists.
>>
>> We're looking at banning the spam IP addresses from being able to
hit
>> the
>> web interface at all. When we have a working alternative, we will
>> look at
>> removing the current ban of using + in address.
> So we have a alternative in place, I pushed a blacklist using
> mod_security and a few DNS blacklist:
> https://github.com/gluster/gluster.org_ansible_configuration/commit/2f4
> c1b8feeae16e1d0b7d6073822a6786ed21dde
>
>
>
>
>> Apologies for the outage and a big shout out to Michael for taking
>> time out
>> of his weekend to debug and fix the issue.
> Well, you can thanks the airport in Prague for being less interesting
> than a spammer attacking us.
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170922/fe2322d3/attachment.html>

On Fri, 22 Sep 2017 at 18:54, Ravishankar N <ravishankar at redhat.com>
wrote:
> Hello,
> Are our servers still facing the overload issue? My replies to
> gluster-users ML are not getting delivered to the list.
>
Same here. Even this is true for gluster-devel as well.

> Regards,
> Ravi
>
>
> On 09/19/2017 10:03 PM, Michael Scherer wrote:
>
> Le samedi 16 septembre 2017 ? 20:48 +0530, Nigel Babu a ?crit :
>
> Hello folks,
>
> We have discovered that for the last few weeks our mailman server was
> used
> for a spam attack. The attacker would make use of the + feature
> offered by
> gmail and hotmail. If you send an email to example at
hotmail.com,example+foo at hotmail.com, example+bar at hotmail.com, it goes to
the same
> inbox. We were constantly hit with requests to subscribe to a few
> inboxes.
> These requests overloaded our mail server so much that it gave up. We
> detected this failure because a postmortem email togluster-infra at
gluster.org bounced. Any emails sent to our mailman
> server
> may have been on hold for the last 24 hours or so. They should be
> processed
> now as your email provider re-attempts.
>
> For the moment, we've banned subscribing with an email address with a
> + in
> the name. If you are already subscribed to the lists with a + in your
> email
> address, you will continue to be able to use the lists.
>
> We're looking at banning the spam IP addresses from being able to hit
> the
> web interface at all. When we have a working alternative, we will
> look at
> removing the current ban of using + in address.
>
> So we have a alternative in place, I pushed a blacklist using
> mod_security and a few DNS
blacklist:https://github.com/gluster/gluster.org_ansible_configuration/commit/2f4
> c1b8feeae16e1d0b7d6073822a6786ed21dde
>
>
>
>
>
> Apologies for the outage and a big shout out to Michael for taking
> time out
> of his weekend to debug and fix the issue.
>
> Well, you can thanks the airport in Prague for being less interesting
> than a spammer attacking us.
>
>
>
>
> _______________________________________________
> Gluster-users mailing listGluster-users at
gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-- 
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170922/61d2784c/attachment.html>

Le mardi 19 septembre 2017 ? 17:33 +0100, Michael Scherer a
?crit?:
> Le samedi 16 septembre 2017 ? 20:48 +0530, Nigel Babu a ?crit?:
> > Hello folks,
> > 
> > We have discovered that for the last few weeks our mailman server
> > was
> > used
> > for a spam attack. The attacker would make use of the + feature
> > offered by
> > gmail and hotmail. If you send an email to example at hotmail.com,
> > example+foo at hotmail.com, example+bar at hotmail.com, it goes to the
> > same
> > inbox. We were constantly hit with requests to subscribe to a few
> > inboxes.
> > These requests overloaded our mail server so much that it gave up.
> > We
> > detected this failure because a postmortem email to
> > gluster-infra at gluster.org bounced. Any emails sent to our mailman
> > server
> > may have been on hold for the last 24 hours or so. They should be
> > processed
> > now as your email provider re-attempts.
> > 
> > For the moment, we've banned subscribing with an email address
with
> > a
> > + in
> > the name. If you are already subscribed to the lists with a + in
> > your
> > email
> > address, you will continue to be able to use the lists.
> > 
> > We're looking at banning the spam IP addresses from being able to
> > hit
> > the
> > web interface at all. When we have a working alternative, we will
> > look at
> > removing the current ban of using + in address.
> 
> So we have a alternative in place, I pushed a blacklist using
> mod_security and a few DNS blacklist:
> https://github.com/gluster/gluster.org_ansible_configuration/commit/2
> f4
> c1b8feeae16e1d0b7d6073822a6786ed21dde
> 
> 
> 
> 
> > Apologies for the outage and a big shout out to Michael for taking
> > time out
> > of his weekend to debug and fix the issue.
> 
> Well, you can thanks the airport in Prague for being less interesting
> than a spammer attacking us.
So, it turned out to have a 2nd problem on the lists server.

Since the 2017 security incident, we have installed a remote syslog
server to store all logs. However, this logs server disk became full
(I still need to investigate why, but I think "lack of proper log
rotation" somewhere down the line).

In turn, the disk being full did cause slowdown on some syslog clients,
even if for now, we only seen issue on postfix for
supercolony.gluster.org. 

As a emergency measure, I did remove logs export for supercolony, and I
will be likely moving the logs server to the community cage and fix the
setup once I am back from PTO next week.
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170925/e7a7e319/attachment.sig>