Shorewall users - Dec 2007 - best practice/How To handel DOS / misconfigured, broadcasting clients

Hi,

today I had an other "nice" situatuation: Our perimeter firewall
stopped
working and I found, that my /var/log-partition was 100% in use. The
message file was about 2.5 GB :-) "normal" for me yet was about up to
500 MB/week.

I found out, that a students Mac Book flooded the lan with broadcasts
and tried to reach one outside server on the internet with port 5354.

His broadcast etc. had been dropped or rejected by the shorewalls
policies and had been loged - so the logfile grews very fast.

Regarding the german apfelwiki (http://www.apfelwiki.de/Main/Port) port
5354 belongs to mdnsresponder which is
http://developer.apple.com/opensource/internet/bonjour.html.

We offer a DMZ for mobile computers from students and academics, which
is directly connected to the perimeter firewall with an own subnet and nic.

What could/would be a good solution or tool or your suggestions to
handle such broadcast storms or situations? I can tell our students a
hundred time, that some service aren''t availabel and that they should
disable e.g. bonjour broadcasting, but why should they care :-) And for
me it isn'' an option to talk to tham after they have "killed"
our firewal.

Thanks for your help and suggestions!

Best Regards

Götz Reinicke
-- 
Götz Reinicke
IT Koordinator

Tel. +49 7141 969 420
Fax  +49 7141 969 55 420
E-Mail goetz.reinicke@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Mathildenstr. 20
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016
Vorsitzender des Aufsichtsrats:
Dr. Christoph Palmer, MdL, Minister a.D.

Geschäftsführer:
Prof. Thomas Schadt









-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4

On Mon, Dec 03, 2007 at 09:13:48AM +0100, G?tz Reinicke
wrote:
> I found out, that a students Mac Book flooded the lan with broadcasts
> and tried to reach one outside server on the internet with port 5354.
> 
> His broadcast etc. had been dropped or rejected by the shorewalls
> policies and had been loged - so the logfile grews very fast.
Why are you even bothering to log this information?
> What could/would be a good solution or tool or your suggestions to
> handle such broadcast storms or situations?
You apparently do not have a problem with network traffic, merely with
logging, so the only problem you need to solve here is logging. You
have two specific problems to tackle here:
> Our perimeter firewall stopped working and I found, that my
> /var/log-partition was 100% in use.
That''s the first one. Why does your firewall stop working when the
filesystem containing /var/log fills up? There is no reason why
netfilter should stop passing packets, so it can''t be directly caused
by shorewall - I''m betting that some other application is at fault
here. If the application cannot be fixed, and you want to keep the
logs, they should not be placed on the same filesystem as the one
which breaks the system when it fills up.


The second problem is that your logging system fills space without
bounds. It would be straightforward to rotate the logs based on size
so that this cannot happen.

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4

Andrew Suffield schrieb:
> On Mon, Dec 03, 2007 at 09:13:48AM +0100, G?tz Reinicke wrote:
>> I found out, that a students Mac Book flooded the lan with broadcasts
>> and tried to reach one outside server on the internet with port 5354.
>>
>> His broadcast etc. had been dropped or rejected by the shorewalls
>> policies and had been loged - so the logfile grews very fast.
> 
> Why are you even bothering to log this information?
Because I''d like to know what''s going on. How would I monitor
hacking or
attack attemps without logging? (May be there is an other way I haven''t
found yet.)
>> What could/would be a good solution or tool or your suggestions to
>> handle such broadcast storms or situations?
> 
> You apparently do not have a problem with network traffic, merely with
> logging, so the only problem you need to solve here is logging. You
> have two specific problems to tackle here:
> 
>> Our perimeter firewall stopped working and I found, that my
>> /var/log-partition was 100% in use.
> 
> That''s the first one. Why does your firewall stop working when the
> filesystem containing /var/log fills up? There is no reason why
> netfilter should stop passing packets, so it can''t be directly
caused
> by shorewall - I''m betting that some other application is at fault
> here. If the application cannot be fixed, and you want to keep the
> logs, they should not be placed on the same filesystem as the one
> which breaks the system when it fills up.
Thats true; the problem for me is, that I have no clue, which app stopps
if the filesystem is full so that shorewall will stop after that. May be
syslog fails and so dose shorewall while trying to log?
> The second problem is that your logging system fills space without
> bounds. It would be straightforward to rotate the logs based on size
> so that this cannot happen.
Thats a good idea! How to do this with syslogd?

Thanks and best regards

Götz
-- 
Götz Reinicke
IT Koordinator

Tel. +49 7141 969 420
Fax  +49 7141 969 55 420
E-Mail goetz.reinicke@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Mathildenstr. 20
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016
Vorsitzender des Aufsichtsrats:
Dr. Christoph Palmer, MdL, Minister a.D.

Geschäftsführer:
Prof. Thomas Schadt


-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4

On Mon, Dec 03, 2007 at 09:55:53AM +0100, G?tz Reinicke
wrote:
> Andrew Suffield schrieb:
> > On Mon, Dec 03, 2007 at 09:13:48AM +0100, G?tz Reinicke wrote:
> >> I found out, that a students Mac Book flooded the lan with
broadcasts
> >> and tried to reach one outside server on the internet with port
5354.
> >>
> >> His broadcast etc. had been dropped or rejected by the shorewalls
> >> policies and had been loged - so the logfile grews very fast.
> > 
> > Why are you even bothering to log this information?
> 
> Because I''d like to know what''s going on. How would I
monitor hacking or
> attack attemps without logging? (May be there is an other way I
haven''t
> found yet.)
The problem with this approach is that you are logging only attempts
which have obviously *failed* since the firewall blocked them, and not
logging any attempts that succeed. This information is unlikely to be
useful.

Logging things that get through the firewall means logging all
traffic, and that''s so hardware-intensive that most people figure it
isn''t worth bothering - it''s still not that likely that you
would gain
anything from it, since there isn''t much you could do with it even if
you did catch something.
> >> Our perimeter firewall stopped working and I found, that my
> >> /var/log-partition was 100% in use.
> > 
> > That''s the first one. Why does your firewall stop working
when the
> > filesystem containing /var/log fills up? There is no reason why
> > netfilter should stop passing packets, so it can''t be
directly caused
> > by shorewall - I''m betting that some other application is at
fault
> > here. If the application cannot be fixed, and you want to keep the
> > logs, they should not be placed on the same filesystem as the one
> > which breaks the system when it fills up.
> 
> Thats true; the problem for me is, that I have no clue, which app stopps
> if the filesystem is full so that shorewall will stop after that. May be
> syslog fails and so dose shorewall while trying to log?
Shorewall isn''t involved. The logging is coming from the kernel
directly to syslog, which does not hang when the userspace tool fails
to fetch data - it uses a static ring buffer in kernel memory, and
simply discards old data. Whatever broke the system is more subtle,
and you''re going to have fun trying to figure it out. It may be
necessary to clone the system and force it to fail so that you can
examine the problem more closely.
> > The second problem is that your logging system fills space without
> > bounds. It would be straightforward to rotate the logs based on size
> > so that this cannot happen.
> 
> Thats a good idea! How to do this with syslogd?
syslogd doesn''t rotate logs itself, that''s always done by
external
tools. Your vendor probably shipped something involving cron and
possibly logrotate to handle it. If it''s logrotate (common on the
mainstream Redhat/SuSE/Debian derivatives, but embedded/turnkey
systems vary), then examine the man page and the ''size''
option. Note
that you may also have to fiddle with the cron job so that logrotate
is invoked more frequently - on Debian systems (for example), it''s
daily, and you''d want it to be more like every five minutes.

But it''s all heavily system-dependant. YMMV. In a pinch you can do it
with just cron and a couple of lines of shell.

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4