CentOS - Jun 2007 - ip_conntrack table filling up, dropping packets

Hi, my ip_conntrack table is filling up and now my server is dropping
packets. I'm running CentOS release 4.4 (Final) on a fairly busy
webserver.  The table is full of various connections, including a lot
of "ESTABLISHED" tcp connections from my webserver (the src is my
webserver ip), and some other random connections to my webserver, and
many "ASSURED" connections.  So why is it filling up? I changed the
default timeout value like so:

echo 36000 >
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established

but I don't think that's had any effect. any thoughts? what additional
info can I provide that would be helpful?    I did find a script that
clears out some of the stale connections using hping2, but I don't
know if that's really a great solution to this problem.

 cat  /proc/sys/net/ipv4/ip_conntrack_max     # 34576

after cleaning out the ip_conntrack table using an hping2 script:
 cat /proc/net/ip_conntrack | wc -l         # 3702     --  this number
was around 34000 before I cleared it out because it was dropping
packets. rebooting the machine, of course, clears it out.


I've spent many hours banging my head against the wall trying to
figure this out, reading in google groups and in various forums, to no
avail.   My webserver does send out emails to a few thousand
registered users (if they opt it for the email) every day.

Thank you for your time!  I hope I sent this to the right list.  This
looked like the right one.  Sorry in advance if I made a mistake.

Michelson

Hi Michelson, I have that problem also on one of my FW box. What i did is i
created a cronjob that reload the iptables rule. In this case you dont drop
any connections and you dont need to reboot your box. So far its working on
our production deployed FW.

Note: You need to find out how frequent you do this on a weeks.

Cheers!


On 6/12/07, yossarian1 at gmail.com <yossarian1 at gmail.com>
wrote:
>
> Hi, my ip_conntrack table is filling up and now my server is dropping
> packets. I'm running CentOS release 4.4 (Final) on a fairly busy
> webserver.  The table is full of various connections, including a lot
> of "ESTABLISHED" tcp connections from my webserver (the src is my
> webserver ip), and some other random connections to my webserver, and
> many "ASSURED" connections.  So why is it filling up? I changed
the
> default timeout value like so:
>
> echo 36000 >
> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
>
> but I don't think that's had any effect. any thoughts? what
additional
> info can I provide that would be helpful?    I did find a script that
> clears out some of the stale connections using hping2, but I don't
> know if that's really a great solution to this problem.
>
> cat  /proc/sys/net/ipv4/ip_conntrack_max     # 34576
>
> after cleaning out the ip_conntrack table using an hping2 script:
> cat /proc/net/ip_conntrack | wc -l         # 3702     --  this number
> was around 34000 before I cleared it out because it was dropping
> packets. rebooting the machine, of course, clears it out.
>
>
> I've spent many hours banging my head against the wall trying to
> figure this out, reading in google groups and in various forums, to no
> avail.   My webserver does send out emails to a few thousand
> registered users (if they opt it for the email) every day.
>
> Thank you for your time!  I hope I sent this to the right list.  This
> looked like the right one.  Sorry in advance if I made a mistake.
>
> Michelson
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>


-- 
Mike Calizo
Registered Linux User # 365113

_________________________________________________
Even the longest journey has to start with a small first-step
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20070616/4776eda7/attachment-0001.html>

If your server isn't having a problem, then why not bump up the
conntrack number?  I've bumped mine up to 2097152.  I can't remember
where, but I remember reading a pdf article on iptables and how many
connections a specific server with X amount of CPU's and X amount of
memory can handle.

[root at firewall1 ~]# cat /proc/sys/net/ipv4/ip_conntrack_max
2097152

-matt

On 6/15/07, Michael Calizo <mike.calizo at gmail.com>
wrote:
> Hi Michelson, I have that problem also on one of my FW box. What i did is i
> created a cronjob that reload the iptables rule. In this case you dont drop
> any connections and you dont need to reboot your box. So far its working on
> our production deployed FW.
>
> Note: You need to find out how frequent you do this on a weeks.
>
> Cheers!
>
>
> On 6/12/07, yossarian1 at gmail.com <yossarian1 at gmail.com> wrote:
> > Hi, my ip_conntrack table is filling up and now my server is dropping
> > packets. I'm running CentOS release 4.4 (Final) on a fairly busy
> > webserver.  The table is full of various connections, including a lot
> > of "ESTABLISHED" tcp connections from my webserver (the src
is my
> > webserver ip), and some other random connections to my webserver, and
> > many "ASSURED" connections.  So why is it filling up? I
changed the
> > default timeout value like so:
> >
> > echo 36000 >
> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
> >
> > but I don't think that's had any effect. any thoughts? what
additional
> > info can I provide that would be helpful?    I did find a script that
> > clears out some of the stale connections using hping2, but I don't
> > know if that's really a great solution to this problem.
> >
> > cat  /proc/sys/net/ipv4/ip_conntrack_max     # 34576
> >
> > after cleaning out the ip_conntrack table using an hping2 script:
> > cat /proc/net/ip_conntrack | wc -l         # 3702     --  this number
> > was around 34000 before I cleared it out because it was dropping
> > packets. rebooting the machine, of course, clears it out.
> >
> >
> > I've spent many hours banging my head against the wall trying to
> > figure this out, reading in google groups and in various forums, to no
> > avail.   My webserver does send out emails to a few thousand
> > registered users (if they opt it for the email) every day.
> >
> > Thank you for your time!  I hope I sent this to the right list.  This
> > looked like the right one.  Sorry in advance if I made a mistake.
> >
> > Michelson
> > _______________________________________________
> > CentOS mailing list
> > CentOS at centos.org
> > http://lists.centos.org/mailman/listinfo/centos
> >
>
>
>
> --
> Mike Calizo
> Registered Linux User # 365113
>
> _________________________________________________
> Even the longest journey has to start with a small first-step
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>

On 6/12/07, yossarian1 at gmail.com <yossarian1 at gmail.com>
wrote:
>
> Hi, my ip_conntrack table is filling up and now my server is dropping
> packets. I'm running CentOS release 4.4 (Final) on a fairly busy
> webserver.  The table is full of various connections, including a lot
> of "ESTABLISHED" tcp connections from my webserver (the src is my
> webserver ip), and some other random connections to my webserver, and
> many "ASSURED" connections.  So why is it filling up? I changed
the
> default timeout value like so:
>
> echo 36000 >
> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
>
> but I don't think that's had any effect. any thoughts? what
additional
> info can I provide that would be helpful?    I did find a script that
> clears out some of the stale connections using hping2, but I don't
> know if that's really a great solution to this problem.

I have seen this in connection with some dreadful internet worm affecting
Windows stations in the last hours. This particular worm seems related to
DEL.EXE file modifications. :(


-- 
Eduardo Grosclaude
Universidad Nacional del Comahue
Neuquen, Argentina
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20070615/37d2b341/attachment-0001.html>