Shorewall users - Apr 2004 - connection state tracking problem with approx 3000 hosts

Hello all,

I''m the systems administrator as a small cable ISP.  I have been using 
shorewall for small network firewalls for a couple years now - it''s a 
wonderful tool.

- Shorewall version - 1.4.8
- more config info attached

Recently I decided to replace our ailing backbone router and decided that I 
wanted the flexibility that a linux box affords me.  I decided to use 
shorewall for my basic packet filtering needs because shorewall''s zones
make
it much easier for me to deal with the number of interfaces and networks on 
this router (5 interfaces 6 zones with two of those interfaces available to 
connect to the net via the same upstream provider but over two physically 
diverse routes in the event of a failure on one leg or the other - I am 
moving to BGP once I get this conntrack issue straightened out.)

The setup was reasonably straight forward, but I''m running into a
problem that
I haven''t run into before - my ip_conntrack table is filling up.  I
have a
gig of ram in the box which permits  /proc/sys/net/ipv4/ip_conntrack_max to 
be set to 57344.  When shorewall was first started up, I was seeing around 
19000 tracked connections in /proc/net/ip_conntrack but over time (last two 
weeks) that number just keeps getting larger and larger I''m currently 
averaging around 

So, hopefully that''s enough background (see my attached config info). 
My
guess is that some timeout is set far too high and the conntrack table is not 
clearing out old connections that are no longer valid or that I have somehow 
misconfigured my system and I am missing the connection close that would 
clear that entry from my conntrack table.  The one that gives me pause is 
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established but 
according to the documentation this is set this high so that long running 
connections are not killed.  As this is an ISP, I''m not sure how much I
can
muck with that setting.

If neither is the case, then I am simply tracking state for too much traffic 
and I''m going to need to disable state tracking for most of my traffic
- this
would be totally acceptable as well, but I''m not sure how to do it for 
specific shorewall rules.  If anyone can tell me whether there is a
"correct"
way to disable the "--state" part of individual rules I''d
love to hear it - I
don''t want to do a crummy hack.

Other than this issue, I''m successfully moving about 30Mbps for over
3000 IPs
and collecting accounting data using nprobe and exporting it to a netflow 
collector for doing billing based upon usage.  The box isn''t even
breaking a
sweat - my whole problem is memory. 
 
-- 
Mason Schmitt
Systems Administrator
Sunwave Internet
ph: (250) 832-9711
www.sunwave.net

Mason Schmitt wrote:

> 
> The setup was reasonably straight forward, but I''m running into a
problem that
> I haven''t run into before - my ip_conntrack table is filling up. 
I have a
> gig of ram in the box which permits  /proc/sys/net/ipv4/ip_conntrack_max to
> be set to 57344. 
Which is only 320 * 57344 = 18,350,080 bytes. Surely you can afford to 
set it higher than that!!! You *ARE* setting it and not just accepting 
the default, aren''t you?

If you are not setting it, then you certainly should be. And you''ll
want
to set the hashsize larger also (at about 1/8th of the table size). The 
latter can''t be done via /proc but rather must be set when the 
ip_conntrack module is loaded, either in /etc/shorewall/modules or in 
/etc/modules.conf.

  When shorewall was first started up, I was seeing
around
> 19000 tracked connections in /proc/net/ip_conntrack but over time (last two
> weeks) that number just keeps getting larger and larger I''m
currently
> averaging around 
Yes?
> 
> If neither is the case, then I am simply tracking state for too much
traffic
> and I''m going to need to disable state tracking for most of my
traffic
Then you don''t use Shorewall.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net

On April 30, 2004 12:46 pm, Tom Eastep wrote:
> Which is only 320 * 57344 = 18,350,080 bytes. Surely you can afford to
> set it higher than that!!! You *ARE* setting it and not just accepting
> the default, aren''t you?
>
No I was not explicitly setting it.  The only bit of documentation I have read 
on the subject is from 
http://iptables-tutorial.frozentux.net/iptables-tutorial.html#THECONNTRACKENTRIES

"The default value held by this entry varies heavily depending on how much 
memory you have. On 128 MB of RAM you will get 8192 possible entries, and at 
256 MB of RAM, you will get 16376 entries. You can read and set your settings 
through the /proc/sys/net/ipv4/ip_conntrack_max setting."

Thank you for pointing out that each conntrack entry is only 320 bytes, that 
makes this whole process much easier.
> If you are not setting it, then you certainly should be. And
you''ll want
> to set the hashsize larger also (at about 1/8th of the table size). The
> latter can''t be done via /proc but rather must be set when the
> ip_conntrack module is loaded, either in /etc/shorewall/modules or in
> /etc/modules.conf.
>
I did see this tip in a previous post when going through the archives, I will 
do that as well.

-- 
Mason Schmitt
Systems Administrator
Sunwave Internet
ph: (250) 832-9711
www.sunwave.net

Mason Schmitt wrote:

> 
> Thank you for pointing out that each conntrack entry is only 320 bytes,
that
> makes this whole process much easier.
> 
On any given system, you can see the size of these entries:

gateway:/usr/share/shorewall# cat /proc/slabinfo  | grep conntrack
ip_conntrack          52    360    320   13   30    1
gateway:/usr/share/shorewall#

In this case:

52   entries are in use
360  entries are allocated
320  is the length in bytes of each entry
> 
> I did see this tip in a previous post when going through the archives, I
will
> do that as well.
> 
Good. I believe that suitably sizing your connection tracking table 
should solve your problem.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net