I have just completed the installation of a new firewall running Shorewall 1.4 on Mandrake 9.2 for our campus network. It appears to be running fairly well so far, but is generating significantly more log entries than our previous linux 2.0.x firewall... Our previous firewall enjoyed more than 6 years of 24/7 operation with no downtime before we finally decided it needed more horsepower, and I''d like to do what I can to ensure the new one will be similarly long- lived. Our bandwidth usage is growing constantly, so that is my main concern. Are there any sizing guidelines for the linux 2.4 firewall? Currently we have: PPro 200 with 96 Mb RAM Mandrake 9.2 with 2.4.22 kernel 100Mb FD connection to LAN (tulip) 10Mb FD connection to net (e100) Shoreline 1.4 with ~20 rules, a few NAT entries. Typical average traffic: 200 pkts/sec, 75kBytes/sec Peak traffic ~500kBytes/sec (5 minute avg) We have approx 500 total users, with about 250 peak concurrent users. I''m in the process of setting up logwatch to deal with 100+ Mb of logs each day. Any other tools I can look at for gauging the performance, etc? -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Shawn Wright, I.T. Manager Shawnigan Lake School http://www.sls.bc.ca swright@sls.bc.ca
Shawn Wright wrote:> I''m in the process of setting up logwatch to deal with 100+ Mb of logs > each day. Any other tools I can look at for gauging the performance, > etc?You know that you are not a helpless bystander WRT the volume of log messages that a Shorewall-configured firewall produces. There are a significant number of controls available on Shorewall logging. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net
Shawn Wright wrote:> Any other tools I can look at for gauging the performance, etc?You probably want to keep track of your connection tracking table usage: gateway:~# cat /proc/sys/net/ipv4/ip_conntrack_max 16384 gateway:~# grep ip_conntrack /proc/slabinfo ip_conntrack 107 240 320 17 20 1 gateway:~# The first command shows that I have a maximum of 16384 entries in my connection tracking table. The second entry shows that: a) There are 107 active entries. b) There are 240 allocated entries. c) Each entry is 320 bytes long. You can dynamically change the size of the table by echoing new values into /proc/sys/net/ipv4/ip_conntrack_max. Such dynamic updates don''t survive a reboot of course so you probably want to use your distro''s method of setting these values (with Mandrake, it''s /etc/sysctl.conf). -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net
On 19 Apr 2004 at 10:39, Tom Eastep wrote:> Shawn Wright wrote: > > Any other tools I can look at for gauging the performance, etc? > > You probably want to keep track of your connection tracking table usage: > > gateway:~# cat /proc/sys/net/ipv4/ip_conntrack_max > 16384 > gateway:~# grep ip_conntrack /proc/slabinfo > ip_conntrack 107 240 320 17 20 1 > gateway:~# > > The first command shows that I have a maximum of 16384 entries in my > connection tracking table. > > The second entry shows that: > > a) There are 107 active entries. > b) There are 240 allocated entries. > c) Each entry is 320 bytes long. > > You can dynamically change the size of the table by echoing new values > into /proc/sys/net/ipv4/ip_conntrack_max. Such dynamic updates don''t > survive a reboot of course so you probably want to use your distro''s > method of setting these values (with Mandrake, it''s /etc/sysctl.conf).Tom, Thanks for this info. Below are my current stats, but the machine was rebooted last night, so the numbers probably haven''t peaked yet. Our heaviest traffic will be later today. [root@fw console]# cat /proc/sys/net/ipv4/ip_conntrack_max 6144 [root@fw console]# grep ip_conntrack /proc/slabinfo ip_conntrack 622 1920 320 56 160 1 I''m guessing that the default of 6144 could probably be bumped up a bit. But are there other factors to consider before doing so? -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Shawn Wright, I.T. Manager Shawnigan Lake School http://www.sls.bc.ca swright@sls.bc.ca
Shawn Wright wrote:> On 19 Apr 2004 at 10:39, Tom Eastep wrote: > > >>Shawn Wright wrote: >> >>>Any other tools I can look at for gauging the performance, etc? >> >>You probably want to keep track of your connection tracking table usage: >> >>gateway:~# cat /proc/sys/net/ipv4/ip_conntrack_max >>16384 >>gateway:~# grep ip_conntrack /proc/slabinfo >>ip_conntrack 107 240 320 17 20 1 >>gateway:~# >> >>The first command shows that I have a maximum of 16384 entries in my >>connection tracking table. >> >>The second entry shows that: >> >>a) There are 107 active entries. >>b) There are 240 allocated entries. >>c) Each entry is 320 bytes long. >> >>You can dynamically change the size of the table by echoing new values >>into /proc/sys/net/ipv4/ip_conntrack_max. Such dynamic updates don''t >>survive a reboot of course so you probably want to use your distro''s >>method of setting these values (with Mandrake, it''s /etc/sysctl.conf). > > > Tom, Thanks for this info. > > Below are my current stats, but the machine was rebooted last night, so > the numbers probably haven''t peaked yet. Our heaviest traffic will be later > today. > > [root@fw console]# cat /proc/sys/net/ipv4/ip_conntrack_max > 6144 > [root@fw console]# grep ip_conntrack /proc/slabinfo > ip_conntrack 622 1920 320 56 160 1 > > I''m guessing that the default of 6144 could probably be bumped up a bit. > But are there other factors to consider before doing so?The only factor is the amount of memory that you can afford to dedicate to connection tracking. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net
Tom Eastep wrote:> > The only factor is the amount of memory that you can afford to dedicate > to connection tracking. >Also, if you change the value dramatically, you might also want to bump up the size of the connection tracking hash table. By default, it is sized as 1/8th the max conntrack entries (actually, the max entries are sized at 8 timies the hash table size). The hash table is created at boot/module-load time so you must set it''s size in /etc/modules.conf: ip_conntrack_core hashsize=<number> -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net
Tom Eastep wrote:> Tom Eastep wrote: > >> >> The only factor is the amount of memory that you can afford to >> dedicate to connection tracking. >> > > Also, if you change the value dramatically, you might also want to bump > up the size of the connection tracking hash table. By default, it is > sized as 1/8th the max conntrack entries (actually, the max entries are > sized at 8 timies the hash table size). The hash table is created at > boot/module-load time so you must set it''s size in /etc/modules.conf: > > ip_conntrack_core hashsize=<number> >Ooops -- stop the presses. That should be: ip_conntrack hashsize=<number> And if you let Shorewall load modules for you, you can add the "hashsize=<number>" part to the entry for ip_conntrack in /etc/shorewall/modules. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net
Greetings, Having a little trouble trying to figure out how to do incoming dcc transfers in irc. Outgoing works fine such as me doing dcc send but doing dcc get doesnt seem to work. Im not sure if this is a firewall issue or not. Any thoughts? Regards, Jason Harrison
bahadunn wrote:> Greetings, > > Having a little trouble trying to figure out how to do incoming dcc > transfers in irc. Outgoing works fine such as me doing dcc send but > doing dcc get doesnt seem to work. Im not sure if this is a firewall > issue or not. Any thoughts?Are you loading ip_conntrack_irc and ip_nat_irc? -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net
Yes both those modules are loaded. On Mon, 2004-04-19 at 20:38, Tom Eastep wrote:> bahadunn wrote: > > > Greetings, > > > > Having a little trouble trying to figure out how to do incoming dcc > > transfers in irc. Outgoing works fine such as me doing dcc send but > > doing dcc get doesnt seem to work. Im not sure if this is a firewall > > issue or not. Any thoughts? > > Are you loading ip_conntrack_irc and ip_nat_irc? > > -Tom
bahadunn wrote:> Yes both those modules are loaded. >and is the channel you are on using port 6667 or another port? -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net
Hi, After some investigation via tcpdump and tests with other people I think the problem is not me but my friend im dccing with. Thanks for your time. Regards, Jason On Mon, 2004-04-19 at 20:45, Tom Eastep wrote:> bahadunn wrote: > > > Yes both those modules are loaded. > > > > and is the channel you are on using port 6667 or another port? > > -Tom
On Tuesday 20 April 2004 02:51, bahadunn wrote:> After some investigation via tcpdump and tests with other people I think > the problem is not me but my friend im dccing with.Exactly. DCC works by sending the IP address and a port of the dcc sender to the receiver which then connects to that ip:port, which means if you allow all outgoing traffic you''re on the safe side. You only need the connection tracking and nat modules for sending a dcc yourself so the firewall know which port to forward to you because a dcc is incoming. Alex P.S.: On a sidenote, please don''t reply to a post starting a new topic, this breaks threading which is done by In-Reply-To and References headers. Thx
On 19 Apr 2004 at 10:39, Tom Eastep wrote:> Shawn Wright wrote: > > Any other tools I can look at for gauging the performance, etc? > > You probably want to keep track of your connection tracking table usage: > > gateway:~# cat /proc/sys/net/ipv4/ip_conntrack_max > 16384 > gateway:~# grep ip_conntrack /proc/slabinfo > ip_conntrack 107 240 320 17 20 1 > gateway:~# > > The first command shows that I have a maximum of 16384 entries in my > connection tracking table. > > The second entry shows that: > > a) There are 107 active entries. > b) There are 240 allocated entries. > c) Each entry is 320 bytes long. > > You can dynamically change the size of the table by echoing new values > into /proc/sys/net/ipv4/ip_conntrack_max. Such dynamic updates don''t > survive a reboot of course so you probably want to use your distro''s > method of setting these values (with Mandrake, it''s /etc/sysctl.conf).Ok, now were digging into the past a little, but I''m still trying to get a handle on estimated memory usage on our firewall. Our old one was crippled with 96Mb, and was using over 200Mb of swap continuously before it was taken down this week. Our current box is running with 256Mb, and is doing much better so for, but still seems to be using a lot of memory for not that many connections in use, and it probably has not seen a peak period of use yet. Here are the stats: [root@proxy4 net]# free total used free shared buffers cached Mem: 255464 250376 5088 0 44 174816 -/+ buffers/cache: 75516 179948 Swap: 819176 0 819176 [root@proxy4 net]# cat /proc/sys/net/ipv4/ip_conntrack_max 16384 [root@proxy4 net]# grep conntrack /proc/slabinfo ip_conntrack 871 1008 320 83 84 1 : 124 62 This is after less than 24 hours. Previously I have seen the allocated # of conntrack entries to be >1900. I assume the kernel will reclaim these after a period of time? If so, is there a way to determine the peak allocated/used entries? What other factors are contributing to using this much memory with so few connections? Or am I just misinterpreting the output from ''free'' here? Thanks. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Shawn Wright, I.T. Manager Shawnigan Lake School http://www.sls.bc.ca swright@sls.bc.ca
On Thu, 2004-11-18 at 14:26 -0800, Shawn Wright wrote:> > Ok, now were digging into the past a little, but I''m still trying to get a > handle on estimated memory usage on our firewall. Our old one was > crippled with 96Mb, and was using over 200Mb of swap continuously > before it was taken down this week. Our current box is running with > 256Mb, and is doing much better so for, but still seems to be using a lot of > memory for not that many connections in use, and it probably has not > seen a peak period of use yet. Here are the stats: > > [root@proxy4 net]# free > total used free shared buffers cached > Mem: 255464 250376 5088 0 44 174816 > -/+ buffers/cache: 75516 179948 > Swap: 819176 0 819176 > > [root@proxy4 net]# cat /proc/sys/net/ipv4/ip_conntrack_max > 16384 > > [root@proxy4 net]# grep conntrack /proc/slabinfo > ip_conntrack 871 1008 320 83 84 1 : 124 62 > > This is after less than 24 hours. Previously I have seen the allocated # of > conntrack entries to be >1900. I assume the kernel will reclaim these after > a period of time?Entries are freed as the connections that they represent are closed. You can of course see the actual conntrack entries using "shorewall show connections". IIRC, the quantum of memory allocation in the slab cache is a slab -- if you "head /proc/slabinfo", you can see the column headings and should be able to determine the size of a slab by doing a little math. Empty slabs are subject to reclamation by the kernel.> If so, is there a way to determine the peak > allocated/used entries?Not that I''m aware of.> > What other factors are contributing to using this much memory with so > few connections? Or am I just misinterpreting the output from ''free'' here? >Why don''t you look at the rest of /proc/slabinfo and see what other caches have a lot of allocated entries? -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
On 18 Nov 2004 at 15:06, Tom Eastep wrote:> On Thu, 2004-11-18 at 14:26 -0800, Shawn Wright wrote: > > > > > [root@proxy4 net]# grep conntrack /proc/slabinfo > > ip_conntrack 871 1008 320 83 84 1 : 124 62 > > > > This is after less than 24 hours. Previously I have seen the allocated # of > > conntrack entries to be >1900. I assume the kernel will reclaim these after > > a period of time? > > Entries are freed as the connections that they represent are closed. You > can of course see the actual conntrack entries using "shorewall show > connections". IIRC, the quantum of memory allocation in the slab cache > is a slab -- if you "head /proc/slabinfo", you can see the column > headings and should be able to determine the size of a slab by doing a > little math. Empty slabs are subject to reclamation by the kernel.The headings weren''t in there, but ''man slabinfo'' gave me all I could want and more... I think :-)> > If so, is there a way to determine the peak > > allocated/used entries? > > Not that I''m aware of.Looks like this might do it with the "high water mark"? (from man slabinfo) "Kernels compiled with slab cache statistics will also have "(statis- tics)" in the first line of output, and will have 5 additional columns, namely: the high water mark of active objects; the number of times objects have been allocated; the number of times the cache has grown (new pages added to this cache); the number of times the cache has been reaped (unused pages removed from this cache); and the number of times there was an error allocating new pages to this cache. If slab cache statistics are not enabled for this kernel, these columns will not be shown. "> > What other factors are contributing to using this much memory with so > > few connections? Or am I just misinterpreting the output from ''free'' here? > > Why don''t you look at the rest of /proc/slabinfo and see what other > caches have a lot of allocated entries?Thanks for the tip. Looks like I found the other users: xfs_inode 53064 53073 408 5897 5897 1 : 124 62 linvfs_icache 53064 53064 352 4824 4824 1 : 124 62 If I understand this correctly, each of these two are using ~20Mb. Perhaps the XFS filesystem is not the best for this application... We''ve been using it successfully for busy fileservers for several years, so I always use it by default now. Also, I have this: ip_dst_cache 2168 2420 192 109 121 1 : 252 126 ip_conntrack 1050 1236 320 103 103 1 : 124 62 So I''m guessing I have two ip_dst_cache entries for each ip_conntrack, which seems to make sense. Thanks for all the help - I''ll continue to monitor stats for a few days. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Shawn Wright, I.T. Manager Shawnigan Lake School http://www.sls.bc.ca swright@sls.bc.ca
On Nov 18, 2004, at 6:06 PM, Tom Eastep wrote:> On Thu, 2004-11-18 at 14:26 -0800, Shawn Wright wrote: > >> >> Ok, now were digging into the past a little, but I''m still trying to >> get a >> handle on estimated memory usage on our firewall. Our old one was >> crippled with 96Mb, and was using over 200Mb of swap continuously >> before it was taken down this week. Our current box is running with >> 256Mb, and is doing much better so for, but still seems to be using a >> lot of >> memory for not that many connections in use, and it probably has not >> seen a peak period of use yet. Here are the stats: >> >> [root@proxy4 net]# free >> total used free shared buffers >> cached >> Mem: 255464 250376 5088 0 44 >> 174816 >> -/+ buffers/cache: 75516 179948 >> Swap: 819176 0 819176>> >> What other factors are contributing to using this much memory with so >> few connections? Or am I just misinterpreting the output from ''free'' >> here? >> >Just for info: you''re (shawn) only using about 75MB here; there''s 175Mb of disk cache there. So, you have about 175Mb free. You obviously don''t want to get to a point where you''re swapping in and out kernel data structures, but you seem to have a lot of headroom, since a lot of that 75MB is user-land stuff, and maybe just a small portion is related to firewall activities.. -SteveK