LARTC - Aug 2005 - Hardware Configuration Ideas

Hey guys,

I am planning to buy some components for a Linux router that will handle the
Internet access of 200 computers (includes tc shaping) and some inter
sub-network routing (at least 100MBps per eth - and there are 3 eth cards).

I was thinking of a:
Pentium 4 - 3GHz
256 or 512MB RAM
Network Cards.

Now - I wonder what is more important: the processor speed or the amount of
RAM.

And can you point me to some good Network Cards that you have used and are
not so expensive. Some Intel, etc. I have no clue about this subject...


Maybe this discussion can be extended to a list of best practices to set up
a performant Linux Router from the hardware point of view.

Thanks in advance,
Mihai

Hello, Mihai,

On Mon, 15 Aug 2005 23:53:38 +0300
"Mihai Vlad" <mihaivlad@web-profile.net> wrote:
> Hey guys,
> 
> I am planning to buy some components for a Linux router that will
> handle the Internet access of 200 computers (includes tc shaping) and
> some inter sub-network routing (at least 100MBps per eth - and there
> are 3 eth cards).
> 
> I was thinking of a:
> Pentium 4 - 3GHz
> 256 or 512MB RAM
> Network Cards.
This is more than enough - I''d suggest a PIII/1GHz with 512MB RAM for
those kind of needs, which is also a bit generous.

[ cut ]
> And can you point me to some good Network Cards that you have used
> and are not so expensive. Some Intel, etc. I have no clue about this
> subject...
Network cards - YMMV, but I''d go for Intel EtherXpress Pro/100 for the
100Mbit, and something based on BCM57xx for the gigabit speeds.

My experience dictates that for now the 2.4.30+ kernels are more
suitable for tc/QoS setups.

Hope it helped,
Nickola


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Memory will most definitely be your problem.  I think you could get away with a
fairly low end processor (read < 1 GHz) but you will need a lot of memory
depending on how much you want to do.  I have a router in place that I was
running out of memory for the connection tracking sub system.  I ended up
allocating 1 GB of RAM to just connection tracking.  In fact you need 1 GB (or
very close to) to be able to track 65535 connections.  You may think this is way
over kill, but not really.  Keep in mind that connections tend to hang around on
average 10 minutes after they are closed b/c not all systems out there close
them correctly and thus they have to time out (10 minutes).  You can get away
with less RAM but you need to watch your DMESG to make sure that you
don''t see any issues with your connection tracking table filling up, it
acts like a FIFO if memory serves.  If you are not doing muc
 h in the way if *VERY* *ADVANCED* firewalling, just basic source and / or
destination IP v
alidation and / or source and / or destination port validation will not need
much of a processor.  In fact I''d try it with a 500 MHz to 1 GHz
system, what ever is the most economical that you can get your hands on.

Another problem that you may run in to will be filling your ARP table.  The
kernel space ARP table is not very large at all, only like 64 or maybe up to 255
IP MAC pairs.  I want to say it''s closer to 64.  Thus you may want to
take a look at using the ARP Daemon for Linux to offload the ARP cache to thus
avoiding this issue.  Basically how it works (from what I''ve read) is
you reduce the number of times you query the ARP cache in kernel to 0 which will
cause the kernel to query the user space daemon for the ARP data.  The user
space daemon does it''s own ARPing to make sure that it has the
information to hand to the kernel.  The main advantage of the user space daemon
is that it can handle LOTS of ARP entries, well beyond 1024 (I think).

Something else you might consider would be some managed switches so that you
could bond your connections out of the router to them thus ensuring that a cable
failure (disconnection) will not take the router down.  If you plug everything
in to the managed switch and set up some VLANs you can easily do everything that
you are wanting to do over the bonded connections with VLANs on top.  The VLAN
interfaces in Linux look like another network interface that you can do all the
routing that you want off of.

If the client systems you are going to be firewalling for are business systems I
might suggest building two of these routers and setting them up as a VRRP router
to ensure that the ""router is alwayse up and useable.  This is much
easier through managed switches too as you don''t have to cable as much
to the physical routers.

In short get memory and a lower end proc to save the money for a 2nd identical
router.



Grant. . . .

Mihai Vlad wrote:
> Hey guys,
> 
> I am planning to buy some components for a Linux router that will handle
the
> Internet access of 200 computers (includes tc shaping) and some inter
> sub-network routing (at least 100MBps per eth - and there are 3 eth cards).
> 
> I was thinking of a:
> Pentium 4 - 3GHz
> 256 or 512MB RAM
> Network Cards.
> 
> Now - I wonder what is more important: the processor speed or the amount of
> RAM.
> 
> And can you point me to some good Network Cards that you have used and are
> not so expensive. Some Intel, etc. I have no clue about this subject...
> 
> 
> Maybe this discussion can be extended to a list of best practices to set up
> a performant Linux Router from the hardware point of view.
> 
> Thanks in advance,
> Mihai
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

On Tue, 16 Aug 2005 11:38:06 -0500 "Taylor, Grant"
<gtaylor@riverviewtech.net>
wrote:
>I ended up
>+allocating 1 GB of RAM to just connection tracking.  In fact you need 1 GB
(or
>+very close to) to be able to track 65535 connections.
You don''t. Maybe that''s conntrack''s default, but you
can set it to a higher
number manually. The required memory is approx 400b per connection (depends on
iptables/kernel compile time options). The rather conservative default (hashsize
= 1/16384th of RAM) is for a generic system. For more info look at
ip_conntrack_core.c

65535 connections need about 25MB in RAM, so before starting iptables, do
modprobe ip_conntrack hashsize=8192
(contrack_max is auto-set to 8*hashsize, this is the recommended relation). In
fact my distro Shurdix automatically sets up larger hashsize than the default,
depending on system memory.

You can change conntrack_max while the module is loaded (sysctl
net.ipv4.netfilter.ip_conntrack_max), but you can''t change the hashsize
this
way. If the relation is other than 1:8, you might experience performance
problems (I don''t know details, this is recommended on various places
on the
net).
>Another problem that you may run in to will be filling your ARP table.  The
>+kernel space ARP table is not very large at all, only like 64 or maybe up
to
255
>+IP MAC pairs.
This is also tunable, per sysctl, somewhere like
net.ipv4.neigh.default.gc_thresh[123]. Unfortunately poorly documented, I had to
look at the source to realize this, and I don''t remember what means
what.
>In short get memory and a lower end proc to save the money for a 2nd
identical
>router.
While a redundant system is indeed a good idea, I recommend making sure the
router is rock stable. This doesn''t necessarily require high-end / fast
hardware, it is recommended to stress test it before going live
(memtest/cpuburn/whatever).

My tip is not to use "primitive" network cards like those based on
rtl8139 which
you require high bandwidth. This has the most noticeable impact on performance.
I have ok experience with 3com''s, I''ve heard intels are even
better.

Yours sincerely,
Peter

-- 
http://www.shurdix.org - Linux distribution for routers and firewalls

Also,
....

On Tue, 16 Aug 2005 11:38:06 -0500 "Taylor, Grant"
<gtaylor@riverviewtech.net>
wrote:
>+If you are not doing much in the way if *VERY* *ADVANCED*
>+firewalling, just basic source and / or destination IP v
>alidation and / or source and / or destination port validation will not need
>+much of a processor.  In fact I''d try it with a 500 MHz to 1 GHz
system, what
>+ever is the most economical that you can get your hands on.
Yes. In fact most cases of "advanced" firewalling only mean that you
have a
stupid fw-design, like hundreds/thousands of rules in one chain :-). Usually can
be optimised by using sub-chains, ipset and/or ipt_ACCOUNT.

Yours sincerely,
Peter

-- 
http://www.shurdix.org - Linux distribution for routers and firewalls

> Yes. In fact most cases of "advanced" firewalling only mean that
you have a
> stupid fw-design, like hundreds/thousands of rules in one chain :-).
Usually can
> be optimised by using sub-chains, ipset and/or ipt_ACCOUNT.
If someone has hundreds of rules in one chain (with out a _*VERY*_ good reason
and even then) they need to be shot on the spot.  For performance reasons such a
chain should be broken out in to a tree of chains an subchains that are jumped
to in an attempt to minimize the number of rules that have to be traversed to
get a match on any given packet.

What I was referring to by advanced firewalling was such things as running
things like "-p udp -s 0.0.0.0/32 -d 255.255.255.255/32 --sport 68 --dport
67 -m addrtype --src-type broadcast -m pkttype --pkt-type broadcast" for
DHCP requests. or complex SSH Brute Force prevention chains / rules, or recent
lists to control what types of traffic will be valid based on what you have sent
or is not valid b/c you have not sent any thing, or should packets with the
reset flag have the ack flat set or not, etc.



Grant. . . .

> You don''t. Maybe that''s conntrack''s default, but
you can set it to a higher
> number manually. The required memory is approx 400b per connection (depends
on
> iptables/kernel compile time options). The rather conservative default
(hashsize
> = 1/16384th of RAM) is for a generic system. For more info look at
> ip_conntrack_core.c
> 
> 65535 connections need about 25MB in RAM, so before starting iptables, do
> modprobe ip_conntrack hashsize=8192
> (contrack_max is auto-set to 8*hashsize, this is the recommended relation).
In
> fact my distro Shurdix automatically sets up larger hashsize than the
default,
> depending on system memory.
Hmm, I did not have much time to solve this problem at the time and
documentation was hard to come by at the time and what I did find was old. 
Alass I was not subscribed to this list to ask for help either.  Note things
have changed sense then.  :)
> While a redundant system is indeed a good idea, I recommend making sure the
> router is rock stable. This doesn''t necessarily require high-end /
fast
> hardware, it is recommended to stress test it before going live
> (memtest/cpuburn/whatever).
> 
> My tip is not to use "primitive" network cards like those based
on rtl8139 which
> you require high bandwidth. This has the most noticeable impact on
performance.
> I have ok experience with 3com''s, I''ve heard intels are
even better.
I would agree to both points.  I have had good luck with the rtl8139s on Cable /
DSL and T1 routers but I would want something better (3C905x cards) for a much
higher bandwidth installation.  The redundant (identical) system is for those
cases where the cleaning crew and / or momma nature and / or Mr Murphy have
their way with your box.  We have all had it happen (or will) in some way or
another at some time.  It is not "if" a box will fail in some way, but
rather "when".  The failure may not be any thing you could prevent.  I
think the stores in Florida this year are a good example of that.



Grant. . . .

Thank you all for your help!

The reason why I asked for your help was because I had the following
problem:

Router:
1.4 GHZ AMD
256 RAM
Realtek 8139 NIC
Slackware
2.6 kernel

No iptables rules just iproute2 routing.
I have 2 sub-networks and the router is configured as a "router on a
stick".
I tested a simple routing from one sub-network to another (FTP transfer)
(the traffic enters and leaves the router using the same NIC) and the
maximum transfer rate is 50 Mbit (instead of 100 Mbit as expected).

I wonder how am I able to route a Gigabit network ... Someone told me that
he solved the problem using FreeBSD as it has a better performance on this
field...

I am bound to Linux as it has a superb implementation of QoS. You have
nothing like this in BSD.

I want to ask you: what is the maximum transfer rate that you can obtain on
your configurations (with and without heavy firewalling / marking / tc).

Thanks again for your input!

Mihai

On Wed, 17 Aug 2005 18:46:35 +0300 "Mihai Vlad"
<mihaivlad@web-profile.net>
wrote:
>The reason why I asked for your help was because I had the following
>problem:
[cut]
>Realtek 8139 NIC
Yes, this is indeed a problem. Cards based on this chipset don''t scale
well. Try
3coms or intels. If you''re short on money, dlink dfe-530tx might do
too.

(I work for none of the mentioned companies :-)).
>Mihai
Yours sincerely,
Peter

-- 
http://www.shurdix.org - Linux distribution for routers and firewalls

> Yes, this is indeed a problem. Cards based on this chipset don''t
scale well. Try
> 3coms or intels. If you''re short on money, dlink dfe-530tx might
do too.
Um, the DFE-530TX+ is a (rebranded) Realtek8139 chipset card so I would expect
that the performance would be comparable to the Realtek cards.  I personally
have not used them but I have heard wonderful things about DEC Tulup cards.  If
you are interested I have some 3Com 905 cards I''d be willing to part
with for a very reasonable price.  (Please email me off the list if you are
interested in the NICs.)



Grant. . . .

Taylor, Grant schrieb:
>>Yes, this is indeed a problem. Cards based on this chipset
don''t scale well. Try
>>3coms or intels. If you''re short on money, dlink dfe-530tx
might do too.
> 
> Um, the DFE-530TX+ is a (rebranded) Realtek8139 chipset card so I would
> expect that the performance would be comparable to the Realtek cards.
> I personally have not used them but I have heard wonderful things about
> DEC Tulip cards.
Some of the DFE-5..TX cards are indeed tulip based and have very good
performance for a reasonable price. Unfortunately sources on the net
disagree which cards have which chipset, so I recommend looking at them
(I bought a new tulip based card here in Germany for 7 Euros). It pays
to request a look at the chipsets of the cheaper cards, they might not
even advertise their chipsets as tulip.


Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/

On Wed, 17 Aug 2005 14:18:47 -0500 "Taylor, Grant"
<gtaylor@riverviewtech.net>
wrote:

>Um, the DFE-530TX+ is a (rebranded) Realtek8139 chipset card so I would
expect
>that the performance would be comparable to the Realtek cards.
I don''t have one here, but I vaguely remember they use a different
kernel module
(8139too vs. via-rhine) so I think it''s a different chipset. I only
mentioned
them because some time in distant past I solved some performance problems by
switching from from 8139 to the dlink, and the dlink costs less than a half of
entry-level 3com.
>Grant. . . .
-- 
http://www.shurdix.org - Linux distribution for routers and firewalls