Hi, I wonder about the performance of a Linux box used as router (I guest I''m not the first :). Althought I know it mainly depends on the hardware, I''m trying to find some references on the topic or comparations with other routing solutions (FreeBSD box used as router, Cisco, etc). For example, http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf (althought is related with Linux-briding more than with Linux-routing) shows in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps can be achieved. Anybody knows any other similar analysis, please? Best regards, -------------------- Fermín Galán Márquez CTTC - Centre Tecnològic de Telecomunicacions de Catalunya Parc Mediterrani de la Tecnologia, Av. del Canal Olímpic s/n, 08860 Castelldefels, Spain Room 1.02 Tel : +34 93 645 29 12 Fax : +34 93 645 29 01 Email address: fermin.galan@cttc.es
Hi, Maybe: Khan, Sohel; Waheed, Abdul (2003): High Performance Routing on PCshttp://www.ccse.kfupm.edu.sa/~sohel/networking/references/Routing.pdf A rule of thumb: - with current COTS hardware and (standard) PCI Bus, you can reach the maximum of the PCI bus bandwidth. That''s 1 GB/s, e.h. two NICs with 500 Meg/s each ( one in and one out ) - with PCI-X and in the future PCI-express you''ll for sure be able to reach more performance. I didnt find a sponsor for a test-lab yet :) - in DoS secnarios it may get worse :/ I heavily depends on driver type (polling and NAPI preferred). The problem with the performace is _always_ the number of interrupts, nothing else is a bottleneck (well, we didn''t talk about thousands of iptables rules yet, but you ask for a ''maximum''). - The question you have to ask in high-performance scenarios is not "MBit/s" but MPPS (megapackets per seconds). FreeBSD and Linux broke the 1 MPPS barrier some time ago (on dual xeons). rgds, Andreas Fermín Galán Márquez wrote:> Hi, > > I wonder about the performance of a Linux box used as router (I guest I''m > not the first :). Althought I know it mainly depends on the hardware, I''m > trying to find some references on the topic or comparations with other > routing solutions (FreeBSD box used as router, Cisco, etc). For example, > http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf > (althought is related with Linux-briding more than with Linux-routing) shows > in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps > can be achieved. > > Anybody knows any other similar analysis, please? > > Best regards, > > -------------------- > Fermín Galán Márquez > CTTC - Centre Tecnològic de Telecomunicacions de Catalunya > Parc Mediterrani de la Tecnologia, Av. del Canal Olímpic s/n, 08860 > Castelldefels, Spain > Room 1.02 > Tel : +34 93 645 29 12 > Fax : +34 93 645 29 01 > Email address: fermin.galan@cttc.es > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >
Fermín Galán Márquez wrote:> Hi, > > I wonder about the performance of a Linux box used as router (I guest I''m > not the first :). Althought I know it mainly depends on the hardware, I''m > trying to find some references on the topic or comparations with other > routing solutions (FreeBSD box used as router, Cisco, etc). For example, > http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf > (althought is related with Linux-briding more than with Linux-routing) shows > in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps > can be achieved.On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express gigabit cards (but that was with 1500 byte packets). Never tried more although the box has 6 interfaces capable of gigabit, 4 of them attached via PCI-Express. Regards, Carl-Daniel -- http://www.hailfinger.org/
> > I wonder about the performance of a Linux box used as router (I guest I''m > > not the first :). Althought I know it mainly depends on the hardware, I''m > > trying to find some references on the topic or comparations with other > > routing solutions (FreeBSD box used as router, Cisco, etc). For example, > > http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf > > (althought is related with Linux-briding more than with Linux-routing) shows > > in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps > > can be achieved. > > On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express > gigabit cards (but that was with 1500 byte packets). Never tried more > although the box has 6 interfaces capable of gigabit, 4 of them attached > via PCI-Express.But that''s _only_ 83333 packets/s isn''t it. -- damjan | дамјан This is my jabber ID --> damjan@bagra.net.mk -- not my mail address, it''s a Jabber ID --^ :)
Damjan wrote:>>> I wonder about the performance of a Linux box used as router (I guest I''m >>> not the first :). Althought I know it mainly depends on the hardware, I''m >>> trying to find some references on the topic or comparations with other >>> routing solutions (FreeBSD box used as router, Cisco, etc). For example, >>> http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf >>> (althought is related with Linux-briding more than with Linux-routing) shows >>> in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps >>> can be achieved. >> On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express >> gigabit cards (but that was with 1500 byte packets). Never tried more >> although the box has 6 interfaces capable of gigabit, 4 of them attached >> via PCI-Express. > > But that''s _only_ 83333 packets/s isn''t it.Hm. How do you arrive at that result? I get twice the numbers. nic a: 1 gbit in -> nic b: 1 gbit out nic b: 1 gbit in -> nic a: 1 gbit out total 2 gbit 2 gbit /(1500*8 bit/frame) ~ 160k packets/s Please note that I did not test with smaller frame sizes, so 1Mp/s may be possible (I''ll test that if I have some spare time). Regards, Carl-Daniel -- http://www.hailfinger.org/
On Thu, Jun 01, 2006 at 02:44:57AM +0200, Carl-Daniel Hailfinger wrote:> Damjan wrote: > >>> I wonder about the performance of a Linux box used as router (I guest I''m > >>> not the first :). Althought I know it mainly depends on the hardware, I''m > >>> trying to find some references on the topic or comparations with other > >>> routing solutions (FreeBSD box used as router, Cisco, etc). For example, > >>> http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf > >>> (althought is related with Linux-briding more than with Linux-routing) shows > >>> in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps > >>> can be achieved. > >> On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express > >> gigabit cards (but that was with 1500 byte packets). Never tried more > >> although the box has 6 interfaces capable of gigabit, 4 of them attached > >> via PCI-Express. > > > > But that''s _only_ 83333 packets/s isn''t it. > > Hm. How do you arrive at that result? I get twice the numbers. > nic a: 1 gbit in -> nic b: 1 gbit out > nic b: 1 gbit in -> nic a: 1 gbit out > total 2 gbit > 2 gbit /(1500*8 bit/frame) ~ 160k packets/s > > Please note that I did not test with smaller frame sizes, so 1Mp/s > may be possible (I''ll test that if I have some spare time).what if you test inbound and outbound at the same time - the cards should be capable of full duplex ?> > > Regards, > Carl-Daniel > -- > http://www.hailfinger.org/ > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >_______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Alexander Samad wrote:> On Thu, Jun 01, 2006 at 02:44:57AM +0200, Carl-Daniel Hailfinger wrote: >> Damjan wrote: >>>>> I wonder about the performance of a Linux box used as router (I guest I''m >>>>> not the first :). Althought I know it mainly depends on the hardware, I''m >>>>> trying to find some references on the topic or comparations with other >>>>> routing solutions (FreeBSD box used as router, Cisco, etc). For example, >>>>> http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf >>>>> (althought is related with Linux-briding more than with Linux-routing) shows >>>>> in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps >>>>> can be achieved. >>>> On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express >>>> gigabit cards (but that was with 1500 byte packets). Never tried more >>>> although the box has 6 interfaces capable of gigabit, 4 of them attached >>>> via PCI-Express. >>> But that''s _only_ 83333 packets/s isn''t it. >> Hm. How do you arrive at that result? I get twice the numbers. >> nic a: 1 gbit in -> nic b: 1 gbit out >> nic b: 1 gbit in -> nic a: 1 gbit out >> total 2 gbit >> 2 gbit /(1500*8 bit/frame) ~ 160k packets/s >> >> Please note that I did not test with smaller frame sizes, so 1Mp/s >> may be possible (I''ll test that if I have some spare time). > > what if you test inbound and outbound at the same time - the cards > should be capable of full duplex ?I tested 1 gbit in and 1 gbit out per nic at the same time. That''s how I arrived at my results. Regards, Carl-Daniel -- http://www.hailfinger.org/
On Thu, Jun 01, 2006 at 04:03:29AM +0200, Carl-Daniel Hailfinger wrote:> Alexander Samad wrote: > > On Thu, Jun 01, 2006 at 02:44:57AM +0200, Carl-Daniel Hailfinger wrote: > >> Damjan wrote: > >>>>> I wonder about the performance of a Linux box used as router (I guest I''m > >>>>> not the first :). Althought I know it mainly depends on the hardware, I''m > >>>>> trying to find some references on the topic or comparations with other > >>>>> routing solutions (FreeBSD box used as router, Cisco, etc). For example, > >>>>> http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf > >>>>> (althought is related with Linux-briding more than with Linux-routing) shows > >>>>> in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps > >>>>> can be achieved. > >>>> On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express > >>>> gigabit cards (but that was with 1500 byte packets). Never tried more > >>>> although the box has 6 interfaces capable of gigabit, 4 of them attached > >>>> via PCI-Express. > >>> But that''s _only_ 83333 packets/s isn''t it. > >> Hm. How do you arrive at that result? I get twice the numbers. > >> nic a: 1 gbit in -> nic b: 1 gbit out > >> nic b: 1 gbit in -> nic a: 1 gbit out > >> total 2 gbit > >> 2 gbit /(1500*8 bit/frame) ~ 160k packets/s > >> > >> Please note that I did not test with smaller frame sizes, so 1Mp/s > >> may be possible (I''ll test that if I have some spare time). > > > > what if you test inbound and outbound at the same time - the cards > > should be capable of full duplex ? > > I tested 1 gbit in and 1 gbit out per nic at the same time. That''s > how I arrived at my results.sorry I might be being very dense on this, but 2 nics 1G in and out shouldn''t that be 4gbit / (1500*8 bit/frame) ~ 320k packets/s My presumption is that the nic can send and recieve at the same time> > Regards, > Carl-Daniel > -- > http://www.hailfinger.org/ > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >_______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Alexander Samad wrote:> On Thu, Jun 01, 2006 at 04:03:29AM +0200, Carl-Daniel Hailfinger wrote: >> Alexander Samad wrote: >>> On Thu, Jun 01, 2006 at 02:44:57AM +0200, Carl-Daniel Hailfinger wrote: >>>> Damjan wrote: >>>>>>> I wonder about the performance of a Linux box used as router (I guest I''m >>>>>>> not the first :). Althought I know it mainly depends on the hardware, I''m >>>>>>> trying to find some references on the topic or comparations with other >>>>>>> routing solutions (FreeBSD box used as router, Cisco, etc). For example, >>>>>>> http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf >>>>>>> (althought is related with Linux-briding more than with Linux-routing) shows >>>>>>> in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps >>>>>>> can be achieved. >>>>>> On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express >>>>>> gigabit cards (but that was with 1500 byte packets). Never tried more >>>>>> although the box has 6 interfaces capable of gigabit, 4 of them attached >>>>>> via PCI-Express. >>>>> But that''s _only_ 83333 packets/s isn''t it. >>>> Hm. How do you arrive at that result? I get twice the numbers. >>>> nic a: 1 gbit in -> nic b: 1 gbit out >>>> nic b: 1 gbit in -> nic a: 1 gbit out >>>> total 2 gbit >>>> 2 gbit /(1500*8 bit/frame) ~ 160k packets/s >>>> >>>> Please note that I did not test with smaller frame sizes, so 1Mp/s >>>> may be possible (I''ll test that if I have some spare time). >>> what if you test inbound and outbound at the same time - the cards >>> should be capable of full duplex ? >> I tested 1 gbit in and 1 gbit out per nic at the same time. That''s >> how I arrived at my results. > sorry I might be being very dense on this, but 2 nics 1G in and out > shouldn''t that be > 4gbit / (1500*8 bit/frame) ~ 320k packets/sNo, because you can count each packet passing through the router only once. If the machine works as a router, each entering packet also has to leave, so if the router has 2 interfaces A+B, you can have 1 Gbit from A to B and 1 Gbit from B to A. Your calculation would be correct if the machine is a server and generates and consumes all traffic locally.> My presumption is that the nic can send and recieve at the same timeYes. Regards, Carl-Daniel -- http://www.hailfinger.org/
x>On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express gigabit cards (but that was with 1500 byte packets). Never tried more although the box has 6 interfaces capable of gigabit, >4 of them attached via PCI-Express. What NIC''s are you using? Are they multiport or do you have several pci-express single port cards? Andy Registered Office: J.O. Sims Ltd, Pudding Lane, Pinchbeck, Spalding, Lincs. PE11 3TJ Company reg No: 2084187 Vat reg No: GB 437 4621 47 Tel: +44 (0) 1775 842100 Fax: +44 (0) 1775 842101 Web: www.josims.com Email:enquiries@josims.com The information contained in this e-mail is confidential and is intended for the addressee only. The contents of this e-mail must not be disclosed or copied without the sender''s consent. If you are not the intended recipient of the message, please notify the sender immediately, and delete the message. The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the company. No commitment may be inferred from the contents unless explicitly stated. The company does not take any responsibility for the personal views of the author. This message has been scanned for viruses before sending, but the company does not accept any responsibility for infection and recommends that you scan any attachments.JOSEDV001TAG
On Wed, 2006-05-31 at 22:27 +0200, Andreas John wrote:> Hi, > > Maybe: > Khan, Sohel; Waheed, Abdul (2003): High Performance Routing on > PCshttp://www.ccse.kfupm.edu.sa/~sohel/networking/references/Routing.pdf > > A rule of thumb: > - with current COTS hardware and (standard) PCI Bus, you can reach the > maximum of the PCI bus bandwidth. That''s 1 GB/s, e.h. two NICs with 500 > Meg/s each ( one in and one out ) > - with PCI-X and in the future PCI-express you''ll for sure be able to > reach more performance. I didnt find a sponsor for a test-lab yet :) > - in DoS secnarios it may get worse :/ I heavily depends on driver type > (polling and NAPI preferred).ofcouse prefered. Does it exsist a list of driver/nic combos that are know to support NAPI on linux on stock kernels ? -- Ronny Aasen <list@datapart-as.no>
Andrew Lyon wrote:> >On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express > >gigabit cards (but that was with 1500 byte packets). Never tried more > >although the box has 6 interfaces capable of gigabit, >4 of them attached > >via PCI-Express. > > What NIC''s are you using? Are they multiport or do you have several > pci-express single port cards?Single-Port SK-9E21D with sky2 driver version 0.13a. I''m going to retry with SK-9E82 dual port cards soon. Regards, Carl-Daniel -- http://www.hailfinger.org/
Hi I''m sure that Robert can provide us with some interesting numbers. I have just tested routing performance on a AMD opteron 270 (dual core), here I can route 400 kpps (tg3 netcards on PCI-X). I use the kernel module "pktgen" to generate the packets (64 bytes in size). Cheers, Jesper Brouer -- ------------------------------------------------------------------- MSc. Master of Computer Science Dept. of Computer Science, University of Copenhagen Author of http://www.adsl-optimizer.dk ------------------------------------------------------------------- On Wed, 31 May 2006, Andreas John wrote:> Hi, > > Maybe: > Khan, Sohel; Waheed, Abdul (2003): High Performance Routing on > PCshttp://www.ccse.kfupm.edu.sa/~sohel/networking/references/Routing.pdf > > A rule of thumb: > - with current COTS hardware and (standard) PCI Bus, you can reach the > maximum of the PCI bus bandwidth. That''s 1 GB/s, e.h. two NICs with 500 > Meg/s each ( one in and one out ) > - with PCI-X and in the future PCI-express you''ll for sure be able to > reach more performance. I didnt find a sponsor for a test-lab yet :) > - in DoS secnarios it may get worse :/ I heavily depends on driver type > (polling and NAPI preferred). The problem with the performace is > _always_ the number of interrupts, nothing else is a bottleneck (well, > we didn''t talk about thousands of iptables rules yet, but you ask for a > ''maximum''). > - The question you have to ask in high-performance scenarios is not > "MBit/s" but MPPS (megapackets per seconds). FreeBSD and Linux broke the > 1 MPPS barrier some time ago (on dual xeons). > > rgds, > Andreas > > Fermín Galán Márquez wrote: >> Hi, >> >> I wonder about the performance of a Linux box used as router (I guest I''m >> not the first :). Althought I know it mainly depends on the hardware, I''m >> trying to find some references on the topic or comparations with other >> routing solutions (FreeBSD box used as router, Cisco, etc). For example, >> http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf >> (althought is related with Linux-briding more than with Linux-routing) shows >> in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps >> can be achieved. >> >> Anybody knows any other similar analysis, please? >> >> Best regards, >> >> -------------------- >> Fermín Galán Márquez >> CTTC - Centre Tecnològic de Telecomunicacions de Catalunya >> Parc Mediterrani de la Tecnologia, Av. del Canal Olímpic s/n, 08860 >> Castelldefels, Spain >> Room 1.02 >> Tel : +34 93 645 29 12 >> Fax : +34 93 645 29 01 >> Email address: fermin.galan@cttc.es_______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Damjan wrote:>>> On an AMD Athlon64 3200+ (2 GHz) I was able to saturate 2 PCI-Express >>> gigabit cards (but that was with 1500 byte packets). Never tried more >>> although the box has 6 interfaces capable of gigabit, 4 of them attached >>> via PCI-Express. >> >> But that''s _only_ 83333 packets/s isn''t it. > >Hm. How do you arrive at that result? I get twice the numbers. >nic a: 1 gbit in -> nic b: 1 gbit out >nic b: 1 gbit in -> nic a: 1 gbit out >total 2 gbit >2 gbit /(1500*8 bit/frame) ~ 160k packets/s > >Please note that I did not test with smaller frame sizes, so 1Mp/s >may be possible (I''ll test that if I have some spare time).I''ve done some benchmarks on a Sunfire x2100 with 2 port PCI Express ethernet cards. It switches 800KPPS for 64B packets. Regards Mohan
Fermín Galán Márquez skrev:> Hi, > > I wonder about the performance of a Linux box used as router (I guest I''m > not the first :). Althought I know it mainly depends on the hardware, I''m > trying to find some references on the topic or comparations with other > routing solutions (FreeBSD box used as router, Cisco, etc). For example, > http://facweb.cti.depaul.edu/jyu/Publications/Yu-Linux-TSM2004.pdf > (althought is related with Linux-briding more than with Linux-routing) shows > in Figure 14 that with an AMD Duron 1.3GHz 512M RAM a throughput of 90 Mbps > can be achieved. > > Anybody knows any other similar analysis, please? > > Best regards, > > -------------------- > Fermín Galán Márquez > CTTC - Centre Tecnològic de Telecomunicacions de Catalunya > Parc Mediterrani de la Tecnologia, Av. del Canal Olímpic s/n, 08860 > Castelldefels, Spain > Room 1.02 > Tel : +34 93 645 29 12 > Fax : +34 93 645 29 01 > Email address: fermin.galan@cttc.es > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >This was seen on the mailing list a couple of years ago, doesnt say much but it shows what could be done. On Mon, 02 Dec 2002 22:30:10 +0100 Anton Tinchev <atl@unixsol.org> wrote:> > Hi, > > first i wonna thank you for the great work. > > I have few slack boxes with several 3com cards that acts as routers. > > Some of them has 50+ vlans, 100 000+ routing entries, full BGP (zebra) with 10+ peers > > and routes 50-70 mb/s traffic. Everithing is rock solid, few months uptimes. >Sounds pretty impressive, really. I admire such setups.> > I wona to upgrade some of my cards and need advice what to use. > > On 100+mb/s interrups killing my boxes - 20 000+/s (yes, coalescing, i know:)) > > What to use? tigon2 or tigon3 for gigabit? (3c985 or 3c996) >None of them! Or at least not tigon3! I''ve tried to use one (3c996-T), and I experienced strange system lockups. The board is a dual Tyan Tiger MP with couple of Athlon MP 1600+. It was just hanging from time to time with completely no output of any kind. Just rock solid lockup. :/ Anyway, I changed to a good old 3c905C and now I don''t have any problems. Well, I''m serving at half of your rate, but anyway. So, I would suggest using HP equipment. At least I''ve heard that it works quote well.