freebsd stable - Oct 2004 - Packet passing performance study on exotic hardware.

The opportunity presented itelf for me to test packet passing ability
on some fairly exotic hardware.  The motherboard I really wanted to
test not only had separate memory busses for each cpu, but also had
two separate PCI-X busses (one slot each).  To this, I added two
intel pro/1000 gigabit ethernet cards (PCI-X versions).

I had two sets of processors to test: two 246's and two 240's.

The packets in this test are all minimal 64 byte UDP packets.

My first goal was to determine the DDOS stability of FreeBSD 5.3, and
Linux on this hardware.  I was using amd64 binaries for both FreeBSD
and linux.

Right out of the box (with polling), Linux passed 550 kpps (kilo
packets wer second).  Full data rate would be 1.9 mpps.  On linux, the
240 processors passed only 450 kppps (which is somewhat expected).

Right out of the box, FreeBSD 5.3 (with polling) passed about 200
kpps.  net.isr.enable=1 increased that without polling to about 220
kpps (although livelock ensued without polling as packet load
increased).  With excessive tuning, we got FreeBSD 5.3 to pass 270
kpps.  This included polling, nmbclusters, net.isr, and some em
patches.  I can't see where to get more performance.

To compare, we loaded FreeBSD-5.3 ia32 and achieved almost identical
performance.

Then, also to compare, we loaded FreeBSD-4.10 ia32 and it promptly
passed 550 kpps (almost identical to the linux performance) (with
polling).

Some interesting things about 5.3(-BETA4) in this environment:

  - without polling, it definately livelocks.

  - with polling and excessive packets, it doesn't "receive" the
full
    load of packets.  In netstat -w, they show as input "errors"
    although the number of "errors" isn't strictly related to the
    number of dropped packets.  It's just some large number that
    generally increases with the number of dropped packets.

  - With net.isr and not polling, both cpus are used (220 kpps)

  - With net.isr and polling, one cpu is used (270 kpps, one cpu free
    for other tasks)

  - It's worth noting that only FreeBSD 5.3 used two cpus to pass
    packets at any time.  Neither linux nor 4.10 used the other cpu.

  - hz and polling tuning options didn't really change packets passed
    significantly.

During the next week, I will continue testing with full simulated
routing tables, random packets and packets between 350 and 550 bytes
(average ISP out/in packet sizes).  I will add to this report then.
If anyone has tuning advice for FreeBSD 5.3, I'd like to hear it.

Dave.

-- 
===========================================================================|David
Gilbert, Independent Contractor.       | Two things can only be     |
|Mail:       dave@daveg.ca                    |  equal if and only if they |
|http://daveg.ca                              |   are precisely opposite.  |
=========================================================GLO================

David Gilbert wrote:
> The opportunity presented itelf for me to test packet passing ability
> on some fairly exotic hardware.  The motherboard I really wanted to
> test not only had separate memory busses for each cpu, but also had
> two separate PCI-X busses (one slot each).  To this, I added two
> intel pro/1000 gigabit ethernet cards (PCI-X versions).
> 
> I had two sets of processors to test: two 246's and two 240's.
> 
> The packets in this test are all minimal 64 byte UDP packets.
> 
> My first goal was to determine the DDOS stability of FreeBSD 5.3, and
> Linux on this hardware.  I was using amd64 binaries for both FreeBSD
> and linux.
> 
> Right out of the box (with polling), Linux passed 550 kpps (kilo
> packets wer second).  Full data rate would be 1.9 mpps.  On linux, the
> 240 processors passed only 450 kppps (which is somewhat expected).
> 
> Right out of the box, FreeBSD 5.3 (with polling) passed about 200
> kpps.  net.isr.enable=1 increased that without polling to about 220
> kpps (although livelock ensued without polling as packet load
> increased).  With excessive tuning, we got FreeBSD 5.3 to pass 270
> kpps.  This included polling, nmbclusters, net.isr, and some em
> patches.  I can't see where to get more performance.
> 
> To compare, we loaded FreeBSD-5.3 ia32 and achieved almost identical
> performance.
> 
> Then, also to compare, we loaded FreeBSD-4.10 ia32 and it promptly
> passed 550 kpps (almost identical to the linux performance) (with
> polling).
> 
> Some interesting things about 5.3(-BETA4) in this environment:
> 
>   - without polling, it definately livelocks.
> 
>   - with polling and excessive packets, it doesn't "receive"
the full
>     load of packets.  In netstat -w, they show as input "errors"
>     although the number of "errors" isn't strictly related to
the
>     number of dropped packets.  It's just some large number that
>     generally increases with the number of dropped packets.
> 
>   - With net.isr and not polling, both cpus are used (220 kpps)
> 
>   - With net.isr and polling, one cpu is used (270 kpps, one cpu free
>     for other tasks)
> 
>   - It's worth noting that only FreeBSD 5.3 used two cpus to pass
>     packets at any time.  Neither linux nor 4.10 used the other cpu.
> 
>   - hz and polling tuning options didn't really change packets passed
>     significantly.
> 
> During the next week, I will continue testing with full simulated
> routing tables, random packets and packets between 350 and 550 bytes
> (average ISP out/in packet sizes).  I will add to this report then.
> If anyone has tuning advice for FreeBSD 5.3, I'd like to hear it.
> 
> Dave.
> 
Interesting results.  One thing to note is that a severe bug in the 
if_em driver was fixed for BETA7.  The symptoms of this bug include
apparent livelock of the machine during heavy xmit load.  You might
want to update and re-run your tests.

Scott

>>>>> "Scott" == Scott Long <scottl@FreeBSD.org>
writes:
Scott> Interesting results.  One thing to note is that a severe bug in
Scott> the if_em driver was fixed for BETA7.  The symptoms of this bug
Scott> include apparent livelock of the machine during heavy xmit
Scott> load.  You might want to update and re-run your tests.

Sorry.  I should have made it clear that I applied the patches to the
em from the tree by hand.

Dave.

-- 
===========================================================================|David
Gilbert, Independent Contractor.       | Two things can only be     |
|Mail:       dave@daveg.ca                    |  equal if and only if they |
|http://daveg.ca                              |   are precisely opposite.  |
=========================================================GLO================

David Gilbert wrote:
>The opportunity presented itelf for me to test packet passing ability
>on some fairly exotic hardware.  The motherboard I really wanted to
>test not only had separate memory busses for each cpu, but also had
>two separate PCI-X busses (one slot each).  To this, I added two
>intel pro/1000 gigabit ethernet cards (PCI-X versions).
>
>I had two sets of processors to test: two 246's and two 240's.
>
>The packets in this test are all minimal 64 byte UDP packets.
>
>My first goal was to determine the DDOS stability of FreeBSD 5.3, and
>Linux on this hardware.  I was using amd64 binaries for both FreeBSD
>and linux.
>
>Right out of the box (with polling), Linux passed 550 kpps (kilo
>packets wer second).  Full data rate would be 1.9 mpps.  On linux, the
>240 processors passed only 450 kppps (which is somewhat expected).
>
>Right out of the box, FreeBSD 5.3 (with polling) passed about 200
>kpps.  net.isr.enable=1 increased that without polling to about 220
>kpps (although livelock ensued without polling as packet load
>increased).  With excessive tuning, we got FreeBSD 5.3 to pass 270
>kpps.  This included polling, nmbclusters, net.isr, and some em
>patches.  I can't see where to get more performance.
>
>To compare, we loaded FreeBSD-5.3 ia32 and achieved almost identical
>performance.
>
>Then, also to compare, we loaded FreeBSD-4.10 ia32 and it promptly
>passed 550 kpps (almost identical to the linux performance) (with
>polling).
>
>Some interesting things about 5.3(-BETA4) in this environment:
>
>  - without polling, it definately livelocks.
>
>  - with polling and excessive packets, it doesn't "receive"
the full
>    load of packets.  In netstat -w, they show as input "errors"
>    although the number of "errors" isn't strictly related to
the
>    number of dropped packets.  It's just some large number that
>    generally increases with the number of dropped packets.
>
>  
>
Have you used "sysctl hw.em0.stats=1" and/or "sysctl
hw.em1.stats=1"
before and after running the test to obtain snapshots of the detailed 
error statistics (they're logged by the kernel to /var/log/messages)?  
Perhaps those would be enlightening.

The fixed bug in the em driver for BETA7 may significantly help (see 
Scott Long's response prior to mine).

If you try BETA7 without polling but with SMP, do you get better results 
if you increase hw.em0.rx_int_delay and hw.em1.rx_int_delay above 0?

Have you set sysctls kern.random.sys.harvest.ethernet=0 and 
kern.random.sys.harvest.interrupt=0?

I don't know if it will have any effect in your situation, but have you 
increased net.inet.ip.intr_queue_maxlen?

Hope this helps,
Guy

-- 
Guy Helmer, Ph.D., Principal System Architect, Palisade Systems, Inc.
ghelmer@palisadesys.com
http://www.palisadesys.com/~ghelmer

David Gilbert wrote:
> During the next week, I will continue testing with full simulated
> routing tables, random packets and packets between 350 and 550 bytes
> (average ISP out/in packet sizes).  I will add to this report then.
> If anyone has tuning advice for FreeBSD 5.3, I'd like to hear it.
Three things:

  sysctl net.inet.ip.fastforwarding=1

  Don't use SMP for packet forwarding.  It doesn't help anything and
  introduces only locking overhead.

  Upgrade to the latest RELENG_5, there are a couple of fixes for things
  that may hurt you here.  Especially there is a fix for the transmit
  queues on the em() driver.

-- 
Andre

At 10:08 AM 08/10/2004, David Gilbert wrote:
>Right out of the box, FreeBSD 5.3 (with polling) passed about 200
>kpps.  net.isr.enable=1 increased that without polling to about 220
Did you have kern.polling.idle_poll at 0 or 1 ? In my tests a few weeks ago 
this seemed to make a difference, but the load avg gets messed up.  Also, 
HZ does seem to make a difference at least in my tests on BETA5.

         ---Mike

David Gilbert wrote:
> Right out of the box, FreeBSD 5.3 (with polling) passed about 200
> kpps.
Was this with debug.mpsafenet enabled and all debugging (WITNESS and such)
turned off?

/Daniel Eriksson