I'm trying to tune 5.x for maximum router performance, in terms of
packets per second.
I'm starting to hit errors at 150Kpps.
input (em0) output
packets errs bytes packets errs bytes colls
1 0 66 1 0 178 0
75751 3391 5124070 1 0 298 0
150150 0 9609602 1 0 178 0
150150 19837 10809670 1 0 186 0
150150 0 9609602 1 0 178 0
150150 19840 10809466 1 0 186 0
150150 0 9609602 1 0 178 0
150150 19842 10809466 1 0 186 0
146029 0 9345858 1 0 178 0
141518 32596 10206142 1 0 186 0
Set-up:
RELENG_5 as of Apr 03 21:33
* P4 3.0GHz / 2GB DDR400 RAM
* Supermicro P4SCi with 2 on-board Intel gige
* Additional 2-port Intel gige on 64-bit PCI-X
With this many interfaces I have disabled unnecessary devices in the
BIOS such as USB to reduce IRQ sharing.
APIC also seems to do quite some renumbering of IRQs, no idea if this
is a problem or not.
Kernel config additions over GENERIC:
options DEVICE_POLLING #network device mixed interrupt-polling handling
options HZ=1000 #for polling
options TCP_SIGNATURE #TCP MD5 signatures
options FAST_IPSEC #IPSec needed for TCP MD5
options IPFILTER #ipfilter support
options IPFILTER_LOG #ipfilter logging
device crypto # crypto needed for TCP MD5
device vlan # 802.1q ethernet VLAN tagging
device carp # common address resolution protocol
/etc/sysctl.conf:
kern.polling.enable=1
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1
Empty ipfilter ruleset.
[ Full dmesg and kernel config etc at http://www.rackred.com/temp/ ]
My test set-up is to use a Linux based click-router with udpgen on
one side, and a Linux based click-router with udpcount on the other
side.
[ SRC host ] ---- [em0 router under test em3] ---- [ DST host ]
So traffic test is passing data through the router.
Information gleaned from the archives suggests tweaking sysctl's
such as net.isr.enable and disabling harvest options but these
haven't really made any noticeable difference for me.
Does anyone have any suggestions?
I'm sure I have read of other people getting better performance than
this.
-Paul-
--
Paul Civati <paul(at)racksense.com>
Rack Sense Ltd - Managed/Business hosting - www.racksense.com
RackRed - Value SSL certificates and servers - www.rackred.com
> I'm starting to hit errors at 150Kpps.I'm not sure why you're hitting errors at that speed. But here are a few suggestions:> options DEVICE_POLLING #network device mixed > interrupt-polling handling > options HZ=1000 #for pollingThat's too low. 2000 is the minimum you should consider. See below for how to tune this in /boot/loader.conf.> /etc/sysctl.conf: > kern.polling.enable=1 > net.inet.ip.forwarding=1 > net.inet.ip.fastforwarding=1You probably want: kern.random.sys.harvest.ethernet=0 kern.random.sys.hervest.interrupt=0> Does anyone have any suggestions?In /boot/loader.conf, consider: kern.ipc.nmbclusters="262144" kern.hz="2000" vm.kmem_size="536870912" vm.kmem_size_max="536870912" What are your current values for vm.kmem_size and vm.kmem_size_max? DS
Hi, Paul. Do sysctl kern.random.sys.harvest.ethernet=0 sysctl kern.random.sys.harvest.interrupt=0 affect to your router's performance? and, disabling interrupt moderation of em(4) affects the router too? Please see the previous discussion of interrupt moderation at freebsd-current. http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1279567+0+archive/2004/freebsd-current/20041121.freebsd-current http://docs.freebsd.org/cgi/getmsg.cgi?fetch=215083+0+archive/2004/freebsd-current/20041128.freebsd-current http://www.freebsd.org/cgi/query-pr.cgi?pr=74986 -- Shunsuke SHINOMIYA <shino@fornext.org>
> RELENG_5 as of Apr 03 21:33 > > * P4 3.0GHz / 2GB DDR400 RAM > * Supermicro P4SCi with 2 on-board Intel gige > * Additional 2-port Intel gige on 64-bit PCI-X > > With this many interfaces I have disabled unnecessary devices in the > BIOS such as USB to reduce IRQ sharing.Interesting. I'm using 4.11-RELEASE on a P4 3.0G / 1GB DDR400, on a SuperMicro P4SC8 (which is basically the P4SCi with SCSI). No additional 2-port card. It's a fairly similar setup. No problems with high traffic rates. (For the purposes of this discussion, all traffic except ssh to the router is traffic /thru/ the router). input (Total) output packets errs bytes packets errs bytes colls 107439 0 72162014 144813 0 142032833 0 101704 0 68046409 135931 0 128359873 0 104645 0 65768204 141990 0 129334101 0 112468 0 75986846 150024 0 144615912 0 106497 0 72030483 142558 0 141768086 0 99765 0 70045449 133374 0 132466526 0 96067 0 63798208 125844 0 125487982 0 101886 0 70674519 134926 0 134179234 0 103760 0 69877737 135941 0 137523002 0 112911 0 79450478 154102 0 151303406 0 116146 0 77471891 156882 0 152534206 0 The apparent high output rate is due to vlan devices, where the output is attributed to both vlan* and em1 (dedicated to vlans). The input and the output are actually both approximately the input numbers. We had noticed at one point that we were starting to see em1 report drops which were in turn being reported as errs by the vlan devices. We were pushing right near a gigabit of traffic at the time, so I had written it off as just pushing the router to the edge. But let's re-examine that. Now, if I start pumping more PPS, I do indeed see errs: input (Total) output packets errs bytes packets errs bytes colls drops 97267 0 66675536 122086 0 131168641 0 0 104247 0 75413955 130001 0 142561178 0 0 95617 0 64612833 121289 0 127119179 0 0 253471 0 77582829 397208 10000 139671386 0 10000 289454 0 71304600 483657 9443 129348488 0 9443 288084 0 71948183 486654 8725 135213379 0 8725 287875 0 68385123 491482 7460 123001416 0 7460 290068 0 72583882 487009 10140 135995424 0 10140 283690 0 66749355 489184 5460 120351717 0 5460 287656 0 71174008 494372 6546 135130910 0 6546 294036 0 73220341 485374 12531 131770769 0 12531 292414 0 74153414 489359 10575 139045664 0 10575 293152 0 72187276 485446 11945 129875438 0 11945 288516 0 73740228 484401 8261 138960063 0 8261 286625 0 70340602 476896 11569 126852274 0 11569 226024 0 70356111 374375 3985 135568639 0 3985 97423 0 67559069 123148 0 131424813 0 0 99617 0 70842045 126110 0 134722149 0 0 109440 0 75528823 139232 0 148738405 0 0 99994 0 69608728 128823 0 130991145 0 0 This is running a little "udpblast" program that sends min size packets in addition to the production traffic load on the router. I find this interesting, because the aggregate traffic through the router is clearly not anywhere near a gigabit. So it does appear that there is some sort of inadvertent cap on PPS here. Now, a few notes. 1) I notice that /your/ errors are showing as input errors. Mine seem to show up as output errors. 2) We need to remember that the design of the P4SC{8,i} is a bit crappy, in that the onboard ports consist of one CSA port (no problem here) and one PCI port - which Supermicro wisely placed on the 32-bit, 33 MHz legacy PCI bus. This could potentially limit the throughput on that port. 3) The driver tunable EM_TIDV looks quite promising as a first guess as to what to tune for /my/ issue. 4) I think playing with EM_MAX_TXD and EM_MAX_RXD may be useful to both you and I as well. 5) Don't know about FreeBSD 5. You're on your own there. :-) ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.