Folks,
My box is Dell PowerEdge 4600 with:
aac0 Dell PERC 3/Di with 10x 70Gb HDDs,
em0 82544EI,
bge0,
fxp0,
6GB RAM of which only 4GB being used,
This is a Usenet server running INN 2.4.1 (release) connected to
Cisco GSR 12416 with Gigabit Ethernet em0; physical connection is
1000SX Full-Duplex.
Yesterday I upgraded my news server from 5.1 to 5.2.1 and ran into
performance problem.
Want to say from the beginning that during upgrade I transferred all
my sysctl and other settings from 5.1 into 5.2.1
The box has about 40 NNTP peers (more that 300 TCP connections) and
traffic flow must be at least 160mbit/100mbit in/out repectively.
However after upgrade these number dropped down t/80mbit (input/output
respectively), which is not normal for a Usenet server. I tried to do
some FTP with nearby servers - along with news traffic it gave something
like 380kbytes/sec transfer rate, so total output traffic doesn't exceed
100mbit/sec.
I noticed that box started getting input errors on em0.
In order to fix that I increased EM_MAX_TXD from 256 to 1024 and
EM_MAX_RXD from 256 to 2048.
After that Input errors disappeared but I continued getting intr_queue_drops
counter increasing.
I also observe full Send-Q for many TCP connections.
I increased intr_queue_maxlen several times, from 400 up to 1200 with 200
increments, but it's still increasing.
I changed kern.hz from 100 to 1000; also played with hw.em0.{tx,rx}_int_delay -
neither one helped.
Here are my modified sysctl values:
kern.ipc.maxsockbuf=524288
net.inet.ip.intr_queue_maxlen=1200
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=131072
net.inet.tcp.tcbhashsize=16384
net.inet.tcp.rfc1323=1
net.isr.enable=1
net.inet.tcp.path_mtu_discovery=0
net.inet.tcp.delayed_ack=0
net.inet.tcp.newreno=0
net.inet6.ip6.accept_rtadv=1
Also from loader.conf:
kern.ipc.nmbclusters=64512
kern.ipc.nmbufs=129024
net.inet.tcp.tcbhashsize=16384
kern.maxdsiz=1342177280 # deprecated in 5.2.1
if_em_load="YES" # made this driver as a module to play
with tunables in if_em.h
kern.hz="1000"
My friend from OpenBSD core team advised to try to increase IFQ_MAXLEN from 50
to 500,
but I have a feeling it's not gonna help.
If anybody has any ideas or suggestions, all are welcome and very appreciated.
--
Thanks and Regards,
Igor.
If TCP/IP handshaking was less formal,
perhaps SYN/ACK would be YO!/SUP! instead...