freebsd stable - Oct 2006 - Userland freezes during heavy packet forwarding workloads

Hello!

We (pfSense developers) have noticed an interesting problem where
userland stops functioning under high packet forwarding workloads.
Userland applications such as sshd and lighttpd freeze but userland
resumes after the network load eases.

We tested with both PF enabled and disabled and the problem remains in
either case so chances are that its not a firewalling / pfil issue.

Machines tested so far:

PC Engines WRAP 266 mhz, 3 sis nics
Soekris Engineering 4801 266 mhz, 3 sis nics
AMD K6-2 450MHZ, 384 MB ram, 5 XL nics (3c905c)

FreeBSD versions tested and are affected:
 FreeBSD 6.1-RELEASE as of October 10th, 2006
 FreeBSD 6.2-PRERELEASE as of October 10th, 2006

FreeBSD versions tested and are not affected:
 FreeBSD 4.11-RELEASE

Kernel configuration file:
[http://cvs.pfsense.com/cgi-bin/cvsweb.cgi/tools/builder_scripts/conf/pfSense_wrap.6?rev=1.36;content-type=text%2Fplain]

Copy of /etc/sysctl.conf in use:
[http://www.pfsense.com/~sullrich/logs/sysctl.conf]

Is there anything we can do to prevent userland from stopping
completely when under heavy load?  Is this a bug?

Unfortunately, I cannot reproduce this behavior in FreeBSD 4.X.

Thanks in advance!

Scott

On Oct 12, 2006, at 2:11 PM, Scott Ullrich wrote:
> We (pfSense developers) have noticed an interesting problem where
> userland stops functioning under high packet forwarding workloads.
> Userland applications such as sshd and lighttpd freeze but userland
> resumes after the network load eases.
[ ...note that the links you posted included the trailing square  
bracket,
      and thus were broken, although easily fixable by hand... ]
> Is there anything we can do to prevent userland from stopping
> completely when under heavy load?  Is this a bug?
You're probably experiencing some form of "livelock"; your
hardware
isn't horribly fast, and if the NICs are generating interrupts  
rapidly enough due to the high rate of packet forwarding, you're not  
going to have a lot of spare CPU available to run userland tasks.

I notice you're compiling in support for DEVICE_POLLING; does  
enabling it do anything to help the responsiveness of the userland  
tasks under high network load?  You might try adjusting HZ from 100  
to 250 or so...I found that made a decent tradeoff between packet  
delay and scheduler overhead on similar Soerkis 4801 or VIA C3/EPIA-M  
hardware to yours.

-- 
-Chuck