Hey -STABLE, I've got a client who we've setup a FreeBSD cluster for with about a dozens servers, all behind two front end proxies/LBs/firewalls which also act as NAT gateways for the internal servers. On the active front end proxy we've started seeing "fatal: socket: No buffer space available" errors during high-peak times. I can see in vmstat -z that this is what is getting denied: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP tcp_inpcb: 392, 32770, 19398, 13372,1449734621,6312858, 0 We've got a lot of the other values bumped, and it appears to be this input limit that is getting hit. There are no other non-zero FAILed counters except 64 and 128 buckets which I believe are normal. I cannot seem to find the sysctl (or equiv) that controls this limit though, or even what it is. Anyone know? I'm obviously in need of this specific answer, but overall is there a codex of vmstat -z's items that explains this that I have just not found in my searches? This isn't the first time I've had to dig into a value like this to increase it's limit, but this time I'm not turning anything up. Any thoughts/ideas appreciated! -- Adam Strohl http://www.ateamsystems.com/
Check the output of 'netstat -mb', maybe you're also running out of mbufs? Adrian On 30 October 2012 06:21, Adam Strohl <adams-freebsd at ateamsystems.com> wrote:> Hey -STABLE, > > I've got a client who we've setup a FreeBSD cluster for with about a dozens > servers, all behind two front end proxies/LBs/firewalls which also act as > NAT gateways for the internal servers. > > On the active front end proxy we've started seeing "fatal: socket: No buffer > space available" errors during high-peak times. I can see in vmstat -z > that this is what is getting denied: > > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP > tcp_inpcb: 392, 32770, 19398, 13372,1449734621,6312858, 0 > > We've got a lot of the other values bumped, and it appears to be this input > limit that is getting hit. There are no other non-zero FAILed counters > except 64 and 128 buckets which I believe are normal. > > I cannot seem to find the sysctl (or equiv) that controls this limit though, > or even what it is. Anyone know? > > I'm obviously in need of this specific answer, but overall is there a codex > of vmstat -z's items that explains this that I have just not found in my > searches? This isn't the first time I've had to dig into a value like this > to increase it's limit, but this time I'm not turning anything up. > > Any thoughts/ideas appreciated! > > -- > Adam Strohl > http://www.ateamsystems.com/ > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
On 10/30/12 06:21, Adam Strohl wrote:> Hey -STABLE, > > I've got a client who we've setup a FreeBSD cluster for with about a > dozens servers, all behind two front end proxies/LBs/firewalls which > also act as NAT gateways for the internal servers. > > On the active front end proxy we've started seeing "fatal: socket: No > buffer space available" errors during high-peak times. I can see in > vmstat -z that this is what is getting denied: > > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP > tcp_inpcb: 392, 32770, 19398, 13372,1449734621,6312858, 0 > > We've got a lot of the other values bumped, and it appears to be this > input limit that is getting hit. There are no other non-zero FAILed > counters except 64 and 128 buckets which I believe are normal. > > I cannot seem to find the sysctl (or equiv) that controls this limit > though, or even what it is. Anyone know?kern.ipc.maxsockets controls this limit. See in_pcbinfo_init() for details. Regards, Navdeep