freebsd stable - Jan 2015 - sonewconn: pcb 0x???: Listen queue overflow [Was: Re: lagg(8) causes ghost queue with igb(4)]

Bez?glich Harald Schmalzbauer's Nachricht vom 16.01.2015 20:12
(localtime):
>  Hi all,
>
> while investigating a watchdog timeout problem (on FreeBSD-10.1-stable
> r276295) I noticed "ghost" queues consuming interrupts with
> igb(4)[82576] if igb(4) is member of lagg(4) (laggproto loadbalance
> lagghash l2). Mysteriously only sometimes (1 of 2 runs).
>
> Sending machine has only one ssh session open, where I do the following:
> 'dd if=/dev/zero | nc vegashare 3333' (of course on vegashare is a
> listener [nc -l vegashare 3333 > /dev/null])
>
> I'm transfering 123.5*10^6 Bytes/sec and on the sender, systat reports:
> 2682 igb0:que 0 (1 of 4 queues causes moderate irq load hw.igb.aim=1,
> otherhise it were 16k irqs/s, nothing on the other 3 queues)
>
> But roughly every second run, I see a another queue consuming much more
> irqs/s while transferring exactly the same at exactly the same speed:
> 7740 igb0:que 0
> 2640 igb0:que 3
>
> Here's again one queue with 2.6k irqs/s, but another one with 7.7k
> irqs/s which I can't understand what this is doing. It's useless
for
> sure, because with only one queue I get the same payload transported on
> the same hardware with the same speed and 3 queues idle?
I noticed another mysterium, at least for me. I'd highly appreciate if
somebody could give me a hint how I can understand the following lines

sonewconn: pcb 0xfffff801b85907a8: Listen queue overflow: 151 already in
queue awaiting acceptance (1 occurrences)
sonewconn: pcb 0xfffff801b85907a8: Listen queue overflow: 151 already in
queue awaiting acceptance (220 occurrences)

I have never seen them before and I guess it's related to my other queue
mysterium.
They occured not during tests, so not during artificial (high) load, but
regular low avarage load.

Thanks in advance,

-Harry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20150118/cfa520ae/attachment.sig>

Bez?glich Harald Schmalzbauer's Nachricht vom 18.01.2015 13:11
(localtime):
>  Bez?glich Harald Schmalzbauer's Nachricht vom 16.01.2015 20:12
> (localtime):
>>  Hi all,
>>
>> while investigating a watchdog timeout problem (on FreeBSD-10.1-stable
>> r276295) I noticed "ghost" queues consuming interrupts with
>> igb(4)[82576] if igb(4) is member of lagg(4) (laggproto loadbalance
>> lagghash l2). Mysteriously only sometimes (1 of 2 runs).
>>
>> Sending machine has only one ssh session open, where I do the
following:
>> 'dd if=/dev/zero | nc vegashare 3333' (of course on vegashare
is a
>> listener [nc -l vegashare 3333 > /dev/null])
>>
>> I'm transfering 123.5*10^6 Bytes/sec and on the sender, systat
reports:
>> 2682 igb0:que 0 (1 of 4 queues causes moderate irq load hw.igb.aim=1,
>> otherhise it were 16k irqs/s, nothing on the other 3 queues)
>>
>> But roughly every second run, I see a another queue consuming much more
>> irqs/s while transferring exactly the same at exactly the same speed:
>> 7740 igb0:que 0
>> 2640 igb0:que 3
>>
>> Here's again one queue with 2.6k irqs/s, but another one with 7.7k
>> irqs/s which I can't understand what this is doing. It's
useless for
>> sure, because with only one queue I get the same payload transported on
>> the same hardware with the same speed and 3 queues idle?
> I noticed another mysterium, at least for me. I'd highly appreciate if
> somebody could give me a hint how I can understand the following lines
>
> sonewconn: pcb 0xfffff801b85907a8: Listen queue overflow: 151 already in
> queue awaiting acceptance (1 occurrences)
> sonewconn: pcb 0xfffff801b85907a8: Listen queue overflow: 151 already in
> queue awaiting acceptance (220 occurrences)
>
> I have never seen them before and I guess it's related to my other
queue
> mysterium.
Sorry, I should have asked someone to type that into a internet search
portal for me before posting here ;-)
The pcb address is a unix domain socket, so completely unrelated to my
igb(4) queue mysterium!

Please excuse, thanks,

-Harry


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20150118/3833bf07/attachment.sig>

Hi!
> I noticed another mysterium, at least for me. I'd highly appreciate if
> somebody could give me a hint how I can understand the following lines
> 
> sonewconn: pcb 0xfffff801b85907a8: Listen queue overflow: 151 already in
> queue awaiting acceptance (1 occurrences)
You have a socket open in the LISTEN state. Incoming connections
must be accept()'ed. If the process that LISTENs is too busy
to accept, the so-called 'listen-queue' is growing.

I have found

kern.ipc.soacceptqueue: 128

and I assume that this is the max. number of incoming connections waiting
for accept before the next incoming connection will cause that error
(and is not successful any more).

I do not know how to match the pcb (protocol control block) back
to the process/LISTEN in question, but maybe someone else knows this ?

-- 
pi at opsec.eu            +49 171 3101372                         5 years to go
!