freebsd stable - Oct 2016 - 11.0 stuck on high network load

On 10/13/16 5:17 PM, Slawa Olhovchenkov wrote:
> On Thu, Oct 13, 2016 at 05:06:00PM +0200, Julien Charbon wrote:
> 
>>>> will give you that trace in the core, and without INVARIANT
then it is
>>>> better to use dtrace:
>>>>
>>>> $ cat tcp-twstart-dropped.d
>>>> fbt::tcp_twstart:entry
>>>> /args[0]->t_inpcb->inp_flags & 0x04000000/
>>>> {
>>>>   stack();
>>>>   printf("INP_DROPPED in tcp_twstart: %x",
args[0]->t_inpcb->inp_flags);
>>>> }
>>>
>>> Same code may be insert there too, IMHO.
>>
>>  Hmm, I don't think so:
>>
>>  - If you have INVARIANT, the kernel will panic in tcp_twstart() or
>> tcp_detach() and you will have everything you need to debug.
>>  - If you don't, dtrace is the right tool to use in all cases
anyway.
> 
> dtrace don't executed in may case w/ diagnostic "dtrace:
processing
> aborted: Abort due to systemic unresponsiveness". This is for
> tcp_close. May be tcp_twstart will be more successuful, may be not.
 It does and will.
> Also, using dtrace too complex in production (need complex startup
> under screen and capture output) and for many peoples.
> kdb_backtrace() have too less administrative overhead.
 I still think it is overkill.  The main goal of this change is to fix a
quite tricky and old TCP stack locking issue.  Let's try to do that
first, it is complex enough by itself.

 Once the fix is validated and pushed, feel free to propose your own
patch/review to add kdb_backtrace(), log(), etc.. to get other devs
point of view.

 I don't remember who said: "Never ever optimize error cases"...

--
Julien

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20161013/350f0616/attachment.sig>

On Thu, Oct 13, 2016 at 06:14:29PM +0200, Julien Charbon wrote:
> On 10/13/16 5:17 PM, Slawa Olhovchenkov wrote:
> > On Thu, Oct 13, 2016 at 05:06:00PM +0200, Julien Charbon wrote:
> > 
> >>>> will give you that trace in the core, and without
INVARIANT then it is
> >>>> better to use dtrace:
> >>>>
> >>>> $ cat tcp-twstart-dropped.d
> >>>> fbt::tcp_twstart:entry
> >>>> /args[0]->t_inpcb->inp_flags & 0x04000000/
> >>>> {
> >>>>   stack();
> >>>>   printf("INP_DROPPED in tcp_twstart: %x",
args[0]->t_inpcb->inp_flags);
> >>>> }
> >>>
> >>> Same code may be insert there too, IMHO.
> >>
> >>  Hmm, I don't think so:
> >>
> >>  - If you have INVARIANT, the kernel will panic in tcp_twstart()
or
> >> tcp_detach() and you will have everything you need to debug.
> >>  - If you don't, dtrace is the right tool to use in all cases
anyway.
> > 
> > dtrace don't executed in may case w/ diagnostic "dtrace:
processing
> > aborted: Abort due to systemic unresponsiveness". This is for
> > tcp_close. May be tcp_twstart will be more successuful, may be not.
> 
>  It does and will.
> 
> > Also, using dtrace too complex in production (need complex startup
> > under screen and capture output) and for many peoples.
> > kdb_backtrace() have too less administrative overhead.
> 
>  I still think it is overkill.  The main goal of this change is to fix a
> quite tricky and old TCP stack locking issue.  Let's try to do that
> first, it is complex enough by itself.
> 
>  Once the fix is validated and pushed, feel free to propose your own
> patch/review to add kdb_backtrace(), log(), etc.. to get other devs
> point of view.
> 
>  I don't remember who said: "Never ever optimize error
cases"...
This is not optimeze error cases, this is error recovery and
diagnostic of error cases in other subsystems.

Currently FreeBSD internals too complex for just always trust on
correct of other subsystem or do panic on any incosystency.

INVARIANTS too expensive now (20Gbit drops to 8Gbits).

PS: I am applay patch. Wait till monday.

Thanks very match for this hard work!