freebsd stable - Aug 2017 - mlx4en, timer irq @100%... (11.0 stuck on high network load ???)

Hi,

On 8/8/17 10:31 AM, Hans Petter Selasky wrote:
> On 08/08/17 10:06, Ben RUBSON wrote:
>>> On 08 Aug 2017, at 10:02, Hans Petter Selasky <hps at
selasky.org> wrote:
>>>
>>> On 08/08/17 10:00, Ben RUBSON wrote:
>>>> kgdb) print *twq_2msl.tqh_first
>>>> $2 = {
>>>>    tw_inpcb = 0xfffff8031c570740,
>>>
>>> print *twq_2msl.tqh_first->tw_inpcb
>>
> 
> Here is the conclusion:
> 
> The following code is going in an infinite loop:
> 
> 
>>         for (;;) {
>>                 TW_RLOCK(V_tw_lock);
>>                 tw = TAILQ_FIRST(&V_twq_2msl);
>>                 if (tw == NULL || (!reuse && (tw->tw_time -
ticks) >
>> 0)) {
>>                         TW_RUNLOCK(V_tw_lock);
>>                         break;
>>                 }
>>                 KASSERT(tw->tw_inpcb != NULL, ("%s:
tw->tw_inpcb =>> NULL",
>>                     __func__));
>>
>>                 inp = tw->tw_inpcb;
>>                 in_pcbref(inp);
>>                 TW_RUNLOCK(V_tw_lock);
>>
>>                 if (INP_INFO_TRY_RLOCK(&V_tcbinfo)) {
>>
>>                         INP_WLOCK(inp);
>>                         tw = intotw(inp);
>>                         if (in_pcbrele_wlocked(inp)) {
> 
> in_pcbrele_wlocked() returns (1) because INP_FREED (16) is set in
> inp->inp_flags2. I guess you have invariants disabled, because the
> KASSERT() below should have caused a panic.
> 
>>                                 KASSERT(tw == NULL, ("%s: held
last inp "
>>                                     "reference but tw not
NULL",
>> __func__));
>>                                 INP_INFO_RUNLOCK(&V_tcbinfo);
>>                                 continue;
>>                         }
> 
> This is a regression issue after:
> 
>> commit 5630210a7f1dbbd903b77b2aef939cd47c63da58
>> Author: jch <jch at FreeBSD.org>
>> Date:   Thu Oct 30 08:53:56 2014 +0000
>>
>>     Fix a race condition in TCP timewait between tcp_tw_2msl_reuse()
and
>>     tcp_tw_2msl_scan().  This race condition drives unplanned timewait
>>     timeout cancellation.  Also simplify implementation by holding
inpcb
>>     reference and removing tcptw reference counting.
> 
> Suggested fix attached.
 I agree we your conclusion.  Just for the record, more precisely this
regression seems to have been introduced with:

commit b02d40ddcda08b51a49e5667e6808f5dc5ec0472
Author: fabient <fabient at FreeBSD.org>
Date:   Wed Nov 25 14:45:43 2015 +0000

    The r241129 description was wrong that the scenario is possible
    only for read locks on pcbs. The same race can happen with write
    lock semantics as well.

    The race scenario:

    - Two threads (1 and 2) locate pcb with writer semantics
(INPLOOKUP_WLOCKPCB)
     and do in_pcbref() on it.
    - 1 and 2 both drop the inp hash lock.
    - Another thread (3) grabs the inp hash lock. Then it runs in_pcbfree(),
     which wlocks the pcb. They must happen faster than 1 or 2 come
INP_WLOCK()!
    - 1 and 2 congest in INP_WLOCK().
    - 3 does in_pcbremlists(), drops hash lock, and runs
in_pcbrele_wlocked(),
     which doesn't free the pcb due to two references on it.
     Then it unlocks the pcb.
    - 1 (or 2) gets wlock on the pcb, runs in_pcbrele_wlocked(), which
doesn't
     report inp as freed, due to 2 (or 1) still helding extra reference
on it.
     The thread tries to do smth with a disconnected pcb and crashes.

    Submitted by:   emeric.poupon at stormshield.eu
    Reviewed by:    gleb@
    MFC after:      1 week
    Sponsored by: Stormshield
    Tested by: Cassiano Peixoto, Stormshield

Notes:
    svn path=/head/; revision=291301

 Before this change in_pcbrele_wlocked() returned 1 only on the last
reference which is what tcp_tw_2msl_scan() currently expects.

 Thus good catch, and your patch looks good.  I am going to just verify
the other in_pcbrele_wlocked() calls in TCP stack.

 Thanks.

--
Julien

> On 08 Aug 2017, at 13:33, Julien Charbon <jch at freebsd.org> wrote:
> 
> Hi,
> 
> On 8/8/17 10:31 AM, Hans Petter Selasky wrote:
>> 
>> 
>> Suggested fix attached.
> 
> I agree we your conclusion.  Just for the record, more precisely this
> regression seems to have been introduced with:
> (...)
> Thus good catch, and your patch looks good.  I am going to just verify
> the other in_pcbrele_wlocked() calls in TCP stack.
Julien, do you plan to make this fix reach 11.0-p12 ?

Many thx !

Ben