freebsd stable - Sep 2016 - 11.0 stuck on high network load

On Thu, Sep 15, 2016 at 12:06:33PM +0300, Slawa Olhovchenkov wrote:
> On Thu, Sep 15, 2016 at 11:59:38AM +0300, Konstantin Belousov wrote:
> 
> > On Thu, Sep 15, 2016 at 12:35:04AM +0300, Slawa Olhovchenkov wrote:
> > > On Sun, Sep 04, 2016 at 06:46:12PM -0700, hiren panchasara wrote:
> > > 
> > > > On 09/05/16 at 12:57P, Slawa Olhovchenkov wrote:
> > > > > I am try using 11.0 on Dual E5-2620 (no X2APIC).
> > > > > Under high network load and may be addtional
conditional system go to
> > > > > unresponsible state -- no reaction to network and
console (USB IPMI
> > > > > emulation). INVARIANTS give to high overhad. Is this
exist some way to
> > > > > debug this?
> > > > 
> > > > Can you panic it from console to get to db> to get
backtrace and other
> > > > info when it goes unresponsive?
> > > 
> > > ipmi console don't respond (chassis power diag don't
react)
> > > login on sol console stuck on *tcp.
> > 
> > Is 'login' you reference is the ipmi client state, or you mean
login(1)
> > on the wedged host ?
> 
> on the wedged host
> 
> > If BMC stops responding simultaneously with the host, I would suspect
> > the hardware platform issues instead of a software problem.  Do you
have
> > dedicated LAN port for BMC ?
> 
> Yes.
> But BMC emulate USB keyboard and this is may be lock inside USB
> system.
> "ipmi console don't respond" must be read as "ipmi
console runnnig and
> attached but system don't react to keypress on this console".
> at the sime moment system respon to `enter` on ipmi sol console, but
> after enter `root` stuck in login in the '*tcp' state (I think this
is
> NIS related).
~^B don't break to debuger.
But I can login to sol console.
All system work tooooo slooooow.
Some cores displayed by top as 100% idle.
Some cores displayed by top as 100% interrupt.

last pid: 16631;  load averages:  9.00,  9.08,  9.35    up 0+06:26:34  19:53:13
832 processes: 22 running, 765 sleeping, 42 waiting, 3 lock
CPU 0:   0.0% user,  0.0% nice,  0.0% system,  100% interrupt,  0.0% idle
CPU 1:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 2:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 3:   0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 4:   0.0% user,  0.0% nice,  0.0% system,  100% interrupt,  0.0% idle
CPU 5:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 6:   0.0% user,  0.0% nice,  0.0% system,  100% interrupt,  0.0% idle
CPU 7:   0.0% user,  0.0% nice,  0.0% system,  100% interrupt,  0.0% idle
CPU 8:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 9:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 10:  0.0% user,  0.0% nice,  0.0% system,  100% interrupt,  0.0% idle
CPU 11:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 1564M Active, 5169M Inact, 114G Wired, 4548M Free
ARC: 107G Total, 98G MFU, 9476M MRU, 160K Anon, 150M Header, 254M Other
Swap: 32G Total, 32G Free

Now I am trying to

# pmcstat -c 0 -S CPU_CLK_UNHALTED_CORE -l 10 -O sample0.out

load: 10.32  cmd: pmcstat 16878 [runnable] 4632.42r 0.00u 0.00s 0% 2940k

I am still waiting

17.09.2016 1:18, Slawa Olhovchenkov ?????:
> ~^B don't break to debuger.
Make sure your kernel config has:

# Solaris implements a new BREAK which is initiated by a character
# sequence CR ~ ^b which is similar to a familiar pattern used on
# Sun servers by the Remote Console.  There are FreeBSD extensions:
# CR ~ ^p requests force panic and CR ~ ^r requests a clean reboot.
options         ALT_BREAK_TO_DEBUGGER

And don't forget that tilde (~) should follow CR (Enter).

On 09/16/16 at 09:18P, Slawa Olhovchenkov wrote:
> On Thu, Sep 15, 2016 at 12:06:33PM +0300, Slawa Olhovchenkov wrote:
> 
> > On Thu, Sep 15, 2016 at 11:59:38AM +0300, Konstantin Belousov wrote:
> > 
> > > On Thu, Sep 15, 2016 at 12:35:04AM +0300, Slawa Olhovchenkov
wrote:
> > > > On Sun, Sep 04, 2016 at 06:46:12PM -0700, hiren panchasara
wrote:
> > > > 
> > > > > On 09/05/16 at 12:57P, Slawa Olhovchenkov wrote:
> > > > > > I am try using 11.0 on Dual E5-2620 (no X2APIC).
> > > > > > Under high network load and may be addtional
conditional system go to
> > > > > > unresponsible state -- no reaction to network and
console (USB IPMI
> > > > > > emulation). INVARIANTS give to high overhad. Is
this exist some way to
> > > > > > debug this?
> > > > > 
> > > > > Can you panic it from console to get to db> to get
backtrace and other
> > > > > info when it goes unresponsive?
> > > > 
> > > > ipmi console don't respond (chassis power diag don't
react)
> > > > login on sol console stuck on *tcp.
> > > 
> > > Is 'login' you reference is the ipmi client state, or you
mean login(1)
> > > on the wedged host ?
> > 
> > on the wedged host
> > 
> > > If BMC stops responding simultaneously with the host, I would
suspect
> > > the hardware platform issues instead of a software problem.  Do
you have
> > > dedicated LAN port for BMC ?
> > 
> > Yes.
> > But BMC emulate USB keyboard and this is may be lock inside USB
> > system.
> > "ipmi console don't respond" must be read as "ipmi
console runnnig and
> > attached but system don't react to keypress on this console".
> > at the sime moment system respon to `enter` on ipmi sol console, but
> > after enter `root` stuck in login in the '*tcp' state (I think
this is
> > NIS related).
> 
> ~^B don't break to debuger.
> But I can login to sol console.
You can probably:
debug.kdb.enter: set to enter the debugger

or force a panic and get vmcore:
debug.kdb.panic: set to panic the kernel

Cheers,
Hiren
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 603 bytes
Desc: not available
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160916/31c263f2/attachment.sig>