freebsd stable - Mar 2016 - kernel: mps0: Out of chain frames, consider increasing hw.mps.max_chains.

> On Mar 6, 2016, at 10:04 PM, Slawa Olhovchenkov <slw at zxy.spb.ru>
wrote:
> 
> On Sun, Mar 06, 2016 at 06:20:06PM -0800, Scott Long wrote:
> 
>> 
>>> On Mar 6, 2016, at 1:27 PM, Slawa Olhovchenkov <slw at
zxy.spb.ru> wrote:
>>> 
>>> On Sun, Mar 06, 2016 at 01:10:42PM -0800, Scott Long wrote:
>>> 
>>>> Hi,
>>>> 
>>>> The message is harmless, it's a reminder that you should
tune the kernel for your workload.  When the message is triggered, it means that
a potential command was deferred, likely for only a few microseconds, and then
everything moved on as normal.
>>>> 
>>>> A command uses anywhere from 0 to a few dozen chain frames per
I/o, depending on the size of the io.  The chain frame memory is allocated at
boot so that it's always available, not allocated on the fly.  When I wrote
this driver, I felt that it would be wasteful to reserve memory for a worst case
scenario of all large io's by default, so I put in this deferral system with
a console reminder to for tuning.
>>>> 
>>>> Yes, you actually do have 900 io's outstanding.  The
controller buffers the io requests and allows the system to queue up much more
than what sata disks might allow on their own.  It's debatable if this is
good or bad, but it's tunable as well.
>>>> 
>>>> Anyways, the messages should not cause alarm.  Either tune up
the chain frame count, or tune down the max io count.
>>> 
>>> I am don't know depends or not, but I see dramaticaly
performance drop
>>> at time of this messages.
>>> 
>> 
>> Good to know.  Part of the performance drop might be because of the
slowness of printing to the console.
> 
> no, on console print may be one per minute
> 
The one-per-minute prints are by design.  I should probably make it print once
and then increment a sysctl counter.
>>> How I can calculate buffers numbers?
>> 
>> If your system is new enough to have mpsutil, please run it ?mpsutil
>> show iocfacts?.
> 
> As I see mpsutil present only on -HEAD.
> Can I compile it on 10-STABLE?
> 
Yes, I believe it should compile on 10, but I have not tried it recently.
>> If not, then boot your system with bootverbose and send me the output.
> 
> I can do this day ago.
> 
>>> I am have very heavy I/O.
>> 
>> Out of curiosity, do you redefine MAXPHYS/DFLTPHYS in your kernel
config?
> 
> no
> 
>>> This allocated one for all controllers, or allocated for every
controller?
>> 
>> It?s per-controller.
>> 
>> I?ve thought about making the tuning be dynamic at runtime.  I
>> implemented similar dynamic tuning for other drivers, but it seemed
>> overly complex for low benefit.  Implementing it for this driver
>> would be possible but require some significant code changes.
> 
> What cause of chain_free+io_cmds_active << max_chains?
> One cmd can use many chains?
Yes.  A request uses and active command, and depending on the size of the I/O,
it might use several chain frames.

Scott

On Sun, Mar 06, 2016 at 11:40:55PM -0800, Scott Long wrote:
> 
> > On Mar 6, 2016, at 10:04 PM, Slawa Olhovchenkov <slw at
zxy.spb.ru> wrote:
> > 
> > On Sun, Mar 06, 2016 at 06:20:06PM -0800, Scott Long wrote:
> > 
> >> 
> >>> On Mar 6, 2016, at 1:27 PM, Slawa Olhovchenkov <slw at
zxy.spb.ru> wrote:
> >>> 
> >>> On Sun, Mar 06, 2016 at 01:10:42PM -0800, Scott Long wrote:
> >>> 
> >>>> Hi,
> >>>> 
> >>>> The message is harmless, it's a reminder that you
should tune the kernel for your workload.  When the message is triggered, it
means that a potential command was deferred, likely for only a few microseconds,
and then everything moved on as normal.
> >>>> 
> >>>> A command uses anywhere from 0 to a few dozen chain frames
per I/o, depending on the size of the io.  The chain frame memory is allocated
at boot so that it's always available, not allocated on the fly.  When I
wrote this driver, I felt that it would be wasteful to reserve memory for a
worst case scenario of all large io's by default, so I put in this deferral
system with a console reminder to for tuning.
> >>>> 
> >>>> Yes, you actually do have 900 io's outstanding.  The
controller buffers the io requests and allows the system to queue up much more
than what sata disks might allow on their own.  It's debatable if this is
good or bad, but it's tunable as well.
> >>>> 
> >>>> Anyways, the messages should not cause alarm.  Either tune
up the chain frame count, or tune down the max io count.
> >>> 
> >>> I am don't know depends or not, but I see dramaticaly
performance drop
> >>> at time of this messages.
> >>> 
> >> 
> >> Good to know.  Part of the performance drop might be because of
the slowness of printing to the console.
> > 
> > no, on console print may be one per minute
> > 
> 
> The one-per-minute prints are by design.  I should probably make it print
once and then increment a sysctl counter.
I.e. this is can't be cause slowness of printing to the console?
> >>> How I can calculate buffers numbers?
> >> 
> >> If your system is new enough to have mpsutil, please run it
?mpsutil
> >> show iocfacts?.
> > 
> > As I see mpsutil present only on -HEAD.
> > Can I compile it on 10-STABLE?
> > 
> 
> Yes, I believe it should compile on 10, but I have not tried it recently.
# mpsutil show iocfacts
       MaxChainDepth: 128
             WhoInit: 0x4
       NumberOfPorts: 1
      MaxMSIxVectors: 0
       RequestCredit: 1720
           ProductID: 0x2713
     IOCCapabilities: 0x185c
           FWVersion: 0x0f000000
 IOCRequestFrameSize: 32
       MaxInitiators: 1
          MaxTargets: 256
     MaxSasExpanders: 11
       MaxEnclosures: 12
       ProtocolFlags: 0x2
  HighPriorityCredit: 116
MaxRepDescPostQDepth: 65504
      ReplyFrameSize: 32
          MaxVolumes: 2
        MaxDevHandle: 286
MaxPersistentEntries: 128
        MinDevHandle: 9
> >> If not, then boot your system with bootverbose and send me the
output.
> > 
> > I can do this day ago.
> > 
> >>> I am have very heavy I/O.
> >> 
> >> Out of curiosity, do you redefine MAXPHYS/DFLTPHYS in your kernel
config?
> > 
> > no
> > 
> >>> This allocated one for all controllers, or allocated for every
controller?
> >> 
> >> It?s per-controller.
> >> 
> >> I?ve thought about making the tuning be dynamic at runtime.  I
> >> implemented similar dynamic tuning for other drivers, but it
seemed
> >> overly complex for low benefit.  Implementing it for this driver
> >> would be possible but require some significant code changes.
> > 
> > What cause of chain_free+io_cmds_active << max_chains?
> > One cmd can use many chains?
> 
> Yes.  A request uses and active command, and depending on the size of the
I/O,
> it might use several chain frames.
> 
> Scott
>