freebsd stable - Mar 2016 - kernel: mps0: Out of chain frames, consider increasing hw.mps.max_chains.

On Tue, Mar 08, 2016 at 10:34:23AM -0800, Scott Long wrote:
> 
> > On Mar 8, 2016, at 10:07 AM, Slawa Olhovchenkov <slw at
zxy.spb.ru> wrote:
> > 
> > On Mon, Mar 07, 2016 at 02:10:12PM +0300, Slawa Olhovchenkov wrote:
> > 
> >>>>>> This allocated one for all controllers, or
allocated for every controller?
> >>>>> 
> >>>>> It?s per-controller.
> >>>>> 
> >>>>> I?ve thought about making the tuning be dynamic at
runtime.  I
> >>>>> implemented similar dynamic tuning for other drivers,
but it seemed
> >>>>> overly complex for low benefit.  Implementing it for
this driver
> >>>>> would be possible but require some significant code
changes.
> >>>> 
> >>>> What cause of chain_free+io_cmds_active <<
max_chains?
> >>>> One cmd can use many chains?
> >>> 
> >>> Yes.  A request uses and active command, and depending on the
size of the I/O,
> >>> it might use several chain frames.
> > 
> > I am play with max_chains and like significant cost of handling
> > max_chains: with 8192 system resonded badly vs 2048. Now try 3192,
> > response like with 2048.
> 
> Hi, I?m not sure I understand what you?re saying.  You said that you tried
8192, but the system still complained of being out of chain frames?  Now you are
trying fewer, only 3192?
With 8192 system not complained of being out of chain frames, but like
need more CPU power to handle this chain list -- traffic graf (this
host servered HTTP by nginx) have many "jerking", with 3192 traffic
graf is more smooth.

> On Mar 8, 2016, at 10:48 AM, Slawa Olhovchenkov <slw at zxy.spb.ru>
wrote:
> 
> On Tue, Mar 08, 2016 at 10:34:23AM -0800, Scott Long wrote:
> 
>> 
>>> On Mar 8, 2016, at 10:07 AM, Slawa Olhovchenkov <slw at
zxy.spb.ru> wrote:
>>> 
>>> On Mon, Mar 07, 2016 at 02:10:12PM +0300, Slawa Olhovchenkov wrote:
>>> 
>>>>>>>> This allocated one for all controllers, or
allocated for every controller?
>>>>>>> 
>>>>>>> It?s per-controller.
>>>>>>> 
>>>>>>> I?ve thought about making the tuning be dynamic at
runtime.  I
>>>>>>> implemented similar dynamic tuning for other
drivers, but it seemed
>>>>>>> overly complex for low benefit.  Implementing it
for this driver
>>>>>>> would be possible but require some significant code
changes.
>>>>>> 
>>>>>> What cause of chain_free+io_cmds_active <<
max_chains?
>>>>>> One cmd can use many chains?
>>>>> 
>>>>> Yes.  A request uses and active command, and depending on
the size of the I/O,
>>>>> it might use several chain frames.
>>> 
>>> I am play with max_chains and like significant cost of handling
>>> max_chains: with 8192 system resonded badly vs 2048. Now try 3192,
>>> response like with 2048.
>> 
>> Hi, I?m not sure I understand what you?re saying.  You said that you
tried 8192, but the system still complained of being out of chain frames?  Now
you are trying fewer, only 3192?
> 
> With 8192 system not complained of being out of chain frames, but like
> need more CPU power to handle this chain list -- traffic graf (this
> host servered HTTP by nginx) have many "jerking", with 3192
traffic
> graf is more smooth.
Hi,

The CPU overhead of doing more chain frames is nil.  They are just objects in a
list, and processing the list is O(1), not O(n).  What you are likely seeing is
other problems with VM and VFS-BIO system struggling to deal with the amount of
I/O that you are doing.  Depending on what kind I/O you are doing (buffered
filesystem reads/writes, memory mapped I/O, unbuffered I/O) there are limits and
high/low water marks on how much I/O can be outstanding, and when the limits are
reached processes are put to sleep and then race back in when they are woken up.
This causes poor, oscillating system behavior.  There?s some tuning you can do
to increase the limits, but yes, it?s a problem that behaves poorly in an
untuned system.

Scott