freebsd stable - Feb 2012 - mpslsi0 : Trying sleep, but thread marked as sleeping prohibited

Hi,

I am doing some code changes in mps dirver. While working on those changes, I
come to know about something which is new to me.
Some expert help is required to clarify my doubt.

1. When any irq is register with FreeBSD OS, it sets " TDP_NOSLEEPING"
pflag. It means though irq in freebsd is treated as thread,
We cannot sleep in IRQ because of " "TDP_NOSLEEPING " set.
2. In mps driver we have below code snippet in ISR routine.


    mps_dprint(sc, MPS_TRACE, "%s\n", __func__);
    mps_lock(sc);
    mps_intr_locked(data);
    mps_unlock(sc);

I wonder why there is no issue with above code ? Theoretical we cannot sleep in
ISR. (as explained in #1)
Any thoughts ?


3. I recently added few place msleep() instead of DELAY in ISR context and I see
" Trying sleep, but thread marked as sleeping prohibited".


` Kashyap

Adding Ed Schouten and Jorg Wunsch  as I see there are author of msleep/mtx
related APIs.
> -----Original Message-----
> From: Desai, Kashyap
> Sent: Wednesday, February 22, 2012 7:37 PM
> To: freebsd-scsi@freebsd.org; freebsd-stable
> Cc: 'Kenneth D. Merry'; McConnell, Stephen; Justin T. Gibbs
> Subject: mpslsi0 : Trying sleep, but thread marked as sleeping
> prohibited
> 
> Hi,
> 
> I am doing some code changes in mps dirver. While working on those
> changes, I come to know about something which is new to me.
> Some expert help is required to clarify my doubt.
> 
> 1. When any irq is register with FreeBSD OS, it sets "
TDP_NOSLEEPING"
> pflag. It means though irq in freebsd is treated as thread,
> We cannot sleep in IRQ because of " "TDP_NOSLEEPING " set.
> 2. In mps driver we have below code snippet in ISR routine.
> 
> 
>     mps_dprint(sc, MPS_TRACE, "%s\n", __func__);
>     mps_lock(sc);
>     mps_intr_locked(data);
>     mps_unlock(sc);
> 
> I wonder why there is no issue with above code ? Theoretical we cannot
> sleep in ISR. (as explained in #1)
> Any thoughts ?
> 
> 
> 3. I recently added few place msleep() instead of DELAY in ISR context
> and I see
> " Trying sleep, but thread marked as sleeping prohibited".
> 
> 
> ` Kashyap

On Wed, Feb 22, 2012 at 07:36:42PM +0530, Desai, Kashyap
wrote:
> Hi,
> 
> I am doing some code changes in mps dirver. While working on those changes,
I come to know about something which is new to me.
> Some expert help is required to clarify my doubt.
> 
> 1. When any irq is register with FreeBSD OS, it sets "
TDP_NOSLEEPING" pflag. It means though irq in freebsd is treated as thread,
> We cannot sleep in IRQ because of " "TDP_NOSLEEPING " set.
> 2. In mps driver we have below code snippet in ISR routine.
> 
> 
>     mps_dprint(sc, MPS_TRACE, "%s\n", __func__);
>     mps_lock(sc);
>     mps_intr_locked(data);
>     mps_unlock(sc);
> 
> I wonder why there is no issue with above code ? Theoretical we cannot
sleep in ISR. (as explained in #1)
> Any thoughts ?
> 
> 
> 3. I recently added few place msleep() instead of DELAY in ISR context and
I see
> " Trying sleep, but thread marked as sleeping prohibited".
> 
FreeBSD has several basic ways to prevent a thread from executing on CPU.
They mostly fall into two categories: bounded sleep, sometimes called
blocking, and unbounded sleep, usually abbreviated as sleep. The bounded
there refers to amount of code executed by other thread that hold resource
preventing blocked thread from making a progress.

Examples of the blocking primitives are mutexes, rw locks and rm locks.
The blocking is not counted as sleeping, so interrupt threads, which are
designated as non-sleeping, still can lock mutexes.

Examples of the sleeping primitives are msleep(), sx locks, lockmgr locks
and conditional variables.

In essence, the locking facilities are split into several classes that
form the hierarchy, and you cannot legally obtain the lock of higher class
while holding a lock of lower class:
	spin mutexes -> blocking locks -> sleeping locks.
It establishes some meta-order on the all locks.

Does this make sense ?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120222/82938dad/attachment.pgp

Hi Kashyap,

* Desai, Kashyap <Kashyap.Desai@lsi.com>, 20120222
18:51:
> Adding Ed Schouten and Jorg Wunsch  as I see there are author of
> msleep/mtx related APIs.
Am I? :-)
> 1. When any irq is register with FreeBSD OS, it sets "
TDP_NOSLEEPING"
> pflag. It means though irq in freebsd is treated as thread,
> We cannot sleep in IRQ because of " "TDP_NOSLEEPING " set.
> 2. In mps driver we have below code snippet in ISR routine.
> 
> 
>     mps_dprint(sc, MPS_TRACE, "%s\n", __func__);
>     mps_lock(sc);
>     mps_intr_locked(data);
>     mps_unlock(sc);
> 
> I wonder why there is no issue with above code ? Theoretical we cannot
> sleep in ISR. (as explained in #1)
> Any thoughts ?
The TDP_NOSLEEPING flag only disallows sleeping of an indeterminate
amount of time. Locking a mutex is allowed, as it can only cause the
thread to be blocked for a small amount of time, waiting for another
thread to unlock it.
> 3. I recently added few place msleep() instead of DELAY in ISR context
> and I see
> " Trying sleep, but thread marked as sleeping prohibited".
Which makes sense, as msleep() can be used to sleep for indefinitely.

-- 
 Ed Schouten <ed@80386.nl>
 WWW: http://80386.nl/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120223/f25e175e/attachment.pgp

> -----Original Message-----
> From: Ed Schouten [mailto:ed@80386.nl]
> Sent: Thursday, February 23, 2012 3:16 PM
> To: Desai, Kashyap
> Cc: freebsd-scsi@freebsd.org; freebsd-stable; joerg@FreeBSD.org; Kenneth
> D. Merry; McConnell, Stephen; Justin T. Gibbs
> Subject: Re: mpslsi0 : Trying sleep, but thread marked as sleeping
> prohibited
> 
> Hi Kashyap,
> 
> * Desai, Kashyap <Kashyap.Desai@lsi.com>, 20120222 18:51:
> > Adding Ed Schouten and Jorg Wunsch  as I see there are author of
> > msleep/mtx related APIs.
> 
> Am I? :-)
I am new to FreeBSD and desperate to know the answer. :-). Thanks for your
help.
> 
> > 1. When any irq is register with FreeBSD OS, it sets "
TDP_NOSLEEPING"
> > pflag. It means though irq in freebsd is treated as thread, We cannot
> > sleep in IRQ because of " "TDP_NOSLEEPING " set.
> > 2. In mps driver we have below code snippet in ISR routine.
> >
> >
> >     mps_dprint(sc, MPS_TRACE, "%s\n", __func__);
> >     mps_lock(sc);
> >     mps_intr_locked(data);
> >     mps_unlock(sc);
> >
> > I wonder why there is no issue with above code ? Theoretical we cannot
> > sleep in ISR. (as explained in #1) Any thoughts ?
> 
> The TDP_NOSLEEPING flag only disallows sleeping of an indeterminate
> amount of time. Locking a mutex is allowed, as it can only cause the
> thread to be blocked for a small amount of time, waiting for another
> thread to unlock it.
So does this assumption that another thread will release mutex fast enough ?
Same as msleep() this can be applicable here as well. I mean another thread 
_can_ (if some bad drivers) take long time to release mutex.
> 
> > 3. I recently added few place msleep() instead of DELAY in ISR context
> > and I see " Trying sleep, but thread marked as sleeping
prohibited".
> 
> Which makes sense, as msleep() can be used to sleep for indefinitely.
This part is clear. ! I agree with all experts view. 
> 
> --
>  Ed Schouten <ed@80386.nl>
>  WWW: http://80386.nl/

On Thu, 23 Feb 2012 18:45:29 +0530
"Desai, Kashyap" <Kashyap.Desai@lsi.com> wrote:
> 
> 
> > -----Original Message-----
> > From: Ed Schouten [mailto:ed@80386.nl]
> > Sent: Thursday, February 23, 2012 3:16 PM
> > To: Desai, Kashyap
> > Cc: freebsd-scsi@freebsd.org; freebsd-stable; joerg@FreeBSD.org;
> > Kenneth D. Merry; McConnell, Stephen; Justin T. Gibbs
> > Subject: Re: mpslsi0 : Trying sleep, but thread marked as sleeping
> > prohibited
> > 
> > Hi Kashyap,
> > 
> > * Desai, Kashyap <Kashyap.Desai@lsi.com>, 20120222 18:51:
> > > Adding Ed Schouten and Jorg Wunsch  as I see there are author of
> > > msleep/mtx related APIs.
> > 
> > Am I? :-)
> I am new to FreeBSD and desperate to know the answer. :-). Thanks for
> your help.
> > 
> > > 1. When any irq is register with FreeBSD OS, it sets "
> > > TDP_NOSLEEPING" pflag. It means though irq in freebsd is
treated
> > > as thread, We cannot sleep in IRQ because of "
"TDP_NOSLEEPING "
> > > set. 2. In mps driver we have below code snippet in ISR routine.
> > >
> > >
> > >     mps_dprint(sc, MPS_TRACE, "%s\n", __func__);
> > >     mps_lock(sc);
> > >     mps_intr_locked(data);
> > >     mps_unlock(sc);
> > >
> > > I wonder why there is no issue with above code ? Theoretical we
> > > cannot sleep in ISR. (as explained in #1) Any thoughts ?
> > 
> > The TDP_NOSLEEPING flag only disallows sleeping of an indeterminate
> > amount of time. Locking a mutex is allowed, as it can only cause the
> > thread to be blocked for a small amount of time, waiting for another
> > thread to unlock it.
> So does this assumption that another thread will release mutex fast
> enough ? Same as msleep() this can be applicable here as well. I mean
> another thread  _can_ (if some bad drivers) take long time to release
> mutex.
> 
>
Bad drivers can just defererence wild pointer to satisfy their need for
wrongdoing. For that reason let us keep them out of conversation.

mtx locks were designed to protect small sections of code where thread
is only allowed to block waiting for other thread to release the lock
of similar class. While threads are blocked, they get benefit of
adaptive sleep and priority propagation and should take precaution of
never taking the path of code that can cause them to sleep. Assertions
are there to help code authors with that.

sleep locks are by definition unbound. There is no spinning, no
priority propagation. Holders are free to take, say, page faults and go
to long journey to disk and back, etc. Hardly the stuff _anyone_ would
want to do from interrupt handler, thread or otherwise. 

 
> > > 3. I recently added few place msleep() instead of DELAY in ISR
> > > context and I see " Trying sleep, but thread marked as
sleeping
> > > prohibited".
> > 
> > Which makes sense, as msleep() can be used to sleep for
> > indefinitely.
> This part is clear. ! I agree with all experts view. 
> > 
> > --
> >  Ed Schouten <ed@80386.nl>
> >  WWW: http://80386.nl/
> _______________________________________________
> freebsd-scsi@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to
> "freebsd-scsi-unsubscribe@freebsd.org"

-- 
Alexander Kabaev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 188 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120223/d0a33b06/signature.pgp

> -----Original Message-----
> From: Alexander Kabaev [mailto:kabaev@gmail.com]
> Sent: Thursday, February 23, 2012 8:42 PM
> To: Desai, Kashyap
> Cc: Ed Schouten; freebsd-stable; joerg@FreeBSD.org; Kenneth D. Merry;
> freebsd-scsi@freebsd.org; Justin T. Gibbs; McConnell, Stephen
> Subject: Re: mpslsi0 : Trying sleep, but thread marked as sleeping
> prohibited
> 
> On Thu, 23 Feb 2012 18:45:29 +0530
> "Desai, Kashyap" <Kashyap.Desai@lsi.com> wrote:
> 
> >
> >
> > > -----Original Message-----
> > > From: Ed Schouten [mailto:ed@80386.nl]
> > > Sent: Thursday, February 23, 2012 3:16 PM
> > > To: Desai, Kashyap
> > > Cc: freebsd-scsi@freebsd.org; freebsd-stable; joerg@FreeBSD.org;
> > > Kenneth D. Merry; McConnell, Stephen; Justin T. Gibbs
> > > Subject: Re: mpslsi0 : Trying sleep, but thread marked as
sleeping
> > > prohibited
> > >
> > > Hi Kashyap,
> > >
> > > * Desai, Kashyap <Kashyap.Desai@lsi.com>, 20120222 18:51:
> > > > Adding Ed Schouten and Jorg Wunsch  as I see there are
author of
> > > > msleep/mtx related APIs.
> > >
> > > Am I? :-)
> > I am new to FreeBSD and desperate to know the answer. :-). Thanks for
> > your help.
> > >
> > > > 1. When any irq is register with FreeBSD OS, it sets "
> > > > TDP_NOSLEEPING" pflag. It means though irq in freebsd
is treated
> > > > as thread, We cannot sleep in IRQ because of "
"TDP_NOSLEEPING "
> > > > set. 2. In mps driver we have below code snippet in ISR
routine.
> > > >
> > > >
> > > >     mps_dprint(sc, MPS_TRACE, "%s\n", __func__);
> > > >     mps_lock(sc);
> > > >     mps_intr_locked(data);
> > > >     mps_unlock(sc);
> > > >
> > > > I wonder why there is no issue with above code ? Theoretical
we
> > > > cannot sleep in ISR. (as explained in #1) Any thoughts ?
> > >
> > > The TDP_NOSLEEPING flag only disallows sleeping of an
indeterminate
> > > amount of time. Locking a mutex is allowed, as it can only cause
the
> > > thread to be blocked for a small amount of time, waiting for
another
> > > thread to unlock it.
> > So does this assumption that another thread will release mutex fast
> > enough ? Same as msleep() this can be applicable here as well. I mean
> > another thread  _can_ (if some bad drivers) take long time to release
> > mutex.
> >
> >
> Bad drivers can just defererence wild pointer to satisfy their need for
> wrongdoing. For that reason let us keep them out of conversation.

OK. 
> 
> mtx locks were designed to protect small sections of code where thread
> is only allowed to block waiting for other thread to release the lock of
> similar class. While threads are blocked, they get benefit of adaptive
> sleep and priority propagation and should take precaution of never
> taking the path of code that can cause them to sleep. Assertions are
> there to help code authors with that.
> 
> sleep locks are by definition unbound. There is no spinning, no priority
> propagation. Holders are free to take, say, page faults and go to long
> journey to disk and back, etc.
I understood your above lines.
> Hardly the stuff _anyone_ would want to
> do from interrupt handler, thread or otherwise.
So the way mps driver does in interrupt handler is as below.

mps_lock(sc);
mps_intr_locked(data);
mps_unlock(sc);

We hold the mtx lock in Interrupt handler and do whole bunch of work(this is bit
lengthy work) under that.
It looks mps driver is miss using mtx_lock. Are we ?

~ Kashyap
> 
> 
> > > > 3. I recently added few place msleep() instead of DELAY in
ISR
> > > > context and I see " Trying sleep, but thread marked as
sleeping
> > > > prohibited".
> > >
> > > Which makes sense, as msleep() can be used to sleep for
> > > indefinitely.
> > This part is clear. ! I agree with all experts view.
> > >
> > > --
> > >  Ed Schouten <ed@80386.nl>
> > >  WWW: http://80386.nl/
> > _______________________________________________
> > freebsd-scsi@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> > To unsubscribe, send any mail to
> > "freebsd-scsi-unsubscribe@freebsd.org"
> 
> 
> --
> Alexander Kabaev