What I''d like to be able to do is include functions such as rw_enter in fbt scripting. The problem I''m faced with is that functions such as these appear to not be present in fbt, only rw_enter_sleep... I glanced over at lockstat but rw-acquired is on the wrong side of the lock for me. What I''m trying to do is provoke a race condition and the easiest way that I could think of would be to force the thread to sleep as its about to enter the protected section of code. Is there any chance of there being RFE-able work here to provide an fbt or other probe pre-lock on the "leading" edge of functions such as rw_enter? Darren
rw_enter, mutex_enter, et al. are assembly language functions that do not engage in the typical function call save/restore register stuff that DTrace keys from. The only things you''re able to watch via DTrace from those functions is the C functions they call out to if the fast path is not possible. lockstat works its magic by rewriting some code in those functions at certain places (e.g., .rw_read_enter_lockstat_patch_point). Some of the patch points are early but some are quite late (as is the case with rw_enter). See $SRC/sparc/v9/mp/lock_prim.s for the details of the SPARC implementation. Jim --- ----- Original Message ----- From: Darren Reed <Darren.Reed at Sun.COM> Date: Wednesday, February 18, 2009 3:11 pm Subject: [dtrace-discuss] dtrace and locking... To: dtrace-discuss at opensolaris.org> What I''d like to be able to do is include functions such as > rw_enter in fbt scripting. The problem I''m faced with is > that functions such as these appear to not be present in > fbt, only rw_enter_sleep... > > I glanced over at lockstat but rw-acquired is on the wrong > side of the lock for me. > > What I''m trying to do is provoke a race condition and > the easiest way that I could think of would be to force > the thread to sleep as its about to enter the protected > section of code. > > Is there any chance of there being RFE-able work here > to provide an fbt or other probe pre-lock on the "leading" > edge of functions such as rw_enter? > > Darren > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
On 18/02/09 03:34 PM, James Litchfield wrote:> rw_enter, mutex_enter, et al. are assembly language functions > that do not engage in the typical function call save/restore register > stuff that DTrace keys from. The only things you''re able to watch via > DTrace from those functions is the C functions they call out to if the > fast path is not possible.I suppose what I''m seeking is an answer on whether or not it is possible to implement the fbt probes for such functions despite their implementation in the kernel being outside the realm of those that are normally associated with fbt. Well, I suppose this is just software, so theoretically anything is possible. But is this something that would be entertained? Darren> > lockstat works its magic by rewriting some code in those functions at > certain places (e.g., .rw_read_enter_lockstat_patch_point). Some of the > patch points are early but some are quite late (as is the case with rw_enter). > > See $SRC/sparc/v9/mp/lock_prim.s for the details of the SPARC implementation. > > Jim > --- > > ----- Original Message ----- > From: Darren Reed <Darren.Reed at Sun.COM> > Date: Wednesday, February 18, 2009 3:11 pm > Subject: [dtrace-discuss] dtrace and locking... > To: dtrace-discuss at opensolaris.org > > >> What I''d like to be able to do is include functions such as >> rw_enter in fbt scripting. The problem I''m faced with is >> that functions such as these appear to not be present in >> fbt, only rw_enter_sleep... >> >> I glanced over at lockstat but rw-acquired is on the wrong >> side of the lock for me. >> >> What I''m trying to do is provoke a race condition and >> the easiest way that I could think of would be to force >> the thread to sleep as its about to enter the protected >> section of code. >> >> Is there any chance of there being RFE-able work here >> to provide an fbt or other probe pre-lock on the "leading" >> edge of functions such as rw_enter? >> >> Darren >> >> _______________________________________________ >> dtrace-discuss mailing list >> dtrace-discuss at opensolaris.org
Darren Reed wrote:> What I''d like to be able to do is include functions such as > rw_enter in fbt scripting. The problem I''m faced with is > that functions such as these appear to not be present in > fbt, only rw_enter_sleep... > > I glanced over at lockstat but rw-acquired is on the wrong > side of the lock for me. > > What I''m trying to do is provoke a race condition and > the easiest way that I could think of would be to force > the thread to sleep as its about to enter the protected > section of code.I am not sure you could force a thread to sleep from the DTrace script. Did you consider using the chill() action ? [You could put a probe either in the caller of the rw_enter function or in rw_enter_sleep and have it spin on the cpu for few milliseconds using chill()] In my experiments i have found that the chill() action does widen the race windows. Pramod> > Is there any chance of there being RFE-able work here > to provide an fbt or other probe pre-lock on the "leading" > edge of functions such as rw_enter? > > Darren > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
On 19/02/09 05:42 AM, Pramod Batni wrote:> Darren Reed wrote: >> What I''d like to be able to do is include functions such as >> rw_enter in fbt scripting. The problem I''m faced with is >> that functions such as these appear to not be present in >> fbt, only rw_enter_sleep... >> >> I glanced over at lockstat but rw-acquired is on the wrong >> side of the lock for me. >> >> What I''m trying to do is provoke a race condition and >> the easiest way that I could think of would be to force >> the thread to sleep as its about to enter the protected >> section of code. > > I am not sure you could force a thread to sleep from the DTrace script.raise(SIGSTOP) seems like it would do the trick...> > Did you consider using the chill() action ? [You could put a probe > either in the caller of the rw_enter function or in rw_enter_sleep > and have it > spin on the cpu for few milliseconds using chill()] > In my experiments i have found that the chill() action does widen > the race windows.Yes, I can add a specific probe into the code to enable me to do what I want, but that doesn''t help me debug something when I''m not able to recompile the code with a special new dtrace probe in it. Darren>> Is there any chance of there being RFE-able work here >> to provide an fbt or other probe pre-lock on the "leading" >> edge of functions such as rw_enter? >> >> Darren >> >> _______________________________________________ >> dtrace-discuss mailing list >> dtrace-discuss at opensolaris.org >
On 02/20/09 04:13, Darren Reed wrote:> On 19/02/09 05:42 AM, Pramod Batni wrote: >> Darren Reed wrote: >>> What I''d like to be able to do is include functions such as >>> rw_enter in fbt scripting. The problem I''m faced with is >>> that functions such as these appear to not be present in >>> fbt, only rw_enter_sleep... >>> >>> I glanced over at lockstat but rw-acquired is on the wrong >>> side of the lock for me. >>> >>> What I''m trying to do is provoke a race condition and >>> the easiest way that I could think of would be to force >>> the thread to sleep as its about to enter the protected >>> section of code. >> >> I am not sure you could force a thread to sleep from the DTrace >> script. > > raise(SIGSTOP) > > seems like it would do the trick...If the protected section of code has a cv_wait_sig/cv_waituntil_sig/cv_timedwait_sig. If you enable the rw_enter_sleep:entry probe and in the action raise(SIGSTOP) the thread will not stop with the stack leading to rw_enter_sleep().> >> >> Did you consider using the chill() action ? [You could put a probe >> either in the caller of the rw_enter function or in rw_enter_sleep >> and have it >> spin on the cpu for few milliseconds using chill()] >> In my experiments i have found that the chill() action does widen >> the race windows. > > Yes, I can add a specific probe into the code to enable me > to do what I want, but that doesn''t help me debug something > when I''m not able to recompile the code with a special new > dtrace probe in it.I realise i was not precise enough. Apologies. What I meant is enable the probe rw_enter_sleep::entry and in the action clause for that probe call chill(); Pramod> > Darren > >>> Is there any chance of there being RFE-able work here >>> to provide an fbt or other probe pre-lock on the "leading" >>> edge of functions such as rw_enter? >>> >>> Darren >>> >>> _______________________________________________ >>> dtrace-discuss mailing list >>> dtrace-discuss at opensolaris.org >> >
On 19/02/09 10:42 PM, Pramod Batni wrote:> > > On 02/20/09 04:13, Darren Reed wrote: >> On 19/02/09 05:42 AM, Pramod Batni wrote: >>> Darren Reed wrote: >>>> What I''d like to be able to do is include functions such as >>>> rw_enter in fbt scripting. The problem I''m faced with is >>>> that functions such as these appear to not be present in >>>> fbt, only rw_enter_sleep... >>>> >>>> I glanced over at lockstat but rw-acquired is on the wrong >>>> side of the lock for me. >>>> >>>> What I''m trying to do is provoke a race condition and >>>> the easiest way that I could think of would be to force >>>> the thread to sleep as its about to enter the protected >>>> section of code. >>> >>> I am not sure you could force a thread to sleep from the DTrace >>> script. >> >> raise(SIGSTOP) >> >> seems like it would do the trick... > > If the protected section of code has a > cv_wait_sig/cv_waituntil_sig/cv_timedwait_sig. > If you enable the rw_enter_sleep:entry probe and in the action > raise(SIGSTOP) the thread > will not stop with the stack leading to rw_enter_sleep().So I discovered... sigh...>>> Did you consider using the chill() action ? [You could put a probe >>> either in the caller of the rw_enter function or in >>> rw_enter_sleep and have it >>> spin on the cpu for few milliseconds using chill()] >>> In my experiments i have found that the chill() action does widen >>> the race windows. >> >> Yes, I can add a specific probe into the code to enable me >> to do what I want, but that doesn''t help me debug something >> when I''m not able to recompile the code with a special new >> dtrace probe in it. > I realise i was not precise enough. Apologies. > What I meant is enable the probe rw_enter_sleep::entry and in the > action clause for that probe call chill()Which is only useful if rw_enter() needs to take the slow path. Darren