dtrace discuss - Feb 2007 - Asynchronous signal handling in fasttrap provider

Hi,

I was going through the implementation of the fasttrap provider. I found
that when displaced execution happens (ie., when the instruction traced is
being executed out of the scratch space) if an asynchrnous signal comes we
need to defer the delivery.

Can somebody explain why this is necessary and how is it implemented?

Thanks in advance
Saravanan S
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070214/a82eccb6/attachment.html>

Recall that displaced execution refers to relocating the original instruction
to memory at a different location.

There are some possibilities for exactly what memory we use for thes
displaced instructions. One option is that the kernel could allocate memory
on the process''s behalf by mapping memory into the process; we could
then
have scratch space for every traced instruction.  This would work (and was
seriously explored at one point), but mapping memory on a process''s
behalf
is a serious pain and at least a little rude (no matter what door_call(3C)
tells you).

We observed that a given thread can only be executing one instruction at any
given moment so we reserved a part of the per thread structure (ulwp_t) to
hold our displaced instructions. There was, however, a catch. When a thread is
interrupted by a signal and there''s an associated signal handler, that
thread''s execution will be diverted to the handler; when the handler is
done, executing returns to the interrupted instruction. If that instruction
was in the per-thread scratch space, it would be possible for that scratch
space to have been overwritten if a traced instruction was executed from
the signal handler. In other words, a given thread can have multiple contexts
active each of which could be executing from the scratch space.

To resolve this problem, we needed to make sure that the interrupted context
wouldn''t return to the displaced instruction scratch space. For
synchronous
signals -- ones caused by the executed instruction itself -- we simply back
up the program counter so that the same instruction would be executed again.
For asynchronous signals -- ones caused by some external activity -- we defer
their reception until after the thread has exited the per-thread scratch
space, and it''s fine to defer this signals briefly since there are no
guarantees about when exactly they''ll be handled.

To see the code that deals with signals and the pid provider look for
dtrace_safe_synchronous_signal() for synchronous signals, and for
dtrace_safe_defer_signal() for asynchronous signals (for the curious,
the ''dtrace_safe_'' prefix indicates that the function is
associated with
DTrace, but can safely be traced by the fbt provider).

If, by some fluke, you think this was an overly terse response, check out
this presentation where I delved into some the implementation details of
the pid provider:

  http://blogs.sun.com/ahl/entry/pid_provider_exposed

Adam

On Wed, Feb 14, 2007 at 10:11:50AM +0900, Saravanan S
wrote:
> Hi,
> 
> I was going through the implementation of the fasttrap provider. I found
> that when displaced execution happens (ie., when the instruction traced is
> being executed out of the scratch space) if an asynchrnous signal comes we
> need to defer the delivery.
> 
> Can somebody explain why this is necessary and how is it implemented?
> 
> Thanks in advance
> Saravanan S
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

Hi,

Thanks Adam for the detailed explanation(not terse). Apologies for not
posting to the group earlier..Some questions below...excuse me if my
questions seems dump.


On 2/14/07, Adam Leventhal <ahl at eng.sun.com>
wrote:
>
> Recall that displaced execution refers to relocating the original
> instruction
> to memory at a different location.
>
> There are some possibilities for exactly what memory we use for thes
> displaced instructions. One option is that the kernel could allocate
> memory
> on the process''s behalf by mapping memory into the process; we
could then
> have scratch space for every traced instruction.  This would work (and was
> seriously explored at one point), but mapping memory on a
process''s behalf
>
> is a serious pain and at least a little rude (no matter what door_call(3C)
> tells you).
>
> We observed that a given thread can only be executing one instruction at
> any
> given moment so we reserved a part of the per thread structure (ulwp_t) to
>
> hold our displaced instructions.

I looked at the ulwp_t structure and found that 40 bytes were allocated for
the scratch space in i386. I understand we need to have space for the traced
instruction,JMP to the next instruction and the alternate instruction
sequence ending in a trap required for handling aysynchronous
signal.Asshown clearly in fasttrap_isa.c''s fasttrap_pid_probe function
this accounts
for 37 bytes. Does the remaining three bytes allocated of the total 40 bytes
is used for someother purpose?

There was, however, a catch. When a thread is
> interrupted by a signal and there''s an associated signal handler,
that
> thread''s execution will be diverted to the handler; when the
handler is
> done, executing returns to the interrupted instruction. If that
> instruction
> was in the per-thread scratch space, it would be possible for that scratch
> space to have been overwritten if a traced instruction was executed from
> the signal handler. In other words, a given thread can have multiple
> contexts
> active each of which could be executing from the scratch space.

I am not able to understand what you mean by "overwritten if a traced
instruction was executed from the signal handler". Does this mean that
signal handler instructions(code) can also be dtraced using offset probes?
.Or am i completely missing something?

To resolve this problem, we needed to make sure that the interrupted
context
> wouldn''t return to the displaced instruction scratch space. For
> synchronous
> signals -- ones caused by the executed instruction itself -- we simply
> back
> up the program counter so that the same instruction would be executed
> again.
> For asynchronous signals -- ones caused by some external activity -- we
> defer
> their reception until after the thread has exited the per-thread scratch
> space, and it''s fine to defer this signals briefly since there are
no
> guarantees about when exactly they''ll be handled.

In the code I find that when we reach dtrace_safe_defer_signal on an
asynchronous signal we set the pc(eip) to the alternate execution
sequence(t->t_dtrace_npc) which will trap in to the kernel. In the function
dtrace_user_probe as part of the trap handling code we reset the AST flag.

if (curthread->t_dtrace_ast) {
aston(curthread);
curthread->t_sig_check = 1;
}
We then go forward and set the pc(eip) to curthread->t_dtrace_npc which i
seems to point to the alternate instruction sequence.So do we again trap in
to the kernel? Seems like i am missing something here, please elucidate.


To see the code that deals with signals and the pid provider look
for
> dtrace_safe_synchronous_signal() for synchronous signals, and for
> dtrace_safe_defer_signal() for asynchronous signals (for the curious,
> the ''dtrace_safe_'' prefix indicates that the function is
associated with
> DTrace, but can safely be traced by the fbt provider).

I find that in fbt_provide_module function of the fbt provider functions
belonging to dtrace module are not eligible for instrumentation.

if (strcmp(modname, "dtrace") == 0)
     return;

Doesn''t this mean dtrace_safe_ prefix functions are also not eligible
as
this symbols should also belong to the dtrace module?

If, by some fluke, you think this was an overly terse response, check
out
> this presentation where I delved into some the implementation details of
> the pid provider:
>
>   http://blogs.sun.com/ahl/entry/pid_provider_exposed

 I had gone through the presenstation and it was really good. It would be
really nice if you could add USDT implementation specifics also in the
slides. I did read your blog entries on USDT implementation specifics
as well which were very informative
http://blogs.sun.com/ahl/entry/user_land_tracing_gets_better
http://blogs.sun.com/ahl/entry/the_mysteries_of_init
Thank you for the same.

Thanks
Saravanan S

Adam
>
> On Wed, Feb 14, 2007 at 10:11:50AM +0900, Saravanan S wrote:
> > Hi,
> >
> > I was going through the implementation of the fasttrap provider. I
found
> > that when displaced execution happens (ie., when the instruction
traced
> is
> > being executed out of the scratch space) if an asynchrnous signal
comes
> we
> > need to defer the delivery.
> >
> > Can somebody explain why this is necessary and how is it implemented?
> >
> > Thanks in advance
> > Saravanan S
>
> > _______________________________________________
> > dtrace-discuss mailing list
> > dtrace-discuss at opensolaris.org
>
> --
> Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl
>


-- 
Subscribe LKG_INDIA Group :
Its Linux Kernel oriented group, started by bunch of
guys from India.This group discusses the technical details of Linux
Kernel, helping the members to share and enhance their knowledge on Linux
Kernel and general OS concepts.
Be a member of it if you are Linux Kernel Geek.
Visit the LKG_INDIA home page:
http://groups.yahoo.com/group/lkg_india/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070216/2efa2de7/attachment.html>

On Fri, Feb 16, 2007 at 09:59:05AM +0900, Saravanan S
wrote:
> I looked at the ulwp_t structure and found that 40 bytes were allocated for
> the scratch space in i386. I understand we need to have space for the
traced
> instruction,JMP to the next instruction and the alternate instruction
> sequence ending in a trap required for handling aysynchronous
> signal.Asshown clearly in fasttrap_isa.c''s fasttrap_pid_probe
function
> this accounts
> for 37 bytes. Does the remaining three bytes allocated of the total 40
bytes
> is used for someother purpose?
We only need 37 bytes, but since it was so close to a multiple of 8 I
couldn''t
help but round up...
> I am not able to understand what you mean by "overwritten if a traced
> instruction was executed from the signal handler". Does this mean that
> signal handler instructions(code) can also be dtraced using offset probes?
> .Or am i completely missing something?
Signal handlers are often just C functions which themselves may be traced
using the pid provider. Executing a traced function will overwrite the
scratch space so we can''t redirect the flow of control while
we''re actively
executing instructions in the scratch space.
> In the code I find that when we reach dtrace_safe_defer_signal on an
> asynchronous signal we set the pc(eip) to the alternate execution
> sequence(t->t_dtrace_npc) which will trap in to the kernel. In the
function
> dtrace_user_probe as part of the trap handling code we reset the AST flag.
> 
> if (curthread->t_dtrace_ast) {
> aston(curthread);
> curthread->t_sig_check = 1;
> }
> We then go forward and set the pc(eip) to curthread->t_dtrace_npc which
i
> seems to point to the alternate instruction sequence.So do we again trap in
> to the kernel? Seems like i am missing something here, please elucidate.
The alternate instruction sequence traps into the kernel (I''d suggest
going
through the presentation I referenced before as it explains this in more
detail).
> I find that in fbt_provide_module function of the fbt provider functions
> belonging to dtrace module are not eligible for instrumentation.
> 
> if (strcmp(modname, "dtrace") == 0)
>     return;
> 
> Doesn''t this mean dtrace_safe_ prefix functions are also not
eligible as
> this symbols should also belong to the dtrace module?
These symbols are in the unix module.
> >If, by some fluke, you think this was an overly terse response, check
out
> >this presentation where I delved into some the implementation details
of
> >the pid provider:
> >
> >  http://blogs.sun.com/ahl/entry/pid_provider_exposed
> 
> I had gone through the presenstation and it was really good. It would be
> really nice if you could add USDT implementation specifics also in the
> slides. I did read your blog entries on USDT implementation specifics
> as well which were very informative
> http://blogs.sun.com/ahl/entry/user_land_tracing_gets_better
> http://blogs.sun.com/ahl/entry/the_mysteries_of_init
> Thank you for the same.
My pleasure. I''m glad to hear they were useful. May I ask why
you''re
investigating the internals?

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On 2/19/07, Adam Leventhal <ahl at eng.sun.com>
wrote:
>
> On Fri, Feb 16, 2007 at 09:59:05AM +0900, Saravanan S wrote:
> > I looked at the ulwp_t structure and found that 40 bytes were
allocated
> for
> > the scratch space in i386. I understand we need to have space for the
> traced
> > instruction,JMP to the next instruction and the alternate instruction
> > sequence ending in a trap required for handling aysynchronous
> > signal.Asshown clearly in fasttrap_isa.c''s fasttrap_pid_probe
function
> > this accounts
> > for 37 bytes. Does the remaining three bytes allocated of the total 40
> bytes
> > is used for someother purpose?
>
> We only need 37 bytes, but since it was so close to a multiple of 8 I
> couldn''t
> help but round up...
>
> > I am not able to understand what you mean by "overwritten if a
traced
> > instruction was executed from the signal handler". Does this mean
that
> > signal handler instructions(code) can also be dtraced using offset
> probes?
> > .Or am i completely missing something?
>
> Signal handlers are often just C functions which themselves may be traced
> using the pid provider. Executing a traced function will overwrite the
> scratch space so we can''t redirect the flow of control while
we''re
> actively
> executing instructions in the scratch space.

Thanks for the explanation.
> In the code I find that when we reach dtrace_safe_defer_signal on an
> > asynchronous signal we set the pc(eip) to the alternate execution
> > sequence(t->t_dtrace_npc) which will trap in to the kernel. In the
> function
> > dtrace_user_probe as part of the trap handling code we reset the AST
> flag.
> >
> > if (curthread->t_dtrace_ast) {
> > aston(curthread);
> > curthread->t_sig_check = 1;
> > }
> > We then go forward and set the pc(eip) to curthread->t_dtrace_npc
which
> i
> > seems to point to the alternate instruction sequence.So do we again
trap
> in
> > to the kernel? Seems like i am missing something here, please
elucidate.
>
> The alternate instruction sequence traps into the kernel (I''d
suggest
> going
> through the presentation I referenced before as it explains this in more
> detail).
>
> > I find that in fbt_provide_module function of the fbt provider
functions
> > belonging to dtrace module are not eligible for instrumentation.
> >
> > if (strcmp(modname, "dtrace") == 0)
> >     return;
> >
> > Doesn''t this mean dtrace_safe_ prefix functions are also not
eligible as
> > this symbols should also belong to the dtrace module?
>
> These symbols are in the unix module.
>
> > >If, by some fluke, you think this was an overly terse response,
check
> out
> > >this presentation where I delved into some the implementation
details
> of
> > >the pid provider:
> > >
> > >  http://blogs.sun.com/ahl/entry/pid_provider_exposed
> >
> > I had gone through the presenstation and it was really good. It would
be
> > really nice if you could add USDT implementation specifics also in the
> > slides. I did read your blog entries on USDT implementation specifics
> > as well which were very informative
> > http://blogs.sun.com/ahl/entry/user_land_tracing_gets_better
> > http://blogs.sun.com/ahl/entry/the_mysteries_of_init
> > Thank you for the same.
>
> My pleasure. I''m glad to hear they were useful. May I ask why
you''re
> investigating the internals?

Firstly, i am interested in knowing the internals of Dtrace. I am also trying to
investigate how much easy it is to port Dtrace on a different
architecture or a OS apart from
SPARC/x86. I did find that the source code is clearly organised in to ISA
specific and platform independent code.I did learn that Dtrace was ported on
to Mac OS X(Xray) for the x86 architecture.  I tried to find the Xray source
code,
but i was not able to. I am not sure whether the Xray  code is open or
not. It would be nice if
you, the dtrace community and the Apple Dtrace team can
share your thoughts about porting Dtrace to a different
architecture/OS, the possible pitfalls that could
be encountered etc.,

Thanks
Saravanan S
> Adam
>
> --
> Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl
>


-- 
Subscribe LKG_INDIA Group :
Its Linux Kernel oriented group, started by bunch of
guys from India.This group discusses the technical details of Linux
Kernel, helping the members to share and enhance their knowledge on Linux
Kernel and general OS concepts.
Be a member of it if you are Linux Kernel Geek.
Visit the LKG_INDIA home page:
http://groups.yahoo.com/group/lkg_india/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070219/01c7ac28/attachment.html>

On Mon, Feb 19, 2007 at 01:29:24PM +0900, Saravanan S
wrote:
> Firstly, i am interested in knowing the internals of Dtrace. I am also 
> trying to investigate how much easy it is to port Dtrace on a different
> architecture or a OS apart from SPARC/x86.
Did you have a particular architecture or OS in mind?
> I tried to find the Xray source code,
> but i was not able to. I am not sure whether the Xray  code is open or not.
I imagine the Xray code won''t be made public, but the changes to CDDL
files
certainly should be once Apple ships derived binaries.
> It would be nice if
> you, the dtrace community and the Apple Dtrace team can
> share your thoughts about porting Dtrace to a different
> architecture/OS, the possible pitfalls that could
> be encountered etc.,
At one point the Apple team had made noises about putting together a porting
guide. I don''t think there''s been any progress, but perhaps if
they''re asked
nicely they could be persuaded to put something together.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On 2/19/07, Adam Leventhal <ahl at eng.sun.com>
wrote:
>
> On Mon, Feb 19, 2007 at 01:29:24PM +0900, Saravanan S wrote:
> > Firstly, i am interested in knowing the internals of Dtrace. I am also
> > trying to investigate how much easy it is to port Dtrace on a
different
> > architecture or a OS apart from SPARC/x86.
>
> Did you have a particular architecture or OS in mind?

Not really. I was thinking of PowerPC and Solaris, as there already
exists a opensolaris
community for porting opensolaris on PowerPC. Honestly speaking I have not even
tried to see the Power PC solaris code yet. But first I would like to
know the internals
of  Dtrace on x86(or any one architecture) before proceeding any further.
> I tried to find the Xray source code,
> > but i was not able to. I am not sure whether the Xray  code is open or
> not.
>
> I imagine the Xray code won''t be made public, but the changes to
CDDL
> files
> certainly should be once Apple ships derived binaries.

Ok. I will try to keep track of the same.
> It would be nice if
> > you, the dtrace community and the Apple Dtrace team can
> > share your thoughts about porting Dtrace to a different
> > architecture/OS, the possible pitfalls that could
> > be encountered etc.,
>
> At one point the Apple team had made noises about putting together a
> porting
> guide. I don''t think there''s been any progress, but
perhaps if they''re
> asked
> nicely they could be persuaded to put something together.

Does this mean,I can mail the Apple team asking for a porting guide.
If so,can you
please share their email ids.

Thanks
Saravanan S

Adam
>
> --
> Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070219/f11498b8/attachment.html>

On Mon, Feb 19, 2007 at 05:23:25PM +0900, Saravanan S
wrote:
> Does this mean,I can mail the Apple team asking for a porting guide.
> If so,can you please share their email ids.
I''m sure some of them are subscribed to this list so feel free to use
this
forum to make your appeal.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Feb 19, 2007, at 10:44 AM, Adam Leventhal wrote:
> On Mon, Feb 19, 2007 at 05:23:25PM +0900, Saravanan S wrote:
>> Does this mean,I can mail the Apple team asking for a porting guide.
>> If so,can you please share their email ids.
>
> I''m sure some of them are subscribed to this list so feel free to
> use this
> forum to make your appeal.
Saravanan,

	We do not have a porting guide, like many a project, that fell under
the pressure of "Is it done yet?" :-)

	As you may be aware, Apple is quite protective of any information that
hasn''t been publicly disclosed, I will not be able to answer many  
questions about
our implementation until after it ships.

	In general, I''ve found the dtrace code to be exceptionally well  
written.
My only complaint to date is the fact that vowels apparently cost $20  
each when
naming structure fields :-).

	If you''re porting from scratch, I would concentrate on getting just
one
provider working, say syscall or fbt. You can comment out large  
sections of the
code while bringing up dtrace. DTrace will work with quite a bit of  
its functionality
missing, and you can incrementally add them back in as you gain  
understanding
of the inner workings. The test suite is your friend!

	There will likely be several places where you have mismatches between
whatever OS you are porting to and the Solaris implementation.  
Depending on how
often you plan on sync''ing with the Solaris source, you may find it  
advantageous
to write emulation libraries instead of rewriting the Solaris code.

	If you can delay making decisions on how to handle those situations
until later (when you have a better understanding of how DTrace  
works), you
may save yourself a few code rewrites. Bringing up syscall or fbt  
should be
possible without hitting any major mismatches, every OS I know of has  
either
system calls or function calls :-).

	James M

On Feb 19, 2007, at 3:01 PM, James McIlree wrote:
>
> On Feb 19, 2007, at 10:44 AM, Adam Leventhal wrote:
>
>> On Mon, Feb 19, 2007 at 05:23:25PM +0900, Saravanan S wrote:
>>> Does this mean,I can mail the Apple team asking for a porting
guide.
>>> If so,can you please share their email ids.
>>
>> I''m sure some of them are subscribed to this list so feel free
to
>> use this
>> forum to make your appeal.
>
> Saravanan,
>
> 	We do not have a porting guide, like many a project, that fell under
> the pressure of "Is it done yet?" :-)
>
> 	As you may be aware, Apple is quite protective of any information  
> that
> hasn''t been publicly disclosed, I will not be able to answer many
> questions about
> our implementation until after it ships.
>
> 	In general, I''ve found the dtrace code to be exceptionally well  
> written.
> My only complaint to date is the fact that vowels apparently cost  
> $20 each when
> naming structure fields :-).
>
> 	If you''re porting from scratch, I would concentrate on getting  
> just one
> provider working, say syscall or fbt. You can comment out large  
> sections of the
> code while bringing up dtrace. DTrace will work with quite a bit of  
> its functionality
> missing, and you can incrementally add them back in as you gain  
> understanding
> of the inner workings. The test suite is your friend!
>
> 	There will likely be several places where you have mismatches between
> whatever OS you are porting to and the Solaris implementation.  
> Depending on how
> often you plan on sync''ing with the Solaris source, you may find
it
> advantageous
> to write emulation libraries instead of rewriting the Solaris code.
>
> 	If you can delay making decisions on how to handle those situations
> until later (when you have a better understanding of how DTrace  
> works), you
> may save yourself a few code rewrites. Bringing up syscall or fbt  
> should be
> possible without hitting any major mismatches, every OS I know of  
> has either
> system calls or function calls :-).
>
> 	James M
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
Saravanan,
Everything James said, and ....
I started the Mac OS X DTrace port on the *user* side, bringing up  
libdtrace and the dtrace(1) command line tool. Depending on your OS,  
lex/yacc "features" will require some taming. With a working dtrace 
(1), you can begin to think about a rudimentary kernel device driver  
that can answer to open/close/ioctl. The beautiful craftsmanship of  
the DTrace code makes all of this a pleasant and fun exercise.

SCP
--
Steve Peters
scp at mac.com

On 2/20/07, James McIlree <jmcilree at apple.com>
wrote:
>
>
>         As you may be aware, Apple is quite protective of any information
> that
> hasn''t been publicly disclosed, I will not be able to answer many
> questions about
> our implementation until after it ships.

I understand that. I will be waiting for the  shipment of the derived
binaries.
>
>         If you''re porting from scratch, I would concentrate on
getting
> just one
> provider working, say syscall or fbt. You can comment out large
> sections of the code while bringing up dtrace. DTrace will work with quite
> a bit of
> its functionality missing, and you can incrementally add them back in as
> you gain
> understanding of the inner workings. The test suite is your friend!

Thanks for the tips! I will try to use them.
>
>         If you can delay making decisions on how to handle those
> situations
> until later (when you have a better understanding of how DTrace
> works), you may save yourself a few code rewrites. Bringing up syscall or
> fbt
> should be possible without hitting any major mismatches, every OS I know
> of has
> either system calls or function calls :-).

Thats precisely my plan, first to investigate the internals of Dtrace
on one architecture(x86)
and estimate the things necessary for porting.  And slowly bringing up
provider by provider.

Thanks again for your valuable tips.

- Saravanan S

        James M
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070220/2a91bcc9/attachment.html>