I''m on Sol10u4. Sun Studio12.
I''m trying to add some custom probes to an app (which I''ve
done
successfully in the past) and this time it is just being obstinate.
The application in question is Apache. I''ve added a few probes in
here and there. Now, Apache''s build system is a bit intricate relying
on apr to pull the strings.
The one issues I have is that it creates a lot of archive (*.a) files
that get linked together in the end. As dtrace can''t instrument those
I''ve added just kept track of each object being archived and later
link the httpd process by linking all the .o files explicitly.
So, I have a rather simple set of probes (whittled down dramatically
for troubleshooting this issue):
(apache_probes.d):
provider ap {
/* Explicit, core */
probe internal__redirect(char *, char *);
probe process__request__entry(uintptr_t, char *);
probe process__request__return(uintptr_t, char *, uint32_t);
probe read__request__entry(uintptr_t);
probe read__request__success(uintptr_t, char *, char *, char *,
uint32_t);
probe read__request__failure(uintptr_t);
};
I generate the header using dtrace -h and use the explicit macros
instead of the generic DTRACE_PROBE#() ones.
The objects all compile. If I attempt to link, I get the expected
unresolved dependencies. I then run:
dtrace -64 -G -a apache_probes.d -o probes.o <....... really long list
of objects ....>
I then link httpd using all the objects and probes.o which links
successfully and runs well (Apache server behaving as expected), but I
see no probes. Looking into the bowels of the dof_init stuff I see an
environment variable and set it:
# DTRACE_DOF_INIT_DEBUG=1 ./httpd -f ~/a.conf -X
dtrace DOF httpd: DTrace ioctl succeeded for DOF at 4c2608httpd
So... it would appear something is wrong with the DOF that is being
ioctl''d into the helper during dof_init. I''m totally stuck.
I have
instrumentation for a plethora of other apps and have only hit the
issue here. I could use some assistance troubleshooting. Anyone have
an ideas?
Thanks!
--
Theo Schlossnagle
Esoteric Curio -- http://lethargy.org/
OmniTI Computer Consulting, Inc. -- http://omniti.com/
Hey Theo,> The one issues I have is that it creates a lot of archive (*.a) files > that get linked together in the end. As dtrace can''t instrument those > I''ve added just kept track of each object being archived and later > link the httpd process by linking all the .o files explicitly.Do I understand correctly that you''ve worked around that part of the problem by not using archive files? I realize that''s an annoying issue with USDT and one we''d like to address.> The objects all compile. If I attempt to link, I get the expected > unresolved dependencies. I then run: > dtrace -64 -G -a apache_probes.d -o probes.o <....... really long list > of objects ....>What unresolved dependencies were you expecting? The ones for the dummy functions in the generated header file?> I then link httpd using all the objects and probes.o which links > successfully and runs well (Apache server behaving as expected), but I > see no probes. Looking into the bowels of the dof_init stuff I see an > environment variable and set it: > > # DTRACE_DOF_INIT_DEBUG=1 ./httpd -f ~/a.conf -X > dtrace DOF httpd: DTrace ioctl succeeded for DOF at 4c2608httpd > > So... it would appear something is wrong with the DOF that is being > ioctl''d into the helper during dof_init. I''m totally stuck. I have > instrumentation for a plethora of other apps and have only hit the > issue here. I could use some assistance troubleshooting. Anyone have > an ideas?Can you send me (offline) the binary that contains the DOF section? Using elfdump(1), one can extract the DOF object from its ELF section, and then use mdb(1) on that file to verify the DOF (mdb has a built-in DOF target mode). I''d give you the specifics of how that all works, but I''m going to have to reacquaint myself with them using this binary. :-) Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl
On Apr 29, 2008, at 1:57 AM, Adam Leventhal wrote:> Hey Theo, > >> The one issues I have is that it creates a lot of archive (*.a) files >> that get linked together in the end. As dtrace can''t instrument >> those >> I''ve added just kept track of each object being archived and later >> link the httpd process by linking all the .o files explicitly. > > Do I understand correctly that you''ve worked around that part of the > problem by not using archive files? I realize that''s an annoying > issue with USDT and one we''d like to address.Yes. That same technique is used in PostgreSQL with success. Each subsystem in that is actually linked as ld -r -o SUBSYS.o $(OBJS). I tried that here and during the the long process of troubleshooting I resorted to just using the original objects.> >> The objects all compile. If I attempt to link, I get the expected >> unresolved dependencies. I then run: >> dtrace -64 -G -a apache_probes.d -o probes.o <....... really long >> list >> of objects ....> > > What unresolved dependencies were you expecting? The ones for the > dummy functions in the generated header file?Yes. Undefined first referenced symbol in file __dtrace_ap___read__request__entry /home/jesus/src/apache-2.2.x/ server/protocol.o __dtrace_ap___read__request__success /home/jesus/src/apache-2.2.x/ server/protocol.o __dtrace_ap___process__request__return modules/http/http_request.o __dtrace_ap___read__request__failure /home/jesus/src/apache-2.2.x/ server/protocol.o __dtrace_ap___internal__redirect modules/http/http_request.o __dtrace_ap___process__request__entry modules/http/http_request.o And, as expected, running dtrace -G "fixes" these in the source objects.> >> I then link httpd using all the objects and probes.o which links >> successfully and runs well (Apache server behaving as expected), >> but I >> see no probes. Looking into the bowels of the dof_init stuff I see >> an >> environment variable and set it: >> >> # DTRACE_DOF_INIT_DEBUG=1 ./httpd -f ~/a.conf -X >> dtrace DOF httpd: DTrace ioctl succeeded for DOF at 4c2608httpd >> >> So... it would appear something is wrong with the DOF that is being >> ioctl''d into the helper during dof_init. I''m totally stuck. I have >> instrumentation for a plethora of other apps and have only hit the >> issue here. I could use some assistance troubleshooting. Anyone >> have >> an ideas? > > Can you send me (offline) the binary that contains the DOF section? > Using elfdump(1), one can extract the DOF object from its ELF > section, and then use mdb(1) on that file to verify the DOF (mdb has > a built-in DOF target mode). I''d give you the specifics of how that > all works, but I''m going to have to reacquaint myself with them > using this binary. :-)Sure. -- Theo Schlossnagle Principal/CEO OmniTI Computer Consulting, Inc. Web Applications & Internet Architectures w: http://omniti.com p: +1.443.325.1357 x201 f: +1.410.872.4911