Hi, We have a large application that uses dozens of shared objects that I''m trying to add dtrace probes into. Ideally, I''d like to have a single provider with a bunch of probes that could be triggered at various points throughout all of the libraries. However, this is impossible because you have to run dtrace -G against all of the objects for a provider at once. I''m trying to work around this by having a separate provider for each shared object, autogenerated from a template, so each provider has a bunch of probes with the same names. I''ve got this to work providing only one shared object is linked with dtrace, but as soon as 2 shared objects are linked with dtrace I get weird results - according to dtrace -l the second library to load seems to get it''s probes named as for the first library. dtrace -l if only libdcmge.so is compiled with dtrace probes in it: 5892 metadcmge5665 libdcmge.so mw_mge_media_format_lookup trc-flow 5893 metadcmge5665 libdcmge.so mw_mge_mega_build_term_name trc-flow 5894 metadcmge5665 libdcmge.so mw_mge_check_term_numbering trc-flow ... dtrace-l if both libnbase.so and libdcmge.so are compiled with probes in them: 7242 metanbase11957 libnbase.so nbb_dispatch_process ips-dispatch 7243 metanbase11957 libnbase.so yadf_log_connection_failure trc-flow 7244 metanbase11957 libnbase.so yadf_start_keepalives trc-flow ... 49479 metanbase11957 libdcmge.so nbb_dispatch_process ips-dispatch 49480 metanbase11957 libdcmge.so yadf_log_connection_failure trc-flow 49481 metanbase11957 libdcmge.so yadf_start_keepalives trc-flow Is the problem that you can you only have one USDT provider per-executable, or is it re-using the same probe names with different providers that causes the problem, or something else? Cheers, Mike -- This message posted from opensolaris.org
Hi Mike, I did a little experiment to try to replicate what you''re describing. I made libfoo.so which had a foo provider with a probe foo and libbar.so with a bar provider and a bar probe. This all seems to work: # dtrace -l | grep `pgrep main` 58121 bar140134 libbar.so bar bar 58122 foo140134 libfoo.so foo foo # dtrace -n foo -n bar dtrace: description ''foo'' matched 1 probe dtrace: description ''bar'' matched 1 probe CPU ID FUNCTION:NAME 1 58121 bar:bar 1 58122 foo:foo ... Is there any way you could provide binaries which exhibit this problem? Further, I confirmed that it''s possible to have a provider of the same name in two different shared objects which I think is a better solution to your problem. The only subtlety is that you have to generate a different .o file from the provider definition .d file for each shared object: $ dtrace -h -s provider.d $ cc -c bar.c foo.c bar.c: foo.c: $ dtrace -G -o bar_provider.o -s provider.d bar.o $ dtrace -G -o foo_provider.o -s provider.d foo.o $ cc -G -o libbar.so bar.o bar_provider.o $ cc -G -o libfoo.so foo.o foo_provider.o Please let me know if you have problems with that. Adam On Thu, Feb 28, 2008 at 04:27:54AM -0800, Mike Bell wrote:> Hi, > > We have a large application that uses dozens of shared objects that I''m trying to add dtrace probes into. Ideally, I''d like to have a single provider with a bunch of probes that could be triggered at various points throughout all of the libraries. However, this is impossible because you have to run dtrace -G against all of the objects for a provider at once. > > I''m trying to work around this by having a separate provider for each shared object, autogenerated from a template, so each provider has a bunch of probes with the same names. > > I''ve got this to work providing only one shared object is linked with dtrace, but as soon as 2 shared objects are linked with dtrace I get weird results - according to dtrace -l the second library to load seems to get it''s probes named as for the first library. > > dtrace -l if only libdcmge.so is compiled with dtrace probes in it: > > 5892 metadcmge5665 libdcmge.so mw_mge_media_format_lookup trc-flow > 5893 metadcmge5665 libdcmge.so mw_mge_mega_build_term_name trc-flow > 5894 metadcmge5665 libdcmge.so mw_mge_check_term_numbering trc-flow > ... > > dtrace-l if both libnbase.so and libdcmge.so are compiled with probes in them: > 7242 metanbase11957 libnbase.so nbb_dispatch_process ips-dispatch > 7243 metanbase11957 libnbase.so yadf_log_connection_failure trc-flow > 7244 metanbase11957 libnbase.so yadf_start_keepalives trc-flow > ... > 49479 metanbase11957 libdcmge.so nbb_dispatch_process ips-dispatch > 49480 metanbase11957 libdcmge.so yadf_log_connection_failure trc-flow > 49481 metanbase11957 libdcmge.so yadf_start_keepalives trc-flow > > Is the problem that you can you only have one USDT provider per-executable, or is it re-using the same probe names with different providers that causes the problem, or something else? > > Cheers, > Mike > > > -- > This message posted from opensolaris.org > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- Adam Leventhal, Fishworks http://blogs.sun.com/ahl
Hi Adam, Thanks for looking at this - I have repeated your experiment and confirm that it works fine here too (including having the same provider shared across two shared libs). So the problem I am seeing must be something to do with what happens when our application loads libraries. To test different scenarios, I have craeted C source files main, foo and bar. foo and bar are compiled to shared objects and main is compiled to an excutable. If foo and bar have dtrace probes in them, but main does not, you get the following behaviour: foo and bar linked to main by specifying -l on the cc command line: all is well foo and bar dynamically loaded using dlopen from main: all is well foo and bar dynamically loaded using dlopen with the RTLD_GLOBAL flag: broken behaviour - probe foo used for both foo and bar. It gets worse if main also uses dtrace. In this case you get broken behaviour regardless of how foo and bar are linked - you see main''s probe used for all of main, foo and bar.>From what I''ve seen, it looks like dtrace uses the global data __SUNW_dof to determine where to insert the probes. I think the problem arises when the dtrace init function picks the incorrect __SUNW_dof from the symbol table.Does that sound plausible? Mike -- This message posted from opensolaris.org
After some back and forth off the list, Mike and I determined that this was a bug, but one that was fixed in 2005: 6276101 dtrace -G behaves strangely with multiple scripts The latest patch for DTrace should include this fix. Adam On Fri, Feb 29, 2008 at 07:30:34AM -0800, Mike Bell wrote:> Hi Adam, > > Thanks for looking at this - I have repeated your experiment and confirm that it works fine here too (including having the same provider shared across two shared libs). So the problem I am seeing must be something to do with what happens when our application loads libraries. > > To test different scenarios, I have craeted C source files main, foo and bar. foo and bar are compiled to shared objects and main is compiled to an excutable. > > If foo and bar have dtrace probes in them, but main does not, you get the following behaviour: > foo and bar linked to main by specifying -l on the cc command line: all is well > foo and bar dynamically loaded using dlopen from main: all is well > foo and bar dynamically loaded using dlopen with the RTLD_GLOBAL flag: broken behaviour - probe foo used for both foo and bar. > > It gets worse if main also uses dtrace. In this case you get broken behaviour regardless of how foo and bar are linked - you see main''s probe used for all of main, foo and bar. > > >From what I''ve seen, it looks like dtrace uses the global data __SUNW_dof to determine where to insert the probes. I think the problem arises when the dtrace init function picks the incorrect __SUNW_dof from the symbol table. > > Does that sound plausible? > > Mike > > > -- > This message posted from opensolaris.org > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- Adam Leventhal, Fishworks http://blogs.sun.com/ahl