I''m trying to do some userspace tracing on a server-process with the pid provider. My problem is, that the only probes the pid-provider lists for the server-process (to which I attach dtrace with "-p nnnn") are coming from "ld.so.1". There''s not a single one from my modules. If I''m specifying "a.out" (or any of our shared-objects) as the module, I only get "No probe matches description". Tracing with the syscall-provider shows no problem, but is way too fine-grained for my purpose. When I''m attaching mdb to the server-process, "::objects" and "::nm" show me lots of our stuff. The server-process runs in a non-global zone, I''m trying to run dtrace in the global zone. The mdb-commands ran in the same zone as the server-process. With our Solaris 10 (11/06) local-zone dtrace should work, but I only get "DTrace device not available in local zone" as response :( Any idea what I''m doing wrong? Is there a restriction regarding cross-zone tracing with the pid-provider? Thomas -- This message posted from opensolaris.org
Hi Thomas, Can you send the invocations of dtrace(1M) that you''re using? What do you see if you do this: dtrace -l -n pid<yourpid>:::entry Adam On Mon, Aug 25, 2008 at 07:58:50AM -0700, Thomas Mohme wrote:> I''m trying to do some userspace tracing on a server-process with the pid provider. > My problem is, that the only probes the pid-provider lists for the server-process (to which I attach dtrace with "-p nnnn") are coming from "ld.so.1". > There''s not a single one from my modules. > If I''m specifying "a.out" (or any of our shared-objects) as the module, I only get "No probe matches description". > Tracing with the syscall-provider shows no problem, but is way too fine-grained for my purpose. > When I''m attaching mdb to the server-process, "::objects" and "::nm" show me lots of our stuff. > > The server-process runs in a non-global zone, I''m trying to run dtrace in the global zone. > The mdb-commands ran in the same zone as the server-process. > > With our Solaris 10 (11/06) local-zone dtrace should work, but I only get "DTrace device not available in local zone" as response :( > > Any idea what I''m doing wrong? > Is there a restriction regarding cross-zone tracing with the pid-provider? > > Thomas > > > -- > This message posted from opensolaris.org > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- Adam Leventhal, Fishworks http://blogs.sun.com/ahl
Hi Adam, I was wrong in stating that only probes from "ld.so.1" are seen... "dtrace -l -n pid[i]nnnn[/i]:::entry" gives me this: ID PROVIDER MODULE FUNCTION NAME 45042 pid[i]nnnn[/i] ld.so.1 avl_walk entry 45043 pid[i]nnnn[/i] ld.so.1 _cerror64 entry ... 45501 pid[i]nnnn[/i] ld.so.1 __systemcall6 entry 45502 pid[i]nnnn[/i] ld.so.1 _cerror entry 45994 pid[i]nnnn[/i] LM1`ld.so.1 avl_walk entry 45995 pid[i]nnnn[/i] LM1`ld.so.1 _cerror64 entry ... 46453 pid[i]nnnn[/i] LM1`ld.so.1 __systemcall6 entry 46454 pid[i]nnnn[/i] LM1`ld.so.1 _cerror entry 46455 pid[i]nnnn[/i] libsocket.so.1 bindresvport entry 46456 pid[i]nnnn[/i] libsocket.so.1 _nss_initf_bootparams entry ... 46592 pid[i]nnnn[/i] libsocket.so.1 __xnet_sendto entry 46593 pid[i]nnnn[/i] libsocket.so.1 __xnet_getsockopt entry 46594 pid[i]nnnn[/i] libnsl.so.1 thr_get_storage entry 46595 pid[i]nnnn[/i] libnsl.so.1 sig_mutex_lock entry ... 48455 pid[i]nnnn[/i] libnsl.so.1 setmodulus_192_0 entry 48456 pid[i]nnnn[/i] libnsl.so.1 __gen_common_dhkeys_g entry 48457 pid[i]nnnn[/i] librt.so.1 close entry 48458 pid[i]nnnn[/i] librt.so.1 fork entry ... 48544 pid[i]nnnn[/i] librt.so.1 sigtimedwait entry 48545 pid[i]nnnn[/i] librt.so.1 sigqueue entry 48546 pid[i]nnnn[/i] libpam.so.1 pam_trace_iname entry 48547 pid[i]nnnn[/i] libpam.so.1 pam_settrace entry ... 48587 pid[i]nnnn[/i] libpam.so.1 __pam_display_msg entry 48588 pid[i]nnnn[/i] libpam.so.1 __pam_get_authtok entry 48589 pid[i]nnnn[/i] libl.so.1 allprint entry 48590 pid[i]nnnn[/i] libl.so.1 sprint entry ... 48600 pid[i]nnnn[/i] libl.so.1 yyreject_e entry 48601 pid[i]nnnn[/i] libl.so.1 yyless_e entry 48602 pid[i]nnnn[/i] libdb2.so.1 sqlaprep entry 48603 pid[i]nnnn[/i] libdb2.so.1 __1cUsqlaprep_parm_errors6FpnKsqlagi_inp_hpch_h_ entry ... 58214 pid[i]nnnn[/i] libdb2.so.1 __ex_deregister_at_exit entry 58215 pid[i]nnnn[/i] libdb2.so.1 __cplus_fini_at_exit entry 58216 pid[i]nnnn[/i] libcobrts.so.2 I_DYNLNKLOADER entry 58217 pid[i]nnnn[/i] libcobrts.so.2 ld_dynlnk_ph_load entry ... 60485 pid[i]nnnn[/i] libcobrts.so.2 freepath entry 60486 pid[i]nnnn[/i] libcobrts.so.2 name_resolve entry 60487 pid[i]nnnn[/i] libcobcrtn.so.2 _cv2p2b entry 60488 pid[i]nnnn[/i] libcobcrtn.so.2 _cv3p2b entry ... 61713 pid[i]nnnn[/i] libcobcrtn.so.2 CBL_FN_FRACTION0PART entry 61714 pid[i]nnnn[/i] libcobcrtn.so.2 CBL_FN_SQRT entry 61715 pid[i]nnnn[/i] libcobmisc.so.2 MFaslmf_getlic entry 61716 pid[i]nnnn[/i] libcobmisc.so.2 aslmjmp entry ... 61845 pid[i]nnnn[/i] libcobmisc.so.2 asodm_init_nls entry 61846 pid[i]nnnn[/i] libcobmisc.so.2 asodm_nls_get entry 61847 pid[i]nnnn[/i] libc.so.1 .stret1 entry 61848 pid[i]nnnn[/i] libc.so.1 .stret2 entry ... 64356 pid[i]nnnn[/i] libc.so.1 statvfs64 entry 64357 pid[i]nnnn[/i] libc.so.1 mmap64 entry 64358 pid[i]nnnn[/i] libaio.so.1 _aio_forkinit entry 64359 pid[i]nnnn[/i] libaio.so.1 __uaio_init entry ... 64431 pid[i]nnnn[/i] libaio.so.1 _aio_alloc_worker entry 64432 pid[i]nnnn[/i] libaio.so.1 _aio_free_worker entry 64433 pid[i]nnnn[/i] libmd5.so.1 MD5Init entry 64434 pid[i]nnnn[/i] libmd5.so.1 MD5Update entry 64435 pid[i]nnnn[/i] libmd5.so.1 MD5Final entry 64436 pid[i]nnnn[/i] libmd5.so.1 md5_calc entry 64437 pid[i]nnnn[/i] libmd5.so.1 MD5Transform entry 64438 pid[i]nnnn[/i] libmd5.so.1 Encode entry 64439 pid[i]nnnn[/i] libcmd.so.1 free_thr_data entry 64440 pid[i]nnnn[/i] libcmd.so.1 deflt_init entry ... 64458 pid[i]nnnn[/i] libcmd.so.1 sumepi entry 64459 pid[i]nnnn[/i] libcmd.so.1 sumout entry 64460 pid[i]nnnn[/i] libCrun.so.1 __1cH__CimplGxstackIget_prev6M_p1_ entry 64461 pid[i]nnnn[/i] libCrun.so.1 __1cH__CimplMget_cur_xptr6F_rpn0AGxstack__ entry ... 64618 pid[i]nnnn[/i] libCrun.so.1 __ex_deregister_at_exit entry 64619 pid[i]nnnn[/i] libCrun.so.1 __cplus_fini_at_exit entry 64620 pid[i]nnnn[/i] libdb2install.so.1 __1cbDsqloConstructFullPathFromLink6Fkpkc2kIkpc_v_ entry 64621 pid[i]nnnn[/i] libdb2install.so.1 sqloSetDB2InstallLogFacility entry 64622 pid[i]nnnn[/i] libdb2install.so.1 sqloSetDB2InstallLogLevel entry 64623 pid[i]nnnn[/i] libdb2install.so.1 sqloInstallPath entry 64624 pid[i]nnnn[/i] libdb2install.so.1 sqleGetLevelInfo entry 64625 pid[i]nnnn[/i] libdb2install.so.1 __ex_deregister_at_exit entry 64626 pid[i]nnnn[/i] libdb2install.so.1 __cplus_fini_at_exit entry 64627 pid[i]nnnn[/i] libdb2g11n.so.1 __1cRsqlnlsUnicodeConv6FIIpvppCLpnLsqlnlscvcbx_pb_i_ entry 64628 pid[i]nnnn[/i] libdb2g11n.so.1 __1cbDsqlnlsProcessSavedUnicodeChar6FkIpCpnLsqlnlscvcbx_1pb44_i_ entry ... 64787 pid[i]nnnn[/i] libdb2g11n.so.1 __ex_deregister_at_exit entry 64788 pid[i]nnnn[/i] libdb2g11n.so.1 __cplus_fini_at_exit entry 64789 pid[i]nnnn[/i] libdb2locale.so.1 __1cUlocaleCmpNoModifiers6Fpkc1_i_ entry 64790 pid[i]nnnn[/i] libdb2locale.so.1 __1cRsqloFindLocMapRow6Fpkc_pnElmap__ entry ... 64811 pid[i]nnnn[/i] libdb2locale.so.1 __ex_deregister_at_exit entry 64812 pid[i]nnnn[/i] libdb2locale.so.1 __cplus_fini_at_exit entry 64813 pid[i]nnnn[/i] libdb2osse.so.1 ecfErrorGetNumCodes entry 64814 pid[i]nnnn[/i] libdb2osse.so.1 ecfGetNumProducts entry ... 65355 pid[i]nnnn[/i] libdb2osse.so.1 __ex_deregister_at_exit entry 65356 pid[i]nnnn[/i] libdb2osse.so.1 __cplus_fini_at_exit entry 65357 pid[i]nnnn[/i] libdb2genreg.so.1 __1cKGenRegBase2t5B6M_v_ entry 65358 pid[i]nnnn[/i] libdb2genreg.so.1 __1cKGenRegBase2T5B6M_v_ entry ... 65488 pid[i]nnnn[/i] libdb2genreg.so.1 __ex_deregister_at_exit entry 65489 pid[i]nnnn[/i] libdb2genreg.so.1 __cplus_fini_at_exit entry 65490 pid[i]nnnn[/i] libdb2trcapi.so.1 __1cNchkCrashPoint6FHLH_b_ entry 65491 pid[i]nnnn[/i] libdb2trcapi.so.1 __1cKtraceCrash6FHLHpLLppv_v_ entry ... 65562 pid[i]nnnn[/i] libdb2trcapi.so.1 __ex_deregister_at_exit entry 65563 pid[i]nnnn[/i] libdb2trcapi.so.1 __cplus_fini_at_exit entry 65564 pid[i]nnnn[/i] libdb2dascmn.so.1 peekNLString entry 65565 pid[i]nnnn[/i] libdb2dascmn.so.1 readNLString entry ... 65729 pid[i]nnnn[/i] libdb2dascmn.so.1 __ex_deregister_at_exit entry 65730 pid[i]nnnn[/i] libdb2dascmn.so.1 __cplus_fini_at_exit entry 65731 pid[i]nnnn[/i] libm.so.1 default_errno_func entry 65732 pid[i]nnnn[/i] libm.so.1 __libm_errno entry ... 65789 pid[i]nnnn[/i] libm.so.1 __libm__rem_pio2m entry 65790 pid[i]nnnn[/i] libm.so.1 __libm__k_tan entry 65791 pid[i]nnnn[/i] libresolv.so.2 daemon entry 65792 pid[i]nnnn[/i] libresolv.so.2 strsep entry ... 66388 pid[i]nnnn[/i] libresolv.so.2 res_mkupdrec entry 66389 pid[i]nnnn[/i] libresolv.so.2 res_freeupdrec entry 66390 pid[i]nnnn[/i] libcobscreen.so.2 mF_addch entry 66391 pid[i]nnnn[/i] libcobscreen.so.2 addtobuf entry ... 66479 pid[i]nnnn[/i] libcobscreen.so.2 pop entry 66480 pid[i]nnnn[/i] libcobscreen.so.2 jumpto entry 66481 pid[i]nnnn[/i] libm.so.1 __acosf entry 66482 pid[i]nnnn[/i] libm.so.1 __asinf entry ... 66523 pid[i]nnnn[/i] libm.so.1 __tanl entry 66524 pid[i]nnnn[/i] libm.so.1 __tanhl entry 66525 pid[i]nnnn[/i] libkstat.so.1 kstat_zalloc entry 66526 pid[i]nnnn[/i] libkstat.so.1 kstat_chain_free entry ... 66532 pid[i]nnnn[/i] libkstat.so.1 kstat_lookup entry 66533 pid[i]nnnn[/i] libkstat.so.1 kstat_data_lookup entry 66534 pid[i]nnnn[/i] libkvm.so.1 fail entry 66535 pid[i]nnnn[/i] libkvm.so.1 kvm_open entry ... 66555 pid[i]nnnn[/i] libkvm.so.1 kvm_getcmd32 entry 66556 pid[i]nnnn[/i] libkvm.so.1 kvm_getcmd entry 66557 pid[i]nnnn[/i] libdb2osse_db2.so.1 sqloPdbCommFncInit entry 66558 pid[i]nnnn[/i] libdb2osse_db2.so.1 sqloPdbCommFncTerm entry ... 66617 pid[i]nnnn[/i] libdb2osse_db2.so.1 __ex_deregister_at_exit entry 66618 pid[i]nnnn[/i] libdb2osse_db2.so.1 __cplus_fini_at_exit entry 66619 pid[i]nnnn[/i] libelf.so.1 _elf32_entsz entry 66620 pid[i]nnnn[/i] libelf.so.1 elf32_fsize entry ... 66860 pid[i]nnnn[/i] libelf.so.1 nlist entry 66861 pid[i]nnnn[/i] libelf.so.1 _elf_findop entry 66862 pid[i]nnnn[/i] libc_psr.so.1 memmove entry 66863 pid[i]nnnn[/i] libc_psr.so.1 memcpy entry ... 66869 pid[i]nnnn[/i] libc_psr.so.1 __urem64 entry 66870 pid[i]nnnn[/i] libc_psr.so.1 __rem64 entry 66871 pid[i]nnnn[/i] IBMOSauthclient.so __1cbAosplugin_validate_password6Fpkcl1ll1lLppv_i_ entry 66872 pid[i]nnnn[/i] IBMOSauthclient.so __1cZosplugin_validatePassword6Fpkcl1ll1l1l1lLppvppcpl_i_ entry ... 66881 pid[i]nnnn[/i] IBMOSauthclient.so __ex_deregister_at_exit entry 66882 pid[i]nnnn[/i] IBMOSauthclient.so __cplus_fini_at_exit entry 66883 pid[i]nnnn[/i] libmd5_psr.so.1 MD5Init entry 66884 pid[i]nnnn[/i] libmd5_psr.so.1 MD5Update entry 66885 pid[i]nnnn[/i] libmd5_psr.so.1 MD5Final entry 66886 pid[i]nnnn[/i] libmd5_psr.so.1 md5_calc entry 66887 pid[i]nnnn[/i] libmd5_psr.so.1 MD5Transform entry 66888 pid[i]nnnn[/i] libmd5_psr.so.1 Encode entry 66889 pid[i]nnnn[/i] libscf.so.1 scf_setup_error entry 66890 pid[i]nnnn[/i] libscf.so.1 scf_set_error entry ... 67170 pid[i]nnnn[/i] libscf.so.1 valid_encoded_value entry 67171 pid[i]nnnn[/i] libscf.so.1 scf_validate_encoded_value entry 67172 pid[i]nnnn[/i] libdoor.so.1 _door_init entry 67173 pid[i]nnnn[/i] libdoor.so.1 door_unref_func entry ... 67184 pid[i]nnnn[/i] libdoor.so.1 door_create_func entry 67185 pid[i]nnnn[/i] libdoor.so.1 door_create_server entry 67186 pid[i]nnnn[/i] libuutil.so.1 avl_walk entry 67187 pid[i]nnnn[/i] libuutil.so.1 avl_first entry ... 67292 pid[i]nnnn[/i] libuutil.so.1 uu_strtoint entry 67293 pid[i]nnnn[/i] libuutil.so.1 uu_strtouint entry 67294 pid[i]nnnn[/i] libmp.so.2 mp_gcd entry 67295 pid[i]nnnn[/i] libmp.so.2 mp_invert entry ... 67329 pid[i]nnnn[/i] libmp.so.2 mp_mtox entry 67330 pid[i]nnnn[/i] libmp.so.2 mp_mfree entry 67331 pid[i]nnnn[/i] libicclib.so METAC_CRYPTO_set_mem_ex_functions entry 67332 pid[i]nnnn[/i] libicclib.so METAC_CRYPTO_get_mem_ex_functions entry ... 67490 pid[i]nnnn[/i] libicclib.so METAC_fips_dss_prng_ctx_free entry 67491 pid[i]nnnn[/i] libicclib.so METAC_fips_dss_prng_add_seed entry 67492 pid[i]nnnn[/i] libcrypto.so.0.9.7 C97C_CRYPTO_get_new_lockid entry 67493 pid[i]nnnn[/i] libcrypto.so.0.9.7 C97C_CRYPTO_num_locks entry ... 69819 pid[i]nnnn[/i] libcrypto.so.0.9.7 C97C_KRB5_AUTHENT_new entry 69820 pid[i]nnnn[/i] libcrypto.so.0.9.7 C97C_KRB5_AUTHENT_free entry Obviously there''s more than only "ld.so.1". Unfortunately "dtrace -l -n pid[i]nnnn[/i]:a.out::entry" gives me this: dtrace: failed to match pid[i]nnnn[/i]:stub.exe::entry: No probe matches description while "nm stub.exe | fgrep FUNC" gives me this: [95] | 139988| 0|FUNC |GLOB |0 |UNDEF |_exit [135] | 74256| 56|FUNC |GLOB |0 |11 |_fini [108] | 140288| 0|FUNC |WEAK |0 |UNDEF |_get_exit_frame_monitor [130] | 74200| 56|FUNC |GLOB |0 |10 |_init [154] | 140264| 0|FUNC |GLOB |0 |UNDEF |_mFexeload [117] | 140276| 0|FUNC |GLOB |0 |UNDEF |_mFexeunload [119] | 69064| 288|FUNC |GLOB |0 |9 |_start [146] | 69368| 804|FUNC |GLOB |0 |9 |action [137] | 70192| 348|FUNC |GLOB |0 |9 |action2 [139] | 70560| 2360|FUNC |GLOB |0 |9 |action3 [109] | 139964| 0|FUNC |GLOB |0 |UNDEF |atexit [106] | 140120| 0|FUNC |GLOB |0 |UNDEF |atoi [136] | 140012| 0|FUNC |GLOB |0 |UNDEF |bdog_changemystate [113] | 140156| 0|FUNC |GLOB |0 |UNDEF |bdog_changemystate2 [110] | 140240| 0|FUNC |GLOB |0 |UNDEF |bdog_close [158] | 140180| 0|FUNC |GLOB |0 |UNDEF |bdog_onlywithparent [156] | 140168| 0|FUNC |GLOB |0 |UNDEF |bdog_open [131] | 72936| 204|FUNC |GLOB |0 |9 |berta [115] | 140084| 0|FUNC |GLOB |0 |UNDEF |close [148] | 140228| 0|FUNC |GLOB |0 |UNDEF |cyCloseMPX [124] | 140216| 0|FUNC |GLOB |0 |UNDEF |cyLoopMPX [127] | 140204| 0|FUNC |GLOB |0 |UNDEF |cyOpenStubMPX [134] | 139976| 0|FUNC |GLOB |0 |UNDEF |exit [121] | 140000| 0|FUNC |GLOB |0 |UNDEF |getenv [152] | 73160| 444|FUNC |GLOB |0 |9 |hugo [141] | 73624| 576|FUNC |GLOB |0 |9 |main [140] | 140144| 0|FUNC |GLOB |0 |UNDEF |memcmp [129] | 140096| 0|FUNC |GLOB |0 |UNDEF |memcpy [93] | 140108| 0|FUNC |GLOB |0 |UNDEF |memset [122] | 140036| 0|FUNC |GLOB |0 |UNDEF |mepkid_call [112] | 140192| 0|FUNC |GLOB |0 |UNDEF |mepkid_init [138] | 140252| 0|FUNC |GLOB |0 |UNDEF |puts [89] | 140072| 0|FUNC |GLOB |0 |UNDEF |recv [116] | 140060| 0|FUNC |GLOB |0 |UNDEF |send [114] | 140048| 0|FUNC |GLOB |0 |UNDEF |setjmp [128] | 140132| 0|FUNC |GLOB |0 |UNDEF |sprintf [153] | 140024| 0|FUNC |GLOB |0 |UNDEF |strcmp>From this I conclude that the needed symbols aren''t stripped from our module(s).So I expect DTrace to "see" at least functions "main", "action", etc. When I do similar checks with a hello-world program in the global zone only everything behaves as expected. "mdb -p [i]nnnn[/i]" followed by "::objects" gives me this:> ::objectsBASE LIMIT SIZE NAME 10000 14000 4000 [i]path-to-our-modules[/i]/stub.exe ff3b0000 ff3de000 2e000 /lib/ld.so.1 ff380000 ff386000 6000 [i]path-to-our-modules[/i]/syk006.so ff360000 ff366000 6000 [i]path-to-our-modules[/i]/kidinit.so ff330000 ff340000 10000 [i]path-to-our-modules[/i]/kidlib.so ff300000 ff306000 6000 [i]path-to-our-modules[/i]/ezasoket.so ff2e0000 ff2e2000 2000 [i]path-to-our-modules[/i]/syhtsp.so ff2b0000 ff2be000 e000 [i]path-to-our-modules[/i]/blib.so ff290000 ff294000 4000 [i]path-to-our-modules[/i]/sykcon.so ff3a0000 ff3a2000 2000 /lib/libdl.so.1 ff250000 ff25c000 c000 /lib/libsocket.so.1 ff180000 ff212000 92000 /lib/libnsl.so.1 ff160000 ff166000 6000 /lib/librt.so.1 ff140000 ff148000 8000 /lib/libpam.so.1 ff120000 ff122000 2000 /usr/lib/libl.so.1 ff288000 ff28c000 4000 /lib/libthread.so.1 ff278000 ff27c000 4000 /lib/libpthread.so.1 fd400000 fd800000 400000 /opt/IBM/db2/V9.1/lib32/libdb2.so.1 ff000000 ff092000 92000 /opt/microfocus/cobol/lib/libcobrts.so.2 ff0c0000 ff104000 44000 /opt/microfocus/cobol/lib/libcobcrtn.so.2 fefb0000 fefd4000 24000 /opt/microfocus/cobol/lib/libcobmisc.so.2 fd300000 fd3d8000 d8000 /lib/libc.so.1 fef80000 fef88000 8000 /lib/libaio.so.1 fef60000 fef62000 2000 /lib/libmd5.so.1 fd2e0000 fd2e4000 4000 /lib/libcmd.so.1 fd2a0000 fd2ac000 c000 /usr/lib/libCrun.so.1 fd280000 fd284000 4000 /opt/IBM/db2/V9.1/lib32/libdb2install.so.1 fcc00000 fcc6e000 6e000 /opt/IBM/db2/V9.1/lib32/libdb2g11n.so.1 fd250000 fd266000 16000 /opt/IBM/db2/V9.1/lib32/libdb2locale.so.1 fc900000 fcb46000 246000 /opt/IBM/db2/V9.1/lib32/libdb2osse.so.1 fcbb0000 fcbe4000 34000 /opt/IBM/db2/V9.1/lib32/libdb2genreg.so.1 fc8e0000 fc8e8000 8000 /opt/IBM/db2/V9.1/lib32/libdb2trcapi.so.1 fc8b0000 fc8ca000 1a000 /opt/IBM/db2/V9.1/lib32/libdb2dascmn.so.1 ... So mdb sees not only the "standard"-libraries, but also our modules, which DTrace seems to refuse Any suggestions are welcome. Thx, Thomas -- This message posted from opensolaris.org
As I got questions about the behaviour with the "Z"-flag: It is exactly the same as without the flag - for listing the probes as well as for doing the trace. Thomas -- This message posted from opensolaris.org
A clarification on the -Z flag; this is not a panacea and it doesn''t do
anything zany or ill-defined. It simply does this:
Permit probe descriptions that match zero probes. If the
-Z option is not specified, dtrace reports an error and
exits if any probe descriptions specified in D program
files (-s option) or on the command line (-P, -m, -f,
-n, or -i options) contain descriptions that do not
match any known probes.
In some situations you may need to use the -Z flag because the probe you
want to enable doesn''t yet exist. For example, if a program dynamically
opens
a module, specifying a probe in that as-to-yet unopened library will generate
an error. The -Z flag will suppress that error -- but you must still be
careful to accurately specify the probe. Here''s an example of
it''s misuse:
# dtrace -n ''pid$target::mane:entry'' -c date
dtrace: invalid probe specifier pid$target::mane:entry: probe description
pid741533::mane:entry does not match any probes
# dtrace -Z -n ''pid$target::mane:entry'' -c date
dtrace: description ''pid$target::mane:entry'' matched 0 probes
Tue Aug 26 09:17:26 PDT 2008
dtrace: pid 741535 has exited
I hope that helps.
Adam
--
Adam Leventhal, Fishworks http://blogs.sun.com/ahl
On Tue, Aug 26, 2008 at 12:57:46AM -0700, Thomas Mohme wrote:> Unfortunately "dtrace -l -n pid[i]nnnn[/i]:a.out::entry" gives me this: > > dtrace: failed to match pid[i]nnnn[/i]:stub.exe::entry: No probe matches description > > > while "nm stub.exe | fgrep FUNC" gives me this: > [snip] > > From this I conclude that the needed symbols aren''t stripped from our > module(s). So I expect DTrace to "see" at least functions "main", "action", > etc. > > So mdb sees not only the "standard"-libraries, but also our modules, which > DTrace seems to refuse > > Any suggestions are welcome.Hey Thomas, Can you explain a bit more about how zones fits into the picture (what works or doesn''t work from what zones on processes in what zones, etc)? Also, can you send me a private email with the output of the following invocation: dtrace -l -n pid123:a.out:entry -x debug Thanks. Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl
Hi Adam, I continue being interested in solving the problem - just was out of office yesterday. I get this strange behavior when (as non-root with dtrace-proc and dtrace-user privs) I try to trace a process in a non-global zone (running there under my account, same privs) from the global zone. This already is my "fallback" path, because my first attempt (to do the tracing in the non-global zone where the process is running) failed with "dtrace: failed to initialize dtrace: DTrace device not available in local zone" I get the same effect with a variant of the classic "hello, world"-program when trying to trace it in the same manner (cross-zone, as described above). But everything works as expected when I''m running both the "hello, world"-program and dtrace in the global zone. Unfortunately running the server-process in the global-zone isn''t a practicable path, because this process depends heavy on the versions of installed third-party libraries, which differ between the zones. Thomas -- This message posted from opensolaris.org
Hi Thomas, Thanks for the debug data - I think I know what''s going on now (for those who are wondering - I contacted Thomas offline to get some debug and ask some questions).> I get this strange behavior when (as non-root with dtrace-proc and dtrace-user privs) I try to trace a process in a non-global zone (running there under my account, same privs) from the global zone. > > This already is my "fallback" path, because my first attempt (to do the tracing in the non-global zone where the process is running) failed with "dtrace: failed to initialize dtrace: DTrace device not available in local zone"DTrace doesn''t work in a zone until Solaris 10 08/07 so that''s why you''re seeing this (I believe you''re running Solaris 10 11/06).> I get the same effect with a variant of the classic "hello, world"-program when trying to trace it in the same manner (cross-zone, as described above). > But everything works as expected when I''m running both the "hello, world"-program and dtrace in the global zone. > Unfortunately running the server-process in the global-zone isn''t a practicable path, because this process depends heavy on the versions of installed third-party libraries, which differ between the zones.The reason why you''re seeing this is NFS (NFS experts correct me if I''m wrong here). I understand that the failure cases occur when the binaries are located in an NFS mount in the non-global zone. In the current implementation a zone cannot access files associated with NFS mounts in other zones and this includes the global zone. Therefore when process X runs in zone Y we cannot access its constituent files for pertinent debug data from another zone even if that is the global zone. I think this screws you for debugging objects which are running from non-global zone based NFS mounts but doing the debugging in the global zone. I think you have two solutions: 1) Upgrade to a later update of Solaris 10 and then you can do your debugging all in the non-global zone. 2) Run off local filesystems in the non-global zone. Let me know if you have any questions (or anyone with NFS/Zone expertise feel free to chip in here). Jon.
On Thu, Aug 28, 2008 at 12:27:34AM -0700, Thomas Mohme wrote:> Hi Adam, > > I continue being interested in solving the problem - just was out of office yesterday. > > > I get this strange behavior when (as non-root with dtrace-proc and > dtrace-user privs) I try to trace a process in a non-global zone > (running there under my account, same privs) from the global zone. > > > This already is my "fallback" path, because my first attempt (to do > the tracing in the non-global zone where the process is running) > failed with "dtrace: failed to initialize dtrace: DTrace device not > available in local zone" > > I get the same effect with a variant of the classic "hello, > world"-program when trying to trace it in the same manner (cross-zone, > as described above). > > But everything works as expected when I''m running both the "hello, > world"-program and dtrace in the global zone. > > Unfortunately running the server-process in the global-zone isn''t a > practicable path, because this process depends heavy on the versions > of installed third-party libraries, which differ between the zones.Do you have the "PRIV_PROC_ZONE" privilege? (you can check by running: % ppriv $$ ... flags = <none> E: basic,proc_zone I: basic,proc_zone P: basic,proc_zone L: all % ). With zones, there''s no guarantee that UIDs inside a local zone matches the UIDs in the global zone. So the system prevents processes in the global zone from effecting processes in local zones. With this privilege on your global zone process, you''ll be able to act on processes in other zones with your UID just as if they were in the global zone. It grants no other rights. The dtrace(1M) consumer needs to be able to attach to a process like a debugger to set up some of it''s probes. Without PRIV_PROC_ZONE, it''s not possible to do so. Cheers, - jonathan
Hi Jon,> DTrace doesn''t work in a zone until Solaris 10 08/07 so that''s > why you''re seeing this (I believe you''re running Solaris 10 11/06).I think our systems engineers came up with this assumption (DTrace in zones works in 11/06) because after first noticing the problem they googled with "dtrace zone". The first hit is "Using DTrace in a Non-Global Zone" in the Solaris [Express] Sytem Administration Guide. The limitpriv property of the zonecfg command mentioned there is available since 11/06. I think this was the cause of the misunderstanding. Maybe this historical "restriction" and DTrace-Zone-NFS restriction find their way into the documentation (or, if already there, to a more prominent place) ;) But maybe this is a stumbling block only to Solaris newbies like me and these restrictions are (at least in parts) well known to the pros.> I think this screws you for debugging objects which are running from > non-global zone based NFS mounts but doing the debugging in the > global zone. I think you have two solutions: > > 1) Upgrade to a later update of Solaris 10 and then you can do > your debugging all in the non-global zone.I''ll definitely push this since 11/06 isn''t that fresh anyway. Unfortunately it''s a big shop with real-world restrictions and real-world systems engineers, so the mills grind slowly . . .> 2) Run off local filesystems in the non-global zone.My workaround is to copy the binaries to a local filesystem in the non-global zone after building them. It''s a bit ugly but bearable. By all means it''s way better than any other alternative. Once again, thank you for your support! Thomas -- This message posted from opensolaris.org
Hi Jonathan,> Do you have the "PRIV_PROC_ZONE" privilege? (you can check by running: > > % ppriv $$ > ... > flags = <none> > E: basic,proc_zone > I: basic,proc_zone > P: basic,proc_zone > L: all > % > )."ppriv $$" in the global zone gives me 14567: -bash flags = <none> E: basic,dtrace_proc,dtrace_user,file_dac_read,proc_owner,proc_zone I: basic,dtrace_proc,dtrace_user,file_dac_read,proc_owner,proc_zone P: basic,dtrace_proc,dtrace_user,file_dac_read,proc_owner,proc_zone L: all> With zones, there''s no guarantee that UIDs inside a local zone matches the UIDs > in the global zone. So the system prevents processes in the global zone from > effecting processes in local zones.My UID is the same in the global and in the local zone.> With this privilege on your global zone process, you''ll be able to act on > processes in other zones with your UID just as if they were in the global > zone. It grants no other rights. > > The dtrace(1M) consumer needs to be able to attach to a process like a debugger > to set up some of it''s probes. Without PRIV_PROC_ZONE, it''s not possible > to do so.After some initial confusion Jon Haslam found out that the reason for the absence of DTrace''s cross-zone-ability is just the too old 11/06 release we''re using. Thomas -- This message posted from opensolaris.org
Hi Thomas, Apologies for not following up on the list to your reply. I''m glad that you can now move forward with your DTrace based endeavors and hope that this wasn''t too much of a pain to get sorted.> Maybe this historical "restriction" and DTrace-Zone-NFS restriction find their way into the documentation (or, if already there, to a more prominent place) ;) > But maybe this is a stumbling block only to Solaris newbies like me and these restrictions are (at least in parts) well known to the pros.Don''t believe it! We do our best to make sure that all this complementary technology works well together and, for the most part, it does. However, there are always different combinations that we may not have tried or thought about (read "I" for "we" in that last sentence really) so there are always new "restrictions" to understand and address. That''s part of the fun though :-) . Jon.