Howdy,
I am trying to use the Java 6 hotspot provider''s object-alloc probe to
determine where objects are being allocated in an application. When I
use the attached DTrace script [1], I keep seeing unresolved symbols
and "couldn''t find start" errors in the output:
libjvm.so`__1cNSharedRuntimeYdtrace_object_alloc_base6FpnGThread_pnHoopDesc__i_+0x7d
libjvm.so`__1cNSharedRuntimeTdtrace_object_alloc6FpnHoopDesc__i_+0x4f
libjvm.so`__1cNCollectedHeapbCpost_allocation_setup_common6FnLKlassHandle_pnIHeapWord_I_v_+0x121
libjvm.so`__1cOtypeArrayKlassIallocate6MipnGThread__pnQtypeArrayOopDesc__+0x19b
libjvm.so`__1cKoopFactoryNnew_typeArray6FnJBasicType_ipnGThread__pnQtypeArrayOopDesc__+0x2f
libjvm.so`__1cSInterpreterRuntimeInewarray6FpnKJavaThread_nJBasicType_i_v_+0x33
<couldn''t find start>
java/nio/ByteBuffer.allocate(I)Ljava/nio/ByteBuffer;
sun/nio/cs/StreamDecoder.<init>(Ljava/io/InputStream;Ljava/lang/Object;Ljava/nio/charset/CharsetDecoder;)V
sun/nio/cs/StreamDecoder.<init>(Ljava/io/InputStream;Ljava/lang/Object;Ljava/nio/charset/Charset;)V
sun/nio/cs/StreamDecoder.forInputStreamReader(Ljava/io/InputStream;Ljava/lang/Object;Ljava/lang/String;)Lsun/nio/cs/StreamDecoder;
0xfb402da7
0xfb402f0d
0xfb402f0d
0xfb402f0d
0xfb402da7
0xfb402da7
0xfb66e4d4
0xf7308980
4448736
Does anyone happen to know what is going on here? Without the method
names leading up to the actual allocations, it is somewhat difficult
to pinpoint who is keeping the garbage collector busy.
Thanks for any insight,
- Ryan
--
UNIX Administrator
http://prefetch.net
[1]
hotspot*:::object-alloc
/ copyinstr(arg1,arg2) == "[I" || copyinstr(arg1,arg2) ==
"[C" ||
copyinstr(arg1,arg2) == "[B" /
{
@stacks[jstack()] = sum(arg3);
}
profile:::tick-60s
{
printa(@stacks);
trunc(@stacks);
}
Matty wrote:> Howdy, > > I am trying to use the Java 6 hotspot provider''s object-alloc probe to > determine where objects are being allocated in an application. When I > use the attached DTrace script [1], I keep seeing unresolved symbols > and "couldn''t find start" errors in the output: > libjvm.so`__1cNSharedRuntimeYdtrace_object_alloc_base6FpnGThread_pnHoopDesc__i_+0x7d > libjvm.so`__1cNSharedRuntimeTdtrace_object_alloc6FpnHoopDesc__i_+0x4f > libjvm.so`__1cNCollectedHeapbCpost_allocation_setup_common6FnLKlassHandle_pnIHeapWord_I_v_+0x121 > libjvm.so`__1cOtypeArrayKlassIallocate6MipnGThread__pnQtypeArrayOopDesc__+0x19b > libjvm.so`__1cKoopFactoryNnew_typeArray6FnJBasicType_ipnGThread__pnQtypeArrayOopDesc__+0x2f > libjvm.so`__1cSInterpreterRuntimeInewarray6FpnKJavaThread_nJBasicType_i_v_+0x33 > <couldn''t find start> > java/nio/ByteBuffer.allocate(I)Ljava/nio/ByteBuffer; > sun/nio/cs/StreamDecoder.<init>(Ljava/io/InputStream;Ljava/lang/Object;Ljava/nio/charset/CharsetDecoder;)V > sun/nio/cs/StreamDecoder.<init>(Ljava/io/InputStream;Ljava/lang/Object;Ljava/nio/charset/Charset;)V > sun/nio/cs/StreamDecoder.forInputStreamReader(Ljava/io/InputStream;Ljava/lang/Object;Ljava/lang/String;)Lsun/nio/cs/StreamDecoder; > 0xfb402da7 > 0xfb402f0d > 0xfb402f0d > 0xfb402f0d > 0xfb402da7 > 0xfb402da7 > 0xfb66e4d4 > 0xf7308980 > 4448736Just curious, can you run and check the following: pargs <jvm_pid> dtrace -n ''syscall:::entry/pid==xxxx/{@[jstack(40)] = count();}'' If I recall correct you might need -XX:+ExtendedDTraceProbes for object-alloc which is much finer degree of debug. Try to add that to your JVM options and see if it is helping you. stefan
Matty''s email at 10/12/2007 9:49 PM, said:> Howdy, > > I am trying to use the Java 6 hotspot provider''s object-alloc probe to > determine where objects are being allocated in an application. When I > use the attached DTrace script [1], I keep seeing unresolved symbols > and "couldn''t find start" errors in the output: > libjvm.so`__1cNSharedRuntimeYdtrace_object_alloc_base6FpnGThread_pnHoopDesc__i_+0x7d > libjvm.so`__1cNSharedRuntimeTdtrace_object_alloc6FpnHoopDesc__i_+0x4f > libjvm.so`__1cNCollectedHeapbCpost_allocation_setup_common6FnLKlassHandle_pnIHeapWord_I_v_+0x121 > libjvm.so`__1cOtypeArrayKlassIallocate6MipnGThread__pnQtypeArrayOopDesc__+0x19b > libjvm.so`__1cKoopFactoryNnew_typeArray6FnJBasicType_ipnGThread__pnQtypeArrayOopDesc__+0x2f > libjvm.so`__1cSInterpreterRuntimeInewarray6FpnKJavaThread_nJBasicType_i_v_+0x33 > <couldn''t find start> > java/nio/ByteBuffer.allocate(I)Ljava/nio/ByteBuffer; > sun/nio/cs/StreamDecoder.<init>(Ljava/io/InputStream;Ljava/lang/Object;Ljava/nio/charset/CharsetDecoder;)V > sun/nio/cs/StreamDecoder.<init>(Ljava/io/InputStream;Ljava/lang/Object;Ljava/nio/charset/Charset;)V > sun/nio/cs/StreamDecoder.forInputStreamReader(Ljava/io/InputStream;Ljava/lang/Object;Ljava/lang/String;)Lsun/nio/cs/StreamDecoder; > 0xfb402da7 > 0xfb402f0d > 0xfb402f0d > 0xfb402f0d > 0xfb402da7 > 0xfb402da7 > 0xfb66e4d4 > 0xf7308980 > 4448736 > > Does anyone happen to know what is going on here? Without the method > names leading up to the actual allocations, it is somewhat difficult > to pinpoint who is keeping the garbage collector busy. > > Thanks for any insight, > - RyanThe default buffer size passed the the Java helper is too small. Change your jstack() to be something like: jstack(40,1024) This gives a 1024 byte buffer. I would not make it arbitrarily large, but I have never needed more than 2048 in the hundreds of times I have looked at Java apps. That should help you be able to resolve all of the symbols (well actually, they are all resolved - just nowhere to put them). Thanks, Jarod
Hey Jarod,
thanks for tip. True, the buffer size simple needs to be increased. Tracing BEA
Weblogic Portal 10 would require you for start a 2048 buffer size and 40 frames.
See in 1024 what you get:
libdvmti.so`_method_entry+0x16c
com/sun/tools/dtrace/internal/Tracker._method_entry*
sun/misc/URLClassPath.getLoader*
java/net/URLClassLoader$2.run*
I2CAdapter
StubRoutines (1)
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`JVM_DoPrivileged+0x500
libjava.so`Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedAction_2Ljava_security_AccessControlContext_2+0x14
java/security/AccessController.doPrivileged
java/security/AccessController.doPrivileged
C2IAdapter
java/net/URLClassLoader.findResource*
java/lang/ClassLoader.getResource*
java/lang/ClassLoader.getResource*
java/lang/ClassLoader.getResource*
weblogic/utils/classloaders/GenericClassLoader.getResourceInternal*
I2CAdapter
weblogic/utils/classloaders/GenericClassLoader.getResource
weblogic/utils/classloaders/FilteringClassLoader.getResource
C2IAdapter
java/lang/ClassLoader.getResource*
I2CAdapter
weblogic/utils/classloaders/GenericClassLoader.getResource
org/apache/commons/discovery/jdk/JDK12Hooks.getResources
org/apache/commons/discovery/resource/DiscoverResources$1.getNextResources
org/apache/commons/discovery/resource/DiscoverResources$1.getNextResource
0xf8c05874
0xf8c057a8
0xf8c05874
0xf8c05874
0xf8c05c70
0xf8c05874
0xf8c057a8
0xf8c05874
0xf8c05874
0xf8c05874
0xf8c05764
0xf8c05764
0xf8c05764
12738
We are trying to debug a problem with lots of calls from java/net/URLClassLoader
and dvm is really useful on this case. We are using here the trick with the
dvmpipe to enable/disable the agent when we need otherwise we will need to have
a long coffee break :)
Cheers,
Stefan
--
This message posted from opensolaris.org