dtrace discuss - Feb 2007 - Hotspot provider system crashes, class data sharing and automatic server detection gone wrong

HI,

I''ve been looking at the hotspot provider and wanted to demo some of 
it''s features. Running on nevada 56 I''m getting a  lot of
system hangs
:( The system was rock solid until I started playing with the hotspot 
provider, it seems particularly unhappy about the class-loaded probe for 
some reason, though it''s hung with other method-entry probes. Anyone 
else seeing this?

The following script was run, as I wanted to demonstrate Class data sharing.

$ dtrace -Zqn ''hotspot*:::class-loaded{ @loaded[arg2, copyinstr(arg0, 
arg1), arg3? "Shared":"Unique"] = count(); 
@total[arg3?"Shared":"Unique"] = count();}
END{printa("%-8d %40s %-10s
%8 at d\n", @loaded); printa("%-20s %@d\n", at total);}''

$ java -jar /usr/java/demo/jfc/SwingSet2/SwingSet2.jar

Strangely all the classes where unique. Checked to see if I had a class 
archive, I do:
/usr/jdk/instances/jdk1.6.0/jre/lib/i386/client/classes.jsa

Then looked to see what the hotspot VM thought:
bash-3.00$ java -help
Usage: java [-options] class [args...]
          :
    -client       to select the "client" VM
    -server       to select the "server" VM
    -hotspot      is a synonym for the "client" VM  [deprecated]
                  The default VM is server,
                  because you are running on a server-class machine.

So appears it thinks my new dual core Sony Vaio with 2 Gig Ram is a 
server :( This is due to the joys of Smart tuning introduced in J2SE 
5.0, which I''m sure is a good thing but perhaps the Automatic Server 
detection needs revisiting, now that desktop machines are getting a lot 
beefier :)

If I run [note the -client directive]:
$ java -client -jar /usr/java/demo/jfc/SwingSet2/SwingSet2.jar

I now get over 1800 shared classes loaded as reported by the dtrace 
probe above. Unfortunately I now seem to crash even more often. Has the 
client VM option been as heavily tested on nevada as the default server 
VM option?

JR

I''ve been playing with this and it seems that the combination of using 
-Z and hotspot* makes the system hang a lot. I wanted to use these 
options so I could have the probe running before I started up different 
applications. Sticking to no -Z and hotspot$target seems to be a bit 
more stable.

Curious if I''m the only one seeing this? Is it a nevada 56 issue?
JR


John Rice wrote:
> Hi,
>
> I''ve been looking at the hotspot provider and wanted to demo some
of
> it''s features. Running on nevada 56 I''m getting a  lot of
system hangs
> :( The system was rock solid until I started playing with the hotspot 
> provider, it seems particularly unhappy about the class-loaded probe 
> for some reason, though it''s hung with other method-entry probes. 
> Anyone else seeing this?
>
> The following script was run, as I wanted to demonstrate Class data 
> sharing.
>
> $ dtrace -Zqn ''hotspot*:::class-loaded{ @loaded[arg2,
copyinstr(arg0,
> arg1), arg3? "Shared":"Unique"] = count(); 
> @total[arg3?"Shared":"Unique"] = count();}
END{printa("%-8d %40s %-10s
> %8 at d\n", @loaded); printa("%-20s %@d\n", at
total);}''
>
> $ java -jar /usr/java/demo/jfc/SwingSet2/SwingSet2.jar
>
> Strangely all the classes where unique. Checked to see if I had a 
> class archive, I do:
> /usr/jdk/instances/jdk1.6.0/jre/lib/i386/client/classes.jsa
>
> Then looked to see what the hotspot VM thought:
> bash-3.00$ java -help
> Usage: java [-options] class [args...]
>          :
>    -client       to select the "client" VM
>    -server       to select the "server" VM
>    -hotspot      is a synonym for the "client" VM  [deprecated]
>                  The default VM is server,
>                  because you are running on a server-class machine.
>
> So appears it thinks my new dual core Sony Vaio with 2 Gig Ram is a 
> server :( This is due to the joys of Smart tuning introduced in J2SE 
> 5.0, which I''m sure is a good thing but perhaps the Automatic
Server
> detection needs revisiting, now that desktop machines are getting a 
> lot beefier :)
>
> If I run [note the -client directive]:
> $ java -client -jar /usr/java/demo/jfc/SwingSet2/SwingSet2.jar
>
> I now get over 1800 shared classes loaded as reported by the dtrace 
> probe above. Unfortunately I now seem to crash even more often. Has 
> the client VM option been as heavily tested on nevada as the default 
> server VM option?
>
> JR
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

Hi John,

John Rice wrote:
> I''ve been playing with this and it seems that the combination of
using
> -Z and hotspot* makes the system hang a lot. I wanted to use these 
> options so I could have the probe running before I started up different 
> applications. Sticking to no -Z and hotspot$target seems to be a bit 
> more stable.
> Curious if I''m the only one seeing this? Is it a nevada 56 issue?
I''m running snv_57 on a dual-core opteron. I''m using java
"1.6.0-b105"

>> I''ve been looking at the hotspot provider and wanted to demo
some of
>> it''s features. Running on nevada 56 I''m getting a 
lot of system hangs
>> :( The system was rock solid until I started playing with the hotspot 
>> provider, it seems particularly unhappy about the class-loaded probe 
>> for some reason, though it''s hung with other method-entry
probes.
>> Anyone else seeing this?
>>
>> The following script was run, as I wanted to demonstrate Class data 
>> sharing.
>>
>> $ dtrace -Zqn ''hotspot*:::class-loaded{ @loaded[arg2,
copyinstr(arg0,
>> arg1), arg3? "Shared":"Unique"] = count(); 
>> @total[arg3?"Shared":"Unique"] = count();}
END{printa("%-8d %40s %-10s
>> %8 at d\n", @loaded); printa("%-20s %@d\n", at
total);}''
>> $ java -jar /usr/java/demo/jfc/SwingSet2/SwingSet2.jar
the D above gave me

^C
0                                java/lang/Object Unique            1
Unique               1

and the jar died with this collection of stack traces:

core ''core'' of 10516:   java -jar
/usr/java/demo/jfc/SwingSet2/SwingSet2.jar
-----------------  lwp# 1 / thread# 1  --------------------
  feef8d37 __lwp_wait (3, 8046550) + 7
  feef4b43 _thrp_join (3, 0, 80465a0, 1) + 5d
  feef4c83 thr_join (3, 0, 80465a0) + 23
  08058a47 ContinueInNewThread (8052260, 50000, 0, 8046e48, 8046e00) + 4f
  08051f59 main     (0, 806af9c, 8046ed4) + aa1
  0805142a _start   (3, 8047050, 8047055, 804705a, 0, 8047085) + 7a
-----------------  lwp# 2 / thread# 2  --------------------
  feef8a07 __pollsys (fe31bd40, 2, 0, 0) + 7
  feeb41fe pselect  (5, fe31be20, fef32260, fef32260, 0, 0) + 19e
  feeb450e select   (5, fe31be20, 0, 0, 0) + 7e
  fe515fb0 SplashEventLoop (fe551f08) + 11c
  fe516453 SplashScreenThread (fe551f08) + df
  feef7d82 _thr_setup (fee42400) + 52
  feef7fe0 _lwp_start (fee42400, 0, 0, 0, 0, 0)
-----------------  lwp# 3 / thread# 3  --------------------
  feeed7f4 sigacthandler (b, fe0ad744, fe0ad544) + 14
  --- called from signal handler with signal 11 (SIGSEGV) ---
  feef1188 mutex_lock_impl (fef316e0, 0) + 18
  feef13a2 mutex_lock (fef316e0) + 1d
  feea4de4 malloc   (b) + 2c
  fe8aca9f __1cCosGmalloc6FI_pv_ (b) + 2b
  fea60c5e __1cLClassLoaderLadd_package6Fpkci_b_ (80f7cc8, 1) + 62
  fe8ec0bc 
__1cLClassLoaderOload_classfile6FnMsymbolHandle_pnGThread__nTinstanceKlassHandle__
(fe0ada30, fee22da4) + 25c
  fe8e1b70 
__1cQSystemDictionaryTload_instance_class6FnMsymbolHandle_nGHandle_pnGThread__nTinstanceKlassHandle__
(fe0adaf8, fee22da4, 0) + 314
  fe8af3ac 
__1cQSystemDictionarybEresolve_instance_class_or_null6FnMsymbolHandle_nGHandle_2pnGThread__pnMklassOopDesc__
(fee22da4, 0, 0, 80f7800) + 514
  fe8aa9db 
__1cQSystemDictionaryPresolve_or_null6FnMsymbolHandle_nGHandle_2pnGThread__pnMklassOopDesc__
(fee22da4, 0, 0, 80f7800) + 63
  fec95002 
__1cQSystemDictionaryPresolve_or_fail6FnMsymbolHandle_nGHandle_2bpnGThread__pnMklassOopDesc__
(fee22da4, 0, 0, 1, 80f7800) + 2a
  fec951fa 
__1cQSystemDictionaryPresolve_or_fail6FnMsymbolHandle_bpnGThread__pnMklassOopDesc__
(fee22da4, 1, 80f7800) + 32
  fe9db82a __1cQSystemDictionarybCinitialize_preloaded_classes6FpnGThread__v_ 
(80f7800) + 36
  fe9dc26b __1cQSystemDictionaryKinitialize6FpnGThread__v_ (80f7800) + c7
  fe99ec6a __1cIUniverseHgenesis6FpnGThread__v_ (80f7800) + afa
  fe9a0080 __1cOuniverse2_init6F_v_ (fedde000, 80f7800, 80f7800, fe0adec4, 
feca0b3c, fee0dff8) + 30
  fe9b5134 __1cMinit_globals6F_i_ (fee0dff8, fe0adf40, fedde000, fe0adec4, 1, 
0) + 70
  feca0b3c __1cHThreadsJcreate_vm6FpnOJavaVMInitArgs_pb_i_ (fe0adf40, 
fe0adf04) + 1c4
  fe9bd412 JNI_CreateJavaVM (fe0adfc4, fe0adfc8, fe0adf40) + a6
  080523bf JavaMain (8046e48) + 15f
  feef7d82 _thr_setup (fe040000) + 52
  feef7fe0 _lwp_start (fe040000, 0, 0, 0, 0, 0)
-----------------  lwp# 4 / thread# 4  --------------------
  feef8dd7 ___lwp_cond_wait (80fdef8, 80fdee0) + 7
  fec3621a __1cHMonitorEwait6Mbl_b_ (80fde88, 1, 0) + 46a
  fe8aab8d __1cNGCTaskManagerIget_task6MI_pnGGCTask__ (80fde48, 0) + 89
  fe99acb1 __1cMGCTaskThreadDrun6M_v_ (80fe000) + 16d
  fec40007 java_start (80fe000) + d3
  feef7d82 _thr_setup (fe040400) + 52
  feef7fe0 _lwp_start (fe040400, 0, 0, 0, 0, 0)
-----------------  lwp# 5 / thread# 5  --------------------
  feef8dd7 ___lwp_cond_wait (80fdef8, 80fdee0) + 7
  fec3621a __1cHMonitorEwait6Mbl_b_ (80fde88, 1, 0) + 46a
  fe8aab8d __1cNGCTaskManagerIget_task6MI_pnGGCTask__ (80fde48, 1) + 89
  fe99acb1 __1cMGCTaskThreadDrun6M_v_ (80ff000) + 16d
  fec40007 java_start (80ff000) + d3
  feef7d82 _thr_setup (fe040800) + 52
  feef7fe0 _lwp_start (fe040800, 0, 0, 0, 0, 0)



Running -hotspot gave this thread stack in the core (and no splash screen)

-----------------  lwp# 3 / thread# 3  --------------------
  feeed7f4 sigacthandler (b, fe28d6a4, fe28d4a4) + 14
  --- called from signal handler with signal 11 (SIGSEGV) ---
  feef1188 mutex_lock_impl (fef316e0, 0) + 18
  feef13a2 mutex_lock (fef316e0) + 1d
  feea5940 free     (8110f38) + 20
  fe83bfeb __1cCosEfree6Fpv_v_ (8110f38) + 1b
  fe8d624f __1cICHeapObj2k6Fpv_v_ (8110f38) + 1b
  feb1d12e 
__1cQPlaceholderEntrySremove_seen_thread6MpnGThread_nQPlaceholderTablePclassloadAction__b_
(810a820, 80f8000, 1) + 7a
  fe881a8d 
__1cQSystemDictionarybEresolve_instance_class_or_null6FnMsymbolHandle_nGHandle_2pnGThread__pnMklassOopDesc__
(80f8874, 0, 0, 80f8000) + 649
  fe88143b 
__1cQSystemDictionaryPresolve_or_null6FnMsymbolHandle_nGHandle_2pnGThread__pnMklassOopDesc__
(80f8874, 0, 0, 80f8000) + 63
  feb1be23 
__1cQSystemDictionaryVresolve_super_or_fail6FnMsymbolHandle_1nGHandle_2bpnGThread__pnMklassOopDesc__
(80f8864, 80f8874, 0, 0, 0, 80f8000) + 16b
  fe8827fa 
__1cQSystemDictionaryRload_shared_class6FnTinstanceKlassHandle_nGHandle_pnGThread__1_
(fe28d9c8, 80f8860, 0) + 20a
  fe882578 
__1cQSystemDictionaryRload_shared_class6FnMsymbolHandle_nGHandle_pnGThread__nTinstanceKlassHandle__
(fe28da2c, fec56bf4, 0) + 6c
  fe8824a6 
__1cQSystemDictionaryTload_instance_class6FnMsymbolHandle_nGHandle_pnGThread__nTinstanceKlassHandle__
(fe28daf8, fec56bf4, 0) + 2fa
  fe881958 
__1cQSystemDictionarybEresolve_instance_class_or_null6FnMsymbolHandle_nGHandle_2pnGThread__pnMklassOopDesc__
(fec56bf4, 0, 0, 80f8000) + 514
  fe88143b 
__1cQSystemDictionaryPresolve_or_null6FnMsymbolHandle_nGHandle_2pnGThread__pnMklassOopDesc__
(fec56bf4, 0, 0, 80f8000) + 63
  feb1baba 
__1cQSystemDictionaryPresolve_or_fail6FnMsymbolHandle_nGHandle_2bpnGThread__pnMklassOopDesc__
(fec56bf4, 0, 0, 1, 80f8000) + 2a
  feb1bcb2 
__1cQSystemDictionaryPresolve_or_fail6FnMsymbolHandle_bpnGThread__pnMklassOopDesc__
(fec56bf4, 1, 80f8000) + 32
  fe880a86 __1cQSystemDictionarybCinitialize_preloaded_classes6FpnGThread__v_ 
(80f8000) + 5e
  fe880967 __1cQSystemDictionaryKinitialize6FpnGThread__v_ (80f8000) + c7
  fe875d2a __1cIUniverseHgenesis6FpnGThread__v_ (80f8000) + afa
  fe8751a4 __1cOuniverse2_init6F_v_ (fec32000, 80f8000, 80f8000, fe28dec4, 
feb26df0, fec44dc8) + 30
  fe840d8c __1cMinit_globals6F_i_ (fec44dc8, fe28df40, fec32000, fe28dec4, 1, 
0) + 70
  feb26df0 __1cHThreadsJcreate_vm6FpnOJavaVMInitArgs_pb_i_ (fe28df40, 
fe28df04) + 1c4
  fe83a11e JNI_CreateJavaVM (fe28dfc4, fe28dfc8, fe28df40) + a6
  080523bf JavaMain (8046e38) + 15f
  feef7d82 _thr_setup (fe220000) + 52
  feef7fe0 _lwp_start (fe220000, 0, 0, 0, 0, 0)


and
^C
0                            java/io/Serializable Shared            1
0                                java/lang/Object Shared            1
Shared               2



I can''t run that demo with java 1.5.0.


cheers,
James C. McPherson
--
Solaris kernel software engineer, system admin and troubleshooter
               http://www.jmcp.homeunix.com/blog
Find me on LinkedIn @ http://www.linkedin.com/in/jamescmcpherson

Hi John,

I also observed the similar hangs on nevada 41
(but I am not sure it is related to -Z).
We upgraded to nevada 58 and seems there are no more hangs.

-katya

John Rice wrote:
> I''ve been playing with this and it seems that the combination of
using
> -Z and hotspot* makes the system hang a lot. I wanted to use these 
> options so I could have the probe running before I started up different 
> applications. Sticking to no -Z and hotspot$target seems to be a bit 
> more stable.
> 
> Curious if I''m the only one seeing this? Is it a nevada 56 issue?
> JR
> 
> 
> John Rice wrote:
> 
>> Hi,
>>
>> I''ve been looking at the hotspot provider and wanted to demo
some of
>> it''s features. Running on nevada 56 I''m getting a 
lot of system hangs
>> :( The system was rock solid until I started playing with the hotspot 
>> provider, it seems particularly unhappy about the class-loaded probe 
>> for some reason, though it''s hung with other method-entry
probes.
>> Anyone else seeing this?
>>
>> The following script was run, as I wanted to demonstrate Class data 
>> sharing.
>>
>> $ dtrace -Zqn ''hotspot*:::class-loaded{ @loaded[arg2,
copyinstr(arg0,
>> arg1), arg3? "Shared":"Unique"] = count(); 
>> @total[arg3?"Shared":"Unique"] = count();}
END{printa("%-8d %40s %-10s
>> %8 at d\n", @loaded); printa("%-20s %@d\n", at
total);}''
>>
>> $ java -jar /usr/java/demo/jfc/SwingSet2/SwingSet2.jar
>>
>> Strangely all the classes where unique. Checked to see if I had a 
>> class archive, I do:
>> /usr/jdk/instances/jdk1.6.0/jre/lib/i386/client/classes.jsa
>>
>> Then looked to see what the hotspot VM thought:
>> bash-3.00$ java -help
>> Usage: java [-options] class [args...]
>>          :
>>    -client       to select the "client" VM
>>    -server       to select the "server" VM
>>    -hotspot      is a synonym for the "client" VM 
[deprecated]
>>                  The default VM is server,
>>                  because you are running on a server-class machine.
>>
>> So appears it thinks my new dual core Sony Vaio with 2 Gig Ram is a 
>> server :( This is due to the joys of Smart tuning introduced in J2SE 
>> 5.0, which I''m sure is a good thing but perhaps the Automatic
Server
>> detection needs revisiting, now that desktop machines are getting a 
>> lot beefier :)
>>
>> If I run [note the -client directive]:
>> $ java -client -jar /usr/java/demo/jfc/SwingSet2/SwingSet2.jar
>>
>> I now get over 1800 shared classes loaded as reported by the dtrace 
>> probe above. Unfortunately I now seem to crash even more often. Has 
>> the client VM option been as heavily tested on nevada as the default 
>> server VM option?
>>
>> JR
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
> 
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Ekaterina Pavlova,
VMSQE Team, St. Petersburg.
http://blogs.sun.com/vmrobot/
http://blogs.sun.com/theaquarium_ru/

Thanks Jim and Ekaterina - too late now for me to risk an upgrade as
I''m
travelling today and will do the demo tomorrow :) So think I will stick 
to describing the feature and demo the method-entry stuff instead for 
this part of the talk.

I''ll upgrade to 58 when I get home and post up any findings.

JR

Ekaterina Pavlova wrote:
> Hi John,
>
> I also observed the similar hangs on nevada 41
> (but I am not sure it is related to -Z).
> We upgraded to nevada 58 and seems there are no more hangs.
>
> -katya
>
> John Rice wrote:
>> I''ve been playing with this and it seems that the combination
of
>> using -Z and hotspot* makes the system hang a lot. I wanted to use 
>> these options so I could have the probe running before I started up 
>> different applications. Sticking to no -Z and hotspot$target seems to 
>> be a bit more stable.
>>
>> Curious if I''m the only one seeing this? Is it a nevada 56
issue?
>> JR
>>
>>
>> John Rice wrote:
>>
>>> Hi,
>>>
>>> I''ve been looking at the hotspot provider and wanted to
demo some of
>>> it''s features. Running on nevada 56 I''m getting a
lot of system
>>> hangs :( The system was rock solid until I started playing with the
>>> hotspot provider, it seems particularly unhappy about the 
>>> class-loaded probe for some reason, though it''s hung with
other
>>> method-entry probes. Anyone else seeing this?
>>>
>>> The following script was run, as I wanted to demonstrate Class data
>>> sharing.
>>>
>>> $ dtrace -Zqn ''hotspot*:::class-loaded{ @loaded[arg2, 
>>> copyinstr(arg0, arg1), arg3? "Shared":"Unique"]
= count();
>>> @total[arg3?"Shared":"Unique"] = count();}
END{printa("%-8d %40s
>>> %-10s %8 at d\n", @loaded); printa("%-20s %@d\n", at
total);}''
>>>
>>> $ java -jar /usr/java/demo/jfc/SwingSet2/SwingSet2.jar
>>>
>>> Strangely all the classes where unique. Checked to see if I had a 
>>> class archive, I do:
>>> /usr/jdk/instances/jdk1.6.0/jre/lib/i386/client/classes.jsa
>>>
>>> Then looked to see what the hotspot VM thought:
>>> bash-3.00$ java -help
>>> Usage: java [-options] class [args...]
>>>          :
>>>    -client       to select the "client" VM
>>>    -server       to select the "server" VM
>>>    -hotspot      is a synonym for the "client" VM 
[deprecated]
>>>                  The default VM is server,
>>>                  because you are running on a server-class machine.
>>>
>>> So appears it thinks my new dual core Sony Vaio with 2 Gig Ram is a
>>> server :( This is due to the joys of Smart tuning introduced in
J2SE
>>> 5.0, which I''m sure is a good thing but perhaps the
Automatic Server
>>> detection needs revisiting, now that desktop machines are getting a
>>> lot beefier :)
>>>
>>> If I run [note the -client directive]:
>>> $ java -client -jar /usr/java/demo/jfc/SwingSet2/SwingSet2.jar
>>>
>>> I now get over 1800 shared classes loaded as reported by the dtrace
>>> probe above. Unfortunately I now seem to crash even more often. Has
>>> the client VM option been as heavily tested on nevada as the
default
>>> server VM option?
>>>
>>> JR
>>> _______________________________________________
>>> dtrace-discuss mailing list
>>> dtrace-discuss at opensolaris.org
>>
>>
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>