dtrace discuss - Apr 2006 - Improving performance of Java applications.

Hi All,

I read about a challenge happend in Java One conf last year on improving 
performance of Java applications at
http://emea.cobalt.com/software/solaris/javaone_challenge.jsp

Can anyone know what was the strategy followed to make this happen?  I 
mean people who ever volunteered there should have had a step by step plan.

I want to know who was volunteered for this and the approach they 
followed during that time.

Thanks,
Gangadhar.

Hi Gangadhar,

Jarod Jenson patiently used DTrace to find various wins last year. Perhaps
Jarod could be persuaded to comment on some of the techniques he used.
Alternatively, you can come to our talk about DTrace and Java at JavaOne
5/18/2006 at 2:45:

 
https://www28.cplan.com/javaone06_cv_124_1/session_details.jsp?isid=277311&ilocation_id=124-1&ilanguage=english

Adam

On Fri, Apr 21, 2006 at 11:55:38PM +0530, Gangadhar Mylapuram
wrote:
> Hi All,
> 
> I read about a challenge happend in Java One conf last year on improving 
> performance of Java applications at
> http://emea.cobalt.com/software/solaris/javaone_challenge.jsp
> 
> Can anyone know what was the strategy followed to make this happen?  I 
> mean people who ever volunteered there should have had a step by step plan.
> 
> I want to know who was volunteered for this and the approach they 
> followed during that time.
> 
> Thanks,
> Gangadhar.
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

Adam Leventhal''s email at 4/21/2006 1:55 PM,
said:
> Hi Gangadhar,
> 
> Jarod Jenson patiently used DTrace to find various wins last year. Perhaps
> Jarod could be persuaded to comment on some of the techniques he used.
> Alternatively, you can come to our talk about DTrace and Java at JavaOne
> 5/18/2006 at 2:45:
> 
Yep, I was the guy that got roped into this. We may even do it again 
this year. So the basic technique was to stand at the booth for 8 hours 
per day for several days straight and try to not rip the mic from the 
dude in the booth behind us that yelled the exact same annoying pitch 
every 15 minutes on the minute. The same joke gets old when heard 32 
times per day.

Seriously though, I have no specific step by step method to do this. 
Each app was wildly different and required a different approach. This is 
the beauty of DTrace. Never did I have to worry if there would be an 
application that we couldn''t profile and find _something_. With
systemic
observability, there will always be something (app, container, JVM, OS, 
etc.) that we can find some improvement.

All scripts were written on the fly, and I used absolutely no 
pre-written scripts. The sad part is that after JavaOne; the system was 
taken (actually it was pretty much taken while we were still working) 
and we lost all of the data including the scripts I did write (95% of 
hat I do is cmd line).

This year (if we do it), I plan on preserving these as there were 
several that were pretty dang nifty.

Over the past 2.5 years, I have used DTrace on over 200 distinct 
applications at more than 80 companies (I quit keeping track last year). 
It never gets boring. Every app is different, and there is always 
something we can find.

I wish I could say that there is some magic cookbook or silver bullet 
script, but I haven''t found it yet. All I can say is that you have to 
use DTrace often. Using DTrace is simple, learning how to approach 
problems with such a powerful tool is where the real trick is. It takes 
time.

Thanks,

Jarod

G''Day Folks,

On Fri, 21 Apr 2006, Jarod Jenson wrote:
[...]
> I wish I could say that there is some magic cookbook or silver bullet
> script, but I haven''t found it yet. All I can say is that you have
to
> use DTrace often. Using DTrace is simple, learning how to approach
> problems with such a powerful tool is where the real trick is. It takes
> time.
Excellent description! I''ve tried to address this point in the
DTraceToolkit FAQ, by saying that while these scripts can help, they
aren''t magical - you still need experience and understanding. I need to
go further and discuss the mentality required.

There have been times when I''ve used a DTraceToolkit script to solve a
client problem directly. Great. But there have been other times when the
scripts only confirm that the system is performing well, and the
client-obvious value was had from the application specific scripts and
one-liners I wrote on-site. (you can argue that checking the system and
finding nothing has value, and it does; but nothing beats finding _the_
problem itself).

If you ever see me teach DTrace, I focus on convaying mentality - the
different approaches that you can take when tackling a mystery problem.
I encourage students to bring in their application problems to class,
so that we can discuss mentality as a group. It''s crucial. If I taught
a
syntax-only class (which would be easy), many students would go back to
work and realise they still don''t know how to go about tackling
problems.
Focusing only on existing scripts (the DTraceToolkit), wouldn''t be
right
either. Using DTrace is mostly about mentality, strategy, approach,
understanding, experience and coffee.

There are some great performance observability mentalities documented in
Solaris Internals 2nd ed, vol 2 (not yet available), many of which make
use of DTrace (and many of which I wrote :).

I do plan on writing some silver-bullet-like scripts (eg, a system
dashboard), but I''m planning on being very careful on how they are
presented. I''ll make it very, very clear that they are not the be all
and end all of performance monitoring.

The silver bullet to performance problems is DTrace itself. My scripts
give you different ways to fire it, but you need to know what to aim at.

cheers,

Brendan

[Sydney, Australia]

Jarod,

Java application is a userland process(JVM). There are D-trace java 
agents to observe the flow of application.
I am not looking for the scripts. Scripts we can write on the fly based 
on requriement. What exactly I am
looking for is the approach.

For example, if we say a performance issue, few things we generally look 
for are
memory leaks, resource availability (memory ) , load on the system like 
that...

For Java applications what do we look for.

Basically I want to avoid re-inventing the wheel.

Thanks,
Gangadhar.

Jarod Jenson wrote:
>
>
> Adam Leventhal''s email at 4/21/2006 1:55 PM, said:
>
>> Hi Gangadhar,
>>
>> Jarod Jenson patiently used DTrace to find various wins last year. 
>> Perhaps
>> Jarod could be persuaded to comment on some of the techniques he used.
>> Alternatively, you can come to our talk about DTrace and Java at
JavaOne
>> 5/18/2006 at 2:45:
>>
>
> Yep, I was the guy that got roped into this. We may even do it again 
> this year. So the basic technique was to stand at the booth for 8 
> hours per day for several days straight and try to not rip the mic 
> from the dude in the booth behind us that yelled the exact same 
> annoying pitch every 15 minutes on the minute. The same joke gets old 
> when heard 32 times per day.
>
> Seriously though, I have no specific step by step method to do this. 
> Each app was wildly different and required a different approach. This 
> is the beauty of DTrace. Never did I have to worry if there would be 
> an application that we couldn''t profile and find _something_. With
> systemic observability, there will always be something (app, 
> container, JVM, OS, etc.) that we can find some improvement.
>
> All scripts were written on the fly, and I used absolutely no 
> pre-written scripts. The sad part is that after JavaOne; the system 
> was taken (actually it was pretty much taken while we were still 
> working) and we lost all of the data including the scripts I did write 
> (95% of hat I do is cmd line).
>
> This year (if we do it), I plan on preserving these as there were 
> several that were pretty dang nifty.
>
> Over the past 2.5 years, I have used DTrace on over 200 distinct 
> applications at more than 80 companies (I quit keeping track last 
> year). It never gets boring. Every app is different, and there is 
> always something we can find.
>
> I wish I could say that there is some magic cookbook or silver bullet 
> script, but I haven''t found it yet. All I can say is that you have
to
> use DTrace often. Using DTrace is simple, learning how to approach 
> problems with such a powerful tool is where the real trick is. It 
> takes time.
>
> Thanks,
>
> Jarod
>
>
>
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

On 4/24/06, Gangadhar Mylapuram <Gangadhar.M at sun.com>
wrote:
> Jarod,
>
> Java application is a userland process(JVM). There are D-trace java
> agents to observe the flow of application.
> I am not looking for the scripts. Scripts we can write on the fly based
> on requriement. What exactly I am
> looking for is the approach.
>
> For example, if we say a performance issue, few things we generally look
> for are
> memory leaks, resource availability (memory ) , load on the system like
> that...
>
> For Java applications what do we look for.
>
> Basically I want to avoid re-inventing the wheel.
Jarod addresses some of this in his ACM Queue interview with Kirk McCusick:

http://acmqueue.com/modules.php?name=Content&pa=showpage&pid=363&page=5

Pages 5 and 6 have the most interesting information WRT Java (although
the whole article''s worth reading.)

It seems to me the best way to know what to look for is to know how
the JVM works, at least to some extent.  For example, if you know that
the JVM does garbage collection, and you know that garbage collection
is an expensive operation, then you know that looking at object
allocations might turn up something interesting.

Having said that, though, I''d love to hear anything more Jarod would
like to add, as I need to learn something about using DTrace to look
at the performance of Java applications, too.

Chad Mynhier