dtrace discuss - Mar 2008 - Strange Java/hibernate problem

Hi people, this is my first post to the list.

I''m trying to diagnose a weird performance problem.

I have a big sun server runing Solaris. On top of it
there''s a JBoss running an app that interacts with a
db.

The performance (to say it in a nice way) leaves a lot
to be desired but the server doesn''t look busy.

if we move the app to a linux server (runing in a WAY
smaller hardware) the performance increases a lot.
Like 6-10 times more transactions/minute.

The big server is runing Solaris 9, so we grabbed a
smaller Sun runing Solaris 10 and put the aplication
on top of it. We got the same ugly performance as with
Solaris 9. So it wasn''t the OS version and it wasn''t
the JBoss or aplication code (in linux it runs great).
It must be something Solaris-specific.

I did some basic dtracing and found that there are
some "yield storms" once every 5-10 seconds. That is,
the yield syscalls were normal, then they would climb
up to several thounsands.

I also did a basic profiling, like this

profile-1ms
/execname=="java"/
{
     @[jstack()]=count;
}

tick-30s
{
      exit(0);
}

most of the stack traces showed the application on
some hibernate (http://www.hibernate.org) method.

I also did dome lock tracing and saw many of the
threads spent a lot of time blocking on locks

The app is running on jdk 1.5, so I have no hotspot
provider. When we tried using the dvm and started
jboss, after 8 minutes it hadn''t finished the
deployement, so we cancelled it and tried using dtrace
"as it is". Today, we tried using jdk 1.6, but the
performance was even worse. Since this is a production
system, we had to pull it out before we could perform
any tests.

I guess the problem is in the native code
(specifically, some kind of contention there), but I
don''t know exactly how to narrow the search and where
to look (I''m more of an unix/tcpip sysadmin, my jva
knowledge is very limited).

Any suggestions, pointers, and such would be very
apreciated.


Fer


     
____________________________________________________________________________________
Looking for last minute shopping deals?  
Find them fast with Yahoo! Search. 
http://tools.search.yahoo.com/newsearch/category.php?category=shopping

On Wed, Mar 26, 2008 at 06:39:14PM -0700, Fernando Gleiser
wrote:
> Hi people, this is my first post to the list.
> 
> I''m trying to diagnose a weird performance problem.
> 
> [...]
> 
> Any suggestions, pointers, and such would be very
> apreciated.
Could you please start (and show us) with:
- prstat (while running your application)
- prstat -mL -p <pid of java process>

Regards
przemol

-- 
http://przemol.blogspot.com/





















----------------------------------------------------------------------
Poprowadz swoj klub do zwyciestwa!
Sprawdz >>> http://link.interia.pl/f1d63

This all sounds very familiar, I have had a similar problem but the net result
was to move to AMD h/w as single threaded speed was the issue. However I had
great improvemnts in runnign with a 32bit JVM and largepages.


--
This message posted from opensolaris.org

How many cores and how much physical memory do you have on
the three machines you were testing?   Have you tried turning
on the garbage collection logging?  I''d suggest
the VM command line flags

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps

That will log to stdout.  To redirect to a separate file,
you can use

-Xloggc:filename

You can reference this thread and send the logs to the public
mailing list

hotspot-gc-use at openjdk.java.net

and I''ll take a look.  Has any type of report been filed with 
Sun support?


--
This message posted from opensolaris.org