In our environment we have many legacy application servers running apache/jserv. There is a web server front end, then a couple of load-balanced java servers on the backside. One of the problems we are faced with is hung or stuck jvms. I have looked at the java process with the ps command, and there are many times when URL(s) do not respond, yet the java looks healthy, at least from the OS point of view. The usual cure for this situation is to restart the JVM, then the URLs come right back up. Are any of you aware of tools for monitoring apache jserv, either from localhost or by connecting to port 8008 over the network? I really want to find out if there is a way to detect a "sick" JVM other than getting a bunch of down URL alerts on my phone.
Sean Carolan wrote:> Are any of you aware of tools for monitoring apache jserv, either from > localhost or by connecting to port 8008 over the network? I really > want to find out if there is a way to detect a "sick" JVM other than > getting a bunch of down URL alerts on my phone.How about setting up a cron to monitor it and auto restart if it's not responding? wget -q --timeout=30 http://localhost:8008/ -O /dev/null || (command to restart jserv) I'm not familiar with jserv but it looks like it serves a purpose similar to tomcat, just a much older code base. I've used similar scripts for tomcat in the past, such as this cron */5 * * * * /bin/grep -q \"java.lang.OutOfMemoryError: Java heap space\" /logs/tomcat/web/1/run.latest.log 1>/dev/null 2>&1 && export MYHOST=`hostname -s` && echo \"Restarting Tomcat on $MYHOST - OutOfMemoryError\" | mail -s \"[$MYHOST] Restarting Tomcat due to OutOfMemoryError\" my at email.address && /etc/init.d/appname restart 1>/dev/null 2>&1 When the app had a memory leak it helped me sleep better at night, as the apps were restarting about 10x a day until the developers found the source of the leak. Fortunately the app restarted reliably 99.9% of the time. Another place I worked at before that had an app that was so poor that you had to babysit it all the time, the number of things that could go wrong was really too much work for us to automate it in a reasonable amount of time. Ideally the "load balancer" should be performing "health checks" against the back end nodes and stop sending traffic to them if they are no longer healthy. I'm not familiar with the load balancing method your using, only with bigger load balancers like F5 BigIP. nate
> In our environment we have many legacy application servers runningapache/jserv. There is a web server front end, then a couple of load-balanced java servers on the backside. One of the problems we are faced with is hung or stuck jvms. I have looked at the java process with the ps command, and there are many times when URL(s) do not respond, yet the java looks healthy, at least from the OS point of view. The usual cure for this situation is to restart the JVM, then the URLs come right back up.> > Are any of you aware of tools for monitoring apache jserv, either fromlocalhost or by connecting to port 8008 over the network? I really want to find out if there is a way to detect a "sick" JVM other than getting a bunch of down URL alerts on my phone. Hello, I don't know jserv or about eventual specific jserv support, but Hyperic might be part of the answer. I know it can provide metrics about Tomcat and JVMs, and application/server specific plugins can be written without too much effort. It's by far the most powerful/complete monitoring solution I've seen, providing support out of the box or with plugins for a bunch of services and applications. Check http://support.hyperic.com/confluence/display/hypcomm/HyperForge/#HyperFORGE-pluginforge for existing plugins. Perhaps what you want can be done with a JMX plugin ? Really worth a try anyway...
On Tue, Jun 10, 2008 at 5:07 PM, Sean Carolan <scarolan at gmail.com> wrote: <snip>> Are any of you aware of tools for monitoring apache jserv, either from > localhost or by connecting to port 8008 over the network? I really > want to find out if there is a way to detect a "sick" JVM other than > getting a bunch of down URL alerts on my phone.The Sun JDK provides tools for pulling info from a running JVM. http://java.sun.com/javase/6/webnotes/trouble/other/tools6-Unix.html Not jserv (time for an upgrade?) specific, but attaches directly to the underlying JVM. The one I've played with is jmap. Of course, you need to be running a recent JVM. I do not proclaim to be an expert with these tools, but it may be of some use in your situation. -- Jeff
Sean Carolan wrote:> In our environment we have many legacy application servers running > apache/jserv. There is a web server front end, then a couple of > load-balanced java servers on the backside. One of the problems we > are faced with is hung or stuck jvms. I have looked at the java > process with the ps command, and there are many times when URL(s) do > not respond, yet the java looks healthy, at least from the OS point of > view. The usual cure for this situation is to restart the JVM, then > the URLs come right back up. > > Are any of you aware of tools for monitoring apache jserv, either from > localhost or by connecting to port 8008 over the network? I really > want to find out if there is a way to detect a "sick" JVM other than > getting a bunch of down URL alerts on my phone. >Being far from an expert, if restarting a service or script is what's needed, you might find SIM[1] helpful YMMV, -R [1]http://rfxnetworks.com/sim.php