Hi all,
I'm running CentOS 5.5 Final, Java version "1.6.0_17" OpenJDK
Runtime
Environment (IcedTea6 1.7.5) (rhel-1.16.b17.el5-x86_64) OpenJDK 64-Bit
Server VM (build 14.0-b16, mixed mode) installed via Yum.
We have a java application, packaged as a jar, running on our servers
which, periodically, crawls RSS feeds and writes the articles to a
database.
Randomly, and seemingly without cause, these processes will die, not
through the application exiting, or due to my killing it, but due to
something that seems to kill without leaving a trace.
My first step in diagnosing this was to log all output from the
application, as well as sending stderr and stdout to a logfile, but
none of these output logs contain anything that would indicate why
these processes have died.
My next instinct was the kernel-level out of memory killer, but the
system is never low on memory (8GB installed, routinely showing 6.5GB
in free cache) and usually has somewhere between 1GB and 3GB of memory
free at any given point in time.
I next thought the system could be hitting bad memory, segfaulting,
and killing the process because of that, but I've mirrored the system
on an identically configured server in a different datacentre, and the
processes are still being killed.
The java virtual machine does trigger JVM core dumps on exit, so the
process is being killed by something, but the JVM dumps don't have any
useful information.
My question is: does anyone know what might be causing it, and where I
should start looking to diagnose the cause?
Thanks.