CentOS - Dec 2010 - Server unresponsive until reboot, memory exhausted

I'm having an issue with an apache web server running the latest CentOS5
kernel (this issue is not new to the kernel). After a few days/weeks of
running the server will become unresponsive and will require a physical
reboot in order to come back online. The system is so unresponsive when
the issue occurs that login at console is not even possible. 

I have atop installed and have looked back before the crash to see what
happened process wise and I can see the http starts using a lot of
memory and CPU usage. The vmcommit jumps from 1.8 GB to 4.8GB in a
matter of a few minutes. The VSIZE of the httpd process jumps from 8.1GB
to 36.9GB. So apache is doing something -- but how can I get historical
data for this? I also see that paging is very active, probably why the 
server is unresponsive. I have looked through the apache logs and system 
logs and there is nothing
obvious that is consuming all that memory. I know of the server-status
module for apache but that is only useful if you can get to the server
during the crash (I can't) and doesn't have any historical data. 

The issue occurs seemingly randomly, last time in the middle of the night 
with little or no user traffic. 

James

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20101228/be4c7138/attachment-0001.html>

On Tue, Dec 28, 2010 at 9:59 AM, james <james at wintercastle.net>
wrote:
> I'm having an issue with an apache web server running the latest
CentOS5
> kernel (this issue is not new to the kernel). After a few days/weeks of
> running the server will become unresponsive and will require a physical
> reboot in order to come back online. The system is so unresponsive when the
> issue occurs that login at console is not even possible.
Do you have everything *else* updated? And what kind of web service
are you running?

There's a lot of third party freeware and commercial tools that was
not written with any kind of resource management in mind, and which
may require a simple web server restart on a regular baris to free
memory. (MusicBrainz: I remember porting MusicBrainz.....)
> I have atop installed and have looked back before the crash to see what
> happened process wise and I can see the http starts using a lot of memory
> and CPU usage. The vmcommit jumps from 1.8 GB to 4.8GB in a matter of a few
> minutes. The VSIZE of the httpd process jumps from 8.1GB to 36.9GB. So
> apache is doing something -- but how can I get historical data for this? I
Look in /var/log/http/*.
> also see that paging is very active, probably why the server is
> unresponsive. I have looked through the apache logs and system logs and
> there is nothing obvious that is consuming all that memory. I know of the
> server-status module for apache but that is only useful if you can get to
> the server during the crash (I can't) and doesn't have any
historical data.
>
> The issue occurs seemingly randomly, last time in the middle of the night
> with little or no user traffic.
Do you have a search engine scanning your web server?

>Do you have everything *else* updated? And what kind of web service
>are you running?
>There's a lot of third party freeware and commercial tools that was
>not written with any kind of resource management in mind, and which
>may require a simple web server restart on a regular baris to free
>memory. (MusicBrainz: I remember porting MusicBrainz.....)
Yes, all the packages are up to date. General web services -- static HTML, 
and the rest is mainly wordpress. 

You may be right about the restart, but I would like to know WHAT is 
crashing my web server regardless. We are not running any shiftily coded 
sites or apps on this server that I'm aware of (obviously something is 
shifty!). Is anyone aware of any other methods for drilling into the 
problem?
>Look in /var/log/http/*.
Yes, these are the web server logs I am referring to having checked. 
>Do you have a search engine scanning your web server?
Hmm, no, nothing systematic. The usual crawlers out there but nothing we 
are doing. 
_______________________________________________
CentOS mailing list
CentOS at centos.org
http://lists.centos.org/mailman/listinfo/centos

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20101228/572acd24/attachment-0001.html>