Hello, Over the weekend, I upgraded one of my servers that runs mysql and pdns to 5.3. Previous to the update I have not had an issue form this server. But since, I have had mysql die multiple times from oom-killer. $ uname -a Linux rack2a 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux Unfortunately, the server has been rebooted by others before I have been able to look at it while the problem is occurring. But I have found this in the logs: http://pastebin.centos.org/25553 Running sar -A: http://pastebin.centos.org/25556 Since the server was rebooted, the amount of swap is used is 0. How do I determine what process is/was chewing up the memory on this server? What should I be looking for to narrow this down? Thanks, Rick
Rick Barnes wrote:> How do I determine what process is/was chewing up the memory on this > server? What should I be looking for to narrow this down?Setup a script to monitor memory usage and alert you when swap usage starts getting high. Or setup a monitor that just runs ps auxw or something and sends output to a file so the next time it happens you know how much memory everything was using at the time. nate
On Mon, 2009-04-13 at 10:31 -0400, Rick Barnes wrote:> Hello, > > Over the weekend, I upgraded one of my servers that runs mysql and pdns > to 5.3. Previous to the update I have not had an issue form this server. > But since, I have had mysql die multiple times from oom-killer. > > $ uname -a > Linux rack2a 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 > x86_64 x86_64 x86_64 GNU/Linux > > Unfortunately, the server has been rebooted by others before I have been > able to look at it while the problem is occurring. But I have found this > in the logs: > > http://pastebin.centos.org/25553 > > Running sar -A: > > http://pastebin.centos.org/25556 > > Since the server was rebooted, the amount of swap is used is 0. > > How do I determine what process is/was chewing up the memory on this > server? What should I be looking for to narrow this down? > > Thanks, > Rick--- You found your problem already. MYSQL. There could be something helping it also. When SQL server begins running out of RAM there could be many things to blame on it doing so. What actually does mysql do? As in what do you use it for. Try ps ?auxf | grep mysqld. Something else you can use is "mytop". You can use the below as a script and it works in an infinite loop or just use the command only. What you need to see is the running queries. ################## #!/bin/bash while [ 1 ] do mysql -N -u root -ppassword -e 'show processlist' |grep -v 'show processlist' sleep 2 done ################## JohnStanley
On Mon, 2009-04-13 at 10:31 -0400, Rick Barnes wrote:> Hello, > > I have had mysql die multiple times from oom-killer. >> Thanks, > Rick--- In seeing what you do I would be looking at the queries being made. SPROCs, Triggers, Views. Also very import is if that one particular data base is doing graffing "running predictions" as you all do GIS and Mapping. Predictions can be huge memory burners. Last thing does it run as a virtual machine or on real hardware? JohnStanley