Hello I have a centos Xen virtual server that runs shorewall(version 4.4.2.3). It seems like every few weeks my server freezes. This means I cannot ssh into it or ping it. My service provider suggested that I run a logging program using cron to see if I can find out what is going wrong. So I have written a cron job that runs every minute and runs top in batch mode. My batch mode top outputs its results every 5 seconds for 55 seconds The server has frozen again. After rebooting from the Control panel I have examined my top logs and have discovered that the last log has a zombie process. ie 8408 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 iptables <defunct> The top of Top displays as top - 09:30:02 up 6 days, 1:27, 1 user, load average: 0.00, 0.03, 0.00 Tasks: 71 total, 3 running, 67 sleeping, 0 stopped, 1 zombie Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 98.8%id, 0.6%wa, 0.0%hi, 0.0%si, 0.3%st Mem: 384.191M total, 379.086M used, 5228.000k free, 142.023M buffers Swap: 1023.992M total, 36.000k used, 1023.957M free, 151.355M cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8336 root 20 0 11672 9620 1648 R 40.0 2.4 0:00.20 perl Notice that there is a perl process taking 40% of the cpu. Immediately after this top output the whole virtual server froze. (This was the last Top output until virtual server reboot) I have attached the last top output file, and also the previous log file for comparison purposes. I have also attached a file from /var/log/messages which shows output immediately before and after the freeze. My shorewall is ordinary except that it reboots every 5 minutes(using cron) because a user does not have a static ip address but they do have a dns address. My rules file has a reference to this dns address, rather than an ip address. Note that the last messages output timestamp matches quite closely with the last top output. messages > Jun 12 09:30:02 vessel2 kernel: [523658.639435] Advised path = 120.138.19.35 -> 120.138.18.100 Top logfile > top - 09:30:02 up 6 days, 1:27, 1 user, load average: 0.00, 0.03, 0.00 This may point towards iptables stopping top(and everything else) in some way. I do not know why my server is freezing, but I am assuming it is because iptables has run amok and has locked up the kernel. Is this a good assumption? Is it a shorewall problem or an iptables problem? What can I do about it? Are my shorewall restarts causing the problem? I wanted a static ip address but I wasn''t able to get one for this user. I don''t have physical access to the host server. My provider has said that my virtual server is taking about 50% of the cpu, but they can''t see into the server much more than that. Thank you Peter McGregor ------------------------------------------------------------------------------ EditLive Enterprise is the world''s most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
> Are my shorewall restarts causing the problem? > I wanted a static ip address but I wasn''t able to get one for this user.Can you use their MAC address? ------------------------------------------------------------------------------ EditLive Enterprise is the world''s most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
On 17/06/11 11:56, David Watkins wrote:>> Are my shorewall restarts causing the problem? >> I wanted a static ip address but I wasn''t able to get one for this user. > > Can you use their MAC address? >If you are running dhcp on your server, you can make the dhcp server assign a static ip address to their particular MAC address. Ian. ------------------------------------------------------------------------------ EditLive Enterprise is the world''s most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
On Fri, 2011-06-17 at 22:22 +1200, peter mcgregor wrote:> > This may point towards iptables stopping top(and everything else) in > some way.> > I do not know why my server is freezing, but I am assuming it is because > iptables has run amok and has locked up the kernel. > > Is this a good assumption?I doubt it.> > Is it a shorewall problem or an iptables problem?Not enough information to say.> > What can I do about it?Have you configured a STARTUP_LOG in shorewall.conf? If not, you should so you can see all of the Shorewall messages being generated at the time of the hang. I would also recommend upgrading to the latest version of Shorewall, as yours is almost two years old. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ EditLive Enterprise is the world''s most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
On 17/06/11 10:56 PM, David Watkins wrote:>> Are my shorewall restarts causing the problem? >> I wanted a static ip address but I wasn''t able to get one for this user. > Can you use their MAC address? >It is a good suggestion but I thought about this for a while, did some investigation and realised that mac address''s can be easily spoofed. cheers ------------------------------------------------------------------------------ EditLive Enterprise is the world''s most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
On 17/06/11 10:56 PM, David Watkins wrote:>> Are my shorewall restarts causing the problem? >> I wanted a static ip address but I wasn''t able to get one for this user. > Can you use their MAC address?It is a good suggestion but I thought about this for a while, did some investigation and realised that mac address''s can be easily spoofed. cheers ------------------------------------------------------------------------------ EditLive Enterprise is the world''s most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
On 18/06/11 1:06 AM, Ian Barton wrote:> On 17/06/11 11:56, David Watkins wrote: >>> Are my shorewall restarts causing the problem? >>> I wanted a static ip address but I wasn''t able to get one for this user. >> Can you use their MAC address? >> > If you are running dhcp on your server, you can make the dhcp server > assign a static ip address to their particular MAC address. > > Ian. >Its not my server and for some reason their ISP does not want to give out a static ip address. I''ve been told that it may be because the country is very controlled. eg they are not allowed to use skype. Peter ------------------------------------------------------------------------------ EditLive Enterprise is the world''s most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
> >> What can I do about it? > Have you configured a STARTUP_LOG in shorewall.conf? If not, you > should so you can see all of the Shorewall messages being generated at > the time of the hang. I would also recommend upgrading to the latest > version of Shorewall, as yours is almost two years old. > > -Tom >I''ve upgraded shorewall to the latest version and created a startup log as suggested. I''ve currently stopped the firewall restarts as the dynamic ip address won''t be used for about the next 6 months. It also helps to verify that it is a firewall related problem(Although I am reasonably sure of this.) In 6 months time I may have to come up with another solution. cheers peter ------------------------------------------------------------------------------ EditLive Enterprise is the world''s most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev