Felix Botner
2010-Jun-03 08:05 UTC
[Xen-users] Xen 3.4.3 - randomly crash of Dom0 and DomU
Hi everyone,
from time to time (but at least one crash in 24h) the system crashes (neither
Dom0 nor DomU is accessible via network or console) with no usefull
information in any log file and i have no idea what the problem could be.
Log files (syslog, kern.log, messages, xend.log) on both systems, Dom0 and
DomU, show nothing, no kernel oops or anything.
The setup is:
* a Dom0 (64Bit) with a 2.6.32-xen, Debian Sid based kernel
(linux-source-2.6.32 2.6.32-13) with xen-3.4.3 (build with
the standard options)
* and one paravirtualized Debian Etch based linux DomU (32Bit) with a
Debian Etch based kernel 2.6.18-xen (also tried other Debian Lenny based
DomU Kernels, 2.6.26-xen and 2.6.30-xen)
* DomU''s network interface is connected via bridge
* xen.independent_wallclock=1 and ntpd in DomU (due to many
"clocksource/2: Time went backwards" log messages)
DomU cgf:
name = ''name''
kernel = "/opt/ucs2.2_xen/vmlinuz-2.6.18-ucs153-xen-686"
memory = 2500
vcpus = 1
disk = [''file:/var/lib/xen/name/disk,xvda,w'' ]
vif = [ ''mac=00:16:3e:8e:8b:08'' ]
boot = ''d''
ramdisk = "/opt/ucs2.2_xen/initrd.img-2.6.18-ucs153-xen-686"
root = "/dev/xvda1 ro"
extra = ''xencons=tty console=hvc0''
Dom0 /etc/xen/xend-config.sxp:
(xen-api-server ((unix)))
(xend-http-server no)
(xend-unix-server no)
(xend-unix-xmlrpc-server yes)
(xend-tcp-xmlrpc-server no)
(xend-relocation-server no)
(xend-relocation-ssl-server no)
(network-script network-bridge)
(vif-script vif-bridge)
(dom0-min-mem 196)
(enable-dom0-ballooning yes)
(dom0-cpus 0)
(vncpasswd '''')
Is there any known problem running a 32Bit DomU in a 64 Bit Dom0?
Could anyone give me a hint, how to debug this situation or where i can find
more information?
Feel free to ask for more information about the system or the setup.
Many thanks
--
Felix Botner
Open Source Software Engineer
Univention GmbH
Linux for your business
Mary-Somerville-Str.1
28359 Bremen
Tel. : +49 421 22232-0
Fax : +49 421 22232-99
botner@univention.de>
http://www.univention.de
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Felix Botner
2010-Jun-07 07:58 UTC
[Xen-users] Xen 3.4.3 - randomly crash of Dom0 and DomU
Hi everyone,
from time to time (but at least one crash in 24h) the system crashes (neither
Dom0 nor DomU is accessible via network or console) with no usefull
information in any log file and i have no idea what the problem could be.
Log files (syslog, kern.log, messages, xend.log) on both systems, Dom0 and
DomU, show nothing, no kernel oops or anything.
The setup is:
* a Dom0 (64Bit) with a 2.6.32-xen, Debian Sid based kernel
(linux-source-2.6.32 2.6.32-13) with xen-3.4.3 (build with
the standard options)
* and one paravirtualized Debian Etch based linux DomU (32Bit) with a
Debian Etch based kernel 2.6.18-xen (also tried other Debian Lenny based
DomU Kernels, 2.6.26-xen and 2.6.30-xen)
* DomU''s network interface is connected via bridge
DomU cgf:
name = ''name''
kernel = "/opt/ucs2.2_xen/vmlinuz-2.6.18-ucs153-xen-686"
memory = 2500
vcpus = 1
disk = [''file:/var/lib/xen/name/disk,xvda,w'' ]
vif = [ ''mac=00:16:3e:8e:8b:08'' ]
boot = ''d''
ramdisk = "/opt/ucs2.2_xen/initrd.img-2.6.18-ucs153-xen-686"
root = "/dev/xvda1 ro"
extra = ''xencons=tty console=hvc0''
Dom0 /etc/xen/xend-config.sxp:
(xen-api-server ((unix)))
(xend-http-server no)
(xend-unix-server no)
(xend-unix-xmlrpc-server yes)
(xend-tcp-xmlrpc-server no)
(xend-relocation-server no)
(xend-relocation-ssl-server no)
(network-script network-bridge)
(vif-script vif-bridge)
(dom0-min-mem 196)
(enable-dom0-ballooning yes)
(dom0-cpus 0)
(vncpasswd '''')
grub/menu.lst
title Xen 3.4.3 / UCS, kernel 2.6.32-ucs5-xen-amd64
root (hd0,0)
kernel /xen-3.4.3.gz dom0_mem=3072M
module /vmlinuz-2.6.32-ucs5-xen-amd64
root=/dev/mapper/vg_ucs-rootfs ro root2fstype=ext3
splash=silent video=vga16:off nosplash rootdelay=5
clock=pit loglevel=0 nopat console=tty0 fbcon=map:2
module /initrd.img-2.6.32-ucs5-xen-amd64
Is there any known problem running a 32Bit DomU in a 64 Bit Dom0?
Could anyone give me a hint, how to debug this situation or where i can find
more information?
Feel free to ask for more information about the system or the setup.
Many thanks
--
Felix Botner
Open Source Software Engineer
Univention GmbH
Linux for your business
Mary-Somerville-Str.1
28359 Bremen
Tel. : +49 421 22232-0
Fax : +49 421 22232-99
botner@univention.de>
http://www.univention.de
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Felix Botner
2010-Jun-08 09:06 UTC
[Xen-users] Xen 3.4.3 - randomly crash of Dom0 and DomU
Hi everyone,
from time to time (but at least one crash in 24h) the system crashes (neither
Dom0 nor DomU is accessible via network or console) with no usefull
information in any log file and i have no idea what the problem could be.
Log files (syslog, kern.log, messages, xend.log) on both systems, Dom0 and
DomU, show nothing, no kernel oops or anything.
The setup is:
* a Dom0 (64Bit) with a 2.6.32-xen, Debian Sid based kernel
(linux-source-2.6.32 2.6.32-13) with xen-3.4.3 (build with
the standard options)
* and one paravirtualized Debian Etch based linux DomU (32Bit) with a
Debian Etch based kernel 2.6.18-xen (also tried other Debian Lenny based
DomU Kernels, 2.6.26-xen and 2.6.30-xen)
* DomU''s network interface is connected via bridge
* tomcat/java are running on both systems
DomU cgf:
name = ''name''
kernel = "/opt/ucs2.2_xen/vmlinuz-2.6.18-ucs153-xen-686"
memory = 2500
vcpus = 1
disk = [''file:/var/lib/xen/name/disk,xvda,w'' ]
vif = [ ''mac=00:16:3e:8e:8b:08'' ]
boot = ''d''
ramdisk = "/opt/ucs2.2_xen/initrd.img-2.6.18-ucs153-xen-686"
root = "/dev/xvda1 ro"
extra = ''xencons=tty console=hvc0''
Dom0 /etc/xen/xend-config.sxp:
(xen-api-server ((unix)))
(xend-http-server no)
(xend-unix-server no)
(xend-unix-xmlrpc-server yes)
(xend-tcp-xmlrpc-server no)
(xend-relocation-server no)
(xend-relocation-ssl-server no)
(network-script network-bridge)
(vif-script vif-bridge)
(dom0-min-mem 196)
(enable-dom0-ballooning yes)
(dom0-cpus 0)
(vncpasswd '''')
grub/menu.lst
title Xen 3.4.3 / UCS, kernel 2.6.32-ucs5-xen-amd64
root (hd0,0)
kernel /xen-3.4.3.gz dom0_mem=3072M
module /vmlinuz-2.6.32-ucs5-xen-amd64
root=/dev/mapper/vg_ucs-rootfs ro root2fstype=ext3
splash=silent video=vga16:off nosplash rootdelay=5
clock=pit loglevel=0 nopat console=tty0 fbcon=map:2
module /initrd.img-2.6.32-ucs5-xen-amd64
I also have a script running, collecting the output of ps aux, free , ... and
everything seems ok except this (but ps aux is showing this even hours before
the crash):
root 8661 0.0 0.4 95988 10364 ? S Jun05 0:00 /usr/bin/python2.5
/usr/sbin/xend start
root 8665 10449177 1.0 221868 22708 ? Sl Jun05
21120206:30 /usr/bin/python2.5 /usr/sbin/xend star
Is there any known problem running a 32Bit DomU in a 64 Bit Dom0?
Could anyone give me a hint, how to debug this situation or where i can find
more information?
Feel free to ask for more information about the system or the setup.
Many thanks
--
Felix Botner
Open Source Software Engineer
Univention GmbH
Linux for your business
Mary-Somerville-Str.1
28359 Bremen
Tel. : +49 421 22232-0
Fax : +49 421 22232-99
botner@univention.de>
http://www.univention.de
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
-------------------------------------------------------
--
Felix Botner
Open Source Software Engineer
Univention GmbH
Linux for your business
Mary-Somerville-Str.1
28359 Bremen
Tel. : +49 421 22232-0
Fax : +49 421 22232-99
botner@univention.de>
http://www.univention.de
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Jonathan Tripathy
2010-Jun-15 06:52 UTC
Re: [Xen-users] Xen 3.4.3 - randomly crash of Dom0 and DomU
On 07/06/10 08:58, Felix Botner wrote:> Hi everyone, > > from time to time (but at least one crash in 24h) the system crashes (neither > Dom0 nor DomU is accessible via network or console) with no usefull > information in any log file and i have no idea what the problem could be. > Log files (syslog, kern.log, messages, xend.log) on both systems, Dom0 and > DomU, show nothing, no kernel oops or anything. > > The setup is: > > * a Dom0 (64Bit) with a 2.6.32-xen, Debian Sid based kernel > (linux-source-2.6.32 2.6.32-13) with xen-3.4.3 (build with > the standard options) > > * and one paravirtualized Debian Etch based linux DomU (32Bit) with a > Debian Etch based kernel 2.6.18-xen (also tried other Debian Lenny based > DomU Kernels, 2.6.26-xen and 2.6.30-xen) > > * DomU''s network interface is connected via bridge > > DomU cgf: > name = ''name'' > kernel = "/opt/ucs2.2_xen/vmlinuz-2.6.18-ucs153-xen-686" > memory = 2500 > vcpus = 1 > disk = [''file:/var/lib/xen/name/disk,xvda,w'' ] > vif = [ ''mac=00:16:3e:8e:8b:08'' ] > boot = ''d'' > ramdisk = "/opt/ucs2.2_xen/initrd.img-2.6.18-ucs153-xen-686" > root = "/dev/xvda1 ro" > extra = ''xencons=tty console=hvc0'' > > Dom0 /etc/xen/xend-config.sxp: > (xen-api-server ((unix))) > (xend-http-server no) > (xend-unix-server no) > (xend-unix-xmlrpc-server yes) > (xend-tcp-xmlrpc-server no) > (xend-relocation-server no) > (xend-relocation-ssl-server no) > (network-script network-bridge) > (vif-script vif-bridge) > (dom0-min-mem 196) > (enable-dom0-ballooning yes) > (dom0-cpus 0) > (vncpasswd '''') > > grub/menu.lst > title Xen 3.4.3 / UCS, kernel 2.6.32-ucs5-xen-amd64 > root (hd0,0) > kernel /xen-3.4.3.gz dom0_mem=3072M > module /vmlinuz-2.6.32-ucs5-xen-amd64 > root=/dev/mapper/vg_ucs-rootfs ro root2fstype=ext3 > splash=silent video=vga16:off nosplash rootdelay=5 > clock=pit loglevel=0 nopat console=tty0 fbcon=map:2 > module /initrd.img-2.6.32-ucs5-xen-amd64 > > > Is there any known problem running a 32Bit DomU in a 64 Bit Dom0? > Could anyone give me a hint, how to debug this situation or where i can find > more information? > > Feel free to ask for more information about the system or the setup. > > Many thanks > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-usersSo I did a test to see if I could replicate what Fexlix was experiencing. I left 3 DomUs running (2PV and 1 HVM) on my Ubuntu 10.04 Dom0 running overnight (with 2 of them pinging an internet server, the other one was a pfsense firewall running in HVM), and in the morning the server had indeed crashed (Had to hard reset it). The kernel I''m using is Jerem''y 2.6.32.15 pvops kernel. Both Dom0 and DomUs are 64-bit. I''m using Xen 3.4.3. Xend.log says "Xend exited with status 0)" at around 4:12am in the morning. The last thing that Xend-debug says is this (no time given): 1. Exception in thread HttpServer: 2. Traceback (most recent call last): 3. File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner 4. self.run() 5. File "/usr/lib/python2.6/threading.py", line 484, in run 6. self.__target(*self.__args, **self.__kwargs) 7. File "/usr/lib/python2.6/dist-packages/xen/web/httpserver.py", line 312, in run 8. (sock, addr) = self.accept() 9. File "/usr/lib/python2.6/dist-packages/xen/web/httpserver.py", line 331, in accept 10. return self.socket.accept() 11. File "/usr/lib/python2.6/socket.py", line 195, in accept 12. sock, addr = self._sock.accept() 13. error: [Errno 22] Invalid argument Any help is appreciated I should probably mention that for some reason when my system starts up, or when I try and start Xend manually, the init script launch 2 xend processes. Could this have caused the problem? Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jonathan Tripathy
2010-Jun-15 07:07 UTC
Re: [Xen-users] Xen 3.4.3 - randomly crash of Dom0 and DomU
On 15/06/10 07:52, Jonathan Tripathy wrote:> > On 07/06/10 08:58, Felix Botner wrote: >> Hi everyone, >> >> from time to time (but at least one crash in 24h) the system crashes (neither >> Dom0 nor DomU is accessible via network or console) with no usefull >> information in any log file and i have no idea what the problem could be. >> Log files (syslog, kern.log, messages, xend.log) on both systems, Dom0 and >> DomU, show nothing, no kernel oops or anything. >> >> The setup is: >> >> * a Dom0 (64Bit) with a 2.6.32-xen, Debian Sid based kernel >> (linux-source-2.6.32 2.6.32-13) with xen-3.4.3 (build with >> the standard options) >> >> * and one paravirtualized Debian Etch based linux DomU (32Bit) with a >> Debian Etch based kernel 2.6.18-xen (also tried other Debian Lenny based >> DomU Kernels, 2.6.26-xen and 2.6.30-xen) >> >> * DomU''s network interface is connected via bridge >> >> DomU cgf: >> name = ''name'' >> kernel = "/opt/ucs2.2_xen/vmlinuz-2.6.18-ucs153-xen-686" >> memory = 2500 >> vcpus = 1 >> disk = [''file:/var/lib/xen/name/disk,xvda,w'' ] >> vif = [ ''mac=00:16:3e:8e:8b:08'' ] >> boot = ''d'' >> ramdisk = "/opt/ucs2.2_xen/initrd.img-2.6.18-ucs153-xen-686" >> root = "/dev/xvda1 ro" >> extra = ''xencons=tty console=hvc0'' >> >> Dom0 /etc/xen/xend-config.sxp: >> (xen-api-server ((unix))) >> (xend-http-server no) >> (xend-unix-server no) >> (xend-unix-xmlrpc-server yes) >> (xend-tcp-xmlrpc-server no) >> (xend-relocation-server no) >> (xend-relocation-ssl-server no) >> (network-script network-bridge) >> (vif-script vif-bridge) >> (dom0-min-mem 196) >> (enable-dom0-ballooning yes) >> (dom0-cpus 0) >> (vncpasswd '''') >> >> grub/menu.lst >> title Xen 3.4.3 / UCS, kernel 2.6.32-ucs5-xen-amd64 >> root (hd0,0) >> kernel /xen-3.4.3.gz dom0_mem=3072M >> module /vmlinuz-2.6.32-ucs5-xen-amd64 >> root=/dev/mapper/vg_ucs-rootfs ro root2fstype=ext3 >> splash=silent video=vga16:off nosplash rootdelay=5 >> clock=pit loglevel=0 nopat console=tty0 fbcon=map:2 >> module /initrd.img-2.6.32-ucs5-xen-amd64 >> >> >> Is there any known problem running a 32Bit DomU in a 64 Bit Dom0? >> Could anyone give me a hint, how to debug this situation or where i can find >> more information? >> >> Feel free to ask for more information about the system or the setup. >> >> Many thanks >> >> >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users > So I did a test to see if I could replicate what Fexlix was experiencing. > > I left 3 DomUs running (2PV and 1 HVM) on my Ubuntu 10.04 Dom0 running > overnight (with 2 of them pinging an internet server, the other one > was a pfsense firewall running in HVM), and in the morning the server > had indeed crashed (Had to hard reset it). The kernel I''m using is > Jerem''y 2.6.32.15 pvops kernel. Both Dom0 and DomUs are 64-bit. I''m > using Xen 3.4.3. > > Xend.log says "Xend exited with status 0)" at around 4:12am in the > morning. > The last thing that Xend-debug says is this (no time given): > > 1. > Exception in thread HttpServer: > 2. > Traceback (most recent call last): > 3. > File "/usr/lib/python2.6/threading.py", line 532, in > __bootstrap_inner > 4. > self.run() > 5. > File "/usr/lib/python2.6/threading.py", line 484, in run > 6. > self.__target(*self.__args, **self.__kwargs) > 7. > File "/usr/lib/python2.6/dist-packages/xen/web/httpserver.py", > line 312, in run > 8. > (sock, addr) = self.accept() > 9. > File "/usr/lib/python2.6/dist-packages/xen/web/httpserver.py", > line 331, in accept > 10. > return self.socket.accept() > 11. > File "/usr/lib/python2.6/socket.py", line 195, in accept > 12. > sock, addr = self._sock.accept() > 13. > error: [Errno 22] Invalid argument > > > > Any help is appreciated > > I should probably mention that for some reason when my system starts > up, or when I try and start Xend manually, the init script launch 2 > xend processes. Could this have caused the problem? > > Thanks > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-usersOk so a little bit of an update. My pfsense HVM machine has a PCI NIC passed through to it. I ran my test again, but this time the system crashed in front of me. But what actually happens, is that my Dom0 crashed (can''t move mouse or keyboard) however its video still works. I can see the pings failing on the screen, which indicates that the pfsense HVM has also crashed. I have no way to restart the system except by power. Seems to me like a PCI host controller fault?? Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users