I''m having a very weird problem that started about a week ago. I have a couple sles11 xen 3.4 servers. The first server has a local 400gb disk provided by our NAS. My VMs are file based. For some reason that server has started to hang randomly. I thought something was wrong with the server, possibly the hardware. I moved the 400gb disk and assigned it to a second sles11 xen 3.4 server and brought the VMs up on that server. Now it''s hanging! I don''t see anything in /var/log/messages. When this happens the server is still pingable. I can get on the server''s console, and hit enter and the cursor moves down, but it''s not usable. I''ve forced an fcsk on the 400gb drive and it comes back clean. Below is the last thing in the log before I have to restart the server. Can anyone suggest troubleshooting for something like this? I''m stuck. Help is appreciated. James May 21 12:30:00 xen06 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/console/3/0 May 21 12:30:00 xen06 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vfb/3/0 May 21 12:30:00 xen06 logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/3/768 May 21 12:30:00 xen06 logger: /etc/xen/scripts/vif-bridge: offline XENBUS_PATH=backend/vif/3/0 May 21 12:30:00 xen06 kernel: br0: port 3(vif3.0) entering disabled state May 21 12:30:00 xen06 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vbd/3/768 May 21 12:30:00 xen06 kernel: br0: port 3(vif3.0) entering disabled state May 21 12:30:00 xen06 logger: /etc/xen/scripts/vif-bridge: brctl delif br0 vif3.0 failed May 21 12:30:00 xen06 ifdown: vif3.0 May 21 12:30:00 xen06 logger: /etc/xen/scripts/vif-bridge: ifconfig vif3.0 down failed May 21 12:30:00 xen06 ifdown: Interface not available and no configuration found. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge offline for vif3.0, bridge br0. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:00 xen06 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vif/3/0 May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:01 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:30:01 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:31:29 xen06 libvirtd: 12:31:29.391: error : xenInotifyXendDomainsDirLookup:163 : internal error finding dom on config list May 21 12:31:29 xen06 libvirtd: 12:31:29.464: error : xenInotifyXendDomainsDirLookup:163 : internal error finding dom on config list May 21 12:32:42 xen06 libvirtd: 12:32:42.463: error : xenInotifyXendDomainsDirLookup:163 : internal error finding dom on config list May 21 12:32:42 xen06 libvirtd: 12:32:42.577: error : xenInotifyXendDomainsDirLookup:163 : internal error finding dom on config list May 21 12:33:39 xen06 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/19/768 May 21 12:33:39 xen06 logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/19/0 May 21 12:33:39 xen06 kernel: device tap19.0 entered promiscuous mode May 21 12:33:39 xen06 kernel: br0: topology change detected, propagating May 21 12:33:39 xen06 kernel: br0: port 2(tap19.0) entering forwarding state May 21 12:33:39 xen06 kernel: device vif19.0 entered promiscuous mode May 21 12:33:39 xen06 kernel: br0: topology change detected, propagating May 21 12:33:39 xen06 kernel: br0: port 3(vif19.0) entering forwarding state May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:39 xen06 logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking. May 21 12:33:39 xen06 ifup: vif19.0 May 21 12:33:40 xen06 logger: /etc/xen/scripts/block: Writing backend/vbd/19/768/physical-device 8:10 to xenstore. May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:40 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. May 21 12:33:40 xen06 ifup: No configuration found for vif19.0 May 21 12:33:40 xen06 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif19.0, bridge br0. May 21 12:33:40 xen06 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/19/0/hotplug-status connected to xenstore. May 21 12:33:40 xen06 logger: /etc/xen/scripts/block: Writing backend/vbd/19/768/hotplug-status connected to xenstore. May 21 12:33:40 xen06 kernel: (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/19/768 May 21 12:33:40 xen06 kernel: (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 May 21 12:34:01 xen06 kernel: br0: port 2(tap19.0) entering disabled state May 21 12:34:01 xen06 kernel: device tap19.0 left promiscuous mode May 21 12:34:01 xen06 kernel: br0: port 2(tap19.0) entering disabled state May 21 12:34:01 xen06 kernel: blkback: ring-ref 8, event-channel 5, protocol 2 (x86_32-abi) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, 2010-05-21 at 13:57 -0400, James Pifer wrote:> I''m having a very weird problem that started about a week ago. > > I have a couple sles11 xen 3.4 servers. The first server has a local > 400gb disk provided by our NAS. My VMs are file based. For some reason > that server has started to hang randomly. > > I thought something was wrong with the server, possibly the hardware. I > moved the 400gb disk and assigned it to a second sles11 xen 3.4 server > and brought the VMs up on that server. > > Now it''s hanging! I don''t see anything in /var/log/messages. > When this happens the server is still pingable. > I can get on the server''s console, and hit enter and the cursor moves > down, but it''s not usable. > > I''ve forced an fcsk on the 400gb drive and it comes back clean. > > Below is the last thing in the log before I have to restart the server. > > Can anyone suggest troubleshooting for something like this? I''m stuck. > > Help is appreciated. >A little more information. We did start making some disk changes on these servers. I was going to be moving from file based to physical disk based VMs from our SAN. We added two 20gb disks at LUN1 and LUN2 and moved the 400gb disk to the end at LUN50. I modified fstab to load the 400 gb disk at the same mount point. If/when the system hangs again we''ll try moving removing the 20 gb disks and have only the 400 gb disk. Don''t see how this will change anything though. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> A little more information. We did start making some disk changes on > these servers. I was going to be moving from file based to physical disk > based VMs from our SAN. > > We added two 20gb disks at LUN1 and LUN2 and moved the 400gb disk to the > end at LUN50. I modified fstab to load the 400 gb disk at the same mount > point. > > If/when the system hangs again we''ll try moving removing the 20 gb disks > and have only the 400 gb disk. Don''t see how this will change anything > though. > > Thanks, > James >Still having some problems with this. I''ve now had it happen on a couple different xen servers. I thought it was related to a specific domU, but the latest event did not involve that one. Basically the server hangs, but network activity, even on the domUs, is still there. IE, I can ping both the host and the VMs, but I can''t do anything on the host or the VMs. The latest one took a week before it popped up. Happened sometime yesterday or last night. Since pings still work, I''m not sure when it happened. Any suggestion on how to troubleshoot this? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
What about console access? Can you do anything from there? -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of James Pifer Sent: 01 June 2010 12:58 To: xen-users@lists.xensource.com Subject: Re: [Xen-users] Server hanging - Please help!> A little more information. We did start making some disk changes on > these servers. I was going to be moving from file based to physicaldisk> based VMs from our SAN. > > We added two 20gb disks at LUN1 and LUN2 and moved the 400gb disk tothe> end at LUN50. I modified fstab to load the 400 gb disk at the samemount> point. > > If/when the system hangs again we''ll try moving removing the 20 gbdisks> and have only the 400 gb disk. Don''t see how this will change anything > though. > > Thanks, > James >Still having some problems with this. I''ve now had it happen on a couple different xen servers. I thought it was related to a specific domU, but the latest event did not involve that one. Basically the server hangs, but network activity, even on the domUs, is still there. IE, I can ping both the host and the VMs, but I can''t do anything on the host or the VMs. The latest one took a week before it popped up. Happened sometime yesterday or last night. Since pings still work, I''m not sure when it happened. Any suggestion on how to troubleshoot this? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am Dienstag 01 Juni 2010, 13:58:22 schrieb James Pifer:> Any suggestion on how to troubleshoot this?You might run xen on the serial console of your machine as you did not have access to your VGA buffers / console. http://en.opensuse.org/How_to_Capture_Xen_Hypervisor_and_Kernel_Messages_using_a_Serial_Cable http://www.xen.org/files/xensummit_intel09/xen-debugging.pdf Xen GDB setup http://zhigang.org/files/docbook/xen-faq.html#setup_gdb> The latest one took a week before it popped up. Happened sometime > yesterday or last night. Since pings still work, I''m not sure when it > happened.You may use service based monitoring like Nagios offers it (or - if that''s to overloadad for you) by a script which does some "real" with your DomUs. hth best regards, Niels. -- --- Niels Dettenbach --- Syndicat IT&Internet http://www.syndicat.com T.-Muentzer.-Str. 2, 37308 Heilbad Heiligenstadt - DE --- Kryptoinfo: PGP public key ID 651CA20D Fingerprint: 55E0 4DCD B04C 4A49 1586 88AE 54DC 4465 651C A20D https://syndicat.com/pub_key.asc --- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2010-06-01 at 13:51 +0100, Ian Tobin wrote:> What about console access? Can you do anything from there? >I can get to the console using a Remote Insight Board, but you can''t do anything. You can see it, and hit enter, the cursor moves down, but it''s hosed. Can''t do anything. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2010-06-01 at 15:02 +0200, Niels Dettenbach wrote:> Am Dienstag 01 Juni 2010, 13:58:22 schrieb James Pifer: > > Any suggestion on how to troubleshoot this? > > You might run xen on the serial console of your machine as you did not have > access to your VGA buffers / console. > > http://en.opensuse.org/How_to_Capture_Xen_Hypervisor_and_Kernel_Messages_using_a_Serial_Cable > http://www.xen.org/files/xensummit_intel09/xen-debugging.pdf > > Xen GDB setup > http://zhigang.org/files/docbook/xen-faq.html#setup_gdb >The biggest problem for me is that I''m remote to the servers. So a serial connection will not work for me. Had another server lockup today. The weird thing is that it''s not just xen that gets hosed. The whole box is hosed, except the host and the VMs still respond to pings, and you can still view the console, although you can''t do anything in it. It''s very strange. I''m trying to find out if it''s related to the SAN or not, but part of the problem is I''m not able to recreate the issue. I''m also running a cat to a file of messages and xend.log, so I''m just waiting to see when it happens again. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I try setup Win7 in Domain U. Next config for domainU: # grep -v ^# /etc/xen/win7.hvm | grep -vx '''' kernel = "hvmloader" builder=''hvm'' memory = 2048 name = "win7" vif = [ ''type=ioemu, bridge=eth0'' ] disk = [ ''file:/vm/win7.img,hda,w'', ''file:/file/Windows7_ru_en_x86_x64.iso,hdc:cdrom,r'' ] device_model = ''qemu-dm'' boot="d" sdl=0 opengl=1 vnc=1 vncpasswd='''' stdvga=0 serial=''pty'' tsc_mode=0 localtime=3 usb=1 vncviewer connect to Domain :1 and show white rectangle. Then # xm create win7.hvm Using config file "./win7.hvm". Error: Device 0 (vif) could not be connected. Hotplug scripts not working. How fix it ??? # brctl show bridge name bridge id STP enabled interfaces eth0 8000.002191d6f05a no peth0 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
jpp@jppozzi.dyndns.org
2010-Jun-05 19:39 UTC
Re: [Xen-users] white rectangle without image
Le samedi 05 juin 2010 à 21:42 +0400, doom a écrit :> I try setup Win7 in Domain U. > Next config for domainU: > > # grep -v ^# /etc/xen/win7.hvm | grep -vx '''' > kernel = "hvmloader" > builder=''hvm'' > memory = 2048 > name = "win7" > vif = [ ''type=ioemu, bridge=eth0'' ] > disk = [ ''file:/vm/win7.img,hda,w'', ''file:/file/Windows7_ru_en_x86_x64.iso,hdc:cdrom,r'' ] > device_model = ''qemu-dm'' > boot="d" > sdl=0 > opengl=1 > vnc=1 > vncpasswd='''' > stdvga=0 > serial=''pty'' > tsc_mode=0 > localtime=3 > usb=1 >Hello, No network device. Regards JPPO _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Здравствуйте, Jpp. Вы писали 5 июня 2010 г., 23:39:08:> Le samedi 05 juin 2010 à 21:42 +0400, doom a écrit : >> I try setup Win7 in Domain U. >> Next config for domainU: >> >> # grep -v ^# /etc/xen/win7.hvm | grep -vx '''' >> kernel = "hvmloader" >> builder=''hvm'' >> memory = 2048 >> name = "win7" >> vif = [ ''type=ioemu, bridge=eth0'' ] >> disk = [ ''file:/vm/win7.img,hda,w'', ''file:/file/Windows7_ru_en_x86_x64..iso,hdc:cdrom,r'' ] >> device_model = ''qemu-dm'' >> boot="d" >> sdl=0 >> opengl=1 >> vnc=1 >> vncpasswd='''' >> stdvga=0 >> serial=''pty'' >> tsc_mode=0 >> localtime=3 >> usb=1 >>> Hello,> No network device.> Regards> JPPOHmmm... but in config file - eth0... I don''t undestand.how fix it ? -- С уважением, doom mailto:doom28@mail.ru _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I fix it. I switch options in kernel Hard link <*> Network Bridging and XEN-> Backend Drivers in Dom0 <*> Thanks for Boris D. And this error disappear, but appear another. Error: Device 768 (vbd) could not be connected. Path closed or removed during hotplug add: backend/vbd/9/768 state: 1>> opengl=1 >> vnc=1 >> vncpasswd='''' >> stdvga=0 >> serial=''pty'' >> tsc_mode=0 >> localtime=3 >> usb=1 >>> Hello,> No network device.> Regards> JPPO-- С уважением, doom mailto:doom28@mail.ru _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I fix it. I switch options in kernel Hard link <*> Network Bridging and XEN-> Backend Drivers in Dom0 <*> Thanks for Boris D. And this error disappear, but appear another. Error: Device 768 (vbd) could not be connected. Path closed or removed during hotplug add: backend/vbd/9/768 state: 1>> opengl=1 >> vnc=1 >> vncpasswd='''' >> stdvga=0 >> serial=''pty'' >> tsc_mode=0 >> localtime=3 >> usb=1 >>> Hello,> No network device.> Regards> JPPO-- С уважением, doom mailto:doom28@mail.ru _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users