CentOS virt - Jun 2012 - Failing to start or create VM, cannot connect to hypervisor host

Greetings -

I shutdown one of my Centos 6.2 VMs for some offline maintenance and am
now unable to get it to restart.  I am also unable to create and start a
new VM.  The host system is Centos 6.2, fully up to date.  I have been
searching Google for two days and have not been successful in getting a VM
to start.  I have restarted libvirtd, but did not want to shutdown my
other two running VMs and reboot my host system, unless I felt confident
that it would solve the problem.  The other two VMs on the box are still
running fine, but I am also unable to connect to a console in VirtManger
with them.  I was able to connect to all these VM consoles from the host
system just a week or so ago.  The only changes that I had recently made
to the host system was to add some restrictions in my /etc/hosts.allow and
hosts.deny files and installed Fail2ban.  I have reversed all of the
changes to my hosts.allow/deny files and have stopped the Fail2ban
service.  Still unable to start or create a VM.  I am at the end of my
rope here and am hoping someone will be able to direct me with where I
should be looking next.  My error and log messages are listed below. 
Thanks.

Jeff




Trying to start the VM using VirtManager at the console of the host
results in the following error.

error:  Failed to start domain SequoiaVM
error:  internal error process exited while connecting to monitor: char
device redirected to /dev/pts/7
inet_listen_opts:  bind(ipv4, 127.0.0.1, 5902) : Cannot assign requested
address
inet_listen_opts:  FAILED


Each attempt at a restart results in the following set of messages in
/var/log/messages

Jun 16 13:45:04 earth kernel: device vnet2 entered promiscuous mode
Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering learning state
Jun 16 13:45:04 earth qemu-kvm: Could not find keytab file:
/etc/qemu/krb5.tab: No such file or directory
Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
Jun 16 13:45:04 earth kernel: device vnet2 left promiscuous mode
Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
Jun 16 13:45:04 earth kernel: libfcoe_device_notification:
NETDEV_UNREGISTER vnet2


Log messages in /var/log/libvirt/libvirtd.log are similar to the console

2012-06-16 19:56:39.807+0000: 32175: error : virNetSocketReadWire:911 :
End of file while reading data: Input/output error
2012-06-16 20:15:23.672+0000: 32176: error : qemuMonitorOpenUnix:290 :
failed to connect to monitor socket: No such process
2012-06-16 20:15:23.672+0000: 32176: error :
qemuProcessWaitForMonitor:1289 : internal error process exited while
connecting to monitor: char device redirected to /dev/pts/7
inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested address
inet_listen_opts: FAILED


Log messages in /var/log/libvirt/qemu/SequoiaVM.log are similar

2012-06-16 13:45:04.572: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
4,sockets=4,cores=1,threads=1 -name SequoiaVM -uuid
1ca3a14b-cd4b-f371-b3fe-1cd518480c09 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/SequoiaVM.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -boot order=dc,menu=on -drive
file=/dev/vg_mei/lv_sequoiaroot,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0
-drive
file=/dev/vg_mei/lv_sequoiaswap,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/dev/vg_mei/lv_sequoiavar,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2
-drive
file=/dev/vg_mei/lv_sequoiahome,if=none,id=drive-virtio-disk3,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=virtio-disk3
-drive
file=/dev/vg_mei/lv_sequoiaecosystem,if=none,id=drive-virtio-disk4,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4
-drive
file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9d:b2:49,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -device
usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device
intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa
char device redirected to /dev/pts/7
inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested address
inet_listen_opts: FAILED
2012-06-16 13:45:04.872: shutting down


I attempted to create a new VM (TempSequoiaVM) via VirtManager which also
failed to be created and started.  These are the messages in
/var/log/libvirt/qemu/TempSequoiaVM.log

2012-06-16 13:15:23.168: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
2,sockets=2,cores=1,threads=1 -name TempSequoiaVM -uuid
ef6c99c7-7fea-e24d-447c-cf98d6ffbec3 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/TempSequoiaVM.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot
-no-shutdown -drive
file=/dev/vg_mei/lv_TempSequoia,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
-drive
file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:59:9e,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -device
usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device
intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
char device redirected to /dev/pts/7
inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested address
inet_listen_opts: FAILED
2012-06-16 13:15:23.673: shutting down

A) Could be a perm issue for the krb5.tab file for the qemu system user.

B) Have you tried 'touch /etc/qemu/krb5.tab'  if the file doesn't
truly
exist?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20120616/aa2cf11d/attachment-0006.html>

On 06/16/2012 11:14 PM, jboyce at meridianenv.com wrote:
> Greetings -
>
> I shutdown one of my Centos 6.2 VMs for some offline maintenance and am
> now unable to get it to restart.  I am also unable to create and start a
> new VM.  The host system is Centos 6.2, fully up to date.  I have been
> searching Google for two days and have not been successful in getting a VM
> to start.  I have restarted libvirtd, but did not want to shutdown my
> other two running VMs and reboot my host system, unless I felt confident
> that it would solve the problem.  The other two VMs on the box are still
> running fine, but I am also unable to connect to a console in VirtManger
> with them.  I was able to connect to all these VM consoles from the host
> system just a week or so ago.  The only changes that I had recently made
> to the host system was to add some restrictions in my /etc/hosts.allow and
> hosts.deny files and installed Fail2ban.  I have reversed all of the
> changes to my hosts.allow/deny files and have stopped the Fail2ban
> service.  Still unable to start or create a VM.  I am at the end of my
> rope here and am hoping someone will be able to direct me with where I
> should be looking next.  My error and log messages are listed below.
> Thanks.
>
> Jeff
>
>
>
>
> Trying to start the VM using VirtManager at the console of the host
> results in the following error.
>
> error:  Failed to start domain SequoiaVM
> error:  internal error process exited while connecting to monitor: char
> device redirected to /dev/pts/7
> inet_listen_opts:  bind(ipv4, 127.0.0.1, 5902) : Cannot assign requested
> address
> inet_listen_opts:  FAILED
>
>
> Each attempt at a restart results in the following set of messages in
> /var/log/messages
>
> Jun 16 13:45:04 earth kernel: device vnet2 entered promiscuous mode
> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering learning state
> Jun 16 13:45:04 earth qemu-kvm: Could not find keytab file:
> /etc/qemu/krb5.tab: No such file or directory
> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
> Jun 16 13:45:04 earth kernel: device vnet2 left promiscuous mode
> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
> Jun 16 13:45:04 earth kernel: libfcoe_device_notification:
> NETDEV_UNREGISTER vnet2
>
>
> Log messages in /var/log/libvirt/libvirtd.log are similar to the console
>
> 2012-06-16 19:56:39.807+0000: 32175: error : virNetSocketReadWire:911 :
> End of file while reading data: Input/output error
> 2012-06-16 20:15:23.672+0000: 32176: error : qemuMonitorOpenUnix:290 :
> failed to connect to monitor socket: No such process
> 2012-06-16 20:15:23.672+0000: 32176: error :
> qemuProcessWaitForMonitor:1289 : internal error process exited while
> connecting to monitor: char device redirected to /dev/pts/7
> inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested
address
> inet_listen_opts: FAILED
>
>
> Log messages in /var/log/libvirt/qemu/SequoiaVM.log are similar
>
> 2012-06-16 13:45:04.572: starting up
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
> 4,sockets=4,cores=1,threads=1 -name SequoiaVM -uuid
> 1ca3a14b-cd4b-f371-b3fe-1cd518480c09 -nodefconfig -nodefaults -chardev
>
socket,id=charmonitor,path=/var/lib/libvirt/qemu/SequoiaVM.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -boot order=dc,menu=on -drive
>
file=/dev/vg_mei/lv_sequoiaroot,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
> -device
>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0
> -drive
>
file=/dev/vg_mei/lv_sequoiaswap,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native
> -device
>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1
> -drive
>
file=/dev/vg_mei/lv_sequoiavar,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native
> -device
>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2
> -drive
>
file=/dev/vg_mei/lv_sequoiahome,if=none,id=drive-virtio-disk3,format=raw,cache=none,aio=native
> -device
>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=virtio-disk3
> -drive
>
file=/dev/vg_mei/lv_sequoiaecosystem,if=none,id=drive-virtio-disk4,format=raw,cache=none,aio=native
> -device
>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4
> -drive
> file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
>
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9d:b2:49,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -usb -device
> usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device
> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa
> char device redirected to /dev/pts/7
> inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested
address
> inet_listen_opts: FAILED
> 2012-06-16 13:45:04.872: shutting down
>
>
> I attempted to create a new VM (TempSequoiaVM) via VirtManager which also
> failed to be created and started.  These are the messages in
> /var/log/libvirt/qemu/TempSequoiaVM.log
>
> 2012-06-16 13:15:23.168: starting up
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
> 2,sockets=2,cores=1,threads=1 -name TempSequoiaVM -uuid
> ef6c99c7-7fea-e24d-447c-cf98d6ffbec3 -nodefconfig -nodefaults -chardev
>
socket,id=charmonitor,path=/var/lib/libvirt/qemu/TempSequoiaVM.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot
> -no-shutdown -drive
>
file=/dev/vg_mei/lv_TempSequoia,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
> -device
>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
> -drive
> file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
> -device
> ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
> -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device
>
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:59:9e,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -usb -device
> usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device
> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
> char device redirected to /dev/pts/7
> inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested
address
> inet_listen_opts: FAILED
> 2012-06-16 13:15:23.673: shutting down
>
>
>
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt
>
FWIW I updated my VMs (4) and host on one of my servers yesterday and 
rebooted. All the VMs came up normally and can be accessed over the 
network or by the VM Manager.

ChrisG



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20120617/48e41d74/attachment-0006.html>

Sorry for the mess on the message threading; responding from the daily
digest.

Thanks for the suggestion.  I'll research any option that may provide a
lead to a solution right now.  I have not tried to touch
/etc/qemu/krb5.tab.  After a little research on what this file is for, I
don't see that it would help me since I have never set up the system to
use kerberos.  I don't even have an /etc/qemu directory on my host.  I
would suspect that part of the error is a side effect of something else
that is the main cause.  Please keep coming with the suggestions though. 
I am working on a plan B to make the data files available to the staff on
Monday morning.

Jeff


> Message: 1
> Date: Sat, 16 Jun 2012 14:14:04 -0700 (PDT)
> From: jboyce at meridianenv.com
> Subject: [CentOS-virt] Failing to start or create VM,	cannot connect
> to hypervisor host To: centos-virt at centos.org
> Message-ID:
> <21752.184.157.241.237.1339881244.squirrel at www.meridianenv.com>
> Content-Type: text/plain;charset=iso-8859-1
>
>
> Greetings -
>
>
> I shutdown one of my Centos 6.2 VMs for some offline maintenance and am
> now unable to get it to restart.  I am also unable to create and start a
> new VM.  The host system is Centos 6.2, fully up to date.  I have been
> searching Google for two days and have not been successful in getting a
> VM
> to start.  I have restarted libvirtd, but did not want to shutdown my other
> two running VMs and reboot my host system, unless I felt confident that it
> would solve the problem.  The other two VMs on the box are still running
> fine, but I am also unable to connect to a console in VirtManger with
> them.  I was able to connect to all these VM consoles from the host system
> just a week or so ago.  The only changes that I had recently made to the
> host system was to add some restrictions in my /etc/hosts.allow and
> hosts.deny files and installed Fail2ban.  I have reversed all of the
> changes to my hosts.allow/deny files and have stopped the Fail2ban
> service.  Still unable to start or create a VM.  I am at the end of my
> rope here and am hoping someone will be able to direct me with where I
> should be looking next.  My error and log messages are listed below.
> Thanks.
>
>
> Jeff
>
>
>
>
>
> Trying to start the VM using VirtManager at the console of the host
> results in the following error.
>
> error:  Failed to start domain SequoiaVM
> error:  internal error process exited while connecting to monitor: char
> device redirected to /dev/pts/7 inet_listen_opts:  bind(ipv4, 127.0.0.1,
> 5902) : Cannot assign requested
> address inet_listen_opts:  FAILED
>
>
>
> Each attempt at a restart results in the following set of messages in
> /var/log/messages
>
>
> Jun 16 13:45:04 earth kernel: device vnet2 entered promiscuous mode
> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering learning state
> Jun 16 13:45:04 earth qemu-kvm: Could not find keytab file:
> /etc/qemu/krb5.tab: No such file or directory
> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
> Jun 16 13:45:04 earth kernel: device vnet2 left promiscuous mode
> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
> Jun 16 13:45:04 earth kernel: libfcoe_device_notification:
> NETDEV_UNREGISTER vnet2
>
>
>
> Log messages in /var/log/libvirt/libvirtd.log are similar to the console
>
>
> 2012-06-16 19:56:39.807+0000: 32175: error : virNetSocketReadWire:911 :
> End of file while reading data: Input/output error
> 2012-06-16 20:15:23.672+0000: 32176: error : qemuMonitorOpenUnix:290 :
> failed to connect to monitor socket: No such process 2012-06-16
> 20:15:23.672+0000: 32176: error :
> qemuProcessWaitForMonitor:1289 : internal error process exited while
> connecting to monitor: char device redirected to /dev/pts/7
> inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested
> address inet_listen_opts: FAILED
>
>
>
> Log messages in /var/log/libvirt/qemu/SequoiaVM.log are similar
>
>
> 2012-06-16 13:45:04.572: starting up
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
> 4,sockets=4,cores=1,threads=1 -name SequoiaVM -uuid
> 1ca3a14b-cd4b-f371-b3fe-1cd518480c09 -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/SequoiaVM.monitor,server,
> nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -boot order=dc,menu=on -drive
> file=/dev/vg_mei/lv_sequoiaroot,if=none,id=drive-virtio-disk0,format=raw,c
> ache=none,aio=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
> tio-disk0 -drive
> file=/dev/vg_mei/lv_sequoiaswap,if=none,id=drive-virtio-disk1,format=raw,c
> ache=none,aio=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=vir
> tio-disk1 -drive
> file=/dev/vg_mei/lv_sequoiavar,if=none,id=drive-virtio-disk2,format=raw,ca
> che=none,aio=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=vir
> tio-disk2 -drive
> file=/dev/vg_mei/lv_sequoiahome,if=none,id=drive-virtio-disk3,format=raw,c
> ache=none,aio=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=vir
> tio-disk3 -drive
>
file=/dev/vg_mei/lv_sequoiaecosystem,if=none,id=drive-virtio-disk4,format>
raw,cache=none,aio=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=vir
> tio-disk4 -drive
> file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>  -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9d:b2:49,bus=pci.0,add
> r=0x3 -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0
> -vnc 127.0.0.1:2 -vga cirrus -device
> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa char device redirected
> to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign
> requested address inet_listen_opts: FAILED
> 2012-06-16 13:45:04.872: shutting down
>
>
>
> I attempted to create a new VM (TempSequoiaVM) via VirtManager which also
>  failed to be created and started.  These are the messages in
> /var/log/libvirt/qemu/TempSequoiaVM.log
>
>
> 2012-06-16 13:15:23.168: starting up
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
> 2,sockets=2,cores=1,threads=1 -name TempSequoiaVM -uuid
> ef6c99c7-7fea-e24d-447c-cf98d6ffbec3 -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/TempSequoiaVM.monitor,se
> rver,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-reboot
> -no-shutdown -drive
> file=/dev/vg_mei/lv_TempSequoia,if=none,id=drive-virtio-disk0,format=raw,c
> ache=none,aio=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
> tio-disk0,bootindex=2 -drive
> file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>  -device
> ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
> -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:59:9e,bus=pci.0,add
> r=0x3 -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0
> -vnc 127.0.0.1:2 -vga cirrus -device
> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 char device redirected
> to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign
> requested address inet_listen_opts: FAILED
> 2012-06-16 13:15:23.673: shutting down
>
>
>
>
>
>
> ------------------------------
>
>
> Message: 2
> Date: Sat, 16 Jun 2012 18:01:52 -0400
> From: Travis <corasian at gmail.com>
> Subject: Re: [CentOS-virt] Failing to start or create VM, cannot
> connect to hypervisor host To: Discussion about the virtualization on
> CentOS
> <centos-virt at centos.org>
> Message-ID:
> <CAKbLq_KcmO=72Qq3x=sDpMC21717tS=eumgqL_-LOoMP8-HC7A at
mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
>
> A) Could be a perm issue for the krb5.tab file for the qemu system user.
>
>
> B) Have you tried 'touch /etc/qemu/krb5.tab'  if the file
doesn't truly
> exist?

Responding to the daily digest again.

Well, I finally pinpointed the problem and fixed it.  But to answer you 
curiosity question, the host is running a hardware raid 5 (Dell T610 with a 
PERC H700).

My problem was a result of a change made to my network configuration almost 
2 weeks ago.  When I brought the network backup, I just ifup'ed the em1 and 
br0 interfaces and everything was working.  At least everything that was 
obvious at the time.  In reality, I didn't have a loopback interface 
working, and I hadn't needed to look any any of my guests through 
VirtManager since then to notice the problem.  As soon as I issued a service 
network restart to the host system, everything came back normal and I was 
able to connect to a console on my guest systems.  It took hours of tracking 
down false leads from the log files, and learning how to interpret others 
before I got this pinpointed.  I am now very thankful for keeping fairly 
detailed notes on all the changes to my system, as cross checking the logs 
against my notes helped my pinpoint the problem.

Jeff

> ------------------------------
>
> Message: 2
> Date: Sun, 17 Jun 2012 11:19:41 -0700
> From: aurfalien <aurfalien at gmail.com>
> Subject: Re: [CentOS-virt] Failing to start or create VM, cannot
> connect to hypervisor host
> To: Discussion about the virtualization on CentOS
> <centos-virt at centos.org>
> Message-ID: <B7DCC3C3-C847-4E18-B016-4FBA20F12815 at gmail.com>
> Content-Type: text/plain; charset=us-ascii
>
> Curious, is this system (the host) on a software raid?
>
> - aurf
> On Jun 17, 2012, at 10:35 AM, jboyce at meridianenv.com wrote:
>
>> Sorry for the mess on the message threading; responding from the daily
>> digest.
>>
>> Thanks for the suggestion.  I'll research any option that may
provide a
>> lead to a solution right now.  I have not tried to touch
>> /etc/qemu/krb5.tab.  After a little research on what this file is for,
I
>> don't see that it would help me since I have never set up the
system to
>> use kerberos.  I don't even have an /etc/qemu directory on my host.
I
>> would suspect that part of the error is a side effect of something else
>> that is the main cause.  Please keep coming with the suggestions
though.
>> I am working on a plan B to make the data files available to the staff
on
>> Monday morning.
>>
>> Jeff
>>
>>
>>
>>> Message: 1
>>> Date: Sat, 16 Jun 2012 14:14:04 -0700 (PDT)
>>> From: jboyce at meridianenv.com
>>> Subject: [CentOS-virt] Failing to start or create VM, cannot
connect
>>> to hypervisor host To: centos-virt at centos.org
>>> Message-ID:
>>> <21752.184.157.241.237.1339881244.squirrel at
www.meridianenv.com>
>>> Content-Type: text/plain;charset=iso-8859-1
>>>
>>>
>>> Greetings -
>>>
>>>
>>> I shutdown one of my Centos 6.2 VMs for some offline maintenance
and am
>>> now unable to get it to restart.  I am also unable to create and
start a
>>> new VM.  The host system is Centos 6.2, fully up to date.  I have
been
>>> searching Google for two days and have not been successful in
getting a
>>> VM
>>> to start.  I have restarted libvirtd, but did not want to shutdown
my
>>> other
>>> two running VMs and reboot my host system, unless I felt confident
that
>>> it
>>> would solve the problem.  The other two VMs on the box are still
running
>>> fine, but I am also unable to connect to a console in VirtManger
with
>>> them.  I was able to connect to all these VM consoles from the host
>>> system
>>> just a week or so ago.  The only changes that I had recently made
to the
>>> host system was to add some restrictions in my /etc/hosts.allow and
>>> hosts.deny files and installed Fail2ban.  I have reversed all of
the
>>> changes to my hosts.allow/deny files and have stopped the Fail2ban
>>> service.  Still unable to start or create a VM.  I am at the end of
my
>>> rope here and am hoping someone will be able to direct me with
where I
>>> should be looking next.  My error and log messages are listed
below.
>>> Thanks.
>>>
>>>
>>> Jeff
>>>
>>>
>>>
>>>
>>>
>>> Trying to start the VM using VirtManager at the console of the host
>>> results in the following error.
>>>
>>> error:  Failed to start domain SequoiaVM
>>> error:  internal error process exited while connecting to monitor:
char
>>> device redirected to /dev/pts/7 inet_listen_opts:  bind(ipv4,
127.0.0.1,
>>> 5902) : Cannot assign requested
>>> address inet_listen_opts:  FAILED
>>>
>>>
>>>
>>> Each attempt at a restart results in the following set of messages
in
>>> /var/log/messages
>>>
>>>
>>> Jun 16 13:45:04 earth kernel: device vnet2 entered promiscuous mode
>>> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering learning
state
>>> Jun 16 13:45:04 earth qemu-kvm: Could not find keytab file:
>>> /etc/qemu/krb5.tab: No such file or directory
>>> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled
state
>>> Jun 16 13:45:04 earth kernel: device vnet2 left promiscuous mode
>>> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled
state
>>> Jun 16 13:45:04 earth kernel: libfcoe_device_notification:
>>> NETDEV_UNREGISTER vnet2
>>>
>>>
>>>
>>> Log messages in /var/log/libvirt/libvirtd.log are similar to the
console
>>>
>>>
>>> 2012-06-16 19:56:39.807+0000: 32175: error :
virNetSocketReadWire:911 :
>>> End of file while reading data: Input/output error
>>> 2012-06-16 20:15:23.672+0000: 32176: error :
qemuMonitorOpenUnix:290 :
>>> failed to connect to monitor socket: No such process 2012-06-16
>>> 20:15:23.672+0000: 32176: error :
>>> qemuProcessWaitForMonitor:1289 : internal error process exited
while
>>> connecting to monitor: char device redirected to /dev/pts/7
>>> inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign
requested
>>> address inet_listen_opts: FAILED
>>>
>>>
>>>
>>> Log messages in /var/log/libvirt/qemu/SequoiaVM.log are similar
>>>
>>>
>>> 2012-06-16 13:45:04.572: starting up
>>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
>>> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
>>> 4,sockets=4,cores=1,threads=1 -name SequoiaVM -uuid
>>> 1ca3a14b-cd4b-f371-b3fe-1cd518480c09 -nodefconfig -nodefaults
-chardev
>>>
socket,id=charmonitor,path=/var/lib/libvirt/qemu/SequoiaVM.monitor,server,
>>> nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc
>>> -no-shutdown -boot order=dc,menu=on -drive
>>>
file=/dev/vg_mei/lv_sequoiaroot,if=none,id=drive-virtio-disk0,format=raw,c
>>> ache=none,aio=native -device
>>>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
>>> tio-disk0 -drive
>>>
file=/dev/vg_mei/lv_sequoiaswap,if=none,id=drive-virtio-disk1,format=raw,c
>>> ache=none,aio=native -device
>>>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=vir
>>> tio-disk1 -drive
>>>
file=/dev/vg_mei/lv_sequoiavar,if=none,id=drive-virtio-disk2,format=raw,ca
>>> che=none,aio=native -device
>>>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=vir
>>> tio-disk2 -drive
>>>
file=/dev/vg_mei/lv_sequoiahome,if=none,id=drive-virtio-disk3,format=raw,c
>>> ache=none,aio=native -device
>>>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=vir
>>> tio-disk3 -drive
>>>
file=/dev/vg_mei/lv_sequoiaecosystem,if=none,id=drive-virtio-disk4,format>>>
raw,cache=none,aio=native -device
>>>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=vir
>>> tio-disk4 -drive
>>>
file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>>> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>>> -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
>>>
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9d:b2:49,bus=pci.0,add
>>> r=0x3 -chardev pty,id=charserial0 -device
>>> isa-serial,chardev=charserial0,id=serial0 -usb -device 
>>> usb-tablet,id=input0
>>> -vnc 127.0.0.1:2 -vga cirrus -device
>>> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
>>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa char device
redirected
>>> to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot
assign
>>> requested address inet_listen_opts: FAILED
>>> 2012-06-16 13:45:04.872: shutting down
>>>
>>>
>>>
>>> I attempted to create a new VM (TempSequoiaVM) via VirtManager
which
>>> also
>>> failed to be created and started.  These are the messages in
>>> /var/log/libvirt/qemu/TempSequoiaVM.log
>>>
>>>
>>> 2012-06-16 13:15:23.168: starting up
>>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
>>> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
>>> 2,sockets=2,cores=1,threads=1 -name TempSequoiaVM -uuid
>>> ef6c99c7-7fea-e24d-447c-cf98d6ffbec3 -nodefconfig -nodefaults
-chardev
>>>
socket,id=charmonitor,path=/var/lib/libvirt/qemu/TempSequoiaVM.monitor,se
>>> rver,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc 
>>> base=utc
>>> -no-reboot
>>> -no-shutdown -drive
>>>
file=/dev/vg_mei/lv_TempSequoia,if=none,id=drive-virtio-disk0,format=raw,c
>>> ache=none,aio=native -device
>>>
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
>>> tio-disk0,bootindex=2 -drive
>>>
file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>>> -device
>>>
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
>>> -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device
>>>
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:59:9e,bus=pci.0,add
>>> r=0x3 -chardev pty,id=charserial0 -device
>>> isa-serial,chardev=charserial0,id=serial0 -usb -device 
>>> usb-tablet,id=input0
>>> -vnc 127.0.0.1:2 -vga cirrus -device
>>> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
>>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 char device
redirected
>>> to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot
assign
>>> requested address inet_listen_opts: FAILED
>>> 2012-06-16 13:15:23.673: shutting down
>>>