CentOS virt - Oct 2014 - virsh list hangs / guests not starting automatically

I followed the wiki[1] to create a KVM virtual machine using bridged
network on CentOS 6.5.  It seemed to work fine on initial setup.
(FWIW I'm trying to run a MythBuntu guest.) However, after a reboot,
it doesn't auto-start the VMs.

Shortly after boot, if I go into "virsh", then do a "list",
it just
hangs.  Likewise, if I go into "virt manager", it just hangs with the
message "connecting".

Kernel version is: 2.6.32-431.29.2.el6.x86_64

Relevant package versions:

libvirt.x86_64                       0.10.2-29.el6_5.12
libvirt-client.x86_64                0.10.2-29.el6_5.12
libvirt-python.x86_64                0.10.2-29.el6_5.12
python-virtinst.noarch               0.600.0-18.el6
virt-manager.x86_64                  0.9.0-19.el6
virt-top.x86_64                      1.0.4-3.15.el6
virt-viewer.x86_64                   0.5.6-8.el6_5.3
qemu-img.x86_64                      2:0.12.1.2-2.415.el6_5.14
qemu-kvm.x86_64                      2:0.12.1.2-2.415.el6_5.14

CPU is a Xeon E3-1230v3.  I have the virtualization setting enabled in the BIOS.

I googled on this, and saw a bunch of talk about two years ago
regarding issues with the libvirt packages having a deadlock bug.  But
I think the versions of the relevant packages that I have installed
are new enough to have fixes for that.

I also happened across an earlier post to this list[2], where it
seemed someone was having a similar problem.  I was previously
attempting to use balance-rr and 802.3ad bonding modes on my host.
However, I just changed to using "active-backup" and the problem
remains.

I have in /etc/libvirt/libvirtd.conf the following three lines (the
rest is stock, i.e. all comments):
log_level = 2
log_filters=""
log_outputs="1:file:/var/log/libvirt/libvirt.log"

Below I posted the contents of the libvirt log file after doing a
"service libvirt start".

Anyone ever fought this before?

Thanks!

[1] http://wiki.centos.org/HowTos/KVM
[2] http://lists.centos.org/pipermail/centos-virt/2014-March/003722.html

/var/log/libvirt/libvirt.log output:
2014-10-14 16:47:11.150+0000: 4657: info : libvirt version: 0.10.2,
package: 29.el6_5.12 (CentOS BuildSystem <http://bugs.centos.org>,
2014-09-01-13:44:02, c6b8.bsys.dev.centos.org)
2014-10-14 16:47:11.150+0000: 4657: info :
virNetlinkEventServiceStart:517 : starting netlink event service with
protocol 0
2014-10-14 16:47:11.151+0000: 4657: info :
virNetlinkEventServiceStart:517 : starting netlink event service with
protocol 15
2014-10-14 16:47:11.154+0000: 4668: info :
dnsmasqCapsSetFromBuffer:667 : dnsmasq version is 2.48, --bind-dynamic
is NOT present, SO_BINDTODEVICE is NOT in use
2014-10-14 16:47:11.157+0000: 4668: info :
networkReloadIptablesRules:1925 : Reloading iptables rules
2014-10-14 16:47:11.157+0000: 4668: info : networkRefreshDaemons:1287
: Refreshing network daemons
2014-10-14 16:47:11.278+0000: 4668: info : networkStartNetwork:2422 :
Starting up network 'default'
2014-10-14 16:47:11.290+0000: 4668: info :
virStorageBackendVolOpenCheckMode:1085 : Skipping special dir '.'
2014-10-14 16:47:11.290+0000: 4668: info :
virStorageBackendVolOpenCheckMode:1085 : Skipping special dir '..'
2014-10-14 16:47:11.352+0000: 4668: info : qemudStartup:754 : Unable
to create cgroup for driver: No such device or address
2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
Configured cgroup controller 'cpu'
2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
Configured cgroup controller 'cpuacct'
2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
Configured cgroup controller 'cpuset'
2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
Configured cgroup controller 'memory'
2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
Configured cgroup controller 'devices'
2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
Configured cgroup controller 'blkio'
2014-10-14 16:47:11.509+0000: 4668: info :
virDomainLoadAllConfigs:14696 : Scanning for configs in
/var/run/libvirt/qemu
2014-10-14 16:47:11.527+0000: 4668: info :
virDomainLoadAllConfigs:14696 : Scanning for configs in
/etc/libvirt/qemu
2014-10-14 16:47:11.527+0000: 4668: info :
virDomainLoadAllConfigs:14718 : Loading config file 'mythbuntu.xml'
2014-10-14 16:47:11.529+0000: 4668: info : qemuDomainSnapshotLoad:484
: Scanning for snapshots for domain mythbuntu in
/var/lib/libvirt/qemu/snapshot/mythbuntu

I just wanted to follow-up to add that eventually, the virtual machine
did start, and now "virsh list" works as expected.  But it took nearly
30 minutes.  The updated libvirt.log is shown below.  Notice the huge
jump in time, from 16:47 to 17:14.  (Side question: it appears the
timestamps are UTC, rather than my local time, any way to address
that?)


2014-10-14 16:47:11.527+0000: 4668: info :
virDomainLoadAllConfigs:14718 : Loading config file 'mythbuntu.xml'
2014-10-14 16:47:11.529+0000: 4668: info : qemuDomainSnapshotLoad:484
: Scanning for snapshots for domain mythbuntu in
/var/lib/libvirt/qemu/snapshot/mythbuntu
2014-10-14 17:14:41.751+0000: 4668: info : virNetDevProbeVnetHdr:94 :
Enabling IFF_VNET_HDR
2014-10-14 17:14:41.805+0000: 4668: info :
virSecurityDACSetOwnership:296 : Setting DAC user and group on
'/home/kvm/mythbuntu.img' to '107:107'
2014-10-14 17:14:41.806+0000: 4668: info :
virSecurityDACSetOwnership:296 : Setting DAC user and group on
'/mnt/mythtv1/mythbackend_recordings' to '107:107'
2014-10-14 17:14:42.084+0000: 4668: info : lxcSecurityInit:1380 :
lxcSecurityInit (null)
2014-10-14 17:14:42.084+0000: 4668: info :
virDomainLoadAllConfigs:14696 : Scanning for configs in
/var/run/libvirt/lxc
2014-10-14 17:14:42.084+0000: 4668: info :
virDomainLoadAllConfigs:14696 : Scanning for configs in
/etc/libvirt/lxc
2014-10-14 17:14:42.089+0000: 4659: error : virFileReadAll:462 :
Failed to open file '/proc/4836/stat': No such file or directory
2014-10-14 17:14:42.090+0000: 4660: error : virFileReadAll:462 :
Failed to open file '/proc/8017/stat': No such file or directory
2014-10-14 17:26:34.679+0000: 4661: info : remoteDispatchAuthList:2398
: Bypass polkit auth for privileged client pid:11343,uid:0




On Tue, Oct 14, 2014 at 12:08 PM, Matt Garman <matthew.garman at
gmail.com> wrote:
> I followed the wiki[1] to create a KVM virtual machine using bridged
> network on CentOS 6.5.  It seemed to work fine on initial setup.
> (FWIW I'm trying to run a MythBuntu guest.) However, after a reboot,
> it doesn't auto-start the VMs.
>
> Shortly after boot, if I go into "virsh", then do a
"list", it just
> hangs.  Likewise, if I go into "virt manager", it just hangs with
the
> message "connecting".
>
> Kernel version is: 2.6.32-431.29.2.el6.x86_64
>
> Relevant package versions:
>
> libvirt.x86_64                       0.10.2-29.el6_5.12
> libvirt-client.x86_64                0.10.2-29.el6_5.12
> libvirt-python.x86_64                0.10.2-29.el6_5.12
> python-virtinst.noarch               0.600.0-18.el6
> virt-manager.x86_64                  0.9.0-19.el6
> virt-top.x86_64                      1.0.4-3.15.el6
> virt-viewer.x86_64                   0.5.6-8.el6_5.3
> qemu-img.x86_64                      2:0.12.1.2-2.415.el6_5.14
> qemu-kvm.x86_64                      2:0.12.1.2-2.415.el6_5.14
>
> CPU is a Xeon E3-1230v3.  I have the virtualization setting enabled in the
BIOS.
>
> I googled on this, and saw a bunch of talk about two years ago
> regarding issues with the libvirt packages having a deadlock bug.  But
> I think the versions of the relevant packages that I have installed
> are new enough to have fixes for that.
>
> I also happened across an earlier post to this list[2], where it
> seemed someone was having a similar problem.  I was previously
> attempting to use balance-rr and 802.3ad bonding modes on my host.
> However, I just changed to using "active-backup" and the problem
> remains.
>
> I have in /etc/libvirt/libvirtd.conf the following three lines (the
> rest is stock, i.e. all comments):
> log_level = 2
> log_filters=""
> log_outputs="1:file:/var/log/libvirt/libvirt.log"
>
> Below I posted the contents of the libvirt log file after doing a
> "service libvirt start".
>
> Anyone ever fought this before?
>
> Thanks!
>
> [1] http://wiki.centos.org/HowTos/KVM
> [2] http://lists.centos.org/pipermail/centos-virt/2014-March/003722.html
>
> /var/log/libvirt/libvirt.log output:
> 2014-10-14 16:47:11.150+0000: 4657: info : libvirt version: 0.10.2,
> package: 29.el6_5.12 (CentOS BuildSystem <http://bugs.centos.org>,
> 2014-09-01-13:44:02, c6b8.bsys.dev.centos.org)
> 2014-10-14 16:47:11.150+0000: 4657: info :
> virNetlinkEventServiceStart:517 : starting netlink event service with
> protocol 0
> 2014-10-14 16:47:11.151+0000: 4657: info :
> virNetlinkEventServiceStart:517 : starting netlink event service with
> protocol 15
> 2014-10-14 16:47:11.154+0000: 4668: info :
> dnsmasqCapsSetFromBuffer:667 : dnsmasq version is 2.48, --bind-dynamic
> is NOT present, SO_BINDTODEVICE is NOT in use
> 2014-10-14 16:47:11.157+0000: 4668: info :
> networkReloadIptablesRules:1925 : Reloading iptables rules
> 2014-10-14 16:47:11.157+0000: 4668: info : networkRefreshDaemons:1287
> : Refreshing network daemons
> 2014-10-14 16:47:11.278+0000: 4668: info : networkStartNetwork:2422 :
> Starting up network 'default'
> 2014-10-14 16:47:11.290+0000: 4668: info :
> virStorageBackendVolOpenCheckMode:1085 : Skipping special dir '.'
> 2014-10-14 16:47:11.290+0000: 4668: info :
> virStorageBackendVolOpenCheckMode:1085 : Skipping special dir '..'
> 2014-10-14 16:47:11.352+0000: 4668: info : qemudStartup:754 : Unable
> to create cgroup for driver: No such device or address
> 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
> Configured cgroup controller 'cpu'
> 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
> Configured cgroup controller 'cpuacct'
> 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
> Configured cgroup controller 'cpuset'
> 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
> Configured cgroup controller 'memory'
> 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
> Configured cgroup controller 'devices'
> 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 :
> Configured cgroup controller 'blkio'
> 2014-10-14 16:47:11.509+0000: 4668: info :
> virDomainLoadAllConfigs:14696 : Scanning for configs in
> /var/run/libvirt/qemu
> 2014-10-14 16:47:11.527+0000: 4668: info :
> virDomainLoadAllConfigs:14696 : Scanning for configs in
> /etc/libvirt/qemu
> 2014-10-14 16:47:11.527+0000: 4668: info :
> virDomainLoadAllConfigs:14718 : Loading config file 'mythbuntu.xml'
> 2014-10-14 16:47:11.529+0000: 4668: info : qemuDomainSnapshotLoad:484
> : Scanning for snapshots for domain mythbuntu in
> /var/lib/libvirt/qemu/snapshot/mythbuntu