Re: [libvirt-users] libvirtd not accepting connections

Michael C Cambria
Sat, 03 Jun 2017 21:20:47 -0000

On 06/03/2017 04:22 PM, Martin Kletzander wrote:
> On Sat, Jun 03, 2017 at 09:22:58AM -0400, Michael C Cambria wrote:
>>
>>
>> On 06/02/2017 09:53 AM, Michael C. Cambria wrote:
>>>
>>>
>>> On 06/02/2017 09:43 AM, Martin Kletzander wrote:
>>>> [adding back the ML, you probably hit reply instead of
reply-all, this
>>>> way other people might help if they know more]
>>>>
>>>> On Fri, Jun 02, 2017 at 08:10:01AM -0400, Michael C. Cambria
wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> libvirtd never seems to get notified that there is work to
do.
>>>>> journalct
>>>>> -f indicated that nothing was logged when connections were
attempted
>>>>> via
>>>>> virsh.
>>>>>
>>>>> I also tried 'LIBVIRT_DEBUG=1 libvirtd --verbose'
and once startup
>>>>> finished, there were no more log entries even though virsh
attempts
>>>>> were
>>>>> made.
>>>>>
>>>>
>>>> That's because it gets overridden by the configuration
files. This
>>>> might be a bug, but it's not related to what's
happening.
>>>>
>>>>> "ps ax" shows about a dozen
"qemu-system-alpha" processes.  I don't
>>>>> know
>>>>> if it matters but I didn't expect to see this.  I
didn't
>>>>> intentionally
>>>>> configure alpha emulations (assuming that's what it is)
and certainly
>>>>> don't want to waste resources having it running.
>>>>>
>>>>
>>>> Libvirt caches the capabilities of the emulators it can find in
your
>>>> system in order not to waste resources.  These processes are
>>>> expected to
>>>> go away after they reply with all libvirt asks them for.
However, it
>>>> seems like the initialization cannot be completed precisely due
to the
>>>> fact that these processes don't communicate.
>>>>
>>>> There might be some details about qemu-system-alpha that are
different
>>>> when compared to, e.g. qemu-system-x86 and libvirt is not (yet)
>>>> adapted
>>>> to them, but I installed that emulator and libvirt daemon runs
as
>>>> usual.  It looks like a problem in QEMU.  Could you, as a
workaround,
>>>> try uninstalling that qemu binary from your system and
restarting the
>>>> service?
>>>>
>>
>> With qemu-system-alpha gone, I  now see qemu-system-arm.  With all
>> qemu-system-* removed (using dnf remove) and stopping, then starting
>> libvirtd, I still see qemu-system-arm even though the executable
doesn't
>> exist:
>>
>> # ls -al /usr/bin | grep qemu-system
>> -rwxr-xr-x.   1 root root     9993416 Apr 13 21:10 qemu-system-i386
>> -rwxr-xr-x.   1 root root    10036032 Apr 13 21:10 qemu-system-x86_64
>> # systemctl status libvirtd.service
>> ● libvirtd.service - Virtualization daemon
>>    Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
>> vendor preset: enabled)
>>    Active: active (running) since Sat 2017-06-03 09:12:19 EDT; 8min ago
>>      Docs: man:libvirtd(8)
>>            http://libvirt.org
>>  Main PID: 5965 (libvirtd)
>>     Tasks: 23 (limit: 4915)
>>    Memory: 128.2M
>>       CPU: 411ms
>>    CGroup: /system.slice/libvirtd.service
>>            ├─1537 /usr/bin/qemu-system-arm -S -no-user-config
>> -nodefaults -nographic -M none -qmp
>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n
>>            ├─1638 /usr/bin/qemu-system-arm -S -no-user-config
>> -nodefaults -nographic -M none -qmp
>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n
>>            ├─5965 /usr/sbin/libvirtd
>>            ├─5999 /usr/bin/qemu-system-i386 -S -no-user-config
>> -nodefaults -nographic -M none -qmp
>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,
>>            └─6001 /usr/bin/qemu-system-i386 -S -no-user-config
>> -nodefaults -nographic -M none -qmp
>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,
>>
>> Jun 03 09:12:19 example.com systemd[1]: Starting Virtualization
>> daemon...
>> Jun 03 09:12:19 example.com systemd[1]: Started Virtualization daemon.
>>
>
> Those look like leftovers from the previous run.  I have no idea why
> they keep running, probably the same bug why they are not responding.
I dnf remove'ed every qemu-system-* in /usr/bin that dnf allowed me to.
The only things are:

$ ls -al /usr/bin | grep qemu-system
-rwxr-xr-x.   1 root root     9993416 Apr 13 21:10 qemu-system-i386
-rwxr-xr-x.   1 root root    10036032 Apr 13 21:10 qemu-system-x86_64
$

Is there a way to remove them?  I still have the original problem,
though at least arm and alpha don't show up.

$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
    Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
    Active: active (running) since Sat 2017-06-03 16:59:40 EDT; 7min ago
      Docs: man:libvirtd(8)
            http://libvirt.org
  Main PID: 1200 (libvirtd)
     Tasks: 20 (limit: 4915)
    Memory: 101.4M
       CPU: 361ms
    CGroup: /system.slice/libvirtd.service
            ├─1200 /usr/sbin/libvirtd
            ├─1481 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
            └─1658 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.

Jun 03 16:59:32 eastie.fid4.com systemd[1]: Starting Virtualization
daemon...
Jun 03 16:59:40 eastie.fid4.com systemd[1]: Started Virtualization daemon.
$

> Would you mind trying one more thing:  Stopping the service, killing all
> these processes, and then starting it?  Let's leave the QEMU issue on a
> side until we figure this out first, then we can focus on the second
> issue.  I must say this is quite bizarre.
>

As requested...

$ sudo systemctl stop libvirtd.service
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
    Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
    Active: inactive (dead) since Sat 2017-06-03 17:09:16 EDT; 1s ago
      Docs: man:libvirtd(8)
            http://libvirt.org
   Process: 1200 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
(code=exited, status=0/SUCCESS)
  Main PID: 1200 (code=exited, status=0/SUCCESS)
     Tasks: 3 (limit: 4915)
    Memory: 92.4M
       CPU: 386ms
    CGroup: /system.slice/libvirtd.service
            ├─1481 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
            └─1658 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.

Jun 03 16:59:32 exampl.com systemd[1]: Starting Virtualization daemon...
Jun 03 16:59:40 example.com systemd[1]: Started Virtualization daemon.
Jun 03 17:09:16 example.com systemd[1]: Stopping Virtualization daemon...
Jun 03 17:09:16 example.com systemd[1]: Stopped Virtualization daemon.
$ ps ax | grep virt
  1481 ?        Sl     0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
  1658 ?        S      0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
  3111 pts/1    S+     0:00 grep --color=auto virt
$ sudo kill -9 1481
$ sudo kill -9 1658
$ ps ax | grep virt
  3120 pts/1    S+     0:00 grep --color=auto virt
$ sudo systemctl start libvirtd.service
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
    Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
    Active: active (running) since Sat 2017-06-03 17:09:52 EDT; 4s ago
      Docs: man:libvirtd(8)
            http://libvirt.org
  Main PID: 3124 (libvirtd)
     Tasks: 20 (limit: 4915)
    Memory: 103.4M
       CPU: 232ms
    CGroup: /system.slice/libvirtd.service
            ├─3124 /usr/sbin/libvirtd
            ├─3158 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
            └─3160 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.

Jun 03 17:09:52 example.com systemd[1]: Starting Virtualization daemon...
Jun 03 17:09:52 example.com systemd[1]: Started Virtualization daemon.
$ ps ax | grep virt
  3124 ?        Ssl    0:00 /usr/sbin/libvirtd
  3158 ?        Sl     0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
  3160 ?        S      0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
  3166 pts/1    S+     0:00 grep --color=auto virt
$

Problem still exists.

I also tried stopping libvirtd, renaming both qemu-system-i386 and
qemu-system-x86_64, start libvirtd.  Things get further along; dnsmasq
log messages show up.

$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
    Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
    Active: active (running) since Sat 2017-06-03 17:14:25 EDT; 7s ago
      Docs: man:libvirtd(8)
            http://libvirt.org
  Main PID: 3246 (libvirtd)
     Tasks: 21 (limit: 4915)
    Memory: 107.4M
       CPU: 300ms
    CGroup: /system.slice/libvirtd.service
            ├─3158 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
            ├─3160 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
            ├─3246 /usr/sbin/libvirtd
            ├─3457 /sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper
            └─3458 /sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper

Jun 03 17:14:25 example.com libvirtd[3246]: Failed to probe capabilities
for /usr/bin/qemu-kvm: internal error: Child process (LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-k
Jun 03 17:14:26 example.com dnsmasq[3457]: started, version 2.76
cachesize 150
Jun 03 17:14:26 example.com dnsmasq[3457]: compile time options: IPv6
GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset
auth DNSSEC loop-detect inotify
Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, IP range
192.168.122.2 -- 192.168.122.254, lease time 1h
Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, sockets bound
exclusively to interface virbr0
Jun 03 17:14:26 example.com dnsmasq[3457]: reading /etc/resolv.conf
Jun 03 17:14:26 example.com dnsmasq[3457]: using nameserver 172.16.8.1#53
Jun 03 17:14:26 example.com dnsmasq[3457]: read /etc/hosts - 2 addresses
Jun 03 17:14:26 example.com dnsmasq[3457]: read
/var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: read
/var/lib/libvirt/dnsmasq/default.hostsfile


$ sudo virsh list
  Id    Name                           State
----------------------------------------------------

$

As you can see, none of my vm's show up.  I'm guessing the
"qemu-kvm:
internal error" results from renaming qemu-system-x86_64??

Could something else (selinux?) be involved?
>>
>>>> Also, what versions of libvirt and qemu do you have installed?
>>>
>>> # LIBVIRT_DEBUG=1 libvirtd --verbose
>>> 2017-06-02 00:16:30.317+0000: 18088: info : libvirt version: 2.2.1,
>>> package: 1.fc25 (Fedora Project, 2017-05-10-22:06:21,
>>> buildvm-29.phx2.fedoraproject.org)
>>>
>>> I'll check on qemu as soon as I can get to the machine.  The
version
>>> should be the latest one gets via 'dnf update' on fedora 25
>>
>>
>> # /usr/sbin/libvirtd --version
>> /usr/sbin/libvirtd (libvirt) 2.2.1
>> # qemu-x86_64 --version
>> qemu-x86_64 version 2.7.1(qemu-2.7.1-6.fc25), Copyright (c) 2003-2016
>> Fabrice Bellard and the QEMU Project developers
>>
>> [ gdb output deleted]