Martin Kletzander
2017-Jun-03 20:22 UTC
Re: [libvirt-users] libvirtd not accepting connections
On Sat, Jun 03, 2017 at 09:22:58AM -0400, Michael C Cambria wrote:> > >On 06/02/2017 09:53 AM, Michael C. Cambria wrote: >> >> >> On 06/02/2017 09:43 AM, Martin Kletzander wrote: >>> [adding back the ML, you probably hit reply instead of reply-all, this >>> way other people might help if they know more] >>> >>> On Fri, Jun 02, 2017 at 08:10:01AM -0400, Michael C. Cambria wrote: >>>> >>>> Hi, >>>> >>>> libvirtd never seems to get notified that there is work to do. >>>> journalct >>>> -f indicated that nothing was logged when connections were attempted >>>> via >>>> virsh. >>>> >>>> I also tried 'LIBVIRT_DEBUG=1 libvirtd --verbose' and once startup >>>> finished, there were no more log entries even though virsh attempts >>>> were >>>> made. >>>> >>> >>> That's because it gets overridden by the configuration files. This >>> might be a bug, but it's not related to what's happening. >>> >>>> "ps ax" shows about a dozen "qemu-system-alpha" processes. I don't >>>> know >>>> if it matters but I didn't expect to see this. I didn't intentionally >>>> configure alpha emulations (assuming that's what it is) and certainly >>>> don't want to waste resources having it running. >>>> >>> >>> Libvirt caches the capabilities of the emulators it can find in your >>> system in order not to waste resources. These processes are expected to >>> go away after they reply with all libvirt asks them for. However, it >>> seems like the initialization cannot be completed precisely due to the >>> fact that these processes don't communicate. >>> >>> There might be some details about qemu-system-alpha that are different >>> when compared to, e.g. qemu-system-x86 and libvirt is not (yet) adapted >>> to them, but I installed that emulator and libvirt daemon runs as >>> usual. It looks like a problem in QEMU. Could you, as a workaround, >>> try uninstalling that qemu binary from your system and restarting the >>> service? >>> > >With qemu-system-alpha gone, I now see qemu-system-arm. With all >qemu-system-* removed (using dnf remove) and stopping, then starting >libvirtd, I still see qemu-system-arm even though the executable doesn't >exist: > ># ls -al /usr/bin | grep qemu-system >-rwxr-xr-x. 1 root root 9993416 Apr 13 21:10 qemu-system-i386 >-rwxr-xr-x. 1 root root 10036032 Apr 13 21:10 qemu-system-x86_64 ># systemctl status libvirtd.service >● libvirtd.service - Virtualization daemon > Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; >vendor preset: enabled) > Active: active (running) since Sat 2017-06-03 09:12:19 EDT; 8min ago > Docs: man:libvirtd(8) > http://libvirt.org > Main PID: 5965 (libvirtd) > Tasks: 23 (limit: 4915) > Memory: 128.2M > CPU: 411ms > CGroup: /system.slice/libvirtd.service > ├─1537 /usr/bin/qemu-system-arm -S -no-user-config >-nodefaults -nographic -M none -qmp >unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n > ├─1638 /usr/bin/qemu-system-arm -S -no-user-config >-nodefaults -nographic -M none -qmp >unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n > ├─5965 /usr/sbin/libvirtd > ├─5999 /usr/bin/qemu-system-i386 -S -no-user-config >-nodefaults -nographic -M none -qmp >unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server, > └─6001 /usr/bin/qemu-system-i386 -S -no-user-config >-nodefaults -nographic -M none -qmp >unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server, > >Jun 03 09:12:19 example.com systemd[1]: Starting Virtualization daemon... >Jun 03 09:12:19 example.com systemd[1]: Started Virtualization daemon. >Those look like leftovers from the previous run. I have no idea why they keep running, probably the same bug why they are not responding. Would you mind trying one more thing: Stopping the service, killing all these processes, and then starting it? Let's leave the QEMU issue on a side until we figure this out first, then we can focus on the second issue. I must say this is quite bizarre.> >>> Also, what versions of libvirt and qemu do you have installed? >> >> # LIBVIRT_DEBUG=1 libvirtd --verbose >> 2017-06-02 00:16:30.317+0000: 18088: info : libvirt version: 2.2.1, >> package: 1.fc25 (Fedora Project, 2017-05-10-22:06:21, >> buildvm-29.phx2.fedoraproject.org) >> >> I'll check on qemu as soon as I can get to the machine. The version >> should be the latest one gets via 'dnf update' on fedora 25 > > ># /usr/sbin/libvirtd --version >/usr/sbin/libvirtd (libvirt) 2.2.1 ># qemu-x86_64 --version >qemu-x86_64 version 2.7.1(qemu-2.7.1-6.fc25), Copyright (c) 2003-2016 >Fabrice Bellard and the QEMU Project developers > >[ gdb output deleted]
Michael C Cambria
2017-Jun-03 21:20 UTC
Re: [libvirt-users] libvirtd not accepting connections
On 06/03/2017 04:22 PM, Martin Kletzander wrote:> On Sat, Jun 03, 2017 at 09:22:58AM -0400, Michael C Cambria wrote: >> >> >> On 06/02/2017 09:53 AM, Michael C. Cambria wrote: >>> >>> >>> On 06/02/2017 09:43 AM, Martin Kletzander wrote: >>>> [adding back the ML, you probably hit reply instead of reply-all, this >>>> way other people might help if they know more] >>>> >>>> On Fri, Jun 02, 2017 at 08:10:01AM -0400, Michael C. Cambria wrote: >>>>> >>>>> Hi, >>>>> >>>>> libvirtd never seems to get notified that there is work to do. >>>>> journalct >>>>> -f indicated that nothing was logged when connections were attempted >>>>> via >>>>> virsh. >>>>> >>>>> I also tried 'LIBVIRT_DEBUG=1 libvirtd --verbose' and once startup >>>>> finished, there were no more log entries even though virsh attempts >>>>> were >>>>> made. >>>>> >>>> >>>> That's because it gets overridden by the configuration files. This >>>> might be a bug, but it's not related to what's happening. >>>> >>>>> "ps ax" shows about a dozen "qemu-system-alpha" processes. I don't >>>>> know >>>>> if it matters but I didn't expect to see this. I didn't >>>>> intentionally >>>>> configure alpha emulations (assuming that's what it is) and certainly >>>>> don't want to waste resources having it running. >>>>> >>>> >>>> Libvirt caches the capabilities of the emulators it can find in your >>>> system in order not to waste resources. These processes are >>>> expected to >>>> go away after they reply with all libvirt asks them for. However, it >>>> seems like the initialization cannot be completed precisely due to the >>>> fact that these processes don't communicate. >>>> >>>> There might be some details about qemu-system-alpha that are different >>>> when compared to, e.g. qemu-system-x86 and libvirt is not (yet) >>>> adapted >>>> to them, but I installed that emulator and libvirt daemon runs as >>>> usual. It looks like a problem in QEMU. Could you, as a workaround, >>>> try uninstalling that qemu binary from your system and restarting the >>>> service? >>>> >> >> With qemu-system-alpha gone, I now see qemu-system-arm. With all >> qemu-system-* removed (using dnf remove) and stopping, then starting >> libvirtd, I still see qemu-system-arm even though the executable doesn't >> exist: >> >> # ls -al /usr/bin | grep qemu-system >> -rwxr-xr-x. 1 root root 9993416 Apr 13 21:10 qemu-system-i386 >> -rwxr-xr-x. 1 root root 10036032 Apr 13 21:10 qemu-system-x86_64 >> # systemctl status libvirtd.service >> ● libvirtd.service - Virtualization daemon >> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; >> vendor preset: enabled) >> Active: active (running) since Sat 2017-06-03 09:12:19 EDT; 8min ago >> Docs: man:libvirtd(8) >> http://libvirt.org >> Main PID: 5965 (libvirtd) >> Tasks: 23 (limit: 4915) >> Memory: 128.2M >> CPU: 411ms >> CGroup: /system.slice/libvirtd.service >> ├─1537 /usr/bin/qemu-system-arm -S -no-user-config >> -nodefaults -nographic -M none -qmp >> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n >> ├─1638 /usr/bin/qemu-system-arm -S -no-user-config >> -nodefaults -nographic -M none -qmp >> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n >> ├─5965 /usr/sbin/libvirtd >> ├─5999 /usr/bin/qemu-system-i386 -S -no-user-config >> -nodefaults -nographic -M none -qmp >> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server, >> └─6001 /usr/bin/qemu-system-i386 -S -no-user-config >> -nodefaults -nographic -M none -qmp >> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server, >> >> Jun 03 09:12:19 example.com systemd[1]: Starting Virtualization >> daemon... >> Jun 03 09:12:19 example.com systemd[1]: Started Virtualization daemon. >> > > Those look like leftovers from the previous run. I have no idea why > they keep running, probably the same bug why they are not responding.I dnf remove'ed every qemu-system-* in /usr/bin that dnf allowed me to. The only things are: $ ls -al /usr/bin | grep qemu-system -rwxr-xr-x. 1 root root 9993416 Apr 13 21:10 qemu-system-i386 -rwxr-xr-x. 1 root root 10036032 Apr 13 21:10 qemu-system-x86_64 $ Is there a way to remove them? I still have the original problem, though at least arm and alpha don't show up. $ sudo systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2017-06-03 16:59:40 EDT; 7min ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 1200 (libvirtd) Tasks: 20 (limit: 4915) Memory: 101.4M CPU: 361ms CGroup: /system.slice/libvirtd.service ├─1200 /usr/sbin/libvirtd ├─1481 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. └─1658 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. Jun 03 16:59:32 eastie.fid4.com systemd[1]: Starting Virtualization daemon... Jun 03 16:59:40 eastie.fid4.com systemd[1]: Started Virtualization daemon. $> Would you mind trying one more thing: Stopping the service, killing all > these processes, and then starting it? Let's leave the QEMU issue on a > side until we figure this out first, then we can focus on the second > issue. I must say this is quite bizarre. >As requested... $ sudo systemctl stop libvirtd.service $ sudo systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sat 2017-06-03 17:09:16 EDT; 1s ago Docs: man:libvirtd(8) http://libvirt.org Process: 1200 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS) Main PID: 1200 (code=exited, status=0/SUCCESS) Tasks: 3 (limit: 4915) Memory: 92.4M CPU: 386ms CGroup: /system.slice/libvirtd.service ├─1481 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. └─1658 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. Jun 03 16:59:32 exampl.com systemd[1]: Starting Virtualization daemon... Jun 03 16:59:40 example.com systemd[1]: Started Virtualization daemon. Jun 03 17:09:16 example.com systemd[1]: Stopping Virtualization daemon... Jun 03 17:09:16 example.com systemd[1]: Stopped Virtualization daemon. $ ps ax | grep virt 1481 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 1658 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 3111 pts/1 S+ 0:00 grep --color=auto virt $ sudo kill -9 1481 $ sudo kill -9 1658 $ ps ax | grep virt 3120 pts/1 S+ 0:00 grep --color=auto virt $ sudo systemctl start libvirtd.service $ sudo systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2017-06-03 17:09:52 EDT; 4s ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 3124 (libvirtd) Tasks: 20 (limit: 4915) Memory: 103.4M CPU: 232ms CGroup: /system.slice/libvirtd.service ├─3124 /usr/sbin/libvirtd ├─3158 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. └─3160 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. Jun 03 17:09:52 example.com systemd[1]: Starting Virtualization daemon... Jun 03 17:09:52 example.com systemd[1]: Started Virtualization daemon. $ ps ax | grep virt 3124 ? Ssl 0:00 /usr/sbin/libvirtd 3158 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 3160 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 3166 pts/1 S+ 0:00 grep --color=auto virt $ Problem still exists. I also tried stopping libvirtd, renaming both qemu-system-i386 and qemu-system-x86_64, start libvirtd. Things get further along; dnsmasq log messages show up. $ sudo systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2017-06-03 17:14:25 EDT; 7s ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 3246 (libvirtd) Tasks: 21 (limit: 4915) Memory: 107.4M CPU: 300ms CGroup: /system.slice/libvirtd.service ├─3158 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. ├─3160 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. ├─3246 /usr/sbin/libvirtd ├─3457 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper └─3458 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper Jun 03 17:14:25 example.com libvirtd[3246]: Failed to probe capabilities for /usr/bin/qemu-kvm: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-k Jun 03 17:14:26 example.com dnsmasq[3457]: started, version 2.76 cachesize 150 Jun 03 17:14:26 example.com dnsmasq[3457]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, sockets bound exclusively to interface virbr0 Jun 03 17:14:26 example.com dnsmasq[3457]: reading /etc/resolv.conf Jun 03 17:14:26 example.com dnsmasq[3457]: using nameserver 172.16.8.1#53 Jun 03 17:14:26 example.com dnsmasq[3457]: read /etc/hosts - 2 addresses Jun 03 17:14:26 example.com dnsmasq[3457]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: read /var/lib/libvirt/dnsmasq/default.hostsfile $ sudo virsh list Id Name State ---------------------------------------------------- $ As you can see, none of my vm's show up. I'm guessing the "qemu-kvm: internal error" results from renaming qemu-system-x86_64?? Could something else (selinux?) be involved?>> >>>> Also, what versions of libvirt and qemu do you have installed? >>> >>> # LIBVIRT_DEBUG=1 libvirtd --verbose >>> 2017-06-02 00:16:30.317+0000: 18088: info : libvirt version: 2.2.1, >>> package: 1.fc25 (Fedora Project, 2017-05-10-22:06:21, >>> buildvm-29.phx2.fedoraproject.org) >>> >>> I'll check on qemu as soon as I can get to the machine. The version >>> should be the latest one gets via 'dnf update' on fedora 25 >> >> >> # /usr/sbin/libvirtd --version >> /usr/sbin/libvirtd (libvirt) 2.2.1 >> # qemu-x86_64 --version >> qemu-x86_64 version 2.7.1(qemu-2.7.1-6.fc25), Copyright (c) 2003-2016 >> Fabrice Bellard and the QEMU Project developers >> >> [ gdb output deleted]
Martin Kletzander
2017-Jun-03 21:47 UTC
Re: [libvirt-users] libvirtd not accepting connections
On Sat, Jun 03, 2017 at 05:20:47PM -0400, Michael C Cambria wrote:>I also tried stopping libvirtd, renaming both qemu-system-i386 and >qemu-system-x86_64, start libvirtd. Things get further along; dnsmasq >log messages show up. > >$ sudo systemctl status libvirtd.service >● libvirtd.service - Virtualization daemon > Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; >vendor preset: enabled) > Active: active (running) since Sat 2017-06-03 17:14:25 EDT; 7s ago > Docs: man:libvirtd(8) > http://libvirt.org > Main PID: 3246 (libvirtd) > Tasks: 21 (limit: 4915) > Memory: 107.4M > CPU: 300ms > CGroup: /system.slice/libvirtd.service > ├─3158 /usr/bin/qemu-system-i386 -S -no-user-config >-nodefaults -nographic -M none -qmp >unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait >-pidfile /var/lib/libvirt/qemu/capabilities. > ├─3160 /usr/bin/qemu-system-i386 -S -no-user-config >-nodefaults -nographic -M none -qmp >unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait >-pidfile /var/lib/libvirt/qemu/capabilities. > ├─3246 /usr/sbin/libvirtd > ├─3457 /sbin/dnsmasq >--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro >--dhcp-script=/usr/libexec/libvirt_leaseshelper > └─3458 /sbin/dnsmasq >--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro >--dhcp-script=/usr/libexec/libvirt_leaseshelper > >Jun 03 17:14:25 example.com libvirtd[3246]: Failed to probe capabilities >for /usr/bin/qemu-kvm: internal error: Child process (LC_ALL=C >PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-k >Jun 03 17:14:26 example.com dnsmasq[3457]: started, version 2.76 >cachesize 150 >Jun 03 17:14:26 example.com dnsmasq[3457]: compile time options: IPv6 >GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset >auth DNSSEC loop-detect inotify >Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, IP range >192.168.122.2 -- 192.168.122.254, lease time 1h >Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, sockets bound >exclusively to interface virbr0 >Jun 03 17:14:26 example.com dnsmasq[3457]: reading /etc/resolv.conf >Jun 03 17:14:26 example.com dnsmasq[3457]: using nameserver 172.16.8.1#53 >Jun 03 17:14:26 example.com dnsmasq[3457]: read /etc/hosts - 2 addresses >Jun 03 17:14:26 example.com dnsmasq[3457]: read >/var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses >Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: read >/var/lib/libvirt/dnsmasq/default.hostsfile > > >$ sudo virsh list > Id Name State >---------------------------------------------------- > >$ > >As you can see, none of my vm's show up. I'm guessing the "qemu-kvm: >internal error" results from renaming qemu-system-x86_64?? >Yeah, feel free to move them back.>Could something else (selinux?) be involved?It should not. Now we know it is QEMU, but becuase it is disrupting libvirt in its initialization phase, it looks like nothing works. You could grab a backtrace of the qemu process and try on qemu-devel ML. Also, if you want to make sure (or see after what command gets stuck), you can enable debug logs for libvirt into some file and then grep that for QEMU_MONITOR_ and you'll see. [1] http://wiki.libvirt.org/page/DebugLogs [2] I'm not sure about the whole thing to look for, but this is enough)