libvirt users - Jun 2017 - Re: libvirtd not accepting connections

On Sun, Jun 04, 2017 at 06:42:39PM -0400, Michael C Cambria
wrote:
>I've upgraded from Fedora 20; probably missed a merge of rpmnew with
>existing .conf; permission problem, some other mistake along the way to
>Fedora 25.
>
Yeah, probably some 'rpm -qV' (or whatever the command to verify all
packages is) could help as well.
>Is there a "how to" similar to [1] that lets one qemu to log that
it was
>invoked and how far it got?
>
>I removed qemu (dnf remote qemu; sudo dnf remove qemu-common)
>build qemu 2.2-maint (assuming this relates to 2:2.7.1-6.fc25) from
>github sources
>installed qemu from sources (into /usr/local/bin)
>
>Things are a bit better.  Where something like "sudo virsh
pool-list"
>would just hang before, now my storage pools actually are listed.  No
>luck listing my VM's, but "virsh list" and "virsh list
--all" do not
>hang like before:
>
># virsh list
Are you sure you didn't miss the --all?
>  Id    Name                           State
>----------------------------------------------------
>
># virsh pool-list
>  Name                 State      Autostart
>-------------------------------------------
>  default              active     yes
>  Downloads            active     yes
>  guest_images_lvm     active     yes
>
>#
>
>VM xml is /etc/libvirt/qemu.  The network, virbr0 is in
>/etc/libvirt/qemu/networks, and that gets created just fine.  All have
>root:root owner:group:
>
The VMs are not visible because the XML cannot be parsed if the binaries
are not on the system (the XML contains the whole path).  Also, I think
this works because libvirt doesn't look into /usr/local/bin, but I might
be wrong.  Check whether 'virsh capabilities' tells you something about
any emulator.

You can try installing from source, but putting it in /usr/bin, you can
also remove that installation, put back the one from the package and try
running:

  { for i in qmp_capabilities query-commands quit; do echo
  "{'execute':'$i'}"; done } | qemu-system-x86_64
-nographic -nodefaults
  -no-user-config -M none -qmp stdio

And see whether the QEMU process quits, what it outputs and if it gets
stuck, you can attach gdb and see what it's waiting for.  Or maybe try
running it with strace.

You can also do a thing I used to do a lot.  You can rename
/usr/bin/qemu-system-x86_64 (for example) and create a script with that
filename that for example execs as qemu in strace with the output of
strace put in some file, or similar.  I can't think of anything else for
now, sorry.

Have a nice day,
Martin

P.S.: If qemu gets stuc even with that easy command, then it's
      definitely its fault.

On 06/05/2017 10:46 AM, Martin Kletzander wrote:
> On Sun, Jun 04, 2017 at 06:42:39PM -0400, Michael C Cambria wrote:
>> I've upgraded from Fedora 20; probably missed a merge of rpmnew
with
>> existing .conf; permission problem, some other mistake along the way to
>> Fedora 25.
>>
>
> Yeah, probably some 'rpm -qV' (or whatever the command to verify
all
> packages is) could help as well.
"rpm -aV" didn't turn up anything obvious:

S.5....T.  c /etc/libvirt/libvirtd.conf was modified for logging:
# diff libvirtd.conf libvirtd.conf.rpmnew
1,4d0
< log_level = 1
< log_filters="1:remote 1:event 1:json 1:rpc 1:qemu"
< log_outputs="1:file:/var/log/libvirt/libvirtd.log"
< #
#
>> Is there a "how to" similar to [1] that lets one qemu to log
that it was
>> invoked and how far it got?
>>
>> I removed qemu (dnf remote qemu; sudo dnf remove qemu-common)
>> build qemu 2.2-maint (assuming this relates to 2:2.7.1-6.fc25) from
>> github sources
>> installed qemu from sources (into /usr/local/bin)
>>
>> Things are a bit better.  Where something like "sudo virsh
pool-list"
>> would just hang before, now my storage pools actually are listed.  No
>> luck listing my VM's, but "virsh list" and "virsh
list --all" do not
>> hang like before:
>>
>> # virsh list
>
> Are you sure you didn't miss the --all?
>
Yes I'm sure.  I wish that was all it was <g>
>>  Id    Name State
>> ----------------------------------------------------
>>
>> # virsh pool-list
>>  Name                 State      Autostart
>> -------------------------------------------
>>  default              active     yes
>>  Downloads            active     yes
>>  guest_images_lvm     active     yes
>>
>> #
>>
>> VM xml is /etc/libvirt/qemu.  The network, virbr0 is in
>> /etc/libvirt/qemu/networks, and that gets created just fine. All have
>> root:root owner:group:
>>
>
> The VMs are not visible because the XML cannot be parsed if the binaries
> are not on the system (the XML contains the whole path).  Also, I think
> this works because libvirt doesn't look into /usr/local/bin, but I
might
> be wrong.  Check whether 'virsh capabilities' tells you something
about
> any emulator.
>
> You can try installing from source, but putting it in /usr/bin, 

I put qemu-system-x86_64 and qemu-x86_64 in /usr/bin as requested. No 
luck.  It seems /usr/local/bin/qemu-xxx is found (see CGroup: output)

$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
    Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; 
vendor preset: enabled)
    Active: active (running) since Mon 2017-06-05 18:21:20 EDT; 3s ago
      Docs: man:libvirtd(8)
            http://libvirt.org
  Main PID: 7076 (libvirtd)
     Tasks: 19 (limit: 4915)
    Memory: 124.7M
       CPU: 3.649s
    CGroup: /system.slice/libvirtd.service
            ├─7076 /usr/sbin/libvirtd
            └─7208 /usr/local/bin/qemu-system-sh4eb -S -no-user-config 
-nodefaults -nographic -M none -qmp 
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait 
-pidfile /var/lib/libvirt/qemu/capabi

$ sudo virsh list --all
  Id    Name                           State
----------------------------------------------------

$ sudo virsh pool-list
  Name                 State      Autostart
-------------------------------------------
  default              active     yes
  Downloads            active     yes
  guest_images_lvm     active     yes

$
> you can
> also remove that installation, put back the one from the package and try
> running:
>
>  { for i in qmp_capabilities query-commands quit; do echo
>  "{'execute':'$i'}"; done } | qemu-system-x86_64
-nographic -nodefaults
>  -no-user-config -M none -qmp stdio
>
> And see whether the QEMU process quits, what it outputs and if it gets
> stuck, you can attach gdb and see what it's waiting for.  Or maybe try
> running it with strace.
This completed almost instantly.
> You can also do a thing I used to do a lot.  You can rename
> /usr/bin/qemu-system-x86_64 (for example) and create a script with that
> filename that for example execs as qemu in strace with the output of
> strace put in some file, or similar.  I can't think of anything else
for
> now, sorry.
I'll try this ASAP.   First I'll look at all the QEMU_MONITOR_* log 
entries that contain "error" that I *am* getting with qemu installed 
from source (in /usr/local/bin/qemu-*)

2017-06-05 23:06:39.884+0000: 15559: info : 
virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=11 
events=1
2017-06-05 23:06:39.884+0000: 15559: info : virObjectRef:296 : 
OBJECT_REF: obj=0x7f0a603d33b0
2017-06-05 23:06:39.884+0000: 15559: info : qemuMonitorIOProcess:429 : 
QEMU_MONITOR_IO_PROCESS: mon=0x7f0a603d33b0 buf={"id":
"libvirt-41",
"error": {"class": "GenericError",
"desc": "this feature or command is
not currently supported"}}^M
  len=120
2017-06-05 23:06:39.884+0000: 15559: debug : 
qemuMonitorJSONIOProcessLine:191 : Line [{"id":
"libvirt-41", "error":
{"class": "GenericError", "desc": "this
feature or command is not
currently supported"}}]
2017-06-05 23:06:39.884+0000: 15559: debug : virJSONValueFromString:1604 
: string={"id": "libvirt-41", "error":
{"class": "GenericError", "desc":
"this feature or command is not currently supported"}}
2017-06-05 23:06:39.884+0000: 15559: debug : 
virJSONParserHandleStartMap:1478 : parser=0x7ffe4440afd0

> Have a nice day,
> Martin
>
Thank you, you too!

On Mon, Jun 05, 2017 at 07:52:58PM -0400, Michael C Cambria
wrote:
>
>
>On 06/05/2017 10:46 AM, Martin Kletzander wrote:
>> On Sun, Jun 04, 2017 at 06:42:39PM -0400, Michael C Cambria wrote:
>>> I've upgraded from Fedora 20; probably missed a merge of rpmnew
with
>>> existing .conf; permission problem, some other mistake along the
way to
>>> Fedora 25.
>>>
>>
>> Yeah, probably some 'rpm -qV' (or whatever the command to
verify all
>> packages is) could help as well.
>
>"rpm -aV" didn't turn up anything obvious:
>
>S.5....T.  c /etc/libvirt/libvirtd.conf was modified for logging:
># diff libvirtd.conf libvirtd.conf.rpmnew
>1,4d0
>< log_level = 1
>< log_filters="1:remote 1:event 1:json 1:rpc 1:qemu"
>< log_outputs="1:file:/var/log/libvirt/libvirtd.log"
>< #
>#
>
>>> Is there a "how to" similar to [1] that lets one qemu to
log that it was
>>> invoked and how far it got?
>>>
>>> I removed qemu (dnf remote qemu; sudo dnf remove qemu-common)
>>> build qemu 2.2-maint (assuming this relates to 2:2.7.1-6.fc25) from
>>> github sources
>>> installed qemu from sources (into /usr/local/bin)
>>>
>>> Things are a bit better.  Where something like "sudo virsh
pool-list"
>>> would just hang before, now my storage pools actually are listed. 
No
>>> luck listing my VM's, but "virsh list" and
"virsh list --all" do not
>>> hang like before:
>>>
>>> # virsh list
>>
>> Are you sure you didn't miss the --all?
>>
>Yes I'm sure.  I wish that was all it was <g>
>
>>>  Id    Name State
>>> ----------------------------------------------------
>>>
>>> # virsh pool-list
>>>  Name                 State      Autostart
>>> -------------------------------------------
>>>  default              active     yes
>>>  Downloads            active     yes
>>>  guest_images_lvm     active     yes
>>>
>>> #
>>>
>>> VM xml is /etc/libvirt/qemu.  The network, virbr0 is in
>>> /etc/libvirt/qemu/networks, and that gets created just fine. All
have
>>> root:root owner:group:
>>>
>>
>> The VMs are not visible because the XML cannot be parsed if the
binaries
>> are not on the system (the XML contains the whole path).  Also, I think
>> this works because libvirt doesn't look into /usr/local/bin, but I
might
>> be wrong.  Check whether 'virsh capabilities' tells you
something about
>> any emulator.
>>
>> You can try installing from source, but putting it in /usr/bin,
>
>
>I put qemu-system-x86_64 and qemu-x86_64 in /usr/bin as requested. No
>luck.  It seems /usr/local/bin/qemu-xxx is found (see CGroup: output)
>
>$ sudo systemctl status libvirtd.service
>● libvirtd.service - Virtualization daemon
>    Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
>vendor preset: enabled)
>    Active: active (running) since Mon 2017-06-05 18:21:20 EDT; 3s ago
>      Docs: man:libvirtd(8)
>            http://libvirt.org
>  Main PID: 7076 (libvirtd)
>     Tasks: 19 (limit: 4915)
>    Memory: 124.7M
>       CPU: 3.649s
>    CGroup: /system.slice/libvirtd.service
>            ├─7076 /usr/sbin/libvirtd
>            └─7208 /usr/local/bin/qemu-system-sh4eb -S -no-user-config
>-nodefaults -nographic -M none -qmp
>unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
>-pidfile /var/lib/libvirt/qemu/capabi
>
So this is now only with the qemu from source installed on the system?
>$ sudo virsh list --all
>  Id    Name                           State
>----------------------------------------------------
>
>$ sudo virsh pool-list
>  Name                 State      Autostart
>-------------------------------------------
>  default              active     yes
>  Downloads            active     yes
>  guest_images_lvm     active     yes
>
>$
>
>> you can
>> also remove that installation, put back the one from the package and
try
>> running:
>>
>>  { for i in qmp_capabilities query-commands quit; do echo
>>  "{'execute':'$i'}"; done } |
qemu-system-x86_64 -nographic -nodefaults
>>  -no-user-config -M none -qmp stdio
>>
>> And see whether the QEMU process quits, what it outputs and if it gets
>> stuck, you can attach gdb and see what it's waiting for.  Or maybe
try
>> running it with strace.
>
>This completed almost instantly.
>
And that is with the QEMU installed form the package, right?
>> You can also do a thing I used to do a lot.  You can rename
>> /usr/bin/qemu-system-x86_64 (for example) and create a script with that
>> filename that for example execs as qemu in strace with the output of
>> strace put in some file, or similar.  I can't think of anything
else for
>> now, sorry.
>
>I'll try this ASAP.   First I'll look at all the QEMU_MONITOR_* log
>entries that contain "error" that I *am* getting with qemu
installed
>from source (in /usr/local/bin/qemu-*)
>
>2017-06-05 23:06:39.884+0000: 15559: info :
>virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=11
>events=1
>2017-06-05 23:06:39.884+0000: 15559: info : virObjectRef:296 :
>OBJECT_REF: obj=0x7f0a603d33b0
>2017-06-05 23:06:39.884+0000: 15559: info : qemuMonitorIOProcess:429 :
>QEMU_MONITOR_IO_PROCESS: mon=0x7f0a603d33b0 buf={"id":
"libvirt-41",
>"error": {"class": "GenericError",
"desc": "this feature or command is
>not currently supported"}}^M
>  len=120
Well, at least QEMU_MONITOR_ messages are showing, finally.  The non-x86
machines might be missing lot of commands that libvirt needs.  Like
'query-commands' which we need to know which commands we can run (bit of
a catch 22 right there).  But these should not happen with x86, x86_64,
arm, arm64 (or aarch64), and few others.  Those are being maintained
continuously.
>2017-06-05 23:06:39.884+0000: 15559: debug :
>qemuMonitorJSONIOProcessLine:191 : Line [{"id":
"libvirt-41", "error":
>{"class": "GenericError", "desc": "this
feature or command is not
>currently supported"}}]
>2017-06-05 23:06:39.884+0000: 15559: debug : virJSONValueFromString:1604
>: string={"id": "libvirt-41", "error":
{"class": "GenericError", "desc":
>"this feature or command is not currently supported"}}
>2017-06-05 23:06:39.884+0000: 15559: debug :
>virJSONParserHandleStartMap:1478 : parser=0x7ffe4440afd0
>
>
>> Have a nice day,
>> Martin
>>
>Thank you, you too!
>
>
>_______________________________________________
>libvirt-users mailing list
>libvirt-users@redhat.com
>https://www.redhat.com/mailman/listinfo/libvirt-users