libvirt users - Dec 2021 - Qemu monitor info tlb gives unable to encode message payload

On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke
wrote:
> Hi,
> 
> the command
>     virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb"
> fails with "error: Unable to encode message payload".
> 
> I found a bugtracker entry for a similiar error [1], but I don't if
this is the same error (message too large). I also don't know how large an
info tlb message is.
> Preferably I would not have to recompile libvirt just to issue this monitor
command..
Libvirt unfortunately limits strings to 4MiB:

const REMOTE_STRING_MAX = 4194304;

And the reply from qemu-monitor-command is a single string. Now
internally we process JSON messages up to 10 MiB so one could argue that
we should increase the size for the 'qemu-monitor-command' reply up to
10MiB. This could be straightforward but it's questionable whether it's
worth it.
> Then I thought about circumventing the error by connecting directly to the
qemu monitor via netcat, but I found a thread [2] that says I cannot add my own
"-monitor tcp:..." to the Qemu commandline arguments.
IIRC at that point qemu wasn't able to handle two monitor connections.
At this point it is possible to have two concurrent connections to the
montitor. Obviously things may break and you get to keep the pieces if
it breaks.

By adding:

  <qemu:commandline>
    <qemu:arg value='-qmp'/>
    <qemu:arg value='tcp:127.0.0.1:1235'/>
  </qemu:commandline>


I then get a connection from qemu on the socket when starting the VM:

  $ nc -l -p 1235
  {"QMP": {"version": {"qemu": {"micro":
93, "minor": 1, "major": 6}, "package":
"v6.2.0-rc3"}, "capabilities": ["oob"]}}

I can then start conversing with the monitor:

  {"execute":"qmp_capabilities"}
  {"return": {}}
 
{"execute":"human-monitor-command","arguments":{"command-line":"info"}}
  {"return": "info balloon  - ....


Out of curiousity, what do you specifically need 'info tlb' for?

On 12/8/21 12:09, Peter Krempa wrote:
> CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know the
content is safe.
>
>
> On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke wrote:
>> Hi,
>>
>> the command
>>      virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info
tlb"
>> fails with "error: Unable to encode message payload".
>>
>> I found a bugtracker entry for a similiar error [1], but I don't if
this is the same error (message too large). I also don't know how large an
info tlb message is.
>> Preferably I would not have to recompile libvirt just to issue this
monitor command..
> Libvirt unfortunately limits strings to 4MiB:
>
> const REMOTE_STRING_MAX = 4194304;
>
> And the reply from qemu-monitor-command is a single string. Now
> internally we process JSON messages up to 10 MiB so one could argue that
> we should increase the size for the 'qemu-monitor-command' reply up
to
> 10MiB. This could be straightforward but it's questionable whether
it's
> worth it.
Thanks for the clarification! I will try to go for the 2nd monitor
then.
>> Then I thought about circumventing the error by connecting directly to
the qemu monitor via netcat, but I found a thread [2] that says I cannot add my
own "-monitor tcp:..." to the Qemu commandline arguments.
> IIRC at that point qemu wasn't able to handle two monitor connections.
> At this point it is possible to have two concurrent connections to the
> montitor. Obviously things may break and you get to keep the pieces if
> it breaks.
>
> By adding:
>
>    <qemu:commandline>
>      <qemu:arg value='-qmp'/>
>      <qemu:arg value='tcp:127.0.0.1:1235'/>
>    </qemu:commandline>
When I add this to the config via virsh edit and then do a shutdown + 
reboot, I get a kernel panic.
I put the corresponding dmesg log on gist: 
https://gist.github.com/klockeph/323f3e0d23a3254ef98a65fd7c8f5a1c
>
> I then get a connection from qemu on the socket when starting the VM:
>
>    $ nc -l -p 1235
>    {"QMP": {"version": {"qemu":
{"micro": 93, "minor": 1, "major": 6},
"package": "v6.2.0-rc3"}, "capabilities":
["oob"]}}
>
> I can then start conversing with the monitor:
>
>    {"execute":"qmp_capabilities"}
>    {"return": {}}
>   
{"execute":"human-monitor-command","arguments":{"command-line":"info"}}
>    {"return": "info balloon  - ....
>
>
> Out of curiousity, what do you specifically need 'info tlb' for?
>
I am currently investigating a performance issue where a Windows VM is 
very slow in accessing qemu ivshmem.
So I want to get some better understanding how this memory is mapped 
into the VM (if it's paginated or continuous etc.), and hoped that info 
tlb could help me there. But I don't know what the output of the command 
is because it never ran. :D

I would prefer using a "real" profiler for this but the only one I
tried
so far (AMD uProf) crashed the VM with a bluescreen...

On 12/8/21 12:09, Peter Krempa wrote
> 
> On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke wrote:
>> Hi,
>>
>> the command
>>      virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info
tlb"
>> fails with "error: Unable to encode message payload".
>>
>> I found a bugtracker entry for a similiar error [1], but I don't if
this is the same error (message too large). I also don't know how large an
info tlb message is.
>> Preferably I would not have to recompile libvirt just to issue this
monitor command..
> 
> Libvirt unfortunately limits strings to 4MiB:
> 
> const REMOTE_STRING_MAX = 4194304;
> 
> And the reply from qemu-monitor-command is a single string. Now
> internally we process JSON messages up to 10 MiB so one could argue that
> we should increase the size for the 'qemu-monitor-command' reply up
to
> 10MiB. This could be straightforward but it's questionable whether
it's
> worth it.
> 
Thanks for the clarification! I will try to go for the 2nd monitor then.
>> Then I thought about circumventing the error by connecting directly to
the qemu monitor via netcat, but I found a thread [2] that says I cannot add my
own "-monitor tcp:..." to the Qemu commandline arguments.
> 
> IIRC at that point qemu wasn't able to handle two monitor connections.
> At this point it is possible to have two concurrent connections to the
> montitor. Obviously things may break and you get to keep the pieces if
> it breaks.
> 
> By adding:
> 
>    <qemu:commandline>
>      <qemu:arg value='-qmp'/>
>      <qemu:arg value='tcp:127.0.0.1:1235'/>
>    </qemu:commandline>
> 
When I add this to the config via virsh edit and then do a shutdown + 
reboot, I get a kernel panic.
I put the corresponding dmesg log on gist: 
https://gist.github.com/klockeph/323f3e0d23a3254ef98a65fd7c8f5a1c
> 
> 
> Out of curiousity, what do you specifically need 'info tlb' for?
> 
I am currently investigating a performance issue where a Windows VM is 
very slow in accessing qemu ivshmem.
So I want to get some better understanding how this memory is mapped 
into the VM (if it's paginated or continuous etc.), and hoped that info 
tlb could help me there. But I don't know what the output of the command 
is because it never ran. :D

I would prefer using a "real" profiler for this but the only one I
tried
so far (AMD uProf) crashed the VM with a bluescreen...