Philipp Klocke
2021-Dec-08 12:32 UTC
Qemu monitor info tlb gives unable to encode message payload
On 12/8/21 12:09, Peter Krempa wrote:> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke wrote: >> Hi, >> >> the command >> virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb" >> fails with "error: Unable to encode message payload". >> >> I found a bugtracker entry for a similiar error [1], but I don't if this is the same error (message too large). I also don't know how large an info tlb message is. >> Preferably I would not have to recompile libvirt just to issue this monitor command.. > Libvirt unfortunately limits strings to 4MiB: > > const REMOTE_STRING_MAX = 4194304; > > And the reply from qemu-monitor-command is a single string. Now > internally we process JSON messages up to 10 MiB so one could argue that > we should increase the size for the 'qemu-monitor-command' reply up to > 10MiB. This could be straightforward but it's questionable whether it's > worth it.Thanks for the clarification! I will try to go for the 2nd monitor then.>> Then I thought about circumventing the error by connecting directly to the qemu monitor via netcat, but I found a thread [2] that says I cannot add my own "-monitor tcp:..." to the Qemu commandline arguments. > IIRC at that point qemu wasn't able to handle two monitor connections. > At this point it is possible to have two concurrent connections to the > montitor. Obviously things may break and you get to keep the pieces if > it breaks. > > By adding: > > <qemu:commandline> > <qemu:arg value='-qmp'/> > <qemu:arg value='tcp:127.0.0.1:1235'/> > </qemu:commandline>When I add this to the config via virsh edit and then do a shutdown + reboot, I get a kernel panic. I put the corresponding dmesg log on gist: https://gist.github.com/klockeph/323f3e0d23a3254ef98a65fd7c8f5a1c> > I then get a connection from qemu on the socket when starting the VM: > > $ nc -l -p 1235 > {"QMP": {"version": {"qemu": {"micro": 93, "minor": 1, "major": 6}, "package": "v6.2.0-rc3"}, "capabilities": ["oob"]}} > > I can then start conversing with the monitor: > > {"execute":"qmp_capabilities"} > {"return": {}} > {"execute":"human-monitor-command","arguments":{"command-line":"info"}} > {"return": "info balloon - .... > > > Out of curiousity, what do you specifically need 'info tlb' for? >I am currently investigating a performance issue where a Windows VM is very slow in accessing qemu ivshmem. So I want to get some better understanding how this memory is mapped into the VM (if it's paginated or continuous etc.), and hoped that info tlb could help me there. But I don't know what the output of the command is because it never ran. :D I would prefer using a "real" profiler for this but the only one I tried so far (AMD uProf) crashed the VM with a bluescreen...
Daniel P. Berrangé
2021-Dec-08 13:15 UTC
Qemu monitor info tlb gives unable to encode message payload
On Wed, Dec 08, 2021 at 01:32:48PM +0100, Philipp Klocke wrote:> On 12/8/21 12:09, Peter Krempa wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > > > On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke wrote: > > > Hi, > > > > > > the command > > > virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb" > > > fails with "error: Unable to encode message payload". > > > > > > I found a bugtracker entry for a similiar error [1], but I don't if this is the same error (message too large). I also don't know how large an info tlb message is. > > > Preferably I would not have to recompile libvirt just to issue this monitor command.. > > Libvirt unfortunately limits strings to 4MiB: > > > > const REMOTE_STRING_MAX = 4194304; > > > > And the reply from qemu-monitor-command is a single string. Now > > internally we process JSON messages up to 10 MiB so one could argue that > > we should increase the size for the 'qemu-monitor-command' reply up to > > 10MiB. This could be straightforward but it's questionable whether it's > > worth it. > Thanks for the clarification! I will try to go for the 2nd monitor then. > > > Then I thought about circumventing the error by connecting directly to the qemu monitor via netcat, but I found a thread [2] that says I cannot add my own "-monitor tcp:..." to the Qemu commandline arguments. > > IIRC at that point qemu wasn't able to handle two monitor connections. > > At this point it is possible to have two concurrent connections to the > > montitor. Obviously things may break and you get to keep the pieces if > > it breaks. > > > > By adding: > > > > <qemu:commandline> > > <qemu:arg value='-qmp'/> > > <qemu:arg value='tcp:127.0.0.1:1235'/> > > </qemu:commandline> > > When I add this to the config via virsh edit and then do a shutdown + > reboot, I get a kernel panic. > I put the corresponding dmesg log on gist: > https://gist.github.com/klockeph/323f3e0d23a3254ef98a65fd7c8f5a1cI can't see how that is related to adding an extra -qmp parameter, since it doesn't affect the guest ABI FWIW, exposing -qmp over a tcp socket is totally insecure - any local process can connect to that. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|