libvirt users - Aug 2012 - virDomainMemoryPeek: bad behavior under workload

Greetings,

I am working on a platform for analysis automation.
I need to run several Virtual Environments concurrently and record
information about their behavior.

I wrote some months ago about the capability of reading the Memory
during the Environment's execution (in paused state).
What do I need is the complete linear memory image, byte per byte,
nothing special; I will give this output to tools and parsers like
Volatility to get the value from it.

I looked around and the only way to get the memory in such a way is
using the QEMU monitor command `pmemsave`.
I am using libvirt through its Python bindings and the
virDomainQemuMonitorCommand seems not to be exposed by the API so, as
suggested in some mails I read into the mailig list, I switched to
virDomainMemoryPeek.

Using this function keeps up to 14-16 seconds to read 512Mb of memory
with the 64Kb limitation and 2-3 seconds with the 1Mb one; but the
most annoying thing is that I can't run several environment
concurrently as the function keeps failing.
Here's the typical output:

  File "/home/nox/workspace/NOX/src/NOX/hooks.py", line 134, in
trigger
    hook.trigger(event)
  File "/home/nox/workspace/NOX/src/NOX/hooks.py", line 33, in trigger
    self.handlers[event]()
  File "/home/nox/workspace/NOX/hooks/volatility.py", line 81, in
memory_dump
    for block in Memory(self.ctx):
  File "/home/see/workspace/NOX/src/NOX/lib/libtools.py", line 179, in
next
    libvirt.VIR_MEMORY_PHYSICAL)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1759, in
memoryPeek
    ret = libvirtmod.virDomainMemoryPeek(self._o, start, size, flags)
SystemError: error return without exception set

I can't run more than 3 environments concurrently on a Xeon Quad with
8Gb of memory.

I guess the RPC reply goes in timeout because the system is under
heavy load but I'm not sure as the error output is quite obscure.
Is there any solution to this issue? Is it possible to raise the RPC
reply timeout value so that, even if slowly, I eventually get the
memory dump?

If through virsh I use the QEMU `pmemsave` command, I get the memory
dump in less than one second; is there any way to obtain the same
performance?

Thanks anyway for making libvirt the great tool it is!

NoxDaFox

On Fri, Aug 31, 2012 at 03:23:18PM +0300, NoxDaFox
wrote:
> Greetings,
> 
> I am working on a platform for analysis automation.
> I need to run several Virtual Environments concurrently and record
> information about their behavior.
> 
> I wrote some months ago about the capability of reading the Memory
> during the Environment's execution (in paused state).
> What do I need is the complete linear memory image, byte per byte,
> nothing special; I will give this output to tools and parsers like
> Volatility to get the value from it.
If you want the complete memory image, perhaps you can just run the
virDomainCoreDump() command, with the VIR_DUMP_MEMORY_ONLY flag
(though this flag only works on very recent QEMU)
> I looked around and the only way to get the memory in such a way is
> using the QEMU monitor command `pmemsave`.
> I am using libvirt through its Python bindings and the
> virDomainQemuMonitorCommand seems not to be exposed by the API so, as
> suggested in some mails I read into the mailig list, I switched to
> virDomainMemoryPeek.
> 
> Using this function keeps up to 14-16 seconds to read 512Mb of memory
> with the 64Kb limitation and 2-3 seconds with the 1Mb one; but the
> most annoying thing is that I can't run several environment
> concurrently as the function keeps failing.
FYI, the virDomainMemoryPeek command was not really designed with
scalability in mind, in particular not really intended for dumping
the entire of guest memory. Its use case was tools like the virt-dmesg
command, where you just want to peek at a handful of small memory
regions.

> Here's the typical output:
> 
>   File "/home/nox/workspace/NOX/src/NOX/hooks.py", line 134, in
trigger
>     hook.trigger(event)
>   File "/home/nox/workspace/NOX/src/NOX/hooks.py", line 33, in
trigger
>     self.handlers[event]()
>   File "/home/nox/workspace/NOX/hooks/volatility.py", line 81, in
memory_dump
>     for block in Memory(self.ctx):
>   File "/home/see/workspace/NOX/src/NOX/lib/libtools.py", line
179, in next
>     libvirt.VIR_MEMORY_PHYSICAL)
>   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1759,
in memoryPeek
>     ret = libvirtmod.virDomainMemoryPeek(self._o, start, size, flags)
> SystemError: error return without exception set
Hmm, that's a peculiar message to see - I can't find anywhere in the
libvirt code that uses that particular messages, so I'm not sure what
has gone wrong here.
> I can't run more than 3 environments concurrently on a Xeon Quad with
> 8Gb of memory.
> 
> I guess the RPC reply goes in timeout because the system is under
> heavy load but I'm not sure as the error output is quite obscure.
> Is there any solution to this issue? Is it possible to raise the RPC
> reply timeout value so that, even if slowly, I eventually get the
> memory dump?
For the memory peek API, we invoked a QEMU monitor command - we should
not timeout on this at all, unless you are trying to invoke other
monitor commands against the same QEMU process concurrently
> If through virsh I use the QEMU `pmemsave` command, I get the memory
> dump in less than one second; is there any way to obtain the same
> performance?
If virsh works properly, then this suggests the problem is somewhere
in the python code, either libvirt's python binding, or your apps
usage.


Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|