Libguestfs - Apr 2019 - Re: virt-v2v slow when running inside the VM

Hi,

I'm trying to create a clone of a physical Window VM using p2v.

My goal is to create a cloning tools VM that has libguestfs tools installed
and acts as the convertor.
VM conversion works just fine but the conversion rate is significantly
slow(1/3) when running inside the VM compared to when the v2v is run on the
same bare-metal host.

On the host:
./virt-p2v-20190405-w1f4efxy/virt-v2v-conversion-log.txt:virtual copying
rate: 615.9 M bits/sec
./virt-p2v-20190405-w1f4efxy/virt-v2v-conversion-log.txt:real copying rate:
181.8 M bits/sec
>From the Guest VM (On the same bare-metal host)
virt-p2v-20190405-95azj89j/virt-v2v-conversion-log.txt:virtual copying
rate: 185.1 M bits/sec
virt-p2v-20190405-95azj89j/virt-v2v-conversion-log.txt:real copying rate:
62.7 M bits/sec

I understand there are several factors come into play but i tried to make
the VM comparable by making sure enough CPU / memory is given to the VM.
Also the I played by adjusting the disk cache modes for the VM(cache=none,
cache=unsafe).  When the conversion happens there is not much load and
there are no other VMs on this machine.

I ruled out the disk being the bootleneck because when i do "virt-v2v -i
disk" conversion the VM is only slightly off.
For the same disk image,
virt-v2v when running in the host took '75 sec' whereas in the VM it
took
'100 sec'

How to go about debugging this performance issue? Any pointers would be
helpful

thanks
Suresh

On Tue, Apr 09, 2019 at 06:37:46PM -0700, Sureshkumar Kaliannan
wrote:
> Hi,
> 
> I'm trying to create a clone of a physical Window VM using p2v.
> 
> My goal is to create a cloning tools VM that has libguestfs tools installed
> and acts as the convertor.
> VM conversion works just fine but the conversion rate is significantly
> slow(1/3) when running inside the VM compared to when the v2v is run on the
> same bare-metal host.
> 
> On the host:
> ./virt-p2v-20190405-w1f4efxy/virt-v2v-conversion-log.txt:virtual copying
> rate: 615.9 M bits/sec
> ./virt-p2v-20190405-w1f4efxy/virt-v2v-conversion-log.txt:real copying rate:
> 181.8 M bits/sec
> 
> >From the Guest VM (On the same bare-metal host)
> virt-p2v-20190405-95azj89j/virt-v2v-conversion-log.txt:virtual copying
> rate: 185.1 M bits/sec
> virt-p2v-20190405-95azj89j/virt-v2v-conversion-log.txt:real copying rate:
> 62.7 M bits/sec
> 
> I understand there are several factors come into play but i tried to make
> the VM comparable by making sure enough CPU / memory is given to the VM.
> Also the I played by adjusting the disk cache modes for the VM(cache=none,
> cache=unsafe).  When the conversion happens there is not much load and
> there are no other VMs on this machine.
> 
> I ruled out the disk being the bootleneck because when i do "virt-v2v
-i
> disk" conversion the VM is only slightly off.
> For the same disk image,
> virt-v2v when running in the host took '75 sec' whereas in the VM
it took
> '100 sec'
> 
> How to go about debugging this performance issue? Any pointers would be
> helpful
I think this is just KVM vs TCG?  You could try enabling nested KVM to
see if that makes things faster, but it very much depends on your host
CPU.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

thanks Richard,

The experiment was indeed done with nested VM enabled. I am not sure about
the internals, but i thought once overlay is setup the 2 main processes are
sshd and qemu-img convert (reading data from sshd and doing the conversion)
I don't see any of the qemu process running.
Initial overlay setup was pretty quick and rest of the time was spent in
qemu-img convert operation

Suresh

On Wed, Apr 10, 2019 at 6:22 AM Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Tue, Apr 09, 2019 at 06:37:46PM -0700, Sureshkumar Kaliannan wrote:
> > Hi,
> >
> > I'm trying to create a clone of a physical Window VM using p2v.
> >
> > My goal is to create a cloning tools VM that has libguestfs tools
> installed
> > and acts as the convertor.
> > VM conversion works just fine but the conversion rate is significantly
> > slow(1/3) when running inside the VM compared to when the v2v is run
on
> the
> > same bare-metal host.
> >
> > On the host:
> > ./virt-p2v-20190405-w1f4efxy/virt-v2v-conversion-log.txt:virtual
copying
> > rate: 615.9 M bits/sec
> > ./virt-p2v-20190405-w1f4efxy/virt-v2v-conversion-log.txt:real copying
> rate:
> > 181.8 M bits/sec
> >
> > >From the Guest VM (On the same bare-metal host)
> > virt-p2v-20190405-95azj89j/virt-v2v-conversion-log.txt:virtual copying
> > rate: 185.1 M bits/sec
> > virt-p2v-20190405-95azj89j/virt-v2v-conversion-log.txt:real copying
rate:
> > 62.7 M bits/sec
> >
> > I understand there are several factors come into play but i tried to
make
> > the VM comparable by making sure enough CPU / memory is given to the
VM.
> > Also the I played by adjusting the disk cache modes for the
> VM(cache=none,
> > cache=unsafe).  When the conversion happens there is not much load and
> > there are no other VMs on this machine.
> >
> > I ruled out the disk being the bootleneck because when i do
"virt-v2v -i
> > disk" conversion the VM is only slightly off.
> > For the same disk image,
> > virt-v2v when running in the host took '75 sec' whereas in the
VM it took
> > '100 sec'
> >
> > How to go about debugging this performance issue? Any pointers would
be
> > helpful
>
> I think this is just KVM vs TCG?  You could try enabling nested KVM to
> see if that makes things faster, but it very much depends on your host
> CPU.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> libguestfs lets you edit virtual machines.  Supports shell scripting,
> bindings from many languages.  http://libguestfs.org
>