Libguestfs - Apr 2019 - Re: virt-v2v slow when running inside the VM

thanks Richard,

The experiment was indeed done with nested VM enabled. I am not sure about
the internals, but i thought once overlay is setup the 2 main processes are
sshd and qemu-img convert (reading data from sshd and doing the conversion)
I don't see any of the qemu process running.
Initial overlay setup was pretty quick and rest of the time was spent in
qemu-img convert operation

Suresh

On Wed, Apr 10, 2019 at 6:22 AM Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Tue, Apr 09, 2019 at 06:37:46PM -0700, Sureshkumar Kaliannan wrote:
> > Hi,
> >
> > I'm trying to create a clone of a physical Window VM using p2v.
> >
> > My goal is to create a cloning tools VM that has libguestfs tools
> installed
> > and acts as the convertor.
> > VM conversion works just fine but the conversion rate is significantly
> > slow(1/3) when running inside the VM compared to when the v2v is run
on
> the
> > same bare-metal host.
> >
> > On the host:
> > ./virt-p2v-20190405-w1f4efxy/virt-v2v-conversion-log.txt:virtual
copying
> > rate: 615.9 M bits/sec
> > ./virt-p2v-20190405-w1f4efxy/virt-v2v-conversion-log.txt:real copying
> rate:
> > 181.8 M bits/sec
> >
> > >From the Guest VM (On the same bare-metal host)
> > virt-p2v-20190405-95azj89j/virt-v2v-conversion-log.txt:virtual copying
> > rate: 185.1 M bits/sec
> > virt-p2v-20190405-95azj89j/virt-v2v-conversion-log.txt:real copying
rate:
> > 62.7 M bits/sec
> >
> > I understand there are several factors come into play but i tried to
make
> > the VM comparable by making sure enough CPU / memory is given to the
VM.
> > Also the I played by adjusting the disk cache modes for the
> VM(cache=none,
> > cache=unsafe).  When the conversion happens there is not much load and
> > there are no other VMs on this machine.
> >
> > I ruled out the disk being the bootleneck because when i do
"virt-v2v -i
> > disk" conversion the VM is only slightly off.
> > For the same disk image,
> > virt-v2v when running in the host took '75 sec' whereas in the
VM it took
> > '100 sec'
> >
> > How to go about debugging this performance issue? Any pointers would
be
> > helpful
>
> I think this is just KVM vs TCG?  You could try enabling nested KVM to
> see if that makes things faster, but it very much depends on your host
> CPU.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> libguestfs lets you edit virtual machines.  Supports shell scripting,
> bindings from many languages.  http://libguestfs.org
>

On Wed, Apr 10, 2019 at 10:15:43AM -0700, Sureshkumar Kaliannan
wrote:
> thanks Richard,
> 
> The experiment was indeed done with nested VM enabled. I am not sure about
> the internals, but i thought once overlay is setup the 2 main processes are
> sshd and qemu-img convert (reading data from sshd and doing the conversion)
Yes this should be true.  I wouldn't expect copying to be slower.

So initial guess probably wrong.  I wonder if there are some extra
steps such as a slow software bridge or openvswitch on the host?
> I don't see any of the qemu process running.
As per http://libguestfs.org/virt-p2v.1.html#how-virt-p2v-works you
should see qemu-nbd running on the physical server, and qemu-img
running on the conversion server during the copying phase.
> Initial overlay setup was pretty quick and rest of the time was spent in
> qemu-img convert operation
Indeed qemu should exit before copying starts.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

I eliminated software-bridge( using iperf3 bandwidth/latency on the host
and guest was almost the same) and
disk write(using qemu-img convert src dest file both on the host and
guestvm was comprable) being the issue.

Next I profiled both sshd and qemu-img during the conversion phase using
bpftools (profile and tcptop) and this is what i see.

Time (min)

File Size

(KB)

sshd RX

(KB)

tcptop

qemu-img RX

(KB)

tcptop

profile (flamegraph)

0 - 5

5478400

5612971

5601536

qemu-img-0-5.svg

5 - 10

8365440 (+2887040)

5669043

5656517

qemu-img-5-10.svg

10 - 15

8423296 (+57856)

5722376

5708926

qemu-img-10-15.svg

15 - 20

9472128(+1048832)

5708230

5693710

qemu-img-15-20.svg

20 - 25

9540992(+68864)

5723629

5712002

qemu-img-20-25.svg

25 - 30

9617024(+76032)

5721568

5708794

qemu-img-25-30.svg

30 - 35

11263040(+10646016)

5689537

5678589

qemu-img-30-35.svg

35 – 40

11263040(+0)

5722598

5709954

qemu-img-35-40.svg

40 - 45

13252160(+1989120)

5693774

5682506

qemu-img-40-45.svg

45 - 50

15272576(+12020416)

5698998

5686398

qemu-img-45-50.svg

50 - 55

15272704(128)

985500

983322

qemu-img-50-55.svg

Bulk of the write to the "-sda" file completed within 15-20 mins
although
sshd/qemu-img receives constant stream of data from the nbd-server
I'm not sure if this is because of "sparseness" (Not sure if the
sparsifying is handled on the nbd-server side or on the receiving side)

The flamegraph also show not much of disk write happening after 20 mins.
(mostly poll and recv_msg)

I am yet todo a similar experiment when running on the "host" to
actually
compare the behavior.

Does this provide any clues?

thanks
Suresh

On Wed, Apr 10, 2019 at 10:30 AM Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Wed, Apr 10, 2019 at 10:15:43AM -0700, Sureshkumar Kaliannan wrote:
> > thanks Richard,
> >
> > The experiment was indeed done with nested VM enabled. I am not sure
> about
> > the internals, but i thought once overlay is setup the 2 main
processes
> are
> > sshd and qemu-img convert (reading data from sshd and doing the
> conversion)
>
> Yes this should be true.  I wouldn't expect copying to be slower.
>
> So initial guess probably wrong.  I wonder if there are some extra
> steps such as a slow software bridge or openvswitch on the host?
>
> > I don't see any of the qemu process running.
>
> As per http://libguestfs.org/virt-p2v.1.html#how-virt-p2v-works you
> should see qemu-nbd running on the physical server, and qemu-img
> running on the conversion server during the copying phase.
>
> > Initial overlay setup was pretty quick and rest of the time was spent
in
> > qemu-img convert operation
>
> Indeed qemu should exit before copying starts.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
>