Libguestfs - Sep 2020 - Re: [PATCH v2v] v2v: Set the number of vCPUs to same as host number of pCPUs.

So it didn't make any noticable difference in my test.  I wonder if
the test guest I'm using (Fedora 32 using dracut) doesn't use parallel
compression?

However I don't think it can cause a problem and it seems obvious that
it could benefit some cases.

Rich.

This helps mkinitrd which runs pigz (parallel gzip).

Thanks Jean-Louis Dupond for suggesting this change.
---
 common     | 2 +-
 v2v/v2v.ml | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/common b/common
index c840f2e39..ea5278bba 160000
--- a/common
+++ b/common
@@ -1 +1 @@
-Subproject commit c840f2e39d0bb637a98b224c89f6df011e1d4414
+Subproject commit ea5278bba7800e3a6d9c233a0f5f53a9acf244f8
diff --git a/v2v/v2v.ml b/v2v/v2v.ml
index 73edff2c4..7637b1a8c 100644
--- a/v2v/v2v.ml
+++ b/v2v/v2v.ml
@@ -88,6 +88,8 @@ let rec main ()  
   let g = open_guestfs ~identifier:"v2v" () in
   g#set_memsize (g#get_memsize () * 14 / 5);
+  (* Setting the number of vCPUs allows parallel mkinitrd. *)
+  g#set_smp (Sysconf.nr_processors_online ());
   (* The network is only used by the unconfigure_vmware () function. *)
   g#set_network true;
   (match conversion_mode with
-- 
2.27.0

On Fri, Sep 18, 2020 at 10:44:03AM +0100, Richard W.M. Jones
wrote:
> So it didn't make any noticable difference in my test.  I wonder if
> the test guest I'm using (Fedora 32 using dracut) doesn't use
parallel
> compression?
Do you do anything special to optimize storage ? If the thing using
parallel CPUs in the guest is doing I/O you'd likely want to tune
storage at same time. In current QEMU git master there's a change[1]
that sets  viort-blk/virtio-scsi  num_queues to match SMP count,
so the guest OS get fully parallel I/O from meach guest CPU.

[1] a4eef0711b2cf7a7476c3e2c202a414b68a1baa0
> However I don't think it can cause a problem and it seems obvious that
> it could benefit some cases.
What's the default RAM size ?  On hosts with very large numbers of CPUs
(100's) you might need to scale up RAM size too so you keep a sane
RAM-per-CPU ratio.  Or default to host CPUs, but cap it at say 32.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Fri, Sep 18, 2020 at 10:52:58AM +0100, Daniel P. Berrangé
wrote:
> On Fri, Sep 18, 2020 at 10:44:03AM +0100, Richard W.M. Jones wrote:
> > So it didn't make any noticable difference in my test.  I wonder
if
> > the test guest I'm using (Fedora 32 using dracut) doesn't use
parallel
> > compression?
> 
> Do you do anything special to optimize storage ? If the thing using
> parallel CPUs in the guest is doing I/O you'd likely want to tune
> storage at same time. In current QEMU git master there's a change[1]
> that sets  viort-blk/virtio-scsi  num_queues to match SMP count,
> so the guest OS get fully parallel I/O from meach guest CPU.
> 
> [1] a4eef0711b2cf7a7476c3e2c202a414b68a1baa0
I'm not using a new enough qemu to have that and it doesn't appear in
a released version, but I'll test again once we have a new enough
version.
> > However I don't think it can cause a problem and it seems obvious
that
> > it could benefit some cases.
> 
> What's the default RAM size ?  On hosts with very large numbers of CPUs
> (100's) you might need to scale up RAM size too so you keep a sane
> RAM-per-CPU ratio.  Or default to host CPUs, but cap it at say 32.
I'll cap the number of CPUs in this patch.

Thanks,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

On Friday, 18 September 2020 11:44:04 CEST Richard W.M. Jones
wrote:
>    let g = open_guestfs ~identifier:"v2v" () in
>    g#set_memsize (g#get_memsize () * 14 / 5);
> +  (* Setting the number of vCPUs allows parallel mkinitrd. *)
> +  g#set_smp (Sysconf.nr_processors_online ());
IMHO this is not a good idea, for few reasons:

a) it unconditionally uses all the available CPUs with no way to change
it to any other value

b) virt-v2v is run also on systems that are not specifically dedicated
as conversion servers, for example the oVirt hosts (that run the oVirt
VMs); in this context, suddently using more CPUs than requested may
saturate the system resources

IMHO the right approach is to add a --smp N option like available in
other libguestfs tools; this way, the user can decide how many
resources virt-v2v can use.

-- 
Pino Toscano