Libguestfs - Jan 2016 - What are the advantages and disadvantages of running with or without libvirt?

I didn't see what are the main differences in
http://libguestfs.org/guestfs.3.html#backend

Specifically, I'm interested in what is faster (direct sounds faster to
me), and if there are any major restrictions (networking?)

Here's an example command we are running (sorry, Python'ish, but
you'll get
it):

['virt-sysprep', '--connect', 'qemu:///system',
'-a',
u'/zram/3.6/images/lago_basic_suite_3_6_engine_root.qcow2',
'--selinux-relabel', '--hostname',
'lago_basic_suite_3_6_engine',
'--root-password', 'password:123456', '--mkdir',
'/root/.ssh', '--chmod',
'0700:/root/.ssh', '--upload',
'/zram/3.6/id_rsa.pub:/root/.ssh/authorized_keys',
'--run-command', 'chown
root.root /root/.ssh/authorized_keys', '--mkdir',
'/etc/iscsi', '--chmod',
'0755:/etc/iscsi', '--write',
'/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_engine',
'--mkdir', '/etc/selinux', '--chmod',
'0755:/etc/selinux', '--write',
'/etc/selinux/config:SELINUX=enforcing\nSELINUXTYPE=targeted\n',
'--mkdir',
'/etc/sysconfig/network-scripts', '--chmod',
'0755:/etc/sysconfig/network-scripts', '--write',
'/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:03"\nBOOTPROTO="dhcp"\nTYPE="Ethernet"\nONBOOT="yes"\nNAME="eth0"']

TIA,
Y.

There's two parts to this question.

On Tue, Jan 12, 2016 at 05:26:10PM +0200, Yaniv Kaul
wrote:
> I didn't see what are the main differences in
> http://libguestfs.org/guestfs.3.html#backend
The basic concept of the backend is how do we run the libguestfs
appliance
(http://libguestfs.org/guestfs-internals.1.html#architecture).

There are two ways we could run the appliance: either we run qemu
directly as a forked subprocess of the program that is linked to
libguestfs.so (LIBGUESTFS_BACKEND=direct); or we could run qemu via
libvirt (LIBGUESTFS_BACKEND=libvirt).  The two code paths are largely
equivalent but also quite different in their implementation:

https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c
https://github.com/libguestfs/libguestfs/blob/master/src/launch-libvirt.c
> Specifically, I'm interested in what is faster (direct sounds faster to
> me), and if there are any major restrictions (networking?)
You would think that direct is faster, but I benchmarked this a while
back and there's essentially no measurable difference.

There are however big differences, and restrictions.  Off the top of
my head:

 - libvirt implements sVirt (http://selinuxproject.org/page/SVirt) so
   it's considerably more secure for examining untrusted disk images

 - libvirt allows hotplugging of drives, ie. you can call
   guestfs_add_drive_opts after launching the handle

 - there are some differences in how networking is done, although they
   are rather obscure (and networking is not enabled by default
   anyway, so it would not affect you unless the main program called
   guestfs_set_network (g, 1) before guestfs_launch)

 - libvirt is the supported method to run qemu in RHEL

 - libvirt is way more complex and fails much more frequently :-(

It's unfortunately because of the last point that if libvirt fails to
start the qemu appliance, we print an error message telling you to try
with LIBGUESTFS_BACKEND=direct.

Libvirt has different namespaces where you can run guests, eg.  for
Xen you have to use the xen:/// namespace, for qemu you have a choice
(qemu:///session or qemu:///system).

The libguestfs appliance mostly runs in the qemu:///session (non-root
per-user) namespace, except when you run libguestfs as root when it
has to run in the qemu:///system namespace, but that's just a bug in
libvirt - https://bugzilla.redhat.com/show_bug.cgi?id=890291 .

BUT!  I said there were two parts to this question, and this is the
second part ...

There are several other places where libguestfs calls libvirt, and
they are not related to how libguestfs runs its appliance, and that
seems to be what you're hitting here:
> Here's an example command we are running (sorry, Python'ish, but
you'll get
> it):
> 
> ['virt-sysprep', '--connect', 'qemu:///system',
'-a',
> u'/zram/3.6/images/lago_basic_suite_3_6_engine_root.qcow2',
The virt-sysprep --connect option isn't anything to do with how
libguestfs runs its appliance, and it's not even significant in the
virt-sysprep command you've shown here.

The virt-sysprep --connect option affects how virt-sysprep (and other
tools) process the '-d' option.

If you write:

  virt-sysprep -d some_libvirt_domain --mkdir /foo

then virt-sysprep asks libvirt for the XML of `some_libvirt_domain'
(basically equivalent to running `virsh dumpxml some_libvirt_domain')
and it then parses the libvirt XML to find the disks of
`some_libvirt_domain' so it knows where to create the /foo directory.

https://github.com/libguestfs/libguestfs/blob/master/src/libvirt-domain.c#L468

Because libvirt has different namespaces for libvirt domain names, we
allow you to select the namespace using the -c/--connect option, so
you could for example do:

  virt-sysprep -c xen:/// -d some_xen_domain

which is equivalent to doing `virsh -c xen:/// dumpxml some_xen_domain'
and parsing that XML.

This has nothing to do with how we run the libguestfs appliance.

It's complicated ... and often we get confused too.

HTH,

Rich.
> '--selinux-relabel', '--hostname',
'lago_basic_suite_3_6_engine',
> '--root-password', 'password:123456', '--mkdir',
'/root/.ssh', '--chmod',
> '0700:/root/.ssh', '--upload',
> '/zram/3.6/id_rsa.pub:/root/.ssh/authorized_keys',
'--run-command', 'chown
> root.root /root/.ssh/authorized_keys', '--mkdir',
'/etc/iscsi', '--chmod',
> '0755:/etc/iscsi', '--write',
>
'/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_engine',
> '--mkdir', '/etc/selinux', '--chmod',
'0755:/etc/selinux', '--write',
> '/etc/selinux/config:SELINUX=enforcing\nSELINUXTYPE=targeted\n',
'--mkdir',
> '/etc/sysconfig/network-scripts', '--chmod',
> '0755:/etc/sysconfig/network-scripts', '--write',
>
'/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:03"\nBOOTPROTO="dhcp"\nTYPE="Ethernet"\nONBOOT="yes"\nNAME="eth0"']
> 
> TIA,
> Y.
> _______________________________________________
> Libguestfs mailing list
> Libguestfs@redhat.com
> https://www.redhat.com/mailman/listinfo/libguestfs

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Sorry for the spam, but I had to say it, great explanation :), thanks
a lot!

On 01/12 20:01, Richard W.M. Jones wrote:
> There's two parts to this question.
> 
> On Tue, Jan 12, 2016 at 05:26:10PM +0200, Yaniv Kaul wrote:
> > I didn't see what are the main differences in
> > http://libguestfs.org/guestfs.3.html#backend
> 
> The basic concept of the backend is how do we run the libguestfs
> appliance
> (http://libguestfs.org/guestfs-internals.1.html#architecture).
> 
> There are two ways we could run the appliance: either we run qemu
> directly as a forked subprocess of the program that is linked to
> libguestfs.so (LIBGUESTFS_BACKEND=direct); or we could run qemu via
> libvirt (LIBGUESTFS_BACKEND=libvirt).  The two code paths are largely
> equivalent but also quite different in their implementation:
> 
> https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c
> https://github.com/libguestfs/libguestfs/blob/master/src/launch-libvirt.c
> 
> > Specifically, I'm interested in what is faster (direct sounds
faster to
> > me), and if there are any major restrictions (networking?)
> 
> You would think that direct is faster, but I benchmarked this a while
> back and there's essentially no measurable difference.
> 
> There are however big differences, and restrictions.  Off the top of
> my head:
> 
>  - libvirt implements sVirt (http://selinuxproject.org/page/SVirt) so
>    it's considerably more secure for examining untrusted disk images
> 
>  - libvirt allows hotplugging of drives, ie. you can call
>    guestfs_add_drive_opts after launching the handle
> 
>  - there are some differences in how networking is done, although they
>    are rather obscure (and networking is not enabled by default
>    anyway, so it would not affect you unless the main program called
>    guestfs_set_network (g, 1) before guestfs_launch)
> 
>  - libvirt is the supported method to run qemu in RHEL
> 
>  - libvirt is way more complex and fails much more frequently :-(
> 
> It's unfortunately because of the last point that if libvirt fails to
> start the qemu appliance, we print an error message telling you to try
> with LIBGUESTFS_BACKEND=direct.
> 
> Libvirt has different namespaces where you can run guests, eg.  for
> Xen you have to use the xen:/// namespace, for qemu you have a choice
> (qemu:///session or qemu:///system).
> 
> The libguestfs appliance mostly runs in the qemu:///session (non-root
> per-user) namespace, except when you run libguestfs as root when it
> has to run in the qemu:///system namespace, but that's just a bug in
> libvirt - https://bugzilla.redhat.com/show_bug.cgi?id=890291 .
> 
> BUT!  I said there were two parts to this question, and this is the
> second part ...
> 
> There are several other places where libguestfs calls libvirt, and
> they are not related to how libguestfs runs its appliance, and that
> seems to be what you're hitting here:
> 
> > Here's an example command we are running (sorry, Python'ish,
but you'll get
> > it):
> > 
> > ['virt-sysprep', '--connect',
'qemu:///system', '-a',
> > u'/zram/3.6/images/lago_basic_suite_3_6_engine_root.qcow2',
> 
> The virt-sysprep --connect option isn't anything to do with how
> libguestfs runs its appliance, and it's not even significant in the
> virt-sysprep command you've shown here.
> 
> The virt-sysprep --connect option affects how virt-sysprep (and other
> tools) process the '-d' option.
> 
> If you write:
> 
>   virt-sysprep -d some_libvirt_domain --mkdir /foo
> 
> then virt-sysprep asks libvirt for the XML of `some_libvirt_domain'
> (basically equivalent to running `virsh dumpxml some_libvirt_domain')
> and it then parses the libvirt XML to find the disks of
> `some_libvirt_domain' so it knows where to create the /foo directory.
> 
>
https://github.com/libguestfs/libguestfs/blob/master/src/libvirt-domain.c#L468
> 
> Because libvirt has different namespaces for libvirt domain names, we
> allow you to select the namespace using the -c/--connect option, so
> you could for example do:
> 
>   virt-sysprep -c xen:/// -d some_xen_domain
> 
> which is equivalent to doing `virsh -c xen:/// dumpxml some_xen_domain'
> and parsing that XML.
> 
> This has nothing to do with how we run the libguestfs appliance.
> 
> It's complicated ... and often we get confused too.
> 
> HTH,
> 
> Rich.
> 
> > '--selinux-relabel', '--hostname',
'lago_basic_suite_3_6_engine',
> > '--root-password', 'password:123456',
'--mkdir', '/root/.ssh', '--chmod',
> > '0700:/root/.ssh', '--upload',
> > '/zram/3.6/id_rsa.pub:/root/.ssh/authorized_keys',
'--run-command', 'chown
> > root.root /root/.ssh/authorized_keys', '--mkdir',
'/etc/iscsi', '--chmod',
> > '0755:/etc/iscsi', '--write',
> >
'/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_engine',
> > '--mkdir', '/etc/selinux', '--chmod',
'0755:/etc/selinux', '--write',
> >
'/etc/selinux/config:SELINUX=enforcing\nSELINUXTYPE=targeted\n',
'--mkdir',
> > '/etc/sysconfig/network-scripts', '--chmod',
> > '0755:/etc/sysconfig/network-scripts', '--write',
> >
'/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:03"\nBOOTPROTO="dhcp"\nTYPE="Ethernet"\nONBOOT="yes"\nNAME="eth0"']
> > 
> > TIA,
> > Y.
> 
> > _______________________________________________
> > Libguestfs mailing list
> > Libguestfs@redhat.com
> > https://www.redhat.com/mailman/listinfo/libguestfs
> 
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> libguestfs lets you edit virtual machines.  Supports shell scripting,
> bindings from many languages.  http://libguestfs.org
> 
> _______________________________________________
> Libguestfs mailing list
> Libguestfs@redhat.com
> https://www.redhat.com/mailman/listinfo/libguestfs
-- 
David Caro

Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D

Tel.: +420 532 294 605
Email: dcaro@redhat.com
IRC: dcaro|dcaroest@{freenode|oftc|redhat}
Web: www.redhat.com
RHT Global #: 82-62605

On Tue, Jan 12, 2016 at 10:01 PM, Richard W.M. Jones <rjones@redhat.com>
wrote:
>  - libvirt is way more complex and fails much more frequently :-(
>
> It's unfortunately because of the last point that if libvirt fails to
> start the qemu appliance, we print an error message telling you to try
> with LIBGUESTFS_BACKEND=direct.
>

That was one of the motivations. You were hit by it too apparently (
https://bugzilla.redhat.com/show_bug.cgi?id=1271183 ).

I removed the connect line and set the environment variable and all looks
good. I indeed don't see any much different performance difference.

Thanks!
Y.