Libguestfs - Jan 2014 - Re: Notes on building libguestfs in a systemd-nspawn container

On Thu, Jan 30, 2014 at 11:50:35AM +0530, Kashyap Chamarthy
wrote:
> > - Single `make` job timing to compile everything:
> > 
> >     real    31m9.792s
> >     user    17m18.359s
> >     sys     13m17.868s
> 
> For comparison, on the _host_, the same single `make` job timing:
> 
>     real    13m41.440s
>     user    13m5.816s
>     sys     1m9.911s
This is timing the build only?

I'm surprised it is slower in the container.  Is memory or # CPUs
limited?
> > 
> > - `make -k check` is still running as I write this, albeit
> >    a bit slow.
> 
> This just finished (in the container):
> 
>     [. . .]
>     grep -v -E
'^(examples|gnulib|perl/(blib|examples)|po-docs|tests)/' | \
>     grep -v -E '/((guestfs|rc)_protocol\.c)$' | \
>     LC_ALL=C sort > po/POTFILES
>     cd .; \
>     find builder mllib resize sparsify sysprep -name '*.ml' | \
>     LC_ALL=C sort > po/POTFILES-ml
>     make[1]: Leaving directory `/root/libguestfs'
>     make: *** [check-recursive] Error 1
>       GEN      public-submodule-commit
>     make: Target `check' not remade because of errors.
> 
>     real    474m53.630s
>     user    325m54.254s
>     sys     205m58.032s
> 
>     -bash-4.2# git log | head -1
>     commit c841d08d7084db69e81614d54423686cf0566ad6
> 
> 
> Again, for comparison, `make -k check` on _host_:
> 
>     real    63m1.078s
>     user    54m39.393s
>     sys     12m8.130s
Is KVM available in the container?  I've never tried that actually ..

I suppose the next step is to make LIBGUESTFS_BACKEND=libvirt:lxc:///
work!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)

On Thu, Jan 30, 2014 at 10:28:31AM +0000, Richard W.M. Jones
wrote:
> On Thu, Jan 30, 2014 at 11:50:35AM +0530, Kashyap Chamarthy wrote:
> > 
> > Again, for comparison, `make -k check` on _host_:
> > 
> >     real    63m1.078s
> >     user    54m39.393s
> >     sys     12m8.130s
> 
> Is KVM available in the container?  I've never tried that actually ..
I don't believe so. IIRC systemd-nspawn will populate a tmpfs on /dev
with only zero, null, full, random, unrandom, tty nodes.

If you use libvirt to setup the container then you can tell libvirt
to add in the /dev/kvm node too.

This shows how to setup the basic Fedora OS container

 
https://www.berrange.com/posts/2013/08/12/running-a-full-fedora-os-inside-a-libvirt-lxc-guest/

Then update the XML to passthrough the /dev/kvm char device as a
hostdev:

  http://libvirt.org/formatdomain.html#elementsHostDevCaps

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

On 01/30/2014 03:58 PM, Richard W.M. Jones wrote:
> On Thu, Jan 30, 2014 at 11:50:35AM +0530, Kashyap Chamarthy wrote:
>>> - Single `make` job timing to compile everything:
>>>
>>>     real    31m9.792s
>>>     user    17m18.359s
>>>     sys     13m17.868s
>>
>> For comparison, on the _host_, the same single `make` job timing:
>>
>>     real    13m41.440s
>>     user    13m5.816s
>>     sys     1m9.911s
> 
> This is timing the build only?
Yes.

(I wonder if Btrfs matters here.)
> 
> I'm surprised it is slower in the container.  Is memory or # CPUs
> limited?
I haven't done cgroups tuning or deep introspection or any such. It's
just a default invocation of `systemd-nspawn`. That said, from inside
the container:

========-bash-4.2# cat /proc/cpuinfo | grep processor | wc -l
48

-bash-4.2# free -m
             total       used       free     shared    buffers     cached
Mem:         64259      12461      51798         49          1      10757
-/+ buffers/cache:       1703      62556
Swap:        13996          0      13996
-bash-4.2#
========

The machine is  Intel(R) Xeon(R) CPU E7- 4807  @ 1.87GHz, it has 48
processors and 60 GB memory
> 
>>>
>>> - `make -k check` is still running as I write this, albeit
>>>    a bit slow.
>>
>> This just finished (in the container):
>>
>>     [. . .]
>>     grep -v -E
'^(examples|gnulib|perl/(blib|examples)|po-docs|tests)/' | \
>>     grep -v -E '/((guestfs|rc)_protocol\.c)$' | \
>>     LC_ALL=C sort > po/POTFILES
>>     cd .; \
>>     find builder mllib resize sparsify sysprep -name '*.ml' | \
>>     LC_ALL=C sort > po/POTFILES-ml
>>     make[1]: Leaving directory `/root/libguestfs'
>>     make: *** [check-recursive] Error 1
>>       GEN      public-submodule-commit
>>     make: Target `check' not remade because of errors.
>>
>>     real    474m53.630s
>>     user    325m54.254s
>>     sys     205m58.032s
>>
>>     -bash-4.2# git log | head -1
>>     commit c841d08d7084db69e81614d54423686cf0566ad6
>>
>>
>> Again, for comparison, `make -k check` on _host_:
>>
>>     real    63m1.078s
>>     user    54m39.393s
>>     sys     12m8.130s
> 
> Is KVM available in the container?  I've never tried that actually ..
No it isn't (as Dan noted in his next thread)

========-bash-4.2# file /dev/kvm
/dev/kvm: ERROR: cannot open `/dev/kvm' (No such file or directory)
========-bash-4.2# virt-host-validate
  QEMU: Checking for hardware virtualization
     : PASS
  QEMU: Checking for device /dev/kvm
     : FAIL (Check that the 'kvm-intel' or 'kvm-amd' modules are
loaded
& the BIOS has enabled virtualization)
  QEMU: Checking for device /dev/vhost-net
     : WARN (Load the 'vhost_net' module to improve performance of
virtio networking)
  QEMU: Checking for device /dev/net/tun
     : FAIL (Load the 'tun' module to enable networking for QEMU guests)
   LXC: Checking for Linux >= 2.6.26
     : PASS
========
Despite reading from the `systemd-nspawn` man page:

 ". . .kernel modules may not be loaded from within the container."

I purposefully tried from inside the container:
========-bash-4.2# modprobe kvm_intel
-bash-4.2# echo $?
1
-bash-4.2# file /dev/kvm
/dev/kvm: ERROR: cannot open `/dev/kvm' (No such file or directory)
-bash-4.2#
========
> 
> I suppose the next step is to make LIBGUESTFS_BACKEND=libvirt:lxc:///
> work!
> 
> Rich.
> 

-- 
/kashyap

On Thu, Jan 30, 2014 at 04:34:04PM +0530, Kashyap Chamarthy
wrote:
> On 01/30/2014 03:58 PM, Richard W.M. Jones wrote:
> >>> - `make -k check` is still running as I write this, albeit
> >>>    a bit slow.
> >>
> >> This just finished (in the container):
> >>
> >>     [. . .]
> >>     grep -v -E
'^(examples|gnulib|perl/(blib|examples)|po-docs|tests)/' | \
> >>     grep -v -E '/((guestfs|rc)_protocol\.c)$' | \
> >>     LC_ALL=C sort > po/POTFILES
> >>     cd .; \
> >>     find builder mllib resize sparsify sysprep -name
'*.ml' | \
> >>     LC_ALL=C sort > po/POTFILES-ml
> >>     make[1]: Leaving directory `/root/libguestfs'
> >>     make: *** [check-recursive] Error 1
> >>       GEN      public-submodule-commit
> >>     make: Target `check' not remade because of errors.
> >>
> >>     real    474m53.630s
> >>     user    325m54.254s
> >>     sys     205m58.032s
> >>
> >>     -bash-4.2# git log | head -1
> >>     commit c841d08d7084db69e81614d54423686cf0566ad6
> >>
> >>
> >> Again, for comparison, `make -k check` on _host_:
> >>
> >>     real    63m1.078s
> >>     user    54m39.393s
> >>     sys     12m8.130s
> > 
> > Is KVM available in the container?  I've never tried that actually
..
> 
> No it isn't (as Dan noted in his next thread)
> 
> ========> -bash-4.2# file /dev/kvm
> /dev/kvm: ERROR: cannot open `/dev/kvm' (No such file or directory)
> ========> -bash-4.2# virt-host-validate
>   QEMU: Checking for hardware virtualization
>      : PASS
>   QEMU: Checking for device /dev/kvm
>      : FAIL (Check that the 'kvm-intel' or 'kvm-amd'
modules are loaded
> & the BIOS has enabled virtualization)
>   QEMU: Checking for device /dev/vhost-net
>      : WARN (Load the 'vhost_net' module to improve performance of
> virtio networking)
>   QEMU: Checking for device /dev/net/tun
>      : FAIL (Load the 'tun' module to enable networking for QEMU
guests)
>    LXC: Checking for Linux >= 2.6.26
>      : PASS
> ========> 
> Despite reading from the `systemd-nspawn` man page:
> 
>  ". . .kernel modules may not be loaded from within the
container."
> 
> I purposefully tried from inside the container:
With container based virt there is only one kernel image, so any
modules you want must be loaded in the host. Libvirt "passthrough"
of char/block devices simply involves libvirt doing mknod in the
/dev tmpfs it sets up. The container itself is blocked from doing
any 'mknod' calls since that'd be a security risk. Hence you must
list any desired device nodes in the XML config.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|