Libguestfs - Aug 2011 - [virt-devel] End-user review of the native KVM tool

On Tue, Aug 16, 2011 at 09:40:44PM +0530, Kashyap Chamarthy
wrote:
> Hm..it's been 10 mins..I don't see anything more in the file where
I
> redirected stdout of 'febootstrap' cmd.
> 
> Roughly, can you guess how much time this takes?
For me it has so far taken a lot longer.  The problem is that the
number of dependent packages is probably 100s.

If you want something simple to test out which should complete in a
few minutes, try this example:

https://rwmj.wordpress.com/2010/12/10/tip-creating-throwaway-appliances-with-febootstrap/

Also, for libguestfs (~150 packages) we aggressively configure squid
so that it caches everything properly.  Unfortunately just installing
squid is not sufficient, you have to configure it as well:

https://fedoraproject.org/wiki/Extras/MockTricks#Using_Squid_to_Speed_Up_Mock_package_downloads
http://libguestfs.org/FAQ.html#buildspeed

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

On Tue, Aug 16, 2011 at 10:54:37PM +0530, Kashyap Chamarthy
wrote:
> Is there a something I can do to see it is actually progressing and
> not hung somewhere? Or do I just have to show some patience? :)
Maybe --verbose?  I can't remember if that displays all the individual
downloads.  In any case it took about 30 minutes for me to get all the
packages over my rather slow DSL.

Here are the warnings that febootstrap finally displays:

febootstrap: warning: some host files are unreadable by non-root
febootstrap: warning: get your distro to fix these files:
	     /etc/selinux/targeted/modules/active/commit_num
	     /etc/selinux/targeted/modules/active/file_contexts
	     /etc/selinux/targeted/modules/active/file_contexts.template
	     /etc/selinux/targeted/modules/active/homedir_template
	     /etc/selinux/targeted/modules/active/netfilter_contexts
	     /etc/selinux/targeted/modules/active/seusers.final
	     /etc/selinux/targeted/modules/active/users_extra
	     /lib64/dbus-1/dbus-daemon-launch-helper
	     /sbin/unix_update
	     /usr/bin/chfn
	     /usr/bin/chsh
	     /usr/libexec/utempter/utempter
	     /usr/sbin/build-locale-archive
	     /usr/sbin/glibc_post_upgrade.x86_64
	     /usr/sbin/groupadd
	     /usr/sbin/groupdel
	     /usr/sbin/groupmems
	     /usr/sbin/groupmod
	     /usr/sbin/redhat_lsb_trigger.x86_64
	     /usr/sbin/suexec
	     /usr/sbin/tzdata-update
	     /usr/sbin/useradd
	     /usr/sbin/userdel
	     /usr/sbin/usermod
	     /var/lib/hsqldb/sqltool.rc

febootstrap-supermin-helper relies on reading the host files when it
reconstructs the appliance, so it has to skip files that aren't
readable by non-root (assuming you run the appliance as non-root which
is the usual advice).  I think all of the above can probably be
ignored however.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

Another thing is that febootstrap doesn't run any %pre/%post scripts
from RPMs.  For libguestfs we keep an eye on what these scripts are
doing for the subset of RPMs we use, and sometimes add workarounds in
the appliance /init script.

If FreeIPA is doing anything significant in %pre/%post scripts then it
could require some workarounds.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

On Tue, Aug 16, 2011 at 11:21:44PM +0530, Kashyap Chamarthy
wrote:
> I see. From a quick look at the spec file, I do notice  post, preun,
> postun operations where it deals w/ seliux policy modules, and other
> system commands etc. Can you please elaborate a little on what kind
> of workarounds are we talking here?
You have to imagine that the appliance will boot with just the files
specified directly from the RPMs (ie. the %files sections).  Any other
adjustments done by %pre or %post scripts will not be done.

Before you can boot the appliance (see [1]) you need to write a /init
script, which could jump into the existing initscripts/systemd/..., or
it could do its own thing.  We do our own thing in libguestfs[2].

Therefore, this /init script can do all the operations that are needed
by the IPA %post scripts, such as (from the scripts you sent me)
enabling services, running ipa-upgradeconfig, and doing selinux
tweaks.

Interestingly Debian debootstrap has the same problem, and what they
do is to collect together all the scripts and generate an init script
automatically which does all the required operations.  We don't do
this in febootstrap because we wanted a bit more control for
libguestfs.

Rich.

[1]
https://rwmj.wordpress.com/2010/12/10/tip-creating-throwaway-appliances-with-febootstrap/

[2] http://git.annexia.org/?p=libguestfs.git;a=blob;f=appliance/init;hb=HEAD

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

On Tue, Aug 16, 2011 at 11:47:55PM +0530, Kashyap Chamarthy
wrote:
> On 08/16/2011 11:35 PM, Richard W.M. Jones wrote:
> >On Tue, Aug 16, 2011 at 11:21:44PM +0530, Kashyap Chamarthy wrote:
> >>I see. From a quick look at the spec file, I do notice  post,
preun,
> >>postun operations where it deals w/ seliux policy modules, and
other
> >>system commands etc. Can you please elaborate a little on what kind
> >>of workarounds are we talking here?
> >
> >You have to imagine that the appliance will boot with just the files
> >specified directly from the RPMs (ie. the %files sections).  Any other
> >adjustments done by %pre or %post scripts will not be done.
> 
> Just a  question, is there a technical/design reason why febootstrap
> doesn't do this? Or like you mentioned  at the end, it is purely for
> more control for libguestfs ?
It's a technical reason.  We don't install the RPMs in the regular way
(ie. using rpm), but instead we unpack them.  We *used* to install the
RPMs in the regular way into a chroot, but it was a gigantic pain in
the rear the whole time, and we were glad to get away from doing that.

In any case, installing the RPMs like that didn't really solve the
problem, it just added a lot more complication for what is in practice
only a minor issue.

(The way forward in some distant future febootstrap 4 is to install
the RPMs in a virtual machine using kickstart or the equivalent
mechanism on other distros ...)
> Meanwhile, my run of 'febootstrap'  looks like still(2 hrs or so
> now) it's hung. Would you suggest me to wait? or do a ctl+c ; cleaup
> all the febootstrap temp. directories; and retry?
Try the simpler example from the blog to make sure it's generally
working.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/