Libguestfs - Dec 2015 - Which ISO format/layout does virt-builder expect?

I'm running into a bit of trouble using virt-builder and I'm hoping that
someone might be able to point in the right direction.

My build process creates a custom ISO image which is subsequently used to
kickstart an Oracle X4-2 server using PXE.  The ISO is a minimal install and
contains a suite of packages and customization scripts to properly configure our
box during first boot.  The process works great for kickstarting baremetal
machines as well as VMs created with virt-manager or vsphere.

I'd like to be able to use virt-builder to build virtual images
automatically (i.e., take our .ISO and generate a qcow2 of a properly installed
system).  Unfortunately, virt-builder doesn't seem to like the format of my
ISO image.  The tool dies during the "Setting passwords" step because
it's attempting to create a temp file in /etc and my ISO doesn't have an
/etc directory at this point.

Here's a snippet that shows the error message:

[  16.0] Setting passwords
libguestfs: trace: mktemp "/etc/shadow.guestfsXXXXXX"
libguestfs: trace: mktemp = NULL (error)
virt-builder: error: libguestfs error: mktemp: /etc/shadow.guestfsyU5Zwo:
No such file or directory

Is there a particular ISO format that virt-builder expects?

As a test, I used virt-builder to successfully create a centos-6 image.  I
looked in my .cache/virt-builder directory, unxz'd the image and then
mounted the partitions.  It appears that the sanitized centos-6 image is fully
complete (i.e., contains a typical filesystem, has an /etc, an
/etc/passwd...etc.) whereas mine seems more like a linux image that is in a
ready-to-be-installed state.

virt-inspector on my ISO shows:

<?xml version="1.0"?>
<operatingsystems>
  <operatingsystem>
    <root>/dev/sda</root>
    <name>linux</name>
    <arch>x86_64</arch>
    <distro>oraclelinux</distro>
    <major_version>6</major_version>
    <minor_version>0</minor_version>
    <package_format>rpm</package_format>
    <package_management>yum</package_management>
    <format>installer</format>
    <mountpoints>
      <mountpoint dev="/dev/sda">/</mountpoint>
    </mountpoints>
    <filesystems>
      <filesystem dev="/dev/sda">
        <type>iso9660</type>
        <label>iDVA 4.95.0 x86_64</label>
        <uuid>2015-11-20-19-50-52-00</uuid>
      </filesystem>
    </filesystems>
    <applications/>
  </operatingsystem>
</operatingsystems>

Likewise, if I mount my ISO I have a filesystem that looks like:

drwxrwxr-x  3 test test      4096 Nov 19 18:03 images
drwxrwxr-x 11 test test      4096 Nov 19 18:05 iqcustom
-rw-rw-r--  1 test test 966785078 Nov 19 18:08 iqcustom.tar.gz
drwxrwxr-x  2 test test      4096 Nov 19 18:03 isolinux
-rw-rw-r--  1 test test      2368 Nov 19 18:03 ks-dell.cfg
-rw-rw-r--  1 test test     15158 Nov 19 18:03 ks-oracle.cfg
-rw-rw-r--  1 test test     15985 Nov 19 18:03 ks-pxe.cfg
-rw-rw-r--  1 test test      3061 Nov 19 18:03 ks-pxe-oracle-vm.cfg
-rw-rw-r--  1 test test      2176 Nov 19 18:03 ks-vm.cfg
drwxrwxr-x  2 test test     36864 Nov 19 18:04 Packages
drwxrwxr-x  2 test test      4096 Nov 19 18:04 repodata
-r--r--r--  1 test test      3106 Nov 19 18:05 TRANS.TBL
drwxrwxr-x  3 test test      4096 Nov 19 18:05

In contrast, mounting one of the partitions in the centos-6 image shows:

dr-xr-xr-x.  2 root root  4096 Oct 31  2014 bin
drwxr-xr-x.  2 root root  4096 Oct 31  2014 boot
drwxr-xr-x.  2 root root  4096 Oct 31  2014 dev
drwxr-xr-x. 73 root root  4096 Dec  9 16:26 etc
drwxr-xr-x.  2 root root  4096 Sep 23  2011 home
dr-xr-xr-x. 10 root root  4096 Oct 31  2014 lib
dr-xr-xr-x.  9 root root 12288 Oct 31  2014 lib64
drwx------.  2 root root 16384 Oct 31  2014 lost+found
drwxr-xr-x.  2 root root  4096 Sep 23  2011 media
drwxr-xr-x.  2 root root  4096 Sep 23  2011 mnt
drwxr-xr-x.  3 root root  4096 Oct 31  2014 opt
drwxr-xr-x.  2 root root  4096 Oct 31  2014 proc
dr-xr-x---.  2 root root  4096 Oct 31  2014 root
dr-xr-xr-x.  2 root root 12288 Oct 31  2014 sbin
drwxr-xr-x.  2 root root  4096 Oct 31  2014 selinux
drwxr-xr-x.  2 root root  4096 Sep 23  2011 srv
drwxr-xr-x.  2 root root  4096 Oct 31  2014 sys
drwxrwxrwt.  2 root root  4096 Oct 31  2014 tmp
drwxr-xr-x. 13 root root  4096 Oct 31  2014 usr
drwxr-xr-x. 19 root root  4096 Oct 31  2014 var


Note: we build our ISO using mkisofs.  Is there some other tool/process that we
should use to generate ISOs that satisfy virt-builder?  Should I be starting
from a Live CD format?

Cheers!


________________________________

NOTICE: This communication and any attachments may contain confidential and/or
privileged information of IneoQuest Technologies, Inc. ("IneoQuest").
This communication is intended solely for the use of the individual or entity to
which it is addressed. If you are not the intended recipient of this
communication, you are hereby notified that any dissemination, distribution,
copying or other use of this communication or the information it contains is
strictly prohibited and may be unlawful. If you have received this communication
in error, please notify the sender immediately by reply e-mail and destroy all
copies of this communication (including attachments), electronic or otherwise.
If you are a party to a written Non-Disclosure Agreement ("NDA") with
IneoQuest, you should maintain the contents of this communication (including
attachments) in confidence to the full extent specified in the NDA.

On Thu, Dec 10, 2015 at 07:53:55PM +0000, Scott Gorton
wrote:
> My build process creates a custom ISO image which is subsequently
> used to kickstart an Oracle X4-2 server using PXE.  The ISO is a
> minimal install and contains a suite of packages and customization
> scripts to properly configure our box during first boot.  The
> process works great for kickstarting baremetal machines as well as
> VMs created with virt-manager or vsphere.
>
> I'd like to be able to use virt-builder to build virtual images
> automatically (i.e., take our .ISO and generate a qcow2 of a
> properly installed system).  Unfortunately, virt-builder doesn't
> seem to like the format of my ISO image.  The tool dies during the
> "Setting passwords" step because it's attempting to create a
temp
> file in /etc and my ISO doesn't have an /etc directory at this
> point.
ISOs are not modifiable by the Linux kernel :-(  So none of the tools
like guestfish, virt-customize etc etc can work on them.  It doesn't
matter what options you use, the problem is in the Linux ISO driver
(and possibly the ISO format -- I've no idea if it's plausible to
write a driver than can modify ISO).

For virt-p2v, we also need an ISO and a PXE image, and I would dearly
like to use virt-builder to create it.  I tried looking into ways to
build one using virt-builder, but it's a much harder problem than it
appears.

In summary, if you find a way, please let us know :-(

The virt-p2v scripts we have so far are:

https://github.com/libguestfs/libguestfs/blob/master/p2v/virt-p2v-make-disk.in
(documentation:
http://libguestfs.org/virt-p2v-make-disk.1.html)

https://github.com/libguestfs/libguestfs/blob/master/p2v/virt-p2v-make-kickstart.in
(documentation:
http://libguestfs.org/virt-p2v-make-kickstart.1.html)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Please let's keep the replies on the list.]

On Fri, Dec 11, 2015 at 11:58:46AM +0000, Scott Gorton
wrote:
> >ISOs are not modifiable by the Linux kernel :-(  So none of the tools
like guestfish,
> >virt-customize etc etc can work on them.
> 
> I wasn't suggesting that virt-builder or any virt-* command attempt
> to manipulate the ISO.  Instead, I was wondering if I should create
> the ISO with a different filesystem layout.  Mine appears to be an
>  installer image that lays out the filesystem upon first install.
> Yours seems to be a fully baked (live-cd) format.
>
> Maybe the way to go would be to configure a VM using virsh and point
> it at my ISO.  I'll then run through the boot/installation process
> (to install packages and run our customization scripts), shut the
> system down to produce a qcow2.  I can then use your other tools to
> convert the qcow2 into other formats.
In that case I don't understand the problem, but I'll just comment
that virt-customize cannot customize an install CD.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Perhaps an example will clarify.  Let's say that I want to create a
virt-builder repository image for CentOS-6.5-x86_64-bin-DVA1.iso image that I
downloaded off of a CentOS mirror.

1) First, let's mount the ISO to see what's inside

# mkdir /mnt/centos-6.5
# mount -o loop CentOS-6.5-x86_64-bin-DVD1.iso /mnt/centos-6.5
# cd /mnt/centos-6.5
# ls -l

-r--r--r-- 2 root root     14 Nov 29  2013 CentOS_BuildTag
dr-xr-xr-x 3 root root   2048 Nov 29  2013 EFI
-r--r--r-- 2 root root    212 Nov 27  2013 EULA
-r--r--r-- 2 root root  18009 Nov 27  2013 GPL
dr-xr-xr-x 3 root root   2048 Nov 29  2013 images
dr-xr-xr-x 2 root root   2048 Nov 29  2013 isolinux
dr-xr-xr-x 2 root root 655360 Nov 29  2013 Packages
-r--r--r-- 2 root root   1354 Nov 27  2013 RELEASE-NOTES-en-US.html
dr-xr-xr-x 2 root root   4096 Nov 29  2013 repodata
-r--r--r-- 2 root root   1706 Nov 27  2013 RPM-GPG-KEY-CentOS-6
-r--r--r-- 2 root root   1730 Nov 27  2013 RPM-GPG-KEY-CentOS-Debug-6
-r--r--r-- 2 root root   1730 Nov 27  2013 RPM-GPG-KEY-CentOS-Security-6
-r--r--r-- 2 root root   1734 Nov 27  2013 RPM-GPG-KEY-CentOS-Testing-6
-r--r--r-- 1 root root   3380 Nov 29  2013 TRANS.TBL

(Note: it's an install image like the one I've been trying to use)

2) Now following your instructions
(https://rwmj.wordpress.com/2014/09/11/creating-a-local-virt-builder-repository/)
I'll move the iso to my ~/builder directory,  compress it and add an entry
in my index file.

# // unmount the iso and move to builder directory
# umount /mnt/centos-6.5
# mv CentOS-6.5-x86_64-bin-DVD1.iso builder
# cd builder

# // get the original size of the iso
# ls -l CentOS-6.5-x86_64-bin-DVD1.iso
-rwxr--r-- 1 root root 4467982336 Dec 11 11:04 CentOS-6.5-x86_64-bin-DVD1.iso

# // compress it using prescribed settings
# xz --best --block-size=16777216 CentOS-6.5-x86_64-bin-DVD1.iso

# // get the size of the compressed image
# ls -l
-rwxr--r-- 1 root root 4287626520 Dec 11 11:04 CentOS-6.5-x86_64-bin-DVD1.iso.xz

# // create a sha512 checksum
# sha512sum CentOS-6.5-x86_64-bin-DVD1.iso.xz
ea57956dc36afc497e2508d8e5bda983b87c11007301d1d11936281ed3eb5fdff46fe125df8941df56a5f22866218d18fd426017a7adf032d90defc8e20b2ace
CentOS-6.5-x86_64-bin-DVD1.iso.xz

# // update the index file (steps omitted)
# // show the contents of the index file
# cat index
[centos-6.5]
name=centos-6.5
arch=x86_64
file=CentOS-6.5-x86_64-bin-DVD1.iso.xz
checksum[sha512]=ea57956dc36afc497e2508d8e5bda983b87c11007301d1d11936281ed3eb5fdff46fe125df8941df56a5f22866218d18fd426017a7adf032d90defc8e20b2ace
format=raw
size=4467982336
compressed_size=4287626520
expand=/dev/sda3
notes=CentOS-6.5

# // see if virt-builder lists the new image
# virt-builder --list|grep centos
gpg: Signature made Sat 05 Dec 2015 06:41:14 PM EST using RSA key ID E1B768A0
gpg: Good signature from "Richard W.M. Jones
<rjones@redhat.com>"
gpg:                 aka "Richard W.M. Jones <rich@annexia.org>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: F777 4FB1 AD07 4A7E 8C87  67EA 9173 8F73 E1B7 68A0
centos-6.5               x86_64     centos-6.5
centos-6                 x86_64     CentOS 6.6
centos-7.0               x86_64     CentOS 7.0
centos-7.1               x86_64     CentOS 7.1

3) Notice that centos-6.5 appears in list.  Let's try and create a 6.5 image

# virt-builder --password-crypto sha512 centos-6.5
gpg: Signature made Sat 05 Dec 2015 06:41:14 PM EST using RSA key ID E1B768A0
gpg: Good signature from "Richard W.M. Jones
<rjones@redhat.com>"
gpg:                 aka "Richard W.M. Jones <rich@annexia.org>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: F777 4FB1 AD07 4A7E 8C87  67EA 9173 8F73 E1B7 68A0
[   6.0] Downloading: file:///root/builder/CentOS-6.5-x86_64-bin-DVD1.iso.xz
[  19.0] Planning how to build this image
[  19.0] Uncompressing
[  23.0] Opening the new disk
[  25.0] Setting a random seed
virt-builder: warning: random seed could not be set for this type of guest
[  25.0] Setting passwords
virt-builder: error: libguestfs error: mktemp: /etc/shadow.guestfstcTloo:
No such file or directory

If reporting bugs, run virt-builder with debugging enabled and include the
complete output:

  virt-builder -v -x [...]

Notice the " error: libguestfs error: mktemp:
/etc/shadow.guestfstcTloo" error.

The issue is that virt-builder expects the ISO image to contain an /etc
directory.  This image does not.  It's a stock CentOS-6.5 installer image
downloaded from CentOS.

So the question is what do I have to do to create an ISO image that virt-builder
can use?  The images that are present in the virt-builder libguestfs.conf file
(fetching from http://libguestfs.org/download/builder/index.asc), how are THOSE
images built?

________________________________

NOTICE: This communication and any attachments may contain confidential and/or
privileged information of IneoQuest Technologies, Inc. ("IneoQuest").
This communication is intended solely for the use of the individual or entity to
which it is addressed. If you are not the intended recipient of this
communication, you are hereby notified that any dissemination, distribution,
copying or other use of this communication or the information it contains is
strictly prohibited and may be unlawful. If you have received this communication
in error, please notify the sender immediately by reply e-mail and destroy all
copies of this communication (including attachments), electronic or otherwise.
If you are a party to a written Non-Disclosure Agreement ("NDA") with
IneoQuest, you should maintain the contents of this communication (including
attachments) in confidence to the full extent specified in the NDA.