Libguestfs - Mar 2015 - Name resolution not working inside virt-customize

Hi!

I am trying to use virt-customize to add packages to an image. However 
it seems that there is a problem where a resolv.conf is not being 
installed and thus package installation fails.

The command is:

virt-customize -v -x -a trusty-server-cloudimg-amd64-disk1.img 
--run-command "host openstack.org"

I am having a hard time gathering complete output as the command seems 
to corrupt my terminal somehow but I think the relevant lines may be:

[   3.0] Running: host openstack.org
running command:
exec >>'/tmp/builder.log' 2>&1


host openstack.org

libguestfs: trace: sh "exec >>'/tmp/builder.log'
2>&1\n\n\nhost
openstack.org\n"
guestfsd: main_loop: new request, len 0x60
mount --bind /dev /sysroot/dev
mount --bind /dev/pts /sysroot/dev/pts
mount: mount point /sysroot/dev/pts does not exist
mount --bind /proc /sysroot/proc
mount --bind /sys/fs/selinux /sysroot/selinux
mount: mount point /sysroot/selinux does not exist
mount --bind /sys /sysroot/sys
mount --bind /sys/fs/selinux /sysroot/sys/fs/selinux
mount: mount point /sysroot/sys/fs/selinux does not exist
renaming /sysroot/etc/resolv.conf to /sysroot/etc/g1fbx1fs
cp /etc/resolv.conf /sysroot/etc/resolv.conf
cp: cannot stat '/etc/resolv.conf': No such file or directory
/bin/sh -c exec >>'/tmp/builder.log' 2>&1


host openstack.org

And the command fails due to the missing resolv.conf.

I am using version 1.28.6 build from source on ubuntu 14.04. The same 
image file listed above works perfectly when launched as an openstack 
instance.

Thanks,

Lee Revell

On Tue, Mar 10, 2015 at 04:41:22PM -0400, Lee Revell
wrote:
> Hi!
> 
> I am trying to use virt-customize to add packages to an image.
> However it seems that there is a problem where a resolv.conf is not
> being installed and thus package installation fails.
> 
> The command is:
> 
> virt-customize -v -x -a trusty-server-cloudimg-amd64-disk1.img
> --run-command "host openstack.org"
>
> I am having a hard time gathering complete output as the command
> seems to corrupt my terminal somehow but I think the relevant lines
> may be:
> 
> [   3.0] Running: host openstack.org
> running command:
> exec >>'/tmp/builder.log' 2>&1
> 
> 
> host openstack.org
> 
> libguestfs: trace: sh "exec >>'/tmp/builder.log'
2>&1\n\n\nhost
> openstack.org\n"
> guestfsd: main_loop: new request, len 0x60
> mount --bind /dev /sysroot/dev
> mount --bind /dev/pts /sysroot/dev/pts
> mount: mount point /sysroot/dev/pts does not exist
> mount --bind /proc /sysroot/proc
> mount --bind /sys/fs/selinux /sysroot/selinux
> mount: mount point /sysroot/selinux does not exist
> mount --bind /sys /sysroot/sys
> mount --bind /sys/fs/selinux /sysroot/sys/fs/selinux
> mount: mount point /sysroot/sys/fs/selinux does not exist
> renaming /sysroot/etc/resolv.conf to /sysroot/etc/g1fbx1fs
> cp /etc/resolv.conf /sysroot/etc/resolv.conf
> cp: cannot stat '/etc/resolv.conf': No such file or directory
> /bin/sh -c exec >>'/tmp/builder.log' 2>&1
> 
> 
> host openstack.org
> 
> And the command fails due to the missing resolv.conf.
> 
> I am using version 1.28.6 build from source on ubuntu 14.04. The
> same image file listed above works perfectly when launched as an
> openstack instance.
I'm afraid our handling of /etc/resolv.conf inside the chroot is a
large (and continuing) source of problems.  Currently this code is
run:

https://github.com/libguestfs/libguestfs/blob/master/daemon/command.c#L148

There are a couple of thing you could do:

 - Don't rely too heavily on name resolution.

 - Use --firstboot* scripts instead, since they genuinely run in guest
   context, and should therefore work more like you expect.

(and I guess a third one .. suggest a better approach to resolv.conf
handling).

Just for comparison, libguestfs 1.29.29 on Fedora host, plus
virt-builder, with a Fedora guest does this:

----------------------------------------------------------------------
$ virt-builder fedora-21 --install /usr/bin/host --run-command "host
openstack.org"
[   2.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz
[   2.0] Planning how to build this image
[   2.0] Uncompressing
[  13.0] Opening the new disk
[  24.0] Setting a random seed
[  24.0] Installing packages: /usr/bin/host
[  49.0] Running: host openstack.org
[  50.0] Setting passwords
virt-builder: Setting random password of root to K0kkVIZXdtgmKtvw
[  53.0] Finishing off
                   Output file: fedora-21.img
                   Output size: 4.0G
                 Output format: raw
            Total usable space: 5.2G
                    Free space: 4.3G (83%)

$ guestfish --ro -a fedora-21.img -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Fedora release 21 (Twenty One)
/dev/sda3 mounted on /
/dev/sda1 mounted on /boot
><fs> cat /tmp/builder.log 
[lots of yum output deleted]
Installed:
  bind-utils.x86_64 32:9.9.6-8.P1.fc21                                          

Dependency Installed:
  bind-libs.x86_64 32:9.9.6-8.P1.fc21                                           

Dependency Updated:
  bind-libs-lite.x86_64 32:9.9.6-8.P1.fc21                                      
  bind-license.noarch 32:9.9.6-8.P1.fc21                                        

Complete!
openstack.org has address 162.242.140.107
openstack.org mail is handled by 20 mx2.emailsrvr.com.
openstack.org mail is handled by 10 mx1.emailsrvr.com.
----------------------------------------------------------------------

Pino ^^ any ideas?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

On 03/10/2015 05:12 PM, Richard W.M. Jones wrote:
> I'm afraid our handling of /etc/resolv.conf inside the chroot is a
> large (and continuing) source of problems.  Currently this code is
> run:
>
> https://github.com/libguestfs/libguestfs/blob/master/daemon/command.c#L148
>
> There are a couple of thing you could do:
>
>   - Don't rely too heavily on name resolution.
>
>   - Use --firstboot* scripts instead, since they genuinely run in guest
>     context, and should therefore work more like you expect.
Thanks, I'll try these. I actually tried several other options like:

   --upload /home/leer/resolv.conf:/etc/resolv.conf

   --write /etc/resolv.conf:"nameserver 172.16.12.1"

   --run-command "echo 'nameserver 172.16.12.1' >
/etc/resolv.conf"

   --copy-in /home/leer/resolv.conf:/etc/resolv.conf

The first three all failed with some variant on
"'/etc/resolv.conf': No
such file or directory". The fourth apparently has been removed from 
this version or was not compiled in.

Also I believe what I thought was scrollback buffer corruption is 
actually virt-customize making changes to the virtual as I noticed the 
md5sum of the image changed after each failed attempt.

Thanks again,

Lee

On Tue, Mar 10, 2015 at 09:12:12PM +0000, Richard W.M. Jones
wrote:
> On Tue, Mar 10, 2015 at 04:41:22PM -0400, Lee Revell wrote:
> > Hi!
> > 
> > I am trying to use virt-customize to add packages to an image.
> > However it seems that there is a problem where a resolv.conf is not
> > being installed and thus package installation fails.
> > 
> > The command is:
> > 
> > virt-customize -v -x -a trusty-server-cloudimg-amd64-disk1.img
> > --run-command "host openstack.org"
> >
> > I am having a hard time gathering complete output as the command
> > seems to corrupt my terminal somehow but I think the relevant lines
> > may be:
> > 
> > [   3.0] Running: host openstack.org
> > running command:
> > exec >>'/tmp/builder.log' 2>&1
> > 
> > 
> > host openstack.org
> > 
> > libguestfs: trace: sh "exec >>'/tmp/builder.log'
2>&1\n\n\nhost
> > openstack.org\n"
> > guestfsd: main_loop: new request, len 0x60
> > mount --bind /dev /sysroot/dev
> > mount --bind /dev/pts /sysroot/dev/pts
> > mount: mount point /sysroot/dev/pts does not exist
> > mount --bind /proc /sysroot/proc
> > mount --bind /sys/fs/selinux /sysroot/selinux
> > mount: mount point /sysroot/selinux does not exist
> > mount --bind /sys /sysroot/sys
> > mount --bind /sys/fs/selinux /sysroot/sys/fs/selinux
> > mount: mount point /sysroot/sys/fs/selinux does not exist
> > renaming /sysroot/etc/resolv.conf to /sysroot/etc/g1fbx1fs
> > cp /etc/resolv.conf /sysroot/etc/resolv.conf
> > cp: cannot stat '/etc/resolv.conf': No such file or directory
> > /bin/sh -c exec >>'/tmp/builder.log' 2>&1
> > 
> > 
> > host openstack.org
> > 
> > And the command fails due to the missing resolv.conf.
> > 
> > I am using version 1.28.6 build from source on ubuntu 14.04. The
> > same image file listed above works perfectly when launched as an
> > openstack instance.
> 
> I'm afraid our handling of /etc/resolv.conf inside the chroot is a
> large (and continuing) source of problems.  Currently this code is
> run:
> 
> https://github.com/libguestfs/libguestfs/blob/master/daemon/command.c#L148
> 
> There are a couple of thing you could do:
> 
>  - Don't rely too heavily on name resolution.
> 
>  - Use --firstboot* scripts instead, since they genuinely run in guest
>    context, and should therefore work more like you expect.
> 
> (and I guess a third one .. suggest a better approach to resolv.conf
> handling).
We can finally use overlayfs now it is merged

 # cat /etc/resolv.conf
 nameserver 192.168.1.254
 # mkdir /mnt/{overlay,empty,custometc}
 # mount -t overlay overlay -o
lowerdir=/etc,upperdir=/mnt/overlay,workdir=/mnt/empty /mnt/custometc
 # echo "nameserver 192.168.1.1" > /mnt/custometc/resolv.conf
 # cat /mnt/custom/etcresolv.conf
 nameserver 192.168.1.1
 # cat /etc/resolv.conf
 nameserver 192.168.1.254

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

On 03/10/2015 05:12 PM, Richard W.M. Jones wrote:
> Just for comparison, libguestfs 1.29.29 on Fedora host, plus
> virt-builder, with a Fedora guest does this:
Hmm, something weird is going in with the output of virt-customize.

leer@eng-leer:~$ virt-customize -a 
trusty-server-cloudimg-amd64-disk1.img --firstboot update-resolv-conf.sh 
--install mysql-server -v -x > virt-customize.log 2>&1

leer@eng-leer:~$ wc -l virt-customize.log
125 virt-customize.log

leer@eng-leer:~$ head virt-customize.log
Err http://security.ubuntu.com trusty-security InRelease

Err http://archive.ubuntu.com trusty InRelease

Err http://archive.ubuntu.com trusty-updates InRelease

Err http://security.ubuntu.com trusty-security Release.gpg
   Could not resolve 'security.ubuntu.com'
Err http://archive.ubuntu.com trusty Release.gpg
   Could not resolve 'archive.ubuntu.com'

Lee

On 03/10/2015 05:12 PM, Richard W.M. Jones wrote:
> Just for comparison, libguestfs 1.29.29 on Fedora host, plus
> virt-builder, with a Fedora guest does this:
>
libguestfs 1.29.29 on a ubuntu 14.04 host with fedora-21 guest seems 
have no functioning network, it cannot ping the host. Is there some 
configuration required on the host?

leer@eng-leer:~/libguestfs-1.29.29$ virt-builder fedora-21 --run-command 
"ping -c 4 172.16.12.200"
[   1.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz
[   1.0] Planning how to build this image
[   1.0] Uncompressing
[  28.0] Opening the new disk
[  31.0] Setting a random seed
[  31.0] Running: ping -c 4 172.16.12.200
connect: Network is unreachable
virt-builder: error: ping -c 4 172.16.12.200: command exited with an error

Lee