Libguestfs - Sep 2015 - [PATCH 1/2] Allow curl to follow redirects.

Hi guys,

Here are two patches to get opensuse images in virt-builder.

Cédric Bosdonnat (2):
  Allow curl to follow redirects.
  builder: add opensuse images sources

 builder/Makefile.am      |  3 ++-
 builder/downloader.ml    |  4 ++--
 builder/opensuse.conf.in |  3 +++
 builder/opensuse.gpg     | 21 +++++++++++++++++++++
 configure.ac             |  1 +
 5 files changed, 29 insertions(+), 3 deletions(-)
 create mode 100644 builder/opensuse.conf.in
 create mode 100644 builder/opensuse.gpg

-- 
2.1.4

download.opensuse.org uses redirects to point to the relevant mirror.
Adding -L --max-redirs 5 to curl options will get us the requested
images.
---
 builder/downloader.ml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/builder/downloader.ml b/builder/downloader.ml
index 30ca212..46581cb 100644
--- a/builder/downloader.ml
+++ b/builder/downloader.ml
@@ -94,7 +94,7 @@ and download_to t ?(progress_bar = false) ~proxy uri filename 
| _ as protocol -> (* Any other protocol. *)
     let outenv = proxy_envvar protocol proxy in
     (* Get the status code first to ensure the file exists. *)
-    let cmd = sprintf "%s%s%s -g -o /dev/null -I -w
'%%{http_code}' %s"
+    let cmd = sprintf "%s%s%s -L --max-redirs 5 -g -o /dev/null -I -w
'%%{http_code}' %s"
       outenv
       t.curl
       (if verbose () then "" else " -s -S")
@@ -114,7 +114,7 @@ and download_to t ?(progress_bar = false) ~proxy uri
filename        error (f_"failed to download %s: HTTP status code %s")
uri status_code;
 
     (* Now download the file. *)
-    let cmd = sprintf "%s%s%s -g -o %s %s"
+    let cmd = sprintf "%s%s%s -L --max-redirs 5 -g -o %s %s"
       outenv
       t.curl
       (if verbose () then "" else if progress_bar then "
-#" else " -s -S")
-- 
2.1.4

---
 builder/Makefile.am      |  3 ++-
 builder/opensuse.conf.in |  3 +++
 builder/opensuse.gpg     | 21 +++++++++++++++++++++
 configure.ac             |  1 +
 4 files changed, 27 insertions(+), 1 deletion(-)
 create mode 100644 builder/opensuse.conf.in
 create mode 100644 builder/opensuse.gpg

diff --git a/builder/Makefile.am b/builder/Makefile.am
index f48efb0..33acb79 100644
--- a/builder/Makefile.am
+++ b/builder/Makefile.am
@@ -269,7 +269,8 @@ DISTCLEANFILES = .depend
 # virt-builder's default repository
 
 repoconfdir = $(sysconfdir)/xdg/virt-builder/repos.d
-repoconf_DATA = libguestfs.conf libguestfs.gpg
+repoconf_DATA = libguestfs.conf libguestfs.gpg \
+				opensuse.conf opensuse.gpg
 
 install-exec-hook:
 	$(LN_S) xdg/virt-builder $(DESTDIR)$(sysconfdir)/virt-builder
diff --git a/builder/opensuse.conf.in b/builder/opensuse.conf.in
new file mode 100644
index 0000000..80db267
--- /dev/null
+++ b/builder/opensuse.conf.in
@@ -0,0 +1,3 @@
+[opensuse.org]
+uri=http://download.opensuse.org/repositories/Virtualization:/virt-builder-images/images/index
+gpgkey=file://@SYSCONFDIR@/xdg/virt-builder/repos.d/opensuse.gpg
diff --git a/builder/opensuse.gpg b/builder/opensuse.gpg
new file mode 100644
index 0000000..001376f
--- /dev/null
+++ b/builder/opensuse.gpg
@@ -0,0 +1,21 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+Version: GnuPG v1.4.5 (GNU/Linux)
+
+mQENBFImAl0BCACkjaXGvVLHBGTVXVP0khtpUVHqFvCRtaIIMHaX/5oTr3nyehDQ
+Ex9VLsSRcNa0QxtnCHFRQzjWWqe+i6pBginnSjucgmjnIKyJsF4l6R+rwAiinHQX
+C4s6Lqg/wH9xDPRBrMYFqlc/7MVf0Glhk1+lAxgQjolMt+5AbbrWlBbwc/i+++zl
+ES3MaeH8aiwup/ogjhmk0SbCQQ/ib21p3XWBwx2oz/KM6Voq9tKDvMczjzNRY3ZT
+6Di3FsUSKI7kgljiNiuN+675YwqEqxWEJgdE5a7Zb67giH1Ik08b5wQiF5jSAICD
+DxW7/ibWBvZJnqhqQT2xJpLC5VaJqwkN8o83ABEBAAG0PlZpcnR1YWxpemF0aW9u
+IE9CUyBQcm9qZWN0IDxWaXJ0dWFsaXphdGlvbkBidWlsZC5vcGVuc3VzZS5vcmc+
+iQE7BBMBAgAmBQJSJgJdAhsDBQkEHrAABgsJCAcDAgQVAggDBBYCAwECHgECF4AA
+CgkQoZP7tXIXT8ITnwf3SVUUoVjVLFCjhIxdet8BL011cJDwr9TwKEQfq4Ybsq5L
+5Y1/Zk86rTzrVOZrODLwNRIC3fMuegZV5f85KMggXu37Di+UvX+dQW9v1hte+hAT
++gsqb60kOnE/Yacgkb6D3xIzRudAB2q/xfvHl/hgfn416yGI8NvntT7n4Hk9wT28
+9JSFkun0uaessg77aXlAdsqHwdugm9hELeva89OoYoiZ4d9r4ScTMSj0UkNgnh7g
+CyIScZHYqiiOeosUtAX9u1PyUFfFsg9s5snfud7aF48EfXU0RTtZAGKtG4GPDv3q
+bYc5TJ2pQzs9y5Bk/jAMR/QQw8CKglBsn1cjYkKViEYEExECAAYFAlImAl0ACgkQ
+OzARt2udZSO5yACgr6Ei7QZ+PAmg4Mr5db+4M3aepAEAniU33RaTKBCGkwQi6kHr
+4VaII2/E
+=l8DH
+-----END PGP PUBLIC KEY BLOCK-----
diff --git a/configure.ac b/configure.ac
index 739cf06..a440f43 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1722,6 +1722,7 @@ AC_CONFIG_FILES([Makefile
                  bash/Makefile
                  builder/Makefile
                  builder/libguestfs.conf
+                 builder/opensuse.conf
                  builder/test-config/virt-builder/repos.d/test-index.conf
                  builder/test-website/virt-builder/repos.d/libguestfs.conf
                  builder/website/Makefile
-- 
2.1.4

On Tue, Sep 08, 2015 at 09:40:23AM +0200, Cédric Bosdonnat
wrote:
> ---
>  builder/Makefile.am      |  3 ++-
>  builder/opensuse.conf.in |  3 +++
>  builder/opensuse.gpg     | 21 +++++++++++++++++++++
>  configure.ac             |  1 +
>  4 files changed, 27 insertions(+), 1 deletion(-)
>  create mode 100644 builder/opensuse.conf.in
>  create mode 100644 builder/opensuse.gpg
> 
> diff --git a/builder/Makefile.am b/builder/Makefile.am
> index f48efb0..33acb79 100644
> --- a/builder/Makefile.am
> +++ b/builder/Makefile.am
> @@ -269,7 +269,8 @@ DISTCLEANFILES = .depend
>  # virt-builder's default repository
>  
>  repoconfdir = $(sysconfdir)/xdg/virt-builder/repos.d
> -repoconf_DATA = libguestfs.conf libguestfs.gpg
> +repoconf_DATA = libguestfs.conf libguestfs.gpg \
> +				opensuse.conf opensuse.gpg
>  
>  install-exec-hook:
>  	$(LN_S) xdg/virt-builder $(DESTDIR)$(sysconfdir)/virt-builder
> diff --git a/builder/opensuse.conf.in b/builder/opensuse.conf.in
> new file mode 100644
> index 0000000..80db267
> --- /dev/null
> +++ b/builder/opensuse.conf.in
> @@ -0,0 +1,3 @@
> +[opensuse.org]
>
+uri=http://download.opensuse.org/repositories/Virtualization:/virt-builder-images/images/index
> +gpgkey=file://@SYSCONFDIR@/xdg/virt-builder/repos.d/opensuse.gpg
> diff --git a/builder/opensuse.gpg b/builder/opensuse.gpg
> new file mode 100644
> index 0000000..001376f
> --- /dev/null
> +++ b/builder/opensuse.gpg
> @@ -0,0 +1,21 @@
> +-----BEGIN PGP PUBLIC KEY BLOCK-----
> +Version: GnuPG v1.4.5 (GNU/Linux)
> +
> +mQENBFImAl0BCACkjaXGvVLHBGTVXVP0khtpUVHqFvCRtaIIMHaX/5oTr3nyehDQ
> +Ex9VLsSRcNa0QxtnCHFRQzjWWqe+i6pBginnSjucgmjnIKyJsF4l6R+rwAiinHQX
> +C4s6Lqg/wH9xDPRBrMYFqlc/7MVf0Glhk1+lAxgQjolMt+5AbbrWlBbwc/i+++zl
> +ES3MaeH8aiwup/ogjhmk0SbCQQ/ib21p3XWBwx2oz/KM6Voq9tKDvMczjzNRY3ZT
> +6Di3FsUSKI7kgljiNiuN+675YwqEqxWEJgdE5a7Zb67giH1Ik08b5wQiF5jSAICD
> +DxW7/ibWBvZJnqhqQT2xJpLC5VaJqwkN8o83ABEBAAG0PlZpcnR1YWxpemF0aW9u
> +IE9CUyBQcm9qZWN0IDxWaXJ0dWFsaXphdGlvbkBidWlsZC5vcGVuc3VzZS5vcmc+
> +iQE7BBMBAgAmBQJSJgJdAhsDBQkEHrAABgsJCAcDAgQVAggDBBYCAwECHgECF4AA
> +CgkQoZP7tXIXT8ITnwf3SVUUoVjVLFCjhIxdet8BL011cJDwr9TwKEQfq4Ybsq5L
> +5Y1/Zk86rTzrVOZrODLwNRIC3fMuegZV5f85KMggXu37Di+UvX+dQW9v1hte+hAT
> ++gsqb60kOnE/Yacgkb6D3xIzRudAB2q/xfvHl/hgfn416yGI8NvntT7n4Hk9wT28
> +9JSFkun0uaessg77aXlAdsqHwdugm9hELeva89OoYoiZ4d9r4ScTMSj0UkNgnh7g
> +CyIScZHYqiiOeosUtAX9u1PyUFfFsg9s5snfud7aF48EfXU0RTtZAGKtG4GPDv3q
> +bYc5TJ2pQzs9y5Bk/jAMR/QQw8CKglBsn1cjYkKViEYEExECAAYFAlImAl0ACgkQ
> +OzARt2udZSO5yACgr6Ei7QZ+PAmg4Mr5db+4M3aepAEAniU33RaTKBCGkwQi6kHr
> +4VaII2/E
> +=l8DH
> +-----END PGP PUBLIC KEY BLOCK-----
> diff --git a/configure.ac b/configure.ac
> index 739cf06..a440f43 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -1722,6 +1722,7 @@ AC_CONFIG_FILES([Makefile
>                   bash/Makefile
>                   builder/Makefile
>                   builder/libguestfs.conf
> +                 builder/opensuse.conf
>                   builder/test-config/virt-builder/repos.d/test-index.conf
>                   builder/test-website/virt-builder/repos.d/libguestfs.conf
>                   builder/website/Makefile
Series looks good to me.  Pino?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Hi Cédric,

On Tuesday 08 September 2015 09:40:21 Cédric Bosdonnat
wrote:
> Here are two patches to get opensuse images in virt-builder.
> 
> Cédric Bosdonnat (2):
>   Allow curl to follow redirects.
>   builder: add opensuse images sources
> 
>  builder/Makefile.am      |  3 ++-
>  builder/downloader.ml    |  4 ++--
>  builder/opensuse.conf.in |  3 +++
>  builder/opensuse.gpg     | 21 +++++++++++++++++++++
>  configure.ac             |  1 +
>  5 files changed, 29 insertions(+), 3 deletions(-)
>  create mode 100644 builder/opensuse.conf.in
>  create mode 100644 builder/opensuse.gpg
Really nice work! Both patches LGTM.

The only problem is that patch #2 does not apply cleanly after my
SimpleStreams work for virt-builder, so can you please rebase it?
I just pushed patch #1, which applies fine.

I gave the images a try, and a couple of notes so far are:
- mllib/passwords.ml:default_crypto does not know about opensuse, so
  assumes md5 for shadow (there's a warning when setting up passwords);
  should be easy to fix
- none of the images then boot, reporting that /sbin/init cannot be
  executed, with error -28.

Thanks,
-- 
Pino Toscano

On Tuesday 08 September 2015 12:16:37 Pino Toscano
wrote:
> Hi Cédric,
> 
> On Tuesday 08 September 2015 09:40:21 Cédric Bosdonnat wrote:
> > Here are two patches to get opensuse images in virt-builder.
> > 
> > Cédric Bosdonnat (2):
> >   Allow curl to follow redirects.
> >   builder: add opensuse images sources
> > 
> >  builder/Makefile.am      |  3 ++-
> >  builder/downloader.ml    |  4 ++--
> >  builder/opensuse.conf.in |  3 +++
> >  builder/opensuse.gpg     | 21 +++++++++++++++++++++
> >  configure.ac             |  1 +
> >  5 files changed, 29 insertions(+), 3 deletions(-)
> >  create mode 100644 builder/opensuse.conf.in
> >  create mode 100644 builder/opensuse.gpg
> 
> Really nice work! Both patches LGTM.
> 
> The only problem is that patch #2 does not apply cleanly after my
> SimpleStreams work for virt-builder, so can you please rebase it?
> I just pushed patch #1, which applies fine.
> 
> I gave the images a try, and a couple of notes so far are:
> - mllib/passwords.ml:default_crypto does not know about opensuse, so
>   assumes md5 for shadow (there's a warning when setting up passwords);
>   should be easy to fix
This has been fixed with
https://github.com/libguestfs/libguestfs/commit/1b4c1d74d36c942417ea946a561a1964b20a1191
> - none of the images then boot, reporting that /sbin/init cannot be
>   executed, with error -28.
Apparently, even though -28 is -ENOSPC on Linux, it seems like the
issue was the too limited RAM for the guest. The default for qemu is
128 MiB, and bumping it to 512 M made it run.

So, looking forward to see the rebased patch!

Thanks,
-- 
Pino Toscano