Xen devel - Sep 2013 - [PATCH OSSTEST RFC 0/5] Improve README and enable Wheezy

Hi Ian and Ian

With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to run
build-* jobs and test-amd64-amd64-{pv,xl}.

Wei Liu (5):
  README: add curl and netcat to requirement list
  README: describe job dependence
  Debian: install libc6-dev-i386 for Wheezy amd64
  ts-debian-fixup: remove dead code
  ts-debian-fixup: workaround Debian bug #691320

 README            |    8 ++++++++
 ts-debian-fixup   |   32 ++++++++++++++------------------
 ts-xen-build-prep |    4 ++++
 3 files changed, 26 insertions(+), 18 deletions(-)

-- 
1.7.10.4

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 README |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/README b/README
index e9aed52..c845d4d 100644
--- a/README
+++ b/README
@@ -5,6 +5,8 @@ To run osstest in standalone mode:
      tcl8.4 tclx8.4 libsqlite3-tcl
      libdbi-perl libdbd-sqlite3-perl
      pax rsync
+     curl
+     netcat
 
  - Write a config file
     ~/.xen-osstest/config
-- 
1.7.10.4

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 README |    6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/README b/README
index c845d4d..7f90696 100644
--- a/README
+++ b/README
@@ -33,6 +33,12 @@ To run osstest in standalone mode:
                              (strictly, the final `_HOST'' is the
 			      uppercase of the host''s `ident'' in the job)
 
+   However, as test-amd64-{i386,amd64}-xl and other tests depends on
+   some runtime variables generated by build-* jobs, you can run
+   build-* jobs before running actual test jobs. If you don''t want to
do
+   so you need to insert those missing runvars into standalone.db with
+   sqlite3.
+
  - Don''t forget to set the machine to pxeboot in the BIOS.
 
  - Currently you need a serial console at 115200 8n1; see TODO.
-- 
1.7.10.4

Otherwise build-amd64 fails with missing header predefs.h.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index b1e6a6c..66ba000 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -175,6 +175,10 @@ sub prep () {
     target_cmd_root($ho, "chmod -R a+r
/usr/share/git-core/templates");
     # workaround for Debian #595728
 
+    if ($r{arch} eq ''amd64'' && $ho->{Suite} =~
m/wheezy/) {
+	target_install_packages($ho, ''libc6-dev-i386'');
+    }
+
     if ($ho->{Suite} !~ m/lenny/) {
        
target_putfilecontents_stash($ho,60,<<''END'',''/home/osstest/.hgrc'');
 [extensions]
-- 
1.7.10.4

Per c7910040 ("Osstest.pm, toolstack()->; always use "cfg",
not separate
xl configs"), we only pick the config file with .cfg suffix.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-fixup |   18 ------------------
 1 file changed, 18 deletions(-)

diff --git a/ts-debian-fixup b/ts-debian-fixup
index d8b51b7..f001418 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -163,23 +163,6 @@ sub writecfg () {
     target_putfile_root($ho,10, $cfgstash, $cfgfile);
 }
 
-sub xlcfg () {
-    my $cfg_xend= target_var($gho,''cfgpath'');
-    my $cfg_xl= "/etc/xen/$gho->{Name}.xl";
-    store_runvar("$gho->{Guest}_xlpath", $cfg_xl);
-    target_editfile_root($ho, $cfg_xend, undef, $cfg_xl, sub {
-        local ($/)= undef;
-        $_= <EI>;
-        s/^ [ \t]* memory [ \t]* \= [ \t]* [''"] (.+)
[''"] [ \t]*
-          $/memory=$1/mx;
-        s/([[,])\s*\n/$1/g;
-        s/,\s*\]/\]/g;
-        s/([^#\n].*)\n/$1;\n/mg;
-        s/\n\n+/\n/g;
-        print EO or die $!;
-    });
-}
-
 savecfg();
 ether();
 target_kernkind_check($gho);
@@ -188,5 +171,4 @@ console();
 filesystems();
 otherfixupcfg();
 writecfg();
-xlcfg();
 unmount();
-- 
1.7.10.4

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-fixup |   14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/ts-debian-fixup b/ts-debian-fixup
index f001418..f62bb37 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -118,6 +118,20 @@ sub otherfixupcfg () {
     $cfg =~ s/^vcpus.*//mg;
     $cfg .= "\nvcpus = $vcpus\n";
 
+    # Debian bug #691320
+    # Wheezy''s xen-create-image writes incorrect memory= option in
+    # config file. This is fixed in newer version of xen-tools but it
+    # is not likely to go in in short term.
+    # On the other hand we should consider making xl support
''512M'',
+    # ''2G'' etc. But xend will still need this workaround
until Wheezy
+    # is fixed.
+    # The default value 512 comes from ts-debian-install.
+    if ($ho->{Suite} =~ m/wheezy/) {
+	my $memory= guest_var($gho,''memory'',512);
+	$cfg =~ s/^memory.*//mg;
+	$cfg .= "\nmemory = $memory\n";
+    }
+
     # PCI passthrough
     # Look for runvars  
<gn>_pcipassthrough_<devtype>=<hostident>
     # and pass through all matching devices from the specified host.
-- 
1.7.10.4

On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  ts-debian-fixup |   14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/ts-debian-fixup b/ts-debian-fixup
> index f001418..f62bb37 100755
> --- a/ts-debian-fixup
> +++ b/ts-debian-fixup
> @@ -118,6 +118,20 @@ sub otherfixupcfg () {
>      $cfg =~ s/^vcpus.*//mg;
>      $cfg .= "\nvcpus = $vcpus\n";
>  
> +    # Debian bug #691320
> +    # Wheezy''s xen-create-image writes incorrect memory= option
in
> +    # config file. This is fixed in newer version of xen-tools but it
> +    # is not likely to go in in short term.
> +    # On the other hand we should consider making xl support
''512M'',
> +    # ''2G'' etc. But xend will still need this workaround
until Wheezy
> +    # is fixed.
> +    # The default value 512 comes from ts-debian-install.
Wouldn''t it be better to fix the invocation of xne-create-image to use
a
Mb suffix instead of M, AFAICT that would work on both Squeeze and
Wheezy.

I think the fact that Squeeze''s accepted just M was unintentional,
although it''s good that they are willing to fix it to be acceptable
again...
> +    if ($ho->{Suite} =~ m/wheezy/) {
> +	my $memory= guest_var($gho,''memory'',512);
> +	$cfg =~ s/^memory.*//mg;
> +	$cfg .= "\nmemory = $memory\n";
> +    }
> +
>      # PCI passthrough
>      # Look for runvars  
<gn>_pcipassthrough_<devtype>=<hostident>
>      # and pass through all matching devices from the specified host.

On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> Otherwise build-amd64 fails with missing header predefs.h.
This package exists in Squeeze too, it might simplify things to make it
unconditional?

I wonder why it apparently isn''t need for a Squeeze build though. What
32-bit stuff is getting built? 
> 
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  ts-xen-build-prep |    4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/ts-xen-build-prep b/ts-xen-build-prep
> index b1e6a6c..66ba000 100755
> --- a/ts-xen-build-prep
> +++ b/ts-xen-build-prep
> @@ -175,6 +175,10 @@ sub prep () {
>      target_cmd_root($ho, "chmod -R a+r
/usr/share/git-core/templates");
>      # workaround for Debian #595728
>  
> +    if ($r{arch} eq ''amd64'' && $ho->{Suite}
=~ m/wheezy/) {
> +	target_install_packages($ho, ''libc6-dev-i386'');
> +    }
> +
>      if ($ho->{Suite} !~ m/lenny/) {
>         
target_putfilecontents_stash($ho,60,<<''END'',''/home/osstest/.hgrc'');
>  [extensions]

On Thu, Sep 19, 2013 at 06:16:15PM +0100, Ian Campbell
wrote:
> On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > Otherwise build-amd64 fails with missing header predefs.h.
> 
> This package exists in Squeeze too, it might simplify things to make it
> unconditional?
> 
No, it doesn''t exist in Lenny -- I see there''s branches
conditional on
Lenny, do we still use that?
> I wonder why it apparently isn''t need for a Squeeze build though.
What
> 32-bit stuff is getting built? 
> 
I guess that''s because Wheezy introduces multiarch? The thing being
built is QEMU.

Wei.
> > 
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> >  ts-xen-build-prep |    4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/ts-xen-build-prep b/ts-xen-build-prep
> > index b1e6a6c..66ba000 100755
> > --- a/ts-xen-build-prep
> > +++ b/ts-xen-build-prep
> > @@ -175,6 +175,10 @@ sub prep () {
> >      target_cmd_root($ho, "chmod -R a+r
/usr/share/git-core/templates");
> >      # workaround for Debian #595728
> >  
> > +    if ($r{arch} eq ''amd64'' &&
$ho->{Suite} =~ m/wheezy/) {
> > +	target_install_packages($ho, ''libc6-dev-i386'');
> > +    }
> > +
> >      if ($ho->{Suite} !~ m/lenny/) {
> >         
target_putfilecontents_stash($ho,60,<<''END'',''/home/osstest/.hgrc'');
> >  [extensions]
>

On Thu, 2013-09-19 at 19:31 +0100, Wei Liu wrote:
> On Thu, Sep 19, 2013 at 06:16:15PM +0100, Ian Campbell wrote:
> > On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > > Otherwise build-amd64 fails with missing header predefs.h.
> > 
> > This package exists in Squeeze too, it might simplify things to make
it
> > unconditional?
> > 
> 
> No, it doesn''t exist in Lenny -- I see there''s branches
conditional on
> Lenny, do we still use that?
I think it is all Squeeze these days.
> > I wonder why it apparently isn''t need for a Squeeze build
though. What
> > 32-bit stuff is getting built? 
> > 
> 
> I guess that''s because Wheezy introduces multiarch? The thing
being
> built is QEMU.
We''re building 64-bit qemu and failing because of a missing 32-bit
header? That sounds like a bug somewhere, perhaps in the compiler/libc
packaging or maybe in some part of our build system. How does the actual
error look?

Ian.

On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to
run
> build-* jobs and test-amd64-amd64-{pv,xl}.
So I guess you''ve just been through the process of setting up osstest
in
standalone mode? I wonder if you could put that into a short blog or
wiki doc as an illustration for people?

Ian.

On Thu, Sep 19, 2013 at 08:03:17PM +0100, Ian Campbell
wrote:
> On Thu, 2013-09-19 at 19:31 +0100, Wei Liu wrote:
> > On Thu, Sep 19, 2013 at 06:16:15PM +0100, Ian Campbell wrote:
> > > On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > > > Otherwise build-amd64 fails with missing header predefs.h.
> > > 
> > > This package exists in Squeeze too, it might simplify things to
make it
> > > unconditional?
> > > 
> > 
> > No, it doesn''t exist in Lenny -- I see there''s
branches conditional on
> > Lenny, do we still use that?
> 
> I think it is all Squeeze these days.
> 
OK.
> > > I wonder why it apparently isn''t need for a Squeeze
build though. What
> > > 32-bit stuff is getting built? 
> > > 
> > 
> > I guess that''s because Wheezy introduces multiarch? The thing
being
> > built is QEMU.
> 
> We''re building 64-bit qemu and failing because of a missing 32-bit
> header? That sounds like a bug somewhere, perhaps in the compiler/libc
> packaging or maybe in some part of our build system. How does the actual
> error look?
> 
Oh sorrrry, it''s not QEMU, -j32 and interleaving output fooled me. :-(

XEN_TARGET_ARCH=x86_32 make -f blowfish.mk all
make[5]: Entering directory
`/local/scratch/xen.git/tools/tests/x86_emulator''
gcc  -O1 -fno-omit-frame-pointer -m32 -march=i686 -g -fno-strict-aliasing
-std=gnu99 -Wall -Wstrict-prototypes -Wdeclaration-after-statement
-Wno-unused-but-set-variable -Wno-unused-local-typedefs   -D__XEN_TOOLS__ -MMD
-MF .blowfish.bin.d -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
-D_LARGEFILE64_SOURCE -fno-optimize-sibling-calls -mno-tls-direct-seg-refs 
-fno-stack-protector -fno-exceptions -fno-builtin -msoft-float -c blowfish.c
In file included from /usr/include/stdint.h:26:0,
                 from /usr/lib/gcc/x86_64-linux-gnu/4.7/include/stdint.h:3,
                 from blowfish.c:19:
/usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file or
directory
compilation terminated.

Wei.
> Ian.

On Thu, Sep 19, 2013 at 08:05:38PM +0100, Ian Campbell
wrote:
> On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy
to run
> > build-* jobs and test-amd64-amd64-{pv,xl}.
> 
> So I guess you''ve just been through the process of setting up
osstest in
> standalone mode? I wonder if you could put that into a short blog or
> wiki doc as an illustration for people?
> 
I''m actually writing a blog post on this topic, but still far from
complete.

Wei.
> Ian.

On Thu, 2013-09-19 at 20:12 +0100, Wei Liu wrote:
> On Thu, Sep 19, 2013 at 08:03:17PM +0100, Ian Campbell wrote:
> > On Thu, 2013-09-19 at 19:31 +0100, Wei Liu wrote:
> > > On Thu, Sep 19, 2013 at 06:16:15PM +0100, Ian Campbell wrote:
> > > > On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > > > > Otherwise build-amd64 fails with missing header
predefs.h.
> > > > 
> > > > This package exists in Squeeze too, it might simplify things
to make it
> > > > unconditional?
> > > > 
> > > 
> > > No, it doesn''t exist in Lenny -- I see there''s
branches conditional on
> > > Lenny, do we still use that?
> > 
> > I think it is all Squeeze these days.
> > 
> 
> OK.
> 
> > > > I wonder why it apparently isn''t need for a Squeeze
build though. What
> > > > 32-bit stuff is getting built? 
> > > > 
> > > 
> > > I guess that''s because Wheezy introduces multiarch? The
thing being
> > > built is QEMU.
> > 
> > We''re building 64-bit qemu and failing because of a missing
32-bit
> > header? That sounds like a bug somewhere, perhaps in the compiler/libc
> > packaging or maybe in some part of our build system. How does the
actual
> > error look?
> > 
> 
> Oh sorrrry, it''s not QEMU, -j32 and interleaving output fooled me.
:-(
> 
> XEN_TARGET_ARCH=x86_32 make -f blowfish.mk all
> make[5]: Entering directory
`/local/scratch/xen.git/tools/tests/x86_emulator''
> gcc  -O1 -fno-omit-frame-pointer -m32 -march=i686 -g -fno-strict-aliasing
-std=gnu99 -Wall -Wstrict-prototypes -Wdeclaration-after-statement
-Wno-unused-but-set-variable -Wno-unused-local-typedefs   -D__XEN_TOOLS__ -MMD
-MF .blowfish.bin.d -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
-D_LARGEFILE64_SOURCE -fno-optimize-sibling-calls -mno-tls-direct-seg-refs 
-fno-stack-protector -fno-exceptions -fno-builtin -msoft-float -c blowfish.c
> In file included from /usr/include/stdint.h:26:0,
>                  from /usr/lib/gcc/x86_64-linux-gnu/4.7/include/stdint.h:3,
>                  from blowfish.c:19:
> /usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file
or directory
> compilation terminated.
Ah, OK, this is test code for the 32-bit instruction decoder/emulator.
In principal this probably shouldn''t be built if the compiler is not
available, but in the meantime your fix looks sensible.

Ian

On Thu, 2013-09-19 at 20:13 +0100, Wei Liu wrote:
> On Thu, Sep 19, 2013 at 08:05:38PM +0100, Ian Campbell wrote:
> > On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > > With the changes to Debian.pm and ts-debian-fixup I can now use
Wheezy to run
> > > build-* jobs and test-amd64-amd64-{pv,xl}.
> > 
> > So I guess you''ve just been through the process of setting up
osstest in
> > standalone mode? I wonder if you could put that into a short blog or
> > wiki doc as an illustration for people?
> > 
> 
> I''m actually writing a blog post on this topic, but still far from
> complete.
Perfect, thanks! ;-)

On Thu, Sep 19, 2013 at 06:12:26PM +0100, Ian Campbell
wrote:
> On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> >  ts-debian-fixup |   14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> > 
> > diff --git a/ts-debian-fixup b/ts-debian-fixup
> > index f001418..f62bb37 100755
> > --- a/ts-debian-fixup
> > +++ b/ts-debian-fixup
> > @@ -118,6 +118,20 @@ sub otherfixupcfg () {
> >      $cfg =~ s/^vcpus.*//mg;
> >      $cfg .= "\nvcpus = $vcpus\n";
> >  
> > +    # Debian bug #691320
> > +    # Wheezy''s xen-create-image writes incorrect memory=
option in
> > +    # config file. This is fixed in newer version of xen-tools but it
> > +    # is not likely to go in in short term.
> > +    # On the other hand we should consider making xl support
''512M'',
> > +    # ''2G'' etc. But xend will still need this
workaround until Wheezy
> > +    # is fixed.
> > +    # The default value 512 comes from ts-debian-install.
> 
> Wouldn''t it be better to fix the invocation of xne-create-image to
use a
> Mb suffix instead of M, AFAICT that would work on both Squeeze and
> Wheezy.
> 
OK. I tested passing ''Mb'', it did work in Wheezy.

I will drop this one.

Wei.
> I think the fact that Squeeze''s accepted just M was unintentional,
> although it''s good that they are willing to fix it to be
acceptable
> again...
> 
> > +    if ($ho->{Suite} =~ m/wheezy/) {
> > +	my $memory= guest_var($gho,''memory'',512);
> > +	$cfg =~ s/^memory.*//mg;
> > +	$cfg .= "\nmemory = $memory\n";
> > +    }
> > +
> >      # PCI passthrough
> >      # Look for runvars  
<gn>_pcipassthrough_<devtype>=<hostident>
> >      # and pass through all matching devices from the specified host.
>