Xen devel - Nov 2013 - [osstest PATCH] target_cmd_build: honour DistccHosts host property

Apropos of our suggestion to recommission army since marilith is
having trouble.

FYI I''ve just pushed this to pretest.  It seems to WFM in an ad-hoc
run of ts-xen-build worked well.  The main build took 844 seconds (the
timeout is 9000) and the whole script took about 20 mins before
crashing.

Unfortunately there is another problem which is that ts-xen-build has
been made to require root access (which osstest doesn''t have on army).

Ian.

From: Ian Jackson <ian.jackson@eu.citrix.com>
Date: Thu, 28 Nov 2013 18:05:56 +0000
Subject: [PATCH] target_cmd_build: honour DistccHosts host property

If this property is set then the environment variable
 (a) DISTCC_HOSTS is set to the value of the host property
 (a) CCACHE_PREFIX is set to "distcc"
which will result in distcc being used as configured.

distcc had better be installed for such hosts.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm |   11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 233d453..4954d84 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -501,10 +501,19 @@ sub target_editfile_root ($$$;$$) {
 
 sub target_cmd_build ($$$$) {
     my ($ho,$timeout,$builddir,$script) = @_;
-    target_cmd($ho, <<END.$script, $timeout);
+
+    my $distcc_hosts =
get_host_property($ho,''DistccHosts'',undef);
+    my $distcc = defined($distcc_hosts) ? <<END : "";
+        CCACHE_PREFIX=distcc
+        DISTCC_HOSTS="$distcc_hosts"
+        export CCACHE_PREFIX DISTCC_HOSTS
+END
+
+    target_cmd($ho, <<END.$distcc.<<END.$script, $timeout);
 	set -xe
         LC_ALL=C; export LC_ALL
         PATH=/usr/lib/ccache:\$PATH:/usr/lib/git-core
+END
         exec </dev/null
         cd $builddir
 END
-- 
1.7.10.4

Ian Jackson writes ("[osstest PATCH] target_cmd_build: honour DistccHosts
host property"):
> Unfortunately there is another problem which is that ts-xen-build has
> been made to require root access (which osstest doesn''t have on
army).
Here''s a fix to this.

I have not pushed this to pretest yet because I want to be sure we get
a pass to fix the freebsd hostflags problem and I''m not 100.000% sure
that this change will pass.

For now it''s in xenbits.xen.org:/home/iwj/ext/osstest.git#wip.distcc

Ian.

From: Ian Jackson <ian.jackson@eu.citrix.com>
Date: Thu, 28 Nov 2013 18:20:04 +0000
Subject: [PATCH] target_file_exists: Use non-root access for this check

The new call of this in built_stash_file makes ts-*-build require root
access, which is not desirable.

The only other caller of target_file_exists is in ts-xen-install where
it is called on
                        /etc/default/xencommons
                        /etc/sysconfig/xencommons
                        /etc/default/xend
                        /etc/sysconfig/xend
all of which should be accessible as a normal user in a default config.

So change the access to use target_cmd_output rather than
target_cmd_output_root.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 4954d84..3bad5ca 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -457,7 +457,7 @@ sub target_putfilecontents_root_stash ($$$$;$) {
 
 sub target_file_exists ($$) {
     my ($ho,$rfile) = @_;
-    my $out= target_cmd_output_root($ho, "if test -e $rfile; then echo y;
fi");
+    my $out= target_cmd_output($ho, "if test -e $rfile; then echo y;
fi");
     return 1 if $out =~ m/^y$/;
     return 0 if $out !~ m/\S/;
     die "$rfile $out ?";
-- 
1.7.10.4

Ian Jackson writes ("Re: [osstest PATCH] target_cmd_build: honour
DistccHosts host property"):
> Here''s a fix to this.
> 
> I have not pushed this to pretest yet because I want to be sure we get
> a pass to fix the freebsd hostflags problem and I''m not 100.000%
sure
> that this change will pass.
> 
> For now it''s in
xenbits.xen.org:/home/iwj/ext/osstest.git#wip.distcc
It passed my ad-hoc smoke test; ts-xen-build on army now passes for me
with these two patches.

Ian.

On Thu, 2013-11-28 at 18:45 +0000, Ian Jackson wrote:
> Ian Jackson writes ("Re: [osstest PATCH] target_cmd_build: honour
DistccHosts host property"):
> > Here''s a fix to this.
> > 
> > I have not pushed this to pretest yet because I want to be sure we get
> > a pass to fix the freebsd hostflags problem and I''m not
100.000% sure
> > that this change will pass.
> > 
> > For now it''s in
xenbits.xen.org:/home/iwj/ext/osstest.git#wip.distcc
> 
> It passed my ad-hoc smoke test; ts-xen-build on army now passes for me
> with these two patches.
ts-kernel-build was the problematic one WRT timing out. I think,
entirely anecdotally, it was worse overnight if it hit my nightly Xen
build cronjob (at 0028). Now that we have the osstest stuff running that
is a bit obsolete, I could kill it I suppose.

Also, won''t you need to increase the -j flag setting ''shared
max tasks''
for army for this stuff to make any difference?

Ian.