Ian Campbell
2013-Oct-11 15:53 UTC
[PATCH OSSTEST 0/6] Support for serial logs from marilith boxes
The marilith boxes use a conserver (http://www.conserver.com/) setup for
serial access. Our installation exports the logs via http allowing us to
grab them with wget.
Sending debug keys with is handled separately via xenuse. xenuse
ultimately speaks to the conserver too but it abstracts away the IP and
port to use so this is preferred.
With these changes the correct Serial hostprop for a marilith box is:
xenuse;http http://conserver.uk.xensource.com/consoles/
Ian Campbell
2013-Oct-11 15:54 UTC
[PATCH OSSTEST 1/6] Standalone: Collect logs modified in the last hour
On failure, when the console may have been silent for a considerable time, 1s is
not long enough ago.
---
Osstest/JobDB/Standalone.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/Osstest/JobDB/Standalone.pm b/Osstest/JobDB/Standalone.pm
index da35376..d3ff1df 100644
--- a/Osstest/JobDB/Standalone.pm
+++ b/Osstest/JobDB/Standalone.pm
@@ -101,7 +101,8 @@ sub jobdb_check_other_job { } #method
sub jobdb_flight_started_for_log_capture ($$) { #method
my ($mo, $flight) = @_;
- return time - 1; # just the most recent serial log then
+ return time - 60*60; # just the most recent serial log then,
+ # modified in the last hour
}
sub jobdb_enable_log_capture ($) { #method
--
1.7.10.4
Ian Campbell
2013-Oct-11 15:54 UTC
[PATCH OSSTEST 2/6] TestSupport: allow multiple host serial methods
---
Osstest/TestSupport.pm | 25 ++++++++++++++++++++++---
ts-logs-capture | 2 +-
2 files changed, 23 insertions(+), 4 deletions(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index e5343c8..75fb7cc 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -63,6 +63,7 @@ BEGIN {
selecthost get_hostflags get_host_property
power_state power_cycle power_cycle_time
+ serial_fetch_logs
propname_massage
get_stashed open_unique_stashfile
@@ -620,6 +621,26 @@ sub guest_check_ip ($) {
$gho->{DhcpWatch}->check_ip($gho);
}
+#-------------- serial -------------
+
+sub serial_host_setup ($) {
+ my ($ho) = @_;
+ my $methobjs = [ ];
+ my $serialmeth =
get_host_property($ho,''serial'',''noop'');
+ foreach my $meth (split /\;\s*/, $serialmeth) {
+ push @$methobjs, get_host_method_object($ho,''Serial'',$meth);
+ }
+ $ho->{SerialMethobjs} = $methobjs;
+}
+
+sub serial_fetch_logs ($) {
+ my ($ho) = @_;
+ logm("serial: collecting logs for $ho->{Name}");
+ foreach my $mo (@{ $ho->{SerialMethobjs} }) {
+ $mo->fetch_logs();
+ }
+}
+
#---------- power cycling ----------
sub power_cycle_host_setup ($) {
@@ -721,9 +742,7 @@ sub selecthost ($) {
dhcp_watch_setup($ho,$ho);
power_cycle_host_setup($ho);
-
- my $serialmeth =
get_host_property($ho,''serial'',''noop'');
- $ho->{SerialMethobj} =
get_host_method_object($ho,''Serial'',$serialmeth);
+ serial_host_setup($ho);
$ho->{IpStatic} = get_host_property($ho,''ip-addr'');
if (!defined $ho->{IpStatic}) {
diff --git a/ts-logs-capture b/ts-logs-capture
index 9a71db0..2dd95ea 100755
--- a/ts-logs-capture
+++ b/ts-logs-capture
@@ -211,6 +211,6 @@ sub fetch_logs_guest ($) {
}
}
-$ho->{SerialMethobj}->fetch_logs();
+serial_fetch_logs($ho);
fetch_logs_host_guests();
logm("logs captured to $stash");
--
1.7.10.4
Ian Campbell
2013-Oct-11 15:54 UTC
[PATCH OSSTEST 3/6] Serial: Refactor debug key sending into separate request_debug function
This puts the list of keys to send into common code.
---
Osstest/Serial/noop.pm | 4 ++++
Osstest/Serial/sympathy.pm | 42 +++++++++++++++++++++++++-----------------
Osstest/TestSupport.pm | 12 ++++++++++++
3 files changed, 41 insertions(+), 17 deletions(-)
diff --git a/Osstest/Serial/noop.pm b/Osstest/Serial/noop.pm
index 1b980c2..d0de9cd 100644
--- a/Osstest/Serial/noop.pm
+++ b/Osstest/Serial/noop.pm
@@ -39,6 +39,10 @@ sub new {
return bless { }, $class;
}
+sub request_debug {
+ return 0;
+}
+
sub fetch_logs {
my ($mo) = @_;
logm("serial access method \`noop'',".
diff --git a/Osstest/Serial/sympathy.pm b/Osstest/Serial/sympathy.pm
index 3128b68..d6bf425 100644
--- a/Osstest/Serial/sympathy.pm
+++ b/Osstest/Serial/sympathy.pm
@@ -57,21 +57,15 @@ sub new {
return bless $mo, $class;
}
-sub fetch_logs {
- my ($mo) = @_;
-
- my $started= $mjobdb->jobdb_flight_started_for_log_capture($flight);
+sub request_debug {
+ my ($mo,$conswitch,$xenkeys,$guestkeys) = @_;
- my $ho = $mo->{Host};
- my $logpat = $mo->{Pattern};
my $targhost= $mo->{Server};
- logm("requesting debug information");
-
my ($sshopts) = sshopts();
my $sympwrite= sub {
my ($what,$str,$pause) = @_;
- logm("sending $what");
+ logm("sympathy sending $what");
if (!eval {
local ($SIG{''PIPE''}) =
''IGNORE'';
my $sock= $mo->{Socket};
@@ -89,18 +83,32 @@ sub fetch_logs {
}
return 1;
};
+
my $debugkeys= sub {
- my ($what, $keys) = @_;
- foreach my $k (split //, $keys) {
- $sympwrite->("$what debug info request, debug key $k",
$k, 2);
- }
+ my ($what, $keys) = @_;
+ foreach my $k (split //, $keys) {
+ $sympwrite->("$what debug info request, debug key $k", $k,
2);
+ }
};
- $sympwrite->(''request for input to
Xen'',"\x18\x18\x18",1);
- $debugkeys->(''Xen'',"0HMQacdegimnrstuvz");
+
+ $sympwrite->(''request for input to Xen'', $conswitch,
1);
+ $debugkeys->(''Xen'', $xenkeys);
sleep(10);
- $debugkeys->(''guest'',"q");
+ $debugkeys->(''guest'', $guestkeys);
sleep(10);
- $sympwrite->("RET to dom0","\x18\x18\x18\r", 5);
+ $sympwrite->("RET to dom0","$conswitch\r", 5);
+
+ return 1;
+}
+
+sub fetch_logs {
+ my ($mo) = @_;
+
+ my $started= $mjobdb->jobdb_flight_started_for_log_capture($flight);
+
+ my $ho = $mo->{Host};
+ my $logpat = $mo->{Pattern};
+ my $targhost= $mo->{Server};
logm("collecting serial logs since $started from $targhost");
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 75fb7cc..f4ec055 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -635,7 +635,19 @@ sub serial_host_setup ($) {
sub serial_fetch_logs ($) {
my ($ho) = @_;
+
+ logm("serial: requesting debug information from $ho->{Name}");
+
+ foreach my $mo (@{ $ho->{SerialMethobjs} }) {
+ $mo->request_debug("\x18\x18\x18",
+ "0HMQacdegimnrstuvz",
+ "q") or next;
+ # use the first method which supports ->request_debug.
+ last;
+ }
+
logm("serial: collecting logs for $ho->{Name}");
+
foreach my $mo (@{ $ho->{SerialMethobjs} }) {
$mo->fetch_logs();
}
--
1.7.10.4
Ian Campbell
2013-Oct-11 15:54 UTC
[PATCH OSSTEST 4/6] Serial: collect serial logs from an http server
This relies on wget and httpd colluding to set the mtimes on the downloaded
files, which works with at least the one server I care about (the apache on
the conserver handling the marilith boxes)
---
Osstest/Serial/http.pm | 114 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 114 insertions(+)
create mode 100644 Osstest/Serial/http.pm
diff --git a/Osstest/Serial/http.pm b/Osstest/Serial/http.pm
new file mode 100644
index 0000000..cb36f1c
--- /dev/null
+++ b/Osstest/Serial/http.pm
@@ -0,0 +1,114 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Fetch logs from a web directory export.
+
+package Osstest::Serial::http;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+
+use File::Temp;
+use File::Copy;
+
+BEGIN {
+ use Exporter ();
+ our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+ $VERSION = 1.00;
+ @ISA = qw(Exporter);
+ @EXPORT = qw();
+ %EXPORT_TAGS = ( );
+
+ @EXPORT_OK = qw();
+}
+
+sub new {
+ my ($class, $ho, $methname, @args) = @_;
+ my $mo = { Host => $ho, Name => $ho->{Name} };
+ die if @args<1;
+ push @args, "$ho->{Name}.txt*" if @args < 2;
+
+ logm("serial method $methname $mo->{Host}{Name}: @args");
+ ($mo->{Server}, $mo->{Pattern}) = @args;
+ return bless $mo, $class;
+}
+
+sub request_debug {
+ return 0;
+}
+
+sub fetch_logs {
+ my ($mo) = @_;
+
+ my $started= $mjobdb->jobdb_flight_started_for_log_capture($flight);
+
+ my $ho = $mo->{Host};
+ my $logpat = $mo->{Pattern};
+ my $targhost= $mo->{Server};
+
+ logm("serial http from $mo->{Name} fetching $mo->{Pattern} from
$mo->{Server}");
+
+ my $dir = File::Temp->newdir();
+ my $tdir = $dir->dirname;
+
+ my $lrf = "$tdir/log-retrieval.log";
+
+ system_checked(qw(wget -nH --cut-dirs=1 -r -l1 --no-parent),
+ ''-o'', "$lrf",
+ ''-A'', $mo->{Pattern},
+ ''-P'', $tdir,
+ $mo->{Server});
+
+ my $sr = "serial-retrieval.log";
+ my $lr = open_unique_stashfile(\$sr);
+ File::Copy::copy($lrf, $lr);
+
+ my %done;
+ foreach my $logfile (glob "$tdir/$mo->{Pattern}") {
+ my $lh= new IO::File $logfile, ''r'';
+ if (!defined $lh) {
+ $!==&ENOENT or warn "$logfile $!";
+ next;
+ }
+ stat $lh or die "$logfile $!";
+ my $inum= (stat _)[1];
+ my $lfage= (stat _)[9];
+ my $df= $logfile;
+ $df =~ s,.*/,,;
+ if ($lfage < $started) {
+ next if $done{$inum};
+ logm("$df modified $lfage, skipping")
+ unless $done{$inum};
+ $done{$inum}= 1;
+ next;
+ }
+ next if defined $done{$inum} and $done{$inum} >= 2;
+ $done{$inum}= 2;
+
+ $df = "serial-$df";
+ logm("stashing $df");
+
+ my $dh= open_unique_stashfile(\$df);
+ File::Copy::copy($logfile, $dh);
+ }
+ return;
+
+}
+
+1;
--
1.7.10.4
Ian Campbell
2013-Oct-11 15:54 UTC
[PATCH OSSTEST 5/6] Serial: new module to send debug keys via xenuse
No log retreival here, that is handled via a separate conserver.
---
Osstest/Serial/xenuse.pm | 100 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 100 insertions(+)
create mode 100644 Osstest/Serial/xenuse.pm
diff --git a/Osstest/Serial/xenuse.pm b/Osstest/Serial/xenuse.pm
new file mode 100644
index 0000000..6b9d955
--- /dev/null
+++ b/Osstest/Serial/xenuse.pm
@@ -0,0 +1,100 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Send debug keys via xenuse.
+
+package Osstest::Serial::xenuse;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+
+use File::Temp;
+use File::Copy;
+
+BEGIN {
+ use Exporter ();
+ our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+ $VERSION = 1.00;
+ @ISA = qw(Exporter);
+ @EXPORT = qw();
+ %EXPORT_TAGS = ( );
+
+ @EXPORT_OK = qw();
+}
+
+sub new {
+ my ($class, $ho, $methname, @args) = @_;
+ my $mo = { Host => $ho, Name => $ho->{Name} };
+
+ logm("serial method $methname $mo->{Host}{Name}: @args");
+ return bless $mo, $class;
+}
+
+sub request_debug {
+ my ($mo,$conswitch,$xenkeys,$guestkeys) = @_;
+ my $xenuse= $c{XenUsePath} || "xenuse";
+
+ my $ho= $mo->{Host};
+
+ my $writer= sub {
+ my ($what,$str,$pause) = @_;
+ logm("xenuse sending $what");
+ if (!eval {
+ print XENUSEWRITE $str or die $!;
+ sleep($pause);
+ 1;
+ }) {
+ warn "failed to send $what: $@\n";
+ return 0;
+ }
+ return 1;
+ };
+
+ my $debugkeys= sub {
+ my ($what, $keys) = @_;
+ foreach my $k (split //, $keys) {
+ $writer->("$what debug info request, debug key $k", $k, 2);
+ }
+ };
+
+ local ($SIG{''PIPE''}) = ''IGNORE'';
+ open XENUSEWRITE, "|$xenuse -t $ho->{Name}" or die $!;
+ autoflush XENUSEWRITE 1;
+
+ $writer->(''force attach'', "\x05cf", 1); # ^E
c f == force attach
+
+ $writer->(''request for input to Xen'', $conswitch, 1);
+ $debugkeys->(''Xen'', $xenkeys);
+ sleep(10);
+ $debugkeys->(''guest'', $guestkeys);
+ sleep(10);
+ $writer->("RET to dom0","$conswitch\r", 5);
+
+ $writer->(''dettach'', "\x05c.", 1); # ^E c .
== disconnect
+
+ close XENUSEWRITE or die "$? $!";
+
+ return 1;
+}
+
+sub fetch_logs {
+ return;
+}
+
+1;
--
1.7.10.4
Ian Campbell
2013-Oct-11 15:54 UTC
[PATCH OSSTEST 6/6] Serial/http: use mkdtemp instead of File::Temp->newdir
The latter is not available in Squeeze''s File::Temp.
---
Osstest/Serial/http.pm | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/Osstest/Serial/http.pm b/Osstest/Serial/http.pm
index cb36f1c..3094baa 100644
--- a/Osstest/Serial/http.pm
+++ b/Osstest/Serial/http.pm
@@ -24,7 +24,8 @@ use warnings;
use Osstest;
use Osstest::TestSupport;
-use File::Temp;
+use File::Path qw / rmtree /;
+use File::Temp qw / mkdtemp /;
use File::Copy;
BEGIN {
@@ -64,8 +65,7 @@ sub fetch_logs {
logm("serial http from $mo->{Name} fetching $mo->{Pattern} from
$mo->{Server}");
- my $dir = File::Temp->newdir();
- my $tdir = $dir->dirname;
+ my $tdir = mkdtemp("osstest.log.XXXXX") or die $!;
my $lrf = "$tdir/log-retrieval.log";
@@ -93,7 +93,7 @@ sub fetch_logs {
$df =~ s,.*/,,;
if ($lfage < $started) {
next if $done{$inum};
- logm("$df modified $lfage, skipping")
+ logm("$df modified $lfage ($started), skipping")
unless $done{$inum};
$done{$inum}= 1;
next;
@@ -107,8 +107,9 @@ sub fetch_logs {
my $dh= open_unique_stashfile(\$df);
File::Copy::copy($logfile, $dh);
}
- return;
+ rmtree($tdir);
+ return;
}
1;
--
1.7.10.4
Ian Jackson
2013-Oct-11 17:22 UTC
Re: [PATCH OSSTEST 0/6] Support for serial logs from marilith boxes
Ian Campbell writes ("[PATCH OSSTEST 0/6] Support for serial logs from
marilith boxes"):> The marilith boxes use a conserver (http://www.conserver.com/) setup for
> serial access. Our installation exports the logs via http allowing us to
> grab them with wget.
>
> Sending debug keys with is handled separately via xenuse. xenuse
> ultimately speaks to the conserver too but it abstracts away the IP and
> port to use so this is preferred.
>
> With these changes the correct Serial hostprop for a marilith box is:
> xenuse;http http://conserver.uk.xensource.com/consoles/
These all look plausible to me (as do your previous to); please go
ahead and push them.
Ian.
Ian Campbell
2013-Oct-13 09:19 UTC
Re: [PATCH OSSTEST 0/6] Support for serial logs from marilith boxes
On Fri, 2013-10-11 at 18:22 +0100, Ian Jackson wrote:> Ian Campbell writes ("[PATCH OSSTEST 0/6] Support for serial logs from marilith boxes"): > > The marilith boxes use a conserver (http://www.conserver.com/) setup for > > serial access. Our installation exports the logs via http allowing us to > > grab them with wget. > > > > Sending debug keys with is handled separately via xenuse. xenuse > > ultimately speaks to the conserver too but it abstracts away the IP and > > port to use so this is preferred. > > > > With these changes the correct Serial hostprop for a marilith box is: > > xenuse;http http://conserver.uk.xensource.com/consoles/ > > These all look plausible to me (as do your previous to); please go > ahead and push them.I assumed s/to/two/ and pushed all eight: fbcda5f Serial/http: use mkdtemp instead of File::Temp->newdir 5caaf09 Serial: new module to send debug keys via xenuse 660efb2 Serial: collect serial logs from an http server 514d05e Serial: Refactor debug key sending into separate request_debug function acecbdc TestSupport: allow multiple host serial methods cc60e34 Standalone: Collect logs modified in the last hour 5ce9690 Debian: set FANCYTTY=0 so serial logs aren''t full of escape chars on Wheezy a2164c3 TestSupport: fix warning about cacheing_git when !no-reinstall Ian.
Ian Campbell
2013-Oct-13 09:51 UTC
Re: [PATCH OSSTEST 0/6] Support for serial logs from marilith boxes
On Sun, 2013-10-13 at 10:19 +0100, Ian Campbell wrote:> On Fri, 2013-10-11 at 18:22 +0100, Ian Jackson wrote: > > Ian Campbell writes ("[PATCH OSSTEST 0/6] Support for serial logs from marilith boxes"): > > > The marilith boxes use a conserver (http://www.conserver.com/) setup for > > > serial access. Our installation exports the logs via http allowing us to > > > grab them with wget. > > > > > > Sending debug keys with is handled separately via xenuse. xenuse > > > ultimately speaks to the conserver too but it abstracts away the IP and > > > port to use so this is preferred. > > > > > > With these changes the correct Serial hostprop for a marilith box is: > > > xenuse;http http://conserver.uk.xensource.com/consoles/ > > > > These all look plausible to me (as do your previous to); please go > > ahead and push them. > > I assumed s/to/two/ and pushed all eight:I also ran: ./mg-hosts setprops marilith-n5 -- serial ''xenuse;http http://conserver.uk.xensource.com/consoles/'' Ian.