Pkg xen devel - Jun 2014 - how to get debugging output from dom0 kernel

Ian Campbell <ijc at hellion.org.uk> writes:
> On Sun, 2014-06-15 at 11:03 +0200, lee wrote:
>> Hi,
>> 
>> how could I get better debugging output from the dom0 kernel?
>
> Either configure a serial console [0] or add "noreboot" to your
> *hypervisor* command line, so you can see (and perhaps
> transcribe/photograph) the crash before manually rebooting.
The server isn't rebooting automatically, it only gets stuck.  I still
have a monitor connected to it, so I can see what's printed to the
console.  But the output is scrolled off screen by subsequent messages
so that it's gone by the time I notice that it hangs.

If there was a way to disable the subsequent messages, I could take a
picture.  Perhaps I can get output from a serial console if I manage to
set that up ... or maybe I can actually make it crash instead of having
to wait ...

>
> [0] http://wiki.xen.org/wiki/Xen_Serial_Console
>
> Ian.
>
>
>
>
-- 
Knowledge is volatile and fluid.  Software is power.

On Sun, 2014-06-15 at 13:35 +0200, lee wrote:
> Ian Campbell <ijc at hellion.org.uk> writes:
> 
> > On Sun, 2014-06-15 at 11:03 +0200, lee wrote:
> >> Hi,
> >> 
> >> how could I get better debugging output from the dom0 kernel?
> >
> > Either configure a serial console [0] or add "noreboot" to
your
> > *hypervisor* command line, so you can see (and perhaps
> > transcribe/photograph) the crash before manually rebooting.
> 
> The server isn't rebooting automatically, it only gets stuck.  I still
> have a monitor connected to it, so I can see what's printed to the
> console.  But the output is scrolled off screen by subsequent messages
> so that it's gone by the time I notice that it hangs.
> 
> If there was a way to disable the subsequent messages, I could take a
> picture.  Perhaps I can get output from a serial console if I manage to
> set that up ... or maybe I can actually make it crash instead of having
> to wait ...
Serial console would be best. I think the Linux kernel also has a panic
on crash option somewhere, perhaps oops=panic, but it depends rather
what form the initial error takes.

I suppose there are also various software monitoring solutions which
would alert you to something odd in the dom0 dmesg e.g. via email.

Ian.

Ian Campbell <ijc at hellion.org.uk> writes:
> On Sun, 2014-06-15 at 13:35 +0200, lee wrote:
>> Ian Campbell <ijc at hellion.org.uk> writes:
>> 
>> > On Sun, 2014-06-15 at 11:03 +0200, lee wrote:
>> >> Hi,
>> >> 
>> >> how could I get better debugging output from the dom0 kernel?
>> >
>> > Either configure a serial console [0] or add "noreboot"
to your
>> > *hypervisor* command line, so you can see (and perhaps
>> > transcribe/photograph) the crash before manually rebooting.
>> 
>> The server isn't rebooting automatically, it only gets stuck.  I
still
>> have a monitor connected to it, so I can see what's printed to the
>> console.  But the output is scrolled off screen by subsequent messages
>> so that it's gone by the time I notice that it hangs.
>> 
>> If there was a way to disable the subsequent messages, I could take a
>> picture.  Perhaps I can get output from a serial console if I manage to
>> set that up ... or maybe I can actually make it crash instead of having
>> to wait ...
>
> Serial console would be best. I think the Linux kernel also has a panic
> on crash option somewhere, perhaps oops=panic, but it depends rather
> what form the initial error takes.
That's the most difficult option --- the server has a management card
which apparently is able to show you console output in a web browser,
but that seems to require some java plugin to work, and since nothing
java ever works, I haven't been able to use that --- plus I had to get
another network cable first.

Last time it crashed I looked, but only an empty screen was displayed on
the console.  So I upgraded to a 3.14 kernel from backports, and it's
running over 24 hours yet.

It might be a good idea to upgrade the kernel for stable or at least to
place an info into the description that if it crashes, ppl can try a
backports kernel.
> I suppose there are also various software monitoring solutions which
> would alert you to something odd in the dom0 dmesg e.g. via email.
That could be useful for normal operation.  When the kernel has crashed,
such software probably can't do anything anymore.


-- 
Knowledge is volatile and fluid.  Software is power.

lee <lee at yun.yagibdah.de> writes:
> If there was a way to disable the subsequent messages, I could take a
> picture.  Perhaps I can get output from a serial console if I manage to
> set that up ... or maybe I can actually make it crash instead of having
> to wait ...
Ok, I can't make it crash, but this time it went down with a stream of
messages like


aacraid 000:04:00.0: swiotlb buffer is full (sz: 255 bytes)


This happened with '3.14-0.bpo.1-amd64 #1 SMP Debian 3.14.5-1~bpo70+1
(2014-06-05) x86_64 GNU/Linux'.  This seems to be the most recent kernel
available for Wheezy.  There is some indication at [1] that the problem
/might/ be fixed or doesn't come up in 3.4 kernels.


What should I do:

Make a bug report with this message?  Perhaps the problem is already
fixed?

Get the 3.15.1 kernel source from kernel.org and compile it and try it?
Perhaps that doesn't work with the rest of the system?

I need the server to be stable ...


[1]: http://www.spinics.net/lists/linux-scsi/msg70278.html


-- 
Knowledge is volatile and fluid.  Software is power.