Fedora xen - Aug 2006 - [BUG] dmidecode in DomU throws thousands of xen printk''s

Hi All

One more bug... in a seperate email so they don''t get confused.

System is kernel-xen-2.6.17-1.2157_FC5 on a 2p opteron (older 
non-hardware-virt). This happens when I try to run the
''dmidecode'' tool
from within a non-privileged guest. The guest is running the same kernel 
as above. One other note is that the console on this system is serial, 
ttyS0 at 9600bps... which may have something to do with the CPU lockup 
bugs below.

I basically get thousands of the following printk messages, starting with:

(XEN) DOM4: (file=mm.c, line=578) Non-privileged attempt to map I/O space 
000000f0

and counting up and ending with:

(XEN) DOM4: (file=mm.c, line=578) Non-privileged attempt to map I/O space 
000000ff

Then, some of the time, the kernel throws a spinlock timer at the end of 
all of these printks:

Timer ISR/1: Time went backwards: delta=-46186990 delta_cpu=-42186990 
shadow=2112479126776 off=466807746 processed=2112992120678 
cpu_processed=2112988120678
BUG: spinlock lockup on CPU#0, swapper/0, c06e9284 (Not tainted)
  <c04e74b0> __spin_lock_debug+0x7b/0x85  <c04e7514> 
_raw_spin_lock+0x5a/0x69
  <c061eedb> _spin_lock+0x6/0x8  <c0407d80>
timer_interrupt+0x40/0x563
  <c0434533> ktime_get_ts+0x4b/0x51  <c0434484> ktime_get+0x10/0x3a
  <c0442fe3> handle_IRQ_event+0x40/0x82  <c04430b0>
__do_IRQ+0x8b/0xdf
  <c0406336> do_IRQ+0x1a/0x25  <c054c937> evtchn_do_upcall+0x84/0xb8
  <c0404c71> hypervisor_callback+0x3d/0x48  <c04087da>
safe_halt+0x1a/0x2f
  <c0402b98> kernel_thread_helper+0x0/0xb  <c04028a9>
xen_idle+0x45/0x4d
  <c0402945> cpu_idle+0x94/0xad  <c07047cf> start_kernel+0x1c1/0x1c5
  0: 2112992120678
  1: 2112988120678
BUG: soft lockup detected on CPU#1!
  <c0442e2d> softlockup_tick+0x9a/0xab  <c040825d> 
timer_interrupt+0x51d/0x563
  <c0442fe3> handle_IRQ_event+0x40/0x82  <c04430b0>
__do_IRQ+0x8b/0xdf
  <c0406336> do_IRQ+0x1a/0x25  <c054c937> evtchn_do_upcall+0x84/0xb8
  <c0404c71> hypervisor_callback+0x3d/0x48  <c054c8b1> 
force_evtchn_callback+0xa/0xc
  <c04248cf> __do_softirq+0x67/0xee  <c0424995> do_softirq+0x3f/0x63
  <c040633b> do_IRQ+0x1f/0x25  <c054c937> evtchn_do_upcall+0x84/0xb8
  <c0404c71> hypervisor_callback+0x3d/0x48  <c04087da>
safe_halt+0x1a/0x2f
  <c04028a9> xen_idle+0x45/0x4d  <c0402945> cpu_idle+0x94/0xad
BUG: soft lockup detected on CPU#0!
  <c0442e2d> softlockup_tick+0x9a/0xab  <c040825d> 
timer_interrupt+0x51d/0x563
  <c0434533> ktime_get_ts+0x4b/0x51  <c0442fe3>
handle_IRQ_event+0x40/0x82
  <c04430b0> __do_IRQ+0x8b/0xdf  <c0406336> do_IRQ+0x1a/0x25
  <c054c937> evtchn_do_upcall+0x84/0xb8  <c0404c71> 
hypervisor_callback+0x3d/0x48
  <c04087da> safe_halt+0x1a/0x2f  <c0402b98>
kernel_thread_helper+0x0/0xb
  <c04028a9> xen_idle+0x45/0x4d  <c0402945> cpu_idle+0x94/0xad
  <c07047cf> start_kernel+0x1c1/0x1c5

Once, I even had the storage controller (3ware 9500) barf on me due to an 
I/O timeout that it couldn''t recover from. I didn''t have my
console
capture setup for that one, unfortunately.

I assume that this is the smbios hardware address space that dmidecode is 
trying to access via the guest kernel. I''m also guessing that the
printk
has interrupts disabled or something similarly funky, so the flood of them 
causes the CPU lockup messages as well as probably the I/O problems.

I saw some patches on the xen-devel list about adding real smbios data to 
the HVM virtual hosts, but it seemed to be very HVM-specific code, and
I''m
running paravirts. The thread from that is here:
http://lists.xensource.com/archives/html/xen-devel/2006-07/msg00416.html

I''d be happy if dmidecode just returned nothing but the console
didn''t
spew messages like this. As you can imagine, the whole system (all virts) 
is unuseable while this is happening, and it runs for minutes at a time.

Thanks again

-Matt

On Mon, Aug 07, 2006 at 11:40:34PM -0700, Matt C wrote:
> System is kernel-xen-2.6.17-1.2157_FC5 on a 2p opteron (older 
> non-hardware-virt). This happens when I try to run the
''dmidecode'' tool
> from within a non-privileged guest. The guest is running the same kernel 
> as above. One other note is that the console on this system is serial, 
> ttyS0 at 9600bps... which may have something to do with the CPU lockup 
> bugs below.
[snip] 
> I saw some patches on the xen-devel list about adding real smbios data to 
> the HVM virtual hosts, but it seemed to be very HVM-specific code, and
I''m
> running paravirts. The thread from that is here:
> http://lists.xensource.com/archives/html/xen-devel/2006-07/msg00416.html
Indeed, thre is no SMBios data made available to ParaVirt guests so it is
not currently possible to make dmidecide work for PV guests. This may change
in the future, but I don''t think anyone is actively working on SMBios
support
for PV just yet.
> I''d be happy if dmidecode just returned nothing but the console
didn''t
> spew messages like this. As you can imagine, the whole system (all virts) 
> is unuseable while this is happening, and it runs for minutes at a time.
Best to file a bug report about this denial of service in bugzilla.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

> [snip]
>
> Indeed, thre is no SMBios data made available to ParaVirt guests so it is
> not currently possible to make dmidecide work for PV guests. This may
change
> in the future, but I don''t think anyone is actively working on
SMBios support
> for PV just yet.
>
> Best to file a bug report about this denial of service in bugzilla.
>
Filed...

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=201799

Thanks!

-matt

Daniel P. Berrange wrote:
> On Mon, Aug 07, 2006 at 11:40:34PM -0700, Matt C wrote:
>> System is kernel-xen-2.6.17-1.2157_FC5 on a 2p opteron (older 
>> non-hardware-virt). This happens when I try to run the
''dmidecode'' tool
>> from within a non-privileged guest. The guest is running the same
kernel
>> as above. One other note is that the console on this system is serial, 
>> ttyS0 at 9600bps... which may have something to do with the CPU lockup 
>> bugs below.
The guest should not be trying to do those operations.  I can
take a look into tightening our /dev/mem patch, since it does
not seem to happen with some of our Xen kernels but it does
with others.
> Indeed, thre is no SMBios data made available to ParaVirt guests so it is
> not currently possible to make dmidecide work for PV guests. This may
change
> in the future, but I don''t think anyone is actively working on
SMBios support
> for PV just yet.
Yeah, I''m pretty sure we don''t want SMBios data for paravirt
guests.  After all, they''re not running on real hardware...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Rik van Riel wrote:
> Daniel P. Berrange wrote:
>> On Mon, Aug 07, 2006 at 11:40:34PM -0700, Matt C wrote:
>>> System is kernel-xen-2.6.17-1.2157_FC5 on a 2p opteron (older 
>>> non-hardware-virt). This happens when I try to run the
''dmidecode''
>>> tool from within a non-privileged guest. The guest is running the 
>>> same kernel as above. One other note is that the console on this 
>>> system is serial, ttyS0 at 9600bps... which may have something to
do
>>> with the CPU lockup bugs below.
> 
> The guest should not be trying to do those operations.  I can
> take a look into tightening our /dev/mem patch,
The bug seems to have disappeared with the latest rawhide
kernels.  That is, the hypervisor prints one message for
a dmidecode invocation, not thousands...

Matt, is this still an issue or should I try to fix some
more urgent stuff first? :)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan