Xen devel - Apr 2005 - xen-unstable domU smp broken for vcpus > physical cpus

Hi,

Rik mentioned in #xen on irc.oftc.net that his smp domU was crashing.
Mine was working and we started to investigate.  Turns out that using
xen-unstable from 2005-04-01 when domU is launched with vcpus > physical
cpus, domU, and in my case, Xen itself crash.  Attached is the debug log
of Xen going down.

Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I can reproduce reliably.   My setup is vcpus > physical in domU.  SMP 
is disabled in dom0.  Only my linux kernel crashes (in domU).  Attached 
is the oops.

Ryan Harper wrote:
>Hi,
>
>Rik mentioned in #xen on irc.oftc.net that his smp domU was crashing.
>Mine was working and we started to investigate.  Turns out that using
>xen-unstable from 2005-04-01 when domU is launched with vcpus > physical
>cpus, domU, and in my case, Xen itself crash.  Attached is the debug log
>of Xen going down.
>
>Ryan Harper
>Software Engineer; Linux Technology Center
>IBM Corp., Austin, Tx
>(512) 838-9253   T/L: 678-9253
>ryanh@us.ibm.com
>  
>
>------------------------------------------------------------------------
>
>(XEN) Assertion ''(x & PGT_count_mask) != 0'' failed,
line 1087, file mm.c
>(XEN) BUG at mm.c:1087
>(XEN) CPU:    1
>(XEN) EIP:    0808:[<fc520bfb>]      
>(XEN) EFLAGS: 00010292   CONTEXT: hypervisor
>(XEN) eax: fc562191   ebx: f0000000   ecx: 00000000   edx: 00000000
>(XEN) esi: fc5553d0   edi: f0000000   ebp: efffffff   esp: fcff9d24
>(XEN) ds: 0810   es: 0810   fs: 0810   gs: 0810   ss: 0810   cs: 0808
>(XEN) Stack trace from ESP=fcff9d24:
>(XEN)    fc54d1ba fc555eda 0000043f fc555eda 075e0000 00000004 00000000
fc000000
>(XEN)    c00245b4 c00245b8 00000000 fd4b0010 fcffac40 00000958 fcffac40
[fc529678]
>(XEN)    fd4b0010 e0000000 00000001 000006c0 fc56a6c0 0000001b fcff9db8
[fc51eebf]
>(XEN)    fd93faa8 fcffac40 80000002 80000003 80000002 fcffac40 fcffac40
fd4b7810
>(XEN)    07556063 07a56000 00000004 07a56000 fc558248 c0025958 07a56000
[fc52925b]
>(XEN)    c0025958 00000000 07a56000 00000004 00000000 00000004 0000004e
[fc536c35]
>(XEN)    c0025958 07a56000 00000004 00000004 00000040 [fc50f14a] [fc520c1e]
00000808
>(XEN)    0651cd76 00000000 00000000 00000004 00000000 006dc0a0 00000000
006dc0a0
>(XEN)    00000000 a9470320 00000034 0000e76d fcfe7089 fcff3bac fcfe7040
00000040
>(XEN)    [fc50f14a] 00000292 fcfb9040 00000002 00000040 00000001 fcffb640
aec00b55
>(XEN)    00000092 fcffb640 fc561790 fffffffc 00000006 00000092 00003700
00000089
>(XEN)    c10004a0 07a56000 00000063 c0025958 c1000000 00000000 07a56000
000e0003
>(XEN)    c0114a2b 00000061 00010246 c7a4fe3c 00000069 0000007b 0000007b
00000000
>(XEN)    00000000 ff913000 fcffac40 20000000 00000001 00000004 07a56000
07556063
>(XEN)    c0025958 075e0000 c7a4fe1c 0000000c 00000000 00000004 07a56000
07a56000
>(XEN)    fcff9e9c 00000034 fcffac40 00000000 fd4b0d00 00000000 fcffac40
c0025958
>(XEN)    fcff9fb8 fcfe6040 fc558248 [fc5255eb] fcff9fb8 c0025958 fc558248
00000004
>(XEN)    fc561204 fc561204 c7a4fb58 fcfe6040 37fc7061 fcffac40 80000002
fcfe6040
>(XEN)    fcffac40 00037fc7 19ee6b00 fcffac40 c0025958 fcfe6040 fcff9fb8
[fc530262]
>(XEN)    c0025958 00000006 c131ef20 c0114d38 00000000 075e0bfc 00000000
fcfe6040
>(XEN) Call Trace from ESP=fcff9d24:
>(XEN)    [<fc529678>] [<fc51eebf>] [<fc52925b>]
[<fc536c35>] [<fc50f14a>] [<fc520c1e>]
>(XEN)    [<fc50f14a>] [<fc5255eb>] [<fc530262>]
[<fc53981e>]
>
>****************************************
>CPU1 FATAL TRAP: vector = 6 (invalid operand)
>[error_code=0000]
>Aieee! CPU1 is toast...
>****************************************
>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>  
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, 1 Apr 2005, Ryan Harper wrote:
> Rik mentioned in #xen on irc.oftc.net that his smp domU was crashing.
> Mine was working and we started to investigate.  Turns out that using
> xen-unstable from 2005-04-01 when domU is launched with vcpus > physical
> cpus, domU, and in my case, Xen itself crash.  Attached is the debug log
> of Xen going down.
Interesting, I haven''t made Xen itself crash.  Here are the
config options that may be relevant:

CONFIG_XEN=y
CONFIG_ARCH_XEN=y
CONFIG_NO_IDLE_HZ=y

#
# XEN
#
# CONFIG_XEN_PRIVILEGED_GUEST is not set
# CONFIG_XEN_PHYSDEV_ACCESS is not set
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
# CONFIG_XEN_NETDEV_FRONTEND_PIPELINED_TRANSMITTER is not set
# CONFIG_XEN_BLKDEV_TAP is not set
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_X86=y
# CONFIG_X86_64 is not set
CONFIG_HAVE_ARCH_DEV_ALLOC_SKB=y


-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 2 Apr 2005, at 00:25, Ryan Harper wrote:
> Rik mentioned in #xen on irc.oftc.net that his smp domU was crashing.
> Mine was working and we started to investigate.  Turns out that using
> xen-unstable from 2005-04-01 when domU is launched with vcpus > 
> physical
> cpus, domU, and in my case, Xen itself crash.  Attached is the debug 
> log
> of Xen going down.
A link to the xen-syms file would make the backtrace more useful. Also 
great would be to know in which changeset the bug first appeared: was 
it actually 1st April, or is that just the only one you''ve tested with 
vcpus > pcpus and so it could have been there for weeks or months?

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-04-02
01:36]:
> 
> On 2 Apr 2005, at 00:25, Ryan Harper wrote:
> 
> >Rik mentioned in #xen on irc.oftc.net that his smp domU was crashing.
> >Mine was working and we started to investigate.  Turns out that using
> >xen-unstable from 2005-04-01 when domU is launched with vcpus > 
> >physical
> >cpus, domU, and in my case, Xen itself crash.  Attached is the debug 
> >log
> >of Xen going down.
> 
> A link to the xen-syms file would make the backtrace more useful. Also 
Hrm, I always do the install.sh, shouldnt that set up the xen-syms file
correctly?  What do I need to check to make sure the xen-syms file is
setup up for a more useful backtrace?
> great would be to know in which changeset the bug first appeared: was 
> it actually 1st April, or is that just the only one you''ve tested
with
> vcpus > pcpus and so it could have been there for weeks or months?
I''m fairly certain that I''ve had vcpus > pcpus in domU
working fine as
recent as 2005-03-31, but I''ll backtrack and verify.

Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Ryan Harper <ryanh@us.ibm.com> [2005-04-02
02:34]:
> * Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-04-02 01:36]:
> > 
> > On 2 Apr 2005, at 00:25, Ryan Harper wrote:
> > 
> > >Rik mentioned in #xen on irc.oftc.net that his smp domU was
crashing.
> > >Mine was working and we started to investigate.  Turns out that
using
> > >xen-unstable from 2005-04-01 when domU is launched with vcpus >
> > >physical
> > >cpus, domU, and in my case, Xen itself crash.  Attached is the
debug
> > >log
> > >of Xen going down.
> > 
> 
> > great would be to know in which changeset the bug first appeared: was 
> > it actually 1st April, or is that just the only one you''ve
tested with
> > vcpus > pcpus and so it could have been there for weeks or months?
> 
> I''m fairly certain that I''ve had vcpus > pcpus in domU
working fine as
> recent as 2005-03-31, but I''ll backtrack and verify.
I just built the 2005-03-31 snapshot, dom0 is SMP, two vcpus, and booted
domU with 4 vcpus with no problems.  Building 2005-04-01 in a simliar
fashion now yields an oops in domU and one error [1]output from Xen.
I then went to destroy the domU that I created and Xen went down.  And
appearently, I don''t have xen-sysms set up right as nont of the
addresses are translated into function names.

Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

1. (XEN)
  
(file=/home/rharper/work/openhype/xen/unstable/20050401/b/xen/include/asm/mm.h,
    line=160) Error pfn 000073ae0




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 2 Apr 2005, at 17:34, Ryan Harper wrote:
> I just built the 2005-03-31 snapshot, dom0 is SMP, two vcpus, and 
> booted
> domU with 4 vcpus with no problems.  Building 2005-04-01 in a simliar
> fashion now yields an oops in domU and one error [1]output from Xen.
> I then went to destroy the domU that I created and Xen went down.  And
> appearently, I don''t have xen-sysms set up right as nont of the
> addresses are translated into function names.
What are the top-of-tree changeset dates on these two snapshots? It''d 
be good to know what range of changesets to drill down on. The relevant 
changeset info is one of the first things that Xen prints out after the 
vanity banner during boot.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>> I''m fairly certain that I''ve had vcpus > pcpus in
domU working fine as
>> recent as 2005-03-31, but I''ll backtrack and verify.
>
> I just built the 2005-03-31 snapshot, dom0 is SMP, two vcpus, and 
> booted
> domU with 4 vcpus with no problems.  Building 2005-04-01 in a simliar
> fashion now yields an oops in domU and one error [1]output from Xen.
> I then went to destroy the domU that I created and Xen went down.  And
> appearently, I don''t have xen-sysms set up right as nont of the
> addresses are translated into function names.
The bug was actually in a changeset from late on 03-29, so it should 
have caused crashes as far back as the 2005-03-30 snapshot (the one 
taken on the morning on 03-30).

It''s now fixed in the unstable repository - context switching between 
VCPUs of the same domain incorrectly cleared that physical CPU from the 
domain''s bitmap.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sun, 3 Apr 2005, Keir Fraser wrote:
> It''s now fixed in the unstable repository - context switching
between
> VCPUs of the same domain incorrectly cleared that physical CPU from the 
> domain''s bitmap.
Confirmed, booting xenU domains with vcpus=3 works again,
on my single P4 with HT.  This fix will be in FC4 test 2.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel