thr3ads.net - Xen devel - [Xen-devel] Xen 3.2.1-rc1: Assertion ''!(*ar & ( 3<<13))'' failed at traps.c:1305 [Apr 2008]

If this information is useful, please help other people find it:
Share via:

Christopher S. Aker

2008-Apr-07 19:43 UTC

[Xen-devel] Xen 3.2.1-rc1: Assertion ''!(*ar & ( 3<<13))'' failed at traps.c:1305

We''ve set up a few xen-thrash environments, which are a mix of a nodes 
being created and destroyed, created and shutdown, long running 
swap-thrashing nodes, long-running crashme nodes, etc.  About 50 domains 
in total.  We''ve managed to hit a crash!

Xen:
http://www.theshore.net/~caker/xen/BUGfatal_page_fault/

domU:
http://www.theshore.net/~caker/xen/kernels/2.6.24.4-linode8

On this particular host, we''re running all pv_ops guest kernels.  Trace
as follows:

(XEN) traps.c:2540: GPF (9898): ffff828c8014c387 -> ffff828c801be667
(XEN) traps.c:2540: GPF (c708): ffff828c8014c387 -> ffff828c801be667
(XEN) Assertion ''!(*ar & ( 3<<13))'' failed at
traps.c:1305
(XEN) ----[ Xen-3.2.1-rc1  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    6
(XEN) RIP:    e008:[<ffff828c8014c3bf>] read_gate_descriptor+0xa9/0x15b
(XEN) RFLAGS: 0000000000010206   CONTEXT: hypervisor
(XEN) rax: 0000000000cfffff   rbx: ffff8300cee17ecc   rcx: 0000000000cffb00
(XEN) rdx: 000000000000fb00   rsi: ffff8300cfd60100   rdi: 0000000000000070
(XEN) rbp: ffff8300cee17e38   rsp: ffff8300cee17e28   r8:  ffff8300cee17ec8
(XEN) r9:  ffff820000020070   r10: 000000000000ffff   r11: ffff8300cee17ec0
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 00000005f3c04000   cr2: 000000000804e1b4
(XEN) ds: 007b   es: 007b   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff8300cee17e28:
(XEN)    ffff8300cee17f28 ffff8300cfd60100 ffff8300cee17ef8 
ffff828c8014eb27
(XEN)    ffff8300cee17e78 ffff828c801074c0 0000000000000000 
0000000000000000
(XEN)    ffff8300cf28c100 ffff8300cfd76100 ffff8300cfd60100 
0000000000000000
(XEN)    ffff8300cfd76100 ffff8300cfd60100 ffff8300cee17ef8 
ffff828c80141576
(XEN)    ffff8300cee17eb8 00000005e1dc6067 00000005e1a64067 
0000000000cfffff
(XEN)    0000000000000006 ffff8300cee17f28 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 ffff8300cee17f18 
ffff828c8014fad5
(XEN)    ffff8300cfd60100 0000000000000000 00007cff311e80b7 
ffff828c801b367b
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    00000000bf94f358 000000000804c060 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000391fe938 
00000000cc38755e
(XEN)    000000000804c060 0000000000000000 000000000000002c 
0000000d00000070
(XEN)    000000000804d9f2 0000000000000073 0000000000010212 
00000000bf94f338
(XEN)    000000000000007b 5555555555555555 5555555555555555 
5555555555555555
(XEN)    5555555555555555 5555555500000006 ffff8300cfd60100
(XEN) Xen call trace:
(XEN)    [<ffff828c8014c3bf>] read_gate_descriptor+0xa9/0x15b
(XEN)    [<ffff828c8014eb27>] emulate_gate_op+0x50/0xeae
(XEN)    [<ffff828c8014fad5>] do_general_protection+0x150/0x205
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 6:
(XEN) Assertion ''!(*ar & ( 3<<13))'' failed at
traps.c:1305
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) traps.c:2540: GPF (bfd8): ffff828c8014c387 -> ffff828c801be667
(XEN) traps.c:2540: GPF (d760): ffff828c8014c387 -> ffff828c801be667
(XEN) traps.c:2540: GPF (9060): ffff828c8014c387 -> ffff828c801be667
(XEN) traps.c:2540: GPF (8050): ffff828c8014c387 -> ffff828c801be667
(XEN) traps.c:2540: GPF (9898): ffff828c8014c387 -> ffff828c801be667
(XEN) traps.c:2540: GPF (c708): ffff828c8014c387 -> ffff828c801be667
(XEN) Assertion ''!(*ar & ( 3<<13))'' failed at
traps.c:1305
(XEN) ----[ Xen-3.2.1-rc1  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff828c8014c3bf>] read_gate_descriptor+0xa9/0x15b
(XEN) RFLAGS: 0000000000010206   CONTEXT: hypervisor
(XEN) rax: 0000000000cfffff   rbx: ffff8300cf13fecc   rcx: 0000000000cffb00
(XEN) rdx: 000000000000fb00   rsi: ffff8300cfd5c100   rdi: 0000000000000070
(XEN) rdx: 000000000000fb00   rsi: ffff8300cfd5c100   rdi: 0000000000000070
(XEN) rbp: ffff8300cf13fe38   rsp: ffff8300cf13fe28   r8:  ffff8300cf13fec8
(XEN) r9:  ffff820000040070   r10: 000000000000ffff   r11: ffff8300cf13fec0
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 00000005f3c02000   cr2: 000000000804e1b4
(XEN) ds: 007b   es: 007b   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff8300cf13fe28:
Apr  7 15:27:06 opengear-tp3/70.85.129.5 port05  RXDATA: 
<14>dispatchd[2844]: Checking for jobs (HostID 135)
(XEN)    ffff8300cf13ff28 ffff8300cfd5c100 ffff8300cf13fef8 
ffff828c8014eb27
Apr  7 15:27:06 opengear-tp3/70.85.129.5 port05  RXDATA: 
<14>dispatchd[2844]: Going to sleep for 45 seconds
(XEN)    ffff8300cf13fe78 ffff828c801074c0 0000000000000000 
0000000000000000
(XEN)    ffff8300cf2de100 ffff8300cfd76100 ffff8300cfd5c100 
0000000000000000
(XEN)    ffff8300cfd76100 ffff8300cfd5c100 ffff8300cf13fef8 
ffff828c80141576
(XEN)    ffff8300cf13feb8 00000005e253c067 00000005e195b067 
0000000000cfffff
(XEN)    0000000000000001 ffff8300cf13ff28 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 ffff8300cf13ff18 
ffff828c8014fad5
(XEN)    ffff8300cfd5c100 0000000000000000 00007cff30ec00b7 
ffff828c801b367b
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    00000000bf980388 000000000804c060 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 000000003922f968 
00000000cc38755e
(XEN)    000000000804c060 0000000000000000 000000000000002c 
0000000d00000070
(XEN)    000000000804d9f2 0000000000000073 0000000000010212 
00000000bf980368
(XEN)    000000000000007b 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000001 ffff8300cfd5c100
(XEN) Xen call trace:
(XEN)    [<ffff828c8014c3bf>] read_gate_descriptor+0xa9/0x15b
(XEN)    [<ffff828c8014eb27>] emulate_gate_op+0x50/0xeae
(XEN)    [<ffff828c8014fad5>] do_general_protection+0x150/0x205
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) Assertion ''!(*ar & ( 3<<13))'' failed at
traps.c:1305
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

and finally it rebooted itself..

-Chris


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2008-Apr-08 06:46 UTC

head link

Re: [Xen-devel] Xen 3.2.1-rc1: Assertion ''!(*ar & ( 3<<13))'' failed at traps.c:1305

On 7/4/08 20:43, "Christopher S. Aker" <caker@theshore.net>
wrote:
> We''ve set up a few xen-thrash environments, which are a mix of a
nodes
> being created and destroyed, created and shutdown, long running
> swap-thrashing nodes, long-running crashme nodes, etc.  About 50 domains
> in total.  We''ve managed to hit a crash!
> 
> Xen:
> http://www.theshore.net/~caker/xen/BUGfatal_page_fault/
> 
> domU:
> http://www.theshore.net/~caker/xen/kernels/2.6.24.4-linode8
> 
> On this particular host, we''re running all pv_ops guest kernels. 
Trace
> as follows:
> 
> (XEN) traps.c:2540: GPF (9898): ffff828c8014c387 -> ffff828c801be667
> (XEN) traps.c:2540: GPF (c708): ffff828c8014c387 -> ffff828c801be667
> (XEN) Assertion ''!(*ar & ( 3<<13))'' failed at
traps.c:1305
Weird error condition (which may well mean the domU is doomed) has triggered
an unexpected path in Xen. I''ll give the code concerned an audit and at
the
very least I should be able to fix the host crash (hopefully, unless weird
corruption really must have happened).

The GPFs are interesting -- those are very big and random looking segment
selector values (9898 c708), rather like someone is trying load random
numbers into segment registers. It reminds me of the first crash you posted
about, which was a random number turning up in a page directory entry. I
wonder if there is a common cause.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2008-Apr-08 08:50 UTC

head link

Re: [Xen-devel] Xen 3.2.1-rc1: Assertion ''!(*ar & ( 3<<13))'' failed at traps.c:1305

Okay, this one is fixed. The weird GPFs are explained by the fact that you
were running crashme (I missed that on first skimming of your email). The
Xen crash is now fixed in 3.2-testing as changeset 16845.

 Thanks,
 Keir

On 7/4/08 20:43, "Christopher S. Aker" <caker@theshore.net>
wrote:
> We''ve set up a few xen-thrash environments, which are a mix of a
nodes
> being created and destroyed, created and shutdown, long running
> swap-thrashing nodes, long-running crashme nodes, etc.  About 50 domains
> in total.  We''ve managed to hit a crash!
> 
> Xen:
> http://www.theshore.net/~caker/xen/BUGfatal_page_fault/
> 
> domU:
> http://www.theshore.net/~caker/xen/kernels/2.6.24.4-linode8
> 
> On this particular host, we''re running all pv_ops guest kernels. 
Trace
> as follows:
> 
> (XEN) traps.c:2540: GPF (9898): ffff828c8014c387 -> ffff828c801be667
> (XEN) traps.c:2540: GPF (c708): ffff828c8014c387 -> ffff828c801be667
> (XEN) Assertion ''!(*ar & ( 3<<13))'' failed at
traps.c:1305
> (XEN) ----[ Xen-3.2.1-rc1  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    6
> (XEN) RIP:    e008:[<ffff828c8014c3bf>]
read_gate_descriptor+0xa9/0x15b
> (XEN) RFLAGS: 0000000000010206   CONTEXT: hypervisor
> (XEN) rax: 0000000000cfffff   rbx: ffff8300cee17ecc   rcx: 0000000000cffb00
> (XEN) rdx: 000000000000fb00   rsi: ffff8300cfd60100   rdi: 0000000000000070
> (XEN) rbp: ffff8300cee17e38   rsp: ffff8300cee17e28   r8:  ffff8300cee17ec8
> (XEN) r9:  ffff820000020070   r10: 000000000000ffff   r11: ffff8300cee17ec0
> (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
> (XEN) cr3: 00000005f3c04000   cr2: 000000000804e1b4
> (XEN) ds: 007b   es: 007b   fs: 0000   gs: 0000   ss: 0000   cs: e008
> (XEN) Xen stack trace from rsp=ffff8300cee17e28:
> (XEN)    ffff8300cee17f28 ffff8300cfd60100 ffff8300cee17ef8
> ffff828c8014eb27
> (XEN)    ffff8300cee17e78 ffff828c801074c0 0000000000000000
> 0000000000000000
> (XEN)    ffff8300cf28c100 ffff8300cfd76100 ffff8300cfd60100
> 0000000000000000
> (XEN)    ffff8300cfd76100 ffff8300cfd60100 ffff8300cee17ef8
> ffff828c80141576
> (XEN)    ffff8300cee17eb8 00000005e1dc6067 00000005e1a64067
> 0000000000cfffff
> (XEN)    0000000000000006 ffff8300cee17f28 0000000000000000
> 0000000000000000
> (XEN)    0000000000000000 0000000000000000 ffff8300cee17f18
> ffff828c8014fad5
> (XEN)    ffff8300cfd60100 0000000000000000 00007cff311e80b7
> ffff828c801b367b
> (XEN)    0000000000000000 0000000000000000 0000000000000000
> 0000000000000000
> (XEN)    00000000bf94f358 000000000804c060 0000000000000000
> 0000000000000000
> (XEN)    0000000000000000 0000000000000000 00000000391fe938
> 00000000cc38755e
> (XEN)    000000000804c060 0000000000000000 000000000000002c
> 0000000d00000070
> (XEN)    000000000804d9f2 0000000000000073 0000000000010212
> 00000000bf94f338
> (XEN)    000000000000007b 5555555555555555 5555555555555555
> 5555555555555555
> (XEN)    5555555555555555 5555555500000006 ffff8300cfd60100
> (XEN) Xen call trace:
> (XEN)    [<ffff828c8014c3bf>] read_gate_descriptor+0xa9/0x15b
> (XEN)    [<ffff828c8014eb27>] emulate_gate_op+0x50/0xeae
> (XEN)    [<ffff828c8014fad5>] do_general_protection+0x150/0x205
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 6:
> (XEN) Assertion ''!(*ar & ( 3<<13))'' failed at
traps.c:1305
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
> (XEN) traps.c:2540: GPF (bfd8): ffff828c8014c387 -> ffff828c801be667
> (XEN) traps.c:2540: GPF (d760): ffff828c8014c387 -> ffff828c801be667
> (XEN) traps.c:2540: GPF (9060): ffff828c8014c387 -> ffff828c801be667
> (XEN) traps.c:2540: GPF (8050): ffff828c8014c387 -> ffff828c801be667
> (XEN) traps.c:2540: GPF (9898): ffff828c8014c387 -> ffff828c801be667
> (XEN) traps.c:2540: GPF (c708): ffff828c8014c387 -> ffff828c801be667
> (XEN) Assertion ''!(*ar & ( 3<<13))'' failed at
traps.c:1305
> (XEN) ----[ Xen-3.2.1-rc1  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    1
> (XEN) RIP:    e008:[<ffff828c8014c3bf>]
read_gate_descriptor+0xa9/0x15b
> (XEN) RFLAGS: 0000000000010206   CONTEXT: hypervisor
> (XEN) rax: 0000000000cfffff   rbx: ffff8300cf13fecc   rcx: 0000000000cffb00
> (XEN) rdx: 000000000000fb00   rsi: ffff8300cfd5c100   rdi: 0000000000000070
> (XEN) rdx: 000000000000fb00   rsi: ffff8300cfd5c100   rdi: 0000000000000070
> (XEN) rbp: ffff8300cf13fe38   rsp: ffff8300cf13fe28   r8:  ffff8300cf13fec8
> (XEN) r9:  ffff820000040070   r10: 000000000000ffff   r11: ffff8300cf13fec0
> (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
> (XEN) cr3: 00000005f3c02000   cr2: 000000000804e1b4
> (XEN) ds: 007b   es: 007b   fs: 0000   gs: 0000   ss: 0000   cs: e008
> (XEN) Xen stack trace from rsp=ffff8300cf13fe28:
> Apr  7 15:27:06 opengear-tp3/70.85.129.5 port05  RXDATA:
> <14>dispatchd[2844]: Checking for jobs (HostID 135)
> (XEN)    ffff8300cf13ff28 ffff8300cfd5c100 ffff8300cf13fef8
> ffff828c8014eb27
> Apr  7 15:27:06 opengear-tp3/70.85.129.5 port05  RXDATA:
> <14>dispatchd[2844]: Going to sleep for 45 seconds
> (XEN)    ffff8300cf13fe78 ffff828c801074c0 0000000000000000
> 0000000000000000
> (XEN)    ffff8300cf2de100 ffff8300cfd76100 ffff8300cfd5c100
> 0000000000000000
> (XEN)    ffff8300cfd76100 ffff8300cfd5c100 ffff8300cf13fef8
> ffff828c80141576
> (XEN)    ffff8300cf13feb8 00000005e253c067 00000005e195b067
> 0000000000cfffff
> (XEN)    0000000000000001 ffff8300cf13ff28 0000000000000000
> 0000000000000000
> (XEN)    0000000000000000 0000000000000000 ffff8300cf13ff18
> ffff828c8014fad5
> (XEN)    ffff8300cfd5c100 0000000000000000 00007cff30ec00b7
> ffff828c801b367b
> (XEN)    0000000000000000 0000000000000000 0000000000000000
> 0000000000000000
> (XEN)    00000000bf980388 000000000804c060 0000000000000000
> 0000000000000000
> (XEN)    0000000000000000 0000000000000000 000000003922f968
> 00000000cc38755e
> (XEN)    000000000804c060 0000000000000000 000000000000002c
> 0000000d00000070
> (XEN)    000000000804d9f2 0000000000000073 0000000000010212
> 00000000bf980368
> (XEN)    000000000000007b 0000000000000000 0000000000000000
> 0000000000000000
> (XEN)    0000000000000000 0000000000000001 ffff8300cfd5c100
> (XEN) Xen call trace:
> (XEN)    [<ffff828c8014c3bf>] read_gate_descriptor+0xa9/0x15b
> (XEN)    [<ffff828c8014eb27>] emulate_gate_op+0x50/0xeae
> (XEN)    [<ffff828c8014fad5>] do_general_protection+0x150/0x205
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 1:
> (XEN) Assertion ''!(*ar & ( 3<<13))'' failed at
traps.c:1305
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
> 
> and finally it rebooted itself..
> 
> -Chris
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - Apr 2008 - Xen 3.2.1-rc1: Assertion ''!(*ar & ( 3<<13))'' failed at traps.c:1305

[Xen-devel] Xen 3.2.1-rc1: Assertion ''!(*ar & ( 3<<13))'' failed at traps.c:1305

Re: [Xen-devel] Xen 3.2.1-rc1: Assertion ''!(*ar & ( 3<<13))'' failed at traps.c:1305

Re: [Xen-devel] Xen 3.2.1-rc1: Assertion ''!(*ar & ( 3<<13))'' failed at traps.c:1305