Some users are hitting this panic in network_alloc_rx_buffers()
when running a big network intensive app:
/* Check return status of HYPERVISOR_dom_mem_op(). */
if ( rx_mcl[nr_pfns].args[5] != nr_pfns )
panic("Unable to reduce memory reservation\n");
I''m not clear on what is happening here. Are the domains changing
size?
Which memory reservation is being reduced, xen0 or xenU?
All the domains are v2.6.8.1, changeset 1.1331 from Tuesday.
David
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel
This indicates a bug in either XenLinux or in Xen. The network driver must be passing a page frame number to Xen which does not belong to that domain. So someone''s accounting has gone wrong. If you enable debugging in the Xen buil dthen you may get soem useful diagnostic info. -- Keir> > Some users are hitting this panic in network_alloc_rx_buffers() > when running a big network intensive app: > /* Check return status of HYPERVISOR_dom_mem_op(). */ > if ( rx_mcl[nr_pfns].args[5] != nr_pfns ) > panic("Unable to reduce memory reservation\n"); > > I''m not clear on what is happening here. Are the domains changing size? > Which memory reservation is being reduced, xen0 or xenU? > All the domains are v2.6.8.1, changeset 1.1331 from Tuesday. > > David > > > ------------------------------------------------------- > This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 > Project Admins to receive an Apple iPod Mini FREE for your judgement on > who ports your project to Linux PPC the best. Sponsored by IBM. > Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> I''m not clear on what is happening here. Are the domains changing size?Yes they are (by a small, bounded amount). As you may well know, incoming network packets are received into buffers in the backend and those buffers are then page-flipped into the appropriate destination domain. Therefore, domains with virtual ethernet interfaces must return pages to Xen (by reducing their reservation) in return for the pages that are flipped in. The net effect is that both backend and frontend domains vary in size by a small, bounded amount and that copies on the receive path are avoided.> Which memory reservation is being reduced, xen0 or xenU?XenU is reducing its reservation to give receive buffers to the backend driver in Xen0. HTH, Mark> All the domains are v2.6.8.1, changeset 1.1331 from Tuesday. > > David > > > ------------------------------------------------------- > This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 > Project Admins to receive an Apple iPod Mini FREE for your judgement on > who ports your project to Linux PPC the best. Sponsored by IBM. > Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
" debugging in the Xen buil dthen you may get soem useful diagnostic " info. I dunno if its useful, but after xenU panics the output in xm dmesg now has this line repeated for the length of the dmesg buffer: 0 (XEN) (file=traps.c, line=399) Page fault: fc52e2a4 -> fc52e2a6 I''ll setup for a run without writeable page tables next. ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Thu, Sep 23, 2004 at 04:41:52PM -0400, David Becker wrote:> > " debugging in the Xen buil dthen you may get soem useful diagnostic > " info. > > I dunno if its useful, but after xenU panics the output in xm dmesg now > has this line repeated for the length of the dmesg buffer: > > 0 (XEN) (file=traps.c, line=399) Page fault: fc52e2a4 -> fc52e2a6This is most likely a get_user or put_user faulting. Could you run gdb on your xen image and then check which function these addresses are in: disas 0xfc52e2a4 will tell you... christian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
" This is most likely a get_user or put_user faulting. Could you " run gdb on your xen image and then check which function these addresses " are in: disas 0xfc52e2a4 will tell you... " $ gdb xen/xen-syms (gdb) disas 0xfc52e2a4 Dump of assembler code for function __copy_to_user_ll: 0xfc52e270 <__copy_to_user_ll+0>: sub $0x8,%esp 0xfc52e273 <__copy_to_user_ll+3>: mov 0x14(%esp),%ecx 0xfc52e277 <__copy_to_user_ll+7>: mov %esi,(%esp) 0xfc52e27a <__copy_to_user_ll+10>: mov 0x10(%esp),%esi 0xfc52e27e <__copy_to_user_ll+14>: mov %ecx,%edx 0xfc52e280 <__copy_to_user_ll+16>: mov %edi,0x4(%esp) 0xfc52e284 <__copy_to_user_ll+20>: mov 0xc(%esp),%edi 0xfc52e288 <__copy_to_user_ll+24>: cmp $0x7,%ecx 0xfc52e28b <__copy_to_user_ll+27>: jbe 0xfc52e2a4 <__copy_to_user_ll+52> 0xfc52e28d <__copy_to_user_ll+29>: mov %edi,%ecx 0xfc52e28f <__copy_to_user_ll+31>: neg %ecx 0xfc52e291 <__copy_to_user_ll+33>: and $0x7,%ecx 0xfc52e294 <__copy_to_user_ll+36>: sub %ecx,%edx 0xfc52e296 <__copy_to_user_ll+38>: repz movsb %ds:(%esi),%es:(%edi) 0xfc52e298 <__copy_to_user_ll+40>: mov %edx,%ecx 0xfc52e29a <__copy_to_user_ll+42>: shr $0x2,%ecx 0xfc52e29d <__copy_to_user_ll+45>: and $0x3,%edx 0xfc52e2a0 <__copy_to_user_ll+48>: repz movsl %ds:(%esi),%es:(%edi) 0xfc52e2a2 <__copy_to_user_ll+50>: mov %edx,%ecx 0xfc52e2a4 <__copy_to_user_ll+52>: repz movsb %ds:(%esi),%es:(%edi) 0xfc52e2a6 <__copy_to_user_ll+54>: mov (%esp),%esi 0xfc52e2a9 <__copy_to_user_ll+57>: mov %ecx,%eax 0xfc52e2ab <__copy_to_user_ll+59>: mov 0x4(%esp),%edi 0xfc52e2af <__copy_to_user_ll+63>: add $0x8,%esp 0xfc52e2b2 <__copy_to_user_ll+66>: ret 0xfc52e2b3 <__copy_to_user_ll+67>: lea 0x0(%esi),%esi 0xfc52e2b9 <__copy_to_user_ll+73>: lea 0x0(%edi),%edi ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > " debugging in the Xen buil dthen you may get soem useful diagnostic > " info. > > I dunno if its useful, but after xenU panics the output in xm dmesg now > has this line repeated for the length of the dmesg buffer: > > 0 (XEN) (file=traps.c, line=399) Page fault: fc52e2a4 -> fc52e2a6We''ll probably actually know more if you disassemble the 2nd address. This is 2.4 right? I''ve noticed that 2.4 causes occasional page fault / GPF printfs to come out of Xen. I remember thinking that it was slightly odd, but nothing seemed to be broken. Probably worth getting to the bottom of, but it may well turn out to be harmless and unrelated to your panic. It would be useful to print out a bit more information in the panic message. e.g. how many pfns were returned OK. I''m very surprised you''re not getting a message from xen/common/dom_mem_ops.c to say why the page is being refused. My guess would be that Xen thinks that the domain still has a reference to it. Can you get a serial line on one of these machines just so that we can be totally sure you''re getting all the Xen debug output. (I presume ''xm dmesg'' gives you the most recent messages, but I don''t use it myself and hence don;t totally trust it) Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> This is 2.4 right? I''ve noticed that 2.4 causes occasional page > fault / GPF printfs to come out of Xen. I remember thinking that > it was slightly odd, but nothing seemed to be broken. Probably > worth getting to the bottom of, but it may well turn out to be > harmless and unrelated to your panic.The GPF / page fault messages I get from 2.4 actually all turn out to be harmless. The page fault messages come from a get_user in map_ldt_shadow_page, and the GPF messages come when pop''ing an old gs value during a multicall. They''re very occasional and aren''t anything to worry about -- just the sign of a slow path being invoked due to something slightly unusual happening. Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
I turned off CONFIG_XEN_WRITABLE_PAGETABLES and it looks like several runs ran last night without crashing anything. I''ll find out for sure when my user on the west coast gets in. " > 0 (XEN) (file=traps.c, line=399) Page fault: fc52e2a4 -> fc52e2a6 " " We''ll probably actually know more if you disassemble the 2nd " address. The 2nd address is the next instruction. This is 2.6.8.1 for xen0 and xenU. linux-2.4-xenU also crashed which is why we switched xenU to use 2.6 even though it is missing the memory_target feature. The 2.6 crashes have been much more informative. " Can you get a serial line on one of these machines just so that The app runs across a cluster of (in this case) 15 hosts and only one or two VMs panic. So far the users report no shortcuts to reproduce the problem. None of these particular hosts have serial consoles. ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Fri, Sep 24, 2004 at 09:28:54AM -0400, David Becker wrote:> > I turned off CONFIG_XEN_WRITABLE_PAGETABLES and it looks like several runs > ran last night without crashing anything. I''ll find out for sure when > my user on the west coast gets in.Could you try updating to the latest version, with writable pagetables reenabled. Keir has spotted an incorrect check in the writable pagetable code and this could have caused very seldom failures. christian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
" Could you try updating to the latest version, with writable pagetables
" reenabled. Keir has spotted an incorrect check in the writable
xen is faulting the same as the previous pull.
objdump shows the fault address 0xfc52f5b4 in the same function as
the previous pull.
fc52f580 <__copy_to_user_ll>:
fc52f580: 83 ec 08 sub $0x8,%esp
fc52f583: 8b 4c 24 14 mov 0x14(%esp,1),%ecx
fc52f587: 89 34 24 mov %esi,(%esp,1)
fc52f58a: 8b 74 24 10 mov 0x10(%esp,1),%esi
fc52f58e: 89 ca mov %ecx,%edx
fc52f590: 89 7c 24 04 mov %edi,0x4(%esp,1)
fc52f594: 8b 7c 24 0c mov 0xc(%esp,1),%edi
fc52f598: 83 f9 07 cmp $0x7,%ecx
fc52f59b: 76 17 jbe fc52f5b4
<__copy_to_user_ll+0x34>
fc52f59d: 89 f9 mov %edi,%ecx
fc52f59f: f7 d9 neg %ecx
fc52f5a1: 83 e1 07 and $0x7,%ecx
fc52f5a4: 29 ca sub %ecx,%edx
fc52f5a6: f3 a4 repz movsb %ds:(%esi),%es:(%edi)
fc52f5a8: 89 d1 mov %edx,%ecx
fc52f5aa: c1 e9 02 shr $0x2,%ecx
fc52f5ad: 83 e2 03 and $0x3,%edx
fc52f5b0: f3 a5 repz movsl %ds:(%esi),%es:(%edi)
fc52f5b2: 89 d1 mov %edx,%ecx
fc52f5b4: f3 a4 repz movsb %ds:(%esi),%es:(%edi)
fc52f5b6: 8b 34 24 mov (%esp,1),%esi
fc52f5b9: 89 c8 mov %ecx,%eax
fc52f5bb: 8b 7c 24 04 mov 0x4(%esp,1),%edi
fc52f5bf: 83 c4 08 add $0x8,%esp
fc52f5c2: c3 ret
fc52f5c3: 8d b6 00 00 00 00 lea 0x0(%esi),%esi
fc52f5c9: 8d bc 27 00 00 00 00 lea 0x0(%edi,1),%edi
Oddly this time I can''t get gdb to disassemble xen-sysms even though
I built with debug=y:
(gdb) file xen
Reading symbols from xen...(no debugging symbols found)...done.
(gdb) file xen-syms
Reading symbols from xen-syms...(no debugging symbols found)...done.
file says that xen-syms is not stripped and objdump sees the syms in xen-syms
$ file xen xen-syms
xen: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
statically linked, stripped
xen-syms: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
statically linked, not stripped
And here is the full xm dmesg output:
__ __ ____ ___ _ _
\ \/ /___ _ __ |___ \ / _ \ | |__ ___| |_ __ _
\ // _ \ ''_ \ __) || | | |__| ''_ \ / _ \ __/ _` |
/ \ __/ | | | / __/ | |_| |__| |_) | __/ || (_| |
/_/\_\___|_| |_| |_____(_)___/ |_.__/ \___|\__\__,_|
http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory
Xen version 2.0-beta (becker@cs.duke.edu) (gcc version 3.3.4 (Debian
1:3.3.4-6sarge1)) Fri Sep 24 17:37:41 EDT 2004
(XEN) Initialised 1023MB memory (262125 pages) on a 1023MB machine
(XEN) Xen heap size is 10712KB
(XEN) CPU0: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#0: Physical ID: 0, Logical ID: 0
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) found SMP MP-table at 0009e140
(XEN) Memory Reservation 0x9e140, 4096 bytes
(XEN) Memory Reservation 0x9e510, 4096 bytes
(XEN) ACPI: RSDP (v000 IBM ) @ 0x000fdfc0
(XEN) ACPI: RSDT (v001 IBM SERBLADE 0x00001000 IBM 0x45444f43) @ 0x3ffeff80
(XEN) ACPI: FADT (v001 IBM SERBLADE 0x00001000 IBM 0x45444f43) @ 0x3ffeff00
(XEN) ACPI: MADT (v001 IBM SERBLADE 0x00001000 IBM 0x45444f43) @ 0x3ffefe80
(XEN) ACPI: DSDT (v001 IBM SERBLADE 0x00001000 INTL 0x02002025) @ 0x00000000
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 Pentium 4(tm) XEON(tm) APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) Processor #1 Pentium 4(tm) XEON(tm) APIC version 20
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] dfl dfl lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
(XEN) Using ACPI for processor (LAPIC) configuration information
(XEN) Intel MultiProcessor Specification v1.4
(XEN) Virtual Wire compatibility mode.
(XEN) OEM ID: IBM ENSW Product ID: BLADER SMP APIC at: 0xFEE00000
(XEN) I/O APIC #14 Version 17 at 0xFEC00000.
(XEN) I/O APIC #13 Version 17 at 0xFEC01000.
(XEN) I/O APIC #12 Version 17 at 0xFEC02000.
(XEN) Enabling APIC mode: Flat. Using 3 I/O APICs
(XEN) Processors: 2
(XEN) Using scheduler: Borrowed Virtual Time (bvt)
(XEN) Initializing CPU#0
(XEN) Detected 2600.039 MHz processor.
(XEN) CPU0: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#0: Physical ID: 0, Logical ID: 0
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) CPU0 booted
(XEN) enabled ExtINT on CPU#0
(XEN) ESR value before enabling vector: 00000000
(XEN) ESR value after enabling vector: 00000000
(XEN) Booting processor 1/1 eip 90000
(XEN) Initializing CPU#1
(XEN) masked ExtINT on CPU#1
(XEN) ESR value before enabling vector: 00000000
(XEN) ESR value after enabling vector: 00000000
(XEN) CPU1: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#1: Physical ID: 0, Logical ID: 1
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) CPU1 has booted.
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) Setting 14 in the phys_id_present_map
(XEN) ...changing IO-APIC physical APIC ID to 14 ... ok.
(XEN) Setting 13 in the phys_id_present_map
(XEN) ...changing IO-APIC physical APIC ID to 13 ... ok.
(XEN) Setting 12 in the phys_id_present_map
(XEN) ...changing IO-APIC physical APIC ID to 12 ... ok.
(XEN) init IO_APIC IRQs
(XEN) ..TIMER: vector=0x41 pin1=2 pin2=-1
(XEN) number of MP IRQ sources: 12.
(XEN) number of IO-APIC #14 registers: 16.
(XEN) number of IO-APIC #13 registers: 16.
(XEN) number of IO-APIC #12 registers: 16.
(XEN) testing the IO APIC.......................
(XEN)
(XEN) IO APIC #14......
(XEN) .... register #00: 0E000000
(XEN) ....... : physical APIC id: 0E
(XEN) ....... : Delivery Type: 0
(XEN) ....... : LTS : 0
(XEN) .... register #01: 000F0011
(XEN) ....... : max redirection entries: 000F
(XEN) ....... : PRQ implemented: 0
(XEN) ....... : IO APIC version: 0011
(XEN) .... register #02: 0E000000
(XEN) ....... : arbitration: 0E
(XEN) .... IRQ redirection table:
(XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
(XEN) 00 000 00 1 0 0 0 0 0 0 00
(XEN) 01 003 03 0 0 0 0 0 1 1 49
(XEN) 02 003 03 0 0 0 0 0 1 1 41
(XEN) 03 000 00 1 0 0 0 0 0 0 00
(XEN) 04 000 00 1 0 0 0 0 0 0 00
(XEN) 05 000 00 1 0 0 0 0 0 0 00
(XEN) 06 003 03 0 0 0 0 0 1 1 51
(XEN) 07 003 03 1 1 0 1 0 1 1 59
(XEN) 08 003 03 0 0 0 0 0 1 1 61
(XEN) 09 000 00 1 0 0 0 0 0 0 00
(XEN) 0a 000 00 1 0 0 0 0 0 0 00
(XEN) 0b 000 00 1 0 0 0 0 0 0 00
(XEN) 0c 003 03 0 0 0 0 0 1 1 69
(XEN) 0d 003 03 0 0 0 0 0 1 1 71
(XEN) 0e 003 03 0 0 0 0 0 1 1 79
(XEN) 0f 003 03 0 0 0 0 0 1 1 81
(XEN)
(XEN) IO APIC #13......
(XEN) .... register #00: 0D000000
(XEN) ....... : physical APIC id: 0D
(XEN) ....... : Delivery Type: 0
(XEN) ....... : LTS : 0
(XEN) .... register #01: 000F0011
(XEN) ....... : max redirection entries: 000F
(XEN) ....... : PRQ implemented: 0
(XEN) ....... : IO APIC version: 0011
(XEN) .... register #02: 0D000000
(XEN) ....... : arbitration: 0D
(XEN) .... IRQ redirection table:
(XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
(XEN) 00 003 03 1 1 0 1 0 1 1 89
(XEN) 01 003 03 1 1 0 1 0 1 1 91
(XEN) 02 000 00 1 0 0 0 0 0 0 00
(XEN) 03 000 00 1 0 0 0 0 0 0 00
(XEN) 04 003 03 1 1 0 1 0 1 1 99
(XEN) 05 000 00 1 0 0 0 0 0 0 00
(XEN) 06 000 00 1 0 0 0 0 0 0 00
(XEN) 07 000 00 1 0 0 0 0 0 0 00
(XEN) 08 000 00 1 0 0 0 0 0 0 00
(XEN) 09 000 00 1 0 0 0 0 0 0 00
(XEN) 0a 000 00 1 0 0 0 0 0 0 00
(XEN) 0b 000 00 1 0 0 0 0 0 0 00
(XEN) 0c 000 00 1 0 0 0 0 0 0 00
(XEN) 0d 000 00 1 0 0 0 0 0 0 00
(XEN) 0e 000 00 1 0 0 0 0 0 0 00
(XEN) 0f 000 00 1 0 0 0 0 0 0 00
(XEN)
(XEN) IO APIC #12......
(XEN) .... register #00: 0C000000
(XEN) ....... : physical APIC id: 0C
(XEN) ....... : Delivery Type: 0
(XEN) ....... : LTS : 0
(XEN) .... register #01: 000F0011
(XEN) ....... : max redirection entries: 000F
(XEN) ....... : PRQ implemented: 0
(XEN) ....... : IO APIC version: 0011
(XEN) .... register #02: 0C000000
(XEN) ....... : arbitration: 0C
(XEN) .... IRQ redirection table:
(XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
(XEN) 00 000 00 1 0 0 0 0 0 0 00
(XEN) 01 000 00 1 0 0 0 0 0 0 00
(XEN) 02 000 00 1 0 0 0 0 0 0 00
(XEN) 03 000 00 1 0 0 0 0 0 0 00
(XEN) 04 000 00 1 0 0 0 0 0 0 00
(XEN) 05 000 00 1 0 0 0 0 0 0 00
(XEN) 06 000 00 1 0 0 0 0 0 0 00
(XEN) 07 000 00 1 0 0 0 0 0 0 00
(XEN) 08 000 00 1 0 0 0 0 0 0 00
(XEN) 09 000 00 1 0 0 0 0 0 0 00
(XEN) 0a 000 00 1 0 0 0 0 0 0 00
(XEN) 0b 000 00 1 0 0 0 0 0 0 00
(XEN) 0c 000 00 1 0 0 0 0 0 0 00
(XEN) 0d 000 00 1 0 0 0 0 0 0 00
(XEN) 0e 000 00 1 0 0 0 0 0 0 00
(XEN) 0f 000 00 1 0 0 0 0 0 0 00
(XEN) IRQ to pin mappings:
(XEN) IRQ0 -> 0:2
(XEN) IRQ1 -> 0:1
(XEN) IRQ6 -> 0:6
(XEN) IRQ7 -> 0:7
(XEN) IRQ8 -> 0:8
(XEN) IRQ12 -> 0:12
(XEN) IRQ13 -> 0:13
(XEN) IRQ14 -> 0:14
(XEN) IRQ15 -> 0:15
(XEN) IRQ16 -> 1:0
(XEN) IRQ17 -> 1:1
(XEN) IRQ20 -> 1:4
(XEN) .................................... done.
(XEN) Using local APIC timer interrupts.
(XEN) Calibrating APIC timer for CPU0...
(XEN) ..... CPU speed is 2599.9544 MHz.
(XEN) ..... Bus speed is 99.9981 MHz.
(XEN) ..... bus_scale = 0x00006666
(XEN) checking TSC synchronization across CPUs: passed.
(XEN) Time init:
(XEN) .... System Time: 20000499ns
(XEN) .... cpu_freq: 00000000:9AF974B0
(XEN) .... scale: 00000001:89D7146B
(XEN) .... Wall Clock: 1096062327s 310000us
(XEN) PCI: PCI BIOS revision 2.10 entry at 0xfd83c, last bus=4
(XEN) PCI: Using configuration type 1
(XEN) PCI: Probing PCI hardware
(XEN) PCI: Probing PCI hardware (bus 00)
(XEN) PCI: Ignoring BAR0-3 of IDE controller 00:0f.1
(XEN) PCI: Discovered peer bus 01
(XEN) PCI: Discovered peer bus 02
(XEN) PCI->APIC IRQ transform: (B0,I1,P0) -> 20
(XEN) PCI->APIC IRQ transform: (B0,I15,P0) -> 7
(XEN) PCI->APIC IRQ transform: (B1,I1,P0) -> 16
(XEN) PCI->APIC IRQ transform: (B2,I1,P0) -> 17
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen-ELF header found:
''GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=2.0,VIRT_BASE=0xC0000000,LOADER=generic,PT_MODE_WRITABLE''
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Kernel image: 02800000->02bcbc8c
(XEN) Initrd image: 00000000->00000000
(XEN) Dom0 alloc.: 02c00000->05310000
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: c0100000->c04ffb44
(XEN) Init. ramdisk: c0500000->c0500000
(XEN) Phys-Mach map: c0500000->c0509c40
(XEN) Page tables: c050a000->c050d000
(XEN) Start info: c050d000->c050e000
(XEN) Boot stack: c050e000->c050f000
(XEN) TOTAL: c0000000->c0800000
(XEN) ENTRY ADDRESS: c0100000
(XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times
to switch input to Xen).
(XEN) Give DOM0 read access to all PCI devices
(XEN) (file=sched_bvt.c, line=367) Get domain 1 bvt mcu_adv=10, warpback=0,
warpvalue=0, warpl=2000000000, warpu=1000000000
(XEN) (file=sched_bvt.c, line=400) Get domain 1 bvt mcu_adv=10, warpback=0,
warpvalue=0, warpl=2000000000000000, warpu=1000000000000000
(XEN) (file=domain.c, line=250) Releasing task 1
(XEN) (file=sched_bvt.c, line=367) Get domain 2 bvt mcu_adv=10, warpback=0,
warpvalue=0, warpl=2000000000, warpu=1000000000
(XEN) (file=sched_bvt.c, line=400) Get domain 2 bvt mcu_adv=10, warpback=0,
warpvalue=0, warpl=2000000000000000, warpu=1000000000000000
(XEN) (file=domain.c, line=250) Releasing task 3
(XEN) (file=domain.c, line=250) Releasing task 2
(XEN) (file=sched_bvt.c, line=367) Get domain 4 bvt mcu_adv=10, warpback=0,
warpvalue=0, warpl=2000000000, warpu=1000000000
(XEN) (file=sched_bvt.c, line=400) Get domain 4 bvt mcu_adv=10, warpback=0,
warpvalue=0, warpl=2000000000000000, warpu=1000000000000000
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel
> > " Could you try updating to the latest version, with writable pagetables > " reenabled. Keir has spotted an incorrect check in the writable > > xen is faulting the same as the previous pull. > > objdump shows the fault address 0xfc52f5b4 in the same function as > the previous pull.The page faults that are getting logged may well turn out to be harmless. However, it would be interesting to see the call trace leading up to the page fault, so that we can see what''s causing it. Please could you add something akin to the following to the relevant printk: { extern void show_trace(unsigned long *esp); unsigned long *esp; __asm__ __volatile__ ("movl %%esp,%0" : "=r" (esp) : ); show_trace(esp); } It would be very useful to see the panic message coming out of the domain. If you leave an ''xm console'' connected to the domain there''s a good chance that you''ll capture it. Thanks, Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
" > " Could you try updating to the latest version, with writable pagetables " > " reenabled. Keir has spotted an incorrect check in the writable " > " > xen is faulting the same as the previous pull. I am now seeing a new behaviour from xen0. Some of them drop off the net for a minute or so and come back apparently ok, except for lots of console messages from swapper, shown below. I suppose it could be the bug that has been panicking xenU didn''t manage a panic but did corrupt the system.. This network delay effect is on some (but not all) of the xen0 e100 hosts. (The xenU panic I reported before has happened on all types of hosts.) Additionally the clocks are off on the xen0 hosts that show this symptom. They xen0 domains are all running ntp, so I''d guess this symptom is messing up ntp. xen0 is configured with the same amount of mem on all hosts. xen0 has plenty of swap space: $ swapon -s Filename Type Size Used Priority /dev/sda2 partition 987988 9316 -1 xen0 dmesg: swapper: page allocation failure. order:0, mode:0x20 [<c0138448>] __alloc_pages+0x2d7/0x347 [<c01384d7>] __get_free_pages+0x1f/0x3b [<c013b9e6>] kmem_getpages+0x22/0xdc [<c013c632>] cache_grow+0xb9/0x189 [<c013c86b>] cache_alloc_refill+0x169/0x20a [<c013cb0f>] kmem_cache_alloc+0x6c/0x70 [<c031e0da>] alloc_skb_from_cache+0x44/0xdc [<c010a557>] __dev_alloc_skb+0x23/0x39 [<c023e42a>] e100_intr+0x1b0/0x60b [<c031e170>] alloc_skb_from_cache+0xda/0xdc [<c010bc5a>] handle_IRQ_event+0x47/0x80 [<c010bfc2>] do_IRQ+0x93/0x132 [<c01092d0>] evtchn_do_upcall+0xa8/0x111 [<c010d93b>] hypervisor_callback+0x33/0x49 [<c0109226>] force_evtchn_callback+0xa/0xc [<c0118d2a>] try_to_wake_up+0xab/0xbd [<c01248d4>] process_timeout+0x0/0x9 [<c0118d5a>] wake_up_process+0x1e/0x22 [<c0124613>] run_timer_softirq+0xe3/0x1e6 [<c012066b>] __do_softirq+0x93/0x9c [<c01206b9>] do_softirq+0x45/0x47 [<c010c02f>] do_IRQ+0x100/0x132 [<c01092d0>] evtchn_do_upcall+0xa8/0x111 [<c010d93b>] hypervisor_callback+0x33/0x49 [<c0109bcc>] xen_cpu_idle+0x0/0x8c [<c0109bcc>] xen_cpu_idle+0x0/0x8c [<c010eebc>] cpu_idle+0x2a/0x35 [<c046269d>] start_kernel+0x1a0/0x1e9 [<c04622bc>] unknown_bootoption+0x0/0x149 ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > " > " Could you try updating to the latest version, with writable pagetables > " > " reenabled. Keir has spotted an incorrect check in the writable > " > > " > xen is faulting the same as the previous pull. > > I am now seeing a new behaviour from xen0. Some of them drop off the > net for a minute or so and come back apparently ok, except for lots of > console messages from swapper, shown below. I suppose it could be the > bug that has been panicking xenU didn''t manage a panic but did corrupt > the system..The ''error'' you reported isn''t actually fatal, it''s just a GFP_ATOMIC allocation failure from within the e100 interrupt handler. It''s pretty benign really, although it should be hard to trigger if you''ve given DOM0 a reasonable amount of RAM. It''s also weird that it has only just started to appear! I''ve rewritten the skb allocation routines, but the new ones shouldn''t be any more memory hungry. Perhaps it is worth checking the slab caches (e.g., cat /proc/slbinfo | grep xen-skb) and see whether the skb cache, or any other cache, is perhaps leaking?? I think it unlikely though. -- Keir ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, Sep 25, 2004 at 09:45:24PM -0400, David Becker wrote:> > " Could you try updating to the latest version, with writable pagetables > " reenabled. Keir has spotted an incorrect check in the writable > > xen is faulting the same as the previous pull.Ok, this is because you haven''t moved /lib/tls out of the way, at least I can reproduce the Page fault messages from __copy_to_user_ll if I move /lib/tls back. So these Page fault messages are "normal" if you run a debug build. christian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> On Sat, Sep 25, 2004 at 09:45:24PM -0400, David Becker wrote: > > > > " Could you try updating to the latest version, with writable pagetables > > " reenabled. Keir has spotted an incorrect check in the writable > > > > xen is faulting the same as the previous pull. > > Ok, this is because you haven''t moved /lib/tls out of the way, > at least I can reproduce the Page fault messages from __copy_to_user_ll > if I move /lib/tls back. So these Page fault messages are "normal" > if you run a debug build. > > christian >Yeah, everyone should move /lib/tls out of the way. I hoped that a large scary message, mostly in CAPITALS, with asterisks around it, and a 5-second boot delay would be enough to encourage action! :-) -- Keir ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
" " Yeah, everyone should move /lib/tls out of the way. I hoped that a " large scary message, mostly in CAPITALS, with asterisks around it, and " a 5-second boot delay would be enough to encourage action! :-) um, /lib/tls *IS* moved out of the way on all the domains. for some time now. The mv is hard coded in my upgrade script. Could those insns appear elsewhere? ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel