Hi,
I tried to use crash (http://people.redhat.com/~anderson/) with
x86_32 xen domain0 image, which is extracted from whole-machine dump
image made by kexec/kdump.
''kmem -i'' cause some error, but other almost all commands I
tried seem
to work well.
This may be some helps for xen development and troubleshooting.
regards,
---------------------------------------------------------------------------
(... get vmcore with kexec/kdump ...)
$ ./dom0cut_x86_32.py -txen -f -dvmcore -odom0core -xxen-syms -vvmlinux-2.6.16
$ crash vmlinux dom0core
crash 4.0-2.24
Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005 Fujitsu Limited
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: vmlinux
DUMPFILE: dom0core
CPUS: 2
DATE: Wed May 24 18:52:04 2006
UPTIME: 00:04:32
LOAD AVERAGE: 0.46, 0.10, 0.03
TASKS: 26
NODENAME: (none)
RELEASE: 2.6.16.13-xen
VERSION: #3 SMP Wed May 24 10:08:29 JST 2006
MACHINE: i686 (866 Mhz)
MEMORY: 887.4 MB
PANIC: "SysRq : Trigger a crashdump"
PID: 163
COMMAND: "echo"
TASK: c024e050 [THREAD_INFO: f3f0e000]
CPU: 0
STATE: TASK_RUNNING (SYSRQ)
crash> bt
PID: 163 TASK: c024e050 CPU: 0 COMMAND: "echo"
#0 [f3f0ffbc] ret_from_fork at c01050a8
#1 [f3f0ffb8] system_call at c01051d2
EAX: 00000004 EBX: 00000001 ECX: b7e8c000 EDX: 00000002
DS: 007b ESI: 00000002 ES: 007b EDI: 00000000
SS: 007b ESP: bf8ad750 EBP: bf8ad774
CS: 0073 EIP: b7f35a88 ERR: 00000004 EFLAGS: 00000246
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 225145 879.5 MB ----
FREE 216192 844.5 MB 96% of TOTAL MEM
USED 8953 35 MB 3% of TOTAL MEM
SHARED 216640 846.2 MB 96% of TOTAL MEM
kmem: invalid kernel virtual address: 0 type: "inode buffer"
---------------------------------------------------------------------------
--
Kazuo Moriwaka <moriwaka@valinux.co.jp>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Hi, Kazuo Great! I''ll try to use your tool and Horms patch. Best Regards, Akio Takebe>Hi, > >I tried to use crash (http://people.redhat.com/~anderson/) with >x86_32 xen domain0 image, which is extracted from whole-machine dump >image made by kexec/kdump. > >''kmem -i'' cause some error, but other almost all commands I tried seem >to work well. >This may be some helps for xen development and troubleshooting. > >regards, > >--------------------------------------------------------------------------- >(... get vmcore with kexec/kdump ...) >$ ./dom0cut_x86_32.py -txen -f -dvmcore -odom0core -xxen-syms -vvmlinux-2.6 >.16 >$ crash vmlinux dom0core > >crash 4.0-2.24 >Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc. >Copyright (C) 2004, 2005, 2006 IBM Corporation >Copyright (C) 1999-2006 Hewlett-Packard Co >Copyright (C) 2005 Fujitsu Limited >Copyright (C) 2005 NEC Corporation >Copyright (C) 1999, 2002 Silicon Graphics, Inc. >Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. >This program is free software, covered by the GNU General Public License, >and you are welcome to change it and/or distribute copies of it under >certain conditions. Enter "help copying" to see the conditions. >This program has absolutely no warranty. Enter "help warranty" for details. > >GNU gdb 6.1 >Copyright 2004 Free Software Foundation, Inc. >GDB is free software, covered by the GNU General Public License, and you are >welcome to change it and/or distribute copies of it under certain conditions. >Type "show copying" to see the conditions. >There is absolutely no warranty for GDB. Type "show warranty" for details. >This GDB was configured as "i686-pc-linux-gnu"... > > > KERNEL: vmlinux > DUMPFILE: dom0core > CPUS: 2 > DATE: Wed May 24 18:52:04 2006 > UPTIME: 00:04:32 >LOAD AVERAGE: 0.46, 0.10, 0.03 > TASKS: 26 > NODENAME: (none) > RELEASE: 2.6.16.13-xen > VERSION: #3 SMP Wed May 24 10:08:29 JST 2006 > MACHINE: i686 (866 Mhz) > MEMORY: 887.4 MB > PANIC: "SysRq : Trigger a crashdump" > PID: 163 > COMMAND: "echo" > TASK: c024e050 [THREAD_INFO: f3f0e000] > CPU: 0 > STATE: TASK_RUNNING (SYSRQ) > >crash> bt >PID: 163 TASK: c024e050 CPU: 0 COMMAND: "echo" > #0 [f3f0ffbc] ret_from_fork at c01050a8 > #1 [f3f0ffb8] system_call at c01051d2 > EAX: 00000004 EBX: 00000001 ECX: b7e8c000 EDX: 00000002 > DS: 007b ESI: 00000002 ES: 007b EDI: 00000000 > SS: 007b ESP: bf8ad750 EBP: bf8ad774 > CS: 0073 EIP: b7f35a88 ERR: 00000004 EFLAGS: 00000246 >crash> kmem -i > PAGES TOTAL PERCENTAGE > TOTAL MEM 225145 879.5 MB ---- > FREE 216192 844.5 MB 96% of TOTAL MEM > USED 8953 35 MB 3% of TOTAL MEM > SHARED 216640 846.2 MB 96% of TOTAL MEM >kmem: invalid kernel virtual address: 0 type: "inode buffer" >--------------------------------------------------------------------------- >-- >Kazuo Moriwaka <moriwaka@valinux.co.jp>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kazuo Moriwaka wrote:> Hi, > > I tried to use crash (http://people.redhat.com/~anderson/) with > x86_32 xen domain0 image, which is extracted from whole-machine dump > image made by kexec/kdump. > > ''kmem -i'' cause some error, but other almost all commands I tried seem > to work well. > This may be some helps for xen development and troubleshooting. > > regards, >Hello Kazuo, Would it be possible for you to make your vmlinux and dom0core files available for me to work with? Can you put them someplace where I can download them? (Contact me off-list with the particulars...) Thanks, Dave Anderson> > --------------------------------------------------------------------------- > (... get vmcore with kexec/kdump ...) > $ ./dom0cut_x86_32.py -txen -f -dvmcore -odom0core -xxen-syms -vvmlinux-2.6.16 > $ crash vmlinux dom0core > > crash 4.0-2.24 > Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005 Fujitsu Limited > Copyright (C) 2005 NEC Corporation > Copyright (C) 1999, 2002 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for details. > > GNU gdb 6.1 > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i686-pc-linux-gnu"... > > KERNEL: vmlinux > DUMPFILE: dom0core > CPUS: 2 > DATE: Wed May 24 18:52:04 2006 > UPTIME: 00:04:32 > LOAD AVERAGE: 0.46, 0.10, 0.03 > TASKS: 26 > NODENAME: (none) > RELEASE: 2.6.16.13-xen > VERSION: #3 SMP Wed May 24 10:08:29 JST 2006 > MACHINE: i686 (866 Mhz) > MEMORY: 887.4 MB > PANIC: "SysRq : Trigger a crashdump" > PID: 163 > COMMAND: "echo" > TASK: c024e050 [THREAD_INFO: f3f0e000] > CPU: 0 > STATE: TASK_RUNNING (SYSRQ) > > crash> bt > PID: 163 TASK: c024e050 CPU: 0 COMMAND: "echo" > #0 [f3f0ffbc] ret_from_fork at c01050a8 > #1 [f3f0ffb8] system_call at c01051d2 > EAX: 00000004 EBX: 00000001 ECX: b7e8c000 EDX: 00000002 > DS: 007b ESI: 00000002 ES: 007b EDI: 00000000 > SS: 007b ESP: bf8ad750 EBP: bf8ad774 > CS: 0073 EIP: b7f35a88 ERR: 00000004 EFLAGS: 00000246 > crash> kmem -i > PAGES TOTAL PERCENTAGE > TOTAL MEM 225145 879.5 MB ---- > FREE 216192 844.5 MB 96% of TOTAL MEM > USED 8953 35 MB 3% of TOTAL MEM > SHARED 216640 846.2 MB 96% of TOTAL MEM > kmem: invalid kernel virtual address: 0 type: "inode buffer" > --------------------------------------------------------------------------- > -- > Kazuo Moriwaka <moriwaka@valinux.co.jp>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, May 25, 2006 at 09:12:18AM -0400, Dave Anderson wrote:> Kazuo Moriwaka wrote: > > > Hi, > > > > I tried to use crash (http://people.redhat.com/~anderson/) with > > x86_32 xen domain0 image, which is extracted from whole-machine dump > > image made by kexec/kdump. > > > > ''kmem -i'' cause some error, but other almost all commands I tried seem > > to work well. > > This may be some helps for xen development and troubleshooting. > > > > regards, > > > > Hello Kazuo, > > Would it be possible for you to make your vmlinux and dom0core files > available for me to work with? Can you put them someplace where > I can download them? (Contact me off-list with the particulars...)Hi Dave, I haven''t seen Moriwaka-san yet this morning, so I''m not sure if he has sent you the data or not. There is nothing remotely private about it so I''ve put it up (for a short time only) for anyone who is interested to grab. (That is assuming that he is talking about the dump I made for him a few days ago). See: http://packages.vergenet.net/tmp/ -- Horms http://www.vergenet.net/~horms/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kazuo Moriwaka
2006-May-26 06:40 UTC
Re: [Xen-devel] Re: crash with Xen dom0 image from kdump
Hello Dave, From: Dave Anderson <anderson@redhat.com> Subject: Re: [Xen-devel] Re: crash with Xen dom0 image from kdump Date: Thu, 25 May 2006 15:14:54 -0400> Kazuo Moriwaka wrote: > > > Hello Dave, > > > > Thank you very much for your work. Your support for xendump is great > > help for us. > > I''ll put dom0core, vmlinux, System.map at following. > > http://people.valinux.co.jp/~moriwaka/dom0core/dom0core.tar.bz2 > > Please tell me if you have some trouble or need any more information. > > > > Hello Kazuo, > > This is very interesting. I was confused at first, but now > I see that you have created a xendump-like dumpfile from a > kexec/kdump generated dumpfile. This makes kind of a "hybrid" > dumpfile, that the crash utility is a bit confused by. In other > words the dumpfile was generated by kexec/kdump, but the crash > utility "thinks" it''s a xendump. For that reason, the proper > starting backtrace hooks are not searched for. > > For example, here''s "bt" on the panicking process: > > crash> bt > PID: 163 TASK: c024e050 CPU: 0 COMMAND: "echo" > #0 [f3f0ffbc] ret_from_fork at c01050a8 > #1 [f3f0ffb8] system_call at c01051d2 > EAX: 00000004 EBX: 00000001 ECX: b7e8c000 EDX: 00000002 > DS: 007b ESI: 00000002 ES: 007b EDI: 00000000 > SS: 007b ESP: bf8ad750 EBP: bf8ad774 > CS: 0073 EIP: b7f35a88 ERR: 00000004 EFLAGS: 00000246 > crash> > > That obviously not right -- so checking the process stack for > all possible backtrace starting points, we see the kexec > activity: > > crash> bt -T > PID: 163 TASK: c024e050 CPU: 0 COMMAND: "echo" > [f3f0fc1c] get_page_from_freelist at c014a52a > [f3f0fc40] monotonic_clock at c0109b50 > [f3f0fc58] scheduler_tick at c011b4ea > [f3f0fc64] input_init at c02537d0 > [f3f0fc78] __kmap_atomic at c0115538 > [f3f0fc98] get_page_from_freelist at c014a6e8 > [f3f0fc9c] huft_build at c0244000 > [f3f0fca4] __kmap_atomic at c01155be > [f3f0fcc4] get_page_from_freelist at c014a6e8 > [f3f0fd00] __alloc_bootmem_core at c024e050 > [f3f0fd0c] __alloc_bootmem_core at c024e050 > [f3f0fd18] __alloc_pages at c014a7a3 > [f3f0fd58] __handle_mm_fault at c0154c47 > [f3f0fd7c] __d_lookup at c01814c5 > [f3f0fd98] kcons_write_dom0 at c01ddee1 > [f3f0fda4] vsnprintf at c01b5fa9 > [f3f0fdb8] __call_console_drivers at c011e69e > [f3f0fdd4] release_console_sem at c011e9a9 > [f3f0fdfc] vprintk at c011f586 > [f3f0fe20] get_page_from_freelist at c014a6e8 > [f3f0fe40] __kmap_atomic at c01155be > [f3f0fe44] kcons_write_dom0 at c01ddee1 > [f3f0fe6c] printk at c011f69b > [f3f0fe7c] machine_crash_shutdown at c01ddb97 > [f3f0fea8] vprintk at c011f586 > [f3f0febc] sys_kexec_load at c014007b > [f3f0fed4] crash_kexec at c013f525 > [f3f0ff08] crash_kexec at c013f57a > [f3f0ff28] __handle_sysrq at c01d05b6 > [f3f0ff54] write_sysrq_trigger at c019b494 > [f3f0ff6c] vfs_write at c01676a1 > [f3f0ff90] sys_write at c016801b > [f3f0ffb8] syscall_call at c01051d9 > [f3f0ffdc] L6 at c010007b > crash> > > And taking the "crash_kexec" stack hook, the backtrace can > be manually resurrected: > > crash> bt -I c013f525 -S f3f0fed4 > PID: 163 TASK: c024e050 CPU: 0 COMMAND: "echo" > #0 [f3f0fed4] crash_kexec at c013f525 > #1 [f3f0ff28] __handle_sysrq at c01d05b4 > #2 [f3f0ff54] write_sysrq_trigger at c019b48f > #3 [f3f0ff6c] vfs_write at c016769f > #4 [f3f0ff90] sys_write at c0168016 > #5 [f3f0ffb8] system_call at c01051d2 > EAX: 00000004 EBX: 00000001 ECX: b7e8c000 EDX: 00000002 > DS: 007b ESI: 00000002 ES: 007b EDI: 00000000 > SS: 007b ESP: bf8ad750 EBP: bf8ad774 > CS: 0073 EIP: b7f35a88 ERR: 00000004 EFLAGS: 00000246 > crash> > > Given that the crash utility supports the kexec/kdump > dumpfile format, I am curious what happens when you run > crash against that vmcore file? Does it even come > close to working?I tried with Horms''s dump, it stopped ''cause it cannot find linux_banner. I''ll attach the log with debug option. Do you have done any works to support xen itself?> In any case, this "extraction tool" that you have written > to create a xendump-like file, will you be making that > available as a xen utility?Sure, I will.> I will be going away tomorrow for this coming holiday > weekend, so given our timezone differences, I will not > be able to reply until next Tuesday at the earliest. > > Also, I''ve sent you an invitation to the crash-utility > mailing list. This kind of discussion would be very > useful there. (And please feel free to invite others > at valinux as well...) > > Thanks very much, > Dave > > > > > > I note my dom0 image extract script is patching up VCPU context header > > from dom0 VCPU context infos(it''s base, but they are old) in xen, > > ELF core header(latest user registers), and CR3 for dom0 linux kernel > > (it''s work around for gdbserver-xen). > > If crash needs more faithful dump image headers, I''ll try to make it. > > > > thanks, > > > > On 5/25/06, Dave Anderson <anderson@redhat.com> wrote: > > > Kazuo Moriwaka wrote: > > > > > > > Hi, > > > > > > > > I tried to use crash (http://people.redhat.com/~anderson/) with > > > > x86_32 xen domain0 image, which is extracted from whole-machine dump > > > > image made by kexec/kdump. > > > > > > > > ''kmem -i'' cause some error, but other almost all commands I tried seem > > > > to work well. > > > > This may be some helps for xen development and troubleshooting. > > > > > > > > regards, > > > > > > > > > > Hello Kazuo, > > > > > > Would it be possible for you to make your vmlinux and dom0core files > > > available for me to work with? Can you put them someplace where > > > I can download them? (Contact me off-list with the particulars...) > > > > > > Thanks, > > > Dave Anderson > > > > > > > > > > > > > > --------------------------------------------------------------------------- > > > > (... get vmcore with kexec/kdump ...) > > > > $ ./dom0cut_x86_32.py -txen -f -dvmcore -odom0core -xxen-syms -vvmlinux-2.6.16 > > > > $ crash vmlinux dom0core > > > > > > > > crash 4.0-2.24 > > > > Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc. > > > > Copyright (C) 2004, 2005, 2006 IBM Corporation > > > > Copyright (C) 1999-2006 Hewlett-Packard Co > > > > Copyright (C) 2005 Fujitsu Limited > > > > Copyright (C) 2005 NEC Corporation > > > > Copyright (C) 1999, 2002 Silicon Graphics, Inc. > > > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > > > This program is free software, covered by the GNU General Public License, > > > > and you are welcome to change it and/or distribute copies of it under > > > > certain conditions. Enter "help copying" to see the conditions. > > > > This program has absolutely no warranty. Enter "help warranty" for details. > > > > > > > > GNU gdb 6.1 > > > > Copyright 2004 Free Software Foundation, Inc. > > > > GDB is free software, covered by the GNU General Public License, and you are > > > > welcome to change it and/or distribute copies of it under certain conditions. > > > > Type "show copying" to see the conditions. > > > > There is absolutely no warranty for GDB. Type "show warranty" for details. > > > > This GDB was configured as "i686-pc-linux-gnu"... > > > > > > > > KERNEL: vmlinux > > > > DUMPFILE: dom0core > > > > CPUS: 2 > > > > DATE: Wed May 24 18:52:04 2006 > > > > UPTIME: 00:04:32 > > > > LOAD AVERAGE: 0.46, 0.10, 0.03 > > > > TASKS: 26 > > > > NODENAME: (none) > > > > RELEASE: 2.6.16.13-xen > > > > VERSION: #3 SMP Wed May 24 10:08:29 JST 2006 > > > > MACHINE: i686 (866 Mhz) > > > > MEMORY: 887.4 MB > > > > PANIC: "SysRq : Trigger a crashdump" > > > > PID: 163 > > > > COMMAND: "echo" > > > > TASK: c024e050 [THREAD_INFO: f3f0e000] > > > > CPU: 0 > > > > STATE: TASK_RUNNING (SYSRQ) > > > > > > > > crash> bt > > > > PID: 163 TASK: c024e050 CPU: 0 COMMAND: "echo" > > > > #0 [f3f0ffbc] ret_from_fork at c01050a8 > > > > #1 [f3f0ffb8] system_call at c01051d2 > > > > EAX: 00000004 EBX: 00000001 ECX: b7e8c000 EDX: 00000002 > > > > DS: 007b ESI: 00000002 ES: 007b EDI: 00000000 > > > > SS: 007b ESP: bf8ad750 EBP: bf8ad774 > > > > CS: 0073 EIP: b7f35a88 ERR: 00000004 EFLAGS: 00000246 > > > > crash> kmem -i > > > > PAGES TOTAL PERCENTAGE > > > > TOTAL MEM 225145 879.5 MB ---- > > > > FREE 216192 844.5 MB 96% of TOTAL MEM > > > > USED 8953 35 MB 3% of TOTAL MEM > > > > SHARED 216640 846.2 MB 96% of TOTAL MEM > > > > kmem: invalid kernel virtual address: 0 type: "inode buffer" > > > > --------------------------------------------------------------------------- > > > > -- > > > > Kazuo Moriwaka <moriwaka@valinux.co.jp> > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > > > > > > -- > > Kazuo Moriwaka_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel