thr3ads.net - Xen devel - [Xen-devel] crash can''t analyze memory dumpfile of Xen [Sep 2008]

If this information is useful, please help other people find it:
Share via:

Yuji Shimada

2008-Sep-03 06:42 UTC

[Xen-devel] crash can''t analyze memory dumpfile of Xen

I used the following tools to debug Xen.

  xen-3.3-unstable(changeset 17917)
  kexec-tools 2.0.0
  makedumpfile-1.2.8
  crash 4.0-6.3

Extraction of the memory dumpfile was completed normally. 
And I executed the following command to analyze memory dumpfile of Xen.

    # crash /boot/xen-syms-3.3-unstable dumpfile.core

The following errors occurred.

-------------------------------------------
    crash: invalid kernel virtual address:0
    type:"note_buf_t or crash_note_t"
    WARNING: cannot fill note core.

    crash: cannot read elf note core.
-------------------------------------------

I would like to ask some comments why these errors occurred.

I did the following steps:

 1.Boot Xen with the following Dom0 option. 
     "crashkernel=128M@64M"
   And 2nd kernel is loaded by kexec.

 2.On Dom0 Linux, carry out crash dump.
     # echo c > /sys/sysrq-trigger
 
 3.On 2nd kernel, extract memory dumpfile.
     # makedumpfile -E -X /proc/vmcore dumpfile.core

 4.Reboot Xen and execute the following command.
   And the above errors occurred.
     # crash /boot/xen-syms-3.3-unstable dumpfile.core

I can read the same memory dumpfile by crash command with no error to
debug vmlinux.

Thanks,

--
Yuji Shimada

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Itsuro ODA

2008-Sep-03 08:46 UTC

head link

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Hi,

Maybe a structure expected by the crash command was lost from
xen-3.3.0. and the crash command should be fixed for xen-3.3.0.

What arch did you use ?

If you send me xen-syms-3.3-unstable and dumpfile.core
I will investigate more.

Thanks
Itusro Oda

Yuji Shimada said:> I used the following tools to debug Xen.
>
>   xen-3.3-unstable(changeset 17917)
>   kexec-tools 2.0.0
>   makedumpfile-1.2.8
>   crash 4.0-6.3
>
> Extraction of the memory dumpfile was completed normally.
> And I executed the following command to analyze memory dumpfile of Xen.
>
>     # crash /boot/xen-syms-3.3-unstable dumpfile.core
>
> The following errors occurred.
>
> -------------------------------------------
>     crash: invalid kernel virtual address:0
>     type:"note_buf_t or crash_note_t"
>     WARNING: cannot fill note core.
>
>     crash: cannot read elf note core.
> -------------------------------------------
>
> I would like to ask some comments why these errors occurred.
>
> I did the following steps:
>
>  1.Boot Xen with the following Dom0 option.
>      "crashkernel=128M@64M"
>    And 2nd kernel is loaded by kexec.
>
>  2.On Dom0 Linux, carry out crash dump.
>      # echo c > /sys/sysrq-trigger
>
>  3.On 2nd kernel, extract memory dumpfile.
>      # makedumpfile -E -X /proc/vmcore dumpfile.core
>
>  4.Reboot Xen and execute the following command.
>    And the above errors occurred.
>      # crash /boot/xen-syms-3.3-unstable dumpfile.core
>
> I can read the same memory dumpfile by crash command with no error to
> debug vmlinux.
>
> Thanks,
>
> --
> Yuji Shimada
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

-- 
Itsuro ODA <oda@valinux.co.jp>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Itsuro ODA

2008-Sep-05 04:42 UTC

head link

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Hi,

I recieved the dump file via FTP from Yuji and I can reproduced
the problem.

Hmm, it seems the format of the crash note section is not expected.
(and there is another problem; "jiffies" is lost.)
I will check more deeply.

Until the problem is fixed, try the following quick hack.
------------------------------------------------------------------
--- xen_hyper.c.org	2008-09-05 12:48:57.000000000 +0900
+++ xen_hyper.c	2008-09-05 13:32:28.000000000 +0900
@@ -150,7 +150,7 @@
 	 * Do some initialization.
 	 */
 #ifndef IA64
-	xen_hyper_dumpinfo_init();
+//	xen_hyper_dumpinfo_init(); /* XXX: should be fixed !! */
 #endif
 	xhmachdep->pcpu_init();
 	xen_hyper_domain_init();
@@ -1746,9 +1746,11 @@
 			tmp2 = (ulong)jiffies_64;
 			jiffies_64 = (ulonglong)(tmp2 - tmp1);
 		}
-	} else {
+	} else if (symbol_exists("jiffies")) {
 		get_symbol_data("jiffies", sizeof(long), &jiffies);
 		jiffies_64 = (ulonglong)jiffies;
+	} else {
+		jiffies_64 = 0; /* XXX: find alternative !! */
 	}
 
 	return jiffies_64;
------------------------------------------------------------------
(the "dumpinfo" sub command cannot be used.)

Thanks.
Itsuro Oda

On Thu, 04 Sep 2008 16:29:51 +0900
Yuji Shimada <shimada-yxb@necst.nec.co.jp> wrote:
> Hi ODA-san,
> 
> Thank you so much for your reply.
> 
> > What arch did you use ?
> 
> I used x86_64 arch.
> 
> > If you send me xen-syms-3.3-unstable and dumpfile.core
> > I will investigate more.
> 
> Please find "xen-syms-3.3-unstable", attached with the mail.
> If you need "dumpfile.core" to investigate this issue, please let
me know.
> 
> In such case, I think I should send you "dumpfile.core" stored on
> the disk by post, or upload it to your site. Because it is a huge size.
> 
> Thanks,
> 
> --
> Yuji Shimada
-- 
Itsuro ODA <oda@valinux.co.jp>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Itsuro ODA

2008-Oct-03 06:02 UTC

head link

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Hi,

I found the root cause of this problem is that the value of
"PERCPU_SHIFT"
was changed to 13 from 12.

The quick workaround is to apply the following patch to the crash command:
----------------------------------------------------------------------
--- xen_hyper_defs.h.org        2008-10-03 14:46:28.000000000 +0900
+++ xen_hyper_defs.h    2008-10-03 14:46:50.000000000 +0900
@@ -134,7 +134,7 @@
 #endif

 #if defined(X86) || defined(X86_64)
-#define XEN_HYPER_PERCPU_SHIFT 12
+#define XEN_HYPER_PERCPU_SHIFT 13
 #define xen_hyper_per_cpu(var, cpu)  \
        ((ulong)(var) + (((ulong)(cpu))<<XEN_HYPER_PERCPU_SHIFT))
 #elif defined(IA64)
------------------------------------------------------------------------

I need to think the backword compatibility. I wonder how to determine
the value of "PERCPU_SHIFT". The change of "PERCPU_SHIFT"
was made at
a certain point of xen-unstable before xen-3.3 release. The xen version
number (3.3) can''t use as key... I will consider more...

Thanks.
Itsuro Oda

On Fri, 05 Sep 2008 13:42:46 +0900
Itsuro ODA <oda@valinux.co.jp> wrote:
> Hi,
> 
> I recieved the dump file via FTP from Yuji and I can reproduced
> the problem.
> 
> Hmm, it seems the format of the crash note section is not expected.
> (and there is another problem; "jiffies" is lost.)
> I will check more deeply.
> 
> Until the problem is fixed, try the following quick hack.
> ------------------------------------------------------------------
> --- xen_hyper.c.org	2008-09-05 12:48:57.000000000 +0900
> +++ xen_hyper.c	2008-09-05 13:32:28.000000000 +0900
> @@ -150,7 +150,7 @@
>  	 * Do some initialization.
>  	 */
>  #ifndef IA64
> -	xen_hyper_dumpinfo_init();
> +//	xen_hyper_dumpinfo_init(); /* XXX: should be fixed !! */
>  #endif
>  	xhmachdep->pcpu_init();
>  	xen_hyper_domain_init();
> @@ -1746,9 +1746,11 @@
>  			tmp2 = (ulong)jiffies_64;
>  			jiffies_64 = (ulonglong)(tmp2 - tmp1);
>  		}
> -	} else {
> +	} else if (symbol_exists("jiffies")) {
>  		get_symbol_data("jiffies", sizeof(long), &jiffies);
>  		jiffies_64 = (ulonglong)jiffies;
> +	} else {
> +		jiffies_64 = 0; /* XXX: find alternative !! */
>  	}
>  
>  	return jiffies_64;
> ------------------------------------------------------------------
> (the "dumpinfo" sub command cannot be used.)
> 
> Thanks.
> Itsuro Oda
> 
> On Thu, 04 Sep 2008 16:29:51 +0900
> Yuji Shimada <shimada-yxb@necst.nec.co.jp> wrote:
> 
> > Hi ODA-san,
> > 
> > Thank you so much for your reply.
> > 
> > > What arch did you use ?
> > 
> > I used x86_64 arch.
> > 
> > > If you send me xen-syms-3.3-unstable and dumpfile.core
> > > I will investigate more.
> > 
> > Please find "xen-syms-3.3-unstable", attached with the mail.
> > If you need "dumpfile.core" to investigate this issue,
please let me know.
> > 
> > In such case, I think I should send you "dumpfile.core"
stored on
> > the disk by post, or upload it to your site. Because it is a huge
size.
> > 
> > Thanks,
> > 
> > --
> > Yuji Shimada
> 
> -- 
> Itsuro ODA <oda@valinux.co.jp>
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
-- 
Itsuro ODA <oda@valinux.co.jp>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2008-Oct-03 07:03 UTC

head link

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

On 3/10/08 07:02, "Itsuro ODA" <oda@valinux.co.jp> wrote:
> I need to think the backword compatibility. I wonder how to determine
> the value of "PERCPU_SHIFT". The change of
"PERCPU_SHIFT" was made at
> a certain point of xen-unstable before xen-3.3 release. The xen version
> number (3.3) can''t use as key... I will consider more...
Is the problem easier if you don''t try to support every random snapshot
of
xen-unstable? Why not constrain yourself to stable releases (or even random
snapshots of stable branches): then you could indeed just look for >=3.3?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Itsuro ODA

2008-Oct-03 07:21 UTC

head link

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Hi,

Yes, it is one option.
(If I can not find any good idea, I will select this.)

Thanks.
Itsuro Oda

On Fri, 03 Oct 2008 08:03:48 +0100
Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> On 3/10/08 07:02, "Itsuro ODA" <oda@valinux.co.jp> wrote:
> 
> > I need to think the backword compatibility. I wonder how to determine
> > the value of "PERCPU_SHIFT". The change of
"PERCPU_SHIFT" was made at
> > a certain point of xen-unstable before xen-3.3 release. The xen
version
> > number (3.3) can''t use as key... I will consider more...
> 
> Is the problem easier if you don''t try to support every random
snapshot of
> xen-unstable? Why not constrain yourself to stable releases (or even random
> snapshots of stable branches): then you could indeed just look for
>=3.3?
> 
>  -- Keir
> 
> 
-- 
Itsuro ODA <oda@valinux.co.jp>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Itsuro ODA

2008-Oct-06 03:08 UTC

head link

Re: [Crash-utility] Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Hi,

It is a good point. 
Thank you for your suggestion.

Itsuro Oda

On Fri, 03 Oct 2008 08:47:28 -0400
Dave Anderson <anderson@redhat.com> wrote:
> Itsuro ODA wrote:
> > Hi,
> > 
> > I found the root cause of this problem is that the value of
"PERCPU_SHIFT"
> > was changed to 13 from 12.
> > 
> > The quick workaround is to apply the following patch to the crash
command:
> > ----------------------------------------------------------------------
> > --- xen_hyper_defs.h.org        2008-10-03 14:46:28.000000000 +0900
> > +++ xen_hyper_defs.h    2008-10-03 14:46:50.000000000 +0900
> > @@ -134,7 +134,7 @@
> >  #endif
> > 
> >  #if defined(X86) || defined(X86_64)
> > -#define XEN_HYPER_PERCPU_SHIFT 12
> > +#define XEN_HYPER_PERCPU_SHIFT 13
> >  #define xen_hyper_per_cpu(var, cpu)  \
> >         ((ulong)(var) +
(((ulong)(cpu))<<XEN_HYPER_PERCPU_SHIFT))
> >  #elif defined(IA64)
> >
------------------------------------------------------------------------
> > 
> > I need to think the backword compatibility. I wonder how to determine
> > the value of "PERCPU_SHIFT". The change of
"PERCPU_SHIFT" was made at
> > a certain point of xen-unstable before xen-3.3 release. The xen
version
> > number (3.3) can''t use as key... I will consider more...
> 
>  From the crash utility perspective, and looking at the RHEL5 xen sources
> where these hypervisor definitions exist:
> 
>    #define PERCPU_SIZE (1UL << PERCPU_SHIFT)
> 
>    static void __init percpu_init_areas(void)
>    {
>        unsigned int i, data_size = __per_cpu_data_end - __per_cpu_start;
>        unsigned int first_unused;
> 
>        BUG_ON(data_size > PERCPU_SIZE);
> 
> during initialization you could calculate the difference between the
> __per_cpu_data_end and __per_cpu_start symbol values, and if it''s
more
> than the original 4k size (12), then it must be 13.
> 
> Dave
> 
> --
> Crash-utility mailing list
> Crash-utility@redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
-- 
Itsuro ODA <oda@valinux.co.jp>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - Sep 2008 - crash can''t analyze memory dumpfile of Xen

[Xen-devel] crash can''t analyze memory dumpfile of Xen

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen

Re: [Crash-utility] Re: [Xen-devel] crash can''t analyze memory dumpfile of Xen