Apparao, Padmashree K
2008-Mar-06 23:21 UTC
[Xen-users] FW: oprofile 0.9.3 xen symbols incorrect
Hi I am using oprofile 0.9.3 on xen cs 16540 on an Intel system. When I look at the top "hot" functions, I see p2m_change_type being one of the top function in xen-syms. This function is only in the svm (AMD) code and should not appear on an Intel system. I see that this function is not being clled at all when I am running my apps as I have put printk in the functions and they do not show up anywhere in the dmesg. Is it possible that oprofile is picking up the symbols from elsewhere? The /root/.profile/daemonrc file shows the correct xen-syms file, so obviously it is not using that.. (I am profiling a HVM domain using passive-domains in the command line for opcontrol) Thanks -Padma _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Santos, Jose Renato G
2008-Mar-07 05:22 UTC
[Xen-users] RE: oprofile 0.9.3 xen symbols incorrect
Padma,
I received some reports in the past that Xenoprof was generating incorrect
symbols for HVM guests on Intel processors.
Here is a patch provided by Andrew Gallagher that should fix the problem but I
did not have had a chance to test it yet. It has been sitting on my todo list
for a long time.
Could you please check if this fix the problem and let me know.
Thanks
Renato
diff -r 0164d924ceba xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c Wed Feb 13 10:43:13 2008 +0000
+++ b/xen/arch/x86/hvm/vmx/vmx.c Thu Mar 06 21:08:46 2008 -0800
@@ -2511,6 +2511,7 @@ asmlinkage void vmx_vmexit_handler(struc
* (2) NMI
*/
unsigned int intr_info, vector;
+ int saved_eip;
intr_info = __vmread(VM_EXIT_INTR_INFO);
BUG_ON(!(intr_info & INTR_INFO_VALID_MASK));
@@ -2565,7 +2566,10 @@ asmlinkage void vmx_vmexit_handler(struc
(X86_EVENTTYPE_NMI << 8) )
goto exit_and_crash;
HVMTRACE_0D(NMI, v);
+ saved_eip = regs->eip;
+ regs->eip = __vmread(GUEST_RIP);
do_nmi(regs); /* Real NMI, vector 2: normal processing. */
+ regs->eip = saved_eip;
break;
case TRAP_machine_check:
HVMTRACE_0D(MCE, v);
________________________________
From: Apparao, Padmashree K [mailto:padmashree.k.apparao@intel.com]
Sent: Thursday, March 06, 2008 3:21 PM
To: oprofile-list@lists.sourceforge.net; Santos, Jose Renato G;
xen-users@lists.xensource.com
Subject: FW: oprofile 0.9.3 xen symbols incorrect
Hi
I am using oprofile 0.9.3 on xen cs 16540 on an Intel system.
When I look at the top "hot" functions, I see p2m_change_type being
one of the top function in xen-syms. This function is only in the svm (AMD) code
and should not appear on an Intel system. I see that this function is not being
clled at all when I am running my apps as I have put printk in the functions and
they do not show up anywhere in the dmesg.
Is it possible that oprofile is picking up the symbols from elsewhere? The
/root/.profile/daemonrc file shows the correct xen-syms file, so obviously it is
not using that..
(I am profiling a HVM domain using passive-domains in the command line for
opcontrol)
Thanks
-Padma
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Apparao, Padmashree K
2008-Mar-07 06:05 UTC
[Xen-users] RE: oprofile 0.9.3 xen symbols incorrect
Renato,
OOPs that patch kills the VM that is running :-(
-Padma
________________________________
From: Santos, Jose Renato G [mailto:joserenato.santos@hp.com]
Sent: Thursday, March 06, 2008 9:22 PM
To: Apparao, Padmashree K; oprofile-list@lists.sourceforge.net;
xen-users@lists.xensource.com
Subject: RE: oprofile 0.9.3 xen symbols incorrect
Padma,
I received some reports in the past that Xenoprof was generating
incorrect symbols for HVM guests on Intel processors.
Here is a patch provided by Andrew Gallagher that should fix the problem
but I did not have had a chance to test it yet. It has been sitting on
my todo list for a long time.
Could you please check if this fix the problem and let me know.
Thanks
Renato
diff -r 0164d924ceba xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c Wed Feb 13 10:43:13 2008 +0000
+++ b/xen/arch/x86/hvm/vmx/vmx.c Thu Mar 06 21:08:46 2008 -0800
@@ -2511,6 +2511,7 @@ asmlinkage void vmx_vmexit_handler(struc
* (2) NMI
*/
unsigned int intr_info, vector;
+ int saved_eip;
intr_info = __vmread(VM_EXIT_INTR_INFO);
BUG_ON(!(intr_info & INTR_INFO_VALID_MASK));
@@ -2565,7 +2566,10 @@ asmlinkage void vmx_vmexit_handler(struc
(X86_EVENTTYPE_NMI << 8) )
goto exit_and_crash;
HVMTRACE_0D(NMI, v);
+ saved_eip = regs->eip;
+ regs->eip = __vmread(GUEST_RIP);
do_nmi(regs); /* Real NMI, vector 2: normal processing. */
+ regs->eip = saved_eip;
break;
case TRAP_machine_check:
HVMTRACE_0D(MCE, v);
________________________________
From: Apparao, Padmashree K
[mailto:padmashree.k.apparao@intel.com]
Sent: Thursday, March 06, 2008 3:21 PM
To: oprofile-list@lists.sourceforge.net; Santos, Jose Renato G;
xen-users@lists.xensource.com
Subject: FW: oprofile 0.9.3 xen symbols incorrect
Hi
I am using oprofile 0.9.3 on xen cs 16540 on an Intel system.
When I look at the top "hot" functions, I see p2m_change_type
being one of the top function in xen-syms. This function is only in the
svm (AMD) code and should not appear on an Intel system. I see that this
function is not being clled at all when I am running my apps as I have
put printk in the functions and they do not show up anywhere in the
dmesg.
Is it possible that oprofile is picking up the symbols from
elsewhere? The /root/.profile/daemonrc file shows the correct xen-syms
file, so obviously it is not using that..
(I am profiling a HVM domain using passive-domains in the
command line for opcontrol)
Thanks
-Padma
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Santos, Jose Renato G
2008-Mar-07 16:38 UTC
[Xen-users] RE: oprofile 0.9.3 xen symbols incorrect
Padma,
You will have to debug this some more and provide more details.
I suggest that you remove the patch and do some more tests on the original
system.
Try generating a more detailed opreport including the EIP values in addition to
the symbols, and then check what symbol the suspicious EIP corresponds to in the
xen image.
Unfortunately I will not have much time to test this myself
Good luck
Renato
________________________________
From: Apparao, Padmashree K [mailto:padmashree.k.apparao@intel.com]
Sent: Thursday, March 06, 2008 10:05 PM
To: Santos, Jose Renato G; oprofile-list@lists.sourceforge.net;
xen-users@lists.xensource.com
Subject: RE: oprofile 0.9.3 xen symbols incorrect
Renato,
OOPs that patch kills the VM that is running :(
-Padma
________________________________
From: Santos, Jose Renato G [mailto:joserenato.santos@hp.com]
Sent: Thursday, March 06, 2008 9:22 PM
To: Apparao, Padmashree K; oprofile-list@lists.sourceforge.net;
xen-users@lists.xensource.com
Subject: RE: oprofile 0.9.3 xen symbols incorrect
Padma,
I received some reports in the past that Xenoprof was generating incorrect
symbols for HVM guests on Intel processors.
Here is a patch provided by Andrew Gallagher that should fix the problem but I
did not have had a chance to test it yet. It has been sitting on my todo list
for a long time.
Could you please check if this fix the problem and let me know.
Thanks
Renato
diff -r 0164d924ceba xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c Wed Feb 13 10:43:13 2008 +0000
+++ b/xen/arch/x86/hvm/vmx/vmx.c Thu Mar 06 21:08:46 2008 -0800
@@ -2511,6 +2511,7 @@ asmlinkage void vmx_vmexit_handler(struc
* (2) NMI
*/
unsigned int intr_info, vector;
+ int saved_eip;
intr_info = __vmread(VM_EXIT_INTR_INFO);
BUG_ON(!(intr_info & INTR_INFO_VALID_MASK));
@@ -2565,7 +2566,10 @@ asmlinkage void vmx_vmexit_handler(struc
(X86_EVENTTYPE_NMI << 8) )
goto exit_and_crash;
HVMTRACE_0D(NMI, v);
+ saved_eip = regs->eip;
+ regs->eip = __vmread(GUEST_RIP);
do_nmi(regs); /* Real NMI, vector 2: normal processing. */
+ regs->eip = saved_eip;
break;
case TRAP_machine_check:
HVMTRACE_0D(MCE, v);
________________________________
From: Apparao, Padmashree K [mailto:padmashree.k.apparao@intel.com]
Sent: Thursday, March 06, 2008 3:21 PM
To: oprofile-list@lists.sourceforge.net; Santos, Jose Renato G;
xen-users@lists.xensource.com
Subject: FW: oprofile 0.9.3 xen symbols incorrect
Hi
I am using oprofile 0.9.3 on xen cs 16540 on an Intel system.
When I look at the top "hot" functions, I see p2m_change_type being
one of the top function in xen-syms. This function is only in the svm (AMD) code
and should not appear on an Intel system. I see that this function is not being
clled at all when I am running my apps as I have put printk in the functions and
they do not show up anywhere in the dmesg.
Is it possible that oprofile is picking up the symbols from elsewhere? The
/root/.profile/daemonrc file shows the correct xen-syms file, so obviously it is
not using that..
(I am profiling a HVM domain using passive-domains in the command line for
opcontrol)
Thanks
-Padma
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Apparao, Padmashree K
2008-Mar-07 23:47 UTC
[Xen-users] RE: oprofile 0.9.3 xen symbols incorrect
Renato I did what you suggested. Opreport -l showed p2m_change_type () to be one of hot functions. This is an AMD specific function and should not appear when running on Intel platforms. So I did a gdb of xen-syms and the EIP for the p2m_change_type function shows that the actual EIP is iommu_domain_destroy p2m_change_type is 0xffff828c801771d0 ==> Iommu_domain_destroy (which again should NOT be the case) So there is something wrong with the way oprofile is mapping the xen symbols, The command I use is Opcontrol --passive-domains=1 -xen=/boot/xen-sysms -vmlinux=/boot/vmlinux -event=CPU_CLK_UNHALTED:1000000 -start Can you please help in determining why oprofile is getting the symbols wrong? Thanks -Padma ________________________________ From: Santos, Jose Renato G [mailto:joserenato.santos@hp.com] Sent: Friday, March 07, 2008 8:39 AM To: Apparao, Padmashree K; oprofile-list@lists.sourceforge.net; xen-users@lists.xensource.com Subject: RE: oprofile 0.9.3 xen symbols incorrect Padma, You will have to debug this some more and provide more details. I suggest that you remove the patch and do some more tests on the original system. Try generating a more detailed opreport including the EIP values in addition to the symbols, and then check what symbol the suspicious EIP corresponds to in the xen image. Unfortunately I will not have much time to test this myself Good luck Renato ________________________________ From: Apparao, Padmashree K [mailto:padmashree.k.apparao@intel.com] Sent: Thursday, March 06, 2008 10:05 PM To: Santos, Jose Renato G; oprofile-list@lists.sourceforge.net; xen-users@lists.xensource.com Subject: RE: oprofile 0.9.3 xen symbols incorrect Renato, OOPs that patch kills the VM that is running :-( -Padma ________________________________ From: Santos, Jose Renato G [mailto:joserenato.santos@hp.com] Sent: Thursday, March 06, 2008 9:22 PM To: Apparao, Padmashree K; oprofile-list@lists.sourceforge.net; xen-users@lists.xensource.com Subject: RE: oprofile 0.9.3 xen symbols incorrect Padma, I received some reports in the past that Xenoprof was generating incorrect symbols for HVM guests on Intel processors. Here is a patch provided by Andrew Gallagher that should fix the problem but I did not have had a chance to test it yet. It has been sitting on my todo list for a long time. Could you please check if this fix the problem and let me know. Thanks Renato diff -r 0164d924ceba xen/arch/x86/hvm/vmx/vmx.c --- a/xen/arch/x86/hvm/vmx/vmx.c Wed Feb 13 10:43:13 2008 +0000 +++ b/xen/arch/x86/hvm/vmx/vmx.c Thu Mar 06 21:08:46 2008 -0800 @@ -2511,6 +2511,7 @@ asmlinkage void vmx_vmexit_handler(struc * (2) NMI */ unsigned int intr_info, vector; + int saved_eip; intr_info = __vmread(VM_EXIT_INTR_INFO); BUG_ON(!(intr_info & INTR_INFO_VALID_MASK)); @@ -2565,7 +2566,10 @@ asmlinkage void vmx_vmexit_handler(struc (X86_EVENTTYPE_NMI << 8) ) goto exit_and_crash; HVMTRACE_0D(NMI, v); + saved_eip = regs->eip; + regs->eip = __vmread(GUEST_RIP); do_nmi(regs); /* Real NMI, vector 2: normal processing. */ + regs->eip = saved_eip; break; case TRAP_machine_check: HVMTRACE_0D(MCE, v); ________________________________ From: Apparao, Padmashree K [mailto:padmashree.k.apparao@intel.com] Sent: Thursday, March 06, 2008 3:21 PM To: oprofile-list@lists.sourceforge.net; Santos, Jose Renato G; xen-users@lists.xensource.com Subject: FW: oprofile 0.9.3 xen symbols incorrect Hi I am using oprofile 0.9.3 on xen cs 16540 on an Intel system. When I look at the top "hot" functions, I see p2m_change_type being one of the top function in xen-syms. This function is only in the svm (AMD) code and should not appear on an Intel system. I see that this function is not being clled at all when I am running my apps as I have put printk in the functions and they do not show up anywhere in the dmesg. Is it possible that oprofile is picking up the symbols from elsewhere? The /root/.profile/daemonrc file shows the correct xen-syms file, so obviously it is not using that.. (I am profiling a HVM domain using passive-domains in the command line for opcontrol) Thanks -Padma _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users