Xen devel - Sep 2006 - [HVM][XENOPROFILE][RFC][0/3] xenoprofile HVM patches

Keir, 
The following 3 patches allow HVM (SVM and VT) guests to be passively
profiled using the very latest patches from Renato.  These patches apply
to 11470.
Renato''s patches:
http://xenoprof/sourceforge.net  oprofile-0.9.1-xen-r2.patch
And also a patch not posted FAIK
(oprofile-0.9.1-fix-hvm-addr-overlap.patch) which is in the attached
tar.bz2 file.

hvm_xenoprofile_1.patch -
The first patch is merely the same patch that I posted a few weeks ago.
The STGI code is moved from the exits.S files, and into the vmexit
handler.  The extra do_nmi() is removed allowing the host to handle the
NMI when the stgi instruction is executed.  This patch allows the second
patch to properly set "guest NMI" flag for the oprofile code to
determine proper rip/eip and mode.

hvm_xenoprofile_2.patch -
The second patch adds the actual xenoprofile fixes for SVM:

1) hvm function (svm_oprofile_get_eip())
This new hvm table function is called from the op_module_athlon.c code,
and returns the proper eip and "mode" for use in the
xenoprof_log_event() function.  

2) The vmexit handler code now checks the exitcode and if an NMI was
intercepted, then prior to the stgi(), a flag (VGCF_hvm_guest_nmi) is
set in the v->arch.guest_context.flags field.   The VGCF_hvm_guest_nmi
bit is checked during NMI processing via the hvm function callback. 

hvm_xenoprofile_3.patch -
The third patch adds the actual xenoprofile fixes for VT:
I checked (verified by Renato), that the VT traces look ok also.  There
is some code that I am unsure about in the vmx_oprofile_get_eip(): i.e
using the __vmread() to get eip might not be in the proper context, and
then secondly I am not sure how to determine the guest CPL level.

Overall, these patches seem to work fine for 32bit hypervisor, as well
as 64bit hypervisor with 32bit and 64bit HVM guests.  You do need to use
both of Renato''s oprofile patches on top of 0.9.1.  With another
oprofile release soon (this week?), I believe that some of the "r2"
patch will be incorporated into 0.9.2, but not sure which parts.

There certainly might be a better way to obtain the eip/mode, but since
the SVM code is not calling do_nmi() then the host context will handle
the NMI, and there must be some mechanism to properly setup the eip/mode
in the op_model code indirectly.  I am not sure if the VT code requires
the same approach since do_nmi() is called directly, and the eip/mode
could be passed directly to this function.

The attached tar.bz2 file contains some example logs, and some scripts
from Ray Bryant (AMD) for extracting and normalizing each of the
profiling buckets.  There is a brief README for using the scripts. 

We''d like to see this get into 3.0.3 if possible.  Renato has reviewed
the patches, but not actually tested these patches on his platforms.
Feedback appreciated,
Tom Woller




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Tom,

Thanks for the HVM patches.
This is good news

We are now able to profile HVM guests (both SVM and VT)(in passive mode)
using Tom''s patches combined with the attached new version of oprofile.
The attached patch (r3) combines the old oprofile patch (r2) with the
patch "oprofile-0.9.1-fix-hvm-addr-overlap.patch" that Tom mentioned
and
included in his post.
After Tom''s fixes are merged, this is the only patch that will be
needed.

The patch is for oprofile 0.9.1. Oprofile 0.9.2, which will be release
soon, will include XenOprofile support for active domains but it will
not support passive domains. I will generate a patch with passive domain
support for oprofile 0.9.2 when it is released.

I hope Tom patches are accepted and make into Xen 3.0.3

Thanks

Renato
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Woller, Thomas
> Sent: Friday, September 15, 2006 9:19 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] [HVM][XENOPROFILE][RFC][0/3] xenoprofile 
> HVM patches
> 
> Keir,
> The following 3 patches allow HVM (SVM and VT) guests to be 
> passively profiled using the very latest patches from Renato. 
>  These patches apply to 11470.
> Renato''s patches:
> http://xenoprof/sourceforge.net  oprofile-0.9.1-xen-r2.patch 
> And also a patch not posted FAIK
> (oprofile-0.9.1-fix-hvm-addr-overlap.patch) which is in the attached
> tar.bz2 file.
> 
> hvm_xenoprofile_1.patch -
> The first patch is merely the same patch that I posted a few 
> weeks ago.
> The STGI code is moved from the exits.S files, and into the 
> vmexit handler.  The extra do_nmi() is removed allowing the 
> host to handle the NMI when the stgi instruction is executed. 
>  This patch allows the second patch to properly set "guest 
> NMI" flag for the oprofile code to determine proper rip/eip and mode.
> 
> hvm_xenoprofile_2.patch -
> The second patch adds the actual xenoprofile fixes for SVM:
> 
> 1) hvm function (svm_oprofile_get_eip()) This new hvm table 
> function is called from the op_module_athlon.c code, and 
> returns the proper eip and "mode" for use in the
> xenoprof_log_event() function.  
> 
> 2) The vmexit handler code now checks the exitcode and if an 
> NMI was intercepted, then prior to the stgi(), a flag 
> (VGCF_hvm_guest_nmi) is
> set in the v->arch.guest_context.flags field.   The VGCF_hvm_guest_nmi
> bit is checked during NMI processing via the hvm function callback. 
> 
> hvm_xenoprofile_3.patch -
> The third patch adds the actual xenoprofile fixes for VT:
> I checked (verified by Renato), that the VT traces look ok 
> also.  There is some code that I am unsure about in the 
> vmx_oprofile_get_eip(): i.e using the __vmread() to get eip 
> might not be in the proper context, and then secondly I am 
> not sure how to determine the guest CPL level.
> 
> Overall, these patches seem to work fine for 32bit 
> hypervisor, as well as 64bit hypervisor with 32bit and 64bit 
> HVM guests.  You do need to use both of Renato''s oprofile 
> patches on top of 0.9.1.  With another oprofile release soon 
> (this week?), I believe that some of the "r2"
> patch will be incorporated into 0.9.2, but not sure which parts.
> 
> There certainly might be a better way to obtain the eip/mode, 
> but since the SVM code is not calling do_nmi() then the host 
> context will handle the NMI, and there must be some mechanism 
> to properly setup the eip/mode in the op_model code 
> indirectly.  I am not sure if the VT code requires the same 
> approach since do_nmi() is called directly, and the eip/mode 
> could be passed directly to this function.
> 
> The attached tar.bz2 file contains some example logs, and 
> some scripts from Ray Bryant (AMD) for extracting and 
> normalizing each of the profiling buckets.  There is a brief 
> README for using the scripts. 
> 
> We''d like to see this get into 3.0.3 if possible.  Renato has 
> reviewed the patches, but not actually tested these patches 
> on his platforms.
> Feedback appreciated,
> Tom Woller
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Santos, Jose Renato G wrote:
> Tom,
> 
> Thanks for the HVM patches.
> This is good news
> 
> We are now able to profile HVM guests (both SVM and VT)(in passive mode)
> using Tom''s patches combined with the attached new version of
oprofile.
> The attached patch (r3) combines the old oprofile patch (r2) with the
> patch "oprofile-0.9.1-fix-hvm-addr-overlap.patch" that Tom
mentioned and
> included in his post.
> After Tom''s fixes are merged, this is the only patch that will be
> needed.
> 
> The patch is for oprofile 0.9.1. Oprofile 0.9.2, which will be release
> soon, will include XenOprofile support for active domains but it will
> not support passive domains. I will generate a patch with passive domain
> support for oprofile 0.9.2 when it is released.
> 
> I hope Tom patches are accepted and make into Xen 3.0.3
> 
> Thanks
> 
> Renato
Hi all,

OProfile 0.9.2 was released this past Friday, Sept 15. It is available on the 
OProfile website:

http://oprofile.sourceforge.net/news/

-Will

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

A XenOprofile patch for Oprofile 0.9.2 is now available at
http://xenoprof.sourceforge.net.
Oprofile 0.9.2 already supports XenOprofile but it does not include
support for passive domain profiling.
This patch adds passive domain profiling support to Oprofile 0.9.2. 

This version is more recent than the one posted by William Cohen last
week. It includes the latest fixes to support fully vitrualized guests. 

This patch is equivalent to the latest patch for Oprofile 0.9.1
(oprofile-0.9.1-xen-r3.patch) ported to 0.9.2.

Renato

> -----Original Message-----
> From: William Cohen [mailto:wcohen@redhat.com] 
> Sent: Monday, September 18, 2006 11:29 AM
> To: Santos, Jose Renato G
> Cc: Woller, Thomas; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] [XENOPROLIFE] New oprofile patch for 
> XenOProfile
> 
> 
> Hi all,
> 
> OProfile 0.9.2 was released this past Friday, Sept 15. It is 
> available on the OProfile website:
> 
> http://oprofile.sourceforge.net/news/
> 
> -Will
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Renato,

Well, I''d postponed any work with xenoprof until we had a working 3.0.3
version plus the local patches that we are using (includes the xenoprof 
patches that Tom Woller et al have done).

So, I''m now working with an oprofile 0.9.2 plus your patch 
oprofile-0.9.2-xen.patch from the HP open source website.   The particular 
project we are working on requires us to obtain profiles for the HVM guest 
kernel.   Unfortunately, my first few tries at this have been unsuccessful (I 
get lots of "lost kernel sample" messages in the oprofiled.log and no 
reported samples for the guest kernel.  :-(  ).

So I guess my guestion to you is, do you have access to a VT or AMD V enabled 
platform that you can test stuff on?   (I''d like to find a collegue to
work
with on this problem if at all possible.)   I have dedicated VT and AMD 
platforms for my use, so if you have a VT platorm, we can get stuff to work 
correctly there first and then I can work with Tom et al to get the AMD V 
versions to work.

-- 
Ray Bryant
AMD Performance Labs                   Austin, Tx
512-602-0038 (o)                 512-507-7807 (c)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ray,

I have an AMD machine in my Lab which I believe has SVM support.
I can use that if needed, but I will need sometime to get it ready and
running Xen.
I thought Tom has tested oprofile with both AMD and Intel HVM guests and
that things were working fine.
Could you please sync up with Tom and make sure it is not a problem with
your configuration.
Tom can you confirm you were able to successfully run XenOprofile for
HVM guests?
If needed I can try to reproduce your problem here when I have my AMD
machine up.

Regards

Renato 
> -----Original Message-----
> From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com] 
> Sent: Tuesday, November 07, 2006 9:25 AM
> To: xen-devel@lists.xensource.com
> Cc: Santos, Jose Renato G; Woller, Thomas
> Subject: Re: [Xen-devel] [XENOPROLIFE] New oprofile patch for 
> XenOProfile
> 
> Renato,
> 
> Well, I''d postponed any work with xenoprof until we had a 
> working 3.0.3 version plus the local patches that we are 
> using (includes the xenoprof patches that Tom Woller et al have done).
> 
> So, I''m now working with an oprofile 0.9.2 plus your patch 
> oprofile-0.9.2-xen.patch from the HP open source website.   
> The particular 
> project we are working on requires us to obtain profiles for 
> the HVM guest 
> kernel.   Unfortunately, my first few tries at this have been 
> unsuccessful (I 
> get lots of "lost kernel sample" messages in the 
> oprofiled.log and no reported samples for the guest kernel.  :-(  ).
> 
> So I guess my guestion to you is, do you have access to a VT 
> or AMD V enabled 
> platform that you can test stuff on?   (I''d like to find a 
> collegue to work 
> with on this problem if at all possible.)   I have dedicated 
> VT and AMD 
> platforms for my use, so if you have a VT platorm, we can get 
> stuff to work correctly there first and then I can work with 
> Tom et al to get the AMD V versions to work.
> 
> --
> Ray Bryant
> AMD Performance Labs                   Austin, Tx
> 512-602-0038 (o)                 512-507-7807 (c)
> 
> 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I have an AMD machine in my Lab which I believe has SVM support.
> I can use that if needed, but I will need sometime to get it 
> ready and running Xen.
> I thought Tom has tested oprofile with both AMD and Intel HVM 
> guests and that things were working fine.
thanks renato, I believe that the hypervisor portion of xenoprofile is
sending
the correct mode, eip/rip values.  I''ll get with Ray and see if we can
track
down where the issue might be (oprofile or hv, or configuration issue). 
> Could you please sync up with Tom and make sure it is not a 
> problem with your configuration.
> Tom can you confirm you were able to successfully run 
> XenOprofile for HVM guests?
> If needed I can try to reproduce your problem here when I 
> have my AMD machine up.
We''ll definitely let you know if we need some assistance in the
oprofile
area.  
Any idea when will 0.9.3 be out?  I know that some of the distros are 
trying to add your xenoprofile patch on top of 0.9.2 for their releases,
and having 0.9.3 would be easier that already contained the xen patches.
 
  --Tom
> 
> Regards
> 
> Renato 
> 
> > -----Original Message-----
> > From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com]
> > Sent: Tuesday, November 07, 2006 9:25 AM
> > To: xen-devel@lists.xensource.com
> > Cc: Santos, Jose Renato G; Woller, Thomas
> > Subject: Re: [Xen-devel] [XENOPROLIFE] New oprofile patch for 
> > XenOProfile
> > 
> > Renato,
> > 
> > Well, I''d postponed any work with xenoprof until we had a
working
> > 3.0.3 version plus the local patches that we are using 
> (includes the 
> > xenoprof patches that Tom Woller et al have done).
> > 
> > So, I''m now working with an oprofile 0.9.2 plus your patch 
> > oprofile-0.9.2-xen.patch from the HP open source website.   
> > The particular
> > project we are working on requires us to obtain profiles 
> for the HVM 
> > guest
> > kernel.   Unfortunately, my first few tries at this have been 
> > unsuccessful (I
> > get lots of "lost kernel sample" messages in the 
> oprofiled.log and no 
> > reported samples for the guest kernel.  :-(  ).
> > 
> > So I guess my guestion to you is, do you have access to a 
> VT or AMD V 
> > enabled
> > platform that you can test stuff on?   (I''d like to find a 
> > collegue to work 
> > with on this problem if at all possible.)   I have dedicated 
> > VT and AMD
> > platforms for my use, so if you have a VT platorm, we can 
> get stuff to 
> > work correctly there first and then I can work with Tom et 
> al to get 
> > the AMD V versions to work.
> > 
> > --
> > Ray Bryant
> > AMD Performance Labs                   Austin, Tx
> > 512-602-0038 (o)                 512-507-7807 (c)
> > 
> > 
> > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: Woller, Thomas [mailto:thomas.woller@amd.com] 
> Sent: Wednesday, November 08, 2006 7:12 AM
> To: Santos, Jose Renato G; Ray Bryant; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] [XENOPROLIFE] New oprofile patch for 
> XenOProfile
> 
> > I have an AMD machine in my Lab which I believe has SVM support.
> > I can use that if needed, but I will need sometime to get 
> it ready and 
> > running Xen.
> > I thought Tom has tested oprofile with both AMD and Intel 
> HVM guests 
> > and that things were working fine.
> thanks renato, I believe that the hypervisor portion of 
> xenoprofile is sending the correct mode, eip/rip values.  
> I''ll get with Ray and see if we can track down where the 
> issue might be (oprofile or hv, or configuration issue). 
> 
Ok. That seems a good plan.
I will be glad to help if I can when you have more details.
On a related note, I received a report that XenOprofile is missing some
user level samples. I am preparing a patch with the fix and should post
it to the list later today. This should affect kernel samples though.
> > Could you please sync up with Tom and make sure it is not a problem 
> > with your configuration.
> > Tom can you confirm you were able to successfully run 
> XenOprofile for 
> > HVM guests?
> > If needed I can try to reproduce your problem here when I 
> have my AMD 
> > machine up.
> We''ll definitely let you know if we need some assistance in 
> the oprofile area.  
> Any idea when will 0.9.3 be out?  I know that some of the 
> distros are trying to add your xenoprofile patch on top of 
> 0.9.2 for their releases, and having 0.9.3 would be easier 
> that already contained the xen patches.
No I don''t know when 0.9.3 wil be out. I still need to sync up with
William Cohen to get the passive domain patches into the oprofile CVS
tree. I was waiting to finish some cleanups in the patch before sending
it to him, but I did not have had time to work on this yet.  

Regards  

Renato
>  
>   --Tom
> 
> > 
> > Regards
> > 
> > Renato
> > 
> > > -----Original Message-----
> > > From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com]
> > > Sent: Tuesday, November 07, 2006 9:25 AM
> > > To: xen-devel@lists.xensource.com
> > > Cc: Santos, Jose Renato G; Woller, Thomas
> > > Subject: Re: [Xen-devel] [XENOPROLIFE] New oprofile patch for 
> > > XenOProfile
> > > 
> > > Renato,
> > > 
> > > Well, I''d postponed any work with xenoprof until we had
a working
> > > 3.0.3 version plus the local patches that we are using
> > (includes the
> > > xenoprof patches that Tom Woller et al have done).
> > > 
> > > So, I''m now working with an oprofile 0.9.2 plus your
patch
> > > oprofile-0.9.2-xen.patch from the HP open source website.   
> > > The particular
> > > project we are working on requires us to obtain profiles
> > for the HVM
> > > guest
> > > kernel.   Unfortunately, my first few tries at this have been 
> > > unsuccessful (I
> > > get lots of "lost kernel sample" messages in the
> > oprofiled.log and no
> > > reported samples for the guest kernel.  :-(  ).
> > > 
> > > So I guess my guestion to you is, do you have access to a
> > VT or AMD V
> > > enabled
> > > platform that you can test stuff on?   (I''d like to find
a
> > > collegue to work 
> > > with on this problem if at all possible.)   I have dedicated 
> > > VT and AMD
> > > platforms for my use, so if you have a VT platorm, we can
> > get stuff to
> > > work correctly there first and then I can work with Tom et
> > al to get
> > > the AMD V versions to work.
> > > 
> > > --
> > > Ray Bryant
> > > AMD Performance Labs                   Austin, Tx
> > > 512-602-0038 (o)                 512-507-7807 (c)
> > > 
> > > 
> > > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> > 
> > 
> > 
> 
> 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Oprofile user level samples for passive domains are being lost.
The number of of lost samples is most significant when dom0 is idle.   
This patch fix the bug

Thanks

Renato


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wednesday 08 November 2006 10:54, Santos, Jose Renato G wrote:
>
> Ok. That seems a good plan.
> I will be glad to help if I can when you have more details.
> On a related note, I received a report that XenOprofile is missing some
> user level samples. I am preparing a patch with the fix and should post
> it to the list later today. This should affect kernel samples though.
>
"should" or "should not" affect kernel samples?

I went back through my old tests and the last time I had a good profile output 
was with change set 10650 in September.

<snip>

-- 
Ray Bryant
AMD Performance Labs                   Austin, Tx
512-602-0038 (o)                 512-507-7807 (c)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com] 
> Sent: Wednesday, November 08, 2006 11:23 AM
> To: Santos, Jose Renato G
> Cc: Woller, Thomas; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] [XENOPROLIFE] New oprofile patch for 
> XenOProfile
> 
> On Wednesday 08 November 2006 10:54, Santos, Jose Renato G wrote:
> 
> >
> > Ok. That seems a good plan.
> > I will be glad to help if I can when you have more details.
> > On a related note, I received a report that XenOprofile is missing 
> > some user level samples. I am preparing a patch with the fix and 
> > should post it to the list later today. This should affect 
> kernel samples though.
> >
> 
> "should" or "should not" affect kernel samples?
> 
Sorry for the typo. It should NOT affect kernel samples
REnato 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel