Xen devel - Apr 2005 - [PATCH] Xenoprof: Enabling performance profiling in Xen

I am posting patches for xenoprof in the next three messages
  (one patch file in each message).

  These patches enable system-wide statistical profiling
  in Xen environments. Xenoprof was developed at HP Labs by
  modifying and extending the original OProfile code for linux
 
 The 3 patch files are:

  1) xenoprof-1.0-xen-2.0.5.patch:
      - Patch for Xen
  2) xenoprof-1.0-xenolinux-2.6.10.patch
      - Patch for the linux sparse tree in Xen source tree (2.6.10)
  3) xenoprof-1.0-oprofile-0.8.1
      - Patch for Oprofile 0.8.1

  The attached file provides an overview of Xenoprof
  and how to use it.
  The patches were generated from the Xen stable version 2.0.5.
  The oprofile patches were generated from version 0.8.1, but
  can also be applied to the more recent version 0.8.2. 

  I would be very interested in receiving feedback and knowing
  where people are using Xenoprof.

  I hope this will be useful for many of you
  Thanks and happy profiling !

  Jose Renato Santos
  HP Labs

  acknowledgement: Most of this code was written by 
  Aravind  Menon while working at our group at HP Labs under the
  supervision of Yoshio Turner, John Janakiraman, and myself



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Here is part 2 of the patches for Xenoprof
>>   2) xenoprof-1.0-xenolinux-2.6.10.patch
>>       - Patch for the linux sparse tree in Xen source tree (2.6.10)
   Jose Renato Santos
   HP Labs


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I took a quick look at the patches for supporting profiling in Xen and I 
have some questions about the patches.

-Do all domains have to use the same setup for the performance 
monitoring counters? Or is there some virtualization of the performance 
counters?

-How is this going to interact with other performance monitoring 
infrastructure such as perfctr and perfmon? Or is the design going to 
need to be significantly revised for other performance monitoring 
interfaces? I like OProfile as much as anyone else, but I would like to 
see the Xen support allow the other interfaces to performance monitoring 
hardware to work.

-It appears that the multiple samples can be queued up in the 
hypervisor. In the OProfile kernel support when a process exits there is 
a flush of the per cpu buffers to make sure that the VMAs are mapped to 
files and offsets before the the mappings are lost. Should there be a 
flush of the samples from the hypervisor to make sure that they are read 
out before the process exits and the memory maps are lost?

-OProfile analysis tools expect the exectuable to be around when 
analyzing the sample files. Is the oprofile user space making the 
assumption that the domains have the same executables for the active and 
passive domains? Things could be messy if the domains are different, 
e.g. Fedora Core 3 domain and a Rawhide domain.

-Should get rid of xen-2.0.5/xen/arch/x86/oprofile/#Makefile# in the xen 
hypervisor patch.


-Will

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com 
>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
>> William Cohen
>> Sent: Monday, April 11, 2005 1:27 PM
>> To: xen-devel@lists.xensource.com
>> Cc: oprofile-list
>> Subject: [Xen-devel] [PATCH] Xenoprof: Enabling performance 
>> profiling in Xen
>> 
>> 
>> I took a quick look at the patches for supporting profiling 
>> in Xen and I 
>> have some questions about the patches.
>> 
>> -Do all domains have to use the same setup for the performance 
>> monitoring counters? Or is there some virtualization of the 
>> performance 
>> counters?
>> 
    Right now, there is no virtualization of the hardware performance
    counters. Our goal was to have a system wide profiling capability
    that could enable us understand better the behavior of Xen.
    Currently, only one profiling session can be run at a time and the 
    initiator domain for that session (typicaly domain 0) is the 
    domain that specify the performance events to be monitored.
    All other profiled domains just collect and decode samples of the
    same events.
    It would be nice to have hardware performance counter virtualization
    but it is not on our priority list.

>> -How is this going to interact with other performance monitoring 
>> infrastructure such as perfctr and perfmon? Or is the design 
>> going to 
>> need to be significantly revised for other performance monitoring 
>> interfaces? I like OProfile as much as anyone else, but I 
>> would like to 
>> see the Xen support allow the other interfaces to 
>> performance monitoring 
>> hardware to work.
>> 
   Our focus was mostly on enabling Oprofile to work on Xen. 
   Significant changes will be required for other performance 
   monitoring infraestructure, although some of the code could 
   be re-used. We are not planning to work on these in the
   near future. 
>> -It appears that the multiple samples can be queued up in the 
>> hypervisor. In the OProfile kernel support when a process 
>> exits there is 
>> a flush of the per cpu buffers to make sure that the VMAs 
>> are mapped to 
>> files and offsets before the the mappings are lost. Should 
>> there be a 
>> flush of the samples from the hypervisor to make sure that 
>> they are read 
>> out before the process exits and the memory maps are lost?
>> 
   Samples are only queued for passive domains (on the initiator
   queue) (this is done to avoid waking up the initiator to process 
   samples that happen when running passive domains), or when executing 
   a critical part of the Xen code. PC samples for passive domains
   are not decoded into a file/offset and thus there is no issue.
   When the sample happens at a critical session of Xen, there
   is no issue either, since the PC refers to code in the hypervisor.
   When a counter overflows in an active domain, we guarantee that
   the samples in the hypervisor are transferred to the domain
   CPU  buffer, before the domain continue execution
   (i.e. virtually no queue).
   Thus there is no process context switch or exiting processes
   in between a counter overflow and the receipt of a sample in
   the domain CPU buffer (for active domains). The normal flush
   of CPU buffer in the domain when a process exit still applies,
   as in standard linux.

   
>> -OProfile analysis tools expect the exectuable to be around when 
>> analyzing the sample files. Is the oprofile user space making the 
>> assumption that the domains have the same executables for 
>> the active and 
>> passive domains? Things could be messy if the domains are different, 
>> e.g. Fedora Core 3 domain and a Rawhide domain.
>> 
   Right now, passive domains samples do not get assigned to any 
   executable file. They are assigned to a general category domain_n
   (coarse granularity profiling). (i.e. no executable file is needed
   by analysis tools for passive domains). Active domains 
   keep their own sample files and thus there is no issue either.
 
   Ian suggested that we add support to enable the user to specify
   (at the initiator)  a file with symbol mappings for passive domains.
   This may be added in a future version to enable symbol mapping
   for the kernel, but user level symbols would still not be decoded
   for passive domains.
>> -Should get rid of 
>> xen-2.0.5/xen/arch/x86/oprofile/#Makefile# in the xen 
>> hypervisor patch.
>> 
>> 
   Sure! Sorry, I missed that.

   Renato 

  PS: For context, those who did not see the original post, please
check:
 
http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.html
>> -Will
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Friday 08 April 2005 22:10, Santos, Jose Renato G (Jose Renato 
Santos) wrote:
>   I am posting patches for xenoprof in the next three messages
>   (one patch file in each message).
>
>   These patches enable system-wide statistical profiling
>   in Xen environments. Xenoprof was developed at HP Labs by
>   modifying and extending the original OProfile code for linux
>
>  The 3 patch files are:
>
>   1) xenoprof-1.0-xen-2.0.5.patch:
>       - Patch for Xen
>   2) xenoprof-1.0-xenolinux-2.6.10.patch
>       - Patch for the linux sparse tree in Xen source tree (2.6.10)
>   3) xenoprof-1.0-oprofile-0.8.1
>       - Patch for Oprofile 0.8.1
>
>   The attached file provides an overview of Xenoprof
>   and how to use it.
Thanks very much; this will come in very handy.  A few questions:

1) When should the --xen option be specified?  My quick testing shows 
that only when combining with the --start option does opcontrol process 
the --xen option fully.  For example, running "opcontrol --vmlinux=yyy 
--xen=zzz; opcontrol --start" does not work, but "opcontrol 
--vmlinux=xxx; opcontrol --start --xen=zzz" does work for me. 

2) So far I have been able to profile domain0 (running SDET) with 
nothing else running, but I can''t seem to account for another domain 
running at the same time.  For example, I create a domU (dom id 1) 
running a cpu bound task, I start oprofile with --passive-domains=1, 
but I do not get any ticks representing time spent in dom1 (all I want 
is coarse grain here).  In the sequence of commands to start oprofile, 
is there a particular invocation of opcontrol to specify the 
--passive-domains option?  So far I have been doing "oprofile 
--vmlinux=xxx; oprofile --start --xen=zzz --passive-domains=1"

3) So far I cannot get an samples in domains other than domain0.  Most 
likely user error here.  Can a domU be an initiator?  For example, if I 
have two domains, dom0 and dom1, can dom1 do "everything" while
leaving
dom0 totally uninvolved?

Thanks very much!

-Andrew Theurer

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Apr 11, 2005 at 03:37:57PM -0700, Santos, Jose Renato G (Jose Renato
Santos) wrote:
> http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.html
Where can I get the diff against oprofile userspace source, plus a diff
of the changes to the kernel driver?

(It''d be really handy if somebody wrote up a little bit about Xen (no
idea what a domain is) for us OProfile people too)

regards
john

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Thanks very much; this will come in very handy.  A few questions:
> 
> 1) When should the --xen option be specified?  My quick testing shows 
> that only when combining with the --start option does opcontrol process 
> the --xen option fully.  For example, running "opcontrol --vmlinux=yyy
> --xen=zzz; opcontrol --start" does not work, but "opcontrol 
> --vmlinux=xxx; opcontrol --start --xen=zzz" does work for me. 
I''ll fix that and send out a patch. Basically, opcontrol saves its
--vmlinux argument in "/root/.oprofile/daemonrc" from where it reads
it
back later. Currently, it does not save its --xen argument, so you have
to give it when you actually start profiling, i.e. either with --start
or with --start-daemon. 
> 
> 2) So far I have been able to profile domain0 (running SDET) with 
> nothing else running, but I can''t seem to account for another
domain
> running at the same time.  For example, I create a domU (dom id 1) 
> running a cpu bound task, I start oprofile with --passive-domains=1, 
> but I do not get any ticks representing time spent in dom1 (all I want 
> is coarse grain here).  In the sequence of commands to start oprofile, 
> is there a particular invocation of opcontrol to specify the 
> --passive-domains option?  So far I have been doing "oprofile 
> --vmlinux=xxx; oprofile --start --xen=zzz --passive-domains=1"
I think there may be a bug remaining with the user level tools for
passive domains. Just do a dmesg after the profiling session. If the
kernel code is OK, it should print out a line displaying the number of
samples collected for all domains, active and passive. like this

oprofile_arch_init: ret 0, events 8, is_primary 1
pmc: oprofile samples 2727, active 398, passive 2329, other 0, buffering
losses 0, NMI restarted 0

> 3) So far I cannot get an samples in domains other than domain0.  Most 
> likely user error here.  Can a domU be an initiator?  For example, if I 
> have two domains, dom0 and dom1, can dom1 do "everything" while
leaving
> dom0 totally uninvolved?
> 
To get samples at the end of profiling, you have to run opreport in each
of the profiled domains (yuck). This is because we basically run
OProfile in each of the active domains. Right now, I just run a script
which ssh''es to all domains, and collects the profile reports from
them.
So, the profile reports you get are per-domain, and you have to
post-process them if you want to get an integrated report.

Currently, any domain can be initiator. (This needs to be changed). So,
you can have a profiling session with only domU leaving out other
domains.  


Regards

Aravind
> Thanks very much!
> 
> -Andrew Theurer

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

John Levon wrote:
> On Mon, Apr 11, 2005 at 03:37:57PM -0700, Santos, Jose Renato G (Jose
Renato Santos) wrote:
> 
> 
>>http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.html
> 
> 
> Where can I get the diff against oprofile userspace source, plus a diff
> of the changes to the kernel driver?
> 
> (It''d be really handy if somebody wrote up a little bit about Xen
(no
> idea what a domain is) for us OProfile people too)
Xen is a virtualization system. It allows multiple domains (instances of 
the OS) to run on a computer using a hypervisor. Think of a domain as a 
Virtual Machine in the old 360 mainframes. Thus, a physical machine 
could appear to be multiple logical machines, each with a different disk 
image and installation running on it.

Xen uses paravirtualization, so the kernel needs to be modified to make 
calls to the hypervisor rather than directly touch hardware like the 
memory management unit. This is bit different than VMware which emulates 
the machine and the OS doesn''t need to be modified. However, Xen has 
better performance a a result of this. Note that user applications do 
not need to be modified to run on Xen. There is more information about 
Xen at:

http://xen.sf.net/

Here are the start of the threads in the xen-devel mailing list 
discussing the patches to provide oprofile support:

http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00256.html
http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00257.html
http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00259.html

-Will

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

This is my feeble attempt to port xenoprof to xen-unstable.  I really 
did not make a lot of changes, but(!) the one big caveat is that it 
does not work :)

The 2 patches attached are for ./xen and for ./linux-2.6.11-xen-sparse.  
Both should patch cleanly against bk (2005-04-15) and should compile 
fine.  It does boot, but starting profiling causes xen to stop (output 
below).

I''m not sure if this is of any use to anyone (being that it does not 
work yet), but I''d thought "why not" and maybe someone with
more
knowledge would point out some obvious errors I made.

-Andrew



(XEN) CPU:    0
(XEN) EIP:    0808:[<fc53b3e7>]
(XEN) EFLAGS: 00210046   CONTEXT: hypervisor
(XEN) eax: 00000000   ebx: 00000000   ecx: 00000000   edx: 00000000
(XEN) esi: fc551c00   edi: fc5143de   ebp: 00000000   esp: fc503ea8
(XEN) ds: 0810   es: 0810   fs: 0810   gs: 0810   ss: 0810   cs: 0808
(XEN) Stack trace from ESP=fc503ea8:
(XEN)    00000000 8403d000 [fc5143de] [fc53ac43] fc551c00 [fc5143de] 
00000002 00000000
(XEN)    00200096 006618db 00000002 fc551c00 [fc5143de] 00000000 
00000000 fc503f40
(XEN)    fc569e00 fc503f40 00000000 [fc539c77] 00000000 fc569e00 
fc503f40 00000002
(XEN)    fc551c00 fc551c00 fc551c35 [fc52d1e2] fc503f40 00000000 
00000000 00000000
(XEN)    fc551c35 00000000 00000000 [fc536846] fc503f40 fc551c35 
fc551c00 00000000
(XEN)    00000000 00000000 00000000 00000000 fc551c00 fc551c00 
[fc5143de] 00000808
(XEN)    00200046 fc503f80 00000000 00000000 [fc5196c6] fc503f80 
fc551c00 00000000
(XEN)    00000000 00000000 00000000 00000000 00000001 00ef0000 
[fc516f98] 00000808
(XEN)    00200246 fc551c00 0000032a fc552840 00000001 0000004c 9eb24e00 
c0412000
(XEN)    00000001 00000000 00000006 fffffffc c0107458 00000061 00200246 
c0413fc0
(XEN)    00000069 0000007b 0000007b 00000000 00000000 fc551c00
(XEN) Call Trace from ESP=fc503ea8:
(XEN)    [<fc5143de>] [<fc53ac43>] [<fc5143de>]
[<fc5143de>]
[<fc539c77>] [<fc52d1e2>]
(XEN)    [<fc536846>] [<fc5143de>] [<fc5196c6>]
[<fc516f98>]
(XEN) Faulting linear address might be 0000024c
(XEN) ************************************
(XEN) CPU0 FATAL TRAP 14 (page fault), ERROR_CODE 0000, IN INTERRUPT 
CONTEXT.
(XEN) System shutting down -- need manual reset.
(XEN) ************************************



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel