Santos, Jose Renato G
2006-Jul-07 03:15 UTC
[Xen-devel] [PATCH] Xenoprof passive domain support fixes
These patches provide some fixes for xenoprof passive domain support. Currently, passive domain samples are being assigned to the wrong kernel functions. This patch fixes this problem. In addition the patch changes the encoding of domain switch ESCAPE codes (marks used to separate samples in oprofile buffers associated with different domains). Instead of using 2 codes, one for START and one for END of passive domain samples, a single escape CODE value is used to indicate a domain switch (no need for a STOP followed by a START). Finally there some other minor style fixes. Note that these changes affect both the kernel and oprofile tools. The attached oprofile patch is against oprofile 0.8.2, since this was the version used by the original Xiaowei patch. I will port the passive domain support modifications to the latest oprofile 0.9.1 and post it as soon as I have it. Keir, please, apply the kernel patch to xen-unstable. Thanks Renato Signed-off-by: Jose Renato Santos <jsantos@hpl.hp.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yang, Xiaowei
2006-Jul-10 07:18 UTC
[Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
>Currently, passive domain samples are being assigned to the wrongkernel>functions.Can you explain more? Thanks, xiaowei _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Santos, Jose Renato G
2006-Jul-10 18:19 UTC
[Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Xiaowei,
In the original code the logic for encoding domain switches was buggy.
The whole approach of encoding the domain switch as a CPU mode change
does not seem right to me.
The code was somewhat difficult to follow, but I could find at least the
following bug.
This was the original code used to add a domain switch on the CPU
buffer:
+static void xenoprof_handle_passive(void)
+{
+ int i, j;
+
+ for (i = 0; i < pdomains; i++)
+ for (j = 0; j < passive_domains[i].nbuf; j++) {
+ xenoprof_buf_t *buf = p_xenoprof_buf[i][j];
+ if (buf->event_head == buf->event_tail)
+ continue;
+ oprofile_add_pc(IGNORED_PC,
CPU_MODE_PASSIVE_START, passive_domains[i].domain_id);
+ xenoprof_add_pc(buf, 1);
+ oprofile_add_pc(IGNORED_PC,
CPU_MODE_PASSIVE_STOP, passive_domains[i].domain_id);
+ }
+}
+
This code inserts the sequence (ESCAPE_CODE, CPU_MODE_PASSIVE_START,
IGNORED_PC, domain_id) in the CPU buffer, to indicate a domain switch.
Since ESCAPE_CODE = IGNORED_PC = 0, the inserted sequence is (0,
CPU_MODE_PASSIVE_START, 0, domain_id).
Now the code which reads the sample from the CPU buffer and copies them
to the event buffer is the following:
+ for (i = 0; i < available; ++i) {
+ struct op_sample * s &cpu_buf->buffer[cpu_buf->tail_pos];
+
+ if (is_code(s->eip)) {
+ if (s->event < CPU_TRACE_BEGIN) {
+ /* kernel/userspace switch */
+ cpu_mode = s->event;
+ if (state == sb_buffer_start)
+ state = sb_sample_start;
+
+ if (s->event == CPU_MODE_PASSIVE_START)
+ domain_switch DOMAIN_SWITCH_START_EVENT1;
+ else if (s->event =CPU_MODE_PASSIVE_STOP)
+ domain_switch DOMAIN_SWITCH_STOP_EVENT1;
+
+ if (domain_switch !DOMAIN_SWITCH_START_EVENT2)
+ add_cpu_mode_switch(s->event);
+ } else if (s->event == CPU_TRACE_BEGIN) {
+ state = sb_bt_start;
+ add_trace_begin();
+ } else {
+ struct mm_struct * oldmm = mm;
+
+ /* userspace context switch */
+ new = (struct task_struct *)s->event;
+
+ release_mm(oldmm);
+ mm = take_tasks_mm(new);
+ if (mm != oldmm)
+ cookie = get_exec_dcookie(mm);
+ add_user_ctx_switch(new, cookie);
+ }
+ } else {
+ if (domain_switch == DOMAIN_SWITCH_START_EVENT1)
{
(1)>>>>> add_event_entry(s->event);
+ domain_switch DOMAIN_SWITCH_START_EVENT2;
(2)>>>>> } else if (domain_switch =DOMAIN_SWITCH_START_EVENT1) {
+ add_sample_entry(s->eip, s->event);
+ } else if (domain_switch =DOMAIN_SWITCH_STOP_EVENT1) {
+ domain_switch = NO_DOMAIN_SWITCH;
+ } else {
+ if (state >= sb_bt_start &&
+ !add_sample(mm, s, cpu_mode)) {
+ if (state == sb_bt_start) {
+ state = sb_bt_ignore;
+
atomic_inc(&oprofile_stats.bt_lost_no_mapping);
+ }
+ }
+ }
+ }
+
+ increment_tail(cpu_buf);
+ }
The line marked (1)>>>>>, seems to be adding the domain id of the
passive domain switch.
However, because the domain_id was encoded as an "event" with
"eip=0",
this code is not executed when it needs to. The code executes the "if"
part of the statement instead, since "s->eip=0" causes is_code() to
evaluate to true. This should cause samples to be misinterpreted.
Also the if statement on line (2)>>>> will never be executed since
the
condition is inside an "else" for the same condition
(domain_switch == DOMAIN_SWITCH_START_EVENT1). Since the code was buggy
I changed the logic for representing domain switches. In the process of
doing that I eliminated the CPU_MODE_PASSIVE stop, as I have asked you
to do before. There is really no need to encode a START and a STOP. You
just need a mark to separate samples belonging to different domains.
I compared the results of oprofile with --passive-domains option and
with --active-domains while running a TCP network benchmark.
Here are the first few lines reported by oprofile in both cases:
--active-domains option:
29583 22.8896 vmlinux-syms-2.6.16.13-xenU
vmlinux-syms-2.6.16.13-xenU __copy_to_user_ll
4353 3.3681 vmlinux-syms-2.6.16.13-xenU
vmlinux-syms-2.6.16.13-xenU tcp_v4_rcv
2986 2.3104 vmlinux-syms-2.6.16.13-xenU
vmlinux-syms-2.6.16.13-xenU eth_type_trans
2820 2.1820 vmlinux-syms-2.6.16.13-xenU
vmlinux-syms-2.6.16.13-xenU netif_poll
2751 2.1286 vmlinux-syms-2.6.16.13-xenU
vmlinux-syms-2.6.16.13-xenU gnttab_end_foreign_transfer_ref
2402 1.8585 vmlinux-syms-2.6.16.13-xenU
vmlinux-syms-2.6.16.13-xenU network_tx_buf_gc
2332 1.8044 vmlinux-syms-2.6.16.13-xenU
vmlinux-syms-2.6.16.13-xenU network_start_xmit
--passive-domains option:
28805 10.5522 pvmlinux1-syms iowrite32_rep
3706 1.3576 pvmlinux1-syms iret_exc
2642 0.9679 pvmlinux1-syms netlink_recvmsg
2576 0.9437 pvmlinux1-syms ip4_datagram_connect
2541 0.9309 pvmlinux1-syms input_devices_read
2499 0.9155 pvmlinux1-syms hypercall_page
2204 0.8074 pvmlinux1-syms ip_setsockopt
1942 0.7114 pvmlinux1-syms skbuff_ctor
The set of functions reported in each case are completely different,
suggesting that the passive-domain case was in fact assigning samples to
the wrong functions.
After the modifications included in the patch that I submited I got the
following output for oprofile with --passive-domains:
23818 11.8026 pvmlinux3-syms __copy_to_user_ll
3272 1.6214 pvmlinux3-syms tcp_v4_rcv
2312 1.1457 pvmlinux3-syms eth_type_trans
2151 1.0659 pvmlinux3-syms netif_poll
2128 1.0545 pvmlinux3-syms hypercall_page
2024 1.0030 pvmlinux3-syms network_start_xmit
2023 1.0025 pvmlinux3-syms network_tx_buf_gc
1489 0.7378 pvmlinux3-syms ip_queue_xmit
This is much closer to the active-domain case and gives me confident
that the code is now doing the right thing
Anyway, thanks for your help on getting the passive domain support in.
If you had not generated the patch we would not have passive domain
support yet. I do appreciate your effort on getting this into
xen-unstable.
Thanks
Renato
>> -----Original Message-----
>> From: Yang, Xiaowei [mailto:xiaowei.yang@intel.com]
>> Sent: Monday, July 10, 2006 12:18 AM
>> To: Santos, Jose Renato G; Keir Fraser
>> Cc: xen-devel@lists.xensource.com
>> Subject: RE: [PATCH] Xenoprof passive domain support fixes
>>
>> >Currently, passive domain samples are being assigned to the wrong
>> >kernel functions.
>>
>> Can you explain more?
>> Thanks,
>> xiaowei
>>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ray Bryant
2006-Jul-11 19:02 UTC
Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Renato,
One thing I am still puzzled by in all of this is the following: In the
"opreport -x" output, I get entries for both pxen*-syms (which is a
symbolic
link to the xen-syms I supplied on the opreport command line) AND I get
entries for xen-syms itself:
samples| %|
------------------
76273 23.7270 papps2-syms
73306 22.8040 pxen2-syms
63550 19.7691 vmlinux
43024 13.3839 libc-2.4.so
24406 7.5922 xen-syms
17278 5.3748 jbd
12228 3.8039 ext3
9845 3.0626 oprofiled
395 0.1229 qemu-dm
<snip>
Why is that? (I''ve been told the xen-syms samples are samples in the
hypervisor due to dom0 activity, but I''ve been unable to verify this).
Additionally, I find that "opreport -lx" will report "no
symbols" for
papps*-syms:
samples % app name symbol name
76273 23.7738 papps2-syms (no symbols)
19131 5.9630 pxen2-syms l2e_rw_fault
17278 5.3854 jbd (no symbols)
12228 3.8114 ext3 (no symbols)
11840 3.6905 libc-2.4.so vfprintf
10256 3.1967 libc-2.4.so _IO_file_xsputn@@GLIBC_2.2.5
8587 2.6765 xen-syms general_protection
7374 2.2984 pxen2-syms vmx_asm_vmexit_handler
5212 1.6245 pxen2-syms resync_all
5128 1.5984 xen-syms write_cr3
<snip>
unless I do an "ln -s /boot/vmlinux2-syms /boot/papps2-syms". (It
appears
that opreport should be creating papps2-syms instead of vmlinux2-syms??)
Finally, I''m not convinced yet that the sample reports for the HVM
guest
(papps2-syms or pvmlinux2-syms, in this case) are correct. I''m going
to run
some native and xen profile sessions using the same benchmark and see if I
can correlate the results at all.
--
Ray Bryant
AMD Performance Labs Austin, Tx
512-602-0038 (o) 512-507-7807 (c)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Santos, Jose Renato G
2006-Jul-11 20:51 UTC
RE: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Ray, Please, see my comments below.>> -----Original Message----- >> From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com] >> Sent: Tuesday, July 11, 2006 12:02 PM >> To: xen-devel@lists.xensource.com >> Cc: Santos, Jose Renato G; Yang, Xiaowei >> Subject: Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain >> support fixes >> >> Renato, >> >> One thing I am still puzzled by in all of this is the >> following: In the "opreport -x" output, I get entries for >> both pxen*-syms (which is a symbolic link to the xen-syms I >> supplied on the opreport command line) AND I get entries for >> xen-syms itself: >> >> samples| %| >> ------------------ >> 76273 23.7270 papps2-syms >> 73306 22.8040 pxen2-syms >> 63550 19.7691 vmlinux >> 43024 13.3839 libc-2.4.so >> 24406 7.5922 xen-syms >> 17278 5.3748 jbd >> 12228 3.8039 ext3 >> 9845 3.0626 oprofiled >> 395 0.1229 qemu-dm >> <snip> >> >> Why is that? (I''ve been told the xen-syms samples are >> samples in the hypervisor due to dom0 activity, but I''ve >> been unable to verify this). >>Yes, that is correct. Every sample (including the ones in the hypervisor)are associated with a domain ( the current domain running on the CPU as indicated by the Xen macro "current"). Therefore opreport distinguish hypervisor samples based on the domain that was running at the time the sample was generated.>> Additionally, I find that "opreport -lx" will report "no symbols" for >> papps*-syms: >> >> samples % app name symbol name >> 76273 23.7738 papps2-syms (no symbols) >> 19131 5.9630 pxen2-syms l2e_rw_fault >> 17278 5.3854 jbd (no symbols) >> 12228 3.8114 ext3 (no symbols) >> 11840 3.6905 libc-2.4.so vfprintf >> 10256 3.1967 libc-2.4.so >> _IO_file_xsputn@@GLIBC_2.2.5 >> 8587 2.6765 xen-syms general_protection >> 7374 2.2984 pxen2-syms vmx_asm_vmexit_handler >> 5212 1.6245 pxen2-syms resync_all >> 5128 1.5984 xen-syms write_cr3 >> <snip> >> >> unless I do an "ln -s /boot/vmlinux2-syms >> /boot/papps2-syms". (It appears that opreport should be >> creating papps2-syms instead of vmlinux2-syms??) >>papps2-syms represent samples collected in user level for domain2 (i.e. ring 3). Remember that passive domain profiling cannot decode application level samples since domain0 does not know the current memory mappings of user level processes in domain 2. Therefore it is expected that opreport will report "no symbols" for papps2-syms. What is suspicious to me is that opreport is not reporting any samples in the kernel for domain2 (they should have appeared under the name vmlinux2-syms) This is probably a bug. Maybe this is triggered if you do not specify the option --passive-images. Did you specify this option? If not, try running the command with --passive-images=<linux image file for xenU> (e.g. --passive-images=/boot/vmlinux-syms-2.6.16-xenU)>> Finally, I''m not convinced yet that the sample reports for >> the HVM guest (papps2-syms or pvmlinux2-syms, in this case) >> are correct. I''m going to run some native and xen profile >> sessions using the same benchmark and see if I can correlate >> the results at all. >>There is a problem with the current version of xenoprof for passive domains. Samples are being assigned to wrong samples. I posted a patch last week, that fix this problem but it seems that it was not pushed into the main unstable tree yet. Try applying that patch and check if they match what you get from native. (If you cannot find the patch, please let me know I will forward it to you) I would appreciate if you could send me the results of your tests, either if you find problems or if they are successfull. I think not many people have used passive domain support yet, and any feedback would be usefull. Thanks Renato>> -- >> Ray Bryant >> AMD Performance Labs Austin, Tx >> 512-602-0038 (o) 512-507-7807 (c) >> >> >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Markus Armbruster
2006-Jul-12 08:44 UTC
Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Now that xen and kernel support passive domains and the kinks are being ironed out, I wonder whether anybody got a clean userland patch against *current* oprofile. I saw Renato''s patch against 0.8.2, but it doesn''t apply cleanly to 0.9.1. I can certainly merge it myself, but if you did already, please share. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ray Bryant
2006-Jul-12 14:39 UTC
Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
On Wednesday 12 July 2006 03:44, Markus Armbruster wrote:> Now that xen and kernel support passive domains and the kinks are > being ironed out, I wonder whether anybody got a clean userland patch > against *current* oprofile. I saw Renato''s patch against 0.8.2, but > it doesn''t apply cleanly to 0.9.1. I can certainly merge it myself, > but if you did already, please share. >The attached is what I am using for oprofile-0.9.1. It is a version of Renato''s patch. YMMV.> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Ray Bryant AMD Performance Labs Austin, Tx 512-602-0038 (o) 512-507-7807 (c) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ray Bryant
2006-Jul-12 18:40 UTC
Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Hi Renato, On Tuesday 11 July 2006 15:51, Santos, Jose Renato G wrote:> Ray, ><snip>> >> Additionally, I find that "opreport -lx" will report "no symbols" for > >> papps*-syms: > >> > >> samples % app name symbol name > >> 76273 23.7738 papps2-syms (no symbols) > >> 19131 5.9630 pxen2-syms l2e_rw_fault > >> 17278 5.3854 jbd (no symbols) > >> 12228 3.8114 ext3 (no symbols) > >> 11840 3.6905 libc-2.4.so vfprintf > >> 10256 3.1967 libc-2.4.so > >> _IO_file_xsputn@@GLIBC_2.2.5 > >> 8587 2.6765 xen-syms general_protection > >> 7374 2.2984 pxen2-syms vmx_asm_vmexit_handler > >> 5212 1.6245 pxen2-syms resync_all > >> 5128 1.5984 xen-syms write_cr3 > >> <snip> > >> > >> unless I do an "ln -s /boot/vmlinux2-syms > >> /boot/papps2-syms". (It appears that opreport should be > >> creating papps2-syms instead of vmlinux2-syms??) > > papps2-syms represent samples collected in user level for domain2 (i.e. > ring 3). Remember that passive domain profiling cannot decode > application level samples since domain0 does not know the current memory > mappings of user level processes in domain 2. Therefore it is expected > that opreport will report "no symbols" for papps2-syms. >I see. Oops. :-)> What is suspicious to me is that opreport is not reporting any samples > in the kernel for domain2 (they should have appeared under the name > vmlinux2-syms)Perhaps you meant pvmlinux2-syms here?> This is probably a bug. Maybe this is triggered if you do not specify > the option --passive-images. Did you specify this option? If not, try > running the command with --passive-images=<linux image file for xenU> > (e.g. --passive-images=/boot/vmlinux-syms-2.6.16-xenU) >Yes, here is the setup script: opcontrol --vmlinux=/home/raybry/xenbits-unstable.$cs.hg/linux-2.6.16.13-xen/vmlinux opcontrol --start-daemon --active-domains=0 --passive-domains=$passive \ --passive-images=/home/raybry/RH4U2/vmlinux \ --xen=/home/raybry/xenbits-unstable.$cs.hg/xen/xen-syms \ --verbose=all --event=GLOBAL_POWER_EVENTS:100000:1:1:1 where $cs is the current change set I am running and $passive is the passive domain id. Of course, the actual image file for an HVM guest is stored in the / file system of the guest, which in this case is a loopback mounted file. So the vmlinux referenced above is a copy of that file in the host''s file system. There are no samples attributed to pvmlinux2-syms in the oprofiled.log. There are lots of samples attributed to papps2-syms. Now this is all with your patches applied to change set 10428. It''s possible, I suppose, that there are some subtle differences making this incorrect at that change set level. I''ll move up to a more recent change set and try again. Also, I "ported" your oprofile changes forward to 0.9.1, so I could have messed that up. See the message I sent to Markus on xen-devel for a copy of my version of your> >> Finally, I''m not convinced yet that the sample reports for > >> the HVM guest (papps2-syms or pvmlinux2-syms, in this case) > >> are correct. I''m going to run some native and xen profile > >> sessions using the same benchmark and see if I can correlate > >> the results at all. > > There is a problem with the current version of xenoprof for passive > domains. Samples are being assigned to wrong samples. I posted a patch > last week, that fix this problem but it seems that it was not pushed > into the main unstable tree yet. > Try applying that patch and check if they match what you get from > native. > (If you cannot find the patch, please let me know I will forward it to > you) > I would appreciate if you could send me the results of your tests, > either if you find problems or if they are successfull. I think not many > people have used passive domain support yet, and any feedback would be > usefull.I am running with your patches from 2006-07-07 3:15:15 for both xen and oprofile. However, as stated above, I am running under changeset 10428. Let me upgrade and try again.> > Thanks > > RenatoThank you! Best Regards, -- Ray Bryant AMD Performance Labs Austin, Tx 512-602-0038 (o) 512-507-7807 (c) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Santos, Jose Renato G
2006-Jul-13 02:42 UTC
RE: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
>> -----Original Message----- >> From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com] >> Sent: Wednesday, July 12, 2006 11:40 AM >> To: Santos, Jose Renato G >> Cc: xen-devel@lists.xensource.com; Yang, Xiaowei >> Subject: Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain >> support fixes >> >> Hi Renato, >> >> On Tuesday 11 July 2006 15:51, Santos, Jose Renato G wrote: >> > Ray, >> > >> <snip> >> >> > >> Additionally, I find that "opreport -lx" will report >> "no symbols"Ray, I did not notice at the first time that you were using option -x on opreport. According to opreport man page this option should not be used in you did not run opcontrol with --separate option. I am not sure what the behavior would be in this case, but that is what may be causing the kernel samples to be ommited. Try running opreport without the -x option. Thanks for porting the patch to oprofile 0.9.1. I am fixing a few things and plan to post a new version for 0.9.1 in a few days. Renato>> > >> for >> > >> papps*-syms: >> > >> >> > >> samples % app name symbol name >> > >> 76273 23.7738 papps2-syms (no symbols) >> > >> 19131 5.9630 pxen2-syms l2e_rw_fault >> > >> 17278 5.3854 jbd (no symbols) >> > >> 12228 3.8114 ext3 (no symbols) >> > >> 11840 3.6905 libc-2.4.so vfprintf >> > >> 10256 3.1967 libc-2.4.so >> > >> _IO_file_xsputn@@GLIBC_2.2.5 >> > >> 8587 2.6765 xen-syms general_protection >> > >> 7374 2.2984 pxen2-syms >> vmx_asm_vmexit_handler >> > >> 5212 1.6245 pxen2-syms resync_all >> > >> 5128 1.5984 xen-syms write_cr3 >> > >> <snip> >> > >> >> > >> unless I do an "ln -s /boot/vmlinux2-syms >> /boot/papps2-syms". (It >> > >> appears that opreport should be creating papps2-syms instead of >> > >> vmlinux2-syms??) >> > >> > papps2-syms represent samples collected in user level for >> domain2 (i.e. >> > ring 3). Remember that passive domain profiling cannot decode >> > application level samples since domain0 does not know the current >> > memory mappings of user level processes in domain 2. >> Therefore it is >> > expected that opreport will report "no symbols" for papps2-syms. >> > >> >> I see. Oops. :-) >> >> > What is suspicious to me is that opreport is not reporting >> any samples >> > in the kernel for domain2 (they should have appeared under the name >> > vmlinux2-syms) >> >> Perhaps you meant pvmlinux2-syms here? >> >> > This is probably a bug. Maybe this is triggered if you do >> not specify >> > the option --passive-images. Did you specify this option? >> If not, try >> > running the command with --passive-images=<linux image >> file for xenU> >> > (e.g. --passive-images=/boot/vmlinux-syms-2.6.16-xenU) >> > >> >> Yes, here is the setup script: >> >> opcontrol >> --vmlinux=/home/raybry/xenbits-unstable.$cs.hg/linux-2.6.16.1 >> 3-xen/vmlinux >> opcontrol --start-daemon --active-domains=0 >> --passive-domains=$passive \ >> --passive-images=/home/raybry/RH4U2/vmlinux \ >> --xen=/home/raybry/xenbits-unstable.$cs.hg/xen/xen-syms \ >> --verbose=all --event=GLOBAL_POWER_EVENTS:100000:1:1:1 >> >> where $cs is the current change set I am running and >> $passive is the passive domain id. >> >> Of course, the actual image file for an HVM guest is stored >> in the / file system of the >> guest, which in this case is a loopback mounted file. So >> the vmlinux referenced above >> is a copy of that file in the host''s file system. >> >> There are no samples attributed to pvmlinux2-syms in the >> oprofiled.log. There are >> lots of samples attributed to papps2-syms. >> >> Now this is all with your patches applied to change set >> 10428. It''s possible, I suppose, that there are some subtle >> differences making this incorrect at that change set level. >> I''ll move up to a more recent change set and try again. >> Also, I "ported" your oprofile changes forward to 0.9.1, so >> I could have messed that up. See the message I sent to >> Markus on xen-devel for a copy of my version of your >> >> > >> Finally, I''m not convinced yet that the sample reports >> for the HVM >> > >> guest (papps2-syms or pvmlinux2-syms, in this case) are >> correct. >> > >> I''m going to run some native and xen profile sessions using the >> > >> same benchmark and see if I can correlate the results at all. >> > >> > There is a problem with the current version of xenoprof >> for passive >> > domains. Samples are being assigned to wrong samples. I >> posted a patch >> > last week, that fix this problem but it seems that it was >> not pushed >> > into the main unstable tree yet. >> > Try applying that patch and check if they match what you get from >> > native. >> > (If you cannot find the patch, please let me know I will >> forward it to >> > you) >> > I would appreciate if you could send me the results of your tests, >> > either if you find problems or if they are successfull. I >> think not >> > many people have used passive domain support yet, and any feedback >> > would be usefull. >> >> I am running with your patches from 2006-07-07 3:15:15 for >> both xen and oprofile. >> However, as stated above, I am running under changeset >> 10428. Let me upgrade >> and try again. >> >> > >> > Thanks >> > >> > Renato >> >> Thank you! >> >> Best Regards, >> -- >> Ray Bryant >> AMD Performance Labs Austin, Tx >> 512-602-0038 (o) 512-507-7807 (c) >> >> >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ray Bryant
2006-Jul-13 20:30 UTC
Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
On Wednesday 12 July 2006 21:42, Santos, Jose Renato G wrote: <snip>> > Ray, > > I did not notice at the first time that you were using option -x on > opreport. According to opreport man page this option should not be > used in you did not run opcontrol with --separate option. I am not sure > what the behavior would be in this case, but that is what may be causing > the kernel samples to be ommited. Try running opreport without the -x > option. >I''ve tried it all ways with and w/o --separate=kernel, with and w/o -x. No pvmlinux1 samples. :-( Anyway, there aren''t any pvmlinux1 samples in the oprofiled.log either, so this is a problem upstream of the opreport (i. e. either in oprofiled or in the kernel code that sets the mode, etc.) I''m losing a ton of samples someplace: opreport has just shy of 1,000,000 samples accounted for, and the experiment ran for 120 seconds on a 3.00 GHZ Pentium D, so that should be around 3.6 million samples. According to a "grep Sample oprofiled.log | wc" oprofiled saw around 3.4 million samples, so that is not too far off. (The experiment is a kernel compile, repeated over and over, so the guest cpu should be busy all the time.)> Thanks for porting the patch to oprofile 0.9.1. I am fixing a few things > and plan to post a new version for 0.9.1 in a few days. >Cool. I was just trying to help Markus out.> Renato >-- Ray Bryant AMD Performance Labs Austin, Tx 512-602-0038 (o) 512-507-7807 (c) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ray Bryant
2006-Jul-14 14:01 UTC
Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Renato, BTW, the HVM guest I am running (and hence the passive image) is a 64-bit guest. We''ve run experiments with 32 bit guests and we do get pvmlinux samples. There may be other differences between these experiments, but this is the only one we''ve been able to identify thus far. So perhaps there is something broken about kernel samples in a passive domain and 64 bit? -- Ray Bryant AMD Performance Labs Austin, Tx 512-602-0038 (o) 512-507-7807 (c) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
DaSilva, Rosilmildo
2006-Jul-14 14:37 UTC
RE: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Ray, A 64bit guest would never have any entry on pvmlinux?, since the code on opd_kernel.c, line 112 ( of oprofile ), hardcodes the addresses in the range of 0xC0100000 - 0xC060000. The 64bits guest has a complete different range. Renato, seu nome parece um nome Brasileiro. Rosimildo. -----Original Message----- From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com] Sent: Friday, July 14, 2006 9:01 AM To: xen-devel@lists.xensource.com Cc: Santos, Jose Renato G; Yang, Xiaowei; DaSilva, Rosilmildo Subject: Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes Renato, BTW, the HVM guest I am running (and hence the passive image) is a 64-bit guest. We''ve run experiments with 32 bit guests and we do get pvmlinux samples. There may be other differences between these experiments, but this is the only one we''ve been able to identify thus far. So perhaps there is something broken about kernel samples in a passive domain and 64 bit? -- Ray Bryant AMD Performance Labs Austin, Tx 512-602-0038 (o) 512-507-7807 (c) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ray Bryant
2006-Jul-14 14:42 UTC
Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
On Friday 14 July 2006 09:37, DaSilva, Rosilmildo wrote:> Ray, > A 64bit guest would never have any entry on pvmlinux?, since the code on > opd_kernel.c, line 112 ( of oprofile ), hardcodes the addresses in the > range of 0xC0100000 - 0xC060000. The 64bits guest has a complete > different range. > > Renato, seu nome parece um nome Brasileiro. > Rosimildo. >That would certainly explain what I am seeing. I thought Rento''s patch was intended to do away with such hard coded addresses, but perhaps not all of that is done yet. An easy test would be to change the range so it matches what the 64 bit kernel would use. What address range should I use instead (for 64-bit)? -- Ray Bryant AMD Performance Labs Austin, Tx 512-602-0038 (o) 512-507-7807 (c) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Santos, Jose Renato G
2006-Jul-14 16:15 UTC
RE: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Rosimildo, You are correct. This explains the problem that Ray is seeing. We need to fix the kernel code to use the right address range from the image file specified with --passive-images option. And in case the image file is not specified we need to rely on the CPU mode (user/kernel/xen) to assign the sample to the right bin. I will proabbly not have time to work on this before the Ottawa Linux Simposium next week which I will be attending. If nobody provides a fix until them I will work something out when I come back. Xiaowei, any change you could provide a fix for this? Thanks Renato PS- Brasileiro, sim. Bom encontrar outro brasileiro por aqui ...>> -----Original Message----- >> From: DaSilva, Rosilmildo [mailto:rosilmildo.dasilva@amd.com] >> Sent: Friday, July 14, 2006 7:37 AM >> To: Ray Bryant; xen-devel@lists.xensource.com >> Cc: Santos, Jose Renato G; Yang, Xiaowei >> Subject: RE: [Xen-devel] RE: [PATCH] Xenoprof passive domain >> support fixes >> >> Ray, >> A 64bit guest would never have any entry on pvmlinux?, since >> the code on opd_kernel.c, line 112 ( of oprofile ), >> hardcodes the addresses in the range of 0xC0100000 - >> 0xC060000. The 64bits guest has a complete different range. >> >> Renato, seu nome parece um nome Brasileiro. >> Rosimildo. >> >> -----Original Message----- >> From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com] >> Sent: Friday, July 14, 2006 9:01 AM >> To: xen-devel@lists.xensource.com >> Cc: Santos, Jose Renato G; Yang, Xiaowei; DaSilva, Rosilmildo >> Subject: Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain >> support fixes >> >> Renato, >> >> BTW, the HVM guest I am running (and hence the passive >> image) is a 64-bit >> guest. We''ve run experiments with 32 bit guests and we do >> get pvmlinux >> >> samples. There may be other differences between these experiments, >> but >> this is the only one we''ve been able to identify thus far. >> >> So perhaps there is something broken about kernel samples in >> a passive domain and 64 bit? >> >> -- >> Ray Bryant >> AMD Performance Labs Austin, Tx >> 512-602-0038 (o) 512-507-7807 (c) >> >> >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ray Bryant
2006-Jul-14 18:20 UTC
Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Renato, et al,
I hacked my oprofiled as follows
Index: oprofile-xen-0.9.1/daemon/opd_kernel.c
==================================================================---
oprofile-xen-0.9.1.orig/daemon/opd_kernel.c
+++ oprofile-xen-0.9.1/daemon/opd_kernel.c
@@ -109,8 +109,11 @@ void opd_create_passive(char const *name
image = xmalloc(sizeof(struct kernel_image));
image->name = xstrdup(file);
// start/end VM should be more accurate, but it works now
- image->start = 0xc0100000;
- image->end = 0xc0600000;
+ // image->start = 0xc0100000;
+ // image->end = 0xc0600000;
+ // hack for 64 bit test
+ image->start = 0xFFFFFFFF80100000;
+ image->end = 0xFFFFFFFF80600000;
image->id = id;
list_add(&image->list, &passive_vmlinux);
(This is the address range that Rosimildo suggested.)
Now I get pvmlinux samples for my 64-bit guest test case, however, the profile
is not any good.
Almost all of the samples (148747 out 148751) are to a single address (from
the oprofiled.log):
Sample 0x167fe0(0): kern (name /boot/pvmlinux2-syms,
0xffffffff80100000-0xffffffff80600000) ...
Rosimildo''s patched SVM code works correctly and provides approximately
the
same kernel profile under HVM as we get when we run the application under a
kernel natively (where we''ve run two simple applications: kernbench
and
stress vm).
We''d like to get to a point where the SVM and VT versions both work,
but at
the moment, we don''t have the expertise to create the VT fixes,
although
Rosimildo has created some patches for test purposes.
--
Ray Bryant
AMD Performance Labs Austin, Tx
512-602-0038 (o) 512-507-7807 (c)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Woller, Thomas
2006-Jul-14 18:37 UTC
RE: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
Attached patch is what we have been using internally to allow HVM guest profiling on 64bit hv with 32bit hvm guest. Code generated by Rosimildo. This patch is experimental and not ready for inclusion into any repo, but might help with the next set of patches for xenoprofile support. This patch has oprofile, SVM, VT and HVM modifications all rolled into one inclusive patch file. For SVM, if you modify opd_kernel.c image start/end addresses in oprofile 0.9.1 then 64bit samples seem to work. VT does not seem to work with attached patch, but we haven''t spent time debugging why. Mods: 1) move SVM global interrupt to vmexit handler 2) add VGCF_hvm_guest_nmi flag to guest context to distinguish nmi occuring in guest rather than in host 3) oprofile model files modified to use the new VGCF_hvm_guest_nmi flag to determine a bit more properly the mode Thanks Tom _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Santos, Jose Renato G
2006-Jul-16 19:36 UTC
RE: [Xen-devel] RE: [PATCH] Xenoprof passive domain support fixes
> -----Original Message----- > From: Ray Bryant [mailto:raybry@mpdtxmail.amd.com] > Sent: Friday, July 14, 2006 11:20 AM > To: Santos, Jose Renato G > Cc: DaSilva, Rosilmildo; xen-devel@lists.xensource.com; Yang, Xiaowei > Subject: Re: [Xen-devel] RE: [PATCH] Xenoprof passive domain > support fixes > > Renato, et al, > > I hacked my oprofiled as follows > > Index: oprofile-xen-0.9.1/daemon/opd_kernel.c > ==================================================================> --- oprofile-xen-0.9.1.orig/daemon/opd_kernel.c > +++ oprofile-xen-0.9.1/daemon/opd_kernel.c > @@ -109,8 +109,11 @@ void opd_create_passive(char const *name > image = xmalloc(sizeof(struct kernel_image)); > image->name = xstrdup(file); > // start/end VM should be more accurate, but > it works now > - image->start = 0xc0100000; > - image->end = 0xc0600000; > + // image->start = 0xc0100000; > + // image->end = 0xc0600000; > + // hack for 64 bit test > + image->start = 0xFFFFFFFF80100000; > + image->end = 0xFFFFFFFF80600000; > image->id = id; > list_add(&image->list, &passive_vmlinux); > > (This is the address range that Rosimildo suggested.) > > Now I get pvmlinux samples for my 64-bit guest test case, > however, the profile is not any good. > > Almost all of the samples (148747 out 148751) are to a single > address (from the oprofiled.log): > > Sample 0x167fe0(0): kern (name /boot/pvmlinux2-syms, > 0xffffffff80100000-0xffffffff80600000) ... > > Rosimildo''s patched SVM code works correctly and provides > approximately the same kernel profile under HVM as we get > when we run the application under a kernel natively (where > we''ve run two simple applications: kernbench and stress vm). >Thanks for the update. It is good to confirm that XenOprofile is getting the samples right.> We''d like to get to a point where the SVM and VT versions > both work, but at the moment, we don''t have the expertise to > create the VT fixes, although Rosimildo has created some > patches for test purposes. > --Ah, OK. So the problem is specific to x64 and VT processors. It seems you have XenOprofile with passive domain running fine for 32 bit fully virtualized guests (both Intel and AMD) and for x64 AMD, right? I am not sure if I will be able to fix this since I don''t have much expertise on VT. It would be nice if someone at Intel could take a look at this... Xiaowei, any suggestion of who can help on getting XenOprofile passive domain support working for fully virtualized guests on Intel X64? Thanks Renato> Ray Bryant > AMD Performance Labs Austin, Tx > 512-602-0038 (o) 512-507-7807 (c) > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel