Hello again,
simple test case: added the following statement in
xen/arch/x86/mm/p2m.c, in function p2m_mem_access_check():
printk("v->arch.user_regs.eip: 0x%016lx, __vmread(GUEST_RIP):
0x%016lx\n", v->arch.user_regs.eip, __vmread(GUEST_RIP));
(where v is current) which dutifully prints out (at the first page fault):
(XEN) v->arch.user_regs.eip: 0xfffff80002c13a7b, __vmread(GUEST_RIP):
0xfffff8000269ec0a
They''re not equal, and furthermore __vmread(GUEST_RIP) is the correct
one (the same value returned by xc_domain_hvm_getcontext_partial()).
The questions:
1. What''s the Xen-friendliest way to retrieve the _correct_ values for
RIP and all the other registers usually put in a struct hvm_hw_cpu
instance, in p2m_mem_access_check()?
2. What''s the explanation for why things run as they do now?
Thanks,
Razvan Cojocaru
>>> On 06.03.13 at 11:55, Razvan Cojocaru <rzvncj@gmail.com> wrote: > Hello again, > > simple test case: added the following statement in > xen/arch/x86/mm/p2m.c, in function p2m_mem_access_check(): > > printk("v->arch.user_regs.eip: 0x%016lx, __vmread(GUEST_RIP): > 0x%016lx\n", v->arch.user_regs.eip, __vmread(GUEST_RIP)); > > (where v is current) which dutifully prints out (at the first page fault): > > (XEN) v->arch.user_regs.eip: 0xfffff80002c13a7b, __vmread(GUEST_RIP): > 0xfffff8000269ec0a > > They''re not equal, and furthermore __vmread(GUEST_RIP) is the correct > one (the same value returned by xc_domain_hvm_getcontext_partial()). > > The questions: > > 1. What''s the Xen-friendliest way to retrieve the _correct_ values for > RIP and all the other registers usually put in a struct hvm_hw_cpu > instance, in p2m_mem_access_check()?v->arch.user_regs.eip is what you should look at. It''s expected to be in sync with the VMCS value as long as nothing (e.g. emulation) modified the value. There''s code in vmx_asm_vmexit_handler to sync the fields, and in vmx_asm_do_vmentry to sync them back in the opposite direction.> 2. What''s the explanation for why things run as they do now?If the above doesn''t work, you''re likely in bigger trouble. So you will want to hunt down why the user_regs field isn''t what you expect it to be. Jan
> v->arch.user_regs.eip is what you should look at. It''s > expected to be in sync with the VMCS value as long as > nothing (e.g. emulation) modified the value. There''s code in > vmx_asm_vmexit_handler to sync the fields, and in > vmx_asm_do_vmentry to sync them back in the opposite > direction.Thank you for your answer. Isn''t it possible that emulation is indeed involved, since in p2m_mem_access_check() a page fault has occured, and RIP might point to _after_ the offending instruction? Thanks, Razvan Cojocaru
>>> On 06.03.13 at 12:35, Razvan Cojocaru <rzvncj@gmail.com> wrote: >> v->arch.user_regs.eip is what you should look at. It''s >> expected to be in sync with the VMCS value as long as >> nothing (e.g. emulation) modified the value. There''s code in >> vmx_asm_vmexit_handler to sync the fields, and in >> vmx_asm_do_vmentry to sync them back in the opposite >> direction. > > Thank you for your answer. Isn''t it possible that emulation is indeed > involved, since in p2m_mem_access_check() a page fault has occured, and > RIP might point to _after_ the offending instruction?That wouldn''t match your observation (the two values were much farther apart), unless a call or jump got emulated. Further, after emulation, the user_regs.eip value should be the canonical one. And finally, you said you took this right after a page fault occurred, and - just like for any fault - the RIP the fault refers to is the faulting instruction, not the one following it. Jan
>> Thank you for your answer. Isn''t it possible that emulation is indeed >> involved, since in p2m_mem_access_check() a page fault has occured, and >> RIP might point to _after_ the offending instruction? > > That wouldn''t match your observation (the two values were > much farther apart), unless a call or jump got emulated.I''ve noticed that, and have indeed assumed that it was something like a call.> Further, after emulation, the user_regs.eip value should be the > canonical one. > > And finally, you said you took this right after a page fault > occurred, and - just like for any fault - the RIP the fault refers > to is the faulting instruction, not the one following it.Fair enough. Will debug the issue. Thanks, Razvan Cojocaru
> That wouldn''t match your observation (the two values were > much farther apart), unless a call or jump got emulated. > > Further, after emulation, the user_regs.eip value should be the > canonical one.Well, I''ve downloaded and installed a fresh copy of Xen 4.2.0 from the official website, and added the printk() to p2m.c (the only modification I''ve done): 1311 req->vcpu_id = v->vcpu_id; 1312 1313 printk("v->arch.user_regs.eip: 0x%016lx, __vmread(GUEST_RIP):0x%016lx\n", 1314 v->arch.user_regs.eip, __vmread(GUEST_RIP)); 1315 } I then launched my guest VM, and tested it with the xen-access tool from tools/tests/xen-access, which I''ve run like this (1 is the machine ID, as printed out by ''sudo xm list''): sudo ./xen-access 1 write Here''s what the hypervisor wrote (values that are different appear once per each VCPU): (XEN) HVM1: Booting from 0000:7c00 (XEN) v->arch.user_regs.eip: 0x00000000fc002de9, __vmread(GUEST_RIP):0x0000000000007c17 (XEN) v->arch.user_regs.eip: 0x0000000000007c17, __vmread(GUEST_RIP):0x0000000000000667 (XEN) v->arch.user_regs.eip: 0x0000000000000667, __vmread(GUEST_RIP):0x0000000000000116 (XEN) v->arch.user_regs.eip: 0x0000000000000116, __vmread(GUEST_RIP):0x0000000000000116 (XEN) v->arch.user_regs.eip: 0x0000000000000116, __vmread(GUEST_RIP):0x00000000000005ab (XEN) v->arch.user_regs.eip: 0x00000000000005ab, __vmread(GUEST_RIP):0x00000000000001e7 (XEN) v->arch.user_regs.eip: 0x00000000000001e7, __vmread(GUEST_RIP):0x00000000000001f3 (XEN) v->arch.user_regs.eip: 0x00000000000001f3, __vmread(GUEST_RIP):0x0000000000000b71 (XEN) v->arch.user_regs.eip: 0x0000000000000b71, __vmread(GUEST_RIP):0x0000000000000782 (XEN) v->arch.user_regs.eip: 0x0000000000000782, __vmread(GUEST_RIP):0x0000000000000782 (XEN) v->arch.user_regs.eip: 0x0000000000000782, __vmread(GUEST_RIP):0x0000000000000782 Not sure what''s going on, but this time the only code I''ve added is the printk(). Any thoughts? Thanks, Razvan Cojocaru
Furthermore (with the fresh install of Xen 4.2.0), when modifying
xen-access.c to fetch the registers and print RIP out:
624 case MEM_EVENT_REASON_VIOLATION:
625 {
626 struct hvm_hw_cpu hwcpu;
627
628 if (xc_domain_hvm_getcontext_partial(xch, domain_id,
HVM_SAVE_CODE(CPU),
629 req.vcpu_id, &hwcpu,
sizeof(hwcpu)) != 0)
630 {
631 ERROR("xc_domain_hvm_getcontext_partial()
failed\n");
632 interrupted = -1;
633 continue;
634 }
635
636 printf("RIP: 0x%016lx\n", hwCpu.rip);
637 fflush(stdout);
the RIP it prints out is always the __vmread(GUEST_RIP), not
v->arch.user_regs.eip:
xen-access output:
Got event from Xen
RIP: 0x0000000000003272
Got event from Xen
PAGE ACCESS: -w- for GFN 25 (offset 0001ba) gla 00000000000251ba (vcpu 0)
RIP: 0x000000000000fed9
Got event from Xen
PAGE ACCESS: -w- for GFN 0 (offset 00046c) gla 000000000000046c (vcpu 0)
RIP: 0x0000000000000643
Got event from Xen
PAGE ACCESS: -w- for GFN 9f (offset 000e54) gla 000000000009fe54 (vcpu 0)
RIP: 0x0000000000000643
Got event from Xen
PAGE ACCESS: -w- for GFN 32 (offset 000042) gla 0000000000032042 (vcpu 0)
RIP: 0x000000000000009b
Got event from Xen
PAGE ACCESS: -w- for GFN 9a (offset 0003d9) gla 000000000009a3d9 (vcpu 0)
RIP: 0x0000000000000269
corresponding hypervisor log:
(XEN) v->arch.user_regs.eip: 0x0000000000003272,
__vmread(GUEST_RIP):0x0000000000003272
(XEN) v->arch.user_regs.eip: 0x0000000000003272,
__vmread(GUEST_RIP):0x000000000000fed9
(XEN) v->arch.user_regs.eip: 0x000000000000fed9,
__vmread(GUEST_RIP):0x0000000000000643
(XEN) v->arch.user_regs.eip: 0x0000000000000643,
__vmread(GUEST_RIP):0x0000000000000643
(XEN) v->arch.user_regs.eip: 0x0000000000000643,
__vmread(GUEST_RIP):0x000000000000009b
(XEN) v->arch.user_regs.eip: 0x000000000000009b,
__vmread(GUEST_RIP):0x0000000000000269
Hope this helps,
Razvan Cojocaru
>>> On 06.03.13 at 14:49, Razvan Cojocaru <rzvncj@gmail.com> wrote: >> That wouldn''t match your observation (the two values were >> much farther apart), unless a call or jump got emulated. >> >> Further, after emulation, the user_regs.eip value should be the >> canonical one. > > Well, I''ve downloaded and installed a fresh copy of Xen 4.2.0 from the > official website, and added the printk() to p2m.c (the only modification > I''ve done): > > 1311 req->vcpu_id = v->vcpu_id; > 1312 > 1313 printk("v->arch.user_regs.eip: 0x%016lx, > __vmread(GUEST_RIP):0x%016lx\n", > 1314 v->arch.user_regs.eip, __vmread(GUEST_RIP)); > 1315 }I know close to nothing about the paths that can lead here, and hence I can''t assess whether expecting consistency at this point is valid. All I can tell is that right after VM exit (i.e. at the top of vmx_vmexit_handler()) the two values ought to be consistent. Jan
> I know close to nothing about the paths that can lead here, and > hence I can''t assess whether expecting consistency at this point > is valid. All I can tell is that right after VM exit (i.e. at the top of > vmx_vmexit_handler()) the two values ought to be consistent.Yes, that''s true, however what I''m after is a way to have the information associated with the proper RIP _before_ the VM exit (basically all of the information that gets stuffed into the struct hvm_hw_cpu instance by xc_domain_hvm_getcontext_partial()). Specifically, I want that information inside p2m_mem_access_check(). Thanks for your help, Razvan Cojocaru
At 14:33 +0000 on 06 Mar (1362580383), Jan Beulich wrote:> >>> On 06.03.13 at 14:49, Razvan Cojocaru <rzvncj@gmail.com> wrote: > >> That wouldn''t match your observation (the two values were > >> much farther apart), unless a call or jump got emulated. > >> > >> Further, after emulation, the user_regs.eip value should be the > >> canonical one. > > > > Well, I''ve downloaded and installed a fresh copy of Xen 4.2.0 from the > > official website, and added the printk() to p2m.c (the only modification > > I''ve done): > > > > 1311 req->vcpu_id = v->vcpu_id; > > 1312 > > 1313 printk("v->arch.user_regs.eip: 0x%016lx, > > __vmread(GUEST_RIP):0x%016lx\n", > > 1314 v->arch.user_regs.eip, __vmread(GUEST_RIP)); > > 1315 } > > I know close to nothing about the paths that can lead here, and > hence I can''t assess whether expecting consistency at this point > is valid. All I can tell is that right after VM exit (i.e. at the top of > vmx_vmexit_handler()) the two values ought to be consistent.I don''t think so -- v->arch.user_regs is only brought up to date during context switches. For the currently running vcpu you should be looking at guest_cpu_user_regs(). Cheers, Tim.
>>> On 07.03.13 at 17:49, Tim Deegan <tim@xen.org> wrote: > At 14:33 +0000 on 06 Mar (1362580383), Jan Beulich wrote: >> >>> On 06.03.13 at 14:49, Razvan Cojocaru <rzvncj@gmail.com> wrote: >> >> That wouldn''t match your observation (the two values were >> >> much farther apart), unless a call or jump got emulated. >> >> >> >> Further, after emulation, the user_regs.eip value should be the >> >> canonical one. >> > >> > Well, I''ve downloaded and installed a fresh copy of Xen 4.2.0 from the >> > official website, and added the printk() to p2m.c (the only modification >> > I''ve done): >> > >> > 1311 req->vcpu_id = v->vcpu_id; >> > 1312 >> > 1313 printk("v->arch.user_regs.eip: 0x%016lx, >> > __vmread(GUEST_RIP):0x%016lx\n", >> > 1314 v->arch.user_regs.eip, __vmread(GUEST_RIP)); >> > 1315 } >> >> I know close to nothing about the paths that can lead here, and >> hence I can''t assess whether expecting consistency at this point >> is valid. All I can tell is that right after VM exit (i.e. at the top of >> vmx_vmexit_handler()) the two values ought to be consistent. > > I don''t think so -- v->arch.user_regs is only brought up to date during > context switches. For the currently running vcpu you should be looking > at guest_cpu_user_regs().Oh, of course, yes - I was mixing them up quite badly. Sorry for that. Jan
> I don''t think so -- v->arch.user_regs is only brought up to date during > context switches. For the currently running vcpu you should be looking > at guest_cpu_user_regs().Yes, that''s what did it - I discovered that a few hours earlier and wanted to post the solution but you beat me to it :) Thanks, Razvan Cojocaru