Hello, I''m interested in using the mem_event API (or it''s LibVMI wrapper), but I''m interested in capturing write events anywhere in a domU guest''s memory, _and_ not just once for each page write, but every time a page is being written to. To this end, I''ve looked at the xen-access.c example, where at first all the pages are being monitored: xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rx, ~0ull, 0); xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rx, 0, xenaccess->domain_info->max_pages); Then, after an event is received: xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rwx, req.gfn, 1); thus allowing ''rwx'' access to 1 page starting at req.gfn. This strategy indeed allows me to inspect writes to any page of the guest OS, however I can only do this once per page. Once the page is allowed HVMMEM_access_rwx access, any subsequent writes to it become unavailable to my dom0 userspace tool (I''m "unregistering" for that page-specific event). The author of LibVMI''s events code has suggested that I might enable writes to that page, single step past the write instruction, and then disable writes again. Two questions: 1. I haven''t been able to find an example of how single-stepping via the mem_event API might work. Can you point me to some code that does this (or a paper, etc.)? 2. Is there any other way to achieve what I''m after? Thanks, Razvan Cojocaru
> 1. I haven''t been able to find an example of how single-stepping via the > mem_event API might work. Can you point me to some code that does this > (or a paper, etc.)?This is what I got so far (assume the xen-access.c source code file): int main(int argc, char *argv[]) { [...] rc = xc_hvm_set_mem_access(xch, domain_id, default_access, ~0ull, 0); rc = xc_hvm_set_mem_access(xch, domain_id, default_access, 0, xenaccess->domain_info->max_pages); xc_set_hvm_param(xch, domain_id, HVM_PARAM_MEMORY_EVENT_SINGLE_STEP, HVMPME_mode_sync); [...] case MEM_EVENT_REASON_SINGLESTEP: printf("SINGLESTEP: gla: %lx gfn %lx\n", req.gla, req.gfn); // If something to do with gfn/gfa, // stop single-stepping this domain and // start faulting again on page writes. break; case MEM_EVENT_REASON_VIOLATION: [...] if ( default_access != after_first_access ) { rc = xc_hvm_set_mem_access(xch, domain_id, after_first_access, req.gfn, 1); xc_domain_debug_control(xch, domain_id, XEN_DOMCTL_DEBUG_OP_SINGLE_STEP_ON, req.vcpu_id); } [...] } Not quite sure what to do with gla and gfn except print them out at this point, though. Again, the condition for stopping single-step mode is that a page write happened. Any help is appreciated. Thanks, Razvan Cojocaru
Hi, At 16:38 +0200 on 29 Nov (1354207125), Razvan Cojocaru wrote:> I''m interested in using the mem_event API (or it''s LibVMI wrapper), but > I''m interested in capturing write events anywhere in a domU guest''s > memory, _and_ not just once for each page write, but every time a page > is being written to. > > To this end, I''ve looked at the xen-access.c example, where at first all > the pages are being monitored: > > xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rx, ~0ull, 0); > xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rx, 0, > xenaccess->domain_info->max_pages); > > Then, after an event is received: > > xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rwx, req.gfn, 1); > > thus allowing ''rwx'' access to 1 page starting at req.gfn. > > This strategy indeed allows me to inspect writes to any page of the > guest OS, however I can only do this once per page. Once the page is > allowed HVMMEM_access_rwx access, any subsequent writes to it become > unavailable to my dom0 userspace tool (I''m "unregistering" for that > page-specific event).Yes, indeed.> The author of LibVMI''s events code has suggested that I might enable > writes to that page, single step past the write instruction, and then > disable writes again.That might work for single-vcpu guests. On multi-vcpu, you''d have to pause the whole VM, unprotect the page, single-step the one vcpu that trapped, re-protect the page, and unpause the VM. That might be unacceptably slow.> 1. I haven''t been able to find an example of how single-stepping via the > mem_event API might work. Can you point me to some code that does this > (or a paper, etc.)?I think you have to do the single-stepping via the existing debugger API, and the memory event API stuff is just to notify you that it''s happened. Cc''ing the author. Joe, can you help out here?> 2. Is there any other way to achieve what I''m after?You could try: - pause the domain - copy out the contents of the page - use XENMEM_decrease_reservation to remove the page from the guest - unpause the domain Then all accesses to that page will get emulated by Xen and forwarded to qemu, just like happens for emulated MMIO devices. In qemu, you can emulate the read or write access, and do anything else you like at the same time. That won''t work for memory that''s accessed in non-trivial ways (e.g. used for pagetables or descriptor tables) or using instructions that are unsupported/buggy in Xen''s instruction emulator. Cheers, Tim.
Hello, thank you for the quick reply!> That might work for single-vcpu guests. On multi-vcpu, you''d have to > pause the whole VM, unprotect the page, single-step the one vcpu that > trapped, re-protect the page, and unpause the VM. That might be > unacceptably slow.Ah, the simple fact that the words "unacceptably slow" could be employed to describe the situation makes this approach very unlikely to fit my purposes. I''m still interested in how the debugger API works, though. Maybe, just maybe, it''ll just be acceptably slow. :)> You could try: > - pause the domain > - copy out the contents of the page > - use XENMEM_decrease_reservation to remove the page from the guest > - unpause the domain > > Then all accesses to that page will get emulated by Xen and forwarded to > qemu, just like happens for emulated MMIO devices. In qemu, you can > emulate the read or write access, and do anything else you like at the > same time. > > That won''t work for memory that''s accessed in non-trivial ways > (e.g. used for pagetables or descriptor tables) or using instructions > that are unsupported/buggy in Xen''s instruction emulator.Well, I need it to be able to work on _all_ memory, however accessed. Also, with this approach I might have to copy out the contents of all of the domain''s pages in the worst case scenario - isn''t that something to avoid? Thanks, Razvan Cojocaru
At 18:20 +0200 on 29 Nov (1354213227), Razvan Cojocaru wrote:> Hello, thank you for the quick reply! > >That might work for single-vcpu guests. On multi-vcpu, you''d have to > >pause the whole VM, unprotect the page, single-step the one vcpu that > >trapped, re-protect the page, and unpause the VM. That might be > >unacceptably slow. > > Ah, the simple fact that the words "unacceptably slow" could be employed > to describe the situation makes this approach very unlikely to fit my > purposes. > > I''m still interested in how the debugger API works, though. Maybe, just > maybe, it''ll just be acceptably slow. :):) I think it will depend entirely on how much memory you protect and what the guest is doing with it. It should be possible to make the multi-vcpu case of single-stepping work better by having per-cpu EPT tables, if you''re up for hacking away at Xen. That way you wouldn''t have to pause the other vcpus while you made the memory writeable.> >You could try: > > - pause the domain > > - copy out the contents of the page > > - use XENMEM_decrease_reservation to remove the page from the guest > > - unpause the domain > > > >Then all accesses to that page will get emulated by Xen and forwarded to > >qemu, just like happens for emulated MMIO devices. In qemu, you can > >emulate the read or write access, and do anything else you like at the > >same time. > > > >That won''t work for memory that''s accessed in non-trivial ways > >(e.g. used for pagetables or descriptor tables) or using instructions > >that are unsupported/buggy in Xen''s instruction emulator. > > Well, I need it to be able to work on _all_ memory, however accessed.Ah, in that case this won''t work for you. I think the single-step may be your only option. Tim.
On Thu, Nov 29, 2012 at 7:18 AM, Razvan Cojocaru <rzvncj@gmail.com> wrote:>> >> 1. I haven''t been able to find an example of how single-stepping via the >> mem_event API might work. Can you point me to some code that does this >> (or a paper, etc.)? > > > This is what I got so far (assume the xen-access.c source code file): > > int main(int argc, char *argv[]) > { > [...] > rc = xc_hvm_set_mem_access(xch, domain_id, > default_access, ~0ull, 0); > rc = xc_hvm_set_mem_access(xch, domain_id, > default_access, 0, > xenaccess->domain_info->max_pages); > xc_set_hvm_param(xch, domain_id, > HVM_PARAM_MEMORY_EVENT_SINGLE_STEP, > HVMPME_mode_sync); > [...] > case MEM_EVENT_REASON_SINGLESTEP: > printf("SINGLESTEP: gla: %lx gfn %lx\n", > req.gla, req.gfn); > // If something to do with gfn/gfa, > // stop single-stepping this domain and > // start faulting again on page writes. > break; > > case MEM_EVENT_REASON_VIOLATION: > [...] > if ( default_access != after_first_access ) { > rc = xc_hvm_set_mem_access(xch, domain_id, > after_first_access, > req.gfn, 1); > xc_domain_debug_control(xch, domain_id, > XEN_DOMCTL_DEBUG_OP_SINGLE_STEP_ON, > req.vcpu_id); > } > [...] > } > > Not quite sure what to do with gla and gfn except print them out at this point, though. Again, the condition for stopping single-step mode is that a page write happened. Any help is appreciated.After you set single stepping on for a VCPU and resume, you should see a single step mem_event soon after. What is the behavior that you are observing? Are you trying this on an UP or SMP guest? Thanks, AP
> After you set single stepping on for a VCPU and resume, you should see > a single step mem_event soon after. What is the behavior that you are > observing? Are you trying this on an UP or SMP guest?I do see single step mem_events after. I''ve tried it on a SMP HVM Slackware guest. What behaviour am I observing? The gla and gfn values are being printed out - but I don''t know how to detect a write operation based on them. Tim Deegan has kindly suggested that I pass those values to a debugger API that I assume Xen has, but I don''t think there have been any replies on that topic since. Unfortunately I don''t have access to the computer I''ve written the test code on, and it''s a rather complicated setup that I can''t easily duplicate at home, so if you need specific details (gfn/gla values, etc.) I''m afraid I''ll only be able to provide them on Monday. Thanks, Razvan Cojocaru
On Fri, Nov 30, 2012 at 12:09 PM, Razvan Cojocaru <rzvncj@gmail.com> wrote:>> After you set single stepping on for a VCPU and resume, you should see >> a single step mem_event soon after. What is the behavior that you are >> observing? Are you trying this on an UP or SMP guest? > > I do see single step mem_events after. I''ve tried it on a SMP HVM > Slackware guest. What behaviour am I observing? The gla and gfn values > are being printed out - but I don''t know how to detect a write operation > based on them. Tim Deegan has kindly suggested that I pass those valuesYou will not be able to detect a write event based on them as you have marked the page rwx. One option is to mark the page rx and continue single stepping the page. The on the next write to that page should give you a write violation and there you can stop single stepping the page.> to a debugger API that I assume Xen has, but I don''t think there have > been any replies on that topic since. > > Unfortunately I don''t have access to the computer I''ve written the test > code on, and it''s a rather complicated setup that I can''t easily > duplicate at home, so if you need specific details (gfn/gla values, > etc.) I''m afraid I''ll only be able to provide them on Monday. > > Thanks, > Razvan Cojocaru
>> I do see single step mem_events after. I''ve tried it on a SMP HVM >> Slackware guest. What behaviour am I observing? The gla and gfn values >> are being printed out - but I don''t know how to detect a write operation >> based on them. Tim Deegan has kindly suggested that I pass those values > > You will not be able to detect a write event based on them as you have > marked the page rwx. One option is to mark the page rx and continue > single stepping the page. The on the next write to that page should > give you a write violation and there you can stop single stepping the > page.I''m not trying to detect a write event - the write event has already been detected: before the "case MEM_EVENT_REASON_VIOLATION" part of the code, I''ve put "xc_hvm_set_mem_access(xch, domain_id, default_access, 0, xenaccess->domain_info->max_pages);". Default_access, in the xen-access.c file, is rx, and after_first_access is rwx. What I am trying to do is catch the first write operation (that''s why I''m single stepping), not the first write mem_event. My scenario is this: I''d like to mark _all_ of the domain''s pages rw, then when I get a write mem_event, mark only one page rwx, allow that write (single stepping), then mark that one page rw again. This would allow me to catch more than one write per unique domain page, as xen-access.c does. Xen-access.c does not catch subsequent writes to a page once it received a write mem_event about it. Thanks, Razvan Cojocaru
On Fri, Nov 30, 2012 at 12:45 PM, Razvan Cojocaru <rzvncj@gmail.com> wrote:>>> I do see single step mem_events after. I''ve tried it on a SMP HVM >>> Slackware guest. What behaviour am I observing? The gla and gfn values >>> are being printed out - but I don''t know how to detect a write operation >>> based on them. Tim Deegan has kindly suggested that I pass those values >> >> You will not be able to detect a write event based on them as you have >> marked the page rwx. One option is to mark the page rx and continue >> single stepping the page. The on the next write to that page should >> give you a write violation and there you can stop single stepping the >> page. > > I''m not trying to detect a write event - the write event has already > been detected: before the "case MEM_EVENT_REASON_VIOLATION" part of the > code, I''ve put "xc_hvm_set_mem_access(xch, domain_id, default_access, 0, > xenaccess->domain_info->max_pages);". Default_access, in the > xen-access.c file, is rx, and after_first_access is rwx. > > What I am trying to do is catch the first write operation (that''s why > I''m single stepping), not the first write mem_event.Not sure what the difference is between first write operation and first write mem_event.> My scenario is this: I''d like to mark _all_ of the domain''s pages rw, > then when I get a write mem_event, mark only one page rwx, allow that > write (single stepping), then mark that one page rw again. This wouldDo you mean rx? If you mark a page rw you will not get a write mem_event for it.> allow me to catch more than one write per unique domain page, as > xen-access.c does. Xen-access.c does not catch subsequent writes to a > page once it received a write mem_event about it.You could track the EIP and GFN and if you single step more than one instruction you can mark the page RX and stop single stepping. You will get control again on the next write. Though if there are jmp, things could get tricky. Thanks, AP
> Not sure what the difference is between first write operation and > first write mem_event.The difference is this: a write operation will only trigger a write event for a page marked rx. So if we''re in single step mode and a write happens on a page marked rwx, this will not trigger a write mem_event.>> My scenario is this: I''d like to mark _all_ of the domain''s pages rw, >> then when I get a write mem_event, mark only one page rwx, allow that >> write (single stepping), then mark that one page rw again. This would > > Do you mean rx? If you mark a page rw you will not get a write mem_event for it.Yes, I meant rx. Sorry (it''s late where I''m writing from).> You could track the EIP and GFN and if you single step more than one > instruction you can mark the page RX and stop single stepping. You > will get control again on the next write. Though if there are jmp, > things could get tricky.So if I understand you correctly, simply single-stepping for only the duration of one MEM_EVENT_REASON_SINGLESTEP, which should be the write operation (ignoring the gfn/gla fields of the mem_event), should do the trick? Thanks, Razvan Cojocaru
At 22:09 +0200 on 30 Nov (1354313363), Razvan Cojocaru wrote:> > After you set single stepping on for a VCPU and resume, you should see > > a single step mem_event soon after. What is the behavior that you are > > observing? Are you trying this on an UP or SMP guest? > > I do see single step mem_events after. I''ve tried it on a SMP HVM > Slackware guest. What behaviour am I observing? The gla and gfn values > are being printed out - but I don''t know how to detect a write operation > based on them.What more do you want to know? The gla tells you what the address is that''s being written to. If you want the rest of the register state &c, then have a look at either tools/debugger/gdbsx (a full debugger stub for guests; its maintainer is Mukesh Rathor <mukesh.rathor@oracle.com>) or tools/xentrace/xenctx.c (which has some code to print registers, read memory addresses, &c) Cheers, Tim.
> What more do you want to know? The gla tells you what the address is > that''s being written to. If you want the rest of the register state &c, > then have a look at either tools/debugger/gdbsx (a full debugger stub > for guests; its maintainer is Mukesh Rathor <mukesh.rathor@oracle.com>) > or tools/xentrace/xenctx.c (which has some code to print registers, read > memory addresses, &c)Excellent. Will look those up. Thanks, Razvan
On Fri, Nov 30, 2012 at 1:04 PM, Razvan Cojocaru <rzvncj@gmail.com> wrote:>> Not sure what the difference is between first write operation and >> first write mem_event. > > The difference is this: a write operation will only trigger a write > event for a page marked rx. So if we''re in single step mode and a write > happens on a page marked rwx, this will not trigger a write mem_event. > >>> My scenario is this: I''d like to mark _all_ of the domain''s pages rw, >>> then when I get a write mem_event, mark only one page rwx, allow that >>> write (single stepping), then mark that one page rw again. This would >> >> Do you mean rx? If you mark a page rw you will not get a write mem_event for it. > > Yes, I meant rx. Sorry (it''s late where I''m writing from). > >> You could track the EIP and GFN and if you single step more than one >> instruction you can mark the page RX and stop single stepping. You >> will get control again on the next write. Though if there are jmp, >> things could get tricky. > > So if I understand you correctly, simply single-stepping for only the > duration of one MEM_EVENT_REASON_SINGLESTEP, which should be the write > operation (ignoring the gfn/gla fields of the mem_event), should do the > trick?That is correct. Let me know how it goes. Cheers, AP
>> So if I understand you correctly, simply single-stepping for only the >> duration of one MEM_EVENT_REASON_SINGLESTEP, which should be the write >> operation (ignoring the gfn/gla fields of the mem_event), should do the >> trick? > > That is correct. Let me know how it goes.So far, so good. It did work when I tried it. Thank you for your help, Razvan Cojocaru
Seemingly Similar Threads
- [PATCH V2] mem_event: Add support for MEM_EVENT_REASON_MSR
- [PATCH V4] mem_event: Add support for MEM_EVENT_REASON_MSR
- [PATCH 0 of 2] v2: memshare/xenpaging/xen-access fixes for xen-unstable
- [PATCH] [resend] xen-access: Check return values and clean up on errors during init
- [PATCH 0 of 2] xenpaging:speed up page-in