Bcc: Tim Deegan <tjd-xen@phlegethon.org> Subject: Re: [Xen-devel] Guest memory access hooking Reply-To: In-Reply-To: <CAG4Ohu_p-vVF9ZS01PeMqHvscCrrO+UDawK-noaaP8k+MuqHrQ@mail.gmail.com> Hi, At 10:56 -0500 on 13 Nov (1352804161), Cutter 409 wrote:> I''m trying to do some research with malware, and I''m trying to get > notifications on arbitrary guest page accesses (similar to what Ether > does.) I''ve noticed the mem-event API and it seems like it might be close > to what I need, but I can''t find much documentation about how it works or > how to use it.Yes, the mem-event api, and in particular the HVMOP_set_mem_access hypercall, looks like what you want. As you say, there isn''t much documentation for it, except the xen-access.c client and the mailing list archive. CC''ing Aravindh, who has worked on this code most recently and might be able to help with specific questions.> I know that that mem-event API works only with EPT, but is the code to > change permissions modifying the guest page tables, or does it work via > EPT? (Can the guest detect it?)It works by EPT. The guest can''t detect it by looking at its pagetables or page fault patterns, though it might be able to detect it by looking at timings.> I''m also interested monitoring arbitrary page access via the shadow page > tables. I''ve been reading through the code, but if anyone has any insight > or some kind of push in the right direction, I''d really appreciate it.Your best bet is to modify _sh_propagate. Look at how it handles shadow_mode_log_dirty() -- any time a writeable mapping is shadowed, the shadow PTE is made read-only until the guest is actually doing a write, then mark_dirty can be called. You should be able to do the same thing for other kinds of access. Cheers, Tim.
On Thu, Nov 15, 2012 at 4:08 AM, Tim Deegan <tim@xen.org> wrote:> > Bcc: Tim Deegan <tjd-xen@phlegethon.org> > Subject: Re: [Xen-devel] Guest memory access hooking > Reply-To: > In-Reply-To: <CAG4Ohu_p-vVF9ZS01PeMqHvscCrrO+UDawK-noaaP8k+MuqHrQ@mail.gmail.com> > > Hi, > > At 10:56 -0500 on 13 Nov (1352804161), Cutter 409 wrote: > > I''m trying to do some research with malware, and I''m trying to get > > notifications on arbitrary guest page accesses (similar to what Ether > > does.) I''ve noticed the mem-event API and it seems like it might be close > > to what I need, but I can''t find much documentation about how it works or > > how to use it. > > Yes, the mem-event api, and in particular the HVMOP_set_mem_access > hypercall, looks like what you want. As you say, there isn''t much > documentation for it, except the xen-access.c client and the mailing > list archive. > > CC''ing Aravindh, who has worked on this code most recently and might be > able to help with specific questions.Sure, I can help with the specifics of the API usage.> > I know that that mem-event API works only with EPT, but is the code to > > change permissions modifying the guest page tables, or does it work via > > EPT? (Can the guest detect it?) > > It works by EPT. The guest can''t detect it by looking at its pagetables > or page fault patterns, though it might be able to detect it by looking > at timings. > > > I''m also interested monitoring arbitrary page access via the shadow page > > tables. I''ve been reading through the code, but if anyone has any insight > > or some kind of push in the right direction, I''d really appreciate it.It might be useful to get mem-event working with shadow by following Tim''s suggestions to achieve what you are after. Thanks, Aravindh
Hi, I''m finally to a point where I can start looking at this more closely. I''m trying to wrap my head around the shadow code to figure out the right course of action. I''d want HVMOP_set_mem_access to work with both shadow and EPT, so I''d want things to work via p2m somehow. I think I understand this part. * HVMOP_set_mem_access is used to change the p2m_access_t for the target page(s). This should already be implemented I think? * During propagation, I''ll check the p2m map to see if I should mask off any permission bits. * On a shadow paging fault, I''ll check if the fault was caused by p2m permissions, somehow integrating that with the code for read-only guest page tables safely. Questions: * Just for background, am I correct in my understanding that the log_dirty code is used to track which gfns have been written to by the guest, in order to speed up migration? * Are multiple shadow tables maintained per domain? Is there one per VCPU? One shadow table per guest page table? Is it blown away every time the guest changes CR3? I''m having some trouble tracking this down. * How should I clear/update existing shadow entries after changing the p2m_access_t? Can I clear the shadow tables somehow and force everything to be repopulated? Is that insane? Thanks! On Thu, Nov 15, 2012 at 7:08 AM, Tim Deegan <tim@xen.org> wrote:> Bcc: Tim Deegan <tjd-xen@phlegethon.org> > Subject: Re: [Xen-devel] Guest memory access hooking > Reply-To: > In-Reply-To: < > CAG4Ohu_p-vVF9ZS01PeMqHvscCrrO+UDawK-noaaP8k+MuqHrQ@mail.gmail.com> > > Hi, > > At 10:56 -0500 on 13 Nov (1352804161), Cutter 409 wrote: > > I''m trying to do some research with malware, and I''m trying to get > > notifications on arbitrary guest page accesses (similar to what Ether > > does.) I''ve noticed the mem-event API and it seems like it might be close > > to what I need, but I can''t find much documentation about how it works or > > how to use it. > > Yes, the mem-event api, and in particular the HVMOP_set_mem_access > hypercall, looks like what you want. As you say, there isn''t much > documentation for it, except the xen-access.c client and the mailing > list archive. > > CC''ing Aravindh, who has worked on this code most recently and might be > able to help with specific questions. > > > I know that that mem-event API works only with EPT, but is the code to > > change permissions modifying the guest page tables, or does it work via > > EPT? (Can the guest detect it?) > > It works by EPT. The guest can''t detect it by looking at its pagetables > or page fault patterns, though it might be able to detect it by looking > at timings. > > > I''m also interested monitoring arbitrary page access via the shadow page > > tables. I''ve been reading through the code, but if anyone has any insight > > or some kind of push in the right direction, I''d really appreciate it. > > Your best bet is to modify _sh_propagate. Look at how it handles > shadow_mode_log_dirty() -- any time a writeable mapping is shadowed, the > shadow PTE is made read-only until the guest is actually doing a write, > then mark_dirty can be called. You should be able to do the same thing > for other kinds of access. > > Cheers, > > Tim. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hi, At 17:56 -0400 on 22 Apr (1366653364), Cutter 409 wrote:> I''m finally to a point where I can start looking at this more closely. I''m > trying to wrap my head around the shadow code to figure out the right > course of action. > > I''d want HVMOP_set_mem_access to work with both shadow and EPT, so I''d want > things to work via p2m somehow. I think I understand this part. > > * HVMOP_set_mem_access is used to change the p2m_access_t for the target > page(s). This should already be implemented I think?Yep. The shadow code uses the same p2m implementataion as NPT, so that should all be fine.> * During propagation, I''ll check the p2m map to see if I should mask off > any permission bits.Yep. You''ll already be doing a p2m lookup to get the MFN, so you just need to look at the p2ma as well.> * On a shadow paging fault, I''ll check if the fault was caused by p2m > permissions, somehow integrating that with the code for read-only guest > page tables safely.Yes. The common case will be handled in _sh_propagate, which is where the shadow PTE is constructed. For the rest you''ll need to look at the places where the shadow fault handler calls the emulator and DTRT (either before the call or in the callbacks that the emulator uses to access guest memory).> Questions: > > * Just for background, am I correct in my understanding that the log_dirty > code is used to track which gfns have been written to by the guest, in > order to speed up migration?That''s right. It''s also used to track which parts of an emulated framebuffer have been updated, to make VNC connections more efficient.> * Are multiple shadow tables maintained per domain? Is there one per VCPU? > One shadow table per guest page table? Is it blown away every time the > guest changes CR3? I''m having some trouble tracking this down.There''s one set of shadows per domain, shared among all VCPUs. A given page of memory may have multiple shadows though, e.g. if it''s seen both as a top-level pagetables and a leaf pagetable. Shadows are kept around until: - it looks like the page is no longer a pagetable; - the guest explicitly tells us it''s no longer a pagetable; or - we need the memory to shadow some other page. Mostly, a pages''s shadow(s) are kept in sync with any changes the guest makes to the page, by trapping and emulating all writes. For performance, we allow some l1 pagetables to be ''out of sync'' (''oos'' in the code), letting the guest write to the page directly. On guest CR3 writes (and other TLB-flush-related activity) we make sure any OOS shadows are brought up to date.> * How should I clear/update existing shadow entries after changing the > p2m_access_t? Can I clear the shadow tables somehow and force everything to > be repopulated? Is that insane?It depends how often you''re changing the access permissions. sh_remove_all_mappings() and sh_remove_write_access() will try to flush mappings of a single MFN from the shadows, but they can be expensive (e.g. involving a brute-force scan of all shadows) so if you''re going to do a lot of them it may be worth considering batching them up and calling shadow_blow_tables() once instead. Cheers, Tim.