Xen devel - Nov 2012 - Mem_event API and MEM_EVENT_REASON_SINGLESTEP

Hello,

I''m interested in using the mem_event API (or it''s LibVMI
wrapper), but
I''m interested in capturing write events anywhere in a domU
guest''s
memory, _and_ not just once for each page write, but every time a page 
is being written to.

To this end, I''ve looked at the xen-access.c example, where at first
all
the pages are being monitored:

xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rx, ~0ull, 0);
xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rx, 0, 
xenaccess->domain_info->max_pages);

Then, after an event is received:

xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rwx, req.gfn, 1);

thus allowing ''rwx'' access to 1 page starting at req.gfn.

This strategy indeed allows me to inspect writes to any page of the 
guest OS, however I can only do this once per page. Once the page is 
allowed HVMMEM_access_rwx access, any subsequent writes to it become 
unavailable to my dom0 userspace tool (I''m "unregistering"
for that
page-specific event).

The author of LibVMI''s events code has suggested that I might enable 
writes to that page, single step past the write instruction, and then 
disable writes again.

Two questions:

1. I haven''t been able to find an example of how single-stepping via
the
mem_event API might work. Can you point me to some code that does this 
(or a paper, etc.)?

2. Is there any other way to achieve what I''m after?

Thanks,
Razvan Cojocaru

> 1. I haven''t been able to find an example of how single-stepping
via the
> mem_event API might work. Can you point me to some code that does this
> (or a paper, etc.)?
This is what I got so far (assume the xen-access.c source code file):

int main(int argc, char *argv[])
{
[...]
     rc = xc_hvm_set_mem_access(xch, domain_id,
                                default_access, ~0ull, 0);
     rc = xc_hvm_set_mem_access(xch, domain_id,
                                default_access, 0,
                                xenaccess->domain_info->max_pages);
     xc_set_hvm_param(xch, domain_id,
                      HVM_PARAM_MEMORY_EVENT_SINGLE_STEP,
                      HVMPME_mode_sync);
[...]
             case MEM_EVENT_REASON_SINGLESTEP:
                 printf("SINGLESTEP: gla: %lx gfn %lx\n",
                        req.gla, req.gfn);
                 // If something to do with gfn/gfa,
                 // stop single-stepping this domain and
                 // start faulting again on page writes.
                 break;

             case MEM_EVENT_REASON_VIOLATION:
                 [...]
                 if ( default_access != after_first_access ) {
                     rc = xc_hvm_set_mem_access(xch, domain_id,
                                                after_first_access,
                                                req.gfn, 1);
                     xc_domain_debug_control(xch, domain_id,
                          XEN_DOMCTL_DEBUG_OP_SINGLE_STEP_ON,
                          req.vcpu_id);
                 }
[...]
}

Not quite sure what to do with gla and gfn except print them out at this 
point, though. Again, the condition for stopping single-step mode is 
that a page write happened. Any help is appreciated.

Thanks,
Razvan Cojocaru

Hi,

At 16:38 +0200 on 29 Nov (1354207125), Razvan Cojocaru
wrote:
> I''m interested in using the mem_event API (or it''s LibVMI
wrapper), but
> I''m interested in capturing write events anywhere in a domU
guest''s
> memory, _and_ not just once for each page write, but every time a page 
> is being written to.
> 
> To this end, I''ve looked at the xen-access.c example, where at
first all
> the pages are being monitored:
> 
> xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rx, ~0ull, 0);
> xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rx, 0, 
> xenaccess->domain_info->max_pages);
> 
> Then, after an event is received:
> 
> xc_hvm_set_mem_access(xch, domain_id, HVMMEM_access_rwx, req.gfn, 1);
> 
> thus allowing ''rwx'' access to 1 page starting at req.gfn.
> 
> This strategy indeed allows me to inspect writes to any page of the 
> guest OS, however I can only do this once per page. Once the page is 
> allowed HVMMEM_access_rwx access, any subsequent writes to it become 
> unavailable to my dom0 userspace tool (I''m
"unregistering" for that
> page-specific event).
Yes, indeed.
> The author of LibVMI''s events code has suggested that I might
enable
> writes to that page, single step past the write instruction, and then 
> disable writes again.
That might work for single-vcpu guests.  On multi-vcpu, you''d have to
pause the whole VM, unprotect the page, single-step the one vcpu that
trapped, re-protect the page, and unpause the VM.  That might be
unacceptably slow.
> 1. I haven''t been able to find an example of how single-stepping
via the
> mem_event API might work. Can you point me to some code that does this 
> (or a paper, etc.)?
I think you have to do the single-stepping via the existing debugger
API, and the memory event API stuff is just to notify you that it''s
happened. 

Cc''ing the author.  Joe, can you help out here?
> 2. Is there any other way to achieve what I''m after?
You could try:
 - pause the domain
 - copy out the contents of the page
 - use XENMEM_decrease_reservation to remove the page from the guest
 - unpause the domain

Then all accesses to that page will get emulated by Xen and forwarded to
qemu, just like happens for emulated MMIO devices.  In qemu, you can
emulate the read or write access, and do anything else you like at the
same time.

That won''t work for memory that''s accessed in non-trivial ways
(e.g. used for pagetables or descriptor tables) or using instructions
that are unsupported/buggy in Xen''s instruction emulator.

Cheers,

Tim.

Hello, thank you for the quick reply!
> That might work for single-vcpu guests.  On multi-vcpu, you''d have
to
> pause the whole VM, unprotect the page, single-step the one vcpu that
> trapped, re-protect the page, and unpause the VM.  That might be
> unacceptably slow.
Ah, the simple fact that the words "unacceptably slow" could be
employed
to describe the situation makes this approach very unlikely to fit my 
purposes.

I''m still interested in how the debugger API works, though. Maybe, just
maybe, it''ll just be acceptably slow. :)
> You could try:
>   - pause the domain
>   - copy out the contents of the page
>   - use XENMEM_decrease_reservation to remove the page from the guest
>   - unpause the domain
>
> Then all accesses to that page will get emulated by Xen and forwarded to
> qemu, just like happens for emulated MMIO devices.  In qemu, you can
> emulate the read or write access, and do anything else you like at the
> same time.
>
> That won''t work for memory that''s accessed in non-trivial
ways
> (e.g. used for pagetables or descriptor tables) or using instructions
> that are unsupported/buggy in Xen''s instruction emulator.
Well, I need it to be able to work on _all_ memory, however accessed. 
Also, with this approach I might have to copy out the contents of all of 
the domain''s pages in the worst case scenario - isn''t that
something to
avoid?

Thanks,
Razvan Cojocaru

At 18:20 +0200 on 29 Nov (1354213227), Razvan Cojocaru
wrote:
> Hello, thank you for the quick reply!
> >That might work for single-vcpu guests.  On multi-vcpu, you''d
have to
> >pause the whole VM, unprotect the page, single-step the one vcpu that
> >trapped, re-protect the page, and unpause the VM.  That might be
> >unacceptably slow.
> 
> Ah, the simple fact that the words "unacceptably slow" could be
employed
> to describe the situation makes this approach very unlikely to fit my 
> purposes.
> 
> I''m still interested in how the debugger API works, though. Maybe,
just
> maybe, it''ll just be acceptably slow. :)
:) I think it will depend entirely on how much memory you protect and
what the guest is doing with it.

It should be possible to make the multi-vcpu case of single-stepping
work better by having per-cpu EPT tables, if you''re up for hacking away
at Xen.  That way you wouldn''t have to pause the other vcpus while you
made the memory writeable.
> >You could try:
> >  - pause the domain
> >  - copy out the contents of the page
> >  - use XENMEM_decrease_reservation to remove the page from the guest
> >  - unpause the domain
> >
> >Then all accesses to that page will get emulated by Xen and forwarded
to
> >qemu, just like happens for emulated MMIO devices.  In qemu, you can
> >emulate the read or write access, and do anything else you like at the
> >same time.
> >
> >That won''t work for memory that''s accessed in
non-trivial ways
> >(e.g. used for pagetables or descriptor tables) or using instructions
> >that are unsupported/buggy in Xen''s instruction emulator.
> 
> Well, I need it to be able to work on _all_ memory, however accessed. 
Ah, in that case this won''t work for you.  I think the single-step may
be your only option.

Tim.

On Thu, Nov 29, 2012 at 7:18 AM, Razvan Cojocaru <rzvncj@gmail.com>
wrote:
>>
>> 1. I haven''t been able to find an example of how
single-stepping via the
>> mem_event API might work. Can you point me to some code that does this
>> (or a paper, etc.)?
>
>
> This is what I got so far (assume the xen-access.c source code file):
>
> int main(int argc, char *argv[])
> {
> [...]
>     rc = xc_hvm_set_mem_access(xch, domain_id,
>                                default_access, ~0ull, 0);
>     rc = xc_hvm_set_mem_access(xch, domain_id,
>                                default_access, 0,
>                                xenaccess->domain_info->max_pages);
>     xc_set_hvm_param(xch, domain_id,
>                      HVM_PARAM_MEMORY_EVENT_SINGLE_STEP,
>                      HVMPME_mode_sync);
> [...]
>             case MEM_EVENT_REASON_SINGLESTEP:
>                 printf("SINGLESTEP: gla: %lx gfn %lx\n",
>                        req.gla, req.gfn);
>                 // If something to do with gfn/gfa,
>                 // stop single-stepping this domain and
>                 // start faulting again on page writes.
>                 break;
>
>             case MEM_EVENT_REASON_VIOLATION:
>                 [...]
>                 if ( default_access != after_first_access ) {
>                     rc = xc_hvm_set_mem_access(xch, domain_id,
>                                                after_first_access,
>                                                req.gfn, 1);
>                     xc_domain_debug_control(xch, domain_id,
>                          XEN_DOMCTL_DEBUG_OP_SINGLE_STEP_ON,
>                          req.vcpu_id);
>                 }
> [...]
> }
>
> Not quite sure what to do with gla and gfn except print them out at this
point, though. Again, the condition for stopping single-step mode is that a page
write happened. Any help is appreciated.
After you set single stepping on for a VCPU and resume, you should see
a single step mem_event soon after. What is the behavior that you are
observing? Are you trying this on an UP or SMP guest?

Thanks,
AP

> After you set single stepping on for a VCPU and resume, you should see
> a single step mem_event soon after. What is the behavior that you are
> observing? Are you trying this on an UP or SMP guest?
I do see single step mem_events after. I''ve tried it on a SMP HVM
Slackware guest. What behaviour am I observing? The gla and gfn values
are being printed out - but I don''t know how to detect a write
operation
based on them. Tim Deegan has kindly suggested that I pass those values
to a debugger API that I assume Xen has, but I don''t think there have
been any replies on that topic since.

Unfortunately I don''t have access to the computer I''ve written
the test
code on, and it''s a rather complicated setup that I can''t
easily
duplicate at home, so if you need specific details (gfn/gla values,
etc.) I''m afraid I''ll only be able to provide them on Monday.

Thanks,
Razvan Cojocaru

On Fri, Nov 30, 2012 at 12:09 PM, Razvan Cojocaru <rzvncj@gmail.com>
wrote:
>> After you set single stepping on for a VCPU and resume, you should see
>> a single step mem_event soon after. What is the behavior that you are
>> observing? Are you trying this on an UP or SMP guest?
>
> I do see single step mem_events after. I''ve tried it on a SMP HVM
> Slackware guest. What behaviour am I observing? The gla and gfn values
> are being printed out - but I don''t know how to detect a write
operation
> based on them. Tim Deegan has kindly suggested that I pass those values
You will not be able to detect a write event based on them as you have
marked the page rwx. One option is to mark the page rx and continue
single stepping the page. The on the next write to that page should
give you a write violation and there you can stop single stepping the
page.
> to a debugger API that I assume Xen has, but I don''t think there
have
> been any replies on that topic since.
>
> Unfortunately I don''t have access to the computer I''ve
written the test
> code on, and it''s a rather complicated setup that I can''t
easily
> duplicate at home, so if you need specific details (gfn/gla values,
> etc.) I''m afraid I''ll only be able to provide them on
Monday.
>
> Thanks,
> Razvan Cojocaru

>> I do see single step mem_events after. I''ve tried it on a SMP
HVM
>> Slackware guest. What behaviour am I observing? The gla and gfn values
>> are being printed out - but I don''t know how to detect a write
operation
>> based on them. Tim Deegan has kindly suggested that I pass those values
> 
> You will not be able to detect a write event based on them as you have
> marked the page rwx. One option is to mark the page rx and continue
> single stepping the page. The on the next write to that page should
> give you a write violation and there you can stop single stepping the
> page.
I''m not trying to detect a write event - the write event has already
been detected: before the "case MEM_EVENT_REASON_VIOLATION" part of
the
code, I''ve put "xc_hvm_set_mem_access(xch, domain_id,
default_access, 0,
xenaccess->domain_info->max_pages);". Default_access, in the
xen-access.c file, is rx, and after_first_access is rwx.

What I am trying to do is catch the first write operation (that''s why
I''m single stepping), not the first write mem_event.

My scenario is this: I''d like to mark _all_ of the domain''s
pages rw,
then when I get a write mem_event, mark only one page rwx, allow that
write (single stepping), then mark that one page rw again. This would
allow me to catch more than one write per unique domain page, as
xen-access.c does. Xen-access.c does not catch subsequent writes to a
page once it received a write mem_event about it.

Thanks,
Razvan Cojocaru

On Fri, Nov 30, 2012 at 12:45 PM, Razvan Cojocaru <rzvncj@gmail.com>
wrote:
>>> I do see single step mem_events after. I''ve tried it on a
SMP HVM
>>> Slackware guest. What behaviour am I observing? The gla and gfn
values
>>> are being printed out - but I don''t know how to detect a
write operation
>>> based on them. Tim Deegan has kindly suggested that I pass those
values
>>
>> You will not be able to detect a write event based on them as you have
>> marked the page rwx. One option is to mark the page rx and continue
>> single stepping the page. The on the next write to that page should
>> give you a write violation and there you can stop single stepping the
>> page.
>
> I''m not trying to detect a write event - the write event has
already
> been detected: before the "case MEM_EVENT_REASON_VIOLATION" part
of the
> code, I''ve put "xc_hvm_set_mem_access(xch, domain_id,
default_access, 0,
> xenaccess->domain_info->max_pages);". Default_access, in the
> xen-access.c file, is rx, and after_first_access is rwx.
>
> What I am trying to do is catch the first write operation (that''s
why
> I''m single stepping), not the first write mem_event.
Not sure what the difference is between first write operation and
first write mem_event.
> My scenario is this: I''d like to mark _all_ of the
domain''s pages rw,
> then when I get a write mem_event, mark only one page rwx, allow that
> write (single stepping), then mark that one page rw again. This would
Do you mean rx? If you mark a page rw you will not get a write mem_event for it.
> allow me to catch more than one write per unique domain page, as
> xen-access.c does. Xen-access.c does not catch subsequent writes to a
> page once it received a write mem_event about it.
You could track the EIP and GFN and if you single step more than one
instruction you can mark the page RX and stop single stepping. You
will get control again on the next write. Though if there are jmp,
things could get tricky.

Thanks,
AP

> Not sure what the difference is between first write operation and
> first write mem_event.
The difference is this: a write operation will only trigger a write
event for a page marked rx. So if we''re in single step mode and a write
happens on a page marked rwx, this will not trigger a write mem_event.
>> My scenario is this: I''d like to mark _all_ of the
domain''s pages rw,
>> then when I get a write mem_event, mark only one page rwx, allow that
>> write (single stepping), then mark that one page rw again. This would
> 
> Do you mean rx? If you mark a page rw you will not get a write mem_event
for it.
Yes, I meant rx. Sorry (it''s late where I''m writing from).
> You could track the EIP and GFN and if you single step more than one
> instruction you can mark the page RX and stop single stepping. You
> will get control again on the next write. Though if there are jmp,
> things could get tricky.
So if I understand you correctly, simply single-stepping for only the
duration of one MEM_EVENT_REASON_SINGLESTEP, which should be the write
operation (ignoring the gfn/gla fields of the mem_event), should do the
trick?

Thanks,
Razvan Cojocaru

At 22:09 +0200 on 30 Nov (1354313363), Razvan Cojocaru
wrote:
> > After you set single stepping on for a VCPU and resume, you should see
> > a single step mem_event soon after. What is the behavior that you are
> > observing? Are you trying this on an UP or SMP guest?
> 
> I do see single step mem_events after. I''ve tried it on a SMP HVM
> Slackware guest. What behaviour am I observing? The gla and gfn values
> are being printed out - but I don''t know how to detect a write
operation
> based on them.
What more do you want to know?  The gla tells you what the address is
that''s being written to.  If you want the rest of the register state
&c,
then have a look at either tools/debugger/gdbsx (a full debugger stub
for guests; its maintainer is Mukesh Rathor <mukesh.rathor@oracle.com>)
or tools/xentrace/xenctx.c (which has some code to print registers, read
memory addresses, &c)

Cheers,

Tim.

> What more do you want to know?  The gla tells you what the address is
> that''s being written to.  If you want the rest of the register
state &c,
> then have a look at either tools/debugger/gdbsx (a full debugger stub
> for guests; its maintainer is Mukesh Rathor
<mukesh.rathor@oracle.com>)
> or tools/xentrace/xenctx.c (which has some code to print registers, read
> memory addresses, &c)
Excellent. Will look those up.

Thanks,
Razvan

On Fri, Nov 30, 2012 at 1:04 PM, Razvan Cojocaru <rzvncj@gmail.com>
wrote:
>> Not sure what the difference is between first write operation and
>> first write mem_event.
>
> The difference is this: a write operation will only trigger a write
> event for a page marked rx. So if we''re in single step mode and a
write
> happens on a page marked rwx, this will not trigger a write mem_event.
>
>>> My scenario is this: I''d like to mark _all_ of the
domain''s pages rw,
>>> then when I get a write mem_event, mark only one page rwx, allow
that
>>> write (single stepping), then mark that one page rw again. This
would
>>
>> Do you mean rx? If you mark a page rw you will not get a write
mem_event for it.
>
> Yes, I meant rx. Sorry (it''s late where I''m writing
from).
>
>> You could track the EIP and GFN and if you single step more than one
>> instruction you can mark the page RX and stop single stepping. You
>> will get control again on the next write. Though if there are jmp,
>> things could get tricky.
>
> So if I understand you correctly, simply single-stepping for only the
> duration of one MEM_EVENT_REASON_SINGLESTEP, which should be the write
> operation (ignoring the gfn/gla fields of the mem_event), should do the
> trick?
That is correct. Let me know how it goes.

Cheers,
AP

>> So if I understand you correctly, simply single-stepping for only the
>> duration of one MEM_EVENT_REASON_SINGLESTEP, which should be the write
>> operation (ignoring the gfn/gla fields of the mem_event), should do the
>> trick?
> 
> That is correct. Let me know how it goes.
So far, so good. It did work when I tried it.

Thank you for your help,
Razvan Cojocaru