Xen devel - Nov 2013 - PV guest + shadow mode translate causing hang

I am trying to implement mem_access for PV guests using shadow pagetables. I am
enabling shadow mode for pv using the following call:
xc_shadow_control(xch, domain_id,  XEN_DOMCTL_SHADOW_OP_ENABLE,  NULL, 0, NULL,
0, NULL);

This seems to work and I can see a lot of shadow activity if I turn on
debugtrace (xl debug T). However I see that the p2m table does not get
initialized in this shadow mode. So I tried enabling refcount and translate
modes:
xc_shadow_control(xch, domain_id,  XEN_DOMCTL_SHADOW_OP_ENABLE,  NULL, 0, NULL,
XEN_DOMCTL_SHADOW_ENABLE_REFCOUNT | XEN_DOMCTL_SHADOW_ENABLE_TRANSLATE, NULL);

But this is causing the guest and host to hang. I turned on debugtrace and all I
see on the serial console is the following:

1 sh: sh_set_allocation(): current 0 target 1024
2 p2m: p2m_alloc_table(): allocating p2m table
3 p2m: p2m_alloc_table(): populating p2m table
4 p2m: p2m_alloc_table(): p2m table initialised (65536 pages)
5 sh: sh_resync_all(): d=1, v=0
6 sh: sh_update_cr3__guest_4(): d=1 v=0 guest_table=54bc6
7 shdebug: sh_make_shadow(): (54bc6, 13)=>52651
8 sh: set_shadow_status(): d=1, v=0, gmfn=54bc6, type=0000000d, smfn=52651
9 sh: sh_set_toplevel_shadow(): 4/4 [0] gmfn 0x54bc6 smfn 0x52651
10 sh: sh_resync_all(): d=1, v=0
11 sh: sh_page_fault__guest_4(): d:v=1:0 va=0xffff88000fc0bd80 err=2,
rip=ffff82d08018b26a
12 sh: sh_page_fault__guest_4(): not a shadow fault

At this point the guest and host are unresponsive. Any idea what is going on?

This is with an Ubuntu 12.04 PV guest (3.2.0-48-virtual kernel) with the
following configuration:
builder = ''linux''
bootloader ="pygrub"
memory = 256
name = ''ubuntu-min-pv''
vcpus = 1
vif = [ ''type=vif, ip=192.168.0.1'' ]
disk = [''/dev/sdc,raw,xvda,w'']

Thanks,
Aravindh

At 08:15 +0000 on 11 Nov (1384154143), Aravindh Puthiyaparambil (aravindp)
wrote:
> I am trying to implement mem_access for PV guests using shadow pagetables.
I am enabling shadow mode for pv using the following call:
> xc_shadow_control(xch, domain_id,  XEN_DOMCTL_SHADOW_OP_ENABLE,  NULL, 0,
NULL, 0, NULL);
> 
> This seems to work and I can see a lot of shadow activity if I turn on
debugtrace (xl debug T). However I see that the p2m table does not get
initialized in this shadow mode. So I tried enabling refcount and translate
modes:
> xc_shadow_control(xch, domain_id,  XEN_DOMCTL_SHADOW_OP_ENABLE,  NULL, 0,
NULL, XEN_DOMCTL_SHADOW_ENABLE_REFCOUNT | XEN_DOMCTL_SHADOW_ENABLE_TRANSLATE,
NULL);
> 
> But this is causing the guest and host to hang.

The guest I expect, as it''s not expecting Xen to be translating its
pagetables.  The host hanging is unfortunate. :(

What you want to do is something in between the two: you need to
store some p2m-like information for each PV page but you don''t
actually want to do any translation.

One option would be to fix up shadow_mode_translate so that it uses a
1-1 p2m mapping, and then have that mapping be kept up to date with
any alloc/free/map operations.

Another, and I think better, would be to add a third implementation of
the p2m interface, that always return mfn==gfn, but also records
mem_event state.  Then you''d need to add another XEN_DOMCTL_SHADOW
flag to select that mode.

Tim.

>At 08:15 +0000 on 11 Nov (1384154143), Aravindh Puthiyaparambil (aravindp)
>wrote:
>> I am trying to implement mem_access for PV guests using shadow
>pagetables. I am enabling shadow mode for pv using the following call:
>> xc_shadow_control(xch, domain_id,  XEN_DOMCTL_SHADOW_OP_ENABLE,
>NULL,
>> 0, NULL, 0, NULL);
>>
>> This seems to work and I can see a lot of shadow activity if I turn on
>debugtrace (xl debug T). However I see that the p2m table does not get
>initialized in this shadow mode. So I tried enabling refcount and translate
>modes:
>> xc_shadow_control(xch, domain_id,  XEN_DOMCTL_SHADOW_OP_ENABLE,
>NULL,
>> 0, NULL, XEN_DOMCTL_SHADOW_ENABLE_REFCOUNT |
>> XEN_DOMCTL_SHADOW_ENABLE_TRANSLATE, NULL);
>>
>> But this is causing the guest and host to hang.
>
>
>The guest I expect, as it''s not expecting Xen to be translating its
pagetables.
>The host hanging is unfortunate. :(
>
>What you want to do is something in between the two: you need to store
>some p2m-like information for each PV page but you don''t actually
want to do
>any translation.
>
>One option would be to fix up shadow_mode_translate so that it uses a
>1-1 p2m mapping, and then have that mapping be kept up to date with any
>alloc/free/map operations.
>
>Another, and I think better, would be to add a third implementation of the
>p2m interface, that always return mfn==gfn, but also records mem_event
>state.  Then you''d need to add another XEN_DOMCTL_SHADOW flag to
select
>that mode.
OK, I will go down the path of adding another shadow mode. 

There is one other thing I am not clear about. When the mem_access listener is
trying to map guest memory or set access permissions for say a 256MB pv guest,
does it make the calls on gfns ranging from 0 to 256mb or does it need to do it
on the mfn? Or should this gnf -> mfn translation be abstracted by the new
shadow mode for the mem_access listener?

Thanks,
Aravindh