Xen devel - Dec 2007 - [PATCH] unshadow the page table page which are used as data page

The patch deals with the situation which guest OS uses unused page table
pages as data pages and write data to them. The pages will still be
grabbed by Xen as page table pages, and lots of unnecessary page faults
occur. 

The patch will check if the data guest writes to the page table contains
valid mfn or not, if not, we believe it is a data page now and then
unshadow the page.

The patch improves the VT-D NIC performance a lot.

 

Signed-off-by: Xin Xiaohui <xiaohui.xin@intel.com>

     




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

At 10:48 +0800 on 05 Dec (1196851700), Xin, Xiaohui
wrote:
> The patch will check if the data guest writes to the page table contains
> valid mfn or not, if not, we believe it is a data page now and then
> unshadow the page.
Good plan, thanks.
> +static inline void
> +check_for_data_page_unshadow(struct vcpu *v, void *addr,mfn_t mfn)
> +{
> +    gfn_t gfn;
> +    p2m_type_t p2m_type;
> +
> +#if SHADOW_OPTIMIZATIONS & SHOPT_EARLY_UNSHADOW
> +    if ( mfn_to_page(mfn)->shadow_flags & SHF_L2_32 )
> +    {
> +        gfn = _gfn((paddr_t)(((*(u32*)addr) &
(PADDR_MASK&PAGE_MASK))) >>
> +                 PAGE_SHIFT);
Shouldn''t you also check for the _PAGE_PRESENT bit?  Otherwise I think
normal shadow performance could be hurt.
> +        if ( gfn_to_mfn(v->domain, gfn, &p2m_type) == INVALID_MFN )
> +            sh_remove_shadows(v, mfn, 1, 0);
> +    }
> +    else if (mfn_to_page(mfn)->shadow_flags &
> +                (SHF_L4_64|SHF_L2_PAE|SHF_L2H_PAE))
Why not SHF_L3_64, SHF_l2_64 and SHF_L2H_64?
> +    {
> +        gfn = _gfn((paddr_t)(((*(u64*)addr) &
(PADDR_MASK&PAGE_MASK))) >>
> +                 PAGE_SHIFT);
> +        if ( gfn_to_mfn(v->domain, gfn, &p2m_type) == INVALID_MFN )
> +            sh_remove_shadows(v, mfn, 1, 0);
> +    }
> +#endif
> +}
>  
>  int
>  sh_x86_emulate_write(struct vcpu *v, unsigned long vaddr, void *src,
> @@ -4068,7 +4092,8 @@ sh_x86_emulate_write(struct vcpu *v, uns
>          check_for_early_unshadow(v, mfn);
>      else
>          reset_early_unshadow(v);
> -    
> +
> +    check_for_data_page_unshadow(v, addr, mfn);
>      paging_mark_dirty(v->domain, mfn_x(mfn));
This, and its equivalent in sh_x86_emulate_cmpxchg, need to be gated on
bytes() being greater than the size of the read you''ll be doing in
check_for_data_page_unshadow() or you''ll get #PF when a guest writes to
the top word of a page.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems.
[Company #5334508: XenSource UK Ltd, reg''d c/o EC2Y 5EB, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Tim,
Attached is the updated patch which based on some part of your suggestion and
some part of our new thoughts about it. We have re-checked the code path of the
guest write emulate, found that in some extent(not all) the code checks the
valid mfn for the guest written data. But maybe for the optimization, the code
just check valide mfn when PRESENT bit exists. Maybe it can cover most of the
cases, but not all, that''s what we have found in the vt-d iperf test.

To minimize the hurt to other performance of shadow, the patch tries to use the
valid mfn check in the original code, please have a review. I''m not
sure about the cost of the gfn_to_mfn(), and not sure whether we may get some
trade-off. If you have good ideas, please let us know.

Thanks
Xiaohui
-----Original Message-----
From: Tim Deegan [mailto:Tim.Deegan@citrix.com] 
Sent: 2007年12月5日 18:10
To: Xin, Xiaohui
Cc: xen-devel@lists.xensource.com; Kay, Allen M
Subject: Re: [Xen-devel][PATCH] unshadow the page table page which are used as
data page

Hi,

At 10:48 +0800 on 05 Dec (1196851700), Xin, Xiaohui
wrote:
> The patch will check if the data guest writes to the page table contains
> valid mfn or not, if not, we believe it is a data page now and then
> unshadow the page.
Good plan, thanks.
> +static inline void
> +check_for_data_page_unshadow(struct vcpu *v, void *addr,mfn_t mfn)
> +{
> +    gfn_t gfn;
> +    p2m_type_t p2m_type;
> +
> +#if SHADOW_OPTIMIZATIONS & SHOPT_EARLY_UNSHADOW
> +    if ( mfn_to_page(mfn)->shadow_flags & SHF_L2_32 )
> +    {
> +        gfn = _gfn((paddr_t)(((*(u32*)addr) &
(PADDR_MASK&PAGE_MASK))) >>
> +                 PAGE_SHIFT);
Shouldn''t you also check for the _PAGE_PRESENT bit?  Otherwise I think
normal shadow performance could be hurt.
> +        if ( gfn_to_mfn(v->domain, gfn, &p2m_type) == INVALID_MFN )
> +            sh_remove_shadows(v, mfn, 1, 0);
> +    }
> +    else if (mfn_to_page(mfn)->shadow_flags &
> +                (SHF_L4_64|SHF_L2_PAE|SHF_L2H_PAE))
Why not SHF_L3_64, SHF_l2_64 and SHF_L2H_64?
> +    {
> +        gfn = _gfn((paddr_t)(((*(u64*)addr) &
(PADDR_MASK&PAGE_MASK))) >>
> +                 PAGE_SHIFT);
> +        if ( gfn_to_mfn(v->domain, gfn, &p2m_type) == INVALID_MFN )
> +            sh_remove_shadows(v, mfn, 1, 0);
> +    }
> +#endif
> +}
>  
>  int
>  sh_x86_emulate_write(struct vcpu *v, unsigned long vaddr, void *src,
> @@ -4068,7 +4092,8 @@ sh_x86_emulate_write(struct vcpu *v, uns
>          check_for_early_unshadow(v, mfn);
>      else
>          reset_early_unshadow(v);
> -    
> +
> +    check_for_data_page_unshadow(v, addr, mfn);
>      paging_mark_dirty(v->domain, mfn_x(mfn));
This, and its equivalent in sh_x86_emulate_cmpxchg, need to be gated on
bytes() being greater than the size of the read you''ll be doing in
check_for_data_page_unshadow() or you''ll get #PF when a guest writes to
the top word of a page.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems.
[Company #5334508: XenSource UK Ltd, reg''d c/o EC2Y 5EB, UK.]


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, 

At 21:12 +0800 on 07 Dec (1197061925), Xin, Xiaohui
wrote:
> Tim,
>  Attached is the updated patch which based on some part of your
> suggestion and some part of our new thoughts about it. We have
> re-checked the code path of the guest write emulate, found that in
> some extent(not all) the code checks the valid mfn for the guest
> written data. But maybe for the optimization, the code just check
> valide mfn when PRESENT bit exists. Maybe it can cover most of the
> cases, but not all, that''s what we have found in the vt-d iperf
test.
Hmmm.  The new behaviour is slightly nonintuitive, as it lets the guest
write non-present entries without unshadowing only if the bits that
would have been the GFN are in fact a valid GFN (which happens to
include zero).  I think it''s OK, but needs a comment.

Please don''t change validate_gl1e or include level-1 shadow types in
check_for_data_page_unshadow().  Windows, in particular, keeps all sorts
of non-PTE-like values in PTE slots, and we can''t treat those as a
reason to unshadow.
>  To minimize the hurt to other performance of shadow, the patch tries
> to use the valid mfn check in the original code, please have a
> review. I''m not sure about the cost of the gfn_to_mfn(), and not
sure
> whether we may get some trade-off. If you have good ideas, please let
> us know.
gfn_to_mfn() is very cheap when shadow mode is being used. 

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems.
[Company #5334508: XenSource UK Ltd, reg''d c/o EC2Y 5EB, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 14:39 +0000 on 07 Dec (1197038381), Tim Deegan wrote:
> Hmmm.  The new behaviour is slightly nonintuitive, as it lets the guest
> write non-present entries without unshadowing only if the bits that
> would have been the GFN are in fact a valid GFN (which happens to
> include zero).  I think it''s OK, but needs a comment.
On second thoughts, since the emulator change will be doing this check
anyway, best to leave the validate functions alone entirely.

Cheers,

Tim. 

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems.
[Company #5334508: XenSource UK Ltd, reg''d c/o EC2Y 5EB, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, Tim
Heard from Kevin that the Linux kernel writes swap cache entries in swap cache
pages. And the swap cache entries contains only type and offset which seems not
contains valid mfn at all. Does the patch will hurt this? Is there any other
situations that guest write NON-PTE entries in the page tables?

Thanks
Xiaohui

-----Original Message-----
From: Tim Deegan [mailto:Tim.Deegan@citrix.com] 
Sent: 2007年12月7日 22:40
To: Xin, Xiaohui
Cc: xen-devel@lists.xensource.com; Kay, Allen M
Subject: Re: [Xen-devel][PATCH] unshadow the page table page which are used as
data page

Hi, 

At 21:12 +0800 on 07 Dec (1197061925), Xin, Xiaohui
wrote:
> Tim,
>  Attached is the updated patch which based on some part of your
> suggestion and some part of our new thoughts about it. We have
> re-checked the code path of the guest write emulate, found that in
> some extent(not all) the code checks the valid mfn for the guest
> written data. But maybe for the optimization, the code just check
> valide mfn when PRESENT bit exists. Maybe it can cover most of the
> cases, but not all, that''s what we have found in the vt-d iperf
test.
Hmmm.  The new behaviour is slightly nonintuitive, as it lets the guest
write non-present entries without unshadowing only if the bits that
would have been the GFN are in fact a valid GFN (which happens to
include zero).  I think it''s OK, but needs a comment.

Please don''t change validate_gl1e or include level-1 shadow types in
check_for_data_page_unshadow().  Windows, in particular, keeps all sorts
of non-PTE-like values in PTE slots, and we can''t treat those as a
reason to unshadow.
>  To minimize the hurt to other performance of shadow, the patch tries
> to use the valid mfn check in the original code, please have a
> review. I''m not sure about the cost of the gfn_to_mfn(), and not
sure
> whether we may get some trade-off. If you have good ideas, please let
> us know.
gfn_to_mfn() is very cheap when shadow mode is being used. 

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems.
[Company #5334508: XenSource UK Ltd, reg''d c/o EC2Y 5EB, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 12/5/07, Xin, Xiaohui <xiaohui.xin@intel.com>
wrote:
>
>
>
> The patch deals with the situation which guest OS uses unused page table
> pages as data pages and write data to them.
You mean some (HVM) OS do that kind of thing? Which OS is that?

That is horrible problem!! How come they get that idea???

Thanks,
Jun

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Xin, Xiaohui
>Sent: 2007年12月10日 10:48
>
>Hi, Tim
>Heard from Kevin that the Linux kernel writes swap cache 
>entries in swap cache pages. And the swap cache entries 
>contains only type and offset which seems not contains valid 
>mfn at all. Does the patch will hurt this? Is there any other 
>situations that guest write NON-PTE entries in the page tables?
It seems that unshadow has no correctness issue, and only
hurt the performance when unshadowed page is used in page
walk path again next time. At that time, shadow fault path will 
re-walk guest page table to find updated value.

In this context, if windows programs Non-PTE-like values in PTE
like swap usage in Linux, by clearing the present bit, the
performance to inject guest page fault for those special values
are hurt a bit. The difference is on a fast-path N/P check and a
full-path N/P check.

So we need balance the gain and loss here. Tim, how much
difference per your experience for a fast-path N/P? If not much,
maybe it''s worthy doing it then. :-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

It''s a very common thing to do with unused ptes. You can''t
really infer
anything from writes where _PAGE_PRESENT is clear.

 -- Keir

On 10/12/07 02:48, "Xin, Xiaohui" <xiaohui.xin@intel.com> wrote:
> Hi, Tim
> Heard from Kevin that the Linux kernel writes swap cache entries in swap
cache
> pages. And the swap cache entries contains only type and offset which seems
> not contains valid mfn at all. Does the patch will hurt this? Is there any
> other situations that guest write NON-PTE entries in the page tables?
> 
> Thanks
> Xiaohui
> 
> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> Sent: 2007年12月7日 22:40
> To: Xin, Xiaohui
> Cc: xen-devel@lists.xensource.com; Kay, Allen M
> Subject: Re: [Xen-devel][PATCH] unshadow the page table page which are used
as
> data page
> 
> Hi, 
> 
> At 21:12 +0800 on 07 Dec (1197061925), Xin, Xiaohui wrote:
>> Tim,
>>  Attached is the updated patch which based on some part of your
>> suggestion and some part of our new thoughts about it. We have
>> re-checked the code path of the guest write emulate, found that in
>> some extent(not all) the code checks the valid mfn for the guest
>> written data. But maybe for the optimization, the code just check
>> valide mfn when PRESENT bit exists. Maybe it can cover most of the
>> cases, but not all, that''s what we have found in the vt-d
iperf test.
> 
> Hmmm.  The new behaviour is slightly nonintuitive, as it lets the guest
> write non-present entries without unshadowing only if the bits that
> would have been the GFN are in fact a valid GFN (which happens to
> include zero).  I think it''s OK, but needs a comment.
> 
> Please don''t change validate_gl1e or include level-1 shadow types
in
> check_for_data_page_unshadow().  Windows, in particular, keeps all sorts
> of non-PTE-like values in PTE slots, and we can''t treat those as a
> reason to unshadow.
> 
>>  To minimize the hurt to other performance of shadow, the patch tries
>> to use the valid mfn check in the original code, please have a
>> review. I''m not sure about the cost of the gfn_to_mfn(), and
not sure
>> whether we may get some trade-off. If you have good ideas, please let
>> us know.
> 
> gfn_to_mfn() is very cheap when shadow mode is being used.
> 
> Cheers,
> 
> Tim.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Keir Fraser
>Sent: 2007年12月10日 16:25
>
>It''s a very common thing to do with unused ptes. You can''t
really infer
>anything from writes where _PAGE_PRESENT is clear.
>
> -- Keir
Will OS use those unused ptes in performance critical path? If not, this
change may be tolerable for the penalty on that type of usage, due to
unshadow, but benefit normal data page access a lot like encountered
in Xiaohui''s case? For _PAGE_PRESENT clear case, guest page fault
is anyway injected and shadow them just means we can accelerate the
injection by fast path...

Thanks,
Kevin
>
>On 10/12/07 02:48, "Xin, Xiaohui" <xiaohui.xin@intel.com>
wrote:
>
>> Hi, Tim
>> Heard from Kevin that the Linux kernel writes swap cache 
>entries in swap cache
>> pages. And the swap cache entries contains only type and 
>offset which seems
>> not contains valid mfn at all. Does the patch will hurt 
>this? Is there any
>> other situations that guest write NON-PTE entries in the page tables?
>> 
>> Thanks
>> Xiaohui
>> 
>> -----Original Message-----
>> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
>> Sent: 2007年12月7日 22:40
>> To: Xin, Xiaohui
>> Cc: xen-devel@lists.xensource.com; Kay, Allen M
>> Subject: Re: [Xen-devel][PATCH] unshadow the page table page 
>which are used as
>> data page
>> 
>> Hi, 
>> 
>> At 21:12 +0800 on 07 Dec (1197061925), Xin, Xiaohui wrote:
>>> Tim,
>>>  Attached is the updated patch which based on some part of your
>>> suggestion and some part of our new thoughts about it. We have
>>> re-checked the code path of the guest write emulate, found that in
>>> some extent(not all) the code checks the valid mfn for the guest
>>> written data. But maybe for the optimization, the code just check
>>> valide mfn when PRESENT bit exists. Maybe it can cover most of the
>>> cases, but not all, that''s what we have found in the vt-d 
>iperf test.
>> 
>> Hmmm.  The new behaviour is slightly nonintuitive, as it 
>lets the guest
>> write non-present entries without unshadowing only if the bits that
>> would have been the GFN are in fact a valid GFN (which happens to
>> include zero).  I think it''s OK, but needs a comment.
>> 
>> Please don''t change validate_gl1e or include level-1 shadow
types in
>> check_for_data_page_unshadow().  Windows, in particular, 
>keeps all sorts
>> of non-PTE-like values in PTE slots, and we can''t treat those
as a
>> reason to unshadow.
>> 
>>>  To minimize the hurt to other performance of shadow, the 
>patch tries
>>> to use the valid mfn check in the original code, please have a
>>> review. I''m not sure about the cost of the gfn_to_mfn(), 
>and not sure
>>> whether we may get some trade-off. If you have good ideas, 
>please let
>>> us know.
>> 
>> gfn_to_mfn() is very cheap when shadow mode is being used.
>> 
>> Cheers,
>> 
>> Tim.
>
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

So now we CANNOT do any optimization to the guest write (like what we have seen
that the guest just write pure data in an unused page table), right?

Thanks
Xiaohui

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
Sent: 2007年12月10日 16:25
To: Xin, Xiaohui; Tim Deegan
Cc: xen-devel@lists.xensource.com; Kay, Allen M
Subject: Re: [Xen-devel][PATCH] unshadow the page table page which areused as
data page

It''s a very common thing to do with unused ptes. You can''t
really infer
anything from writes where _PAGE_PRESENT is clear.

 -- Keir

On 10/12/07 02:48, "Xin, Xiaohui" <xiaohui.xin@intel.com> wrote:
> Hi, Tim
> Heard from Kevin that the Linux kernel writes swap cache entries in swap
cache
> pages. And the swap cache entries contains only type and offset which seems
> not contains valid mfn at all. Does the patch will hurt this? Is there any
> other situations that guest write NON-PTE entries in the page tables?
> 
> Thanks
> Xiaohui
> 
> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> Sent: 2007年12月7日 22:40
> To: Xin, Xiaohui
> Cc: xen-devel@lists.xensource.com; Kay, Allen M
> Subject: Re: [Xen-devel][PATCH] unshadow the page table page which are used
as
> data page
> 
> Hi, 
> 
> At 21:12 +0800 on 07 Dec (1197061925), Xin, Xiaohui wrote:
>> Tim,
>>  Attached is the updated patch which based on some part of your
>> suggestion and some part of our new thoughts about it. We have
>> re-checked the code path of the guest write emulate, found that in
>> some extent(not all) the code checks the valid mfn for the guest
>> written data. But maybe for the optimization, the code just check
>> valide mfn when PRESENT bit exists. Maybe it can cover most of the
>> cases, but not all, that''s what we have found in the vt-d
iperf test.
> 
> Hmmm.  The new behaviour is slightly nonintuitive, as it lets the guest
> write non-present entries without unshadowing only if the bits that
> would have been the GFN are in fact a valid GFN (which happens to
> include zero).  I think it''s OK, but needs a comment.
> 
> Please don''t change validate_gl1e or include level-1 shadow types
in
> check_for_data_page_unshadow().  Windows, in particular, keeps all sorts
> of non-PTE-like values in PTE slots, and we can''t treat those as a
> reason to unshadow.
> 
>>  To minimize the hurt to other performance of shadow, the patch tries
>> to use the valid mfn check in the original code, please have a
>> review. I''m not sure about the cost of the gfn_to_mfn(), and
not sure
>> whether we may get some trade-off. If you have good ideas, please let
>> us know.
> 
> gfn_to_mfn() is very cheap when shadow mode is being used.
> 
> Cheers,
> 
> Tim.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Unshadowing the whole page is probably not a very good idea. If bit 0 is set
then of course you can do a lot more. But if bit 0 is clear then there is
not necessarily a gpfn to pull out and test.

Perhaps if you knew that the page contained no or very few _PAGE_PRESENT
ptes then you could unshadow? But it''s probably not worth the
book-keeping.

Beyond that you''re into OS-specific heuristics.

 -- Keir

On 10/12/07 08:42, "Xin, Xiaohui" <xiaohui.xin@intel.com> wrote:
> So now we CANNOT do any optimization to the guest write (like what we have
> seen that the guest just write pure data in an unused page table), right?
> 
> Thanks
> Xiaohui
> 
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
> Sent: 2007年12月10日 16:25
> To: Xin, Xiaohui; Tim Deegan
> Cc: xen-devel@lists.xensource.com; Kay, Allen M
> Subject: Re: [Xen-devel][PATCH] unshadow the page table page which areused
as
> data page
> 
> It''s a very common thing to do with unused ptes. You
can''t really infer
> anything from writes where _PAGE_PRESENT is clear.
> 
>  -- Keir
> 
> On 10/12/07 02:48, "Xin, Xiaohui" <xiaohui.xin@intel.com>
wrote:
> 
>> Hi, Tim
>> Heard from Kevin that the Linux kernel writes swap cache entries in
swap
>> cache
>> pages. And the swap cache entries contains only type and offset which
seems
>> not contains valid mfn at all. Does the patch will hurt this? Is there
any
>> other situations that guest write NON-PTE entries in the page tables?
>> 
>> Thanks
>> Xiaohui
>> 
>> -----Original Message-----
>> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
>> Sent: 2007年12月7日 22:40
>> To: Xin, Xiaohui
>> Cc: xen-devel@lists.xensource.com; Kay, Allen M
>> Subject: Re: [Xen-devel][PATCH] unshadow the page table page which are
used
>> as
>> data page
>> 
>> Hi, 
>> 
>> At 21:12 +0800 on 07 Dec (1197061925), Xin, Xiaohui wrote:
>>> Tim,
>>>  Attached is the updated patch which based on some part of your
>>> suggestion and some part of our new thoughts about it. We have
>>> re-checked the code path of the guest write emulate, found that in
>>> some extent(not all) the code checks the valid mfn for the guest
>>> written data. But maybe for the optimization, the code just check
>>> valide mfn when PRESENT bit exists. Maybe it can cover most of the
>>> cases, but not all, that''s what we have found in the vt-d
iperf test.
>> 
>> Hmmm.  The new behaviour is slightly nonintuitive, as it lets the guest
>> write non-present entries without unshadowing only if the bits that
>> would have been the GFN are in fact a valid GFN (which happens to
>> include zero).  I think it''s OK, but needs a comment.
>> 
>> Please don''t change validate_gl1e or include level-1 shadow
types in
>> check_for_data_page_unshadow().  Windows, in particular, keeps all
sorts
>> of non-PTE-like values in PTE slots, and we can''t treat those
as a
>> reason to unshadow.
>> 
>>>  To minimize the hurt to other performance of shadow, the patch
tries
>>> to use the valid mfn check in the original code, please have a
>>> review. I''m not sure about the cost of the gfn_to_mfn(),
and not sure
>>> whether we may get some trade-off. If you have good ideas, please
let
>>> us know.
>> 
>> gfn_to_mfn() is very cheap when shadow mode is being used.
>> 
>> Cheers,
>> 
>> Tim.
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

On Mon, 2007-12-10 at 16:40 +0800, Tian, Kevin wrote:
> Will OS use those unused ptes in performance critical path?
There''s one extremely common case for Linux --- the special
"empty" pte,
"pte_none" (usually all-zeros), which is used in live memory areas to
represent unfaulted fault-on-demand memory.  munmap will write that
value all over live page tables, and in some workloads that''s a
performance-sensitive operation.


Other than that, fork and exit are the obvious cases which actively walk
page tables containing mixed present and swap ptes.  

For fork, any !PAGE_PRESENT swap ptes will be read to bump the refcount
on the swap page.  Any COW ptes in the same page table will be copied
and set read-only at the same time.

But that may not be too bad --- the swap entries in the existing page
table will only be getting read, not written, so they won''t trigger the
unsharing code.  The COW ptes will be written, but those are all valid
ptes so again they won''t trigger any undesirable unsharing.  

And in the new page table, we''ll be mixing writes of the valid ptes and
the swap entries... but the page table won''t be "live" by
then so I
don''t think we''ll have those ptes shadowed at all at that
point.


I _think_ we''re OK for exit, too --- that again only reads, rather than
writes, the swap ptes.


So as long as we''re not triggering an unshare when pte_none is written,
I don''t think this will hit Linux''s hot paths too much.  It
will
definitely hurt the "soft" swapout case where the page is getting
unhooked from one mm but is still in use elsewhere, but if we''re on
that
path then we''re already straying from peak performance.


Cheers,
 Stephen



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Stephen C. Tweedie [mailto:sct@redhat.com] 
>Sent: 2007年12月11日 1:07
>
>Hi,
>
>On Mon, 2007-12-10 at 16:40 +0800, Tian, Kevin wrote:
>
>> Will OS use those unused ptes in performance critical path?
>
>There''s one extremely common case for Linux --- the special 
>"empty" pte,
>"pte_none" (usually all-zeros), which is used in live memory areas
to
>represent unfaulted fault-on-demand memory.  munmap will write that
>value all over live page tables, and in some workloads that''s a
>performance-sensitive operation.
All 0''s means a valid mfn which falls out the unshadow check added
here.
>
>
>Other than that, fork and exit are the obvious cases which 
>actively walk
>page tables containing mixed present and swap ptes.  
Yes.
>
>For fork, any !PAGE_PRESENT swap ptes will be read to bump the refcount
>on the swap page.  Any COW ptes in the same page table will be copied
>and set read-only at the same time.
>
>But that may not be too bad --- the swap entries in the existing page
>table will only be getting read, not written, so they won''t trigger
the
>unsharing code.  The COW ptes will be written, but those are all valid
>ptes so again they won''t trigger any undesirable unsharing.  
Exactly.
>
>And in the new page table, we''ll be mixing writes of the valid ptes
and
>the swap entries... but the page table won''t be "live" by
then so I
>don''t think we''ll have those ptes shadowed at all at that
point.
>
>
>I _think_ we''re OK for exit, too --- that again only reads, rather
than
>writes, the swap ptes.
>
>
>So as long as we''re not triggering an unshare when pte_none is
written,
>I don''t think this will hit Linux''s hot paths too much. 
It will
>definitely hurt the "soft" swapout case where the page is getting
>unhooked from one mm but is still in use elsewhere, but if 
>we''re on that
>path then we''re already straying from peak performance.
>
>
Now I understand Keir''s concern. The unshadow has effect on all
512/1024
entries within same page, and incorrect unshadow heuristics on one valid
pte entry hurts whole 4M/2M virtual range which is too overkill. Maybe we
can pattern the whole page like Keir said on present bit, which however is
not worthy since data page has 50% chance to have present bit set.

So such heuristics seems more meaningful only when present bit is set 
which is already done in existing shadow code.

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel