Xen devel - Apr 2008 - [PATCH] [VTD] Separate VT-d page table from P2M table

Currently VT-d page table shares with P2M table, this patch supports
separate VT-d page table. 
1) add an option (vtd_share) to control whether VT-d page table shares
with P2M table or not.
2) in p2m_set_entry(), map/unmap page in VT-d page table. And also add a
function to free VT-d page table when tear down. 
3) allocate extra memory for VT-d page table when allocate memory for
doamin. 

In addition, when VT-d page table shares with P2M table,  remove the if
conditional judgement for iommu_flush(). Or there is opential issue when
using balloon driver. 

Signed-off-by: Weidong Han <weidong.han@intel.com>
Signed-off-by: Anthony Xu <anthony.xu@intel.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/4/08 09:36, "Han, Weidong" <weidong.han@intel.com> wrote:
> Currently VT-d page table shares with P2M table, this patch supports
> separate VT-d page table.
> 1) add an option (vtd_share) to control whether VT-d page table shares
> with P2M table or not.
Why? Is this just another option that noone will understand.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Separate VT-d page table is by default. Shared VT-d page table may be
easy and good in some cases. So we let them co-exist now. If shared VT-d
page table is not necessary and useless, we can remove it easily in
future.

Randy (weidong)

Keir Fraser wrote:
> On 22/4/08 09:36, "Han, Weidong" <weidong.han@intel.com>
wrote:
> 
>> Currently VT-d page table shares with P2M table, this patch supports
>> separate VT-d page table. 1) add an option (vtd_share) to control
>> whether VT-d page table shares with P2M table or not.
> 
> Why? Is this just another option that noone will understand.
> 
>  -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

What are the tradeoffs? One obvious tradeoff is that separate tables doubles
the memory overhead. What are the advantages of separate tables? I believe
currently we share the pagetables (right?). If so, why would we even
consider moving to separate tables?

 -- Keir

On 22/4/08 10:34, "Han, Weidong" <weidong.han@intel.com> wrote:
> Separate VT-d page table is by default. Shared VT-d page table may be
> easy and good in some cases. So we let them co-exist now. If shared VT-d
> page table is not necessary and useless, we can remove it easily in
> future.
> 
> Randy (weidong)
> 
> Keir Fraser wrote:
>> On 22/4/08 09:36, "Han, Weidong"
<weidong.han@intel.com> wrote:
>> 
>>> Currently VT-d page table shares with P2M table, this patch
supports
>>> separate VT-d page table. 1) add an option (vtd_share) to control
>>> whether VT-d page table shares with P2M table or not.
>> 
>> Why? Is this just another option that noone will understand.
>> 
>>  -- Keir
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Han, Weidong, le Tue 22 Apr 2008 17:34:11 +0800, a écrit
:
> Separate VT-d page table is by default. Shared VT-d page table may be
> easy and good in some cases. So we let them co-exist now. If shared VT-d
> page table is not necessary and useless, we can remove it easily in
> future.
But why unsharing it at all?  Is there any bonus except using more
memory? :)

Samuel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

If EPT supports 2MB page, we need separate VT-d table to let them work
together. What''s more, sharing makes coupling between VT-d page table
and p2m table. In case VT-d spec changes or p2m structure changes,
shared table will be broken.

Randy (Weidong)

Keir Fraser wrote:
> What are the tradeoffs? One obvious tradeoff is that separate tables
> doubles the memory overhead. What are the advantages of separate
> tables? I believe currently we share the pagetables (right?). If so,
> why would we even consider moving to separate tables?
> 
>  -- Keir
> 
> On 22/4/08 10:34, "Han, Weidong" <weidong.han@intel.com>
wrote:
> 
>> Separate VT-d page table is by default. Shared VT-d page table may be
>> easy and good in some cases. So we let them co-exist now. If shared
>> VT-d page table is not necessary and useless, we can remove it
>> easily in future. 
>> 
>> Randy (weidong)
>> 
>> Keir Fraser wrote:
>>> On 22/4/08 09:36, "Han, Weidong"
<weidong.han@intel.com> wrote:
>>> 
>>>> Currently VT-d page table shares with P2M table, this patch
>>>> supports separate VT-d page table. 1) add an option (vtd_share)
to
>>>> control whether VT-d page table shares with P2M table or not.
>>> 
>>> Why? Is this just another option that noone will understand.
>>> 
>>>  -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> If EPT supports 2MB page, we need separate VT-d table to let them work
> together. 
Are you saying that the VT-d 2MB page format is different from the EPT
2MB page format? Or that VTd does not support 2MB pages?

Ian
> What''s more, sharing makes coupling between VT-d page table
> and p2m table. In case VT-d spec changes or p2m structure changes,
> shared table will be broken.
> Randy (Weidong)
> 
> Keir Fraser wrote:
> > What are the tradeoffs? One obvious tradeoff is that separate tables
> > doubles the memory overhead. What are the advantages of separate
> > tables? I believe currently we share the pagetables (right?). If so,
> > why would we even consider moving to separate tables?
> >
> >  -- Keir
> >
> > On 22/4/08 10:34, "Han, Weidong"
<weidong.han@intel.com> wrote:
> >
> >> Separate VT-d page table is by default. Shared VT-d page table may
> be
> >> easy and good in some cases. So we let them co-exist now. If
shared
> >> VT-d page table is not necessary and useless, we can remove it
> >> easily in future.
> >>
> >> Randy (weidong)
> >>
> >> Keir Fraser wrote:
> >>> On 22/4/08 09:36, "Han, Weidong"
<weidong.han@intel.com> wrote:
> >>>
> >>>> Currently VT-d page table shares with P2M table, this
patch
> >>>> supports separate VT-d page table. 1) add an option
(vtd_share)
to
> >>>> control whether VT-d page table shares with P2M table or
not.
> >>>
> >>> Why? Is this just another option that noone will understand.
> >>>
> >>>  -- Keir
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Pratt wrote:
>> If EPT supports 2MB page, we need separate VT-d table to let them
>> work together.
> 
> Are you saying that the VT-d 2MB page format is different from the EPT
> 2MB page format? Or that VTd does not support 2MB pages?
> 
Now VTd does not support 2MB pages.
> Ian
> 
>> What''s more, sharing makes coupling between VT-d page table
>> and p2m table. In case VT-d spec changes or p2m structure changes,
>> shared table will be broken.
> 
>> Randy (Weidong)
>> 
>> Keir Fraser wrote:
>>> What are the tradeoffs? One obvious tradeoff is that separate
tables
>>> doubles the memory overhead. What are the advantages of separate
>>> tables? I believe currently we share the pagetables (right?). If
so,
>>> why would we even consider moving to separate tables?
>>> 
>>>  -- Keir
>>> 
>>> On 22/4/08 10:34, "Han, Weidong"
<weidong.han@intel.com> wrote:
>>> 
>>>> Separate VT-d page table is by default. Shared VT-d page table
may
>>>> be easy and good in some cases. So we let them co-exist now. If
>>>> shared VT-d page table is not necessary and useless, we can
remove
>>>> it easily in future. 
>>>> 
>>>> Randy (weidong)
>>>> 
>>>> Keir Fraser wrote:
>>>>> On 22/4/08 09:36, "Han, Weidong"
<weidong.han@intel.com> wrote:
>>>>> 
>>>>>> Currently VT-d page table shares with P2M table, this
patch
>>>>>> supports separate VT-d page table. 1) add an option
(vtd_share)
>>>>>> to control whether VT-d page table shares with P2M
table or not.
>>>>> 
>>>>> Why? Is this just another option that noone will
understand.
>>>>> 
>>>>>  -- Keir
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/4/08 13:17, "Han, Weidong" <weidong.han@intel.com> wrote:
>> Are you saying that the VT-d 2MB page format is different from the EPT
>> 2MB page format? Or that VTd does not support 2MB pages?
>> 
> 
> Now VTd does not support 2MB pages.
Then don''t allow EPT 2MB mappings for domains which have passthru
devices?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 22/4/08 13:17, "Han, Weidong" <weidong.han@intel.com>
wrote:
> 
>>> Are you saying that the VT-d 2MB page format is different from the
>>> EPT 2MB page format? Or that VTd does not support 2MB pages?
>>> 
>> 
>> Now VTd does not support 2MB pages.
> 
> Then don''t allow EPT 2MB mappings for domains which have passthru
> devices? 
> 
No, EPT is 2MB, VT-d page table is 4KB.

Randy (Weidong)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[Weidong Han]
> Keir Fraser wrote:
>> On 22/4/08 13:17, "Han, Weidong"
<weidong.han@intel.com> wrote:
>> 
>>>> Are you saying that the VT-d 2MB page format is different from
>>>> the EPT 2MB page format? Or that VTd does not support 2MB
pages?
>>>> 
>>> 
>>> Now VTd does not support 2MB pages.
>> 
>> Then don''t allow EPT 2MB mappings for domains which have
passthru
>> devices?
>> 
> No, EPT is 2MB, VT-d page table is 4KB.
So EPT only supports 2MB mappings?  Not 4KB?  That doesn''t sound right.

Another possible inompatibility: The VT-d chipset I''m using only
allows 4-level page tables.  Not sure if similar restrictions might
apply to EPT.  Also, in another project I worked on I found it
advantageous to emulate superpages in the guest even if this was not
supported by the VT hardware --- lower memory footprint, quicker table
lookups.  Such optimizations might be another reason for separating
the tables.

That said, I would really disfavor separating the tables.  There are
enough memory management structures as it is right now.  If the tables
really, really, really need to be separated then don''t make it a boot
time option.  The capabilities of VT-d (and probably also EPT) is
readily available at initialization time, and that''s where the
decision for sharing or not should be done.

	eSk


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Espen Skoglund wrote:
> [Weidong Han]
>> Keir Fraser wrote:
>>> On 22/4/08 13:17, "Han, Weidong"
<weidong.han@intel.com> wrote:
>>> 
>>>>> Are you saying that the VT-d 2MB page format is different
from
>>>>> the EPT 2MB page format? Or that VTd does not support 2MB
pages?
>>>>> 
>>>> 
>>>> Now VTd does not support 2MB pages.
>>> 
>>> Then don''t allow EPT 2MB mappings for domains which have
passthru
>>> devices? 
>>> 
> 
>> No, EPT is 2MB, VT-d page table is 4KB.
> 
> So EPT only supports 2MB mappings?  Not 4KB?  That doesn''t sound
> right. 
No, EPT can support both 4KB and 2MB pages. 

> 
> Another possible inompatibility: The VT-d chipset I''m using only
> allows 4-level page tables.  Not sure if similar restrictions might
> apply to EPT.  Also, in another project I worked on I found it
> advantageous to emulate superpages in the guest even if this was not
> supported by the VT hardware --- lower memory footprint, quicker table
> lookups.  Such optimizations might be another reason for separating
> the tables.
> 
> That said, I would really disfavor separating the tables.  There are
> enough memory management structures as it is right now.  If the tables
> really, really, really need to be separated then don''t make it a
boot
> time option.  The capabilities of VT-d (and probably also EPT) is
> readily available at initialization time, and that''s where the
> decision for sharing or not should be done.
I think the option for sharing or not will be a temporary thing. If
community considers shared table or separate table is useless, we can
remove the one easily.

Randy (Weidong)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>On 22/4/08 13:17, "Han, Weidong" <weidong.han@intel.com>
wrote:
>
>>> Are you saying that the VT-d 2MB page format is different from the
EPT
>>> 2MB page format? Or that VTd does not support 2MB pages?
>>> 
>> 
>> Now VTd does not support 2MB pages.
>
>Then don''t allow EPT 2MB mappings for domains which have 
>passthru devices?
>
EPT 2M page is already supported while VT-d 2M not, so for we have to
seperate the 2 guys if we want to use 2M page in EPT.
and of course we share the page tables if both sides use 4K pages.
-Xin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/4/08 16:43, "Li, Xin B" <xin.b.li@intel.com> wrote:
>> Then don''t allow EPT 2MB mappings for domains which have
>> passthru devices?
>> 
> 
> EPT 2M page is already supported while VT-d 2M not, so for we have to
> seperate the 2 guys if we want to use 2M page in EPT.
> and of course we share the page tables if both sides use 4K pages.
Just share the tables and put up with not having 2MB mappings available to
you if a guest has devices passed through to it? Doesn''t sound so bad
to me,
and you can get the performance back down the line by fixing the VT-d
hardware to understand superpage mappings. We''re going to end up with a
silly number of p2m structures otherwise.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

Attached patch removes the shared option, and only supports separated
VT-d page table.

Randy (Weidong)

Keir Fraser wrote:
> On 22/4/08 16:43, "Li, Xin B" <xin.b.li@intel.com> wrote:
> 
>>> Then don''t allow EPT 2MB mappings for domains which have
passthru
>>> devices? 
>>> 
>> 
>> EPT 2M page is already supported while VT-d 2M not, so for we have to
>> seperate the 2 guys if we want to use 2M page in EPT.
>> and of course we share the page tables if both sides use 4K pages.
> 
> Just share the tables and put up with not having 2MB mappings
> available to you if a guest has devices passed through to it?
Doesn''t
> sound so bad to me, and you can get the performance back down the
> line by fixing the VT-d hardware to understand superpage mappings.
> We''re going to end up with a silly number of p2m structures
otherwise.
> 
>  -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel