I do not have a clear answer about use of super pages in VT-D for the IOMMU. Here is from the release notes for 4.1: http://blog.xen.org/index.php/2011/03/25/xen-4-1-releases/ Further, support for EPT/VTd 1GB/2MB super pages has been added to Xen, reducing the TLB overhead. EPT/VTd page table sharing simplifies the support for Intel''s IOMMU<http://en.wikipedia.org/wiki/IOMMU> by allowing the CPU''s Enhanced Page Table to be directly utilized by the VTd IOMMU. Based on this check in, I also see the output from Xen on my system showing HAP page sizes as 4kB, 2MB, and 1GB - but it does not say which one is in use. http://lists.xen.org/archives/html/xen-changelog/2012-03/msg00140.html 1. Are superpages (2MB, 1GB) enabled by default in Xen? If enabled, what size? 2. On Intel, if superpages are enabled, is that used for the IOMMU? 3. If it is not enabled by default, how does one enable it? Thanks, Santosh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
At 09:18 -0700 on 13 Jul (1342171126), Santosh Jodh wrote:> 1. Are superpages (2MB, 1GB) enabled by default in Xen? If enabled, what size?Yes, they are enabled; for guest memory, Xen uses whatever the tools ask for, which (IIRC) is the largest available for HVM (falling back to smaller pages if no large ones can be found) and 2MB for PV if the guest supports PV superpages.> 2. On Intel, if superpages are enabled, is that used for the IOMMU?Yes. In fact the IOMMU code checks for IOMMU superpage support separately; if the IOMMU and the MMU have the same superpage sizes, they share a table; otherwise the p2m is duplicated in the IOMMU''s format. Cheers, Tim.
Thanks Tim. I understand the value of using superpages for IOMMU. What are the other advantages of sharing the table between the MMU and IOMMU (other than saving table memory)? Thanks, Santosh -----Original Message----- From: Tim Deegan [mailto:tim@xen.org] Sent: Thursday, July 19, 2012 3:23 AM To: Santosh Jodh Cc: xen-devel@lists.xen.org; xiantao.zhang@intel.com; jun.nakajima@intel.com Subject: Re: [Xen-devel] Superpages for VT-D At 09:18 -0700 on 13 Jul (1342171126), Santosh Jodh wrote:> 1. Are superpages (2MB, 1GB) enabled by default in Xen? If enabled, what size?Yes, they are enabled; for guest memory, Xen uses whatever the tools ask for, which (IIRC) is the largest available for HVM (falling back to smaller pages if no large ones can be found) and 2MB for PV if the guest supports PV superpages.> 2. On Intel, if superpages are enabled, is that used for the IOMMU?Yes. In fact the IOMMU code checks for IOMMU superpage support separately; if the IOMMU and the MMU have the same superpage sizes, they share a table; otherwise the p2m is duplicated in the IOMMU''s format. Cheers, Tim.
At 07:01 -0700 on 19 Jul (1342681270), Santosh Jodh wrote:> I understand the value of using superpages for IOMMU. What are the > other advantages of sharing the table between the MMU and IOMMU (other > than saving table memory)?It''s mostly the memory, but also it''s faster to update one table than two, and we don''t have to worry about them getting out of sync with each other. I don''t think there are any deeper reasons. Cheers, Tim.
On Thu, Jul 19, 2012 at 7:53 AM, Tim Deegan <tim@xen.org> wrote:> At 07:01 -0700 on 19 Jul (1342681270), Santosh Jodh wrote: > > I understand the value of using superpages for IOMMU. What are the > > other advantages of sharing the table between the MMU and IOMMU (other > > than saving table memory)? > > It''s mostly the memory, but also it''s faster to update one table than > two, and we don''t have to worry about them getting out of sync with each > other. I don''t think there are any deeper reasons. > > Cheers, > > Tim. > >- The code can be simpler because you don''t need to maintain the same info "guest physical -> host physical" in two different places. - Maybe cache utilization can be better -- Jun Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Is there an easy way to see the mapping sizes being used for a VM in the IOMMU? Thanks, Santosh From: Nakajima, Jun [mailto:jun.nakajima@intel.com] Sent: Friday, July 20, 2012 2:38 PM To: Tim (Xen.org) Cc: Santosh Jodh; xen-devel@lists.xen.org; xiantao.zhang@intel.com Subject: Re: [Xen-devel] Superpages for VT-D On Thu, Jul 19, 2012 at 7:53 AM, Tim Deegan <tim@xen.org<mailto:tim@xen.org>> wrote: At 07:01 -0700 on 19 Jul (1342681270), Santosh Jodh wrote:> I understand the value of using superpages for IOMMU. What are the > other advantages of sharing the table between the MMU and IOMMU (other > than saving table memory)?It''s mostly the memory, but also it''s faster to update one table than two, and we don''t have to worry about them getting out of sync with each other. I don''t think there are any deeper reasons. Cheers, Tim. - The code can be simpler because you don''t need to maintain the same info "guest physical -> host physical" in two different places. - Maybe cache utilization can be better -- Jun Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
At 10:56 -0700 on 23 Jul (1343040996), Santosh Jodh wrote:> Is there an easy way to see the mapping sizes being used for a VM in the IOMMU?Doesn''t look like it. If the IOMMU is sharing the EPT table you can use the ''D'' debug-key to dump the EPT tables, but I don''t think there''s an equivalent for the IOMMU tables. I''m sure it wouldn''t be too hard to do, though. Tim.
I am going to try to add this support. It looks like a new iommu_ops handler would be needed that would do the actual work of dumping the entries - one for AMD and one for Intel. Am I reading this correctly? Or is it better to get the root_table + paging_mode (for AMD) and pgd_maddr + agaw (for Intel) and then do a generic dump? Thanks, Santosh -----Original Message----- From: Tim Deegan [mailto:tim@xen.org] Sent: Tuesday, July 24, 2012 12:42 PM To: Santosh Jodh Cc: Nakajima, Jun; xiantao.zhang@intel.com; xen-devel@lists.xen.org Subject: Re: [Xen-devel] Superpages for VT-D At 10:56 -0700 on 23 Jul (1343040996), Santosh Jodh wrote:> Is there an easy way to see the mapping sizes being used for a VM in the IOMMU?Doesn''t look like it. If the IOMMU is sharing the EPT table you can use the ''D'' debug-key to dump the EPT tables, but I don''t think there''s an equivalent for the IOMMU tables. I''m sure it wouldn''t be too hard to do, though. Tim.
At 13:44 -0700 on 31 Jul (1343742260), Santosh Jodh wrote:> I am going to try to add this support. > > It looks like a new iommu_ops handler would be needed that would do > the actual work of dumping the entries - one for AMD and one for > Intel. Am I reading this correctly?I think that''s correct.> Or is it better to get the root_table + paging_mode (for AMD) and > pgd_maddr + agaw (for Intel) and then do a generic dump?No; the iommu tabels are sufficiebntly arch-specific that it''s best to add arch-specific dump routines. Cheers, Tim.
Thanks for confirming. BTW, would the IOMMU ever have entries above domains max mapped pfn? -----Original Message----- From: Tim Deegan [mailto:tim@xen.org] Sent: Thursday, August 02, 2012 4:11 AM To: Santosh Jodh Cc: xiantao.zhang@intel.com; Nakajima, Jun; xen-devel@lists.xen.org Subject: Re: [Xen-devel] Superpages for VT-D At 13:44 -0700 on 31 Jul (1343742260), Santosh Jodh wrote:> I am going to try to add this support. > > It looks like a new iommu_ops handler would be needed that would do > the actual work of dumping the entries - one for AMD and one for > Intel. Am I reading this correctly?I think that''s correct.> Or is it better to get the root_table + paging_mode (for AMD) and > pgd_maddr + agaw (for Intel) and then do a generic dump?No; the iommu tabels are sufficiebntly arch-specific that it''s best to add arch-specific dump routines. Cheers, Tim.
At 08:56 -0700 on 02 Aug (1343897810), Santosh Jodh wrote:> Thanks for confirming. BTW, would the IOMMU ever have entries above > domains max mapped pfn?I don''t believe so, but potentially some grant-table-style operations might want to map pages in for DMA that don''t need to be mapped in for CPU access. But if you''re writing a dump routine from scratch it should be easy enough (and more efficient) to dump all the entries with a depth-first pass over the trie, rather than individually querying all frames up to max_mapped_pfn. Tim.> -----Original Message----- > From: Tim Deegan [mailto:tim@xen.org] > Sent: Thursday, August 02, 2012 4:11 AM > To: Santosh Jodh > Cc: xiantao.zhang@intel.com; Nakajima, Jun; xen-devel@lists.xen.org > Subject: Re: [Xen-devel] Superpages for VT-D > > At 13:44 -0700 on 31 Jul (1343742260), Santosh Jodh wrote: > > I am going to try to add this support. > > > > It looks like a new iommu_ops handler would be needed that would do > > the actual work of dumping the entries - one for AMD and one for > > Intel. Am I reading this correctly? > > I think that''s correct. > > > Or is it better to get the root_table + paging_mode (for AMD) and > > pgd_maddr + agaw (for Intel) and then do a generic dump? > > No; the iommu tabels are sufficiebntly arch-specific that it''s best to add arch-specific dump routines. > > Cheers, > > Tim. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel