Tom Rotenberg
2009-Nov-29 17:56 UTC
[Xen-devel] Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings
Hi All, I am using a Dell E6400 machine, with IGD (Intel graphics device), which i''m trying to pass-through into a domU with Windows XP. When i assign only the 00:02.0 device, it seems to work ok, but when i try to assign the 00:02.1 device, the whole machine seems to freeze form more then 2 mins, causing a lot of messgaes in dom0, such as: SATA doesn''t respond, etc. After some analysis i made, i discovered the followings: * When calling the (Intel) iommu_assign_device, it checks if the device has any details in the RMRR table, and if so it calls the ''iommu_prepare_rmrr_dev()'' function. * The device 00:02.1 has an RMRR information, which details the whole area between: 0x7dc00000 to 0x80000000 (it contains ~9000 pages) * For each such page, Xen calls the ''intel_iommu_map_page()'' function. This causes the hypercall to take a lot of time to process, and this makes dom0 to stop getting responds from devices, such as SATA, etc,, which may lead to a whole system crash. My question are: 1. Can this hypercall be in some way preemptive, so it won''t stuck the whole system (i noticed that there are other hypercalls, which are preemptive) ? 2. Why can''t the mapping of the pages, be done in batches of pages, instead of mapping each page in a separate call to ''intel_iommu_map_page()'' ? 3. i noticed that for USB devices, the code there ignores the USB RMRR... can this be done also for the 00:02.1 device? Tom _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Han, Weidong
2009-Nov-30 02:30 UTC
[Xen-devel] RE: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings
Tom, Yes, you can add an API to map batches of pages instead of mapping each page in a separate call. RMRR regions of USB controller is ignored because they won''t be used in guest. The RMRR 0x7dc00000 to 0x80000000 is shared by 00:02.0 and 00:02.1, it will be used by IGD in guest. But you can have a try to ignore it for 00:02.1. Regards, Weidong -----Original Message----- From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] Sent: Monday, November 30, 2009 1:56 AM To: xen-devel@lists.xensource.com Cc: Han, Weidong Subject: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings Hi All, I am using a Dell E6400 machine, with IGD (Intel graphics device), which i''m trying to pass-through into a domU with Windows XP. When i assign only the 00:02.0 device, it seems to work ok, but when i try to assign the 00:02.1 device, the whole machine seems to freeze form more then 2 mins, causing a lot of messgaes in dom0, such as: SATA doesn''t respond, etc. After some analysis i made, i discovered the followings: * When calling the (Intel) iommu_assign_device, it checks if the device has any details in the RMRR table, and if so it calls the ''iommu_prepare_rmrr_dev()'' function. * The device 00:02.1 has an RMRR information, which details the whole area between: 0x7dc00000 to 0x80000000 (it contains ~9000 pages) * For each such page, Xen calls the ''intel_iommu_map_page()'' function. This causes the hypercall to take a lot of time to process, and this makes dom0 to stop getting responds from devices, such as SATA, etc,, which may lead to a whole system crash. My question are: 1. Can this hypercall be in some way preemptive, so it won''t stuck the whole system (i noticed that there are other hypercalls, which are preemptive) ? 2. Why can''t the mapping of the pages, be done in batches of pages, instead of mapping each page in a separate call to ''intel_iommu_map_page()'' ? 3. i noticed that for USB devices, the code there ignores the USB RMRR... can this be done also for the 00:02.1 device? Tom _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tom Rotenberg
2009-Nov-30 11:17 UTC
[Xen-devel] Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings
Can u please guid me a little bit, on how to write such function which
will work on a batch of pages?
Should i just enclose the:
"
...
pg_maddr = addr_to_dma_page_maddr(d, (paddr_t)gfn << PAGE_SHIFT_4K,
1);
if ( pg_maddr == 0 )
{
spin_unlock(&hd->mapping_lock);
return -ENOMEM;
}
page = (struct dma_pte *)map_vtd_domain_page(pg_maddr);
pte = page + (gfn & LEVEL_MASK);
pte_present = dma_pte_present(*pte);
dma_set_pte_addr(*pte, (paddr_t)mfn << PAGE_SHIFT_4K);
dma_set_pte_prot(*pte, DMA_PTE_READ | DMA_PTE_WRITE);
/* Set the SNP on leaf page table if Snoop Control available */
if ( iommu_snoop )
dma_set_pte_snp(*pte);
...
"
in a loop?
On Mon, Nov 30, 2009 at 4:30 AM, Han, Weidong <weidong.han@intel.com>
wrote:> Tom,
>
> Yes, you can add an API to map batches of pages instead of mapping each
page in a separate call.
>
> RMRR regions of USB controller is ignored because they won''t be
used in guest. The RMRR 0x7dc00000 to 0x80000000 is shared by 00:02.0 and
00:02.1, it will be used by IGD in guest. But you can have a try to ignore it
for 00:02.1.
>
>
> Regards,
> Weidong
>
> -----Original Message-----
> From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com]
> Sent: Monday, November 30, 2009 1:56 AM
> To: xen-devel@lists.xensource.com
> Cc: Han, Weidong
> Subject: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines,
causes Xen to freeze for >2mins because of RMRR mappings
>
> Hi All,
>
> I am using a Dell E6400 machine, with IGD (Intel graphics device),
> which i''m trying to pass-through into a domU with Windows XP. When
i
> assign only the 00:02.0 device, it seems to work ok, but when i try to
> assign the 00:02.1 device, the whole machine seems to freeze form more
> then 2 mins, causing a lot of messgaes in dom0, such as: SATA
doesn''t
> respond, etc.
>
> After some analysis i made, i discovered the followings:
> * When calling the (Intel) iommu_assign_device, it checks if the
> device has any details in the RMRR table, and if so it calls the
> ''iommu_prepare_rmrr_dev()'' function.
> * The device 00:02.1 has an RMRR information, which details the whole
> area between: 0x7dc00000 to 0x80000000 (it contains ~9000 pages)
> * For each such page, Xen calls the
''intel_iommu_map_page()'' function.
> This causes the hypercall to take a lot of time to process, and this
> makes dom0 to stop getting responds from devices, such as SATA, etc,,
> which may lead to a whole system crash.
>
> My question are:
> 1. Can this hypercall be in some way preemptive, so it won''t stuck
the
> whole system (i noticed that there are other hypercalls, which are
> preemptive) ?
> 2. Why can''t the mapping of the pages, be done in batches of
pages,
> instead of mapping each page in a separate call to
> ''intel_iommu_map_page()'' ?
> 3. i noticed that for USB devices, the code there ignores the USB
> RMRR... can this be done also for the 00:02.1 device?
>
> Tom
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Han, Weidong
2009-Nov-30 14:54 UTC
[Xen-devel] RE: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings
You also need to include below code in the loop:
iommu_flush_cache_entry(pte);
unmap_vtd_domain_page(page);
At the end, need to flush iotlb for all pages:
if ( iommu_flush_iotlb_psi(iommu, domain_iommu_domid(d),
(paddr_t)gfn << PAGE_SHIFT_4K, pages,
----> "pages" means page number
!pte_present, flush_dev_iotlb) )
iommu_flush_write_buffer(iommu);
Regards,
Weidong
-----Original Message-----
From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com]
Sent: Monday, November 30, 2009 7:17 PM
To: Han, Weidong
Cc: xen-devel@lists.xensource.com
Subject: Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes
Xen to freeze for >2mins because of RMRR mappings
Can u please guid me a little bit, on how to write such function which
will work on a batch of pages?
Should i just enclose the:
"
...
pg_maddr = addr_to_dma_page_maddr(d, (paddr_t)gfn << PAGE_SHIFT_4K,
1);
if ( pg_maddr == 0 )
{
spin_unlock(&hd->mapping_lock);
return -ENOMEM;
}
page = (struct dma_pte *)map_vtd_domain_page(pg_maddr);
pte = page + (gfn & LEVEL_MASK);
pte_present = dma_pte_present(*pte);
dma_set_pte_addr(*pte, (paddr_t)mfn << PAGE_SHIFT_4K);
dma_set_pte_prot(*pte, DMA_PTE_READ | DMA_PTE_WRITE);
/* Set the SNP on leaf page table if Snoop Control available */
if ( iommu_snoop )
dma_set_pte_snp(*pte);
...
"
in a loop?
On Mon, Nov 30, 2009 at 4:30 AM, Han, Weidong <weidong.han@intel.com>
wrote:> Tom,
>
> Yes, you can add an API to map batches of pages instead of mapping each
page in a separate call.
>
> RMRR regions of USB controller is ignored because they won''t be
used in guest. The RMRR 0x7dc00000 to 0x80000000 is shared by 00:02.0 and
00:02.1, it will be used by IGD in guest. But you can have a try to ignore it
for 00:02.1.
>
>
> Regards,
> Weidong
>
> -----Original Message-----
> From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com]
> Sent: Monday, November 30, 2009 1:56 AM
> To: xen-devel@lists.xensource.com
> Cc: Han, Weidong
> Subject: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines,
causes Xen to freeze for >2mins because of RMRR mappings
>
> Hi All,
>
> I am using a Dell E6400 machine, with IGD (Intel graphics device),
> which i''m trying to pass-through into a domU with Windows XP. When
i
> assign only the 00:02.0 device, it seems to work ok, but when i try to
> assign the 00:02.1 device, the whole machine seems to freeze form more
> then 2 mins, causing a lot of messgaes in dom0, such as: SATA
doesn''t
> respond, etc.
>
> After some analysis i made, i discovered the followings:
> * When calling the (Intel) iommu_assign_device, it checks if the
> device has any details in the RMRR table, and if so it calls the
> ''iommu_prepare_rmrr_dev()'' function.
> * The device 00:02.1 has an RMRR information, which details the whole
> area between: 0x7dc00000 to 0x80000000 (it contains ~9000 pages)
> * For each such page, Xen calls the
''intel_iommu_map_page()'' function.
> This causes the hypercall to take a lot of time to process, and this
> makes dom0 to stop getting responds from devices, such as SATA, etc,,
> which may lead to a whole system crash.
>
> My question are:
> 1. Can this hypercall be in some way preemptive, so it won''t stuck
the
> whole system (i noticed that there are other hypercalls, which are
> preemptive) ?
> 2. Why can''t the mapping of the pages, be done in batches of
pages,
> instead of mapping each page in a separate call to
> ''intel_iommu_map_page()'' ?
> 3. i noticed that for USB devices, the code there ignores the USB
> RMRR... can this be done also for the 00:02.1 device?
>
> Tom
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Kay, Allen M
2009-Nov-30 19:45 UTC
RE: [Xen-devel] RE: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings
While you are at it, please also add memory attribute to the API too. In the future, we might want to map pages as read only or some other attribute instead of just RW. -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Han, Weidong Sent: Sunday, November 29, 2009 6:30 PM To: Tom Rotenberg; xen-devel@lists.xensource.com Subject: [Xen-devel] RE: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings Tom, Yes, you can add an API to map batches of pages instead of mapping each page in a separate call. RMRR regions of USB controller is ignored because they won''t be used in guest. The RMRR 0x7dc00000 to 0x80000000 is shared by 00:02.0 and 00:02.1, it will be used by IGD in guest. But you can have a try to ignore it for 00:02.1. Regards, Weidong -----Original Message----- From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] Sent: Monday, November 30, 2009 1:56 AM To: xen-devel@lists.xensource.com Cc: Han, Weidong Subject: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings Hi All, I am using a Dell E6400 machine, with IGD (Intel graphics device), which i''m trying to pass-through into a domU with Windows XP. When i assign only the 00:02.0 device, it seems to work ok, but when i try to assign the 00:02.1 device, the whole machine seems to freeze form more then 2 mins, causing a lot of messgaes in dom0, such as: SATA doesn''t respond, etc. After some analysis i made, i discovered the followings: * When calling the (Intel) iommu_assign_device, it checks if the device has any details in the RMRR table, and if so it calls the ''iommu_prepare_rmrr_dev()'' function. * The device 00:02.1 has an RMRR information, which details the whole area between: 0x7dc00000 to 0x80000000 (it contains ~9000 pages) * For each such page, Xen calls the ''intel_iommu_map_page()'' function. This causes the hypercall to take a lot of time to process, and this makes dom0 to stop getting responds from devices, such as SATA, etc,, which may lead to a whole system crash. My question are: 1. Can this hypercall be in some way preemptive, so it won''t stuck the whole system (i noticed that there are other hypercalls, which are preemptive) ? 2. Why can''t the mapping of the pages, be done in batches of pages, instead of mapping each page in a separate call to ''intel_iommu_map_page()'' ? 3. i noticed that for USB devices, the code there ignores the USB RMRR... can this be done also for the 00:02.1 device? Tom _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tom Rotenberg
2009-Dec-02 10:57 UTC
[Xen-devel] Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings
And, the other question: how can i make this hypercall preemptive? as it may still take too much time for an hypercall (taking into account, it needs to map ~10.000 pages...) On Mon, Nov 30, 2009 at 4:54 PM, Han, Weidong <weidong.han@intel.com> wrote:> You also need to include below code in the loop: > iommu_flush_cache_entry(pte); > unmap_vtd_domain_page(page); > > At the end, need to flush iotlb for all pages: > if ( iommu_flush_iotlb_psi(iommu, domain_iommu_domid(d), > (paddr_t)gfn << PAGE_SHIFT_4K, pages, ----> "pages" means page number > !pte_present, flush_dev_iotlb) ) > iommu_flush_write_buffer(iommu); > > Regards, > Weidong > > -----Original Message----- > From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] > Sent: Monday, November 30, 2009 7:17 PM > To: Han, Weidong > Cc: xen-devel@lists.xensource.com > Subject: Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings > > Can u please guid me a little bit, on how to write such function which > will work on a batch of pages? > Should i just enclose the: > " > ... > pg_maddr = addr_to_dma_page_maddr(d, (paddr_t)gfn << PAGE_SHIFT_4K, 1); > if ( pg_maddr == 0 ) > { > spin_unlock(&hd->mapping_lock); > return -ENOMEM; > } > page = (struct dma_pte *)map_vtd_domain_page(pg_maddr); > pte = page + (gfn & LEVEL_MASK); > pte_present = dma_pte_present(*pte); > dma_set_pte_addr(*pte, (paddr_t)mfn << PAGE_SHIFT_4K); > dma_set_pte_prot(*pte, DMA_PTE_READ | DMA_PTE_WRITE); > > /* Set the SNP on leaf page table if Snoop Control available */ > if ( iommu_snoop ) > dma_set_pte_snp(*pte); > ... > " > > in a loop? > > On Mon, Nov 30, 2009 at 4:30 AM, Han, Weidong <weidong.han@intel.com> wrote: >> Tom, >> >> Yes, you can add an API to map batches of pages instead of mapping each page in a separate call. >> >> RMRR regions of USB controller is ignored because they won''t be used in guest. The RMRR 0x7dc00000 to 0x80000000 is shared by 00:02.0 and 00:02.1, it will be used by IGD in guest. But you can have a try to ignore it for 00:02.1. >> >> >> Regards, >> Weidong >> >> -----Original Message----- >> From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] >> Sent: Monday, November 30, 2009 1:56 AM >> To: xen-devel@lists.xensource.com >> Cc: Han, Weidong >> Subject: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings >> >> Hi All, >> >> I am using a Dell E6400 machine, with IGD (Intel graphics device), >> which i''m trying to pass-through into a domU with Windows XP. When i >> assign only the 00:02.0 device, it seems to work ok, but when i try to >> assign the 00:02.1 device, the whole machine seems to freeze form more >> then 2 mins, causing a lot of messgaes in dom0, such as: SATA doesn''t >> respond, etc. >> >> After some analysis i made, i discovered the followings: >> * When calling the (Intel) iommu_assign_device, it checks if the >> device has any details in the RMRR table, and if so it calls the >> ''iommu_prepare_rmrr_dev()'' function. >> * The device 00:02.1 has an RMRR information, which details the whole >> area between: 0x7dc00000 to 0x80000000 (it contains ~9000 pages) >> * For each such page, Xen calls the ''intel_iommu_map_page()'' function. >> This causes the hypercall to take a lot of time to process, and this >> makes dom0 to stop getting responds from devices, such as SATA, etc,, >> which may lead to a whole system crash. >> >> My question are: >> 1. Can this hypercall be in some way preemptive, so it won''t stuck the >> whole system (i noticed that there are other hypercalls, which are >> preemptive) ? >> 2. Why can''t the mapping of the pages, be done in batches of pages, >> instead of mapping each page in a separate call to >> ''intel_iommu_map_page()'' ? >> 3. i noticed that for USB devices, the code there ignores the USB >> RMRR... can this be done also for the 00:02.1 device? >> >> Tom >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Han, Weidong
2009-Dec-03 02:22 UTC
[Xen-devel] RE: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings
I think the issue is not related to preempted hypercall, because 00:02.0 has the same RMRR with 00:02.1, and its pass-through succeeds. One correction, the RMRR (0x7dc00000 to 0x80000000) is not 10,000 pages, is ~1000 pages. Preempted hypercall implementation is a bit of complex. If you want to have a try, you can refer to existed preempted hypercalls (e.g. XENMEM_increase_reservation). Regards, Weidong -----Original Message----- From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] Sent: Wednesday, December 02, 2009 6:58 PM To: Han, Weidong; Keir Fraser Cc: xen-devel@lists.xensource.com Subject: Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings And, the other question: how can i make this hypercall preemptive? as it may still take too much time for an hypercall (taking into account, it needs to map ~10.000 pages...) On Mon, Nov 30, 2009 at 4:54 PM, Han, Weidong <weidong.han@intel.com> wrote:> You also need to include below code in the loop: > iommu_flush_cache_entry(pte); > unmap_vtd_domain_page(page); > > At the end, need to flush iotlb for all pages: > if ( iommu_flush_iotlb_psi(iommu, domain_iommu_domid(d), > (paddr_t)gfn << PAGE_SHIFT_4K, pages, ----> "pages" means page number > !pte_present, flush_dev_iotlb) ) > iommu_flush_write_buffer(iommu); > > Regards, > Weidong > > -----Original Message----- > From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] > Sent: Monday, November 30, 2009 7:17 PM > To: Han, Weidong > Cc: xen-devel@lists.xensource.com > Subject: Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings > > Can u please guid me a little bit, on how to write such function which > will work on a batch of pages? > Should i just enclose the: > " > ... > pg_maddr = addr_to_dma_page_maddr(d, (paddr_t)gfn << PAGE_SHIFT_4K, 1); > if ( pg_maddr == 0 ) > { > spin_unlock(&hd->mapping_lock); > return -ENOMEM; > } > page = (struct dma_pte *)map_vtd_domain_page(pg_maddr); > pte = page + (gfn & LEVEL_MASK); > pte_present = dma_pte_present(*pte); > dma_set_pte_addr(*pte, (paddr_t)mfn << PAGE_SHIFT_4K); > dma_set_pte_prot(*pte, DMA_PTE_READ | DMA_PTE_WRITE); > > /* Set the SNP on leaf page table if Snoop Control available */ > if ( iommu_snoop ) > dma_set_pte_snp(*pte); > ... > " > > in a loop? > > On Mon, Nov 30, 2009 at 4:30 AM, Han, Weidong <weidong.han@intel.com> wrote: >> Tom, >> >> Yes, you can add an API to map batches of pages instead of mapping each page in a separate call. >> >> RMRR regions of USB controller is ignored because they won''t be used in guest. The RMRR 0x7dc00000 to 0x80000000 is shared by 00:02.0 and 00:02.1, it will be used by IGD in guest. But you can have a try to ignore it for 00:02.1. >> >> >> Regards, >> Weidong >> >> -----Original Message----- >> From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] >> Sent: Monday, November 30, 2009 1:56 AM >> To: xen-devel@lists.xensource.com >> Cc: Han, Weidong >> Subject: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings >> >> Hi All, >> >> I am using a Dell E6400 machine, with IGD (Intel graphics device), >> which i''m trying to pass-through into a domU with Windows XP. When i >> assign only the 00:02.0 device, it seems to work ok, but when i try to >> assign the 00:02.1 device, the whole machine seems to freeze form more >> then 2 mins, causing a lot of messgaes in dom0, such as: SATA doesn''t >> respond, etc. >> >> After some analysis i made, i discovered the followings: >> * When calling the (Intel) iommu_assign_device, it checks if the >> device has any details in the RMRR table, and if so it calls the >> ''iommu_prepare_rmrr_dev()'' function. >> * The device 00:02.1 has an RMRR information, which details the whole >> area between: 0x7dc00000 to 0x80000000 (it contains ~9000 pages) >> * For each such page, Xen calls the ''intel_iommu_map_page()'' function. >> This causes the hypercall to take a lot of time to process, and this >> makes dom0 to stop getting responds from devices, such as SATA, etc,, >> which may lead to a whole system crash. >> >> My question are: >> 1. Can this hypercall be in some way preemptive, so it won''t stuck the >> whole system (i noticed that there are other hypercalls, which are >> preemptive) ? >> 2. Why can''t the mapping of the pages, be done in batches of pages, >> instead of mapping each page in a separate call to >> ''intel_iommu_map_page()'' ? >> 3. i noticed that for USB devices, the code there ignores the USB >> RMRR... can this be done also for the 00:02.1 device? >> >> Tom >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Han, Weidong
2009-Dec-08 02:40 UTC
[Xen-devel] RE: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings
Tom, As I know, the 00:02.1 is a little bit legacy, it was used in Win2000 to enabled second display functionality. Recent intel platforms doesn''t include 00:02.1. Can you have a try with guest win2000? Regards, Weidong -----Original Message----- From: Han, Weidong Sent: Thursday, December 03, 2009 10:22 AM To: ''Tom Rotenberg''; Keir Fraser Cc: xen-devel@lists.xensource.com Subject: RE: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings I think the issue is not related to preempted hypercall, because 00:02.0 has the same RMRR with 00:02.1, and its pass-through succeeds. One correction, the RMRR (0x7dc00000 to 0x80000000) is not 10,000 pages, is ~1000 pages. Preempted hypercall implementation is a bit of complex. If you want to have a try, you can refer to existed preempted hypercalls (e.g. XENMEM_increase_reservation). Regards, Weidong -----Original Message----- From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] Sent: Wednesday, December 02, 2009 6:58 PM To: Han, Weidong; Keir Fraser Cc: xen-devel@lists.xensource.com Subject: Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings And, the other question: how can i make this hypercall preemptive? as it may still take too much time for an hypercall (taking into account, it needs to map ~10.000 pages...) On Mon, Nov 30, 2009 at 4:54 PM, Han, Weidong <weidong.han@intel.com> wrote:> You also need to include below code in the loop: > iommu_flush_cache_entry(pte); > unmap_vtd_domain_page(page); > > At the end, need to flush iotlb for all pages: > if ( iommu_flush_iotlb_psi(iommu, domain_iommu_domid(d), > (paddr_t)gfn << PAGE_SHIFT_4K, pages, ----> "pages" means page number > !pte_present, flush_dev_iotlb) ) > iommu_flush_write_buffer(iommu); > > Regards, > Weidong > > -----Original Message----- > From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] > Sent: Monday, November 30, 2009 7:17 PM > To: Han, Weidong > Cc: xen-devel@lists.xensource.com > Subject: Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings > > Can u please guid me a little bit, on how to write such function which > will work on a batch of pages? > Should i just enclose the: > " > ... > pg_maddr = addr_to_dma_page_maddr(d, (paddr_t)gfn << PAGE_SHIFT_4K, 1); > if ( pg_maddr == 0 ) > { > spin_unlock(&hd->mapping_lock); > return -ENOMEM; > } > page = (struct dma_pte *)map_vtd_domain_page(pg_maddr); > pte = page + (gfn & LEVEL_MASK); > pte_present = dma_pte_present(*pte); > dma_set_pte_addr(*pte, (paddr_t)mfn << PAGE_SHIFT_4K); > dma_set_pte_prot(*pte, DMA_PTE_READ | DMA_PTE_WRITE); > > /* Set the SNP on leaf page table if Snoop Control available */ > if ( iommu_snoop ) > dma_set_pte_snp(*pte); > ... > " > > in a loop? > > On Mon, Nov 30, 2009 at 4:30 AM, Han, Weidong <weidong.han@intel.com> wrote: >> Tom, >> >> Yes, you can add an API to map batches of pages instead of mapping each page in a separate call. >> >> RMRR regions of USB controller is ignored because they won''t be used in guest. The RMRR 0x7dc00000 to 0x80000000 is shared by 00:02.0 and 00:02.1, it will be used by IGD in guest. But you can have a try to ignore it for 00:02.1. >> >> >> Regards, >> Weidong >> >> -----Original Message----- >> From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] >> Sent: Monday, November 30, 2009 1:56 AM >> To: xen-devel@lists.xensource.com >> Cc: Han, Weidong >> Subject: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings >> >> Hi All, >> >> I am using a Dell E6400 machine, with IGD (Intel graphics device), >> which i''m trying to pass-through into a domU with Windows XP. When i >> assign only the 00:02.0 device, it seems to work ok, but when i try to >> assign the 00:02.1 device, the whole machine seems to freeze form more >> then 2 mins, causing a lot of messgaes in dom0, such as: SATA doesn''t >> respond, etc. >> >> After some analysis i made, i discovered the followings: >> * When calling the (Intel) iommu_assign_device, it checks if the >> device has any details in the RMRR table, and if so it calls the >> ''iommu_prepare_rmrr_dev()'' function. >> * The device 00:02.1 has an RMRR information, which details the whole >> area between: 0x7dc00000 to 0x80000000 (it contains ~9000 pages) >> * For each such page, Xen calls the ''intel_iommu_map_page()'' function. >> This causes the hypercall to take a lot of time to process, and this >> makes dom0 to stop getting responds from devices, such as SATA, etc,, >> which may lead to a whole system crash. >> >> My question are: >> 1. Can this hypercall be in some way preemptive, so it won''t stuck the >> whole system (i noticed that there are other hypercalls, which are >> preemptive) ? >> 2. Why can''t the mapping of the pages, be done in batches of pages, >> instead of mapping each page in a separate call to >> ''intel_iommu_map_page()'' ? >> 3. i noticed that for USB devices, the code there ignores the USB >> RMRR... can this be done also for the 00:02.1 device? >> >> Tom >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tom Rotenberg
2009-Dec-08 09:04 UTC
[Xen-devel] Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings
Well... i have tested it on Windows XP (win2000 is really nhot that popular), and it''s required for Windows XP, so i don''t think that testing it with a win2000 guest will help, as it''s required on XP. On Tue, Dec 8, 2009 at 4:40 AM, Han, Weidong <weidong.han@intel.com> wrote:> Tom, > > As I know, the 00:02.1 is a little bit legacy, it was used in Win2000 to enabled second display functionality. Recent intel platforms doesn''t include 00:02.1. Can you have a try with guest win2000? > > Regards, > Weidong > > -----Original Message----- > From: Han, Weidong > Sent: Thursday, December 03, 2009 10:22 AM > To: ''Tom Rotenberg''; Keir Fraser > Cc: xen-devel@lists.xensource.com > Subject: RE: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings > > I think the issue is not related to preempted hypercall, because 00:02.0 has the same RMRR with 00:02.1, and its pass-through succeeds. One correction, the RMRR (0x7dc00000 to 0x80000000) is not 10,000 pages, is ~1000 pages. > > Preempted hypercall implementation is a bit of complex. If you want to have a try, you can refer to existed preempted hypercalls (e.g. XENMEM_increase_reservation). > > Regards, > Weidong > > > -----Original Message----- > From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] > Sent: Wednesday, December 02, 2009 6:58 PM > To: Han, Weidong; Keir Fraser > Cc: xen-devel@lists.xensource.com > Subject: Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings > > And, the other question: how can i make this hypercall preemptive? as > it may still take too much time for an hypercall (taking into account, > it needs to map ~10.000 pages...) > > On Mon, Nov 30, 2009 at 4:54 PM, Han, Weidong <weidong.han@intel.com> wrote: >> You also need to include below code in the loop: >> iommu_flush_cache_entry(pte); >> unmap_vtd_domain_page(page); >> >> At the end, need to flush iotlb for all pages: >> if ( iommu_flush_iotlb_psi(iommu, domain_iommu_domid(d), >> (paddr_t)gfn << PAGE_SHIFT_4K, pages, ----> "pages" means page number >> !pte_present, flush_dev_iotlb) ) >> iommu_flush_write_buffer(iommu); >> >> Regards, >> Weidong >> >> -----Original Message----- >> From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] >> Sent: Monday, November 30, 2009 7:17 PM >> To: Han, Weidong >> Cc: xen-devel@lists.xensource.com >> Subject: Re: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings >> >> Can u please guid me a little bit, on how to write such function which >> will work on a batch of pages? >> Should i just enclose the: >> " >> ... >> pg_maddr = addr_to_dma_page_maddr(d, (paddr_t)gfn << PAGE_SHIFT_4K, 1); >> if ( pg_maddr == 0 ) >> { >> spin_unlock(&hd->mapping_lock); >> return -ENOMEM; >> } >> page = (struct dma_pte *)map_vtd_domain_page(pg_maddr); >> pte = page + (gfn & LEVEL_MASK); >> pte_present = dma_pte_present(*pte); >> dma_set_pte_addr(*pte, (paddr_t)mfn << PAGE_SHIFT_4K); >> dma_set_pte_prot(*pte, DMA_PTE_READ | DMA_PTE_WRITE); >> >> /* Set the SNP on leaf page table if Snoop Control available */ >> if ( iommu_snoop ) >> dma_set_pte_snp(*pte); >> ... >> " >> >> in a loop? >> >> On Mon, Nov 30, 2009 at 4:30 AM, Han, Weidong <weidong.han@intel.com> wrote: >>> Tom, >>> >>> Yes, you can add an API to map batches of pages instead of mapping each page in a separate call. >>> >>> RMRR regions of USB controller is ignored because they won''t be used in guest. The RMRR 0x7dc00000 to 0x80000000 is shared by 00:02.0 and 00:02.1, it will be used by IGD in guest. But you can have a try to ignore it for 00:02.1. >>> >>> >>> Regards, >>> Weidong >>> >>> -----Original Message----- >>> From: Tom Rotenberg [mailto:tom.rotenberg@gmail.com] >>> Sent: Monday, November 30, 2009 1:56 AM >>> To: xen-devel@lists.xensource.com >>> Cc: Han, Weidong >>> Subject: Intel IOMMU: Assigning IGD device: 00:02.1 on some machines, causes Xen to freeze for >2mins because of RMRR mappings >>> >>> Hi All, >>> >>> I am using a Dell E6400 machine, with IGD (Intel graphics device), >>> which i''m trying to pass-through into a domU with Windows XP. When i >>> assign only the 00:02.0 device, it seems to work ok, but when i try to >>> assign the 00:02.1 device, the whole machine seems to freeze form more >>> then 2 mins, causing a lot of messgaes in dom0, such as: SATA doesn''t >>> respond, etc. >>> >>> After some analysis i made, i discovered the followings: >>> * When calling the (Intel) iommu_assign_device, it checks if the >>> device has any details in the RMRR table, and if so it calls the >>> ''iommu_prepare_rmrr_dev()'' function. >>> * The device 00:02.1 has an RMRR information, which details the whole >>> area between: 0x7dc00000 to 0x80000000 (it contains ~9000 pages) >>> * For each such page, Xen calls the ''intel_iommu_map_page()'' function. >>> This causes the hypercall to take a lot of time to process, and this >>> makes dom0 to stop getting responds from devices, such as SATA, etc,, >>> which may lead to a whole system crash. >>> >>> My question are: >>> 1. Can this hypercall be in some way preemptive, so it won''t stuck the >>> whole system (i noticed that there are other hypercalls, which are >>> preemptive) ? >>> 2. Why can''t the mapping of the pages, be done in batches of pages, >>> instead of mapping each page in a separate call to >>> ''intel_iommu_map_page()'' ? >>> 3. i noticed that for USB devices, the code there ignores the USB >>> RMRR... can this be done also for the 00:02.1 device? >>> >>> Tom >>> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel