Andrew D. Ball
2006-Nov-16 15:44 UTC
[Fwd: RE: [Xen-devel] Fwd: Need help - Assigning PCI device to HVM Windows domain]
-------- Forwarded Message -------- From: Andrew D. Ball <aball@linux.vnet.ibm.com> Reply-To: aball@linux.vnet.ibm.com To: Petersson, Mats <Mats.Petersson@amd.com> Subject: RE: [Xen-devel] Fwd: Need help - Assigning PCI device to HVM Windows domain Date: Thu, 16 Nov 2006 10:43:48 -0500 For the IOMMU stuff, I wonder how far along the Calgary support is in Xen. Granted, the Intel and AMD versions will become ubiquitous and Calgary almost certainly will not ... Also tricks available with stuff like AGP apertures, right? Peace. Andrew On Thu, 2006-11-16 at 11:50 +0100, Petersson, Mats wrote:> This will not work! > > The reason is that we tell Windows that it''s got X MB of memory, > starting at address 0, like it does in a standalone system. But in > reality, the memory actually given to the virtual machine could be ANY > location in memory (including a completely scattered selection of memory > blocks). > > Since Windows is COMPLETELY unaware of this fact, your driver will give > what it believes is the physical address, in the range of 0..X MB to the > network card when it wants to send a packet of data. This is of course > not going to work if the packet to send isn''t at the physical address > Windows thinks it is. > > Let''s take a trivial example: We give Windows a memory size of 256MB, > located in the contiguous section 256..512MB. The OS then wants to send > a packet on the network card that has the virtual address of 0x40867456 > and the physical address of 0x123456 (just over 1MB), so the driver gets > the address for the packet (0x40867456). It calls the Win32 call > GetPhysicalAddress(), it will get back the address that Windows thinks > is the physical address, because it doesn''t know that we''ve lied about > where the memory is. This means that it will get the address 0x123456. > This address should be in the 256MB..512MB range, but it isn''t, it''s > just over 1MB. So the packet sent will be wrong (if the network card > just grabs the data and there''s no need for some extra "stuff" around > the data-packet, in which case the card will just plain refuse to > work!). > > There are two solutions: > 1. Implement within the driver for the network card an understanding of > the fact that guest-physical memory (i.e. what your Windows OS thinks is > physical addresses) and machine physical addresses (those that are > ACTUALLY in the machine), by adding an extra layer of translation from > guest-physical to machine-physical where the driver is asking for a > physical address. Xen nows how to do this, so you should be able to use > the para-virtual driver calling mechanism to achieve this, but it does > assume that A) You are familiar with windows driver programming, B) that > you have source code for the driver(s) involved with your network card > and C) that you have a driver development kit (DDK) from MS installed on > some machine. > > 2. Wait for IOMMU hardware to be available (and relevant code in Xen) to > allow the hypervisor to map the guest-physical memory to > machine-physical address translation so that the guest doesn''t need to > know the difference. > > -- > Mats > > > -----Original Message----- > > From: xen-devel-bounces@lists.xensource.com > > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of GT > > Sent: 16 November 2006 08:30 > > To: xen-devel@lists.xensource.com > > Cc: gtcharny@gmail.com > > Subject: [Xen-devel] Fwd: Need help - Assigning PCI device to > > HVM Windows domain > > > > I forgot to mention that I allowed MMIO/IO ports access to > > WinXP according > > to NIC''s PCI BARs. > > > > Thanks, > > > > GT > > > > ---------- Forwarded message ---------- > > From: GT <gtcharny@gmail.com> > > Date: Nov 16, 2006 10:21 AM > > Subject: Need help - Assigning PCI device to HVM Windows domain > > To: xen-devel@lists.xensource.com > > <mailto:xen-devel@lists.xensource.com> > > > > Hi, > > > > I am a newbie to Xen and I am trying to assign PCI NIC to > > Windows domain. I performed below steps, with no luck. > > It would be great if someone could point me to the right direction. > > > > I did as follows: > > > > 1) I''ve got WinXP installed over Xen, and got network working > > with pcnet model > > 2) I''ve hidden NIC from dom0 and verified that it does not > > appear with lspci > > 3) I''ve added interception of PCI scan from HVM domain (using > > 0xcf8/0xcfc ports interception), > > and when the PCI bud/dev/fun matched my physical NIC, I > > returned physical vendor/device id. > > 4) I''ve rebuilt/reinstalled all images, but WinXP didn''t > > recognize any NIC device. > > > > Thanks ahead, > > > > GT > > > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Petersson, Mats
2006-Nov-16 15:49 UTC
RE: [Fwd: RE: [Xen-devel] Fwd: Need help - Assigning PCI device to HVM Windows domain]
> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of > Andrew D. Ball > Sent: 16 November 2006 15:44 > To: xen-devel@lists.xensource.com > Subject: [Fwd: RE: [Xen-devel] Fwd: Need help - Assigning PCI > device to HVM Windows domain] > > -------- Forwarded Message -------- > From: Andrew D. Ball <aball@linux.vnet.ibm.com> > Reply-To: aball@linux.vnet.ibm.com > To: Petersson, Mats <Mats.Petersson@amd.com> > Subject: RE: [Xen-devel] Fwd: Need help - Assigning PCI device to HVM > Windows domain > Date: Thu, 16 Nov 2006 10:43:48 -0500 > > For the IOMMU stuff, I wonder how far along the Calgary support is in > Xen. Granted, the Intel and AMD versions will become ubiquitous and > Calgary almost certainly will not ... Also tricks available > with stuff > like AGP apertures, right?Sure, there is some work ongoing for Calgary (according to Muli), but that is primarily targetted towards PV domains rather than HVM domains. I think Mark Langsdorf has been owkring on GART-based IOMMU, but I don''t know what the status is on this... I don''t know when ours or Intels hardware will be available (I hope sooner rather than later, because it will solve a whole bunch of peoples problems...) -- Mats> > Peace. > Andrew > > On Thu, 2006-11-16 at 11:50 +0100, Petersson, Mats wrote: > > This will not work! > > > > The reason is that we tell Windows that it''s got X MB of memory, > > starting at address 0, like it does in a standalone system. But in > > reality, the memory actually given to the virtual machine > could be ANY > > location in memory (including a completely scattered > selection of memory > > blocks). > > > > Since Windows is COMPLETELY unaware of this fact, your > driver will give > > what it believes is the physical address, in the range of > 0..X MB to the > > network card when it wants to send a packet of data. This > is of course > > not going to work if the packet to send isn''t at the > physical address > > Windows thinks it is. > > > > Let''s take a trivial example: We give Windows a memory size > of 256MB, > > located in the contiguous section 256..512MB. The OS then > wants to send > > a packet on the network card that has the virtual address > of 0x40867456 > > and the physical address of 0x123456 (just over 1MB), so > the driver gets > > the address for the packet (0x40867456). It calls the Win32 call > > GetPhysicalAddress(), it will get back the address that > Windows thinks > > is the physical address, because it doesn''t know that we''ve > lied about > > where the memory is. This means that it will get the > address 0x123456. > > This address should be in the 256MB..512MB range, but it isn''t, it''s > > just over 1MB. So the packet sent will be wrong (if the network card > > just grabs the data and there''s no need for some extra > "stuff" around > > the data-packet, in which case the card will just plain refuse to > > work!). > > > > There are two solutions: > > 1. Implement within the driver for the network card an > understanding of > > the fact that guest-physical memory (i.e. what your Windows > OS thinks is > > physical addresses) and machine physical addresses (those that are > > ACTUALLY in the machine), by adding an extra layer of > translation from > > guest-physical to machine-physical where the driver is asking for a > > physical address. Xen nows how to do this, so you should be > able to use > > the para-virtual driver calling mechanism to achieve this, > but it does > > assume that A) You are familiar with windows driver > programming, B) that > > you have source code for the driver(s) involved with your > network card > > and C) that you have a driver development kit (DDK) from MS > installed on > > some machine. > > > > 2. Wait for IOMMU hardware to be available (and relevant > code in Xen) to > > allow the hypervisor to map the guest-physical memory to > > machine-physical address translation so that the guest > doesn''t need to > > know the difference. > > > > -- > > Mats > > > > > -----Original Message----- > > > From: xen-devel-bounces@lists.xensource.com > > > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of GT > > > Sent: 16 November 2006 08:30 > > > To: xen-devel@lists.xensource.com > > > Cc: gtcharny@gmail.com > > > Subject: [Xen-devel] Fwd: Need help - Assigning PCI device to > > > HVM Windows domain > > > > > > I forgot to mention that I allowed MMIO/IO ports access to > > > WinXP according > > > to NIC''s PCI BARs. > > > > > > Thanks, > > > > > > GT > > > > > > ---------- Forwarded message ---------- > > > From: GT <gtcharny@gmail.com> > > > Date: Nov 16, 2006 10:21 AM > > > Subject: Need help - Assigning PCI device to HVM Windows domain > > > To: xen-devel@lists.xensource.com > > > <mailto:xen-devel@lists.xensource.com> > > > > > > Hi, > > > > > > I am a newbie to Xen and I am trying to assign PCI NIC to > > > Windows domain. I performed below steps, with no luck. > > > It would be great if someone could point me to the right > direction. > > > > > > I did as follows: > > > > > > 1) I''ve got WinXP installed over Xen, and got network working > > > with pcnet model > > > 2) I''ve hidden NIC from dom0 and verified that it does not > > > appear with lspci > > > 3) I''ve added interception of PCI scan from HVM domain (using > > > 0xcf8/0xcfc ports interception), > > > and when the PCI bud/dev/fun matched my physical NIC, I > > > returned physical vendor/device id. > > > 4) I''ve rebuilt/reinstalled all images, but WinXP didn''t > > > recognize any NIC device. > > > > > > Thanks ahead, > > > > > > GT > > > > > > > > > > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Muli Ben-Yehuda
2006-Nov-17 07:00 UTC
Re: [Fwd: RE: [Xen-devel] Fwd: Need help - Assigning PCI device to HVM Windows domain]
On Thu, Nov 16, 2006 at 04:49:59PM +0100, Petersson, Mats wrote:> > For the IOMMU stuff, I wonder how far along the Calgary support is in > > Xen. Granted, the Intel and AMD versions will become ubiquitous and > > Calgary almost certainly will not ... Also tricks available > > with stuff > > like AGP apertures, right? > > Sure, there is some work ongoing for Calgary (according to Muli), but > that is primarily targetted towards PV domains rather than HVM > domains.Correct, although in theory it should be fairly straightforward to adapt it to FV domains. You would also need to work out a few things such as interrupt delivery to the guest, but the mapping of IO space to correspond to the guest physical mapping is simple enough. To answer anball''s question, the Xen-Calgary work is on a bit of a hold at the moment as I''m working on bringing up the PCI-e version of Calgary, CalIOC2, and a bunch of other things. I expect to get back to the Xen bits pretty soon. Cheers, Muli _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel