Hi All, dom0 and domU uses swiotlb mechanism to emulate hardware IOMMU, right? And that so because the whole memory is visible to hypervisor only and not to other domains, am i right? My doubt is once the translation table between virtual and physical addresses is set by the swiotlb.How does device becomes aware of this table which is emulated in hardware as a matter of fact using swiotlb.the device does not knows about this table when accessing physical memory, isnt it? Another question is - dom0 and domU they both have their swiotlb mappings.This means Xen also keeps record of a mapping between these two tables in dom0 and domU? Am i right? Doesnt this means wasting a lot of memory in allocating bounce buffers for swiotlb during booting of dom0 and then during booting of domUs? Please bear as i am just starting with the Xen code . Any pointers or links will be really helpful. Thank you ~psr -- --- pradeep singh rautela "Genius is 1% inspiration, and 99% perspiration" - not me :) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of > pradeep singh rautela > Sent: 02 May 2007 13:49 > To: xen-devel; xenkernelnewbies@googlegroups.com > Subject: [Xen-devel] swiotlb and Xen query? > > Hi All, > > dom0 and domU uses swiotlb mechanism to emulate hardware > IOMMU, right? > And that so because the whole memory is visible to hypervisor > only and not to other domains, am i right?Not so. It uses swiotlb to allow 32-bit devices to access memory above 4GB.> > My doubt is once the translation table between virtual and > physical addresses is set by the swiotlb.How does device > becomes aware of this table which is emulated in hardware as > a matter of fact using swiotlb.the device does not knows > about this table when accessing physical memory, isnt it?It works because swiotlb isn''t at all a translation. Instead, it copies the buffer of the requesting driver to a specifically allocated memory region below 4GB. This new (low) memory location is what''s being given to the hardware device, so it never knew anything about the buffer originally having been elsewhere.> > Another question is - > dom0 and domU they both have their swiotlb mappings.This > means Xen also keeps record of a mapping between these two > tables in dom0 and domU? Am i right? Doesnt this means > wasting a lot of memory in allocating bounce buffers for > swiotlb during booting of dom0 and then during booting of domUs?There are no mappings of memory within swiotlb. And by the way, both Dom0 and DomU (in the para-virtual case) knows exactly where the actual physical address is in the machine - the hypervisor checks when the guest writes to its page-tables that the memory is within that guest. -- Mats> > Please bear as i am just starting with the Xen code . > Any pointers or links will be really helpful. > > Thank you > ~psr > > > -- > --- > pradeep singh rautela > > "Genius is 1% inspiration, and 99% perspiration" - not me :) >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Mats, sorry again for complete newbie questions. Please find my further queries inline. On 5/2/07, Petersson, Mats <Mats.Petersson@amd.com> wrote:> > > > dom0 and domU uses swiotlb mechanism to emulate hardware > > IOMMU, right? > > And that so because the whole memory is visible to hypervisor > > only and not to other domains, am i right? > > Not so. It uses swiotlb to allow 32-bit devices to access memory above > 4GB.Ok, perhaps i messed up the question.sorry for that. What i mean is, since a device has no idea of the mappings between domU memory and the machine memory, does this means the swiotlb is used to make sure that memory to which a device has DMAed can be actually mapped to the correct domain? Because device can DMA on to any 32 bit memory as you have correctly stated, Mats and that is till 4GB of machine memory. This memory to which the 32bit device can DMA need to be given to the correct domU isn''t it?Who makes sure this? swiotlb or hypervisor only using grant tables? If not how is zero copying ensured, if it is at all in this case?> > > My doubt is once the translation table between virtual and > > physical addresses is set by the swiotlb.How does device > > becomes aware of this table which is emulated in hardware as > > a matter of fact using swiotlb.the device does not knows > > about this table when accessing physical memory, isnt it? > > It works because swiotlb isn''t at all a translation. Instead, it copies > the buffer of the requesting driver to a specifically allocated memory > region below 4GB.So does this means it copies the buffer from dom0 on to domU using swiotlb whenever there is a DMA transfer? Isn''t that an overhead? copying ? This new (low) memory location is what''s being given> to the hardware device, so it never knew anything about the buffer > originally having been elsewhere. > > > > > Another question is - > > dom0 and domU they both have their swiotlb mappings.This > > means Xen also keeps record of a mapping between these two > > tables in dom0 and domU? Am i right? Doesnt this means > > wasting a lot of memory in allocating bounce buffers for > > swiotlb during booting of dom0 and then during booting of domUs? > > There are no mappings of memory within swiotlb. > > And by the way, both Dom0 and DomU (in the para-virtual case) knows > exactly where the actual physical address is in the machine - the > hypervisor checks when the guest writes to its page-tables that the > memory is within that guest.Thanks a lot Mats but i mean when the device writes to the memory. Thank you once again. ~psr> -- > Mats > > > > > Please bear as i am just starting with the Xen code . > > Any pointers or links will be really helpful. > > > > Thank you > > ~psr > > > > > > -- > > --- > > pradeep singh rautela > > > > "Genius is 1% inspiration, and 99% perspiration" - not me :) > > > > >-- --- pradeep singh rautela "Genius is 1% inspiration, and 99% perspiration" - not me :) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Comments inline, but first let me just state something that may not be clear: The below comments apply to REAL hardware, meaning when a domain (either Dom0 or another domain) is accessing for example a network controller or disk-controller. This is not how most DomU''s operate in the typical setup - they use virtual devices. Virtual devices operate through grant-tables and event-channels, and do not use swiotlb. Let me also clarify another thing: Dom0 is in no way special in any other way than "it''s privileged", so it''s allows to do some of the hypercalls that other domains can''t use. But there''s no difference from the hypervisors perspective in how most of the hypercalls operate for Dom0 and any other domain. For example, any domain can, technically, access a hardware device. It just so happens that the traditional way to configure Xen is to assign all the hardware to Dom0 and then use virtual devices in other domains. But it''s perfectly fine to hide any hardware from Dom0 to assign it to any other domain - only hardware that Dom0 needs to have access to is the parts necessary for it to boot and operate in itself, of course. Domains that have "hardware" are considered "driver domains".> -----Original Message----- > From: pradeep singh rautela [mailto:rautelap@gmail.com] > Sent: 02 May 2007 14:18 > To: Petersson, Mats > Cc: xen-devel; xenkernelnewbies@googlegroups.com > Subject: Re: [Xen-devel] swiotlb and Xen query? > > Hi Mats, > sorry again for complete newbie questions. > Please find my further queries inline. > > On 5/2/07, Petersson, Mats < Mats.Petersson@amd.com > <mailto:Mats.Petersson@amd.com> > wrote: > > > > dom0 and domU uses swiotlb mechanism to emulate hardware > > IOMMU, right? > > And that so because the whole memory is visible to hypervisor > > only and not to other domains, am i right? > > Not so. It uses swiotlb to allow 32-bit devices to > access memory above > 4GB. > > > Ok, perhaps i messed up the question.sorry for that. > What i mean is, since a device has no idea of the mappings > between domU memory and the machine memory, does this means > the swiotlb is used to make sure that memory to which a > device has DMAed can be actually mapped to the correct domain?No. The device also has no idea about the virtual (as used by Windows or Linux for example) to physical mapping in a "real" machine, right? So it is the responsibility of the OS+driver to translate any virtual address into a physical address. Since DomU (and Dom0) can translate it''s own memory addresses, this poses no problem - just take the virtual address of the buffer(s) and translate it into physical address(es). The physical address(es) are then given to the device itself to transfer.> Because device can DMA on to any 32 bit memory as you have > correctly stated, Mats and that is till 4GB of machine > memory. This memory to which the 32bit device can DMA need to > be given to the correct domU isn''t it?Who makes sure this?At the moment, there''s nothing preventing a DomU from DMA''ing to/from the low 4GB of physical memory (and if the device allows it, all of the physical memory in the machine). This is why Dom0 and any driver domain is a potential security hole in Xen. This is of course no different from how it works in any other system without Xen - there''s no prevention of a Windows disk driver writing the entire memory content to disk, for example [in fact, this is almost exactly how Windows Hibernate works].> swiotlb or hypervisor only using grant tables? If not how is > zero copying ensured, if it is at all in this case? > > > > > > > My doubt is once the translation table between virtual and > > physical addresses is set by the swiotlb.How does device > > becomes aware of this table which is emulated in hardware as > > a matter of fact using swiotlb.the device does not knows > > about this table when accessing physical memory, isnt it? > > It works because swiotlb isn''t at all a translation. > Instead, it copies > the buffer of the requesting driver to a specifically > allocated memory > region below 4GB. > > > So does this means it copies the buffer from dom0 on to domU > using swiotlb whenever there is a DMA transfer? Isn''t that an > overhead? copying ?There is no copy from Dom0 to DomU or other way around for driver domains. The swiotlb uses a special modifier on the memory allocation that it uses for "bounce buffers" to allocate memory that is in the low region (below 4GB). But if the device does need to use memory in the low 4GB, and the buffer isn''t in the low 4GB, then the buffer content is copied to a temporary buffer by swiotlb-code. And yes, there is some overhead in this. But it''s less overhead than not being able to transfer the data at all, and relying on some person to walk with a piece of paper with the data written down, and then writing the data into the other machine... ;-)> > > > This new (low) memory location is what''s being given > to the hardware device, so it never knew anything about > the buffer > originally having been elsewhere. > > > > > Another question is - > > dom0 and domU they both have their swiotlb mappings.This > > means Xen also keeps record of a mapping between these two > > tables in dom0 and domU? Am i right? Doesnt this means > > wasting a lot of memory in allocating bounce buffers for > > swiotlb during booting of dom0 and then during > booting of domUs? > > There are no mappings of memory within swiotlb. > > And by the way, both Dom0 and DomU (in the para-virtual > case) knows > exactly where the actual physical address is in the > machine - the > hypervisor checks when the guest writes to its > page-tables that the > memory is within that guest. > > > Thanks a lot Mats but i mean when the device writes to the memory.The device (in general) writes to memory that it''s been told to write to, by the OS in the Domain. And as described above, it''s given the physical address. Note that there are security issues with certain hardware that allows "remote target addressing", meaning that the remote end can decide where in the target system the data ends up. So, with the right hardware + cabling and software, you can over-write (for example) the user-id in the "who''s logged in", so that you become "root" on a machine where you don''t have root privileges. Fortunately, this can not be done on Ethernet controllers! -- Mats> > > Thank you once again. > > ~psr > > > -- > Mats > > > > > Please bear as i am just starting with the Xen code . > > Any pointers or links will be really helpful. > > > > Thank you > > ~psr > > > > > > -- > > --- > > pradeep singh rautela > > > > "Genius is 1% inspiration, and 99% perspiration" - not me :) > > > > > > > > > > -- > --- > pradeep singh rautela > > "Genius is 1% inspiration, and 99% perspiration" - not me :) >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/2/07, Petersson, Mats <Mats.Petersson@amd.com> wrote:> > Comments inline, but first let me just state something that may not be > clear: > > The below comments apply to REAL hardware, meaning when a domain (either > Dom0 or another domain) is accessing for example a network controller or > disk-controller. This is not how most DomU''s operate in the typical > setup - they use virtual devices. Virtual devices operate through > grant-tables and event-channels, and do not use swiotlb. > > Let me also clarify another thing: Dom0 is in no way special in any > other way than "it''s privileged", so it''s allows to do some of the > hypercalls that other domains can''t use. But there''s no difference from > the hypervisors perspective in how most of the hypercalls operate for > Dom0 and any other domain. For example, any domain can, technically, > access a hardware device. It just so happens that the traditional way to > configure Xen is to assign all the hardware to Dom0 and then use virtual > devices in other domains. But it''s perfectly fine to hide any hardware > from Dom0 to assign it to any other domain - only hardware that Dom0 > needs to have access to is the parts necessary for it to boot and > operate in itself, of course. Domains that have "hardware" are > considered "driver domains". > > > > -----Original Message----- > > From: pradeep singh rautela [mailto:rautelap@gmail.com] > > Sent: 02 May 2007 14:18 > > To: Petersson, Mats > > Cc: xen-devel; xenkernelnewbies@googlegroups.com > > Subject: Re: [Xen-devel] swiotlb and Xen query? > > > > Hi Mats, > > sorry again for complete newbie questions. > > Please find my further queries inline. > > > > On 5/2/07, Petersson, Mats < Mats.Petersson@amd.com > > <mailto:Mats.Petersson@amd.com> > wrote: > > > > > > > dom0 and domU uses swiotlb mechanism to emulate hardware > > > IOMMU, right? > > > And that so because the whole memory is visible to hypervisor > > > only and not to other domains, am i right? > > > > Not so. It uses swiotlb to allow 32-bit devices to > > access memory above > > 4GB. > > > > > > Ok, perhaps i messed up the question.sorry for that. > > What i mean is, since a device has no idea of the mappings > > between domU memory and the machine memory, does this means > > the swiotlb is used to make sure that memory to which a > > device has DMAed can be actually mapped to the correct domain? > > No. The device also has no idea about the virtual (as used by Windows or > Linux for example) to physical mapping in a "real" machine, right? So it > is the responsibility of the OS+driver to translate any virtual address > into a physical address. Since DomU (and Dom0) can translate it''s own > memory addresses, this poses no problem - just take the virtual address > of the buffer(s) and translate it into physical address(es). The > physical address(es) are then given to the device itself to transfer. > > > Because device can DMA on to any 32 bit memory as you have > > correctly stated, Mats and that is till 4GB of machine > > memory. This memory to which the 32bit device can DMA need to > > be given to the correct domU isn''t it?Who makes sure this? > > At the moment, there''s nothing preventing a DomU from DMA''ing to/from > the low 4GB of physical memory (and if the device allows it, all of the > physical memory in the machine). This is why Dom0 and any driver domain > is a potential security hole in Xen. This is of course no different from > how it works in any other system without Xen - there''s no prevention of > a Windows disk driver writing the entire memory content to disk, for > example [in fact, this is almost exactly how Windows Hibernate works]. > > > swiotlb or hypervisor only using grant tables? If not how is > > zero copying ensured, if it is at all in this case? > > > > > > > > > > > > My doubt is once the translation table between virtual and > > > physical addresses is set by the swiotlb.How does device > > > becomes aware of this table which is emulated in hardware as > > > a matter of fact using swiotlb.the device does not knows > > > about this table when accessing physical memory, isnt it? > > > > It works because swiotlb isn''t at all a translation. > > Instead, it copies > > the buffer of the requesting driver to a specifically > > allocated memory > > region below 4GB. > > > > > > So does this means it copies the buffer from dom0 on to domU > > using swiotlb whenever there is a DMA transfer? Isn''t that an > > overhead? copying ? > > There is no copy from Dom0 to DomU or other way around for driver > domains. The swiotlb uses a special modifier on the memory allocation > that it uses for "bounce buffers" to allocate memory that is in the low > region (below 4GB).That was really very informative Mats. Thanks you very much for the help. So, i guess swiotlb comes into picture only if there is a DMA transfer from a domU whose p2m mapping turns out to be above the 4GB( assuming device to be 32 bit) of the machine memory. Other wise just normal p2m mapping is enough to find the actual machine address, which can be handed over to the device for DMAing. Am i right? Thank you once again for the patience. Regards ~psr But if the device does need to use memory in the low 4GB, and the buffer> isn''t in the low 4GB, then the buffer content is copied to a temporary > buffer by swiotlb-code. And yes, there is some overhead in this. But > it''s less overhead than not being able to transfer the data at all, and > relying on some person to walk with a piece of paper with the data > written down, and then writing the data into the other machine... ;-)> > > > > > > This new (low) memory location is what''s being given > > to the hardware device, so it never knew anything about > > the buffer > > originally having been elsewhere. > > > > > > > > Another question is - > > > dom0 and domU they both have their swiotlb mappings.This > > > means Xen also keeps record of a mapping between these two > > > tables in dom0 and domU? Am i right? Doesnt this means > > > wasting a lot of memory in allocating bounce buffers for > > > swiotlb during booting of dom0 and then during > > booting of domUs? > > > > There are no mappings of memory within swiotlb. > > > > And by the way, both Dom0 and DomU (in the para-virtual > > case) knows > > exactly where the actual physical address is in the > > machine - the > > hypervisor checks when the guest writes to its > > page-tables that the > > memory is within that guest. > > > > > > Thanks a lot Mats but i mean when the device writes to the memory. > > The device (in general) writes to memory that it''s been told to write > to, by the OS in the Domain. And as described above, it''s given the > physical address. > > Note that there are security issues with certain hardware that allows > "remote target addressing", meaning that the remote end can decide where > in the target system the data ends up. So, with the right hardware + > cabling and software, you can over-write (for example) the user-id in > the "who''s logged in", so that you become "root" on a machine where you > don''t have root privileges. Fortunately, this can not be done on > Ethernet controllers! > > -- > Mats > > > > > > Thank you once again. > > > > ~psr > > > > > > -- > > Mats > > > > > > > > Please bear as i am just starting with the Xen code . > > > Any pointers or links will be really helpful. > > > > > > Thank you > > > ~psr > > > > > > > > > -- > > > --- > > > pradeep singh rautela > > > > > > "Genius is 1% inspiration, and 99% perspiration" - not me :) > > > > > > > > > > > > > > > > > > > -- > > --- > > pradeep singh rautela > > > > "Genius is 1% inspiration, and 99% perspiration" - not me :) > > > > >-- --- pradeep singh rautela "Genius is 1% inspiration, and 99% perspiration" - not me :) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
[snip]> > > That was really very informative Mats. > Thanks you very much for the help. > So, i guess swiotlb comes into picture only if there is a DMA > transfer from a domU whose p2m mapping turns out to be above > the 4GB( assuming device to be 32 bit) of the machine memory. > Other wise just normal p2m mapping is enough to find the > actual machine address, which can be handed over to the > device for DMAing. Am i right?Yes, that''s pretty much how it works. In theory, it could be that devices only work on a 31, 28 or some other number of bit-range, but yes, that''s the principle. -- Mats> > Thank you once again for the patience. > > Regards > ~psr >[snip] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/3/07, Petersson, Mats <Mats.Petersson@amd.com> wrote:> > [snip] > > > > > > That was really very informative Mats. > > Thanks you very much for the help. > > So, i guess swiotlb comes into picture only if there is a DMA > > transfer from a domU whose p2m mapping turns out to be above > > the 4GB( assuming device to be 32 bit) of the machine memory. > > Other wise just normal p2m mapping is enough to find the > > actual machine address, which can be handed over to the > > device for DMAing. Am i right? > > Yes, that''s pretty much how it works. In theory, it could be that > devices only work on a 31, 28 or some other number of bit-range, but > yes, that''s the principle.Ok.Got this correct. I am sure this can cause problems while migrating live domains across cpus. yes/no/thoughts? Thank you ~psr --> Mats > > > > Thank you once again for the patience. > > > > Regards > > ~psr > > > [snip] > > >-- --- pradeep singh rautela "Genius is 1% inspiration, and 99% perspiration" - not me :) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> -----Original Message----- > From: pradeep singh rautela [mailto:rautelap@gmail.com] > Sent: 03 May 2007 10:25 > To: Petersson, Mats > Cc: xen-devel > Subject: Re: [Xen-devel] swiotlb and Xen query? > > > > On 5/3/07, Petersson, Mats <Mats.Petersson@amd.com> wrote: > > [snip] > > > > > > That was really very informative Mats. > > Thanks you very much for the help. > > So, i guess swiotlb comes into picture only if there is a DMA > > transfer from a domU whose p2m mapping turns out to be above > > the 4GB( assuming device to be 32 bit) of the machine memory. > > Other wise just normal p2m mapping is enough to find the > > actual machine address, which can be handed over to the > > device for DMAing. Am i right? > > Yes, that''s pretty much how it works. In theory, it > could be that > devices only work on a 31, 28 or some other number of > bit-range, but > yes, that''s the principle. > > > Ok.Got this correct. > I am sure this can cause problems while migrating live > domains across cpus. > yes/no/thoughts?Yes, you can''t live-migrate (or migrate at all, I think) driver domains, as there is no way to determine if a driver has outstanding DMA requests. You need to shut it down and restart it elsewhere if you want to move it from one machine to another. [Of course not across CPU''s, but if you actually move across physical machine boundary]. [I had a quick look, but I couldn''t actually find any checks for this, but I still believe it holds true that driver domains can''t be saved/restored/migrated]. -- Mats> > Thank you > ~psr > > > > -- > Mats > > > > Thank you once again for the patience. > > > > Regards > > ~psr > > > [snip] > > > > > > > > -- > --- > pradeep singh rautela > > "Genius is 1% inspiration, and 99% perspiration" - not me :) >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel