Xen devel - Jun 2011 - Is there a way to map pv guest pseudo physical address into dom0?

Hi, all

I''m reading the code of QEMU''s virtio infrastructure
(hw/virtio.c) and
Linux''s virtio driver (virtio_blk.c, virtio_net.c).

Virtio drivers utilize scatter gather infrastructure in the Linux
kernel. It stores the physical addresses in scatter list. In Xen''s hvm
case, QEMU can access guest address space with
cpu_physical_memory_map.

However, in Xen''s pv case, this "physical address" will be
"pseudo
physical address". In order to retrieve theirs content, backend must
map them into its own address space. I''m wondering if there is such
infrastructure to get this job done.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 2011-06-27 at 09:47 +0100, Wei Liu wrote:
> Hi, all
> 
> I''m reading the code of QEMU''s virtio infrastructure
(hw/virtio.c) and
> Linux''s virtio driver (virtio_blk.c, virtio_net.c).
> 
> Virtio drivers utilize scatter gather infrastructure in the Linux
> kernel. It stores the physical addresses in scatter list. In Xen''s
hvm
> case, QEMU can access guest address space with
> cpu_physical_memory_map.
> 
> However, in Xen''s pv case, this "physical address" will
be "pseudo
> physical address". In order to retrieve theirs content, backend must
> map them into its own address space. I''m wondering if there is
such
> infrastructure to get this job done.
I think you want one of xc_map_* provided by libxc (see xenctrl.h). (But
avoid xc_map_foreign_batch which is deprecated).

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Jun 27, 2011 at 5:01 PM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> I think you want one of xc_map_* provided by libxc (see xenctrl.h). (But
> avoid xc_map_foreign_batch which is deprecated).
>
AFAICT, they take MFN as parameter, but I have pseudo physical addresses. :-(

Wei.
> Ian.
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 2011-06-27 at 10:10 +0100, Wei Liu wrote:
> On Mon, Jun 27, 2011 at 5:01 PM, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> > I think you want one of xc_map_* provided by libxc (see xenctrl.h).
(But
> > avoid xc_map_foreign_batch which is deprecated).
> >
> 
> AFAICT, they take MFN as parameter, but I have pseudo physical addresses.
:-(
You need to arrange for the Xen specific bit of the virtio frontend to
do the p2m translation. All externally visible addresses from a PV guest
must be in MFN space.

Otherwise you only option is to pause the guest and map his p2m. Which
would obviously suck.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Jun 27, 2011 at 5:17 PM, Ian Campbell
<Ian.Campbell@eu.citrix.com> wrote:
> On Mon, 2011-06-27 at 10:10 +0100, Wei Liu wrote:
>> On Mon, Jun 27, 2011 at 5:01 PM, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
>> > I think you want one of xc_map_* provided by libxc (see
xenctrl.h). (But
>> > avoid xc_map_foreign_batch which is deprecated).
>> >
>>
>> AFAICT, they take MFN as parameter, but I have pseudo physical
addresses. :-(
>
> You need to arrange for the Xen specific bit of the virtio frontend to
> do the p2m translation. All externally visible addresses from a PV guest
> must be in MFN space.
>
That''s not a good approach. Dirvers reside on a higher level from the
"Xen specific bit". Doing this kind of translation will break their
generality. But there seems no other way to get this job done...
> Otherwise you only option is to pause the guest and map his p2m. Which
> would obviously suck.
>
Agreed.

Wei.
> Ian.
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
> > You need to arrange for the Xen specific bit of the virtio frontend to
> > do the p2m translation. All externally visible addresses from a PV
guest
> > must be in MFN space.
> >
> 
> That''s not a good approach. Dirvers reside on a higher level from
the
> "Xen specific bit". Doing this kind of translation will break
their
> generality. But there seems no other way to get this job done...
This is a general problem with operating systems: bus addresses are not
always the same as CPU memory addresses (even though in the easy case of
a simple x86 PC they often are).  This is why OSes already have
interfaces to allocate DMA-able memory and to translate between the
address that CPU will map and the one the peripheral should use.  If
your driver''s not using the right address-space to program its hardware
it is broken; the Xen PV model is just an extreme case of this.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 27 Jun 2011, Tim Deegan wrote:
> At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
> > > You need to arrange for the Xen specific bit of the virtio
frontend to
> > > do the p2m translation. All externally visible addresses from a
PV guest
> > > must be in MFN space.
> > >
> > 
> > That''s not a good approach. Dirvers reside on a higher level
from the
> > "Xen specific bit". Doing this kind of translation will
break their
> > generality. But there seems no other way to get this job done...
> 
> This is a general problem with operating systems: bus addresses are not
> always the same as CPU memory addresses (even though in the easy case of
> a simple x86 PC they often are).  This is why OSes already have
> interfaces to allocate DMA-able memory and to translate between the
> address that CPU will map and the one the peripheral should use.  If
> your driver''s not using the right address-space to program its
hardware
> it is broken; the Xen PV model is just an extreme case of this.
Indeed.

When passing addresses to devices, drivers should use the DMA APIs to
make sure that proper address conversions take place (see
Documentation/DMA-API-HOWTO.txt).
On Xen the swiotlb implementation takes care of doing pfn to mfn
conversions as well.
For example, to allocate a consistent buffer that is addressable by both
a device and the cpu, the correct API is dma_alloc_coherent, that
returns two values: the virtual address which you can use to access it
from the CPU and dma_handle which you pass to the device.

I can very well imagine that virtio drivers are not doing any of this
because they didn''t need to, but it would be probably OK to submit a
patch to convert the drivers to use the proper APIs.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thank you, Tim and Stefano.

On Mon, Jun 27, 2011 at 7:23 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Mon, 27 Jun 2011, Tim Deegan wrote:
>> At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
>> > > You need to arrange for the Xen specific bit of the virtio
frontend to
>> > > do the p2m translation. All externally visible addresses from
a PV guest
>> > > must be in MFN space.
>> > >
>> >
>> > That''s not a good approach. Dirvers reside on a higher
level from the
>> > "Xen specific bit". Doing this kind of translation will
break their
>> > generality. But there seems no other way to get this job done...
>>
>> This is a general problem with operating systems: bus addresses are not
>> always the same as CPU memory addresses (even though in the easy case
of
>> a simple x86 PC they often are).  This is why OSes already have
>> interfaces to allocate DMA-able memory and to translate between the
>> address that CPU will map and the one the peripheral should use.  If
>> your driver''s not using the right address-space to program its
hardware
>> it is broken; the Xen PV model is just an extreme case of this.
>
> Indeed.
>
> When passing addresses to devices, drivers should use the DMA APIs to
> make sure that proper address conversions take place (see
> Documentation/DMA-API-HOWTO.txt).
> On Xen the swiotlb implementation takes care of doing pfn to mfn
> conversions as well.
> For example, to allocate a consistent buffer that is addressable by both
> a device and the cpu, the correct API is dma_alloc_coherent, that
> returns two values: the virtual address which you can use to access it
> from the CPU and dma_handle which you pass to the device.
>
This is the first time to notice swiotlb TBH. It seems to be a good
start for me.

After skimming the code in $KERNEL/drivers/xen/swiotlb-xen.c, I have
further questions.

It seems that swiotlb is used for PCI passthrough in PV, I''m wondering
if paravirt drivers ever use it.
> I can very well imagine that virtio drivers are not doing any of this
> because they didn''t need to, but it would be probably OK to submit
a
> patch to convert the drivers to use the proper APIs.
>
No, they are not doing this.

Do you mean that I replace every allocation/deallocation of virtio
drivers scatter buffer with DMA API?

Then, with ''swiotlb=force'' in pv guest kernel command line,
these
DMA-API will call xen-swiotlb and get proper addresses?

If I don''t add ''swiotlb=force'', which underlying
implementation will
be used? And, which DMA-API will be used in hvm case? will this change
affect hvm guest?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 27 Jun 2011, Wei Liu wrote:
> On Mon, Jun 27, 2011 at 7:23 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Mon, 27 Jun 2011, Tim Deegan wrote:
> >> At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
> >> > > You need to arrange for the Xen specific bit of the
virtio frontend to
> >> > > do the p2m translation. All externally visible addresses
from a PV guest
> >> > > must be in MFN space.
> >> > >
> >> >
> >> > That''s not a good approach. Dirvers reside on a
higher level from the
> >> > "Xen specific bit". Doing this kind of translation
will break their
> >> > generality. But there seems no other way to get this job
done...
> >>
> >> This is a general problem with operating systems: bus addresses
are not
> >> always the same as CPU memory addresses (even though in the easy
case of
> >> a simple x86 PC they often are).  This is why OSes already have
> >> interfaces to allocate DMA-able memory and to translate between
the
> >> address that CPU will map and the one the peripheral should use.
 If
> >> your driver''s not using the right address-space to
program its hardware
> >> it is broken; the Xen PV model is just an extreme case of this.
> >
> > Indeed.
> >
> > When passing addresses to devices, drivers should use the DMA APIs to
> > make sure that proper address conversions take place (see
> > Documentation/DMA-API-HOWTO.txt).
> > On Xen the swiotlb implementation takes care of doing pfn to mfn
> > conversions as well.
> > For example, to allocate a consistent buffer that is addressable by
both
> > a device and the cpu, the correct API is dma_alloc_coherent, that
> > returns two values: the virtual address which you can use to access it
> > from the CPU and dma_handle which you pass to the device.
> >
> 
> This is the first time to notice swiotlb TBH. It seems to be a good
> start for me.
> 
> After skimming the code in $KERNEL/drivers/xen/swiotlb-xen.c, I have
> further questions.
> 
> It seems that swiotlb is used for PCI passthrough in PV, I''m
wondering
> if paravirt drivers ever use it.
> 
> > I can very well imagine that virtio drivers are not doing any of this
> > because they didn''t need to, but it would be probably OK to
submit a
> > patch to convert the drivers to use the proper APIs.
> >
> 
> No, they are not doing this.
> 
> Do you mean that I replace every allocation/deallocation of virtio
> drivers scatter buffer with DMA API?
Yes. Konrad is the most knowledgeable about this, he might have some
suggestions.

> Then, with ''swiotlb=force'' in pv guest kernel command
line, these
> DMA-API will call xen-swiotlb and get proper addresses?
> 
> If I don''t add ''swiotlb=force'', which underlying
implementation will
> be used? And, which DMA-API will be used in hvm case? will this change
> affect hvm guest?
 
The right swiotlb implementation is chosen at run time depending on your
platform. So the user doesn''t need to manually add swiotlb=force to the
command line, and the changes shouldn''t affect performance in the
normal
cases.
--8323329-1526883182-1309189570=:12963
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

--8323329-1526883182-1309189570=:12963--

On Mon, Jun 27, 2011 at 09:35:24PM +0800, Wei Liu wrote:
> Thank you, Tim and Stefano.
> 
> On Mon, Jun 27, 2011 at 7:23 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Mon, 27 Jun 2011, Tim Deegan wrote:
> >> At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
> >> > > You need to arrange for the Xen specific bit of the
virtio frontend to
> >> > > do the p2m translation. All externally visible addresses
from a PV guest
> >> > > must be in MFN space.
> >> > >
> >> >
> >> > That''s not a good approach. Dirvers reside on a
higher level from the
> >> > "Xen specific bit". Doing this kind of translation
will break their
> >> > generality. But there seems no other way to get this job
done...
> >>
> >> This is a general problem with operating systems: bus addresses
are not
> >> always the same as CPU memory addresses (even though in the easy
case of
> >> a simple x86 PC they often are).  This is why OSes already have
> >> interfaces to allocate DMA-able memory and to translate between
the
> >> address that CPU will map and the one the peripheral should use.
 If
> >> your driver''s not using the right address-space to
program its hardware
> >> it is broken; the Xen PV model is just an extreme case of this.
> >
> > Indeed.
> >
> > When passing addresses to devices, drivers should use the DMA APIs to
> > make sure that proper address conversions take place (see
> > Documentation/DMA-API-HOWTO.txt).
> > On Xen the swiotlb implementation takes care of doing pfn to mfn
> > conversions as well.
> > For example, to allocate a consistent buffer that is addressable by
both
> > a device and the cpu, the correct API is dma_alloc_coherent, that
> > returns two values: the virtual address which you can use to access it
> > from the CPU and dma_handle which you pass to the device.
> >
> 
> This is the first time to notice swiotlb TBH. It seems to be a good
> start for me.
> 
> After skimming the code in $KERNEL/drivers/xen/swiotlb-xen.c, I have
> further questions.
> 
> It seems that swiotlb is used for PCI passthrough in PV, I''m
wondering
> if paravirt drivers ever use it.
And when running in Dom0.
> 
> > I can very well imagine that virtio drivers are not doing any of this
> > because they didn''t need to, but it would be probably OK to
submit a
> > patch to convert the drivers to use the proper APIs.
> >
> 
> No, they are not doing this.
> 
> Do you mean that I replace every allocation/deallocation of virtio
> drivers scatter buffer with DMA API?
Use the pci_map_sg calls and follow it with pci_sync calls.
The DMA API mentions how to do it properly. Countless drivers do it.
> 
> Then, with ''swiotlb=force'' in pv guest kernel command
line, these
> DMA-API will call xen-swiotlb and get proper addresses?
Correct. Thought you don''t need to use
''swiotlb=force''. You can
just do ''iommu=soft'' and that will take care of it in PV case.

In Dom0 case there is no need for this argument.
> 
> If I don''t add ''swiotlb=force'', which underlying
implementation will
> be used? And, which DMA-API will be used in hvm case? will this change
> affect hvm guest?
For HVM if the machine has less than 4GB it will use the no_dma implementation.
Which is to say: phys_to_virt and virt_to_phys for all DMA calls.

If you have more than 4GB (or iommu=soft) it will use the native
SWIOTLB implementation.

(And the native SWIOTLB can''t properly do PFN -> MFN or vice
versa).
> 
> Wei.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel