Nouveau - Oct 2024 - [PATCH v1 1/4] mm/hmm: HMM API for P2P DMA to device zone pages

On Wed, Oct 16, 2024 at 12:44:28PM -0300, Jason Gunthorpe
wrote:
> > We are talking about P2P memory here.  How do you manage to get a page
> > that dma_map_page can be used on?  All P2P memory needs to use the P2P
> > aware dma_map_sg as the pages for P2P memory are just fake zone device
> > pages.
> 
> FWIW, I've been expecting this series to be rebased on top of
Leon's
> new DMA API series so it doesn't have this issue..
That's not going to make a difference at this level.
> I'm guessing they got their testing done so far on a system without an
> iommu translation?
IOMMU or not doens't matter much for P2P.  The important difference is
through the host bridge or through a switch.  dma_map_page will work
for P2P through the host brige (assuming the host bridge even support
it as it also lacks the error handling for when not), but it lacks the
handling for P2P through a switch.
> 
> > which also makes it clear that returning a page from the method is
> > not that great, a PFN might work a lot better, e.g.
> > 
> > 	unsigned long (*device_private_dma_pfn)(struct page *page);
> 
> Ideally I think we should not have the struct page * at all through
> these APIs if we can avoid it..
The input page is the device private page that we have at hand
anyway.  Until that scheme is complete redone it is the right kind
of parameter.

On Wed, Oct 16, 2024 at 09:41:03AM -0700, Christoph Hellwig
wrote:
> On Wed, Oct 16, 2024 at 12:44:28PM -0300, Jason Gunthorpe wrote:
> > > We are talking about P2P memory here.  How do you manage to get a
page
> > > that dma_map_page can be used on?  All P2P memory needs to use
the P2P
> > > aware dma_map_sg as the pages for P2P memory are just fake zone
device
> > > pages.
> > 
> > FWIW, I've been expecting this series to be rebased on top of
Leon's
> > new DMA API series so it doesn't have this issue..
> 
> That's not going to make a difference at this level.
I'm not sure what you are asking then.

Patch 2 does pci_p2pdma_add_resource() and so a valid struct page with
a P2P ZONE_DEVICE type exists, and that gets returned back to the
hmm/odp code.

Today odp calls dma_map_page() which only works by chance in limited
cases. With Leon's revision it will call hmm_dma_map_pfn() ->
dma_iova_link() which does call pci_p2pdma_map_type() and should do
the right thing.
> > I'm guessing they got their testing done so far on a system
without an
> > iommu translation?
> 
> IOMMU or not doens't matter much for P2P.  The important difference is
> through the host bridge or through a switch.  dma_map_page will work
> for P2P through the host brige (assuming the host bridge even support
> it as it also lacks the error handling for when not), but it lacks the
> handling for P2P through a switch.
On most x86 systems the BAR/bus address of the P2P memory is the same
as the CPU address, so without an IOMMU translation dma_map_page()
will return the CPU/host physical address which is the same as the
BAR/bus address and that will take the P2P switch path for testing.

Jason