Nouveau - Feb 2025 - [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

On Tue, 2025-02-04 at 09:26 -0400, Jason Gunthorpe
wrote:
> On Tue, Feb 04, 2025 at 10:32:32AM +0100, Thomas Hellstr?m wrote:
> > 
> 
> > 1) Existing users would never use the callback. They can still rely
> > on
> > the owner check, only if that fails we check for callback
> > existence.
> > 2) By simply caching the result from the last checked dev_pagemap,
> > most
> > callback calls could typically be eliminated.
> 
> But then you are not in the locked region so your cache is racy and
> invalid.
I'm not sure I follow? If a device private pfn handed back to the
caller is dependent on dev_pagemap A having a fast interconnect to the
client, then subsequent pfns in the same hmm_range_fault() call must be
able to make the same assumption (pagemap A having a fast
interconnect), else the whole result is invalid?
> 
> > 3) As mentioned before, a callback call would typically always be
> > followed by either migration to ram or a page-table update.
> > Compared to
> > these, the callback overhead would IMO be unnoticeable.
> 
> Why? Surely the normal case should be a callback saying the memory
> can
> be accessed?
Sure, but at least on the xe driver, that means page-table repopulation
since the hmm_range_fault() typically originated from a page-fault.

> 
> > 4) pcie_p2p is already planning a dev_pagemap callback?
> 
> Yes, but it is not a racy validation callback, and it already is
> creating a complicated lifecycle problem inside the exporting the
> driver.
Yeah, I bet there are various reasons against a callback. I just don't
see the performance argument being a main concern. 
> 
> Jason
/Thomas

On Tue, Feb 04, 2025 at 03:29:48PM +0100, Thomas Hellstr?m
wrote:
> On Tue, 2025-02-04 at 09:26 -0400, Jason Gunthorpe wrote:
> > On Tue, Feb 04, 2025 at 10:32:32AM +0100, Thomas Hellstr?m wrote:
> > > 
> > 
> > > 1) Existing users would never use the callback. They can still
rely
> > > on
> > > the owner check, only if that fails we check for callback
> > > existence.
> > > 2) By simply caching the result from the last checked
dev_pagemap,
> > > most
> > > callback calls could typically be eliminated.
> > 
> > But then you are not in the locked region so your cache is racy and
> > invalid.
> 
> I'm not sure I follow? If a device private pfn handed back to the
> caller is dependent on dev_pagemap A having a fast interconnect to the
> client, then subsequent pfns in the same hmm_range_fault() call must be
> able to make the same assumption (pagemap A having a fast
> interconnect), else the whole result is invalid?
But what is the receiver going to do with this device private page?
Relock it again and check again if it is actually OK? Yuk.
> > > 3) As mentioned before, a callback call would typically always be
> > > followed by either migration to ram or a page-table update.
> > > Compared to
> > > these, the callback overhead would IMO be unnoticeable.
> > 
> > Why? Surely the normal case should be a callback saying the memory
> > can
> > be accessed?
> 
> Sure, but at least on the xe driver, that means page-table repopulation
> since the hmm_range_fault() typically originated from a page-fault.
Yes, I expect all hmm_range_fault()'s to be on page fault paths, and
we'd like it to be as fast as we can in the CPU present case..

Jason