Nouveau - May 2021 - [PATCH v8 5/8] mm: Device exclusive memory access

On Tue, May 18, 2021 at 01:27:42PM -0400, Peter Xu wrote:
> I also have a pure and high level question regarding a process fork() when
> there're device exclusive ptes: would the two processes then own the
device
> together?  Is this a real usecase?
If the pages are MAP_SHARED then yes. All VMAs should point at the
same device_exclusive page and all VMA should migrate back to CPU
pages together.
> Indeed it'll be odd for a COW page since for COW page then it means
after
> parent/child writting to the page it'll clone into two, then it's a
mistery on
> which one will be the one that "exclusived owned" by the device..
For COW pages it is like every other fork case.. We can't reliably
write-protect the device_exclusive page during fork so we must copy it
at fork time.

Thus three reasonable choices:
 - Copy to a new CPU page
 - Migrate back to a CPU page and write protect it
 - Copy to a new device exclusive page

Jason

On Tue, May 18, 2021 at 02:33:34PM -0300, Jason Gunthorpe
wrote:
> On Tue, May 18, 2021 at 01:27:42PM -0400, Peter Xu wrote:
> 
> > I also have a pure and high level question regarding a process fork()
when
> > there're device exclusive ptes: would the two processes then own
the device
> > together?  Is this a real usecase?
> 
> If the pages are MAP_SHARED then yes. All VMAs should point at the
> same device_exclusive page and all VMA should migrate back to CPU
> pages together.
Makes sense.  If we keep the anonymous-only in this series (I think it's
good
to separate these), maybe we can drop the !COW case, plus some proper installed
WARN_ON_ONCE()s.
> 
> > Indeed it'll be odd for a COW page since for COW page then it
means after
> > parent/child writting to the page it'll clone into two, then
it's a mistery on
> > which one will be the one that "exclusived owned" by the
device..
> 
> For COW pages it is like every other fork case.. We can't reliably
> write-protect the device_exclusive page during fork so we must copy it
> at fork time.
> 
> Thus three reasonable choices:
>  - Copy to a new CPU page
>  - Migrate back to a CPU page and write protect it
>  - Copy to a new device exclusive page
IMHO the ownership question would really help us to answer this one..

If the device ownership should be kept in parent IMHO option (1) is the best
approach. To be explicit on page copy: we can do best-effort, even if the copy
is during a device atomic operation, perhaps?

If the ownership will be shared, seems option (3) will be easier as I don't
see
a strong reason to do immediate restorinng of ptes; as long as we're careful
on
the refcounting.

Thanks,

-- 
Peter Xu

On Tue, May 18, 2021 at 02:01:36PM -0400, Peter Xu
wrote:
> > > Indeed it'll be odd for a COW page since for COW page then it
means after
> > > parent/child writting to the page it'll clone into two, then
it's a mistery on
> > > which one will be the one that "exclusived owned" by
the device..
> > 
> > For COW pages it is like every other fork case.. We can't reliably
> > write-protect the device_exclusive page during fork so we must copy it
> > at fork time.
> > 
> > Thus three reasonable choices:
> >  - Copy to a new CPU page
> >  - Migrate back to a CPU page and write protect it
> >  - Copy to a new device exclusive page
> 
> IMHO the ownership question would really help us to answer this one..
I'm confused about what device ownership you are talking about

It is just a page and it is tied to some specific pgmap?

If the thing providing the migration stuff goes away then all
device_exclusive pages should revert back to CPU pages and destroy the
pgmap?

Jason