Nouveau - Jun 2022 - [PATCH 03/13] mm: shmem: provide oom badness for shmem files

Am 10.06.22 um 16:16 schrieb Michal Hocko:
> [...]
>>> The primary question is whether it actually helps much or what kind
of
>>> scenarios it can help with and whether we can actually do better
for
>>> those.
>> Well, it does help massively with a standard Linux desktop and GPU
workloads
>> (e.g. games).
>>
>> See what currently happens is that when games allocate for example
textures
>> the memory for that is not accounted against that game. Instead
it's usually
>> the display server (X or Wayland) which most of the shared resources
>> accounts to because it needs to compose a desktop from it and usually
also
>> mmaps it for fallback CPU operations.
> Let me try to understand some more. So the game (or the entity to be
> responsible for the resource) doesn't really allocate the memory but it
> relies on somebody else (from memcg perspective living in a different
> resource domain - i.e. a different memcg) to do that on its behalf.
> Correct? If that is the case then that is certainly not fitting into the
> memcg model then.
More or less: yes, that is one possible use case.? But we could leave 
that one out since it is not the primary use case.

What happens more is that 99% of the resources are only allocated per 
process, but around 1% are shared with somebody else.

But see two comments below of a better description of the problem I'm 
facing.
> I am not really sure there is any reasonable model where you cannot
> really tell who is responsible for the resource.
Well it would be fine with me to leave out those 1% of resources shared 
with different memcgs.

What breaks my neck are those 99% which are allocated by a game and 
could potentially be shared but are most of the time not.
>> So what happens when a games over allocates texture resources is that
your
>> whole desktop restarts because the compositor is killed. This obviously
also
>> kills the game, but it would be much nice if we would be more selective
>> here.
>>
>> For hardware rendering DMA-buf and GPU drivers are used, but for the
>> software fallback shmem files is what is used under the hood as far as
I
>> know. And the underlying problem is the same for both.
> For shmem files the end user of the buffer can preallocate and so own
> the buffer and be accounted for it.
The problem is just that it can easily happen that one process is 
allocating the resource and a different one freeing it.

So just imaging the following example: Process opens X window, get 
reference to the handle of the buffer backing this window for drawing, 
tells X to close the window again and then a bit later closes the buffer 
handle.

In this example the X server would be charged allocating the buffer and 
the client (which is most likely in a different memcg group) is charged 
freeing it.

I could of course add something to struct page to track which memcg (or 
process) it was charged against, but extending struct page is most 
likely a no-go.

Alternative I could try to track the "owner" of a buffer (e.g. a shmem
file), but then it can happen that one processes creates the object and 
another one is writing to it and actually allocating the memory.
>>> Also do not forget that shared file memory is not the only thing
>>> to care about. What about the kernel memory used on behalf of
processes?
>> Yeah, I'm aware of that as well. But at least inside the GPU
drivers we try
>> to keep that in a reasonable ratio.
>>
>>> Just consider the above mentioned memcg driven model. It
doesn't really
>>> require to chase specific files and do some arbitrary math to share
the
>>> responsibility. It has a clear accounting and responsibility model.
>> Ok, how does that work then?
> The memory is accounted to whoever faults that memory in or to the
> allocating context if that is a kernel memory (in most situations).
That's what I had in mind as well. Problem with this approach is that 
file descriptors are currently not informed that they are shared between 
processes.

So to make this work we would need something like attach/detach to 
process in struct file_operations.

And as I noted, this happens rather often. For example a game which 
renders 120 frames per second needs to transfer 120 buffers per second 
between client and X.

So this is not something which could take a lot of time and the file 
descriptor tracking structures in the Linux kernel are not made for this 
either.

I think for now I will try something like this specific for DRM drivers. 
That doesn't solve the shmem file problem, but it at least gives me 
something at hand for the accelerated Linux desktop case.

Regards,
Christian.

On Sat 11-06-22 10:06:18, Christian K?nig wrote:
> Am 10.06.22 um 16:16 schrieb Michal Hocko:
[...]
> > > So what happens when a games over allocates texture resources is
that your
> > > whole desktop restarts because the compositor is killed. This
obviously also
> > > kills the game, but it would be much nice if we would be more
selective
> > > here.
> > > 
> > > For hardware rendering DMA-buf and GPU drivers are used, but for
the
> > > software fallback shmem files is what is used under the hood as
far as I
> > > know. And the underlying problem is the same for both.
> > For shmem files the end user of the buffer can preallocate and so own
> > the buffer and be accounted for it.
> 
> The problem is just that it can easily happen that one process is
allocating
> the resource and a different one freeing it.
> 
> So just imaging the following example: Process opens X window, get
reference
> to the handle of the buffer backing this window for drawing, tells X to
> close the window again and then a bit later closes the buffer handle.
> 
> In this example the X server would be charged allocating the buffer and the
> client (which is most likely in a different memcg group) is charged freeing
> it.
Thanks for the clarification.
> I could of course add something to struct page to track which memcg (or
> process) it was charged against, but extending struct page is most likely a
> no-go.
Struct page already maintains is memcg. The one which has charged it and
it will stay constatnt throughout of the allocation lifetime (cgroup v1
has a concept of the charge migration but this hasn't been adopted in
v2).

We have a concept of active_memcg which allows to charge against a
different memcg than the allocating context. From your example above I
do not think this is really usable for the described usecase as the X is
not aware where the request comes from?
> Alternative I could try to track the "owner" of a buffer (e.g. a
shmem
> file), but then it can happen that one processes creates the object and
> another one is writing to it and actually allocating the memory.
If you can enforce that the owner is really responsible for the
allocation then all should be fine. That would require MAP_POPULATE like
semantic and I suspect this is not really feasible with the existing
userspace. It would be certainly hard to enforce for bad players.
-- 
Michal Hocko
SUSE Labs

Am 13.06.22 um 11:08 schrieb Michel D?nzer:
> On 2022-06-11 10:06, Christian K?nig wrote:
>> Am 10.06.22 um 16:16 schrieb Michal Hocko:
>>> [...]
>>>>> Just consider the above mentioned memcg driven model. It
doesn't really
>>>>> require to chase specific files and do some arbitrary math
to share the
>>>>> responsibility. It has a clear accounting and
responsibility model.
>>>> Ok, how does that work then?
>>> The memory is accounted to whoever faults that memory in or to the
>>> allocating context if that is a kernel memory (in most situations).
>> That's what I had in mind as well. Problem with this approach is
that file descriptors are currently not informed that they are shared between
processes.
>>
>> So to make this work we would need something like attach/detach to
process in struct file_operations.
>>
>> And as I noted, this happens rather often. For example a game which
renders 120 frames per second needs to transfer 120 buffers per second between
client and X.
> FWIW, in the steady state, the game will cycle between a small (generally
2-5) set of buffers. The game will not cause new buffers to be exported &
imported for every frame.
>
> In general, I'd expect dma-buf export & import to happen relatively
rarely, e.g. when a window is opened or resized.
Yeah, on a normal Linux desktop. Just unfortunately not on Android :)

Anyway even when this only happens on game start we can't go over all 
the processes/fds and check where a DMA-buf is opened to account this 
against each process.

We would need to add callbacks for this to make it work halve way reliable.

Christian.

On 2022-06-11 10:06, Christian K?nig wrote:
> Am 10.06.22 um 16:16 schrieb Michal Hocko:
>> [...]
>>>> Just consider the above mentioned memcg driven model. It
doesn't really
>>>> require to chase specific files and do some arbitrary math to
share the
>>>> responsibility. It has a clear accounting and responsibility
model.
>>> Ok, how does that work then?
>> The memory is accounted to whoever faults that memory in or to the
>> allocating context if that is a kernel memory (in most situations).
> 
> That's what I had in mind as well. Problem with this approach is that
file descriptors are currently not informed that they are shared between
processes.
> 
> So to make this work we would need something like attach/detach to process
in struct file_operations.
> 
> And as I noted, this happens rather often. For example a game which renders
120 frames per second needs to transfer 120 buffers per second between client
and X.
FWIW, in the steady state, the game will cycle between a small (generally 2-5)
set of buffers. The game will not cause new buffers to be exported &
imported for every frame.

In general, I'd expect dma-buf export & import to happen relatively
rarely, e.g. when a window is opened or resized.


-- 
Earthling Michel D?nzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer

Am 13.06.22 um 09:45 schrieb Michal Hocko:
> On Sat 11-06-22 10:06:18, Christian K?nig wrote:
>> Am 10.06.22 um 16:16 schrieb Michal Hocko:
> [...]
>> I could of course add something to struct page to track which memcg (or
>> process) it was charged against, but extending struct page is most
likely a
>> no-go.
> Struct page already maintains is memcg. The one which has charged it and
> it will stay constatnt throughout of the allocation lifetime (cgroup v1
> has a concept of the charge migration but this hasn't been adopted in
> v2).
>
> We have a concept of active_memcg which allows to charge against a
> different memcg than the allocating context. From your example above I
> do not think this is really usable for the described usecase as the X is
> not aware where the request comes from?
Well X/Wayland is aware, but not the underlying kernel drivers.

When X/Wayland would want to forward this information to the kernel we 
would need to extend the existing UAPI quite a bit. And that of course 
doesn't help us at all with existing desktops.
>> Alternative I could try to track the "owner" of a buffer
(e.g. a shmem
>> file), but then it can happen that one processes creates the object and
>> another one is writing to it and actually allocating the memory.
> If you can enforce that the owner is really responsible for the
> allocation then all should be fine. That would require MAP_POPULATE like
> semantic and I suspect this is not really feasible with the existing
> userspace. It would be certainly hard to enforce for bad players.
I've tried this today and the result was: "BUG: Bad rss-counter state 
mm:000000008751d9ff type:MM_FILEPAGES val:-571286".

The problem is once more that files are not informed when the process 
clones. So what happened is that somebody called fork() with an 
mm_struct I've accounted my pages to. The result is just that we messed 
up the rss_stats and? the the "BUG..." above.

The key difference between normal allocated pages and the resources here 
is just that we are not bound to an mm_struct in any way.

I could just potentially add a dummy VMA to the mm_struct, but to be 
honest I think that this would just be an absolutely hack.

So I'm running out of ideas how to fix this, except for adding this per 
file oom badness like I proposed.

Regards,
Christian.