Nouveau - Jun 2022 - [PATCH 03/13] mm: shmem: provide oom badness for shmem files

Am 09.06.22 um 14:57 schrieb Michal Hocko:
> On Thu 09-06-22 14:16:56, Christian K?nig wrote:
>> Am 09.06.22 um 11:18 schrieb Michal Hocko:
>>> On Tue 31-05-22 11:59:57, Christian K?nig wrote:
>>>> This gives the OOM killer an additional hint which processes
are
>>>> referencing shmem files with potentially no other accounting
for them.
>>>>
>>>> Signed-off-by: Christian K?nig <christian.koenig at
amd.com>
>>>> ---
>>>>    mm/shmem.c | 6 ++++++
>>>>    1 file changed, 6 insertions(+)
>>>>
>>>> diff --git a/mm/shmem.c b/mm/shmem.c
>>>> index 4b2fea33158e..a4ad92a16968 100644
>>>> --- a/mm/shmem.c
>>>> +++ b/mm/shmem.c
>>>> @@ -2179,6 +2179,11 @@ unsigned long
shmem_get_unmapped_area(struct file *file,
>>>>    	return inflated_addr;
>>>>    }
>>>> +static long shmem_oom_badness(struct file *file)
>>>> +{
>>>> +	return i_size_read(file_inode(file)) >> PAGE_SHIFT;
>>>> +}
>>> This doesn't really represent the in memory size of the file,
does it?
>> Well the file could be partially or fully swapped out as anonymous
memory or
>> the address space only sparse populated, but even then just using the
file
>> size as OOM badness sounded like the most straightforward approach to
me.
> It covers hole as well, right?
Yes, exactly.
>
>> What could happen is that the file is also mmaped and we double
account.
>>
>>> Also the memcg oom handling could be considerably skewed if the
file was
>>> shared between more memcgs.
>> Yes, and that's one of the reasons why I didn't touched the
memcg by this
>> and only affected the classic OOM killer.
> oom_badness is for all oom handlers, including memcg. Maybe I have
> misread an earlier patch but I do not see anything specific to global
> oom handling.
As far as I can see the oom_badness() function is only used in oom_kill.c and in
procfs to return the oom score. Did I missed something?

Regards,
Christian.

On Thu 09-06-22 16:10:33, Christian K?nig wrote:
> Am 09.06.22 um 14:57 schrieb Michal Hocko:
> > On Thu 09-06-22 14:16:56, Christian K?nig wrote:
> > > Am 09.06.22 um 11:18 schrieb Michal Hocko:
> > > > On Tue 31-05-22 11:59:57, Christian K?nig wrote:
> > > > > This gives the OOM killer an additional hint which
processes are
> > > > > referencing shmem files with potentially no other
accounting for them.
> > > > > 
> > > > > Signed-off-by: Christian K?nig <christian.koenig at
amd.com>
> > > > > ---
> > > > >    mm/shmem.c | 6 ++++++
> > > > >    1 file changed, 6 insertions(+)
> > > > > 
> > > > > diff --git a/mm/shmem.c b/mm/shmem.c
> > > > > index 4b2fea33158e..a4ad92a16968 100644
> > > > > --- a/mm/shmem.c
> > > > > +++ b/mm/shmem.c
> > > > > @@ -2179,6 +2179,11 @@ unsigned long
shmem_get_unmapped_area(struct file *file,
> > > > >    	return inflated_addr;
> > > > >    }
> > > > > +static long shmem_oom_badness(struct file *file)
> > > > > +{
> > > > > +	return i_size_read(file_inode(file)) >>
PAGE_SHIFT;
> > > > > +}
> > > > This doesn't really represent the in memory size of the
file, does it?
> > > Well the file could be partially or fully swapped out as
anonymous memory or
> > > the address space only sparse populated, but even then just using
the file
> > > size as OOM badness sounded like the most straightforward
approach to me.
> > It covers hole as well, right?
> 
> Yes, exactly.
So let's say I have a huge sparse shmem file. I will get killed because
the oom_badness of such a file would be large as well...
> > > What could happen is that the file is also mmaped and we double
account.
> > > 
> > > > Also the memcg oom handling could be considerably skewed if
the file was
> > > > shared between more memcgs.
> > > Yes, and that's one of the reasons why I didn't touched
the memcg by this
> > > and only affected the classic OOM killer.
> > oom_badness is for all oom handlers, including memcg. Maybe I have
> > misread an earlier patch but I do not see anything specific to global
> > oom handling.
> 
> As far as I can see the oom_badness() function is only used in
> oom_kill.c and in procfs to return the oom score. Did I missed
> something?
oom_kill.c implements most of the oom killer functionality. Memcg oom
killing is a part of that. Have a look at select_bad_process.

-- 
Michal Hocko
SUSE Labs

Am 09.06.22 um 16:21 schrieb Michal Hocko:
> On Thu 09-06-22 16:10:33, Christian K?nig wrote:
>> Am 09.06.22 um 14:57 schrieb Michal Hocko:
>>> On Thu 09-06-22 14:16:56, Christian K?nig wrote:
>>>> Am 09.06.22 um 11:18 schrieb Michal Hocko:
>>>>> On Tue 31-05-22 11:59:57, Christian K?nig wrote:
>>>>>> This gives the OOM killer an additional hint which
processes are
>>>>>> referencing shmem files with potentially no other
accounting for them.
>>>>>>
>>>>>> Signed-off-by: Christian K?nig <christian.koenig at
amd.com>
>>>>>> ---
>>>>>>     mm/shmem.c | 6 ++++++
>>>>>>     1 file changed, 6 insertions(+)
>>>>>>
>>>>>> diff --git a/mm/shmem.c b/mm/shmem.c
>>>>>> index 4b2fea33158e..a4ad92a16968 100644
>>>>>> --- a/mm/shmem.c
>>>>>> +++ b/mm/shmem.c
>>>>>> @@ -2179,6 +2179,11 @@ unsigned long
shmem_get_unmapped_area(struct file *file,
>>>>>>     	return inflated_addr;
>>>>>>     }
>>>>>> +static long shmem_oom_badness(struct file *file)
>>>>>> +{
>>>>>> +	return i_size_read(file_inode(file)) >>
PAGE_SHIFT;
>>>>>> +}
>>>>> This doesn't really represent the in memory size of the
file, does it?
>>>> Well the file could be partially or fully swapped out as
anonymous memory or
>>>> the address space only sparse populated, but even then just
using the file
>>>> size as OOM badness sounded like the most straightforward
approach to me.
>>> It covers hole as well, right?
>> Yes, exactly.
> So let's say I have a huge sparse shmem file. I will get killed because
> the oom_badness of such a file would be large as well...
Yes, correct. But I of hand don't see how we could improve that accounting.
>>>> What could happen is that the file is also mmaped and we double
account.
>>>>
>>>>> Also the memcg oom handling could be considerably skewed if
the file was
>>>>> shared between more memcgs.
>>>> Yes, and that's one of the reasons why I didn't touched
the memcg by this
>>>> and only affected the classic OOM killer.
>>> oom_badness is for all oom handlers, including memcg. Maybe I have
>>> misread an earlier patch but I do not see anything specific to
global
>>> oom handling.
>> As far as I can see the oom_badness() function is only used in
>> oom_kill.c and in procfs to return the oom score. Did I missed
>> something?
> oom_kill.c implements most of the oom killer functionality. Memcg oom
> killing is a part of that. Have a look at select_bad_process.
Ah! So mem_cgroup_scan_tasks() calls oom_evaluate_task for each task in 
the control group.

Thanks for pointing that out, that was absolutely not obvious to me.

Is that a show stopper? How should we address this?

Christian.

Am 2022-06-09 um 10:21 schrieb Michal Hocko:
> On Thu 09-06-22 16:10:33, Christian K?nig wrote:
>> Am 09.06.22 um 14:57 schrieb Michal Hocko:
>>> On Thu 09-06-22 14:16:56, Christian K?nig wrote:
>>>> Am 09.06.22 um 11:18 schrieb Michal Hocko:
>>>>> On Tue 31-05-22 11:59:57, Christian K?nig wrote:
>>>>>> This gives the OOM killer an additional hint which
processes are
>>>>>> referencing shmem files with potentially no other
accounting for them.
>>>>>>
>>>>>> Signed-off-by: Christian K?nig <christian.koenig at
amd.com>
>>>>>> ---
>>>>>>     mm/shmem.c | 6 ++++++
>>>>>>     1 file changed, 6 insertions(+)
>>>>>>
>>>>>> diff --git a/mm/shmem.c b/mm/shmem.c
>>>>>> index 4b2fea33158e..a4ad92a16968 100644
>>>>>> --- a/mm/shmem.c
>>>>>> +++ b/mm/shmem.c
>>>>>> @@ -2179,6 +2179,11 @@ unsigned long
shmem_get_unmapped_area(struct file *file,
>>>>>>     	return inflated_addr;
>>>>>>     }
>>>>>> +static long shmem_oom_badness(struct file *file)
>>>>>> +{
>>>>>> +	return i_size_read(file_inode(file)) >>
PAGE_SHIFT;
>>>>>> +}
>>>>> This doesn't really represent the in memory size of the
file, does it?
>>>> Well the file could be partially or fully swapped out as
anonymous memory or
>>>> the address space only sparse populated, but even then just
using the file
>>>> size as OOM badness sounded like the most straightforward
approach to me.
>>> It covers hole as well, right?
>> Yes, exactly.
> So let's say I have a huge sparse shmem file. I will get killed because
> the oom_badness of such a file would be large as well...
Would killing processes free shmem files, though? Aren't those 
persistent anyway? In that case, shmem files should not contribute to 
oom_badness at all.

I guess a special case would be files that were removed from the 
filesystem but are still open in some processes.

Regards,
 ? Felix

>
>>>> What could happen is that the file is also mmaped and we double
account.
>>>>
>>>>> Also the memcg oom handling could be considerably skewed if
the file was
>>>>> shared between more memcgs.
>>>> Yes, and that's one of the reasons why I didn't touched
the memcg by this
>>>> and only affected the classic OOM killer.
>>> oom_badness is for all oom handlers, including memcg. Maybe I have
>>> misread an earlier patch but I do not see anything specific to
global
>>> oom handling.
>> As far as I can see the oom_badness() function is only used in
>> oom_kill.c and in procfs to return the oom score. Did I missed
>> something?
> oom_kill.c implements most of the oom killer functionality. Memcg oom
> killing is a part of that. Have a look at select_bad_process.
>