Linux Virtualization - Jul 2018 - [PATCH v35 1/5] mm: support to get hints of free page blocks

On 07/11/2018 05:21 PM, Michal Hocko wrote:
> On Tue 10-07-18 18:44:34, Linus Torvalds wrote:
> [...]
>> That was what I tried to encourage with actually removing the pages
>> form the page list. That would be an _incremental_ interface. You can
>> remove MAX_ORDER-1 pages one by one (or a hundred at a time), and mark
>> them free for ballooning that way. And if you still feel you have tons
>> of free memory, just continue removing more pages from the free list.
> We already have an interface for that. alloc_pages(GFP_NOWAIT, MAX_ORDER
-1).
> So why do we need any array based interface?
Yes, I'm trying to get free pages directly via alloc_pages, so there 
will be no new mm APIs.
I plan to let free page allocation stop when the remaining system free 
memory becomes close to min_free_kbytes (prevent swapping).


Best,
Wei

On Wed 11-07-18 18:52:45, Wei Wang wrote:
> On 07/11/2018 05:21 PM, Michal Hocko wrote:
> > On Tue 10-07-18 18:44:34, Linus Torvalds wrote:
> > [...]
> > > That was what I tried to encourage with actually removing the
pages
> > > form the page list. That would be an _incremental_ interface. You
can
> > > remove MAX_ORDER-1 pages one by one (or a hundred at a time), and
mark
> > > them free for ballooning that way. And if you still feel you have
tons
> > > of free memory, just continue removing more pages from the free
list.
> > We already have an interface for that. alloc_pages(GFP_NOWAIT,
MAX_ORDER -1).
> > So why do we need any array based interface?
> 
> Yes, I'm trying to get free pages directly via alloc_pages, so there
will be
> no new mm APIs.
OK. The above was just a rough example. In fact you would need a more
complex gfp mask. I assume you only want to balloon only memory directly
usable by the kernel so it will be
	(GFP_KERNEL | __GFP_NOWARN) & ~__GFP_RECLAIM
> I plan to let free page allocation stop when the remaining system free
> memory becomes close to min_free_kbytes (prevent swapping).
~__GFP_RECLAIM will make sure you are allocate as long as there is any
memory without reclaim. It will not even poke the kswapd to do the
background work. So I do not think you would need much more than that.

But let me note that I am not really convinced how this (or previous)
approach will really work in most workloads. We tend to cache heavily so
there is rarely any memory free.

-- 
Michal Hocko
SUSE Labs

On Wednesday, July 11, 2018 7:10 PM, Michal Hocko wrote:
> On Wed 11-07-18 18:52:45, Wei Wang wrote:
> > On 07/11/2018 05:21 PM, Michal Hocko wrote:
> > > On Tue 10-07-18 18:44:34, Linus Torvalds wrote:
> > > [...]
> > > > That was what I tried to encourage with actually removing
the
> > > > pages form the page list. That would be an _incremental_
> > > > interface. You can remove MAX_ORDER-1 pages one by one (or a
> > > > hundred at a time), and mark them free for ballooning that
way.
> > > > And if you still feel you have tons of free memory, just
continue
> removing more pages from the free list.
> > > We already have an interface for that. alloc_pages(GFP_NOWAIT,
> MAX_ORDER -1).
> > > So why do we need any array based interface?
> >
> > Yes, I'm trying to get free pages directly via alloc_pages, so
there
> > will be no new mm APIs.
> 
> OK. The above was just a rough example. In fact you would need a more
> complex gfp mask. I assume you only want to balloon only memory directly
> usable by the kernel so it will be
> 	(GFP_KERNEL | __GFP_NOWARN) & ~__GFP_RECLAIM
Sounds good to me, thanks.
> 
> > I plan to let free page allocation stop when the remaining system free
> > memory becomes close to min_free_kbytes (prevent swapping).
> 
> ~__GFP_RECLAIM will make sure you are allocate as long as there is any
> memory without reclaim. It will not even poke the kswapd to do the
> background work. So I do not think you would need much more than that.
"close to min_free_kbytes" - I meant when doing the allocations, we
intentionally reserve some small amount of memory, e.g. 2 free page blocks of
"MAX_ORDER - 1". So when other applications happen to do some
allocation, they may easily get some from the reserved memory left on the free
list. Without that reserved memory, other allocation may cause the system free
memory below the WMARK[MIN], and kswapd would start to do swapping. This is
actually just a small optimization to reduce the probability of causing swapping
(nice to have, but not mandatary because we will allocate free page blocks one
by one).

 > But let me note that I am not really convinced how this (or
previous)
> approach will really work in most workloads. We tend to cache heavily so
> there is rarely any memory free.
With less free memory, the improvement becomes less, but should be nicer than no
optimization. For example, the Linux build workload would cause 4~5 GB (out of
8GB) memory to be used as page cache at the final stage, there is still ~44%
live migration time reduction.

Since we have many cloud customers interested in this feature, I think we can
let them test the usefulness.

Best,
Wei

On Wed, Jul 11, 2018 at 01:09:49PM +0200, Michal Hocko
wrote:
> But let me note that I am not really convinced how this (or previous)
> approach will really work in most workloads. We tend to cache heavily so
> there is rarely any memory free.
It might be that it's worth flushing the cache when VM is
migrating. Or maybe we should implement virtio-tmem or add
transcendent memory support to the balloon.

-- 
MST