Linux Virtualization - Mar 2016 - [Qemu-devel] [RFC qemu 0/4] A PV solution for live migration optimization

On Wed, Mar 09, 2016 at 03:27:54PM +0000, Li, Liang Z
wrote:
> > On Mon, Mar 07, 2016 at 01:40:06PM +0200, Michael S. Tsirkin wrote:
> > > On Mon, Mar 07, 2016 at 06:49:19AM +0000, Li, Liang Z wrote:
> > > > > > No. And it's exactly what I mean. The
ballooned memory is still
> > > > > > processed during live migration without skipping.
The live
> > > > > > migration code is
> > > > > in migration/ram.c.
> > > > >
> > > > > So if guest acknowledged
VIRTIO_BALLOON_F_MUST_TELL_HOST, we
> > can
> > > > > teach qemu to skip these pages.
> > > > > Want to write a patch to do this?
> > > > >
> > > >
> > > > Yes, we really can teach qemu to skip these pages and
it's not hard.
> > > > The problem is the poor performance, this PV solution
> > >
> > > Balloon is always PV. And do not call patches solutions please.
> > >
> > > > is aimed to make it more
> > > > efficient and reduce the performance impact on guest.
> > >
> > > We need to get a bit beyond this.  You are making multiple
changes, it
> > > seems to make sense to split it all up, and analyse each change
> > > separately.
> > 
> > Couldn't agree more.
> > 
> > There are three stages in this optimization:
> > 
> > 1) choosing which pages to skip
> > 
> > 2) communicating them from guest to host
> > 
> > 3) skip transferring uninteresting pages to the remote side on
migration
> > 
> > For (3) there seems to be a low-hanging fruit to amend
> > migration/ram.c:iz_zero_range() to consult /proc/self/pagemap.  This
would
> > work for guest RAM that hasn't been touched yet or which has been
> > ballooned out.
> > 
> > For (1) I've been trying to make a point that skipping clean pages
is much
> > more likely to result in noticable benefit than free pages only.
> > 
> 
> I am considering to drop the pagecache before getting the free pages. 
> 
> > As for (2), we do seem to have a problem with the existing balloon:
> > according to your measurements it's very slow; besides, I guess it
plays badly
> 
> I didn't say communicating is slow. Even this is very slow, my solution
use bitmap instead of
> PFNs, there is fewer data traffic, so it's faster than the existing
balloon which use PFNs.
By how much?
> > with transparent huge pages (as both the guest and the host work with
one
> > 4k page at a time).  This is a problem for other use cases of balloon
(e.g. as a
> > facility for resource management); tackling that appears a more
natural
> > application for optimization efforts.
> > 
> > Thanks,
> > Roman.

> > > > > Yes, we really can teach qemu to skip these pages and
it's not hard.
> > > > > The problem is the poor performance, this PV solution
> > > >
> > > > Balloon is always PV. And do not call patches solutions
please.
> > > >
> > > > > is aimed to make it more
> > > > > efficient and reduce the performance impact on guest.
> > > >
> > > > We need to get a bit beyond this.  You are making multiple
> > > > changes, it seems to make sense to split it all up, and
analyse
> > > > each change separately.
> > >
> > > Couldn't agree more.
> > >
> > > There are three stages in this optimization:
> > >
> > > 1) choosing which pages to skip
> > >
> > > 2) communicating them from guest to host
> > >
> > > 3) skip transferring uninteresting pages to the remote side on
> > > migration
> > >
> > > For (3) there seems to be a low-hanging fruit to amend
> > > migration/ram.c:iz_zero_range() to consult /proc/self/pagemap. 
This
> > > would work for guest RAM that hasn't been touched yet or
which has
> > > been ballooned out.
> > >
> > > For (1) I've been trying to make a point that skipping clean
pages
> > > is much more likely to result in noticable benefit than free
pages only.
> > >
> >
> > I am considering to drop the pagecache before getting the free pages.
> >
> > > As for (2), we do seem to have a problem with the existing
balloon:
> > > according to your measurements it's very slow; besides, I
guess it
> > > plays badly
> >
> > I didn't say communicating is slow. Even this is very slow, my
> > solution use bitmap instead of PFNs, there is fewer data traffic, so
it's
> faster than the existing balloon which use PFNs.
> 
> By how much?
> 
Haven't measured yet. 
To identify a page, 1 bit is needed if using bitmap, 4 Bytes(32bit) is needed if
using PFN,

For a guest with 8GB RAM,  the corresponding free page bitmap size is 256KB.
And the corresponding total PFNs size is 8192KB. Assuming the inflating size
is 7GB, the total PFNs size is 7168KB.

Maybe this is not the point.

Liang
> > > with transparent huge pages (as both the guest and the host work
> > > with one 4k page at a time).  This is a problem for other use
cases
> > > of balloon (e.g. as a facility for resource management); tackling
> > > that appears a more natural application for optimization efforts.
> > >
> > > Thanks,
> > > Roman.

On Thu, Mar 10, 2016 at 01:41:16AM +0000, Li, Liang Z
wrote:
> > > > > > Yes, we really can teach qemu to skip these pages
and it's not hard.
> > > > > > The problem is the poor performance, this PV
solution
> > > > >
> > > > > Balloon is always PV. And do not call patches solutions
please.
> > > > >
> > > > > > is aimed to make it more
> > > > > > efficient and reduce the performance impact on
guest.
> > > > >
> > > > > We need to get a bit beyond this.  You are making
multiple
> > > > > changes, it seems to make sense to split it all up, and
analyse
> > > > > each change separately.
> > > >
> > > > Couldn't agree more.
> > > >
> > > > There are three stages in this optimization:
> > > >
> > > > 1) choosing which pages to skip
> > > >
> > > > 2) communicating them from guest to host
> > > >
> > > > 3) skip transferring uninteresting pages to the remote side
on
> > > > migration
> > > >
> > > > For (3) there seems to be a low-hanging fruit to amend
> > > > migration/ram.c:iz_zero_range() to consult
/proc/self/pagemap.  This
> > > > would work for guest RAM that hasn't been touched yet or
which has
> > > > been ballooned out.
> > > >
> > > > For (1) I've been trying to make a point that skipping
clean pages
> > > > is much more likely to result in noticable benefit than free
pages only.
> > > >
> > >
> > > I am considering to drop the pagecache before getting the free
pages.
> > >
> > > > As for (2), we do seem to have a problem with the existing
balloon:
> > > > according to your measurements it's very slow; besides,
I guess it
> > > > plays badly
> > >
> > > I didn't say communicating is slow. Even this is very slow,
my
> > > solution use bitmap instead of PFNs, there is fewer data traffic,
so it's
> > faster than the existing balloon which use PFNs.
> > 
> > By how much?
> > 
> 
> Haven't measured yet. 
> To identify a page, 1 bit is needed if using bitmap, 4 Bytes(32bit) is
needed if using PFN,
> 
> For a guest with 8GB RAM,  the corresponding free page bitmap size is
256KB.
> And the corresponding total PFNs size is 8192KB. Assuming the inflating
size
> is 7GB, the total PFNs size is 7168KB.
Yes but this is not how balloon works, instead, it will reuse a single
4K page multiple times. We can also trade off more memory for speed
if we want to, it's completely up to guest.
> 
> Maybe this is not the point.
> 
> Liang

> > > > with transparent huge pages (as both the guest and the host
work
> > > > with one 4k page at a time).  This is a problem for other
use cases
> > > > of balloon (e.g. as a facility for resource management);
tackling
> > > > that appears a more natural application for optimization
efforts.
> > > >
> > > > Thanks,
> > > > Roman.