Xen devel - Jan 2007 - [Patch] the interface of invalidating qemu mapcache

HVM balloon driver or something, that''s under development, may decrease
or increase the machine memory that is taken by HVM guest; in IA32/IA32e
host, now Qemu maps the physical memory of HVM guest based on little
blocks of memory (the block size is 64K in IA32 host or 1M in IA32E
host). When HVM balloon driver decreases the reserved machine memory of
HVM guest, Qemu should unmap the related physical memory range of HVM
guest so that when HVM balloon driver increases the reserved machine
memory of HVM guest later, Qemu can map and access the physical memory
of HVM guest correctly.

The patch exports a function invalidate_qemu_mapcache() in
unmodified_drivers/linux-2.6/platform-pci/platform-pci.c, so HVM balloon
driver can use it directly. Note: considering IA32 guest may run on both
IA32 host and IA32E host, the size of the memory block HVM balloon
driver wants to release MUST be 1M bytes and the address of the memory
block MUST be 1M-byte boundary aligned; if there are several such 1M
memory blocks, HVM balloon driver should call invalidate_qemu_mapcache()
several times, 1 block each time. The execution of
invalidate_qemu_mapcache() is not slow -- 1000 times of invocation may
take about 0.4 second.

-- Dexuan

Signed-off-by: Dexuan Cui <dexuan.cui@intel.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/1/07 4:27 pm, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:
> The patch exports a function invalidate_qemu_mapcache() in
> unmodified_drivers/linux-2.6/platform-pci/platform-pci.c, so HVM balloon
> driver can use it directly. Note: considering IA32 guest may run on both
> IA32 host and IA32E host, the size of the memory block HVM balloon
> driver wants to release MUST be 1M bytes and the address of the memory
> block MUST be 1M-byte boundary aligned; if there are several such 1M
> memory blocks, HVM balloon driver should call invalidate_qemu_mapcache()
> several times, 1 block each time. The execution of
> invalidate_qemu_mapcache() is not slow -- 1000 times of invocation may
> take about 0.4 second.
The balloon driver is likely to free up pages scattered all over the memory
map. Since it does this in batches it will probably be easiest to simply
blow the entire mapcache at the end of a batch. This will make your patch a
lot smaller!

Making effective use of 1MB selective zapping will actually be hard for the
balloon driver -- either it will make a zap request for each page it frees
(which will be slow) or it will need to maintain a bitmap it fills in during
its batched work which it then scans-and-zaps at the end. It is probably not
worth the effort.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>
>The balloon driver is likely to free up pages scattered all over the
memory
>map. Since it does this in batches it will probably be easiest to
simply
>blow the entire mapcache at the end of a batch. This will make your
patch a
>lot smaller!
>
Yes, it''s much simpler if we just blow the entire mapcache each time we
release memory to Xen or get some back from Xen.
-Xin 

>Making effective use of 1MB selective zapping will actually be hard for
the
>balloon driver -- either it will make a zap request for each page it
frees
>(which will be slow) or it will need to maintain a bitmap it fills in
during
>its batched work which it then scans-and-zaps at the end. It is
probably not
>worth the effort.
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/1/07 7:33 pm, "Li, Xin B" <xin.b.li@intel.com> wrote:
> Yes, it''s much simpler if we just blow the entire mapcache each
time we
> release memory to Xen or get some back from Xen.
For populating memory, qemu should probably refill the mapcache on demand.
It ought to be catching page faults on access to the mapcache anyway (easy,
just need a SIGSEGV handler). If it detects a mapcache access it should
first attempt to refill that mapcache entry and if that fails then do
something appropriate (crash the domain?).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,
Attached is the updated version. The exported qemu_invalidate_map_cache() just
blows the entire qemu mapcache.
Would you please give some comments? Thanks a lot!

 -- Dexuan
Signed-off-by: Dexuan Cui <dexuan.cui@intel.com>
>-----Original Message-----
>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007年1月27日 17:03
>To: Li, Xin B; Cui, Dexuan; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] Re: [Patch] the interface of invalidating qemu
mapcache
>
>On 26/1/07 7:33 pm, "Li, Xin B" <xin.b.li@intel.com> wrote:
>
>> Yes, it''s much simpler if we just blow the entire mapcache
each time we
>> release memory to Xen or get some back from Xen.
>
>For populating memory, qemu should probably refill the mapcache on demand.
>It ought to be catching page faults on access to the mapcache anyway (easy,
>just need a SIGSEGV handler). If it detects a mapcache access it should
>first attempt to refill that mapcache entry and if that fails then do
>something appropriate (crash the domain?).
>
> -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 27/1/07 10:29 am, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:
> Keir,
> Attached is the updated version. The exported qemu_invalidate_map_cache()
just
> blows the entire qemu mapcache.
> Would you please give some comments? Thanks a lot!
Allocating the PCI resources with an incrementing region_num counter is
pointless (and in fact confusing) given that the BAR for mmio and portio
resources are hardcoded as 1 and 0 (respectively) in the pv-on-hvm driver
code. Also the existing portio resource is (I believe) a placeholder. Thus
you don''t need to create a new resource -- use the first port of
resource
region 0 instead. The existing read/write handler registrations in
platform_io_map() are pointless. You can remove them and replace with a
single-byte write handler to blow the mapcache -- there''s no need to
install
handlers for 2- or 4-byte accesses, nor for read accesses.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,
Attached is the latest version of the patch.
Thanks very much for your comments!

 -- Dexuan
>-----Original Message-----
>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007年1月27日 18:41
>To: Cui, Dexuan; Li, Xin B; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] Re: [Patch] the interface of invalidating qemu
mapcache
>
>On 27/1/07 10:29 am, "Cui, Dexuan" <dexuan.cui@intel.com>
wrote:
>
>> Keir,
>> Attached is the updated version. The exported
qemu_invalidate_map_cache() just
>> blows the entire qemu mapcache.
>> Would you please give some comments? Thanks a lot!
>
>Allocating the PCI resources with an incrementing region_num counter is
>pointless (and in fact confusing) given that the BAR for mmio and portio
>resources are hardcoded as 1 and 0 (respectively) in the pv-on-hvm driver
>code. Also the existing portio resource is (I believe) a placeholder. Thus
>you don''t need to create a new resource -- use the first port of
resource
>region 0 instead. The existing read/write handler registrations in
>platform_io_map() are pointless. You can remove them and replace with a
>single-byte write handler to blow the mapcache -- there''s no need
to install
>handlers for 2- or 4-byte accesses, nor for read accesses.
>
> -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Looks good, I''ll check it in.

 -- Keir

On 27/1/07 11:46 am, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:
> Keir,
> Attached is the latest version of the patch.
> Thanks very much for your comments!
> 
>  -- Dexuan
> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>> Sent: 2007年1月27日 18:41
>> To: Cui, Dexuan; Li, Xin B; xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] Re: [Patch] the interface of invalidating qemu
>> mapcache
>> 
>> On 27/1/07 10:29 am, "Cui, Dexuan"
<dexuan.cui@intel.com> wrote:
>> 
>>> Keir,
>>> Attached is the updated version. The exported
qemu_invalidate_map_cache()
>>> just
>>> blows the entire qemu mapcache.
>>> Would you please give some comments? Thanks a lot!
>> 
>> Allocating the PCI resources with an incrementing region_num counter is
>> pointless (and in fact confusing) given that the BAR for mmio and
portio
>> resources are hardcoded as 1 and 0 (respectively) in the pv-on-hvm
driver
>> code. Also the existing portio resource is (I believe) a placeholder.
Thus
>> you don''t need to create a new resource -- use the first port
of resource
>> region 0 instead. The existing read/write handler registrations in
>> platform_io_map() are pointless. You can remove them and replace with a
>> single-byte write handler to blow the mapcache -- there''s no
need to install
>> handlers for 2- or 4-byte accesses, nor for read accesses.
>> 
>> -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Keir,
> > Attached is the updated version. The exported
qemu_invalidate_map_cache()
> just
> > blows the entire qemu mapcache.
> > Would you please give some comments? Thanks a lot!
> 
> Allocating the PCI resources with an incrementing region_num counter
is
> pointless (and in fact confusing) given that the BAR for mmio and
portio
> resources are hardcoded as 1 and 0 (respectively) in the pv-on-hvm
driver
> code. Also the existing portio resource is (I believe) a placeholder.
Thus
> you don''t need to create a new resource -- use the first port of
resource
> region 0 instead. The existing read/write handler registrations in
> platform_io_map() are pointless. You can remove them and replace with
a
> single-byte write handler to blow the mapcache -- there''s no need
to
> install handlers for 2- or 4-byte accesses, nor for read accesses.
Why are we triggering this with an ioport access rather than just adding
a new message type between xen and qemu-dm? The latter seems rather
cleaner.

Ian 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Pratt wrote:
>>> Keir,
>>> Attached is the updated version. The exported
>>> qemu_invalidate_map_cache() just blows the entire qemu mapcache.
>>> Would you please give some comments? Thanks a lot!
>> 
>> Allocating the PCI resources with an incrementing region_num counter
>> is pointless (and in fact confusing) given that the BAR for mmio and
>> portio resources are hardcoded as 1 and 0 (respectively) in the
>> pv-on-hvm driver code. Also the existing portio resource is (I
>> believe) a placeholder. Thus you don''t need to create a new
resource
>> -- use the first port of resource region 0 instead. The existing
>> read/write handler registrations in platform_io_map() are pointless.
>> You can remove them and replace with a single-byte write handler to
>> blow the mapcache -- there''s no need to install handlers for
2- or
>> 4-byte accesses, nor for read accesses. 
> 
> Why are we triggering this with an ioport access rather than just
> adding a new message type between xen and qemu-dm? The latter seems
> rather cleaner.
> 
> Ian
Since qemu-dm is reponsible for I/O events, I think it''s natural (i.e.
kind of chipset featrue) to construct and send an I/O event to qemu-dm
to trigger mapcache invalidation. It does not mean the guest needs to
use an I/O instruction, but our plan is that Xen sends a mapcache
invalidation message and it''s implemented as an I/O event for HVM
guests.

Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> >> Allocating the PCI resources with an incrementing region_num
counter
> >> is pointless (and in fact confusing) given that the BAR for mmio
and
> >> portio resources are hardcoded as 1 and 0 (respectively) in the
> >> pv-on-hvm driver code. Also the existing portio resource is (I
> >> believe) a placeholder. Thus you don''t need to create a
new
resource
> >> -- use the first port of resource region 0 instead. The existing
> >> read/write handler registrations in platform_io_map() are
pointless.
> >> You can remove them and replace with a single-byte write handler
to
> >> blow the mapcache -- there''s no need to install handlers
for 2- or
> >> 4-byte accesses, nor for read accesses.
> >
> > Why are we triggering this with an ioport access rather than just
> > adding a new message type between xen and qemu-dm? The latter seems
> > rather cleaner.
> 
> Since qemu-dm is reponsible for I/O events, I think it''s natural
(i.e.
> kind of chipset featrue) to construct and send an I/O event to qemu-dm
> to trigger mapcache invalidation. It does not mean the guest needs to
> use an I/O instruction, but our plan is that Xen sends a mapcache
> invalidation message and it''s implemented as an I/O event for HVM
> guests.
Ah, OK -- so this is just for testing purposes. Seems sensible.
Thanks,
Ian



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 27/1/07 3:27 pm, "Nakajima, Jun" <jun.nakajima@intel.com>
wrote:
>> Why are we triggering this with an ioport access rather than just
>> adding a new message type between xen and qemu-dm? The latter seems
>> rather cleaner.
>> 
>> Ian
> 
> Since qemu-dm is reponsible for I/O events, I think it''s natural
(i.e.
> kind of chipset featrue) to construct and send an I/O event to qemu-dm
> to trigger mapcache invalidation. It does not mean the guest needs to
> use an I/O instruction, but our plan is that Xen sends a mapcache
> invalidation message and it''s implemented as an I/O event for HVM
> guests.
If Xen is going to trigger the mapcache zap then the additions to the
pv-on-hvm driver package itself doesn''t make much sense.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Since qemu-dm is reponsible for I/O events, I think it''s
natural
(i.e.
> > kind of chipset featrue) to construct and send an I/O event to
qemu-dm
> > to trigger mapcache invalidation. It does not mean the guest needs
to
> > use an I/O instruction, but our plan is that Xen sends a mapcache
> > invalidation message and it''s implemented as an I/O event for
HVM
> > guests.
> 
> If Xen is going to trigger the mapcache zap then the additions to the
> pv-on-hvm driver package itself doesn''t make much sense.
Yeah, but I can see how its useful hack to have for testability, which
is a good thing -- I''d leave it in place even when we get xen
triggering
the flush.

In fact, I''d recommend we try getting the linux PV mem driver running
on
HVM (using the magic io port write) before starting on the xen side of
things.

Ian



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel