Xen devel - Apr 2006 - Regarding page table management changes from Xen v1 to Xen v2 (and v3)

Hi Folks,

While going through the 2004 OLS presentation on Xen, I noticed that the way PT
management happens changed from v1.2 to v2.0. More particularly in v1.2, guest
would make a hypercall for all the writes to the PT, whilst in v2.0 (and in
v3.0)
, guest tries
to write the PT directly, even knowing that it is mapped RO and would cause a 
fault. Xen then detaches that particular page and maps it RW in guest. Another
access will cause another page fault and which time it will be
validated and re-attached to higher level PT structure.

I am trying to understand the rationale behind this change. In previous case, 
there would be no page faults due to page table updates and only one hypercall.
In the second case, there would be atleast 2 page faults due to PT management
activity, but no hypercalls. Besides, mapping and remapping with different
permissions
imply removing this entry from TLB (which is hopefully being done with invlpg).
Benefit of latter approach only seems to be the removal of xen specific linux 
code. Why was the approach changed? Could someone please shed some light on 
this?

Thanks and best regards,
Himanshu

-- 
-------------------------------------------------------------------------
Himanshu Raj
PhD Student, GaTech (www.cc.gatech.edu/~rhim)
I prefer to receive attachments in an open, non-proprietary format.
-------------------------------------------------------------------------

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26 Apr 2006, at 16:35, Himanshu Raj wrote:
> I am trying to understand the rationale behind this change. In 
> previous case,
> there would be no page faults due to page table updates and only one 
> hypercall.
> In the second case, there would be atleast 2 page faults due to PT 
> management
> activity, but no hypercalls. Besides, mapping and remapping with 
> different permissions
> imply removing this entry from TLB (which is hopefully being done with 
> invlpg).
> Benefit of latter approach only seems to be the removal of xen 
> specific linux
> code. Why was the approach changed? Could someone please shed some 
> light on
> this?
It''s useful for batched updates (e.g., fork()) where the 2 faults are 
amortised across up to 1024 pte changes.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Although, same results can be obtained by doing the v1.2 way, i.e. making one
hypercall requesting these 1024 changes, no?

-Himanshu

On Wed, Apr 26, 2006 at 05:18:28PM +0100, Keir Fraser
wrote:
> 
> On 26 Apr 2006, at 16:35, Himanshu Raj wrote:
> 
> >I am trying to understand the rationale behind this change. In 
> >previous case,
> >there would be no page faults due to page table updates and only one 
> >hypercall.
> >In the second case, there would be atleast 2 page faults due to PT 
> >management
> >activity, but no hypercalls. Besides, mapping and remapping with 
> >different permissions
> >imply removing this entry from TLB (which is hopefully being done with 
> >invlpg).
> >Benefit of latter approach only seems to be the removal of xen 
> >specific linux
> >code. Why was the approach changed? Could someone please shed some 
> >light on
> >this?
> 
> It''s useful for batched updates (e.g., fork()) where the 2 faults
are
> amortised across up to 1024 pte changes.
> 
>  -- Keir
-- 
-------------------------------------------------------------------------
Himanshu Raj
PhD Student, GaTech (www.cc.gatech.edu/~rhim)
I prefer to receive attachments in an open, non-proprietary format.
-------------------------------------------------------------------------

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26 Apr 2006, at 18:40, Himanshu Raj wrote:
> Although, same results can be obtained by doing the v1.2 way, i.e. 
> making one
> hypercall requesting these 1024 changes, no?
>
> -Himanshu
You have to marshal/unmarshal the batched arguments, and insert flushes 
all over the place to ensure the requests get flushed before Linux next 
tries to read any of the ptes. So the new method is fewer changes to 
the guest and no slower for reasonable-sized batches.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks Keir for your prompt reply. 

In another example, mmapped access to a file, the new approach will incur these
two extra faults for every page fault required to bring a file page to page
cache,
right?

Could you tell me another example other than fork when one can use batched PTE
modifications.

Thanks and best regards,
Himanshu

On Wed, Apr 26, 2006 at 06:44:24PM +0100, Keir Fraser
wrote:
> 
> On 26 Apr 2006, at 18:40, Himanshu Raj wrote:
> 
> >Although, same results can be obtained by doing the v1.2 way, i.e. 
> >making one
> >hypercall requesting these 1024 changes, no?
> >
> >-Himanshu
> 
> You have to marshal/unmarshal the batched arguments, and insert flushes 
> all over the place to ensure the requests get flushed before Linux next 
> tries to read any of the ptes. So the new method is fewer changes to 
> the guest and no slower for reasonable-sized batches.
> 
>  -- Keir
-- 
-------------------------------------------------------------------------
Himanshu Raj
PhD Student, GaTech (www.cc.gatech.edu/~rhim)
I prefer to receive attachments in an open, non-proprietary format.
-------------------------------------------------------------------------

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26 Apr 2006, at 18:56, Himanshu Raj wrote:
> In another example, mmapped access to a file, the new approach will 
> incur these
> two extra faults for every page fault required to bring a file page to 
> page cache,
> right?
All the common non-batched paths (i.e., demand faults) use 
update_va_mapping(), so do a single fast hypercall.
> Could you tell me another example other than fork when one can use 
> batched PTE
> modifications.
fork() is the only one for us these days. All others use 
update_va_mapping(), act on pagetables that aren''t pinned (so the guest
can directly update them without faulting) or are infrequent enough we 
do not care.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> >Could you tell me another example other than fork when one can use 
> >batched PTE
> >modifications.
> 
> fork() is the only one for us these days. All others use 
> update_va_mapping(), act on pagetables that aren''t pinned (so the
guest
> can directly update them without faulting) or are infrequent enough we 
> do not care.
Not sure I understand the last part (aren''t pinned ...) - my
assumptions about
page tables are (these apply to both direct mapped and shadow page tables):

1. Always pinned (backed) - so a access to them cannot cause a page unavailable
fault.
2. Always read only to guest - so a read access to them is fine, but a write
access
   will cause a protection fault.

Both of these faults are reflected as a PG fault.

Are you refering to the case when Xen has "detached" the page table
page and
has made it RW for guest?

Thanks for your answers and patience :-).

-Himanshu

-- 
-------------------------------------------------------------------------
Himanshu Raj
PhD Student, GaTech (www.cc.gatech.edu/~rhim)
I prefer to receive attachments in an open, non-proprietary format.
-------------------------------------------------------------------------

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Suse76:/imagesxen # xm create /etc/xen/suse10 -c Using config file
"/etc/xen/suse10".
Started domain suse10
Linux version 2.6.13-15.8-xen (geeko@buildhost) (gcc version 4.0.2 20050901
(prerelease) (SUSE Linux)) #1 SMP Tue Feb 7 11:07:24 UTC 2006 BIOS-provided
physical RAM map:
 Xen: 0000000000000000 - 0000000010000000 (usable) 0MB HIGHMEM available.
264MB LOWMEM available.
ACPI in unprivileged domain disabled
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line:  root=/dev/sda1 ro 5 Initializing CPU#0 PID hash table
entries: 2048 (order: 11, 32768 bytes) Xen reported: 1007.282 MHz processor.
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) Inode-cache
hash table entries: 32768 (order: 5, 131072 bytes) Software IO TLB disabled
vmalloc area: d1000000-fb7fe000, maxmem 34000000
Memory: 249472k/270336k available (2362k kernel code, 12380k reserved, 829k
data, 180k init, 0k highmem) Checking if this processor honours the WP bit
even in supervisor mode... Ok.
Security Framework v1.0.0 initialized
SELinux:  Disabled at boot.
Mount-cache hash table entries: 512
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 256K (64 bytes/line)
Enabling fast FPU save and restore... done.
Checking ''hlt'' instruction... disabled
checking if image is initramfs... it is
Freeing initrd memory: 4270k freed
Brought up 1 CPUs
NET: Registered protocol family 16
Brought up 1 CPUs
ACPI: Subsystem revision 20050408
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
xen_mem: Initialising balloon driver.
PCI: Using ACPI for IRQ routing
PCI: If a device doesn''t work, try "pci=routeirq".  If it
helps, post a
report
PCI: System does not support PCI
PCI: System does not support PCI
TC classifier action (bugs to netdev@vger.kernel.org cc hadi@cyberus.ca)
Grant table initialized
audit: initializing netlink socket (disabled)
audit(1146070099.926:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) Initializing
Cryptographic API
vesafb: abort, cannot ioremap video memory 0x0 @ 0x0
vesafb: probe of vesafb.0 failed with error -5
PNP: No PS/2 controller found. Probing ports directly.
i8042.c: No controller found.
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 64000K size 1024 blocksize
loop: loaded (max 8 devices)
Xen virtual console successfully installed as tty1 Event-channel device
installed.
Successfully initialized TPM backend driver.
Registering block device major 8
netfront: Initialising virtual ethernet driver.
xen_tpm_fr: Initialising the vTPM driver.
mice: PS/2 mouse device common for all mice
md: md driver 0.90.2 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 3.38
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes) TCP
established hash table entries: 16384 (order: 5, 131072 bytes) TCP bind hash
table entries: 16384 (order: 5, 131072 bytes)
TCP: Hash tables configured (established 16384 bind 16384) TCP reno
registered
NET: Registered protocol family 1
NET: Registered protocol family 8
NET: Registered protocol family 20
Freeing unused kernel memory: 180k freed Starting udev Creating devices
Loading ide-disk Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 50MHz system bus speed for PIO modes; override with idebus=xx
Loading sd_mod SCSI subsystem initialized
register_blkdev: cannot get major 8 for sd Loading via82cxxx Loading aic7xxx
Loading processor Loading thermal
FATAL: Error inserting thermal
(/lib/modules/2.6.13-15.8-xen/kernel/drivers/acpi/thermal.ko): No such
device Loading fan
FATAL: Error inserting fan
(/lib/modules/2.6.13-15.8-xen/kernel/drivers/acpi/fan.ko): No such device
Loading reiserfs Waiting for device /dev/sda1 to appear:  ok
rootfs:  major=8 minor=1 devn=2049
Mounting root /dev/sda1
mount: No such device
Kernel panic - not syncing: Attempted to kill init!
_________________________

The config file:

suse76:/imagesxen # more /etc/xen/suse10

kernel = "/boot/vmlinuz-2.6.13-15.8-xen"
ramdisk = "/boot/initrd-2.6.13-15.8-xen"
memory = 256
name = "suse10"
nics = 1
vif = [ ''bridge=xenbr0'' ]
disk = [ ''file:/imagesxen/suse10.img,sda1,w'' ] root =
"/dev/sda1 ro"
extra = "5"
suse76:/imagesxen #

Thanks for your help,

Jimmy


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel