Xen devel - Jan 2006 - x86-64 problem with invalid page fault in linux 2.6.16-rc1

I''m just trying to see if anyone has any clue about this, which only
appears to happen with MP guests:

Since the check for the modules area is gone in 2.6.16''s
vmalloc_fault() (and we appropriately merged this change to
the Xen files), we are now seeing page faults in the module area, where a
subsequent software page table walk shows all
page table entries present, and a get_user from inside the hypervisor''s
or the guest''s page fault handler also succeeds.
The module in the questionable space was loaded significantly before the page
fault occurs, and we never saw a fault
after the system fully booted. Faults of this kind may have existed before, but
would have been hidden by the
vmalloc_fault() handling assuming that another processor would have put in place
the pgd entry meanwhile.

Since I have no clue how such a fault could be raised in the first place (given
that the pgd entry for the modules area
is shared with main kernel code, all lower level entries are shared across
kernel and all processes, and the fault
happens on an access to the modules area from kernel code [guaranteeing that the
system isn''t unintentionally running
with the user mode page tables]).

I can also mostly rule out any sort of hardware problem since the issue is
visible on both Intel and AMD processors.

Thanks for any thoughts, hints, or pointers,
Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On SMP systems we need the guest to handle spurious page-not-present 
faults at any time and at any virtual address. This is a side effect of 
the writable pagetable implementation.

If the vmalloc_fault path no longer covers all of the kernel virtual 
address space then a spurious-fault detection needs to be added before 
oops''ing the kernel.

Alternatively it would be quite reasonable to extend the writable 
pagetable implementation to exclude areas of the virtual address space 
from being subject to batched wrpt updates. Updates to such areas would 
be emulated synchronously. This would be a good thing anyway -- updates 
to kernel va space rarely occur in batches and so emulation would 
improve performance.

  -- Keir

On 20 Jan 2006, at 15:50, Jan Beulich wrote:
> I''m just trying to see if anyone has any clue about this, which
only
> appears to happen with MP guests:
>
> Since the check for the modules area is gone in 2.6.16''s 
> vmalloc_fault() (and we appropriately merged this change to
> the Xen files), we are now seeing page faults in the module area, 
> where a subsequent software page table walk shows all
> page table entries present, and a get_user from inside the 
> hypervisor''s or the guest''s page fault handler also
succeeds.
> The module in the questionable space was loaded significantly before 
> the page fault occurs, and we never saw a fault
> after the system fully booted. Faults of this kind may have existed 
> before, but would have been hidden by the
> vmalloc_fault() handling assuming that another processor would have 
> put in place the pgd entry meanwhile.
>
> Since I have no clue how such a fault could be raised in the first 
> place (given that the pgd entry for the modules area
> is shared with main kernel code, all lower level entries are shared 
> across kernel and all processes, and the fault
> happens on an access to the modules area from kernel code 
> [guaranteeing that the system isn''t unintentionally running
> with the user mode page tables]).
>
> I can also mostly rule out any sort of hardware problem since the 
> issue is visible on both Intel and AMD processors.
>
> Thanks for any thoughts, hints, or pointers,

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>In SMP systems we need the guest to handle spurious page-not-present 
>faults at any time and at any virtual address. This is a side effect of 
>the writable pagetable implementation.
So far it was my understanding that writable page tables are not the normal way
of operating for the guest (I thought
this was only happening in shadow mode). If that assumption is wrong, why get
all the page table pages converted to
readonly in the kernel?
>If the vmalloc_fault path no longer covers all of the kernel virtual 
>address space then a spurious-fault detection needs to be added before 
>oops''ing the kernel.
That would then include the kernel image itself, too. But we never see any page
faults there.

Further, what updating are you talking about - the area for a loaded module gets
mapped once and then never touched
again (and as said in the original mail, the faults are occuring long after the
affected module got loaded)?

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

On Fri, 2006-01-20 at 17:02 +0000, Keir Fraser wrote:
> On SMP systems we need the guest to handle spurious page-not-present 
> faults at any time and at any virtual address. This is a side effect of 
> the writable pagetable implementation.
> 
> If the vmalloc_fault path no longer covers all of the kernel virtual 
> address space then a spurious-fault detection needs to be added before 
> oops''ing the kernel.
OK, I''ve been seeing precisely the same symptoms as Jan on current
xen-unstable HV+dom0.  And with the Fedora kernel build being so
modular, this makes it completely impossible to boot on a 64-bit SMP
box.  But it looks like there''s an easy fix.

The x86_64 vmalloc_fault() path is already doing a soft pagetable walk
to detect if it''s a true or a spurious fault.  It only does that for
the
vmalloc area; if we can get spurious faults in the module area too, then
the same test probably needs to be applied there too.

And indeed, the attached patch fixes the problem entirely for me (so
far, at least.)

--Stephen




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I had already pointed out to Christian that there are a couple of pieces missing
from the 2.6.16-rc? merges (along with
the actual patches that we use for SLES10). The patch you provide here
isn''t sufficient, however - the area that
spurious page faults need to be dealt with is the entire kernel range (otherwise
e.g. memory hotplug code, which
hopefully is or will be used for increasing a VMs memory, may trigger the same
problem). Jan
>>> "Stephen C. Tweedie" <sct@redhat.com> 15.02.06
21:07:37 >>>
Hi,

On Fri, 2006-01-20 at 17:02 +0000, Keir Fraser wrote:
> On SMP systems we need the guest to handle spurious page-not-present 
> faults at any time and at any virtual address. This is a side effect of 
> the writable pagetable implementation.
> 
> If the vmalloc_fault path no longer covers all of the kernel virtual 
> address space then a spurious-fault detection needs to be added before 
> oops''ing the kernel.
OK, I''ve been seeing precisely the same symptoms as Jan on current
xen-unstable HV+dom0.  And with the Fedora kernel build being so
modular, this makes it completely impossible to boot on a 64-bit SMP
box.  But it looks like there''s an easy fix.

The x86_64 vmalloc_fault() path is already doing a soft pagetable walk
to detect if it''s a true or a spurious fault.  It only does that for
the
vmalloc area; if we can get spurious faults in the module area too, then
the same test probably needs to be applied there too.

And indeed, the attached patch fixes the problem entirely for me (so
far, at least.)

--Stephen



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

On Thu, 2006-02-16 at 10:01 +0100, Jan Beulich wrote:
> I had already pointed out to Christian that there are a couple of pieces
missing from the 2.6.16-rc? merges (along with
> the actual patches that we use for SLES10). The patch you provide here
isn''t sufficient, however - the area that
> spurious page faults need to be dealt with is the entire kernel range
(otherwise e.g. memory hotplug code, which
> hopefully is or will be used for increasing a VMs memory, may trigger the
same problem). Jan
OK, thanks: that makes sense.

I notice "Fix upgrade omissions." in today''s xen-unstable.hg
logs; I
assume that that contains the other bits you referred to?

Cheers,
 Stephen



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel