Jan Beulich
2006-Jan-20 15:50 UTC
[Xen-devel] x86-64 problem with invalid page fault in linux 2.6.16-rc1
I''m just trying to see if anyone has any clue about this, which only appears to happen with MP guests: Since the check for the modules area is gone in 2.6.16''s vmalloc_fault() (and we appropriately merged this change to the Xen files), we are now seeing page faults in the module area, where a subsequent software page table walk shows all page table entries present, and a get_user from inside the hypervisor''s or the guest''s page fault handler also succeeds. The module in the questionable space was loaded significantly before the page fault occurs, and we never saw a fault after the system fully booted. Faults of this kind may have existed before, but would have been hidden by the vmalloc_fault() handling assuming that another processor would have put in place the pgd entry meanwhile. Since I have no clue how such a fault could be raised in the first place (given that the pgd entry for the modules area is shared with main kernel code, all lower level entries are shared across kernel and all processes, and the fault happens on an access to the modules area from kernel code [guaranteeing that the system isn''t unintentionally running with the user mode page tables]). I can also mostly rule out any sort of hardware problem since the issue is visible on both Intel and AMD processors. Thanks for any thoughts, hints, or pointers, Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Jan-20 17:02 UTC
Re: [Xen-devel] x86-64 problem with invalid page fault in linux 2.6.16-rc1
On SMP systems we need the guest to handle spurious page-not-present faults at any time and at any virtual address. This is a side effect of the writable pagetable implementation. If the vmalloc_fault path no longer covers all of the kernel virtual address space then a spurious-fault detection needs to be added before oops''ing the kernel. Alternatively it would be quite reasonable to extend the writable pagetable implementation to exclude areas of the virtual address space from being subject to batched wrpt updates. Updates to such areas would be emulated synchronously. This would be a good thing anyway -- updates to kernel va space rarely occur in batches and so emulation would improve performance. -- Keir On 20 Jan 2006, at 15:50, Jan Beulich wrote:> I''m just trying to see if anyone has any clue about this, which only > appears to happen with MP guests: > > Since the check for the modules area is gone in 2.6.16''s > vmalloc_fault() (and we appropriately merged this change to > the Xen files), we are now seeing page faults in the module area, > where a subsequent software page table walk shows all > page table entries present, and a get_user from inside the > hypervisor''s or the guest''s page fault handler also succeeds. > The module in the questionable space was loaded significantly before > the page fault occurs, and we never saw a fault > after the system fully booted. Faults of this kind may have existed > before, but would have been hidden by the > vmalloc_fault() handling assuming that another processor would have > put in place the pgd entry meanwhile. > > Since I have no clue how such a fault could be raised in the first > place (given that the pgd entry for the modules area > is shared with main kernel code, all lower level entries are shared > across kernel and all processes, and the fault > happens on an access to the modules area from kernel code > [guaranteeing that the system isn''t unintentionally running > with the user mode page tables]). > > I can also mostly rule out any sort of hardware problem since the > issue is visible on both Intel and AMD processors. > > Thanks for any thoughts, hints, or pointers,_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2006-Jan-20 17:06 UTC
Re: [Xen-devel] x86-64 problem with invalid page fault in linux 2.6.16-rc1
>In SMP systems we need the guest to handle spurious page-not-present >faults at any time and at any virtual address. This is a side effect of >the writable pagetable implementation.So far it was my understanding that writable page tables are not the normal way of operating for the guest (I thought this was only happening in shadow mode). If that assumption is wrong, why get all the page table pages converted to readonly in the kernel?>If the vmalloc_fault path no longer covers all of the kernel virtual >address space then a spurious-fault detection needs to be added before >oops''ing the kernel.That would then include the kernel image itself, too. But we never see any page faults there. Further, what updating are you talking about - the area for a loaded module gets mapped once and then never touched again (and as said in the original mail, the faults are occuring long after the affected module got loaded)? Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephen C. Tweedie
2006-Feb-15 20:07 UTC
[PATCH] Re: [Xen-devel] x86-64 problem with invalid page fault in linux 2.6.16-rc1
Hi, On Fri, 2006-01-20 at 17:02 +0000, Keir Fraser wrote:> On SMP systems we need the guest to handle spurious page-not-present > faults at any time and at any virtual address. This is a side effect of > the writable pagetable implementation. > > If the vmalloc_fault path no longer covers all of the kernel virtual > address space then a spurious-fault detection needs to be added before > oops''ing the kernel.OK, I''ve been seeing precisely the same symptoms as Jan on current xen-unstable HV+dom0. And with the Fedora kernel build being so modular, this makes it completely impossible to boot on a 64-bit SMP box. But it looks like there''s an easy fix. The x86_64 vmalloc_fault() path is already doing a soft pagetable walk to detect if it''s a true or a spurious fault. It only does that for the vmalloc area; if we can get spurious faults in the module area too, then the same test probably needs to be applied there too. And indeed, the attached patch fixes the problem entirely for me (so far, at least.) --Stephen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2006-Feb-16 09:01 UTC
[PATCH] Re: [Xen-devel] x86-64 problem with invalid page fault in linux 2.6.16-rc1
I had already pointed out to Christian that there are a couple of pieces missing from the 2.6.16-rc? merges (along with the actual patches that we use for SLES10). The patch you provide here isn''t sufficient, however - the area that spurious page faults need to be dealt with is the entire kernel range (otherwise e.g. memory hotplug code, which hopefully is or will be used for increasing a VMs memory, may trigger the same problem). Jan>>> "Stephen C. Tweedie" <sct@redhat.com> 15.02.06 21:07:37 >>>Hi, On Fri, 2006-01-20 at 17:02 +0000, Keir Fraser wrote:> On SMP systems we need the guest to handle spurious page-not-present > faults at any time and at any virtual address. This is a side effect of > the writable pagetable implementation. > > If the vmalloc_fault path no longer covers all of the kernel virtual > address space then a spurious-fault detection needs to be added before > oops''ing the kernel.OK, I''ve been seeing precisely the same symptoms as Jan on current xen-unstable HV+dom0. And with the Fedora kernel build being so modular, this makes it completely impossible to boot on a 64-bit SMP box. But it looks like there''s an easy fix. The x86_64 vmalloc_fault() path is already doing a soft pagetable walk to detect if it''s a true or a spurious fault. It only does that for the vmalloc area; if we can get spurious faults in the module area too, then the same test probably needs to be applied there too. And indeed, the attached patch fixes the problem entirely for me (so far, at least.) --Stephen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephen C. Tweedie
2006-Feb-16 19:57 UTC
Re: [PATCH] Re: [Xen-devel] x86-64 problem with invalid page fault in linux 2.6.16-rc1
Hi, On Thu, 2006-02-16 at 10:01 +0100, Jan Beulich wrote:> I had already pointed out to Christian that there are a couple of pieces missing from the 2.6.16-rc? merges (along with > the actual patches that we use for SLES10). The patch you provide here isn''t sufficient, however - the area that > spurious page faults need to be dealt with is the entire kernel range (otherwise e.g. memory hotplug code, which > hopefully is or will be used for increasing a VMs memory, may trigger the same problem). JanOK, thanks: that makes sense. I notice "Fix upgrade omissions." in today''s xen-unstable.hg logs; I assume that that contains the other bits you referred to? Cheers, Stephen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel