Hi,
I''ve been intensely debugging a dom0 crash right at boot in
pagetable_init() with 256G. We are running 64bit hyp with 32bit dom0.
This on xen 3.1.3, which seems to have fixes from changeset 16548.
It boils down to this:
crash occurs in dom0: mfn_to_pfn() during machine_to_phys_mapping[mfn]
lookup where mfn == 0x3f2ec09 is too big. The table starts at: 0xf5800000
which I see comes from MACH2PHYS_VIRT_START.
Trying to figure the hyp out, I notice phys addr is set as:
d->arch.physaddr_bitsize fls((1UL << 32) -
HYPERVISOR_COMPAT_VIRT_START(d)) - 1
+ (PAGE_SIZE - 2);
which is set to 0x1019, totally baffling me. I''d expect it to be 32,
36, or 64????????
moving along, in alloc_domheap_pages() :
bits = domain_clamp_alloc_bitsize(d, bits ? : (BITS_PER_LONG+PAGE_SHIFT));
domain_clamp_alloc_bitsize(struct domain *d, unsigned int bits)
{
if ( (d == NULL) || !is_pv_32on64_domain(d) )
return bits;
return min(d->arch.physaddr_bitsize, bits);
---> here : physaddr: 0x1019, bits:76 ?????
---> i''m thinking bits should be 36+12 == 48 ??
}
finally, call to alloc_heap_pages(...) returns frames too high, pg
ptr is returned at : ffff82849df30000 (zone_hi == 0x27)
On a side note, is my understanding correct that while a guest pfn must
always be < 36 bits for a PAE 32 guest, the mfn doesn''t have to
be?
thanks
mukesh
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
> I''ve been intensely debugging a dom0 crash right at boot in > pagetable_init() with 256G. We are running 64bit hyp with 32bit > dom0. > This on xen 3.1.3, which seems to have fixes from changeset 16548.256GB is more than a 32b dom0 can handle. Presumably it works if you set dom0_mem= to something smaller? We should probably make the clipping of dom0 memory to something like 64GB automatic. Ian> It boils down to this: > > crash occurs in dom0: mfn_to_pfn() during > machine_to_phys_mapping[mfn] > lookup where mfn == 0x3f2ec09 is too big. The table starts at: > 0xf5800000 > which I see comes from MACH2PHYS_VIRT_START. > > Trying to figure the hyp out, I notice phys addr is set as: > > d->arch.physaddr_bitsize > fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1 > + (PAGE_SIZE - 2); > which is set to 0x1019, totally baffling me. I''d expect it to be32,> 36, or 64???????? > > moving along, in alloc_domheap_pages() : > bits = domain_clamp_alloc_bitsize(d, bits ? : > (BITS_PER_LONG+PAGE_SHIFT)); > > domain_clamp_alloc_bitsize(struct domain *d, unsigned int bits) > { > if ( (d == NULL) || !is_pv_32on64_domain(d) ) > return bits; > return min(d->arch.physaddr_bitsize, bits); > ---> here : physaddr: 0x1019, bits:76 ????? > ---> i''m thinking bits should be 36+12 == 48 ?? > } > > finally, call to alloc_heap_pages(...) returns frames too high, pg > ptr is returned at : ffff82849df30000 (zone_hi == 0x27) > > > On a side note, is my understanding correct that while a guest pfn > must > always be < 36 bits for a PAE 32 guest, the mfn doesn''t have tobe?> > > thanks > mukesh > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/6/08 05:00, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:>> I''ve been intensely debugging a dom0 crash right at boot in >> pagetable_init() with 256G. We are running 64bit hyp with 32bit >> dom0. >> This on xen 3.1.3, which seems to have fixes from changeset 16548. > > 256GB is more than a 32b dom0 can handle. Presumably it works if you set > dom0_mem= to something smaller? > > We should probably make the clipping of dom0 memory to something like > 64GB automatic.Xen ought to have crashed during dom0 build in that case, being unable to allocate enough memory of the restricted address width supported by the 32b dom0 guest. I agree that dom0_mem should be automatically intelligently clamped. Currently nr_pages is clamped to UINT_MAX (!) for a 32b dom0, which is still rather a big number. I''ll have to sketch out exactly what that physaddr_bitsize computation is doing and see if I can make sense of it... -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/6/08 03:07, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:> d->arch.physaddr_bitsize > fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1 > + (PAGE_SIZE - 2); > which is set to 0x1019, totally baffling me. I''d expect it to be 32, > 36, or 64????????PAGE_SIZE should be PAGE_SHIFT. If you fix that then you may discover that dom0 can no longer be built (since there is not enough memory below 64GB to build a ~256GB dom0, which is the default that the dom0 builder will go for) so make sure you have a suitably restrictive dom0_mem= option on your Xen command line. As Ian says we should fix that more nicely. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Trying to figure the hyp out, I notice phys addr is set as: > > d->arch.physaddr_bitsize > fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1 > + (PAGE_SIZE - 2); > which is set to 0x1019, totally baffling me. I''d expect it to be 32, > 36, or 64????????Of course it should be PAGE_SHIFT in here! c/s 14097 screwed this up - sorry, my fault. Signed-off-by: Jan Beulich <jbeulich@novell.com> Index: 2008-05-08/xen/arch/x86/domain.c ==================================================================--- 2008-05-08.orig/xen/arch/x86/domain.c 2008-05-09 12:19:01.000000000 +0200 +++ 2008-05-08/xen/arch/x86/domain.c 2008-06-05 09:31:30.000000000 +0200 @@ -353,7 +353,7 @@ int switch_compat(struct domain *d) d->arch.physaddr_bitsize fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1 - + (PAGE_SIZE - 2); + + (PAGE_SHIFT - 2); return 0; Index: 2008-05-08/xen/arch/x86/domain_build.c ==================================================================--- 2008-05-08.orig/xen/arch/x86/domain_build.c 2008-06-04 08:26:22.000000000 +0200 +++ 2008-05-08/xen/arch/x86/domain_build.c 2008-06-05 09:31:36.000000000 +0200 @@ -366,7 +366,7 @@ int __init construct_dom0( if ( is_pv_32on64_domain(d) ) d->arch.physaddr_bitsize fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1 - + (PAGE_SIZE - 2); + + (PAGE_SHIFT - 2); #endif /* _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/6/08 08:24, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:>> 256GB is more than a 32b dom0 can handle. Presumably it works if you set >> dom0_mem= to something smaller? >> >> We should probably make the clipping of dom0 memory to something like >> 64GB automatic. > > Xen ought to have crashed during dom0 build in that case, being unable to > allocate enough memory of the restricted address width supported by the 32b > dom0 guest. I agree that dom0_mem should be automatically intelligently > clamped. Currently nr_pages is clamped to UINT_MAX (!) for a 32b dom0, which > is still rather a big number.Actually compat guests are good for up to 128GB, so you should be fine with no dom0_mem= parameter. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> Actually compat guests are good for up to 128GB, so you should be fine with > no dom0_mem= parameter. >Real PAE processors are limited to 2^36 = 64GB, and the guest kernel may mask ptes at that point. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/6/08 13:54, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:> Keir Fraser wrote: >> Actually compat guests are good for up to 128GB, so you should be fine with >> no dom0_mem= parameter. >> > > Real PAE processors are limited to 2^36 = 64GB, and the guest kernel may > mask ptes at that point.A paravirtualised kernel needs to have its PAGE_MASK widened. This is a bug we''ve bumped into once or twice. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Keir Fraser <keir.fraser@eu.citrix.com> 05.06.08 14:58 >>> >On 5/6/08 13:54, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote: > >> Keir Fraser wrote: >>> Actually compat guests are good for up to 128GB, so you should be fine with >>> no dom0_mem= parameter. >>> >> >> Real PAE processors are limited to 2^36 = 64GB, and the guest kernel may >> mask ptes at that point. > >A paravirtualised kernel needs to have its PAGE_MASK widened. This is a bug >we''ve bumped into once or twice.Even a native kernel ought to honor the physical address width the processor supports rather than arbitrarily masking off bits just because original PAE only supported 36 bits. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/6/08 14:15, "Jan Beulich" <jbeulich@novell.com> wrote:>> A paravirtualised kernel needs to have its PAGE_MASK widened. This is a bug >> we''ve bumped into once or twice. > > Even a native kernel ought to honor the physical address width the > processor supports rather than arbitrarily masking off bits just because > original PAE only supported 36 bits.Yes, both Intel and AMD have wider PAE addressing these days. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich wrote:>> A paravirtualised kernel needs to have its PAGE_MASK widened. This is a bug >> we''ve bumped into once or twice. >> > > Even a native kernel ought to honor the physical address width the > processor supports rather than arbitrarily masking off bits just because > original PAE only supported 36 bits.Current 2.6 is too strict about this. I''ll queue up a patch to fix it. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt <Ian.Pratt <at> eu.citrix.com> writes:> > > I''ve been intensely debugging a dom0 crash right at boot in > > pagetable_init() with 256G. We are running 64bit hyp with 32bit > > dom0. > > This on xen 3.1.3, which seems to have fixes from changeset 16548. > > 256GB is more than a 32b dom0 can handle. Presumably it works if you set > dom0_mem= to something smaller? > > We should probably make the clipping of dom0 memory to something like > 64GB automatic. > > Ian > >.......... My apologies, I forgot to mention dom0_mem is set to : dom0_mem=569M (XEN) Command line: console=com2,ttyS0 com2=115200,8n1 earlykdb dom0_mem=569M mem=256G Things are fine with mem=128G. I''ve not tried 192G or other number. I just started debugging..... I''ll try with :> d->arch.physaddr_bitsize > > fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1 > > + (PAGE_SIZE - 2); > > which is set to 0x1019, totally baffling me. I''d expect it to be 32, > > 36, or 64????????>PAGE_SIZE should be PAGE_SHIFT. If you fix that then you may discover ...thanks mukesh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mukesh Rathor
2008-Jun-06 02:43 UTC
Re: [PATCH] Re: [Xen-devel] dom0 boot failure with 256G
Jan Beulich wrote:>> Trying to figure the hyp out, I notice phys addr is set as: >> >> d->arch.physaddr_bitsize >> fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1 >> + (PAGE_SIZE - 2); >> which is set to 0x1019, totally baffling me. I''d expect it to be 32, >> 36, or 64???????? > > Of course it should be PAGE_SHIFT in here! c/s 14097 screwed this up > - sorry, my fault. >Yup, that fixed it. Can you pl explain in couple sentences how we are getting at the physaddr_bitsize here? I still don''t understand why 37 is an OK size for a 32-PAE guest (i thought 36 was max). Thanks, Mukesh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> > > On 5/6/08 08:24, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote: > >>> 256GB is more than a 32b dom0 can handle. Presumably it works if you set >>> dom0_mem= to something smaller? >>> >>> We should probably make the clipping of dom0 memory to something like >>> 64GB automatic. >> Xen ought to have crashed during dom0 build in that case, being unable to >> allocate enough memory of the restricted address width supported by the 32b >> dom0 guest. I agree that dom0_mem should be automatically intelligently >> clamped. Currently nr_pages is clamped to UINT_MAX (!) for a 32b dom0, which >> is still rather a big number. > > Actually compat guests are good for up to 128GB, so you should be fine with > no dom0_mem= parameter. > > -- Keir >No, it crashed without dom0_mem paramater after I fixed the physaddr_bitsize in construct_dom0(). It looked like it was allocating almost 256G to dom0... things are fine now with dom0_mem= specified. thanks mukesh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Mukesh Rathor <mukesh.rathor@oracle.com> 06.06.08 04:43 >>> > >Jan Beulich wrote: >>> Trying to figure the hyp out, I notice phys addr is set as: >>> >>> d->arch.physaddr_bitsize >>> fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1 >>> + (PAGE_SIZE - 2); >>> which is set to 0x1019, totally baffling me. I''d expect it to be 32, >>> 36, or 64???????? >> >> Of course it should be PAGE_SHIFT in here! c/s 14097 screwed this up >> - sorry, my fault. >> > >Yup, that fixed it. Can you pl explain in couple sentences how we are getting >at the physaddr_bitsize here?The whole hypervisor hole is used solely for the (compatibility) m2p map when running on a 64-bit hypervisor. Hence the size of that hole, rounded down to a power of two, determines the address restriction a domain must be assigned. A kernel could theoretically specify being able to tolerate a larger hole (though XEN_ELFNOTE_HV_START_LOW), so to increase the physical address width it can get memory from.>I still don''t understand why 37 is an OK size for a 32-PAE guest (i thought 36 >was max).36 was the original max, but even on native hardware you can go beyond that nowadays. On 64-bit Xen, this was a natural extension even independent of confirming hardware support of wider than 36-bit physical addresses for 32-bit code. Of course, we all realize that giving dom0 128G won''t do any good, afair even native Linux can''t reasonably deal with 64G (due to the huge memmap needed in lowmem). But allowing an as wide as possible range to get memory from for the domain is certainly very desirable (I am actually considering ideas on how to overcome even that limit, as in practical uses this is really undesirable to have, since there''s no way to balloon out specific memory [i.e. such below the 37-bit address boundary] from other domains). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mukesh Rathor
2008-Jun-07 00:33 UTC
Re: [PATCH] Re: [Xen-devel] dom0 boot failure with 256G
Jan Beulich wrote:>>>> Mukesh Rathor <mukesh.rathor@oracle.com> 06.06.08 04:43 >>> >> Jan Beulich wrote: >>>> Trying to figure the hyp out, I notice phys addr is set as: >>>> >>>> d->arch.physaddr_bitsize >>>> fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1 >>>> + (PAGE_SIZE - 2); >>>> which is set to 0x1019, totally baffling me. I''d expect it to be 32, >>>> 36, or 64???????? >>> Of course it should be PAGE_SHIFT in here! c/s 14097 screwed this up >>> - sorry, my fault. >>> >> Yup, that fixed it. Can you pl explain in couple sentences how we are getting >> at the physaddr_bitsize here? > > The whole hypervisor hole is used solely for the (compatibility) m2p map > when running on a 64-bit hypervisor. Hence the size of that hole, rounded > down to a power of two, determines the address restriction a domain > must be assigned. A kernel could theoretically specify being able to > tolerate a larger hole (though XEN_ELFNOTE_HV_START_LOW), so to > increase the physical address width it can get memory from. > >> I still don''t understand why 37 is an OK size for a 32-PAE guest (i thought 36 >> was max). > > 36 was the original max, but even on native hardware you can go beyond > that nowadays. On 64-bit Xen, this was a natural extension even > independent of confirming hardware support of wider than 36-bit physical > addresses for 32-bit code. > > Of course, we all realize that giving dom0 128G won''t do any good, afair > even native Linux can''t reasonably deal with 64G (due to the huge > memmap needed in lowmem). But allowing an as wide as possible range > to get memory from for the domain is certainly very desirable (I am > actually considering ideas on how to overcome even that limit, as in > practical uses this is really undesirable to have, since there''s no way > to balloon out specific memory [i.e. such below the 37-bit address > boundary] from other domains). > > JanGot it. Thanks a lot, appreciate your time. Mukesh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel