Hi Keir and Jan -- With the plan to obsolete the x86 32-bit hypervisor at 4.3, when prototyping the "claim" hypercall/subop, can I assume that the CONFIG_X86 code in the hypervisor and, specifically any separation of the concepts of xen_heap from dom_heap, can be ignored? Or will the ARM version of the hypervisor be requiring a similar separation of xen_heap vs dom_heap? If the CONFIG_X86/separation code will no longer be used, shall I submit a patch to delete the code in page_alloc.c and memory.c? Or can you (Keir or Jan) just delete it? Thanks, Dan
>>> Dan Magenheimer <dan.magenheimer@oracle.com> 11/01/12 9:34 PM >>> >With the plan to obsolete the x86 32-bit hypervisor at 4.3,That''s not just a plan, but reality already.>when prototyping the "claim" hypercall/subop, can I assume >that the CONFIG_X86 code in the hypervisor and, specifically >any separation of the concepts of xen_heap from dom_heap, >can be ignored?No, you shouldn''t. Once adding support for memory amounts beyond 5Tb I expect the separation to become meaningful even for x86-64.>Or will the ARM version of the hypervisor be requiring >a similar separation of xen_heap vs dom_heap? > >If the CONFIG_X86/separation code will no longer be used, >shall I submit a patch to delete the code in page_alloc.c >and memory.c? Or can you (Keir or Jan) just delete it?Please don''t. Jan
At 13:34 -0700 on 01 Nov (1351776880), Dan Magenheimer wrote:> With the plan to obsolete the x86 32-bit hypervisor at 4.3, > when prototyping the "claim" hypercall/subop, can I assume > that the CONFIG_X86 code in the hypervisor and, specifically > any separation of the concepts of xen_heap from dom_heap, > can be ignored? > > Or will the ARM version of the hypervisor be requiring > a similar separation of xen_heap vs dom_heap?Yes, 32-bit ARM has this separation. Tim.
> From: Jan Beulich [mailto:jbeulich@suse.com] > Sent: Thursday, November 01, 2012 2:42 PM > To: Dan Magenheimer; keir@xen.org > Cc: xen-devel@lists.xen.org > Subject: Re: 32bit xen and "claim"> >If the CONFIG_X86/separation code will no longer be used, > >shall I submit a patch to delete the code in page_alloc.c > >and memory.c? Or can you (Keir or Jan) just delete it? > > Please don''t.OK. I''m glad I asked rather than assuming.> >when prototyping the "claim" hypercall/subop, can I assume > >that the CONFIG_X86 code in the hypervisor and, specifically > >any separation of the concepts of xen_heap from dom_heap, > >can be ignored? > > No, you shouldn''t. Once adding support for memory amounts beyond 5Tb > I expect the separation to become meaningful even for x86-64.On quick scan, I don''t see anything obvious in the archives that explains why 5Tb is the limit (rather than, say, 4Tb or 8Tb, or some other power of two). Could you provide a pointer to this info or, if you agree it is non-obvious and undocumented, say a few words of explanation? Also, just wondering, should exceeding 5Tb be on the 4.3 features list or is >5Tb physical memory still too far away? Thanks, Dan
> From: Tim Deegan [mailto:tim@xen.org] > Subject: Re: [Xen-devel] 32bit xen and "claim" > > At 13:34 -0700 on 01 Nov (1351776880), Dan Magenheimer wrote: > > With the plan to obsolete the x86 32-bit hypervisor at 4.3, > > when prototyping the "claim" hypercall/subop, can I assume > > that the CONFIG_X86 code in the hypervisor and, specifically > > any separation of the concepts of xen_heap from dom_heap, > > can be ignored? > > > > Or will the ARM version of the hypervisor be requiring > > a similar separation of xen_heap vs dom_heap? > > Yes, 32-bit ARM has this separation.Hmmm... looking at page_alloc.c... does ARM overload CONFIG_X86 to mean CONFIG-32-bitness then? Seems dangerous given that various random chunks of CONFIG_X86 may be disappearing.
>>> On 01.11.12 at 21:55, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: >> >when prototyping the "claim" hypercall/subop, can I assume >> >that the CONFIG_X86 code in the hypervisor and, specifically >> >any separation of the concepts of xen_heap from dom_heap, >> >can be ignored? >> >> No, you shouldn''t. Once adding support for memory amounts beyond 5Tb >> I expect the separation to become meaningful even for x86-64. > > On quick scan, I don''t see anything obvious in the archives > that explains why 5Tb is the limit (rather than, say, 4Tb > or 8Tb, or some other power of two). Could you provide a > pointer to this info or, if you agree it is non-obvious and > undocumented, say a few words of explanation?xen/include/asm-x86/config.h:DIRECTMAP_* (and the comment preceding all those #define-s).> Also, just wondering, should exceeding 5Tb be on the 4.3 > features list or is >5Tb physical memory still too far away?It _is_ on the feature list. So far I just had way too many other issues to deal with, preventing me to get started with that work. Jan
At 13:57 -0700 on 01 Nov (1351778261), Dan Magenheimer wrote:> > From: Tim Deegan [mailto:tim@xen.org] > > Subject: Re: [Xen-devel] 32bit xen and "claim" > > > > At 13:34 -0700 on 01 Nov (1351776880), Dan Magenheimer wrote: > > > With the plan to obsolete the x86 32-bit hypervisor at 4.3, > > > when prototyping the "claim" hypercall/subop, can I assume > > > that the CONFIG_X86 code in the hypervisor and, specifically > > > any separation of the concepts of xen_heap from dom_heap, > > > can be ignored? > > > > > > Or will the ARM version of the hypervisor be requiring > > > a similar separation of xen_heap vs dom_heap? > > > > Yes, 32-bit ARM has this separation. > > Hmmm... looking at page_alloc.c... does ARM overload CONFIG_X86 > to mean CONFIG-32-bitness then?No. CONFIG_X86 doesn''t mean 32-bit, even on x86; it means i386/amd64 as distinct from MIPS/ARM/PPC. Tim.
> From: Tim Deegan [mailto:tim@xen.org] > Sent: Friday, November 02, 2012 3:31 AM > To: Dan Magenheimer > Cc: Keir (Xen.org); Jan Beulich; xen-devel@lists.xen.org > Subject: Re: [Xen-devel] 32bit xen and "claim" > > At 13:57 -0700 on 01 Nov (1351778261), Dan Magenheimer wrote: > > > From: Tim Deegan [mailto:tim@xen.org] > > > Subject: Re: [Xen-devel] 32bit xen and "claim" > > > > > > At 13:34 -0700 on 01 Nov (1351776880), Dan Magenheimer wrote: > > > > With the plan to obsolete the x86 32-bit hypervisor at 4.3, > > > > when prototyping the "claim" hypercall/subop, can I assume > > > > that the CONFIG_X86 code in the hypervisor and, specifically > > > > any separation of the concepts of xen_heap from dom_heap, > > > > can be ignored? > > > > > > > > Or will the ARM version of the hypervisor be requiring > > > > a similar separation of xen_heap vs dom_heap? > > > > > > Yes, 32-bit ARM has this separation. > > > > Hmmm... looking at page_alloc.c... does ARM overload CONFIG_X86 > > to mean CONFIG-32-bitness then? > > No. CONFIG_X86 doesn''t mean 32-bit, even on x86; it means i386/amd64 as > distinct from MIPS/ARM/PPC.Oops, sorry, I was confusing the old code in page_alloc.c that said: #if !defined(__x86_64__) && !defined(__ia64__) when I was reading the new code that says: #if !defined(CONFIG_X86) (why not #ifndef?) Sorry for the ARM-y noise. :-( But as long as I''ve bothered you already... does the ARM port already (or will it soon) support 64-bit versions of ARM? Dan
At 11:00 -0800 on 05 Nov (1352113204), Dan Magenheimer wrote:> But as long as I''ve bothered you already... does the ARM port > already (or will it soon) support 64-bit versions of ARM?Real Soon Now, we hope. We''ve been a bit distracted by having actual hardware to test the 32-bit version on, but 64-bit support is definitely planned. Tim.
> From: Jan Beulich [mailto:JBeulich@suse.com] > Subject: RE: 32bit xen and "claim" > > >>> On 01.11.12 at 21:55, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: > >> >when prototyping the "claim" hypercall/subop, can I assume > >> >that the CONFIG_X86 code in the hypervisor and, specifically > >> >any separation of the concepts of xen_heap from dom_heap, > >> >can be ignored? > >> > >> No, you shouldn''t. Once adding support for memory amounts beyond 5Tb > >> I expect the separation to become meaningful even for x86-64. > > > > On quick scan, I don''t see anything obvious in the archives > > that explains why 5Tb is the limit (rather than, say, 4Tb > > or 8Tb, or some other power of two). Could you provide a > > pointer to this info or, if you agree it is non-obvious and > > undocumented, say a few words of explanation? > > xen/include/asm-x86/config.h:DIRECTMAP_* (and the comment > preceding all those #define-s).Sorry if this is a silly question, but the hex value 0xffff880000000000 looks like a Linux kernel base address so it made me wonder: Do I understand correctly that the 5TB limit only applies because of legacy PV guests?> > Also, just wondering, should exceeding 5Tb be on the 4.3 > > features list or is >5Tb physical memory still too far away? > > It _is_ on the feature list. So far I just had way too many other > issues to deal with, preventing me to get started with that work.Sorry, I should have looked that up first. :-} Does it make sense to have a runtime option that unsets the physical limit but disallows legacy PV guests? If this defaults to false for machines with RAM<=5TB but to true for machines with RAM>5TB, then the feature is "done" (AND we have put a stake in the ground to begin the slow obsolescence of PV functionality). Just an idea... Dan
On 05/11/2012 19:16, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:>> xen/include/asm-x86/config.h:DIRECTMAP_* (and the comment >> preceding all those #define-s). > > Sorry if this is a silly question, but the hex value > 0xffff880000000000 looks like a Linux kernel base address > so it made me wonder: > > Do I understand correctly that the 5TB limit only applies > because of legacy PV guests?Yes. For HVM guests we can direct-map all physical memory, all the time. We will only ever need ephemeral mappings on x86/64 when running PV guests. -- Keir
On 05/11/2012 19:00, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> #if !defined(CONFIG_X86) > > (why not #ifndef?)We have no coding convention to prefer one over the other. Personally I would say they are equally readable. -- Keir
>>> On 05.11.12 at 20:16, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: > Does it make sense to have a runtime option that unsets the > physical limit but disallows legacy PV guests? If this > defaults to false for machines with RAM<=5TB but to true > for machines with RAM>5TB, then the feature is "done" > (AND we have put a stake in the ground to begin the > slow obsolescence of PV functionality).That would be interesting: Mukesh''s PVH code wasn''t even posted yet, i.e. you''re proposing to render systems with more than 5Tb unbootable (for the lack of a - necessarily PV - Dom0 kernel runnable in that environment). But yes, the plan is to extend the 1:1 mapping beyond 5Tb for non-PV guests, and going through actual mapping code only when acting in the context of a (64-bit) PV guest (even 32-bit PV guests can have the full 1:1 mapping). Jan
> From: Jan Beulich [mailto:JBeulich@suse.com] > Sent: Tuesday, November 06, 2012 2:44 AM > To: Dan Magenheimer > Cc: xen-devel@lists.xen.org; keir@xen.org > Subject: RE: 32bit xen and "claim" > > >>> On 05.11.12 at 20:16, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: > > Does it make sense to have a runtime option that unsets the > > physical limit but disallows legacy PV guests? If this > > defaults to false for machines with RAM<=5TB but to true > > for machines with RAM>5TB, then the feature is "done" > > (AND we have put a stake in the ground to begin the > > slow obsolescence of PV functionality). > > That would be interesting: Mukesh''s PVH code wasn''t even > posted yet, i.e. you''re proposing to render systems with more > than 5Tb unbootable (for the lack of a - necessarily PV - Dom0 > kernel runnable in that environment).Good point. BUT... couldn''t a PV dom0 started with dom0_mem=X (where X is smaller than 5GB) still work?> But yes, the plan is to extend the 1:1 mapping beyond 5Tb for > non-PV guests, and going through actual mapping code only > when acting in the context of a (64-bit) PV guest (even 32-bit > PV guests can have the full 1:1 mapping).That makes sense. I guess I am just worried that this will require enough surgery that there will be a long bug tail. And, since 5TB machines will be quite rare for the next year or so, deprecating PV domains on those machines earlier than on smaller machines might have a much smaller impact on the Xen community. But this is all just a suggestion... I will stop now and leave it in your capable hands. Dan
>>> On 06.11.12 at 21:40, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: >> From: Jan Beulich [mailto:JBeulich@suse.com] >> Sent: Tuesday, November 06, 2012 2:44 AM >> To: Dan Magenheimer >> Cc: xen-devel@lists.xen.org; keir@xen.org >> Subject: RE: 32bit xen and "claim" >> >> >>> On 05.11.12 at 20:16, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: >> > Does it make sense to have a runtime option that unsets the >> > physical limit but disallows legacy PV guests? If this >> > defaults to false for machines with RAM<=5TB but to true >> > for machines with RAM>5TB, then the feature is "done" >> > (AND we have put a stake in the ground to begin the >> > slow obsolescence of PV functionality). >> >> That would be interesting: Mukesh''s PVH code wasn''t even >> posted yet, i.e. you''re proposing to render systems with more >> than 5Tb unbootable (for the lack of a - necessarily PV - Dom0 >> kernel runnable in that environment). > > Good point. BUT... couldn''t a PV dom0 started with dom0_mem=X > (where X is smaller than 5GB) still work?No - Xen would still need to be able to access all memory when in the context of such a Dom0 (even address restricting that Dom0 just like we do for 32-bit PV guests wouldn''t help, as e.g. granted pages from DomU-s may need accessing in Xen in the context of Dom0). Jan