I was looking at the definition of privcmd_hypercall:
typedef struct privcmd_hypercall
{
unsigned long op;
unsigned long arg[5];
} privcmd_hypercall_t;
For ppc64, we are using 32-bit management tools, so this is a problem: they
will create structs where long is 32 bits, and the (64-bit) kernel and
hypervisor will expect structs where long is 64 bits.
The standard (and awkward) way of dealing with the privcmd ioctl is to create
an in-kernel privcmd_hypercall_t, copy the 32-bit values into it field by
field, and then pass *that* struct on to privcmd_ioctl(). Of course,
that''s
only for legacy interfaces; for all new interfaces, we can just design them
properly so that their size and alignment doesn''t change.
There are also longs in some of the dom0_op sub-structures. For example:
typedef struct {
/* IN variables. */
domid_t domain;
unsigned long max_memkb;
} dom0_setdomainmaxmem_t;
I suggest that all longs in these structures be converted to u32.
--
Hollis Blanchard
IBM Linux Technology Center
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 28 Sep 2005, at 22:36, Hollis Blanchard wrote:> I was looking at the definition of privcmd_hypercall: > typedef struct privcmd_hypercall > { > unsigned long op; > unsigned long arg[5]; > } privcmd_hypercall_t; > > For ppc64, we are using 32-bit management tools, so this is a problem: > they > will create structs where long is 32 bits, and the (64-bit) kernel and > hypervisor will expect structs where long is 64 bits.We want the args to be ''machine-register sized'' which is why they are longs. Why use 32-bit tools on 64-bit hypervisor? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>>>> "KF" == Keir Fraser <Keir.Fraser@cl.cam.ac.uk> writes:KF> On 28 Sep 2005, at 22:36, Hollis Blanchard wrote: KF> We want the args to be ''machine-register sized'' which is why they KF> are longs. hmm, It appears that 32bits is sufficient, esp since almost everything is expressed in terms of pages (takes us to 16TB). However, we should be forward thinking :). Anyway, since this is a Xen/Kernel header, perhpas we can create a type that represent the Xen/Kernel machine-register size? KF> Why use 32-bit tools on 64-bit hypervisor? Unlike x86_64 and ia64, there is no performance gain by making regular apps 64bit on PPC, so we tend to have almost all our apps be the same 32 bit apps that run on 32 bit kernels. This way, when Xen-PPC32 is ready the same xentools binary will apply. Besides, most systems don''t have a 64bit python or libz installed or readily available. BTW: Is the plan to not support 32bit domain (0 or U) on a 64bit Xen? -JX -- "I got an idea, an idea so smart my head would explode if I even began to know what I was talking about." -- Peter Griffin (Family Guy) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 29 Sep 2005, at 13:41, Jimi Xenidis wrote:> BTW: Is the plan to not support 32bit domain (0 or U) on a 64bit Xen?That''s the case for paravirtualised x86 guests. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Anyway, since this is a Xen/Kernel header, perhpas we can > create a type that represent the Xen/Kernel machine-register size?Since we''re only talking about dom0 ops, it wouldn''t be unreasonable to turn all the long''s into u64''s. This would be pretty straightforward, arguably easier than bothering with a u_reg type or similar. Since the dom0_op interface is not guaranteed stable in the 3.0 series [we assume you have matched tools and hypervisor] we don''t have to get this in before 3.0.0. However, it''s a pretty simple change and if someone who really cared were to knock up a tested patch immediately we''d potentially consider it. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thursday 29 September 2005 08:56, Ian Pratt wrote:> > Since we''re only talking about dom0 ops, it wouldn''t be unreasonable to > turn all the long''s into u64''s. This would be pretty straightforward, > arguably easier than bothering with a u_reg type or similar.start_info_t: tools, kernel xen_parameters_info_t: tools, hypervisor gnttab_setup_table_t: kernel, hypervisor gnttab_transfer_t: kernel, hypervisor struct xen_memory_reservation: tools, kernel, hypervisor struct t_rec: tools, hypervisor struct t_buf: tools, hypervisor (This was just a quick grep for "long" in xen/include/public.) The tools/hypervisor interface seems to be the most important at the moment. I don''t know if anybody has plans to do 32-bit kernel on a 64-bit hypervisor... but why not enable it while we''re in there? -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
[Somehow Keir was dropped from the CC list on my earlier mail; sorry.] On Thursday 29 September 2005 13:17, Hollis Blanchard wrote:> On Thursday 29 September 2005 08:56, Ian Pratt wrote: > > Since we''re only talking about dom0 ops, it wouldn''t be unreasonable to > > turn all the long''s into u64''s. This would be pretty straightforward, > > arguably easier than bothering with a u_reg type or similar. > > start_info_t: tools, kernel > xen_parameters_info_t: tools, hypervisor > gnttab_setup_table_t: kernel, hypervisor > gnttab_transfer_t: kernel, hypervisor > struct xen_memory_reservation: tools, kernel, hypervisor > struct t_rec: tools, hypervisor > struct t_buf: tools, hypervisor > > (This was just a quick grep for "long" in xen/include/public.)Of course, pointers have the same problem. Thankfully, it seems pointers are mostly found in the same list of offenders. (It''s also a little harder to grep for "*", so I could be missing some problems.) -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 29 Sep 2005, at 21:12, Hollis Blanchard wrote:>> start_info_t: tools, kernel >> xen_parameters_info_t: tools, hypervisor >> gnttab_setup_table_t: kernel, hypervisor >> gnttab_transfer_t: kernel, hypervisor >> struct xen_memory_reservation: tools, kernel, hypervisor >> struct t_rec: tools, hypervisor >> struct t_buf: tools, hypervisor >> >> (This was just a quick grep for "long" in xen/include/public.) > > Of course, pointers have the same problem. Thankfully, it seems > pointers are > mostly found in the same list of offenders. (It''s also a little harder > to > grep for "*", so I could be missing some problems.)How would you ''fix'' pointers? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 29 Sep 2005, at 21:12, Hollis Blanchard wrote:>> start_info_t: tools, kernel >> xen_parameters_info_t: tools, hypervisor >> gnttab_setup_table_t: kernel, hypervisor >> gnttab_transfer_t: kernel, hypervisor >> struct xen_memory_reservation: tools, kernel, hypervisor >> struct t_rec: tools, hypervisor >> struct t_buf: tools, hypervisor >> >> (This was just a quick grep for "long" in xen/include/public.) > > Of course, pointers have the same problem. Thankfully, it seems > pointers are > mostly found in the same list of offenders. (It''s also a little harder > to > grep for "*", so I could be missing some problems.)How about writing a script that converts the Xen public headers into a set that, when compiled on 32-bit ppc, match the 64-bit ppc layout? That might be saner than trying to manually finesse our headers around the ppc abi so that building in 32-bit or 64-bit environment both happen to give same binary layout. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Sep 29, 2005 at 11:43:10PM +0100, Keir Fraser wrote:> > On 29 Sep 2005, at 21:12, Hollis Blanchard wrote: > > >>start_info_t: tools, kernel > >>xen_parameters_info_t: tools, hypervisor > >>gnttab_setup_table_t: kernel, hypervisor > >>gnttab_transfer_t: kernel, hypervisor > >>struct xen_memory_reservation: tools, kernel, hypervisor > >>struct t_rec: tools, hypervisor > >>struct t_buf: tools, hypervisor > >> > >>(This was just a quick grep for "long" in xen/include/public.) > > > >Of course, pointers have the same problem. Thankfully, it seems > >pointers are > >mostly found in the same list of offenders. (It''s also a little harder > >to > >grep for "*", so I could be missing some problems.) > > How about writing a script that converts the Xen public headers into a > set that, when compiled on 32-bit ppc, match the 64-bit ppc layout? > That might be saner than trying to manually finesse our headers around > the ppc abi so that building in 32-bit or 64-bit environment both > happen to give same binary layout. >If this type of ABI compatibility is needed, interface data structures should be declared platform-independantly, i.e using ''sized'' types (yeah, field alignment should be taken care of too). That''ll guarantee 32/64 bit independance and that''s good thing to have. Once defined it can be managed in controlled way. There could be penalty involved for using larger data types than neccessary, but that can be resolved having 32-bit(or 64-bit, depending whichever is attractive as basic) hypercall subset. -Andrei _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 30 Sep 2005, at 01:54, Andrei Petrov wrote:>> How about writing a script that converts the Xen public headers into a >> set that, when compiled on 32-bit ppc, match the 64-bit ppc layout? >> That might be saner than trying to manually finesse our headers around >> the ppc abi so that building in 32-bit or 64-bit environment both >> happen to give same binary layout. >> > > If this type of ABI compatibility is needed, interface data structures > should be declared > platform-independantly, i.e using ''sized'' types (yeah, field alignment > should be taken care of too). > That''ll guarantee 32/64 bit independance and that''s good thing to > have. Once defined it can be > managed in controlled way. There could be penalty involved for using > larger data types than neccessary, > but that can be resolved having 32-bit(or 64-bit, depending whichever > is attractive as basic) > hypercall subset.When we previously had this, by defining packed structs, there were plenty of screams that it wasn''t ANSI compliant, and that performance sucks on some architectures. At the time there seemed no good reason for cross-ABI compatibility (we don''t plan to stuff these structs into network packets, for example). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>>>> "KF" == Keir Fraser <Keir.Fraser@cl.cam.ac.uk> writes:KF> On 30 Sep 2005, at 01:54, Andrei Petrov wrote: KF> When we previously had this, by defining packed structs, there were KF> plenty of screams that it wasn''t ANSI compliant, and that performance KF> sucks on some architectures. Tou use a research term, "thats just ka-ka" :) Packed in no way solves the problem of selecting the appropriate types. KF> At the time there seemed no good reason for cross-ABI KF> compatibility (we don''t plan to stuff these structs into network KF> packets, for example). There are 4 ABIs present, Xen (internal), Domain, Managment App, and the ABI (_not_ API) that Xen presents to all. We only wish to formalize the Xen External ABI in a binary form, not by lanuage. -JX -- "I got an idea, an idea so smart my head would explode if I even began to know what I was talking about." -- Peter Griffin (Family Guy) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sep 30, 2005, at 3:03 AM, Keir Fraser wrote:> > On 30 Sep 2005, at 01:54, Andrei Petrov wrote: > >>> How about writing a script that converts the Xen public headers into >>> a >>> set that, when compiled on 32-bit ppc, match the 64-bit ppc layout? >>> That might be saner than trying to manually finesse our headers >>> around >>> the ppc abi so that building in 32-bit or 64-bit environment both >>> happen to give same binary layout. >> >> If this type of ABI compatibility is needed, interface data >> structures should be declared >> platform-independantly, i.e using ''sized'' types (yeah, field >> alignment should be taken care of too).Absolutely agree.>> That''ll guarantee 32/64 bit independance and that''s good thing to >> have. Once defined it can be >> managed in controlled way. There could be penalty involved for using >> larger data types than neccessary, >> but that can be resolved having 32-bit(or 64-bit, depending whichever >> is attractive as basic) >> hypercall subset. > > When we previously had this, by defining packed structs, there were > plenty of screams that it wasn''t ANSI compliant, and that performance > sucks on some architectures. At the time there seemed no good reason > for cross-ABI compatibility (we don''t plan to stuff these structs into > network packets, for example).The complaints I remember were specifically about using GCC''s __attribute__((packed)) extension. Properly laying out the structures and/or using explicit padding is a different solution. Example: typedef struct { /* IN variables. */ domid_t domain; unsigned long max_pfns; void *buffer; /* OUT variables. */ unsigned long num_pfns; } dom0_getmemlist_t; This could be fixed by laying it out like this: typedef struct { /* IN variables. */ u64 max_pfns; u64 buffer; domid_t domain; char _pad[6]; /* OUT variables. */ u64 num_pfns; } dom0_getmemlist_t; Or easier, losing the IN/OUT sections: typedef struct { u64 max_pfns; /* in */ u64 buffer; /* in */ u64 num_pfns; /* out */ domid_t domain; /* in */ } dom0_getmemlist_t; And yes, pointers will need to be "fixed" into u64s. As for performance, it does take an extra memory reference on x86 when assigning from a 32-bit value into a 64-bit field, but I would expect that cost to be insignificant compared to the cost of a hypercall. Maybe doing 64-bit math on x86 would be significantly slower, but that''s not what we''re talking about... -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 30 Sep 2005, at 16:39, Hollis Blanchard wrote:> And yes, pointers will need to be "fixed" into u64s. > > As for performance, it does take an extra memory reference on x86 when > assigning from a 32-bit value into a 64-bit field, but I would expect > that cost to be insignificant compared to the cost of a hypercall. > Maybe doing 64-bit math on x86 would be significantly slower, but > that''s not what we''re talking about...Cross-architecture ''compatibility'' (same binary layout) is not currently an aim for the Xen-public interfaces, and I don''t expect it to become so. If we went down that road we''d have to stipulate things like endianess, which I think we can all agree is not the way to go. I certainly don''t want to wholesale restructure our interfaces just to fortuitously make things match up for 32- and 64-bit ppc (which is what you are actually arguing for, in the guise of more general cross-architecture compatibility). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sep 30, 2005, at 10:45 AM, Keir Fraser wrote:> > Cross-architecture ''compatibility'' (same binary layout) is not > currently an aim for the Xen-public interfaces, and I don''t expect it > to become so. If we went down that road we''d have to stipulate things > like endianess, which I think we can all agree is not the way to go. > > I certainly don''t want to wholesale restructure our interfaces just to > fortuitously make things match up for 32- and 64-bit ppc (which is > what you are actually arguing for, in the guise of more general > cross-architecture compatibility).Yes, I''m not trying to be sneaky about this: this is a problem for ppc64. It could also be a problem for x86-64, except that x86-32 is so limited compared to x86-64 that apparently nobody wants to do this. I think the Linux kernel folks learned the "unsigned long" lesson too late, and now there is quite a lot of fixup code to convert 32-bit userspace structs to 64-bit kernel structs (have a look at linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the same mistakes in Xen... -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 30 Sep 2005, at 17:34, Hollis Blanchard wrote:>> Cross-architecture ''compatibility'' (same binary layout) is not >> currently an aim for the Xen-public interfaces, and I don''t expect it >> to become so. If we went down that road we''d have to stipulate things >> like endianess, which I think we can all agree is not the way to go. >> >> I certainly don''t want to wholesale restructure our interfaces just >> to fortuitously make things match up for 32- and 64-bit ppc (which is >> what you are actually arguing for, in the guise of more general >> cross-architecture compatibility). > > Yes, I''m not trying to be sneaky about this: this is a problem for > ppc64. It could also be a problem for x86-64, except that x86-32 is so > limited compared to x86-64 that apparently nobody wants to do this. > > I think the Linux kernel folks learned the "unsigned long" lesson too > late, and now there is quite a lot of fixup code to convert 32-bit > userspace structs to 64-bit kernel structs (have a look at > linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the > same mistakes in Xen...There''s the rub: we don''t expect to ever want to provide 32-bit x86 ABI compatibility on 64-bit x86 Xen. We will not be supporting 32-bit paravirtualised guests on 64-bit x86 Xen, and we''ve taken the approach of requiring separate 32- and 64-bit toolsets (which isn''t too painful on x86 since full-fledged 64-bit filesystems are quite easy to come by). Really this sounds to me like this is only going to be a problem for ppc. That given, knocking up a translation script to import the interfaces into your 32-bit toolchain does seem worthy of consideration. It wouldn''t allow you to use the same tool binaries on both 32- and 64-bit ppc Xen, but I don''t know how much you actually care about that... -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi,> > I think the Linux kernel folks learned the "unsigned long" lesson too > late, and now there is quite a lot of fixup code to convert 32-bit > userspace structs to 64-bit kernel structs (have a look at > linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the > same mistakes in Xen... >Quick question.... What is this "''unsigned long'' lesson" you are refering to? -- David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 30 Sep 2005, at 17:44, David wrote:>> I think the Linux kernel folks learned the "unsigned long" lesson too >> late, and now there is quite a lot of fixup code to convert 32-bit >> userspace structs to 64-bit kernel structs (have a look at >> linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the >> same mistakes in Xen... >> > Quick question.... What is this "''unsigned long'' lesson" you are > refering to?As a rule, longs are 4 bytes on 32-bit architectures and 8 bytes on 64-bit architectures. If you want to support 32-bit apps on your shiny new 64-bit OS, it is nice if your interface doesn''t contain too many structures with different layouts between 32-bit and 64-bit. If you have pointers and longs, the structures have differenrt layouts and you need a shim layer at the top of the OS to convert 32-bit format into the 64-bit format that the OS uses internally. Which is a pain. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Sep 30, 2005 at 04:45:48PM +0100, Keir Fraser wrote:> > Cross-architecture ''compatibility'' (same binary layout) is not > currently an aim for the Xen-public interfaces, and I don''t expect it > to become so. If we went down that road we''d have to stipulate things > like endianess, which I think we can all agree is not the way to go. > > I certainly don''t want to wholesale restructure our interfaces just to > fortuitously make things match up for 32- and 64-bit ppc (which is what > you are actually arguing for, in the guise of more general > cross-architecture compatibility). >The only practical consideration would be compatibility between 32/64 modes on the same platform for hypercalls. I would expect 32-bit guest to be able to run on xen-64 and that easier to achive if hypercall interface is clean. Seems(need to look more though) it could be done by redefining current data types in such a way so that they will have the same layout on 32-bit platform. -Andrei _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sep 30, 2005, at 11:44 AM, David wrote:>> I think the Linux kernel folks learned the "unsigned long" lesson too >> late, and now there is quite a lot of fixup code to convert 32-bit >> userspace structs to 64-bit kernel structs (have a look at >> linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the >> same mistakes in Xen... >> > Quick question.... What is this "''unsigned long'' lesson" you are > refering to?"long" has the unfortunate tendency to change size between 32- and 64-bit compiles, like a pointer. So if you put that in a structure, the structure''s layout will change between 32- and 64-bit builds. When you make that structure part of the userland/kernel ABI, you will have problems on architectures that can run both 32- and 64-bit binaries, which include PowerPC, MIPS, x86-64, Sparc, and possibly S390 and even IA64. -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sep 30, 2005, at 11:42 AM, Keir Fraser wrote:> > There''s the rub: we don''t expect to ever want to provide 32-bit x86 > ABI compatibility on 64-bit x86 Xen. We will not be supporting 32-bit > paravirtualised guests on 64-bit x86 Xen, and we''ve taken the approach > of requiring separate 32- and 64-bit toolsets (which isn''t too painful > on x86 since full-fledged 64-bit filesystems are quite easy to come > by).That''s great, if all you care about is x86. That is how Linux started of course, and look where we are now... I think portability and well-designed interfaces are nice things to think about earlier rather than later. :)> Really this sounds to me like this is only going to be a problem for > ppc. That given, knocking up a translation script to import the > interfaces into your 32-bit toolchain does seem worthy of > consideration. It wouldn''t allow you to use the same tool binaries on > both 32- and 64-bit ppc Xen, but I don''t know how much you actually > care about that...A translation script seems exceedingly difficult. A separate set of headers would be less so, but extremely fragile. -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jimi Xenidis wrote:> KF> When we previously had this, by defining packed structs, there were > KF> plenty of screams that it wasn''t ANSI compliant, and that performance > KF> sucks on some architectures. > > Tou use a research term, "thats just ka-ka" :) > Packed in no way solves the problem of selecting the appropriate types.no, it''s not ka ka at all. I had huge troubles with plan 9 c and the way that the linux structs were packed via the use of gcc struct packing. I am really glad packed went away. The fact is, Xen has had trouble for me at times, as it is very linux/gcc specific. The one that hit me the hardest was exception frames, and second hardest was the packed keyword. I had to write marshall/unmarshall functions on Plan 9, just as though I were using RPC. It worked out, though, since (see below) I was able to end up with structs with better alignment. I thank the Xen team for removing packed structs. It was a good move. Also, there was some packing that resulted in very suboptimal structure member alignment. In one case, for one struct, a long struct member was odd-byte-aligned 1/2 the time, odd-short-aligned 1/4 the time, and correctly aligned 1/4 the time. There''s no reason to do this. There was also a lot of research disagreement on the plan 9 list about packing, when I brought this issue up; it''s not like all the angels sing from the gcc hymnal. Comments ranged from the politest, along the lines of "using packed is bad design", to less polite :-) There are lots of good arguments on both sides. But it really boils down to, ''which is most portable, using packed or not''. The answer is, not. thanks ron _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Friday 30 September 2005 15:05, Ronald G Minnich wrote:> Jimi Xenidis wrote: > > KF> When we previously had this, by defining packed structs, there were > > KF> plenty of screams that it wasn''t ANSI compliant, and that > > performance KF> sucks on some architectures. > > > > Tou use a research term, "thats just ka-ka" :) > > Packed in no way solves the problem of selecting the appropriate types. > > no, it''s not ka ka at all. I had huge troubles with plan 9 c and the way > that the linux structs were packed via the use of gcc struct packing. I > am really glad packed went away.I think you misunderstand: Jimi is also glad. Read again: the packed GCC attribute does not solve the problem. -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hollis Blanchard wrote:> On Friday 30 September 2005 15:05, Ronald G Minnich wrote: > >>Jimi Xenidis wrote: >> >>> KF> When we previously had this, by defining packed structs, there were >>> KF> plenty of screams that it wasn''t ANSI compliant, and that >>>performance KF> sucks on some architectures. >>> >>>Tou use a research term, "thats just ka-ka" :) >>>Packed in no way solves the problem of selecting the appropriate types. >> >>no, it''s not ka ka at all. I had huge troubles with plan 9 c and the way >>that the linux structs were packed via the use of gcc struct packing. I >>am really glad packed went away. > > > I think you misunderstand: Jimi is also glad. Read again: the packed GCC > attribute does not solve the problem. >That figures. I am really sorry it''s monday. thanks ron p.s. jimix, whereever you may be, I''m sorry. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2005-09-30 at 17:42 +0100, Keir Fraser wrote:> On 30 Sep 2005, at 17:34, Hollis Blanchard wrote: > >> Cross-architecture ''compatibility'' (same binary layout) is not > >> currently an aim for the Xen-public interfaces, and I don''t expect it > >> to become so. If we went down that road we''d have to stipulate things > >> like endianess, which I think we can all agree is not the way to go. > >> > >> I certainly don''t want to wholesale restructure our interfaces just > >> to fortuitously make things match up for 32- and 64-bit ppc (which is > >> what you are actually arguing for, in the guise of more general > >> cross-architecture compatibility). > > > > Yes, I''m not trying to be sneaky about this: this is a problem for > > ppc64. It could also be a problem for x86-64, except that x86-32 is so > > limited compared to x86-64 that apparently nobody wants to do this. > > > > I think the Linux kernel folks learned the "unsigned long" lesson too > > late, and now there is quite a lot of fixup code to convert 32-bit > > userspace structs to 64-bit kernel structs (have a look at > > linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the > > same mistakes in Xen... > > There''s the rub: we don''t expect to ever want to provide 32-bit x86 ABI > compatibility on 64-bit x86 Xen. We will not be supporting 32-bit > paravirtualised guests on 64-bit x86 Xen,... which I''ve previously said and continue to think is going to be something that will eventually be regretted. People are going to want to continue to run in a 32bit OS for legacy reasons for quite a while (even with the compatibility you get on x86_64). Making it so they can''t do mix and match of 32 and 64 bit guests on a single host is going to significantly reduce the utility of Xen. Jeremy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Katz wrote:> On Fri, 2005-09-30 at 17:42 +0100, Keir Fraser wrote: >> There''s the rub: we don''t expect to ever want to provide 32-bit x86 >> ABI compatibility on 64-bit x86 Xen. We will not be supporting 32-bit >> paravirtualised guests on 64-bit x86 Xen, > > ... which I''ve previously said and continue to think is going to be > something that will eventually be regretted. People are going to want > to continue to run in a 32bit OS for legacy reasons for quite a while > (even with the compatibility you get on x86_64). Making it so they > can''t do mix and match of 32 and 64 bit guests on a single host is > going to significantly reduce the utility of Xen. > > Jeremy > >Jeremy, Hi That''s mostly because of the H/W issues. One thing we could do there is to run such paravirtualised 32-bit guests in compatibility mode. But they would need to use 4-level page tables, and there are other minor differences. So those 32-bit guests wouldn''t be really same as the ones running on 32-bit Xen. I''m not sure that''s what you wanted, but it''s doable. I think a better way is to utilize H/W-based virtualization such as Intel Virtualization Technology (VT) or AMD''s Pacifica. That way we should be able to use the same binaries (thus ABI) for both 32-bit non-PAE and PAE domU on 64-bit x86 Xen. With H/W-based virtualization we can run those 32-bit guests cleanly in 32-bit on 64-bit Xen. 32-bit hypercalls from such 32-bit guests should be intercepted by a 32-bit ring0 component that converts "int 0x82" based ones to VMCALL or VMMCALL sent to 64-bit Xen (because such software interrupts won''t cause VMEXIT). There are other issues like impacts to the memory management in Xen, but I think those can be handled as special cases of shadow mode (i.e. don''t make shadow pages). I don''t think it would increase the utility of Xen right away (because this requires new processors), but it might be a sensible thing to do in the near future. For full-virutalization, today we can mix 32 and 64 bit guests on x86_64 Xen (as in the xen-unstable tree) when the H/W-based virtualization is present on the machine. Jun --- Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> start_info_t: tools, kernel > xen_parameters_info_t: tools, hypervisor > gnttab_setup_table_t: kernel, hypervisor > gnttab_transfer_t: kernel, hypervisor > struct xen_memory_reservation: tools, kernel, hypervisor > struct t_rec: tools, hypervisor struct t_buf: tools, hypervisor > > (This was just a quick grep for "long" in xen/include/public.) > > The tools/hypervisor interface seems to be the most important > at the moment. I don''t know if anybody has plans to do 32-bit > kernel on a 64-bit hypervisor... > but why not enable it while we''re in there?There''s rather more to running a 32 bit x86 xen guest on a 64 bit hypervisor then just worrying about the header files. I wouldn''t rule out ever doing it, but there''ll be a significant performance cost (e.g. it will require shadow pagetables) and a significant amount of work. The hypervisor to guest interface is obviously performance critical, and just changing everything into a 64 bit quantity certainly would not make sense from a memory usage, cache footprint and instruction count point of view. Looking through the public guest interface, the current use of ''long'' to get 32 bit quantities on x86_32 and 64 on x86_64 actually makes good sense. However, I don''t think we should be using ''long'' directly in public headers as it makes it harder to override. For example, if we ever support 32 bit guests on a 64 bit hypervisor we may want to use multiple compilation to build a compatibility layer or such like. We should probably typedef something like ''ureg'' that we could override as appropriate. This would also enable the Power folks to build a 32 bit binary tools for a 64 bit hypervisor. [I think I favour using a single typedef like ''ureg'' rather than coming up coming up with different names for all of the different uses of ''long'' as the latter seems faorly pointless] However, since we''re not actually changing the size of any types, this change isn''t essential to rush through before 3.0, though it might be nice as it should be very low risk. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, 2005-10-01 at 11:25 -0700, Nakajima, Jun wrote:> Jeremy Katz wrote: > > On Fri, 2005-09-30 at 17:42 +0100, Keir Fraser wrote: > >> There''s the rub: we don''t expect to ever want to provide 32-bit x86 > >> ABI compatibility on 64-bit x86 Xen. We will not be supporting 32-bit > >> paravirtualised guests on 64-bit x86 Xen, > > > > ... which I''ve previously said and continue to think is going to be > > something that will eventually be regretted. People are going to want > > to continue to run in a 32bit OS for legacy reasons for quite a while > > (even with the compatibility you get on x86_64). Making it so they > > can''t do mix and match of 32 and 64 bit guests on a single host is > > going to significantly reduce the utility of Xen.> That''s mostly because of the H/W issues. One thing we could do there is > to run such paravirtualised 32-bit guests in compatibility mode. But > they would need to use 4-level page tables, and there are other minor > differences. So those 32-bit guests wouldn''t be really same as the ones > running on 32-bit Xen. I''m not sure that''s what you wanted, but it''s > doable.You really want to get to the same 32bit paravirt guest being used regardless of what HV flavor you''re running. Otherwise, you''re in for a compatibility nitemare. What dictates that the guest has to know about the 4 level page tables? That''s like saying that your 32bit binaries running on 64bit Linux need to know that there is a larger address space when instead that gets handled by the kernel.> I think a better way is to utilize H/W-based virtualization such as > Intel Virtualization Technology (VT) or AMD''s Pacifica. That way we > should be able to use the same binaries (thus ABI) for both 32-bit > non-PAE and PAE domU on 64-bit x86 Xen. With H/W-based virtualization we > can run those 32-bit guests cleanly in 32-bit on 64-bit Xen. 32-bit > hypercalls from such 32-bit guests should be intercepted by a 32-bit > ring0 component that converts "int 0x82" based ones to VMCALL or VMMCALL > sent to 64-bit Xen (because such software interrupts won''t cause > VMEXIT). There are other issues like impacts to the memory management in > Xen, but I think those can be handled as special cases of shadow mode > (i.e. don''t make shadow pages). I don''t think it would increase the > utility of Xen right away (because this requires new processors), but it > might be a sensible thing to do in the near future.Right, the problem is that people have bought hardware and they''re going to want to use it. I think that having things work on the hardware I''ve already purchased will help to give people a much better migration path for the future. If I''m going to have to run my legacy 32bit stuff on a different machine anyway, then I can feel more free to look at other platforms for my "future" stuff. Jeremy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thursday 29 September 2005 08:56, Ian Pratt wrote:> > Anyway, since this is a Xen/Kernel header, perhpas we can > > create a type that represent the Xen/Kernel machine-register size? > > Since we''re only talking about dom0 ops, it wouldn''t be unreasonable to > turn all the long''s into u64''s. This would be pretty straightforward, > arguably easier than bothering with a u_reg type or similar. > > Since the dom0_op interface is not guaranteed stable in the 3.0 series > [we assume you have matched tools and hypervisor] we don''t have to get > this in before 3.0.0. However, it''s a pretty simple change and if > someone who really cared were to knock up a tested patch immediately > we''d potentially consider it.I''ve finally tested this patch on x86, which changes only the dom0_ops structures/code. In your mail from yesterday though, you advocate a "ureg" type. I guess you would like to see "ureg" used for both dom0 and hypercalls? -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Katz wrote:> On Sat, 2005-10-01 at 11:25 -0700, Nakajima, Jun wrote: >> Jeremy Katz wrote: >>> On Fri, 2005-09-30 at 17:42 +0100, Keir Fraser wrote: >>>> There''s the rub: we don''t expect to ever want to provide 32-bit x86 >>>> ABI compatibility on 64-bit x86 Xen. We will not be supporting >>>> 32-bit paravirtualised guests on 64-bit x86 Xen, >>> >>> ... which I''ve previously said and continue to think is going to be >>> something that will eventually be regretted. People are going to >>> want to continue to run in a 32bit OS for legacy reasons for quite >>> a while (even with the compatibility you get on x86_64). Making it >>> so they can''t do mix and match of 32 and 64 bit guests on a single >>> host is going to significantly reduce the utility of Xen. > >> That''s mostly because of the H/W issues. One thing we could do there >> is to run such paravirtualised 32-bit guests in compatibility mode. >> But they would need to use 4-level page tables, and there are other >> minor differences. So those 32-bit guests wouldn''t be really same as >> the ones running on 32-bit Xen. I''m not sure that''s what you wanted, >> but it''s doable. > > You really want to get to the same 32bit paravirt guest being used > regardless of what HV flavor you''re running. Otherwise, you''re in > for a compatibility nitemare. What dictates that the guest has to > know about the 4 level page tables? That''s like saying that your > 32bit binaries running on 64bit Linux need to know that there is a > larger address space when instead that gets handled by the kernel. >I fully agree that the same binary of 32bit paravirt guest be used regardless of the HV flavor (i.e. 32-bit or 64-bit). If we get the shadow mode working for those 32-bit paravirt guests, we could avoid using 4-level page tables in the guest kernel (although the processor actually uses 4-level page tables). In that case, however, we will lose the good optimizations for paravirt guests, such as writeable page tables and "late pin, early unpin." (but then we get the save/restore/migration feature) There are other areas to look at, such as, ring1 vs. ring3, but I think we can handle those in Xen.>> I think a better way is to utilize H/W-based virtualization such as >> Intel Virtualization Technology (VT) or AMD''s Pacifica. That way we >> should be able to use the same binaries (thus ABI) for both 32-bit >> non-PAE and PAE domU on 64-bit x86 Xen. With H/W-based >> virtualization we can run those 32-bit guests cleanly in 32-bit on >> 64-bit Xen. 32-bit hypercalls from such 32-bit guests should be >> intercepted by a 32-bit ring0 component that converts "int 0x82" >> based ones to VMCALL or VMMCALL sent to 64-bit Xen (because such >> software interrupts won''t cause VMEXIT). There are other issues like >> impacts to the memory management in Xen, but I think those can be >> handled as special cases of shadow mode (i.e. don''t make shadow >> pages). I don''t think it would increase the utility of Xen right >> away (because this requires new processors), but it might be a >> sensible thing to do in the near future. > > Right, the problem is that people have bought hardware and they''re > going to want to use it. I think that having things work on the > hardware I''ve already purchased will help to give people a much > better migration path for the future. If I''m going to have to run my > legacy 32bit stuff on a different machine anyway, then I can feel > more free to look at other platforms for my "future" stuff. > > JeremyThat''s a very good point, and that way we could provide more complete 32-bit compatibility than the one we get on x86_64 Linux. In terms of ABI/API, since Xen needs to disiguish 32-bit or 64-bit guests anyway at runtime, I don''t think we don''t need to change the size of any types at this point (i.e. before 3.0). Jun --- Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hollis Blanchard
2005-Oct-03 19:16 UTC
Re: [Xen-devel] 32/64-bit hypercall interface - padding
On Sunday 02 October 2005 16:21, Ian Pratt wrote:> The hypervisor to guest interface is obviously performance critical, and > just changing everything into a 64 bit quantity certainly would not make > sense from a memory usage, cache footprint and instruction count point > of view. Looking through the public guest interface, the current use of > ''long'' to get 32 bit quantities on x86_32 and 64 on x86_64 actually > makes good sense. > > However, I don''t think we should be using ''long'' directly in public > headers as it makes it harder to override. For example, if we ever > support 32 bit guests on a 64 bit hypervisor we may want to use multiple > compilation to build a compatibility layer or such like. We should > probably typedef something like ''ureg'' that we could override as > appropriate. This would also enable the Power folks to build a 32 bit > binary tools for a 64 bit hypervisor. > > [I think I favour using a single typedef like ''ureg'' rather than coming > up coming up with different names for all of the different uses of > ''long'' as the latter seems faorly pointless] > > However, since we''re not actually changing the size of any types, this > change isn''t essential to rush through before 3.0, though it might be > nice as it should be very low risk.I''m working on this patch now. However, ureg_t alone will not alleviate the padding issues that the dom0_ops structures have. Since we''re insisting on using types that change size (and alignment requirements), only reordering the fields will help there. Poor structure layout impacts memory usage and cache footprint, as you mention above. -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Monday 03 October 2005 14:11, Nakajima, Jun wrote:> In terms of ABI/API, since Xen needs to disiguish 32-bit or 64-bit > guests anyway at runtime, I don''t think we don''t need to change the size > of any types at this point (i.e. before 3.0).You would instead propose a compatibility layer in Xen? So when a hypercall from a 32-bit guest arrives at a 64-bit hypervisor, Xen code converts the 32-bit structure into a 64-bit one and passes that pointer on to the rest of Xen? And then for return values you''d convert the other way. Hmm, and of course you wouldn''t be able to pass 64-bit addresses back, such as via dom0_tbufcontrol_t. As mentioned previously, this is the approach Linux uses (linux/fs/compat_ioctl.c), and it seems less than ideal to me. Since we have the ability to fix it now (i.e. make the 32-bit and 64-bit ABI identical), shouldn''t we do that rather than this copying/munging layer? -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
You also said you would never support PAE. PAE is second-only to SMP support on the list of customer requirements. There may be a compelling reason not to, but superficially running many 32-bit paravirtualized guests on a 64-bit HV makes a great deal of sense. -Kip> There's the rub: we don't expect to ever want to provide 32-bit x86 ABI > compatibility on 64-bit x86 Xen. We will not be supporting 32-bit > paravirtualised guests on 64-bit x86 Xen, and we've taken the approach > of requiring separate 32- and 64-bit toolsets (which isn't too painful > on x86 since full-fledged 64-bit filesystems are quite easy to come > by). > > Really this sounds to me like this is only going to be a problem for > ppc. That given, knocking up a translation script to import the > interfaces into your 32-bit toolchain does seem worthy of > consideration. It wouldn't allow you to use the same tool binaries on > both 32- and 64-bit ppc Xen, but I don't know how much you actually > care about that... > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> You would instead propose a compatibility layer in Xen? So > when a hypercall from a 32-bit guest arrives at a 64-bit > hypervisor, Xen code converts the 32-bit structure into a > 64-bit one and passes that pointer on to the rest of Xen? And > then for return values you''d convert the other way. Hmm, and > of course you wouldn''t be able to pass 64-bit addresses back, > such as via dom0_tbufcontrol_t.The dom0 API is a xen <-> tools API. That''s *not* part of the gen guest API. Needing 64 bit *capable* tools for a 64 bit hypevisor is clearly a requirement. [The fact that Power wants to implement 64 bit capable tools as a 32 bit user space binary is totally orthogonal (and some might say slightly perverse)]> As mentioned previously, this is the approach Linux uses > (linux/fs/compat_ioctl.c), and it seems less than ideal to > me. Since we have the ability to fix it now (i.e. make the > 32-bit and 64-bit ABI identical), shouldn''t we do that rather > than this copying/munging layer?Although I wouldn''t object to using u64 everywhere in the tools (dom0) API, I most certainly would object to using it everywhere in the 32 bit x86 guest API as it would hurt performance and waste memory. *Definitely*. Just changing the tools API to all u64 seems pointless, as if you''re using a 64 bit hypervisor it seems perfectly reasonable to insist on using 64 bit tools -- every other system binary on your 64 bit redhat/suse install is going to be specially compiled for 64 bit, so why not yout hypervisor tools! If we''re going to make a change at all, switching to using ureg (or similar) rather than ''long'' is the thing to do to. This makes the job of implementing some future compat layer very slightly easier, and helps the Power guys do their funky 64-bit-tools-as-a-32-bit-binary thing. [*] Ian [*] implementing a hypercall compat layer is trivial compared to other overheads you''d have (like shadow page tables). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2005-10-03 at 20:56 +0100, Ian Pratt wrote:> Just changing the tools API to all u64 seems pointless, as if you''re > using a 64 bit hypervisor it seems perfectly reasonable to insist on > using 64 bit tools -- every other system binary on your 64 bit > redhat/suse install is going to be specially compiled for 64 bit, so > why not yout hypervisor tools!Although this is the case for x86_64, it is _NOT_ the case for ppc64 (or sparc64 if someone is ever demented enough to do it ;) ppc64 Linux ships with a largely 32-bit userland + a 64-bit kernel. And since the Xen tools are python, doing 64-bit tools in an environment like that gets very tricky -- the various scripting languages are not done in such a way that you can easily install both 32 and 64-bit versions Jeremy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Monday 03 October 2005 14:56, Ian Pratt wrote:> [*] implementing a hypercall compat layer is trivial compared to other > overheads you''d have (like shadow page tables).I agree with this statement. However, doesn''t that same argument apply to correcting the ABI in the first place? Shadow page tables will overshadow the performance impact of making the ABI 32/64-bit clean. In fact, even a plain old hypercall will also overshadow that performance impact, both in terms of cycle count and cache footprint. So if your choice then is between a compatibility translation layer and altering the interface, I think it''s pretty clear that changing the interface will result in the least amount of additional code (and associated long-term code maintenance). -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hollis Blanchard wrote:> On Monday 03 October 2005 14:11, Nakajima, Jun wrote: >> In terms of ABI/API, since Xen needs to disiguish 32-bit or 64-bit >> guests anyway at runtime, I don''t think we don''t need to change the >> size of any types at this point (i.e. before 3.0). > > You would instead propose a compatibility layer in Xen? So when a > hypercall from a 32-bit guest arrives at a 64-bit hypervisor, Xen > code converts the 32-bit structure into a 64-bit one and passes that > pointer on to the rest of Xen? And then for return values you''d > convert the other way. Hmm, and of course you wouldn''t be able to > pass 64-bit addresses back, such as via dom0_tbufcontrol_t.I don''t think dom0_tbufcontrol_t is a good example (as dicussed). Do you have other examples?> > As mentioned previously, this is the approach Linux uses > (linux/fs/compat_ioctl.c), and it seems less than ideal to me. Since > we have the ability to fix it now (i.e. make the 32-bit and 64-bit > ABI identical), shouldn''t we do that rather than this copying/munging > layer?The 32-bit and 64-bit hypercall ABI cannot be identical on x86 because of the generic ABI difference between 32-bit and 64-bit. Jun --- Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 3 Oct 2005, at 22:11, Hollis Blanchard wrote:> However, doesn''t that same argument apply to correcting the ABI in the > first > place? Shadow page tables will overshadow the performance impact of > making > the ABI 32/64-bit clean.The cost must also be paid by domains that don''t require shadow page tables (32-bit on 32-bit, in particular). The 64-bit dom0_op patch you posted misses 64-bit-ifying the page-frame arrays of GETMEMLIST and GETPAGEFRAMEINFO2. I had a go at 64-bit-ifying the dom0_op interface myself, and gave up at those two because they are particularly invasive to change. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> However, doesn''t that same argument apply to correcting the > ABI in the first place? Shadow page tables will overshadow > the performance impact of making the ABI 32/64-bit clean. > > In fact, even a plain old hypercall will also overshadow that > performance impact, both in terms of cycle count and cache footprint. > > So if your choice then is between a compatibility translation > layer and altering the interface, I think it''s pretty clear > that changing the interface will result in the least amount > of additional code (and associated long-term code maintenance).This would result in doubling the size of the all the p2m and m2p tables, which I really don''t think would be sensible: the physcial memory consumed would be somewhat annoying, but we''d also have to rejig the virtual memory layout to accommodate the larger tables (consuming more lowmem) and this really doesn''t make sense this late in the 3.0 dev cycle. We''d also double the cache foot print of some quite performance critical operations. You don''t need many cache misses to dominate the cost of a system call... Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 3 Oct 2005, at 20:16, Hollis Blanchard wrote:>> However, since we''re not actually changing the size of any types, this >> change isn''t essential to rush through before 3.0, though it might be >> nice as it should be very low risk. > > I''m working on this patch now. > > However, ureg_t alone will not alleviate the padding issues that the > dom0_ops > structures have. Since we''re insisting on using types that change size > (and > alignment requirements), only reordering the fields will help there. > Poor > structure layout impacts memory usage and cache footprint, as you > mention > above.Well, at least this patch will be benign, but on its own it is also *totally* pointless. You can''t use it to get 64-bit struct arrangement on 32-bit builds because it will leave pointers as 32-bit aligned and sized fields. We could macro up pointer fields I suppose: #define XENIF_PTR(type, name) type name So that macro can be overridden to get 64-bit sized and aligned pointer fields? And ureg_t is a name that''s bound to clash with something down the road. Maybe xenreg_t, or even just be explicit about what it is and call it xenif_ulong_t? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> writes:> > Just changing the tools API to all u64 seems pointless, as if you''re > using a 64 bit hypervisor it seems perfectly reasonable to insist on > using 64 bit tools -- every other system binary on your 64 bit > redhat/suse install is going to be specially compiled for 64 bit, so why > not yout hypervisor tools!Actually a 64bit kernel works just fine with a full 32bit userland (with only a few minor exception of some tools which were too broken to be emulated - most notable one is iptables) People using distributions where using true 64bit is unusually hard or broken (like on a unnamed particular big community distribution) tend to use this setup. I use it also regularly for testing. On the other hand if the people need some tools as 64bit anyways then they can install 64bit hypervisor tools too. If you do that you should make it easy to install under a different prefix though.> If we''re going to make a change at all, switching to using ureg (or > similar) rather than ''long'' is the thing to do to. This makes the job of > implementing some future compat layer very slightly easier, and helps > the Power guys do their funky 64-bit-tools-as-a-32-bit-binary thing. [*]There are some traps. e.g. i386 has a different alignment for u64 (4 bytes) rather than the natural 8 bytes used on 64bits which causes many problems (like the iptables issue mentioned above). If you use this way use a new type which is long long __attribute__((aligned(8))), not just long long. -Andi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
"Nakajima, Jun" <jun.nakajima@intel.com> writes:> > The 32-bit and 64-bit hypercall ABI cannot be identical on x86 because > of the generic ABI difference between 32-bit and 64-bit.You use a custom ABI for hypercalls anyways, so you can define it to be the same with some care. That is the approach that is used by some newer subsystems in the Linux kernel. In my experience even people with the best intentions tend to get that wrong at some point because there are some subtle issues so having the additional safety net of a compat layer is still a good idea. It would only emulate the ones that went wrong. -Andi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
ak@suse.de wrote:> "Nakajima, Jun" <jun.nakajima@intel.com> writes: >> >> The 32-bit and 64-bit hypercall ABI cannot be identical on x86 >> because of the generic ABI difference between 32-bit and 64-bit. > > You use a custom ABI for hypercalls anyways, so you can define > it to be the same with some care. That is the approach that is > used by some newer subsystems in the Linux kernel. >In 32-bit, there are a couple of different instructions used for system calls, e.g. "int N", sysenter, and syscall. Some x86 CPUs support only sysenter, and some syscall, and "int N" is always available and it is the one used for 32-bit hypercalls. In 64-bit, the situation is much better, and we are using syscall for hypercalls, which is always available and can be more efficient than "int N". I don''t think we should use "int N" sacrificing the good feature for 64-bit when we can handle 32-bit hypercalls using a compatibility layer. I also think using the existing ABI convention is much better whenever possible and appropriate because we don''t need to re-invent the wheel. After all, the only difference between system call and hyper call basically is the system/hypercall number.> In my experience even people with the best intentions tend to get that > wrong at some point because there are some subtle issues so having the > additional safety net of a compat layer is still a good idea. It would > only emulate the ones that went wrong. > > -AndiI agree. Jun --- Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hollis Blanchard
2005-Oct-04 16:07 UTC
Re: [Xen-devel] 32/64-bit hypercall interface - padding
On Monday 03 October 2005 17:17, Keir Fraser wrote:> On 3 Oct 2005, at 20:16, Hollis Blanchard wrote: > >> However, since we''re not actually changing the size of any types, this > >> change isn''t essential to rush through before 3.0, though it might be > >> nice as it should be very low risk. > > > > I''m working on this patch now. > > > > Well, at least this patch will be benign, but on its own it is also > *totally* pointless. You can''t use it to get 64-bit struct arrangement > on 32-bit builds because it will leave pointers as 32-bit aligned and > sized fields.As part of the patch, I''m converting pointers to also use the new type, and casting the users appropriately.> We could macro up pointer fields I suppose: > #define XENIF_PTR(type, name) type name > So that macro can be overridden to get 64-bit sized and aligned pointer > fields?So you would use the above macro above for x86, and so continue with the existing unstable interface, but this would allow PPC to define its own macro, and thus have a 32/64-bit clean interface? Like this? #define XENIF_PTR(type, name) u64 name And then casts would still be needed to work with those values, right?> And ureg_t is a name that''s bound to clash with something down the > road. Maybe xenreg_t, or even just be explicit about what it is and > call it xenif_ulong_t?I don''t like calling it a "ulong," because it isn''t, and neither is it a "register" size... -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Monday 03 October 2005 16:24, Nakajima, Jun wrote:> Hollis Blanchard wrote: > > As mentioned previously, this is the approach Linux uses > > (linux/fs/compat_ioctl.c), and it seems less than ideal to me. Since > > we have the ability to fix it now (i.e. make the 32-bit and 64-bit > > ABI identical), shouldn''t we do that rather than this copying/munging > > layer? > > The 32-bit and 64-bit hypercall ABI cannot be identical on x86 because > of the generic ABI difference between 32-bit and 64-bit.I am not talking about the standard ABI used by the compiler (ppc32 and ppc64 use different ABIs as well). I am talking about the hypervisor/tools and hypervisor/kernel ABI. If the hypervisor ABI does not contain types that change size, it will be identical for both 32- and 64-bit users. If you meant something else when you said "generic ABI difference," could you explain? -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Monday 03 October 2005 17:00, Keir Fraser wrote:> > The cost must also be paid by domains that don''t require shadow page > tables (32-bit on 32-bit, in particular).The next sentence in my mail said that a normal hypercall would effectively have the same performance impact...> The 64-bit dom0_op patch you posted misses 64-bit-ifying the page-frame > arrays of GETMEMLIST and GETPAGEFRAMEINFO2. I had a go at 64-bit-ifying > the dom0_op interface myself, and gave up at those two because they are > particularly invasive to change.Couldn''t a field be added indicating the size of the resulting type? I have to run right now, but I''ll think about it more. -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4 Oct 2005, at 17:07, Hollis Blanchard wrote:>> We could macro up pointer fields I suppose: >> #define XENIF_PTR(type, name) type name >> So that macro can be overridden to get 64-bit sized and aligned >> pointer >> fields? > > So you would use the above macro above for x86, and so continue with > the > existing unstable interface, but this would allow PPC to define its own > macro, and thus have a 32/64-bit clean interface? Like this? > #define XENIF_PTR(type, name) u64 name > And then casts would still be needed to work with those values, right?You can add pack/unpack macros if you like. On x86: define xenif_pack_ptr(field, var,) field = var define xenif_unpack_ptr(var, field) var = field I don;t want to lose pointer type checks. The manual (u64)(unsigned long) casting is gross and loses type checking.>> And ureg_t is a name that''s bound to clash with something down the >> road. Maybe xenreg_t, or even just be explicit about what it is and >> call it xenif_ulong_t? > > I don''t like calling it a "ulong," because it isn''t, and neither is it > a > "register" size...Then what will you call it? xenppchack_t? :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tuesday 04 October 2005 18:03, Nakajima, Jun wrote:> > I don''t think we should use "int N" sacrificing the good feature for > 64-bit when we can handle 32-bit hypercalls using a compatibility layer. > I also think using the existing ABI convention is much better whenever > possible and appropriate because we don''t need to re-invent the wheel. > After all, the only difference between system call and hyper call > basically is the system/hypercall number.Main 32bit linux solves this problem by mapping a vsyscall page into the address space. The ABI is then just to jump to an appropiate entry point in it. The entry point uses the right instruction then. -Andi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Monday 03 October 2005 17:03, Ian Pratt wrote:> > However, doesn''t that same argument apply to correcting the > > ABI in the first place? Shadow page tables will overshadow > > the performance impact of making the ABI 32/64-bit clean. > > > > In fact, even a plain old hypercall will also overshadow that > > performance impact, both in terms of cycle count and cache footprint. > > > > So if your choice then is between a compatibility translation > > layer and altering the interface, I think it''s pretty clear > > that changing the interface will result in the least amount > > of additional code (and associated long-term code maintenance). > > This would result in doubling the size of the all the p2m and m2p > tables,Would it (honest question)? Those tables aren''t part of the hypercall interface itself, right? So as long as the hypercalls dealing with those tables are modified appropriately, the tables themselves don''t need the change? -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hollis Blanchard
2005-Oct-04 21:16 UTC
Re: [Xen-devel] 32/64-bit hypercall interface -- p2m tables
On Monday 03 October 2005 17:03, Ian Pratt wrote:> > So if your choice then is between a compatibility translation > > layer and altering the interface, I think it''s pretty clear > > that changing the interface will result in the least amount > > of additional code (and associated long-term code maintenance). > > This would result in doubling the size of the all the p2m and m2p > tables,I don''t believe this is true. The tools can create those tables as arrays of 32- or 64-bit mfns depending on the bitsize of the domain being created. Xen would also know the bitsize of the domain, and so could treat the tables as 32- or 64-bit as needed. That of course only matters if people actually do the work to run 32-bit domains on 64-bit hypervisor. Fixing the ABI enables that work, should people want to pursue it later. -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hollis Blanchard
2005-Oct-04 21:41 UTC
Re: [Xen-devel] 32/64-bit hypercall interface - padding
On Tuesday 04 October 2005 11:39, Keir Fraser wrote:> > I don;t want to lose pointer type checks. The manual (u64)(unsigned > long) casting is gross and loses type checking.Fair enough. Actually, another thought... have a look at the attached proof-of-concept patch. That preserves pointer type safety while still guaranteeing size and alignment.> > I don''t like calling it a "ulong," because it isn''t, and neither is it > > a "register" size... > > Then what will you call it? xenppchack_t? :-):) -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hollis Blanchard
2005-Oct-04 21:46 UTC
Re: [Xen-devel] Re: 32/64-bit hypercall interface -- long long alignment
On Tuesday 04 October 2005 07:08, Andi Kleen wrote:> > There are some traps. e.g. i386 has a different alignment for u64 (4 > bytes) rather than the natural 8 bytes used on 64bits which causes > many problems (like the iptables issue mentioned above). If you use > this way use a new type which is long long > __attribute__((aligned(8))), not just long long.Good to know! Thanks for the warning... -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hollis Blanchard wrote:> On Monday 03 October 2005 16:24, Nakajima, Jun wrote: >> Hollis Blanchard wrote: >>> As mentioned previously, this is the approach Linux uses >>> (linux/fs/compat_ioctl.c), and it seems less than ideal to me. Since >>> we have the ability to fix it now (i.e. make the 32-bit and 64-bit >>> ABI identical), shouldn''t we do that rather than this >>> copying/munging layer? >> >> The 32-bit and 64-bit hypercall ABI cannot be identical on x86 >> because of the generic ABI difference between 32-bit and 64-bit. > > I am not talking about the standard ABI used by the compiler (ppc32 > and ppc64 use different ABIs as well). I am talking about the > hypervisor/tools and hypervisor/kernel ABI. > > If the hypervisor ABI does not contain types that change size, it > will be identical for both 32- and 64-bit users. >Okay, I understand what you need. How about the data layout? That''s not so trivial to have the identical ones. For example, the following one would have different data layout on 32-bit and 64-bit. typedef struct vcpu_time_info { /* * Updates to the following values are preceded and followed by an * increment of ''version''. The guest can therefore detect updates by * looking for changes to ''version''. If the least-significant bit of * the version number is set then an update is in progress and the guest * must wait to read a consistent set of values. * The correct way to interact with the version number is similar to * Linux''s seqlock: see the implementations of read_seqbegin/read_seqretry. */ u32 version; u64 tsc_timestamp; /* TSC at last update of time vals. */ u64 system_time; /* Time, in nanosecs, since boot. */ /* * Current system time: * system_time + ((tsc - tsc_timestamp) << tsc_shift) * tsc_to_system_mul * CPU frequency (Hz): * ((10^9 << 32) / tsc_to_system_mul) >> tsc_shift */ u32 tsc_to_system_mul; s8 tsc_shift; } vcpu_time_info_t;> If you meant something else when you said "generic ABI difference," > could you explain?I meant basically the "standard ABI", such as, parameter passing, i.e. which registers are used to pass which arguments. x86_64 xenlinux runs in ring3, and hypercalls are not so different from system calls in terms of the mechanism/implementation. Then when we use fast system calls, we can use the compiler convention to minimize save/restore of the registers. Jun --- Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jimi Xenidis
2005-Oct-04 21:57 UTC
Re: [Xen-devel] 32/64-bit hypercall interface - padding
>>>>> "HB" == Hollis Blanchard <hollisb@us.ibm.com> writes:Nice idea but not sufficient cuz the pointer could be in the worng place you probably want struct { char _pad[sizeof (u64) - sizeof (long)]; vcpu_guest_context_t *ptr; } ctxt; HB> diff -r 85f92475b943 xen/include/public/dom0_ops.h HB> --- a/xen/include/public/dom0_ops.h Mon Oct 3 18:14:02 2005 HB> +++ b/xen/include/public/dom0_ops.h Tue Oct 4 16:37:44 2005 HB> @@ -99,7 +99,10 @@ HB> domid_t domain; HB> u16 vcpu; HB> /* IN/OUT parameters */ HB> - vcpu_guest_context_t *ctxt; HB> + union { HB> + vcpu_guest_context_t *ptr; HB> + u64 _; HB> + } ctxt; HB> } dom0_setdomaininfo_t; HB> #define DOM0_MSR 15 -- "I got an idea, an idea so smart my head would explode if I even began to know what I was talking about." -- Peter Griffin (Family Guy) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hollis Blanchard
2005-Oct-04 22:04 UTC
Re: [Xen-devel] 32/64-bit hypercall interface - padding
On Tuesday 04 October 2005 16:57, Jimi Xenidis wrote:> > Nice idea but not sufficient cuz the pointer could be in the worng > place you probably want > struct { > char _pad[sizeof (u64) - sizeof (long)]; > vcpu_guest_context_t *ptr; > } ctxt;Good catch. And this goes one step further to hide it: #define PAD_POINTER char _pad[sizeof(u64) - sizeof(void*)] typedef struct { ... union { PAD_POINTER; vcpu_guest_context_t *ptr; } ctxt; } dom0_setdomaininfo_t; Or this: #define PAD_POINTER(type, name) \ union { \ char _pad[sizeof(u64) - sizeof(void*)]; \ type *ptr; \ } name; typedef struct { ... PAD_POINTER(vcpu_guest_context_t, ctxt); } dom0_setdomaininfo_t; That looks nice, but it may confuse tools like cscope. Of course, so would the original XENIF_PTR macro idea. So I think I like the first better. -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tuesday 04 October 2005 16:51, Nakajima, Jun wrote:> Okay, I understand what you need. How about the data layout? That''s not > so trivial to have the identical ones. > For example, the following one would have different data layout on > 32-bit and 64-bit. > > typedef struct vcpu_time_info { > u32 version; > u64 tsc_timestamp; /* TSC at last update of time vals. */ > u64 system_time; /* Time, in nanosecs, since boot. */ > u32 tsc_to_system_mul; > s8 tsc_shift; > } vcpu_time_info_t;Is that because of the u64 4-byte alignment Andi mentioned? To use u64 in xen/include/public, we will need to ensure that __aligned__((8)) is used. The padding in some of these structures is unfortunate in terms of wasted (cacheline) space, but for now I will be happy with commonizing the size and layout. This particular structure couldn''t be compacted anyways... However, Keir indicated earlier he was not interested in reordering any structures, as would be needed to reduce wasted padding space. -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4 Oct 2005, at 19:05, Hollis Blanchard wrote:> Would it (honest question)? Those tables aren''t part of the hypercall > interface itself, right? So as long as the hypercalls dealing with > those > tables are modified appropriately, the tables themselves don''t need the > change?Well that''s true. Same holds for the pfn lists passed to GETMEMLIST, GETPAGEFRAMEINFO2, and the memory_op() hypercall. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4 Oct 2005, at 23:04, Hollis Blanchard wrote:> #define PAD_POINTER(type, name) \ > union { \ > char _pad[sizeof(u64) - sizeof(void*)]; \ > type *ptr; \ > } name; > typedef struct { > ... > PAD_POINTER(vcpu_guest_context_t, ctxt); > } dom0_setdomaininfo_t; > > That looks nice, but it may confuse tools like cscope. Of course, so > would the > original XENIF_PTR macro idea. So I think I like the first better.All approaches other than XENIF_PTR are broken in that they do not initialise the high-order bytes of the u64 field to zero. A 32-bit toolset will set only the low 4 bytes, leaving 64-bit hypervisor to read garbage for the high-order bytes. Here is how I would define these macros for ppc: define XENIF_PTR(type, name) union { u64 u; type t; }; define xenif_pack_ptr(field, var) <do type check>; field.u = (unsigned long)var define xenif_unpack_ptr(var, field) <do type check>; var = (void *)(unsigned long)field.u Where <do type check> does the type-check trick demonstrated by our min/max macros in include/xen/kernel.h. Something like: const typeof(var) _x = 0; const typeof(field.t) _y = 0; (void)(&_x == &_y); Note how the correctly-typed field of the union is used only for compile-time checking. Run-time data transfer is done entirely via the u64 field. The compile-time type check has no run-time overhead. For x86, those macros are all defined as the obvious ''no-ops''. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel