In order for Xen to be able to boot on systems with multiple PCI segments (also called domains), a number of changes are necessary to the hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as in most code paths and definitions there were not even provisions for passing a segment number. The hypercall interface changes may need some discussion before applying the patches, in particular - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable, or whether alternatively we should define a replacement one sub- hypercall - whether PHYSDEVOP_manage_pci_* should be deprecated - whether the bit assignments for the four uses of machine_bdf in the domctl interface can be re-defined Additionally, in the AMD IOMMU code there are two places where I was unable to identify how the segment value ought to be retrieved. Since I''m unaware of multi-segment AMD-based systems, imo this should not be a reason to not commit the changes proposed. 1: introduce notion of PCI segments 2: add new physdevop-s 3: adjust domctl interface 4: VT-d specific adjustments 5: AMD-IOMMU specific adjustments 6: Pass-through adjustments 7: config space accessor adjustments Signed-off-by: Jan Beulich <jbeulich@novell.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Sep-05 13:18 UTC
Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment support
>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote: > In order for Xen to be able to boot on systems with multiple PCI segments > (also called domains), a number of changes are necessary to the > hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as > in most code paths and definitions there were not even provisions for > passing a segment number. > > The hypercall interface changes may need some discussion before > applying the patches, in particular > > - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable, > or whether alternatively we should define a replacement one sub- > hypercall > - whether PHYSDEVOP_manage_pci_* should be deprecated > - whether the bit assignments for the four uses of machine_bdf in > the domctl interface can be re-definedNo comment from either of you on the proposed changes? Jan> Additionally, in the AMD IOMMU code there are two places where I > was unable to identify how the segment value ought to be retrieved. > Since I''m unaware of multi-segment AMD-based systems, imo this > should not be a reason to not commit the changes proposed. > > 1: introduce notion of PCI segments > 2: add new physdevop-s > 3: adjust domctl interface > 4: VT-d specific adjustments > 5: AMD-IOMMU specific adjustments > 6: Pass-through adjustments > 7: config space accessor adjustments > > Signed-off-by: Jan Beulich <jbeulich@novell.com>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2011-Sep-05 13:33 UTC
Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment support
On 05/09/2011 14:18, "Jan Beulich" <JBeulich@suse.com> wrote:>>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote: >> In order for Xen to be able to boot on systems with multiple PCI segments >> (also called domains), a number of changes are necessary to the >> hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as >> in most code paths and definitions there were not even provisions for >> passing a segment number. >> >> The hypercall interface changes may need some discussion before >> applying the patches, in particular >> >> - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable, >> or whether alternatively we should define a replacement one sub- >> hypercall >> - whether PHYSDEVOP_manage_pci_* should be deprecated >> - whether the bit assignments for the four uses of machine_bdf in >> the domctl interface can be re-defined > > No comment from either of you on the proposed changes?I''m personally fine with folding segment into the bus field. Otherwise we just end up with more compat cruft. I don''t have an opinion on the PHYSDEVOP_manage_pci_* hypercalls. In fact I don''t know much about them at all. I''ve always considered the domctl interface subject to change, but you don''t seem to redefine anything that already exists? You just give meaning to bits 24-31 of an existing 32-bit parameter? -- Keir> Jan > >> Additionally, in the AMD IOMMU code there are two places where I >> was unable to identify how the segment value ought to be retrieved. >> Since I''m unaware of multi-segment AMD-based systems, imo this >> should not be a reason to not commit the changes proposed. >> >> 1: introduce notion of PCI segments >> 2: add new physdevop-s >> 3: adjust domctl interface >> 4: VT-d specific adjustments >> 5: AMD-IOMMU specific adjustments >> 6: Pass-through adjustments >> 7: config space accessor adjustments >> >> Signed-off-by: Jan Beulich <jbeulich@novell.com> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Sep-05 13:49 UTC
Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment support
>>> On 05.09.11 at 15:33, Keir Fraser <keir.xen@gmail.com> wrote: > On 05/09/2011 14:18, "Jan Beulich" <JBeulich@suse.com> wrote: > >>>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote: >>> In order for Xen to be able to boot on systems with multiple PCI segments >>> (also called domains), a number of changes are necessary to the >>> hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as >>> in most code paths and definitions there were not even provisions for >>> passing a segment number. >>> >>> The hypercall interface changes may need some discussion before >>> applying the patches, in particular >>> >>> - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable, >>> or whether alternatively we should define a replacement one sub- >>> hypercall >>> - whether PHYSDEVOP_manage_pci_* should be deprecated >>> - whether the bit assignments for the four uses of machine_bdf in >>> the domctl interface can be re-defined >> >> No comment from either of you on the proposed changes? > > I''m personally fine with folding segment into the bus field. Otherwise we > just end up with more compat cruft. > > I don''t have an opinion on the PHYSDEVOP_manage_pci_* hypercalls. In fact I > don''t know much about them at all. > > I''ve always considered the domctl interface subject to change, but you don''t > seem to redefine anything that already exists? You just give meaning to bits > 24-31 of an existing 32-bit parameter?I''m trying to avoid incompatible changes when possible (due to out-of-tree consumers like libvirt, and due to the hacks required to use domctl interfaces from the kernel). Now here we need 16 bits, but have two sets of 8 (at bottom and top), hence I''d favor doing an incompatible change here (moving the bdf bits down to 0...15, and using 16...31 for the segment), perhaps renaming the field to machine_sbdf (to force compile-time noticing of the change at least for those that actually use our headers). But as the odd bit assignment could have other (hidden) reasons I coded things first to not do any re-assignments. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2011-Sep-05 14:05 UTC
Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment support
On 05/09/2011 14:49, "Jan Beulich" <JBeulich@suse.com> wrote:>>>> On 05.09.11 at 15:33, Keir Fraser <keir.xen@gmail.com> wrote: >> On 05/09/2011 14:18, "Jan Beulich" <JBeulich@suse.com> wrote: >> >>>>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote: >>>> In order for Xen to be able to boot on systems with multiple PCI segments >>>> (also called domains), a number of changes are necessary to the >>>> hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as >>>> in most code paths and definitions there were not even provisions for >>>> passing a segment number. >>>> >>>> The hypercall interface changes may need some discussion before >>>> applying the patches, in particular >>>> >>>> - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable, >>>> or whether alternatively we should define a replacement one sub- >>>> hypercall >>>> - whether PHYSDEVOP_manage_pci_* should be deprecated >>>> - whether the bit assignments for the four uses of machine_bdf in >>>> the domctl interface can be re-defined >>> >>> No comment from either of you on the proposed changes? >> >> I''m personally fine with folding segment into the bus field. Otherwise we >> just end up with more compat cruft. >> >> I don''t have an opinion on the PHYSDEVOP_manage_pci_* hypercalls. In fact I >> don''t know much about them at all. >> >> I''ve always considered the domctl interface subject to change, but you don''t >> seem to redefine anything that already exists? You just give meaning to bits >> 24-31 of an existing 32-bit parameter? > > I''m trying to avoid incompatible changes when possible (due to > out-of-tree consumers like libvirt,I think the intention is to maintain API compatibility for libxenlight, and have out-of-tree tool stacks/librariues build on top of that. I think there are libvirt bindings to libxenlight now, for example? My conclusion would be you can do the cleaner change to domctl. Interested in Ian Jackson''s view however. -- Keir> and due to the hacks required to > use domctl interfaces from the kernel). Now here we need 16 bits, but > have two sets of 8 (at bottom and top), hence I''d favor doing an > incompatible change here (moving the bdf bits down to 0...15, and > using 16...31 for the segment), perhaps renaming the field to > machine_sbdf (to force compile-time noticing of the change at least > for those that actually use our headers). But as the odd bit assignment > could have other (hidden) reasons I coded things first to not do any > re-assignments. > > Jan >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Sep-14 14:32 UTC
Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment support
>>> On 05.09.11 at 16:05, Keir Fraser <keir@xen.org> wrote: > On 05/09/2011 14:49, "Jan Beulich" <JBeulich@suse.com> wrote: > >>>>> On 05.09.11 at 15:33, Keir Fraser <keir.xen@gmail.com> wrote: >>> On 05/09/2011 14:18, "Jan Beulich" <JBeulich@suse.com> wrote: >>> >>>>>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote: >>>>> In order for Xen to be able to boot on systems with multiple PCI segments >>>>> (also called domains), a number of changes are necessary to the >>>>> hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as >>>>> in most code paths and definitions there were not even provisions for >>>>> passing a segment number. >>>>> >>>>> The hypercall interface changes may need some discussion before >>>>> applying the patches, in particular >>>>> >>>>> - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable, >>>>> or whether alternatively we should define a replacement one sub- >>>>> hypercall >>>>> - whether PHYSDEVOP_manage_pci_* should be deprecated >>>>> - whether the bit assignments for the four uses of machine_bdf in >>>>> the domctl interface can be re-defined >>>> >>>> No comment from either of you on the proposed changes? >>> >>> I''m personally fine with folding segment into the bus field. Otherwise we >>> just end up with more compat cruft. >>> >>> I don''t have an opinion on the PHYSDEVOP_manage_pci_* hypercalls. In fact I >>> don''t know much about them at all. >>> >>> I''ve always considered the domctl interface subject to change, but you don''t >>> seem to redefine anything that already exists? You just give meaning to bits >>> 24-31 of an existing 32-bit parameter? >> >> I''m trying to avoid incompatible changes when possible (due to >> out-of-tree consumers like libvirt, > > I think the intention is to maintain API compatibility for libxenlight, and > have out-of-tree tool stacks/librariues build on top of that. > > I think there are libvirt bindings to libxenlight now, for example? > > My conclusion would be you can do the cleaner change to domctl. Interested > in Ian Jackson''s view however.Ian?>> and due to the hacks required to >> use domctl interfaces from the kernel). Now here we need 16 bits, but >> have two sets of 8 (at bottom and top), hence I''d favor doing an >> incompatible change here (moving the bdf bits down to 0...15, and >> using 16...31 for the segment), perhaps renaming the field to >> machine_sbdf (to force compile-time noticing of the change at least >> for those that actually use our headers). But as the odd bit assignment >> could have other (hidden) reasons I coded things first to not do any >> re-assignments. >> >> Jan >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2011-Sep-20 18:02 UTC
Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment support
Keir Fraser writes ("Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment
support"):> I think the intention is to maintain API compatibility for libxenlight, and
> have out-of-tree tool stacks/librariues build on top of that.
Yes, but we''re not there yet.
> I think there are libvirt bindings to libxenlight now, for example?
There are but I''m not sure of their release status.
> My conclusion would be you can do the cleaner change to domctl. Interested
> in Ian Jackson''s view however.
We need to preserve compatibility with other versions of linux and
qemu. Do these programs need to know about these changes too ?
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Jan Beulich
2011-Sep-21 07:22 UTC
Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment support
>>> On 20.09.11 at 20:02, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote: > Keir Fraser writes ("Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment > support"): >> My conclusion would be you can do the cleaner change to domctl. Interested >> in Ian Jackson''s view however. > > We need to preserve compatibility with other versions of linux and > qemu. Do these programs need to know about these changes too ?I didn''t find any references to those domctl operations there (the kernel certainly shouldn''t have any, despite there being a couple of things that currently can''t be done without the use of domctls in the kernel), and qemu also shouldn''t have anything to do with the operations changed here. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel