Xen devel - May 2007 - [PATCH 2/3][RFC] MSI/MSI-X support for dom0/driver domain

This patch adds msi/msi-x support for domain0.

In native Linux 2.6.18, instead of irq, it is vector that are used when
MSI is enabled. We change it to use irq still when msi is enabled (This
change also happened in 2.6.20 later native Linux).
Add functions to allocate irq.
These two changes are not needed if we update to 2.6.21 kernel.

Remove the ack function in domain0, since xen will do that for us.

Issues:
1) Two phys_ops are added, First is physdev_msi_setup, to notify xen
that the vector is msi-irq type. The second is PHYSDEVOP_msi_format, to
get the msi format from xen side (mainly for target CPU), is it ok to
add them?

2) Current pirq is a global resource, how about make pirq be a
per-domain resource? 
Thanks
Yunhong Jiang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28/5/07 09:01, "Jiang, Yunhong" <yunhong.jiang@intel.com>
wrote:
> 1) Two phys_ops are added, First is physdev_msi_setup, to notify xen
> that the vector is msi-irq type. The second is PHYSDEVOP_msi_format, to
> get the msi format from xen side (mainly for target CPU), is it ok to
> add them?
> 
> 2) Current pirq is a global resource, how about make pirq be a
> per-domain resource?
Hi Jiang,

Yes, my main problem with this patchset is that it further overloads the
global pirq namespace inside Xen. Could we just specify vectors at the Xen
interface? Something like:

vector = PHYSDEVOP_alloc_irq_vector(MSI_IRQ)

Where MSI_IRQ is a specially-chosen value for the irq parameter to indicate
no presence in ''pirq'' namespace (from Xen''s point of
view at least!).

Then you don''t need physdev_msi_setup, as it''s implicit in
specifying
MSI_IRQ above. And then the fact that physdev_msi_format takes a vector
parameter makes more sense.

The new physdev_irq_permission can also provide a translation service into a
new per-domain pirq space. Takes parameters something like:
 @type: MSI, or INTx
 @idx: vector (MSI type), or irq number as specified to alloc_irq_vector
(INTx type)
 @domain: domain to remap to
 @pirq: ''pirq'' in that domain (allow caller to specify, or -1
to
auto-allocate).

Probably call it PHYSDEVOP_map_irq instead. And provide a
PHYSDEVOP_unmap_irq too (takes domain/pirq pair).

Then EVTCHNOP_bind_pirq clearly maps into this neatly! The only question is
how to map the MSIs into the dom0 pirq space. We could probably hook a
special call to map_irq() for dom0 into pci_enable_msi() or similar for
dom0-bound devices.

Overall I think the patches are looking good. I thought about whether we
should push more knowledge of MSI-X into Xen (e.g., so that it can
automatically mask/unmask/set-affinity on MSI-X sources) but I think
that''s
a bad idea after all, especially with VT-d and PCI-IOV coming down the
pipeline which will allow remapping of interrupts in a much nicer manner.

One other general comment is that you need to keep an eye out for coding
style (e.g., make sure you use 4-space soft tabs in most places in Xen;
always use 8-space hard tabs in Linux). Also, where you modify previously
untainted Linux files, make sure they should still compile correctly in
native builds (you may need to sprinkle some ifdef CONFIG_XEN''s around
the
place).

Anyway, good patches for a first cut!

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Keir Fraser
>Sent: 2007年5月28日 17:18
>
>The new physdev_irq_permission can also provide a translation service
>into a
>new per-domain pirq space. Takes parameters something like:
> @type: MSI, or INTx
> @idx: vector (MSI type), or irq number as specified to alloc_irq_vector
>(INTx type)
> @domain: domain to remap to
> @pirq: ''pirq'' in that domain (allow caller to specify, or
-1 to
>auto-allocate).
Who is the very owner to allocate per-domain pirq, domain itself or Xen? 
Or do you mean both sides can allocate by either "caller to specify " 
or ''-1'' for auto-allocation?

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28/5/07 10:42, "Tian, Kevin" <kevin.tian@intel.com> wrote:
>> The new physdev_irq_permission can also provide a translation service
>> into a
>> new per-domain pirq space. Takes parameters something like:
>> @type: MSI, or INTx
>> @idx: vector (MSI type), or irq number as specified to alloc_irq_vector
>> (INTx type)
>> @domain: domain to remap to
>> @pirq: ''pirq'' in that domain (allow caller to
specify, or -1 to
>> auto-allocate).
> 
> Who is the very owner to allocate per-domain pirq, domain itself or Xen?
> Or do you mean both sides can allocate by either "caller to specify
"
> or ''-1'' for auto-allocation?
I mean the caller can decide whether he allocates or whether Xen allocates.
My thinking is that if we made it so that this command needed to be used to
''wire up'' MSIs into dom0''s pirq namespace, and
we''re already implicitly
mapping PHYSDEVOP_alloc_irq_vector()-allocated IRQs into dom0''s pirq
namespace, it may be that dom0 has a region of its irq namespace that it
would like to use for MSIs and, if so, it is best placed to specify the pirq
parameter for itself. But I would expect that in many/most cases we''d
be
happy to leave the allocation to Xen.

If we wanted to just support one or the other doing the allocation, I would
say we should leave it to Xen, and make pirq an output-only field (rather
than in/out, as I specified it).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007年5月28日 18:20
>>
>> Who is the very owner to allocate per-domain pirq, domain itself or
>Xen?
>> Or do you mean both sides can allocate by either "caller to
specify "
>> or ''-1'' for auto-allocation?
>
>I mean the caller can decide whether he allocates or whether Xen
>allocates.
>My thinking is that if we made it so that this command needed to be used
>to
>''wire up'' MSIs into dom0''s pirq namespace, and
we''re already implicitly
>mapping PHYSDEVOP_alloc_irq_vector()-allocated IRQs into dom0''s
>pirq
>namespace, it may be that dom0 has a region of its irq namespace that it
>would like to use for MSIs and, if so, it is best placed to specify the pirq
>parameter for itself. But I would expect that in many/most cases
we''d be
>happy to leave the allocation to Xen.
>
>If we wanted to just support one or the other doing the allocation, I would
>say we should leave it to Xen, and make pirq an output-only field (rather
>than in/out, as I specified it).
>
> -- Keir
I have the concern on xen to be involved in pirq allocation. Consider 
the current case where pirq == irq, for INTx type devices of dom0, the 
pirq number is always derived from ACPI table though static, and Xen 
doesn''t know such static number before dom0 requests to bind. How 
about a MSI device enabled before an INTx only device? Also xen 
doesn''t know the pirq number to be used in the future like hotplug 
case. 

Unless the pirq namespace is really split from irq namespace by a 
non unity mapping style which however also impose changes to other 
INTx devices since all devices need to request pirq allocation from 
Xen then.

So I think it''s reasonable to always let dom0 to own the allocation of 
pirq namespace and then notify Xen to register even for MSI type 
device.

Native linux 2.6.18 has a messed namespace management, to 
actually have both vector and irq shared in same space. Later 2.6.20 
keeps them cleanly separate, with each MSI vector also allocated with 
an irq number. Then the syntax of irq in __do_IRQ always indicates the 
irq number and vector is only used when writing to MSI cap fields.

Attached patch for Xen also sticks to 2.6.20 style, to always allocate an
irq number bound with a MSI vector. By this way, Xen still does 
permission control on irq namespace and within dom0 evtchn is still 
mapped to irq namespace. Is it cleaner? :-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28/5/07 12:33, "Tian, Kevin" <kevin.tian@intel.com> wrote:
> I have the concern on xen to be involved in pirq allocation. Consider
> the current case where pirq == irq, for INTx type devices of dom0, the
> pirq number is always derived from ACPI table though static, and Xen
> doesn''t know such static number before dom0 requests to bind. How
> about a MSI device enabled before an INTx only device? Also xen
> doesn''t know the pirq number to be used in the future like hotplug
> case. 
Xen doesn''t give a crap about the pirq namespace, except for subtle
semantics associated with legacy isa irqs 0-15. Or at least, what little
care it does have can (and likely will) be removed. So it''s up to dom0
whether it wants its pirq namespace to correspond to BIOS-assigned scheme,
usual Linux allocation scheme, GSI space, or whatever. This interface will
let dom0 control how MSI and INTx is plumbed into its pirq space, if
that''s
what it wants. Other domUs will have no need for an association between
their pirq namespace and physical hardware/bios irq numbering -- in this
case it may make sense to leave it to Xen to do the allocation. But even
here, the interface as I described it would allow dom0 to have control over
domU allocation too if it wants it.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007年5月28日 19:48
>
>On 28/5/07 12:33, "Tian, Kevin" <kevin.tian@intel.com>
wrote:
>
>> I have the concern on xen to be involved in pirq allocation. Consider
>> the current case where pirq == irq, for INTx type devices of dom0, the
>> pirq number is always derived from ACPI table though static, and Xen
>> doesn''t know such static number before dom0 requests to bind.
How
>> about a MSI device enabled before an INTx only device? Also xen
>> doesn''t know the pirq number to be used in the future like
hotplug
>> case.
>
>Xen doesn''t give a crap about the pirq namespace, except for subtle
>semantics associated with legacy isa irqs 0-15. Or at least, what little
>care it does have can (and likely will) be removed. So it''s up to
dom0
>whether it wants its pirq namespace to correspond to BIOS-assigned
>scheme,
>usual Linux allocation scheme, GSI space, or whatever. This interface
>will
>let dom0 control how MSI and INTx is plumbed into its pirq space, if
>that''s
>what it wants. Other domUs will have no need for an association
>between
>their pirq namespace and physical hardware/bios irq numbering -- in this
>case it may make sense to leave it to Xen to do the allocation. But even
>here, the interface as I described it would allow dom0 to have control
>over
>domU allocation too if it wants it.
>
> -- Keir
OK, I agree it''s flexible and extensible. But is there any real usage 
model pushing on this? For example, is it better for pciback instance 
to allocate pirq space for domU? Pciback can select whether 
passthrough real irq number or allocate from a new space for 
target domain. To let Xen allocate instead makes it complex.

Also do we support mixed allocation policy to a given domain, 
when using suggested interface. For example, once one domain 
has BIOS-assigned scheme, it can''t request Xen for auto-allocation. 
Or else it''s difficult and complex for domain and Xen to sit on same 
page for shared allocation policy. Maybe we need some per-domain 
flag to control such allocation policy?

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Tian, Kevin
>Sent: 2007年5月28日 20:04
>
>>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>>Sent: 2007年5月28日 19:48
>>
>>Xen doesn''t give a crap about the pirq namespace, except for
subtle
>>semantics associated with legacy isa irqs 0-15. Or at least, what little
>>care it does have can (and likely will) be removed. So it''s up
to dom0
>>whether it wants its pirq namespace to correspond to BIOS-assigned
>>scheme,
>>usual Linux allocation scheme, GSI space, or whatever. This interface
>>will
>>let dom0 control how MSI and INTx is plumbed into its pirq space, if
>>that''s
>>what it wants. Other domUs will have no need for an association
>>between
>>their pirq namespace and physical hardware/bios irq numbering -- in
>this
>>case it may make sense to leave it to Xen to do the allocation. But even
>>here, the interface as I described it would allow dom0 to have control
>>over
>>domU allocation too if it wants it.
>>
>> -- Keir
>
>OK, I agree it''s flexible and extensible. But is there any real
usage
>model pushing on this? For example, is it better for pciback instance
>to allocate pirq space for domU? Pciback can select whether
>passthrough real irq number or allocate from a new space for
>target domain. To let Xen allocate instead makes it complex.
My point is:
	- Xen itself doesn''t take any usage of per-domain pirq namespace
	- Dom0 can stick to BIOS-scheme or a new pirq namespace scheme
	- Dom0 can also decide the scheme of domU by pciback

So why not let dom0 to own allocation for pirq namespace of all domains?

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28/5/07 13:03, "Tian, Kevin" <kevin.tian@intel.com> wrote:
> OK, I agree it''s flexible and extensible. But is there any real
usage
> model pushing on this? For example, is it better for pciback instance
> to allocate pirq space for domU? Pciback can select whether
> passthrough real irq number or allocate from a new space for
> target domain. To let Xen allocate instead makes it complex.
> 
> Also do we support mixed allocation policy to a given domain,
> when using suggested interface. For example, once one domain
> has BIOS-assigned scheme, it can''t request Xen for
auto-allocation.
> Or else it''s difficult and complex for domain and Xen to sit on
same
> page for shared allocation policy. Maybe we need some per-domain
> flag to control such allocation policy?
I think you''re making this more complicated than it really is. My
preference
for Xen doing the allocation for domUs is only because: what if there are
more places than just pciback that can assign device irqs to domUs? The
implementation complexity in Xen of supporting both allocation methods is
tiny: presumably in either case we will maintain an array of pirqs per
domain (maybe 0-255?) and in the case of Xen doing the allocation it just
has to do a find-first-unused-pirq search. Trivial. Even if it doesn''t
have
any users in the first instance (because we decide to let domU see real bios
numbers, or whatever), it''s a simple service to provide in case
it''s useful
in future.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28/5/07 13:12, "Tian, Kevin" <kevin.tian@intel.com> wrote:
> My point is:
> - Xen itself doesn''t take any usage of per-domain pirq namespace
> - Dom0 can stick to BIOS-scheme or a new pirq namespace scheme
> - Dom0 can also decide the scheme of domU by pciback
> 
> So why not let dom0 to own allocation for pirq namespace of all domains?
See the response I just posted. What if pciback is not the only place to
allocate irqs for domUs? Supporting both ways is cheap and easy even if we
don''t make use of that flexibility and dom0 always does the allocation
for
ever more. Case closed! :-)

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007年5月28日 20:14
>
>On 28/5/07 13:03, "Tian, Kevin" <kevin.tian@intel.com>
wrote:
>
>> OK, I agree it''s flexible and extensible. But is there any
real usage
>> model pushing on this? For example, is it better for pciback instance
>> to allocate pirq space for domU? Pciback can select whether
>> passthrough real irq number or allocate from a new space for
>> target domain. To let Xen allocate instead makes it complex.
>>
>> Also do we support mixed allocation policy to a given domain,
>> when using suggested interface. For example, once one domain
>> has BIOS-assigned scheme, it can''t request Xen for
auto-allocation.
>> Or else it''s difficult and complex for domain and Xen to sit
on same
>> page for shared allocation policy. Maybe we need some per-domain
>> flag to control such allocation policy?
>
>I think you''re making this more complicated than it really is. My
>preference
>for Xen doing the allocation for domUs is only because: what if there are
>more places than just pciback that can assign device irqs to domUs?
>The
>implementation complexity in Xen of supporting both allocation methods
>is
>tiny: presumably in either case we will maintain an array of pirqs per
>domain (maybe 0-255?) and in the case of Xen doing the allocation it just
>has to do a find-first-unused-pirq search. Trivial. Even if it
doesn''t have
>any users in the first instance (because we decide to let domU see real
>bios
>numbers, or whatever), it''s a simple service to provide in case
it''s useful
>in future.
>
> -- Keir
Let me re-open the case a bit. :-)

I understand your point, and yes that''s an easy implementation. My 
small concern now is just whether it''s worthy to pull Xen into resource
allocation for which Xen has nothing reference at all. Shouldn''t the 
components to assign device irq better does the allocation based on 
its own policy? For current stage, HVM domain has device model to 
provide ''pirq'' layout and driver domainU has pciback. Even
when later
there''re other places to assign device irqs, I think it''s
still responsibility
of that place to construct the pirq name space for domU. For example, 
how about the simple Xen pirq allocation policy doesn''t satisfy the 
special requirement of that place, like a special prime-number style 
(just kidding)? If such simple, but no-use from Xen POV, interface 
doesn''t have users now and also may not address all possibilities in 
the future, do we need that indeed?

P.S. I''m not against now, and just not used to the style a bit. :-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28/5/07 13:29, "Tian, Kevin" <kevin.tian@intel.com> wrote:
> I understand your point, and yes that''s an easy implementation. My
> small concern now is just whether it''s worthy to pull Xen into
resource
> allocation for which Xen has nothing reference at all. Shouldn''t
the
> components to assign device irq better does the allocation based on
> its own policy? For current stage, HVM domain has device model to
> provide ''pirq'' layout and driver domainU has pciback.
Even when later
> there''re other places to assign device irqs, I think it''s
still responsibility
> of that place to construct the pirq name space for domU. For example,
> how about the simple Xen pirq allocation policy doesn''t satisfy
the
> special requirement of that place, like a special prime-number style
> (just kidding)? If such simple, but no-use from Xen POV, interface
> doesn''t have users now and also may not address all possibilities
in
> the future, do we need that indeed?
You may be right. I just like to keep the hypervisor interfaces as flexible
as possible, to avoid unnecessarily baking in assumptions based on their
initial usage. It''s a pretty small issue actually, since we can get the
same
behaviour by dom0 attempting to map onto pirqs from zero upwards until it
finds one that isn''t already in use.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007年5月28日 20:41
>
>On 28/5/07 13:29, "Tian, Kevin" <kevin.tian@intel.com>
wrote:
>
>> I understand your point, and yes that''s an easy
implementation. My
>> small concern now is just whether it''s worthy to pull Xen into
resource
>> allocation for which Xen has nothing reference at all.
Shouldn''t the
>> components to assign device irq better does the allocation based on
>> its own policy? For current stage, HVM domain has device model to
>> provide ''pirq'' layout and driver domainU has pciback.
Even when later
>> there''re other places to assign device irqs, I think
it''s still responsibility
>> of that place to construct the pirq name space for domU. For example,
>> how about the simple Xen pirq allocation policy doesn''t
satisfy the
>> special requirement of that place, like a special prime-number style
>> (just kidding)? If such simple, but no-use from Xen POV, interface
>> doesn''t have users now and also may not address all
possibilities in
>> the future, do we need that indeed?
>
>You may be right. I just like to keep the hypervisor interfaces as flexible
>as possible, to avoid unnecessarily baking in assumptions based on their
>initial usage. It''s a pretty small issue actually, since we can get
the same
>behaviour by dom0 attempting to map onto pirqs from zero upwards until
>it
>finds one that isn''t already in use.
>
> -- Keir
Yep, exactly a small issue. Let''s expect Yunhong''s next
version after
incorporating your comments. :-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks for the discussion! I will firstly use Keir''s suggestion to let
Xen to allocate the pirq.

Another point is, should we export vector to domain0/driver domain in long run? 

I think vector is an item for cpu, so that when interrupt happen, cpu will jump
to IDT entry index by the vector. But for dom0/domU, it has no idea of IDT at
all, so why should we export vector for domain0/domU? Is the pirq enough?
As for irq/pirq, I think irq is the index to irq_desc( stated at
http://www.webservertalk.com/archive242-2006-5-1471415.html  and
http://marc.info/?l=linux-kernel&m=110021870415938&w=2), while pirq is a
virtual interrupt number  (something like gsi) injected by xen and is
corresponding to physical interrupt source. What''s special in
domain0/domain U is, in normal kernel, the gsi/irq maybe different, while in
domain0/domainU, we can arrange special so that irq/pirq is the same.

With this, for IOAPIC, domain0 will get a pirq from xen for a specific physical
gsi,and then in io_apic_set_pci_routing(), the irq, instead of vector will be
used. For msi(x), the  physdev_msi_format() will pass pirq/domain pair, and xen
will return back content with vector information in it.

Not sure if my understanding is right. Also, I suspect even my understanding is
right, should we do like this, since this will cause a lot of changes to
ioapic-xen.c code (we may have less changes after domain0 switch to latest
kernel).

Thanks
Yunhong Jiang



-----Original Message-----
From: Tian, Kevin 
Sent: 2007年5月28日 20:54
To: ''Keir Fraser''; Jiang, Yunhong;
xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] [PATCH 2/3][RFC] MSI/MSI-X support fordom0/driver
domain
>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007年5月28日 20:41
>
>On 28/5/07 13:29, "Tian, Kevin" <kevin.tian@intel.com>
wrote:
>
>> I understand your point, and yes that''s an easy
implementation. My
>> small concern now is just whether it''s worthy to pull Xen into
resource
>> allocation for which Xen has nothing reference at all.
Shouldn''t the
>> components to assign device irq better does the allocation based on
>> its own policy? For current stage, HVM domain has device model to
>> provide ''pirq'' layout and driver domainU has pciback.
Even when later
>> there''re other places to assign device irqs, I think
it''s still responsibility
>> of that place to construct the pirq name space for domU. For example,
>> how about the simple Xen pirq allocation policy doesn''t
satisfy the
>> special requirement of that place, like a special prime-number style
>> (just kidding)? If such simple, but no-use from Xen POV, interface
>> doesn''t have users now and also may not address all
possibilities in
>> the future, do we need that indeed?
>
>You may be right. I just like to keep the hypervisor interfaces as flexible
>as possible, to avoid unnecessarily baking in assumptions based on their
>initial usage. It''s a pretty small issue actually, since we can get
the same
>behaviour by dom0 attempting to map onto pirqs from zero upwards until
>it
>finds one that isn''t already in use.
>
> -- Keir
Yep, exactly a small issue. Let''s expect Yunhong''s next
version after
incorporating your comments. :-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28/5/07 15:03, "Jiang, Yunhong" <yunhong.jiang@intel.com>
wrote:
> Another point is, should we export vector to domain0/driver domain in long
> run? 
> 
> I think vector is an item for cpu, so that when interrupt happen, cpu will
> jump to IDT entry index by the vector. But for dom0/domU, it has no idea of
> IDT at all, so why should we export vector for domain0/domU? Is the pirq
> enough?
Well, dom0 is going to poke the vector into the MSI field of its PCI device
isn''t it? So it needs to know the vector. In VT-d/PCI-IOV I assume this
will
actually be an abstract value that gets remapped to an appropriate physical
vector. Since Xen is providing the vector number, we can change to providing
a ''virtual vector'' easily when the time comes. If it makes you
happier,
think of it as a cookie that gets provided by alloc_irq_vector() and then is
poked into the PCI device MSI field and also used in calls to
physdev_msi_format and physdev_map_irq.
> As for irq/pirq, I think irq is the index to irq_desc( stated at
> http://www.webservertalk.com/archive242-2006-5-1471415.html  and
> http://marc.info/?l=linux-kernel&m=110021870415938&w=2), while pirq
is a
> virtual interrupt number  (something like gsi) injected by xen and is
> corresponding to physical interrupt source. What''s special in
domain0/domain U
> is, in normal kernel, the gsi/irq maybe different, while in
domain0/domainU,
> we can arrange special so that irq/pirq is the same.
> 
> With this, for IOAPIC, domain0 will get a pirq from xen for a specific
> physical gsi,and then in io_apic_set_pci_routing(), the irq, instead of
vector
> will be used. For msi(x), the  physdev_msi_format() will pass pirq/domain
> pair, and xen will return back content with vector information in it.
Currently the msi_format command you added to Xen takes a domain/vector
pair. I think this is the correct thing to do, so don''t change it.

As for your comments on Linux pirq/irq/gsi management.... Erm, yeah. I
don''t
really fully understand all that ''clever'' intricacy. But
I''m not suggesting
you change it! Let me be clear: especially for dom0, rolling with whatever
policy it currently has for irq namespace management is the right thing to
do. My main point is that we don''t want to bake that into the Xen
interfaces
any more than we have to.
> Not sure if my understanding is right. Also, I suspect even my
understanding
> is right, should we do like this, since this will cause a lot of changes to
> ioapic-xen.c code (we may have less changes after domain0 switch to latest
> kernel).
I''m not really suggesting you make any drastic changes to what
you''ve done
on the Linux side! If you think otherwise then you''ve misunderstood my
intentions (and/or I''ve misunderstood your approach and patches this
far).

I just want the interfaces you''re adding to Xen simplified and made a
bit
more consistent and elegant. This will require some changes to your Linux
patches of course but I would not expect them to be drastic.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sure, you didn''t misunderstand my original patch :) 

I just wanted to exchange some my understanding for the interface between
xen/guest with you, not related to the MSI at all. Sorry for any confusing :$

I will update my patch according to your feedback.

Thanks
Yunhong  Jiang

-----Original Message-----
From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] 
Sent: 2007年5月28日 22:16
To: Jiang, Yunhong; Tian, Kevin; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] [PATCH 2/3][RFC] MSI/MSI-X support fordom0/driver
domain

On 28/5/07 15:03, "Jiang, Yunhong" <yunhong.jiang@intel.com>
wrote:
> Another point is, should we export vector to domain0/driver domain in long
> run? 
> 
> I think vector is an item for cpu, so that when interrupt happen, cpu will
> jump to IDT entry index by the vector. But for dom0/domU, it has no idea of
> IDT at all, so why should we export vector for domain0/domU? Is the pirq
> enough?
Well, dom0 is going to poke the vector into the MSI field of its PCI device
isn''t it? So it needs to know the vector. In VT-d/PCI-IOV I assume this
will
actually be an abstract value that gets remapped to an appropriate physical
vector. Since Xen is providing the vector number, we can change to providing
a ''virtual vector'' easily when the time comes. If it makes you
happier,
think of it as a cookie that gets provided by alloc_irq_vector() and then is
poked into the PCI device MSI field and also used in calls to
physdev_msi_format and physdev_map_irq.
> As for irq/pirq, I think irq is the index to irq_desc( stated at
> http://www.webservertalk.com/archive242-2006-5-1471415.html  and
> http://marc.info/?l=linux-kernel&m=110021870415938&w=2), while pirq
is a
> virtual interrupt number  (something like gsi) injected by xen and is
> corresponding to physical interrupt source. What''s special in
domain0/domain U
> is, in normal kernel, the gsi/irq maybe different, while in
domain0/domainU,
> we can arrange special so that irq/pirq is the same.
> 
> With this, for IOAPIC, domain0 will get a pirq from xen for a specific
> physical gsi,and then in io_apic_set_pci_routing(), the irq, instead of
vector
> will be used. For msi(x), the  physdev_msi_format() will pass pirq/domain
> pair, and xen will return back content with vector information in it.
Currently the msi_format command you added to Xen takes a domain/vector
pair. I think this is the correct thing to do, so don''t change it.

As for your comments on Linux pirq/irq/gsi management.... Erm, yeah. I
don''t
really fully understand all that ''clever'' intricacy. But
I''m not suggesting
you change it! Let me be clear: especially for dom0, rolling with whatever
policy it currently has for irq namespace management is the right thing to
do. My main point is that we don''t want to bake that into the Xen
interfaces
any more than we have to.
> Not sure if my understanding is right. Also, I suspect even my
understanding
> is right, should we do like this, since this will cause a lot of changes to
> ioapic-xen.c code (we may have less changes after domain0 switch to latest
> kernel).
I''m not really suggesting you make any drastic changes to what
you''ve done
on the Linux side! If you think otherwise then you''ve misunderstood my
intentions (and/or I''ve misunderstood your approach and patches this
far).

I just want the interfaces you''re adding to Xen simplified and made a
bit
more consistent and elegant. This will require some changes to your Linux
patches of course but I would not expect them to be drastic.

 -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel