crossbow discuss - Jul 2007 - Any progress with interrupt retargetting

Garrett/Roamer,

Have you made any progress with interrupt retargetting code. Basically,
when a cpu list is specified via ''-C'' option for a NIC/VNIC or
flow,
we need to do two things:
1) Make i_ddi_intr_get_cpuid() work so in case of NIC having a single
    interrupt, we can assign it to one of the specified CPUs if the
    CPU list is specified for a NIC.
2) In case the NIC supports MSI-X interrupt, we should be able to assign
    the interrupt for the Rx ring to one of the specified CPUs.

Which also means that the MAC to driver APIs need to be strong enough to
be able to associate a MSI-X interrupt with a Rx ring (Kais??).

Cheers,
Sunay

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

Garret,
Could you give an update if any progress? I didn''t look into it.

Roamer

Sunay Tripathi wrote:
> Garrett/Roamer,
> 
> Have you made any progress with interrupt retargetting code. Basically,
> when a cpu list is specified via ''-C'' option for a
NIC/VNIC or flow,
> we need to do two things:
> 1) Make i_ddi_intr_get_cpuid() work so in case of NIC having a single
>     interrupt, we can assign it to one of the specified CPUs if the
>     CPU list is specified for a NIC.
> 2) In case the NIC supports MSI-X interrupt, we should be able to assign
>     the interrupt for the Rx ring to one of the specified CPUs.
> 
> Which also means that the MAC to driver APIs need to be strong enough to
> be able to associate a MSI-X interrupt with a Rx ring (Kais??).
> 
> Cheers,
> Sunay
> 
-- 

# telnet (650)-786-6759 (x86759)
Connected to Solaris.Sun.COM.
login: Lu, Yunsong
Last login: January 2, 2007 from beyond.sfbay
Yunsong.Lu at Sun.COM    v1.03    Since Mon Dec. 22, 2003
[Roamer at Solaris Networking]# cd ..

Yunsong (Roamer) Lu wrote:


I''ve not made progress on this.  I was looking at changing intrd, to 
improve its allocation of processor interrupts, but I got stymied when 
the processor folks couldn''t seem to agree on any set of rules for how 
interrupt allocation should be done.  (Same core, different core?  
Results seem to vary based on the particular CPU and the nature of the 
workload.  Makes it hard to code up a set of rules when we don''t even 
know what the rules should be.)

If making i_ddi_intr_get_cpuid() work is a blocking factor for progress, 
then I can work on that.  I didn''t understand that this particular part
of the problem was needing a solution.

    -- Garrett
> Garret,
> Could you give an update if any progress? I didn''t look into it.
>
> Roamer
>
> Sunay Tripathi wrote:
>   
>> Garrett/Roamer,
>>
>> Have you made any progress with interrupt retargetting code. Basically,
>> when a cpu list is specified via ''-C'' option for a
NIC/VNIC or flow,
>> we need to do two things:
>> 1) Make i_ddi_intr_get_cpuid() work so in case of NIC having a single
>>     interrupt, we can assign it to one of the specified CPUs if the
>>     CPU list is specified for a NIC.
>> 2) In case the NIC supports MSI-X interrupt, we should be able to
assign
>>     the interrupt for the Rx ring to one of the specified CPUs.
>>
>> Which also means that the MAC to driver APIs need to be strong enough
to
>> be able to associate a MSI-X interrupt with a Rx ring (Kais??).
>>
>> Cheers,
>> Sunay
>>
>>     
>
>

Garrett D''Amore wrote:
> Yunsong (Roamer) Lu wrote:
> 
> 
> I''ve not made progress on this.  I was looking at changing intrd,
to
> improve its allocation of processor interrupts, but I got stymied when 
> the processor folks couldn''t seem to agree on any set of rules for
how
> interrupt allocation should be done.  (Same core, different core?  
> Results seem to vary based on the particular CPU and the nature of the 
> workload.  Makes it hard to code up a set of rules when we don''t
even
> know what the rules should be.)
> 
> If making i_ddi_intr_get_cpuid() work is a blocking factor for progress, 
> then I can work on that.  I didn''t understand that this particular
part
> of the problem was needing a solution.
This is the particular problem thats needing the most immediate
attention. The intrd gets complicated because we don''t know what
rules to apply but if we can make i_ddi_intr_get_cpuid() work,
the policies expressed by administrator can be honoured.

Cheers,
Sunay
> 
>    -- Garrett
>> Garret,
>> Could you give an update if any progress? I didn''t look into
it.
>>
>> Roamer
>>
>> Sunay Tripathi wrote:
>>  
>>> Garrett/Roamer,
>>>
>>> Have you made any progress with interrupt retargetting code.
Basically,
>>> when a cpu list is specified via ''-C'' option for
a NIC/VNIC or flow,
>>> we need to do two things:
>>> 1) Make i_ddi_intr_get_cpuid() work so in case of NIC having a
single
>>>     interrupt, we can assign it to one of the specified CPUs if the
>>>     CPU list is specified for a NIC.
>>> 2) In case the NIC supports MSI-X interrupt, we should be able to
assign
>>>     the interrupt for the Rx ring to one of the specified CPUs.
>>>
>>> Which also means that the MAC to driver APIs need to be strong
enough to
>>> be able to associate a MSI-X interrupt with a Rx ring (Kais??).
>>>
>>> Cheers,
>>> Sunay
>>>
>>>     
>>
>>   

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

Sunay

For static assignment, currently there is an ioctl interface 
"PCITOOL_DEVICE_SET_INTR" to open a nexus driver and change interrupt 
assignment. This is a userland interface used by pcitool. One can use 
"pcitool -i <i-no> -w <cpu-id>" to statically change
interrupt
assignment. This is the only supported interface. A kernel interface was 
requested by Ethan Solomita (intrd team), but is not present yet. 
Besides from what Govinda (cc''ed on this thread) has described, opening
a nexus driver and changing the interrupt assignment from within the 
kernel (i.e calling pcitool_set_intr) is not recommended/supported 
.Rather they would like to create a ddi_xxx alike interface for such 
task ....

Nitin
 


Sunay Tripathi wrote:
> Garrett/Roamer,
>
> Have you made any progress with interrupt retargetting code. Basically,
> when a cpu list is specified via ''-C'' option for a
NIC/VNIC or flow,
> we need to do two things:
> 1) Make i_ddi_intr_get_cpuid() work so in case of NIC having a single
>     interrupt, we can assign it to one of the specified CPUs if the
>     CPU list is specified for a NIC.
> 2) In case the NIC supports MSI-X interrupt, we should be able to assign
>     the interrupt for the Rx ring to one of the specified CPUs.
>
> Which also means that the MAC to driver APIs need to be strong enough to
> be able to associate a MSI-X interrupt with a Rx ring (Kais??).
>
> Cheers,
> Sunay
>
>

Yes, I think a DDI interface to this is preferred.

I''ve started looking at this.  It looks like DDI_INTROP functions are
needed.  Unfortunately, this portion of the code is really quite
platform specific, and it will take some time to figure out how best to
handle this.  Separate implementations for i86pc, sun4v, and probably
also sun4u, will be required... at minimum.

	-- Garrett

On Mon, 2007-07-30 at 19:37 -0700, Nitin Hande wrote:
> Sunay
> 
> For static assignment, currently there is an ioctl interface 
> "PCITOOL_DEVICE_SET_INTR" to open a nexus driver and change
interrupt
> assignment. This is a userland interface used by pcitool. One can use 
> "pcitool -i <i-no> -w <cpu-id>" to statically change
interrupt
> assignment. This is the only supported interface. A kernel interface was 
> requested by Ethan Solomita (intrd team), but is not present yet. 
> Besides from what Govinda (cc''ed on this thread) has described,
opening
> a nexus driver and changing the interrupt assignment from within the 
> kernel (i.e calling pcitool_set_intr) is not recommended/supported 
> .Rather they would like to create a ddi_xxx alike interface for such 
> task ....
> 
> Nitin
>  
> 
> 
> Sunay Tripathi wrote:
> > Garrett/Roamer,
> >
> > Have you made any progress with interrupt retargetting code.
Basically,
> > when a cpu list is specified via ''-C'' option for a
NIC/VNIC or flow,
> > we need to do two things:
> > 1) Make i_ddi_intr_get_cpuid() work so in case of NIC having a single
> >     interrupt, we can assign it to one of the specified CPUs if the
> >     CPU list is specified for a NIC.
> > 2) In case the NIC supports MSI-X interrupt, we should be able to
assign
> >     the interrupt for the Rx ring to one of the specified CPUs.
> >
> > Which also means that the MAC to driver APIs need to be strong enough
to
> > be able to associate a MSI-X interrupt with a Rx ring (Kais??).
> >
> > Cheers,
> > Sunay
> >
> >   
> 
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

DDI interface thats H/W neutral is perfectly fine. We can use it
from userland where dladm can take the specified CPU list and
bind the MSI-X interrupt to the first CPU in the list from
userland itself.

BTW, does pcitool work well enough on Niagara to test with?

Thanks,
Sunay


Garrett D''Amore wrote:
> Yes, I think a DDI interface to this is preferred.
> 
> I''ve started looking at this.  It looks like DDI_INTROP functions
are
> needed.  Unfortunately, this portion of the code is really quite
> platform specific, and it will take some time to figure out how best to
> handle this.  Separate implementations for i86pc, sun4v, and probably
> also sun4u, will be required... at minimum.
> 
> 	-- Garrett
> 
> On Mon, 2007-07-30 at 19:37 -0700, Nitin Hande wrote:
>> Sunay
>>
>> For static assignment, currently there is an ioctl interface 
>> "PCITOOL_DEVICE_SET_INTR" to open a nexus driver and change
interrupt
>> assignment. This is a userland interface used by pcitool. One can use 
>> "pcitool -i <i-no> -w <cpu-id>" to statically
change interrupt
>> assignment. This is the only supported interface. A kernel interface
was
>> requested by Ethan Solomita (intrd team), but is not present yet. 
>> Besides from what Govinda (cc''ed on this thread) has
described, opening
>> a nexus driver and changing the interrupt assignment from within the 
>> kernel (i.e calling pcitool_set_intr) is not recommended/supported 
>> .Rather they would like to create a ddi_xxx alike interface for such 
>> task ....
>>
>> Nitin
>>  
>>
>>
>> Sunay Tripathi wrote:
>>> Garrett/Roamer,
>>>
>>> Have you made any progress with interrupt retargetting code.
Basically,
>>> when a cpu list is specified via ''-C'' option for
a NIC/VNIC or flow,
>>> we need to do two things:
>>> 1) Make i_ddi_intr_get_cpuid() work so in case of NIC having a
single
>>>     interrupt, we can assign it to one of the specified CPUs if the
>>>     CPU list is specified for a NIC.
>>> 2) In case the NIC supports MSI-X interrupt, we should be able to
assign
>>>     the interrupt for the Rx ring to one of the specified CPUs.
>>>
>>> Which also means that the MAC to driver APIs need to be strong
enough to
>>> be able to associate a MSI-X interrupt with a Rx ring (Kais??).
>>>
>>> Cheers,
>>> Sunay
>>>
>>>   
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
> 
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

Sunay Tripathi wrote:
> DDI interface thats H/W neutral is perfectly fine. We can use it
> from userland where dladm can take the specified CPU list and
> bind the MSI-X interrupt to the first CPU in the list from
> userland itself.
>   
ioctl(PCITOOL_DEVICE_SET_INTR, ..) is a userland interface. It is 
currently used by pcitool and can be used by dladm.
> BTW, does pcitool work well enough on Niagara to test with?
>   
Govinda, is this stable on Niagara ?

Nitin
> Thanks,
> Sunay
>
>
> Garrett D''Amore wrote:
>   
>> Yes, I think a DDI interface to this is preferred.
>>
>> I''ve started looking at this.  It looks like DDI_INTROP
functions are
>> needed.  Unfortunately, this portion of the code is really quite
>> platform specific, and it will take some time to figure out how best to
>> handle this.  Separate implementations for i86pc, sun4v, and probably
>> also sun4u, will be required... at minimum.
>>
>> 	-- Garrett
>>
>> On Mon, 2007-07-30 at 19:37 -0700, Nitin Hande wrote:
>>     
>>> Sunay
>>>
>>> For static assignment, currently there is an ioctl interface 
>>> "PCITOOL_DEVICE_SET_INTR" to open a nexus driver and
change interrupt
>>> assignment. This is a userland interface used by pcitool. One can
use
>>> "pcitool -i <i-no> -w <cpu-id>" to statically
change interrupt
>>> assignment. This is the only supported interface. A kernel
interface was
>>> requested by Ethan Solomita (intrd team), but is not present yet. 
>>> Besides from what Govinda (cc''ed on this thread) has
described, opening
>>> a nexus driver and changing the interrupt assignment from within
the
>>> kernel (i.e calling pcitool_set_intr) is not recommended/supported 
>>> .Rather they would like to create a ddi_xxx alike interface for
such
>>> task ....
>>>
>>> Nitin
>>>  
>>>
>>>
>>> Sunay Tripathi wrote:
>>>       
>>>> Garrett/Roamer,
>>>>
>>>> Have you made any progress with interrupt retargetting code.
Basically,
>>>> when a cpu list is specified via ''-C'' option
for a NIC/VNIC or flow,
>>>> we need to do two things:
>>>> 1) Make i_ddi_intr_get_cpuid() work so in case of NIC having a
single
>>>>     interrupt, we can assign it to one of the specified CPUs if
the
>>>>     CPU list is specified for a NIC.
>>>> 2) In case the NIC supports MSI-X interrupt, we should be able
to assign
>>>>     the interrupt for the Rx ring to one of the specified CPUs.
>>>>
>>>> Which also means that the MAC to driver APIs need to be strong
enough to
>>>> be able to associate a MSI-X interrupt with a Rx ring (Kais??).
>>>>
>>>> Cheers,
>>>> Sunay
>>>>
>>>>   
>>>>         
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>       
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>     
>
>
>

> ioctl(PCITOOL_DEVICE_SET_INTR, ..) is a userland interface. It is 
> currently used by pcitool and can be used by dladm.
>> BTW, does pcitool work well enough on Niagara to test with?
>>   
>
> Govinda, is this stable on Niagara ?
yes, it is supported on all PCI and PCIe based SPARC and x86 systems.

Cheers
GOVINDA

Govinda Tatti wrote:
>> ioctl(PCITOOL_DEVICE_SET_INTR, ..) is a userland interface. It is 
>> currently used by pcitool and can be used by dladm.
>>> BTW, does pcitool work well enough on Niagara to test with?
>>>   
>> Govinda, is this stable on Niagara ?
> yes, it is supported on all PCI and PCIe based SPARC and x86 systems.
Perfect! Then can we just use this interface from dladm? Take the
first CPU specified in the CPU list and bind the MSI-X interrupt
there.

Cheers,
Sunay
> 
> Cheers
> GOVINDA
> 
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow