crossbow discuss - Dec 2008 - Crossbow heads-up to device driver writers

As you may already know, the Crossbow project just integrated in  
Nevada build 105. Crossbow provides major features such as MAC  
virtualization (virtual switching, virtual NICs),  traffic flows,  
resource control, better observability, and significant performance  
gains. More more information about Crossbow, see:

http://opensolaris.org/os/project/crossbow

In order to provide these enhancements, the Crossbow project made  
changes to the GLDv3 driver interface which was originally introduced  
as part of Project Nemo. We did not originally commit these GLDv3  
interfaces in anticipation of these changes. Now that Crossbow is in  
the ON gate, we will start the work needed to make GLDv3 a committed  
interface for 3rd parties'' use. This is one of the top item on our  
priority list.

At this point all the GLDv3 network device drivers already in ON have  
been updated to comply to the GLDv3 changes made by Crossbow. We know  
that several 3rd parties have been experimenting with GLDv3. We really  
appreciate the device driver writers out there who started to port  
their drivers to GLDv3 and giving us feedback on the APIs. If you are  
one of these developers, you have to update and recompile your GLDv3  
driver in order to use it on Nevada build 105 and later. Note again  
that this does not apply to drivers that are already in the ON  
consolidation, since those have been already updated as part of the  
Crossbow project.

If you already have a GLDv3 driver, it should be straightforward to  
port to this updated interface. The main changes made to the driver  
API are the following:

- New MAC_CAPAB_RINGS capability: used by drivers to advertise receive  
and transmit rings, polling entry point for receive rings, and  
transmit entry points for transmit rings. This allows for dynamic host- 
driven polling for more efficient receive processing, host-based  
transmit fanout for better scalability, and hardware classification  
for efficient virtualization. This new capability replaces the  
previous blanking capability.

- The multiple MAC address support has been split between two  
capabilities: additional factory MAC address values are exposed via a  
new  MAC_CAPAB_MULTIFACTADDR capability, multiple unicast MAC address  
slots are programmed via the new MAC_CAPAB_RINGS capability (see above)

- Other minor changes have been made to the signature of other entry  
points of the API.

Note again that the GLDv3 driver API is still not a committed  
interface. Minor changes will still be made to the API as we ramp up  
our efforts to make it a committed interface. We don''t anticipate  
these changes to be major, and we will do our best to give a heads-up  
to the OpenSolaris community when making incompatible changes to the  
API as part of this effort.

As part of making GLDv3 a committed interface, we will gradually write  
more documentation (reference guide, man pages, porting guide, sample  
code, etc) to help device driver writers. These documents will be  
posted on opensolaris.org as they become available.

For the 3rd party device driver writers who have already experimented  
with GLDv3 and want to port their drivers, you can look at the drivers  
in ON (for example ixgbe) which have been already ported to the latest  
API. Diffs of the changes made to the drivers by the Crossbow project  
can be seen in the following webrev:

http://cr.opensolaris.org/~ndroux/crossbow/webrev-103.2

The updated driver sources are also now in the ON consolidation  
starting with build 105.

We hope that this information will help the device driver writers out  
there who want to experiment with the latest OpenSolaris networking  
features. We''re looking forward to receiving your feedback.

Also, if you are using the GLDv3 driver interface from your driver  
please let us know (or send me private email). We''re planning to  
create a separate alias for GLDv3 driver API related discussions,  
announcements, etc, and we can directly add you to the alias once it''s
created.

If you have any questions or comments please drop us a note at crossbow-discuss
at opensolaris.org

Thanks,
Nicolas.

-- 
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
nicolas.droux at sun.com - http://blogs.sun.com/droux

BTW, I took a look at the diffs for just one driver (hxge), and I''m
impressed at how much code was removed.  Kudos!

On Dec 5, 2008, at 3:25 PM, Andrew Gallatin wrote:
> Nicolas Droux wrote:
>
>>
>> For the 3rd party device driver writers who have already experimented
>> with GLDv3 and want to port their drivers, you can look at the  
>> drivers
>> in ON (for example ixgbe) which have been already ported to the  
>> latest
>> API. Diffs of the changes made to the drivers by the Crossbow project
>> can be seen in the following webrev:
>>
>> http://cr.opensolaris.org/~ndroux/crossbow/webrev-103.2
>
> Is there any way for a 3rd party to see the links behind the
> numbered things?  Eg, I''m interested in:
>
> 6761011 Code review Phase II. flow control comments
>
> But the link resolves to what seems to be an internal
> Sun server: (http://monaco.sfbay.sun.com/detail.jsp?cr=6761011)
These were bugs used for tracking purpose during the life of the  
project. Not very interesting at this points since they have been all  
either resolved and closed, or moved to other categories of solaris  
(for example kernel/gld) They were visible through opensolaris.org at  
some point, but they are no longer accessible for some reason. I don''t
think it''s worth fixing at this point.

If you''re interested to see the code review comments, see:

http://www.opensolaris.org/jive/thread.jspa?threadID=83402&tstart=0

Thanks,
Nicolas.

>
>
>
> Thanks,
>
> Drew
> _______________________________________________
> driver-discuss mailing list
> driver-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/driver-discuss
-- 
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
nicolas.droux at sun.com - http://blogs.sun.com/droux

Andrew Gallatin writes:
> >> But the link resolves to what seems to be an internal
> >> Sun server: (http://monaco.sfbay.sun.com/detail.jsp?cr=6761011)
[...]
> I was mainly interested in the ixgbe comments linked above, which
> mentioned some discussion about either enabling or disabling
> flow control.  We enable flow control by default (but provide
> a mechanism by which a hung or unresponsive host will not
> lock down the network indefinately), and I was wondering if the
> flow control discussion would be relevant to our NIC.
The content of that one was rather sparse.  Description:

  - EN0003: Add comment to explain how flow control works (Gopi)

  - EN0004: Fix macro and callers, and move definition to ip header file (Gopi)

Evaluation:

  Remove the need for the ill_flow_blocked flag.

-- 
James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

On Dec 6, 2008, at 1:59 PM, Andrew Gallatin wrote:
> Nicolas Droux wrote:
>> On Dec 5, 2008, at 3:25 PM, Andrew Gallatin wrote:
>>> Nicolas Droux wrote:
>>>
>>>>
>>>> For the 3rd party device driver writers who have already  
>>>> experimented
>>>> with GLDv3 and want to port their drivers, you can look at the
>>>> drivers
>>>> in ON (for example ixgbe) which have been already ported to the
>>>> latest
>>>> API. Diffs of the changes made to the drivers by the Crossbow  
>>>> project
>>>> can be seen in the following webrev:
>>>>
>>>> http://cr.opensolaris.org/~ndroux/crossbow/webrev-103.2
>>>
>>> Is there any way for a 3rd party to see the links behind the
>>> numbered things?  Eg, I''m interested in:
>>>
>>> 6761011 Code review Phase II. flow control comments
>>>
>>> But the link resolves to what seems to be an internal
>>> Sun server: (http://monaco.sfbay.sun.com/detail.jsp?cr=6761011)
>> These were bugs used for tracking purpose during the life of the  
>> project. Not very interesting at this points since they have been  
>> all either resolved and closed, or moved to other categories of  
>> solaris (for example kernel/gld) They were visible through  
>> opensolaris.org at some point, but they are no longer accessible  
>> for some reason. I don''t think it''s worth fixing at
this point.
>> If you''re interested to see the code review comments, see:
>> http://www.opensolaris.org/jive/thread.jspa?threadID=83402&tstart=0
>
> I was mainly interested in the ixgbe comments linked above, which
> mentioned some discussion about either enabling or disabling
> flow control.  We enable flow control by default (but provide
> a mechanism by which a hung or unresponsive host will not
> lock down the network indefinately), and I was wondering if the
> flow control discussion would be relevant to our NIC.
Here are the basics of flow control between the stack and the driver:  
the driver notifies the stack that it can no longer send packets by  
returning the unsent packets from its transmit entry points. The  
driver is then responsible to notify the framework that it can again  
send more packets by invoking the framework mac_tx_update() or  
mac_tx_ring_update() entry points. mac_tx_update() is called by  
drivers which don''t support multiple rings, and mac_tx_ring_update()  
is called by drivers which implement the MAC_CAPAB_RINGS capability.

Nicolas.
>
>
> Thanks,
>
> Drew
>
-- 
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
nicolas.droux at sun.com - http://blogs.sun.com/droux