crossbow discuss - Sep 2007 - SPARC Zones and ip-type=exclusive

Hi All,

I have come across an issue on an newly upgraded system with zones
configured.  The reason for the upgrade was to get to the per zone ip stack.

However after I have configured the zone with the ip-type=exclusive
setting in update 4 and removed the address portion of the net
configuration for the zone however it does not configure the physical
interface in the zone. The error message I receive is "WARNING: unable
to hold network interface ''ce101000'' .: Invalid argument"
Looking
through sunsolve there is nothing that I can see there.

If you attempt to configure an address on the interface within the zone
config the verification fails and you can not save the changes to the zone.

If you allow the zone to boot then there is no network interface
available to plumb in the zone.

I have just built a SPARC domain on a 3800 in the lab and have exactly
the same issue trying to assign a physical interface (not a tagged
interface) into the zone.

I have tested this on an x86 system and this functionality is working as
expected allowing me to configure a tagged interface into the zone and
then be able to configure that within the zone including plumbing and
unplumbing it.

Any help would be greatly appreciated as currently we have customer
zones that are not accessible, and I would prefer not to have to
configure them with the shared ip stack as before.

If you would like access to the lab machine then please let me know.

Regards

Tony

On Thu, Sep 13, 2007 at 02:18:37PM +0100, Tony Marshall
wrote:
> However after I have configured the zone with the ip-type=exclusive
> setting in update 4 and removed the address portion of the net
> configuration for the zone however it does not configure the
> physical interface in the zone. The error message I receive is
> "WARNING: unable to hold network interface
''ce101000'' .: Invalid
> argument" Looking through sunsolve there is nothing that I can see
> there.
The NIC that you assign to the zone must be one with a GLDv3
driver. ce is not such.

dme.

David Edmondson wrote:
> On Thu, Sep 13, 2007 at 02:18:37PM +0100, Tony Marshall wrote:
> 
>>However after I have configured the zone with the ip-type=exclusive
>>setting in update 4 and removed the address portion of the net
>>configuration for the zone however it does not configure the
>>physical interface in the zone. The error message I receive is
>>"WARNING: unable to hold network interface
''ce101000'' .: Invalid
>>argument" Looking through sunsolve there is nothing that I can see
>>there.
> 
> 
> The NIC that you assign to the zone must be one with a GLDv3
> driver. ce is not such.
I maintain a list at 
http://www.opensolaris.org/os/project/crossbow/faq/#ipinst_which_nic

Steffen
> 
> dme.
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

Steffen Weiberle wrote:
> David Edmondson wrote:
>   
>> On Thu, Sep 13, 2007 at 02:18:37PM +0100, Tony Marshall wrote:
>>
>>     
>>> However after I have configured the zone with the ip-type=exclusive
>>> setting in update 4 and removed the address portion of the net
>>> configuration for the zone however it does not configure the
>>> physical interface in the zone. The error message I receive is
>>> "WARNING: unable to hold network interface
''ce101000'' .: Invalid
>>> argument" Looking through sunsolve there is nothing that I can
see
>>> there.
>>>       
>> The NIC that you assign to the zone must be one with a GLDv3
>> driver. ce is not such.
>>     
>
> I maintain a list at 
> http://www.opensolaris.org/os/project/crossbow/faq/#ipinst_which_nic
>   
Your list is stale.. :-)

The following should be added to your "they support IP instances" (and
all other standard GLDv3 features), as of b73.

eri
hme
qfe
iprb
rtls
afe
mxfe
dmfe

That leaves:

ce
ge
dnet
ixge ?
elx
pcelx
spwr

I may have a look at ge, but I don''t have any other fiber based NICs 
(and I don''t even have a ge board), so it might be hard to do.  (Its 
pretty much the same thing that I did for eri, since they both have the 
same MAC controller and only differ in the transceiver.

dnet could easily be done, and would be, if I had any hardware that I 
could test with it.  (I''d also update it for sparc and amd64 at the
same
time.)  If anyone wants to "donate" (especially one of the 4 port
cards)
a card to me, I might have a look at it.

elx and pcelx can''t support full size VLAN frames, so until clearview 
adds support for GLDv3 devices that don''t support VLANs, I
can''t convert
them.

I''ve not looked at spwr.  I''ve never seen one of them.

    -- Garrett
> Steffen
>
>   
>> dme.
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>     
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>

Hi Garrett,

Excellent news. The ge interface is built into the v880 and v890 
motherboards and is fibre!

Thanks,
Steffen

Garrett D''Amore wrote:
> Steffen Weiberle wrote:
> 
>> David Edmondson wrote:
>>  
>>
>>> On Thu, Sep 13, 2007 at 02:18:37PM +0100, Tony Marshall wrote:
>>>
>>>    
>>>
>>>> However after I have configured the zone with the
ip-type=exclusive
>>>> setting in update 4 and removed the address portion of the net
>>>> configuration for the zone however it does not configure the
>>>> physical interface in the zone. The error message I receive is
>>>> "WARNING: unable to hold network interface
''ce101000'' .: Invalid
>>>> argument" Looking through sunsolve there is nothing that I
can see
>>>> there.
>>>>       
>>>
>>> The NIC that you assign to the zone must be one with a GLDv3
>>> driver. ce is not such.
>>>     
>>
>>
>> I maintain a list at 
>> http://www.opensolaris.org/os/project/crossbow/faq/#ipinst_which_nic
>>   
> 
> 
> Your list is stale.. :-)
> 
> The following should be added to your "they support IP instances"
(and
> all other standard GLDv3 features), as of b73.
> 
> eri
> hme
> qfe
> iprb
> rtls
> afe
> mxfe
> dmfe
> 
> That leaves:
> 
> ce
> ge
> dnet
> ixge ?
> elx
> pcelx
> spwr
> 
> I may have a look at ge, but I don''t have any other fiber based
NICs
> (and I don''t even have a ge board), so it might be hard to do. 
(Its
> pretty much the same thing that I did for eri, since they both have the 
> same MAC controller and only differ in the transceiver.
> 
> dnet could easily be done, and would be, if I had any hardware that I 
> could test with it.  (I''d also update it for sparc and amd64 at
the same
> time.)  If anyone wants to "donate" (especially one of the 4 port
cards)
> a card to me, I might have a look at it.
> 
> elx and pcelx can''t support full size VLAN frames, so until
clearview
> adds support for GLDv3 devices that don''t support VLANs, I
can''t convert
> them.
> 
> I''ve not looked at spwr.  I''ve never seen one of them.
> 
>     -- Garrett
> 
>> Steffen
>>
>>  
>>
>>> dme.
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>     
>>
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>   
> 
>

hi
what is the timeline for ce to be support in crossbow?
TIA

David Edmondson wrote:
> On Thu, Sep 13, 2007 at 02:18:37PM +0100, Tony Marshall wrote:
>   
>> However after I have configured the zone with the ip-type=exclusive
>> setting in update 4 and removed the address portion of the net
>> configuration for the zone however it does not configure the
>> physical interface in the zone. The error message I receive is
>> "WARNING: unable to hold network interface
''ce101000'' .: Invalid
>> argument" Looking through sunsolve there is nothing that I can see
>> there.
>>     
>
> The NIC that you assign to the zone must be one with a GLDv3
> driver. ce is not such.
>
> dme.
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>

LaoTsao (Dr. Tsao) wrote:
> hi
> what is the timeline for ce to be support in crossbow?
>   
there is no timeline.

ce is not a gldv3 device.  it would first have to be updated to be one.  
then there would be the effort to convert it to crossbow.

the owners of the ce driver code have not been willing to make any 
forward looking statements about ce and gldv3.  it isn''t even clear to 
me at this point that they will ever do it.  i''m still trying to get a 
word one or the other from them... if they aren''t doing a ce gldv3
port,
then i may revive my shelved effort.  (adding ce support for crossbow 
would be fairly easy once the much harder task of updating it for gldv3 
is done.  however, updating it for gldv3 will have a negative 
performance impact under some workloads unless we can also get soft-lso 
integrated.   i believe roamer lu is working on soft-lso.)

    -- garrett (who has a dodgy shift key)
> TIA
>
> David Edmondson wrote:
>   
>> On Thu, Sep 13, 2007 at 02:18:37PM +0100, Tony Marshall wrote:
>>   
>>     
>>> However after I have configured the zone with the ip-type=exclusive
>>> setting in update 4 and removed the address portion of the net
>>> configuration for the zone however it does not configure the
>>> physical interface in the zone. The error message I receive is
>>> "WARNING: unable to hold network interface
''ce101000'' .: Invalid
>>> argument" Looking through sunsolve there is nothing that I can
see
>>> there.
>>>     
>>>       
>> The NIC that you assign to the zone must be one with a GLDv3
>> driver. ce is not such.
>>
>> dme.
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>   
>>     
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>

On 9/18/07, Garrett D''Amore <Garrett.Damore at sun.com>
wrote:
> LaoTsao (Dr. Tsao) wrote:
> > hi
> > what is the timeline for ce to be support in crossbow?
> >
>
> there is no timeline.
>
> ce is not a gldv3 device.  it would first have to be updated to be one.
> then there would be the effort to convert it to crossbow.
>
> the owners of the ce driver code have not been willing to make any
> forward looking statements about ce and gldv3.  it isn''t even
clear to
> me at this point that they will ever do it.  i''m still trying to
get a
> word one or the other from them... if they aren''t doing a ce gldv3
port,
> then i may revive my shelved effort.  (adding ce support for crossbow
> would be fairly easy once the much harder task of updating it for gldv3
> is done.  however, updating it for gldv3 will have a negative
> performance impact under some workloads unless we can also get soft-lso
> integrated.   i believe roamer lu is working on soft-lso.)
Hi Garrett,

What exactly is soft-lso? Is there a PSARC case that describes this feature?

Thanks,
- Ryan
-- 
UNIX Administrator
http://prefetch.net

hi
there are many many system that use ce
so it will be nice that ce been updated
regards

Garrett D''Amore wrote:
> LaoTsao (Dr. Tsao) wrote:
>   
>> hi
>> what is the timeline for ce to be support in crossbow?
>>   
>>     
>
> there is no timeline.
>
> ce is not a gldv3 device.  it would first have to be updated to be one.  
> then there would be the effort to convert it to crossbow.
>
> the owners of the ce driver code have not been willing to make any 
> forward looking statements about ce and gldv3.  it isn''t even
clear to
> me at this point that they will ever do it.  i''m still trying to
get a
> word one or the other from them... if they aren''t doing a ce gldv3
port,
> then i may revive my shelved effort.  (adding ce support for crossbow 
> would be fairly easy once the much harder task of updating it for gldv3 
> is done.  however, updating it for gldv3 will have a negative 
> performance impact under some workloads unless we can also get soft-lso 
> integrated.   i believe roamer lu is working on soft-lso.)
>
>     -- garrett (who has a dodgy shift key)
>   
>> TIA
>>
>> David Edmondson wrote:
>>   
>>     
>>> On Thu, Sep 13, 2007 at 02:18:37PM +0100, Tony Marshall wrote:
>>>   
>>>     
>>>       
>>>> However after I have configured the zone with the
ip-type=exclusive
>>>> setting in update 4 and removed the address portion of the net
>>>> configuration for the zone however it does not configure the
>>>> physical interface in the zone. The error message I receive is
>>>> "WARNING: unable to hold network interface
''ce101000'' .: Invalid
>>>> argument" Looking through sunsolve there is nothing that I
can see
>>>> there.
>>>>     
>>>>       
>>>>         
>>> The NIC that you assign to the zone must be one with a GLDv3
>>> driver. ce is not such.
>>>
>>> dme.
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>   
>>>     
>>>       
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>   
>>     
>
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>

On 18/09/2007, Matty <matty91 at gmail.com> wrote:
>
> What exactly is soft-lso? Is there a PSARC case that describes this
feature?
>
One assumes this is a modification to the GLDv3 mac module to disect
an LSO packet into MTU size packets such that the driver/hardware can
be LSO-unaware whilst still giving some proportion of the LSO benefit
by allowing large packets to pass down the stack from TCP?

One question would be how the benefit of doing a single DMA-map for
the LSO payload could be done if the packet is already segmented.

  Paul

-- 
Paul Durrant
http://www.linkedin.com/in/pdurrant

Paul Durrant wrote:
> On 18/09/2007, Matty <matty91 at gmail.com> wrote:
>   
>> What exactly is soft-lso? Is there a PSARC case that describes this
feature?
>>
>>     
>
> One assumes this is a modification to the GLDv3 mac module to disect
> an LSO packet into MTU size packets such that the driver/hardware can
> be LSO-unaware whilst still giving some proportion of the LSO benefit
> by allowing large packets to pass down the stack from TCP?
>
> One question would be how the benefit of doing a single DMA-map for
> the LSO payload could be done if the packet is already segmented.
>
>   Paul
>
>   
One might imagine that soft LSO works differently.   Imagine soft-lso as 
a subroutine, which, given a large TCP segment, and the MTU size, builds 
a series of header mblks, and maps the data segment in one operation 
(perhaps making use of multiple DMA cookies, etc. to deal with page 
boundaries, etc.) giving the appearance of LSO to the upper layers, 
while allowing a NIC that only has the simple ability to use more than 1 
descriptor per ethernet frame to use a single DMA setup operation.

This is just conjectural on my part... I''ve not looked at
Roamer''s
actual implementation....

    -- Garrett

On Tue, Sep 25, 2007 at 07:57:02AM -0700, Garrett D''Amore
wrote:
> One might imagine that soft LSO works differently.   Imagine soft-lso as a 
> subroutine, which, given a large TCP segment, and the MTU size, builds a 
> series of header mblks, and maps the data segment in one operation (perhaps
> making use of multiple DMA cookies, etc. to deal with page boundaries, 
> etc.) giving the appearance of LSO to the upper layers, while allowing a 
> NIC that only has the simple ability to use more than 1 descriptor per 
> ethernet frame to use a single DMA setup operation.
The approach you describe would require that a mac driver be updated
to support it. That would be bad. Keeping things in the common layer
is good.

dme.

David Edmondson wrote:
> On Tue, Sep 25, 2007 at 07:57:02AM -0700, Garrett D''Amore wrote:
>   
>> One might imagine that soft LSO works differently.   Imagine soft-lso
as a
>> subroutine, which, given a large TCP segment, and the MTU size, builds
a
>> series of header mblks, and maps the data segment in one operation
(perhaps
>> making use of multiple DMA cookies, etc. to deal with page boundaries, 
>> etc.) giving the appearance of LSO to the upper layers, while allowing
a
>> NIC that only has the simple ability to use more than 1 descriptor per 
>> ethernet frame to use a single DMA setup operation.
>>     
>
> The approach you describe would require that a mac driver be updated
> to support it. That would be bad. Keeping things in the common layer
> is good.
>   
Soft-LSO will require this anyway.  If you don''t push anything down to 
the Mac layer or below, then what advantage is there to LSO versus 
normal IP fragmentation?

You can do *most* of the work in the common layer, by having the mac 
layer "upcall" to common code functionality.  I think that is a good 
compromise.

    -- Garrett

On Tue, Sep 25, 2007 at 08:41:25AM -0700, Garrett D''Amore
wrote:
> David Edmondson wrote:
>> On Tue, Sep 25, 2007 at 07:57:02AM -0700, Garrett D''Amore
wrote:
>>   
>>> One might imagine that soft LSO works differently.   Imagine
soft-lso as
>>> a subroutine, which, given a large TCP segment, and the MTU size,
builds
>>> a series of header mblks, and maps the data segment in one
operation
>>> (perhaps making use of multiple DMA cookies, etc. to deal with page
>>> boundaries, etc.) giving the appearance of LSO to the upper layers,
while
>>> allowing a NIC that only has the simple ability to use more than 1 
>>> descriptor per ethernet frame to use a single DMA setup operation.
>>>     
>>
>> The approach you describe would require that a mac driver be updated
>> to support it. That would be bad. Keeping things in the common layer
>> is good.
>>   
>
> Soft-LSO will require this anyway.
Why? I can accept a large packet in a MAC layer transmit function,
chop it up and then present an mblk chain to the driver in a single
call to its'' transmit routine.

dme.

On 25/09/2007, David Edmondson <dme at sun.com>
wrote:
>
> Why? I can accept a large packet in a MAC layer transmit function,
> chop it up and then present an mblk chain to the driver in a single
> call to its'' transmit routine.
>
You lose the single DMA map this way, which is a good portion of the
advantage of LSO (given how slow the DMA map operation appears to be
on many platforms).

  Paul

-- 
Paul Durrant
http://www.linkedin.com/in/pdurrant

Paul Durrant wrote:
> On 25/09/2007, David Edmondson <dme at sun.com> wrote:
>   
>> Why? I can accept a large packet in a MAC layer transmit function,
>> chop it up and then present an mblk chain to the driver in a single
>> call to its'' transmit routine.
>>
>>     
>
> You lose the single DMA map this way, which is a good portion of the
> advantage of LSO (given how slow the DMA map operation appears to be
> on many platforms).
>   
Right.  If you push it too far up the stack, then it offers no real 
advantage over just letting IP fragment the frames normally. :-)

The meaty part of this work can be in common code that is called by mac 
layer drivers, though.

    -- Garrett
>   Paul
>
>

Paul Durrant wrote:
> On 25/09/2007, David Edmondson <dme at sun.com> wrote:
>> Why? I can accept a large packet in a MAC layer transmit function,
>> chop it up and then present an mblk chain to the driver in a single
>> call to its'' transmit routine.
>>
> 
> You lose the single DMA map this way, which is a good portion of the
> advantage of LSO (given how slow the DMA map operation appears to be
> on many platforms).
> 
>   Paul
> 
So I haven''t looked at soft-LSO code yet but the way to do this
would be along the lines of what you, Dave and Garrett are saying.

The MAC layer is implementing a pseudo H/W layer anyway as part of
Crossbow. The idea is that upper layer can always assume that
features like classificaton, Rx/Tx rings, soft-lso, etc are
supported. The upper layer always have a pointer to a Tx
routine in MAC layer. The driver on the other hand can choose
to be native LSO, soft-lso aware or none. It needs to register
its capability as per the new document Kais and Roamer are
preparing and corresponding functions. In most cases, the MAC
transmit routine can just be a tail call to driver routine or
a tail call to mac_soft_lso() in case driver doesn''t support
it. Some advantages of a function pointer driven approach ....

Cheers,
Sunay

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Garrett D''Amore wrote:
<blockquote cite="mid46F93823.9050802@damore.org"
type="cite">
  <pre wrap="">Paul Durrant wrote:
  </pre>
  <blockquote type="cite">
    <pre wrap="">On 25/09/2007, David Edmondson <a
class="moz-txt-link-rfc2396E"
href="mailto:dme@sun.com"><dme@sun.com></a> wrote:
  
    </pre>
    <blockquote type="cite">
      <pre wrap="">Why? I can accept a large packet in a MAC
layer transmit function,
chop it up and then present an mblk chain to the driver in a single
call to its'' transmit routine.

    
      </pre>
    </blockquote>
    <pre wrap="">You lose the single DMA map this way, which is
a good portion of the
advantage of LSO (given how slow the DMA map operation appears to be
on many platforms).
  
    </pre>
  </blockquote>
  <pre wrap=""><!---->
Right.  If you push it too far up the stack, then it offers no real 
advantage over just letting IP fragment the frames normally. :-)

The meaty part of this work can be in common code that is called by mac 
layer drivers, though.
  </pre>
</blockquote>
<br>
I understand the obvious advantages of doing soft-lso in MAC layer such
as same code supporting all non-hw-lso NICs,
designing/implementing/testing&nbsp; it once etc ...<br>
On the other hand, doing it in the NIC driver has some advantages due
to the fact it has&nbsp; intimate knowledge of the HW among other
things.&nbsp;
This is from the experience we gained from implementing soft-lso in
Neptune Linux driver.<br>
<br>
1) Reduced DMA mapping. Assuming that the LSO packet is presented as a
chain of large mblks (or pages), need to do DMA mapping per large mblk.
You can then chop the the mapped data buffer. Even in&nbsp; Linux
X86&nbsp; where
a page contains 2+ MTU sized packets this would cut down the number of
DMA mappings significantly.&nbsp; For SPARC, the reduction would be even
bigger.<br>
<br>
2) Manufacture the headers in the driver. Could exploit specific HW&nbsp;
features. <br>
<br>
3) Separate the operations which require holding locks from those
don''t. Manufacturing the headers, DMA mapping, calculating pseudo
checksum&nbsp; and IP header checksum (for the HW which do not support
complete checksum), twiking&nbsp; IP id and sequence numbers can be done
lockless. Only the actual posting of the descriptors would require
locking.<br>
<br>
<br>
4) Simpler Interface. As far as the upper layer is concerned, there
would be no distinction between HW and SW LSO. The driver can advertise
as LSO capable.&nbsp; At the driver level, the&nbsp;&nbsp;
difference&nbsp; would be how
LSO packet is processed/described/posted&nbsp; to the HW and that is HW
specific anyway.<br>
<br>
Regards,<br>
Matheos<br>
<br>
<br>
<blockquote cite="mid46F93823.9050802@damore.org"
type="cite">
  <pre wrap="">
    -- Garrett
  </pre>
  <blockquote type="cite">
    <pre wrap="">  Paul

  
    </pre>
  </blockquote>
  <pre wrap=""><!---->
_______________________________________________
crossbow-discuss mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:crossbow-discuss@opensolaris.org">crossbow-discuss@opensolaris.org</a>
<a class="moz-txt-link-freetext"
href="http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss">http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss</a>
  </pre>
</blockquote>
<br>
</body>
</html>

Matheos Worku wrote:
> Garrett D''Amore wrote:
>> Paul Durrant wrote:
>>   
>>> On 25/09/2007, David Edmondson <dme at sun.com> wrote:
>>>   
>>>     
>>>> Why? I can accept a large packet in a MAC layer transmit
function,
>>>> chop it up and then present an mblk chain to the driver in a
single
>>>> call to its'' transmit routine.
>>>>
>>>>     
>>>>       
>>> You lose the single DMA map this way, which is a good portion of
the
>>> advantage of LSO (given how slow the DMA map operation appears to
be
>>> on many platforms).
>>>   
>>>     
>>
>> Right.  If you push it too far up the stack, then it offers no real 
>> advantage over just letting IP fragment the frames normally. :-)
>>
>> The meaty part of this work can be in common code that is called by mac
>> layer drivers, though.
>>   
> 
> I understand the obvious advantages of doing soft-lso in MAC layer such 
> as same code supporting all non-hw-lso NICs, 
> designing/implementing/testing  it once etc ...
> On the other hand, doing it in the NIC driver has some advantages due to 
> the fact it has  intimate knowledge of the HW among other things.  This 
> is from the experience we gained from implementing soft-lso in Neptune 
> Linux driver.
> 
> 1) Reduced DMA mapping. Assuming that the LSO packet is presented as a 
> chain of large mblks (or pages), need to do DMA mapping per large mblk. 
> You can then chop the the mapped data buffer. Even in  Linux X86  where 
> a page contains 2+ MTU sized packets this would cut down the number of 
> DMA mappings significantly.  For SPARC, the reduction would be even bigger.
> 
> 2) Manufacture the headers in the driver. Could exploit specific HW  
> features.
> 
> 3) Separate the operations which require holding locks from those
don''t.
> Manufacturing the headers, DMA mapping, calculating pseudo checksum  and 
> IP header checksum (for the HW which do not support complete checksum), 
> twiking  IP id and sequence numbers can be done lockless. Only the 
> actual posting of the descriptors would require locking.
> 
> 
> 4) Simpler Interface. As far as the upper layer is concerned, there 
> would be no distinction between HW and SW LSO. The driver can advertise 
> as LSO capable.  At the driver level, the   difference  would be how LSO 
> packet is processed/described/posted  to the HW and that is HW specific 
> anyway.
I think you missed my earlier reply. Yes, pushing it down for driver
that support it is good but the functionality does need to be
offered at a common layer. Don''t want to see code in TCP or
IP dealing with LSO or not to LSO. They should just do it based
on advertized MTU. Here is the relevant part from my earlier email

"
The MAC layer is implementing a pseudo H/W layer anyway as part of
Crossbow. The idea is that upper layer can always assume that
features like classificaton, Rx/Tx rings, soft-lso, etc are
supported. The upper layer always have a pointer to a Tx
routine in MAC layer. The driver on the other hand can choose
to be native LSO, soft-lso aware or none. It needs to register
its capability as per the new document Kais and Roamer are
preparing and corresponding functions. In most cases, the MAC
transmit routine can just be a tail call to driver routine or
a tail call to mac_soft_lso() in case driver doesn''t support
it. Some advantages of a function pointer driven approach ....
"

Cheers,
Sunay

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

Sunay,

Having seen you earlier e-mail and now this one. I think
an appropriate way to simplify this is have the mac layer
provide two things:

1) Provide true soft LSO in the mac layer where the H/W driver
     would receive multiple packets to map/unmap and transmit.
     The mac layer can be a client of the library I propose below in #2.

2) Provide a set of mac library routines that the driver could use
     to do the packet chopping that soft LSO requires while the
     H/W performs the single DMA mapping.  Also, the H/W driver
     would advertize that it is LSO capable.

The end result is that you would have the best of both worlds
in having the mac layer provide LSO for all devices and
specific drivers provide it and offering one dma mapping.

Sound reasonable?

Michael

On Sep 25, 2007, at 2:09 PM, Sunay Tripathi wrote:
> Matheos Worku wrote:
>> Garrett D''Amore wrote:
>>> Paul Durrant wrote:
>>>
>>>> On 25/09/2007, David Edmondson <dme at sun.com> wrote:
>>>>
>>>>
>>>>> Why? I can accept a large packet in a MAC layer transmit
function,
>>>>> chop it up and then present an mblk chain to the driver in
a
>>>>> single
>>>>> call to its'' transmit routine.
>>>>>
>>>>>
>>>>>
>>>> You lose the single DMA map this way, which is a good portion
of
>>>> the
>>>> advantage of LSO (given how slow the DMA map operation appears
>>>> to be
>>>> on many platforms).
>>>>
>>>>
>>>
>>> Right.  If you push it too far up the stack, then it offers no real
>>> advantage over just letting IP fragment the frames normally. :-)
>>>
>>> The meaty part of this work can be in common code that is called  
>>> by mac
>>> layer drivers, though.
>>>
>>
>> I understand the obvious advantages of doing soft-lso in MAC layer  
>> such
>> as same code supporting all non-hw-lso NICs,
>> designing/implementing/testing  it once etc ...
>> On the other hand, doing it in the NIC driver has some advantages  
>> due to
>> the fact it has  intimate knowledge of the HW among other things.   
>> This
>> is from the experience we gained from implementing soft-lso in  
>> Neptune
>> Linux driver.
>>
>> 1) Reduced DMA mapping. Assuming that the LSO packet is presented  
>> as a
>> chain of large mblks (or pages), need to do DMA mapping per large  
>> mblk.
>> You can then chop the the mapped data buffer. Even in  Linux X86   
>> where
>> a page contains 2+ MTU sized packets this would cut down the  
>> number of
>> DMA mappings significantly.  For SPARC, the reduction would be  
>> even bigger.
>>
>> 2) Manufacture the headers in the driver. Could exploit specific HW
>> features.
>>
>> 3) Separate the operations which require holding locks from those  
>> don''t.
>> Manufacturing the headers, DMA mapping, calculating pseudo  
>> checksum  and
>> IP header checksum (for the HW which do not support complete  
>> checksum),
>> twiking  IP id and sequence numbers can be done lockless. Only the
>> actual posting of the descriptors would require locking.
>>
>>
>> 4) Simpler Interface. As far as the upper layer is concerned, there
>> would be no distinction between HW and SW LSO. The driver can  
>> advertise
>> as LSO capable.  At the driver level, the   difference  would be  
>> how LSO
>> packet is processed/described/posted  to the HW and that is HW  
>> specific
>> anyway.
>
> I think you missed my earlier reply. Yes, pushing it down for driver
> that support it is good but the functionality does need to be
> offered at a common layer. Don''t want to see code in TCP or
> IP dealing with LSO or not to LSO. They should just do it based
> on advertized MTU. Here is the relevant part from my earlier email
>
> "
> The MAC layer is implementing a pseudo H/W layer anyway as part of
> Crossbow. The idea is that upper layer can always assume that
> features like classificaton, Rx/Tx rings, soft-lso, etc are
> supported. The upper layer always have a pointer to a Tx
> routine in MAC layer. The driver on the other hand can choose
> to be native LSO, soft-lso aware or none. It needs to register
> its capability as per the new document Kais and Roamer are
> preparing and corresponding functions. In most cases, the MAC
> transmit routine can just be a tail call to driver routine or
> a tail call to mac_soft_lso() in case driver doesn''t support
> it. Some advantages of a function pointer driven approach ....
> "
>
> Cheers,
> Sunay
>
> -- 
> Sunay Tripathi
> Distinguished Engineer
> Solaris Core Operating System
> Sun MicroSystems Inc.
>
> Solaris Networking:     http://www.opensolaris.org/os/community/ 
> networking
> Project Crossbow:       http://www.opensolaris.org/os/project/crossbow
>
>
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

Mike,

Sounds very reasonable. Need to check with Roamer and Thiru as
to where they actually are in the implementation.

Cheers,
sunay

Michael Speer wrote:
> Sunay,
> 
> Having seen you earlier e-mail and now this one. I think
> an appropriate way to simplify this is have the mac layer
> provide two things:
> 
> 1) Provide true soft LSO in the mac layer where the H/W driver
>     would receive multiple packets to map/unmap and transmit.
>     The mac layer can be a client of the library I propose below in #2.
> 
> 2) Provide a set of mac library routines that the driver could use
>     to do the packet chopping that soft LSO requires while the
>     H/W performs the single DMA mapping.  Also, the H/W driver
>     would advertize that it is LSO capable.
> 
> The end result is that you would have the best of both worlds
> in having the mac layer provide LSO for all devices and
> specific drivers provide it and offering one dma mapping.
> 
> Sound reasonable?
> 
> Michael
> 
> On Sep 25, 2007, at 2:09 PM, Sunay Tripathi wrote:
> 
>> Matheos Worku wrote:
>>> Garrett D''Amore wrote:
>>>> Paul Durrant wrote:
>>>>
>>>>> On 25/09/2007, David Edmondson <dme at sun.com>
wrote:
>>>>>
>>>>>
>>>>>> Why? I can accept a large packet in a MAC layer
transmit function,
>>>>>> chop it up and then present an mblk chain to the driver
in a single
>>>>>> call to its'' transmit routine.
>>>>>>
>>>>>>
>>>>>>
>>>>> You lose the single DMA map this way, which is a good
portion of the
>>>>> advantage of LSO (given how slow the DMA map operation
appears to be
>>>>> on many platforms).
>>>>>
>>>>>
>>>>
>>>> Right.  If you push it too far up the stack, then it offers no
real
>>>> advantage over just letting IP fragment the frames normally.
:-)
>>>>
>>>> The meaty part of this work can be in common code that is
called by mac
>>>> layer drivers, though.
>>>>
>>>
>>> I understand the obvious advantages of doing soft-lso in MAC layer
such
>>> as same code supporting all non-hw-lso NICs,
>>> designing/implementing/testing  it once etc ...
>>> On the other hand, doing it in the NIC driver has some advantages
due to
>>> the fact it has  intimate knowledge of the HW among other things. 
This
>>> is from the experience we gained from implementing soft-lso in
Neptune
>>> Linux driver.
>>>
>>> 1) Reduced DMA mapping. Assuming that the LSO packet is presented
as a
>>> chain of large mblks (or pages), need to do DMA mapping per large
mblk.
>>> You can then chop the the mapped data buffer. Even in  Linux X86 
where
>>> a page contains 2+ MTU sized packets this would cut down the number
of
>>> DMA mappings significantly.  For SPARC, the reduction would be even
>>> bigger.
>>>
>>> 2) Manufacture the headers in the driver. Could exploit specific HW
>>> features.
>>>
>>> 3) Separate the operations which require holding locks from those
don''t.
>>> Manufacturing the headers, DMA mapping, calculating pseudo checksum
and
>>> IP header checksum (for the HW which do not support complete
checksum),
>>> twiking  IP id and sequence numbers can be done lockless. Only the
>>> actual posting of the descriptors would require locking.
>>>
>>>
>>> 4) Simpler Interface. As far as the upper layer is concerned, there
>>> would be no distinction between HW and SW LSO. The driver can
advertise
>>> as LSO capable.  At the driver level, the   difference  would be
how LSO
>>> packet is processed/described/posted  to the HW and that is HW
specific
>>> anyway.
>>
>> I think you missed my earlier reply. Yes, pushing it down for driver
>> that support it is good but the functionality does need to be
>> offered at a common layer. Don''t want to see code in TCP or
>> IP dealing with LSO or not to LSO. They should just do it based
>> on advertized MTU. Here is the relevant part from my earlier email
>>
>> "
>> The MAC layer is implementing a pseudo H/W layer anyway as part of
>> Crossbow. The idea is that upper layer can always assume that
>> features like classificaton, Rx/Tx rings, soft-lso, etc are
>> supported. The upper layer always have a pointer to a Tx
>> routine in MAC layer. The driver on the other hand can choose
>> to be native LSO, soft-lso aware or none. It needs to register
>> its capability as per the new document Kais and Roamer are
>> preparing and corresponding functions. In most cases, the MAC
>> transmit routine can just be a tail call to driver routine or
>> a tail call to mac_soft_lso() in case driver doesn''t support
>> it. Some advantages of a function pointer driven approach ....
>> "
>>
>> Cheers,
>> Sunay
>>
>> -- 
>> Sunay Tripathi
>> Distinguished Engineer
>> Solaris Core Operating System
>> Sun MicroSystems Inc.
>>
>> Solaris Networking:     
>> http://www.opensolaris.org/os/community/networking
>> Project Crossbow:       http://www.opensolaris.org/os/project/crossbow
>>
>>
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

Garrett D''Amore wrote:
> The following should be added to your "they support IP instances"
(and
> all other standard GLDv3 features), as of b73.
> 
> eri
> hme
> qfe
> iprb
> rtls
> afe
> mxfe
> dmfe
> 
> That leaves:
> 
> ce
> ge
> dnet
> ixge ?
> elx
> pcelx
> spwr
And of course 3rd party monolithic and GLDv2 drivers.

-Seb