crossbow discuss - Feb 2009 - UV fastpath after Crossbow

Hi the Crossbow team,

I am testing my UV fastpath bits and I found several issues that need your 
suggestion:

1. dladm show-usage won''t work if UV fastpath is enabled.

Since the link usage data are collected based on the statistics of the 
mac_client_impl_t, and UV fastpath skips the GLDv3 processing, that
won''t
be available.

I don''t see a solution for this except to document this restriction.

2. VLAN kstats won''t work.

If we enable UV fastpath over VLANs, packets would go directly between the 
lower VLAN stream and the IP stream over softmac, the VNIC (vlan) kstats 
won''t be updated.

This could be work around by changing vnic_m_stat() function, to add up 
the legacy VLAN kstat (e.g., kstat of ce1000).

3. Performance of VLAN is not optimal

In the current Crossbow code, once a VNIC (vlan) is created, the 
underlying mac is started (by the code path 
vnic_dev_create()->vnic_unicast_add()->i_mac_unicast_add()->mac_start())
even when there is no user of this VLAN. As a result, all packets will be 
passed by mac and will be eventually dropped in MAC.

This is not a preferred behavior and causes performance issues for VLANs 
over legacy devices. Even we try to use the fast-path for a VLAN, there 
would be two lower stream over the legacy device: the shared lower stream 
and dedicated IP lower stream over the VLAN, so each packets will be 
copied once and passed to two streams, and the copy on the shared lower 
stream will soon be dropped in the MAC module.

The fix to this is to not call vnic_unicast_add() for VLANs, and delay 
this unicast_add() when mac_unicast_primary_add() is called for this VLAN. 
Is this possible?

Expecting your early reply.

Thanks
- Cathy

Hi cathy,

I''m still thinking of a few things you mentioned in your review
comments
so I haven''t responded to that yet.

but regarding vlans, I got the impression from your previous mail that
you decided to disable fastpath for vlans because the code will be simpler.
did you change your mind?

the changes you mentioned below sound like hacks and might not be easy
to do (especially, moving unicast_add). what''s worse is that this
benefits
no one but vlans on legacy devices. you mentioned all these changes could
be undone when legacy fastpath is removed. but I doubt this can happen
anytime soon nor have I heard of any plans to do so.

because of these reasons, I would say it''s better to *not* enable
fastpath for vlans. In my opinion, maintainability of the framework
far outweigh performance benefits to a few legacy devices.

thanks
eric


On Thu, Feb 12, 2009 at 03:25:52PM -0800, Cathy Zhou
wrote:
> Hi the Crossbow team,
> 
> I am testing my UV fastpath bits and I found several issues that need your 
> suggestion:
> 
> 1. dladm show-usage won''t work if UV fastpath is enabled.
> 
> Since the link usage data are collected based on the statistics of the 
> mac_client_impl_t, and UV fastpath skips the GLDv3 processing, that
won''t
> be available.
> 
> I don''t see a solution for this except to document this
restriction.
> 
> 2. VLAN kstats won''t work.
> 
> If we enable UV fastpath over VLANs, packets would go directly between the 
> lower VLAN stream and the IP stream over softmac, the VNIC (vlan) kstats 
> won''t be updated.
> 
> This could be work around by changing vnic_m_stat() function, to add up 
> the legacy VLAN kstat (e.g., kstat of ce1000).
> 
> 3. Performance of VLAN is not optimal
> 
> In the current Crossbow code, once a VNIC (vlan) is created, the 
> underlying mac is started (by the code path 
>
vnic_dev_create()->vnic_unicast_add()->i_mac_unicast_add()->mac_start())
> even when there is no user of this VLAN. As a result, all packets will be 
> passed by mac and will be eventually dropped in MAC.
> 
> This is not a preferred behavior and causes performance issues for VLANs 
> over legacy devices. Even we try to use the fast-path for a VLAN, there 
> would be two lower stream over the legacy device: the shared lower stream 
> and dedicated IP lower stream over the VLAN, so each packets will be 
> copied once and passed to two streams, and the copy on the shared lower 
> stream will soon be dropped in the MAC module.
> 
> The fix to this is to not call vnic_unicast_add() for VLANs, and delay 
> this unicast_add() when mac_unicast_primary_add() is called for this VLAN. 
> Is this possible?
> 
> Expecting your early reply.
> 
> Thanks
> - Cathy
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

> Hi cathy,
> 
> I''m still thinking of a few things you mentioned in your review
comments
> so I haven''t responded to that yet.
> 
> but regarding vlans, I got the impression from your previous mail that
> you decided to disable fastpath for vlans because the code will be simpler.
> did you change your mind?
> 
I am still struggling between whether or not to support fast-path for 
VLANs. That is exactly the reason I''d like to understand all the issues
involved and make decisions based on that.
> the changes you mentioned below sound like hacks and might not be easy
> to do (especially, moving unicast_add). 
I don''t agree this is a hack that is only needed for VLAN fastpath over
legacy devices. It is just wrong to pass packets to the stack then get 
dropped as long as we have VLAN(VNIC)s created, even those VLAN/VNICs are 
not used.
> what''s worse is that this benefits
> no one but vlans on legacy devices. you mentioned all these changes could
> be undone when legacy fastpath is removed. but I doubt this can happen
> anytime soon nor have I heard of any plans to do so.
> 
I agree. That is tradeoff. I personally also don''t want to make the
code
more complexer, but I am not convinced VLAN performance over legacy 
drivers does not matter either. That is why I have this discussion and to 
see whether there is any other option to simplify the code and reach the 
performance goal.
> because of these reasons, I would say it''s better to *not* enable
> fastpath for vlans. In my opinion, maintainability of the framework
> far outweigh performance benefits to a few legacy devices.
> 
Thanks for starting the discussion.
- Cathy
> thanks
> eric
> 
> 
> On Thu, Feb 12, 2009 at 03:25:52PM -0800, Cathy Zhou wrote:
>> Hi the Crossbow team,
>>
>> I am testing my UV fastpath bits and I found several issues that need
your
>> suggestion:
>>
>> 1. dladm show-usage won''t work if UV fastpath is enabled.
>>
>> Since the link usage data are collected based on the statistics of the 
>> mac_client_impl_t, and UV fastpath skips the GLDv3 processing, that
won''t
>> be available.
>>
>> I don''t see a solution for this except to document this
restriction.
>>
>> 2. VLAN kstats won''t work.
>>
>> If we enable UV fastpath over VLANs, packets would go directly between
the
>> lower VLAN stream and the IP stream over softmac, the VNIC (vlan)
kstats
>> won''t be updated.
>>
>> This could be work around by changing vnic_m_stat() function, to add up
>> the legacy VLAN kstat (e.g., kstat of ce1000).
>>
>> 3. Performance of VLAN is not optimal
>>
>> In the current Crossbow code, once a VNIC (vlan) is created, the 
>> underlying mac is started (by the code path 
>>
vnic_dev_create()->vnic_unicast_add()->i_mac_unicast_add()->mac_start())
>> even when there is no user of this VLAN. As a result, all packets will
be
>> passed by mac and will be eventually dropped in MAC.
>>
>> This is not a preferred behavior and causes performance issues for
VLANs
>> over legacy devices. Even we try to use the fast-path for a VLAN, there
>> would be two lower stream over the legacy device: the shared lower
stream
>> and dedicated IP lower stream over the VLAN, so each packets will be 
>> copied once and passed to two streams, and the copy on the shared lower
>> stream will soon be dropped in the MAC module.
>>
>> The fix to this is to not call vnic_unicast_add() for VLANs, and delay 
>> this unicast_add() when mac_unicast_primary_add() is called for this
VLAN.
>> Is this possible?
>>
>> Expecting your early reply.
>>
>> Thanks
>> - Cathy
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

Cathy Zhou wrote:
>
> I am still struggling between whether or not to support fast-path for 
> VLANs. That is exactly the reason I''d like to understand all the 
> issues involved and make decisions based on that.
A compromise might be to just enable UV fast path only for the primary 
non-vlan.  Any thing else can use the default path through the mac. That 
would fix the kstat issue as well I guess.

Thirumalai

I had a discussion with Eric over phone and here is the summary:

a. since enable/disable usage log requires a DLDIOC_USAGELOG ioctl to the 
kernel, I can just disable fastpath over all softmacs when usage log is 
enabled.

b. We agreed that mac_start() and mac_datapath_setup() should not be 
called when a VLAN/VNIC is created but not used. We could separate them 
from i_mac_unicast_add() but since it is currently tightly associated 
today, separating them may not be trivial. I will file a bug and will 
spend some time on this to see how much work it will be involved.

c. Once (b) is solved. The only issue would be the VLAN kstat problem. As 
I said, I could work around it by adding another LEGACY_CAPABILITY 
callback which can be called by vnic_m_stat() to get the legacy VLAN 
kstats. The changes should be self-contained and should be minimal.

d. Because of (b) won''t be a trivial work, and without (b), VLAN
won''t
benefit from softmac fast-path. For now, I will disable softmac fast-path 
on VLANs for now.

Thanks
- Cathy
> 
>> Hi cathy,
>>
>> I''m still thinking of a few things you mentioned in your
review comments
>> so I haven''t responded to that yet.
>>
>> but regarding vlans, I got the impression from your previous mail that
>> you decided to disable fastpath for vlans because the code will be 
>> simpler.
>> did you change your mind?
>>
> I am still struggling between whether or not to support fast-path for 
> VLANs. That is exactly the reason I''d like to understand all the
issues
> involved and make decisions based on that.
> 
>> the changes you mentioned below sound like hacks and might not be easy
>> to do (especially, moving unicast_add). 
> 
> I don''t agree this is a hack that is only needed for VLAN fastpath
over
> legacy devices. It is just wrong to pass packets to the stack then get 
> dropped as long as we have VLAN(VNIC)s created, even those VLAN/VNICs 
> are not used.
> 
>> what''s worse is that this benefits
>> no one but vlans on legacy devices. you mentioned all these changes
could
>> be undone when legacy fastpath is removed. but I doubt this can happen
>> anytime soon nor have I heard of any plans to do so.
>>
> I agree. That is tradeoff. I personally also don''t want to make
the code
> more complexer, but I am not convinced VLAN performance over legacy 
> drivers does not matter either. That is why I have this discussion and 
> to see whether there is any other option to simplify the code and reach 
> the performance goal.
> 
>> because of these reasons, I would say it''s better to *not*
enable
>> fastpath for vlans. In my opinion, maintainability of the framework
>> far outweigh performance benefits to a few legacy devices.
>>
> 
> Thanks for starting the discussion.
> - Cathy
> 
>> thanks
>> eric
>>
>>
>> On Thu, Feb 12, 2009 at 03:25:52PM -0800, Cathy Zhou wrote:
>>> Hi the Crossbow team,
>>>
>>> I am testing my UV fastpath bits and I found several issues that
need
>>> your suggestion:
>>>
>>> 1. dladm show-usage won''t work if UV fastpath is enabled.
>>>
>>> Since the link usage data are collected based on the statistics of 
>>> the mac_client_impl_t, and UV fastpath skips the GLDv3 processing, 
>>> that won''t be available.
>>>
>>> I don''t see a solution for this except to document this
restriction.
>>>
>>> 2. VLAN kstats won''t work.
>>>
>>> If we enable UV fastpath over VLANs, packets would go directly 
>>> between the lower VLAN stream and the IP stream over softmac, the 
>>> VNIC (vlan) kstats won''t be updated.
>>>
>>> This could be work around by changing vnic_m_stat() function, to
add
>>> up the legacy VLAN kstat (e.g., kstat of ce1000).
>>>
>>> 3. Performance of VLAN is not optimal
>>>
>>> In the current Crossbow code, once a VNIC (vlan) is created, the 
>>> underlying mac is started (by the code path 
>>>
vnic_dev_create()->vnic_unicast_add()->i_mac_unicast_add()->mac_start())
>>> even when there is no user of this VLAN. As a result, all packets 
>>> will be passed by mac and will be eventually dropped in MAC.
>>>
>>> This is not a preferred behavior and causes performance issues for 
>>> VLANs over legacy devices. Even we try to use the fast-path for a 
>>> VLAN, there would be two lower stream over the legacy device: the 
>>> shared lower stream and dedicated IP lower stream over the VLAN, so
>>> each packets will be copied once and passed to two streams, and the
>>> copy on the shared lower stream will soon be dropped in the MAC
module.
>>>
>>> The fix to this is to not call vnic_unicast_add() for VLANs, and 
>>> delay this unicast_add() when mac_unicast_primary_add() is called
for
>>> this VLAN. Is this possible?
>>>
>>> Expecting your early reply.
>>>
>>> Thanks
>>> - Cathy
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
> 
> 
> _________________________________
> clearview-discuss mailing list
> clearview-discuss at opensolaris.org

> a. since enable/disable usage log requires a DLDIOC_USAGELOG ioctl to the 
 > kernel, I can just disable fastpath over all softmacs when usage log is 
 > enabled.
 >
 > b. We agreed that mac_start() and mac_datapath_setup() should not be 
 > called when a VLAN/VNIC is created but not used. We could separate them 
 > from i_mac_unicast_add() but since it is currently tightly associated 
 > today, separating them may not be trivial. I will file a bug and will 
 > spend some time on this to see how much work it will be involved.
 >
 > c. Once (b) is solved. The only issue would be the VLAN kstat problem. As 
 > I said, I could work around it by adding another LEGACY_CAPABILITY 
 > callback which can be called by vnic_m_stat() to get the legacy VLAN 
 > kstats. The changes should be self-contained and should be minimal.
 > 
 > d. Because of (b) won''t be a trivial work, and without (b), VLAN
won''t
 > benefit from softmac fast-path. For now, I will disable softmac fast-path 
 > on VLANs for now.

That all sounds quite reasonable, though I''m unclear how long is
"now" in
"for now".  Is the plan to integrate without VLAN support if (b) is
too
complex, and then later address (b)?

--
meem

On 2009?02?12? 18:13, Peter Memishian wrote:
>  > a. since enable/disable usage log requires a DLDIOC_USAGELOG ioctl to
the
>  > kernel, I can just disable fastpath over all softmacs when usage log
is
>  > enabled.
>  >
>  > b. We agreed that mac_start() and mac_datapath_setup() should not be 
>  > called when a VLAN/VNIC is created but not used. We could separate
them
>  > from i_mac_unicast_add() but since it is currently tightly associated
>  > today, separating them may not be trivial. I will file a bug and will
>  > spend some time on this to see how much work it will be involved.
>  >
>  > c. Once (b) is solved. The only issue would be the VLAN kstat
problem. As
>  > I said, I could work around it by adding another LEGACY_CAPABILITY 
>  > callback which can be called by vnic_m_stat() to get the legacy VLAN 
>  > kstats. The changes should be self-contained and should be minimal.
>  > 
>  > d. Because of (b) won''t be a trivial work, and without (b),
VLAN won''t
>  > benefit from softmac fast-path. For now, I will disable softmac
fast-path
>  > on VLANs for now.
> 
> That all sounds quite reasonable, though I''m unclear how long is
"now" in
> "for now".  Is the plan to integrate without VLAN support if (b)
is too
> complex, and then later address (b)?
> 
That is the plan. Unless that after some evaluation, I found (b) is not 
that difficult to do.

Thanks
- Cathy

On Feb 12, 2009, at 7:08 PM, Cathy Zhou wrote:
> I had a discussion with Eric over phone and here is the summary:
>
> a. since enable/disable usage log requires a DLDIOC_USAGELOG ioctl  
> to the kernel, I can just disable fastpath over all softmacs when  
> usage log is enabled.
>
> b. We agreed that mac_start() and mac_datapath_setup() should not be  
> called when a VLAN/VNIC is created but not used. We could separate  
> them from i_mac_unicast_add() but since it is currently tightly  
> associated today, separating them may not be trivial. I will file a  
> bug and will spend some time on this to see how much work it will be  
> involved.
I disagree with this one. The VNIC, VLAN, or in general MAC client  
creation potentially require the allocation of hardware resources,  
programming the classifier, moving rings between groups, etc,  
depending of MAC client or administrator preferences. If any of these  
operations fail either due to the lack of hardware resources or  
because of other constraints or hardware issues, the VNIC/VLAN  
creation itself should fail at that time.

I agree with Thiru that not having softmac fastpath for VLANs is a  
reasonable compromise.

Nicolas.
>
>
> c. Once (b) is solved. The only issue would be the VLAN kstat  
> problem. As I said, I could work around it by adding another  
> LEGACY_CAPABILITY callback which can be called by vnic_m_stat() to  
> get the legacy VLAN kstats. The changes should be self-contained and  
> should be minimal.
>
> d. Because of (b) won''t be a trivial work, and without (b), VLAN  
> won''t benefit from softmac fast-path. For now, I will disable  
> softmac fast-path on VLANs for now.
>
> Thanks
> - Cathy
>
>>> Hi cathy,
>>>
>>> I''m still thinking of a few things you mentioned in your
review
>>> comments
>>> so I haven''t responded to that yet.
>>>
>>> but regarding vlans, I got the impression from your previous mail  
>>> that
>>> you decided to disable fastpath for vlans because the code will be
>>> simpler.
>>> did you change your mind?
>>>
>> I am still struggling between whether or not to support fast-path  
>> for VLANs. That is exactly the reason I''d like to understand
all
>> the issues involved and make decisions based on that.
>>> the changes you mentioned below sound like hacks and might not be  
>>> easy
>>> to do (especially, moving unicast_add).
>> I don''t agree this is a hack that is only needed for VLAN
fastpath
>> over legacy devices. It is just wrong to pass packets to the stack  
>> then get dropped as long as we have VLAN(VNIC)s created, even those  
>> VLAN/VNICs are not used.
>>> what''s worse is that this benefits
>>> no one but vlans on legacy devices. you mentioned all these  
>>> changes could
>>> be undone when legacy fastpath is removed. but I doubt this can  
>>> happen
>>> anytime soon nor have I heard of any plans to do so.
>>>
>> I agree. That is tradeoff. I personally also don''t want to
make the
>> code more complexer, but I am not convinced VLAN performance over  
>> legacy drivers does not matter either. That is why I have this  
>> discussion and to see whether there is any other option to simplify  
>> the code and reach the performance goal.
>>> because of these reasons, I would say it''s better to *not*
enable
>>> fastpath for vlans. In my opinion, maintainability of the framework
>>> far outweigh performance benefits to a few legacy devices.
>>>
>> Thanks for starting the discussion.
>> - Cathy
>>> thanks
>>> eric
>>>
>>>
>>> On Thu, Feb 12, 2009 at 03:25:52PM -0800, Cathy Zhou wrote:
>>>> Hi the Crossbow team,
>>>>
>>>> I am testing my UV fastpath bits and I found several issues
that
>>>> need your suggestion:
>>>>
>>>> 1. dladm show-usage won''t work if UV fastpath is
enabled.
>>>>
>>>> Since the link usage data are collected based on the statistics
>>>> of the mac_client_impl_t, and UV fastpath skips the GLDv3  
>>>> processing, that won''t be available.
>>>>
>>>> I don''t see a solution for this except to document
this
>>>> restriction.
>>>>
>>>> 2. VLAN kstats won''t work.
>>>>
>>>> If we enable UV fastpath over VLANs, packets would go directly
>>>> between the lower VLAN stream and the IP stream over softmac,
the
>>>> VNIC (vlan) kstats won''t be updated.
>>>>
>>>> This could be work around by changing vnic_m_stat() function,
to
>>>> add up the legacy VLAN kstat (e.g., kstat of ce1000).
>>>>
>>>> 3. Performance of VLAN is not optimal
>>>>
>>>> In the current Crossbow code, once a VNIC (vlan) is created,
the
>>>> underlying mac is started (by the code path vnic_dev_create()- 
>>>> >vnic_unicast_add()->i_mac_unicast_add()->mac_start())
even when
>>>> there is no user of this VLAN. As a result, all packets will be
>>>> passed by mac and will be eventually dropped in MAC.
>>>>
>>>> This is not a preferred behavior and causes performance issues
>>>> for VLANs over legacy devices. Even we try to use the fast-path
>>>> for a VLAN, there would be two lower stream over the legacy  
>>>> device: the shared lower stream and dedicated IP lower stream  
>>>> over the VLAN, so each packets will be copied once and passed
to
>>>> two streams, and the copy on the shared lower stream will soon
be
>>>> dropped in the MAC module.
>>>>
>>>> The fix to this is to not call vnic_unicast_add() for VLANs,
and
>>>> delay this unicast_add() when mac_unicast_primary_add() is
called
>>>> for this VLAN. Is this possible?
>>>>
>>>> Expecting your early reply.
>>>>
>>>> Thanks
>>>> - Cathy
>>>> _______________________________________________
>>>> crossbow-discuss mailing list
>>>> crossbow-discuss at opensolaris.org
>>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>> _________________________________
>> clearview-discuss mailing list
>> clearview-discuss at opensolaris.org
>
>
> _________________________________
> clearview-discuss mailing list
> clearview-discuss at opensolaris.org
-- 
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
droux at sun.com - http://blogs.sun.com/droux

>> b. We agreed that mac_start() and mac_datapath_setup() should not be 
>> called when a VLAN/VNIC is created but not used. We could separate 
>> them from i_mac_unicast_add() but since it is currently tightly 
>> associated today, separating them may not be trivial. I will file a 
>> bug and will spend some time on this to see how much work it will be 
>> involved.
> 
> I disagree with this one. The VNIC, VLAN, or in general MAC client 
> creation potentially require the allocation of hardware resources, 
> programming the classifier, moving rings between groups, etc, depending 
> of MAC client or administrator preferences. If any of these operations 
> fail either due to the lack of hardware resources or because of other 
> constraints or hardware issues, the VNIC/VLAN creation itself should 
> fail at that time.
> 
Perhaps there is some HW resources must be reserved when the VLAN/VNIC is 
created, but that does not mean that we have to tell the the driver to 
start accept the incoming packets.

Can reserving HW resource be separated from "starting HW" (start to 
accept/transmit packets)?

Thanks
- Cathy

On Feb 13, 2009, at 11:43 AM, Cathy Zhou wrote:
>
>>> b. We agreed that mac_start() and mac_datapath_setup() should not  
>>> be called when a VLAN/VNIC is created but not used. We could  
>>> separate them from i_mac_unicast_add() but since it is currently  
>>> tightly associated today, separating them may not be trivial. I  
>>> will file a bug and will spend some time on this to see how much  
>>> work it will be involved.
>> I disagree with this one. The VNIC, VLAN, or in general MAC client  
>> creation potentially require the allocation of hardware resources,  
>> programming the classifier, moving rings between groups, etc,  
>> depending of MAC client or administrator preferences. If any of  
>> these operations fail either due to the lack of hardware resources  
>> or because of other constraints or hardware issues, the VNIC/VLAN  
>> creation itself should fail at that time.
> Perhaps there is some HW resources must be reserved when the VLAN/ 
> VNIC is created, but that does not mean that we have to tell the the  
> driver to start accept the incoming packets.
>
>
> Can reserving HW resource be separated from "starting HW" (start
to
> accept/transmit packets)?
Reserving the resources require the groups and rings to be registered  
with mac, potentially moving rings between groups, and programming the  
classifier, which require the NIC to be started.

I don''t see why dropping the packets between the creation of VLANs/ 
VNICs/MAC clients and their use is a big issue. These entities are not  
created unless requested by the administrator, or as a result of a VM  
instantiation.

Nicolas.
>
>
> Thanks
> - Cathy
-- 
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
droux at sun.com - http://blogs.sun.com/droux

>>>> b. We agreed that mac_start() and mac_datapath_setup() should
not be
>>>> called when a VLAN/VNIC is created but not used. We could
separate
>>>> them from i_mac_unicast_add() but since it is currently tightly
>>>> associated today, separating them may not be trivial. I will
file a
>>>> bug and will spend some time on this to see how much work it
will be
>>>> involved.
>>> I disagree with this one. The VNIC, VLAN, or in general MAC client 
>>> creation potentially require the allocation of hardware resources, 
>>> programming the classifier, moving rings between groups, etc, 
>>> depending of MAC client or administrator preferences. If any of
these
>>> operations fail either due to the lack of hardware resources or 
>>> because of other constraints or hardware issues, the VNIC/VLAN 
>>> creation itself should fail at that time.
>> Perhaps there is some HW resources must be reserved when the VLAN/VNIC 
>> is created, but that does not mean that we have to tell the the driver 
>> to start accept the incoming packets.
>>
>>
>> Can reserving HW resource be separated from "starting HW"
(start to
>> accept/transmit packets)?
> 
> Reserving the resources require the groups and rings to be registered 
> with mac, potentially moving rings between groups, and programming the 
> classifier, which require the NIC to be started.
> 
> I don''t see why dropping the packets between the creation of 
> VLANs/VNICs/MAC clients and their use is a big issue. These entities are 
> not created unless requested by the administrator, or as a result of a 
> VM instantiation.
> 
I think about this more. What would be worst case that the HW resources 
cannot be successfully reserved? We could use the default group and set 
the promiscuous mode, right?

If that could still fail, the worst case is that we cannot bring up the 
interface after the interface is plumbed. Note that would be true for 
physical links when we failed to reserve the HW resources at the moment 
when we try to use that physical links.

Think of the case that several VNICs are created over a physical link, 
when the physical link is the only link that is in use, ideally this 
physical link can use all the HW resources and the HW resource can be 
dynamically re-distributed to other VNICs once those VNICs are started in 
use. I think this flexibility is worth exploring.

Thanks
- Cathy

On Feb 17, 2009, at 12:35 PM, Cathy Zhou wrote:
>>>>> b. We agreed that mac_start() and mac_datapath_setup()
should
>>>>> not be called when a VLAN/VNIC is created but not used. We
could
>>>>> separate them from i_mac_unicast_add() but since it is
currently
>>>>> tightly associated today, separating them may not be
trivial. I
>>>>> will file a bug and will spend some time on this to see how
much
>>>>> work it will be involved.
>>>> I disagree with this one. The VNIC, VLAN, or in general MAC  
>>>> client creation potentially require the allocation of hardware
>>>> resources, programming the classifier, moving rings between  
>>>> groups, etc, depending of MAC client or administrator  
>>>> preferences. If any of these operations fail either due to the
>>>> lack of hardware resources or because of other constraints or  
>>>> hardware issues, the VNIC/VLAN creation itself should fail at  
>>>> that time.
>>> Perhaps there is some HW resources must be reserved when the VLAN/ 
>>> VNIC is created, but that does not mean that we have to tell the  
>>> the driver to start accept the incoming packets.
>>>
>>>
>>> Can reserving HW resource be separated from "starting HW"
(start
>>> to accept/transmit packets)?
>> Reserving the resources require the groups and rings to be  
>> registered with mac, potentially moving rings between groups, and  
>> programming the classifier, which require the NIC to be started.
>> I don''t see why dropping the packets between the creation of
VLANs/
>> VNICs/MAC clients and their use is a big issue. These entities are  
>> not created unless requested by the administrator, or as a result  
>> of a VM instantiation.
> I think about this more. What would be worst case that the HW  
> resources cannot be successfully reserved? We could use the default  
> group and set the promiscuous mode, right?
>
> If that could still fail, the worst case is that we cannot bring up  
> the interface after the interface is plumbed. Note that would be  
> true for physical links when we failed to reserve the HW resources  
> at the moment when we try to use that physical links.
Administrators or MAC clients in general need to be able to specify  
whether they want to be assigned hardware resources (rings, unicast  
address slots), and we cannot fallback  to the default group and  
promiscuous mode is this case.
> Think of the case that several VNICs are created over a physical  
> link, when the physical link is the only link that is in use,  
> ideally this physical link can use all the HW resources and the HW  
> resource can be dynamically re-distributed to other VNICs once those  
> VNICs are started in use. I think this flexibility is worth exploring.
Why optimize for the unlikely case of having created VNICs unused? In  
the long term we are moving to a model where the VNICs are created on  
demand. For example we already  do this with Xen, and we are planning  
to do this for Zones as well. So the case where the VNICs exist but  
are not use is already not the common case, and is going to be even  
less so in the future.

Nicolas.
>
>
> Thanks
> - Cathy
-- 
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
droux at sun.com - http://blogs.sun.com/droux