crossbow discuss - Oct 2006 - Crossbow Man pages (Draft)

Guys,

As promised earlier, the two manpages for Crossbow are attached.

dladm (1M) allows creation/modification/deletion of Virtual NICs
and assigning bandwidth resources to it including binding
them to processors.

netrcm (1M) allows assigning bandwidth and CPU resources to 
flows without creating the VNICs as administrative entity (primarily
used for service consolidation and tradition QOS based usage).

This is the first draft of the man pages that we are targetting for
next release of Crossbow bits. Comments/feedback welcome.

Cheers,
Sunay


-- 
Sunay Tripathi
Sr. Staff Engineer
Solaris Core Networking Technologies 
Sun MicroSystems Inc.

Solaris Networking:	http://www.opensolaris.org/os/community/networking
Project Crossbow:	http://www.opensolaris.org/os/project/crossbow




-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: dladm.1m.txt
URL:
<http://mail.opensolaris.org/pipermail/crossbow-discuss/attachments/20061024/a64905a9/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: netrcm.1m.txt
URL:
<http://mail.opensolaris.org/pipermail/crossbow-discuss/attachments/20061024/a64905a9/attachment-0001.txt>

On 10/24/06, Sunay Tripathi <Sunay.Tripathi at eng.sun.com>
wrote:
>
> As promised earlier, the two manpages for Crossbow are attached.
>
Any progress on a driver API doc.?

  Paul

-- 
Paul Durrant
http://www.linkedin.com/in/pdurrant

Paul Durrant wrote On 10/24/06 01:40,:
> On 10/24/06, Sunay Tripathi <Sunay.Tripathi at eng.sun.com> wrote:
>
>>
>> As promised earlier, the two manpages for Crossbow are attached.
>>
>
> Any progress on a driver API doc.?

quite a bit of progress.
I''ll sanitize and send by early next week.

Thanks,

    Kais.
>
>   Paul
>

> > Any progress on a driver API doc.?
>
> quite a bit of progress.
> I''ll sanitize and send by early next week.
>
Cool. Look forward to it.

Thanks,

  Paul

-- 
Paul Durrant
http://www.linkedin.com/in/pdurrant

On 10/24/06, Sunay Tripathi <Sunay.Tripathi at eng.sun.com>
wrote:
> Guys,
>
> As promised earlier, the two manpages for Crossbow are attached.
The comments/questions are the type of things that I will ask as I am
reading the man page, including the "See Also" section.  I suspect
that some of the questions reflect some places where clarity could be
added and others indicate RFE''s to be filed.
>
> dladm (1M) allows creation/modification/deletion of Virtual NICs
> and assigning bandwidth resources to it including binding
> them to processors.
It seems as though you cannot set a limit and a guarantee.  Is this
correct?   For instance, if I want to guarantee 10 Mb/s but want to
set a limit of 50 Mb/s, can it be done?

If I have a physical NIC that has multiple tagged VLANs on it, can I
create a virtual NIC that is associated with one of the VLANs?

Are there any related commands that can tell you how much of a given
CPU or processor set is being consumed by a given VNIC?

Are there any related commands that can tell the administrator how
much bandwidth is being used, ideally relative to the guarantee and/or
limit?  Output similar to that found by the *stat commands would be
useful.  Something similar to "netrcm show-flow" may do it.

If a random mac address is chosen, are there any sort of guarantees
that it will not collide with other "random" NICs or VNICs on the
network segment?
> netrcm (1M) allows assigning bandwidth and CPU resources to
> flows without creating the VNICs as administrative entity (primarily
> used for service consolidation and tradition QOS based usage).
If a flow is being constricted by a limit, is there any way that I can
tell that this is why my performance is bad?  I''m going with the
assumption that the admin before me (who has long been forgotten) set
up a limit for a particular reason that has also long been forgotten.
I think that show-flow will do this, but an example would probably be
useful.
> This is the first draft of the man pages that we are targetting for
> next release of Crossbow bits. Comments/feedback welcome.
This all looks very interesting.  I look forward to finding the time
to try out the goodness that crossbow brings.

Mike

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

Mike,
> It seems as though you cannot set a limit and a guarantee.  Is this
> correct?   For instance, if I want to guarantee 10 Mb/s but want to
> set a limit of 50 Mb/s, can it be done?
That''s correct. They are currently mutually exclusive. Do you have a  
particular use case in mind?
>
> If I have a physical NIC that has multiple tagged VLANs on it, can I
> create a virtual NIC that is associated with one of the VLANs?
Yes. If you want to associate a VLAN with a particular VNIC, you  
simply create a VLAN interface on top of that VNIC. Currently VLAN  
processing is implemented above the VNIC MAC layer by dls. In the  
future we will move VLAN processing into the VNIC layer itself, but  
this is not yet captured by the man page.
>
> Are there any related commands that can tell you how much of a given
> CPU or processor set is being consumed by a given VNIC?
In general we don''t have that capability for individual kernel  
threads. But this is an interesting idea that we should investigate  
further as part of the Crossbow observability story.
>
> Are there any related commands that can tell the administrator how
> much bandwidth is being used, ideally relative to the guarantee and/or
> limit?  Output similar to that found by the *stat commands would be
> useful.  Something similar to "netrcm show-flow" may do it.
Yes, we already have a "netrcm show-vnic -s" and "netrcm
show-flow -
s" will show the VNIC and flow statistics, respectively. We''re
trying
to define new options of netstat which would show the combined  
information.
>
> If a random mac address is chosen, are there any sort of guarantees
> that it will not collide with other "random" NICs or VNICs on the
> network segment?
We''re still researching the best way to do this.
>
>> netrcm (1M) allows assigning bandwidth and CPU resources to
>> flows without creating the VNICs as administrative entity (primarily
>> used for service consolidation and tradition QOS based usage).
>
> If a flow is being constricted by a limit, is there any way that I can
> tell that this is why my performance is bad?  I''m going with the
> assumption that the admin before me (who has long been forgotten) set
> up a limit for a particular reason that has also long been forgotten.
> I think that show-flow will do this, but an example would probably be
> useful.
What we could do as a first step is show how much of the allocated  
bandwidth is used by a VNIC or flow, as discussed above, and allowing  
the output to be sorted by limit usage, in addition to bandwidth  
consumption.
>
>> This is the first draft of the man pages that we are targetting for
>> next release of Crossbow bits. Comments/feedback welcome.
>
> This all looks very interesting.  I look forward to finding the time
> to try out the goodness that crossbow brings.
Thanks for the good suggestions and comments,
Nicolas.
>
> Mike
>
> -- 
> Mike Gerdts
> http://mgerdts.blogspot.com/
> _______________________________________________
> networking-discuss mailing list
> networking-discuss at opensolaris.org
-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

[Bcc''ing networking-discuss at opensolaris.org, let''s continue
the
discussion on crossbow-discuss at opensolaris.org]

A couple of additional items came up while I was making some progress  
on the VNIC design and implementation.

- Editorial: the "-m share" option will not be provided for our next  
release, so it should be removed from that man page to avoid confusion.

- I don''t think the model we''ve been considering of exposing
MAC
address slot numbers to dladm(1M) users is really useful, and it adds  
unnecessary complexity to the user interface as well as the  
underlying kernel implementation.

A user wants to assign one of the factory MAC addresses to a VNIC. I  
don''t see why the actual slot which ends-up being assigned to a VNIC  
when it is created matters [1,2]. It would be sufficient to always  
choose the slot number for the user when a VNIC is created, and store  
that slot number with the rest of the VNIC configuration. That  
approach would allow the same factory MAC address to be assigned to  
the VNIC when the host OS restarts, and also avoid exposing MAC  
address slots to dladm(1M) users.

For these reasons I propose we remove the -m option from "dladm show- 
dev", and the -n option from "dladm create-vnic".

Thanks,
Nicolas.

Footnotes:

[1] Internally, VNICs will still be able to distinguish between the  
primary unicast address of the underlying NIC, and the additional  
factory MAC addresses. This will be needed when we allow a NIC to be  
shared by VNICs and plumbed directly, i.e. the "bge0" data-link will  
in this case become a VNIC with the default unicast address of the  
bge0 NIC.

[2] If we provide "-m shared" in the future, the grouping of VNICs  
sharing the same MAC factory MAC address could still be expressed  
without requiring exposing the MAC address slots. A factory MAC  
address slot would be selected automatically when a group is first  
created.

On Oct 24, 2006, at 1:05 AM, Sunay Tripathi wrote:
> Guys,
>
> As promised earlier, the two manpages for Crossbow are attached.
>
> dladm (1M) allows creation/modification/deletion of Virtual NICs
> and assigning bandwidth resources to it including binding
> them to processors.
>
> netrcm (1M) allows assigning bandwidth and CPU resources to
> flows without creating the VNICs as administrative entity (primarily
> used for service consolidation and tradition QOS based usage).
>
> This is the first draft of the man pages that we are targetting for
> next release of Crossbow bits. Comments/feedback welcome.
>
> Cheers,
> Sunay
>
>
> -- 
> Sunay Tripathi
> Sr. Staff Engineer
> Solaris Core Networking Technologies
> Sun MicroSystems Inc.
>
> Solaris Networking:	http://www.opensolaris.org/os/community/networking
> Project Crossbow:	http://www.opensolaris.org/os/project/crossbow
>
>
>
>
> <dladm.1m.txt>
> <netrcm.1m.txt>
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/crossbow-discuss
-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

Sunay-

Would it be useful to add some output from dladm show-vnic to the
examples?
> 
> EXAMPLES
>      Example 1:	Configuring an Aggregation
> 
>      To	configure an aggregate data-link with vinc-id 1 over the dev-
>      ices bge0 and bge1, enter the following command:
> 
>      # dladm create-aggr -d bge0 -d bge1 1
> 
> +     Example 2: Configuring VNICs
> 
> +     To create two VNICs interfaces with vinc-ids 1 and 2
> +     over a single physical device bge0, enter the following com-
> +     mands:
> 
> +     # dladm create-vnic -d bge0 1
> +     # dladm create-vnic -d bge0 2
> +     The new links will be called vnic1 and vnic2.
         # dladm show-vnic
         #         device  speed
         # vnic1   bge0    1000  Mbps
         # vnic2   bge0    1000  Mbps

The same would go for netrcm show-flow.

-Mike

The thing that could be of interest to the user would be, I think, the 
total number of
(factory or otherwise) addresses supported on the device and among them 
how many
of them are free for user to use.

-krgopi
> [Bcc''ing networking-discuss at opensolaris.org, let''s
continue the
> discussion on crossbow-discuss at opensolaris.org]
>
> A couple of additional items came up while I was making some progress 
> on the VNIC design and implementation.
>
> - Editorial: the "-m share" option will not be provided for our
next
> release, so it should be removed from that man page to avoid confusion.
>
> - I don''t think the model we''ve been considering of
exposing MAC
> address slot numbers to dladm(1M) users is really useful, and it adds 
> unnecessary complexity to the user interface as well as the underlying 
> kernel implementation.
>
> A user wants to assign one of the factory MAC addresses to a VNIC. I 
> don''t see why the actual slot which ends-up being assigned to a
VNIC
> when it is created matters [1,2]. It would be sufficient to always 
> choose the slot number for the user when a VNIC is created, and store 
> that slot number with the rest of the VNIC configuration. That 
> approach would allow the same factory MAC address to be assigned to 
> the VNIC when the host OS restarts, and also avoid exposing MAC 
> address slots to dladm(1M) users.
>
> For these reasons I propose we remove the -m option from "dladm 
> show-dev", and the -n option from "dladm create-vnic".
>
> Thanks,
> Nicolas.
>
> Footnotes:
>
> [1] Internally, VNICs will still be able to distinguish between the 
> primary unicast address of the underlying NIC, and the additional 
> factory MAC addresses. This will be needed when we allow a NIC to be 
> shared by VNICs and plumbed directly, i.e. the "bge0" data-link
will
> in this case become a VNIC with the default unicast address of the 
> bge0 NIC.
>
> [2] If we provide "-m shared" in the future, the grouping of
VNICs
> sharing the same MAC factory MAC address could still be expressed 
> without requiring exposing the MAC address slots. A factory MAC 
> address slot would be selected automatically when a group is first 
> created.
>
> On Oct 24, 2006, at 1:05 AM, Sunay Tripathi wrote:
>
>> Guys,
>>
>> As promised earlier, the two manpages for Crossbow are attached.
>>
>> dladm (1M) allows creation/modification/deletion of Virtual NICs
>> and assigning bandwidth resources to it including binding
>> them to processors.
>>
>> netrcm (1M) allows assigning bandwidth and CPU resources to
>> flows without creating the VNICs as administrative entity (primarily
>> used for service consolidation and tradition QOS based usage).
>>
>> This is the first draft of the man pages that we are targetting for
>> next release of Crossbow bits. Comments/feedback welcome.
>>
>> Cheers,
>> Sunay
>>
>>
>> --Sunay Tripathi
>> Sr. Staff Engineer
>> Solaris Core Networking Technologies
>> Sun MicroSystems Inc.
>>
>> Solaris Networking:    
>> http://www.opensolaris.org/os/community/networking
>> Project Crossbow:    http://www.opensolaris.org/os/project/crossbow
>>
>>
>>
>>
>> <dladm.1m.txt>
>> <netrcm.1m.txt>
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://opensolaris.org/mailman/listinfo/crossbow-discuss
>
> --Nicolas Droux, Solaris Kernel Networking
> Sun Microsystems, Inc. http://blogs.sun.com/droux
>
>
>
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/crossbow-discuss

On Oct 27, 2006, at 6:54 PM, Rajagopal Kunhappan wrote:
> The thing that could be of interest to the user would be, I think,  
> the total number of
> (factory or otherwise) addresses supported on the device and among  
> them how many
> of them are free for user to use.
Can you say more about a use case you have in mind which would  
benefit from that additional information?  How do you propose this  
information be displayed? Also I''m not clear about what you mean  
specifically about "or otherwise" in your comment above.

Thanks,
Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

Nicolas Droux wrote:
>
> On Oct 27, 2006, at 6:54 PM, Rajagopal Kunhappan wrote:
>
>> The thing that could be of interest to the user would be, I think, 
>> the total number of
>> (factory or otherwise) addresses supported on the device and among 
>> them how many
>> of them are free for user to use.
>
> Can you say more about a use case you have in mind which would benefit 
> from that additional information?  How do you propose this information 
> be displayed? Also I''m not clear about what you mean specifically 
> about "or otherwise" in your comment above.
If a user only wants to use the multiple mac addresses supported by the 
hardware
(and not by setting the NIC in promisc mode), he may want to know how 
many of them
are available and how many have been used up. Where can the user get 
this type of
information? Maybe this should be part of the command to display all the 
properties
of the link?
 
By "otherwise", I meant the empty slots available in bge, e1000g, etc 
for mac addresses.

-krgopi
>
> Thanks,
> Nicolas.
>
> --Nicolas Droux, Solaris Kernel Networking
> Sun Microsystems, Inc. http://blogs.sun.com/droux
>
>
>

On Oct 30, 2006, at 8:25 PM, Rajagopal Kunhappan wrote:
> If a user only wants to use the multiple mac addresses supported by  
> the hardware
> (and not by setting the NIC in promisc mode), he may want to know  
> how many of them
> are available and how many have been used up. Where can the user  
> get this type of
> information? Maybe this should be part of the command to display  
> all the properties
> of the link?
> By "otherwise", I meant the empty slots available in bge, e1000g,
> etc for mac addresses.
This is a different issue than factory MAC addresses. But I agree  
with you that if this information is provided, it should be displayed  
along with data on the hardware TX/RX rings usage, classifier  
entries, etc. In that case the entire unicast address table and flags  
should be displayed, not just factory MAC addresses. Maybe we could  
provide this as part of a more generic ''dladm show-link -v''
option.

Going back to the factory MAC addresses selection, is it useful to  
allow the administrator to give a specific factory MAC address to  
use? or should we always pick the next available address from the  
list? Is there a good use case for this? Someone mentioned support  
for "static DHCP" offline, which would require a specific MAC address
to be assigned to a VNIC. Any other opinions on this?

Thanks,
Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

* Nicolas.Droux at Sun.COM [2006-10-31 05:36:31]
> Going back to the factory MAC addresses selection, is it useful to
> allow the administrator to give a specific factory MAC address to
> use? or should we always pick the next available address from the
> list? Is there a good use case for this? Someone mentioned support
> for "static DHCP" offline, which would require a specific MAC
> address to be assigned to a VNIC. Any other opinions on this?
I wouldn''t care about slot number, but I may well want to use a
specific address.

Imagine installing a machine and configuring additional vnics, each of
which is using a factory mac address.  I map the vnics to services and
I''m happy.

Now I re-install the machine.  At the same time, I''m removing one of
the services that previously ran there.  I''d like to ensure that the
remaining services retain the originally assigned mac addresses in
order that my (external) mac based filtering continues to work
properly.

To do this, it seems that I need to be able to specify which of the
addresses is used at vnic creation time to remain sane.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

Nicolas Droux writes:
> Going back to the factory MAC addresses selection, is it useful to  
> allow the administrator to give a specific factory MAC address to  
> use? or should we always pick the next available address from the  
> list? Is there a good use case for this? Someone mentioned support  
> for "static DHCP" offline, which would require a specific MAC
address
> to be assigned to a VNIC. Any other opinions on this?
It''s not too uncommon to want to manage MAC addresses as a way of
managing IP address assignment.  It''s certainly how RARP works (and,
sadly, despite its obvious shortcomings, a lot of customers still seem
to depend on RARP), and it''s what DHCPv4 does by default.

DHCPv4 has a better option available -- Client IDs -- and with DHCPv6,
the MAC address is no longer really relevant because Client ID (DUID)
assignment is a requirement (and automatic).

The MAC address also shows up in the EUI-64 construction which is used
for IPv6 stateless address autoconfiguration.  I doubt anyone actually
_cares_, but it''s possible.

Also, there are apparently some load balancers on the market that
require you to configure MAC addresses for your servers.  On these
things, you''d _definitely_ want to be able to control the MAC address
used.

Finally, there are apparently a handful of people who believe in
static ARP configuration and MAC-based filtering as some sort of
"security" mechanism.

The bottom line is that though there are better alternatives in all of
these cases that don''t rely on MAC address stability or control, real
customers have attachments to MAC address assignment mechanisms.
You''ll probably want to get feedback from customers if you do
something less predictable here.

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

On Oct 31, 2006, at 6:09 AM, James Carlson wrote:
> The bottom line is that though there are better alternatives in all of
> these cases that don''t rely on MAC address stability or control,
real
> customers have attachments to MAC address assignment mechanisms.
> You''ll probably want to get feedback from customers if you do
> something less predictable here.
Thanks David and Jim for the additional feedback. I agree that these  
cases are sufficient justification to allow the slot number  
containing factory MAC addresses to be specified. I will keep this  
dladm(1M) option on the table for now.

I''ll follow-up later with an updated proposal to display the list of  
unicast MAC address slots, as discussed earlier in this thread.

Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

> 
> On Oct 31, 2006, at 6:09 AM, James Carlson wrote:
> 
> > The bottom line is that though there are better alternatives in all of
> > these cases that don''t rely on MAC address stability or
control, real
> > customers have attachments to MAC address assignment mechanisms.
> > You''ll probably want to get feedback from customers if you do
> > something less predictable here.
> 
> Thanks David and Jim for the additional feedback. I agree that these  
> cases are sufficient justification to allow the slot number  
> containing factory MAC addresses to be specified. I will keep this  
> dladm(1M) option on the table for now.
> 
> I''ll follow-up later with an updated proposal to display the list
of
> unicast MAC address slots, as discussed earlier in this thread.
Are you suggesting some change in the man pages or about the implementation
itself? BTW, what Jim mentioned is similar to the feedback we had heard
from customers. Several of them manage mac addresses in the same way
as IP addresses. But based on what Dave mentioned, we do need to 
incorporate the likes of bge (and similar NICs) which have more than
1 slot for mac addresses but all the addresses are not necessarily 
factory assigned. 

So we probably need some changes to explicitly control the user
assigned mac address in an empty slot. Some of the possible
semantics can be separation of creation/destruction of the VNIC
from the assigning a mac address to the empty slot. This would allow
user to explicitly fill the slots and then build/destroy VNICs
on top of that.

Alternatively we can go down the path of tying a VNIC to a randomly generated
MAC address for the life of the VNIC only. If there is an empty
slot, then the mac address goes there (and we show it as user generated
or random when asked to display all slots) but as soon as VNIC is
destroyed, the associated mac address also disappears from the slot.

Dave/Jim, any opinion on this?

Cheers,
Sunay
> 
> Nicolas.
> 
> -- 
> Nicolas Droux, Solaris Kernel Networking
> Sun Microsystems, Inc. http://blogs.sun.com/droux
> 
> 
> 
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/crossbow-discuss
> 

-- 
Sunay Tripathi
Sr. Staff Engineer
Solaris Core Networking Technologies 
Sun MicroSystems Inc.

Solaris Networking:	http://www.opensolaris.org/os/community/networking
Project Crossbow:	http://www.opensolaris.org/os/project/crossbow

Sunay Tripathi writes:
> So we probably need some changes to explicitly control the user
> assigned mac address in an empty slot. Some of the possible
> semantics can be separation of creation/destruction of the VNIC
> from the assigning a mac address to the empty slot. This would allow
> user to explicitly fill the slots and then build/destroy VNICs
> on top of that.
> 
> Alternatively we can go down the path of tying a VNIC to a randomly
generated
> MAC address for the life of the VNIC only. If there is an empty
> slot, then the mac address goes there (and we show it as user generated
> or random when asked to display all slots) but as soon as VNIC is
> destroyed, the associated mac address also disappears from the slot.
> 
> Dave/Jim, any opinion on this?
If I understand what you''re suggesting for that alternative, even
factory-supplied addresses could be destroyed when tearing down a
VNIC.  That doesn''t quite sound right to me.

Other than that, I think the models that sound easier to administer
are the ones where the MAC address "slots" stay stable as visible
objects.  Whether VNICs themselves are assigned to slots, built on top
of them, or are identically the same objects as "slots" (and thus
always available), I''m not sure it matters a great deal.

It probably just needs to be simple.  The split of traditional BSD
ifconfig into Solaris-style IP-only ifconfig plus dladm for the rest
is a substantial complication of the administrative model.  Adding
more layers wouldn''t, I think, be a good thing.

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

> Sunay Tripathi writes:
> > So we probably need some changes to explicitly control the user
> > assigned mac address in an empty slot. Some of the possible
> > semantics can be separation of creation/destruction of the VNIC
> > from the assigning a mac address to the empty slot. This would allow
> > user to explicitly fill the slots and then build/destroy VNICs
> > on top of that.
> > 
> > Alternatively we can go down the path of tying a VNIC to a randomly
generated
> > MAC address for the life of the VNIC only. If there is an empty
> > slot, then the mac address goes there (and we show it as user
generated
> > or random when asked to display all slots) but as soon as VNIC is
> > destroyed, the associated mac address also disappears from the slot.
> > 
> > Dave/Jim, any opinion on this?
> 
> If I understand what you''re suggesting for that alternative, even
> factory-supplied addresses could be destroyed when tearing down a
> VNIC.  That doesn''t quite sound right to me.
No no, the factory assigned one are never destroyed. We leave them
alone and explicitly allow users to choose them. The issue is the
NICs that can deal with more than one mac address but only 1 address
is factory assigned and rest are empty slots. Should we show the
empty slots and allow the user to fill them and manage them as if they were
factory assigned. Or should we just hide the empty slots and
when a new VNIC gets created, we use one of the empty slots
under the hood. The 2nd option would be simpler and probably
makes more sense but Dave might have other opinions in Xen world.
> Other than that, I think the models that sound easier to administer
> are the ones where the MAC address "slots" stay stable as visible
> objects.  Whether VNICs themselves are assigned to slots, built on top
> of them, or are identically the same objects as "slots" (and thus
> always available), I''m not sure it matters a great deal.
For factory assigned ones, yes the above model is what makes sense.
> It probably just needs to be simple.  The split of traditional BSD
> ifconfig into Solaris-style IP-only ifconfig plus dladm for the rest
> is a substantial complication of the administrative model.  Adding
> more layers wouldn''t, I think, be a good thing.
Agreed. Although, ifconfig has become sufficiently complicated that 
dladm is needed to makes things sane again. At this point, even
clueful people are not able to understand the ifconfig man page.

Cheers,
Sunay
> 
> -- 
> James Carlson, KISS Network                    <james.d.carlson at
sun.com>
> Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
> MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677
> 

-- 
Sunay Tripathi
Sr. Staff Engineer
Solaris Core Networking Technologies 
Sun MicroSystems Inc.

Solaris Networking:	http://www.opensolaris.org/os/community/networking
Project Crossbow:	http://www.opensolaris.org/os/project/crossbow

On Oct 31, 2006, at 2:06 PM, Sunay Tripathi wrote:
> No no, the factory assigned one are never destroyed. We leave them
> alone and explicitly allow users to choose them. The issue is the
> NICs that can deal with more than one mac address but only 1 address
> is factory assigned and rest are empty slots. Should we show the
> empty slots and allow the user to fill them and manage them as if  
> they were
> factory assigned. Or should we just hide the empty slots and
> when a new VNIC gets created, we use one of the empty slots
> under the hood. The 2nd option would be simpler and probably
> makes more sense but Dave might have other opinions in Xen world.
I was planning to allow the administrators to see all the slots  
(those that contain factory MAC addresses and the others as well) so  
that they can plan their resource allocation if desired (e.g. "do I  
have more factory MAC addresses?", "can I assign a VNIC its own MAC  
address without causing the NIC to go to promiscuous mode".) However  
the administration will be kept simple and will not require these  
slots numbers to be specified by default.

Let''s say we we have have a NIC without multiple factory MAC  
addresses. In that case VNIC will  find the next available available  
slot, assign the MAC address (fixed value or random) to the slot, and  
assign that slot to the VNIC. This selection would be done  
automatically. The MAC address value will be persistent across  
reboots, the slot number  that was assigned to the VNIC will not be.

If a NIC has factory MAC addresses, they will be available to a  
subset of the MAC address slots (as per existing multiple MAC address  
support). When a VNIC is created and the administrators want a  
factory MAC address to be assigned to the VNIC, one of these factory  
MAC address slots will be assigned to the VNIC. By default, the VNIC  
will choose the next available MAC address.

Optionally, the administrator will be able to specify the factory MAC  
address slot to assign to the VNIC. In this case, the slot that  
contained the factory MAC address will be persistent, not the MAC  
address value.

It seems that this allows the requirements we''ve been discussing so  
far to be met without introducing complexity for the simplest VNIC  
creation cases.

Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

* Sunay.Tripathi at eng.sun.com [2006-10-31 20:48:02]
> Dave/Jim, any opinion on this?
Factory assigned MAC addresses are of questionably use for Xen, due to
the ability to live migrate domains between physical systems (the
domain takes the MAC address with it when it moves).  I suspect that
we would not recommend their use, as keeping track of MAC addresses
that belong to a NIC in system A but are in use by domains that are
now on systems B, C, and D is over-ambitious.

Xen already has code to randomly generate MAC addresses and, for
consistency with other dom0 implementations, we would probably choose
to use that rather than a VNIC generated random MAC address (unless
there was a compelling reason to do otherwise).

Solaris on Xen itself almost certainly won''t care about MAC address
slots, though I can well imagine that a system administrator who wants
to ensure the best performance will wish to view the hardware
supported slots and their use by the system.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

On Oct 31, 2006, at 3:37 PM, David Edmondson wrote:
> Xen already has code to randomly generate MAC addresses and, for
> consistency with other dom0 implementations, we would probably choose
> to use that rather than a VNIC generated random MAC address (unless
> there was a compelling reason to do otherwise).
That sounds fine.
> Solaris on Xen itself almost certainly won''t care about MAC
address
> slots, though I can well imagine that a system administrator who wants
> to ensure the best performance will wish to view the hardware
> supported slots and their use by the system.
We will provide that information via an option of dladm show-link.

Thanks,
Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

David Edmondson wrote On 10/31/06 14:37,:
>Xen already has code to randomly generate MAC addresses and, for
>consistency with other dom0 implementations, we would probably choose
>to use that rather than a VNIC generated random MAC address (unless
>there was a compelling reason to do otherwise).
>  
>
would it be possible for VNICs to borrow Xen''s way to ensure non MAC
address
duplication?
any standard being followed here?

thanks,

    Kais.

> > Solaris on Xen itself almost certainly won''t care about MAC
address
 > > slots, though I can well imagine that a system administrator who
wants
 > > to ensure the best performance will wish to view the hardware
 > > supported slots and their use by the system.
 > 
 > We will provide that information via an option of dladm show-link.

As an aside: I''d request that we move in the direction of show-wifi and
other new dladm display commands, and use the -o <field>,... syntax to
allow the administrator to choose what fields to display. Note that other
formatting changes to bring show-link in-line with other "show"
commands
are already being done as part of Clearview[1].

[1] As per section 4.1 of
    http://opensolaris.org/os/project/clearview/uv-design.pdf

-- 
meem

* Kais.Belgaied at Sun.COM [2006-11-01 01:49:24]
> would it be possible for VNICs to borrow Xen''s way to ensure non
MAC address
> duplication?
The Xen approach uses a OUI that is allocated to Xensource, Inc., so
it would be inappropriate to borrow that.  The LDoms team has an OUI
allocated for this purpose I believe, but I don''t know how they plan
to manage it.  The Crossbow project could apply for an OUI.

For the trailing 3 bytes, Xen simply generates random numbers (0x0-0x7f
for the first byte, 0x0-0xff for the other two).  There''s no attempt
to detect a clash, even on a single system.

It''s very common for administrators to specify the MAC address to use,
which means that the problem of managing the address space moves
elsewhere, obviously.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

On Nov 1, 2006, at 5:35 AM, David Edmondson wrote:
> The Xen approach uses a OUI that is allocated to Xensource, Inc., so
> it would be inappropriate to borrow that.  The LDoms team has an OUI
> allocated for this purpose I believe, but I don''t know how they
plan
> to manage it.  The Crossbow project could apply for an OUI.
Yes, it looks like that would be the safest way to go.

In addition, we could also add an option to dladm(1M) allowing the  
user to specify a OUI when asking for a random MAC address to be  
assigned to a VNIC.

Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

On Oct 31, 2006, at 10:32 PM, Peter Memishian wrote:
>> We will provide that information via an option of dladm show-link.
>
> As an aside: I''d request that we move in the direction of
show-wifi
> and
> other new dladm display commands, and use the -o <field>,... syntax
to
> allow the administrator to choose what fields to display. Note that  
> other
> formatting changes to bring show-link in-line with other "show"  
> commands
> are already being done as part of Clearview[1].
>
> [1] As per section 4.1 of
>     http://opensolaris.org/os/project/clearview/uv-design.pdf
 From what I gathered from show-wifi, the -o option you are referring  
to is used to identify a field, which ends-up being displayed as one  
of the columns of a particular link.

The kind of information we are talking about here wouldn''t easily fit  
that model. We need to display a per-link arrays of slots with each  
slot having itself a set of fields associated with them. So I''m not  
convinced that -o is the right answer in this case.

Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

> 
> On Oct 31, 2006, at 2:06 PM, Sunay Tripathi wrote:
> 
> > No no, the factory assigned one are never destroyed. We leave them
> > alone and explicitly allow users to choose them. The issue is the
> > NICs that can deal with more than one mac address but only 1 address
> > is factory assigned and rest are empty slots. Should we show the
> > empty slots and allow the user to fill them and manage them as if  
> > they were
> > factory assigned. Or should we just hide the empty slots and
> > when a new VNIC gets created, we use one of the empty slots
> > under the hood. The 2nd option would be simpler and probably
> > makes more sense but Dave might have other opinions in Xen world.
> 
> I was planning to allow the administrators to see all the slots  
> (those that contain factory MAC addresses and the others as well) so  
> that they can plan their resource allocation if desired (e.g. "do I  
> have more factory MAC addresses?", "can I assign a VNIC its own
MAC
> address without causing the NIC to go to promiscuous mode".) However  
> the administration will be kept simple and will not require these  
> slots numbers to be specified by default.
For the time being, just plan on showing the factory configured mac
addresses along the lines of what man page is showing. Showing the
empty slots is part of the bigger plan where a user can see all the
available hardware resources including the free ones. Things include
Rx/Tx rings, MSI-X interrupts, DMA channels, empty slots for mac
addresses etc. I think we need to figure out how much of the H/W
internals we need to expose.

For the time being, the user can specify the ''-H'' option if he
wants
to ensure that a H/W slot or H/W classification is used for the mac
address.
> 
> Let''s say we we have have a NIC without multiple factory MAC  
> addresses. In that case VNIC will  find the next available available  
> slot, assign the MAC address (fixed value or random) to the slot, and  
> assign that slot to the VNIC. This selection would be done  
> automatically. The MAC address value will be persistent across  
> reboots, the slot number  that was assigned to the VNIC will not be.
Thats great. Just for consistancy sake, can you atleast ensure
that the VNICs that were using a H/W slot will get the same across
a reboot unless something else has changed.
> If a NIC has factory MAC addresses, they will be available to a  
> subset of the MAC address slots (as per existing multiple MAC address  
> support). When a VNIC is created and the administrators want a  
> factory MAC address to be assigned to the VNIC, one of these factory  
> MAC address slots will be assigned to the VNIC. By default, the VNIC  
> will choose the next available MAC address.
Yes, that should be the plan.
> Optionally, the administrator will be able to specify the factory MAC  
> address slot to assign to the VNIC. In this case, the slot that  
> contained the factory MAC address will be persistent, not the MAC  
> address value.
I am not sure whats the need of this. 

Cheers,
Sunay
> It seems that this allows the requirements we''ve been discussing
so
> far to be met without introducing complexity for the simplest VNIC  
> creation cases.
> 
> Nicolas.
> 
> -- 
> Nicolas Droux, Solaris Kernel Networking
> Sun Microsystems, Inc. http://blogs.sun.com/droux
> 
> 
> 

-- 
Sunay Tripathi
Sr. Staff Engineer
Solaris Core Networking Technologies 
Sun MicroSystems Inc.

Solaris Networking:	http://www.opensolaris.org/os/community/networking
Project Crossbow:	http://www.opensolaris.org/os/project/crossbow

On Nov 2, 2006, at 4:40 AM, Sunay Tripathi wrote:
> For the time being, just plan on showing the factory configured mac
> addresses along the lines of what man page is showing. Showing the
> empty slots is part of the bigger plan where a user can see all the
> available hardware resources including the free ones. Things include
> Rx/Tx rings, MSI-X interrupts, DMA channels, empty slots for mac
> addresses etc. I think we need to figure out how much of the H/W
> internals we need to expose.
>
> For the time being, the user can specify the ''-H'' option
if he wants
> to ensure that a H/W slot or H/W classification is used for the mac
> address.
We''re in agreement on the end goal, and I think it''s fine
providing
only that subset of functionality today until we have a better  
understanding on how to display the remaining of the hardware resources.
>> Optionally, the administrator will be able to specify the factory MAC
>> address slot to assign to the VNIC. In this case, the slot that
>> contained the factory MAC address will be persistent, not the MAC
>> address value.
>
> I am not sure whats the need of this.
To allow the same factory MAC addresses to be assigned to the VNICs  
when the host OS reboots.

Thanks,
Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

> > [1] As per section 4.1 of
 > >     http://opensolaris.org/os/project/clearview/uv-design.pdf
 > 
 >  From what I gathered from show-wifi, the -o option you are referring  
 > to is used to identify a field, which ends-up being displayed as one  
 > of the columns of a particular link.
 > 
 > The kind of information we are talking about here wouldn''t easily
fit
 > that model. We need to display a per-link arrays of slots with each  
 > slot having itself a set of fields associated with them. So I''m
not
 > convinced that -o is the right answer in this case.

This sounds like an odd fit for the current show-link model regardless,
since the slot (rather than the link) seems to be the object being shown.
That said, I''m not advocating show-slot just yet ;-)  This probably
needs
some more thought.

Leaving that aside for a moment: for properties that are directly
associated with the link, I have a strong preference for us to:

	* Provide stable -o <field> names that allow these to be
	  displayed.

	* Indicate that the default output for show-link (without use
	  of -o) is unstable, allowing us to change it to provide as
	  much useful info as possible in 80 columns over time.

	* As appropriate, create single-letter options that correspond
	  to common -o <field1>,<field2>,... that are 80-column friendly,
	  to make things more convenient to administer.

-- 
meem

> dladm (1M) allows creation/modification/deletion of Virtual NICs
 > and assigning bandwidth resources to it including binding
 > them to processors.

FWIW, as part of WiFi, I overhauled the dladm manpage to be structured
more similarly to zfs(1M) and zonecfg(1M).  That is, subcommand options
are now grouped under each subcommand, instead of in a shared section.
This makes it easier to figure out what options go with which subcommands.

For reference, the updated manpage is included in the tarball at:

  http://opensolaris.org/os/community/networking/gldv3-wifi-docs.tar.gz

--
meem

Nicolas Droux wrote On 10/25/06 02:06,:
> Mike,
> 
>> It seems as though you cannot set a limit and a guarantee.  Is this
>> correct?   For instance, if I want to guarantee 10 Mb/s but want to
>> set a limit of 50 Mb/s, can it be done?
> 
> 
> That''s correct. They are currently mutually exclusive. Do you have
a
> particular use case in mind?
I don''t remember seeing Mike''s response, but I would agree.

I want to provision a number of services and insure they get a minimum amount,
but not exceed a
maximum as well. With them being mutually exclusive, the minimum is
''guarenteed'' by the sum of the
maximums. But that is not ideal.

My ''customer'' expects a certain amount, and I will allow more
if it is available, but not too much.

The challenges I see include:

Making sure the minimums don''t exceed what is currently available.

Not allowing a dynamic reconfiguration of an interface out of the system if the
sum of the minimums
would be exceeded.

How does link speed really correlate to maximum bandwidth available?

What happens when link speed changes (goes down)?



As a reference point, ZFS supports maximum (quota) *and* minimum (reservation)
concurrently.

Steffen

On 11/7/06, Steffen Weiberle <Steffen.Weiberle at sun.com>
wrote:
> Nicolas Droux wrote On 10/25/06 02:06,:
> > Mike,
> >
> >> It seems as though you cannot set a limit and a guarantee.  Is
this
> >> correct?   For instance, if I want to guarantee 10 Mb/s but want
to
> >> set a limit of 50 Mb/s, can it be done?
> >
> >
> > That''s correct. They are currently mutually exclusive. Do you
have a
> > particular use case in mind?
>
> I don''t remember seeing Mike''s response, but I would
agree.
I don''t think I responded... I was more specifically looking for
clarity in the man page.
> My ''customer'' expects a certain amount, and I will allow
more if it is available, but not too much.
This is a key area of concern in other areas of resource management.
So far I haven''t gotten to a point where bandwidth restrictions are a
big concern.  In my environments, the key concern with "giving too
much (CPU|memory)" is that the expectation is that this amount will
always be available.  This may or may not be the case, depending on
which other workloads are consolidated to the machine.  The best
mechanism that I have today (with the appropriate levels of
granularity) is to guarantee the minimum and monitor actual usage with
the plan to periodically review actual utilization vs. guaranteed
allocation.
> As a reference point, ZFS supports maximum (quota) *and* minimum
(reservation) concurrently.
I see this as very desirable in most every case where a resource is
being controlled.  I just don''t have the business case to jump up and
down saying I need it for networking today.  At a minimum, please
don''t rule out this for future implementation.

Mike

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

On Nov 7, 2006, at 7:14 PM, Mike Gerdts wrote:
> I see this as very desirable in most every case where a resource is
> being controlled.  I just don''t have the business case to jump up
and
> down saying I need it for networking today.  At a minimum, please
> don''t rule out this for future implementation.
I''d like to clarify that we haven''t completely ruled out this
feature. Both you and Steffen gave some great feedback in earlier  
emails, and we should certainly consider providing that functionality  
if it can be useful and fits the current usage model.

Thanks for your comments,
Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux