crossbow discuss - Jan 2009 - [networking-discuss] [brussels-dev] ipadm wish-list

Sebastien Roy wrote:
> On Fri, 2009-01-30 at 17:48 +0100, Darren Reed wrote:
>> I think that''s going to an extreme...
>> But if networking things start providing front ends to stuff in SMF,
>> the equivalent to "editting a text file" might be
"running svccfg" to
>> directly manipulate the property rather than use a front end?
>
> In the example you gave with VNIC configuration, wasn''t the
solution to
> run "dladm show-vnic -P" to discover that you in fact had a VNIC
> persistently configured (explaining why you couldn''t create it
again),
> followed by "dladm delete-vnic <vnic>" to delete that VNIC?
None of those suggestions were proferred..

"dladm show-vnic" (with or without -P) failed to display it.
But "dladm delete-vnic vnic0" did work...
I didn''t want to have to do that - at least I didn''t expect
to need to do it...

Doing "cat /etc/dladm/datalink.conf" did, however, display it.
mmm, those old fashioned administrative interfaces :-)

And now I see the "bug" (I turned on my brain, sorry.)

If you do "dladm create-vnic -l e1000g0 vnic0" but replace
your e1000g0 with another NIC (that then becomes e1000g0),
the MAC address changes and the entry in datalink.conf no
longer matches what is in the system, so it does not create
the new vnic0.

So while vnic0 follows the card around if you change slots,
when you actually replace it, you get unexpected behaviour...

While people might not do this normally with hardware, when
you''re running Solaris inside a virtual environment (such as
VirtualBox), adding and removing NICs is a lot less of a
hassle... and typically you get a random MAC address each
time you add a new NIC...

So to tie all this together...

If I''ve got a NIC I want to replace that has a vnic, I can either...
1) edit /etc/dladm/datalink.conf and update the vnic entry with
   the new MAC address[#], shutdown, replace NIC and boot..
or
2) shutdown, replace the NIC, boot single user, use dladm to
   delete the vnic, use dladm to recreate the vnic and reboot
   again so that I can verify the system starts up correctly

For the virtualbox/vmware case, there may be no option but (2)
because the software may not allow you to know the MAC address
until you''ve added it to the host - if you just stick to their
GUI. If you looked into their .conf files, you may be able to
record and reapply the MAC address from what was there previously.
But that''s using methods to which are just as bad as editting the
datalink.conf file directly, so they shouldn''t be relied upon.

I think there''s room for improvement here... but as for the
temporary vs persistent thing... in this case, it seems likely
that I want to change the persistent store only (the MAC address
in datalink.conf or ...) and leave the running configuration
alone (maybe I''ve ssh''d in over the NIC that I want to
replace.)

There''s at least a man page bug because the documentation for
create-vnic in dladm(1m) doesn''t mention that the vnic is tied
to the MAC address of the NIC. The impression that you get is
that by saying "-l e1000g0", you the vnic you are creating to
the thing with the *name* e1000g0. Maybe we need the equivalent
of "soft" vs "hard" links with vnic''s and interface
names?
e.g. a "soft link" with dladm ties a vnic to the named interface
(c.f. "ln -s foo bar" but a "hard link" with dladm ties it
to
the MAC address (like "ln foo bar" does with inodes.)

Darren
# - unlike software based NICs, cards usually have the MAC
    address printed on them somewhere, so you know what MAC
    address it will have before you plug it in...

Darren Reed wrote:
> There''s at least a man page bug because the documentation for
> create-vnic in dladm(1m) doesn''t mention that the vnic is tied
> to the MAC address of the NIC. The impression that you get is
> that by saying "-l e1000g0", you the vnic you are creating to
> the thing with the *name* e1000g0. Maybe we need the equivalent
> of "soft" vs "hard" links with vnic''s and
interface names?
> e.g. a "soft link" with dladm ties a vnic to the named interface
> (c.f. "ln -s foo bar" but a "hard link" with dladm ties
it to
> the MAC address (like "ln foo bar" does with inodes.)
The VNIC is not tied to the MAC address of the underlying NIC unless the 
underlying NIC provides multiple hardware factory MAC addresses, which 
is not the case for e1000g. If you create a VNIC on e1000g and don''t 
specify a MAC address value, we''ll generate a random MAC address and 
assign it to the VNIC. If later you delete and re-create a VNIC on top 
of that same NIC or a different NIC, a new random MAC address will be 
generated regardless. The random MAC address associated with a VNIC will 
be persistant across reboot as long as you don''t delete and re-create 
that VNIC.

The single factory primary MAC address of the underlying NIC will be 
used only when plumbing the interface, when creating VLAN data-links on 
top of the NIC, and by aggr for LACP traffic or if automatic MAC address 
assignment is specified.

So the issue is not the MAC address, the issue is the association 
between the VNIC and the underlying data-link, not the data-link name. 
Note that the same is true for all pseudo links, aggr, VLAN, etc, so 
it''s not an issue specific to VNICs.

I''m not convinced that having the association be with the name of the 
data-link would be any better, since that approach would have his own 
set of issues.

Nicolas.

Well I think I''m going nuts. This is the second, serious-seeming,
problem
that I''ve run into with crossbow and I''ll be damned if I can
reproduce the
thing. What I thought made it happen doesn''t now and all that
I''ve done
is reboot the damn thing after running some dladm commands. So either
I did something else that I''m not aware of or it was just transitory,
I''ve
got no clue.

Darren

Darren Reed wrote:
> Well I think I''m going nuts. This is the second, serious-seeming,
problem
> that I''ve run into with crossbow and I''ll be damned if I
can reproduce the
> thing. What I thought made it happen doesn''t now and all that
I''ve done
> is reboot the damn thing after running some dladm commands. So either
> I did something else that I''m not aware of or it was just
transitory, I''ve
> got no clue.
Next time you hit one of these issues it would be interesting to see a 
dump of /etc/dladm/datalink.conf, and even /etc/path_to_inst. Also you 
mentioned in an earlier email that you were running from a VM, if so 
please also list any changes made to the VM config during these experiments.

Thanks,
Nicolas.