crossbow discuss - Sep 2006 - ?: shared vs. exclusive network stacks

With crossbow''s network virtualization, as I understand it each VNIC 
gets its own stack. Is it possible to have multiple VNICs share a 
stack? I''m not sure what benefit there would be of having multiple 
VNICs share a stack, except that there will be less stacks. If a 
single zone has multiple VNICs assigned, across multiple NICs, it may 
be sufficient to only have one stack.

Similarly, will all non-VNIC interfaces share the single default stack?

Thanks
Steffen

Steffen Weiberle wrote:
> With crossbow''s network virtualization, as I understand it each
VNIC
> gets its own stack. Is it possible to have multiple VNICs share a stack? 
> I''m not sure what benefit there would be of having multiple VNICs
share
> a stack, except that there will be less stacks. If a single zone has 
> multiple VNICs assigned, across multiple NICs, it may be sufficient to 
> only have one stack.
It''s a bit different.
Each VNIC gets some network resources; a separate squeue.

But the relationship between (V)NICs and stack/IP instances is a bit 
different.
When a zone is configured to be an exclusive IP zone, it is assigned one 
or more datalink names. Those datalinks could be NICs (e.g., bge0) VLANs 
(e.g., bge33000) or VNICs.
Thus each zone can have multiple VNICs; this might make sense when the 
VNICs are on different NICs.

Note that the VLAN part doesn''t work with bge at the moment. We need to
have a /dev/bge33000 to do that.
> Similarly, will all non-VNIC interfaces share the single default stack?
No. See above.

    Erik

Steffen Weiberle wrote:
> With crossbow''s network virtualization, as I understand it each
VNIC
> gets its own stack. Is it possible to have multiple VNICs share a stack? 
> I''m not sure what benefit there would be of having multiple VNICs
share
> a stack, except that there will be less stacks. If a single zone has 
> multiple VNICs assigned, across multiple NICs, it may be sufficient to 
> only have one stack.
Sure, it''s possible. VNICs to the rest of the system are treated as 
regular NICs, and multiple VNICs can be plumbed from the same stack. You 
are giving a good example that illustrates where this would be useful. 
The zone in this case could do some forwarding/packet filtering between 
the VNICs.
> Similarly, will all non-VNIC interfaces share the single default stack?
They don''t have to. VNICs and non-VNICs can be used by any stack.

Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

Erik Nordmark wrote On 09/11/06 17:12,:
> Steffen Weiberle wrote:
> 
>> With crossbow''s network virtualization, as I understand it
each VNIC
>> gets its own stack. Is it possible to have multiple VNICs share a 
>> stack? I''m not sure what benefit there would be of having
multiple
>> VNICs share a stack, except that there will be less stacks. If a 
>> single zone has multiple VNICs assigned, across multiple NICs, it may 
>> be sufficient to only have one stack.
> 
> It''s a bit different.
> Each VNIC gets some network resources; a separate squeue.
OK, separate from the squeue per CPU without crossbow, as I understand it.
> 
> But the relationship between (V)NICs and stack/IP instances is a bit 
> different.
> When a zone is configured to be an exclusive IP zone, it is assigned one 
> or more datalink names. Those datalinks could be NICs (e.g., bge0) VLANs 
> (e.g., bge33000) or VNICs.
So the ''exclusive'' keyword'' assigns a separate stack?
> Thus each zone can have multiple VNICs; this might make sense when the 
> VNICs are on different NICs.
> 
> Note that the VLAN part doesn''t work with bge at the moment. We
need to
> have a /dev/bge33000 to do that.
> 
>> Similarly, will all non-VNIC interfaces share the single default stack?
> 
> No. See above.
If the ''exclusive'' keyword is never used, is there only one
stack? So,
thinking about why I''m even asking, there are several reasons.

Different MTUs (but that should be per NIC, unless newer NICs allow it 
per ''partition''.

Routing (and where IP makes decisions)

ndd settings (will separate stacks now allow different default 
settings for things like TCP parameters and default buffer sizes?)

Some of my questions and confusion comes from Sunay''s presentation 
referred to in section 9 of his Aug 24 blog entry, especially page 12.
http://blogs.sun.com/sunay

While IP QoS will be a nice feature, I am trying to understand how 
crossbow addresses a lot of the thing we have dealt with over the 
years where configuring the stack for a specific application need 
tended to affect the whole system. And, of course, how it addresses 
the routing issues with zones.

Thanks,
Steffen
> 
>     Erik

Nicolas Droux wrote On 09/11/06 17:31,:
> Steffen Weiberle wrote:
> 
>> With crossbow''s network virtualization, as I understand it
each VNIC
>> gets its own stack. Is it possible to have multiple VNICs share a 
>> stack? I''m not sure what benefit there would be of having
multiple
>> VNICs share a stack, except that there will be less stacks. If a 
>> single zone has multiple VNICs assigned, across multiple NICs, it may 
>> be sufficient to only have one stack.
> 
> 
> Sure, it''s possible. VNICs to the rest of the system are treated
as
> regular NICs, and multiple VNICs can be plumbed from the same stack. You 
> are giving a good example that illustrates where this would be useful. 
> The zone in this case could do some forwarding/packet filtering between 
> the VNICs.
> 
>> Similarly, will all non-VNIC interfaces share the single default stack?
> 
> 
> They don''t have to. VNICs and non-VNICs can be used by any stack.
What determines which stack? The ''stacktype'' setting is per
zone, not
per net?

I realize this is all a little early in the delivery cycle. But what 
are some of the design goals or expectations? For example:

Interfaces using the same stack will have traffic between interfaces 
go via loopback, while traffic between stacks will use routing tables 
because the souce IP stack does not know where the destination IP 
stack is located.

Thanks,
Steffen

PS. need to find a or some bge interfaces. All I have are qfe, hme, 
nge, and temporarily e1000g.
> 
> Nicolas.
>

Steffen,
> >> With crossbow''s network virtualization, as I understand
it each VNIC
> >> gets its own stack. Is it possible to have multiple VNICs share a 
> >> stack? I''m not sure what benefit there would be of having
multiple
> >> VNICs share a stack, except that there will be less stacks. If a 
> >> single zone has multiple VNICs assigned, across multiple NICs, it
may
> >> be sufficient to only have one stack.
> > 
> > It''s a bit different.
> > Each VNIC gets some network resources; a separate squeue.
> 
> OK, separate from the squeue per CPU without crossbow, as I understand it.
Yes. The VNIC by itself addresses the data path and resources associated
with it. Squeue is one such resources. So with Crossbow, you get a 
squeue per VNIC but in addition you also get part of NIC resources
(Rx/Tx rings, DMA channels, MSI-X interrupt, MAC address in case the
NIC has more than 1 MAC address), the CPU resources on which your
processing can happen for that particular NIC etc. Basically, for traffic
flowing through the VNIC, you get dedicated NIC resources, queues, kernel
threads and CPUs they run on so the traffic for that VNIC can run
without any impact from other traffic. All of this applies to pretty 
much all form of virtualizations including Xen, Zone, and ldoms.

Now both Xen and ldom have their own network stack for guest domains
but zones in their current for uses a common shared stack. Thats where
stack instances comes in allow you virtualize the network layer and
transport layer for Zones so you can have your own routing tables,
tunables related to TCP/UDP/IP and configuration etc. The configuration
includes and is not limited to how you get your IP address for the
NIC/VNIC assigned to the container, the default route, dns servers etc.
The big part is that individual Zones can be now connected to different
VLANs or subnets.
> > But the relationship between (V)NICs and stack/IP instances is a bit 
> > different.
> > When a zone is configured to be an exclusive IP zone, it is assigned
one
> > or more datalink names. Those datalinks could be NICs (e.g., bge0)
VLANs
> > (e.g., bge33000) or VNICs.
> 
> So the ''exclusive'' keyword'' assigns a separate
stack?
Yes.
> > Thus each zone can have multiple VNICs; this might make sense when the
> > VNICs are on different NICs.
> > 
> > Note that the VLAN part doesn''t work with bge at the moment.
We need to
> > have a /dev/bge33000 to do that.
> > 
> >> Similarly, will all non-VNIC interfaces share the single default
stack?
> > 
> > No. See above.
> 
> If the ''exclusive'' keyword is never used, is there only
one stack? So,
> thinking about why I''m even asking, there are several reasons.
Yes as of now.
> Different MTUs (but that should be per NIC, unless newer NICs allow it 
> per ''partition''.
The stack instance gives you ability to change things IP and above.
The VNICs themselves will allow you to change/tweak things where it
makes sense.
> Routing (and where IP makes decisions)
> 
> ndd settings (will separate stacks now allow different default 
> settings for things like TCP parameters and default buffer sizes?)
For network layer and above.
> Some of my questions and confusion comes from Sunay''s presentation
> referred to in section 9 of his Aug 24 blog entry, especially page 12.
> http://blogs.sun.com/sunay
If you mean not enough details, that would be true. The slides
are not meant for standalone. They are used along with a presenter.
For more standalone details, see the text in the blog. We will
be putting more details out soon.
> While IP QoS will be a nice feature, I am trying to understand how 
> crossbow addresses a lot of the thing we have dealt with over the 
> years where configuring the stack for a specific application need 
> tended to affect the whole system. And, of course, how it addresses 
> the routing issues with zones.
I think the blog answers to some extent how Crossbow addresses QOS
in a different manner including administering things. Are there
anything specific you want to know. Meanwhile, we are working on
putting more details out soon.

Thanks,
Sunay
> 
> Thanks,
> Steffen
> 
> > 
> >     Erik
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/crossbow-discuss
> 

-- 
Sunay Tripathi
Sr. Staff Engineer
Solaris Core Networking Technologies 
Sun MicroSystems Inc.

Solaris Networking:	http://www.opensolaris.org/os/community/networking
Project Crossbow:	http://www.opensolaris.org/os/project/crossbow

Steffen Weiberle wrote:
>>> Similarly, will all non-VNIC interfaces share the single default
stack?
>>
>> They don''t have to. VNICs and non-VNICs can be used by any
stack.
> 
> What determines which stack? The ''stacktype'' setting is
per zone, not
> per net?
The stack is determined when the (v)NIC is assigned to an exclusive 
stack non-global zone by defining the physical property of a net resource.
> 
> I realize this is all a little early in the delivery cycle. But what are 
> some of the design goals or expectations? For example:
> 
> Interfaces using the same stack will have traffic between interfaces go 
> via loopback, while traffic between stacks will use routing tables 
> because the souce IP stack does not know where the destination IP stack 
> is located.
Yes, and in the latter case, non-global zones with exclusive stacks 
running on the same host can communicate through the VNIC loopback path, 
for VNICs that are defined on top of the same underlying physical NIC.

Nicolas.

-- 
Nicolas Droux, Solaris Kernel Networking
Sun Microsystems, Inc. http://blogs.sun.com/droux

On 9/13/06, Sunay Tripathi <Sunay.Tripathi at eng.sun.com>
wrote:
>
> Yes. The VNIC by itself addresses the data path and resources associated
> with it. Squeue is one such resources. So with Crossbow, you get a
> squeue per VNIC but in addition you also get part of NIC resources
> (Rx/Tx rings, DMA channels, MSI-X interrupt, MAC address in case the
> NIC has more than 1 MAC address), the CPU resources on which your
> processing can happen for that particular NIC etc. Basically, for traffic
> flowing through the VNIC, you get dedicated NIC resources, queues, kernel
> threads and CPUs they run on so the traffic for that VNIC can run
> without any impact from other traffic. All of this applies to pretty
> much all form of virtualizations including Xen, Zone, and ldoms.
>
Sunay,

  I assume that there will be an interface added to Nemo to allow the
h/w driver to participate in VNIC creation, since you talk about
dedicated DMA channels and MSI-X interrupts. You also talk about this
in your blog entry.
  Do you have any info. on that interface yet?

  Paul

-- 
Paul Durrant
http://www.linkedin.com/in/pdurrant

Paul Durrant wrote On 09/13/06 01:15,:
> On 9/13/06, Sunay Tripathi <Sunay.Tripathi at eng.sun.com> wrote:
>
>>
>> Yes. The VNIC by itself addresses the data path and resources
associated
>> with it. Squeue is one such resources. So with Crossbow, you get a
>> squeue per VNIC but in addition you also get part of NIC resources
>> (Rx/Tx rings, DMA channels, MSI-X interrupt, MAC address in case the
>> NIC has more than 1 MAC address), the CPU resources on which your
>> processing can happen for that particular NIC etc. Basically, for 
>> traffic
>> flowing through the VNIC, you get dedicated NIC resources, queues, 
>> kernel
>> threads and CPUs they run on so the traffic for that VNIC can run
>> without any impact from other traffic. All of this applies to pretty
>> much all form of virtualizations including Xen, Zone, and ldoms.
>>
>
> Sunay,
>
>   I assume that there will be an interface added to Nemo to allow the
> h/w driver to participate in VNIC creation, since you talk about

yes, but not directly.
the h/w driver exposes through MAC its the separate Rx rings,
it Tx fifos, interfaces to  map MSI/X interrupt numbers to per-ring 
events, interface to
program multiple MAC addresses, interface to program the
receive traffic steering engine on the hardware, etc ...
Then the VNIC driver invokes the MAC functions to use the resources and
capabilities above.
The h/w and driver don''t know if those resources are assigned to a
VNIC, or
being used for different purposes (e.g. load balancing, etc ...) by the 
upper layers.

    Kais
> dedicated DMA channels and MSI-X interrupts. You also talk about this
> in your blog entry.
>   Do you have any info. on that interface yet?
>
>   Paul
>

Steffen Weiberle wrote:
>> But the relationship between (V)NICs and stack/IP instances is a bit 
>> different.
>> When a zone is configured to be an exclusive IP zone, it is assigned 
>> one or more datalink names. Those datalinks could be NICs (e.g., bge0) 
>> VLANs (e.g., bge33000) or VNICs.
> 
> So the ''exclusive'' keyword'' assigns a separate
stack?
Correct.
> If the ''exclusive'' keyword is never used, is there only
one stack? So,
> thinking about why I''m even asking, there are several reasons.
The term "stack" might be a bit overloaded. And to try to avoid 
confusion we will actually rename the project formerly known as "stack 
instances" as "IP instances".

So, if the ''exclusive'' property is not set in any zonecfg,
then there is
only one IP instance in the system; shared by all the zones.
> Different MTUs (but that should be per NIC, unless newer NICs allow it 
> per ''partition''.
Each IP instance has completely different MTUs.

In the shared IP instance we can already have a different MTU per NIC, 
and also per logical interface (e.g., bge0:1 can have a different MTU 
than bge0:2).
> Routing (and where IP makes decisions)
Each IP instance has completely different routing table.

As far as the shared IP instance is concerned, zones was designed the 
concept was a shared IP instance, hence a shared IP routing table.
However, we are doing some short-term fixes to make it somewhat easier 
to have different default routes for different zones, since customers 
are using this. Those fixes will appear in Nevada hence in OpenSolaris 
and express real soon now.
> ndd settings (will separate stacks now allow different default settings 
> for things like TCP parameters and default buffer sizes?)
The IP as well as TCP, UDP, SCTP ndd settings are separate for each IP 
instance.

But the shared IP instance will continue to have shared ndd settings.

    Erik

> > Different MTUs (but that should be per NIC, unless newer NICs allow it
 > > per ''partition''.
 > 
 > Each IP instance has completely different MTUs.
 > 
 > In the shared IP instance we can already have a different MTU per NIC, 
 > and also per logical interface (e.g., bge0:1 can have a different MTU 
 > than bge0:2).

Aside from tunnels, I''ve never been able to make any sense of being
able
to set MTU on a per-address (logical interface) basis.

-- 
meem

Peter.Memishian at sun.com wrote:
>  > > Different MTUs (but that should be per NIC, unless newer NICs
allow it
>  > > per ''partition''.
>  > 
>  > Each IP instance has completely different MTUs.
>  > 
>  > In the shared IP instance we can already have a different MTU per
NIC,
>  > and also per logical interface (e.g., bge0:1 can have a different MTU
>  > than bge0:2).
> 
> Aside from tunnels, I''ve never been able to make any sense of
being able
> to set MTU on a per-address (logical interface) basis.
Agreed.

I don''t know if anybody uses this for zones, but I don''t
understand what
purpose it would serve.

    Erik

On Fri, 2006-09-15 at 00:42 -0400, Peter.Memishian at Sun.COM
wrote:
>  > > Different MTUs (but that should be per NIC, unless newer NICs
allow it
>  > > per ''partition''.
>  > 
>  > Each IP instance has completely different MTUs.
>  > 
>  > In the shared IP instance we can already have a different MTU per
NIC,
>  > and also per logical interface (e.g., bge0:1 can have a different MTU
>  > than bge0:2).
> 
> Aside from tunnels, I''ve never been able to make any sense of
being able
> to set MTU on a per-address (logical interface) basis.
Per logical interface MTU doesn''t make any more sense for tunnels.  The
MTU that matters for a tunnel link is the path MTU to the tunnel''s
destination, which is shared by all "logical interfaces" that are
configured on the tunnel.

-Seb

> > Aside from tunnels, I''ve never been able to make any sense of
being able
 > > to set MTU on a per-address (logical interface) basis.
 > 
 > Per logical interface MTU doesn''t make any more sense for
tunnels.  The
 > MTU that matters for a tunnel link is the path MTU to the
tunnel''s
 > destination, which is shared by all "logical interfaces" that
are
 > configured on the tunnel.

So why did we add this feature?  (Sadly, `ifconfig'' only allows the
per-address MTU to be manipulated, not the interface MTU.)

-- 
meem

On Fri, 2006-09-15 at 01:39 -0400, Peter.Memishian at Sun.COM
wrote:
>  > Per logical interface MTU doesn''t make any more sense for
tunnels.  The
>  > MTU that matters for a tunnel link is the path MTU to the
tunnel''s
>  > destination, which is shared by all "logical interfaces"
that are
>  > configured on the tunnel.
> 
> So why did we add this feature?  (Sadly, `ifconfig'' only allows
the
> per-address MTU to be manipulated, not the interface MTU.)
I thought that the Solaris ifconfig mtu token had always operated on
ipif''s.  I''m not sure why it was designed this way.

In any case, now that we have dladm and soon will have link properties,
I think it would make sense to allow setting per-link MTU using the
dladm properties interface rather than fixing ifconfig''s notion of
per-address MTU.

-Seb

Sebastien Roy wrote:
>On Fri, 2006-09-15 at 01:39 -0400, Peter.Memishian at Sun.COM wrote:
>  
>
>> > Per logical interface MTU doesn''t make any more sense for
tunnels.  The
>> > MTU that matters for a tunnel link is the path MTU to the
tunnel''s
>> > destination, which is shared by all "logical interfaces"
that are
>> > configured on the tunnel.
>>
>>So why did we add this feature?  (Sadly, `ifconfig'' only allows
the
>>per-address MTU to be manipulated, not the interface MTU.)
>>    
>>
>
>I thought that the Solaris ifconfig mtu token had always operated on
>ipif''s.  I''m not sure why it was designed this way.
>
>In any case, now that we have dladm and soon will have link properties,
>I think it would make sense to allow setting per-link MTU using the
>dladm properties interface rather than fixing ifconfig''s notion of
>per-address MTU.
>  
>
Given that ifconfig allows what dladm will allow elsewhere, is
there any benefit to us from making Solaris closer to others
here or is there more potential for harm from people trying
to do the per-address MTU thing and getting it wrong without
realising it?

Darren

Peter.Memishian at Sun.COM wrote On 09/15/06 00:42,:
>  > > Different MTUs (but that should be per NIC, unless newer NICs
allow it
>  > > per ''partition''.
>  > 
>  > Each IP instance has completely different MTUs.
>  > 
>  > In the shared IP instance we can already have a different MTU per
NIC,
>  > and also per logical interface (e.g., bge0:1 can have a different MTU
>  > than bge0:2).
> 
> Aside from tunnels, I''ve never been able to make any sense of
being able
> to set MTU on a per-address (logical interface) basis.
I don''t have my own use case either, but I am just trying to 
anticipate the types of questions and configuration options folks 
might use. I am coming at this from the per-zone perspective, and 
having seem some of the things folks have asked for or tried to do, in 
a limited world, as this becomes more widespread a lot of things not 
yet anticipated may come up.

Steffen

Sebastien Roy writes:
> On Fri, 2006-09-15 at 01:39 -0400, Peter.Memishian at Sun.COM wrote:
> >  > Per logical interface MTU doesn''t make any more sense
for tunnels.  The
> >  > MTU that matters for a tunnel link is the path MTU to the
tunnel''s
> >  > destination, which is shared by all "logical
interfaces" that are
> >  > configured on the tunnel.
> > 
> > So why did we add this feature?  (Sadly, `ifconfig'' only
allows the
> > per-address MTU to be manipulated, not the interface MTU.)
> 
> I thought that the Solaris ifconfig mtu token had always operated on
> ipif''s.  I''m not sure why it was designed this way.
The one case where it could have made _some_ sense is with logical
interfaces that represent different subnets -- locally on the same
wire, but with different characteristics.

For example, suppose you have an interface that supports jumbo
frames, and you want to use large datagrams for local communication on
that subnet, but you don''t want to suffer through path MTU discovery
for other nodes.  You could in theory set up a logical interface with
a different address and subnet for that, and set up the routes to
point to that subnet.

Or perhaps there are some local destinations that cannot handle large
datagrams.  In that case, path MTU discovery would not solve the
problem, but you could put them on a separate subnet with a different
MTU.

The problem is that, well, it doesn''t actually make sense.  Routes
point to output (physical) interfaces, not logical interfaces, and
there''s no way such a scheme could work with forwarding.  Connecting
devices with different MTUs to the same subnetwork is just wrong.  ;-}
> In any case, now that we have dladm and soon will have link properties,
> I think it would make sense to allow setting per-link MTU using the
> dladm properties interface rather than fixing ifconfig''s notion of
> per-address MTU.
"Rather than" here means we need to be prepared to support hackery
based on per-LIF MTU.

Should we perhaps consider making "ifconfig foo0:N mtu" whine at the
user and getting rid of it over time?

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

On Fri, 2006-09-15 at 15:12 +0800, Darren Reed wrote:
> Given that ifconfig allows what dladm will allow elsewhere, is
> there any benefit to us from making Solaris closer to others
> here or is there more potential for harm from people trying
> to do the per-address MTU thing and getting it wrong without
> realising it?
There is certainly benefit from making Solaris interfaces compatible
with other OSs.  The only thing I would worry about is that some
customers out there may be depending on the per-ipif MTU setting in a
way that we haven''t touched upon in this discussion or that we
haven''t
conceived of.  That may be unlikely.

There are at least two bugs that document the need to fix ifconfig, so
that may be the right thing to do.  See:

6341693 ifconfig mtu should operate on the physical interface, not individual
ipif''s
4823410 IP has an inconsistent view of link mtu

I implemented a fix for this at one time (4823410 has the suggested
fix), but had to back-off as at the time, I was told that zones depended
on per-ipif setting.  I''m not confident that this was accurate
advice...

-Seb

On Fri, 2006-09-15 at 07:54 -0400, James Carlson wrote:
> > In any case, now that we have dladm and soon will have link
properties,
> > I think it would make sense to allow setting per-link MTU using the
> > dladm properties interface rather than fixing ifconfig''s
notion of
> > per-address MTU.
> 
> "Rather than" here means we need to be prepared to support
hackery
> based on per-LIF MTU.
Indeed.
> Should we perhaps consider making "ifconfig foo0:N mtu" whine at
the
> user and getting rid of it over time?
Sure, that sounds like a plan.  What about ifconfig foo0 mtu?

-Seb

Sebastien Roy writes:
> > Should we perhaps consider making "ifconfig foo0:N mtu"
whine at the
> > user and getting rid of it over time?
> 
> Sure, that sounds like a plan.  What about ifconfig foo0 mtu?
That can still mean "set IP''s physical interface MTU value,"
can''t it?

Or are you saying that IP should not have an adjustable MTU apart from
the link MTU controlled by dladm?

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

On Fri, 2006-09-15 at 10:11 -0400, James Carlson wrote:
> Sebastien Roy writes:
> > > Should we perhaps consider making "ifconfig foo0:N mtu"
whine at the
> > > user and getting rid of it over time?
> > 
> > Sure, that sounds like a plan.  What about ifconfig foo0 mtu?
> 
> That can still mean "set IP''s physical interface MTU
value," can''t it?
That''s not what it does today.  It only sets the 0th ipif mtu without
affecting any other ipifs on the interface:

seb(/)# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
index 1
        inet 127.0.0.1 netmask ff000000
ath0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500
index 2
        inet 24.34.158.167 netmask fffff800 broadcast 24.34.159.255
        ether 0:b:6b:4e:8f:87
seb(/)# ifconfig ath0 addif 1.2.3.4 up
Created new logical interface ath0:1
seb(/)# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
index 1
        inet 127.0.0.1 netmask ff000000
ath0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500
index 2
        inet 24.34.158.167 netmask fffff800 broadcast 24.34.159.255
        ether 0:b:6b:4e:8f:87
ath0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index
2
        inet 1.2.3.4 netmask ff000000 broadcast 1.255.255.255
seb(/)# ifconfig ath0 mtu 1450
seb(/)# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
index 1
        inet 127.0.0.1 netmask ff000000
ath0: flags=1001004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,FIXEDMTU>
mtu 1450 index 2
        inet 24.34.158.167 netmask fffff800 broadcast 24.34.159.255
        ether 0:b:6b:4e:8f:87
ath0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index
2
        inet 1.2.3.4 netmask ff000000 broadcast 1.255.255.255

If we''re to change it, it ought to be changed to affect IP''s
physical
interface MTU value.
> Or are you saying that IP should not have an adjustable MTU apart from
> the link MTU controlled by dladm?
It probably should, but we''d need to make an incompatible change to
ifconfig in order to implement it.  That may be acceptable given the
limited utility of an ipif specific MTU setting.

-Seb

On Fri, 2006-09-15 at 10:32 -0400, James Carlson wrote:
> Sebastien Roy writes:
> > On Fri, 2006-09-15 at 10:11 -0400, James Carlson wrote:
> > > Sebastien Roy writes:
> > > > > Should we perhaps consider making "ifconfig foo0:N
mtu" whine at the
> > > > > user and getting rid of it over time?
> > > > 
> > > > Sure, that sounds like a plan.  What about ifconfig foo0
mtu?
> > > 
> > > That can still mean "set IP''s physical interface
MTU value," can''t it?
> > 
> > That''s not what it does today.  It only sets the 0th ipif mtu
without
> > affecting any other ipifs on the interface:
> 
> Sure.  I''m suggesting that in this future world, "ifconfig
foo0 mtu"
> would set IP''s MTU for the physical interface.  "ifconfig
foo0:1 mtu"
> would return an error.
Yes.  I was confused by the word "still" above.  I agree with that.
-Seb

Sebastien Roy writes:
> On Fri, 2006-09-15 at 10:11 -0400, James Carlson wrote:
> > Sebastien Roy writes:
> > > > Should we perhaps consider making "ifconfig foo0:N
mtu" whine at the
> > > > user and getting rid of it over time?
> > > 
> > > Sure, that sounds like a plan.  What about ifconfig foo0 mtu?
> > 
> > That can still mean "set IP''s physical interface MTU
value," can''t it?
> 
> That''s not what it does today.  It only sets the 0th ipif mtu
without
> affecting any other ipifs on the interface:
Sure.  I''m suggesting that in this future world, "ifconfig foo0
mtu"
would set IP''s MTU for the physical interface.  "ifconfig foo0:1
mtu"
would return an error.
> If we''re to change it, it ought to be changed to affect
IP''s physical
> interface MTU value.
That''s just what I said ...
> > Or are you saying that IP should not have an adjustable MTU apart from
> > the link MTU controlled by dladm?
> 
> It probably should, but we''d need to make an incompatible change
to
> ifconfig in order to implement it.  That may be acceptable given the
> limited utility of an ipif specific MTU setting.
OK.  I was trying to figure out whether you disagreed with allowing IP
to adjust its MTU within the limits set by the link layer, or if it
was _only_ the per-ipif issue that was a concern.

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

Peter.Memishian at Sun.COM wrote:
>  > > Aside from tunnels, I''ve never been able to make any
sense of being able
>  > > to set MTU on a per-address (logical interface) basis.
>  > 
>  > Per logical interface MTU doesn''t make any more sense for
tunnels.  The
>  > MTU that matters for a tunnel link is the path MTU to the
tunnel''s
>  > destination, which is shared by all "logical interfaces"
that are
>  > configured on the tunnel.
> 
> So why did we add this feature?  (Sadly, `ifconfig'' only allows
the
> per-address MTU to be manipulated, not the interface MTU.)
I think it has been there since 1991.
Probably for historical reasons - in the early days of Solaris 2.0 
development the kernel could only plumb one Ethernet interface.
Thus in order to test things like IP forwarding and path MTU discovery 
the only thing that was available were logical interfaces. Hence a mtu 
per ipif_t made sense for testing back then.

But I don''t think it makes any sense for to have a different MTU for an
address; what does potentially make sense is to have a different MTU per 
route. And we already have that in the route command with the -mtu option.

Hence I think we can consider just having a MTU per network interface 
name aka ill_t.

    Erik