crossbow discuss - Sep 2006 - Design review of IP Instances part of Crossbow

The IP instances (formerly known as "Stack Instances") piece of
crossbow
is now ready for design review comments.

There are two documents which describe the design
	si-interfaces - a high-level design focusing on the problem the
	project solves, and what the user-visible changes are

	si-design - the design of the code changes. Assumes the reader
	has read si-interfaces already!

These documents are available at
http://www.opensolaris.org/os/project/crossbow/Docs/pdf_filename_si-interfaces.pdf
and
http://www.opensolaris.org/os/project/crossbow/Docs/si-design.pdf

Please send any comments to crossbow-discuss at opensolaris.org

Note that the design described in these documents differs slightly from 
the bits that can be downloaded from opensolaris.org (e.g.,
"stacktype"
vs. "ip-type", and no IP addresses for the "net" resource in
zonecfg).
Those changes will be available in the next version of bits we make 
available on opensolaris.org.

    Erik

Hi Erik,

I have taken a look at the si-interfaces doc, and it gives a great 
background and some insight that really help out.

Some questions:

1. Will there be any way to have zones share an IP instance but not 
share it with the global zone? Not sure what the real use case would 
be, but what comes to mind is zones that should be able to 
communication with each other within the system, but the need to 
insure they are really separated from the global zone. The simples 
would be similar to the model today, where all zones have a shared IP 
instance, except for the global zone. A further extention could be 
sets of zones have shared IP instances.

2. Any way to set the default to be exclusive, and possibly not allow 
an over-ride that there can not be a non-global zone that shares the 
IP instance with the global zone? I see so far that the global will 
always be shared.

3. What is the ''restricted'' property-name for?

4. I had created zones with not network. Later I went in and added 
nets with physical=vnic* and af=inet. When the zones rebooted, they 
came up with DHCP IP addresses. Fortunately, I did not do enough to 
consume the full DHCP range, otherwise I would have done a DoS on our 
DHCP server :)  Is this automatic DHCP a function of crossbow, or 
network auto magic? If the latter, does it kick in even on previously 
run zones. If crossbow, is there a way to disable this, or might it be 
wise to not have DHCP a default--leave the zone with no network, but 
an admin can ifconfig from within if able to zlogin into the zone?

5. A Non-Goal is the IP Policy Framework and IPQoS. So anyone who has 
implemented IPQoS will be able to continue to use it in the global 
zone and for all zones that share the IP instance with the global 
zone. Is that correct?

Thanks
Steffen

Erik Nordmark wrote On 09/18/06 19:53,:
> 
> The IP instances (formerly known as "Stack Instances") piece of
crossbow
> is now ready for design review comments.
> 
> There are two documents which describe the design
>     si-interfaces - a high-level design focusing on the problem the
>     project solves, and what the user-visible changes are
> 
>     si-design - the design of the code changes. Assumes the reader
>     has read si-interfaces already!
> 
> These documents are available at
>
http://www.opensolaris.org/os/project/crossbow/Docs/pdf_filename_si-interfaces.pdf
> 
> and
> http://www.opensolaris.org/os/project/crossbow/Docs/si-design.pdf
> 
> Please send any comments to crossbow-discuss at opensolaris.org
> 
> Note that the design described in these documents differs slightly from 
> the bits that can be downloaded from opensolaris.org (e.g.,
"stacktype"
> vs. "ip-type", and no IP addresses for the "net"
resource in zonecfg).
> Those changes will be available in the next version of bits we make 
> available on opensolaris.org.
> 
>     Erik
> 
> _______________________________________________
> zones-discuss mailing list
> zones-discuss at opensolaris.org

Steffen Weiberle wrote:
> 1. Will there be any way to have zones share an IP instance but not 
> share it with the global zone? Not sure what the real use case would be, 
> but what comes to mind is zones that should be able to communication 
> with each other within the system, but the need to insure they are 
> really separated from the global zone. The simples would be similar to 
> the model today, where all zones have a shared IP instance, except for 
> the global zone. A further extention could be sets of zones have shared 
> IP instances.
I''m not sure of a real use case either, which is why there
isn''t such a
capability.

Having two zones communicate without the global zone being involved is 
something we will be able to do with CrossBow VNICs and vSwitches. The 
global zone would create the VNICs on top of the vSwitches, and then use 
zonecfg to hand them out to exclusive stack zones.
If the global zone doesn''t have a VNIC on those vSwitches, then it will
not see any packets.

Note that in all cases the non-global zones have to trust the global 
zone, since it is the global zone admin which controls the kernel and 
everything else. But we can ensure IP separation.
> 2. Any way to set the default to be exclusive, and possibly not allow an 
> over-ride that there can not be a non-global zone that shares the IP 
> instance with the global zone? I see so far that the global will always 
> be shared.
What problem are you trying to solve with this?
The reason for having the shared IP instance is so that we can provide 
backwards compatibility for zones that already have been configured - 
they will continue to use the shared IP instance to ensure they see the 
same behavior as in S10 today.
> 3. What is the ''restricted'' property-name for?
It doesn''t exist in the documents that I sent out for review.
(It does exist in the current CrossBow code, but it will be gone in the 
next code drop to opensolaris.org).
> 4. I had created zones with not network. Later I went in and added nets 
> with physical=vnic* and af=inet. When the zones rebooted, they came up 
> with DHCP IP addresses. Fortunately, I did not do enough to consume the 
> full DHCP range, otherwise I would have done a DoS on our DHCP server 
> :)  Is this automatic DHCP a function of crossbow, or network auto 
> magic? If the latter, does it kick in even on previously run zones. If 
> crossbow, is there a way to disable this, or might it be wise to not 
> have DHCP a default--leave the zone with no network, but an admin can 
> ifconfig from within if able to zlogin into the zone?
That is the behavior of the current CrossBow code on opensolaris.org, 
but it is not the behavior described in the documents.
The described behavior (which will be in the next code drop) is that 
zonecfg just specifies the datalink names (with set physical=...). Then 
the same mechanisms (the sysidtools) that are used to configure 
networking in the global zone will run in the exclusive-IP zone.
Thus at that point in time you can choose to do DHCP. (In fact, I think 
the current sysidtools default to reverse ARP configuration; DHCP might 
be a more sensible default.)
> 5. A Non-Goal is the IP Policy Framework and IPQoS. So anyone who has 
> implemented IPQoS will be able to continue to use it in the global zone 
> and for all zones that share the IP instance with the global zone. Is 
> that correct?
Correct.

And when CrossBow resource controls ships, those can be used to do most 
of what IPQoS can do but with much much less performance overhead.

    Erik

Hi Erik,

Erik Nordmark wrote On 09/19/06 23:05,:
> Steffen Weiberle wrote:
> 
>> 1. Will there be any way to have zones share an IP instance but not 
>> share it with the global zone? Not sure what the real use case would 
>> be, but what comes to mind is zones that should be able to 
>> communication with each other within the system, but the need to 
>> insure they are really separated from the global zone. The simples 
>> would be similar to the model today, where all zones have a shared IP 
>> instance, except for the global zone. A further extention could be 
>> sets of zones have shared IP instances.
> 
> I''m not sure of a real use case either, which is why there
isn''t such a
> capability.
> 
> Having two zones communicate without the global zone being involved is 
> something we will be able to do with CrossBow VNICs and vSwitches. The 
> global zone would create the VNICs on top of the vSwitches, and then use 
> zonecfg to hand them out to exclusive stack zones.
> If the global zone doesn''t have a VNIC on those vSwitches, then it
will
> not see any packets.
Thanks. This will do the same or similar. One concern on my part is 
the overhead of so many different IP instances, but maybe it is small 
relative to all the other processing going on.
> Note that in all cases the non-global zones have to trust the global 
> zone, since it is the global zone admin which controls the kernel and 
> everything else. But we can ensure IP separation.
Agree 100%. More trust is required here than in some other 
virtualization techniques.
>> 2. Any way to set the default to be exclusive, and possibly not allow 
>> an over-ride that there can not be a non-global zone that shares the 
>> IP instance with the global zone? I see so far that the global will 
>> always be shared.
> 
> What problem are you trying to solve with this?
> The reason for having the shared IP instance is so that we can provide 
> backwards compatibility for zones that already have been configured - 
> they will continue to use the shared IP instance to ensure they see the 
> same behavior as in S10 today.
The problem is fully addressing bullet 2 in Section 2 below Figure 2. 
"We can not guarantee that packets can not accidentally show up in a 
different zone..."

Since all zones depend on and must trust the global zone, this is to 
eliminate a potential leak to or from the global zone. This is why 
above I was interested in a separate stack for the GZ, and then have 
the option of shared or exclusive stacks for NGZs (by extention, 
subsets of them).
>> 3. What is the ''restricted'' property-name for?
> 
> It doesn''t exist in the documents that I sent out for review.
> (It does exist in the current CrossBow code, but it will be gone in the 
> next code drop to opensolaris.org).
So it is going away?
>> 4. I had created zones with not network. Later I went in and added 
>> nets with physical=vnic* and af=inet. When the zones rebooted, they 
>> came up with DHCP IP addresses. Fortunately, I did not do enough to 
>> consume the full DHCP range, otherwise I would have done a DoS on our 
>> DHCP server :)  Is this automatic DHCP a function of crossbow, or 
>> network auto magic? If the latter, does it kick in even on previously 
>> run zones. If crossbow, is there a way to disable this, or might it be 
>> wise to not have DHCP a default--leave the zone with no network, but 
>> an admin can ifconfig from within if able to zlogin into the zone?
> 
> That is the behavior of the current CrossBow code on opensolaris.org, 
> but it is not the behavior described in the documents.
> The described behavior (which will be in the next code drop) is that 
> zonecfg just specifies the datalink names (with set physical=...). Then 
> the same mechanisms (the sysidtools) that are used to configure 
> networking in the global zone will run in the exclusive-IP zone.
> Thus at that point in time you can choose to do DHCP. (In fact, I think 
> the current sysidtools default to reverse ARP configuration; DHCP might 
> be a more sensible default.)
1. What happens if the zone is given an interface after initial 
configruation? (That was my case--my default config files don''t 
include a network. I later go in an add the net, depending on what I 
am testing. Sysidcfg based configuration is already done with.)

2. I *strongly* believe that DHCP should *not* be the default. In my 
experience, datacenters don''t run DHCP. If zones are mis-configured on 
a LAN with DHCP, they could consume a lease accidentally, as happened 
with me. This could lead to a DoS of IP addresses. Imagine me bringing 
in a system which consumes 50 DHCP address for my VNICs. Once others 
come in with their laptops, all addresses are gone. Have to call the 
network group to clear the leases. Not good :)
>> 5. A Non-Goal is the IP Policy Framework and IPQoS. So anyone who has 
>> implemented IPQoS will be able to continue to use it in the global 
>> zone and for all zones that share the IP instance with the global 
>> zone. Is that correct?
> 
> Correct.
> 
> And when CrossBow resource controls ships, those can be used to do most 
> of what IPQoS can do but with much much less performance overhead.
When is the next code drop?

Thanks
Steffen
> 
>     Erik
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/crossbow-discuss

> 2. I *strongly* believe that DHCP should *not* be the default. In my 
 > experience, datacenters don''t run DHCP. If zones are
mis-configured on
 > a LAN with DHCP, they could consume a lease accidentally, as happened 
 > with me. This could lead to a DoS of IP addresses. Imagine me bringing 
 > in a system which consumes 50 DHCP address for my VNICs. Once others 
 > come in with their laptops, all addresses are gone. Have to call the 
 > network group to clear the leases. Not good :)

DHCP has facilities for ensuring that only specific clients can get
specific addresses, so I don''t see the DoS issue.  On the other hand,
the
horror of trying to manage network configuration without a centralized
source like DHCP seems unimaginable to me.  Note that DHCP -- despite the
name -- does not require that address assignment be dynamic.

--
meem

> 1. What happens if the zone is given an interface after initial 
> configruation? (That was my case--my default config files don''t
include a
> network. I later go in an add the net, depending on what I am testing. 
> Sysidcfg based configuration is already done with.)
Zones with an exclusive IP instance would need to manually configure
those interfaces in the same way you would with the global zone if
interfaces were added later.  For example, manually create
/etc/hostname.<ifname>, change the filtering configuration, etc.
> 2. I *strongly* believe that DHCP should *not* be the default. In my 
> experience, datacenters don''t run DHCP. If zones are
mis-configured on a LAN
> with DHCP, they could consume a lease accidentally, as happened with me.
This
> could lead to a DoS of IP addresses. Imagine me bringing in a system which 
> consumes 50 DHCP address for my VNICs. Once others come in with their 
> laptops, all addresses are gone. Have to call the network group to clear
the
> leases. Not good :)
For the initial installation, you can also supply though the IP
information via the sysicfg(4) file.  Is that not suitable for
datacenter use?

dsc

David.Comay at sun.com wrote On 09/20/06 12:58,:
>> 1. What happens if the zone is given an interface after initial 
>> configruation? (That was my case--my default config files
don''t
>> include a network. I later go in an add the net, depending on what I 
>> am testing. Sysidcfg based configuration is already done with.)
> 
> Zones with an exclusive IP instance would need to manually configure
> those interfaces in the same way you would with the global zone if
> interfaces were added later.  For example, manually create
> /etc/hostname.<ifname>, change the filtering configuration, etc.
> 
>> 2. I *strongly* believe that DHCP should *not* be the default. In my 
>> experience, datacenters don''t run DHCP. If zones are
mis-configured on
>> a LAN with DHCP, they could consume a lease accidentally, as happened 
>> with me. This could lead to a DoS of IP addresses. Imagine me bringing 
>> in a system which consumes 50 DHCP address for my VNICs. Once others 
>> come in with their laptops, all addresses are gone. Have to call the 
>> network group to clear the leases. Not good :)
> 
> For the initial installation, you can also supply though the IP
> information via the sysicfg(4) file.  Is that not suitable for
> datacenter use?
Yes, it is. But that is not what happened.

1. Configured zone without net
2. Booted zone and configured with sysidcfg
3. Modified zone configuration with net: interface=xyz
    (I did not forget to add an IP address, I intentionally did not
     give it one because I wanted to see what control I would have
     in the zone to configure *any* IP address.)
4. Rebooted zone
5. Zone came up as DHCP client.

# cat /export/zones/z1/root/etc/sysidcfg
system_locale=C
terminal=xterm
network_interface=NONE {
         hostname=z1
}
security_policy=NONE
name_service=none
timezone=US/Eastern
root_password=TRdYpxPKWfFOw
nfs4_domain=dynamic

I recognize this is an edge case, and that is why I originally asked 
whether this is a function of crossbow, or of something else, such as 
network automagic (when a system goes online, detects a network, but 
is not given an IP address, it automatically goes into
''auto-dhcp''
mode). Both /etc/hostname.xyz0 and /etc/dhcp.xyz0 were created. Makes 
it look more like NWAM, which I haven''t played with.

Thanks
Steffen
> 
> dsc
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/crossbow-discuss

Steffen Weiberle wrote:
> Thanks. This will do the same or similar. One concern on my part is the 
> overhead of so many different IP instances, but maybe it is small 
> relative to all the other processing going on.
It seems like most of the networking overhead for a separate zone comes 
from inetd creating the normal set of services (which results in a fair 
number of sockets hence stream heads, queues, etc).
>>> 2. Any way to set the default to be exclusive, and possibly not
allow
>>> an over-ride that there can not be a non-global zone that shares
the
>>> IP instance with the global zone? I see so far that the global will
>>> always be shared.
>>
>> What problem are you trying to solve with this?
>> The reason for having the shared IP instance is so that we can provide 
>> backwards compatibility for zones that already have been configured - 
>> they will continue to use the shared IP instance to ensure they see 
>> the same behavior as in S10 today.
> 
> The problem is fully addressing bullet 2 in Section 2 below Figure 2. 
> "We can not guarantee that packets can not accidentally show up in a 
> different zone..."
That is ensured by configuring the zones to have exclusive IP instances 
plus the IP instances mechanisms in the kernel.
But we still allow the configuration of zones that shared IP with the 
global zone i.e. the customer can choose whether they want IP-level 
separation or not.
Given that we need to trust the global zone admin in any case, giving 
her the option between shared and exclusive seems reasonable.
> Since all zones depend on and must trust the global zone, this is to 
> eliminate a potential leak to or from the global zone. This is why above 
> I was interested in a separate stack for the GZ, and then have the 
> option of shared or exclusive stacks for NGZs (by extention, subsets of 
> them).
I understand what you are saying, but as long as the memory overhead for 
each IP instance is not too large, it is sufficient to give each zone 
its own IP instance.

De-coupling the IP instances from the zones to have a general 
many-to-one mapping makes configuration much much more complex. You''d 
have create/name the IP instance and then attach the zones to it. Thus 
similar to how resource pool configuration works. If we really really 
need that, we can do it, but I''d prefer the simpler model that we have 
for IP instances now.
>>> 3. What is the ''restricted'' property-name for?
>>
>> It doesn''t exist in the documents that I sent out for review.
>> (It does exist in the current CrossBow code, but it will be gone in 
>> the next code drop to opensolaris.org).
> 
> So it is going away?
Yes.
Consider it already gone.

> 1. What happens if the zone is given an interface after initial 
> configruation? (That was my case--my default config files don''t
include
> a network. I later go in an add the net, depending on what I am testing. 
> Sysidcfg based configuration is already done with.)
The same as when you halt the global zone, plug in a new Ethernet card, 
and power it back on.
> 2. I *strongly* believe that DHCP should *not* be the default. In my 
> experience, datacenters don''t run DHCP. If zones are
mis-configured on a
> LAN with DHCP, they could consume a lease accidentally, as happened with 
> me. This could lead to a DoS of IP addresses. Imagine me bringing in a 
> system which consumes 50 DHCP address for my VNICs. Once others come in 
> with their laptops, all addresses are gone. Have to call the network 
> group to clear the leases. Not good :)
The goal is to use the sysidtool functionality as it evolves i.e., avoid 
doing something different for IP instances.

The easiest way to use it to configure IP instances is to deposit a 
sysidcfg file in /etc/ in the zone after it is installed and before the 
first boot.
That way you can specify timezone, root password, name service 
configuration (like for the shared-IP zones today) *plus* IP configuration.

I think it remains an open question how Solaris in general will need be 
configured by default in the future. As we see more virtualization 
(LDOMs, Xen, Zones) in datacenters it might be more common to start 
using DHCP (with fixed assignments) since this allows more easy movement 
of the zones/domains. But this is something we need to work on 
independently of IP instances.
> When is the next code drop?
We don''t have a hard schedule. The plan is to do one based on onnv_49
so
it will take a few weeks after onnv_49 is available.

    Erik

Steffen Weiberle wrote:
> Yes, it is. But that is not what happened.
> 
> 1. Configured zone without net
> 2. Booted zone and configured with sysidcfg
> 3. Modified zone configuration with net: interface=xyz
>    (I did not forget to add an IP address, I intentionally did not
>     give it one because I wanted to see what control I would have
>     in the zone to configure *any* IP address.)
> 4. Rebooted zone
> 5. Zone came up as DHCP client.
The current crossbow code on opensolaris.org does not support sysidcfg 
configuration of IP parameters for exclusive-IP zones.
We expect the next code drop to do so (but there might be some extra 
steps necessary.)

Hence the "design review" - not "review of how the current code
drop
works" ;-)

   Erik

Erik,

Looking over the IPFilter changes, a couple of them don''t appear to be
what I would consider correct.

Firstly, for each proxy (ip_*pxy.c) .c file that has had variables brought
into the ipf_stack_t, the correct architecture should be to have a stack
instance structure for the proxy .c file and to allocate this when the
proxy''s _init() function is called.  Some of the fields currently in
the
per stack instance thing are initialised, are never changed and have
no visibility (ifs_ftppxyfr, ifs_h323_fr, ifs_ipsecfr, ifs_netbiosfr,
ifs_raudiofr, ifs_rcmdfr.)  Going with that, there are accompanying
_init variables that may also not need to be per-instance.

Related to this are ifs_ap_proxylist and ifs_ap_proxies.  The array,
ifs_ap_proxies is staticly configured data that is compiled in and is
never changed.  The variable ifs_ap_proxylist is for proxies that are
loaded externally using a dedicated LKM.

Looking at the IPFilter variables, ifs_fr_refcnt doesn''t need this.
Actually, looking at its presence today, it can probably be rm''d.

There''s one other area of concern - the radix.c file.

Given this, the design document is based on a pre-surya kernel?

Darren

Darren.Reed at sun.com wrote:
> Erik,
> 
> Looking over the IPFilter changes, a couple of them don''t appear
to be
> what I would consider correct.
> 
> Firstly, for each proxy (ip_*pxy.c) .c file that has had variables brought
> into the ipf_stack_t, the correct architecture should be to have a stack
> instance structure for the proxy .c file and to allocate this when the
> proxy''s _init() function is called.  Some of the fields currently
in the
> per stack instance thing are initialised, are never changed and have
> no visibility (ifs_ftppxyfr, ifs_h323_fr, ifs_ipsecfr, ifs_netbiosfr,
> ifs_raudiofr, ifs_rcmdfr.)  Going with that, there are accompanying
> _init variables that may also not need to be per-instance.
I think you are suggesting that each proxy header file declare an 
instance structure, and then the proxy''s apr_init gets to allocate and 
initialize a new instance of that structure.

Question is how ipf would then identify the instance and what it would 
pass to functions like apr_new.
One possibility is to have apr_init return a pointer to the instance 
structure it passed in, and have ip_proxy.c keep that as a void pointer 
for the calls to the other apr_functions.
Did you have something different in mind?
> Related to this are ifs_ap_proxylist and ifs_ap_proxies.  The array,
> ifs_ap_proxies is staticly configured data that is compiled in and is
> never changed.  The variable ifs_ap_proxylist is for proxies that are
> loaded externally using a dedicated LKM.
ifs_ap_proxies and ifs_ap_proxylist are almost static - apr_ref is 
modified. Are you suggesting we should keep apr_ref in a separate 
list/array?
> Looking at the IPFilter variables, ifs_fr_refcnt doesn''t need
this.
> Actually, looking at its presence today, it can probably be rm''d.
ifs_fr_refcnt doesn''t seem to be needed in snv_49 any more.

We will audit all the foo_stack_t structures for fields that are no 
longer referenced at all; the structures have been built based on the 
global variables that existed whether or not they were used.

> There''s one other area of concern - the radix.c file.
> 
> Given this, the design document is based on a pre-surya kernel?
Yes.
I''ve discussed with Sowmini how to handle radix.c (we didn''t
merge with
a post-Surya kernel until last week.) Seems like at least mask_rnhead 
should be per instance.

    Erik

Erik Nordmark wrote:
> Darren.Reed at sun.com wrote:
>
>> Erik,
>>
>> Looking over the IPFilter changes, a couple of them don''t
appear to be
>> what I would consider correct.
>>
>> Firstly, for each proxy (ip_*pxy.c) .c file that has had variables 
>> brought
>> into the ipf_stack_t, the correct architecture should be to have a
stack
>> instance structure for the proxy .c file and to allocate this when the
>> proxy''s _init() function is called.  Some of the fields
currently in the
>> per stack instance thing are initialised, are never changed and have
>> no visibility (ifs_ftppxyfr, ifs_h323_fr, ifs_ipsecfr, ifs_netbiosfr,
>> ifs_raudiofr, ifs_rcmdfr.)  Going with that, there are accompanying
>> _init variables that may also not need to be per-instance.
>
>
> I think you are suggesting that each proxy header file declare an 
> instance structure, and then the proxy''s apr_init gets to allocate
and
> initialize a new instance of that structure.

Correct, for those cases where this makes sense.
> Question is how ipf would then identify the instance and what it would 
> pass to functions like apr_new.
> One possibility is to have apr_init return a pointer to the instance 
> structure it passed in, and have ip_proxy.c keep that as a void 
> pointer for the calls to the other apr_functions.
> Did you have something different in mind?

Initially I was thinking that each proxy would be responsible for
maintaining its own collection of stack instance pointers.  This will
however generate a needlessly high amount of code duplication.

Having the apr_init function return a pointer is a better idea.
But where does that get stored?
Putting it in the aproxy_t structure seems like a fine idea except
that this would imply an aproxy_t per instance (which I currently
don''t believe serves any purpose at all.)  The end result is that
this calls for another list of things, per instance.

The ap_session_t would be an appropriate place to store this
pointer for easy retrieval after appr_new() has been called and
that would need to search to find the correct instance pointer.

Only 3 of the proxies include data structures that could be
useful per instance: FTP, IPsec and PPTP.
>> Related to this are ifs_ap_proxylist and ifs_ap_proxies.  The array,
>> ifs_ap_proxies is staticly configured data that is compiled in and is
>> never changed.  The variable ifs_ap_proxylist is for proxies that are
>> loaded externally using a dedicated LKM.
>
>
> ifs_ap_proxies and ifs_ap_proxylist are almost static - apr_ref is 
> modified. Are you suggesting we should keep apr_ref in a separate 
> list/array?

On the contrary, apr_ref needs to stay where it is for simplicity, otherwise
you need to invent a new mechanism to track whether or not a proxy is in
use.

If/when it becomes possible to load a kernel module in a non-global zone
then it would be time to consider making some changes on a per-zone
basis.

Darren

Darren.Reed at Sun.COM wrote:

{Responding out of order.]

 >> ifs_ap_proxies and ifs_ap_proxylist are almost static - apr_ref is
 >> modified. Are you suggesting we should keep apr_ref in a separate
 >> list/array?
 >
 >
 > On the contrary, apr_ref needs to stay where it is for simplicity,
 > otherwise
 > you need to invent a new mechanism to track whether or not a proxy is in
 > use.

If apr_ref needs to stay, then we must have one ap_proxies and 
ap_proxylist per IP instance AFAICT.
Do you see an other solution?

 > If/when it becomes possible to load a kernel module in a non-global zone
 > then it would be time to consider making some changes on a per-zone
 > basis.

when = never

If you want different kernel modules for different "containers" the 
solution is having different kernels i.e. Xen or Sparc LDOMS.
> Initially I was thinking that each proxy would be responsible for
> maintaining its own collection of stack instance pointers.  This will
> however generate a needlessly high amount of code duplication.
> 
> Having the apr_init function return a pointer is a better idea.
> But where does that get stored?
> Putting it in the aproxy_t structure seems like a fine idea except
> that this would imply an aproxy_t per instance (which I currently
> don''t believe serves any purpose at all.)  The end result is that
> this calls for another list of things, per instance.
But due to apr_ref we need per-instance aproxy_t in any case.
> The ap_session_t would be an appropriate place to store this
> pointer for easy retrieval after appr_new() has been called and
> that would need to search to find the correct instance pointer.
> 
> Only 3 of the proxies include data structures that could be
> useful per instance: FTP, IPsec and PPTP.
All of them have two variables that are modified. E.g.,
	frentry_t		ifs_netbiosfr;
	int			ifs_netbios_proxy_init;

If you remove all those unneeded ones from Nevada, then IP Instances 
doesn''t have to virtualize them.

    Erik

Erik Nordmark wrote:
> Darren.Reed at Sun.COM wrote:
>
> {Responding out of order.]
>
>  >> ifs_ap_proxies and ifs_ap_proxylist are almost static - apr_ref
is
>  >> modified. Are you suggesting we should keep apr_ref in a separate
>  >> list/array?
>  >
>  >
>  > On the contrary, apr_ref needs to stay where it is for simplicity,
>  > otherwise
>  > you need to invent a new mechanism to track whether or not a proxy 
> is in
>  > use.
>
> If apr_ref needs to stay, then we must have one ap_proxies and 
> ap_proxylist per IP instance AFAICT.
> Do you see an other solution?

This reference count is used to indicate whether there are any
users of the proxy if an attempt to remove it is made.

There is nothing special or sacred here that must be different
from one instance to another.
>> Initially I was thinking that each proxy would be responsible for
>> maintaining its own collection of stack instance pointers.  This will
>> however generate a needlessly high amount of code duplication.
>>
>> Having the apr_init function return a pointer is a better idea.
>> But where does that get stored?
>> Putting it in the aproxy_t structure seems like a fine idea except
>> that this would imply an aproxy_t per instance (which I currently
>> don''t believe serves any purpose at all.)  The end result is
that
>> this calls for another list of things, per instance.
>
>
> But due to apr_ref we need per-instance aproxy_t in any case.

I disagree.
>> The ap_session_t would be an appropriate place to store this
>> pointer for easy retrieval after appr_new() has been called and
>> that would need to search to find the correct instance pointer.
>>
>> Only 3 of the proxies include data structures that could be
>> useful per instance: FTP, IPsec and PPTP.
>
>
> All of them have two variables that are modified. E.g.,
>     frentry_t        ifs_netbiosfr;
>     int            ifs_netbios_proxy_init;
>
> If you remove all those unneeded ones from Nevada, then IP Instances 
> doesn''t have to virtualize them.

If you need to virtualise these then I''d question the model being used
to select which globally scoped variables are per-instance and which
are not.

The two above (for example), should be initialised when the proxy loads
or is initialised for the first time and that is it.  If they were made
per-instance then you''d have n copies of exactly the same data in each
instance.  It can''t be changed through any configuration nor can it be
queried.

To be more explicit, only the ftp proxy has an externaly visible and
accessible variable that can be tuned from user space, although that
list could be expanded in time.

The only thing that needs to be virtualised here are tunables that
are exposed through to the systems administrator of a zone.

Darren

Darren.Reed at Sun.COM wrote:
> This reference count is used to indicate whether there are any
> users of the proxy if an attempt to remove it is made.
> 
> There is nothing special or sacred here that must be different
> from one instance to another.
Yes, but that isn''t the criteria we use for what is separated or not in
IP instances.

In order to minimize the complexity added by IP Instances we use a 
criteria where any state which is modified is made to be per IP instance.

In my opinion, if we tried to find "this would be harmless is shared 
between stacks" then we''d end up with an unsupportable mess.

The only places we do something differently is where there is a need to 
share things, such as in ipsec_loader() where we have a single kernel to 
load kernel modules into, and similar things.

> If you need to virtualise these then I''d question the model being
used
> to select which globally scoped variables are per-instance and which
> are not.
See the criteria above.
Perhaps we should get together and talk about that criteria. Do you have 
some time this week?
> To be more explicit, only the ftp proxy has an externaly visible and
> accessible variable that can be tuned from user space, although that
> list could be expanded in time.
Then why not, independently of IP instances, delete the global modified 
state from the other proxies? They all have cut and paste of a kmutex_t 
and an "initialized" variable. None of which are actually used.
> The only thing that needs to be virtualised here are tunables that
> are exposed through to the systems administrator of a zone.
That isn''t the criteria we use.

    Erik

Erik Nordmark wrote:
> Darren.Reed at Sun.COM wrote:
>
>> This reference count is used to indicate whether there are any
>> users of the proxy if an attempt to remove it is made.
>>
>> There is nothing special or sacred here that must be different
>> from one instance to another.
>
>
> Yes, but that isn''t the criteria we use for what is separated or
not
> in IP instances.
>
> In order to minimize the complexity added by IP Instances we use a 
> criteria where any state which is modified is made to be per IP instance.
>
> In my opinion, if we tried to find "this would be harmless is shared 
> between stacks" then we''d end up with an unsupportable mess.
>
> The only places we do something differently is where there is a need 
> to share things, such as in ipsec_loader() where we have a single 
> kernel to load kernel modules into, and similar things.

ok.  In that case the code in ip_proxy.c:appr_del() needs to be changed
such that it only returns 0 if apr_ref is 0 in every instance ipf exists in,
not just the "current" one.

Darren

Erik,

Here are my belated comments on the IP Instances design.
> There are two documents which describe the design
> 	si-interfaces - a high-level design focusing on the problem the
> 	project solves, and what the user-visible changes are
A general comment that in both documents page numbers seem to be
missing.

Figures 1 and 2 - I think the pictures are a little misleading the
second picture displays an example of two applications (web servers?)
both binding to INADDR_ANY on port 80.  Of course, that''s also a very
viable example for the current zones model.  Perhaps a better example
might be one of those applications designated as "Internet facing" and
another might be "Intranet-facing" or "Internal".

Page 6, Section 4 - I''m not sure if this is the proper place for a
discussion on security or if it warrants a separate section, but I
think the document should discuss the security implications of a zone
using a shared stack versus an exclusive stack.  For example, I think
including the table you had sent in private email some time ago which
documented the various attack vectors and how each type of zone
addresses the type of attack would be helpful in understanding the
trade-offs.

(I know some of this is headed to be discussed in Section 9 of the
second document but it seems at least some of the discussion belongs
here as well.)

Page 6, Section 5 - In the second paragraph, you mention there are no
planned layer two changes.  Does that mean for the initial design that
there will be no filtering done at that layer?  Again, perhaps this
merits discussion in a separate "Security" section but it''s
important
to understand what sorts of capabilities a privileged user in an
exclusive stack zone has versus one in a zone using the shared stack.

Page 7, Section 6 - In #4, leveraging zonename(1) in this way does seem
strange.  I know we discussed at one time a command to retrieve and
interpret the values of ZONE_ATTR_* kernel attributes although in this
particular case, it seems the only one that makes sense from inside the
zone itself is whether this zone is tied to an exclusive stack.

What about using /sbin/netstrategy and smf_netstrategy() in
/lib/svc/share/smf_include.sh for this purpose?

Page 7, Section 7 - Given the recent discussion with a customer, would
it make sense to make it clear here that there is a *single* shared IP
and the project does not allow for "islands" of shared IP instances?

Page 8, Section 9 - When does the enforcement of only allowing a
"physical" property for exclusive IP zones?  Does it occur when one
tries to complete the addition of a "net" resource by specifying
"end"
subcommand?  Or does it occur when "verify" is performed on the whole
configuration (either the explicit command or the implicit verification
that takes place prior to commit?)

In the example given for the shared IP zone, I would specify add the
/24 prefix length to the address parameter (it''s not required but
definitely encourage it) and also the missing "end" subcommand.

In the example given for the exclusive IP zone, there is also a missing
"end" subcommand.

Page 9, Section 10 - BrandZ added their BRAND column after the existing
PATH column but you''re introducing it before.  It seems the "IP
TYPE"
should follow both PATH and BRAND.  Also, updating the output for first
two "list" examples to account for the merge with BrandZ would be
helpful.

In the third example, the command listed is "list -i" but earlier
"-l"
is defined for this purpose.

With respect to "list -l", I''m not sure if this really
belongs in
zoneadm(1M) or somewhere else.  In some ways, I''d rather see it as a
new option to dladm(1M) instead.

Will the list of link names printed be the ones actually registered
with the kernel though zone_add_ifname() or the subset of those link
names actually plumbed by the zone with the exclusive stack?  I assume
the former but please verify that.

In this respect, it seems that "list -l" is printing one of that
several pieces of zone runtime state which we don''t currently print
through zoneadm(1M) or any other command.  The zones team has discussed
in the past introducing a more general "zoneadm status" or
"zoneadm
info" facility but that''s not yet designed.  Another alternative
is to
begin supporting a "list -o <format>" in zoneadm(1M) and allow
the link
names to be a supported type there.

Finally, in the case where multiple "net" resources have been
specified
with an exclusive IP zone, are the link names separated by whitespace?

Page 9, Section 11 - As I mentioned earlier, zonename(1) seems the
wrong interface for obtaining this information.  What about using
/sbin/netstrategy and smf_netstrategy()?

Page 9, Section 12 - There should be some discussion of the packaging
changes somewhere in one or both of the documents.  In the particular
case of IP Filter, is the proposal to move all of the contents of
SUNWipfr into a non-hollow package?

Page 15, Section 14 - This is more of a code review question but are
the additional privileges being granted to an exclusive stack zone
being hard-coded in either libzonecfg or zoneadmd(1M) or are you
extending the XML uses for each brand?

Page 16, Section 15 - Is the additional configuration that is being
introduced part of this case or a future case?  If the latter, how will
this project and TX interact?

Page 16, Section 17 - Could you provide a bit more detail on each of
these interfaces?  In particular, does zone_get_ifnum() return the
number of link names registered via zone_add_ifname()?  Is the second
parameter of zone_get_iflist() a single string containing the link
names registered in a concatenated form?

More generally, can zone_getattr() be used in place of these
zone_get_if*() functions?

Should these functions be dealing with "link names" rather than
"interfaces" to avoid confusion with the interfaces that ifconfig(1M)
deals with?
> 	si-design - the design of the code changes. Assumes the reader
> 	has read si-interfaces already!
Pages 14, Section 7.1 - More of a code review comment but please update
truss(1) to account for the additional argument to zone_create().

Pages 14-15, Section 7.2 - As mentioned above, could you please provide
some more details on these new system calls?  Can zone_getattr() be
used instead of the proposed zone_get_if*() functions?

Also, could you list the changes to the proposed zone_t structure?

With regard to the third bullet, please see my concerns above about the
introduction of "list -l".  I think this should be part of a general
zone status/health facility or perhaps something that dladm(1M) can
print about the link names and how their assignment zone-wise.

With respect to the fourth bullet, can the global zone plumb an
interface which has been "assigned" to an exclusive IP zone but not
yet
plumbed by that zone?

Page 15, Section 7.3 - As I asked about in the earlier document, is
"zonecfg verify" the only place where address property verification is
done or is it also done at end of adding or changing a "net" resource?

And as I asked earlier, are the additional privileges and devices being
hard-coded intop libzonecfg or zoneadm(1M) or are you extending the XML
uses for each brand?

Page 16, Section 9 - Providing the table of attack vectors here along
with some discussion would be very helpful.

dsc

> With regard to the third bullet, please see my concerns above about the
 > introduction of "list -l".  I think this should be part of a
general
 > zone status/health facility or perhaps something that dladm(1M) can
 > print about the link names and how their assignment zone-wise.

Displaying the zone with dladm show-link seems a nice approach to me.

-- 
meem

Peter Memishian wrote:
>  > With regard to the third bullet, please see my concerns above about
the
>  > introduction of "list -l".  I think this should be part of
a general
>  > zone status/health facility or perhaps something that dladm(1M) can
>  > print about the link names and how their assignment zone-wise.
> 
> Displaying the zone with dladm show-link seems a nice approach to me.
Which project is planning on making GLDv3 zone aware?
IP Instances isn''t.

    Erik

> >  > With regard to the third bullet, please see my concerns above
about the
 > >  > introduction of "list -l".  I think this should be
part of a general
 > >  > zone status/health facility or perhaps something that dladm(1M)
can
 > >  > print about the link names and how their assignment zone-wise.
 > > 
 > > Displaying the zone with dladm show-link seems a nice approach to me.
 > 
 > Which project is planning on making GLDv3 zone aware?
 > IP Instances isn''t.

Dunno.  My comment was about the overall user experience, not about which
project does what.

-- 
meem

Peter Memishian wrote:
>  > >  > With regard to the third bullet, please see my concerns
above about the
>  > >  > introduction of "list -l".  I think this should
be part of a general
>  > >  > zone status/health facility or perhaps something that
dladm(1M) can
>  > >  > print about the link names and how their assignment
zone-wise.
>  > > 
>  > > Displaying the zone with dladm show-link seems a nice approach
to me.
>  > 
>  > Which project is planning on making GLDv3 zone aware?
>  > IP Instances isn''t.
> 
> Dunno.  My comment was about the overall user experience, not about which
> project does what.
Having layer N or subsystem X, which isn''t otherwise aware of 
virtualization, be the place to look for how virtualization is set up 
seems quite odd to me.

Thus if instead of using zonecfg to configure the boundary of the zone, 
things were spread across different commands (use dladm to assign link 
names to zones, use some zpool command to assign pools to zones, use zfs 
commands to assign zfs filesystems to zones), the I can see the point.

But given that zonecfg is used to specify the boundary of the zone 
(which link names, which pools, which file systems), it seems very 
unnatural that in order to look at what assignment was made one would 
have to use separate commands (be they dladm/pooladm/zfs). It seems 
natural to be able to look at that using a zone* command.

Note that zoneadm list is a bit idiosyncratic. It displays some things 
that are properties of the running state of the zone, which can not be 
found elsewhere (the zoneid and the state), but zoneadm list also 
displays a bunch of properties of the zones that are fixed (the uuid, 
brand, zonepath, etc) do not change as zoneadm manipulates the zone.

''list -i'' religiously follows this idiosyncratic approach ;-)

    Erik

On Fri 03 Nov 2006 at 09:42AM, Erik Nordmark wrote:
> Thus if instead of using zonecfg to configure the boundary of the zone, 
> things were spread across different commands (use dladm to assign link 
> names to zones, use some zpool command to assign pools to zones, use zfs 
> commands to assign zfs filesystems to zones), the I can see the point.
This is not correct.  Several major commands let you alter the boundary
of the zone at runtime, by design.  The difference between these
commands and zonecfg is that only zonecfg lets you do that persistently.
> But given that zonecfg is used to specify the boundary of the zone 
> (which link names, which pools, which file systems), it seems very 
> unnatural that in order to look at what assignment was made one would 
> have to use separate commands (be they dladm/pooladm/zfs). It seems 
> natural to be able to look at that using a zone* command.
You can priocntl a zone, you can mount things into it, you
can ifconfig things into it, you can use poolbind to alter its
pool binding, etc.  This is by design.
> Note that zoneadm list is a bit idiosyncratic. It displays some things 
> that are properties of the running state of the zone, which can not be 
> found elsewhere (the zoneid and the state), but zoneadm list also 
> displays a bunch of properties of the zones that are fixed (the uuid, 
> brand, zonepath, etc) do not change as zoneadm manipulates the zone.
> 
> ''list -i'' religiously follows this idiosyncratic approach
;-)
We have a plan to add ''zoneadm info'' or some such to display
all the
runtime attributes of running zones.  Hopefully we''ll get to that in
the
next 12 months or so.  I''d request that you hold off on adding list -l
until we design ''info''.

It''s up to the network experts to decide whether dladm -z or some such 
should exist to provide zone scoping.

The design principal is: If ''zone'' is an important property of
the object
being managed, then it should be considered for display, since
administrators may need to be aware of that information, or may need
an easy way to get it.

        -dp

-- 
Daniel Price - Solaris Kernel Engineering - dp at eng.sun.com - blogs.sun.com/dp

Dan Price wrote:
>> ''list -i'' religiously follows this idiosyncratic
approach ;-)
> 
> We have a plan to add ''zoneadm info'' or some such to
display all the
> runtime attributes of running zones.  Hopefully we''ll get to that
in the
> next 12 months or so.  I''d request that you hold off on adding
list -l
> until we design ''info''.
The problem is that we have requests from the users of IP Instances 
(folks that have used the bits on opensolaris.org) to have a way of 
displaying which datalink names are assigned to which running zones.

If you start work on ''zonadm info'' in 12 moths when would you
expect to
have it integrated?
> It''s up to the network experts to decide whether dladm -z or some
such
> should exist to provide zone scoping.
> 
> The design principal is: If ''zone'' is an important
property of the object
> being managed, then it should be considered for display, since
> administrators may need to be aware of that information, or may need
> an easy way to get it.
But it isn''t dladm that manages the assignment of datalink names to the
zones. It is configured in zonecfg and the implementation uses 
devfs/devnames. Thus dladm is not a natural place.

    Erik

Erik Nordmark wrote:
> Dan Price wrote:
>
>>> ''list -i'' religiously follows this idiosyncratic
approach ;-)
>>
>>
>> We have a plan to add ''zoneadm info'' or some such to
display all the
>> runtime attributes of running zones.  Hopefully we''ll get to
that in the
>> next 12 months or so.  I''d request that you hold off on adding
list -l
>> until we design ''info''.
>
>
> The problem is that we have requests from the users of IP Instances 
> (folks that have used the bits on opensolaris.org) to have a way of 
> displaying which datalink names are assigned to which running zones.

Could "ifconfig" be modified to report all network interfaces that
are assigned to a zone?

There is also another tool I hacked up earlier in the year called
"ifgrep" that you could do "ifgrep -z global" or whatever
with...
Is it worth doing something more worthwhile with that?

Darren

I just subscribed to this alias, apologies if I''m missing anything
from this thread...

On 11/4/06, Darren.Reed at sun.com <Darren.Reed at sun.com>
wrote:
> Erik Nordmark wrote:
>
> > Dan Price wrote:
> >
> >>> ''list -i'' religiously follows this
idiosyncratic approach ;-)
> >>
> >>
> >> We have a plan to add ''zoneadm info'' or some
such to display all the
> >> runtime attributes of running zones.  Hopefully we''ll get
to that in the
> >> next 12 months or so.  I''d request that you hold off on
adding list -l
> >> until we design ''info''.
> >
> >
> > The problem is that we have requests from the users of IP Instances
> > (folks that have used the bits on opensolaris.org) to have a way of
> > displaying which datalink names are assigned to which running zones.
>
>
> Could "ifconfig" be modified to report all network interfaces
that
> are assigned to a zone?
I''d like to express interest in this as well.  Just last week I came
across the need for this, and was disappointed to learn that it (or
something similar) is not there.

Eric

> There is also another tool I hacked up earlier in the year called
> "ifgrep" that you could do "ifgrep -z global" or
whatever with...
> Is it worth doing something more worthwhile with that?
>
> Darren
>
> _______________________________________________
> networking-discuss mailing list
> networking-discuss at opensolaris.org
>

-- 
Eric Enright

On Fri 03 Nov 2006 at 04:57PM, Erik Nordmark wrote:
> Dan Price wrote:
> 
> >>''list -i'' religiously follows this idiosyncratic
approach ;-)
> >
> >We have a plan to add ''zoneadm info'' or some such to
display all the
> >runtime attributes of running zones.  Hopefully we''ll get to
that in the
> >next 12 months or so.  I''d request that you hold off on adding
list -l
> >until we design ''info''.
> 
> The problem is that we have requests from the users of IP Instances 
> (folks that have used the bits on opensolaris.org) to have a way of 
> displaying which datalink names are assigned to which running zones.
> 
> If you start work on ''zonadm info'' in 12 moths when would
you expect to
> have it integrated?
By "get to it" I mean "have it finished"

        -dp

-- 
Daniel Price - Solaris Kernel Engineering - dp at eng.sun.com - blogs.sun.com/dp

Eric Enright wrote:
> I just subscribed to this alias, apologies if I''m missing anything
> from this thread...
Some of this was discussed a few months back.
> I''d like to express interest in this as well.  Just last week I
came
> across the need for this, and was disappointed to learn that it (or
> something similar) is not there.
Did you look at the description of ''zoneadm list -l'' in the 
si-interfaces.pdf?

The next code drop will include that functionality, and it was added in 
response to a request for better observability on the crossbow-discuss list.

    Erik

Darren.Reed at Sun.COM wrote:
> Could "ifconfig" be modified to report all network interfaces
that
> are assigned to a zone?
I assume you mean in the global zone; ifconfig -a inside a zone (global 
or not) does report all the network interfaces that are configured.

But that would be quite odd.

The reason is that ifconfig in the global is not involved in configuring 
IP for the exclusive-IP zones; that is done by ifconfig running inside 
the exclusive-IP zones.
This is by design different than the IP configuration for the shared-IP 
zones; those are both configurable as well as observable using ifconfig 
in the global zone.

Furthermore, since the primary purpose of IP Instances is IP-level 
isolation (between different zones that are connected to different LANs 
or different VLANs), the design is to have no IP-level sharing. Having 
ifconfig "leak" information from non-global zones to the global zone 
might at least make it appear that some leakage is possible.

    Erik

> The reason is that ifconfig in the global is not involved in configuring IP
> for the exclusive-IP zones; that is done by ifconfig running inside the 
> exclusive-IP zones.
> This is by design different than the IP configuration for the shared-IP 
> zones; those are both configurable as well as observable using ifconfig in 
> the global zone.
>
> Furthermore, since the primary purpose of IP Instances is IP-level
isolation
> (between different zones that are connected to different LANs or different 
> VLANs), the design is to have no IP-level sharing. Having ifconfig
"leak"
> information from non-global zones to the global zone might at least make it
> appear that some leakage is possible.
Yes, that''s one of the reasons I suggested having dladm(1M) be the
place to display this information since it''s where links are
administered in general, even the ones that will be handed off to
exclusive-stack zones.

dsc

David.Comay at Sun.COM wrote:
> Yes, that''s one of the reasons I suggested having dladm(1M) be the
> place to display this information since it''s where links are
> administered in general, even the ones that will be handed off to
> exclusive-stack zones.
David,

If we want any form of internal consistency, wouldn''t we also need to 
change were we assign datalink names from zonecfg to dladm?

Thus no more ''net'' resource in zonecfg for exclusive-IP zones,
but
instead some
	dladm set-zone zoneA bge1

Only having dladm show it, and not be able to affect it, seems quite 
inconsistent.


*If* we believe that this is the right way to go, aren''t we opening up 
pandoraz box to pull out more and more from zonecfg?

I''m failing to understand the architectural or design principle that 
underlies your suggestion.

    Erik

----- Original Message ----- 
From: "Erik Nordmark" <Erik.Nordmark at
Sun.COM>
> Darren.Reed at Sun.COM wrote:
> 
>> Could "ifconfig" be modified to report all network interfaces
that
>> are assigned to a zone?
> 
> I assume you mean in the global zone; ifconfig -a inside a zone (global 
> or not) does report all the network interfaces that are configured.
Yes I do, and more specifically, I was referring to the shared
stack model, not the exclusive stack model.

I can''t see any reason why or for ifconfig, in the global zone, should
report on interfaces that belong to other zones where they are using
the exclusive zone model.

Darren

Darren Reed wrote:
> ----- Original Message ----- From: "Erik Nordmark"
<Erik.Nordmark at Sun.COM>
> 
>> Darren.Reed at Sun.COM wrote:
>>
>>> Could "ifconfig" be modified to report all network
interfaces that
>>> are assigned to a zone?
>>
>> I assume you mean in the global zone; ifconfig -a inside a zone 
>> (global or not) does report all the network interfaces that are 
>> configured.
> 
> Yes I do, and more specifically, I was referring to the shared
> stack model, not the exclusive stack model.
> 
> I can''t see any reason why or for ifconfig, in the global zone,
should
> report on interfaces that belong to other zones where they are using
> the exclusive zone model.
Here''s one reason: consistency.  All users in the GZ can see some
inforamtion
about non-global zones (e.g. "ps").  Privileged GZ users can see all
info about
non-global zones, and need to do so in order to manage them.

I am not certain that observation of exclusive-IP-instances from the GZ should
be
the default, but it should be possible.

--------------------------------------------------------------------------
Jeff VICTOR              Sun Microsystems            jeff.victor @ sun.com
OS Ambassador            Sr. Technical Specialist
Solaris 10 Zones FAQ:    http://www.opensolaris.org/os/community/zones/faq
--------------------------------------------------------------------------

Jeff Victor wrote:
> Here''s one reason: consistency.  All users in the GZ can see some 
> inforamtion about non-global zones (e.g. "ps").  Privileged GZ
users can
> see all info about non-global zones, and need to do so in order to 
> manage them.
But the exclusive-IP behavior is quite different from the shared-IP 
behavior; it offers complete IP isolation between different zones/IP 
instances.

The argument that it should have consistent behavior could also be used 
to argue that it shouldn''t offer IP isolation. I''m sure that
isn''t the
type of consistency we desire.
> I am not certain that observation of exclusive-IP-instances from the GZ 
> should be the default, but it should be possible.
They can be observed from the global zone using zoneadm list -l instead 
of ifconfig -a, which is used for the shared-IP zones.
(And if the global admin wants to look deeper than that output, then for 
both the exclusive and shared cases things behave the same in that e.g. 
netstat in the global zone does not report information for other zones.)

    Erik

Darren Reed wrote:
> ----- Original Message ----- From: "Erik Nordmark"
<Erik.Nordmark at Sun.COM>
>> Darren.Reed at Sun.COM wrote:
>>
>>> Could "ifconfig" be modified to report all network
interfaces that
>>> are assigned to a zone?
>>
>> I assume you mean in the global zone; ifconfig -a inside a zone 
>> (global or not) does report all the network interfaces that are 
>> configured.
> 
> Yes I do, and more specifically, I was referring to the shared
> stack model, not the exclusive stack model.
OK.

IP Instances doesn''t change that part.

    Erik
> I can''t see any reason why or for ifconfig, in the global zone,
should
> report on interfaces that belong to other zones where they are using
> the exclusive zone model.
> 
> Darren
>

Eric Enright wrote:
> I''d like to express interest in this as well.  Just last week I
came
> across the need for this, and was disappointed to learn that it (or
> something similar) is not there.
Would
	zoneadm list -l
as specified (with example output) in
  http://www.opensolaris.org/os/project/crossbow/Docs/si-interfaces.pdf

be sufficient?

    Erik

Erik Nordmark writes:
> Jeff Victor wrote:
> > Here''s one reason: consistency.  All users in the GZ can see
some
> > inforamtion about non-global zones (e.g. "ps").  Privileged
GZ users can
> > see all info about non-global zones, and need to do so in order to 
> > manage them.
> 
> But the exclusive-IP behavior is quite different from the shared-IP 
> behavior; it offers complete IP isolation between different zones/IP 
> instances.
I don''t think that argument works on two counts.  First, exclusive-IP
behavior does not offer complete IP isolation, because you can''t (for
instance) install your own copy of Firewall-1 or Cisco VPN into a
non-global exclusive-IP zone.  Some things do still require global
zone administration.  Second, "ps" shows processes that the user in
the global zone cannot ''administer'' by way of kill(2), so they
are at
least as isolated as IP instances, but they''re still of interest to
global zone administrators who want a global view of the system.

All that said, I think making ifconfig list the interfaces present in
exclusive-IP zones, given the design of ifconfig, would be
prohibitively difficult.  It''d have no access to the DLPI nodes, which
is where it gets some of its information, and the ioctls it uses for
tunnels and the like won''t work well if the zones have independent
control of the interfaces.  (It''d work "for now," but I think
it''d end
up representing more confusion with Clearview, as there''d be no easy
way to coordinate interface names across multiple zones, so
ifta_lifr_name would be ambiguous.)

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

> If we want any form of internal consistency, wouldn''t we also need
to change
> were we assign datalink names from zonecfg to dladm?
>
> Thus no more ''net'' resource in zonecfg for exclusive-IP
zones, but instead
> some
> 	dladm set-zone zoneA bge1
>
> Only having dladm show it, and not be able to affect it, seems quite 
> inconsistent.
As Dan pointed out, there are already other commands such as
ifconfig(1M) and mount(1M) which manipulate or observe resources
assigned to a zones so using dladm(1M) wouldn''t be that inconsistent.

I think the objection is around adding a resource-specific option to
"zoneadm list".  Rather than waiting for the "info" command,
what about
introducing a "-o <format>" option instead where
"datalinks" or
something like that can be used to indicate you wish to select the data
links of each zone as the information to be printed?

dsc

James Carlson wrote:
> I don''t think that argument works on two counts.  First,
exclusive-IP
> behavior does not offer complete IP isolation, because you can''t
(for
> instance) install your own copy of Firewall-1 or Cisco VPN into a
> non-global exclusive-IP zone. 
Agreed you can''t do that. But how does that make IP packets leak
between
different exclusive-IP zones?
Perhaps we have a different definition of what "IP isolation" means?
To me the critical property is that there is no IP packet leakage.
> Some things do still require global
> zone administration.  Second, "ps" shows processes that the user
in
> the global zone cannot ''administer'' by way of kill(2), so
they are at
> least as isolated as IP instances, but they''re still of interest
to
> global zone administrators who want a global view of the system.
I tried the kill and AFAICT root in the global zone can kill a process 
in a non-global zone:
bilen# ptree 103436
100996 gnome-terminal
   100999 csh
     101003 -csh
       103331 zlogin 49bge
         103332 -sh
           103338 csh
             103436 cat
bilen# kill 103436
(and the cat process in the 49bge zone died).
> All that said, I think making ifconfig list the interfaces present in
> exclusive-IP zones, given the design of ifconfig, would be
> prohibitively difficult.  It''d have no access to the DLPI nodes,
which
> is where it gets some of its information, and the ioctls it uses for
> tunnels and the like won''t work well if the zones have independent
> control of the interfaces.  (It''d work "for now," but I
think it''d end
> up representing more confusion with Clearview, as there''d be no
easy
> way to coordinate interface names across multiple zones, so
> ifta_lifr_name would be ambiguous.)
I agree it would be a pain to implement. zone_enter() would be one way.

But the key thing to me is the consistency between where things can be 
observed and where they can be modified.

    Erik

Erik Nordmark writes:
> James Carlson wrote:
> 
> > I don''t think that argument works on two counts.  First,
exclusive-IP
> > behavior does not offer complete IP isolation, because you
can''t (for
> > instance) install your own copy of Firewall-1 or Cisco VPN into a
> > non-global exclusive-IP zone. 
> 
> Agreed you can''t do that. But how does that make IP packets leak
between
> different exclusive-IP zones?
It doesn''t make the packets leak.  It makes the administrative
activities leak.
> Perhaps we have a different definition of what "IP isolation"
means?
> To me the critical property is that there is no IP packet leakage.
I don''t think it''s the only property.
> > Some things do still require global
> > zone administration.  Second, "ps" shows processes that the
user in
> > the global zone cannot ''administer'' by way of
kill(2), so they are at
> > least as isolated as IP instances, but they''re still of
interest to
> > global zone administrators who want a global view of the system.
> 
> I tried the kill and AFAICT root in the global zone can kill a process 
> in a non-global zone:
OK.  I must be misremembering this.  I thought the restriction was
more complex than that.
> > All that said, I think making ifconfig list the interfaces present in
> > exclusive-IP zones, given the design of ifconfig, would be
> > prohibitively difficult.  It''d have no access to the DLPI
nodes, which
> > is where it gets some of its information, and the ioctls it uses for
> > tunnels and the like won''t work well if the zones have
independent
> > control of the interfaces.  (It''d work "for now,"
but I think it''d end
> > up representing more confusion with Clearview, as there''d be
no easy
> > way to coordinate interface names across multiple zones, so
> > ifta_lifr_name would be ambiguous.)
> 
> I agree it would be a pain to implement. zone_enter() would be one way.
> 
> But the key thing to me is the consistency between where things can be 
> observed and where they can be modified.
We already have RFEs filed against other utilities because they don''t
show non-global zone activity (see, for example, CR 6369726).  I think
the lines here are pretty blurry.

In some usage models, the global zone administrator "owns"
everything.  Even if he can''t directly control things from the global
zone (and must log into the non-global zone to turn services on and
off), he wants to see a view of the system that includes everything.

In other usage models, the global zone administrator just provides
"infrastructure" and has no business looking at non-global zones.  And
we''ve had requests to lock down the global zone so it can''t
look where
it shouldn''t.

Given that there are some networking things that must be administered
in the global zone alone even when exclusive stack instances are in
use, it doesn''t seem unreasonable to me to say that the administrator
of the global zone should be able to list related information without
entering the non-global zone.

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

David.Comay at sun.com wrote:
>> If we want any form of internal consistency, wouldn''t we also
need to
>> change were we assign datalink names from zonecfg to dladm?
>>
>> Thus no more ''net'' resource in zonecfg for
exclusive-IP zones, but
>> instead some
>>     dladm set-zone zoneA bge1
>>
>> Only having dladm show it, and not be able to affect it, seems quite 
>> inconsistent.
> 
> As Dan pointed out, there are already other commands such as
> ifconfig(1M) and mount(1M) which manipulate or observe resources
> assigned to a zones so using dladm(1M) wouldn''t be that
inconsistent.
Yes, but those provide for manipulation (aka change) and observability 
in the same place.

 > Rather than waiting for the "info" command, what about
 > introducing a "-o <format>" option instead where
"datalinks" or
 > something like that can be used to indicate you wish to select the data
 > links of each zone as the information to be printed?

Wouldn''t this set a precedent that other things be available as -o?
E.g. -o uuid, -o brand, ...

Only introducing ''-o datalinks'' and never introducing any
other -o
specifiers bit instead moveingto an info subcommand for subsequent ones 
seems odd.

If you commit to adding other -o specifiers in short order, then I can 
see the benefit of introducing ''-o datalinks'' as part of IP
Instances.
If not, then this seems like a dead end to me.

    Erik

James Carlson wrote:
> Erik Nordmark writes:
>> But the key thing to me is the consistency between where things can be 
>> observed and where they can be modified.
> 
> We already have RFEs filed against other utilities because they
don''t
> show non-global zone activity (see, for example, CR 6369726).  I think
> the lines here are pretty blurry.
> 
> In some usage models, the global zone administrator "owns"
> everything.  Even if he can''t directly control things from the
global
> zone (and must log into the non-global zone to turn services on and
> off), he wants to see a view of the system that includes everything.
Do you have an example of that?
> In other usage models, the global zone administrator just provides
> "infrastructure" and has no business looking at non-global zones.
And
> we''ve had requests to lock down the global zone so it
can''t look where
> it shouldn''t.
I know the about is quite blurry - I sure wish zone administration was 
more self-consistent.
> Given that there are some networking things that must be administered
> in the global zone alone even when exclusive stack instances are in
> use, it doesn''t seem unreasonable to me to say that the
administrator
> of the global zone should be able to list related information without
> entering the non-global zone.
ifconfig displays network interface names used by IP and IP addresses 
and related information.

zoneadm list -l displays the datalink names assigned to an exclusive-IP 
zone.

Are you saying that the datalink names are insufficient for the 
administration the global zone would need to do for the exlusive-IP zone?

There are things external to the system (such a firewalls) that might 
need to be configured with IP addresses, and I can see the same thing 
being true for the global zone (e.g. the global zone might run a 
firewall in front of the non-global zone down the road).
But I don''t see that particular type of configuration as an argument
for
being able to do ifconfig -a in the global zone and see the non-global 
information, any more than there being a requirement for a router 
outside the system being able to do ifconfig -a and see the IP 
configuration of other systems on the network.

Thus I am trying to understand what the architectural or design 
principle is that makes you conclude that showing IP address 
configuration for exclusive-IP zones in ifconfig in the global zone.

     Erik

Erik Nordmark writes:
> James Carlson wrote:
> > Erik Nordmark writes:
> 
> >> But the key thing to me is the consistency between where things
can be
> >> observed and where they can be modified.
> > 
> > We already have RFEs filed against other utilities because they
don''t
> > show non-global zone activity (see, for example, CR 6369726).  I think
> > the lines here are pretty blurry.
> > 
> > In some usage models, the global zone administrator "owns"
> > everything.  Even if he can''t directly control things from
the global
> > zone (and must log into the non-global zone to turn services on and
> > off), he wants to see a view of the system that includes everything.
> 
> Do you have an example of that?
I''m not sure I understand the question.  Is CR 6369726 a suitable
example?  If not, then what are you asking?
> > Given that there are some networking things that must be administered
> > in the global zone alone even when exclusive stack instances are in
> > use, it doesn''t seem unreasonable to me to say that the
administrator
> > of the global zone should be able to list related information without
> > entering the non-global zone.
> 
> ifconfig displays network interface names used by IP and IP addresses 
> and related information.
> 
> zoneadm list -l displays the datalink names assigned to an exclusive-IP 
> zone.
> 
> Are you saying that the datalink names are insufficient for the 
> administration the global zone would need to do for the exlusive-IP zone?
Yes, if the administrator of the global zone needs to know what
networks are involved in order to make a change.

For example, if the administrator of the global zone has Firewall-1
installed, he''s going to need to configure IP details in the global
zone.  I don''t see how he can do that if he doesn''t have
access to
them.
> There are things external to the system (such a firewalls) that might 
> need to be configured with IP addresses, and I can see the same thing 
> being true for the global zone (e.g. the global zone might run a 
> firewall in front of the non-global zone down the road).
Right.
> But I don''t see that particular type of configuration as an
argument for
> being able to do ifconfig -a in the global zone and see the non-global 
> information, any more than there being a requirement for a router 
> outside the system being able to do ifconfig -a and see the IP 
> configuration of other systems on the network.
It depends on the administrator''s mental model for the system.

Seeing IP addresses for other systems on the same network would not be
a rational thing to expect out of ifconfig.  Seeing IP addresses
configured inside the very same box on the very same running kernel,
though, may well be a rational expectation.
> Thus I am trying to understand what the architectural or design 
> principle is that makes you conclude that showing IP address 
> configuration for exclusive-IP zones in ifconfig in the global zone.
The design principle is that we have different tools for different
tasks, and those tools are expected to give you a coherent view of the
system.  I do expect to use the resource control commands to control
resources that I''m going to delegate to zones.  I do expect to be able
to view mount points and devices in use on the local system using
tools designed for those purposes, regardless of what zone is
involved.

I don''t expect to use one tool to list datalinks and IP interfaces
when they''re in my local zone (dladm, ifconfig) and a completely
different tool when they''re in the same system but in some other
zone.  That''s not the model we had been using, which is why ifconfig
currently does show interfaces in non-global zones, even though you
can''t "use" those interfaces in the global zone.

The exception is where zonecfg sets up boundary conditions for a given
zone, and thus must take on tasks from other subsystems, but even in
those cases, I expect the ''native'' (non-zones) tools to work
as well
with those resources as with ones in the global zone.

In fact, I don''t see what it has to do with zoneadm.  The question is
whether there''s a single global view of the resources on the system.
If the answer is "no," then I suspect we''ll end up needing to
find a
way to provide that view, because that hasn''t been a broadly accepted
answer in the past.

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

>> As Dan pointed out, there are already other commands such as
>> ifconfig(1M) and mount(1M) which manipulate or observe resources
>> assigned to a zones so using dladm(1M) wouldn''t be that
inconsistent.
>
> Yes, but those provide for manipulation (aka change) and observability in
the
> same place.
However, dladm(1M) is still the place (from the global zone) where such
datalinks are instantiated, right?  For example, isn''t that the method
in which VNICs are created is through dladm?  Or aggregated links?
>> Rather than waiting for the "info" command, what about
>> introducing a "-o <format>" option instead where
"datalinks" or
>> something like that can be used to indicate you wish to select the data
>> links of each zone as the information to be printed?
>
> Wouldn''t this set a precedent that other things be available as
-o?
> E.g. -o uuid, -o brand, ...
Yes, it would.
> Only introducing ''-o datalinks'' and never introducing any
other -o specifiers
> bit instead moveingto an info subcommand for subsequent ones seems odd.
>
> If you commit to adding other -o specifiers in short order, then I can see 
> the benefit of introducing ''-o datalinks'' as part of IP
Instances. If not,
> then this seems like a dead end to me.
Although there isn''t an approved case on this yet, I believe there has
been some general agreement that such an option may be useful
(especially since the default "list -v" output is getting rather
wide).

Can we hold off on introducing the "list -l" functionality from this
design until we can resolve how to deal with the more general
observability issue?  Perhaps either through a "zoneadm info" or a
"zoneadm list -o" mechanism?

dsc

>> I tried the kill and AFAICT root in the global zone can kill a process
>> in a non-global zone:
>
> OK.  I must be misremembering this.  I thought the restriction was
> more complex than that.
Within the global zone, the ability to kill a process in a non-global
zone is controlled by the "proc_zone" privilege.  Normally, only a
user
with all privileges will have this ability unless modified via RBAC.

dsc

David.Comay at Sun.COM writes:
> >> I tried the kill and AFAICT root in the global zone can kill a
process
> >> in a non-global zone:
> >
> > OK.  I must be misremembering this.  I thought the restriction was
> > more complex than that.
> 
> Within the global zone, the ability to kill a process in a non-global
> zone is controlled by the "proc_zone" privilege.  Normally, only
a user
> with all privileges will have this ability unless modified via RBAC.
Thanks.  ;-}

I _knew_ it wasn''t as simple as killing a process in the global zone.

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

James Carlson wrote:
>>> In some usage models, the global zone administrator
"owns"
>>> everything.  Even if he can''t directly control things from
the global
>>> zone (and must log into the non-global zone to turn services on and
>>> off), he wants to see a view of the system that includes
everything.
>> Do you have an example of that?
> 
> I''m not sure I understand the question.  Is CR 6369726 a suitable
> example?  If not, then what are you asking?
Sorry, I misread "want" as "need" in the sense of being a
show-stopper.
> For example, if the administrator of the global zone has Firewall-1
> installed, he''s going to need to configure IP details in the
global
> zone.  I don''t see how he can do that if he doesn''t have
access to
> them.
Sure. But that is analogous to the external firewall.
We could decide that we want zones/containers/domains on the same system 
to be different, but I think there is value in following a network model 
for network components. After all the network is the computer(tm) ;-)
> It depends on the administrator''s mental model for the system.
Agreed. My point is that the model for an exclusive-IP zone is different 
in important aspects than the shared-IP zones.

We could try to hide this by pretending that (parts of) ifconfig 
behavior is the same, but I''m far from certain that is a good idea.

But the suggestion (made at PSARC) to use dladm to both
  - assign datalink names to zones
and
  - observe them (in e.g. show-link)

is one which satisfies the consistency between manipulation and 
observation. (And zonecfg can specify things as well; dladm can be used 
to manipulate and observe the running state.)

    Erik

Erik Nordmark writes:
> > It depends on the administrator''s mental model for the
system.
> 
> Agreed. My point is that the model for an exclusive-IP zone is different 
> in important aspects than the shared-IP zones.
And in other important aspects it''s the same: it''s still a
single
shared kernel and a single set of hardware resources.

For that reason, I don''t believe it''s entirely wrong for users
to want
to be able to answer questions such as "what addresses are being used
by this node?"

There''s a fair argument to be had for a _fully_ virtualized
environment such as Domains or a paravirtualized one such as Xen that
the global view either doesn''t exist or is "hard" to obtain. 
I think
it''s very much harder for that same argument to hold when
we''re
talking about a single instance of Solaris -- no matter how many
network stacks are involved.

Anyway, as I said at the beginning, I think making ifconfig work this
way would be very hard to do, and likely would not work well.  Though
users often think of ifconfig as the sole way to interact with
networking interfaces (because that''s the way it works everywhere but
Solaris), I don''t think that''s reasonably doable here, even if
it''s
something that might be wanted.
> We could try to hide this by pretending that (parts of) ifconfig 
> behavior is the same, but I''m far from certain that is a good
idea.
> 
> But the suggestion (made at PSARC) to use dladm to both
>   - assign datalink names to zones
> and
>   - observe them (in e.g. show-link)
> 
> is one which satisfies the consistency between manipulation and 
> observation. (And zonecfg can specify things as well; dladm can be used 
> to manipulate and observe the running state.)
Right.  The difference is that the zonecfg is just behaving as a
repository for configuration that properly "belongs" to some other
subsystem, rather than behaving as the configuration tool itself.

(Yeah, there''s a fuzzy line here as well.)

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

From: "James Carlson" <James.D.Carlson at
Sun.COM>
> Erik Nordmark writes:
>> > It depends on the administrator''s mental model for the
system.
>>
>> Agreed. My point is that the model for an exclusive-IP zone is
different
>> in important aspects than the shared-IP zones.
>
> And in other important aspects it''s the same: it''s still
a single
> shared kernel and a single set of hardware resources.
>
> For that reason, I don''t believe it''s entirely wrong for
users to want
> to be able to answer questions such as "what addresses are being used
> by this node?"
...
> Anyway, as I said at the beginning, I think making ifconfig work this
> way would be very hard to do, and likely would not work well.  Though
> users often think of ifconfig as the sole way to interact with
> networking interfaces (because that''s the way it works everywhere
but
> Solaris), I don''t think that''s reasonably doable here,
even if it''s
> something that might be wanted.
So how would you propose a system administrator, in the global
zone, bring together all of the network addresses configured for
his machine?

Does this now require zlogin and a loop through all of the zones
that are currently booted?

Will this be "too hard" for those that want to do it?

Thinking on this, would it be "nice" if there was some special command
line switch for the likes of netstat/ifconfig and others that caused them to
iterate through all of the present stack instances and report on them?

Darren

David.Comay at Sun.COM wrote:
> Erik,
> 
> Here are my belated comments on the IP Instances design.
And here are my belated responses. But we''ve already acted on the 
comments that affect the design and code, and I''ll make sure the Zones 
documentation covers the other documentation items.
>> There are two documents which describe the design
>>     si-interfaces - a high-level design focusing on the problem the
>>     project solves, and what the user-visible changes are
> 
> A general comment that in both documents page numbers seem to be
> missing.
Good point.

But I suspect the remaining shelf-life of the design document is very
limited. I''ll definitely take your comments on missing pieces in the
documents as strong hints that we should ensure that the zones book be
clear on these points.
> Figures 1 and 2 - I think the pictures are a little misleading the
> second picture displays an example of two applications (web servers?)
> both binding to INADDR_ANY on port 80.  Of course, that''s also a
very
> viable example for the current zones model.  Perhaps a better example
> might be one of those applications designated as "Internet
facing" and
> another might be "Intranet-facing" or "Internal".
Figure 1 shows how zones can be used today.
Figure 2 shows what customers want from IP-level isolation between VLANs 
(and the white space between the two instances of IP is critical).
Clearly zones today doesn''t provide that isolation.
Even if we ignore that (a bit foolish it might be), in figure two the 
two different zones use different IP subnets and as a side effect they 
need to use different (default or otherwise) routes.
AFAIK we don''t actually claim to support different routes for different
zones.
Do you still think this is misleading?
> Page 6, Section 4 - I''m not sure if this is the proper place for a
> discussion on security or if it warrants a separate section, but I
> think the document should discuss the security implications of a zone
> using a shared stack versus an exclusive stack.  For example, I think
> including the table you had sent in private email some time ago which
> documented the various attack vectors and how each type of zone
> addresses the type of attack would be helpful in understanding the
> trade-offs.
We''ll put this in the zones book. We need to describe the tradeoffs 
between using shared-IP and exclusive-IP zones there, and the network 
security differences is one part of that.
> Page 6, Section 5 - In the second paragraph, you mention there are no
> planned layer two changes.  Does that mean for the initial design that
> there will be no filtering done at that layer?  Again, perhaps this
> merits discussion in a separate "Security" section but
it''s important
> to understand what sorts of capabilities a privileged user in an
> exclusive stack zone has versus one in a zone using the shared stack.
Correct. I''ll make sure the zones book covers this topic.
> Page 7, Section 6 - In #4, leveraging zonename(1) in this way does seem
> strange.  I know we discussed at one time a command to retrieve and
> interpret the values of ZONE_ATTR_* kernel attributes although in this
> particular case, it seems the only one that makes sense from inside the
> zone itself is whether this zone is tied to an exclusive stack.
We''ve discussed this in a separate email thread.
As your college indicated given that this is a private interface it 
isn''t a big deal. If some new general command for getting ZONE_ATTR_*
is
created we can just switch to using that instead. (The only place which 
calls zonename -t is in smf_include.sh thus it would be easy to change 
when that day comes.)
> What about using /sbin/netstrategy and smf_netstrategy() in
> /lib/svc/share/smf_include.sh for this purpose?
Overloading them would probably be a mistake. We also have Xen adding 
its own mechanisms to net-physical. Once we get a perspective across all 
of the virtualization technologies then we can try to refactor how we do 
configuration.
> Page 7, Section 7 - Given the recent discussion with a customer, would
> it make sense to make it clear here that there is a *single* shared IP
> and the project does not allow for "islands" of shared IP
instances?
Will do in the zones book.
> Page 8, Section 9 - When does the enforcement of only allowing a
> "physical" property for exclusive IP zones?  Does it occur when
one
> tries to complete the addition of a "net" resource by specifying
"end"
> subcommand?  Or does it occur when "verify" is performed on the
whole
> configuration (either the explicit command or the implicit verification
> that takes place prior to commit?)
The latter. (It is done is the verify command in zonecfg.)
> In the example given for the shared IP zone, I would specify add the
> /24 prefix length to the address parameter (it''s not required but
> definitely encourage it) and also the missing "end" subcommand.
> 
> In the example given for the exclusive IP zone, there is also a missing
> "end" subcommand.
I''ll make sure the examples in the zones book are correct on these
points.
> Page 9, Section 10 - BrandZ added their BRAND column after the existing
> PATH column but you''re introducing it before.  It seems the
"IP TYPE"
> should follow both PATH and BRAND.  Also, updating the output for first
> two "list" examples to account for the merge with BrandZ would be
> helpful.
IP (a short form of IP TYPE) is at the end of the line.
> In the third example, the command listed is "list -i" but earlier
"-l"
> is defined for this purpose.
Noted.
> With respect to "list -l", I''m not sure if this really
belongs in
> zoneadm(1M) or somewhere else.  In some ways, I''d rather see it as
a
> new option to dladm(1M) instead.
We concluded with PSARC that adding support for this to dladm (as a 
property) makes more sense.
Thus there is no more zoneadm list -l.
> Will the list of link names printed be the ones actually registered
> with the kernel though zone_add_ifname() or the subset of those link
> names actually plumbed by the zone with the exclusive stack?  I assume
> the former but please verify that.
Even with dladm this doesn''t change. It is the list that has been 
granted to the zone, whether or not the zone is using them.
> In this respect, it seems that "list -l" is printing one of that
> several pieces of zone runtime state which we don''t currently
print
> through zoneadm(1M) or any other command.  The zones team has discussed
> in the past introducing a more general "zoneadm status" or
"zoneadm
> info" facility but that''s not yet designed.  Another
alternative is to
> begin supporting a "list -o <format>" in zoneadm(1M) and
allow the link
> names to be a supported type there.
This comment might be important for the future direction. What I got 
from PSARC was the strong opinion that such observation would be done 
with sub-system specific commands. Thus mount(1m) should be used to 
observe that is mounted in a zone, pool* to observe what the running 
pool configuration is for a zone, etc.

If we follow that advice going forward, then zoneadm status/info would 
just be for purely zones specific information i.e. information 
maintained by the zones subsystem and no other place.
> Page 9, Section 11 - As I mentioned earlier, zonename(1) seems the
> wrong interface for obtaining this information.  What about using
> /sbin/netstrategy and smf_netstrategy()?
See above.
> Page 9, Section 12 - There should be some discussion of the packaging
> changes somewhere in one or both of the documents.  In the particular
> case of IP Filter, is the proposal to move all of the contents of
> SUNWipfr into a non-hollow package?
Packaging changes are important, but not central to the design.

After the pfhooks putback SUNWipfr no longer contain kernel components, 
so for Nevada are just changing that (as well as SUNWcnetr) to no longer 
be HOLLOW. (Packaging for S10 updates warrants a separate thread.)
> Page 15, Section 14 - This is more of a code review question but are
> the additional privileges being granted to an exclusive stack zone
> being hard-coded in either libzonecfg or zoneadmd(1M) or are you
> extending the XML uses for each brand?
Hard coded. There is nothing brand specific about what privileges would 
be granted.
The devices are brand specific, so for that we have extended the brand 
XML syntax.
> Page 16, Section 15 - Is the additional configuration that is being
> introduced part of this case or a future case?  If the latter, how will
> this project and TX interact?
For updated discussion of TX see section 19 in the PSARC commitment 
materials (also available at 
http://www.opensolaris.org/os/project/crossbow/Docs/si-interfaces.pdf )
> Page 16, Section 17 - Could you provide a bit more detail on each of
> these interfaces?  In particular, does zone_get_ifnum() return the
> number of link names registered via zone_add_ifname()?  Is the second
> parameter of zone_get_iflist() a single string containing the link
> names registered in a concatenated form?
Best done by reviewing the code.
But the answers are
	Yes
	No, it is an array of chat[LIFNAMSIZ].

And as Jerry pointed out in his code review, we can drop the ifnum 
function since the iflist function returns the number of entries already.
> More generally, can zone_getattr() be used in place of these
> zone_get_if*() functions?
Not easily.
If we just look at the "get" part, there isn''t support in
zone_getattr
to handle an array of structured elements of unknown size - all existing 
attributes are either fixed size or have a fixed upper bound. Thus one 
would have to add a way to determine the size of the attribute. And 
since one also wants to handle the size changing between the "get size 
of X" and "get X", one might need to have "get X" be
able to return the
current size (as we do in zone_get_iflist()).

If we were to solve this, then the next question would be why not to use 
zone_setattr() for setting it. That one would be even more problematic.
In order to support dladm''s model of zone as a property of a datalink 
name, we need to be able to do add/remove and not just a replacement of 
the whole list. (One can in theory use get+set plus some new 
zone_attr_lock/unlock to implement atomic add/remove, but now we seem to 
adding way too much complexity.)

For those reasons it doesn''t seem worth-while to try to force-fit this 
into zone_getattr/setattr.
> Should these functions be dealing with "link names" rather than
> "interfaces" to avoid confusion with the interfaces that
ifconfig(1M)
> deals with?
Good point.
We''ve renamed them to have "datalink" in their names.
Jerry also pointed out that we should be more careful about terminology 
like "ifname" and "interface name" in the code, so we have
done that as
well. (Basically the general concept which corresponds to physical=bge0 
is something we call "network interface (name)". Those could be for 
shared-, or exclusive-IP zones. But when it comes to adding a datalink 
name for an exclusive IP zone we call the zone_add_datalink() syscall.)

>>     si-design - the design of the code changes. Assumes the reader
>>     has read si-interfaces already!
> 
> Pages 14, Section 7.1 - More of a code review comment but please update
> truss(1) to account for the additional argument to zone_create().
Good point (and done).
> Pages 14-15, Section 7.2 - As mentioned above, could you please provide
> some more details on these new system calls?  Can zone_getattr() be
> used instead of the proposed zone_get_if*() functions?
See above.
> Also, could you list the changes to the proposed zone_t structure?
OK. FWIW it is
+       /*
+        * zone_iflist is protected by zone_lock
+        */
+       struct ifnamelist *zone_iflist;
+       netstack_t      *zone_netstack;
+
> With regard to the third bullet, please see my concerns above about the
> introduction of "list -l".  I think this should be part of a
general
> zone status/health facility or perhaps something that dladm(1M) can
> print about the link names and how their assignment zone-wise.
See above.
> With respect to the fourth bullet, can the global zone plumb an
> interface which has been "assigned" to an exclusive IP zone but
not yet
> plumbed by that zone?
The "used" is a typo. The semantics are "assigned".
> Page 15, Section 7.3 - As I asked about in the earlier document, is
> "zonecfg verify" the only place where address property
verification is
> done or is it also done at end of adding or changing a "net"
resource?
See above. I''ll update the design document.
> And as I asked earlier, are the additional privileges and devices being
> hard-coded intop libzonecfg or zoneadm(1M) or are you extending the XML
> uses for each brand?
> 
> Page 16, Section 9 - Providing the table of attack vectors here along
> with some discussion would be very helpful.
Will do that in an appropriate form in the zones book.

David.Comay at Sun.COM wrote:
> Erik,
>
> One additional comment I meant to include is that I think it would be
> useful to add a paragraph on what is possible today with the current
> stack in terms of sharing a link versus what will be possible with IP
> instances (using separate physical NICs or VLANs) versus what will be
> possible once VNIC support is introduced.  Once VNICs are available,
> there will be a lot more flexibility but until then, the use of IP
> instances will be limited by the physical hardware or the use of
> VLANs.

Will do that in an appropriate form in the zones book.

    Erik

> > Should these functions be dealing with "link names" rather
than
 > > "interfaces" to avoid confusion with the interfaces that
ifconfig(1M)
 > > deals with?
 > 
 > Good point.
 > We''ve renamed them to have "datalink" in their names.
 > Jerry also pointed out that we should be more careful about terminology 
 > like "ifname" and "interface name" in the code, so we
have done that as
 > well. (Basically the general concept which corresponds to physical=bge0 
 > is something we call "network interface (name)". Those could be
for
 > shared-, or exclusive-IP zones. But when it comes to adding a datalink 
 > name for an exclusive IP zone we call the zone_add_datalink() syscall.)

Glad to hear this.  Consistent and correct use of these terms will be
increasingly important given upcoming technologies like datalink vanity
naming and VNICs (among others).

--
meem