xen discuss - Jan 2007 - IPv4 configuration for Solaris guest domains

I''d like to enhance Solaris network configuration slightly to allow
guest domain IPv4 details to be passed from the domain building
control tools through to the relevant components in Solaris
(svc:/network/physical and uts/common/fs/nfs/nfs_dlinet.c).

Attached is a proposal, roughly in the form of a fasttrack.

Diffs for a prototype are at:

      http://www.dme.org/solaris/xen-net-config/

The diffs are broadly against nv56.  The bootloader changes are not
included in the diff - shout if you really want to see them.

I''ve set "followup-to" to "xen-discuss" but
I''ll monitor the other
lists as well.




dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

How can this proposal use xen- as a prefix for boot properties yet we 
can''t use a similar naming in the SMF FMRI''s for the daemons ?

This is inconsistent application of "not using the xen word" in user 
interfaces.

--
Darren J Moffat

Is the /sbin/devprop utility delivered on both SPARC and x86 it seems to 
be generally useful outside of this case and you are proposing it as a 
Committed interface.

--
Darren J Moffat

* Darren.Moffat@Sun.COM [2007-01-23 14:32:32]
> How can this proposal use xen- as a prefix for boot properties yet we
> can''t use a similar naming in the SMF FMRI''s for the
daemons ?
>
> This is inconsistent application of "not using the xen word" in
user
> interfaces.
I''m happy to change it to "xpv".

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

* Darren.Moffat@Sun.COM [2007-01-23 14:35:22]
> Is the /sbin/devprop utility delivered on both SPARC and x86
Yes.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

David Edmondson wrote:
> * Darren.Moffat@Sun.COM [2007-01-23 14:35:22]
>> Is the /sbin/devprop utility delivered on both SPARC and x86
> 
> Yes.
Cool, if that was stated in the materials and I just missed it oops from 
me, if not please make it explicit when the case is submitted.

-- 
Darren J Moffat

* Darren.Moffat@Sun.COM [2007-01-23 14:47:20]
> David Edmondson wrote:
>> * Darren.Moffat@Sun.COM [2007-01-23 14:35:22]
>>> Is the /sbin/devprop utility delivered on both SPARC and x86
>>
>> Yes.
>
> Cool, if that was stated in the materials and I just missed it oops
> from me, if not please make it explicit when the case is submitted.
It''s not stated sorry (I thought that it was implicit), it''s
added
now.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

Hi David,

David Edmondson wrote:
>I''d like to enhance Solaris network configuration slightly to allow
>guest domain IPv4 details to be passed from the domain building
>control tools through to the relevant components in Solaris
>(svc:/network/physical and uts/common/fs/nfs/nfs_dlinet.c).
>
>Attached is a proposal, roughly in the form of a fasttrack.
>  
>
I''d like to make a suggestion or two...

> ...
>
>| Variable   | Type   | Meaning                           | Examples        
|
>|------------+--------+-----------------------------------+-----------------------------|
>| ip         | string | IPv4 address of the domain        |
"10.6.39.40"                |
>| netmask    | string | netmask of the domain             |
"255.255.255.0"             |
>| gateway    | string | default router for the domain     |
"10.6.39.1"                 |
>| dhcp       | string | is DHCP enabled?                  | "on"
"off"                  |
>| vif        | list   | network interfaces specifications | see below       
|
>| nfs_root   | string | pathname of NFS root filesystem   |
"pushrod:/export/root/xen1" |
>| nfs_server | string | IPv4 address of the NFS server    |
"10.6.39.80"                |
>  
>
Can we expand this to also cover peer-to-peer connections,
such as tunnels, etc?

I''ll continue reading the rest of it later (timeout interrupt)..

Darren

David Edmondson wrote:
> I''d like to enhance Solaris network configuration slightly to
allow
> guest domain IPv4 details to be passed from the domain building
> control tools through to the relevant components in Solaris
> (svc:/network/physical and uts/common/fs/nfs/nfs_dlinet.c).
> 
> Attached is a proposal, roughly in the form of a fasttrack.
I''m confused by your statement that the DHCP configuration support is 
only for diskless systems (and tied to OBP). The DHCP support in 
network/physical works for diskless systems - x86 and sparc.
Are you talking about diskless Xen domains using DHCP and NFS?

Is the proposal a short-term approach or something we should live with 
forever?

I personally think it makes sense to be able to convey the IP 
configuration to a domU using the standard DHCP protocol; for static 
configuration from the perspective of dom0 this means that the DHCP 
server runs in dom0 and serves up that static information to the domU. 
(To do that we need some intercept of the DHCP packets down in the 
vnic/gld layer, thus it can''t be the short-term solution.)

    Erik

David Edmondson wrote:
> I''d like to enhance Solaris network configuration slightly to
allow
> guest domain IPv4 details to be passed from the domain building
> control tools through to the relevant components in Solaris
> (svc:/network/physical and uts/common/fs/nfs/nfs_dlinet.c).
> 
> Attached is a proposal, roughly in the form of a fasttrack.
I forgot to ask: what about IPv6?

    Erik

* Erik.Nordmark@Sun.COM [2007-01-23 20:30:29]
> David Edmondson wrote:
>> I''d like to enhance Solaris network configuration slightly to
allow
>> guest domain IPv4 details to be passed from the domain building
>> control tools through to the relevant components in Solaris
>> (svc:/network/physical and uts/common/fs/nfs/nfs_dlinet.c).
>>
>> Attached is a proposal, roughly in the form of a fasttrack.
>
> I''m confused by your statement that the DHCP configuration support
> is only for diskless systems (and tied to OBP).  The DHCP support in
> network/physical works for diskless systems - x86 and sparc.
Sorry, I missed the fact that support for setting bootp-response based
on grub provided data had been added.  I''ll update the proposal.
> Are you talking about diskless Xen domains using DHCP and NFS?
Right now that is not supported, as neither the kernel nor the domain
building code are able to perform the DHCP discovery necessary to find
a local IP address to use to mount the root filesystem.

As you suggest below we could add that support to do the discovery in
the domain building tools and pass the data on.  That was not part of
this proposal.
> Is the proposal a short-term approach or something we should live
> with forever?
The mechanism for specifying parameters in the guest domain
configuration file is expected to be long lived.

The details of how those parameters are passed through to Solaris can
be changed, with the ease of change increasing further along the
pipeline (i.e. changing the Xen tools requires external interaction,
changing the kernel implementation is relatively easy, changing the
service scripts is easier still).
> I personally think it makes sense to be able to convey the IP
> configuration to a domU using the standard DHCP protocol;
I agree.  However, using the static configuration mechanism is very
common in the existing Xen community.

The domU can run a DHCP client itself - this works today if
/etc/dhcp.xnf0, etc. are present inside the domain.

The proposal would also allow the dom0 administrator to indicate that
the domU should run a DHCP client in the absence of any other
configuration data within the domain.

Without implementing the bootp-response pass through, DHCP with NFS
root would not work (only static or RARP).  Linux avoids this by
providing a simple DHCP client within the kernel.
> for static configuration from the perspective of dom0 this means
> that the DHCP server runs in dom0 and serves up that static
> information to the domU.
This is possible but not covered by the proposal.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

* Erik.Nordmark-UdXhSnd/wVw@public.gmane.org [2007-01-23
22:04:27]
> David Edmondson wrote:
>> I''d like to enhance Solaris network configuration slightly to
allow
>> guest domain IPv4 details to be passed from the domain building
>> control tools through to the relevant components in Solaris
>> (svc:/network/physical and uts/common/fs/nfs/nfs_dlinet.c).
>>
>> Attached is a proposal, roughly in the form of a fasttrack.
>
> I forgot to ask: what about IPv6?
The current Xen tools don''t support declaring any IPv6 related
details.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

On Tue, Jan 23, 2007 at 01:46:07PM +0000, David Edmondson
wrote:
> I''d like to enhance Solaris network configuration slightly to
allow
> guest domain IPv4 details to be passed from the domain building
> control tools through to the relevant components in Solaris
> (svc:/network/physical and uts/common/fs/nfs/nfs_dlinet.c).
>
> Attached is a proposal, roughly in the form of a fasttrack.
>
> Diffs for a prototype are at:
>
>       http://www.dme.org/solaris/xen-net-config/
>
> The diffs are broadly against nv56.  The bootloader changes are not
> included in the diff - shout if you really want to see them.
>
> I''ve set "followup-to" to "xen-discuss" but
I''ll monitor the other
> lists as well.
>
Content-Description: xen-ip-config.txt
>          IPv4 Network Configuration Enhancements for Xen Guest Domains
>          ============================================================>
> Author: David Edmondson <dme@sun.com>
> Date: 2007/01/23 13:24:31
>
> $Id: xen-ip-config.org,v 1.1 2007/01/23 13:24:08 dme Exp $
>
>
...
> 7 /sbin/devprop
> ###############
>
> : Usage: devprop [-n device-path] [-vq] [-{b|i|l|s}] [property [...]]
>
is there any reason that you couldn''t just add this functionality
to prtconf?  prtconf already accepts a device path.  all you''d have to
do is add a new parameter to specify property names.  course i''m not
sure what all the other parameters to devprop above do so i don''t
know if they map to existing prtconf functionality.

ed

* Darren.Reed@Sun.COM [2007-01-23 17:13:27]
> Can we expand this to also cover peer-to-peer connections,
> such as tunnels, etc?
Sure.  That is not part of this proposal though, which aims to support
the existing Xen tools mechanism.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

* edward.pilatowicz@sun.com [2007-01-24 01:56:10]
>> 7 /sbin/devprop
>> ###############
>>
>> : Usage: devprop [-n device-path] [-vq] [-{b|i|l|s}] [property [...]]
>>
>
> is there any reason that you couldn''t just add this functionality
to
> prtconf?  prtconf already accepts a device path.  all you''d have
to
> do is add a new parameter to specify property names.  course i''m
not
> sure what all the other parameters to devprop above do so i don''t
> know if they map to existing prtconf functionality.
I could do this.  I''ll look and see how it would fit.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

Dave,
> Xen guest domain configuration files are fragments of Python[1] code.
> Certain variable names are "well known" to the configuration
> infrastructure and can be set to control the creation of the guest
> domain.
So is the intent that these Xen configuration files are used to control
the IP configuration of the Solaris domUs at each domU boot?  If this
is the case, what happens during the Solaris installation (or following
a sys-unconfig) when the domU boots up and "ifconfig -a plumb" returns
one or more of the interfaces specified in the "vif" property?  Will
the sysidtool(1M) framework prompt the user to configure the xnf*
interface(s)?

I can see the advantage of having this be a property under Xen
configuration (particularly in the migration case) although as Erik
pointed out in his reply, it would really be nice if we could be using
DHCP for this as it''s my hope that we can use that to provide a (locked
down) configuration for zones with an exclusive IP stack as well.
> 3 Accessing the Configuration Data in a Guest Domain
> ####################################################
> A new utility is provided to allow the examination of system
> properties from within shell scripts (/sbin/devprop).
Given the network centric aspect of these properties, did you consider
extending the existing /sbin/netstrategy for this purpose?  Or are
there other Xen supplied properties (not related to networking) which a
guest domain may wish to retrieve from a shell script?

dsc

Firstly, I should say that this proposal arose out of a need to fit
with the existing Xen tools implementation and their common use.  I
agree that DHCP is a better approach as a bootstrap mechanism for
domain configuration and will work to improve that support, but that''s
not how the existing Xen tools and their users are wired.

* David.Comay@sun.com [2007-01-24 08:56:16]
>> Xen guest domain configuration files are fragments of Python[1]
>> code.  Certain variable names are "well known" to the
configuration
>> infrastructure and can be set to control the creation of the guest
>> domain.
>
> So is the intent that these Xen configuration files are used to control
> the IP configuration of the Solaris domUs at each domU boot?
Yes.  This mechanism is commonly used for exactly this purpose with
the existing Xen tools (and Linux).
> If this is the case, what happens during the Solaris installation
> (or following a sys-unconfig) when the domU boots up and "ifconfig
> -a plumb" returns one or more of the interfaces specified in the
> "vif" property?  Will the sysidtool(1M) framework prompt the user
to
> configure the xnf* interface(s)?
I''m not sure what the right approach is in this case.

In the proposal (and the prototype) the configuration details
specified in the domain configuration are used by Solaris only if
Solaris finds no other configuration details.

If we imagine an already installed system, the presence of
/etc/hostname.xnf0 in the filesystem of the domain would mean that
svc:/network/physical would ignore the parameters specified in the
domain configuration - it would use /etc/hostname.xnf0 in preference.

So, considering the installation scenario, if sysidtool uses the
domain configuration details to "prime" the Solaris configuration
(i.e. create /etc/hostname.xnf0, etc.) then subsequent changes to the
domain configuration would not be reflected in the behaviour of
svc:/network/physical - it would continue to prefer the details
recorded in /etc/hostname.xnf0.

Of course, this is a reflection of the proposal and prototype - it
could be changed.
> I can see the advantage of having this be a property under Xen
> configuration (particularly in the migration case) although as Erik
> pointed out in his reply, it would really be nice if we could be
> using DHCP for this as it''s my hope that we can use that to
provide
> a (locked down) configuration for zones with an exclusive IP stack
> as well.
DHCP works well for the diskfull case today (with the caveat above,
that configuration within the domain filesystem overrides any
externally specified configuration).

The diskless DHCP case does not work well, as Solaris relies the
bootloader (well, some part of the boot process which runs pre-kernel)
to perform the initial DHCP discovery phase.  We would need to add the
DHCP discovery to the domain builder and arrange to create the
relevant bootp-response property.
>> 3 Accessing the Configuration Data in a Guest Domain
>> ####################################################
>
>> A new utility is provided to allow the examination of system
>> properties from within shell scripts (/sbin/devprop).
>
> Given the network centric aspect of these properties, did you
> consider extending the existing /sbin/netstrategy for this purpose?
I have some changes to /sbin/netstrategy so that it can report the
fact that IP configuration details came from boot properties, but I''m
unsure about how useful that really is.  I didn''t pursue the use of
this through all of the other services (such as
svc:/system/identity:node) yet.

netstrategy is also quite a narrow tool - it''s used to discover the
style of configuration that took place rather than the details of that
configuration.
> Or are there other Xen supplied properties (not related to
> networking) which a guest domain may wish to retrieve from a shell
> script?
Other Xen supplied properties exist today but I''m not aware of anyone
needing to use them from within shell scripts.  devprop isn''t specific
to Xen properties though, it''s possible to examine any properties from
the device tree.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

David Edmondson wrote:
> DHCP works well for the diskfull case today (with the caveat above,
> that configuration within the domain filesystem overrides any
> externally specified configuration).
> 
> The diskless DHCP case does not work well, as Solaris relies the
> bootloader (well, some part of the boot process which runs pre-kernel)
> to perform the initial DHCP discovery phase.  We would need to add the
> DHCP discovery to the domain builder and arrange to create the
> relevant bootp-response property.
Are you saying that we can''t rely on inetboot doing dhcp and passing
the
results to the kernel, but the kernel must be able to do dhcp?

Is the kernel (without inetboot) capable of doing reverse ARP/bootparams 
network boot?

(Separately from the above kernel issues there is how we''d build and 
configure the DHCP service in dom0, but that seems orthogonal.)

   Erik

* Erik.Nordmark@Sun.COM [2007-01-24 18:51:01]
> David Edmondson wrote:
>
>> DHCP works well for the diskfull case today (with the caveat above,
>> that configuration within the domain filesystem overrides any
>> externally specified configuration).
>>
>> The diskless DHCP case does not work well, as Solaris relies the
>> bootloader (well, some part of the boot process which runs pre-kernel)
>> to perform the initial DHCP discovery phase.  We would need to add the
>> DHCP discovery to the domain builder and arrange to create the
>> relevant bootp-response property.
>
> Are you saying that we can''t rely on inetboot doing dhcp and
passing
> the results to the kernel,
For Xen guest domains, yes.

Xen guest domains don''t boot via any of the previous mechanisms - the
kernel and boot archive are actually loaded from the filesystem by
some user-level code running in dom0 (the domain builder) which then
asks Xen to create a new domain based around them.  As a consequence
inetboot is never run.
> but the kernel must be able to do dhcp?
That would be an obvious solution.
> Is the kernel (without inetboot) capable of doing reverse
> ARP/bootparams network boot?
Yes.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

Updated proposal, based on feedback the last time around.

dme.