I have attempted to reverse-engineer as much of the protocol as is readily evident from examination of libxl and tools/hotplug/Linux. It''s a very poor set of protocols. We really need to fix this in 4.3. I have also adjusted the documentation of the "script=" parameters in the xlu block config string to correspond to what I understand is currently implemented. Before we freeze for 4.2 we should confirm that this is (a) compatible with xend (b) what we want. Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> --- docs/misc/xl-disk-configuration.txt | 7 +- docs/misc/xl-scripts-block.txt | 246 +++++++++++++++++++++++++++++++++++ 2 files changed, 251 insertions(+), 2 deletions(-) diff --git a/docs/misc/xl-disk-configuration.txt b/docs/misc/xl-disk-configuration.txt index 5da5e11..86c16be 100644 --- a/docs/misc/xl-disk-configuration.txt +++ b/docs/misc/xl-disk-configuration.txt @@ -160,7 +160,10 @@ script=<script> --------------- Specifies that <target> is not a normal host path, but rather -information to be interpreted by /etc/xen/scripts/block-<script>. +information to be interpreted by the executable program <script>, +(looked for in /etc/xen/scripts, if it doesn''t contain a slash). + +These scripts are normally called "block-<script>". @@ -204,7 +207,7 @@ Supported values: iscsi: nbd: enbd: drbd: In xend and old versions of libxl it was necessary to specify the "script" (see above) with a prefix. For compatibility, these four prefixes are recognised as specifying the corresponding script. They -are equivalent to "script=<script>". +are equivalent to "script=block-<script>". <deprecated-prefix>: diff --git a/docs/misc/xl-scripts-block.txt b/docs/misc/xl-scripts-block.txt new file mode 100644 index 0000000..d6a5123 --- /dev/null +++ b/docs/misc/xl-scripts-block.txt @@ -0,0 +1,246 @@ + ----------------------- + DEVICE SCRIPT INTERFACE + ----------------------- + +Guest devices can be specified in [lib]xl to be provided with the help +of a "script". + +This document describes the interface between the toolstack and device +scripts. This protocol was partially reverse-engineered from the xend +infrastructure during the development of libxl; the interface is +intended to be compatible between libxl and xend. This document +describes the protocol implemented by libxl. + +This description is applicable to Linux. + +Only block and network devices support these scripts. + +Note that we are consider deprecating this protocol after the release +of Xen 4.2, or providing a more sophisticated replacement for it. + + +============+BLOCK SCRIPTS +============+ +When a script is specifed, the <target> value from the +domain configuration (the pdev_path value in the libxl API) is not +interpreted by libxl. Instead, it is passed as an opaque parameter to +the script. + +The block is responsible for taking the pdev_path (<target> from the +xl configuration), henceforth known as the "params", and providing a +block device in the toolstack domain. + +Block scripts are optional; if not specified, the pdev_path is used as +the path of the intended block device or file in the driver domain. + + + +NETWORK SCRIPTS +==============+ +When a guest''s networking is being set up, it appears in the driver +domain as a virtual network device (a "vif" or "tap" device). + +The network script is responsible for plumbing this virtual network +device through to the driver domain''s networking arrangements. For +example, enslaving the virtual device to a suitable bridge, creating +routes to it, or whatever is necessary. + +The script is also responsible for renaming the interface as may be +desired. + + + +====+MODEL +====+ +We will define a "connection" which is the making available or a +single block or network device, relating to a single guest, to a +single facility (in the guest or toolstack or service domain). + +It may be necessary to make multiple connections for one guest device; +even multiple connections which overlap in time. + +Such connection is created by an "add" execution of the script and +destroyed by a "remove" execution. The script is invoked in the +driver domain (usually dom0). + + +========+EXECUTION +========+ +Environment +----------- + +The scripts are invoked with the following environment variables: + + script=<value specified in domain configuration> + + This may or may not be a fully qualified path. Scripts should + not use this information as the form is not reliable. + + XENBUS_TYPE=vbd [block devices] + XENBUS_TYPE=vif [network devices] + + XENBUS_PATH=<backend> + + This is a path in xenstore which is used extensively to + communicate between the script and the rest of the backend + system. This may be a relative or absolute xenstore path. + + See below for the contents and usage of this value. + + XENBUS_BASE_PATH=backend + + Do not use this value. + + vif=<devname> [network vif devices] + INTERFACE=<devname> [network tap devices] + + Current name of the virtual device in the toolstack domain. + + +Arguments +--------- + +Block scripts: + The script will get one argument, "add" or "remove" + +Network scripts: + The script will get two arguments. They vary according to + whether the script is running with respect to a vif or + a tap virtual device: + + arguments for add arguments for remove + vif device "online" "type_if=vif" "offline" "type_if=vif" + tap device "add" "type_if=tap" "remove" "type_if=tap" + + +Exit status and error handling +------------------------------ + +The script should exit zero if it is successful. If it fails it +should clean everything up, try write an error message to the +hotplug-error path (see below), and and exit 1 or >=126 or die from a +signal. + +Failure of "remove" scripts will be logged and reported but there are +no arrangements to re-invoke a failed script. + + +Locking, timeouts, concurrency +------------------------------ + +Scripts should not block indefinitely. If a script takes longer than +30s to exit it will be sent a SIGKILL which will result in the +operation being considered failed. + +The scripts may be invoked concurrently for the same or different +resources, and the same or different guests, and must do any necessary +locking. + + + +=======+XENSTORE +=======+ +The <backend> directory in xenstore contains keys and values used to +communicate between the script and the rest of the infrastructure. + +The contents of this path are shared between the script and other +parts of the infrastructure. The script should therefore not write +arbitrary subpaths for its own purposes. + +The following paths are defined: + + +All devices +----------- + + <backend>/hotplug-error + + If a script is about to fail, it may write an error message + string to this path. The string will be reported to the user. + + +Block devices +------------- + + <backend>/params + + The params. Ie, the <target> string from the configuration file, + which corresponds to the "pdev_path" in the libxl API. + + This is not interpreted by the toolstack and the script may + define whatever syntax it likes for it. Newlines, whitespace, + backslashes, quotes, and so forth, should be avoided as + specifying them from a config file may be difficult. Consult the + xl configuration API documentation for information about the + exact syntactic transformations in xl. + + The value is written by the toolstack before the add script is + run and remains present until the remove script has finished. It + should not be modified by the script. + + <backend>/mode = "r" | "w" + + The device is to be readonly or read/write, respectively. + Written by the toolstack; should not be modified. + + <backend>/physical-device + + The main result of the add script. This key does not exist on + entry to add. The add script should, if successful, write this + key with the major and minor number of the block device + (as accessible in the driver domain) in the format "%x:%x". + + +Other information that may be useful to the script is also present. +The block script does not need to pay attention to this if it doesn''t +want to. These keys include: + + <backend>/removable = "0" | "1" [block devices] + + Indicates whether the device is being presented to the guest as a + removable device. + + <backend>/device-type = "disk" | "cdrom" [block devices] + + +Network devices +--------------- + +All of these values are provided by the toolstack to the script and +should not be changed by the script: + + <backend>/vifname [network devices] + + Name that the interface should be renamed to. + + <backend>/mac + + mac parameter from the libxl API. + Ethernet address which is to be used by the guest. + + <backend>/ip + + ip parameter from the libxl API. + IP address which is to be used by the guest. + + <backend>/bridge + + bridge parameter from the libxl API. + Bridge to which the network should be enslaved (not interpreted + by the libxl toolstack). + + <backend>/rate = "%"PRIu64",%"PRIu32"" + + rate_bytes_per_interval and rate_interval_usecs parameters + from the libxl API. + The network script is responsible for enforcing this. + -- tg: (1069650..) t/xen/xl.docs.scripts (depends on: base)
> diff --git a/docs/misc/xl-disk-configuration.txt b/docs/misc/xl-disk-configuration.txt > index 5da5e11..86c16be 100644 > --- a/docs/misc/xl-disk-configuration.txt > +++ b/docs/misc/xl-disk-configuration.txt > @@ -160,7 +160,10 @@ script=<script> > --------------- > > Specifies that <target> is not a normal host path, but rather > -information to be interpreted by /etc/xen/scripts/block-<script>. > +information to be interpreted by the executable program <script>, > +(looked for in /etc/xen/scripts, if it doesn''t contain a slash). > + > +These scripts are normally called "block-<script>". > > > > @@ -204,7 +207,7 @@ Supported values: iscsi: nbd: enbd: drbd: > In xend and old versions of libxl it was necessary to specify the > "script" (see above) with a prefix. For compatibility, these four > prefixes are recognised as specifying the corresponding script. They > -are equivalent to "script=<script>". > +are equivalent to "script=block-<script>". > > > <deprecated-prefix>:Should I pull these hunks into my block-script patch?
Ian Campbell writes ("Re: [Xen-devel] [PATCH] docs: document hotplug script
protocol"):> > diff --git a/docs/misc/xl-disk-configuration.txt
b/docs/misc/xl-disk-configuration.txt
> > index 5da5e11..86c16be 100644
...> > -are equivalent to "script=<script>".
> > +are equivalent to "script=block-<script>".
...> Should I pull these hunks into my block-script patch?
That''s a good idea.
Ian.
On Mon, 2012-07-30 at 17:56 +0100, Ian Jackson wrote:> diff --git a/docs/misc/xl-scripts-block.txt b/docs/misc/xl-scripts-block.txt > new file mode 100644 > index 0000000..d6a5123 > --- /dev/null > +++ b/docs/misc/xl-scripts-block.txtIt''d be nice to use markdown.> @@ -0,0 +1,246 @@ > + ----------------------- > + DEVICE SCRIPT INTERFACE > + ----------------------- > + > +Guest devices can be specified in [lib]xl to be provided with the help > +of a "script". > + > +This document describes the interface between the toolstack and device > +scripts. This protocol was partially reverse-engineered from the xend > +infrastructure during the development of libxl; the interface is > +intended to be compatible between libxl and xend. This document > +describes the protocol implemented by libxl. > + > +This description is applicable to Linux. > + > +Only block and network devices support these scripts. > + > +Note that we are consider deprecating this protocol after the releaseconsidering> +of Xen 4.2, or providing a more sophisticated replacement for it.I''m not a huge fan of these sorts of TODO list statement scattered around docs, they quickly become out of date or irrelevant.> + > + > +============> +BLOCK SCRIPTS > +============> + > +When a script is specifed, the <target> value from thespecified> +domain configuration (the pdev_path value in the libxl API) is not > +interpreted by libxl. Instead, it is passed as an opaque parameter to > +the script. > + > +The block is responsible for taking the pdev_path (<target> from the > +xl configuration), henceforth known as the "params", and providing a > +block device in the toolstack domain.Do you mean backend domain here?> + > +Block scripts are optional; if not specified, the pdev_path is used as > +the path of the intended block device or file in the driver domain. > + > + > + > +NETWORK SCRIPTS > +==============> + > +When a guest''s networking is being set up, it appears in the driver > +domain as a virtual network device (a "vif" or "tap" device). > + > +The network script is responsible for plumbing this virtual network > +device through to the driver domain''s networking arrangements. For > +example, enslaving the virtual device to a suitable bridge, creating > +routes to it, or whatever is necessary. > + > +The script is also responsible for renaming the interface as may be > +desired.Should we mention here and in the block section that the script is responsible for tearing stuff down as well as setting it up?> +====> +MODEL > +====> + > +We will define a "connection" which is the making available or a > +single block or network device, relating to a single guest, to a > +single facility (in the guest or toolstack or service domain). > + > +It may be necessary to make multiple connections for one guest device; > +even multiple connections which overlap in time. > + > +Such connection is created by an "add" execution of the script and > +destroyed by a "remove" execution. The script is invoked in the > +driver domain (usually dom0). > + > + > +========> +EXECUTION > +========> + > +Environment > +----------- > + > +The scripts are invoked with the following environment variables: > + > + script=<value specified in domain configuration> > + > + This may or may not be a fully qualified path. Scripts should > + not use this information as the form is not reliable.Looking at the kernel I think this is only supplied by netback in the traditional udev based model. libxl does indeed supply it in both cases when it runs the hotplug scripts. Only the vif-setup script currently makes use of it. AFAIK vif-setup is only called from the udev based mechanism to shell out to $script, libxl will call $script itself directly. We could therefore declare this variable to be internal to the udev based hotplug script infrastructure and not for use by "users".> + XENBUS_TYPE=vbd [block devices] > + XENBUS_TYPE=vif [network devices] > + > + XENBUS_PATH=<backend> > + > + This is a path in xenstore which is used extensively to > + communicate between the script and the rest of the backend > + system. This may be a relative or absolute xenstore path. > + > + See below for the contents and usage of this value. > + > + XENBUS_BASE_PATH=backend > + > + Do not use this value. > + > + vif=<devname> [network vif devices] > + INTERFACE=<devname> [network tap devices] > + > + Current name of the virtual device in the toolstack domain."Initial name" perhaps, since strictly speaking someone might have already renamed it (although the scripts are a bi stuffed if someone does)> + > + > +Arguments > +--------- > + > +Block scripts: > + The script will get one argument, "add" or "remove" > + > +Network scripts: > + The script will get two arguments. They vary according to > + whether the script is running with respect to a vif or > + a tap virtual device: > + > + arguments for add arguments for remove > + vif device "online" "type_if=vif" "offline" "type_if=vif" > + tap device "add" "type_if=tap" "remove" "type_if=tap"True, sigh.> +Exit status and error handling > +------------------------------ > + > +The script should exit zero if it is successful. If it fails it > +should clean everything up, try write an error message to the > +hotplug-error path (see below), and and exit 1 or >=126 or die from a > +signal. > + > +Failure of "remove" scripts will be logged and reported but there are > +no arrangements to re-invoke a failed script.Is it worth referencing the helper libraries which we supply (xen-hotplug-common.sh, {block,vif}-common.sh etc). Although it''s true that scripts can just do all this themselves it would be more consistent and likely to be done correctly if people would use the supplied functions.> + > + > +Locking, timeouts, concurrency > +------------------------------ > + > +Scripts should not block indefinitely. If a script takes longer than > +30s to exit it will be sent a SIGKILL which will result in the > +operation being considered failed.30s? LIBXL_{INIT,DESTROY,HOTPLUG}_TIMEOUT are all 10s (I''m not sure which if any actually applies here). It''s also a bit toolstack specific, IIRC xapi has a much longer timeout (it also has its own hotplug scripts but eventually I suppose we''d like to converge)> + > +The scripts may be invoked concurrently for the same or different > +resources, and the same or different guests, and must do any necessary > +locking.Again point to the helpers in locking.h and *-common.sh?> + > + > + > +=======> +XENSTORE > +=======> + > +The <backend> directory in xenstore contains keys and values used to > +communicate between the script and the rest of the infrastructure. > + > +The contents of this path are shared between the script and other > +parts of the infrastructure. The script should therefore not write > +arbitrary subpaths for its own purposes. > + > +The following paths are defined: > + > + > +All devices > +----------- > + > + <backend>/hotplug-error > + > + If a script is about to fail, it may write an error message > + string to this path. The string will be reported to the user. > + > + > +Block devices > +------------- > + > + <backend>/params > + > + The params. Ie, the <target> string from the configuration file, > + which corresponds to the "pdev_path" in the libxl API. > + > + This is not interpreted by the toolstack and the script may > + define whatever syntax it likes for it. Newlines, whitespace, > + backslashes, quotes, and so forth, should be avoided as > + specifying them from a config file may be difficult. Consult the > + xl configuration API documentation for information about the > + exact syntactic transformations in xl.Reference xl-disk-configuration specifically?> + > + The value is written by the toolstack before the add script is > + run and remains present until the remove script has finished. It > + should not be modified by the script. > + > + <backend>/mode = "r" | "w" > + > + The device is to be readonly or read/write, respectively.Maybe "read-only"? (perhaps it is alloneword, I''m not sure)> + Written by the toolstack; should not be modified. > + > + <backend>/physical-device > + > + The main result of the add script. This key does not exist on > + entry to add. The add script should, if successful, write this > + key with the major and minor number of the block device > + (as accessible in the driver domain) in the format "%x:%x". > + > + > +Other information that may be useful to the script is also present. > +The block script does not need to pay attention to this if it doesn''t > +want to. These keys include: > + > + <backend>/removable = "0" | "1" [block devices] > + > + Indicates whether the device is being presented to the guest as a > + removable device. > + > + <backend>/device-type = "disk" | "cdrom" [block devices] > + > + > +Network devices > +--------------- > + > +All of these values are provided by the toolstack to the script and > +should not be changed by the script: > + > + <backend>/vifname [network devices]I guess "[network devices]" (and "[block devices]" above) predate you splitting them into their own sections?> + > + Name that the interface should be renamed to. > + > + <backend>/mac > + > + mac parameter from the libxl API. > + Ethernet address which is to be used by the guest. > + > + <backend>/ip > + > + ip parameter from the libxl API. > + IP address which is to be used by the guest. > + > + <backend>/bridge > + > + bridge parameter from the libxl API. > + Bridge to which the network should be enslaved (not interpreted > + by the libxl toolstack).I suppose "ip" and "bridge" are something like the block scripts'' "params" node, in so much as we could consider making them more explicitly generic and opaque to the toolstack in the future. In practice at the moment bridge is the parameter for vif-bridge and ip is the parameter for vif-{route,nat}. Or something like that anyway.> + > + <backend>/rate = "%"PRIu64",%"PRIu32"" > + > + rate_bytes_per_interval and rate_interval_usecs parameters > + from the libxl API. > + The network script is responsible for enforcing this.Actually, netback does this itself in practice. Ian.