Justin T. Gibbs
2012-Feb-15 05:06 UTC
[PATCH 0 of 4 v2] blkif.h: Document protocol and existing extensions
This patch series attempts to document the blkif PV interface and
the various extensions to it that are out in the wild.
Changes in v2:
patch 2 (blkif.h: Provide more complete documentation of the blkif interface.):
o Mark backend device identification section as private to the
backend driver.
o Refer to docs/misc/vbd-interface.txt for the format of the
virtual-device front-end node.
o Correct field size for the virtual-device front-end node.
patch 3 (blkif.h: Document the RedHat and Citrix blkif multi-page
ring extensions.):
o Correct node names for the RedHat/Amazon multi-ring extension. The
previous patch mistakenly documented the now defunct FreeBSD extension.
o Clarify the note on multi-page ring interoperability to indicate that
identical ring parameters should be published to the XenStore for
all supported schemes.
v1 patch 4 (Deleted):
o Remove patch that added the XEN_*_MAJOR definitions.
patch 4 (blkif.h: Define and document the request number/size/segments
extension.)
o Bump __XEN_LATEST_INTERFACE_VERSION to 0x00040201 and use this version
to guard the change to BLKIF_MAX_SEGMENTS_PER_REQUEST.
--
Justin
o Remove trailing whitespace.
o Remove a blank line so that a comment block is adjacent to the
code it documents.
No functional changes.
Signed-off-by: Justin T. Gibbs <justing@spectralogic.com>
diff -r 0ba87b95e80b -r 0e07382ca5fc xen/include/public/io/blkif.h
--- a/xen/include/public/io/blkif.h Mon Feb 13 17:26:08 2012 +0000
+++ b/xen/include/public/io/blkif.h Tue Feb 14 21:54:09 2012 -0700
@@ -1,8 +1,8 @@
/******************************************************************************
* blkif.h
- *
+ *
* Unified block-device I/O interface for Xen guest OSes.
- *
+ *
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the
"Software"), to
* deal in the Software without restriction, including without limitation the
@@ -35,7 +35,7 @@
* notification can be made conditional on req_event (i.e., the generic
* hold-off mechanism provided by the ring macros). Backends must set
* req_event appropriately (e.g., using RING_FINAL_CHECK_FOR_REQUESTS()).
- *
+ *
* Back->front notifications: When enqueuing a new response, sending a
* notification can be made conditional on rsp_event (i.e., the generic
* hold-off mechanism provided by the ring macros). Frontends must set
@@ -133,7 +133,7 @@
*/
#define BLKIF_MAX_SEGMENTS_PER_REQUEST 11
-/*
+/*
* NB. first_sect and last_sect in blkif_request_segment, as well as
* sector_number in blkif_request, are always expressed in 512-byte units.
* However they must be properly aligned to the real sector size of the
@@ -192,7 +192,6 @@ typedef struct blkif_response blkif_resp
/*
* Generate blkif ring structures and types.
*/
-
DEFINE_RING_TYPES(blkif, struct blkif_request, struct blkif_response);
#define VDISK_CDROM 0x1
Justin T. Gibbs
2012-Feb-15 05:06 UTC
[PATCH 2 of 4 v2] blkif.h: Provide more complete documentation of the blkif interface
o Document the XenBus nodes used in this protocol.
o Add a state diagram illustrating the roles and responsibilities
of both the front and backend during startup.
o Correct missed BLKIF_OP_TRIM => BLKIF_OP_DISCARD conversion in a comment.
No functional changes.
Signed-off-by: Justin T. Gibbs <justing@spectralogic.com>
diff -r 0e07382ca5fc -r 2ecf6eaf5d86 xen/include/public/io/blkif.h
--- a/xen/include/public/io/blkif.h Tue Feb 14 21:54:09 2012 -0700
+++ b/xen/include/public/io/blkif.h Tue Feb 14 21:54:09 2012 -0700
@@ -22,6 +22,7 @@
* DEALINGS IN THE SOFTWARE.
*
* Copyright (c) 2003-2004, Keir Fraser
+ * Copyright (c) 2012, Spectra Logic Corporation
*/
#ifndef __XEN_PUBLIC_IO_BLKIF_H__
@@ -48,32 +49,253 @@
#define blkif_sector_t uint64_t
/*
+ * Feature and Parameter Negotiation
+ * ================================+ * The two halves of a Xen block driver
utilize nodes within the XenStore to
+ * communicate capabilities and to negotiate operating parameters. This
+ * section enumerates these nodes which reside in the respective front and
+ * backend portions of the XenStore, following the XenBus convention.
+ *
+ * All data in the XenStore is stored as strings. Nodes specifying numeric
+ * values are encoded in decimal. Integer value ranges listed below are
+ * expressed as fixed sized integer types capable of storing the conversion
+ * of a properly formated node string, without loss of information.
+ *
+ * Any specified default value is in effect if the corresponding XenBus node
+ * is not present in the XenStore.
+ *
+ * XenStore nodes in sections marked "PRIVATE" are solely for use by
the
+ * driver side whose XenBus tree contains them.
+ *
+ * See the XenBus state transition diagram below for details on when XenBus
+ * nodes must be published and when they can be queried.
+ *
+ *****************************************************************************
+ * Backend XenBus Nodes
+ *****************************************************************************
+ *
+ *------------------ Backend Device Identification (PRIVATE) ------------------
+ *
+ * mode
+ * Values: "r" (read only), "w" (writable)
+ *
+ * The read or write access permissions to the backing store to be
+ * granted to the frontend.
+ *
+ * params
+ * Values: string
+ *
+ * A free formatted string providing sufficient information for the
+ * backend driver to open the backing device. (e.g. the path to the
+ * file or block device representing the backing store.)
+ *
+ * type
+ * Values: "file", "phy", "tap"
+ *
+ * The type of the backing device/object.
+ *
+ *--------------------------------- Features ---------------------------------
+ *
+ * feature-barrier
+ * Values: 0/1 (boolean)
+ * Default Value: 0
+ *
+ * A value of "1" indicates that the backend can process
requests
+ * containing the BLKIF_OP_WRITE_BARRIER request opcode. Requests
+ * of this type may still be returned at any time with the
+ * BLKIF_RSP_EOPNOTSUPP result code.
+ *
+ * feature-flush-cache
+ * Values: 0/1 (boolean)
+ * Default Value: 0
+ *
+ * A value of "1" indicates that the backend can process
requests
+ * containing the BLKIF_OP_FLUSH_DISKCACHE request opcode. Requests
+ * of this type may still be returned at any time with the
+ * BLKIF_RSP_EOPNOTSUPP result code.
+ *
+ * feature-discard
+ * Values: 0/1 (boolean)
+ * Default Value: 0
+ *
+ * A value of "1" indicates that the backend can process
requests
+ * containing the BLKIF_OP_DISCARD request opcode. Requests
+ * of this type may still be returned at any time with the
+ * BLKIF_RSP_EOPNOTSUPP result code.
+ *
+ *------------------------- Backend Device Properties -------------------------
+ *
+ * discard-aligment
+ * Values: <uint32_t>
+ * Default Value: 0
+ * Notes: 1, 2
+ *
+ * The offset, in bytes from the beginning of the virtual block device,
+ * to the first, addressable, discard extent on the underlying device.
+ *
+ * discard-granularity
+ * Values: <uint32_t>
+ * Default Value: <"sector-size">
+ * Notes: 1
+ *
+ * The size, in bytes, of the individually addressable discard extents
+ * of the underlying device.
+ *
+ * discard-secure
+ * Values: 0/1 (boolean)
+ * Default Value: 0
+ *
+ * A value of "1" indicates that the backend can process
BLKIF_OP_DISCARD
+ * requests with the BLKIF_DISCARD_SECURE flag set.
+ *
+ * info
+ * Values: <uint32_t> (bitmap)
+ *
+ * A collection of bit flags describing attributes of the backing
+ * device. The VDISK_* macros define the meaning of each bit
+ * location.
+ *
+ * sector-size
+ * Values: <uint32_t>
+ *
+ * The native sector size, in bytes, of the backend device.
+ *
+ * sectors
+ * Values: <uint64_t>
+ *
+ * The size of the backend device, expressed in units of its native
+ * sector size ("sector-size").
+ *
+ *****************************************************************************
+ * Frontend XenBus Nodes
+ *****************************************************************************
+ *
+ *----------------------- Request Transport Parameters -----------------------
+ *
+ * event-channel
+ * Values: <uint32_t>
+ *
+ * The identifier of the Xen event channel used to signal activity
+ * in the ring buffer.
+ *
+ * ring-ref
+ * Values: <uint32_t>
+ *
+ * The Xen grant reference granting permission for the backend to map
+ * the sole page in a single page sized ring buffer.
+ *
+ * protocol
+ * Values: string (XEN_IO_PROTO_ABI_*)
+ * Default Value: XEN_IO_PROTO_ABI_NATIVE
+ *
+ * The machine ABI rules governing the format of all ring request and
+ * response structures.
+ *
+ *------------------------- Virtual Device Properties -------------------------
+ *
+ * device-type
+ * Values: "disk", "cdrom",
"floppy", etc.
+ *
+ * virtual-device
+ * Values: <uint32_t>
+ *
+ * A value indicating the physical device to virtualize within the
+ * frontend''s domain. (e.g. "The first ATA disk",
"The third SCSI
+ * disk", etc.)
+ *
+ * See docs/misc/vbd-interface.txt for details on the format of this
+ * value.
+ *
+ * Notes
+ * -----
+ * (1) Devices that support discard functionality may internally allocate
+ * space (discardable extents) in units that are larger than the
+ * exported logical block size.
+ * (2) The discard-alignment parameter allows a physical device to be
+ * partitioned into virtual devices that do not necessarily begin or
+ * end on a discardable extent boundary.
+ */
+
+/*
+ * STATE DIAGRAMS
+ *
+ *****************************************************************************
+ * Startup *
+ *****************************************************************************
+ *
+ * Tool stack creates front and back nodes with state XenbusStateInitialising.
+ *
+ * Front Back
+ * ================================= ====================================+ *
XenbusStateInitialising XenbusStateInitialising
+ * o Query virtual device o Query backend device identification
+ * properties. data.
+ * o Setup OS device instance. o Open and validate backend device.
+ * o Publish backend features.
+ * |
+ * |
+ * V
+ * XenbusStateInitWait
+ *
+ * o Query backend features.
+ * o Allocate and initialize the
+ * request ring.
+ * |
+ * |
+ * V
+ * XenbusStateInitialised
+ *
+ * o Connect to the request ring and
+ * event channel.
+ * o Publish backend device properties.
+ * |
+ * |
+ * V
+ * XenbusStateConnected
+ *
+ * o Query backend device properties.
+ * o Finalize OS virtual device
+ * instance.
+ * |
+ * |
+ * V
+ * XenbusStateConnected
+ *
+ * Note: Drivers that do not support any optional features can skip certain
+ * states in the state machine:
+ *
+ * o A frontend may transition to XenbusStateInitialised without
+ * waiting for the backend to enter XenbusStateInitWait.
+ *
+ * o A backend may transition to XenbusStateInitialised, bypassing
+ * XenbusStateInitWait, without waiting for the frontend to first
+ * enter the XenbusStateInitialised state.
+ *
+ * Drivers that support optional features must tolerate these additional
+ * state transition paths. In general this means performing the work of
+ * any skipped state transition, if it has not already been performed,
+ * in addition to the work associated with entry into the current state.
+ */
+
+/*
* REQUEST CODES.
*/
#define BLKIF_OP_READ 0
#define BLKIF_OP_WRITE 1
/*
- * Recognised only if "feature-barrier" is present in backend xenbus
info.
- * The "feature-barrier" node contains a boolean indicating whether
barrier
- * requests are likely to succeed or fail. Either way, a barrier request
- * may fail at any time with BLKIF_RSP_EOPNOTSUPP if it is unsupported by
- * the underlying block-device hardware. The boolean simply indicates whether
- * or not it is worthwhile for the frontend to attempt barrier requests.
- * If a backend does not recognise BLKIF_OP_WRITE_BARRIER, it should *not*
- * create the "feature-barrier" node!
+ * All writes issued prior to a request with the BLKIF_OP_WRITE_BARRIER
+ * operation code ("barrier request") must be completed prior to the
+ * execution of the barrier request. All writes issued after the barrier
+ * request must not execute until after the completion of the barrier request.
+ *
+ * Optional. See "feature-barrier" XenBus node documentation above.
*/
#define BLKIF_OP_WRITE_BARRIER 2
/*
- * Recognised if "feature-flush-cache" is present in backend xenbus
- * info. A flush will ask the underlying storage hardware to flush its
- * non-volatile caches as appropriate. The "feature-flush-cache"
node
- * contains a boolean indicating whether flush requests are likely to
- * succeed or fail. Either way, a flush request may fail at any time
- * with BLKIF_RSP_EOPNOTSUPP if it is unsupported by the underlying
- * block-device hardware. The boolean simply indicates whether or not it
- * is worthwhile for the frontend to attempt flushes. If a backend does
- * not recognise BLKIF_OP_WRITE_FLUSH_CACHE, it should *not* create the
- * "feature-flush-cache" node!
+ * Commit any uncommitted contents of the backing device''s volatile
cache
+ * to stable storage.
+ *
+ * Optional. See "feature-flush-cache" XenBus node documentation
above.
*/
#define BLKIF_OP_FLUSH_DISKCACHE 3
/*
@@ -82,47 +304,24 @@
*/
#define BLKIF_OP_RESERVED_1 4
/*
- * Recognised only if "feature-discard" is present in backend xenbus
info.
- * The "feature-discard" node contains a boolean indicating whether
trim
- * (ATA) or unmap (SCSI) - conviently called discard requests are likely
- * to succeed or fail. Either way, a discard request
- * may fail at any time with BLKIF_RSP_EOPNOTSUPP if it is unsupported by
- * the underlying block-device hardware. The boolean simply indicates whether
- * or not it is worthwhile for the frontend to attempt discard requests.
- * If a backend does not recognise BLKIF_OP_DISCARD, it should *not*
- * create the "feature-discard" node!
+ * Indicate to the backend device that a region of storage is no longer in
+ * use, and may be discarded at any time without impact to the client. If
+ * the BLKIF_DISCARD_SECURE flag is set on the request, all copies of the
+ * discarded region on the device must be rendered unrecoverable before the
+ * command returns.
*
- * Discard operation is a request for the underlying block device to mark
- * extents to be erased. However, discard does not guarantee that the blocks
- * will be erased from the device - it is just a hint to the device
- * controller that these blocks are no longer in use. What the device
- * controller does with that information is left to the controller.
- * Discard operations are passed with sector_number as the
- * sector index to begin discard operations at and nr_sectors as the number of
- * sectors to be discarded. The specified sectors should be discarded if the
- * underlying block device supports trim (ATA) or unmap (SCSI) operations,
- * or a BLKIF_RSP_EOPNOTSUPP should be returned.
- * More information about trim/unmap operations at:
+ * This operation is analogous to performing a trim (ATA) or unamp (SCSI),
+ * command on a native device.
+ *
+ * More information about trim/unmap operations can be found at:
* http://t13.org/Documents/UploadedDocuments/docs2008/
* e07154r6-Data_Set_Management_Proposal_for_ATA-ACS2.doc
* http://www.seagate.com/staticfiles/support/disc/manuals/
* Interface%20manuals/100293068c.pdf
- * The backend can optionally provide these extra XenBus attributes to
- * further optimize the discard functionality:
- * ''discard-aligment'' - Devices that support discard
functionality may
- * internally allocate space in units that are bigger than the exported
- * logical block size. The discard-alignment parameter indicates how many bytes
- * the beginning of the partition is offset from the internal allocation
unit''s
- * natural alignment. Do not confuse this with natural disk alignment offset.
- * ''discard-granularity'' - Devices that support discard
functionality may
- * internally allocate space using units that are bigger than the logical block
- * size. The discard-granularity parameter indicates the size of the internal
- * allocation unit in bytes if reported by the device. Otherwise the
- * discard-granularity will be set to match the device''s physical
block size.
- * It is the minimum size you can discard.
- * ''discard-secure'' - All copies of the discarded sectors
(potentially created
- * by garbage collection) must also be erased. To use this feature, the flag
- * BLKIF_DISCARD_SECURE must be set in the blkif_request_discard.
+ *
+ * Optional. See "feature-discard", "discard-alignment",
+ * "discard-granularity", and "discard-secure" in the
XenBus node
+ * documentation above.
*/
#define BLKIF_OP_DISCARD 5
@@ -147,6 +346,9 @@ struct blkif_request_segment {
uint8_t first_sect, last_sect;
};
+/*
+ * Starting ring element for any I/O request.
+ */
struct blkif_request {
uint8_t operation; /* BLKIF_OP_??? */
uint8_t nr_segments; /* number of segments */
@@ -158,7 +360,7 @@ struct blkif_request {
typedef struct blkif_request blkif_request_t;
/*
- * Cast to this structure when blkif_request.operation == BLKIF_OP_TRIM
+ * Cast to this structure when blkif_request.operation == BLKIF_OP_DISCARD
* sizeof(struct blkif_request_discard) <= sizeof(struct blkif_request)
*/
struct blkif_request_discard {
Justin T. Gibbs
2012-Feb-15 05:06 UTC
[PATCH 3 of 4 v2] blkif.h: Document the RedHat and Citrix blkif multi-page ring extensions
No functional changes. Signed-off-by: Justin T. Gibbs <justing@spectralogic.com> diff -r 2ecf6eaf5d86 -r 321810c42d55 xen/include/public/io/blkif.h --- a/xen/include/public/io/blkif.h Tue Feb 14 21:54:09 2012 -0700 +++ b/xen/include/public/io/blkif.h Tue Feb 14 21:54:09 2012 -0700 @@ -123,12 +123,31 @@ * of this type may still be returned at any time with the * BLKIF_RSP_EOPNOTSUPP result code. * + *----------------------- Request Transport Parameters ------------------------ + * + * max-ring-page-order + * Values: <uint32_t> + * Default Value: 0 + * Notes: 1, 3 + * + * The maximum supported size of the request ring buffer in units of + * lb(machine pages). (e.g. 0 == 1 page, 1 = 2 pages, 2 == 4 pages, + * etc.). + * + * max-ring-pages + * Values: <uint32_t> + * Default Value: 1 + * Notes: 2, 3 + * + * The maximum supported size of the request ring buffer in units of + * machine pages. The value must be a power of 2. + * *------------------------- Backend Device Properties ------------------------- * * discard-aligment * Values: <uint32_t> * Default Value: 0 - * Notes: 1, 2 + * Notes: 4, 5 * * The offset, in bytes from the beginning of the virtual block device, * to the first, addressable, discard extent on the underlying device. @@ -136,7 +155,7 @@ * discard-granularity * Values: <uint32_t> * Default Value: <"sector-size"> - * Notes: 1 + * Notes: 4 * * The size, in bytes, of the individually addressable discard extents * of the underlying device. @@ -180,10 +199,20 @@ * * ring-ref * Values: <uint32_t> + * Notes: 6 * * The Xen grant reference granting permission for the backend to map * the sole page in a single page sized ring buffer. * + * ring-ref%u + * Values: <uint32_t> + * Notes: 6 + * + * For a frontend providing a multi-page ring, a "num-ring-pages" sized + * list of nodes, each containing a Xen grant reference granting + * permission for the backend to map the page of the ring located + * at page index "%u". Page indexes are zero based. + * * protocol * Values: string (XEN_IO_PROTO_ABI_*) * Default Value: XEN_IO_PROTO_ABI_NATIVE @@ -191,6 +220,25 @@ * The machine ABI rules governing the format of all ring request and * response structures. * + * ring-page-order + * Values: <uint32_t> + * Default Value: 0 + * Maximum Value: MAX(ffs(max-ring-pages) - 1, max-ring-page-order) + * Notes: 1, 3 + * + * The size of the frontend allocated request ring buffer in units + * of lb(machine pages). (e.g. 0 == 1 page, 1 = 2 pages, 2 == 4 pages, + * etc.). + * + * num-ring-pages + * Values: <uint32_t> + * Default Value: 1 + * Maximum Value: MAX(max-ring-pages,(0x1 << max-ring-page-order)) + * Notes: 2, 3 + * + * The size of the frontend allocated request ring buffer in units of + * machine pages. The value must be a power of 2. + * *------------------------- Virtual Device Properties ------------------------- * * device-type @@ -208,12 +256,26 @@ * * Notes * ----- - * (1) Devices that support discard functionality may internally allocate + * (1) Multi-page ring buffer scheme first developed in the Citrix XenServer + * PV drivers. + * (2) Multi-page ring buffer scheme first used in some RedHat distributions + * including a distribution deployed on certain nodes of the Amazon + * EC2 cluster. + * (3) Support for multi-page ring buffers was implemented independently, + * in slightly different forms, by both Citrix and RedHat/Amazon. + * For full interoperability, block front and backends should publish + * identical ring parameters, adjusted for unit differences, to the + * XenStore nodes used in both schemes. + * (4) Devices that support discard functionality may internally allocate * space (discardable extents) in units that are larger than the * exported logical block size. - * (2) The discard-alignment parameter allows a physical device to be + * (5) The discard-alignment parameter allows a physical device to be * partitioned into virtual devices that do not necessarily begin or * end on a discardable extent boundary. + * (6) When there is only a single page allocated to the request ring, + * ''ring-ref'' is used to communicate the grant reference for this + * page to the backend. When using a multi-page ring, the ''ring-ref'' + * node is not created. Instead ''ring-ref0'' - ''ring-refN'' are used. */ /* @@ -231,20 +293,26 @@ * o Query virtual device o Query backend device identification * properties. data. * o Setup OS device instance. o Open and validate backend device. - * o Publish backend features. + * o Publish backend features and + * transport parameters. * | * | * V * XenbusStateInitWait * - * o Query backend features. + * o Query backend features and + * transport parameters. * o Allocate and initialize the * request ring. + * o Publish transport parameters + * that will be in effect during + * this connection. * | * | * V * XenbusStateInitialised * + * o Query frontend transport parameters. * o Connect to the request ring and * event channel. * o Publish backend device properties. @@ -261,20 +329,26 @@ * V * XenbusStateConnected * - * Note: Drivers that do not support any optional features can skip certain - * states in the state machine: + * Note: Drivers that do not support any optional features, or the negotiation + * of transport parameters, can skip certain states in the state machine: * * o A frontend may transition to XenbusStateInitialised without - * waiting for the backend to enter XenbusStateInitWait. + * waiting for the backend to enter XenbusStateInitWait. In this + * case, default transport parameters are in effect and any + * transport parameters published by the frontend must contain + * their default values. * * o A backend may transition to XenbusStateInitialised, bypassing * XenbusStateInitWait, without waiting for the frontend to first - * enter the XenbusStateInitialised state. + * enter the XenbusStateInitialised state. In this case, default + * transport parameters are in effect and any transport parameters + * published by the backend must contain their default values. * - * Drivers that support optional features must tolerate these additional - * state transition paths. In general this means performing the work of - * any skipped state transition, if it has not already been performed, - * in addition to the work associated with entry into the current state. + * Drivers that support optional features and/or transport parameter + * negotiation must tolerate these additional state transition paths. + * In general this means performing the work of any skipped state + * transition, if it has not already been performed, in addition to the + * work associated with entry into the current state. */ /*
Justin T. Gibbs
2012-Feb-15 05:06 UTC
[PATCH 4 of 4 v2] blkif.h: Define and document the request number/size/segments extension
Note: As of __XEN_INTERFACE_VERSION__ 0x00040201 the definition of
BLKIF_MAX_SEGMENTS_PER_REQUEST has changed. Drivers must be updated
to, at minimum, use BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK, before being
recompiled with a __XEN_INTERFACE_VERSION greater than or equal to
this value.
This extension first appeared in the FreeBSD Operating System.
Signed-off-by: Justin T. Gibbs <justing@spectralogic.com>
diff -r 321810c42d55 -r adcb465964c5 xen/include/public/io/blkif.h
--- a/xen/include/public/io/blkif.h Tue Feb 14 21:54:09 2012 -0700
+++ b/xen/include/public/io/blkif.h Tue Feb 14 21:54:09 2012 -0700
@@ -142,6 +142,32 @@
* The maximum supported size of the request ring buffer in units of
* machine pages. The value must be a power of 2.
*
+ * max-requests <uint32_t>
+ * Default Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE)
+ * Maximum Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE * max-ring-pages)
+ *
+ * The maximum number of concurrent, logical requests that will be
+ * issued by the backend.
+ *
+ * Note: A logical request may span multiple ring entries.
+ *
+ * max-request-segments
+ * Values: <uint8_t>
+ * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK
+ * Maximum Value: BLKIF_MAX_SEGMENTS_PER_REQUEST
+ *
+ * The maximum value of blkif_request.nr_segments supported by
+ * the backend.
+ *
+ * max-request-size
+ * Values: <uint32_t>
+ * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK * PAGE_SIZE
+ * Maximum Value: BLKIF_MAX_SEGMENTS_PER_REQUEST * PAGE_SIZE
+ *
+ * The maximum amount of data, in bytes, that can be referenced by a
+ * request type that accesses frontend memory (currently BLKIF_OP_READ,
+ * BLKIF_OP_WRITE, or BLKIF_OP_WRITE_BARRIER).
+ *
*------------------------- Backend Device Properties -------------------------
*
* discard-aligment
@@ -239,6 +265,33 @@
* The size of the frontend allocated request ring buffer in units of
* machine pages. The value must be a power of 2.
*
+ * max-requests
+ * Values: <uint32_t>
+ * Default Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE)
+ * Maximum Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE * max-ring_pages)
+ *
+ * The maximum number of concurrent, logical requests that will be
+ * issued by the frontend.
+ *
+ * Note: A logical request may span multiple ring entries.
+ *
+ * max-request-segments
+ * Values: <uint8_t>
+ * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK
+ * Maximum Value: MIN(255, backend/max-request-segments)
+ *
+ * The maximum value the frontend will set in the
+ * blkif_request.nr_segments field.
+ *
+ * max-request-size
+ * Values: <uint32_t>
+ * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK * PAGE_SIZE
+ * Maximum Value: max-request-segments * PAGE_SIZE
+ *
+ * The maximum amount of data, in bytes, that can be referenced by
+ * a request type that accesses frontend memory (currently BLKIF_OP_READ,
+ * BLKIF_OP_WRITE, or BLKIF_OP_WRITE_BARRIER).
+ *
*------------------------- Virtual Device Properties -------------------------
*
* device-type
@@ -400,11 +453,28 @@
#define BLKIF_OP_DISCARD 5
/*
- * Maximum scatter/gather segments per request.
+ * Maximum scatter/gather segments associated with a request header block.
* This is carefully chosen so that sizeof(blkif_ring_t) <= PAGE_SIZE.
* NB. This could be 12 if the ring indexes weren''t stored in the same
page.
*/
-#define BLKIF_MAX_SEGMENTS_PER_REQUEST 11
+#define BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK 11
+
+/*
+ * Maximum scatter/gather segments associated with a segment block.
+ */
+#define BLKIF_MAX_SEGMENTS_PER_SEGMENT_BLOCK 14
+
+#if __XEN_INTERFACE_VERSION__ >= 0x00040201
+/*
+ * Maximum scatter/gather segments per request (header + segment blocks).
+ */
+#define BLKIF_MAX_SEGMENTS_PER_REQUEST 255
+#else
+/*
+ * Maximum scatter/gather segments per request (header block only).
+ */
+#define BLKIF_MAX_SEGMENTS_PER_REQUEST BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK
+#endif
/*
* NB. first_sect and last_sect in blkif_request_segment, as well as
@@ -419,9 +489,25 @@ struct blkif_request_segment {
/* @last_sect: last sector in frame to transfer (inclusive). */
uint8_t first_sect, last_sect;
};
+typedef struct blkif_request_segment blkif_request_segment_t;
/*
* Starting ring element for any I/O request.
+ *
+ * One or more segment blocks can be inserted into the request ring
+ * just after a blkif_request_t, allowing requests to operate on
+ * up to BLKIF_MAX_SEGMENTS_PER_REQUEST.
+ *
+ * BLKIF_SEGS_TO_BLOCKS() can be used on blkif_requst.nr_segments
+ * to determine the number of contiguous ring entries associated
+ * with this request.
+ *
+ * Note: Due to the way Xen request rings operate, the producer and
+ * consumer indices of the ring must be incremented by the
+ * BLKIF_SEGS_TO_BLOCKS() value of the associated request.
+ * (e.g. a response to a 3 ring entry request must also consume
+ * 3 entries in the ring, even though only the first ring entry
+ * in the response has any data.)
*/
struct blkif_request {
uint8_t operation; /* BLKIF_OP_??? */
@@ -429,11 +515,22 @@ struct blkif_request {
blkif_vdev_t handle; /* only for read/write requests */
uint64_t id; /* private guest value, echoed in resp */
blkif_sector_t sector_number;/* start sector idx on disk (r/w only) */
- struct blkif_request_segment seg[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ blkif_request_segment_t seg[BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK];
};
typedef struct blkif_request blkif_request_t;
/*
+ * A segment block is a ring request structure that contains only
+ * segment data.
+ *
+ * sizeof(struct blkif_segment_block) <= sizeof(struct blkif_request)
+ */
+struct blkif_segment_block {
+ blkif_request_segment_t seg[BLKIF_MAX_SEGMENTS_PER_SEGMENT_BLOCK];
+};
+typedef struct blkif_segment_block blkif_segment_block_t;
+
+/*
* Cast to this structure when blkif_request.operation == BLKIF_OP_DISCARD
* sizeof(struct blkif_request_discard) <= sizeof(struct blkif_request)
*/
@@ -470,6 +567,21 @@ typedef struct blkif_response blkif_resp
*/
DEFINE_RING_TYPES(blkif, struct blkif_request, struct blkif_response);
+/*
+ * Index to, and treat as a segment block, an entry in the ring.
+ */
+#define BLKRING_GET_SEG_BLOCK(_r, _idx) \
+ (((blkif_segment_block_t *)RING_GET_REQUEST(_r, _idx))->seg)
+
+/*
+ * The number of ring request blocks required to handle an I/O
+ * request containing _segs segments.
+ */
+#define BLKIF_SEGS_TO_BLOCKS(_segs) \
+ ((((_segs - BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK) \
+ + (BLKIF_MAX_SEGMENTS_PER_SEGMENT_BLOCK - 1)) \
+ / BLKIF_MAX_SEGMENTS_PER_SEGMENT_BLOCK) + /*header_block*/1)
+
#define VDISK_CDROM 0x1
#define VDISK_REMOVABLE 0x2
#define VDISK_READONLY 0x4
diff -r 321810c42d55 -r adcb465964c5 xen/include/public/xen-compat.h
--- a/xen/include/public/xen-compat.h Tue Feb 14 21:54:09 2012 -0700
+++ b/xen/include/public/xen-compat.h Tue Feb 14 21:54:09 2012 -0700
@@ -27,7 +27,7 @@
#ifndef __XEN_PUBLIC_XEN_COMPAT_H__
#define __XEN_PUBLIC_XEN_COMPAT_H__
-#define __XEN_LATEST_INTERFACE_VERSION__ 0x00040200
+#define __XEN_LATEST_INTERFACE_VERSION__ 0x00040201
#if defined(__XEN__) || defined(__XEN_TOOLS__)
/* Xen is built with matching headers and implements the latest interface. */
Jan Beulich
2012-Feb-15 08:14 UTC
Re: [PATCH 0 of 4 v2] blkif.h: Document protocol and existing extensions
>>> On 15.02.12 at 06:06, "Justin T. Gibbs" <justing@spectralogic.com> wrote: > patch 4 (blkif.h: Define and document the request number/size/segments > extension.) > o Bump __XEN_LATEST_INTERFACE_VERSION to 0x00040201 and use this versionWhy? It got bumped very recently to 0x040200, your change could well go under that same version. (Which is not to say that I''m now suddenly agreeing to stuff under io/ being controlled by this).> to guard the change to BLKIF_MAX_SEGMENTS_PER_REQUEST.Jan
Justin T. Gibbs
2012-Feb-15 13:03 UTC
Re: [PATCH 0 of 4 v2] blkif.h: Document protocol and existing extensions
On Feb 15, 2012, at 1:14 AM, Jan Beulich wrote:>>>> On 15.02.12 at 06:06, "Justin T. Gibbs" <justing@spectralogic.com> wrote: >> patch 4 (blkif.h: Define and document the request number/size/segments >> extension.) >> o Bump __XEN_LATEST_INTERFACE_VERSION to 0x00040201 and use this version > > Why? It got bumped very recently to 0x040200, your change could > well go under that same version.When reviewing the history of xen-compat.h, it appeared that the version was bumped for each incompatible change. Considering there are 8 bits of sub-minor space, why wouldn''t it be bumped? It allows an integrator of new headers to review all change sets associated with this one header file to understand the work required for an import.> (Which is not to say that I''m now > suddenly agreeing to stuff under io/ being controlled by this).:-) -- Justin
Jan Beulich
2012-Feb-15 13:16 UTC
Re: [PATCH 0 of 4 v2] blkif.h: Document protocol and existing extensions
>>> On 15.02.12 at 14:03, "Justin T. Gibbs" <justing@spectralogic.com> wrote:> On Feb 15, 2012, at 1:14 AM, Jan Beulich wrote: > >>>>> On 15.02.12 at 06:06, "Justin T. Gibbs" <justing@spectralogic.com> wrote: >>> patch 4 (blkif.h: Define and document the request number/size/segments >>> extension.) >>> o Bump __XEN_LATEST_INTERFACE_VERSION to 0x00040201 and use this version >> >> Why? It got bumped very recently to 0x040200, your change could >> well go under that same version. > > When reviewing the history of xen-compat.h, it appeared that the version was > bumped > for each incompatible change. Considering there are 8 bits of sub-minor > space, why > wouldn''t it be bumped? It allows an integrator of new headers to review all > change sets > associated with this one header file to understand the work required for an > import.Then be it that way if it is to Keir''s liking. Jan
Keir Fraser
2012-Feb-15 17:11 UTC
Re: [PATCH 0 of 4 v2] blkif.h: Document protocol and existing extensions
On 15/02/2012 13:16, "Jan Beulich" <JBeulich@suse.com> wrote:>>> Why? It got bumped very recently to 0x040200, your change could >>> well go under that same version. >> >> When reviewing the history of xen-compat.h, it appeared that the version was >> bumped >> for each incompatible change. Considering there are 8 bits of sub-minor >> space, why >> wouldn''t it be bumped? It allows an integrator of new headers to review all >> change sets >> associated with this one header file to understand the work required for an >> import. > > Then be it that way if it is to Keir''s liking.Yep it''s fine by me. -- Keir
Jan Beulich
2012-Feb-16 08:42 UTC
Re: [PATCH 4 of 4 v2] blkif.h: Define and document the request number/size/segments extension
>>> On 15.02.12 at 06:06, "Justin T. Gibbs" <justing@spectralogic.com> wrote: > Note: As of __XEN_INTERFACE_VERSION__ 0x00040201 the definition of > BLKIF_MAX_SEGMENTS_PER_REQUEST has changed. Drivers must be updated > to, at minimum, use BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK, before being > recompiled with a __XEN_INTERFACE_VERSION greater than or equal to > this value. > > This extension first appeared in the FreeBSD Operating System.Any pointer to their implementation?> Signed-off-by: Justin T. Gibbs <justing@spectralogic.com> > > diff -r 321810c42d55 -r adcb465964c5 xen/include/public/io/blkif.h > --- a/xen/include/public/io/blkif.h Tue Feb 14 21:54:09 2012 -0700 > +++ b/xen/include/public/io/blkif.h Tue Feb 14 21:54:09 2012 -0700 > @@ -142,6 +142,32 @@ > * The maximum supported size of the request ring buffer in units of > * machine pages. The value must be a power of 2. > * > + * max-requests <uint32_t> > + * Default Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE) > + * Maximum Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE * max-ring-pages) > + * > + * The maximum number of concurrent, logical requests that will be > + * issued by the backend. > + * > + * Note: A logical request may span multiple ring entries. > + * > + * max-request-segments > + * Values: <uint8_t> > + * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK > + * Maximum Value: BLKIF_MAX_SEGMENTS_PER_REQUEST > + * > + * The maximum value of blkif_request.nr_segments supported by > + * the backend. > + * > + * max-request-size > + * Values: <uint32_t> > + * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK * PAGE_SIZE > + * Maximum Value: BLKIF_MAX_SEGMENTS_PER_REQUEST * PAGE_SIZEIn particular, wrt the question about a reference to their implementation, I don''t see why both max-request-segments and max-request-size are necessary here and below. What is the intended behavior when the two aren''t in sync?> + * > + * The maximum amount of data, in bytes, that can be referenced by a > + * request type that accesses frontend memory (currently BLKIF_OP_READ, > + * BLKIF_OP_WRITE, or BLKIF_OP_WRITE_BARRIER). > + * > *------------------------- Backend Device Properties ------------------------- > * > * discard-aligment > @@ -239,6 +265,33 @@ > * The size of the frontend allocated request ring buffer in units of > * machine pages. The value must be a power of 2. > * > + * max-requests > + * Values: <uint32_t> > + * Default Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE) > + * Maximum Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE * max-ring_pages)max-ring-pages? Jan> + * > + * The maximum number of concurrent, logical requests that will be > + * issued by the frontend. > + * > + * Note: A logical request may span multiple ring entries. > + * > + * max-request-segments > + * Values: <uint8_t> > + * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK > + * Maximum Value: MIN(255, backend/max-request-segments) > + * > + * The maximum value the frontend will set in the > + * blkif_request.nr_segments field. > + * > + * max-request-size > + * Values: <uint32_t> > + * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK * PAGE_SIZE > + * Maximum Value: max-request-segments * PAGE_SIZE > + * > + * The maximum amount of data, in bytes, that can be referenced by > + * a request type that accesses frontend memory (currently BLKIF_OP_READ, > + * BLKIF_OP_WRITE, or BLKIF_OP_WRITE_BARRIER). > + * > *------------------------- Virtual Device Properties ------------------------- > * > * device-type > @@ -400,11 +453,28 @@ > #define BLKIF_OP_DISCARD 5 > > /* > - * Maximum scatter/gather segments per request. > + * Maximum scatter/gather segments associated with a request header block. > * This is carefully chosen so that sizeof(blkif_ring_t) <= PAGE_SIZE. > * NB. This could be 12 if the ring indexes weren''t stored in the same > page. > */ > -#define BLKIF_MAX_SEGMENTS_PER_REQUEST 11 > +#define BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK 11 > + > +/* > + * Maximum scatter/gather segments associated with a segment block. > + */ > +#define BLKIF_MAX_SEGMENTS_PER_SEGMENT_BLOCK 14 > + > +#if __XEN_INTERFACE_VERSION__ >= 0x00040201 > +/* > + * Maximum scatter/gather segments per request (header + segment blocks). > + */ > +#define BLKIF_MAX_SEGMENTS_PER_REQUEST 255 > +#else > +/* > + * Maximum scatter/gather segments per request (header block only). > + */ > +#define BLKIF_MAX_SEGMENTS_PER_REQUEST BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK > +#endif > > /* > * NB. first_sect and last_sect in blkif_request_segment, as well as > @@ -419,9 +489,25 @@ struct blkif_request_segment { > /* @last_sect: last sector in frame to transfer (inclusive). */ > uint8_t first_sect, last_sect; > }; > +typedef struct blkif_request_segment blkif_request_segment_t; > > /* > * Starting ring element for any I/O request. > + * > + * One or more segment blocks can be inserted into the request ring > + * just after a blkif_request_t, allowing requests to operate on > + * up to BLKIF_MAX_SEGMENTS_PER_REQUEST. > + * > + * BLKIF_SEGS_TO_BLOCKS() can be used on blkif_requst.nr_segments > + * to determine the number of contiguous ring entries associated > + * with this request. > + * > + * Note: Due to the way Xen request rings operate, the producer and > + * consumer indices of the ring must be incremented by the > + * BLKIF_SEGS_TO_BLOCKS() value of the associated request. > + * (e.g. a response to a 3 ring entry request must also consume > + * 3 entries in the ring, even though only the first ring entry > + * in the response has any data.) > */ > struct blkif_request { > uint8_t operation; /* BLKIF_OP_??? */ > @@ -429,11 +515,22 @@ struct blkif_request { > blkif_vdev_t handle; /* only for read/write requests */ > uint64_t id; /* private guest value, echoed in resp */ > blkif_sector_t sector_number;/* start sector idx on disk (r/w only) */ > - struct blkif_request_segment seg[BLKIF_MAX_SEGMENTS_PER_REQUEST]; > + blkif_request_segment_t seg[BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK]; > }; > typedef struct blkif_request blkif_request_t; > > /* > + * A segment block is a ring request structure that contains only > + * segment data. > + * > + * sizeof(struct blkif_segment_block) <= sizeof(struct blkif_request) > + */ > +struct blkif_segment_block { > + blkif_request_segment_t seg[BLKIF_MAX_SEGMENTS_PER_SEGMENT_BLOCK]; > +}; > +typedef struct blkif_segment_block blkif_segment_block_t; > + > +/* > * Cast to this structure when blkif_request.operation == BLKIF_OP_DISCARD > * sizeof(struct blkif_request_discard) <= sizeof(struct blkif_request) > */ > @@ -470,6 +567,21 @@ typedef struct blkif_response blkif_resp > */ > DEFINE_RING_TYPES(blkif, struct blkif_request, struct blkif_response); > > +/* > + * Index to, and treat as a segment block, an entry in the ring. > + */ > +#define BLKRING_GET_SEG_BLOCK(_r, _idx) \ > + (((blkif_segment_block_t *)RING_GET_REQUEST(_r, _idx))->seg) > + > +/* > + * The number of ring request blocks required to handle an I/O > + * request containing _segs segments. > + */ > +#define BLKIF_SEGS_TO_BLOCKS(_segs) \ > + ((((_segs - BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK) \ > + + (BLKIF_MAX_SEGMENTS_PER_SEGMENT_BLOCK - 1)) \ > + / BLKIF_MAX_SEGMENTS_PER_SEGMENT_BLOCK) + /*header_block*/1) > + > #define VDISK_CDROM 0x1 > #define VDISK_REMOVABLE 0x2 > #define VDISK_READONLY 0x4 > diff -r 321810c42d55 -r adcb465964c5 xen/include/public/xen-compat.h > --- a/xen/include/public/xen-compat.h Tue Feb 14 21:54:09 2012 -0700 > +++ b/xen/include/public/xen-compat.h Tue Feb 14 21:54:09 2012 -0700 > @@ -27,7 +27,7 @@ > #ifndef __XEN_PUBLIC_XEN_COMPAT_H__ > #define __XEN_PUBLIC_XEN_COMPAT_H__ > > -#define __XEN_LATEST_INTERFACE_VERSION__ 0x00040200 > +#define __XEN_LATEST_INTERFACE_VERSION__ 0x00040201 > > #if defined(__XEN__) || defined(__XEN_TOOLS__) > /* Xen is built with matching headers and implements the latest interface. > */ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel
Justin T. Gibbs
2012-Feb-17 00:41 UTC
Re: [PATCH 4 of 4 v2] blkif.h: Define and document the request number/size/segments extension
On Feb 16, 2012, at 1:42 AM, Jan Beulich wrote:> >>> On 15.02.12 at 06:06, "Justin T. Gibbs" <justing@spectralogic.com> wrote: > > Note: As of __XEN_INTERFACE_VERSION__ 0x00040201 the definition of > > BLKIF_MAX_SEGMENTS_PER_REQUEST has changed. Drivers must be updated > > to, at minimum, use BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK, before being > > recompiled with a __XEN_INTERFACE_VERSION greater than or equal to > > this value. > > > > This extension first appeared in the FreeBSD Operating System. > > Any pointer to their implementation?http://svnweb.freebsd.org/base/head/sys/dev/xen/ ...> > + * max-request-segments > > + * Values: <uint8_t> > > + * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK > > + * Maximum Value: BLKIF_MAX_SEGMENTS_PER_REQUEST > > + * > > + * The maximum value of blkif_request.nr_segments supported by > > + * the backend. > > + * > > + * max-request-size > > + * Values: <uint32_t> > > + * Default Value: BLKIF_MAX_SEGMENTS_PER_HEADER_BLOCK * PAGE_SIZE > > + * Maximum Value: BLKIF_MAX_SEGMENTS_PER_REQUEST * PAGE_SIZE > > In particular, wrt the question about a reference to their implementation, > I don''t see why both max-request-segments and max-request-size are > necessary here and below. What is the intended behavior when the two > aren''t in sync?The resources needed to deal with fragmentation and overall I/O request size are distinct. The goal of the interface is to explicity indicate both the maximum size and maximum fragmentation (i.e. how many underutilized segments are allowed) for an I/O. The FreeBSD block front implementation reserves an extra segment so that a maximum sized I/O can be transferred even if it does not start on a page boundary. I can imagine many reasons why a customized system may need to handle much higher fragmentation levels. "Over provising" the segment count means you can handle this without a copy to coalesce the fragments.> > @@ -239,6 +265,33 @@ > > * The size of the frontend allocated request ring buffer in units of > > * machine pages. The value must be a power of 2. > > * > > + * max-requests > > + * Values: <uint32_t> > > + * Default Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE) > > + * Maximum Value: BLKIF_MAX_RING_REQUESTS(PAGE_SIZE * max-ring_pages) > > > > max-ring-pages?Yes. Nice catch. -- Justin