Eric Blake
2023-Apr-13 22:02 UTC
[Libguestfs] [PATCH v3 0/6] NBD 64-bit extensions (spec only)
v2 was here: https://lists.debian.org/nbd/2022/11/msg00030.html The bulk of the changes since then are: - forbid NBD_OPT_EXPORT_NAME once extended headers are negotiated (Wouter) - consistently use 'maximum payload', rather than a haphazard mix of 'maximum block payload' (Vladimir) At this point, I want to make sure we are happy with the spec before re-posting patches for qemu and libnbd (there were some slight tweaks to the spec compared to v2, which will be easy enough to code, but I want to make sure there won't be any other tweaks needed based on review comments to this proposal). I could still be persuaded that if extended headers are negotiated, then the server should ONLY send back 64-bit block status replies (as written here, and as I implemented in v2 patches for qemu and libnbd, I specified that the server can choose either 32- or 64-bit replies, based on what fits best, and regardless of whether the client made a request with effect length less than 32 bits; clients must therefore be prepared to handle both types of replies). I'm still not sure whether patch 6 is desirable; it didn't get any feedback in my v2 posting. In relation to v2, the subject line for patch 2 changed (the 'git backport-diff' tool doesn't handle that well), and the interdiff changes look bigger than they really are due to reflowed paragraphs. [----] : patches are identical [####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively 001/6:[----] [-C] 'spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length' 002/6:[down] 'spec: Change maximum block size to maximum payload size' 003/6:[0237] [FC] 'spec: Add NBD_OPT_EXTENDED_HEADERS' 004/6:[----] [-C] 'spec: Allow 64-bit block status results' 005/6:[----] [-C] 'spec: Introduce NBD_FLAG_BLOCK_STATUS_PAYLOAD' 006/6:[0062] [FC] 'RFC: spec: Introduce NBD_REPLY_TYPE_OFFSET_HOLE_EXT' Eric Blake (6): spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length spec: Change maximum block size to maximum payload size spec: Add NBD_OPT_EXTENDED_HEADERS spec: Allow 64-bit block status results spec: Introduce NBD_FLAG_BLOCK_STATUS_PAYLOAD RFC: spec: Introduce NBD_REPLY_TYPE_OFFSET_HOLE_EXT doc/proto.md | 738 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 549 insertions(+), 189 deletions(-) base-commit: fd2c5a60e36ef0ea0d7fd6f8a6bd908d24d5801d -- 2.39.2
Eric Blake
2023-Apr-13 22:02 UTC
[Libguestfs] [PATCH v3 1/6] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length
The spec was silent on how many extents a server could reply with.
However, both qemu and nbdkit (the two server implementations known to
have implemented the NBD_CMD_BLOCK_STATUS extension) implement a hard
cap, and will truncate the amount of extents in a reply to avoid
sending a client a reply so large that the client would treat it as a
denial of service attack. Clients currently have no way during
negotiation to request such a limit of the server, so it is easier to
just document this as a restriction on viable server implementations
than to add yet another round of handshaking. Also, mentioning
amplification effects is worthwhile.
When qemu first implemented NBD_CMD_BLOCK_STATUS for the
base:allocation context (qemu commit e7b1948d51, Mar 2018), it behaved
as if NBD_CMD_FLAG_REQ_ONE were always passed by the client, and never
responded with more than one extent. Later, when adding its
qemu:dirty-bitmap:XYZ context extension (qemu commit 3d068aff16, Jun
2018), it added a cap to 128k extents (1M+4 bytes), and that cap was
applied to base:allocation once qemu started sending multiple extents
for that context as well (qemu commit fb7afc797e, Jul 2018). Qemu
extents are never smaller than 512 bytes (other than an exception at
the end of a file whose size is not aligned to 512), but even so, a
request for just under 4G of block status could produce 8M extents,
resulting in a reply of 64M if it were not capped smaller.
When nbdkit first implemented NBD_CMD_BLOCK_STATUS (nbdkit 4ca66f70a5,
Mar 2019), it did not impose any restriction on the number of extents
in the reply chunk. But because it allows extents as small as one
byte, it is easy to write a server that can amplify a client's request
of status over 1M of the image into a reply over 8M in size, and it
was very easy to demonstrate that a hard cap was needed to avoid
crashing clients or otherwise killing the connection (a bad server
impacting the client negatively). So nbdkit enforced a bound of 1M
extents (8M+4 bytes, nbdkit commit 6e0dc839ea, Jun 2019). [Unrelated
to this patch, but worth noting for history: nbdkit's situation also
has to deal with the fact that it is designed for plugin server
implementations; and not capping the number of extents in a reply also
posed a problem to nbdkit as the server, where a plugin could exhaust
memory and kill the server, unrelated to any size constraints enforced
by a client.]
Since the limit chosen by these two implementations is different, and
since nbdkit has versions that were not limited, add this as a SHOULD
NOT instead of MUST NOT constraint on servers implementing block
status. It does not matter that qemu picked a smaller limit that it
truncates to, since we have already documented that the server may
truncate for other reasons (such as it being inefficient to collect
that many extents in the first place). But documenting the limit now
becomes even more important in the face of a future addition of 64-bit
requests, where a client's request is no longer bounded to 4G and
could thereby produce even more than 8M extents for the corner case
when every 512 bytes is a new extent, if it were not for this
recommendation.
Signed-off-by: Eric Blake <eblake at redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov at yandex-team.ru>
---
v2: Add wording about amplification effect
v3: Add R-b.
---
doc/proto.md | 51 +++++++++++++++++++++++++++++++--------------------
1 file changed, 31 insertions(+), 20 deletions(-)
diff --git a/doc/proto.md b/doc/proto.md
index 060deba..0cb84f2 100644
--- a/doc/proto.md
+++ b/doc/proto.md
@@ -1820,6 +1820,12 @@ MUST initiate a hard disconnect.
the different contexts need not have the same number of extents or
cumulative extent length.
+ Servers SHOULD NOT send more than 2^20 extents in a single reply
+ chunk; in other words, the size of
+ `NBD_REPLY_TYPE_BLOCK_STATUS` should not be more than 4 + 8*2^20
+ (8,388,612 bytes), even if this requires that the server truncate
+ the response in relation to the *length* requested by the client.
+
Even if the client did not use the `NBD_CMD_FLAG_REQ_ONE` flag in
its request, the server MAY return fewer descriptors in the reply
than would be required to fully specify the whole range of requested
@@ -2182,26 +2188,31 @@ The following request types exist:
`NBD_REPLY_TYPE_BLOCK_STATUS` chunk represent consecutive portions
of the export starting from specified *offset*. If the client used
the `NBD_CMD_FLAG_REQ_ONE` flag, each chunk contains exactly one
- descriptor where the *length* of the descriptor MUST NOT be greater
- than the *length* of the request; otherwise, a chunk MAY contain
- multiple descriptors, and the final descriptor MAY extend beyond
- the original requested size if the server can determine a larger
- length without additional effort. On the other hand, the server MAY
- return less data than requested. However the server MUST return at
- least one status descriptor (and since each status descriptor has
- a non-zero length, a client can always make progress on a
- successful return). The server SHOULD use different *status*
- values between consecutive descriptors where feasible, although
- the client SHOULD be prepared to handle consecutive descriptors
- with the same *status* value. The server SHOULD use descriptor
- lengths that are an integer multiple of 512 bytes where possible
- (the first and last descriptor of an unaligned query being the
- most obvious places for an exception), and MUST use descriptor
- lengths that are an integer multiple of any advertised minimum
- block size. The status flags are intentionally defined so that a
- server MAY always safely report a status of 0 for any block,
- although the server SHOULD return additional status values when
- they can be easily detected.
+ descriptor where the *length* of the descriptor MUST NOT be
+ greater than the *length* of the request; otherwise, a chunk MAY
+ contain multiple descriptors, and the final descriptor MAY extend
+ beyond the original requested size if the server can determine a
+ larger length without additional effort. On the other hand, the
+ server MAY return less data than requested. In particular, a
+ server SHOULD NOT send more than 2^20 status descriptors in a
+ single chunk. However the server MUST return at least one status
+ descriptor, and since each status descriptor has a non-zero
+ length, a client can always make progress on a successful return.
+
+ The server SHOULD use different *status* values between
+ consecutive descriptors where feasible, although the client SHOULD
+ be prepared to handle consecutive descriptors with the same
+ *status* value. The server SHOULD use descriptor lengths that are
+ an integer multiple of 512 bytes where possible (the first and
+ last descriptor of an unaligned query being the most obvious
+ places for an exception), in part to avoid an amplification effect
+ where a series of smaller descriptors can cause the server's reply
+ to occupy more bytes than the *length* of the client's request.
+ The server MUST use descriptor lengths that are an integer
+ multiple of any advertised minimum block size. The status flags
+ are intentionally defined so that a server MAY always safely
+ report a status of 0 for any block, although the server SHOULD
+ return additional status values when they can be easily detected.
If an error occurs, the server SHOULD set the appropriate error
code in the error field of an error chunk. However, if the error
--
2.39.2
Eric Blake
2023-Apr-13 22:02 UTC
[Libguestfs] [PATCH v3 2/6] spec: Change maximum block size to maximum payload size
Commit 9f30fedb improved the spec to allow non-payload requests like
NBD_CMD_TRIM that exceed any advertised maximum block size. Take this
one step further by documenting that the server may use NBD_EOVERFLOW
as a hint to the client when a non-payload request is oversize (while
permitting NBD_EINVAL for back-compat), and by rewording the text to
explicitly call out that what is being advertised is the maximum
payload length, not maximum block size. Furthermore, favor the term
'maximum payload size' instead of 'maximum block size', as the
real
limitation here is how many bytes are sent in one direction as part of
the command (a maximum payload size may be related to maximum block
size, but existing implementations of both servers and clients that
actually implement NBD_INFO_BLOCK_SIZE have generally been advertising
things like a 32M or 64M data cap, and not an underlying block size
constraint).
Document existing practice that structured replies can slightly exceed
payload size (a client requesting a 32M NBD_CMD_READ can get a single
NBD_REPLY_TYPE_OFFSET_DATA of size 32M+8 bytes, rather than the server
splitting it across two chunks); the only hard limit here is that on
client/server pairs that permit larger payloads than 32M, the reply
type still has a 32-bit limit on payload size (no known client or
server actually tries to do an NBD_CMD_READ of 4G-1 bytes, but the
spec shouldn't prevent it).
This becomes more important when we add 64-bit extensions, where it
becomes possible to extend `NBD_CMD_BLOCK_STATUS` to have both an
effect length (how much of the image does the client want status on -
may be larger than 32 bits) and an optional payload length (a way to
filter the response to a subset of negotiated metadata contexts). In
the shorter term, it means that a server may (but not must) accept a
read request larger than the maximum block size if it can use
structured replies to keep each chunk of the response under the
maximum payload limits.
Signed-off-by: Eric Blake <eblake at redhat.com>
---
v3: Completely drop 'maximum block length', and reword things even
more to emphasize maximum payload. Clarify that a server's structured
reply CAN exceed the maximum payload by the amount of overhead
required by the reply type.
---
doc/proto.md | 148 ++++++++++++++++++++++++++++++---------------------
1 file changed, 87 insertions(+), 61 deletions(-)
diff --git a/doc/proto.md b/doc/proto.md
index 0cb84f2..2651f13 100644
--- a/doc/proto.md
+++ b/doc/proto.md
@@ -747,8 +747,8 @@ text unless the client insists on TLS.
During transmission phase, several operations are constrained by the
export size sent by the final `NBD_OPT_EXPORT_NAME` or `NBD_OPT_GO`,
-as well as by three block size constraints defined here (minimum,
-preferred, and maximum).
+as well as by three block size constraints defined here (minimum
+block, preferred block, and maximum payload).
If a client can honour server block size constraints (as set out below
and under `NBD_INFO_BLOCK_SIZE`), it SHOULD announce this during the
@@ -774,15 +774,15 @@ learn the server's constraints without committing to
them.
If block size constraints have not been advertised or agreed on
externally, then a server SHOULD support a default minimum block size
-of 1, a preferred block size of 2^12 (4,096), and a maximum block size
-that is effectively unlimited (0xffffffff, or the export size if that
-is smaller), while a client desiring maximum interoperability SHOULD
-constrain its requests to a minimum block size of 2^9 (512), and limit
-`NBD_CMD_READ` and `NBD_CMD_WRITE` commands to a maximum block size of
-2^25 (33,554,432). A server that wants to enforce block sizes other
-than the defaults specified here MAY refuse to go into transmission
-phase with a client that uses `NBD_OPT_EXPORT_NAME` (via a hard
-disconnect) or which uses `NBD_OPT_GO` without requesting
+of 1, a preferred block size of 2^12 (4,096), and a maximum
+payload size that is at least 2^25 (33,554,432) (even if the export
+size is smaller); while a client desiring maximum interoperability
+SHOULD constrain its requests to a minimum block size of 2^9 (512),
+and limit `NBD_CMD_READ` and `NBD_CMD_WRITE` commands to a maximum
+payload size of 2^25 (33,554,432). A server that wants to enforce block
+sizes other than the defaults specified here MAY refuse to go into
+transmission phase with a client that uses `NBD_OPT_EXPORT_NAME` (via
+a hard disconnect) or which uses `NBD_OPT_GO` without requesting
`NBD_INFO_BLOCK_SIZE` (via an error reply of
`NBD_REP_ERR_BLOCK_SIZE_REQD`); but servers SHOULD NOT refuse clients
that do not request sizing information when the server supports
@@ -820,17 +820,45 @@ the preferred block size for that export. The server MAY
advertise an
export size that is not an integer multiple of the preferred block
size.
-The maximum block size represents the maximum length that the server
-is willing to handle in one request. If advertised, it MAY be
-something other than a power of 2, but MUST be either an integer
-multiple of the minimum block size or the value 0xffffffff for no
-inherent limit, MUST be at least as large as the smaller of the
-preferred block size or export size, and SHOULD be at least 2^20
-(1,048,576) if the export is that large. For convenience, the server
-MAY advertise a maximum block size that is larger than the export
-size, although in that case, the client MUST treat the export size as
-the effective maximum block size (as further constrained by a nonzero
-offset).
+The maximum payload size represents the maximum payload length that
+the server is willing to handle in one request from the client. If
+advertised, it MAY be something other than a power of 2, but MUST be
+at least as large as the preferred block size, and SHOULD be at least
+2^20 (1,048,576) if the export is that large. Advertising a maximum
+payload size of 0xffffffff is permitted when the server does not have
+a fixed limit on client request payloads. Typically, the advertised
+maximum payload length is independent of the export size, even though
+the actual payloads for read and write cannot successfully exceed the
+constraints given by the export size and offset of a request.
+Notwithstanding any maximum payload size advertised, either the server
+or the client MAY initiate a hard disconnect if a payload length of
+either a request or a reply would be large enough to be deemed a
+denial of service attack; however, for maximum portability, any
+payload not exceeding 2^25 (33,554,432) bytes SHOULD NOT be considered
+a denial of service attack, even if that length is larger than the
+advertised maximum payload size.
+
+For commands that require a payload in either direction and where the
+client controls the payload length (`NBD_CMD_WRITE`, or `NBD_CMD_READ`
+with simple replies), the client MUST NOT request a length larger than
+the maximum payload size. For replies where the payload length is
+controlled by the server (`NBD_CMD_BLOCK_STATUS` without the flag
+`NBD_CMD_FLAG_REQ_ONE`, or `NBD_CMD_READ` when structured replies are
+negotiated), the server MAY exceed the maximum payload by the fixed
+amount of overhead required in the structured reply (for example, a
+server that advertises a maximum payload of 2^25 bytes may return
+2^25+8 payload bytes in a single `NBD_REPLY_TYPE_OFFSET_DATA` chunk,
+rather than splitting the reply across two chunks), although it MUST
+honor any additional payload constraints documented for a particular
+command. For commands that do not require a payload in either
+direction (such as `NBD_CMD_TRIM` or `NBD_CMD_WRITE_ZEROES`), the
+client MAY request an effect length larger than the maximum payload
+size; the server SHOULD NOT disconnect, but MAY reply with an
+`NBD_EOVERFLOW` or `NBD_EINVAL` error if the oversize request would
+require too many server resources when compared to the same command
+with an effect length limited to the maximum payload size (such as an
+implementation of `NBD_CMD_WRITE_ZEROES` that utilizes a scratch
+buffer).
Where a transmission request can have a nonzero *offset* and/or
*length* (such as `NBD_CMD_READ`, `NBD_CMD_WRITE`, or `NBD_CMD_TRIM`),
@@ -838,24 +866,9 @@ the client MUST ensure that *offset* and *length* are
integer
multiples of any advertised minimum block size, and SHOULD use integer
multiples of any advertised preferred block size where possible. For
those requests, the client MUST NOT use a *length* which, when added to
-*offset*, would exceed the export size. Also for NBD_CMD_READ,
-NBD_CMD_WRITE, NBD_CMD_CACHE and NBD_CMD_WRITE_ZEROES (except for
-when NBD_CMD_FLAG_FAST_ZERO is set), the client MUST NOT use a *length*
-larger than any advertised maximum block size.
-The server SHOULD report an `NBD_EINVAL` error if
-the client's request is not aligned to advertised minimum block size
-boundaries, or is larger than the advertised maximum block size.
-Notwithstanding any maximum block size advertised, either the server
-or the client MAY initiate a hard disconnect if the payload of an
-`NBD_CMD_WRITE` request or `NBD_CMD_READ` reply would be large enough
-to be deemed a denial of service attack; however, for maximum
-portability, any *length* less than 2^25 (33,554,432) bytes SHOULD NOT
-be considered a denial of service attack (even if the advertised
-maximum block size is smaller). For all other commands, where the
-*length* is not reflected in the payload (such as `NBD_CMD_TRIM` or
-`NBD_CMD_WRITE_ZEROES`), a server SHOULD merely fail the command with
-an `NBD_EINVAL` error for a client that exceeds the maximum block size,
-rather than initiating a hard disconnect.
+*offset*, would exceed the export size. The server SHOULD report an
+`NBD_EINVAL` error if the client's request is not aligned to advertised
+minimum block size boundaries or would exceed the export size.
## Metadata querying
@@ -1597,7 +1610,7 @@ during option haggling in the fixed newstyle negotiation.
- 16 bits, `NBD_INFO_BLOCK_SIZE`
- 32 bits, minimum block size
- 32 bits, preferred block size
- - 32 bits, maximum block size
+ - 32 bits, maximum payload size
* `NBD_REP_META_CONTEXT` (4)
@@ -1740,12 +1753,15 @@ unrecognized flags.
#### Structured reply types
-These values are used in the "type" field of a structured reply.
-Some chunk types can additionally be categorized by role, such as
-*error chunks* or *content chunks*. Each type determines how to
-interpret the "length" bytes of payload. If the client receives
-an unknown or unexpected type, other than an *error chunk*, it
-MUST initiate a hard disconnect.
+These values are used in the "type" field of a structured reply.
Some
+chunk types can additionally be categorized by role, such as *error
+chunks* or *content chunks*. Each type determines how to interpret
+the "length" bytes of payload. If the client receives an unknown or
+unexpected type, other than an *error chunk*, it MUST initiate a hard
+disconnect. A server MUST NOT send a chunk where any variable-length
+portion of the chunk is larger than any advertised maximum payload
+size (however, the overall chunk may exceed the maximum payload by the
+small amount of fixed-length overhead inherent in the chunk type).
* `NBD_REPLY_TYPE_NONE` (0)
@@ -1908,7 +1924,9 @@ The following request types exist:
If structured replies were not negotiated, then a read request
MUST always be answered by a simple reply, as documented above
(using magic 0x67446698 `NBD_SIMPLE_REPLY_MAGIC`, and containing
- length bytes of data according to the client's request).
+ length bytes of data according to the client's request), which in
+ turn means any client request with a length larger than the
+ maximum payload size will fail.
If an error occurs, the server SHOULD set the appropriate error code
in the error field. The server MAY then initiate a hard disconnect.
@@ -1938,13 +1956,18 @@ The following request types exist:
request, but MAY send the content chunks in any order (the client
MUST reassemble content chunks into the correct order), and MAY
send additional content chunks even after reporting an error
- chunk. Note that a request for more than 2^32 - 8 bytes (if
- permitted by block size constraints) MUST be split into at least
- two chunks, so as not to overflow the length field of a reply
- while still allowing space for the offset of each chunk. When no
- error is detected, the server MUST send enough data chunks to
- cover the entire region described by the offset and length of the
- client's request.
+ chunk. A server MAY support read requests larger than the maximum
+ payload size by splitting the response across multiple chunks (in
+ particular, a request for more than 2^32 - 8 bytes containing data
+ rather than holes MUST be split to avoid overflowing the 32-bit
+ `NBD_REPLY_TYPE_OFFSET_DATA` length field); however, the server is
+ also permitted to reject large read requests up front, so a client
+ should be prepared to retry with smaller requests if a large
+ request fails.
+
+ When no error is detected, the server MUST send enough data chunks
+ to cover the entire region described by the offset and length of
+ the client's request.
To minimize traffic, the server MAY use a content or error chunk
as the final chunk by setting the `NBD_REPLY_FLAG_DONE` flag, but
@@ -2117,11 +2140,12 @@ The following request types exist:
If the server advertised `NBD_FLAG_SEND_FAST_ZERO` but
`NBD_CMD_FLAG_FAST_ZERO` is not set, then the server MUST NOT fail
with `NBD_ENOTSUP`, even if the operation is no faster than a
- corresponding `NBD_CMD_WRITE`. Conversely, if
- `NBD_CMD_FLAG_FAST_ZERO` is set, the server MUST fail quickly with
- `NBD_ENOTSUP` unless the request can be serviced in less time than
- a corresponding `NBD_CMD_WRITE`, and SHOULD NOT alter the contents
- of the export when returning this failure. The server's
+ corresponding `NBD_CMD_WRITE`. Conversely, if `NBD_CMD_FLAG_FAST_ZERO`
+ is set, the server SHOULD NOT fail with `NBD_EOVERFLOW` regardless of
+ the client length, MUST fail quickly with `NBD_ENOTSUP` unless the
+ request can be serviced in less time than a corresponding
+ `NBD_CMD_WRITE`, and SHOULD NOT alter the contents of the export when
+ returning an `NBD_ENOTSUP` failure. The server's
determination on whether to fail a fast request MAY depend on a
number of factors, such as whether the request was suitably
aligned, on whether the `NBD_CMD_FLAG_NO_HOLE` flag was present,
@@ -2274,7 +2298,9 @@ SHOULD return `NBD_EPERM` if it receives a write or trim
request on a
read-only export.
The server SHOULD NOT return `NBD_EOVERFLOW` except as documented in
-response to `NBD_CMD_READ` when `NBD_CMD_FLAG_DF` is supported.
+response to `NBD_CMD_READ` when `NBD_CMD_FLAG_DF` is supported, or when
+a command without payload requests a length larger than an advertised
+maximum payload length.
The server SHOULD NOT return `NBD_ENOTSUP` except as documented in
response to `NBD_CMD_WRITE_ZEROES` when `NBD_CMD_FLAG_FAST_ZERO` is
--
2.39.2
Eric Blake
2023-Apr-13 22:02 UTC
[Libguestfs] [PATCH v3 3/6] spec: Add NBD_OPT_EXTENDED_HEADERS
Add a new negotiation feature where the client and server agree to use
larger packet headers on every packet sent during transmission phase.
This has two purposes: first, it makes it possible to perform
operations like trim, write zeroes, and block status on more than 2^32
bytes in a single command. While at it, this is a great opportunity
to declare that any client opting in to extended headers will use
NBD_OPT_GO instead of the weaker NBD_OPT_EXPORT_NAME.
Second, when structured replies are active, clients have to deal with
the difference between 16- and 20-byte headers of simple
vs. structured replies, which impacts performance if the client must
perform multiple syscalls to first read the magic before knowing if
there are even additional bytes to read to learn a payload length. In
extended header mode, all headers are the same width and there are no
simple replies permitted. The layout of the reply header is more like
the request header; and including the client's offset in the reply
makes it easier to convert between absolute and relative offsets for
replies to NBD_CMD_READ. Similarly, by having extended mode use a
power-of-2 sizing, it becomes easier to manipulate arrays of headers
without worrying about an individual header crossing a cache line.
However, note that this change only affects the headers; data payloads
can still be unaligned (for example, a client performing 1-byte reads
or writes). We would need to negotiate yet another extension if we
wanted to ensure that all NBD transmission packets started on an
8-byte boundary after option haggling has completed.
Note that NBD_INFO_BLOCK_SIZE still advertises only a 32-bit value for
maximum payload size. As of this patch, it is assumed that most
clients will still be limiting NBD_CMD_READ requests to a much smaller
cap of 32M (the portable payload limit if nothing was advertised) or
64M (nbdkit chose this limit, and a client may know it is talking to
an nbdkit server even without explicit advertisement); so keeping a
32-bit payload size is still fine even while benefittng from a 64-bit
effect size. At any rate, this patch intentionally does not worry
about a client and server that can mutually agree to a maximum payload
larger than 4G by some means other than NBD_INFO_BLOCK_SIZE (we can
standardize it as a future extension if someone demonstrates a use
case for it); meanwhile, NBD_REPLY_TYPE_OFFSET_DATA will work as-is
with larger payload (since its length is part of the extended header),
but for holes, a server would have to either add a 64-bit counterpart
to NBD_REPLY_TYPE_OFFSET_HOLE or split its hole responses over
multiple reply chunks.
But for NBD_CMD_BLOCK_STATUS, it is very useful to be able to report a
status extent with a size larger than 32-bits, in some cases even if
the client's request was for an effect length smaller than 32-bits
(such as when it is known that the entire image is not sparse).
Consideration was given on whether to require that once extended
headers are negotiated, only a new 64-bit block status reply type can
be used; but in proof-of-concept implementations, I found it easy
enough to have both clients and servers to support both size block
status replies at all times, rather than being picky about the reply
type corresponding to whether extended headers were negotiated.
This spec addition was done in parallel with proof of concept
implementations in qemu (server and client), libnbd (client), and
nbdkit (server).
Signed-off-by: Eric Blake <eblake at redhat.com>
---
v3: prohibit NBD_OPT_EXPORT_NAME, clarify more regarding
NBD_REP_ERR_EXT_HEADER_REQD, wording tweaks
---
doc/proto.md | 502 +++++++++++++++++++++++++++++++++++++--------------
1 file changed, 371 insertions(+), 131 deletions(-)
diff --git a/doc/proto.md b/doc/proto.md
index 2651f13..9098c42 100644
--- a/doc/proto.md
+++ b/doc/proto.md
@@ -280,36 +280,55 @@ a soft disconnect.
### Transmission
-There are three message types in the transmission phase: the request,
-the simple reply, and the structured reply chunk. The
+There are two general message types in the transmission phase: the
+request (simple or extended), and the reply (simple, structured, or
+extended). The determination of which message headers to use is
+determined during handshaking phase, based on whether
+`NBD_OPT_STRUCTURED_REPLY` or `NBD_OPT_EXTENDED_HEADERS` was requested
+by the client and given a successful response by the server. The
transmission phase consists of a series of transactions, where the
client submits requests and the server sends corresponding replies
with either a single simple reply or a series of one or more
-structured reply chunks per request. The phase continues until
-either side terminates transmission; this can be performed cleanly
-only by the client.
+structured or extended reply chunks per request. The phase continues
+until either side terminates transmission; this can be performed
+cleanly only by the client.
Note that without client negotiation, the server MUST use only simple
replies, and that it is impossible to tell by reading the server
traffic in isolation whether a data field will be present; the simple
reply is also problematic for error handling of the `NBD_CMD_READ`
-request. Therefore, structured replies can be used to create a
-context-free server stream; see below.
+request. Therefore, structured or extended replies can be used to
+create a context-free server stream; see below.
+
+The results of client negotiation also determine whether the client
+and server will utilize only compact requests and replies, or whether
+both sides will use only extended packets. Compact messages are the
+default, but inherently limit single transactions to a 32-bit window
+starting at a 64-bit offset. Extended messages make it possible to
+perform 64-bit transactions (although typically only for commands that
+do not include a data payload). Furthermore, when only structured
+replies have been negotiated, compact messages require the client to
+perform partial reads to determine which reply packet style (16-byte
+simple or 20-byte structured) is on the wire before knowing the length
+of the rest of the reply, which can reduce client performance. With
+extended messages, all packet headers have a fixed length of 32 bytes,
+and although this results in more traffic over the network, the
+resulting layout is friendlier for performance.
Replies need not be sent in the same order as requests (i.e., requests
-may be handled by the server asynchronously), and structured reply
-chunks from one request may be interleaved with reply messages from
-other requests; however, there may be constraints that prevent
-arbitrary reordering of structured reply chunks within a given reply.
+may be handled by the server asynchronously), and structured or
+extended reply chunks from one request may be interleaved with reply
+messages from other requests; however, there may be constraints that
+prevent arbitrary reordering of reply chunks within a given reply.
Clients SHOULD use a cookie that is distinct from all other currently
pending transactions, but MAY reuse cookies that are no longer in
flight; cookies need not be consecutive. In each reply message
-(whether simple or structured), the server MUST use the same value for
-cookie as was sent by the client in the corresponding request,
-treating the cookie as an opaque field. In this way, the client can
-correlate which request is receiving a response. Note that earlier
-versions of this specification referred to a client's cookie as a
-handle.
+(whether simple, structured, or extended), the server MUST use the
+same value for cookie as was sent by the client in the corresponding
+request, treating the cookie as an opaque field. In this way, the
+client can correlate which request is receiving a response. Note that
+earlier versions of this specification referred to a client's cookie
+as a handle.
#### Ordering of messages and writes
@@ -346,7 +365,10 @@ may be useful.
#### Request message
-The request message, sent by the client, looks as follows:
+The compact request message is sent by the client when extended
+transactions are not in use (either the client did not request
+extended headers during negotiation, or the server responded that
+`NBD_OPT_EXTENDED_HEADERS` is unsupported), and looks as follows:
C: 32 bits, 0x25609513, magic (`NBD_REQUEST_MAGIC`)
C: 16 bits, command flags
@@ -356,19 +378,54 @@ C: 64 bits, offset (unsigned)
C: 32 bits, length (unsigned)
C: (*length* bytes of data if the request is of type `NBD_CMD_WRITE`)
+In the compact style, the client MUST NOT use the
+`NBD_CMD_FLAG_PAYLOAD_LEN` flag; and the only command where *length*
+represents payload length instead of effect length is `NBD_CMD_WRITE`.
+
+If negotiation agreed on extended transactions with
+`NBD_OPT_EXTENDED_HEADERS`, the client instead uses extended requests:
+
+C: 32 bits, 0x21e41c71, magic (`NBD_EXTENDED_REQUEST_MAGIC`)
+C: 16 bits, command flags
+C: 16 bits, type
+C: 64 bits, cookie
+C: 64 bits, offset (unsigned)
+C: 64 bits, payload/effect length (unsigned)
+C: (*length* bytes of data if *flags* includes `NBD_CMD_FLAG_PAYLOAD_LEN`)
+
+With extended headers, the meaning of the *length* field depends on
+whether *flags* contains `NBD_CMD_FLAG_PAYLOAD_LEN` (that many
+additional bytes of payload are present), or if the flag is absent
+(there is no payload, and *length* instead is an effect length
+describing how much of the export the request operates on). The
+command `NBD_CMD_WRITE` MUST use the flag `NBD_CMD_FLAG_PAYLOAD_LEN`
+in this mode; while other commands SHOULD avoid the flag if the
+server has not indicated extension suppport for payloads on that
+command. A server SHOULD initiate hard disconnect if a client sets
+the `NBD_CMD_FLAG_PAYLOAD_LEN` flag and uses a *length* larger than
+a server's advertised or default maximum payload length (capped at
+32 bits by the constraints of `NBD_INFO_BLOCK_SIZE`); in all other
+cases, a server SHOULD gracefully consume *length* bytes of payload
+(even if it then replies with an `NBD_EINVAL` failure because the
+particular command was not expecting a payload), and proceed with
+the next client command. Thus, only when *length* is used as an
+effective length will it utilize a full 64-bit value.
+
#### Simple reply message
The simple reply message MUST be sent by the server in response to all
-requests if structured replies have not been negotiated using
-`NBD_OPT_STRUCTURED_REPLY`. If structured replies have been negotiated, a
simple
+requests if neither structured replies (`NBD_OPT_STRUCTURED_REPLY`)
+nor extended headers (`NBD_OPT_EXTENDED_HEADERS`) have been
+negotiated. If structured replies have been negotiated, a simple
reply MAY be used as a reply to any request other than `NBD_CMD_READ`,
-but only if the reply has no data payload. The message looks as
-follows:
+but only if the reply has no data payload. If extended headers have
+been negotiated, a simple reply MUST NOT be used. The message looks
+as follows:
S: 32 bits, 0x67446698, magic (`NBD_SIMPLE_REPLY_MAGIC`; used to be
`NBD_REPLY_MAGIC`)
S: 32 bits, error (MAY be zero)
-S: 64 bits, cookie
+S: 64 bits, cookie (MUST match the request)
S: (*length* bytes of data if the request is of type `NBD_CMD_READ` and
*error* is zero)
@@ -418,9 +475,9 @@ on the chunks received.
A structured reply chunk message looks as follows:
S: 32 bits, 0x668e33ef, magic (`NBD_STRUCTURED_REPLY_MAGIC`)
-S: 16 bits, flags
+S: 16 bits, reply flags
S: 16 bits, type
-S: 64 bits, cookie
+S: 64 bits, cookie (MUST match the request)
S: 32 bits, length of payload (unsigned)
S: *length* bytes of payload data (if *length* is nonzero)
@@ -428,6 +485,47 @@ The use of *length* in the reply allows context-free
division of
the overall server traffic into individual reply messages; the
*type* field describes how to further interpret the payload.
+#### Extended reply chunk message
+
+One implementation drawback of the original structured replies via
+`NBD_OPT_STRUCTURED_REPLY` is that a client must read the magic number
+to know whether it will then be reading a 16-byte simple or 20-byte
+structured header from the server. Another is that an array of
+20-byte structures is not always cache-line friendly, compared to
+power-of-2 sizing. Therefore, a client may instead attempt to
+negotiate extended headers via `NBD_OPT_EXTENDED_HEADERS` in place of
+structured replies.
+
+If extended headers are negotiated, all server replies MUST use an
+extended reply; the server MUST NOT use simple replies even when there
+is no payload. Most fields in the extended reply have the same
+semantics as their counterparts in structured replies (including the
+use of `NBD_REPLY_FLAG_DONE` to end a sequence of one or more chunks
+comprising the overall reply). The extended reply has an additional
+field *offset* which MUST match the *offset* of the client's request
+(even in reply types such as `NBD_REPLY_TYPE_OFFSET_DATA` where the
+payload itself contains a different offset representing the middle of
+the buffer).
+
+The other difference between extended replies and structured replies
+is that the *length* field is 64 bits to match the layout of the
+extended request header. However, note that while the request flags
+determine whether a request uses a 32-bit payload length or a 64-bit
+effect length, at this time the reply length is used solely for
+payload lengths and so in practice will never exceed a 32-bit value
+(see `NBD_INFO_BLOCK_SIZE`); a client MAY initiate hard disconnect if
+a server's reply length is unexpectedly large.
+
+An extended reply chunk message looks as follows:
+
+S: 32 bits, 0x6e8a278c, magic (`NBD_EXTENDED_REPLY_MAGIC`)
+S: 16 bits, reply flags
+S: 16 bits, type
+S: 64 bits, cookie (MUST match the request)
+S: 64 bits, offset (MUST match the request)
+S: 64 bits, length of payload (unsigned)
+S: *length* bytes of payload data (if *length* is nonzero)
+
#### Terminating the transmission phase
There are two methods of terminating the transmission phase:
@@ -843,22 +941,22 @@ client controls the payload length (`NBD_CMD_WRITE`, or
`NBD_CMD_READ`
with simple replies), the client MUST NOT request a length larger than
the maximum payload size. For replies where the payload length is
controlled by the server (`NBD_CMD_BLOCK_STATUS` without the flag
-`NBD_CMD_FLAG_REQ_ONE`, or `NBD_CMD_READ` when structured replies are
-negotiated), the server MAY exceed the maximum payload by the fixed
-amount of overhead required in the structured reply (for example, a
-server that advertises a maximum payload of 2^25 bytes may return
-2^25+8 payload bytes in a single `NBD_REPLY_TYPE_OFFSET_DATA` chunk,
-rather than splitting the reply across two chunks), although it MUST
-honor any additional payload constraints documented for a particular
-command. For commands that do not require a payload in either
-direction (such as `NBD_CMD_TRIM` or `NBD_CMD_WRITE_ZEROES`), the
-client MAY request an effect length larger than the maximum payload
-size; the server SHOULD NOT disconnect, but MAY reply with an
-`NBD_EOVERFLOW` or `NBD_EINVAL` error if the oversize request would
-require too many server resources when compared to the same command
-with an effect length limited to the maximum payload size (such as an
-implementation of `NBD_CMD_WRITE_ZEROES` that utilizes a scratch
-buffer).
+`NBD_CMD_FLAG_REQ_ONE`, or `NBD_CMD_READ` when structured replies or
+extended headers are negotiated), the server MAY exceed the maximum
+payload by the fixed amount of overhead required in the structured
+reply (for example, a server that advertises a maximum payload of 2^25
+bytes may return 2^25+8 payload bytes in a single
+`NBD_REPLY_TYPE_OFFSET_DATA` chunk, rather than splitting the reply
+across two chunks), although it MUST honor any additional payload
+constraints documented for a particular command. For commands that do
+not require a payload in either direction (such as `NBD_CMD_TRIM` or
+`NBD_CMD_WRITE_ZEROES`), the client MAY request an effect length
+larger than the maximum payload size; the server SHOULD NOT
+disconnect, but MAY reply with an `NBD_EOVERFLOW` or `NBD_EINVAL`
+error if the oversize request would require too many server resources
+when compared to the same command with an effect length limited to the
+maximum payload size (such as an implementation of
+`NBD_CMD_WRITE_ZEROES` that utilizes a scratch buffer).
Where a transmission request can have a nonzero *offset* and/or
*length* (such as `NBD_CMD_READ`, `NBD_CMD_WRITE`, or `NBD_CMD_TRIM`),
@@ -901,12 +999,11 @@ The procedure works as follows:
- During transmission, a client can then indicate interest in metadata
for a given region by way of the `NBD_CMD_BLOCK_STATUS` command,
where *offset* and *length* indicate the area of interest. On
- success, the server MUST respond with one structured reply chunk of
- type `NBD_REPLY_TYPE_BLOCK_STATUS` per metadata context selected
- during negotiation, where each reply chunk is a list of one or more
- consecutive extents for that context. Each extent comes with a
- *flags* field, the semantics of which are defined by the metadata
- context.
+ success, the server MUST respond with one status type reply chunk
+ per metadata context selected during negotiation, where each reply
+ chunk is a list of one or more consecutive extents for that context.
+ Each extent comes with a *flags* field, the semantics of which are
+ defined by the metadata context.
The client's requested *length* is only a hint to the server, so the
cumulative extent length contained in a chunk of the server's reply
@@ -927,9 +1024,10 @@ same export name for the subsequent `NBD_OPT_GO` (or
sending `NBD_CMD_BLOCK_STATUS` without selecting at least one metadata
context.
-The reply to the `NBD_CMD_BLOCK_STATUS` request MUST be sent as a
-structured reply; this implies that in order to use metadata querying,
-structured replies MUST be negotiated first.
+The reply to a successful `NBD_CMD_BLOCK_STATUS` request MUST be sent
+via reply chunks; this implies that in order to use metadata querying,
+either structured replies or extended headers MUST be negotiated
+first.
Metadata contexts are identified by their names. The name MUST consist
of a namespace, followed by a colon, followed by a leaf-name. The
@@ -1104,12 +1202,12 @@ The field has the following format:
- bit 5, `NBD_FLAG_SEND_TRIM`: exposes support for `NBD_CMD_TRIM`.
- bit 6, `NBD_FLAG_SEND_WRITE_ZEROES`: exposes support for
`NBD_CMD_WRITE_ZEROES` and `NBD_CMD_FLAG_NO_HOLE`.
-- bit 7, `NBD_FLAG_SEND_DF`: do not fragment a structured reply. The
- server MUST set this transmission flag to 1 if the
- `NBD_CMD_READ` request supports the `NBD_CMD_FLAG_DF` flag, and
- MUST leave this flag clear if structured replies have not been
- negotiated. Clients MUST NOT set the `NBD_CMD_FLAG_DF` request
- flag unless this transmission flag is set.
+- bit 7, `NBD_FLAG_SEND_DF`: do not fragment a structured or extended
+ reply. The server MUST set this transmission flag to 1 if the
+ `NBD_CMD_READ` request supports the `NBD_CMD_FLAG_DF` flag, and MUST
+ leave this flag clear if neither structured replies nor extended
+ headers have been negotiated. Clients MUST NOT set the
+ `NBD_CMD_FLAG_DF` request flag unless this transmission flag is set.
- bit 8, `NBD_FLAG_CAN_MULTI_CONN`: Indicates that the server operates
entirely without cache, or that the cache it uses is shared among all
connections to the given device. In particular, if this flag is
@@ -1167,11 +1265,13 @@ of the newstyle negotiation.
A major problem of this option is that it does not support the
return of error messages to the client in case of problems. To
- remedy this, `NBD_OPT_GO` has been introduced (see below).
- A client thus SHOULD use `NBD_OPT_GO` in preference to
- `NBD_OPT_EXPORT_NAME` but SHOULD fall back to `NBD_OPT_EXPORT_NAME`
- if `NBD_OPT_GO` is not supported (not falling back will prevent
- it from connecting to old servers).
+ remedy this, `NBD_OPT_GO` has been introduced (see below). A
+ client thus SHOULD use `NBD_OPT_GO` in preference to
+ `NBD_OPT_EXPORT_NAME` but SHOULD fall back to
+ `NBD_OPT_EXPORT_NAME` if `NBD_OPT_GO` is not supported (not
+ falling back will prevent it from connecting to old servers).
+ Additionally, a client that has used `NBD_OPT_EXTENDED_HEADERS` to
+ negotiate extended headers MUST NOT use this option.
* `NBD_OPT_ABORT` (2)
@@ -1219,10 +1319,10 @@ of the newstyle negotiation.
When this command succeeds, the server MUST NOT preserve any
negotiation state (such as a request for
- `NBD_OPT_STRUCTURED_REPLY`, or metadata contexts from
- `NBD_OPT_SET_META_CONTEXT`) issued before this command. A client
- SHOULD defer all stateful option requests until after it
- determines whether encryption is available.
+ `NBD_OPT_STRUCTURED_REPLY` or `NBD_OPT_EXTENDED_HEADERS`, or
+ metadata contexts from `NBD_OPT_SET_META_CONTEXT`) issued before
+ this command. A client SHOULD defer all stateful option requests
+ until after it determines whether encryption is available.
See the section on TLS above for further details.
@@ -1303,6 +1403,10 @@ of the newstyle negotiation.
`NBD_REP_ERR_BLOCK_SIZE_REQD` error SHOULD ensure it first
sends an `NBD_INFO_BLOCK_SIZE` information reply in order
to help avoid a potentially unnecessary round trip.
+ - `NBD_REP_ERR_EXT_HEADER_REQD`: The server requires the client to
+ utilize extended headers. While this may make it easier to
+ implement a server with fewer considerations for backwards
+ compatibility, it limits connections to only recent clients.
Additionally, if TLS has not been initiated, the server MAY reply
with `NBD_REP_ERR_TLS_REQD` (instead of `NBD_REP_ERR_UNKNOWN`) to
@@ -1313,7 +1417,10 @@ of the newstyle negotiation.
export.
For backwards compatibility, clients SHOULD be prepared to also
- handle `NBD_REP_ERR_UNSUP` by falling back to using `NBD_OPT_EXPORT_NAME`.
+ handle `NBD_REP_ERR_UNSUP` by falling back to using
+ `NBD_OPT_EXPORT_NAME`; however, if the server has acknowledged a
+ client's request for `NBD_OPT_EXTENDED_HEADERS`, the server MUST
+ NOT fail `NBD_OPT_GO` with `NBD_REP_ERR_UNSUP`.
Other errors (such as `NBD_REP_ERR_SHUTDOWN`) are also possible,
as permitted elsewhere in this document, with no constraints on
@@ -1357,6 +1464,9 @@ of the newstyle negotiation.
server MUST use structured replies to the `NBD_CMD_READ`
transmission request. Other extensions that require structured
replies may now be negotiated.
+ - `NBD_REP_ERR_EXT_HEADER_REQD`: The client has already
+ successfully negotiated extended headers, which takes precedence
+ over this option.
- For backwards compatibility, clients SHOULD be prepared to also
handle `NBD_REP_ERR_UNSUP`; in this case, no structured replies
will be sent.
@@ -1364,9 +1474,10 @@ of the newstyle negotiation.
It is envisioned that future extensions will add other new
requests that may require a data payload in the reply. A server
that supports such extensions SHOULD NOT advertise those
- extensions until the client negotiates structured replies; and a
- client MUST NOT make use of those extensions without first
- enabling the `NBD_OPT_STRUCTURED_REPLY` extension.
+ extensions until the client has negotiated either structured
+ replies or extended headers; and a client MUST NOT make use of
+ those extensions without first enabling support for reply
+ payloads.
If the client requests `NBD_OPT_STARTTLS` after this option, it
MUST renegotiate structured replies and any other dependent
@@ -1377,9 +1488,10 @@ of the newstyle negotiation.
Return a list of `NBD_REP_META_CONTEXT` replies, one per context,
followed by an `NBD_REP_ACK` or an error.
- This option SHOULD NOT be requested unless structured replies have
- been negotiated first. If a client attempts to do so, a server
- MAY send `NBD_REP_ERR_INVALID`.
+ This option SHOULD NOT be requested unless structured replies or
+ extended headers have been negotiated first. If a client attempts
+ to do so, a server MAY send `NBD_REP_ERR_INVALID` or
+ `NBD_REP_ERR_EXT_HEADER_REQD`.
Data:
- 32 bits, length of export name.
@@ -1461,9 +1573,10 @@ of the newstyle negotiation.
they are interested in are selected with the final query that they
sent.
- This option MUST NOT be requested unless structured replies have
- been negotiated first. If a client attempts to do so, a server
- SHOULD send `NBD_REP_ERR_INVALID`.
+ This option MUST NOT be requested unless structured replies or
+ extended headers have been negotiated first. If a client attempts
+ to do so, a server SHOULD send `NBD_REP_ERR_INVALID` or
+ `NBD_REP_ERR_EXT_HEADER_REQD`.
A client MUST NOT send `NBD_CMD_BLOCK_STATUS` unless within the
negotiation phase it sent `NBD_OPT_SET_META_CONTEXT` at least
@@ -1500,6 +1613,48 @@ of the newstyle negotiation.
option does not select any metadata context, provided the client
then does not attempt to issue `NBD_CMD_BLOCK_STATUS` commands.
+* `NBD_OPT_EXTENDED_HEADERS` (11)
+
+ The client wishes to use extended headers during the transmission
+ phase. The client MUST NOT send any additional data with the
+ option, and the server SHOULD reject a request that includes data
+ with `NBD_REP_ERR_INVALID`.
+
+ When successful, this option takes precedence over structured
+ replies, and the server MUST also support `NBD_OPT_GO`. A client
+ MAY request structured replies first, although a server SHOULD
+ support this option even if structured replies are not negotiated.
+ Once extended headers are successfully requested, a client MUST
+ NOT use `NBD_OPT_EXPORT_NAME`.
+
+ It is envisioned that future extensions will add other new
+ requests that support a data payload in the request or reply. A
+ server that supports such extensions SHOULD NOT advertise those
+ extensions until the client has negotiated extended headers; and a
+ client MUST NOT make use of those extensions without first
+ enabling support for reply payloads.
+
+ The server replies with the following, or with an error permitted
+ elsewhere in this document:
+
+ - `NBD_REP_ACK`: Extended headers have been negotiated; the client
+ MUST use the 32-byte extended request header, with proper use of
+ `NBD_CMD_FLAG_PAYLOAD_LEN` for all commands sending a payload;
+ and the server MUST use the 32-byte extended reply header.
+ - For backwards compatibility, clients SHOULD be prepared to also
+ handle `NBD_REP_ERR_UNSUP`; in this case, only the compact
+ transmission headers will be used.
+
+ Note that a response of `NBD_REP_ERR_BLOCK_SIZE_REQD` does not
+ make sense in response to this command, but a server MAY fail with
+ that error for a later `NBD_OPT_GO` without a client request for
+ `NBD_INFO_BLOCK_SIZE`, since the use of extended headers provides
+ more incentive for a client to promise to obey block size
+ constraints.
+
+ If the client requests `NBD_OPT_STARTTLS` after this option, it
+ MUST renegotiate extended headers.
+
#### Option reply types
These values are used in the "reply type" field, sent by the server
@@ -1679,6 +1834,16 @@ case that data is an error message string suitable for
display to the user.
The request or the reply is too large to process.
+* `NBD_REP_ERR_EXT_HEADER_REQD` (2^31 + 10)
+
+ The server is unwilling to proceed with the given request (such as
+ `NBD_OPT_GO` or `NBD_OPT_SET_META_CONTEXT`) for a client that has
+ not yet requested extended headers (via
+ `NBD_OPT_EXTENDED_HEADERS`). This error is also used to indicate
+ that the server is unable to downgrade to structured replies (via
+ `NBD_OPT_STRUCTURED_REPLY`) if extended headers have already been
+ enabled.
+
### Transmission phase
#### Flag fields
@@ -1732,11 +1897,22 @@ valid may depend on negotiation during the handshake
phase.
`NBD_CMD_WRITE`, then the server MUST fail quickly with an error of
`NBD_ENOTSUP`. The client MUST NOT set this unless the server advertised
`NBD_FLAG_SEND_FAST_ZERO`.
+- bit 5, `NBD_CMD_FLAG_PAYLOAD_LEN`; only valid if extended headers
+ were negotiated via `NBD_OPT_EXTENDED_HEADERS`. If set, the
+ *length* field of the header describes a (nonzero) payload length of
+ data to be sent alongside the header; if the flag is clear, the
+ header *length* is instead an effect length for the command's
+ interaction with the export, and there is no payload after the
+ header. With extended headers, the flag MUST be set for
+ `NBD_CMD_WRITE` (as the write command always sends a payload of the
+ bytes to be written); for other commands, the flag will trigger an
+ `NBD_EINVAL` error unless the server has advertised support for an
+ extension payload form for the command.
-##### Structured reply flags
+##### Structured and extended header reply flags
This field of 16 bits is sent by the server as part of every
-structured reply.
+structured or extended header reply.
- bit 0, `NBD_REPLY_FLAG_DONE`; the server MUST clear this bit if
more structured reply chunks will be sent for the same client
@@ -1753,9 +1929,10 @@ unrecognized flags.
#### Structured reply types
-These values are used in the "type" field of a structured reply.
Some
-chunk types can additionally be categorized by role, such as *error
-chunks* or *content chunks*. Each type determines how to interpret
+These values are used in the "type" field of a structured reply, and
+when extended headers are negotiated. Some chunk types can
+additionally be categorized by role, such as *error chunks*, *content
+chunks*, or *status chunks*. Each type determines how to interpret
the "length" bytes of payload. If the client receives an unknown or
unexpected type, other than an *error chunk*, it MUST initiate a hard
disconnect. A server MUST NOT send a chunk where any variable-length
@@ -1804,13 +1981,27 @@ small amount of fixed-length overhead inherent in the
chunk type).
64 bits: offset (unsigned)
32 bits: hole size (unsigned, MUST be nonzero)
+ At this time, although servers that support extended headers are
+ permitted to accept client requests for `NBD_CMD_READ` with an
+ effect length larger than any advertised maximum payload size by
+ splitting the reply into multiple chunks, portable clients SHOULD
+ NOT request a read *length* larger than 32 bits (corresponding to
+ the maximum payload constraint implied by `NBD_INFO_BLOCK_SIZE`),
+ and therefore a 32-bit constraint on the *hole size* does not
+ represent an arbitrary limitation. Should a future scenario arise
+ where it can be demonstrated that a client and server would benefit
+ from an extension allowing a maximum payload size to be larger than
+ 32 bits, that extension would also introduce a counterpart reply
+ type that can express a 64-bit *hole size*.
+
* `NBD_REPLY_TYPE_BLOCK_STATUS` (5)
- *length* MUST be 4 + (a positive integer multiple of 8). This reply
- represents a series of consecutive block descriptors where the sum
- of the length fields within the descriptors is subject to further
- constraints documented below. A successful block status request MUST
- have exactly one status chunk per negotiated metadata context ID.
+ This chunk type is in the status chunk category. *length* MUST be
+ 4 + (a positive integer multiple of 8). This reply represents a
+ series of consecutive block descriptors where the sum of the length
+ fields within the descriptors is subject to further constraints
+ documented below. A successful block status request MUST have
+ exactly one status chunk per negotiated metadata context ID.
The payload starts with:
@@ -1852,6 +2043,43 @@ small amount of fixed-length overhead inherent in the
chunk type).
extent information at the first offset not covered by a
reduced-length reply.
+* `NBD_REPLY_TYPE_BLOCK_STATUS_EXT` (6)
+
+ This chunk type is in the status chunk category. *length* MUST be
+ 8 + (a positive multiple of 16). The semantics of this chunk mirror
+ those of `NBD_REPLY_TYPE_BLOCK_STATUS`, other than the use of a
+ larger *extent length* field, added padding in each descriptor to
+ ease alignment, and the addition of a *descriptor count* field that
+ can be used for easier client processing. This chunk type MUST NOT
+ be used unless extended headers were negotiated with
+ `NBD_OPT_EXTENDED_HEADERS`.
+
+ If the *descriptor count* field contains 0, the number of subsequent
+ descriptors is determined solely by the *length* field of the reply
+ header. However, the server MAY populate the *descriptor count*
+ field with the number of descriptors that follow; when doing this,
+ the server MUST ensure that the header *length* is equal to
+ *descriptor count* * 16 + 8.
+
+ The payload starts with:
+
+ 32 bits, metadata context ID
+ 32 bits, descriptor count
+
+ and is followed by a list of one or more descriptors, each with this
+ layout:
+
+ 64 bits, length of the extent to which the status below
+ applies (unsigned, MUST be nonzero)
+ 32 bits, padding (MUST be zero)
+ 32 bits, status flags
+
+ Note that when extended headers are in use, the client MUST be
+ prepared for the server to use either the compact or extended chunk
+ type, regardless of whether the client's hinted effect length was
+ more or less than 32 bits; but the server MUST use exactly one of
+ the two chunk types per negotiated metacontext ID.
+
All error chunk types have bit 15 set, and begin with the same
*error*, *message length*, and optional *message* fields as
`NBD_REPLY_TYPE_ERROR`. If nonzero, *message length* indicates
@@ -1866,8 +2094,9 @@ remaining structured fields at the end.
and the client MAY NOT make any assumptions about partial
success. This type SHOULD NOT be used more than once in a
structured reply. Valid as a reply to any request. Note that
- *message length* MUST NOT exceed the 4096 bytes string length
- limit.
+ *message length* MUST NOT exceed the 4096 bytes string length limit,
+ and therefore there is no need for a counterpart extended-length
+ error chunk type.
The payload is structured as:
@@ -1914,19 +2143,21 @@ The following request types exist:
A read request. Length and offset define the data to be read. The
server MUST reply with either a simple reply or a structured
- reply, according to whether the structured replies have been
- negotiated using `NBD_OPT_STRUCTURED_REPLY`. The client SHOULD NOT
- request a read length of 0; the behavior of a server on such a
+ reply, according to whether structured replies
+ (`NBD_OPT_STRUCTURED_REPLY`) or extended headers
+ (`NBD_OPT_EXTENDED_HEADERS`) were negotiated. The client SHOULD
+ NOT request a read length of 0; the behavior of a server on such a
request is unspecified although the server SHOULD NOT disconnect.
*Simple replies*
- If structured replies were not negotiated, then a read request
- MUST always be answered by a simple reply, as documented above
- (using magic 0x67446698 `NBD_SIMPLE_REPLY_MAGIC`, and containing
- length bytes of data according to the client's request), which in
- turn means any client request with a length larger than the
- maximum payload size will fail.
+ If neither structured replies nor extended headers were
+ negotiated, then a read request MUST always be answered by a
+ simple reply, as documented above (using magic 0x67446698
+ `NBD_SIMPLE_REPLY_MAGIC`, and containing length bytes of data
+ according to the client's request), which in turn means any client
+ request with a length larger than the maximum payload size will
+ fail.
If an error occurs, the server SHOULD set the appropriate error code
in the error field. The server MAY then initiate a hard disconnect.
@@ -1939,13 +2170,15 @@ The following request types exist:
*Structured replies*
- If structured replies are negotiated, then a read request MUST
- result in a structured reply with one or more chunks (each using
- magic 0x668e33ef `NBD_STRUCTURED_REPLY_MAGIC`), where the final
- chunk has the flag `NBD_REPLY_FLAG_DONE`, and with the following
- additional constraints.
+ If structured replies or extended headers are negotiated, then a
+ read request MUST result in a reply with one or more structured
+ chunks (each using `NBD_STRUCTURED_REPLY_MAGIC` or
+ `NBD_EXTENDED_REPLY_MAGIC` according to what was negotiated),
+ where the final chunk has the flag `NBD_REPLY_FLAG_DONE`, and with
+ the following additional constraints.
- The server MAY split the reply into any number of content chunks;
+ The server MAY split the reply into any number of content chunks
+ (`NBD_REPLY_TYPE_OFFSET_DATA` and `NBD_REPLY_TYPE_OFFSET_HOLE`);
each chunk MUST describe at least one byte, although to minimize
overhead, the server SHOULD use chunks with lengths and offsets as
an integer multiple of 512 bytes, where possible (the first and
@@ -1958,12 +2191,12 @@ The following request types exist:
send additional content chunks even after reporting an error
chunk. A server MAY support read requests larger than the maximum
payload size by splitting the response across multiple chunks (in
- particular, a request for more than 2^32 - 8 bytes containing data
- rather than holes MUST be split to avoid overflowing the 32-bit
- `NBD_REPLY_TYPE_OFFSET_DATA` length field); however, the server is
- also permitted to reject large read requests up front, so a client
- should be prepared to retry with smaller requests if a large
- request fails.
+ particular, if extended headers are not in use, a request for more
+ than 2^32 - 8 bytes containing data rather than holes MUST be
+ split to avoid overflowing the `NBD_REPLY_TYPE_OFFSET_DATA` length
+ field); however, the server is also permitted to reject large read
+ requests up front, so a client should be prepared to retry with
+ smaller requests if a large request fails.
When no error is detected, the server MUST send enough data chunks
to cover the entire region described by the offset and length of
@@ -2189,10 +2422,10 @@ The following request types exist:
* `NBD_CMD_BLOCK_STATUS` (7)
- A block status query request. Length and offset define the range
- of interest. The client SHOULD NOT request a status length of 0;
- the behavior of a server on such a request is unspecified although
- the server SHOULD NOT disconnect.
+ A block status query request. The effect length is a hint to the
+ server about the range of interest. The client SHOULD NOT request
+ a status length of 0; the behavior of a server on such a request
+ is unspecified although the server SHOULD NOT disconnect.
A client MUST NOT send `NBD_CMD_BLOCK_STATUS` unless within the
negotiation phase it sent `NBD_OPT_SET_META_CONTEXT` at least
@@ -2200,26 +2433,27 @@ The following request types exist:
same export name used to enter transmission phase, and where the
server returned at least one metadata context without an error.
This in turn requires the client to first negotiate structured
- replies. For a successful return, the server MUST use a structured
- reply, containing exactly one chunk of type
- `NBD_REPLY_TYPE_BLOCK_STATUS` per selected context id, where the
- status field of each descriptor is determined by the flags field
- as defined by the metadata context. The server MAY send chunks in
- a different order than the context ids were assigned in reply to
- `NBD_OPT_SET_META_CONTEXT`.
+ replies or extended headers. For a successful return, the server
+ MUST use one reply chunk per selected context id (only
+ `NBD_REPLY_TYPE_BLOCK_STATUS` for structured replies, and either
+ `NBD_REPLY_TYPE_BLOCK_STATUS` or `NBD_REPLY_TYPE_BLOCK_STATUS_EXT`
+ for extended headers). The status field of each descriptor is
+ determined by the flags field as defined by the metadata context.
+ The server MAY send chunks in a different order than the context
+ ids were assigned in reply to `NBD_OPT_SET_META_CONTEXT`.
- The list of block status descriptors within the
- `NBD_REPLY_TYPE_BLOCK_STATUS` chunk represent consecutive portions
- of the export starting from specified *offset*. If the client used
- the `NBD_CMD_FLAG_REQ_ONE` flag, each chunk contains exactly one
- descriptor where the *length* of the descriptor MUST NOT be
- greater than the *length* of the request; otherwise, a chunk MAY
- contain multiple descriptors, and the final descriptor MAY extend
- beyond the original requested size if the server can determine a
- larger length without additional effort. On the other hand, the
- server MAY return less data than requested. In particular, a
- server SHOULD NOT send more than 2^20 status descriptors in a
- single chunk. However the server MUST return at least one status
+ The list of block status descriptors within a given status chunk
+ represent consecutive portions of the export starting from
+ specified *offset*. If the client used the `NBD_CMD_FLAG_REQ_ONE`
+ flag, each chunk contains exactly one descriptor where the
+ *length* of the descriptor MUST NOT be greater than the *length*
+ of the request; otherwise, a chunk MAY contain multiple
+ descriptors, and the final descriptor MAY extend beyond the
+ original requested size if the server can determine a larger
+ length without additional effort. On the other hand, the server
+ MAY return less data than requested. In particular, a server
+ SHOULD NOT send more than 2^20 status descriptors in a single
+ chunk. However the server MUST return at least one status
descriptor, and since each status descriptor has a non-zero
length, a client can always make progress on a successful return.
@@ -2421,6 +2655,10 @@ implement the following features:
Clients that do not implement querying for block size constraints
SHOULD abide by the rules laid out in the section "Block size
constraints", above.
+- Servers that implement extended headers but desire interoperability
+ with older client implementations SHOULD NOT use the
+ `NBD_REP_ERR_EXT_HEADER_REQD` error response to `NBD_OPT_GO`, but
+ should gracefully support compact headers.
### Future considerations
@@ -2430,6 +2668,8 @@ implementations are not yet ready to support them:
- Structured replies; the Linux kernel currently does not yet implement
them.
+- Extended headers; these are similar to structured replies, but is
+ new enough that many clients and servers do not yet implement them.
## About this file
--
2.39.2
Eric Blake
2023-Apr-13 22:02 UTC
[Libguestfs] [PATCH v3 4/6] spec: Allow 64-bit block status results
There are some potential extension metadata contexts that would
benefit from a 64-bit status value. For example, Zoned Block Devices
(see https://zonedstorage.io/docs/linux/zbd-api) may want to return
the relative offset of where the next write will occur within the
zone, where a zone may be larger than 4G; creating a metacontext
"zbd:offset" that returns a 64-bit offset seems nicer than creating
two metacontexts "zbd:offset_lo" and "zbd:offset_hi" that
each return
only 32 bits of the answer.
While the addition of extended headers superficially justified leaving
room in NBD_REPLY_TYPE_BLOCK_STATUS_EXT for the purpose of alignment,
it also has the nice benefit of being useful to allow extension
metadata contexts that can actually take advantage of the padding (and
remembering that since network byte order is big-endian, the padding
is in the correct location). To ensure maximum backwards
compatibility, require that all contexts in the "base:" namespace (so
far, just "base:allocation") will only utilize 32-bit status.
---
doc/proto.md | 62 +++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 49 insertions(+), 13 deletions(-)
diff --git a/doc/proto.md b/doc/proto.md
index 9098c42..ab1b066 100644
--- a/doc/proto.md
+++ b/doc/proto.md
@@ -995,7 +995,10 @@ The procedure works as follows:
during transmission, the client MUST select one or more metadata
contexts with the `NBD_OPT_SET_META_CONTEXT` command. If needed, the
client can use `NBD_OPT_LIST_META_CONTEXT` to list contexts that the
- server supports.
+ server supports. Most metadata contexts expose no more than 32 bits
+ of information, but some metadata contexts have associated data that
+ is 64 bits in length; using such contexts requires the client to
+ first negotiate extended headers with `NBD_OPT_EXTENDED_HEADERS`.
- During transmission, a client can then indicate interest in metadata
for a given region by way of the `NBD_CMD_BLOCK_STATUS` command,
where *offset* and *length* indicate the area of interest. On
@@ -1053,7 +1056,7 @@ third-party namespaces are currently registered:
Save in respect of the `base:` namespace described below, this specification
requires no specific semantics of metadata contexts, except that all the
information they provide MUST be representable within the flags field as
-defined for `NBD_REPLY_TYPE_BLOCK_STATUS`. Likewise, save in respect of
+defined for `NBD_REPLY_TYPE_BLOCK_STATUS_EXT`. Likewise, save in respect of
the `base:` namespace, the syntax of query strings is not specified by this
document, other than the recommendation that the empty leaf-name makes
sense as a wildcard for a client query during `NBD_OPT_LIST_META_CONTEXT`,
@@ -1120,7 +1123,9 @@ should make no assumption as to its contents or stability.
For the `base:allocation` context, the remainder of the flags field is
reserved. Servers SHOULD set it to all-zero; clients MUST ignore
-unknown flags.
+unknown flags. Because fewer than 32 flags are defined, this metadata
+context does not require the use of `NBD_OPT_EXTENDED_HEADERS`, and a
+server can use `NBD_REPLY_TYPE_BLOCK_STATUS` to return results.
## Values
@@ -1493,6 +1498,18 @@ of the newstyle negotiation.
to do so, a server MAY send `NBD_REP_ERR_INVALID` or
`NBD_REP_ERR_EXT_HEADER_REQD`.
+ A server MAY support extension contexts that produce status values
+ that require more than 32 bits. The server MAY advertise such
+ contexts even if the client has not yet negotiated extended
+ headers, although it SHOULD then conclude the overall response
+ with the `NBD_REP_ERR_EXT_HEADER_REQD` error to inform the client
+ that extended headers are required to make full use of all
+ contexts advertised. However, since none of the contexts defined
+ in the "base:" namespace provide more than 32 bits of status, a
+ server MUST NOT use this failure mode when the response is limited
+ to the "base:" namespace; nor may the server use this failure
mode
+ when the client has already negotiated extended headers.
+
Data:
- 32 bits, length of export name.
- String, name of export for which we wish to list metadata
@@ -1578,6 +1595,13 @@ of the newstyle negotiation.
to do so, a server SHOULD send `NBD_REP_ERR_INVALID` or
`NBD_REP_ERR_EXT_HEADER_REQD`.
+ If a client requests a metadata context that utilizes 64-bit
+ status, but has not yet negotiated extended headers, the server
+ MUST either omit that context from its successful reply, or else
+ fail the request with `NBD_REP_ERR_EXT_HEADER_REQD`. The server
+ MUST NOT use this failure for a client request that is limited to
+ contexts in the "base:" namespace.
+
A client MUST NOT send `NBD_CMD_BLOCK_STATUS` unless within the
negotiation phase it sent `NBD_OPT_SET_META_CONTEXT` at least
once, and where the final time it was sent, it referred to the
@@ -2043,16 +2067,23 @@ small amount of fixed-length overhead inherent in the
chunk type).
extent information at the first offset not covered by a
reduced-length reply.
+ For an extension metadata context that documents that the status
+ value may potentially occupy 64 bits, a server MUST NOT use this
+ reply type unless the most-significant 32 bits of all *status*
+ values included in this reply are all zeroes. Note that if the
+ client did not negotiate extended headers, then the server already
+ guaranteed during the handshake phase that no metadata contexts
+ utilizing a 64-bit status value were negotiated.
+
* `NBD_REPLY_TYPE_BLOCK_STATUS_EXT` (6)
This chunk type is in the status chunk category. *length* MUST be
8 + (a positive multiple of 16). The semantics of this chunk mirror
those of `NBD_REPLY_TYPE_BLOCK_STATUS`, other than the use of a
- larger *extent length* field, added padding in each descriptor to
- ease alignment, and the addition of a *descriptor count* field that
- can be used for easier client processing. This chunk type MUST NOT
- be used unless extended headers were negotiated with
- `NBD_OPT_EXTENDED_HEADERS`.
+ larger *extent length* field and a 64-bit *status* field, and the
+ addition of a *descriptor count* field that can be used for easier
+ client processing. This chunk type MUST NOT be used unless extended
+ headers were negotiated with `NBD_OPT_EXTENDED_HEADERS`.
If the *descriptor count* field contains 0, the number of subsequent
descriptors is determined solely by the *length* field of the reply
@@ -2071,14 +2102,19 @@ small amount of fixed-length overhead inherent in the
chunk type).
64 bits, length of the extent to which the status below
applies (unsigned, MUST be nonzero)
- 32 bits, padding (MUST be zero)
- 32 bits, status flags
+ 64 bits, status flags
Note that when extended headers are in use, the client MUST be
prepared for the server to use either the compact or extended chunk
- type, regardless of whether the client's hinted effect length was
- more or less than 32 bits; but the server MUST use exactly one of
- the two chunk types per negotiated metacontext ID.
+ type for metadata contexts, regardless of whether the client's
+ hinted effect length was more or less than 32 bits; but the server
+ MUST use exactly one of the two chunk types per negotiated
+ metacontext ID. However, the server MUST use the extended chunk
+ type when responding to an extension metadata context that utilizes
+ a 64-bit status code where the resulting *status* value is not
+ representable in 32 bits. For metadata contexts that only return a
+ 32-bit status (including all contexts in the "base:" namespace),
the
+ most-significant 32 bits of *status* MUST be all zeroes.
All error chunk types have bit 15 set, and begin with the same
*error*, *message length*, and optional *message* fields as
--
2.39.2
Eric Blake
2023-Apr-13 22:02 UTC
[Libguestfs] [PATCH v3 5/6] spec: Introduce NBD_FLAG_BLOCK_STATUS_PAYLOAD
NBD_CMD_BLOCK_STATUS currently forces the server to reply to all
metacontext ids that the client negotiated via
NBD_OPT_SET_META_CONTEXT. But since extended headers make it easy for
the client to pass command payloads, we can allow for a client to
negotiate multiple metacontexts up front but express dynamic interest
in varying subsets of those contexts over the life of the connection,
for less wasted effort in responding to NBD_CMD_BLOCK_STATUS. This
works by having the command payload supply an effect length and a list
of ids the client is currently interested in.
Signed-off-by: Eric Blake <eblake at redhat.com>
---
doc/proto.md | 62 +++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 49 insertions(+), 13 deletions(-)
diff --git a/doc/proto.md b/doc/proto.md
index ab1b066..ddf0a89 100644
--- a/doc/proto.md
+++ b/doc/proto.md
@@ -399,17 +399,20 @@ additional bytes of payload are present), or if the flag
is absent
(there is no payload, and *length* instead is an effect length
describing how much of the export the request operates on). The
command `NBD_CMD_WRITE` MUST use the flag `NBD_CMD_FLAG_PAYLOAD_LEN`
-in this mode; while other commands SHOULD avoid the flag if the
-server has not indicated extension suppport for payloads on that
-command. A server SHOULD initiate hard disconnect if a client sets
-the `NBD_CMD_FLAG_PAYLOAD_LEN` flag and uses a *length* larger than
-a server's advertised or default maximum payload length (capped at
-32 bits by the constraints of `NBD_INFO_BLOCK_SIZE`); in all other
-cases, a server SHOULD gracefully consume *length* bytes of payload
-(even if it then replies with an `NBD_EINVAL` failure because the
-particular command was not expecting a payload), and proceed with
-the next client command. Thus, only when *length* is used as an
-effective length will it utilize a full 64-bit value.
+in this mode; most other commands omit it, although some like
+`NBD_CMD_BLOCK_STATUS` optionally support the flag in order to allow
+the client to pass additional information in the payload (where the
+command documents what the payload will contain, including the
+possibility of a separate effect length). A server SHOULD initiate
+hard disconnect if a client sets the `NBD_CMD_FLAG_PAYLOAD_LEN` flag
+and uses a *length* larger than a server's advertised or default
+maximum payload length (capped at 32 bits by the constraints of
+`NBD_INFO_BLOCK_SIZE`); in all other cases, a server SHOULD gracefully
+consume *length* bytes of payload (even if it then replies with an
+`NBD_EINVAL` failure because the particular command was not expecting
+a payload), and proceed with the next client command. Thus, only when
+*length* is used as an effective length will it utilize a full 64-bit
+value.
#### Simple reply message
@@ -1240,6 +1243,19 @@ The field has the following format:
will be faster than a regular write). Clients MUST NOT set the
`NBD_CMD_FLAG_FAST_ZERO` request flag unless this transmission flag
is set.
+- bit 12, `NBD_FLAG_BLOCK_STATUS_PAYLOAD`: Indicates that the server
+ understands the use of the `NBD_CMD_FLAG_PAYLOAD_LEN` flag to
+ `NBD_CMD_BLOCK_STATUS` to allow the client to request that the
+ server filters its response to a specific subset of negotiated
+ metacontext ids passed in via a client payload, rather than the
+ default of replying to all metacontext ids. Servers MUST NOT
+ advertise this bit unless the client successfully negotiates
+ extended headers via `NBD_OPT_EXTENDED_HEADERS`, and SHOULD NOT
+ advertise this bit in response to `NBD_OPT_EXPORT_NAME` or
+ `NBD_OPT_GO` if the client does not negotiate metacontexts with
+ `NBD_OPT_SET_META_CONTEXT`; clients SHOULD NOT set the
+ `NBD_CMD_FLAG_PAYLOAD_LEN` flag for `NBD_CMD_BLOCK_STATUS` unless
+ this transmission flag is set.
Clients SHOULD ignore unknown flags.
@@ -1930,8 +1946,11 @@ valid may depend on negotiation during the handshake
phase.
header. With extended headers, the flag MUST be set for
`NBD_CMD_WRITE` (as the write command always sends a payload of the
bytes to be written); for other commands, the flag will trigger an
- `NBD_EINVAL` error unless the server has advertised support for an
- extension payload form for the command.
+ `NBD_EINVAL` error unless the command documents an optional payload
+ form for the command and the server has implemented that form (an
+ example being `NBD_CMD_BLOCK_STATUS` providing a payload form for
+ restricting the response to a particular metacontext id, when the
+ server advertises `NBD_FLAG_BLOCK_STATUS_PAYLOAD`).
##### Structured and extended header reply flags
@@ -2478,6 +2497,23 @@ The following request types exist:
The server MAY send chunks in a different order than the context
ids were assigned in reply to `NBD_OPT_SET_META_CONTEXT`.
+ If extended headers were negotiated, a server MAY optionally
+ advertise, via the transmission flag
+ `NBD_FLAG_BLOCK_STATUS_PAYLOAD`, that it supports an alternative
+ request form where the client sets `NBD_CMD_FLAG_PAYLOAD_LEN` in
+ order to pass a payload that informs the server to limit its
+ replies to the metacontext id(s) in the client's request payload,
+ rather than giving an answer on all possible metacontext ids. If
+ the server does not support the payload form, or detects duplicate
+ or unknown metacontext ids in the client's payload, the server
+ MUST gracefully consume the client's payload before failing with
+ `NBD_EINVAL`. The payload form MUST occupy 8 + n*4 bytes, where n
+ is the number of metacontext ids the client is interested in (as
+ implied by the payload length), laid out as:
+
+ 64 bits, effect length
+ n * 32 bits, list of metacontext ids to use
+
The list of block status descriptors within a given status chunk
represent consecutive portions of the export starting from
specified *offset*. If the client used the `NBD_CMD_FLAG_REQ_ONE`
--
2.39.2
Eric Blake
2023-Apr-13 22:02 UTC
[Libguestfs] [PATCH v3 6/6] RFC: spec: Introduce NBD_REPLY_TYPE_OFFSET_HOLE_EXT
Rather than requiring all servers and clients to have a 32-bit limit
on maximum NBD_CMD_READ/WRITE sizes, we can choose to standardize
support for a 64-bit single I/O transaction now.
NBD_REPLY_TYPE_OFFSET_DATA can already handle a large reply, but
NBD_REPLY_TYPE_OFFSET_HOLE needs a 64-bit counterpart.
By standardizing this, all clients must be prepared to support both
types of hole type replies, even though most server implementations of
extended replies are likely to only send one hole type.
---
As this may mean a corner-case that gets less testing, I have
separated it into a separate optional patch. I implemented it in my
proof-of-concept, but am happy to drop this patch for what actually
goes upstream.
In particular, if we foresee clients and servers that WANT to support
a payload larger than 4G, it may be worth introducing an NBD_INFO_*
that supplies 64-bit block sizing information, rather than our current
inherent 32-bit limit of NBD_INFO_BLOCK_SIZE, at the same time as we
introduce this reply type.
---
doc/proto.md | 73 ++++++++++++++++++++++++++++++----------------------
1 file changed, 42 insertions(+), 31 deletions(-)
diff --git a/doc/proto.md b/doc/proto.md
index ddf0a89..9fa471a 100644
--- a/doc/proto.md
+++ b/doc/proto.md
@@ -2024,18 +2024,27 @@ small amount of fixed-length overhead inherent in the
chunk type).
64 bits: offset (unsigned)
32 bits: hole size (unsigned, MUST be nonzero)
- At this time, although servers that support extended headers are
- permitted to accept client requests for `NBD_CMD_READ` with an
- effect length larger than any advertised maximum payload size by
- splitting the reply into multiple chunks, portable clients SHOULD
- NOT request a read *length* larger than 32 bits (corresponding to
- the maximum payload constraint implied by `NBD_INFO_BLOCK_SIZE`),
- and therefore a 32-bit constraint on the *hole size* does not
- represent an arbitrary limitation. Should a future scenario arise
- where it can be demonstrated that a client and server would benefit
- from an extension allowing a maximum payload size to be larger than
- 32 bits, that extension would also introduce a counterpart reply
- type that can express a 64-bit *hole size*.
+* `NBD_REPLY_TYPE_OFFSET_HOLE_EXT` (3)
+
+ This chunk type is in the content chunk category. *length* MUST be
+ exactly 16. The semantics of this chunk mirror those of
+ `NBD_REPLY_TYPE_OFFSET_HOLE`, other than the use of a larger *hole
+ size* field. This chunk type MUST NOT be used unless extended
+ headers were negotiated with `NBD_OPT_EXTENDED_HEADERS`.
+
+ The payload is structured as:
+
+ 64 bits: offset (unsigned)
+ 64 bits: hole size (unsigned, MUST be nonzero)
+
+ Note that even though extended headers are in use, a server may
+ enforce a maximum effect length on `NBD_CMD_READ` comparable to an
+ advertised maximum payload length (rather than having to split a
+ large effect length over multiple chunks); in this case, no valid
+ `NBD_CMD_READ` will have an effect *length* large enough to require
+ the use of this chunk type. However, a client using extended
+ headers MUST be prepared for the server to use either the compact or
+ extended chunk type.
* `NBD_REPLY_TYPE_BLOCK_STATUS` (5)
@@ -2233,25 +2242,27 @@ The following request types exist:
the following additional constraints.
The server MAY split the reply into any number of content chunks
- (`NBD_REPLY_TYPE_OFFSET_DATA` and `NBD_REPLY_TYPE_OFFSET_HOLE`);
- each chunk MUST describe at least one byte, although to minimize
- overhead, the server SHOULD use chunks with lengths and offsets as
- an integer multiple of 512 bytes, where possible (the first and
- last chunk of an unaligned read being the most obvious places for
- an exception). The server MUST NOT send content chunks that
- overlap with any earlier content or error chunk, and MUST NOT send
- chunks that describe data outside the offset and length of the
- request, but MAY send the content chunks in any order (the client
- MUST reassemble content chunks into the correct order), and MAY
- send additional content chunks even after reporting an error
- chunk. A server MAY support read requests larger than the maximum
- payload size by splitting the response across multiple chunks (in
- particular, if extended headers are not in use, a request for more
- than 2^32 - 8 bytes containing data rather than holes MUST be
- split to avoid overflowing the `NBD_REPLY_TYPE_OFFSET_DATA` length
- field); however, the server is also permitted to reject large read
- requests up front, so a client should be prepared to retry with
- smaller requests if a large request fails.
+ (`NBD_REPLY_TYPE_OFFSET_DATA` and `NBD_REPLY_TYPE_OFFSET_HOLE` for
+ structured replies, additionally `NBD_REPLY_TYPE_OFFSET_HOLE_EXT`
+ for extended headers); each chunk MUST describe at least one byte,
+ although to minimize overhead, the server SHOULD use chunks with
+ lengths and offsets as an integer multiple of 512 bytes, where
+ possible (the first and last chunk of an unaligned read being the
+ most obvious places for an exception). The server MUST NOT send
+ content chunks that overlap with any earlier content or error
+ chunk, and MUST NOT send chunks that describe data outside the
+ offset and length of the request, but MAY send the content chunks
+ in any order (the client MUST reassemble content chunks into the
+ correct order), and MAY send additional content chunks even after
+ reporting an error chunk. A server MAY support read requests
+ larger than the maximum payload size by splitting the response
+ across multiple chunks (in particular, if extended headers are not
+ in use, a request for more than 2^32 - 8 bytes containing data
+ rather than holes MUST be split to avoid overflowing the 32-bit
+ `NBD_REPLY_TYPE_OFFSET_DATA` length field); however, the server is
+ also permitted to reject large read requests up front, so a client
+ should be prepared to retry with smaller requests if a large
+ request fails.
When no error is detected, the server MUST send enough data chunks
to cover the entire region described by the offset and length of
--
2.39.2
Possibly Parallel Threads
- [PATCH v2 3/6] spec: Add NBD_OPT_EXTENDED_HEADERS
- [PATCH v2 5/6] spec: Introduce NBD_FLAG_BLOCK_STATUS_PAYLOAD
- [nbdkit PATCH 0/7] Implement structured replies in nbd plugin
- [PATCH v2 2/6] spec: Tweak description of maximum block size
- [nbdkit PATCH v2 0/5] structured replies/.extents for nbd plugin