Eric Blake
2022-Nov-14 22:41 UTC
[Libguestfs] [cross-project PATCH v2] NBD 64-bit extensions
This is a cover letter for a set of multi-project patch series all designed to implement 64-bit operations in NBD, and demonstrate interoperability of the new extension between projects. v1 of the project was attempted nearly a year ago: https://lists.nongnu.org/archive/html/qemu-devel/2021-12/msg00453.html Since then, I've addressed a lot of preliminary cleanups in libnbd to make it easier to test things, and incorporated review comments issued on v1, including: - no orthogonality between simple/structured replies and 64-bit mode: once extended headers are negotiated, all transmission traffic (both from client to server and server to client) uses just one header size - add support for the client to pass a payload on commands to the server, and demonstrate it by further implementing a way to pass a flag with NBD_CMD_BLOCK_STATUS that says the client is passing a payload to request status of just a subset of the negotiated contexts, rather than all possible contexts that were earlier negotiated during NBD_OPT_SET_META_CONTEXT - tweaks to the header layouts: tweak block status to provide 64-bit flags values (although "base:allocation" will remain usable with just 32-bit flags); reply with offset rather than padding and with fields rearranged for maximal sharing between client and server layouts - word-smithing of the NBD proposed protocol; split things into several smaller patches, where we can choose how much to take - more unit tests added in qemu and libnbd - the series end with RFC patches on whether to support 64-bit hole responses to NBD_CMD_READ, even though our current limitations say servers don't have to accept more than a 32M read request and therefore will never have that big of a hole to read) -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Eric Blake
2022-Nov-14 22:46 UTC
[Libguestfs] [PATCH v2 0/6] NBD spec changes for 64-bit extensions
This is the NBD spec series; there are matching qemu and libnbd patches that implement the changes in this series. I'm happy to drop the RFC patches from all three, but wanted the conversation on whether it makes sense to have 64-bit holes during NBD_CMD_READ first (it would make more sense if we had a way for a client and server to agree on a single-transaction buffer limit much larger than 32M). Eric Blake (6): spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length spec: Tweak description of maximum block size spec: Add NBD_OPT_EXTENDED_HEADERS spec: Allow 64-bit block status results spec: Introduce NBD_FLAG_BLOCK_STATUS_PAYLOAD RFC: spec: Introduce NBD_REPLY_TYPE_OFFSET_HOLE_EXT doc/proto.md | 698 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 521 insertions(+), 177 deletions(-) -- 2.38.1
Eric Blake
2022-Nov-14 22:48 UTC
[Libguestfs] [PATCH v2 00/15] qemu patches for 64-bit NBD extensions
This series implements the spec changes in a counterpart NBD series, and has been tested to be interoperable with libnbd implementing the same spec. I'm not too happy with the RFC patch at the end, but implemented it for discussion. Given the release timing, this would be qemu 8.0 material if we are happy with the direction the spec is headed in. Eric Blake (15): nbd/client: Add safety check on chunk payload length nbd/server: Prepare for alternate-size headers nbd: Prepare for 64-bit request effect lengths nbd: Add types for extended headers nbd/server: Refactor handling of request payload nbd/server: Refactor to pass full request around nbd/server: Initial support for extended headers nbd/server: Support 64-bit block status nbd/client: Initial support for extended headers nbd/client: Accept 64-bit block status chunks nbd/client: Request extended headers during negotiation nbd/server: Prepare for per-request filtering of BLOCK_STATUS nbd/server: Add FLAG_PAYLOAD support to CMD_BLOCK_STATUS RFC: nbd/client: Accept 64-bit hole chunks RFC: nbd/server: Send 64-bit hole chunk docs/interop/nbd.txt | 1 + include/block/nbd.h | 163 +++-- nbd/nbd-internal.h | 8 +- block/nbd.c | 102 ++- nbd/client-connection.c | 1 + nbd/client.c | 132 +++- nbd/common.c | 14 +- nbd/server.c | 636 +++++++++++++----- qemu-nbd.c | 3 + block/trace-events | 1 + nbd/trace-events | 11 +- tests/qemu-iotests/223.out | 18 +- tests/qemu-iotests/233.out | 5 + tests/qemu-iotests/241.out | 3 + tests/qemu-iotests/307.out | 15 +- .../tests/nbd-qemu-allocation.out | 3 +- 16 files changed, 797 insertions(+), 319 deletions(-) -- 2.38.1
Eric Blake
2022-Nov-14 22:51 UTC
[Libguestfs] [libnbd PATCH v2 00/23] libnbd 64-bit NBD extensions
This series is posted alongside a spec change to NBD, and interoperable with changes posted to qemu-nbd/qemu-storage-daemon. The RFC patch at the end is optional; ineroperability with qemu works only when either both projects omit the RFC patch, or when both projects include it (if only one of the two RFC projects include it, the protocol is incompatible between the two, but at least client and server gracefully detect the bug rather than SEGV'ing). Eric Blake (23): block_status: Refactor array storage internal: Refactor layout of replies in sbuf protocol: Add definitions for extended headers states: Prepare to send 64-bit requests states: Prepare to receive 64-bit replies states: Break deadlock if server goofs on extended replies generator: Add struct nbd_extent in prep for 64-bit extents block_status: Track 64-bit extents internally block_status: Accept 64-bit extents during block status api: Add [aio_]nbd_block_status_64 api: Add several functions for controlling extended headers copy: Update nbdcopy to use 64-bit block status dump: Update nbddump to use 64-bit block status info: Expose extended-headers support through nbdinfo info: Update nbdinfo --map to use 64-bit block status examples: Update copy-libev to use 64-bit block status ocaml: Add example for 64-bit extents generator: Actually request extended headers api: Add nbd_[aio_]opt_extended_headers() interop: Add test of 64-bit block status api: Add nbd_can_block_status_payload() api: Add nbd_[aio_]block_status_filter() RFC: pread: Accept 64-bit holes docs/libnbd.pod | 18 +- info/nbdinfo.pod | 21 +- sh/nbdsh.pod | 2 +- lib/internal.h | 42 +- lib/nbd-protocol.h | 120 ++-- generator/API.ml | 532 +++++++++++++++--- generator/API.mli | 1 + generator/C.ml | 24 +- generator/GoLang.ml | 24 + generator/Makefile.am | 3 +- generator/OCaml.ml | 18 +- generator/Python.ml | 20 +- generator/state_machine.ml | 50 +- generator/states-issue-command.c | 33 +- .../states-newstyle-opt-extended-headers.c | 110 ++++ generator/states-newstyle-opt-starttls.c | 7 +- .../states-newstyle-opt-structured-reply.c | 3 +- generator/states-newstyle.c | 3 + generator/states-reply-simple.c | 4 +- generator/states-reply-structured.c | 279 ++++++--- generator/states-reply.c | 57 +- lib/aio.c | 7 +- lib/flags.c | 11 + lib/handle.c | 25 +- lib/opt.c | 44 ++ lib/rw.c | 250 +++++++- python/t/110-defaults.py | 1 + python/t/120-set-non-defaults.py | 2 + python/t/465-block-status-64.py | 56 ++ ocaml/examples/Makefile.am | 3 +- ocaml/examples/extents64.ml | 42 ++ ocaml/helpers.c | 20 + ocaml/nbd-c.h | 3 +- ocaml/tests/Makefile.am | 1 + ocaml/tests/test_110_defaults.ml | 2 + ocaml/tests/test_120_set_non_defaults.ml | 3 + ocaml/tests/test_465_block_status_64.ml | 58 ++ tests/Makefile.am | 4 + tests/meta-base-allocation.c | 111 +++- tests/pwrite-extended.c | 112 ++++ examples/copy-libev.c | 21 +- examples/server-flags.c | 7 +- interop/Makefile.am | 18 + interop/block-status-payload.c | 241 ++++++++ interop/block-status-payload.sh | 80 +++ interop/large-status.c | 186 ++++++ interop/large-status.sh | 49 ++ interop/opt-extended-headers.c | 153 +++++ interop/opt-extended-headers.sh | 29 + .gitignore | 4 + copy/nbd-ops.c | 22 +- dump/dump.c | 27 +- fuzzing/libnbd-fuzz-wrapper.c | 22 +- golang/Makefile.am | 1 + golang/handle.go | 8 +- golang/libnbd_110_defaults_test.go | 8 + golang/libnbd_120_set_non_defaults_test.go | 12 + golang/libnbd_465_block_status_64_test.go | 119 ++++ info/can.c | 14 + info/info-can.sh | 30 + info/info-packets.sh | 17 +- info/main.c | 7 +- info/map.c | 67 +-- info/show.c | 9 +- 64 files changed, 2920 insertions(+), 357 deletions(-) create mode 100644 generator/states-newstyle-opt-extended-headers.c create mode 100644 python/t/465-block-status-64.py create mode 100644 ocaml/examples/extents64.ml create mode 100644 ocaml/tests/test_465_block_status_64.ml create mode 100644 tests/pwrite-extended.c create mode 100644 interop/block-status-payload.c create mode 100755 interop/block-status-payload.sh create mode 100644 interop/large-status.c create mode 100755 interop/large-status.sh create mode 100644 interop/opt-extended-headers.c create mode 100755 interop/opt-extended-headers.sh create mode 100644 golang/libnbd_465_block_status_64_test.go -- 2.38.1