similar to: [PATCH libnbd] PROPOSAL Add nbdcp (NBD copying) tool.

I will be following up to this email with four separate threads each addressed to the appropriate single list, with proposed changes to: - the NBD protocol - qemu: both server and client - libnbd: client - nbdkit: server The feature in question adds a new optional NBD_INFO_ packet to the NBD_OPT_GO portion of handshake, adding up to 16 bits of information that the server can advertise to the

This program allows you to turn a network block device source into a FUSE filesystem containing a virtual file: $ nbdkit memory 128M $ mkdir mp $ nbdfuse mp/ramdisk nbd://localhost & $ ls -l mp total 0 -rw-rw-rw-. 1 rjones rjones 134217728 Oct 12 15:09 ramdisk $ dd if=/dev/urandom bs=1M count=128 of=mp/ramdisk conv=notrunc,nocreat 128+0 records in 128+0 records out

This replaces 13/13 of my v2 series; and now that it has pretty good testsuite coverage and demonstrable performance improvement to nbdinfo, I'm going ahead and pushing this now. We may still want to add further nbd_opt_* commands for other fine-grained tuning of negotiation, but for now, I think things have stabilized on this end, and I can return to polishing .list_exports on the nbdkit

Fixes bugs with JSON output in the previous patch, and adds some tests. Rich.

Some of these affect attempts to connect to older qemu-nbd versions, some of them were triggered by manual edits to qemu-nbd source code to provoke various other compliant (if uncommon) server behaviors. Eric Blake (4): starttls: Skip error payload if falling back to unencrypted states: Reject payload to NBD_REP_ACK meta-context: Skip error payload if server lacks meta_context states: Add

This patch is a straightforward refactoring of libnbd?s nbdcopy program, and not very interesting. However I have plans for nbdcopy (see full todo below). I would like to use this utility for virt-v2v as a replacement for ?qemu-img convert?. qemu-img has caused us a series of problems: - change in zeroing behaviour caused a big performance regression - qemu-img reads extents up-front which

This is a cover letter to a series of patches being proposed in tandem to four different projects: - nbd: Document a new NBD_CMD_FLAG_FAST_ZERO command flag - qemu: Implement the flag for both clients and server - libnbd: Implement the flag for clients - nbdkit: Implement the flag for servers, including the nbd passthrough client If you want to test the patches together, I've pushed a

This works, but there's no time saving and I'm still investigating whether it does what I think it does. Nevertheless I thought I would post it because it (probably) implements the idea I had last night outlined in: https://www.redhat.com/archives/libguestfs/2019-June/msg00010.html The meat of the change is patch 4. Patch 5 is an example which I would probably fold into patch 4 for

A compliant server should not send NBD_REPLY_TYPE_BLOCK_STATUS unless we successfully negotiated a meta context. And our default strictness settings refuse to let us send NBD_CMD_BLOCK_STATUS unless we negotiated a meta context. But when you mix non-default settings (using nbd_set_strict to disable STRICT_COMMANDS) to send a block status without having negotiated it, coupled with a non-compliant

nbdkit 1.2 as a server waits for read() to see EOF, even after the client has sent NBD_CMD_DISC. That was fixed in mbdkit 1.4; and most modern NBD servers are smarter than this (they close() the write end of their traffic soon after NBD_CMD_DISC). But it's easy enough to revert nbdkit commit c70616f8 to get back to a server with the same behavior as the older nbdkit, at which point both

Well, I'm not quite done (I still want to get nbdinfo to work on a single nbd connection for all cases when reading the heads of the file is not required), but I'm happy with patches 1-11, and 12-13 show where I'm headed for getting NBD_OPT_INFO to work. Posting now to see if some of the earlier patches are ready to commit while I continue working on the latter half. Eric Blake (13):

On 9/26/20 11:36 AM, Richard W.M. Jones wrote: > --- > info/Makefile.am | 4 + > info/info-map-base-allocation-json.sh | 52 ++++++++++ > info/info-map-base-allocation.sh | 49 ++++++++++ > info/nbdinfo.c | 134 ++++++++++++++++++++++++-- > info/nbdinfo.pod | 39 +++++++- > 5 files changed, 271

See companion patch: Subject: [PATCH nbdkit] server: Move size parsing code (nbdkit_parse_size) to common/include This is the second part of the patch. It adds the new human_size_parse function to libnbd and then uses it for parsing --queue-size, --request-size and --sparse. The main complication here is that there was already a common/utils/human-size.h header which ends up (eventually)

When we added support for .extents, we had nbdkit unconditionally support structured replies if the client requests them, and the plugin's .can_extents has no impact on what the server advertises. However, while the plugin API doesn't care whether the client requested SR, there are still integration situations where not advertising SR can be useful (such as comparison on what a client does

For these simple compile tests where we want to ensure that a basic external program could be compiled using libnbd, we shouldn't include <config.h>. This is because we want to test here that <libnbd.h> can stand alone, without needing anything defined by the GNU autotools infrastructure. Of course we can use <config.h> in other test programs where we aren't evaluating

On 08/03/2018 02:28 PM, Nir Soffer wrote: > File systems not supporting FALLOC_FL_ZERO_RANGE yet fall back to manual > zeroing. > > We can avoid this by combining two fallocate calls: > > fallocate(FALLOC_FL_PUNCH_HOLE) > fallocate(0) > > Based on my tests this is much more efficient compared to manual > zeroing. The idea came from this qemu patch: >

As mentioned in commits 176fc4ea and 609c25f0, our original plan in adding a flags argument to nbd_shutdown was to let us specify different behaviors at the libnbd level, rather than NBD protocol flags (for that, the user has nbd_aio_disconnect). But when we later parameterized OFlags to accept various bitmasks (commit f891340b), we failed to mark nbd_shutdown as using a different bitmask than

The DF flag is only available to clients that negotiated structured replies, and even then, the spec does not require that it be implemented. However, since our current implementation can't fragment NBD_CMD_READ replies, we trivially implement the flag (by ignoring it); we don't even have to pass it on to the plugins. Enhance some documentation about sparse reads while at it (when we

I'm posting this now, as I'm at the end of a workday and I got things working for manual experimentation. Still to do: - write interop tests for qemu-nbd and nbdkit (including my proposed patch addition of qemu-nbd -A to show qemu:allocation-depth) - figure out if we can make 'nbdinfo --map' use the new API to automatically select all contexts advertised by the server Eric Blake

We should have only one NBD protocol file. Let's make nbdkit's version the canonical one, and use it in libnbd. Rich.