search for: nbdcopy

This patch is a straightforward refactoring of libnbd?s nbdcopy program, and not very interesting. However I have plans for nbdcopy (see full todo below). I would like to use this utility for virt-v2v as a replacement for ?qemu-img convert?. qemu-img has caused us a series of problems: - change in zeroing behaviour caused a big performance regression - q...

...ARGET_IS_ZERO_OPTION copy: fix layout of "long_options" table copy: rewrap error message about stuck NBD server copy/file-ops.c | 4 +- copy/main.c | 44 ++++++++++---------- copy/multi-thread-copying.c | 2 +- copy/nbd-ops.c | 5 ++- copy/nbdcopy.h | 2 +- 5 files changed, 30 insertions(+), 27 deletions(-) base-commit: 30d8e6414bdeff079394552e4227d80304b08532

...--- a/copy/nbd-ops.c +++ b/copy/nbd-ops.c @@ -482,7 +482,10 @@ nbd_ops_get_extents (struct rw *rw, size_t index, /* The server should always make progress. */ if (offset == old_offset) { - fprintf (stderr, "%s: NBD server is broken: it is not returning extent information.\nTry nbdcopy --no-extents as a workaround.\n", + fprintf (stderr, + "%s: NBD server is broken: it is not returning extent " + "information.\n" + "Try nbdcopy --no-extents as a workaround.\n", rw->name); e...

libnbd's copy/copy-nbd-error.sh was triggering an assertion failure in nbdkit: $ nbdcopy -- [ nbdkit --exit-with-parent -v --filter=error pattern 5M error-pread-rate=0.5 ] null: ... nbdkit: pattern.2: debug: error-inject: pread count=262144 offset=4718592 nbdkit: pattern.2: debug: pattern: pread count=262144 offset=4718592 nbdkit: pattern.1: debug: error-inject: pread count=262144 offs...

See companion patch: Subject: [PATCH nbdkit] server: Move size parsing code (nbdkit_parse_size) to common/include This is the second part of the patch. It adds the new human_size_parse function to libnbd and then uses it for parsing --queue-size, --request-size and --sparse. The main complication here is that there was already a common/utils/human-size.h header which ends up (eventually)

On Thu, Mar 09, 2023 at 09:50:00AM +0000, Richard W.M. Jones wrote: > When a highly multi-threaded program such as nbdcopy encounters an > error, there is a race condition in the library which can cause an > assertion failure and thus a core dump: > > (1) An error occurs on one of the threads. nbdcopy calls exit(3). > > (2) In lib/errors.c, the destructor calls pthread_key_delete. > > (3) An...

When a highly multi-threaded program such as nbdcopy encounters an error, there is a race condition in the library which can cause an assertion failure and thus a core dump: (1) An error occurs on one of the threads. nbdcopy calls exit(3). (2) In lib/errors.c, the destructor calls pthread_key_delete. (3) Another thread which is still running also...

When a highly multi-threaded program such as nbdcopy encounters an error, there is a race condition in the library which can cause an assertion failure and thus a core dump: (1) An error occurs on one of the threads. nbdcopy calls exit(3). (2) In lib/errors.c, the destructor calls pthread_key_delete. (3) Another thread which is still running also...

When a highly multi-threaded program such as nbdcopy encounters an error, there is a race condition in the library which can cause an assertion failure and thus a core dump: (1) An error occurs on one of the threads. nbdcopy calls exit(3). (2) In lib/errors.c, the destructor calls pthread_key_delete. (3) Another thread which is still running also...

On Thu, Mar 09, 2023 at 08:44:51AM +0000, Richard W.M. Jones wrote: > When a highly multi-threaded program such as nbdcopy encounters an > error, there is a race condition in the library which can cause an > assertion failure and thus a core dump: > > (1) An error occurs on one of the threads. nbdcopy calls exit(3). > > (2) In lib/errors.c, the destructor calls pthread_key_delete. > > (3) An...

When a highly multi-threaded program such as nbdcopy encounters an error, there is a race condition in the library which can cause an assertion failure and thus a core dump: (1) An error occurs on one of the threads. nbdcopy calls exit(3). (2) In lib/errors.c, the destructor calls pthread_key_delete. (3) Another thread which is still running also...

This version simply removes the call to pthread_key_destroy. It fixes the problem and allows us to leave the asserts alone so we can still catch real errors. Of course this leaks pthread_key_t in the case where you dlclose() the library. glibc has a limit of 128 thread-specific keys (and the first 32 are somehow "fast" in a way I could quite follow), so that's a thing. Rich.

...--- a/copy/nbd-ops.c +++ b/copy/nbd-ops.c @@ -482,7 +482,10 @@ nbd_ops_get_extents (struct rw *rw, size_t index, /* The server should always make progress. */ if (offset == old_offset) { - fprintf (stderr, "%s: NBD server is broken: it is not returning extent information.\nTry nbdcopy --no-extents as a workaround.\n", + fprintf (stderr, + "%s: NBD server is broken: it is not returning extent " + "information.\n" + "Try nbdcopy --no-extents as a workaround.\n", rw->name); e...

I'm out of time this weekend, but while trying to write a test for an nbdkit bug fix (a nasty assertion failure when the client disconnects uncleanly without NBD_CMD_DISC, which is what nbdcopy does if it gets an EIO read response from the server), I realized it is extremely hard to trigger this using nbdsh, but easy to do in C or the Go bindings. In the Go bindings, we intentionally coded things so that the Go structure knows if its underlying C pointer is live or not; it exposes a way...

...n/verify.html If the output is produced with unordered writes, but the complete output needs to be verified with a hash *chain*, that still allows for some level of asynchrony. The start of the hashing need not be delayed until after the end of output, only after the start of output. For example, nbdcopy could maintain the highest offset up to which the output is contiguous, and on a separate thread, it could be hashing the output up to that offset. Considering a gigantic output, as yet unassembled blocks could likely not be buffered in memory (that's why the writes are unordered in the first...

...put is produced with unordered writes, but the complete output > needs to be verified with a hash *chain*, that still allows for some > level of asynchrony. The start of the hashing need not be delayed until > after the end of output, only after the start of output. > > For example, nbdcopy could maintain the highest offset up to which the > output is contiguous, and on a separate thread, it could be hashing the > output up to that offset. > > Considering a gigantic output, as yet unassembled blocks could likely > not be buffered in memory (that's why the writes ar...

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2172516 I figured I'd work on the libnbd line wrapping in shorter waves than how long the nbdkit patch series was. Laszlo Laszlo Ersek (4): copy: rename (LONG|SHORT)_OPTIONS to (LONG|SHORT)_OPTIONS_OPTION copy: rename <purpose>_OPTION to OPT_<purpose> copy: fix layout of "long_options" table copy: rewrap

This is the first part of a pair of patch series which aim to let us use nbdkit_parse_size (or rather, an equivalent common function) in nbdcopy, so we can write: nbdcopy --request-size=32M ... We can't do that now which was annoying me earlier in the week. This commit creates a new function called human_size_parse which is basically nbdkit_parse_size, and turns nbdkit_parse_size into a wrapper around it. Rich.

...acde..d2f415dc47 100644 --- a/copy/main.c +++ b/copy/main.c @@ -39,7 +39,7 @@ #include <libnbd.h> #include "ispowerof2.h" -#include "human-size.h" +#include "../utils/human-size.h" #include "minmax.h" #include "version.h" #include "nbdcopy.h" diff --git a/info/show.c b/info/show.c index 920bbb0a27..99b7b5b60a 100644 --- a/info/show.c +++ b/info/show.c @@ -29,7 +29,7 @@ #include <libnbd.h> #include "ansi-colours.h" -#include "human-size.h" +#include "../utils/human-size.h" #include "s...

On Mon, Feb 27, 2023 at 08:42:23AM -0600, Eric Blake wrote: > Or intentionally choose a hash that can be computed out-of-order, such > as a Merkle Tree. But we'd need a standard setup for all parties to > agree on how the hash is to be computed and checked, if it is going to > be anything more than just a linear hash of the entire guest-visible > contents. Unfortunately I