search for: may_trim

In preparation for adding other optional flag arguments to the python bindings, start by making the existing 'may_trim' flag to 'zero' be optional. That is, the plugin need not define the parameter if it does not make any semantic difference (ie. if the plugin ignores the hint and never trims); while if the parameter exists, we now pass it as a keyword argument rather than as a positional argument. Th...

In preparation for adding other optional flag arguments to the python bindings, start by making the existing 'may_trim' flag to 'zero' be optional. That is, the plugin need not define the parameter if it does not make any semantic difference (ie. if the plugin ignores the hint and never trims); while if the parameter exists, we now pass it as a keyword argument rather than as a positional argument. Th...

...ugins/file/file.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/plugins/file/file.c b/plugins/file/file.c index b6e33de..a7c07fb 100644 --- a/plugins/file/file.c +++ b/plugins/file/file.c @@ -243,7 +243,7 @@ file_zero (void *handle, uint32_t count, uint64_t offset, int may_trim) if (may_trim) { r = fallocate (h->fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, offset, count); - if (r == -1 && errno != EOPNOTSUPP) { + if (r == -1 && errno != EOPNOTSUPP && errno != ENODEV) { nbdkit_error ("zero: %m"); }...

...re lines of code (and that's without even implementing the cache/can_cache callback present in Rich's proposal). Thus, this is an RFC to see which pieces we like best out of the various proposals, rather than something that should be applied as-is. Eric Blake (5): python: Let zero's may_trim parameter be optional python: Expose can_zero callback python: Update internals to plugin API level 2 python: Expose FUA support python: Add can_fast_zero support plugins/python/nbdkit-python-plugin.pod | 122 ++++++++-- plugins/python/python.c | 286 ++++++++++++++++++++++...

...if ((flags & NBDKIT_FLAG_FUA) && file_flush (handle, 0) == -1) + return -1; + return 0; } @@ -323,20 +350,20 @@ do_fallocate(int fd, int mode, off_t offset, off_t len) /* Write zeroes to the file. */ static int -file_zero (void *handle, uint32_t count, uint64_t offset, int may_trim) +file_zero (void *handle, uint32_t count, uint64_t offset, uint32_t flags) { struct handle *h = handle; int r; #ifdef FALLOC_FL_PUNCH_HOLE - if (h->can_punch_hole && may_trim) { + if (h->can_punch_hole && (flags & NBDKIT_FLAG_MAY_TRIM)) { r = do_fallocate...

...;<fs> mount /dev/sda1 / # ><fs> [etc] +include Nbdkit + $disk = "\0" * (1024 * 1024) def config(key, value) @@ -50,3 +52,12 @@ def pwrite(h, buf, offset) # Hmm, is this using bytes or chars? XXX $disk[offset, buf.length] = buf end + +def zero(h, count, offset, may_trim) + if may_trim + $disk[offset, count] = "\0" * count + else + set_error(Errno::EOPNOTSUPP) + raise "falling back to pwrite" + end +end diff --git a/plugins/ruby/nbdkit-ruby-plugin.pod b/plugins/ruby/nbdkit-ruby-plugin.pod index 0131574..6cf8e97 100644 --- a/plugins/r...

...(initially all # zero bytes). There is one disk per nbdkit instance, so if you # reconnect to the same server you should see the same disk. You @@ -56,3 +59,11 @@ def pwrite(h, buf, offset): global disk end = offset + len (buf) disk[offset:end] = buf + +def zero(h, count, offset, may_trim): + global disk + if may_trim: + disk[offset:offset+count] = bytearray(count) + else: + nbdkit.set_error(errno.EOPNOTSUPP) + raise Exception diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod index 8b3d08c..b78c0e2 100644 ---...

...(initially all # zero bytes). There is one disk per nbdkit instance, so if you # reconnect to the same server you should see the same disk. You @@ -56,3 +59,11 @@ def pwrite(h, buf, offset): global disk end = offset + len (buf) disk[offset:end] = buf + +def zero(h, count, offset, may_trim): + global disk + if may_trim: + disk[offset:offset+count] = bytearray(count) + else: + nbdkit.set_error(errno.EOPNOTSUPP) + raise Exception diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod index a1e3d2b..1c57e15 100644 ---...

...count, uint64_t offset) > +{ > + write_delay (); > + return next->pwrite (nxdata, buf, count, offset); > +} > + > +/* Zero data. */ > +static int > +delay_zero (struct nbdkit_next *next, void *nxdata, > + void *handle, uint32_t count, uint64_t offset, int may_trim) > +{ > + write_delay (); > + return next->zero (nxdata, count, offset, may_trim); If next->zero() fails with EOPNOTSUPP, that means we will delay once in trying the underlying command, and again for each iteration of the fallback loop as it calls delay_pwrite(). Is that okay, or...

On 11/22/19 3:46 PM, Nir Soffer wrote: >>>> and for zero (once fast zero is supported later in the series), it could >>>> look like: >>>> >>>> def zero(h, count, offset, may_trim=False, fua=False, fast=False): >>> >>> This is nicer - but not extensible. >> >> Why not? Any future flag additions would still appear as new key=value >> kwargs. > > Because there is no **kwargs argument, you will get TypeError when adding new > argum...

Begin the language binding followups to my new .zero callback, since Rich was indeed correct that we want them. I'm more familiar with python and perl (at least to the point that I was able to modify the appropriate example files and prove to myself that the bindings worked), so I've started with those. I'm less familiar with ruby and ocaml, so I've left those for tomorrow (it

...: because it is making frequent requests there > > is no timeout. > > > > (2) I suspect that because we don't have trim support that this is > > actually causing the disk to get fully allocated on the target. > > > > The NBD requests are sent with may_trim=1 so we could turn these > > into trim requests, but obviously cannot do that while there is no > > trim support. > > > > It sounds like nbdkit is emulating trim with zero instead of noop. > > I'm not sure why qemu-img is trying to do, I hope the nbd main...

...-def pread(h, count, offset): +def pread(h, count, offset, flags): global disk return disk[offset:offset+count] -def pwrite(h, buf, offset): +def pwrite(h, buf, offset, flags): global disk end = offset + len(buf) disk[offset:end] = buf -def zero(h, count, offset, may_trim): +def zero(h, count, offset, flags): global disk - if may_trim: + if flags & nbdkit.FLAG_MAY_TRIM: disk[offset:offset+count] = bytearray(count) else: nbdkit.set_error(errno.EOPNOTSUPP) diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdki...

...the server does not support trim. > > > > What qemu-img sends shouldn't be a NBD_CMD_TRIM request (which is indeed > > advisory), but a NBD_CMD_WRITE_ZEROES request. qemu-img relies on the > > image actually being zeroed after this. > > > > So it seems that may_trim=1 is wrong, since trim cannot replace zero. Note that the current plugin ignores may_trim. It is not used at all, so it's not relevant to this problem. However this flag actually corresponds to the inverse of NBD_CMD_FLAG_NO_HOLE which is defined by the NBD spec as: bit 1, NBD_CMD_FLAG_...

...-def pread(h, count, offset): +def pread(h, count, offset, flags): global disk return disk[offset:offset+count] -def pwrite(h, buf, offset): +def pwrite(h, buf, offset, flags): global disk end = offset + len(buf) disk[offset:end] = buf -def zero(h, count, offset, may_trim): +def zero(h, count, offset, flags): global disk - if may_trim: + if flags & nbdkit.FLAG_MAY_TRIM: disk[offset:offset+count] = bytearray(count) else: nbdkit.set_error(errno.EOPNOTSUPP) diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdki...

...rtions(+), 4 deletions(-) > > > >diff --git a/plugins/file/file.c b/plugins/file/file.c > >index b6e33de..a7c07fb 100644 > >--- a/plugins/file/file.c > >+++ b/plugins/file/file.c > >@@ -243,7 +243,7 @@ file_zero (void *handle, uint32_t count, uint64_t offset, int may_trim) > > if (may_trim) { > > r = fallocate (h->fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, > > offset, count); > >- if (r == -1 && errno != EOPNOTSUPP) { > >+ if (r == -1 && errno != EOPNOTSUPP && errno != ENODEV) { > &...

The upstream protocol recently promoted NBD_CMD_WRITE_ZEROES from experimental to a documented extension. Exposing support for this allows plugin writers to create sparse files when driven by a client that knows how to use the extension; meanwhile, even if a plugin does not support this extension, the server benefits from less network traffic from the client. Eric Blake (5): protocol: Support

Tested via: term1$ qemu-nbd -k $PWD/sock -t -f raw -x foo junk --trace=nbd_\* term2$ ./nbdkit -f -v -e bar nbd socket=$PWD/sock export=foo term3$ qemu-io -t none -f raw nbd://localhost:10809/bar --trace=nbd_\* and checking the traces to see that 'w 0 1' vs. 'w -f 0 1' was able to influence whether the FUA flag showed up at the server in term1. Still to go: figure out how to

...here >>>> is no timeout. >>>> >>>> (2) I suspect that because we don't have trim support that this is >>>> actually causing the disk to get fully allocated on the target. >>>> >>>> The NBD requests are sent with may_trim=1 so we could turn these >>>> into trim requests, but obviously cannot do that while there is no >>>> trim support. In fact, if a trim request guarantees that you can read back zeroes regardless of what was previously on the block device, then that is precisely what...

Fix some things I noticed while reviewing v1, and follow Rich's idea to add a new nbdkit_set_error() utility function with a binding for Python users to request a particular error (rather than being forced to live with whatever stale value is in errno after all the intermediate binding glue code). I could not easily find out how to register a C function callable from perl bindings, and have