similar to: [PATCH nbdkit] Experiment with parallel python plugin

On Thu, Aug 06, 2020 at 11:22:00PM +0300, Nir Soffer wrote: > This is a quick hack to experiment with parallel threading model in the > python plugin. > > Changes: > > - Use aligned buffers to make it possible to use O_DIRECT. Using > parallel I/O does not buy us much when using buffered I/O. pwrite() > copies data to the page cache, and pread() reads data from the

This is a quick hack to experiment with parallel threading model in the python plugin. Changes: - Use aligned buffers to make it possible to use O_DIRECT. Using parallel I/O does not buy us much when using buffered I/O. pwrite() copies data to the page cache, and pread() reads data from the page cache. - Disable extents in the file plugin. This way we can compare it with the python

If the plugin .pread method did not fill the buffer with data then the contents of the heap could be leaked back to the client. To avoid this create a thread-local data buffer which is initialized to zero and expanded (with zeroes) as required. This buffer is shared between pread and pwrite which makes the code a little bit simpler. Also this may improve locality by reusing the same memory for

On 4/23/19 10:09 AM, Richard W.M. Jones wrote: > If the plugin .pread method did not fill the buffer with data then the > contents of the heap could be leaked back to the client. To avoid > this create a thread-local data buffer which is initialized to zero > and expanded (with zeroes) as required. > > This buffer is shared between pread and pwrite which makes the code a >

When accepting a connection on a TCP or Unix domain socket we recorded the peer address in both the thread_data struct and thread-local storage. But for no reason because it was never used anywhere. Since we were only allocating a ‘struct sockaddr’ (rather than a ‘struct sockaddr_storage’) it's likely that some peer addresses would have been truncated. Remove all this code, it had no

On 2/11/20 11:15 AM, Richard W.M. Jones wrote: > Since we're going to be calling this function a lot, add a short alias > for it. > --- > server/internal.h | 1 + > server/public.c | 6 +++--- > 2 files changed, 4 insertions(+), 3 deletions(-) > > diff --git a/server/internal.h b/server/internal.h > index a1fa7309..1e7b4cf0 100644 > --- a/server/internal.h

There was a theoretic race in this example: If the server was very fast at handling commands then it's possible that in a call such as: cookie = nbd_aio_pread_callback (..., callback, ...); buffers[i].cookie = cookie; nbd_aio_pread_callback finished and calls the callback before returning. buffers[i].cookie would therefore not be set, but the callback() function was checking the list of

--- .gitignore | 1 + README | 2 + configure.ac | 9 + examples/Makefile.am | 22 ++ examples/glib-main-loop.c | 501 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 535 insertions(+) diff --git a/.gitignore b/.gitignore index edbf941..79b95b5 100644 --- a/.gitignore +++ b/.gitignore @@ -41,6 +41,7 @@ Makefile.in

On 7/15/19 1:38 PM, Richard W.M. Jones wrote: > --- > .gitignore | 1 + > README | 2 + > configure.ac | 9 + > examples/Makefile.am | 22 ++ > examples/glib-main-loop.c | 501 ++++++++++++++++++++++++++++++++++++++ > 5 files changed, 535 insertions(+) > Rough review (since I've never used a glib main loop

--- .gitignore | 1 + README | 2 + configure.ac | 9 + examples/Makefile.am | 22 ++ examples/glib-main-loop.c | 511 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 545 insertions(+) diff --git a/.gitignore b/.gitignore index ce02aef..9a8ba37 100644 --- a/.gitignore +++ b/.gitignore @@ -41,6 +41,7 @@ Makefile.in

Version 1 was here: https://www.redhat.com/archives/libguestfs/2019-April/msg00144.html Version 2 makes a couple of much larger changes: The OCaml patch changes the API of the pread method so it matches what other language bindings are already doing, ie. get the language plugin to return a newly allocated buffer, check it is long enough, copy out the data. The server patch implements a

We noticed while writing various libnbd tests that when the delay filter is in use, there are scenarios where we had to resort to SIGKILL to get rid of nbdkit, because it was non-responsive to SIGINT. I'm still trying to figure out the best way to add testsuite coverage of this, but already proved to myself that it works from the command line, under two scenarios that both used to cause long

On Tue, Aug 13, 2019 at 10:06:06PM -0500, Eric Blake wrote: > On 8/13/19 5:36 PM, Richard W.M. Jones wrote: > > The definition of functions that take a callback is changed so that > > the callback and user_data are combined into a single structure, eg: > > > > int64_t nbd_aio_pread (struct nbd_handle *h, > > void *buf, size_t count, uint64_t offset,

The original nbd_aio_* (non-callback) functions are removed and replaced with the renamed callback variants. This is a simple mechanical change to the API: (1) Any existing call to nbd_aio_*_callback can simply be renamed to nbd_aio_* (2) Any existing call to nbd_aio_* must have two extra NULL parameters added before the final flags parameter. In non-C languages, only change (1) is

On 8/13/19 5:36 PM, Richard W.M. Jones wrote: > The definition of functions that take a callback is changed so that > the callback and user_data are combined into a single structure, eg: > > int64_t nbd_aio_pread (struct nbd_handle *h, > void *buf, size_t count, uint64_t offset, > - int (*completion_callback) (/*..*/), void *user_data, > +

This drops all FUA and flush requests. --- filters/fua/nbdkit-fua-filter.pod | 27 +++++++++++++++++---- filters/fua/fua.c | 39 +++++++++++++++++++++++++++---- tests/test-fua.sh | 33 ++++++++++++++++---------- 3 files changed, 79 insertions(+), 20 deletions(-) diff --git a/filters/fua/nbdkit-fua-filter.pod b/filters/fua/nbdkit-fua-filter.pod index

--- .gitignore | 1 + examples/Makefile.am | 12 + examples/concurrent-writer.c | 450 +++++++++++++++++++++++++++++++++++ 3 files changed, 463 insertions(+) diff --git a/.gitignore b/.gitignore index 30438c1..e4dad91 100644 --- a/.gitignore +++ b/.gitignore @@ -41,6 +41,7 @@ Makefile.in /docs/libnbd-api.3 /docs/libnbd-api.pod /examples/batched-read-write

Since Eric's improvements to the command queue in commit 6af72b8 (and following) there's now a queue of commands waiting to be issued stored in the handle, and there's no need to issue commands only from the ready state. We can therefore remove the want_to_send logic, queue as many commands as possible, and don't need to test if the socket is ready for POLLOUT. This commit also

It is necessary to kick the state machine any time another thread blocks due to write() encountering a full buffer, to get the reader thread to learn that it must now poll on POLLOUT rather than just POLLIN. POSIX only provides two ways to do this: either poll() on a second fd (our pipe-to-self trick), or interrupt the polling with EINTR due to delivery of a signal. So, I played with the latter

My test server was in very eavy load, running kolab's components: postfix, cyrus-imap and openldap A second test machine was sending, and reading emails as fast as it can. The CPU was never idle ! I left the server at about 17H30 and at 22H I find it "uncommitted" (difficult to work when the partition in read-only !) However the mount command report it as RW ! The /boot is still RW