search for: in_flight

...y tracking the count of commands we have queued up to send, as well as the count of commands where we are waiting on the server's response. Update tests/aio-parallel* and examples/batched-read-write to use nbd's own in-flight counter instead of reimplementing it ourselves. Note that h->in_flight is only ever updated while the lock is held; but we may want to consider also making it atomic and therefore readable as a lock-less function. --- examples/batched-read-write.c | 17 +++++--------- generator/generator | 42 ++++++++++++++++++++++++++--------- lib/aio.c...

If virtqueue is full, we put forget requests on a list and these forgets are dispatched later using a worker. As of now we don't count these forgets in fsvq->in_flight variable. This means when queue is being drained, we have to have special logic to first drain these pending requests and then wait for fsvq->in_flight to go to zero. By counting pending forgets in fsvq->in_flight, we can get rid of special logic and just wait for in_flight to go to zero. Wo...

...ed-reads-and-writes.c +++ b/examples/threaded-reads-and-writes.c @@ -41,20 +41,16 @@ static int64_t exportsize; /* Number of commands that can be "in flight" at the same time on each * connection. (Therefore the total number of requests in flight may - * be up to NR_MULTI_CONN * MAX_IN_FLIGHT). qemu's NBD client can - * have up to 16 requests in flight. - * - * Some servers do not support multiple requests in flight and may - * deadlock or even crash if this is larger than 1, but common NBD - * servers should be OK. + * be up to NR_MULTI_CONN * MAX_IN_FLIGHT). See libnbd(3) sectio...

Noticed while thinking about the recent threads wondering if we need a more efficient lookup from cookie back to command. Both of these fix bugs, but are tricky enough that I'm posting for review. Eric Blake (2): lib: Decrement in_flight at response, not retirement lib: Do O(1) rather than O(n) queue insertion generator/states-issue-command.c | 2 ++ generator/states-reply.c | 14 +++++++------- generator/states.c | 11 +++++++++-- lib/aio.c | 6 ++++-- lib/internal.h...

...+---------- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c index 43224db8d9ed..b5ba83ef1914 100644 --- a/fs/fuse/virtio_fs.c +++ b/fs/fuse/virtio_fs.c @@ -35,6 +35,7 @@ struct virtio_fs_vq { struct fuse_dev *fud; bool connected; long in_flight; + struct completion in_flight_zero; /* No inflight requests */ char name[24]; } ____cacheline_aligned_in_smp; @@ -85,6 +86,8 @@ static inline void dec_in_flight_req(struct virtio_fs_vq *fsvq) { WARN_ON(fsvq->in_flight <= 0); fsvq->in_flight--; + if (!fsvq->in_flight) + compl...

Hi Miklos, Here are few small cleanups for virtiofs for 5.5. I had received some comments from Michael Tsirkin on original virtiofs patches and these cleanups are result of these comments. Thanks Vivek Vivek Goyal (3): virtiofs: Use a common function to send forget virtiofs: Do not send forget request "struct list_head" element virtiofs: Use completions while waiting for queue

On Thu, Sep 05, 2019 at 03:48:49PM -0400, Vivek Goyal wrote: > +static void virtio_fs_drain_queue(struct virtio_fs_vq *fsvq) > +{ > + WARN_ON(fsvq->in_flight < 0); > + > + /* Wait for in flight requests to finish.*/ > + while (1) { > + spin_lock(&fsvq->lock); > + if (!fsvq->in_flight) { > + spin_unlock(&fsvq->lock); > + break; > + } > + spin_unlock(&fsvq->lock); > + usleep_range(1000, 200...

...5AM -0400, Vivek Goyal wrote: > On Fri, Sep 06, 2019 at 11:52:10AM +0100, Stefan Hajnoczi wrote: > > On Thu, Sep 05, 2019 at 03:48:49PM -0400, Vivek Goyal wrote: > > > +static void virtio_fs_drain_queue(struct virtio_fs_vq *fsvq) > > > +{ > > > + WARN_ON(fsvq->in_flight < 0); > > > + > > > + /* Wait for in flight requests to finish.*/ > > > + while (1) { > > > + spin_lock(&fsvq->lock); > > > + if (!fsvq->in_flight) { > > > + spin_unlock(&fsvq->lock); > > > + break; > > &...

On Thu, Sep 05, 2019 at 03:48:49PM -0400, Vivek Goyal wrote: > +static void virtio_fs_drain_queue(struct virtio_fs_vq *fsvq) > +{ > + WARN_ON(fsvq->in_flight < 0); > + > + /* Wait for in flight requests to finish.*/ > + while (1) { > + spin_lock(&fsvq->lock); > + if (!fsvq->in_flight) { > + spin_unlock(&fsvq->lock); > + break; > + } > + spin_unlock(&fsvq->lock); > + usleep_range(1000, 200...

...a list and retry submission later with the help of a worker. Thanks Vivek Vivek Goyal (5): virtiofs: Do not end request in submission context virtiofs: No need to check fpq->connected state virtiofs: Set FR_SENT flag only after request has been sent virtiofs: Count pending forgets as in_flight forgets virtiofs: Retry request submission from worker context fs/fuse/virtio_fs.c | 165 +++++++++++++++++++++++++++++--------------- 1 file changed, 111 insertions(+), 54 deletions(-) -- 2.20.1

...ad pool internally, and if you exceed those limits + * then something will break. + */ +#define NR_MULTI_CONN 8 + +/* Number of commands that can be "in flight" at the same time on each + * connection. (Therefore the total number of requests in flight may + * be up to NR_MULTI_CONN * MAX_IN_FLIGHT). See libnbd(3) section + * "Issuing multiple in-flight requests". + */ +#define MAX_IN_FLIGHT 64 + +/* The size of reads and writes. */ +#define BUFFER_SIZE (1024*1024) + +/* Number of commands we issue (per thread). */ +#define NR_CYCLES 1000000 + +/* Reader thread. */ +struct thread_s...

...* The single NBD handle. */ +static struct nbd_handle *nbd; + +/* Buffers used for the test. */ +static char *in, *out; +static int64_t packetsize; + +static int +try_deadlock (void *arg) +{ + struct pollfd fds[1]; + struct nbd_connection *conn; + size_t i; + int64_t handles[2], done; + size_t in_flight; /* counts number of requests in flight */ + int dir, r; + + /* The single thread "owns" the connection. */ + conn = nbd_get_connection (nbd, 0); + + /* Issue commands. */ + in_flight = 0; + handles[0] = nbd_aio_pread (conn, in, packetsize, 0); + if (handles[0] == -1) { +...

...ad pool internally, and if you exceed those limits + * then something will break. + */ +#define NR_MULTI_CONN 8 + +/* Number of commands that can be "in flight" at the same time on each + * connection. (Therefore the total number of requests in flight may + * be up to NR_MULTI_CONN * MAX_IN_FLIGHT). qemu's NBD client can + * have up to 16 requests in flight. + * + * Some servers do not support multiple requests in flight and may + * deadlock or even crash if this is larger than 1, but common NBD + * servers should be OK. + */ +#define MAX_IN_FLIGHT 16 + +/* Number of commands we issue (...

v1: https://www.redhat.com/archives/libguestfs/2019-June/msg00014.html I pushed a few bits which are uncontroversial. The main changes since v1 are: An extra patch removes the want_to_send / check for nbd_aio_is_ready in examples/threaded-reads-and-writes.c. This logic was wrong since commit 6af72b87 as was pointed out by Eric in his review. Comments and structure of

...@@ static void __do_IRQ_guest(int vector) { d = action->guest[i]; if ( (action->ack_type != ACKTYPE_NONE) && - !test_and_set_bit(irq, &d->pirq_mask) ) + !test_and_set_bit(irq, &d->pirq_mask[0]) ) action->in_flight++; send_guest_pirq(d, irq); } @@ -235,7 +235,7 @@ static void __set_eoi_ready(irq_desc_t * if ( !(desc->status & IRQ_GUEST) || (action->in_flight != 0) || - !test_and_clear_bit(cpu, &action->cpu_eoi_map) ) + !cpu_test_and_clear(cpu,...

...D handle. */ +static struct nbd_handle *nbd; + +/* Buffers used for the test. */ +static char *in, *out; +static int64_t packetsize; + +static int +try_deadlock (void *arg) +{ + struct pollfd fds[1]; + struct nbd_connection *conn; + char buf[512]; + size_t i, j; + int64_t handles[2]; + size_t in_flight; /* counts number of requests in flight */ + int dir, r, cmd; + bool want_to_send; + + /* The single thread "owns" the connection. */ + nbd_set_debug (nbd, true); + conn = nbd_get_connection (nbd, 0); + + /* Issue commands. */ + in_flight = 0; + fprintf (stderr, " * be...

...irtio_fs_request_dispatch_work(struct work_struct *work) } } +/* + * Returns 1 if queue is full and sender should wait a bit before sending + * next request, 0 otherwise. + */ +static int send_forget_request(struct virtio_fs_vq *fsvq, + struct virtio_fs_forget *forget, + bool in_flight) +{ + struct scatterlist sg; + struct virtqueue *vq; + int ret = 0; + bool notify; + + spin_lock(&fsvq->lock); + if (!fsvq->connected) { + if (in_flight) + dec_in_flight_req(fsvq); + kfree(forget); + goto out; + } + + sg_init_one(&sg, forget, sizeof(*forget)); + vq = fsvq->vq;...

...2f345e5..d5730a50b303 100644 --- a/fs/fuse/virtio_fs.c +++ b/fs/fuse/virtio_fs.c @@ -63,6 +63,55 @@ static inline struct fuse_pqueue *vq_to_fpq(struct virtqueue *vq) return &vq_to_fsvq(vq)->fud->pq; } +static void virtio_fs_drain_queue(struct virtio_fs_vq *fsvq) +{ + WARN_ON(fsvq->in_flight < 0); + + /* Wait for in flight requests to finish.*/ + while (1) { + spin_lock(&fsvq->lock); + if (!fsvq->in_flight) { + spin_unlock(&fsvq->lock); + break; + } + spin_unlock(&fsvq->lock); + usleep_range(1000, 2000); + } + + flush_work(&fsvq->done_work); +...

I still need to wire in the use of *_notify functions into nbdkit to prove whether it makes the code any faster or easier to maintain, but at least the added example shows one good use case for the new API. Eric Blake (6): api: Add nbd_aio_in_flight generator: Allow DEAD state actions to run generator: Allow Int64 in callbacks states: Prepare for aio notify callback api: Add new nbd_aio_FOO_notify functions examples: New example for strict read validations .gitignore | 1 + docs/libnbd.pod...

We have no control over the user piling up lots of commands faster than the server can accept (h->cmds_to_issue), or delaying retiring those batched commands (h->cmds_in_flight), hence our use of O(n) list insertion can be noticeable, since the growth of n can be unbounded from our viewpoint. It's easy enough to track a tail pointer to keep insertion O(1), to match that these two lists are used as queues; cmds_to_issue is always consumed with O(1) lookup/deletion, an...