similar to: [PATCH 1/1] nbd/server: push pending frames after sending reply

In the recent commit 3842a080 to add SEND_MORE support, I blindly implemented the tls code as: if (SEND_MORE) { cork send } else { send uncork } because it showed improvements for my test case of aio-parallel-load from libnbd. But that test sticks to 64k I/O requests. But with further investigation, I've learned that even though gnutls corking works great for smaller

We disabled Nagle's algorithm to allow less latency in our responses reaching the client; but as a side effect, it leads to more network overhead when we send a reply split across more than one write(). Take advantage of various means for grouping related writes (Linux' MSG_MORE for sockets, gnutls' corking for TLS) to send a larger packet, and adjust callers to pass in our internal

On Fri, Mar 24, 2023 at 02:41:20PM -0500, Eric Blake wrote: > On Fri, Mar 24, 2023 at 11:47:20AM +0100, Florian Westphal wrote: > > qemu-nbd doesn't set TCP_NODELAY on the tcp socket. Replying to myself, WHY aren't we setting TCP_NODELAY on the socket? > > And surprisingly, qemu IS using corking on the client side: >

As mentioned in the previous patch, sending small packets as soon as possible leads to network packet overhead. Grouping related writes under corking appears to help everything but unencrypted Unix plain sockets. I tested with appropriate combinations from: $ nbdkit {-p 10810,-U -} \ {--tls=require --tls-verify-peer --tls-psk=./keys.psk,} memory size=64m \ --run

Slightly RFC, as I need more time to investigate why Unix sockets appeared to degrade with this patch. But as TCP sockets (over loopback to localhost) and TLS sessions (regardless of underlying Unix or TCP) both showed improvements, this looks like a worthwhile series. Eric Blake (2): server: Add support for corking server: Cork around grouped transmission send()s server/internal.h | 3

This time around, the numbers are indeed looking better than in v1; and I like the interface better. Eric Blake (2): server: Prefer send() over write() server: Group related transmission send()s server/internal.h | 7 +++- server/connections.c | 51 +++++++++++++++++++++++++--- server/crypto.c | 11 ++++--

Any time we reply to NBD_CMD_READ or NBD_CMD_BLOCK_STATUS, we end up calling conn->send() more than once. Now that we've disabled Nagle's algorithm, this implies that we try harder to send the small header immediately, rather than batching it with the rest of the payload, which causes more overhead in the amount of actual network traffic. For interfaces that support corking (gnutls, or

libnbd's copy/copy-nbd-error.sh was triggering an assertion failure in nbdkit: $ nbdcopy -- [ nbdkit --exit-with-parent -v --filter=error pattern 5M error-pread-rate=0.5 ] null: ... nbdkit: pattern.2: debug: error-inject: pread count=262144 offset=4718592 nbdkit: pattern.2: debug: pattern: pread count=262144 offset=4718592 nbdkit: pattern.1: debug: error-inject: pread count=262144

I'm a bit too busy nowadays because of school so I haven't had time to look at everyone's bug reports and some other mails. They're still marked as unread in my mailbox, so hopefully I'll get to them soon. And I don't mind you resend them if I don't answer in a few weeks :) Changes: - Fixed maildir syncing errors about wrong UIDs - Maildir was re-read all the time,

Since commit 86fdb48c6a5362d66865493d9d2172166f99722e we have stored the connection object in thread-local storage. In this very large, but mostly mechanical change we stop passing the connection pointer around everywhere, and instead use the value stored in thread-local storage. This assumes a 1-1 mapping between the connection and the current thread which is true in *most* places.

> As a streaming server, it's fairly crucial for icecast to > send out data with as low a delay as possible (many clients > don't care, but some do). That's why we use TCP_NODELAY - we > actually WANT to send out data as soon as we can. Nagle is inherently unsuited for streams. NODELAY was (imho) ment for connections for which Nagle isn't sufficient and CORK is not

Checking for poll(POLLIN) only wakes us up when the server sends more bytes over the wire. But the way gnutls is implemented, it reads as many non-blocking bytes as possible from the wire before then parsing where the encoded message boundaries lie within those bytes. As a result, we may already have the server's next reply in memory without needing to block on the server sending yet more

I'm using qemu-nbd to setup the device with an offset into the image. Using libguestfs version 1.21.22 Code for adding the drive: char *server[] = { "localhost:3000", NULL }; if (guestfs_add_drive_opts (g, "" /* export name - see below */, GUESTFS_ADD_DRIVE_OPTS_FORMAT, "raw", GUESTFS_ADD_DRIVE_OPTS_PROTOCOL, "nbd",

Since we disabled Nagle's algorithm we may send very small packets over the wire in some situations where we are calling send(2) from states that are responsible for small parts of the protocol. By setting the MSG_MORE flag we can indicate to the kernel that more data will follow (usually) immediately and so it can append the data to the same outgoing packet. Although there is some

Now that ->sendpage() has been removed, MSG_SENDPAGE_NOTLAST can be cleaned up. Things were converted to use MSG_MORE instead, but the protocol sendpage stubs still convert MSG_SENDPAGE_NOTLAST to MSG_MORE, which is now unnecessary. Signed-off-by: David Howells <dhowells at redhat.com> cc: "David S. Miller" <davem at davemloft.net> cc: Eric Dumazet <edumazet at

We free transmitted packets in ndo_start_xmit() in the past to get better performance in the past. One side effect is that skb_orphan() needs to be called in ndo_start_xmit() which makes sk_wmem_alloc not accurate in fact. For TCP protocol, this means several optimization could not work well such as TCP small queue and auto corking. This can lead extra low throughput of small packets stream.

We free transmitted packets in ndo_start_xmit() in the past to get better performance in the past. One side effect is that skb_orphan() needs to be called in ndo_start_xmit() which makes sk_wmem_alloc not accurate in fact. For TCP protocol, this means several optimization could not work well such as TCP small queue and auto corking. This can lead extra low throughput of small packets stream.

We can only process 1 packet at one time during sendmsg(). This often lead bad cache utilization under heavy load. So this patch tries to do some batching during rx before submitting them to host network stack. This is done through accepting MSG_MORE as a hint from sendmsg() caller, if it was set, batch the packet temporarily in a linked list and submit them all once MSG_MORE were cleared. Tests

We can only process 1 packet at one time during sendmsg(). This often lead bad cache utilization under heavy load. So this patch tries to do some batching during rx before submitting them to host network stack. This is done through accepting MSG_MORE as a hint from sendmsg() caller, if it was set, batch the packet temporarily in a linked list and submit them all once MSG_MORE were cleared. Tests

Following up to cf1a3045, if we know that we have more requests to transmit (because the user was queueing up requests while we were busy elsewhere), and our current request is short (a non-write, or a write with a small payload), then our current command can be batched with the next command. The numbers here were not as dramatic and may be merely in the noise; over three runs of: $ time