Richard W.M. Jones
2023-Feb-02 15:06 UTC
[Libguestfs] [PATCH nbdkit] curl: Enable multi-conn for read-only connections
Multi-conn is enabled only when we know the connection is read-only: $ ./nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: true $ ./nbdkit curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: false This improves performance. Comparing before and after this commit shows approximate doubling of performance: Benchmark 1: nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" Time (mean ? ?): 943.8 ms ? 18.8 ms [User: 316.2 ms, System: 1029.7 ms] Range (min ? max): 923.7 ms ? 989.2 ms 10 runs Benchmark 2: ~/d/nbdkit/nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" Time (mean ? ?): 455.0 ms ? 6.2 ms [User: 542.2 ms, System: 1824.7 ms] Range (min ? max): 449.1 ms ? 471.6 ms 10 runs Summary ' ~/d/nbdkit/nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" ' ran 2.07 ? 0.05 times faster than ' nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" ' See also: https://listman.redhat.com/archives/libguestfs/2023-February/030581.html --- plugins/curl/curldefs.h | 1 + plugins/curl/curl.c | 14 ++++++++++++++ 2 files changed, 15 insertions(+) diff --git a/plugins/curl/curldefs.h b/plugins/curl/curldefs.h index a2c521c50..16b6071cc 100644 --- a/plugins/curl/curldefs.h +++ b/plugins/curl/curldefs.h @@ -68,6 +68,7 @@ extern const char *user_agent; /* The per-connection handle. */ struct curl_handle { CURL *c; + int readonly; bool accept_range; int64_t exportsize; char errbuf[CURL_ERROR_SIZE]; diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c index 42aba4216..2f34d1dc1 100644 --- a/plugins/curl/curl.c +++ b/plugins/curl/curl.c @@ -465,6 +465,7 @@ curl_open (int readonly) nbdkit_error ("calloc: %m"); return NULL; } + h->readonly = readonly; h->c = curl_easy_init (); if (h->c == NULL) { @@ -781,6 +782,18 @@ curl_get_size (void *handle) return h->exportsize; } +/* Multi-conn is safe for read-only connections, but HTTP does not + * have any concept of flushing so we cannot use it for read-write + * connections. + */ +static int +curl_can_multi_conn (void *handle) +{ + struct curl_handle *h = handle; + + return !! h->readonly; +} + /* NB: The terminology used by libcurl is confusing! * * WRITEFUNCTION / write_cb is used when reading from the remote server @@ -924,6 +937,7 @@ static struct nbdkit_plugin plugin = { .open = curl_open, .close = curl_close, .get_size = curl_get_size, + .can_multi_conn = curl_can_multi_conn, .pread = curl_pread, .pwrite = curl_pwrite, }; -- 2.39.0
Eric Blake
2023-Feb-02 15:18 UTC
[Libguestfs] [PATCH nbdkit] curl: Enable multi-conn for read-only connections
On Thu, Feb 02, 2023 at 03:06:30PM +0000, Richard W.M. Jones wrote:> Multi-conn is enabled only when we know the connection is read-only: > > $ ./nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn > can_multi_conn: true > $ ./nbdkit curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn > can_multi_conn: false > > This improves performance.Reviewed-by: Eric Blake <eblake at redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org