Libguestfs - Jul 2020 - [PATCH nbdkit v2] curl: Implement header and cookie scripts.

Evolution of this patch series:
https://www.redhat.com/archives/libguestfs/2020-July/thread.html#00073

Instead of auth-script, this implements header-script and
cookie-script.  It can be used for similar purposes but the
implementation is somewhat saner.

Rich.

This rather complex feature solves a problem for certain web services
that require a cookie or token for access, especially one which must
be periodically renewed.

For motivation for this feature see the included documentation, and
item (1)(b) here:

https://www.redhat.com/archives/libguestfs/2020-July/msg00069.html
---
 plugins/curl/nbdkit-curl-plugin.pod | 142 ++++++++++++
 plugins/curl/Makefile.am            |   2 +
 tests/Makefile.am                   |  47 ++++
 plugins/curl/curldefs.h             |  76 +++++++
 plugins/curl/curl.c                 | 116 +++++++---
 plugins/curl/scripts.c              | 330 ++++++++++++++++++++++++++++
 tests/test-curl-cookie-script.c     | 143 ++++++++++++
 tests/test-curl-header-script.c     | 165 ++++++++++++++
 .gitignore                          |   2 +
 9 files changed, 991 insertions(+), 32 deletions(-)

diff --git a/plugins/curl/nbdkit-curl-plugin.pod
b/plugins/curl/nbdkit-curl-plugin.pod
index 22c07f39..52875988 100644
--- a/plugins/curl/nbdkit-curl-plugin.pod
+++ b/plugins/curl/nbdkit-curl-plugin.pod
@@ -81,6 +81,14 @@ command line is not secure on shared machines.  Use the
alternate
 C<+FILENAME> syntax to pass it in a file, C<-> to read the cookie
 interactively, or C<-FD> to read it from a file descriptor.
 
+=item B<cookie-script=>SCRIPT
+
+=item B<cookie-script-renew=>SECS
+
+Run C<SCRIPT> (a command or shell script fragment) to generate the
+HTTP/HTTPS cookies.  C<cookie-script> cannot be used with
C<cookie>.
+See L</HEADER AND COOKIE SCRIPTS> below.
+
 =item B<header=>HEADER
 
 (nbdkit E<ge> 1.22)
@@ -106,6 +114,14 @@ requests, even when following a redirect, which can cause
headers
 (eg. containing sensitive authorization information) to be sent to
 hosts other than the one originally requested.
 
+=item B<header-script=>SCRIPT
+
+=item B<header-script-renew=>SECS
+
+Run C<SCRIPT> (a command or shell script fragment) to generate the
+HTTP/HTTPS headers.  C<header-script> cannot be used with
C<header>.
+See L</HEADER AND COOKIE SCRIPTS> below.
+
 =item B<password=>PASSWORD
 
 Set the password to use when connecting to the remote server.
@@ -226,6 +242,132 @@ user-agent header.
 
 =back
 
+=head1 HEADER AND COOKIE SCRIPTS
+
+While the C<header> and C<cookie> parameters can be used to specify
+static headers and cookies which are used in every HTTP/HTTPS request,
+the alternate C<header-script> and C<cookie-script> parameters can
be
+used to run an external script or program to generate headers and/or
+cookies.  This is particularly useful to access services which require
+an authorization token.  In addition the C<header-script-renew> and
+C<cookie-script-renew> parameters allow you to renew the authorization
+token by rerunning the script periodically.
+
+C<header-script> is incompatible with C<header>, and
C<cookie-script>
+is incompatible with C<cookie>.
+
+=head2 Header script
+
+The header script should print zero or more HTTP headers, each line of
+output in the same format as the C<header> parameter.  The headers
+printed by the script are passed to L<CURLOPT_HTTPHEADER(3)>.
+
+In the following example, an imaginary web service requires
+authentication using a token fetched from a separate login server.
+The token expires after 60 seconds, so we also tell the plugin that it
+must renew the token (by re-running the script) if more than 45
+seconds have elapsed since the last request:
+
+ nbdkit curl https://service.example.com/disk.img \
+        header-script='
+          echo -n "Authorization: Bearer "
+          curl -s -X POST https://auth.example.com/login |
+               jq -r .token
+        ' \
+        header-script-renew=50
+
+=head2 Cookie script
+
+The cookie script should print a single line in the same format as the
+C<cookie> parameter.  This is passed to L<CURLOPT_COOKIE(3)>.
+
+=head2 Header and cookie script shell variables
+
+Within the C<header-script> and C<cookie-script> the following
shell
+variables are available:
+
+=over 4
+
+=item C<$iteration>
+
+The number of times that the script has been called.  The first time
+the script is called this contains C<0>.
+
+=item C<$url>
+
+The URL as passed to the plugin.
+
+=back
+
+=head2 Example: VMware ESXi cookies
+
+VMware ESXi’s web server can expose both VMDK and raw format disk
+images, but requires you to log in using HTTP Basic Authentication.
+While you can use the C<user> and C<password> parameters to send
HTTP
+Basic Authentication headers in every request, tests have shown that
+it is faster to accept the cookie which the server returns and send
+that instead.  (It is not clear why it is faster, but one theory is
+that VMware has to do a more expensive username and password check
+each time.)
+
+The web server can be accessed as below.  Since the cookie expires
+after a certain period of time, we use C<cookie-script-renew>, and
+because the server uses a self-signed certificate we must use
+I<--insecure> and C<sslverify=false>.
+
+ SERVER=esx.example.com
+ DCPATH=data
+ DS=datastore1
+ GUEST=guest-name
+
URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
+ 
+ nbdkit curl "$URL" \
+        cookie-script='
+            curl --head -s --insecure -u root:password "$url" |
+                 sed -ne '{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }'
+        ' \
+        cookie-script-renew=500 \
+        sslverify=false
+
+=head2 Example: Docker Hub authorization tokens
+
+Accessing objects like container layers from Docker Hub requires that
+you first fetch an authorization token, even for anonymous access.
+These tokens expire after about 5 minutes (300 seconds) so must be
+periodically renewed.
+
+You will need this authorization script (F</tmp/auth.sh>):
+
+ #!/bin/sh -
+ IMAGE=library/fedora
+ curl -s
"https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull"
|
+      jq -r .token
+
+You will also need this script to get the blobSum of the layer
+(F</tmp/blobsum.sh>):
+
+ #!/bin/sh -
+ TOKEN=`/tmp/auth.sh`
+ IMAGE=library/fedora
+ curl -s -X GET -H "Authorization: Bearer $TOKEN" \
+      "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
+      jq -r '.fsLayers[0].blobSum'
+
+Both scripts must be executable, and both can be run on their own to
+check they are working.  To run nbdkit:
+
+ IMAGE=library/fedora
+ BLOBSUM=`/tmp/blobsum.sh`
+ URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
+ 
+ nbdkit curl "$URL" \
+        header-script=' echo -n "Authorization: Bearer ";
/tmp/auth.sh ' \
+        header-script-renew=200 \
+        --filter=gzip
+
+Note that this exposes a tar file over NBD.  See also
+L<nbdkit-tar-filter(1)>.
+
 =head1 DEBUG FLAG
 
 =over 4
diff --git a/plugins/curl/Makefile.am b/plugins/curl/Makefile.am
index ddf1a215..0dd78199 100644
--- a/plugins/curl/Makefile.am
+++ b/plugins/curl/Makefile.am
@@ -38,6 +38,8 @@ if HAVE_CURL
 plugin_LTLIBRARIES = nbdkit-curl-plugin.la
 
 nbdkit_curl_plugin_la_SOURCES = \
+	curldefs.h \
+	scripts.c \
 	curl.c \
 	$(top_srcdir)/include/nbdkit-plugin.h \
 	$(NULL)
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b830d80e..2641910b 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -502,6 +502,10 @@ if HAVE_CURL
 TESTS += test-curl-file.sh
 EXTRA_DIST += test-curl-file.sh
 LIBGUESTFS_TESTS += test-curl
+LIBNBD_TESTS += \
+	test-curl-header-script \
+	test-curl-cookie-script \
+	$(NULL)
 
 test_curl_SOURCES = \
 	test-curl.c \
@@ -524,6 +528,49 @@ test_curl_LDADD = \
 	libtest.la \
 	$(LIBGUESTFS_LIBS) \
 	$(NULL)
+
+test_curl_header_script_SOURCES = \
+	test-curl-header-script.c \
+	web-server.c \
+	web-server.h \
+	$(NULL)
+test_curl_header_script_CPPFLAGS = \
+	-I$(top_srcdir)/common/utils \
+	$(NULL)
+test_curl_header_script_CFLAGS = \
+	$(WARNINGS_CFLAGS) \
+	$(LIBNBD_CFLAGS) \
+	$(PTHREAD_CFLAGS) \
+	$(NULL)
+test_curl_header_script_LDFLAGS = \
+	$(top_builddir)/common/utils/libutils.la \
+	$(PTHREAD_LIBS) \
+	$(NULL)
+test_curl_header_script_LDADD = \
+	$(LIBNBD_LIBS) \
+	$(NULL)
+
+test_curl_cookie_script_SOURCES = \
+	test-curl-header-script.c \
+	web-server.c \
+	web-server.h \
+	$(NULL)
+test_curl_cookie_script_CPPFLAGS = \
+	-I$(top_srcdir)/common/utils \
+	$(NULL)
+test_curl_cookie_script_CFLAGS = \
+	$(WARNINGS_CFLAGS) \
+	$(LIBNBD_CFLAGS) \
+	$(PTHREAD_CFLAGS) \
+	$(NULL)
+test_curl_cookie_script_LDFLAGS = \
+	$(top_builddir)/common/utils/libutils.la \
+	$(PTHREAD_LIBS) \
+	$(NULL)
+test_curl_cookie_script_LDADD = \
+	$(LIBNBD_LIBS) \
+	$(NULL)
+
 endif HAVE_CURL
 endif HAVE_MKE2FS_WITH_D
 
diff --git a/plugins/curl/curldefs.h b/plugins/curl/curldefs.h
new file mode 100644
index 00000000..5ec03231
--- /dev/null
+++ b/plugins/curl/curldefs.h
@@ -0,0 +1,76 @@
+/* nbdkit
+ * Copyright (C) 2014-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#ifndef NBDKIT_CURLDEFS_H
+#define NBDKIT_CURLDEFS_H
+
+extern const char *url;
+
+extern const char *cainfo;
+extern const char *capath;
+extern char *cookie;
+extern const char *cookie_script;
+extern unsigned cookie_script_renew;
+extern struct curl_slist *headers;
+extern const char *header_script;
+extern unsigned header_script_renew;
+extern char *password;
+extern long protocols;
+extern const char *proxy;
+extern char *proxy_password;
+extern const char *proxy_user;
+extern bool sslverify;
+extern bool tcp_keepalive;
+extern bool tcp_nodelay;
+extern uint32_t timeout;
+extern const char *unix_socket_path;
+extern const char *user;
+extern const char *user_agent;
+
+/* The per-connection handle. */
+struct curl_handle {
+  CURL *c;
+  bool accept_range;
+  int64_t exportsize;
+  char errbuf[CURL_ERROR_SIZE];
+  char *write_buf;
+  uint32_t write_count;
+  const char *read_buf;
+  uint32_t read_count;
+  struct curl_slist *headers_copy;
+};
+
+/* scripts.c */
+extern int do_scripts (struct curl_handle *h);
+extern void scripts_unload (void);
+
+#endif /* NBDKIT_CURLDEFS_H */
diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c
index 50eef1a8..8731a506 100644
--- a/plugins/curl/curl.c
+++ b/plugins/curl/curl.c
@@ -48,9 +48,11 @@
 
 #include <nbdkit-plugin.h>
 
-#include "cleanup.h"
 #include "ascii-ctype.h"
 #include "ascii-string.h"
+#include "cleanup.h"
+
+#include "curldefs.h"
 
 /* Macro CURL_AT_LEAST_VERSION was added in 2015 (Curl 7.43) so if the
  * macro isn't present then Curl is very old.
@@ -61,24 +63,29 @@
 #endif
 #endif
 
-static const char *url = NULL;  /* required */
+/* Plugin configuration. */
+const char *url = NULL;         /* required */
 
-static const char *cainfo = NULL;
-static const char *capath = NULL;
-static char *cookie = NULL;
-static struct curl_slist *headers = NULL;
-static char *password = NULL;
-static long protocols = CURLPROTO_ALL;
-static const char *proxy = NULL;
-static char *proxy_password = NULL;
-static const char *proxy_user = NULL;
-static bool sslverify = true;
-static bool tcp_keepalive = false;
-static bool tcp_nodelay = true;
-static uint32_t timeout = 0;
-static const char *unix_socket_path = NULL;
-static const char *user = NULL;
-static const char *user_agent = NULL;
+const char *cainfo = NULL;
+const char *capath = NULL;
+char *cookie = NULL;
+const char *cookie_script = NULL;
+unsigned cookie_script_renew = 0;
+struct curl_slist *headers = NULL;
+const char *header_script = NULL;
+unsigned header_script_renew = 0;
+char *password = NULL;
+long protocols = CURLPROTO_ALL;
+const char *proxy = NULL;
+char *proxy_password = NULL;
+const char *proxy_user = NULL;
+bool sslverify = true;
+bool tcp_keepalive = false;
+bool tcp_nodelay = true;
+uint32_t timeout = 0;
+const char *unix_socket_path = NULL;
+const char *user = NULL;
+const char *user_agent = NULL;
 
 /* Use '-D curl.verbose=1' to set. */
 int curl_debug_verbose = 0;
@@ -98,11 +105,12 @@ curl_load (void)
 static void
 curl_unload (void)
 {
-  free (password);
-  free (proxy_password);
   free (cookie);
   if (headers)
     curl_slist_free_all (headers);
+  free (password);
+  free (proxy_password);
+  scripts_unload ();
   curl_global_cleanup ();
 }
 
@@ -202,6 +210,16 @@ curl_config (const char *key, const char *value)
       return -1;
   }
 
+  else if (strcmp (key, "cookie-script") == 0) {
+    cookie_script = value;
+  }
+
+  else if (strcmp (key, "cookie-script-renew") == 0) {
+    if (nbdkit_parse_unsigned ("cookie-script-renew", value,
+                               &cookie_script_renew) == -1)
+      return -1;
+  }
+
   else if (strcmp (key, "header") == 0) {
     headers = curl_slist_append (headers, value);
     if (headers == NULL) {
@@ -210,6 +228,16 @@ curl_config (const char *key, const char *value)
     }
   }
 
+  else if (strcmp (key, "header-script") == 0) {
+    header_script = value;
+  }
+
+  else if (strcmp (key, "header-script-renew") == 0) {
+    if (nbdkit_parse_unsigned ("header-script-renew", value,
+                               &header_script_renew) == -1)
+      return -1;
+  }
+
   else if (strcmp (key, "password") == 0) {
     free (password);
     if (nbdkit_read_password (value, &password) == -1)
@@ -300,6 +328,26 @@ curl_config_complete (void)
     return -1;
   }
 
+  if (headers && header_script) {
+    nbdkit_error ("header and header-script cannot be used at the same
time");
+    return -1;
+  }
+
+  if (!header_script && header_script_renew) {
+    nbdkit_error ("header-script-renew cannot be used without
header-script");
+    return -1;
+  }
+
+  if (cookie && cookie_script) {
+    nbdkit_error ("cookie and cookie-script cannot be used at the same
time");
+    return -1;
+  }
+
+  if (!cookie_script && cookie_script_renew) {
+    nbdkit_error ("cookie-script-renew cannot be used without
cookie-script");
+    return -1;
+  }
+
   return 0;
 }
 
@@ -307,7 +355,11 @@ curl_config_complete (void)
   "cainfo=<CAINFO>            Path to Certificate Authority
file.\n" \
   "capath=<CAPATH>            Path to directory with CA
certificates.\n" \
   "cookie=<COOKIE>            Set HTTP/HTTPS cookies.\n" \
+  "cookie-script=<SCRIPT>     Script to set HTTP/HTTPS
cookies.\n" \
+  "cookie-script-renew=<SECS> Time to renew HTTP/HTTPS
cookies.\n" \
   "header=<HEADER>            Set HTTP/HTTPS header.\n" \
+  "header-script=<SCRIPT>     Script to set HTTP/HTTPS
headers.\n" \
+  "header-script-renew=<SECS> Time to renew HTTP/HTTPS
headers.\n" \
   "password=<PASSWORD>        The password for the user
account.\n" \
   "protocols=PROTO,PROTO,..   Limit protocols allowed.\n" \
   "proxy=<PROXY>              Set proxy URL.\n" \
@@ -322,18 +374,6 @@ curl_config_complete (void)
   "user=<USER>                The user to log in as.\n" \
   "user-agent=<USER-AGENT>    Send user-agent header for
HTTP/HTTPS."
 
-/* The per-connection handle. */
-struct curl_handle {
-  CURL *c;
-  bool accept_range;
-  int64_t exportsize;
-  char errbuf[CURL_ERROR_SIZE];
-  char *write_buf;
-  uint32_t write_count;
-  const char *read_buf;
-  uint32_t read_count;
-};
-
 /* Translate CURLcode to nbdkit_error. */
 #define display_curl_error(h, r, fs, ...)                       \
   do {                                                          \
@@ -450,7 +490,11 @@ curl_open (int readonly)
 
   /* Get the file size and also whether the remote HTTP server
    * supports byte ranges.
+   *
+   * We must run the scripts if necessary and set headers in the
+   * handle.
    */
+  if (do_scripts (h) == -1) goto err;
   h->accept_range = false;
   curl_easy_setopt (h->c, CURLOPT_NOBODY, 1); /* No Body, not nobody! */
   curl_easy_setopt (h->c, CURLOPT_HEADERFUNCTION, header_cb);
@@ -608,6 +652,8 @@ curl_close (void *handle)
   struct curl_handle *h = handle;
 
   curl_easy_cleanup (h->c);
+  if (h->headers_copy)
+    curl_slist_free_all (h->headers_copy);
   free (h);
 }
 
@@ -638,6 +684,9 @@ curl_pread (void *handle, void *buf, uint32_t count,
uint64_t offset)
   CURLcode r;
   char range[128];
 
+  /* Run the scripts if necessary and set headers in the handle. */
+  if (do_scripts (h) == -1) return -1;
+
   /* Tell the write_cb where we want the data to be written.  write_cb
    * will update this if the data comes in multiple sections.
    */
@@ -699,6 +748,9 @@ curl_pwrite (void *handle, const void *buf, uint32_t count,
uint64_t offset)
   CURLcode r;
   char range[128];
 
+  /* Run the scripts if necessary and set headers in the handle. */
+  if (do_scripts (h) == -1) return -1;
+
   /* Tell the read_cb where we want the data to be read from.  read_cb
    * will update this if the data comes in multiple sections.
    */
diff --git a/plugins/curl/scripts.c b/plugins/curl/scripts.c
new file mode 100644
index 00000000..5d961391
--- /dev/null
+++ b/plugins/curl/scripts.c
@@ -0,0 +1,330 @@
+/* nbdkit
+ * Copyright (C) 2014-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/* Header and cookie scripts. */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <time.h>
+#include <assert.h>
+#include <pthread.h>
+
+#include <curl/curl.h>
+
+#include <nbdkit-plugin.h>
+
+#include "ascii-ctype.h"
+#include "cleanup.h"
+#include "utils.h"
+
+#include "curldefs.h"
+
+/* This lock protects internal state in this file. */
+static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
+
+/* Last time header-script or cookie-script was run. */
+static time_t header_last = 0;
+static time_t cookie_last = 0;
+static bool header_script_has_run = false;
+static bool cookie_script_has_run = false;
+static unsigned header_iteration = 0;
+static unsigned cookie_iteration = 0;
+
+/* Last set of headers and cookies generated by the scripts. */
+static struct curl_slist *headers_from_script = NULL;
+static char *cookies_from_script = NULL;
+
+void
+scripts_unload (void)
+{
+  curl_slist_free_all (headers_from_script);
+  free (cookies_from_script);
+}
+
+static int run_header_script (struct curl_handle *);
+static int run_cookie_script (struct curl_handle *);
+
+/* This is called from any thread just before we make a curl request.
+ *
+ * Because the thread model is NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS
+ * we can be assured of exclusive access to curl_handle here.
+ */
+int
+do_scripts (struct curl_handle *h)
+{
+  time_t now;
+  struct curl_slist *p;
+
+  /* Return quickly without acquiring the lock if this feature is not
+   * being used.
+   */
+  if (!header_script && !cookie_script)
+    return 0;
+
+  ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
+
+  /* Run or re-run header-script if we need to. */
+  if (header_script) {
+    time (&now);
+    if (!header_script_has_run ||
+        (header_script_renew > 0 && now - header_last >=
header_script_renew)) {
+      if (run_header_script (h) == -1)
+        return -1;
+      header_last = now;
+      header_script_has_run = true;
+    }
+  }
+
+  /* Run or re-run cookie-script if we need to. */
+  if (cookie_script) {
+    time (&now);
+    if (!cookie_script_has_run ||
+        (cookie_script_renew > 0 && now - cookie_last >=
cookie_script_renew)) {
+      if (run_cookie_script (h) == -1)
+        return -1;
+      cookie_last = now;
+      cookie_script_has_run = true;
+    }
+  }
+
+  /* Set headers and cookies in the handle.
+   *
+   * When calling CURLOPT_HTTPHEADER we have to keep the list around
+   * because unfortunately curl doesn't take a copy.  Since we don't
+   * know which other threads might be using it, we must make a copy
+   * of the global list (headers_from_script) per handle
+   * (h->headers_copy).  For CURLOPT_COOKIE, curl internally takes a
+   * copy so we don't need to do this.
+   */
+  if (h->headers_copy) {
+    curl_easy_setopt (h->c, CURLOPT_HTTPHEADER, NULL);
+    curl_slist_free_all (h->headers_copy);
+    h->headers_copy = NULL;
+  }
+  for (p = headers_from_script; p != NULL; p = p->next) {
+    h->headers_copy = curl_slist_append (h->headers_copy, p->data);
+    if (h->headers_copy == NULL) {
+      nbdkit_error ("curl_slist_append: %m");
+      return -1;
+    }
+  }
+  curl_easy_setopt (h->c, CURLOPT_HTTPHEADER, h->headers_copy);
+
+  curl_easy_setopt (h->c, CURLOPT_COOKIE, cookies_from_script);
+
+  return 0;
+}
+
+/* This is called with the lock held when we must run or re-run the
+ * header-script.
+ */
+static int
+run_header_script (struct curl_handle *h)
+{
+  int fd;
+  char tmpfile[] = "/tmp/errorsXXXXXX";
+  FILE *fp;
+  CLEANUP_FREE char *cmd = NULL, *line = NULL;
+  size_t len = 0, linelen = 0, nr_headers = 0;
+
+  assert (header_script != NULL); /* checked by caller */
+
+  /* Reset the list of headers. */
+  curl_slist_free_all (headers_from_script);
+  headers_from_script = NULL;
+
+  /* Create a temporary file for the errors so we can redirect them
+   * into nbdkit_error.
+   */
+  fd = mkstemp (tmpfile);
+  if (fd == -1) {
+    nbdkit_error ("mkstemp");
+    return -1;
+  }
+  close (fd);
+
+  /* Generate the full script with the local $url variable. */
+  fp = open_memstream (&cmd, &len);
+  if (fp == NULL) {
+    nbdkit_error ("open_memstream: %m");
+    return -1;
+  }
+  fprintf (fp, "exec </dev/null\n");    /* Avoid stdin leaking
(nbdkit -s). */
+  fprintf (fp, "exec 2>%s\n", tmpfile); /* Catch errors to a
temporary file. */
+  fprintf (fp, "url=");                 /* Set the shell variables.
*/
+  shell_quote (url, fp);
+  putc ('\n', fp);
+  fprintf (fp, "iteration=%u\n", header_iteration++);
+  putc ('\n', fp);
+  fprintf (fp, "%s", header_script);    /* The script or command. */
+  if (fclose (fp) == EOF) {
+    nbdkit_error ("memstream failed");
+    return -1;
+  }
+
+  /* Run the script and read the headers. */
+  nbdkit_debug ("curl: running header-script");
+  fp = popen (cmd, "r");
+  if (fp == NULL) {
+    nbdkit_error ("popen: %m");
+    return -1;
+  }
+  while ((len = getline (&line, &linelen, fp)) != -1) {
+    /* Remove trailing \n and whitespace. */
+    while (len > 0 && ascii_isspace (line[len-1]))
+      line[--len] = '\0';
+    if (len == 0)
+      continue;
+
+    headers_from_script = curl_slist_append (headers_from_script, line);
+    if (headers_from_script == NULL) {
+      nbdkit_error ("curl_slist_append: %m");
+      pclose (fp);
+      return -1;
+    }
+    nr_headers++;
+  }
+
+  /* If the command failed, this should return EOF and the error
+   * message should be in the temporary file (but we only read the
+   * first line).
+   */
+  if (pclose (fp) == EOF) {
+    fp = fopen (tmpfile, "r");
+    if ((len = getline (&line, &linelen, fp)) >= 0) {
+      if (len > 0 && line[len-1] == '\n')
+        line[len-1] = '\0';
+      nbdkit_error ("header-script failed: %s", line);
+    }
+    else
+      nbdkit_error ("header-script failed");
+    return -1;
+  }
+
+  nbdkit_debug ("header-script returned %zu header(s)", nr_headers);
+  return 0;
+}
+
+/* This is called with the lock held when we must run or re-run the
+ * cookie-script.
+ */
+static int
+run_cookie_script (struct curl_handle *h)
+{
+  int fd;
+  char tmpfile[] = "/tmp/errorsXXXXXX";
+  FILE *fp;
+  CLEANUP_FREE char *cmd = NULL, *line = NULL;
+  size_t len = 0, linelen = 0;
+
+  assert (cookie_script != NULL); /* checked by caller */
+
+  /* Reset the cookies. */
+  free (cookies_from_script);
+  cookies_from_script = NULL;
+
+  /* Create a temporary file for the errors so we can redirect them
+   * into nbdkit_error.
+   */
+  fd = mkstemp (tmpfile);
+  if (fd == -1) {
+    nbdkit_error ("mkstemp");
+    return -1;
+  }
+  close (fd);
+
+  /* Generate the full script with the local $url variable. */
+  fp = open_memstream (&cmd, &len);
+  if (fp == NULL) {
+    nbdkit_error ("open_memstream: %m");
+    return -1;
+  }
+  fprintf (fp, "exec </dev/null\n");    /* Avoid stdin leaking
(nbdkit -s). */
+  fprintf (fp, "exec 2>%s\n", tmpfile); /* Catch errors to a
temporary file. */
+  fprintf (fp, "url=");                 /* Set the shell variable. */
+  shell_quote (url, fp);
+  putc ('\n', fp);
+  fprintf (fp, "iteration=%u\n", cookie_iteration++);
+  putc ('\n', fp);
+  fprintf (fp, "%s", cookie_script);    /* The script or command. */
+  if (fclose (fp) == EOF) {
+    nbdkit_error ("memstream failed");
+    return -1;
+  }
+
+  /* Run the script and read the cookies. */
+  nbdkit_debug ("curl: running cookie-script");
+  fp = popen (cmd, "r");
+  if (fp == NULL) {
+    nbdkit_error ("popen: %m");
+    return -1;
+  }
+  len = getline (&line, &linelen, fp);
+  if (len > 0) {
+    /* Remove trailing \n and whitespace. */
+    while (len > 0 && ascii_isspace (line[len-1]))
+      line[--len] = '\0';
+    if (len > 0) {
+      cookies_from_script = strdup (line);
+      if (cookies_from_script == NULL) {
+        nbdkit_error ("strdup");
+        pclose (fp);
+        return -1;
+      }
+    }
+  }
+
+  /* If the command failed, this should return EOF and the error
+   * message should be in the temporary file (but we only read the
+   * first line).
+   */
+  if (pclose (fp) == EOF) {
+    fp = fopen (tmpfile, "r");
+    if ((len = getline (&line, &linelen, fp)) >= 0) {
+      if (len > 0 && line[len-1] == '\n')
+        line[len-1] = '\0';
+      nbdkit_error ("cookie-script failed: %s", line);
+    }
+    else
+      nbdkit_error ("cookie-script failed");
+    return -1;
+  }
+
+  nbdkit_debug ("cookie-script returned %scookies",
+                cookies_from_script ? "" : "no ");
+  return 0;
+}
diff --git a/tests/test-curl-cookie-script.c b/tests/test-curl-cookie-script.c
new file mode 100644
index 00000000..481207b5
--- /dev/null
+++ b/tests/test-curl-cookie-script.c
@@ -0,0 +1,143 @@
+/* nbdkit
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <libnbd.h>
+
+#include "cleanup.h"
+#include "web-server.h"
+
+#include "test.h"
+
+static int iteration;
+
+#define SCRIPT \
+  "echo iteration=$iteration"
+
+static void
+check_request (const char *request)
+{
+  char expected[64];
+
+  /* Check the Cookie header. */
+  snprintf (expected, sizeof expected,
+            "\r\nCookie: iteration=%u\r\n", iteration);
+  if (strcasestr (request, expected) == NULL) {
+    fprintf (stderr, "%s: no/incorrect iteration cookie in
request\n",
+             program_name);
+    exit (EXIT_FAILURE);
+  }
+}
+
+static char buf[512];
+
+int
+main (int argc, char *argv[])
+{
+  const char *sockpath;
+  struct nbd_handle *nbd;
+  CLEANUP_FREE char *usp_param = NULL;
+
+#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH
+  fprintf (stderr, "%s: curl does not support
CURLOPT_UNIX_SOCKET_PATH\n",
+           program_name);
+  exit (77);
+#endif
+
+  sockpath = web_server ("disk", check_request);
+  if (sockpath == NULL) {
+    fprintf (stderr, "%s: could not start web server thread\n",
program_name);
+    exit (EXIT_FAILURE);
+  }
+
+  nbd = nbd_create ();
+  if (nbd == NULL) {
+    fprintf (stderr, "%s\n", nbd_get_error ());
+    exit (EXIT_FAILURE);
+  }
+
+  /* We expect that connecting will cause a HEAD request (to find the
+   * size).  $iteration will be 0.
+   */
+  iteration = 0;
+
+  /* Start nbdkit. */
+  if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) ==
-1) {
+    perror ("asprintf");
+    exit (EXIT_FAILURE);
+  }
+  char *args[] = {
+    "nbdkit", "-s", "--exit-with-parent",
"-v",
+    "curl",
+    "-D", "curl.verbose=1",
+    "http://localhost/disk",
+    "cookie-script=" SCRIPT,
+    "cookie-script-renew=1",
+    usp_param, /* unix-socket-path=... */
+    NULL
+  };
+  if (nbd_connect_command (nbd, args) == -1) {
+    fprintf (stderr, "%s\n", nbd_get_error ());
+    exit (EXIT_FAILURE);
+  }
+
+  /* Sleep the script will be called again.  $iteration will be 1. */
+  sleep (2);
+  iteration = 1;
+
+  /* Make a request. */
+  if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
+    fprintf (stderr, "%s\n", nbd_get_error ());
+    exit (EXIT_FAILURE);
+  }
+
+  /* Sleep again and make another request.  $iteration will be 2. */
+  sleep (2);
+  iteration = 2;
+
+  if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
+    fprintf (stderr, "%s\n", nbd_get_error ());
+    exit (EXIT_FAILURE);
+  }
+
+  nbd_close (nbd);
+  exit (EXIT_SUCCESS);
+}
diff --git a/tests/test-curl-header-script.c b/tests/test-curl-header-script.c
new file mode 100644
index 00000000..a151af05
--- /dev/null
+++ b/tests/test-curl-header-script.c
@@ -0,0 +1,165 @@
+/* nbdkit
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <libnbd.h>
+
+#include "cleanup.h"
+#include "web-server.h"
+
+#include "test.h"
+
+static int iteration;
+
+#define SCRIPT \
+  "if [ $iteration -eq 0 ]; then echo X-Test: hello; fi\n" \
+  "echo X-Iteration: $iteration\n" \
+  "echo 'X-Empty;'\n"
+
+static void
+check_request (const char *request)
+{
+  char expected[64];
+
+  /* Check the iteration header. */
+  snprintf (expected, sizeof expected, "\r\nX-Iteration: %u\r\n",
iteration);
+  if (strcasestr (request, expected) == NULL) {
+    fprintf (stderr, "%s: no/incorrect X-Iteration header in
request\n",
+             program_name);
+    exit (EXIT_FAILURE);
+  }
+
+  /* Check the test header, only sent when $iteration = 0. */
+  if (iteration == 0) {
+    if (strcasestr (request, "\r\nX-Test: hello\r\n") == NULL) {
+      fprintf (stderr, "%s: no X-Test header in request\n",
program_name);
+      exit (EXIT_FAILURE);
+    }
+  }
+  else {
+    if (strcasestr (request, "\r\nX-Test:") != NULL) {
+      fprintf (stderr, "%s: X-Test header sent but not expected\n",
+               program_name);
+      exit (EXIT_FAILURE);
+    }
+  }
+
+  /* Check the empty header. */
+  if (strcasestr (request, "\r\nX-Empty:\r\n") == NULL) {
+    fprintf (stderr, "%s: no X-Empty header in request\n",
program_name);
+    exit (EXIT_FAILURE);
+  }
+}
+
+static char buf[512];
+
+int
+main (int argc, char *argv[])
+{
+  const char *sockpath;
+  struct nbd_handle *nbd;
+  CLEANUP_FREE char *usp_param = NULL;
+
+#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH
+  fprintf (stderr, "%s: curl does not support
CURLOPT_UNIX_SOCKET_PATH\n",
+           program_name);
+  exit (77);
+#endif
+
+  sockpath = web_server ("disk", check_request);
+  if (sockpath == NULL) {
+    fprintf (stderr, "%s: could not start web server thread\n",
program_name);
+    exit (EXIT_FAILURE);
+  }
+
+  nbd = nbd_create ();
+  if (nbd == NULL) {
+    fprintf (stderr, "%s\n", nbd_get_error ());
+    exit (EXIT_FAILURE);
+  }
+
+  /* We expect that connecting will cause a HEAD request (to find the
+   * size).  $iteration will be 0.
+   */
+  iteration = 0;
+
+  /* Start nbdkit. */
+  if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) ==
-1) {
+    perror ("asprintf");
+    exit (EXIT_FAILURE);
+  }
+  char *args[] = {
+    "nbdkit", "-s", "--exit-with-parent",
"-v",
+    "curl",
+    "-D", "curl.verbose=1",
+    "http://localhost/disk",
+    "header-script=" SCRIPT,
+    "header-script-renew=1",
+    usp_param, /* unix-socket-path=... */
+    NULL
+  };
+  if (nbd_connect_command (nbd, args) == -1) {
+    fprintf (stderr, "%s\n", nbd_get_error ());
+    exit (EXIT_FAILURE);
+  }
+
+  /* Sleep the script will be called again.  $iteration will be 1. */
+  sleep (2);
+  iteration = 1;
+
+  /* Make a request. */
+  if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
+    fprintf (stderr, "%s\n", nbd_get_error ());
+    exit (EXIT_FAILURE);
+  }
+
+  /* Sleep again and make another request.  $iteration will be 2. */
+  sleep (2);
+  iteration = 2;
+
+  if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
+    fprintf (stderr, "%s\n", nbd_get_error ());
+    exit (EXIT_FAILURE);
+  }
+
+  nbd_close (nbd);
+  exit (EXIT_SUCCESS);
+}
diff --git a/.gitignore b/.gitignore
index a39aa675..255a97a5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -113,6 +113,8 @@ plugins/*/*.3
 /tests/stamp-ssh-user-key
 /tests/test-connect
 /tests/test-curl
+/tests/test-curl-cookie-script
+/tests/test-curl-header-script
 /tests/test-data
 /tests/test-delay
 /tests/test-exit-with-parent
-- 
2.27.0

On 7/15/20 3:53 PM, Richard W.M. Jones wrote:
> This rather complex feature solves a problem for certain web services
> that require a cookie or token for access, especially one which must
> be periodically renewed.
> 
> For motivation for this feature see the included documentation, and
> item (1)(b) here:
> 
> https://www.redhat.com/archives/libguestfs/2020-July/msg00069.html
> ---
I see you pushed this in the meantime, but a late review is better than 
none ;)  As usual, a stream-of-consciousness single pass through:
>   
> +=head1 HEADER AND COOKIE SCRIPTS
> +
> +While the C<header> and C<cookie> parameters can be used to
specify
> +static headers and cookies which are used in every HTTP/HTTPS request,
> +the alternate C<header-script> and C<cookie-script> parameters
can be
> +used to run an external script or program to generate headers and/or
> +cookies.  This is particularly useful to access services which require
> +an authorization token.  In addition the C<header-script-renew> and
> +C<cookie-script-renew> parameters allow you to renew the
authorization
> +token by rerunning the script periodically.
What happens if you have header-script-renew=5, but a very slow client 
that sits idle more more than 5 seconds at a time?  Are you burning CPU 
time every five seconds just to throw it away, or are you letting it 
idle until the start of each client command where you check whether the 
time since last renew is larger than the limit.  (I guess that's 
implementation, not worth documenting, so I'll see if I spot it below)
> +
> +C<header-script> is incompatible with C<header>, and
C<cookie-script>
> +is incompatible with C<cookie>.
> +
> +=head2 Header script
> +
> +The header script should print zero or more HTTP headers, each line of
> +output in the same format as the C<header> parameter.  The headers
> +printed by the script are passed to L<CURLOPT_HTTPHEADER(3)>.
> +
> +In the following example, an imaginary web service requires
> +authentication using a token fetched from a separate login server.
> +The token expires after 60 seconds, so we also tell the plugin that it
> +must renew the token (by re-running the script) if more than 45
> +seconds have elapsed since the last request:
> +
> + nbdkit curl https://service.example.com/disk.img \
> +        header-script='
> +          echo -n "Authorization: Bearer "
'echo -n' is not 100% portable; better might be using 'printf'
> +          curl -s -X POST https://auth.example.com/login |
> +               jq -r .token
> +        ' \
> +        header-script-renew=50
Example is inconsistent with text (45 vs. 50), but it looks like you 
answered my question above, and DO let the script idle when the client 
is idle.

> +=head2 Example: VMware ESXi cookies
> +
> +VMware ESXi’s web server can expose both VMDK and raw format disk
> +images, but requires you to log in using HTTP Basic Authentication.
> +While you can use the C<user> and C<password> parameters to
send HTTP
> +Basic Authentication headers in every request, tests have shown that
> +it is faster to accept the cookie which the server returns and send
> +that instead.  (It is not clear why it is faster, but one theory is
> +that VMware has to do a more expensive username and password check
> +each time.)
The perils of having to reverse-engineer closed-source software behavior ;)
> +
> +The web server can be accessed as below.  Since the cookie expires
> +after a certain period of time, we use C<cookie-script-renew>, and
> +because the server uses a self-signed certificate we must use
> +I<--insecure> and C<sslverify=false>.
> +
> + SERVER=esx.example.com
> + DCPATH=data
> + DS=datastore1
> + GUEST=guest-name
> +
URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
> +
> + nbdkit curl "$URL" \
> +        cookie-script='
> +            curl --head -s --insecure -u root:password "$url" |
> +                 sed -ne '{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }'
> +        ' \
Umm, this is nesting '' inside ''.   You probably need
'\'' in the sed line.
> +        cookie-script-renew=500 \
> +        sslverify=false
> +
> +=head2 Example: Docker Hub authorization tokens
> +
> +Accessing objects like container layers from Docker Hub requires that
> +you first fetch an authorization token, even for anonymous access.
> +These tokens expire after about 5 minutes (300 seconds) so must be
> +periodically renewed.
> +
> +You will need this authorization script (F</tmp/auth.sh>):
> +
> + #!/bin/sh -
> + IMAGE=library/fedora
> + curl -s
"https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull"
|
> +      jq -r .token
> +
> +You will also need this script to get the blobSum of the layer
> +(F</tmp/blobsum.sh>):
> +
> + #!/bin/sh -
> + TOKEN=`/tmp/auth.sh`
> + IMAGE=library/fedora
> + curl -s -X GET -H "Authorization: Bearer $TOKEN" \
> +      "https://registry-1.docker.io/v2/$IMAGE/manifests/latest"
|
> +      jq -r '.fsLayers[0].blobSum'
> +
> +Both scripts must be executable, and both can be run on their own to
> +check they are working.  To run nbdkit:
> +
> + IMAGE=library/fedora
> + BLOBSUM=`/tmp/blobsum.sh`
> + URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
> +
> + nbdkit curl "$URL" \
> +        header-script=' echo -n "Authorization: Bearer ";
/tmp/auth.sh ' \
Another s/echo -n/printf/
> +        header-script-renew=200 \
> +        --filter=gzip
> +
> +Note that this exposes a tar file over NBD.  See also
> +L<nbdkit-tar-filter(1)>.
> +
>   =head1 DEBUG FLAG
>   
> +++ b/plugins/curl/curl.c
> @@ -48,9 +48,11 @@
>   
>   #include <nbdkit-plugin.h>
>   
> -#include "cleanup.h"
>   #include "ascii-ctype.h"
>   #include "ascii-string.h"
> +#include "cleanup.h"
> +
> +#include "curldefs.h"
>   
>   /* Macro CURL_AT_LEAST_VERSION was added in 2015 (Curl 7.43) so if the
>    * macro isn't present then Curl is very old.
> @@ -61,24 +63,29 @@
>   #endif
>   #endif
>   
> -static const char *url = NULL;  /* required */
> +/* Plugin configuration. */
> +const char *url = NULL;         /* required */
>   
> -static const char *cainfo = NULL;
> -static const char *capath = NULL;
> -static char *cookie = NULL;
> -static struct curl_slist *headers = NULL;
> -static char *password = NULL;
> -static long protocols = CURLPROTO_ALL;
> -static const char *proxy = NULL;
> -static char *proxy_password = NULL;
> -static const char *proxy_user = NULL;
> -static bool sslverify = true;
> -static bool tcp_keepalive = false;
> -static bool tcp_nodelay = true;
> -static uint32_t timeout = 0;
> -static const char *unix_socket_path = NULL;
> -static const char *user = NULL;
> -static const char *user_agent = NULL;
> +const char *cainfo = NULL;
> +const char *capath = NULL;
> +char *cookie = NULL;
> +const char *cookie_script = NULL;
> +unsigned cookie_script_renew = 0;
> +struct curl_slist *headers = NULL;
> +const char *header_script = NULL;
> +unsigned header_script_renew = 0;
> +char *password = NULL;
All of these variables are already guaranteed zero-initialized without 
being explicit,
> +long protocols = CURLPROTO_ALL;
although consistency with this initialization (which is not necessarily 
0) makes sense.
> @@ -450,7 +490,11 @@ curl_open (int readonly)
>   
>     /* Get the file size and also whether the remote HTTP server
>      * supports byte ranges.
> +   *
> +   * We must run the scripts if necessary and set headers in the
> +   * handle.
>      */
> +  if (do_scripts (h) == -1) goto err;
Okay, code matches my guess above - you DO check on each client 
interaction, and run the script as-needed, rather than kicking off a 
background thread that runs the script on a blind timeout cycle.
> +++ b/plugins/curl/scripts.c
> +
> +  /* Set headers and cookies in the handle.
> +   *
> +   * When calling CURLOPT_HTTPHEADER we have to keep the list around
> +   * because unfortunately curl doesn't take a copy.  Since we
don't
> +   * know which other threads might be using it, we must make a copy
> +   * of the global list (headers_from_script) per handle
> +   * (h->headers_copy).  For CURLOPT_COOKIE, curl internally takes a
> +   * copy so we don't need to do this.
The comment on memory life cycles is very helpful (and I suspect you 
went through several iterations before figuring out an arrangement that 
works)

> +
> +/* This is called with the lock held when we must run or re-run the
> + * header-script.
> + */
> +static int
> +run_header_script (struct curl_handle *h)
> +{
> +
> +/* This is called with the lock held when we must run or re-run the
> + * cookie-script.
> + */
> +static int
> +run_cookie_script (struct curl_handle *h)
> +{
> +  int fd;
These two look similar, is it worth refactoring into a common helper 
routine?
> +
> +  nbdkit_debug ("cookie-script returned %scookies",
> +                cookies_from_script ? "" : "no ");
This would fail ease-of-translation if we used gettext, but we don't ;)

Overall a pretty slick solution!

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org