Libguestfs - Apr 2020 - Re: [PATCH nbdkit UNFINISHED] Add the ability to write plugins in golang.

Sorry Dan, but I really do dislike golang with a passion :-)

Here is a patch that allows you to write nbdkit plugins in golang.  As
with C, OCaml and Rust, you can write a plugin in Go which compiles
directly to a .so file that can be loaded into golang, so in that
sense it works completely differently from scripting language plugins
like Perl and Python where there's an nbdkit-<lang>-plugin that
intermediates between nbdkit and the user’s code.

With that said, there are many problems.  The root cause of most of
them is that you cannot pass Go pointers to C, perhaps because the
golang developers never heard of registering extra GC roots[1], even
though that is common in non-Blub languages like OCaml and Haskell.
This leads to awkward complications nicely summarised in this page:

https://eli.thegreenplace.net/2019/passing-callbacks-and-pointers-to-cgo/

This requires that every callback has a wrapper across 3 files.  Oh
and these wrappers cannot be in a library module, they must be copied
into the plugin source code, AND they must be commented out by the
end-user by hand if the callback is not called.  (There may be a way
around the latter issues, but I could not work out how.)  This means
plugins have tons of duplicated source code.

Other issues that are not related to that one:

 - We have to link the nbdkit module with
   -Wl,--unresolved-symbols=ignore-in-object-files because of the way
   nbdkit plugins have deliberately unresolved symbols.  See the
   libnbdkit.so proposal on the mailing list for another way to solve
   this.

 - Be nice if ./configure could check that golang >= 1.5 since that
   was the first version that introduced shared libraries.

 - Since initialization is not synchronous in golang, you cannot rely
   on anything being initialized in the plugin before nbdkit starts
   calling in.  For this reason I worked around it by having
   plugin_init() call a start up function (func init_plugin()) where
   all golang initialization must be done.  Otherwise:

      nbdkit:                          golang plugin:

      dlopen ("plugin.so");
                                       let's start initializing
      init = dlsym ("plugin_init");
                                       return &plugin (uninitialized!)
      plugin = init ();
      plugin.load ()
                                       hey, I'm still initializing!

 - Related to the previous point: Be nice to move plugin_init() into
   the nbdkit module.  However I don't believe this is possible
   because this function has to call into the plugin (in main module).

 - Tests are a joke at the moment.  We would really need a test which
   properly exercises threads / parallel client connections, so we can
   be sure that the nbdkit thread & golang goroutine models do not
   conflict in some way.  (I don't think they do, but need to check).

 - Current test func pluginPRead() needs to be completed.

 - Documentation needs fixing.  I didn't want to write too many docs
   until I know finally how plugins would work.

Rich.

[1] Reading the proposal here confirms my suspicions, since there is a
much simpler, better and more obvious solution than what is proposed:
https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md

Similar to C, OCaml and Rust, this is not a plugin per se.  Instead
it's more of a method and set of tests around writing plugins in
golang.  They are standalone programs that compile into shared objects
that nbdkit can then load (so there is no "go plugin" between nbdkit
and the user plugin, unlike in scripting languages like Perl).
---
 plugins/golang/nbdkit-golang-plugin.pod       | 34 ++++++++
 configure.ac                                  | 32 +++++++
 plugins/golang/Makefile.am                    | 75 ++++++++++++++++
 plugins/golang/test/wrappers.h                | 40 +++++++++
 plugins/golang/config-test.go                 | 38 +++++++++
 .../src/libguestfs.org/nbdkit/nbdkit.go       | 57 +++++++++++++
 plugins/golang/test/init.go                   | 67 +++++++++++++++
 plugins/golang/test/run-test.sh               | 41 +++++++++
 plugins/golang/test/test.go                   | 82 ++++++++++++++++++
 plugins/golang/test/wrappers.go               | 85 +++++++++++++++++++
 .gitignore                                    |  2 +
 README                                        |  4 +
 12 files changed, 557 insertions(+)

diff --git a/plugins/golang/nbdkit-golang-plugin.pod
b/plugins/golang/nbdkit-golang-plugin.pod
new file mode 100644
index 00000000..b449a830
--- /dev/null
+++ b/plugins/golang/nbdkit-golang-plugin.pod
@@ -0,0 +1,34 @@
+=head1 NAME
+
+nbdkit-golang-plugin - writing nbdkit plugins in Go
+
+=head1 SYNOPSIS
+
+ nbdkit /path/to/plugin.so [arguments...]
+
+=head1 DESCRIPTION
+
+This manual page describes how to write nbdkit plugins in compiled
+Golang code.  Go plugins are compiled to F<*.so> files (the same as
+plugins written in C) and are used in the same way.
+
+XXX MORE DOCS
+
+
+
+=head1 VERSION
+
+Golang plugins first appeared in nbdkit 1.20.
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-plugin(3)>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones
+
+=head1 COPYRIGHT
+
+Copyright (C) 2020 Red Hat Inc.
diff --git a/configure.ac b/configure.ac
index c5db962c..4b1c4cd6 100644
--- a/configure.ac
+++ b/configure.ac
@@ -49,6 +49,7 @@ LT_INIT
 
 dnl List of plugins and filters.
 lang_plugins="\
+        golang \
         lua \
         ocaml \
         perl \
@@ -754,6 +755,34 @@ AS_IF([test "x$enable_lua" != "xno"],[
 ])
 AM_CONDITIONAL([HAVE_LUA],[test "x$enable_lua" = "xyes"])
 
+dnl Check for golang.
+AC_ARG_ENABLE([golang],
+    AS_HELP_STRING([--disable-golang], [disable Go language plugin]),
+        [],
+        [enable_golang=yes])
+AS_IF([test "x$enable_golang" != "xno"],[
+    AC_CHECK_PROG([GOLANG],[go],[go],[no])
+    AS_IF([test "x$GOLANG" != "xno"],[
+        AC_MSG_CHECKING([if $GOLANG is usable])
+        AS_IF([$GOLANG run $srcdir/plugins/golang/config-test.go
2>&AS_MESSAGE_LOG_FD],[
+            AC_MSG_RESULT([yes])
+
+            # Substitute some golang environment.
+            GOOS=`$GOLANG env GOOS`
+            GOARCH=`$GOLANG env GOARCH`
+            GOROOT=`$GOLANG env GOROOT`
+            AC_SUBST([GOOS])
+            AC_SUBST([GOARCH])
+            AC_SUBST([GOROOT])
+        ],[
+            AC_MSG_RESULT([no])
+            AC_MSG_WARN([golang ($GOLANG) is installed but not usable])
+            GOLANG=no
+        ])
+    ])
+],[GOLANG=no])
+AM_CONDITIONAL([HAVE_GOLANG],[test "x$GOLANG" != "xno"])
+
 dnl Check for curl (only if you want to compile the curl plugin).
 AC_ARG_WITH([curl],
     [AS_HELP_STRING([--without-curl],
@@ -1002,6 +1031,7 @@ AC_CONFIG_FILES([Makefile
                  plugins/file/Makefile
                  plugins/floppy/Makefile
                  plugins/full/Makefile
+                 plugins/golang/Makefile
                  plugins/guestfs/Makefile
                  plugins/gzip/Makefile
                  plugins/info/Makefile
@@ -1125,6 +1155,8 @@ feature "vddk ...................................
" \
 echo
 echo "Languages:"
 echo
+feature "go ..................................... " \
+        test "x$HAVE_GOLANG_TRUE" = "x"
 feature "lua .................................... " \
         test "x$HAVE_LUA_TRUE" = "x"
 feature "ocaml .................................. " \
diff --git a/plugins/golang/Makefile.am b/plugins/golang/Makefile.am
new file mode 100644
index 00000000..961495f8
--- /dev/null
+++ b/plugins/golang/Makefile.am
@@ -0,0 +1,75 @@
+# nbdkit
+# Copyright (C) 2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+include $(top_srcdir)/common-rules.mk
+
+EXTRA_DIST = \
+	nbdkit-golang-plugin.pod \
+	src/libguestfs.org/nbdkit/nbdkit.go \
+	test/init.go \
+	test/run-test.sh \
+	test/test.go \
+	test/wrappers.go \
+	test/wrappers.h \
+	$(NULL)
+
+if HAVE_GOLANG
+
+# There is nothing to build.  Everything is statically compiled and
+# linked together when we compile the test.
+
+TESTS = test/run-test.sh
+check_DATA = test/nbdkit-gotest-plugin.so
+
+test/nbdkit-gotest-plugin.so: src/libguestfs.org/nbdkit/nbdkit.go test/init.go
test/test.go test/wrappers.go test/wrappers.h
+	cd test && \
+	GOPATH="$(abs_builddir)" \
+	$(GOLANG) build \
+	    -o nbdkit-gotest-plugin.so -buildmode=c-shared
+
+CLEANFILES += \
+	test/nbdkit-gotest-plugin.so \
+	test/nbdkit-gotest-plugin.h \
+	$(NULL)
+
+if HAVE_POD
+
+man_MANS = nbdkit-golang-plugin.3
+CLEANFILES += $(man_MANS)
+
+nbdkit-golang-plugin.3: nbdkit-golang-plugin.pod
+	$(PODWRAPPER) --section=3 --man $@ \
+	    --html $(top_builddir)/html/$@.html \
+	    $<
+
+endif HAVE_POD
+
+endif HAVE_GOLANG
diff --git a/plugins/golang/test/wrappers.h b/plugins/golang/test/wrappers.h
new file mode 100644
index 00000000..ea654631
--- /dev/null
+++ b/plugins/golang/test/wrappers.h
@@ -0,0 +1,40 @@
+/* Go wrappers
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+extern void *open_wrapper (int readonly);
+extern void close_wrapper (void *handle);
+extern int64_t get_size_wrapper (void *handle);
+extern int pread_wrapper (void *handle, void *buf,
+                          uint32_t count, uint64_t offset, uint32_t flags);
+extern int pwrite_wrapper (void *handle, const void *buf,
+                           uint32_t count, uint64_t offset, uint32_t flags);
+// XXX ALL WRAPPERS
diff --git a/plugins/golang/config-test.go b/plugins/golang/config-test.go
new file mode 100644
index 00000000..0f5cfe6b
--- /dev/null
+++ b/plugins/golang/config-test.go
@@ -0,0 +1,38 @@
+/* Go configuration test
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/* Used by ./configure to check golang is functional. */
+
+package main
+
+func main() {
+}
diff --git a/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go
b/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go
new file mode 100644
index 00000000..c21eabb3
--- /dev/null
+++ b/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go
@@ -0,0 +1,57 @@
+/* Go helper functions.
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+package nbdkit
+
+/*
+#cgo pkg-config: nbdkit
+#cgo LDFLAGS: -Wl,--unresolved-symbols=ignore-in-object-files
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#define NBDKIT_API_VERSION 2
+#include <nbdkit-plugin.h>
+
+// cgo cannot call varargs functions.
+void
+_nbdkit_debug (const char *s)
+{
+  nbdkit_debug ("%s", s);
+}
+
+*/
+import "C"
+
+func Debug(s string) {
+	C._nbdkit_debug(C.CString(s))
+}
diff --git a/plugins/golang/test/init.go b/plugins/golang/test/init.go
new file mode 100644
index 00000000..bfb02f62
--- /dev/null
+++ b/plugins/golang/test/init.go
@@ -0,0 +1,67 @@
+/* Initialize the plugin.
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+package main
+
+/*
+#cgo pkg-config: nbdkit
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#define NBDKIT_API_VERSION 2
+#include <nbdkit-plugin.h>
+*/
+import "C"
+import "unsafe"
+
+var plugin C.struct_nbdkit_plugin
+
+//export plugin_init
+func plugin_init() *C.struct_nbdkit_plugin {
+	// Initialization is asynchronous, but need it to happen
+	// synchronously, so we need to provide a function that the
+	// user code can call to set plugin fields.
+	init_plugin()
+
+	// Set up the hidden plugin fields as for C.
+	struct_size := C.ulong(unsafe.Sizeof(plugin))
+	plugin._struct_size = struct_size
+	plugin._api_version = C.NBDKIT_API_VERSION
+	plugin._thread_model = C.NBDKIT_THREAD_MODEL_PARALLEL
+	p := (*C.struct_nbdkit_plugin)(C.malloc(struct_size))
+	*p = plugin
+	return p
+}
+
+// This must exist, but it is never called.
+func main() {}
diff --git a/plugins/golang/test/run-test.sh b/plugins/golang/test/run-test.sh
new file mode 100755
index 00000000..5c10f1b7
--- /dev/null
+++ b/plugins/golang/test/run-test.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# nbdkit
+# Copyright (C) 2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+set -e
+set -x
+
+if ! qemu-img --version; then
+    echo "qemu-img is required to run this test"
+    exit 1
+fi
+
+../../nbdkit -f -v test/nbdkit-gotest-plugin.so --run 'qemu-img info
$nbd'
diff --git a/plugins/golang/test/test.go b/plugins/golang/test/test.go
new file mode 100644
index 00000000..c9544abc
--- /dev/null
+++ b/plugins/golang/test/test.go
@@ -0,0 +1,82 @@
+/* Test plugin.
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+package main
+
+/*
+#cgo pkg-config: nbdkit
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#define NBDKIT_API_VERSION 2
+#include <nbdkit-plugin.h>
+#include "wrappers.h"
+*/
+import "C"
+import "unsafe"
+import "libguestfs.org/nbdkit"
+
+//export pluginOpen
+func pluginOpen(readonly C.int) unsafe.Pointer {
+	nbdkit.Debug("golang code running in the .open callback")
+	// Cannot return a golang pointer, need to allocate a C obj.
+	return unsafe.Pointer(C.CString("hello"))
+}
+
+//export pluginClose
+func pluginClose(handle unsafe.Pointer) {
+	nbdkit.Debug("golang code running in the .close callback")
+	C.free(handle)
+}
+
+//export pluginGetSize
+func pluginGetSize(handle unsafe.Pointer) C.int64_t {
+	nbdkit.Debug("golang code running in the .get_size callback")
+	return 1024 * 1024
+}
+
+//export pluginPRead
+func pluginPRead(handle unsafe.Pointer, buf []byte, count C.uint32_t,
+	offset C.uint64_t, flags C.uint32_t) C.int {
+	nbdkit.Debug("golang code running in the .pread callback")
+	// XXX
+	return 0
+}
+
+func init_plugin() {
+	plugin.name = C.CString("test")
+	plugin.open = (*[0]byte)(C.open_wrapper)
+	plugin.close = (*[0]byte)(C.close_wrapper)
+	plugin.get_size = (*[0]byte)(C.get_size_wrapper)
+	plugin.pread = (*[0]byte)(C.pread_wrapper)
+}
diff --git a/plugins/golang/test/wrappers.go b/plugins/golang/test/wrappers.go
new file mode 100644
index 00000000..e7e0d3a9
--- /dev/null
+++ b/plugins/golang/test/wrappers.go
@@ -0,0 +1,85 @@
+/* Go wrappers
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+package main
+
+/*
+#cgo pkg-config: nbdkit
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#define NBDKIT_API_VERSION 2
+#include <nbdkit-plugin.h>
+#include "wrappers.h"
+
+extern void *pluginOpen ();
+void *
+open_wrapper (int readonly)
+{
+  return pluginOpen (readonly);
+}
+
+extern void pluginClose ();
+void
+close_wrapper (void *handle)
+{
+  pluginClose (handle);
+}
+
+extern int64_t pluginGetSize ();
+int64_t
+get_size_wrapper (void *handle)
+{
+  return pluginGetSize (handle);
+}
+
+extern int pluginPRead ();
+int
+pread_wrapper (void *handle, void *buf, uint32_t count, uint64_t offset,
+               uint32_t flags)
+{
+  return pluginPRead (handle, buf, count, offset, flags);
+}
+
+//extern int pluginPWrite ();
+//int
+//pwrite_wrapper (void *handle, const void *buf, uint32_t count, uint64_t
offset,
+//                uint32_t flags)
+//{
+//  return pluginPWrite (handle, buf, count, offset, flags);
+//}
+
+// XXX ALL WRAPPERS
+
+*/
+import "C"
diff --git a/.gitignore b/.gitignore
index c44fb40d..7e71df35 100644
--- a/.gitignore
+++ b/.gitignore
@@ -71,6 +71,8 @@ plugins/*/*.3
 /plugins/example4/nbdkit-example4-plugin
 /plugins/eval/call.c
 /plugins/eval/methods.c
+/plugins/golang/test/nbdkit-gotest-plugin.h
+/plugins/golang/test/nbdkit-gotest-plugin.so
 /plugins/ocaml/nbdkit-ocamlexample-plugin.so
 /plugins/rust/Cargo.lock
 /plugins/rust/Cargo.toml
diff --git a/README b/README
index 05f1e060..1e05846d 100644
--- a/README
+++ b/README
@@ -148,6 +148,10 @@ For the Rust plugin:
 
  - cargo (other dependencies will be downloaded at build time)
 
+To be able to write plugins in golang:
+
+ - go >= 1.5
+
 For bash tab completion:
 
  - bash-completion >= 1.99
-- 
2.25.0

On Fri, Apr 10, 2020 at 02:51:51PM +0100, Richard W.M. Jones
wrote:
> Here is a patch that allows you to write nbdkit plugins in golang.  As
> with C, OCaml and Rust, you can write a plugin in Go which compiles
> directly to a .so file that can be loaded into golang, so in that
into *nbdkit*

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On Fri, Apr 10, 2020 at 02:51:52PM +0100, Richard W.M. Jones
wrote:
> Similar to C, OCaml and Rust, this is not a plugin per se.  Instead
> it's more of a method and set of tests around writing plugins in
> golang.  They are standalone programs that compile into shared objects
> that nbdkit can then load (so there is no "go plugin" between
nbdkit
> and the user plugin, unlike in scripting languages like Perl).
Why did you choose this approach ?

Looking at the code below there's the general boilerplate that
will be approx the same for all plugins with cut+paste tedium
across projects, and then there's the "interesting" code in
test.go

The methods in test.go though look quite unappealing from a
Go programmer's POV, as the API contracts are full of CGo
types and unsafe pointers.

To make something that is attractive for Go programmers, I
think it needs to hide all the low level CGo stuff entirely.

Implementing a nbdkit plugin should require nothing more
that providing pure Go code that satisfies  a well defined
Go "interface" type definition. There should not be any
copy+paste of example boilerplate, nor any use of CGo.

As an illustration, consider an interface and basic infrastructure:


  package nbdkitplugin


  type NBDKitFeature int
  const (
      NBDKitFeatureGetSize NBDKitFeature = iota
      NBDKitFeaturePread
      ...other optional methods...
  )

  type NBDKitPlugin interface {
      NBDGetFeatures() []NBDKitFeature
      NBDOpen(readonly bool)
      NBDClose()
      NBDGetSize() int64
      NBDPRead(count uint32, offset uint64, flags uint32) ([]byte)
      ....many other methods...
  }

  var plugin *C.struct_nbdkit_plugin
  var pluginImpl NBDKitPlugin

  func PluginInit(name string, impl NBDKitPlugin) {
      pluginImpl = impl

      features = impl.GetFeatures()
      plugin.name = C.CString(name)
      plugin.open = (*[0]byte)(C.open_wrapper)
      plugin.close = (*[0]byte)(C.close_wrapper)

      for _, feature := range features {
          switch feature {
              case NBDKitFeatureGetSize:
                plugin.get_size = (*[0]byte)(C.get_size_wrapper)
              case NBDKitFeaturePread:
                plugin.pread = (*[0]byte)(C.pread_wrapper)
          }
      }
      
  }

  ....all the methods like C.pread_wrapper/C.get_size_wrapper
      need to invoke pluginImpl methods....


  // This type implements all methods in NBDKitPlugin, with
  // no-op impls. This means that people implementing plugins
  // don't need to implement every method in the interface,
  // only the few they care about
  type PluginBase struct {
  }

  // We don't implement anytrhing by default
  func (plugin *PluginBase) NBDGetFeatures() []NBDKitFeature {
        return []NBDKitFeature{}
  }

  func (plugin *PluginBase) NBDOpen(readonly bool) {}
 
  func (plugin *PluginBase) NBDClose() {}
 
  func (plugin *PluginBase) NBDGetSize() int64 { return 0 }
  }
 
  func (plugin *PluginBase) NBDPRead(count uint32, offset uint64, flags uint32)
[]byte {
 	return []byte{}
  }

  ...  no-op impls of all other methods for NBDKitPlugin interface...


This code above would all be a standalone go module that is just
imported.


Now an impl of a plugin becomes just one single file

  import (
     libguestfs.org/nbdkitplugin
  )


  type TestPlugin struct {
      nbdkitplugin.PluginBase   // This provides no-op impls of all methods
      ...blah...
  }

  func NewTestPlugin() NBDKitPlugin {
      return &TestPlugin{
         ....blah...
      }
  }

  // This declares which methods we're actually implementing
  func (plugin *TestPlugin) NBDGetFeatures() []NBDKitFeature {
        return []NBDKitFeature{
	    NBDKitFeatureGetSize,
	    NBDKitFeaturePread,
	}
  }

  func (plugin *TestPlugin) NBDOpen(readonly bool) {
 	nbdkit.Debug("golang code running in the .open callback")
  }
 
  func  (plugin *TestPlugin) NBDClose() {
 	nbdkit.Debug("golang code running in the .close callback")
  }
 
  func (plugin *TestPlugin) NBDGetSize() int64 {
 	nbdkit.Debug("golang code running in the .get_size callback")
 	return 1024 * 1024
  }
 
  func (plugin *TestPlugin) NBDPRead(count uint32, offset uint64, flags uint32)
[]byte {
 	nbdkit.Debug("golang code running in the .pread callback")
 	return []byte{}
  }

   ... don't need to implement any other methods, since PluginBase
       satisfies the interface contract....

  func init() {
       nbdkitplugin.PluginInitialize("test", NewTestPlugin())
  }


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|