Richard W.M. Jones
2020-Apr-10 13:51 UTC
[Libguestfs] [PATCH nbdkit UNFINISHED] Add the ability to write plugins in golang.
Sorry Dan, but I really do dislike golang with a passion :-) Here is a patch that allows you to write nbdkit plugins in golang. As with C, OCaml and Rust, you can write a plugin in Go which compiles directly to a .so file that can be loaded into golang, so in that sense it works completely differently from scripting language plugins like Perl and Python where there's an nbdkit-<lang>-plugin that intermediates between nbdkit and the user’s code. With that said, there are many problems. The root cause of most of them is that you cannot pass Go pointers to C, perhaps because the golang developers never heard of registering extra GC roots[1], even though that is common in non-Blub languages like OCaml and Haskell. This leads to awkward complications nicely summarised in this page: https://eli.thegreenplace.net/2019/passing-callbacks-and-pointers-to-cgo/ This requires that every callback has a wrapper across 3 files. Oh and these wrappers cannot be in a library module, they must be copied into the plugin source code, AND they must be commented out by the end-user by hand if the callback is not called. (There may be a way around the latter issues, but I could not work out how.) This means plugins have tons of duplicated source code. Other issues that are not related to that one: - We have to link the nbdkit module with -Wl,--unresolved-symbols=ignore-in-object-files because of the way nbdkit plugins have deliberately unresolved symbols. See the libnbdkit.so proposal on the mailing list for another way to solve this. - Be nice if ./configure could check that golang >= 1.5 since that was the first version that introduced shared libraries. - Since initialization is not synchronous in golang, you cannot rely on anything being initialized in the plugin before nbdkit starts calling in. For this reason I worked around it by having plugin_init() call a start up function (func init_plugin()) where all golang initialization must be done. Otherwise: nbdkit: golang plugin: dlopen ("plugin.so"); let's start initializing init = dlsym ("plugin_init"); return &plugin (uninitialized!) plugin = init (); plugin.load () hey, I'm still initializing! - Related to the previous point: Be nice to move plugin_init() into the nbdkit module. However I don't believe this is possible because this function has to call into the plugin (in main module). - Tests are a joke at the moment. We would really need a test which properly exercises threads / parallel client connections, so we can be sure that the nbdkit thread & golang goroutine models do not conflict in some way. (I don't think they do, but need to check). - Current test func pluginPRead() needs to be completed. - Documentation needs fixing. I didn't want to write too many docs until I know finally how plugins would work. Rich. [1] Reading the proposal here confirms my suspicions, since there is a much simpler, better and more obvious solution than what is proposed: https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md
Richard W.M. Jones
2020-Apr-10 13:51 UTC
[Libguestfs] [PATCH nbdkit UNFINISHED] Add the ability to write plugins in golang.
Similar to C, OCaml and Rust, this is not a plugin per se. Instead it's more of a method and set of tests around writing plugins in golang. They are standalone programs that compile into shared objects that nbdkit can then load (so there is no "go plugin" between nbdkit and the user plugin, unlike in scripting languages like Perl). --- plugins/golang/nbdkit-golang-plugin.pod | 34 ++++++++ configure.ac | 32 +++++++ plugins/golang/Makefile.am | 75 ++++++++++++++++ plugins/golang/test/wrappers.h | 40 +++++++++ plugins/golang/config-test.go | 38 +++++++++ .../src/libguestfs.org/nbdkit/nbdkit.go | 57 +++++++++++++ plugins/golang/test/init.go | 67 +++++++++++++++ plugins/golang/test/run-test.sh | 41 +++++++++ plugins/golang/test/test.go | 82 ++++++++++++++++++ plugins/golang/test/wrappers.go | 85 +++++++++++++++++++ .gitignore | 2 + README | 4 + 12 files changed, 557 insertions(+) diff --git a/plugins/golang/nbdkit-golang-plugin.pod b/plugins/golang/nbdkit-golang-plugin.pod new file mode 100644 index 00000000..b449a830 --- /dev/null +++ b/plugins/golang/nbdkit-golang-plugin.pod @@ -0,0 +1,34 @@ +=head1 NAME + +nbdkit-golang-plugin - writing nbdkit plugins in Go + +=head1 SYNOPSIS + + nbdkit /path/to/plugin.so [arguments...] + +=head1 DESCRIPTION + +This manual page describes how to write nbdkit plugins in compiled +Golang code. Go plugins are compiled to F<*.so> files (the same as +plugins written in C) and are used in the same way. + +XXX MORE DOCS + + + +=head1 VERSION + +Golang plugins first appeared in nbdkit 1.20. + +=head1 SEE ALSO + +L<nbdkit(1)>, +L<nbdkit-plugin(3)>. + +=head1 AUTHORS + +Richard W.M. Jones + +=head1 COPYRIGHT + +Copyright (C) 2020 Red Hat Inc. diff --git a/configure.ac b/configure.ac index c5db962c..4b1c4cd6 100644 --- a/configure.ac +++ b/configure.ac @@ -49,6 +49,7 @@ LT_INIT dnl List of plugins and filters. lang_plugins="\ + golang \ lua \ ocaml \ perl \ @@ -754,6 +755,34 @@ AS_IF([test "x$enable_lua" != "xno"],[ ]) AM_CONDITIONAL([HAVE_LUA],[test "x$enable_lua" = "xyes"]) +dnl Check for golang. +AC_ARG_ENABLE([golang], + AS_HELP_STRING([--disable-golang], [disable Go language plugin]), + [], + [enable_golang=yes]) +AS_IF([test "x$enable_golang" != "xno"],[ + AC_CHECK_PROG([GOLANG],[go],[go],[no]) + AS_IF([test "x$GOLANG" != "xno"],[ + AC_MSG_CHECKING([if $GOLANG is usable]) + AS_IF([$GOLANG run $srcdir/plugins/golang/config-test.go 2>&AS_MESSAGE_LOG_FD],[ + AC_MSG_RESULT([yes]) + + # Substitute some golang environment. + GOOS=`$GOLANG env GOOS` + GOARCH=`$GOLANG env GOARCH` + GOROOT=`$GOLANG env GOROOT` + AC_SUBST([GOOS]) + AC_SUBST([GOARCH]) + AC_SUBST([GOROOT]) + ],[ + AC_MSG_RESULT([no]) + AC_MSG_WARN([golang ($GOLANG) is installed but not usable]) + GOLANG=no + ]) + ]) +],[GOLANG=no]) +AM_CONDITIONAL([HAVE_GOLANG],[test "x$GOLANG" != "xno"]) + dnl Check for curl (only if you want to compile the curl plugin). AC_ARG_WITH([curl], [AS_HELP_STRING([--without-curl], @@ -1002,6 +1031,7 @@ AC_CONFIG_FILES([Makefile plugins/file/Makefile plugins/floppy/Makefile plugins/full/Makefile + plugins/golang/Makefile plugins/guestfs/Makefile plugins/gzip/Makefile plugins/info/Makefile @@ -1125,6 +1155,8 @@ feature "vddk ................................... " \ echo echo "Languages:" echo +feature "go ..................................... " \ + test "x$HAVE_GOLANG_TRUE" = "x" feature "lua .................................... " \ test "x$HAVE_LUA_TRUE" = "x" feature "ocaml .................................. " \ diff --git a/plugins/golang/Makefile.am b/plugins/golang/Makefile.am new file mode 100644 index 00000000..961495f8 --- /dev/null +++ b/plugins/golang/Makefile.am @@ -0,0 +1,75 @@ +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +include $(top_srcdir)/common-rules.mk + +EXTRA_DIST = \ + nbdkit-golang-plugin.pod \ + src/libguestfs.org/nbdkit/nbdkit.go \ + test/init.go \ + test/run-test.sh \ + test/test.go \ + test/wrappers.go \ + test/wrappers.h \ + $(NULL) + +if HAVE_GOLANG + +# There is nothing to build. Everything is statically compiled and +# linked together when we compile the test. + +TESTS = test/run-test.sh +check_DATA = test/nbdkit-gotest-plugin.so + +test/nbdkit-gotest-plugin.so: src/libguestfs.org/nbdkit/nbdkit.go test/init.go test/test.go test/wrappers.go test/wrappers.h + cd test && \ + GOPATH="$(abs_builddir)" \ + $(GOLANG) build \ + -o nbdkit-gotest-plugin.so -buildmode=c-shared + +CLEANFILES += \ + test/nbdkit-gotest-plugin.so \ + test/nbdkit-gotest-plugin.h \ + $(NULL) + +if HAVE_POD + +man_MANS = nbdkit-golang-plugin.3 +CLEANFILES += $(man_MANS) + +nbdkit-golang-plugin.3: nbdkit-golang-plugin.pod + $(PODWRAPPER) --section=3 --man $@ \ + --html $(top_builddir)/html/$@.html \ + $< + +endif HAVE_POD + +endif HAVE_GOLANG diff --git a/plugins/golang/test/wrappers.h b/plugins/golang/test/wrappers.h new file mode 100644 index 00000000..ea654631 --- /dev/null +++ b/plugins/golang/test/wrappers.h @@ -0,0 +1,40 @@ +/* Go wrappers + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +extern void *open_wrapper (int readonly); +extern void close_wrapper (void *handle); +extern int64_t get_size_wrapper (void *handle); +extern int pread_wrapper (void *handle, void *buf, + uint32_t count, uint64_t offset, uint32_t flags); +extern int pwrite_wrapper (void *handle, const void *buf, + uint32_t count, uint64_t offset, uint32_t flags); +// XXX ALL WRAPPERS diff --git a/plugins/golang/config-test.go b/plugins/golang/config-test.go new file mode 100644 index 00000000..0f5cfe6b --- /dev/null +++ b/plugins/golang/config-test.go @@ -0,0 +1,38 @@ +/* Go configuration test + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* Used by ./configure to check golang is functional. */ + +package main + +func main() { +} diff --git a/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go b/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go new file mode 100644 index 00000000..c21eabb3 --- /dev/null +++ b/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go @@ -0,0 +1,57 @@ +/* Go helper functions. + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +package nbdkit + +/* +#cgo pkg-config: nbdkit +#cgo LDFLAGS: -Wl,--unresolved-symbols=ignore-in-object-files + +#include <stdio.h> +#include <stdlib.h> + +#define NBDKIT_API_VERSION 2 +#include <nbdkit-plugin.h> + +// cgo cannot call varargs functions. +void +_nbdkit_debug (const char *s) +{ + nbdkit_debug ("%s", s); +} + +*/ +import "C" + +func Debug(s string) { + C._nbdkit_debug(C.CString(s)) +} diff --git a/plugins/golang/test/init.go b/plugins/golang/test/init.go new file mode 100644 index 00000000..bfb02f62 --- /dev/null +++ b/plugins/golang/test/init.go @@ -0,0 +1,67 @@ +/* Initialize the plugin. + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +package main + +/* +#cgo pkg-config: nbdkit + +#include <stdio.h> +#include <stdlib.h> + +#define NBDKIT_API_VERSION 2 +#include <nbdkit-plugin.h> +*/ +import "C" +import "unsafe" + +var plugin C.struct_nbdkit_plugin + +//export plugin_init +func plugin_init() *C.struct_nbdkit_plugin { + // Initialization is asynchronous, but need it to happen + // synchronously, so we need to provide a function that the + // user code can call to set plugin fields. + init_plugin() + + // Set up the hidden plugin fields as for C. + struct_size := C.ulong(unsafe.Sizeof(plugin)) + plugin._struct_size = struct_size + plugin._api_version = C.NBDKIT_API_VERSION + plugin._thread_model = C.NBDKIT_THREAD_MODEL_PARALLEL + p := (*C.struct_nbdkit_plugin)(C.malloc(struct_size)) + *p = plugin + return p +} + +// This must exist, but it is never called. +func main() {} diff --git a/plugins/golang/test/run-test.sh b/plugins/golang/test/run-test.sh new file mode 100755 index 00000000..5c10f1b7 --- /dev/null +++ b/plugins/golang/test/run-test.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +set -e +set -x + +if ! qemu-img --version; then + echo "qemu-img is required to run this test" + exit 1 +fi + +../../nbdkit -f -v test/nbdkit-gotest-plugin.so --run 'qemu-img info $nbd' diff --git a/plugins/golang/test/test.go b/plugins/golang/test/test.go new file mode 100644 index 00000000..c9544abc --- /dev/null +++ b/plugins/golang/test/test.go @@ -0,0 +1,82 @@ +/* Test plugin. + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +package main + +/* +#cgo pkg-config: nbdkit + +#include <stdio.h> +#include <stdlib.h> + +#define NBDKIT_API_VERSION 2 +#include <nbdkit-plugin.h> +#include "wrappers.h" +*/ +import "C" +import "unsafe" +import "libguestfs.org/nbdkit" + +//export pluginOpen +func pluginOpen(readonly C.int) unsafe.Pointer { + nbdkit.Debug("golang code running in the .open callback") + // Cannot return a golang pointer, need to allocate a C obj. + return unsafe.Pointer(C.CString("hello")) +} + +//export pluginClose +func pluginClose(handle unsafe.Pointer) { + nbdkit.Debug("golang code running in the .close callback") + C.free(handle) +} + +//export pluginGetSize +func pluginGetSize(handle unsafe.Pointer) C.int64_t { + nbdkit.Debug("golang code running in the .get_size callback") + return 1024 * 1024 +} + +//export pluginPRead +func pluginPRead(handle unsafe.Pointer, buf []byte, count C.uint32_t, + offset C.uint64_t, flags C.uint32_t) C.int { + nbdkit.Debug("golang code running in the .pread callback") + // XXX + return 0 +} + +func init_plugin() { + plugin.name = C.CString("test") + plugin.open = (*[0]byte)(C.open_wrapper) + plugin.close = (*[0]byte)(C.close_wrapper) + plugin.get_size = (*[0]byte)(C.get_size_wrapper) + plugin.pread = (*[0]byte)(C.pread_wrapper) +} diff --git a/plugins/golang/test/wrappers.go b/plugins/golang/test/wrappers.go new file mode 100644 index 00000000..e7e0d3a9 --- /dev/null +++ b/plugins/golang/test/wrappers.go @@ -0,0 +1,85 @@ +/* Go wrappers + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +package main + +/* +#cgo pkg-config: nbdkit + +#include <stdio.h> +#include <stdlib.h> + +#define NBDKIT_API_VERSION 2 +#include <nbdkit-plugin.h> +#include "wrappers.h" + +extern void *pluginOpen (); +void * +open_wrapper (int readonly) +{ + return pluginOpen (readonly); +} + +extern void pluginClose (); +void +close_wrapper (void *handle) +{ + pluginClose (handle); +} + +extern int64_t pluginGetSize (); +int64_t +get_size_wrapper (void *handle) +{ + return pluginGetSize (handle); +} + +extern int pluginPRead (); +int +pread_wrapper (void *handle, void *buf, uint32_t count, uint64_t offset, + uint32_t flags) +{ + return pluginPRead (handle, buf, count, offset, flags); +} + +//extern int pluginPWrite (); +//int +//pwrite_wrapper (void *handle, const void *buf, uint32_t count, uint64_t offset, +// uint32_t flags) +//{ +// return pluginPWrite (handle, buf, count, offset, flags); +//} + +// XXX ALL WRAPPERS + +*/ +import "C" diff --git a/.gitignore b/.gitignore index c44fb40d..7e71df35 100644 --- a/.gitignore +++ b/.gitignore @@ -71,6 +71,8 @@ plugins/*/*.3 /plugins/example4/nbdkit-example4-plugin /plugins/eval/call.c /plugins/eval/methods.c +/plugins/golang/test/nbdkit-gotest-plugin.h +/plugins/golang/test/nbdkit-gotest-plugin.so /plugins/ocaml/nbdkit-ocamlexample-plugin.so /plugins/rust/Cargo.lock /plugins/rust/Cargo.toml diff --git a/README b/README index 05f1e060..1e05846d 100644 --- a/README +++ b/README @@ -148,6 +148,10 @@ For the Rust plugin: - cargo (other dependencies will be downloaded at build time) +To be able to write plugins in golang: + + - go >= 1.5 + For bash tab completion: - bash-completion >= 1.99 -- 2.25.0
Richard W.M. Jones
2020-Apr-10 13:57 UTC
Re: [Libguestfs] [PATCH nbdkit UNFINISHED] Add the ability to write plugins in golang.
On Fri, Apr 10, 2020 at 02:51:51PM +0100, Richard W.M. Jones wrote:> Here is a patch that allows you to write nbdkit plugins in golang. As > with C, OCaml and Rust, you can write a plugin in Go which compiles > directly to a .so file that can be loaded into golang, so in thatinto *nbdkit* Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Daniel P. Berrangé
2020-Apr-15 16:27 UTC
Re: [Libguestfs] [PATCH nbdkit UNFINISHED] Add the ability to write plugins in golang.
On Fri, Apr 10, 2020 at 02:51:52PM +0100, Richard W.M. Jones wrote:> Similar to C, OCaml and Rust, this is not a plugin per se. Instead > it's more of a method and set of tests around writing plugins in > golang. They are standalone programs that compile into shared objects > that nbdkit can then load (so there is no "go plugin" between nbdkit > and the user plugin, unlike in scripting languages like Perl).Why did you choose this approach ? Looking at the code below there's the general boilerplate that will be approx the same for all plugins with cut+paste tedium across projects, and then there's the "interesting" code in test.go The methods in test.go though look quite unappealing from a Go programmer's POV, as the API contracts are full of CGo types and unsafe pointers. To make something that is attractive for Go programmers, I think it needs to hide all the low level CGo stuff entirely. Implementing a nbdkit plugin should require nothing more that providing pure Go code that satisfies a well defined Go "interface" type definition. There should not be any copy+paste of example boilerplate, nor any use of CGo. As an illustration, consider an interface and basic infrastructure: package nbdkitplugin type NBDKitFeature int const ( NBDKitFeatureGetSize NBDKitFeature = iota NBDKitFeaturePread ...other optional methods... ) type NBDKitPlugin interface { NBDGetFeatures() []NBDKitFeature NBDOpen(readonly bool) NBDClose() NBDGetSize() int64 NBDPRead(count uint32, offset uint64, flags uint32) ([]byte) ....many other methods... } var plugin *C.struct_nbdkit_plugin var pluginImpl NBDKitPlugin func PluginInit(name string, impl NBDKitPlugin) { pluginImpl = impl features = impl.GetFeatures() plugin.name = C.CString(name) plugin.open = (*[0]byte)(C.open_wrapper) plugin.close = (*[0]byte)(C.close_wrapper) for _, feature := range features { switch feature { case NBDKitFeatureGetSize: plugin.get_size = (*[0]byte)(C.get_size_wrapper) case NBDKitFeaturePread: plugin.pread = (*[0]byte)(C.pread_wrapper) } } } ....all the methods like C.pread_wrapper/C.get_size_wrapper need to invoke pluginImpl methods.... // This type implements all methods in NBDKitPlugin, with // no-op impls. This means that people implementing plugins // don't need to implement every method in the interface, // only the few they care about type PluginBase struct { } // We don't implement anytrhing by default func (plugin *PluginBase) NBDGetFeatures() []NBDKitFeature { return []NBDKitFeature{} } func (plugin *PluginBase) NBDOpen(readonly bool) {} func (plugin *PluginBase) NBDClose() {} func (plugin *PluginBase) NBDGetSize() int64 { return 0 } } func (plugin *PluginBase) NBDPRead(count uint32, offset uint64, flags uint32) []byte { return []byte{} } ... no-op impls of all other methods for NBDKitPlugin interface... This code above would all be a standalone go module that is just imported. Now an impl of a plugin becomes just one single file import ( libguestfs.org/nbdkitplugin ) type TestPlugin struct { nbdkitplugin.PluginBase // This provides no-op impls of all methods ...blah... } func NewTestPlugin() NBDKitPlugin { return &TestPlugin{ ....blah... } } // This declares which methods we're actually implementing func (plugin *TestPlugin) NBDGetFeatures() []NBDKitFeature { return []NBDKitFeature{ NBDKitFeatureGetSize, NBDKitFeaturePread, } } func (plugin *TestPlugin) NBDOpen(readonly bool) { nbdkit.Debug("golang code running in the .open callback") } func (plugin *TestPlugin) NBDClose() { nbdkit.Debug("golang code running in the .close callback") } func (plugin *TestPlugin) NBDGetSize() int64 { nbdkit.Debug("golang code running in the .get_size callback") return 1024 * 1024 } func (plugin *TestPlugin) NBDPRead(count uint32, offset uint64, flags uint32) []byte { nbdkit.Debug("golang code running in the .pread callback") return []byte{} } ... don't need to implement any other methods, since PluginBase satisfies the interface contract.... func init() { nbdkitplugin.PluginInitialize("test", NewTestPlugin()) } Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Maybe Matching Threads
- Re: [PATCH nbdkit UNFINISHED] Add the ability to write plugins in golang.
- [PATCH nbdkit v2] Add the ability to write plugins in golang.
- [PATCH nbdkit 0/2 v2] golang: Compile against the local nbdkit build.
- Re: [PATCH nbdkit v2] Add the ability to write plugins in golang.
- [PATCH nbdkit v3 0/2] golang: Compile against the local nbdkit build.