search for: enable_libfuzz

libFuzzer is Clang's fuzzer, and alternative to using AFL: https://llvm.org/docs/LibFuzzer.html I implemented an alternative method of fuzzing for libnbd earlier today and it's pretty simple: https://github.com/libguestfs/libnbd/commit/c19a6fbae9a21a7d4693418706c59e81ed256875 However it's considerably more difficult to use libFuzzer with non-library code -- in this case nbdkit.

v1 was here: https://www.redhat.com/archives/libguestfs/2019-November/msg00003.html This version depends on: https://www.redhat.com/archives/libguestfs/2019-November/msg00004.html and this series: https://www.redhat.com/archives/libguestfs/2019-November/msg00009.html The delta has been reduced slightly because of changes made possible by cleaning up and fixing the quit path in nbdkit. It's

...CES = nbdkit.c +nbdkit_CFLAGS = $(WARNINGS_CFLAGS) +nbdkit_LDADD = libnbdkit.la -nbdkit_SOURCES = \ +lib_LTLIBRARIES = libnbdkit.la +libnbdkit_la_SOURCES = \ backend.c \ background.c \ captive.c \ @@ -70,10 +74,10 @@ nbdkit_SOURCES = \ $(top_srcdir)/include/nbdkit-filter.h \ $(NULL) if ENABLE_LIBFUZZER -nbdkit_SOURCES += fuzzer.c +libnbdkit_la_SOURCES += fuzzer.c endif -nbdkit_CPPFLAGS = \ +libnbdkit_la_CPPFLAGS = \ -Dbindir=\"$(bindir)\" \ -Dlibdir=\"$(libdir)\" \ -Dmandir=\"$(mandir)\" \ @@ -86,21 +90,22 @@ nbdkit_CPPFLAGS = \ -I$(top_srcdir)/common/pr...

...ctions->get_extent = nbdkit_get_extent; +} +#endif + int main (int argc, char *argv[]) { @@ -178,6 +222,9 @@ main (int argc, char *argv[]) perror ("expecting stdin/stdout to be opened"); exit (EXIT_FAILURE); } +#ifdef WINDOWS_COMPAT + init_functions (); +#endif #if !ENABLE_LIBFUZZER threadlocal_init (); @@ -772,6 +819,7 @@ open_plugin_so (size_t i, const char *name, int short_name) bool free_filename = false; void *dl; struct nbdkit_plugin *(*plugin_init) (void); + void *(*functions_init) (struct nbdkit_functions *); char *error; if (short_name) { @@ -79...

Eric: Did you want to take this one any further? It might be one that we save for > 1.18: https://www.redhat.com/archives/libguestfs/2020-February/thread.html#00206 Another thing I've been thinking about for some time is splitting .config_complete into .config_complete + .get_ready (new name TBD). At the moment .config_complete is both the place where we finish processing config, and

...eletions(-) diff --git a/server/main.c b/server/main.c index 8aa8b497..3bc59781 100644 --- a/server/main.c +++ b/server/main.c @@ -671,10 +671,25 @@ main (int argc, char *argv[]) /* Select the correct thread model based on config. */ lock_init_thread_model (); - set_up_quit_pipe (); -#if !ENABLE_LIBFUZZER - set_up_signals (); -#endif + /* If the user has mixed up -p/--run/-s/-U/--vsock options, then + * give an error. + * + * XXX Actually the server could easily be extended to handle both + * TCP/IP and Unix sockets, or even multiple TCP/IP ports. + */ + if ((port && unixsocke...

...tils/libutils.la \ > + $(NULL) > + > +if USE_LINKER_SCRIPT_FOR_SERVER > +# We have to disable the linker script for libFuzzer because Clang > +# adds loads of fuzzer and ASAN-related symbols that are required by > +# the plugins but which our linker script tries to hide. > +if !ENABLE_LIBFUZZER > +libnbdkit_la_LDFLAGS += -Wl,--version-script=$(srcdir)/libnbdkit.syms > +endif I don't know if you tested libfuzzer, but assume that you'll either test that before pushing this, or have some followup patches as needed. > +endif When nesting Makefile.am conditionals, I like...

...m b/server/Makefile.am index 9351fefc..b6728511 100644 --- a/server/Makefile.am +++ b/server/Makefile.am @@ -33,6 +33,7 @@ include $(top_srcdir)/common-rules.mk EXTRA_DIST = nbdkit.syms +pkglib_LTLIBRARIES = nbdkit-shim-dlopen.la sbin_PROGRAMS = nbdkit nbdkit_SOURCES = \ @@ -71,6 +72,17 @@ if ENABLE_LIBFUZZER nbdkit_SOURCES += fuzzer.c endif +nbdkit_shim_dlopen_la_SOURCES = shim.c +nbdkit_shim_dlopen_la_CPPFLAGS = \ + -I$(top_srcdir)/common/utils \ + $(NULL) +nbdkit_shim_dlopen_la_CFLAGS = $(WARNINGS_CFLAGS) +nbdkit_shim_dlopen_la_LIBADD = $(DL_LIBS) +nbdkit_shim_dlopen_la_LDFLAGS = \ + -module -no...

...BADD = \ + $(top_builddir)/common/utils/libutils.la \ + $(NULL) + +if USE_LINKER_SCRIPT_FOR_SERVER +# We have to disable the linker script for libFuzzer because Clang +# adds loads of fuzzer and ASAN-related symbols that are required by +# the plugins but which our linker script tries to hide. +if !ENABLE_LIBFUZZER +libnbdkit_la_LDFLAGS += -Wl,--version-script=$(srcdir)/libnbdkit.syms +endif +endif diff --git a/server/Makefile.am b/server/Makefile.am index 4c789934..ad0de9b1 100644 --- a/server/Makefile.am +++ b/server/Makefile.am @@ -80,6 +80,7 @@ nbdkit_CPPFLAGS = \ -Dfilterdir=\"$(filterdir)\&quot...

The patches following do indeed allow you to compile nbdkit.exe, but it does not actually work yet. I'm posting this experimental series more as a work in progress and to get feedback. Note this does not require Windows itself to build or test. You can cross-compile it using mingw64-* packages on Fedora or Debian, and test it [spoiler alert: it fails] using Wine. Rich.

This is the previous 5/9 patch posted earlier today, split into reviewable chunks. This passes bisection with -x 'make && make check', but I didn't work very hard on the commit messages, so I refer you back to the original patch to explain how it works: https://www.redhat.com/archives/libguestfs/2020-March/msg00248.html Rich.

I pushed a few of the msys2 patches upstream. I changed the way that $(SHARED_LDFLAGS) works so it's more to my liking, and the others were pushed unchanged. Three patches remain which I'm posting on the mailing list for proper review. Rich.

...8282af 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -40,8 +40,12 @@ EXTRA_DIST = libnbdkit.syms lib_LTLIBRARIES = libnbdkit.la libnbdkit_la_SOURCES = \ + extents.c \ init.c \ lib.h \ + parse.c \ + password.c \ + path.c \ $(NULL) libnbdkit_la_CPPFLAGS = \ @@ -69,3 +73,26 @@ if !ENABLE_LIBFUZZER libnbdkit_la_LDFLAGS += -Wl,--version-script=$(srcdir)/libnbdkit.syms endif endif + +# Unit testing + +TESTS = test-public + +check_PROGRAMS = test-public + +test_public_SOURCES = \ + test-public.c \ + extents.c \ + parse.c \ + password.c \ + $(NULL) +test_public_CPPFLAGS = \ + -I$(top_srcdir)...

Also available here: https://github.com/rwmjones/nbdkit/tree/2020-windows-mingw This is the port to Windows using native Windows APIs (not MSYS or Cygwin). This patch series is at the point where it basically now works. I can run the server with the memory plugin, and access it remotely using guestfish, creating filesystems and so on without any apparent problems. Nevertheless there are many

...quot; - -#include "internal.h" +#include "exit-with-parent.h" #include "nbd-protocol.h" +#include "strndup.h" +#include "syslog.h" + +#include "internal.h" #include "options.h" -#include "exit-with-parent.h" #ifdef ENABLE_LIBFUZZER #define main fuzzer_main diff --git a/server/public.c b/server/public.c index d10d466e..1f7e1af0 100644 --- a/server/public.c +++ b/server/public.c @@ -54,6 +54,8 @@ #include "ascii-ctype.h" #include "ascii-string.h" #include "get-current-dir-name.h" +#include &q...

I'm a lot happier with this version: no mucking with dlmopen(). It does add a bit more coupling between nbdkit proper and the vddk plugin (namely, nbdkit now exports a new function that the vddk plugin relies on), but patch 2 adds testsuite coverage of the new function so we shouldn't regress. Patch 1 and 2 are new, patch 3 is unchanged from when Rich posted it in v2, and patch 4 is

...quot; - -#include "internal.h" +#include "exit-with-parent.h" #include "nbd-protocol.h" +#include "strndup.h" +#include "syslog.h" + +#include "internal.h" #include "options.h" -#include "exit-with-parent.h" #ifdef ENABLE_LIBFUZZER #define main fuzzer_main diff --git a/server/public.c b/server/public.c index d10d466e..1f7e1af0 100644 --- a/server/public.c +++ b/server/public.c @@ -54,6 +54,8 @@ #include "ascii-ctype.h" #include "ascii-string.h" #include "get-current-dir-name.h" +#include &q...

Also available here: https://github.com/rwmjones/nbdkit/tree/2020-windows-mingw-nolib After a lot of work I have made the port to Windows work without using a separate library. Instead, on Windows only, we build an "import library" (library of stubs) which resolves references to nbdkit_* functions in the main program and fixes up the plugin, basically the first technique outlined in

This creates libnbdkit.so as discussed in the following thread: https://www.redhat.com/archives/libguestfs/2020-March/thread.html#00203 test-delay-shutdown.sh fails for unclear reasons. This series starts by reverting "tests: Don't strand hung nbdkit processes" which is because several other tests fail randomly unless I revert this patch. I didn't investigate this yet so it