Libguestfs - Apr 2019 - [PATCH nbdkit] build: Use dlsym as sentinel function for -ldl.

When testing which “dl library” we must use for dl* symbols, autoconf
runs a test similar to:

  $ cat conftest.c
  char dlopen ();
  int main () { return dlopen (); }
  $ gcc -o conftest $CFLAGS conftest.c [try various -ldl options here]

When using ‘CFLAGS="-fsanitize=address"’ this succeeds even if no dl
libraries are used at all, since it appears that using this option
causes dlopen to be included in the final binary implicitly.  This
causes configure to set DL_LIBS='', but unfortunately dlsym is not
included implicitly and so linking fails.

Although I believe this is a bug in GCC 9, as a workaround use dlsym
as the sentinel function instead.

The new output from ‘./configure CFLAGS="-fsanitize=address"’ is:

  checking for library containing dlsym... -ldl
  [...]
  checking for dladdr... yes

See also: https://bugzilla.redhat.com/show_bug.cgi?id=1702761
---
 configure.ac | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/configure.ac b/configure.ac
index 4624bf8..1d3aa7b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -224,11 +224,11 @@ AS_IF([test "x$nbdkit_cv_func_printf_percent_m"
= xyes],
     [Define to 1 if vfprintf supports %m.])])
 
 old_LIBS="$LIBS"
-AC_SEARCH_LIBS([dlopen], [dl dld], [
-        AS_IF([test "x$ac_cv_search_dlopen" != "xnone
required"],
-	    [DL_LIBS="$ac_cv_search_dlopen"], [DL_LIBS=])
+AC_SEARCH_LIBS([dlsym], [dl dld], [
+        AS_IF([test "x$ac_cv_search_dlsym" != "xnone
required"],
+	    [DL_LIBS="$ac_cv_search_dlsym"], [DL_LIBS=])
         AC_SUBST([DL_LIBS])
-    ], [AC_MSG_ERROR([unable to find the dlopen() function])
+    ], [AC_MSG_ERROR([unable to find the dlsym() function])
 ])
 LIBS="$old_LIBS"
 
-- 
2.20.1

On 4/24/19 3:42 PM, Richard W.M. Jones wrote:
> When testing which “dl library” we must use for dl* symbols, autoconf
> runs a test similar to:
> 
>   $ cat conftest.c
>   char dlopen ();
>   int main () { return dlopen (); }
>   $ gcc -o conftest $CFLAGS conftest.c [try various -ldl options here]
> 
> When using ‘CFLAGS="-fsanitize=address"’ this succeeds even if no
dl
> libraries are used at all, since it appears that using this option
> causes dlopen to be included in the final binary implicitly.  This
> causes configure to set DL_LIBS='', but unfortunately dlsym is not
> included implicitly and so linking fails.
> 
> Although I believe this is a bug in GCC 9, as a workaround use dlsym
> as the sentinel function instead.
Not just gcc 9; I reproduced it on gcc 8 in Fedora 29. But I agree that
it is odd that the use of -fsanitize=address is sufficient to pull in a
subset of symbols that you normally only get from the explicit library
link line.  At any rate,
> 
> The new output from ‘./configure CFLAGS="-fsanitize=address"’ is:
> 
>   checking for library containing dlsym... -ldl
>   [...]
>   checking for dladdr... yes
> 
> See also: https://bugzilla.redhat.com/show_bug.cgi?id=1702761
> ---
>  configure.ac | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
this change looks correct. ACK.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org