Libguestfs - Feb 2020 - Re: [nbdkit PATCH v5 4/4] vddk: Drive library loading from libdir parameter.

Unfortunately this series doesn't work with the real VDDK library :-(

I see:

  nbdkit: debug: vddk: config_complete
  nbdkit: debug: dlopen shim prefix set to
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64

Then a little bit later during VDDK initialization it fails with:

  nbdkit: debug: VixDiskLib: Could not load default plugins from
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so:
Cannot open library: libcrypto.so.1.0.2: cannot open shared object file: No such
file or directory.

I was curious if the replacement dlopen() was being called at all, and
it is certainly called from plugins/vddk/vddk.c:

  nbdkit: debug: dlopen:
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.6
flags=2

It's also called from VDDK itself:

  nbdkit: debug: dlopen:
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so
flags=257

but it's not being explicitly called for libcrypto.so because
immediately after the above message I see the error:

  nbdkit: debug: VixDiskLib: Could not load default plugins from
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so:
Cannot open library: libcrypto.so.1.0.2: cannot open shared object file: No such
file or directory.

So I guess libcrypto is actually loaded from a DT_NEEDED entry:

  $ eu-readelf -d
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so
| grep NEEDED
    NEEDED            Shared library: [libpthread.so.0]
    NEEDED            Shared library: [libstdc++.so.6]
    NEEDED            Shared library: [libdl.so.2]
    NEEDED            Shared library: [libcrypto.so.1.0.2]
    NEEDED            Shared library: [libssl.so.1.0.2]
    NEEDED            Shared library: [libexpat.so]
    NEEDED            Shared library: [libz.so.1]
    NEEDED            Shared library: [libvmacore.so]
    NEEDED            Shared library: [libvmomi.so]
    NEEDED            Shared library: [libvim-types.so]
    NEEDED            Shared library: [librt.so.1]
    NEEDED            Shared library: [libgcc_s.so.1]
    NEEDED            Shared library: [libc.so.6]
    NEEDED            Shared library: [ld-linux-x86-64.so.2]

which I suppose doesn't actually go through dlopen ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

On 2/17/20 12:17 PM, Richard W.M. Jones wrote:
> Unfortunately this series doesn't work with the real VDDK library :-(
> 
> I see:
> 
>    nbdkit: debug: vddk: config_complete
>    nbdkit: debug: dlopen shim prefix set to
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64
> 
> Then a little bit later during VDDK initialization it fails with:
> 
>    nbdkit: debug: VixDiskLib: Could not load default plugins from
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so:
Cannot open library: libcrypto.so.1.0.2: cannot open shared object file: No such
file or directory.
> 
> I was curious if the replacement dlopen() was being called at all, and
> it is certainly called from plugins/vddk/vddk.c:
> 
>    nbdkit: debug: dlopen:
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.6
flags=2
> 
> It's also called from VDDK itself:
> 
>    nbdkit: debug: dlopen:
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so
flags=257
That's nice to know...
> 
> but it's not being explicitly called for libcrypto.so because
> immediately after the above message I see the error:
> 
>    nbdkit: debug: VixDiskLib: Could not load default plugins from
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so:
Cannot open library: libcrypto.so.1.0.2: cannot open shared object file: No such
file or directory.
> 
> So I guess libcrypto is actually loaded from a DT_NEEDED entry:
...but that's indeed frustrating.
> 
>    $ eu-readelf -d
/home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so
| grep NEEDED
>      NEEDED            Shared library: [libpthread.so.0]
>      NEEDED            Shared library: [libstdc++.so.6]
>      NEEDED            Shared library: [libdl.so.2]
>      NEEDED            Shared library: [libcrypto.so.1.0.2]
>      NEEDED            Shared library: [libssl.so.1.0.2]
>      NEEDED            Shared library: [libexpat.so]
>      NEEDED            Shared library: [libz.so.1]
>      NEEDED            Shared library: [libvmacore.so]
>      NEEDED            Shared library: [libvmomi.so]
>      NEEDED            Shared library: [libvim-types.so]
>      NEEDED            Shared library: [librt.so.1]
>      NEEDED            Shared library: [libgcc_s.so.1]
>      NEEDED            Shared library: [libc.so.6]
>      NEEDED            Shared library: [ld-linux-x86-64.so.2]
> 
> which I suppose doesn't actually go through dlopen ...
And we were wondering if the dlopen() with an absolute file name would 
help the dynamic linker alter its search path for its dependencies. 
That may have answered our question, but not the way we wanted. :(

la_objsearch() seems like it should catch these DT_NEEDED entries, but 
given the fact that glibc does not yet support DT_AUDIT for 
single-process rather than global auditing, we're stuck with having to 
set LD_AUDIT in the environment (which is no different than having to 
set LD_LIBRARY_PATH in the environment).


Next idea: how hard would it be to re-exec nbdkit?  Here's what I'm 
envisioning:

nbdkit proper exposes some way to let a plugin query argv[] (necessary, 
so that re-exec gets the same arguments).  Re-exec is only safe at a 
point prior to the end of .config_complete, and must be done in a way 
that does not cause infinite recursion on re-exec attempts.

In vddk's .config_complete, if the key reexec= is absent, libdir=... is 
present, and dlopen()/init of libvixDiskLib.so fails, then we grab argv 
from nbdkit and re-exec 'LD_LIBRARY_PATH=libdir... nbdkit vddk 
same_args... reexec=true'.  Conversely, if key reexec= is present, then 
.config_complete calls setenv() to reset LD_LIBRARY_PATH to its former 
state.  .config_complete runs prior to the point where nbdkit fork()s 
the --run child, so the child will see the original environment rather 
than the one rewritten by the re-exec attempt, limiting our temporary 
LD_LIBRARY_PATH pollution to just the nbdkit process.

In the meantime, patch 3 (delaying from the dlopen() of libvixDiskLib.so 
from .load over to .config_complete) is worthwhile to push now, even if 
it doesn't change the status quo (I didn't change it from what you
posted).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

On Mon, Feb 17, 2020 at 01:58:53PM -0600, Eric Blake
wrote:
> In the meantime, patch 3 (delaying from the dlopen() of
> libvixDiskLib.so from .load over to .config_complete) is worthwhile
> to push now, even if it doesn't change the status quo (I didn't
> change it from what you posted).
I'll push "vddk: Delay loading VDDK until config_complete." in
a minute.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org