Libguestfs - Feb 2020 - Re: alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

On 2/14/20 3:29 PM, Eric Blake wrote:
> On 2/14/20 1:02 PM, Eric Blake wrote:
> 
>> Writing my own dlopen() wrapper directly in nbdkit seems like a 
>> non-starter (my override has to come from a shared library before it 
>> can replace the shared version that would be imported from -ldl, at 
>> least for all subsequent shared library loads that want to bind to the 
>> override).
> 
> Maybe I spoke too soon. I've tried another approach that looks like it 
> will do what I want: put my shim dlopen() in a shared library, but link 
> nbdkit against that shared library PRIOR to -ldl (so that name lookup 
> for dlopen resolves there first).  The shim library in turn depends on 
> -ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get to the
real
> dlopen.  And by linking it directly into nbdkit, rather than into the 
> nbdkit-vddk-plugin.so that gets loaded later, the first bound dlopen() 
> in use for all subsequent loads is from my shim.  It's still a bit less
> clean than I'd like (it requires tighter coupling between nbdkit and 
> nbdkit-vddk-plugin.so than what used to exist), but the fact that it 
> works without dlmopen() or LD_LIBRARY_PATH is in its favor.  I'm now 
> polishing up the experiment, and will post the patch when it's ready.
Progress report: I've posted a v4 series that relies on a shared library 
in the main executable; but I'm still trying to see if I can further 
reduce things (maybe with -rdynamic) so that the main binary itself 
provides the dlopen() override without needing an auxiliary shared library.
https://www.redhat.com/archives/libguestfs/2020-February/msg00162.html
> But after spending more than an hour playing with la_objsearch() and
reading 'man rtld-audit', it looks like an audit library cannot be
triggered in glibc except by listing it in LD_AUDIT in the environment during
exec - which is back to the same problem we have with needing LD_LIBRARY_PATH in
the environment.  Furthermore, although I know that glibc's audit interface
is slightly different from the Solaris version it copied from, the Solaris
documentation states that an audit library has some rather tough restrictions
(including that using 'malloc' is unsafe,
https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-3.html#scrolltoc
"Some system interfaces assume that the interfaces are the only instance of
their implementation within a process. Examples of such implementations are
signals and malloc(3C). Audit libraries should avoid using such interfaces, as
doing so can inadvertently alter the behavior of the application.").  But
Solaris also stated that a library could serve as an audit entry point without
LD_AUDIT if it is registered locally, via -Wl,-paudit.so.1 when creating the
shared library
(https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-18.html#scrolltoc);
it doesn't seem that this functionality exists with glibc
(/usr/lib64/libaudit.so on Linux has nothing to do with rtld-audit).
I'm just now noticing that 'man ld' reports that you may pass
'--audit
LIB' during linking to add a DT_DEPAUDIT dependency on a library 
implementing the audit interface, which sounds like it might be an 
alternative to LD_AUDIT for getting a library with la_objsearch() to 
actually do something (but doesn't obviate the need for la_obsearch() to 
be in a separate library, rather than part of the main executable, 
unless a library can be reused as its own audit library...).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

* Eric Blake:
> I'm just now noticing that 'man ld' reports that you may pass
'--audit
> LIB' during linking to add a DT_DEPAUDIT dependency on a library
> implementing the audit interface, which sounds like it might be an
> alternative to LD_AUDIT for getting a library with la_objsearch() to
> actually do something (but doesn't obviate the need for la_obsearch()
> to be in a separate library, rather than part of the main executable,
> unless a library can be reused as its own audit library...).
DT_AUDIT support has yet to be implemented in glibc:

  <https://sourceware.org/bugzilla/show_bug.cgi?id=24943>
  <https://sourceware.org/ml/libc-alpha/2019-08/msg00705.html>

If you go on record saying that you need this, maybe someone will review
the patch.  Sorry. 8-(

Florian

On 2/17/20 9:12 AM, Florian Weimer wrote:
> * Eric Blake:
> 
>> I'm just now noticing that 'man ld' reports that you may
pass '--audit
>> LIB' during linking to add a DT_DEPAUDIT dependency on a library
>> implementing the audit interface, which sounds like it might be an
>> alternative to LD_AUDIT for getting a library with la_objsearch() to
>> actually do something (but doesn't obviate the need for
la_obsearch()
>> to be in a separate library, rather than part of the main executable,
>> unless a library can be reused as its own audit library...).
> 
> DT_AUDIT support has yet to be implemented in glibc:
> 
>    <https://sourceware.org/bugzilla/show_bug.cgi?id=24943>
>    <https://sourceware.org/ml/libc-alpha/2019-08/msg00705.html>
> 
> If you go on record saying that you need this, maybe someone will review
> the patch.  Sorry. 8-(
Another followup: nbdkit-vddk-plugin.so is now using a re-exec setup 
[1], for several reasons:
1. all our other ideas tried so far (such as a dlopen() interposition in 
the main nbdkit binary) touched more files and required more exported 
APIs; we managed to get re-exec working with changes limited to just the 
plugin (other than a minor change to nbdkit to quit messing with argv[] 
so that /proc/self/cmdline would stay stable - but that did not require 
a new API).
2. it turns out that overriding dlopen() was insufficient to work around 
VDDK's setup [2].  It _did_ solve the initial dlopen() performed by 
VDDK, but that library in turn had DT_NEEDED entries for bare library 
names, which dlopen() does NOT affect but which la_objsearch() should.
3. for nbdkit, we want to minimize the number of binary files shipped; 
the re-exec solution works with just the nbdkit binary and the 
nbdkit-vddk-plugin.so.  Any solution that requires a third file to be 
shipped (be that a shared library providing dlopen, or a LD_AUDIT 
library, or otherwise) is less palatable than the 2-binary solution that 
our re-exec solution provides.

[1] https://github.com/libguestfs/nbdkit/commit/0c7ac4e655b
[2] https://www.redhat.com/archives/libguestfs/2020-February/msg00184.html

So with that said, here's a question I just thought of:

If your patch for glibc support for DT_AUDIT is incorporated, is it 
possible to mark a shared library as its own audit library via DT_AUDIT? 
  That is, if nbdkit-vddk-plugin.so can provide entry points for _both_ 
the nbdkit interface (which satisfies dlopen() from the nbdkit binary) 
and la_version/la_objsearch() (which satisfy the requirements for use 
from the audit code in ld.so), _and_ during the compilation of 
nbdkit-vddk-plugin.so, we marked the library as its own DT_AUDIT entry, 
would the mere act of dlopen("nbdkit-vddk-plugin.so") from nbdkit be 
sufficient to trigger audit actions such as la_objsearch() on all 
subsequent shared loads (whether by dlopen() or DT_NEEDED) performed by 
nbdkit-vddk-plugin.so and its descendant loaded libraries?  Because if 
so, we would have a use case where a single binary, set up to act as its 
own audit library, might be sufficient to hook the shared object search 
path without needing any of environment variable modification, a process 
re-exec, or a third shipping binary - in which case that would indeed be 
a nicer solution than the current re-exec solution we committed today 
(of course, nbdkit would not be able to rely on that solution except on 
systems with new enough glibc to support DT_AUDIT).

I guess even without DT_AUDIT support, I could at least answer the 
question of whether a single .so can be used to satisfy both dlopen() 
and LD_AUDIT interfaces at once by setting LD_AUDIT (where the only 
remaining gap is figuring out when glibc can let DT_AUDIT have the same 
effect).  During my earlier attempts to get a working dlopen() override, 
I didn't consider any solution that required setting LD_AUDIT, but now 
that I proved dlopen() override alone was not enough for the case at 
hand, having to re-exec to set LD_AUDIT is no worse than having to 
re-exec to set LD_LIBRARY_PATH as a fallback to systems where glibc does 
not support DT_AUDIT.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org