R devel - May 1998 - dependencies in .so files

Thomas Lumley writes:
 > 
 > 
 > I've been trying to get Matt Calder's S compiler to work in R
using R
 > COMPILE and R SHLIB so it will be fairly platform-independent.  
 > 
 > I'm sure someone has claimed in the past that it is possible/easy to
 > dyn.load() two dynamic libraries and have one call functions in the other.
 > I can't get it to work (Red Hat 5.0/R0.61.1ish). It doesn't seem
to matter
 > what order I load them in, they can't find each other.
 > 
 > Any ideas?
 
 >From the dlopen man page:
 void *dlopen (const char *filename, int flag);
 ................................................
 RTLD_GLOBAL may be or'ed with flag
 in which case the external symbols defined in the library will be
 made available to subsequently loaded libraries.
 
 So, I try: 
 1) I modify src/unix/dynload.c changing the flag of dlopen
    from RTLD_LAZY to (RTLD_LAZY | RTLD_GLOBAL)
 2) build two simple shared libraries the second calling a routine 
    in the first.
 ---------------------------a.c-------------------------------
 #include <stdio.h>
 
 void a(int *i) {
    printf("now in a.so; incrementing i\n");
    *i=*i+1;
 }
 
 --------------------------b.c--------------------------------
 #include <stdio.h>
 
 extern void a(int *i);
 
 void b(int *i) {
    printf("in b.so; calling function a in a.so; i=%d\n",*i);
    a(i);
    printf("in b.so; now i=%d\n",*i);
 }
 
 -------------------------Makefile------------------------------
 all: a.so b.so
 
 a.o: a.c
 	gcc -fPIC -c a.c
 	
 a.so: a.o
 	gcc -shared -o a.so a.o
 	
 b.o: b.c
 	gcc -fPIC -c b.c
 	
 b.so: b.o
 	gcc -shared -o b.so b.o	
 	
 -------------------------------------------------------------------------
 
 Things seems ok.
 
 -------------------------------------------------------------------------
 sirio[~/tmp/prova]% R
 
 R : Copyright 1998, Robert Gentleman and Ross Ihaka
 Version 0.61.3  (May 3, 1998)
 
 R is free software and comes with ABSOLUTELY NO WARRANTY.
 You are welcome to redistribute it under certain conditions.
 Type    "license()" for details.
 
 Type    "demo()" for some demos, "help()" for on-line help,
or
         "help.start()" for a HTML browser interface to help.
 
 
 > dyn.load("a.so")
 > dyn.load("b.so")
 > y <- .C("b",as.integer(0))
 in b.so; calling function a in a.so; i=0
 now in a.so; incrementing i
 in b.so; now i=1
 > y
 [[1]]
 [1] 1           
 ----------------------------------------------------------------------------
 
 Before introducing the RTLD_GLOBAL flag:
 ----------------------------------------------------------------------------
 sirio[~/tmp/prova]% R
 
 R : Copyright 1998, Robert Gentleman and Ross Ihaka
 Version 0.61.3  (May 3, 1998)
 
 R is free software and comes with ABSOLUTELY NO WARRANTY.
 You are welcome to redistribute it under certain conditions.
 Type    "license()" for details.
 
 Type    "demo()" for some demos, "help()" for on-line help,
or
         "help.start()" for a HTML browser interface to help.
 
 > dyn.load("a.so")
 > dyn.load("b.so")
 > y <- .C("b",as.integer(0))
 in b.so; calling function a in a.so; i=0
 /usr/local/src/R-0.61.3/bin/R.binary: can't resolve symbol 'a'
 sirio[~/tmp/prova]%                                             
 
 ----------------------------------------------------------------------------
 I am using R-0.61.3 under Linux 2.30.
 
 guido masarotto
 

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To:
r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

> Luke Tierney writes:
 > 
 > guido@sirio.stat.unipd.it wrote:
 > >  So, I try: 
 > >  1) I modify src/unix/dynload.c changing the flag of dlopen
 > >     from RTLD_LAZY to (RTLD_LAZY | RTLD_GLOBAL)
 > 
 > I'm not sure this is properly and reliably supported on all UNIX
 > platforms. It doesn't seem to be in the standard Windows and Mac
 > compilers.  That is, MS VC++ and Borland C++ don't seem to let you
 > creade DLL's with undefined symbols. Watcom does seem to allow it, but
 > I'm not certain how reliable it is there.
 If I understand, under Windows , you have to follow your strategy B (recursive 
 shared libraries). 
 
 > 
 > Also you need to consider what happens at runtime if a function is
 > called and finds an unresolved symbol. On some systems a signal might
 > be sent that can be caught (HP-UX sends SIGABRT, but I'm not sure I
 > would want to catch all those), but this will need to be checked out
 > on every different system. Having any problems reported at load time
 > seems like a safer solution -- I have gone the other way and switched
 > to using RTLD_NOW for this reason (and because that seems the least
 > common denominator accross UNIX/Win32/Mac).
 I agree. This point is however indipendent from  linking a
 shared library which needs another shared library. This has
 to do with the RTLD_LAZY flag not with the RTLD_GLOBAL one. 
 In the example I used the RTLD_LAZY flag only because the current version of
 R uses it.
 The use of RTLD_NOW will also make debugging easier.
 An example was the missing routines
 problem in the locfit package (a couple of messages a month ago).
 I found out that the shared library misses a couple of routines
 compiling the package under Windows. But, under Unix, no 
 errors in building the library.
 But, if you try to use the R function which calls
 a C function with an unresolved routine you get a segmentation fault
 (or at least, I got it).
 
 > The alternative is to use the OS's ld or equivalent to create a
 > library that knows what it needs to call at link time by telling ld
 > what other shared libraries it needs; in this case
 > 
 > >  	
 > >  a.so: a.o
 > >  	gcc -shared -o a.so a.o
 > >  	
 > >  b.o: b.c
 > >  	gcc -fPIC -c b.c
 > >  	
 > >  b.so: b.o
 > >  	gcc -shared -o b.so b.o	
 > >  	
 > 
 > would change to
 > 
 > ...
 > b.so: b.o
 > 	gcc -shared -o b.so b.o	a.so
 > 
 > Mutually recursive libraries can be handled this way on most UNIX
 > systems I think; for Win32 there are some standard tricks involving
 > .def and .lib files (that I can't remember in detail but I think I
 > found them once on MS's web site) for making DLL's that are
mutually
 > recursive. I haven't looked into mutual recursion on the Mac.
 > 
 Under Linux, the problem with this approach is that the dynamic
 linker (ld.so) must find a.so at runtime. I have just tried
 on the previous example but I can't load the so builded b.so in R.
 guido m.


-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To:
r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

This modified version of b.c in Guido's example works for me (R-0.61.3,
RedHat Linux 5.0) without modifying the R source.  R keeps an internal
table of symbols which is accessible via the R_FindSymbol function
(defined in src/unix/dynload.c), so you can use this to return a
pointer to a function in a previously loaded library.

Pardon my C if I've done something stupid here.
Martyn.

--------------------------b.c--------------------------------
#include <stdio.h>

typedef int (*DL_FUNC) ();

DL_FUNC R_FindSymbol(char *);

void b(int *i) {
    char afun[] = "a";
    DL_FUNC fun;

    printf("in b.so; calling function a in a.so; i=%d\n",*i);
    fun = R_FindSymbol(afun);
    fun(i);
    printf("in b.so; now i=%d\n",*i);
}
---------------------------------------------------------------
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To:
r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

Martin Maechler writes:
> 
> >>>>> "Guido" == Guido Masarotto
<guido@sirio.stat.unipd.it> writes:
> 
>     >> From the dlopen man page:
>     Guido> void *dlopen (const char *filename, int flag);
>     Guido> ................................................
>     Guido> RTLD_GLOBAL may be or'ed with flag
>     Guido> in which case the external symbols defined in the library
will be
>     Guido> made available to subsequently loaded libraries.
>  
>     Guido> So, I try: 
>     Guido> 1) I modify src/unix/dynload.c changing the flag of dlopen
>     Guido> from RTLD_LAZY to (RTLD_LAZY | RTLD_GLOBAL)
> 
> R-core: 
> 	Shall we do this, for 0.62?  -- or even try to have "configure"
find out
> 	if  RTLD_GLOBAL  is supported on the given system?
If I can give suggestions:
1) Independently from the RTLD_GLOBAL flag, I will change RTLD_LAZY to
   RTLD_NOW. As Luke T. summarized yesterday in this way a shared library
   which cannot resolve all its symbols (in R.binary or in other libraries)
   cannot be load. This give three   advantages:
   a) At least under Linux, if I call a dynamic linked routine which
      cannot resolve its dependencies, I get a segmentation fault and
      R exit. Of course, we can try to intercept this problem but
      as also Luke noted, it is difficult to be sure that all the systems 
      will trigs an  appropriate signal; in addition
   b) debugging of a shared library becomes simple; at least, we are
      sure that all the functions in the library can be called.
   c) This correspond to the standard for not Unix machine; I am
      sure for Windows but, if I remember, Luke said also for the Mac.
   Perhaps, if a library cannot be loaded, the corresponding dlerror()
   string should be used for giving to the user some information
   on why.
2) Coming to RTDL_GLOBAL; here the question is the one raised by
   Luke: portability. I suppose that the solution suggested by
   Luke (use "recursive" shared libraries)  is more portable than
mine.
   After a first failure, I was   able to run my "silly" example
setting
   the LD_LIBRARY_PATH shell variable,i.e., instructing ld.so about where 
   to search.
   
> Guido:  Have you ever looked at 0.62  (and porting it to Win32) ?
> 	(get it from <CRAN>/src/devel/R-snapshot.tar.gz)
I had no time. I downloaded a snapshot next week but I didn't make
any trial. Is the unix/system.c changed a lot? I have just looked
to it now and it seems quite similar to the 0.61.x one.

To all the R people: have a nice day.
guido m.
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To:
r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._