llvm dev - Dec 2016 - [lld] RFC: Finding shared libraries on OpenBSD

On OpenBSD we still use the "classic" SunOS 4 shared library
versioning scheme where the major and minor number are part of the
library name (and recorded in DT_NEEDED entries).  For example the
shared libc on the OpenBSD-current is named libc.so.89.2.  With this
scheme, linker has to pick the pick the library with the highest major
and minor (within the highest major version); the symbolic links
present on most other ELF-based systems don't exist on OpenBSD.  The
diff below implements this search algorithm.  It follows the same
approach of iterating over files in a directory as the implementation
in the native (ld.bfd derived) toolchain.

This is an RFC as I'm not sure the diff is the best way to implement
this.  For one thing, iterating over files is probably undesirable on
non-OpenBSD systems.  And the searching code should probably be moved
to its own function.


Index: ELF/DriverUtils.cpp
==================================================================---
ELF/DriverUtils.cpp	(revision 290066)
+++ ELF/DriverUtils.cpp	(working copy)
@@ -153,9 +153,34 @@
     return findFromSearchPaths(Name.substr(1));
 
   for (StringRef Dir : Config->SearchPaths) {
-    if (!Config->Static)
+    if (!Config->Static) {
       if (Optional<std::string> S = findFile(Dir, "lib" + Name
+ ".so"))
         return S;
+
+      const StringRef LibName = (Twine("lib") + Name +
".so.").str();
+      int MaxMaj = -1, MaxMin = -1;
+      std::error_code EC;
+      for (fs::directory_iterator LI(Dir, EC), LE;
+	   !EC && LI != LE; LI = LI.increment(EC)) {
+        StringRef FilePath = LI->path();
+	StringRef FileName = path::filename(FilePath);
+	if (!(FileName.startswith(LibName)))
+	  continue;
+	std::pair<StringRef, StringRef> MajMin +	 
FileName.substr(LibName.size()).split('.');
+	int Maj, Min;
+	if (MajMin.first.getAsInteger(10, Maj) || Maj < 0)
+	  continue;
+	if (MajMin.second.getAsInteger(10, Min) || Min < 0)
+	  continue;
+	if (Maj > MaxMaj)
+	  MaxMaj = Maj, MaxMin = Min;
+	if (MaxMaj == Maj && Min > MaxMin)
+	  MaxMin = Min;
+      }
+      if (MaxMaj >= 0)
+	return findFile(Dir, LibName + Twine(MaxMaj) + "." + Twine(MaxMin));
+    }
     if (Optional<std::string> S = findFile(Dir, "lib" + Name +
".a"))
       return S;
   }

Hi Mark,

If we have to do this, or LLD doesn't work on OpenBSD, I think we need to
do this. But can I ask one question? I wonder why OpenBSD systems don't
have symbolic links unlike the other Unix-like systems in the first place.

On Mon, Dec 19, 2016 at 2:27 PM, Mark Kettenis via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On OpenBSD we still use the "classic" SunOS 4 shared library
> versioning scheme where the major and minor number are part of the
> library name (and recorded in DT_NEEDED entries).  For example the
> shared libc on the OpenBSD-current is named libc.so.89.2.  With this
> scheme, linker has to pick the pick the library with the highest major
> and minor (within the highest major version); the symbolic links
> present on most other ELF-based systems don't exist on OpenBSD.  The
> diff below implements this search algorithm.  It follows the same
> approach of iterating over files in a directory as the implementation
> in the native (ld.bfd derived) toolchain.
>
> This is an RFC as I'm not sure the diff is the best way to implement
> this.  For one thing, iterating over files is probably undesirable on
> non-OpenBSD systems.  And the searching code should probably be moved
> to its own function.
>
>
> Index: ELF/DriverUtils.cpp
> ==================================================================> ---
ELF/DriverUtils.cpp (revision 290066)
> +++ ELF/DriverUtils.cpp (working copy)
> @@ -153,9 +153,34 @@
>      return findFromSearchPaths(Name.substr(1));
>
>    for (StringRef Dir : Config->SearchPaths) {
> -    if (!Config->Static)
> +    if (!Config->Static) {
>        if (Optional<std::string> S = findFile(Dir, "lib" +
Name + ".so"))
>          return S;
> +
> +      const StringRef LibName = (Twine("lib") + Name +
".so.").str();
> +      int MaxMaj = -1, MaxMin = -1;
> +      std::error_code EC;
> +      for (fs::directory_iterator LI(Dir, EC), LE;
> +          !EC && LI != LE; LI = LI.increment(EC)) {
> +        StringRef FilePath = LI->path();
> +       StringRef FileName = path::filename(FilePath);
> +       if (!(FileName.startswith(LibName)))
> +         continue;
> +       std::pair<StringRef, StringRef> MajMin > +        
FileName.substr(LibName.size()).split('.');
> +       int Maj, Min;
> +       if (MajMin.first.getAsInteger(10, Maj) || Maj < 0)
> +         continue;
> +       if (MajMin.second.getAsInteger(10, Min) || Min < 0)
> +         continue;
> +       if (Maj > MaxMaj)
> +         MaxMaj = Maj, MaxMin = Min;
> +       if (MaxMaj == Maj && Min > MaxMin)
> +         MaxMin = Min;
> +      }
> +      if (MaxMaj >= 0)
> +       return findFile(Dir, LibName + Twine(MaxMaj) + "." +
> Twine(MaxMin));
> +    }
>      if (Optional<std::string> S = findFile(Dir, "lib" +
Name + ".a"))
>        return S;
>    }
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161219/89503514/attachment.html>

> From: Rui Ueyama <ruiu at google.com>
> Date: Mon, 19 Dec 2016 20:07:39 -0600
> 
> Hi Mark,
> 
> If we have to do this, or LLD doesn't work on OpenBSD, I think we need
to
> do this. But can I ask one question? I wonder why OpenBSD systems don't
> have symbolic links unlike the other Unix-like systems in the first place.
I suppose the main reason is to minimize the differences between ELF
and a.out systems.  The transition was done on an architecture by
architecture basis and took several years to complete.

Whenever the subject came up in the past, there were concerns about
managing the symlinks.  But the real problem is that the system
libraries get built without an DT_SONAME.  That results in the wrong
name being recorded in DT_NEEDED entries, breaking the versioning
scheme.

Anyway, I'll bring it up on the OpenBSD side, and see if we can work
something out.  It's been a while since the issue came up last.

Thanks,

Mark
> On Mon, Dec 19, 2016 at 2:27 PM, Mark Kettenis via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> 
> > On OpenBSD we still use the "classic" SunOS 4 shared library
> > versioning scheme where the major and minor number are part of the
> > library name (and recorded in DT_NEEDED entries).  For example the
> > shared libc on the OpenBSD-current is named libc.so.89.2.  With this
> > scheme, linker has to pick the pick the library with the highest major
> > and minor (within the highest major version); the symbolic links
> > present on most other ELF-based systems don't exist on OpenBSD. 
The
> > diff below implements this search algorithm.  It follows the same
> > approach of iterating over files in a directory as the implementation
> > in the native (ld.bfd derived) toolchain.
> >
> > This is an RFC as I'm not sure the diff is the best way to
implement
> > this.  For one thing, iterating over files is probably undesirable on
> > non-OpenBSD systems.  And the searching code should probably be moved
> > to its own function.
> >
> >
> > Index: ELF/DriverUtils.cpp
> > ==================================================================>
> --- ELF/DriverUtils.cpp (revision 290066)
> > +++ ELF/DriverUtils.cpp (working copy)
> > @@ -153,9 +153,34 @@
> >      return findFromSearchPaths(Name.substr(1));
> >
> >    for (StringRef Dir : Config->SearchPaths) {
> > -    if (!Config->Static)
> > +    if (!Config->Static) {
> >        if (Optional<std::string> S = findFile(Dir,
"lib" + Name + ".so"))
> >          return S;
> > +
> > +      const StringRef LibName = (Twine("lib") + Name +
".so.").str();
> > +      int MaxMaj = -1, MaxMin = -1;
> > +      std::error_code EC;
> > +      for (fs::directory_iterator LI(Dir, EC), LE;
> > +          !EC && LI != LE; LI = LI.increment(EC)) {
> > +        StringRef FilePath = LI->path();
> > +       StringRef FileName = path::filename(FilePath);
> > +       if (!(FileName.startswith(LibName)))
> > +         continue;
> > +       std::pair<StringRef, StringRef> MajMin > > +      
FileName.substr(LibName.size()).split('.');
> > +       int Maj, Min;
> > +       if (MajMin.first.getAsInteger(10, Maj) || Maj < 0)
> > +         continue;
> > +       if (MajMin.second.getAsInteger(10, Min) || Min < 0)
> > +         continue;
> > +       if (Maj > MaxMaj)
> > +         MaxMaj = Maj, MaxMin = Min;
> > +       if (MaxMaj == Maj && Min > MaxMin)
> > +         MaxMin = Min;
> > +      }
> > +      if (MaxMaj >= 0)
> > +       return findFile(Dir, LibName + Twine(MaxMaj) + "." +
> > Twine(MaxMin));
> > +    }
> >      if (Optional<std::string> S = findFile(Dir, "lib"
+ Name + ".a"))
> >        return S;
> >    }
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >