pkg kde talk - Mar 2011 - Handling BIC-without-SONAME-bump in KDE SC libraries

Hello,

I personally can''t comment on how much upstream improved after the last
discussion (iirc, on kde-core-devel). However, it would be naive to think that 
we will never face BIC-without-SONAME-bump problems in the future. Solving 
them all upstream would be great but it''s simply too optimistic
scenario due
to many factors.

Generally I can think of two ways of dealing with this. The first is our 
policy at the moment.

1. [ Package level ]. Rename the library package and add Breaks/Replaces 
against the previous versions, keep library SONAME (and filenames) unchanged.

Pros:

a) binary compatible with upstream and other distributions (at least in 
theory);

b) ensures users have no old BIC libs (and apps linked to them) around. If an 
app ends up directly or indirectly linked to the old and new library with the 
same symbols, bad things may happen (e.g. recent libpng case and see below);

c) does not require patching of upstream source.

Cons:

a) apt unfriendly because two library packages cannot be co-installed. Library 
packages are typically deep in the dependency chain so apt may require to 
force removal/installation in some cases;

b) makes partial upgrade to new KDE releases very painful. I.e. it renders 
some 3rd party applications uninstallable due to conflicting library packages;

2. [ Library level ]. Change library SONAME (e.g. add debN suffix, specifics 
to be discussed) and rename the package. No Breaks/Replaces needed as there 
are no conflicting files.

Pros:

a) apt friendly as it''s a recommended way to deal with SONAME changes;

b) new KDE releases should no longer break 3rd party applications (at least at 
the package management level);

Cons:

a) custom SONAMEs and library filenames will be binary incompatible with the 
rest of the world;

b) if conflicting libraries are loaded in the same app memory space (unlikely 
though), it might lead to crashes at runtime;

c) crashes might occur due incompatibilities in the runtime client/server 
protocols (if there are ones);

d) requires patching of upstream source (i.e. altering SOVERSION in the 
CMakeLists.txt);

So what I am proposing is the switch to 2). I do believe that user experience 
is more important than SONAME compatibility with the rest of the world (which 
is theoretical anyway, at least in the KDE land). IMHO, other cons are not a 
big deal either as KDE is not libpng and runtime conflicts are more unlikely 
than likely.

So what do you think and which way do you prefer? Maybe we could think of 
something better?

-- 
Modestas Vainius <modax at debian.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.alioth.debian.org/pipermail/pkg-kde-talk/attachments/20110315/da2c08de/attachment.pgp>

Hello,

On antradienis 15 Kovas 2011 00:23:14 Modestas Vainius wrote:
> 2. [ Library level ]. Change library SONAME (e.g. add debN suffix,
> specifics to be discussed) and rename the package. No Breaks/Replaces
> needed as there are no conflicting files.
As an amendment to this plan, we could start versioning the symbols of BIC-
prone libraries (i.e. basically everything not kde*libs). This would help us 
avoid the following con:
> b) if conflicting libraries are loaded in the same app memory space
> (unlikely though), it might lead to crashes at runtime;
It''s still an open question how much symbol versioning would affect
people
building from source (e.g. kde developers or kdesrc-build users).
What''s more,
it''s still unknown how much effort (i.e. upstream code patching) this
would
need.

If we were to move forward with this plan, we need to agree about naming of 
custom SOVERSION and symbol versions. Please respond to this mail even if it 
was a short yes/no answer with small remarks. My proposal would be:

SOVERSION: ${upsteamSOVERSION}debN (where N is a number (starting from 1) 
bumped after each BIC-without-SONAME-bump)

symbol versions: DEB_${upsteamSOVERSION}[_N] where N is the same as above if 
there was at least one BIC-without-SONAME-bump. Otherwise _N part would be 
omitted. Whereas symbol versioning could be avoided if there was no BIC-
without-SONAME-bump, is an open technical question (that would be somewhat 
more binary compatible with the rest of world, at least temporary).


-- 
Modestas Vainius <modax at debian.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.alioth.debian.org/pipermail/pkg-kde-talk/attachments/20110316/d0a1a757/attachment.pgp>

On Tuesday 15 March 2011 19:44:28 Modestas Vainius
wrote:
> Hello,
> 
> On antradienis 15 Kovas 2011 00:23:14 Modestas Vainius wrote:
> > 2. [ Library level ]. Change library SONAME (e.g. add debN suffix,
> > specifics to be discussed) and rename the package. No Breaks/Replaces
> > needed as there are no conflicting files.
> 
> As an amendment to this plan, we could start versioning the symbols of BIC-
> prone libraries (i.e. basically everything not kde*libs). This would help
> us
> 
> avoid the following con:
> > b) if conflicting libraries are loaded in the same app memory space
> > (unlikely though), it might lead to crashes at runtime;
> 
> It''s still an open question how much symbol versioning would
affect people
> building from source (e.g. kde developers or kdesrc-build users).
What''s
> more, it''s still unknown how much effort (i.e. upstream code
patching)
> this would need.
> 
> If we were to move forward with this plan, we need to agree about naming of
> custom SOVERSION and symbol versions. Please respond to this mail even if
> it was a short yes/no answer with small remarks. My proposal would be:
> 
> SOVERSION: ${upsteamSOVERSION}debN (where N is a number (starting from 1)
> bumped after each BIC-without-SONAME-bump)
> 
> symbol versions: DEB_${upsteamSOVERSION}[_N] where N is the same as above
> if there was at least one BIC-without-SONAME-bump. Otherwise _N part would
> be omitted. Whereas symbol versioning could be avoided if there was no
> BIC- without-SONAME-bump, is an open technical question (that would be
> somewhat more binary compatible with the rest of world, at least
> temporary).
As for me, sounds good.

I also want to offer myself to put this on a webpage/wiki once it gets 
completely decided (or even before it''s needed somehow).

Regards, Lisandro.

-- 
Hiroshima ''45,
Chernobyl ''86,
Windows   ''95.
 Grafitti en Niceto Vega 5940,
 Buenos Aires. De una foto de
 Mario Gallo.

Lisandro Dami?n Nicanor P?rez Meyer
http://perezmeyer.com.ar/
http://perezmeyer.blogspot.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.alioth.debian.org/pipermail/pkg-kde-talk/attachments/20110315/6027bb6d/attachment.pgp>

Hello,

as SONAME changes + symbol versioning seems to be the most viable option, I 
did some experiments on what end effect it might have and how all this would
look like.

Glossary
--------

unstable-ABI library - a library which is prone to break BC without SONAME 
change.

Undefined (UNDEF) symbol - a plain symbol reference. It will be resolved to 
the implementation/definition in the external shared library by dynamic 
linker.

Conclusions
-----------

1) ABI-unstable libraries should get all symbols versioned from the start. 
That''s because undefined unversioned symbols may bind to any symbol of
the
same name in any library which dynamic linker (recursively) loaded [1]:

 | In case only the object file with the reference does not use
 | versioning but the object with the definition does, then the reference
 | only matches the base definition. [ ...] If there is no symbol definition
 | with such an version index and there is exactly one version for which this
 | symbol is defined, then this version is accepted

It does not matter if a real symbol in the library is versioned or not. The 
first same-name real symbol seen by dynamic linker will be bound to 
the unversioned UNDEF. It means that we basically have no control over this 
unless we version the symbols.

2) Unlike unversioned (Base) symbols, versioned UNDEF symbols have version 
*and* originating library information assigned to them. According to 
[1]:

 | The last case is if the object with the references uses symbol
 | versions but the object with the definitions has none.  In this case a
 | matching symbol is accepted unless the object''s name matches the one
 | in the Elfxx_Verneed entry.  In the latter case this is a fatal error.

However, I determined that this is not the case since middle 1999 [2]. This 
situation is not a fatal error but merely a warning [3]. However, I''m
pretty
sure that it''s a bug in the dynamic linker which may get fixed one day.
Respective assert() is in place [4] but it does not fire in release builds.

3) Versioned symbols cannot be easily "moved" from one library to
another down
the library dependency tree without breakage (as it was done e.g. with 
"libkutils4 -> libkcmutils4 libkemoticons4 libkidletime4
libkprintutils4"
split). But this should be a non-issue for unstable-ABI libraries.

Having in mind the points 1 and 2 above, all symbols in the unstable-ABI 
libraries should get a unique-per-ABI version. As soon as upstream breaks BC 
without bumping SOVERSION, we bump SONAME and change a package name. However, 
initially, we keep original upstream SONAME. This strategy allows to keep some 
compatibility with the rest of the world as long as upstream does not break BC 
without bumping SOVERSION so we do not have to bump SONAME ourselves. In 
particular:

 a) external binaries (which supposedly have unversioned UNDEF symbols as 
pushed by upstream, see point 1) will work with unstable-ABI libraries as 
packaged by Debian;

 b) binaries with versioned UNDEF symbols (i.e. linked against our unstable-
ABI libraries) will work with unstable-ABI libraries as built from upstream 
sources (kdesrc-build) or packaged by other distros which do not use  
incompatible symbol versioning. That''s because of point 2 and as long
as the
linker bug is not fixed.

So how to handle BIC?
---------------------

As said in previous mails, we need to choose a custom string for our 
SONAMEs (that is either SOVERSION or OUTPUT_NAME suffix, whatever toolchain 
eats better) and symbol versions. Let it be "abiN" (where N is an
integer) for
SONAME bumps and ABI_N for symbol versions respectively (with _0 for unchanged 
SOVERSION).

So in 4.6, we add symbol versions ABI_0 to all unstable-ABI libraries which 
kept BC throughout 4.4 -> 4.6 cycle. Otherwise, if libs broke BC, we rename 
their packages, bump their SONAMEs by adding "abi1" suffix and set
their
symbol versions to ABI_1.

As 4.6 is going to be a transitional release from unversioned to versioned 
symbols, we might consider adding "Breaks" against old libs (<<
4:4.6) to all
future "abi1" packages. That''s to avoid weird crashes if both
old unversioned
and new versioned BIC libraries end up getting loaded in the same process 
space. While it''s possible, it''s also highly unlikely because
unstable-ABI
libs are not very popular (unlike e.g. libjpeg). So let''s not do this
"Breaks"
stuff at first and see what happens.

Implementation details
----------------------

In order to minimize patching, I propose the following scheme. All unstable-
ABI library packages would be marked with a custom "X-Custom-ABI: N"
(or
something like that) debian/control field. Then we would need to implement a 
cmake script which would parse debian/control, bump SONAME and add 
appropriate -Wl,--version-script,script-name to the linker command line on-
the-fly. This script would be shipped in pkg-kde-tools to be INCLUDEd from the 
bottom of /CMakeLists.txt of the each source package that needs this 
capability. That would effectively limit us to one patch per source package 
while the rest would be conveniently "configured" in debian/control.

[1] http://www.akkadia.org/drepper/symbol-versioning
[2]
http://sourceware.org/git/?p=glibc.git;a=commit;h=9a8fcca0b33c26759134a545ac45251df53418a3
[3] <apppath>: <libpath>: no version information available (required
by <apppath>)
[4]
http://sourceware.org/git/?p=glibc.git;a=blob;f=elf/dl-lookup.c;hb=glibc-2.13#l168

-- 
Modestas Vainius <modax at debian.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.alioth.debian.org/pipermail/pkg-kde-talk/attachments/20110325/1e6f852b/attachment.pgp>