Xen users - Sep 2005 - Debian: Move /lib/tls out of the way makes a lot of binaries to break

Hi list,

in order to test a bit around with XEN 2.0.7 I''ve made a domU out of my
existing workstation installation. That''s a mixture with most packages
from testing and some hand-selected packages from unstable. If I move
/lib/tls out of the way (eg. to /lib/tls.disabled) I get the following
error when running a lot of binaries (like ls, sshd, mysqld and so on):
> ls: relocation error: /lib/libpthread.so.0: symbol errno, version
> GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
Running ldconfig does not help (I was very happy to see mv still working
;)). Then I''ve set up a second Debian (this time sarge) domU (using
debootstrap) to do a quick test and in this domU I can move /lib/tls
away without problems! I''ve not tried to update this domU to testing,
but this will be the next test.
I''ve searched the web and news using google but wasn''t able to
find
something matching, so maybe someone here has seen something like this
in the past.

Thanks in advance!

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 9/14/05, Markus Schuster <ma.schuster@gmx.de>
wrote:
> in order to test a bit around with XEN 2.0.7 I''ve made a domU out
of my
> existing workstation installation. That''s a mixture with most
packages
> from testing and some hand-selected packages from unstable. If I move
> /lib/tls out of the way (eg. to /lib/tls.disabled) I get the following
I''ve filed a wishlist bug at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=328183 to include the
Glibc patch [1] some how in Debian proper.

[1] http://wiki.xensource.com/xenwiki/XenSpecificGlibc

-- 
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nicholas Lee schrieb:
> I''ve filed a wishlist bug at
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=328183 to include the
> Glibc patch [1] some how in Debian proper.
Thanks for your reply!
I will try to integrate the patch into the libc6 source package and
build my own libc6 package. I hope the diff applies cleanly to the
Debian source as I don''t really understand assembler :)

Greetins,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

At 05:58 PM 9/13/2005, you wrote:
>On 9/14/05, Markus Schuster <ma.schuster@gmx.de> wrote:
> > in order to test a bit around with XEN 2.0.7 I''ve made a domU
out of my
> > existing workstation installation. That''s a mixture with most
packages
> > from testing and some hand-selected packages from unstable. If I move
> > /lib/tls out of the way (eg. to /lib/tls.disabled) I get the following
>
>I''ve filed a wishlist bug at
>http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=328183 to include the
>Glibc patch [1] some how in Debian proper.
>
>[1] http://wiki.xensource.com/xenwiki/XenSpecificGlibc
I''m a bit rusty with assembly language, but Debian''s current
glibc-2.3.5-6
seems to have included this patch already, if -DNO_TLS_DIRECT_SEG_REFS is 
specified with gcc.  I''m compiling libc6 on a spare server right now to
test it.  (It''s going REALLY slow as it is a slow server - ATHLON 800.)

Will report back in a day or two.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

At 12:52 PM 9/17/2005, you wrote:
>At 05:58 PM 9/13/2005, you wrote:
>
>>On 9/14/05, Markus Schuster <ma.schuster@gmx.de> wrote:
>> > in order to test a bit around with XEN 2.0.7 I''ve made a
domU out of my
>> > existing workstation installation. That''s a mixture with
most packages
>> > from testing and some hand-selected packages from unstable. If I
move
>> > /lib/tls out of the way (eg. to /lib/tls.disabled) I get the
following
>>
>>I''ve filed a wishlist bug at
>>http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=328183 to include the
>>Glibc patch [1] some how in Debian proper.
>>
>>[1] http://wiki.xensource.com/xenwiki/XenSpecificGlibc
>
>I''m a bit rusty with assembly language, but Debian''s
current glibc-2.3.5-6
>seems to have included this patch already, if -DNO_TLS_DIRECT_SEG_REFS is 
>specified with gcc.  I''m compiling libc6 on a spare server right
now to
>test it.  (It''s going REALLY slow as it is a slow server - ATHLON
800.)
>
>Will report back in a day or two.
Oddest thing...  the latest Debian libc6-i686 package (2.3.5-6) doesn''t
generate any TLS warning with Xen.  So I don''t have to compile my own 
version after all - the compilation failed anyway :-).  Could others 
confirm this!?

I have been stuck with Mercurial 0.6b because 0.6c (presume 0.7 too) 
requires libc6 2.3.5.  Now I can upgrade!  :-) 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

At 07:22 PM 9/17/2005, you wrote:
>At 12:52 PM 9/17/2005, you wrote:
>
>>At 05:58 PM 9/13/2005, you wrote:
>>
>>>On 9/14/05, Markus Schuster <ma.schuster@gmx.de> wrote:
>>> > in order to test a bit around with XEN 2.0.7 I''ve
made a domU out of my
>>> > existing workstation installation. That''s a mixture
with most packages
>>> > from testing and some hand-selected packages from unstable. If
I move
>>> > /lib/tls out of the way (eg. to /lib/tls.disabled) I get the
following
>>>
>>>I''ve filed a wishlist bug at
>>>http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=328183 to include
the
>>>Glibc patch [1] some how in Debian proper.
>>>
>>>[1] http://wiki.xensource.com/xenwiki/XenSpecificGlibc
>>
>>I''m a bit rusty with assembly language, but Debian''s
current
>>glibc-2.3.5-6 seems to have included this patch already, if 
>>-DNO_TLS_DIRECT_SEG_REFS is specified with gcc.  I''m compiling
libc6 on a
>>spare server right now to test it.  (It''s going REALLY slow as
it is a
>>slow server - ATHLON 800.)
>>
>>Will report back in a day or two.
>
>Oddest thing...  the latest Debian libc6-i686 package (2.3.5-6)
doesn''t
>generate any TLS warning with Xen.  So I don''t have to compile my
own
>version after all - the compilation failed anyway :-).  Could others 
>confirm this!?
>
>I have been stuck with Mercurial 0.6b because 0.6c (presume 0.7 too) 
>requires libc6 2.3.5.  Now I can upgrade!  :-)
Ah, strike that.  The warning message is in the console, not 
/var/log/messages where I was looking... back to compiling.  :-(


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Andy Lee schrieb:
>>> I''m a bit rusty with assembly language, but
Debian''s current
>>> glibc-2.3.5-6 seems to have included this patch already, if
>>> -DNO_TLS_DIRECT_SEG_REFS is specified with gcc.  
> [..]
> back to compiling.  :-(
Would that meen one has to recompile all packages with
-DNO_TLS_DIRECT_SEG_REFS or should it be enough to recompile libc6 with
that parameter?
I really hope you are able to solve that puzzle. I got the sources for
libc6 but it really doesn''t look like I expected it to do, so I pushed
that on my ToDo list :)

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sunday 18 September 2005 12:59, Markus Schuster wrote:
>
> Would that meen one has to recompile all packages with
> -DNO_TLS_DIRECT_SEG_REFS or should it be enough to recompile libc6 with
> that parameter?
> I really hope you are able to solve that puzzle. I got the sources for
> libc6 but it really doesn''t look like I expected it to do, so I
pushed
> that on my ToDo list :)
-DNO_TLS_DIRECT_SEG_REFS defines a preprocessor macro, so unless the 
application/library is using that macro to switch between different ASM/Code 
blocks, it won''t do anything.

IOW, that macro most likely only applies to glibc.

Starting with gcc-3.4 there''s also an gcc switch to prevent the
compiler from
generating non-xen compatible code on its own:
"-mno-tls-direct-seg-refs"

So: recompile your glibc. check if some apps still make xen complain 
about /lib/tls, those are either statically linked against the previous glibc 
or contain incompatible memory acesses themself, generated by inline asm or 
gcc optimizations.

recompiling those apps with CFLAGS=-mno-tls-direct-seg-refs shoud fix it.

/Ernst

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 9/18/05, Ernst Bachmann <e.bachmann@xebec.de>
wrote:
> So: recompile your glibc. check if some apps still make xen complain
> about /lib/tls, those are either statically linked against the previous
glibc
> or contain incompatible memory acesses themself, generated by inline asm or
> gcc optimizations.
> 
> recompiling those apps with CFLAGS=-mno-tls-direct-seg-refs shoud fix it.
Might be a good idea if a list of these applications can be built up somewhere.

-- 
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

At 03:59 AM 9/18/2005, you wrote:
>Andy Lee schrieb:
> >>> I''m a bit rusty with assembly language, but
Debian''s current
> >>> glibc-2.3.5-6 seems to have included this patch already, if
> >>> -DNO_TLS_DIRECT_SEG_REFS is specified with gcc.
> > [..]
> > back to compiling.  :-(
>
>Would that meen one has to recompile all packages with
>-DNO_TLS_DIRECT_SEG_REFS or should it be enough to recompile libc6 with
>that parameter?
>I really hope you are able to solve that puzzle. I got the sources for
>libc6 but it really doesn''t look like I expected it to do, so I
pushed
>that on my ToDo list :)
Ok, success!  Here is what I did.  First, a total failure - I tried to 
apt-build libc6 (2.3.5) on Sarge.  After numerous tries (errors, upgrade 
packages, errors, etc), I finally gave up... decided I should compile on 
Sid, since 2.3.5 is from the Debian testing branch.

So I started over with a Sid install.  This time, it compiled 
successfully.  I changed debian/rules to include
"-mno-tls-direct-seg-refs"
in BUILD_CFLIGS and HOST_CFLAGS, and fudged debian/control to make 
libc6-i686 dependent on libc6 (>= 2.3.5-6), so it could be installed with 
other libc6 packages of the same version.  Here''s the result:

http://members.dslextreme.com/users/andylee/libc6-i686_2.3.5-6.xen_i386.deb

With it, my Sid DomU doesn''t give the TLS warning.

Earlier I mentioned that I needed this to install Mercurial 0.6c.  But 
during the long compiling time, I compiled hg 0.6c successfully on my libc 
2.3.2 environmetn, so I didn''t need libc 2.3.5 after all. 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ernst Bachmann schrieb:
> So: recompile your glibc. check if some apps still make xen complain 
> about /lib/tls, those are either statically linked against the previous
glibc
> or contain incompatible memory acesses themself, generated by inline asm or
> gcc optimizations.
> 
> recompiling those apps with CFLAGS=-mno-tls-direct-seg-refs shoud fix it.
Thanks a lot for all your help! Now I have a XEN-working libc6 debian
package. And it looks like most of my apps don''t make problems so I
don''t have to recompile them. Only mplayer and the nvidia-glx
initscript
(what is not needed within a XEN domU of course) are triggering the tls
error message.

@Andy Lee
I had no problems rebuilding the libc6 package on sarge. Here is what I
did (maybe usefull for others, too):
# apt-get build-dep libc6
$ apt-get source libc6
Then I''ve edited the debian/rules file and added
''-mno-tls-direct-seg-refs'' to both BUILD_CFLAGS and
HOST_CFLAGS.
Then I did a
$ dpkg-buildpackage -rfakeroot
And voilá, after some time of compiling I got some debian packages as
expected. I''ve installed libc6 and libc6-i686 and put both packages on
hold so they are not updated.

Greetings,
Markus



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nicholas Lee schrieb:
>>So: recompile your glibc. check if some apps still make xen complain
>>about /lib/tls, those are either statically linked against the previous
glibc
>>or contain incompatible memory acesses themself, generated by inline asm
or
>>gcc optimizations.
>>
>>recompiling those apps with CFLAGS=-mno-tls-direct-seg-refs shoud fix
it.
> 
> 
> Might be a good idea if a list of these applications can be built up
somewhere.
What about the XEN wiki?

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Markus Schuster schrieb:
> @Andy Lee
> I had no problems rebuilding the libc6 package on sarge. 
I''m sorry for the typo, of course I did the build on testing (etch),
not
sarge.

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users