Fedora xen - Dec 2006 - 4gb seg fixup, process on FC6 domU

Hi,

We are running xen on fc6 host and all domU are also fc6, the only issue we
are currently facing is getting lot of error messages in logs

Dec 20 13:06:37  kernel: 4gb seg fixup, process syslogd (pid 873), cs:ip
73:00259ea0
Dec 20 13:06:37  kernel: 4gb seg fixup, process klogd (pid 876), cs:ip
73:001e63a0
Dec 20 13:06:37  kernel: 4gb seg fixup, process klogd (pid 876), cs:ip
73:00268e50
Dec 20 13:06:37  kernel: 4gb seg fixup, process klogd (pid 876), cs:ip
73:001e63a0
Dec 20 13:06:42  kernel: printk: 21 messages suppressed.
Dec 20 13:06:42  kernel: 4gb seg fixup, process master (pid 997), cs:ip
73:003b3cfc
Dec 20 13:06:48  kernel: printk: 30 messages suppressed.
Dec 20 13:06:48  kernel: 4gb seg fixup, process sshd (pid 1121), cs:ip
73:003a4cfc
Dec 20 13:06:50  kernel: printk: 57 messages suppressed.
Dec 20 13:06:50  kernel: 4gb seg fixup, process sshd (pid 1121), cs:ip
73:003a4cfc

getting these error eventually for all processes and on all fc6 domU, I have
tried the fix mention in one of thread ie ..

I''ve done all of that without success. Any other ideas?

I''ve renamed /lib/tls to /lib/tls.disabled and echo ''hwcap 0
nosegneg'' >
/etc/ld.so.conf.d/libc6-xen.conf

But still getting these error message in domU log file and also on
console when access it via console which makes domU console unusable.

Any help in this regards, will be greatly appreciated.

Thanks. Askar

On Wed, Dec 20, 2006 at 06:28:03PM +0500, Asrai khn
wrote:
> Hi,
> 
> We are running xen on fc6 host and all domU are also fc6, the only issue we
> are currently facing is getting lot of error messages in logs
> 
> Dec 20 13:06:37  kernel: 4gb seg fixup, process syslogd (pid 873), cs:ip
> 73:00259ea0
> Dec 20 13:06:37  kernel: 4gb seg fixup, process klogd (pid 876), cs:ip
> 73:001e63a0
> Dec 20 13:06:37  kernel: 4gb seg fixup, process klogd (pid 876), cs:ip
> 73:00268e50
> Dec 20 13:06:37  kernel: 4gb seg fixup, process klogd (pid 876), cs:ip
> 73:001e63a0
> Dec 20 13:06:42  kernel: printk: 21 messages suppressed.
> Dec 20 13:06:42  kernel: 4gb seg fixup, process master (pid 997), cs:ip
> 73:003b3cfc
> Dec 20 13:06:48  kernel: printk: 30 messages suppressed.
> Dec 20 13:06:48  kernel: 4gb seg fixup, process sshd (pid 1121), cs:ip
> 73:003a4cfc
> Dec 20 13:06:50  kernel: printk: 57 messages suppressed.
> Dec 20 13:06:50  kernel: 4gb seg fixup, process sshd (pid 1121), cs:ip
> 73:003a4cfc
> 
> getting these error eventually for all processes and on all fc6 domU, I
have
> tried the fix mention in one of thread ie ..
> 
> I''ve done all of that without success. Any other ideas?
> 
> I''ve renamed /lib/tls to /lib/tls.disabled and echo
''hwcap 0 nosegneg'' >
> /etc/ld.so.conf.d/libc6-xen.conf
Both of those are redundant/wrong for FC6.

First, there is no separate /lib/tls anymore - FC6 ditched old pthreads
implementation, and the new NPTL is the default in /lib. So renaming the
empty /lib/tls directory is irrelevant.

Second, the ''hwcap 0 nosegneg'' stuff is automatically
installed by the
Xen kernel RPM, eg

  $ rpm -qf /etc/ld.so.conf.d/kernelcap-2.6.18-1.2868.fc6.conf
  kernel-xen-2.6.18-1.2868.fc6

  $ cat /etc/ld.so.conf.d/kernelcap-2.6.18-1.2868.fc6.conf 
  # This directive teaches ldconfig to search in nosegneg subdirectories
  # and cache the DSOs there with extra bit 0 set in their hwcap match
  # fields.  In Xen guest kernels, the vDSO tells the dynamic linker to
  # search in nosegneg subdirectories and to match this extra hwcap bit
  # in the ld.so.cache file.
  hwcap 0 nosegneg
> But still getting these error message in domU log file and also on
> console when access it via console which makes domU console unusable.
What kernel version are you running & what kernel-xen & libc RPMs are 
installed in your guest showing this problem. Also are they successfully
(eg, rpm -V kernel-xen  libc)

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Hi Dan,




Both of those are redundant/wrong for FC6.
>
> First, there is no separate /lib/tls anymore - FC6 ditched old pthreads
> implementation, and the new NPTL is the default in /lib. So renaming the
> empty /lib/tls directory is irrelevant.
>
> Second, the ''hwcap 0 nosegneg'' stuff is automatically
installed by the
> Xen kernel RPM, eg
>
>   $ rpm -qf /etc/ld.so.conf.d/kernelcap-2.6.18-1.2868.fc6.conf
>   kernel-xen-2.6.18-1.2868.fc6

When i do this on host  machine gives me...

rpm -qf /etc/ld.so.conf.d/kernelcap-2.6.18-1.2849.fc6.conf
kernel-xen-2.6.18-1.2849.fc6

but gives nothing on guest, i don''t have /etc/ld.so.conf.d/kernelcap-
2.6.18-1.2849.fc6.conf on guests

  $ cat
/etc/ld.so.conf.d/kernelcap-2.6.18-1.2868.fc6.conf
>   # This directive teaches ldconfig to search in nosegneg subdirectories
>   # and cache the DSOs there with extra bit 0 set in their hwcap match
>   # fields.  In Xen guest kernels, the vDSO tells the dynamic linker to
>   # search in nosegneg subdirectories and to match this extra hwcap bit
>   # in the ld.so.cache file.
>   hwcap 0 nosegneg
>
> > But still getting these error message in domU log file and also on
> > console when access it via console which makes domU console unusable.
>
> What kernel version are you running & what kernel-xen & libc RPMs
are
> installed in your guest showing this problem. Also are they successfully
> (eg, rpm -V kernel-xen  libc)

I''m launching  guest using xm create vms.cfg

and vms.cfg is as under

kernel = "/boot/vmlinuz-2.6.18-1.2849.fc6xen"
ramdisk = "/boot/initrd-2.6.18-1.2849.fc6xen-domU.img"
memory = 128
name = "xxxx"
#vif = [ ''mac=00:16:3e:11:69:85, bridge=xenbr0'' ]
vif = [ ''ip=66.xxx.xxx.xxx'' ]
disk = [''tap:aio:/var/uml/vm3/root,sda1,w'',
''tap:aio:/var/uml/vm3/var,sda2,w'',
''tap:aio:/var/uml/vm3/swap,sda3,w'']
root = "/dev/sda1 ro"
on_reboot   = ''restart''
on_crash    = ''restart''

So the kernel running on guest ... and i have also copied the xen kernel
modules /lib/modules/ on guest disk image by mounting it.

uname -r
2.6.18-1.2849.fc6xen

Secondly the default initrd for xen kernel that come with fc6 fails to start
our guest so i have created initrd manually by ...

mkinitrd -f --with=xennet --preload=xenblk /boot/initrd-
2.6.18-1.2849.fc6xen-domU.img 2.6.18-1.2849.fc6xen

and then use this initrd for ''ramdisk'' in vms.cfg

Thanks.

On 12/20/06, Daniel P. Berrange <berrange@redhat.com>
wrote:
>
> On Wed, Dec 20, 2006 at 07:19:06PM +0500, Asrai khn wrote:
> > Hi Dan,
> >
> >
> > Both of those are redundant/wrong for FC6.
> > >
> > >First, there is no separate /lib/tls anymore - FC6 ditched old
pthreads
> > >implementation, and the new NPTL is the default in /lib. So
renaming
> the
> > >empty /lib/tls directory is irrelevant.
> > >
> > >Second, the ''hwcap 0 nosegneg'' stuff is
automatically installed by the
> > >Xen kernel RPM, eg
> > >
> > >  $ rpm -qf /etc/ld.so.conf.d/kernelcap-2.6.18-1.2868.fc6.conf
> > >  kernel-xen-2.6.18-1.2868.fc6
> >
> >
> > When i do this on host  machine gives me...
> >
> > rpm -qf /etc/ld.so.conf.d/kernelcap-2.6.18-1.2849.fc6.conf
> > kernel-xen-2.6.18-1.2849.fc6
> >
> > but gives nothing on guest, i don''t have
/etc/ld.so.conf.d/kernelcap-
> > 2.6.18-1.2849.fc6.conf on guests
>
> So, have you actually got a kernel-xen installed in your guest ? It
> sounds very much like you dont....

yes we don''t have xen kernel  on guest  coz we  are not using pygrub.
> I''m launching  guest using xm create vms.cfg
> >
> > and vms.cfg is as under
> >
> > kernel = "/boot/vmlinuz-2.6.18-1.2849.fc6xen"
> > ramdisk = "/boot/initrd-2.6.18-1.2849.fc6xen-domU.img"
>
> The fact that you are explicitly listing kernels from Dom0 instead of
> using the regular pygrub  bootloader, also suggests to me you''ve
not
> got kernel-xen installed in your guest image

True, no pygrub
> memory = 128
> > name = "xxxx"
> > #vif = [ ''mac=00:16:3e:11:69:85, bridge=xenbr0'' ]
> > vif = [ ''ip=66.xxx.xxx.xxx'' ]
> > disk = [''tap:aio:/var/uml/vm3/root,sda1,w'',
> > ''tap:aio:/var/uml/vm3/var,sda2,w'',
''tap:aio:/var/uml/vm3/swap,sda3,w'']
> > root = "/dev/sda1 ro"
> > on_reboot   = ''restart''
> > on_crash    = ''restart''
> >
> > So the kernel running on guest ... and i have also copied the xen
kernel
> > modules /lib/modules/ on guest disk image by mounting it.
>
> If you''re not using pygrub, then you''ll at least need to
make sure the
> correct ''kernelcap'' file is installed in the guest, with
file name
> matching the kernel version.

What do you mean by install ''kernelcap'' file on guest?
you want me to create this file manually in the specific path and then echo
''something'' to it?

Thanks. Askar

Dan.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
> -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/
>               -=|
> |=-               Projects: http://freshmeat.net/~danielpb/
> -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505  -=|
>

On Wed, Dec 20, 2006 at 07:37:41PM +0500, Asrai khn
wrote:
> On 12/20/06, Daniel P. Berrange <berrange@redhat.com> wrote:
> >On Wed, Dec 20, 2006 at 07:19:06PM +0500, Asrai khn wrote:
> >
> >If you''re not using pygrub, then you''ll at least need
to make sure the
> >correct ''kernelcap'' file is installed in the guest,
with file name
> >matching the kernel version.
> 
> 
> What do you mean by install ''kernelcap'' file on guest?
> you want me to create this file manually in the specific path and then echo
> ''something'' to it?
Actually i''ve realized its not the name of the config name which
matters.
The issue is you need to run ''ldconfig'' after creating the
config file so
that it updates its library cache

  echo "hwcap 0 nosegneg" > /etc/ld.conf.so.d/nosegneg.conf
  ldconfig


Once you''ve done that, you can verify the linker knows about Xen
variant
by looking for libc mapping. eg before I add the hwcap stuff it shows:

  # ldconfig -v -p 2>&1 | grep libc.so
        libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6

Once I create the hwcap setting & re-run ldconfig it shows

  # ldconfig -v -p 2>&1 | grep libc.so
        libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/libc.so.6
        libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

On 12/20/06, Daniel P. Berrange <berrange@redhat.com>
wrote:
>
> On Wed, Dec 20, 2006 at 07:37:41PM +0500, Asrai khn wrote:
> > On 12/20/06, Daniel P. Berrange <berrange@redhat.com> wrote:
> > >On Wed, Dec 20, 2006 at 07:19:06PM +0500, Asrai khn wrote:
> > >
> > >If you''re not using pygrub, then you''ll at least
need to make sure the
> > >correct ''kernelcap'' file is installed in the
guest, with file name
> > >matching the kernel version.
> >
> >
> > What do you mean by install ''kernelcap'' file on
guest?
> > you want me to create this file manually in the specific path and then
> echo
> > ''something'' to it?
>
> Actually i''ve realized its not the name of the config name which
matters.
> The issue is you need to run ''ldconfig'' after creating
the config file so
> that it updates its library cache
>
>   echo "hwcap 0 nosegneg" > /etc/ld.conf.so.d/nosegneg.conf
>   ldconfig

Right, I did this and run ''ldconfig'' afterward but  its
/etc/ld.so.conf.d/nosegneg.conf not /etc/ld.conf.so.d/nosegneg.conf , may be
typo

Once you''ve done that, you can verify the linker knows about Xen
variant
> by looking for libc mapping. eg before I add the hwcap stuff it shows:
>
>   # ldconfig -v -p 2>&1 | grep libc.so
>         libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6

I get this before echo thing...

ldconfig -v -p 2>&1 | grep libc.so
        libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6


Once I create the hwcap setting & re-run ldconfig it
shows
>
>   # ldconfig -v -p 2>&1 | grep libc.so
>         libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
> => /lib/i686/nosegneg/libc.so.6
>         libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6

After hwcap I get ..

ldconfig -v -p 2>&1 | grep libc.so
        libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/libc.so.6
        libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6

But still getting those  ''4gb seg fixup, process'' errors

ldconfig is enough not need to reboot the guest right?

Thanks. Askar

Regards,
> Dan.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
> -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/
>               -=|
> |=-               Projects: http://freshmeat.net/~danielpb/
> -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505  -=|
>

On Wed, Dec 20, 2006 at 07:58:46PM +0500, Asrai khn
wrote:
> On 12/20/06, Daniel P. Berrange <berrange@redhat.com> wrote:
> >
> >On Wed, Dec 20, 2006 at 07:37:41PM +0500, Asrai khn wrote:
> >> On 12/20/06, Daniel P. Berrange <berrange@redhat.com> wrote:
> >> >On Wed, Dec 20, 2006 at 07:19:06PM +0500, Asrai khn wrote:
> >> >
> >> >If you''re not using pygrub, then you''ll at
least need to make sure the
> >> >correct ''kernelcap'' file is installed in the
guest, with file name
> >> >matching the kernel version.
> >>
> >>
> >> What do you mean by install ''kernelcap'' file on
guest?
> >> you want me to create this file manually in the specific path and
then
> >echo
> >> ''something'' to it?
> >
> >Actually i''ve realized its not the name of the config name
which matters.
> >The issue is you need to run ''ldconfig'' after
creating the config file so
> >that it updates its library cache
> >
> >  echo "hwcap 0 nosegneg" >
/etc/ld.conf.so.d/nosegneg.conf
> >  ldconfig
> 
> 
> Right, I did this and run ''ldconfig'' afterward but  its
> /etc/ld.so.conf.d/nosegneg.conf not /etc/ld.conf.so.d/nosegneg.conf , may
be
> typo
> 
> Once you''ve done that, you can verify the linker knows about Xen
variant
> >by looking for libc mapping. eg before I add the hwcap stuff it shows:
> >
> >  # ldconfig -v -p 2>&1 | grep libc.so
> >        libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6
> 
> 
> I get this before echo thing...
> 
> ldconfig -v -p 2>&1 | grep libc.so
>        libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6
> 
> 
> Once I create the hwcap setting & re-run ldconfig it shows
> >
> >  # ldconfig -v -p 2>&1 | grep libc.so
> >        libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux
2.6.9)
> >=> /lib/i686/nosegneg/libc.so.6
> >        libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6
> 
> 
> After hwcap I get ..
> 
> ldconfig -v -p 2>&1 | grep libc.so
>        libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
=>
> /lib/i686/nosegneg/libc.so.6
>        libc.so.6 (libc6, OS ABI: Linux 2.6.9) => /lib/libc.so.6
> 
> But still getting those  ''4gb seg fixup, process'' errors
> 
> ldconfig is enough not need to reboot the guest right?
Nope. All you''ve done is update the linker cache. This means any new
processes
you''ve spawned will use the correct libc.  Any old processes will still
have
the old (Xen unfriendly)  libc mapped into memory. A reboot is the only way
to get everything using the new settings I''m afraid.



NB, even with this change you''ll see the 4gb fixup messages if you run
any
mono processes - eg beagle or tomboy - this is a bug in the Mono VM for
which there isn''t any fix yet.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

>
> Hi Dan,



Nope. All you''ve done is update the linker cache. This means any
new
> processes
> you''ve spawned will use the correct libc.  Any old processes will
still
> have
> the old (Xen unfriendly)  libc mapped into memory. A reboot is the only
> way
> to get everything using the new settings I''m afraid.

Cool after reboot i''m am  getting those errors not in logs neither when
connect to guest via console :)

So the solution

echo "hwcap 0 nosegneg" > /etc/ld.conf.so.d/nosegneg.conf

work for us.

NB, even with this change you''ll see the 4gb fixup messages if you run
any
> mono processes - eg beagle or tomboy - this is a bug in the Mono VM for
> which there isn''t any fix yet.

We don''t use mono process beagle or tomboy.

I very much appreciate your help :)

Thanks. Askar

Regards,
> Dan.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
> -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/
>               -=|
> |=-               Projects: http://freshmeat.net/~danielpb/
> -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505  -=|
>

Hi,

On 12/20/06, Asrai khn <asraikhn@gmail.com> wrote:
> So the solution
>
> echo "hwcap 0 nosegneg" > /etc/ld.conf.so.d/nosegneg.conf
>
> work for us.
don''t forget the ldconfig. people searching the list archive might
wonder why it doesn''t work for them because they just follow this mail
here...

> > NB, even with this change you''ll see the 4gb fixup messages
if you run any
> > mono processes - eg beagle or tomboy - this is a bug in the Mono VM
for
> > which there isn''t any fix yet.
>
> We don''t use mono process beagle or tomboy.
I also had this annoying problem but also on Domain 0. When reading
the "fine print" I saw the hint on mono causing theses things.

removed beagle and everything works fine.


thanks,
Henning

Hi,

Let me clear the type which we are doing from the very start of this thread

on FC6 instead of

echo "hwcap 0 nosegneg" > /etc/ld.conf.so.d/nosegneg.conf  <---
type in the
directory name

do this ...

echo hwcap 0 nosegneg" > /etc/ld.so.conf.d/nosegneg.conf

and then

ldconfig

Thanks. Askar


On 12/20/06, Henning Sprang <henning_sprang@gmx.de>
wrote:
>
> Hi,
>
> On 12/20/06, Asrai khn <asraikhn@gmail.com> wrote:
> > So the solution
> >
> > echo "hwcap 0 nosegneg" > /etc/ld.conf.so.d/nosegneg.conf
> >
> > work for us.
>
> don''t forget the ldconfig. people searching the list archive might
> wonder why it doesn''t work for them because they just follow this
mail
> here...
>
>
> > > NB, even with this change you''ll see the 4gb fixup
messages if you run
> any
> > > mono processes - eg beagle or tomboy - this is a bug in the Mono
VM
> for
> > > which there isn''t any fix yet.
> >
> > We don''t use mono process beagle or tomboy.
>
> I also had this annoying problem but also on Domain 0. When reading
> the "fine print" I saw the hint on mono causing theses things.
>
> removed beagle and everything works fine.
>
>
> thanks,
> Henning
>
> --
> Fedora-xen mailing list
> Fedora-xen@redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-xen
>

On 12/20/06, Asrai khn <asraikhn@gmail.com> wrote:
> do this ...
>
> echo hwcap 0 nosegneg" > /etc/ld.so.conf.d/nosegneg.conf
or better

echo "hwcap 0 nosegneg" > /etc/ld.so.conf.d/nosegneg.conf

:)
>
> and then
>
> ldconfig
Yep.

And remove beagle when you want to get rid of all warnings...

Henning