Lustre discuss - May 2006 - lustre kernel package is misbuilt for RHEL4/centos-4

Hello again,

I''m not a big fan of replying to myself but this issue is kind of
important
and there was zero replies.

First a small update, the issue is still there in the 1.4.5.2 and 1.4.6 binary 
packages as available.

Impact:
It is impossible to build external modules against the lustre kernels on el4. 
This is specifically a problem for me when it comes to 3w-9xxx and IBGD. The 
former is required to talk to my raid cards and the later is required for my 
client machines to be usable (they run mpi over ib using ibgd).

Work around:
What I''ve had to do so far is to rebuild the lustre kernel (I did it
with make
rpm from the kernel-source package). This gives me a lustre patched kernel 
that I can build modules against. After that lustre modules and userspace 
must be rebuilt which caused me more head ache then rebuilding the kernel but 
it was doable.

Unpleasant side effects for my work around:
* the kernel package as generated by make rpm in the kernel tree is not as 
hightech as the one generated by rebuilding the .src.rpm.
* when rebuilding lustre userspace and modules I had to do a ./configure + 
make rpms that involved choices, I no longer know that I have a well 
configured lustre setup. Atleast it''s probably not identical to the
prebuilt
one.

Suggestion:
Build your el4 packages on el4 in the future (not on, and I''m guessing,
sles9)
or atleast on centos-4.

/Peter

On Friday 10 February 2006 16:13, Peter Kjellstr?m
wrote:
> Hello,
>
> OS: centos-4.2-x86_64 with 2.6.9-22smp lustre kernel from 1.4.5.1
>
> Short summary: lustre works well but when I try to build modules against
> the lustre kernel (using kernel-source from lustre) there is a versionmagic
> problem. (probably related to lustre being built with gcc-3.3 and RHEL with
> gcc-3.4)
>
> RHEL uses gcc-3.4 and as such all version magic is as:
>
> modinfo e1000.ko | grep vermagic
> vermagic:       2.6.9-11.ELsmp SMP gcc-3.4
>
> On a machine with a lustre kernel (from 1.4.5.1) I have this:
>
> modinfo e1000.ko | grep magic
> vermagic:       2.6.9-22.EL_lustre.1.4.5.1smp SMP gcc-3.3
> ...
-- 
------------------------------------------------------------
  Peter Kjellstr?m               | E-mail: cap@nsc.liu.se
  National Supercomputer Centre  |
  Sweden                         | http://www.nsc.liu.se
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: not available
Url :
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20060303/b29fad18/attachment.bin

Hello,

OS: centos-4.2-x86_64 with 2.6.9-22smp lustre kernel from 1.4.5.1

Short summary: lustre works well but when I try to build modules against the 
lustre kernel (using kernel-source from lustre) there is a versionmagic 
problem. (probably related to lustre being built with gcc-3.3 and RHEL with 
gcc-3.4)

RHEL uses gcc-3.4 and as such all version magic is as:

modinfo e1000.ko | grep vermagic
vermagic:       2.6.9-11.ELsmp SMP gcc-3.4

On a machine with a lustre kernel (from 1.4.5.1) I have this:

modinfo e1000.ko | grep magic
vermagic:       2.6.9-22.EL_lustre.1.4.5.1smp SMP gcc-3.3

All is usually well until one tries to build something against this kernel 
(ie. installing and using kernel-source for the lustre kernel).

I tried to build IBGD-1.8.2 from mellanox (Infiniband stack) and this is what 
I get trying to modprobe such a module (it did build fine):

modprobe mlxsys
FATAL: Error inserting mlxsys 
(/lib/modules/2.6.9-22.EL_lustre.1.4.5.1smp/kernel/drivers/infiniband/mlxsys.ko):
Invalid module format
FATAL: Error running install command for mlxsys

on startup it''s a little more verbose as to why it doesn''t
like the module:

mlxsys: version magic ''2.6.9-22.EL_lustre.1.4.5.1smp SMP
gcc-3.4'' should be
''2.6.9-22.EL_lustre.1.4.5.1smp SMP gcc-3.3''

this is also confirmed with modinfo:
modinfo mlxsys.ko | grep vermagic
vermagic:       2.6.9-22.EL_lustre.1.4.5.1smp SMP gcc-3.4

What to do...
 Peter K

-- 
------------------------------------------------------------
  Peter Kjellstr?m               |
  National Supercomputer Centre  |
  Sweden                         | http://www.nsc.liu.se
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20060210/f43d9d54/attachment.bin

On Mar 3, 2006, at 4:16 AM, Peter Kjellstr?m wrote:
> Hello again,
>
> I''m not a big fan of replying to myself but this issue is kind of 
> important
> and there was zero replies.

Peter -- as a small hack, you can edit version.h or versionmagic.h (not 
exactly sure which one) in the kernel tree to ignore the GCC version it 
computes and hardcode it to gcc-3.3. I''ve had good luck with that 
working for me in the past, YMMV of course.

Nic

On Tuesday 07 March 2006 01:37, Nicholas Henke wrote:
> On Mar 3, 2006, at 4:16 AM, Peter Kjellstr?m wrote:
> > Hello again,
> >
> > I''m not a big fan of replying to myself but this issue is
kind of
> > important
> > and there was zero replies.
>
> Peter -- as a small hack, you can edit version.h or versionmagic.h (not
> exactly sure which one) in the kernel tree to ignore the GCC version it
> computes and hardcode it to gcc-3.3. I''ve had good luck with that
> working for me in the past, YMMV of course.
I did try to force it... it broke =(

there is, as the manpage for modprobe says, a reason for the vermagic field.

thanks anyway,
 Peter
>
> Nic
-- 
------------------------------------------------------------
  Peter Kjellstr?m               |
  National Supercomputer Centre  |
  Sweden                         | http://www.nsc.liu.se
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: not available
Url :
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20060307/75dee607/attachment.bin