Hi All,
we are using the Mellanox mem based IB
interface (bought from HP) for lustre file system
access
using "lustre-1.4.6.2" , kernel-2.6.9-34
we have applied the patch from Mellanox posed
in bugzilla on "IBGD-1.8.2"
recompiled the kernel, lustre IB modules, but
still occasionally facing the kernel panic.
you can see this bug in buzilla
https://bugzilla.clusterfs.com/show_bug.cgi?id=7246
just want to know anybody using luster over
mem based IB cards with out this issue, pls
share you experience and configuration.
thanks,
-bala-
____________________________________________________________________________________
Don''t pick lemons.
See all the new 2007 cars at Yahoo! Autos.
http://autos.yahoo.com/new_cars.html
On Friday 04 May 2007, Bala wrote:> just want to know anybody using luster over > mem based IB cards with out this issue, pls > share you experience and configuration.We have been running lustre-1.4.6.4 and kernel 2.6.9-34 with the voltaire ib-stack in production since October 2006 without any issues related to the infiniband networking. The file system has been performing very well with sustained io-rates between 300 and 400 megabytes/second for weeks on end which isn''t bad for a <$100K file system setup. r. -- The Computer Center, University of Troms?, N-9037 TROMS? Norway. phone:+47 77 64 41 07, fax:+47 77 64 41 00 Roy Dragseth, High Performance Computing System Administrator Direct call: +47 77 64 62 56. email: royd@cc.uit.no
Oh, I forgot to say specifically: We do use mem based ib-cards: 06:02.0 PCI bridge: Mellanox Technologies MT23108 PCI Bridge (rev a1) 07:00.0 InfiniBand: Mellanox Technologies MT23108 InfiniHost (rev a1) The cards is branded as Topspin cards, but we switched to the voltaire stack as it plays better with the voltaire switch monitoring capabilities. r.
Hi Roy,
Thanks for sharing your experiance, only
difference is you are using lustre-1.4.6.4 we
may need to try the same, what is your voltaire
stack version, we are using IBGD-1.8.2, here is the
steps we follow to setup lustre
1. Install the Lustre RPMS
#rpm -ivh
kernel-smp-2.6.9-34.EL_lustre.1.4.6.2.x86_64.rpm
kernel-source-2.6.9-34.EL_lustre.1.4.6.2.x86_64.rpm
lustre-source-1.4.6.2-2.6.9_34.EL_lustre.1.4.6.2smp.x86_64.rpm
2. Change the grub.conf to boot lustre kernel by
default and reboot
3. Build the lustre kernel with out infiniband support
4. Build the mellonax Gen1 src
#cd IBGD-1.8.2-srpt
#./install.sh
5. Install the mellonax Gen1 RPMS
6. Configure and install lustre modules
7. Update the config file
/etc/infiniband/ifcfg-ib0
8. set network parameters in /etc/modprobe.conf
#options lnet networks=openib0
9. Create lustre dir and put upcall script
#mkdir /etc/lustre
#mkdir /media/data1
#mkdir /media/data2
create a upcall script
/etc/lustre/lustre_upcall.sh
10. reboot the machines and mount the lustre
partitions
#lconf --node hostname02-ib0 /etc/lustre/config.xml
could you see any errors in above steps, or could you
mind passing us your steps to setup the lustre that
works fine.
In the same clustre we have machines with mem free IB
cards, those machines are no kernel panic issues, that
is the reason we are interested to see your
config/setup.
thanks,
-bala-
--- Roy Dragseth <Roy.Dragseth@cc.uit.no> wrote:
> Oh, I forgot to say specifically: We do use mem
> based ib-cards:
>
> 06:02.0 PCI bridge: Mellanox Technologies MT23108
> PCI Bridge (rev a1)
> 07:00.0 InfiniBand: Mellanox Technologies MT23108
> InfiniHost (rev a1)
>
> The cards is branded as Topspin cards, but we
> switched to the voltaire stack
> as it plays better with the voltaire switch
> monitoring capabilities.
>
> r.
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@clusterfs.com
>
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss>
____________________________________________________________________________________
TV dinner still cooling?
Check out "Tonight''s Picks" on Yahoo! TV.
http://tv.yahoo.com/
We use version 3.5.5 of the voltaire stack, exact version of the rpm is
$ rpm -q ibhost-hpc
ibhost-hpc-3.5.5_16-S1.k2.6.9_34.EL_lustre1.4.6.4
If I remember correctly, we built the software in this order:
1. build kernel rpm w/lustre patches
2. boot up the kernel from stage 1.
3. build ibhost-hpc rpm
4. build lustre rpms pointing to the build source of the kernel and
ibhost-hpc.
The config steps for enabling the ib network layer is different than yours,
but they seems to functional equivalent.
r.
--
The Computer Center, University of Troms?, N-9037 TROMS? Norway.
phone:+47 77 64 41 07, fax:+47 77 64 41 00
Roy Dragseth, High Performance Computing System Administrator
Direct call: +47 77 64 62 56. email: royd@cc.uit.no
On Wednesday 04 July 2007, Bala wrote:> Thanks Roy, as you mentioned we have tested with > voltaire stack, and now the kernel panic issue is > not there.Great to hear that. Glad to be at help, r. -- The Computer Center, University of Troms?, N-9037 TROMS? Norway. phone:+47 77 64 41 07, fax:+47 77 64 41 00 Roy Dragseth, High Performance Computing System Administrator Direct call: +47 77 64 62 56. email: royd@cc.uit.no
Thanks Roy, as you mentioned we have tested with voltaire stack, and now the kernel panic issue is not there. -bala- --- Roy Dragseth <roy.dragseth@cc.uit.no> wrote:> We use version 3.5.5 of the voltaire stack, exact > version of the rpm is > > $ rpm -q ibhost-hpc > ibhost-hpc-3.5.5_16-S1.k2.6.9_34.EL_lustre1.4.6.4 > > If I remember correctly, we built the software in > this order: > > 1. build kernel rpm w/lustre patches > 2. boot up the kernel from stage 1. > 3. build ibhost-hpc rpm > 4. build lustre rpms pointing to the build source of > the kernel and > ibhost-hpc. > > The config steps for enabling the ib network layer > is different than yours, > but they seems to functional equivalent. > > r. > > > -- > > The Computer Center, University of Troms?, N-9037 > TROMS? Norway. > phone:+47 77 64 41 07, fax:+47 77 64 > 41 00 > Roy Dragseth, High Performance Computing System > Administrator > Direct call: +47 77 64 62 56. email: > royd@cc.uit.no > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com >https://mail.clusterfs.com/mailman/listinfo/lustre-discuss>____________________________________________________________________________________ Don''t pick lemons. See all the new 2007 cars at Yahoo! Autos. http://autos.yahoo.com/new_cars.html