Terry Rutledge
2008-Jun-12 23:14 UTC
[Lustre-discuss] Announce: Lustre 1.6.5 is available!
Hi all, At long last, Lustre 1.6.5 is available on the Sun Download Center Site. http://www.sun.com/software/products/lustre/get.jsp The change log and release notes can be read here: http://wiki.lustre.org/index.php?title=Change_Log_1.6 Thank you for your assistance; as always, you can report issues via Bugzilla (https://bugzilla.lustre.org/) Happy downloading! -- The Lustre Team --
Uhm, what happened to the infiniband support in the cfs kernel rpms? 1.6.4.3 had infiniband support but the drivers aren''t there for the 1.6.5 versions for rhel5. Thanks, - David Brown On Thu, Jun 12, 2008 at 4:14 PM, Terry Rutledge <Terry.Rutledge at sun.com> wrote:> Hi all, > > At long last, Lustre 1.6.5 is available on the Sun Download Center Site. > > http://www.sun.com/software/products/lustre/get.jsp > > The change log and release notes can be read here: > > http://wiki.lustre.org/index.php?title=Change_Log_1.6 > > Thank you for your assistance; as always, you can report issues via > Bugzilla (https://bugzilla.lustre.org/) > > Happy downloading! > > -- The Lustre Team -- > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Brian J. Murrell
2008-Jun-13 17:44 UTC
[Lustre-discuss] Announce: Lustre 1.6.5 is available!
On Fri, 2008-06-13 at 10:08 -0700, David Brown wrote:> Uhm, what happened to the infiniband support in the cfs kernel rpms? > 1.6.4.3 had infiniband support but the drivers aren''t there for the > 1.6.5 versions for rhel5.Starting with 1.6.5 we are supporting the "out-of-[vendor-]tree" OFED drivers. 1.6.5 is supposed to ship with the OFED 1.3 kernel-ib (and -devel) RPMs and the Lustre ko2iblnd kernel module built likewise. You should have the ko2iblnd module in the lustre-modules RPM. You are just missing the needed kernel-ib RPM. Terry, any idea why the kernel-ib and kernel-ib-devel RPMs didn''t make it up to the SDLC? b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080613/6342326b/attachment-0001.bin
A related question: is there an easy way of building the Lustre kernel RPMs for the RHEL 5.2 kernel? Maybe a bug in bugzilla with patches or so? Thanks, best regards, Erich On Freitag 13 Juni 2008, Brian J. Murrell wrote:> On Fri, 2008-06-13 at 10:08 -0700, David Brown wrote: > > Uhm, what happened to the infiniband support in the cfs kernel rpms? > > 1.6.4.3 had infiniband support but the drivers aren''t there for the > > 1.6.5 versions for rhel5. > > Starting with 1.6.5 we are supporting the "out-of-[vendor-]tree" OFED > drivers. 1.6.5 is supposed to ship with the OFED 1.3 kernel-ib (and > -devel) RPMs and the Lustre ko2iblnd kernel module built likewise. You > should have the ko2iblnd module in the lustre-modules RPM. You are just > missing the needed kernel-ib RPM. > > Terry, any idea why the kernel-ib and kernel-ib-devel RPMs didn''t make > it up to the SDLC? > > b.
On Sun, Jun 15, 2008 at 3:25 PM, Erich Focht <efocht at hpce.nec.com> wrote:> > A related question: is there an easy way of building the Lustre kernel > RPMs for the RHEL 5.2 kernel? Maybe a bug in bugzilla with patches or so? >In the past I''ve simply added the appropriate things to the rhel kernel spec file, the CFS guys have a very different build suite (I''m unsure how it works I''ve never been able to reproduce it). After modifying the rhel spec and putting the added ''lustre'' config and lustre kernel patches in the SOURCE directory all you have to do is call rpmbuild. This patch was made against an old 2.6.18-5x kernel not the 2.6.18-9x of rhel 5.2. So Some things in the patch will have to be modified. Also, this was used with 1.6.4 not 1.6.5 so if they added some extra patches in the series those need to be added to the appropriate part of the spec file. rpmbuild -ba --with lustre kernel-2.6.spec --- kernel-2.6.spec 2007-11-14 07:26:34.000000000 -0800 +++ kernel-2.6-lustre.spec 2008-01-10 14:34:30.000000000 -0800 @@ -15,6 +15,8 @@ %define with_smp %{?_without_smp: 0} %{?!_without_smp: 1} # kernel-PAE (only valid for i686) %define with_pae %{?_without_pae: 0} %{?!_without_pae: 1} +# kernel-lustre (only for lustre servers) +%define with_lustre %{?_without_lustre: 0} %{?!_without_lustre: 1} # kernel-xen (only valid for i686, x86_64 and ia64) %define with_xen %{?_without_xen: 0} %{?!_without_xen: 1} # kernel-kdump (only valid for ppc64) @@ -84,6 +86,7 @@ %if %{with_baseonly} %define with_smp 0 %define with_pae 0 +%define with_lustre 0 %define with_xen 0 %define with_kdump 0 %define with_debug 0 @@ -367,6 +370,9 @@ #Source67: kernel-%{kversion}-sparc64.config #Source68: kernel-%{kversion}-sparc64-smp.config +Source70: kernel-%{kversion}-x86_64-lustre.config +Source71: kernel-%{kversion}-i686-lustre.config + Source80: config-rhel-generic Source82: config-rhel-ppc64-generic Source83: config-olpc-generic @@ -1601,6 +1607,22 @@ Patch21879: linux-2.6-isdn-i4l-fix-memory-overruns.patch Patch21880: linux-2.6-ppc-chrp-fix-possible-strncmp-null-pointer-usage.patch Patch21881: linux-2.6-nfs-potential-file-corruption-issue-when-writing.patch + +# lustre patches +patch25000: lustre_version.patch +patch25001: vfs_races-2.6-rhel5.patch +patch25002: i_filter_data.patch +patch25003: jbd-jcberr-2.6.18-vanilla.patch +patch25004: export_symbols-2.6.18-vanilla.patch +patch25005: dev_read_only-2.6.18-vanilla.patch +patch25006: export-2.6.18-vanilla.patch +patch25007: 8kstack-2.6.12.patch +patch25008: export-show_task-2.6.18-vanilla.patch +patch25009: sd_iostats-2.6-rhel4.patch +patch25010: export_symbol_numa-2.6-fc5.patch +patch25011: jbd-stats-2.6-rhel5.patch +# end lustre patches + # adds rhel version info to version.h Patch99990: linux-2.6-rhel-version-h.patch # empty final patch file to facilitate testing of kernel patches @@ -1738,6 +1760,42 @@ This package provides kernel headers and makefiles sufficient to build modules against the SMP kernel package. +%package lustre +Summary: The Linux kernel compiled for Lustre. +Group: System Environment/Kernel +Provides: kernel = %{version} +Provides: kernel-drm = 4.3.0 +Provides: kernel-%{_target_cpu} = %{rpmversion}-%{release}lustre +Prereq: %{kernel_prereq} +Conflicts: %{kernel_dot_org_conflicts} +Conflicts: %{package_conflicts} +# upto and including kernel 2.4.9 rpms, the 4Gb+ kernel was called kernel-enterprise +# now that the smp kernel offers this capability, obsolete the old kernel +Obsoletes: kernel-enterprise < 2.4.10 +# We can''t let RPM do the dependencies automatic because it''ll then pick up +# a correct but undesirable perl dependency from the module headers which +# isn''t required for the kernel proper to function +AutoReq: no +AutoProv: yes + +%description lustre +This package includes a Lustre version of the Linux kernel. It is +required only on machines that want to be lustre servers. + +%package lustre-devel +Summary: Development package for building kernel modules to match the Lustre kernel. +Group: System Environment/Kernel +Provides: kernel-lustre-devel-%{_target_cpu} = %{rpmversion}-%{release} +Provides: kernel-devel-%{_target_cpu} = %{rpmversion}-%{release}lustre +Provides: kernel-devel = %{rpmversion}-%{release}lustre +AutoReqProv: no +Prereq: /usr/bin/find + +%description lustre-devel +This package provides kernel headers and makefiles sufficient to build modules +against the Lustre kernel. + + %if %{?debugbuildsenabled} %package debug Summary: The Linux kernel compiled with extra debugging enabled. @@ -3217,6 +3275,20 @@ %patch21879 -p1 %patch21880 -p1 %patch21881 -p1 + +#lustre patches +%patch25000 -p1 +%patch25001 -p1 +%patch25002 -p1 +%patch25003 -p1 +%patch25004 -p1 +%patch25005 -p1 +%patch25006 -p1 +%patch25007 -p1 +%patch25008 -p1 +%patch25009 -p1 +%patch25010 -p1 +%patch25011 -p1 # correction of SUBLEVEL/EXTRAVERSION in top-level source tree Makefile # patch the Makefile to include rhel version info %patch99990 -p1 @@ -3582,6 +3654,10 @@ BuildKernel %make_target %kernel_image smp %endif +%if %{with_lustre} +BuildKernel %make_target %kernel_image lustre +%endif + %if %{includexen} %if %{with_xen} BuildKernel %xen_target %xen_image xen @@ -3739,6 +3815,24 @@ /sbin/weak-modules --add-kernel %{KVERREL}smp || exit $? fi +%post lustre +/sbin/new-kernel-pkg --package kernel-lustre --mkinitrd --depmod --install %{KVERREL}lustre || exit $? +if [ -x /sbin/weak-modules ] +then + /sbin/weak-modules --add-kernel %{KVERREL}lustre || exit $? +fi + +%post lustre-devel +if [ -f /etc/sysconfig/kernel ] +then + . /etc/sysconfig/kernel || exit $? +fi +if [ "$HARDLINK" != "no" -a -x /usr/sbin/hardlink ] ; then + pushd /usr/src/kernels/%{KVERREL}-lustre-%{_target_cpu} > /dev/null + /usr/bin/find . -type f | while read f; do hardlink -c /usr/src/kernels/*FC*/$f $f ; done + popd > /dev/null +fi + %post smp-devel if [ -f /etc/sysconfig/kernel ] then @@ -3854,6 +3948,13 @@ /sbin/weak-modules --remove-kernel %{KVERREL}smp || exit $? fi +%preun lustre +/sbin/new-kernel-pkg --rminitrd --rmmoddep --remove %{KVERREL}lustre || exit $? +if [ -x /sbin/weak-modules ] +then + /sbin/weak-modules --remove-kernel %{KVERREL}lustre || exit $? +fi + %preun PAE /sbin/new-kernel-pkg --rminitrd --rmmoddep --remove %{KVERREL}PAE || exit $? if [ -x /sbin/weak-modules ] @@ -4026,6 +4127,49 @@ /usr/src/kernels/%{KVERREL}smp-%{_target_cpu} %endif +%if %{with_lustre} +%if "%{_enable_debug_packages}" == "1" +%ifnarch noarch +%package lustre-debuginfo +Summary: Debug information for package %{name}-lustre +Group: Development/Debug +Requires: %{name}-debuginfo-common-%{_target_cpu} = %{KVERREL} +Provides: %{name}-%lustre-debuginfo-%{_target_cpu} = %{KVERREL} +%description lustre-debuginfo +This package provides debug information for package %{name}-lustre +This is required to use SystemTap with %{name}-lustre-%{KVERREL}. +%files lustre-debuginfo +%defattr(-,root,root) +%if "%{elf_image_install_path}" != "" +/usr/lib/debug/%{elf_image_install_path}/*-%{KVERREL}lustre.debug +%endif +/usr/lib/debug/lib/modules/%{KVERREL}lustre +/usr/lib/debug/usr/src/kernels/%{KVERREL}-lustre-%{_target_cpu} +%endif +%endif + +%files lustre +%defattr(-,root,root) +/%{image_install_path}/vmlinuz-%{KVERREL}lustre +/boot/System.map-%{KVERREL}lustre +/boot/symvers-%{KVERREL}lustre.gz +/boot/config-%{KVERREL}lustre +%dir /lib/modules/%{KVERREL}lustre +/lib/modules/%{KVERREL}lustre/kernel +/lib/modules/%{KVERREL}lustre/build +/lib/modules/%{KVERREL}lustre/source +/lib/modules/%{KVERREL}lustre/extra +/lib/modules/%{KVERREL}lustre/updates +/lib/modules/%{KVERREL}lustre/weak-updates +%ghost /boot/initrd-%{KVERREL}lustre.img +%config(noreplace) /etc/modprobe.d/blacklist-firewire + +%files lustre-devel +%defattr(-,root,root) +%verify(not mtime) /usr/src/kernels/%{KVERREL}-lustre-%{_target_cpu} +/usr/src/kernels/%{KVERREL}lustre-%{_target_cpu} +%endif + %if %{debugbuildsenabled} %if %{with_debug} %if "%{_enable_debug_packages}" == "1"
Brian J. Murrell
2008-Jun-16 14:48 UTC
[Lustre-discuss] Announce: Lustre 1.6.5 is available!
On Mon, 2008-06-16 at 00:25 +0200, Erich Focht wrote:> A related question: is there an easy way of building the Lustre kernel > RPMs for the RHEL 5.2 kernel?Which kernel version is RHEL 5.2? b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080616/c744982b/attachment.bin
Current version of rhel 5.2 is: [root at localhost ~]# rpm -qa | grep kernel kernel-xen-2.6.18-92.1.1.el5 I''m running xen stuff currently... On Mon, Jun 16, 2008 at 7:48 AM, Brian J. Murrell <Brian.Murrell at sun.com> wrote:> On Mon, 2008-06-16 at 00:25 +0200, Erich Focht wrote: >> A related question: is there an easy way of building the Lustre kernel >> RPMs for the RHEL 5.2 kernel? > > Which kernel version is RHEL 5.2? > > b. > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
For those who are asking, I was able to build OFED 1.3.1 against the
1.6.5 release (after some hacking of the process). Considering centos
5.2 (with OFED 1.3) isn''t released yet, someone might find it useful.
Of course no warranties or guarantees :).
http://research.yale.edu/hpc/OFED_1.3.1_lustre_1.6.5.tar.gz
(Redhat 5.1, lustre 1.6.5, x86_64)
By the way, are lustre kernels tested for compatibility with the
latest OFED? This seemed to be more work than usual.
Thanks,
jeff
On Fri, Jun 13, 2008 at 1:44 PM, Brian J. Murrell <Brian.Murrell at
sun.com> wrote:> On Fri, 2008-06-13 at 10:08 -0700, David Brown wrote:
>> Uhm, what happened to the infiniband support in the cfs kernel rpms?
>> 1.6.4.3 had infiniband support but the drivers aren''t there
for the
>> 1.6.5 versions for rhel5.
>
> Starting with 1.6.5 we are supporting the "out-of-[vendor-]tree"
OFED
> drivers. 1.6.5 is supposed to ship with the OFED 1.3 kernel-ib (and
> -devel) RPMs and the Lustre ko2iblnd kernel module built likewise. You
> should have the ko2iblnd module in the lustre-modules RPM. You are just
> missing the needed kernel-ib RPM.
>
> Terry, any idea why the kernel-ib and kernel-ib-devel RPMs didn''t
make
> it up to the SDLC?
>
> b.
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
--
Jeff Blasius / jeff.blasius at yale.edu
Phone: (203)432-9940 51 Prospect Rm. 011
High Performance Computing (HPC)
UNIX Systems Administrator, Linux Systems Design & Support (LSDS)
Yale University Information Technology Services (ITS)
Greenseid, Joseph M.
2008-Jun-17 19:54 UTC
[Lustre-discuss] Announce: Lustre 1.6.5 is available!
Thanks, Jeff. I''ve been hoping to try this, so I will give your bundle
a shot in the morning and let you know how it goes.
--Joe
________________________________
From: lustre-discuss-bounces at lists.lustre.org on behalf of Jeff Blasius
Sent: Mon 6/16/2008 11:06 PM
To: David Brown; Terry Rutledge; lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] Announce: Lustre 1.6.5 is available!
For those who are asking, I was able to build OFED 1.3.1 against the
1.6.5 release (after some hacking of the process). Considering centos
5.2 (with OFED 1.3) isn''t released yet, someone might find it useful.
Of course no warranties or guarantees :).
http://research.yale.edu/hpc/OFED_1.3.1_lustre_1.6.5.tar.gz
(Redhat 5.1, lustre 1.6.5, x86_64)
By the way, are lustre kernels tested for compatibility with the
latest OFED? This seemed to be more work than usual.
Thanks,
jeff
On Fri, Jun 13, 2008 at 1:44 PM, Brian J. Murrell <Brian.Murrell at
sun.com> wrote:> On Fri, 2008-06-13 at 10:08 -0700, David Brown wrote:
>> Uhm, what happened to the infiniband support in the cfs kernel rpms?
>> 1.6.4.3 had infiniband support but the drivers aren''t there
for the
>> 1.6.5 versions for rhel5.
>
> Starting with 1.6.5 we are supporting the "out-of-[vendor-]tree"
OFED
> drivers. 1.6.5 is supposed to ship with the OFED 1.3 kernel-ib (and
> -devel) RPMs and the Lustre ko2iblnd kernel module built likewise. You
> should have the ko2iblnd module in the lustre-modules RPM. You are just
> missing the needed kernel-ib RPM.
>
> Terry, any idea why the kernel-ib and kernel-ib-devel RPMs didn''t
make
> it up to the SDLC?
>
> b.
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
--
Jeff Blasius / jeff.blasius at yale.edu
Phone: (203)432-9940 51 Prospect Rm. 011
High Performance Computing (HPC)
UNIX Systems Administrator, Linux Systems Design & Support (LSDS)
Yale University Information Technology Services (ITS)
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080617/8c7083d9/attachment-0001.html
Install step is: rpm -Uvh --nodeps e2fsprogs-devel-1.40.7.sun3-0redhat.x86_64.rpm rpm -Uvh e2fsprogs-1.40.7.sun3-0redhat.x86_64.rpm cd ../PyXML/ tar -zxvf PyXML-0.8.4.tar.gz cd PyXML-0.8.4 python setup.py build python setup.py install cd ../../Expect rpm -ivh expect-5.42.1-1.src.rpm cd ../1.6.5/ rpm -ivh kernel-lustre-smp-2.6.9-67.0.7.EL_lustre.1.6.5.x86_64.rpm rpm -ivh lustre-ldiskfs-3.0.4-2.6.9_67.0.7.EL_lustre. 1.6.5smp.x86_64.rpm rpm -ivh lustre-modules-1.6.5-2.6.9_67.0.7.EL_lustre. 1.6.5smp.x86_64.rpm when install lustre-modules, it displays warning: WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_create_cq WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_addr WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_dereg_mr WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_reject WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_disconnect WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_route WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_bind_addr WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_create_qp WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_destroy_cq WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_create_id WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_listen WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_qp WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_get_dma_mr WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_alloc_pd WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_connect WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_modify_qp WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_id WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_accept WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_dealloc_pd Please tell me why? Thank you On Fri, Jun 13, 2008 at 7:14 AM, Terry Rutledge <Terry.Rutle... at Sun.COM> wrote:> Hi all, > > At long last, Lustre 1.6.5 is available on the Sun Download Center Site. > > http://www.sun.com/software/products/lustre/get.jsp > > The change log and release notes can be read here: > > http://wiki.lustre.org/index.php?title=Change_Log_1.6 > > Thank you for your assistance; as always, you can report issues via > Bugzilla (https://bugzilla.lustre.org/) > > Happy downloading! > > ?-- The Lustre Team -- > _______________________________________________ > Lustre-discuss mailing list > Lustre-disc... at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discuss
Charles Taylor
2008-Jun-18 10:34 UTC
[Lustre-discuss] Announce: Lustre 1.6.5 is available!
FWIW, we grabbed these and are running them on an AMD 2356 platform box now (just a client). We had tried 1.6.5 + CentOS 5.1 + OFED 1.3 and the node was consistently crashing in the ko2iblnd module (no, I don''t have the stack trace). The RPMs seemed to just be the OFED 1.3.1 RPMS which were useful and installed fine on our 5.1 node but we still had to build a lustre-patched kernel to match the modules and then re-build lustre against the whole thing (OFED 1.3.1 + patched kernel). Is that what you intended? I was hoping the kernel rpm might be in there too. :) So far it is working much better with the OFED 1.3.1 RPMs from Jeff. Thanks, Charlie Taylor UF HPC Center On Jun 17, 2008, at 3:54 PM, Greenseid, Joseph M. wrote:> Thanks, Jeff. I''ve been hoping to try this, so I will give your > bundle a shot in the morning and let you know how it goes. > > --Joe > > From: lustre-discuss-bounces at lists.lustre.org on behalf of Jeff > Blasius > Sent: Mon 6/16/2008 11:06 PM > To: David Brown; Terry Rutledge; lustre-discuss at lists.lustre.org > Subject: Re: [Lustre-discuss] Announce: Lustre 1.6.5 is available! > > For those who are asking, I was able to build OFED 1.3.1 against the > 1.6.5 release (after some hacking of the process). Considering centos > 5.2 (with OFED 1.3) isn''t released yet, someone might find it useful. > Of course no warranties or guarantees :). > > http://research.yale.edu/hpc/OFED_1.3.1_lustre_1.6.5.tar.gz > (Redhat 5.1, lustre 1.6.5, x86_64) > > By the way, are lustre kernels tested for compatibility with the > latest OFED? This seemed to be more work than usual. > > Thanks, > jeff > > On Fri, Jun 13, 2008 at 1:44 PM, Brian J. Murrell <Brian.Murrell at sun.com > > wrote: > > On Fri, 2008-06-13 at 10:08 -0700, David Brown wrote: > >> Uhm, what happened to the infiniband support in the cfs kernel > rpms? > >> 1.6.4.3 had infiniband support but the drivers aren''t there for the > >> 1.6.5 versions for rhel5. > > > > Starting with 1.6.5 we are supporting the "out-of-[vendor-]tree" > OFED > > drivers. 1.6.5 is supposed to ship with the OFED 1.3 kernel-ib (and > > -devel) RPMs and the Lustre ko2iblnd kernel module built > likewise. You > > should have the ko2iblnd module in the lustre-modules RPM. You > are just > > missing the needed kernel-ib RPM. > > > > Terry, any idea why the kernel-ib and kernel-ib-devel RPMs didn''t > make > > it up to the SDLC? > > > > b. > > > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > > > > > -- > Jeff Blasius / jeff.blasius at yale.edu > Phone: (203)432-9940 51 Prospect Rm. 011 > High Performance Computing (HPC) > UNIX Systems Administrator, Linux Systems Design & Support (LSDS) > Yale University Information Technology Services (ITS) > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080618/614b077f/attachment.html
Enrico Morelli
2008-Jun-26 08:39 UTC
[Lustre-discuss] Announce: Lustre 1.6.5 is available!
On Wed, 18 Jun 2008 03:11:38 -0700 (PDT) Johnlya <johnlya at gmail.com> wrote:> Install step is: > rpm -Uvh --nodeps e2fsprogs-devel-1.40.7.sun3-0redhat.x86_64.rpm > rpm -Uvh e2fsprogs-1.40.7.sun3-0redhat.x86_64.rpm > cd ../PyXML/ > tar -zxvf PyXML-0.8.4.tar.gz > cd PyXML-0.8.4 > python setup.py build > python setup.py install > cd ../../Expect > rpm -ivh expect-5.42.1-1.src.rpm > cd ../1.6.5/ > rpm -ivh kernel-lustre-smp-2.6.9-67.0.7.EL_lustre.1.6.5.x86_64.rpm > rpm -ivh lustre-ldiskfs-3.0.4-2.6.9_67.0.7.EL_lustre. > 1.6.5smp.x86_64.rpm > rpm -ivh lustre-modules-1.6.5-2.6.9_67.0.7.EL_lustre. > 1.6.5smp.x86_64.rpm > > when install lustre-modules, it displays warning: > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_create_cq > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_addr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_dereg_mr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_reject > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_disconnect > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_route > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_bind_addr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_create_qp > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_destroy_cq > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_create_id > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_listen > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_qp > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_get_dma_mr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_alloc_pd > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_connect > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_modify_qp > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_id > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_accept > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_dealloc_pd > > Please tell me why? > Thank youBecause, I think, the installation process tries to load the new modules in the current kernel. So you have to reboot the machine using the new kernel.> > > On Fri, Jun 13, 2008 at 7:14 AM, Terry Rutledge > <Terry.Rutle... at Sun.COM> wrote: > > Hi all, > > > > At long last, Lustre 1.6.5 is available on the Sun Download Center > > Site. > > > > http://www.sun.com/software/products/lustre/get.jsp > > > > The change log and release notes can be read here: > > > > http://wiki.lustre.org/index.php?title=Change_Log_1.6 > > > > Thank you for your assistance; as always, you can report issues via > > Bugzilla (https://bugzilla.lustre.org/) > > > > Happy downloading! > > > > ?-- The Lustre Team -- > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-disc... at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- ------------------------------------------------------------------- (o_ (o_ //\ Coltivate Linux che tanto Windows si pianta da solo. (/)_ V_/_ +------------------------------------------------------------------+ | ENRICO MORELLI | email: morelli at CERM.UNIFI.IT | | * * * * | phone: +39 055 4574269 | | University of Florence | fax : +39 055 4574253 | | CERM - via Sacconi, 6 - 50019 Sesto Fiorentino (FI) - ITALY | +------------------------------------------------------------------+
The reason is that the set of rpms published for lustre 1.6.5 on suns website is incomplete. The OFED modules that you get by installing the kernel-ib package are the ones that are mentioned below as missing. It will all still work as long as you only use gigabit ethernet, but when you want to use infiniband, then you are missing the essential bits, and have to either build it yourself or pay SUN or your favorite system integrator to build the kernel-ib rpm for you. Somebody published an inofficial tarball that is supposed to fit with redhat enterprise linux 5.2 but right now I could not find it again by googling. The filename was OFED_1.3.1_lustre_1.6.5.tar.gz and the url to the file from my browser cache is http://research.yale.edu/hpc/OFED_1.3.1_lustre_1.6.5.tar.gz . I have not had a chance to test it yet. A relevant bugzilla entry might be https://bugzilla.lustre.org/show_bug.cgi?id=15316 Michael -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org on behalf of Enrico Morelli Sent: Thu 6/26/2008 1:39 AM To: lustre-discuss at lists.lustre.org Subject: Re: [Lustre-discuss] Announce: Lustre 1.6.5 is available! On Wed, 18 Jun 2008 03:11:38 -0700 (PDT) Johnlya <johnlya at gmail.com> wrote:> Install step is: > rpm -Uvh --nodeps e2fsprogs-devel-1.40.7.sun3-0redhat.x86_64.rpm > rpm -Uvh e2fsprogs-1.40.7.sun3-0redhat.x86_64.rpm > cd ../PyXML/ > tar -zxvf PyXML-0.8.4.tar.gz > cd PyXML-0.8.4 > python setup.py build > python setup.py install > cd ../../Expect > rpm -ivh expect-5.42.1-1.src.rpm > cd ../1.6.5/ > rpm -ivh kernel-lustre-smp-2.6.9-67.0.7.EL_lustre.1.6.5.x86_64.rpm > rpm -ivh lustre-ldiskfs-3.0.4-2.6.9_67.0.7.EL_lustre. > 1.6.5smp.x86_64.rpm > rpm -ivh lustre-modules-1.6.5-2.6.9_67.0.7.EL_lustre. > 1.6.5smp.x86_64.rpm > > when install lustre-modules, it displays warning: > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_create_cq > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_addr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_dereg_mr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_reject > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_disconnect > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_route > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_bind_addr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_create_qp > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_destroy_cq > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_create_id > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_listen > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_qp > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_get_dma_mr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_alloc_pd > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_connect > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_modify_qp > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_id > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_accept > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_dealloc_pd > > Please tell me why? > Thank youBecause, I think, the installation process tries to load the new modules in the current kernel. So you have to reboot the machine using the new kernel.> > > On Fri, Jun 13, 2008 at 7:14 AM, Terry Rutledge > <Terry.Rutle... at Sun.COM> wrote: > > Hi all, > > > > At long last, Lustre 1.6.5 is available on the Sun Download Center > > Site. > > > > http://www.sun.com/software/products/lustre/get.jsp > > > > The change log and release notes can be read here: > > > > http://wiki.lustre.org/index.php?title=Change_Log_1.6 > > > > Thank you for your assistance; as always, you can report issues via > > Bugzilla (https://bugzilla.lustre.org/) > > > > Happy downloading! > > > > ?-- The Lustre Team -- > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-disc... at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- ------------------------------------------------------------------- (o_ (o_ //\ Coltivate Linux che tanto Windows si pianta da solo. (/)_ V_/_ +------------------------------------------------------------------+ | ENRICO MORELLI | email: morelli at CERM.UNIFI.IT | | * * * * | phone: +39 055 4574269 | | University of Florence | fax : +39 055 4574253 | | CERM - via Sacconi, 6 - 50019 Sesto Fiorentino (FI) - ITALY | +------------------------------------------------------------------+ _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080626/5d7cd6e7/attachment.html
Andreas Dilger
2008-Jun-28 07:07 UTC
[Lustre-discuss] Announce: Lustre 1.6.5 is available!
On Jun 26, 2008 08:44 -0700, Michael Will wrote:> when you want to use infiniband, then you are missing the essential bits, > and have to either build it yourself or pay SUN or your favorite system > integrator to build the kernel-ib rpm for you.That isn''t quite true - the omission of the IB support from 1.6.5 was a mistake, and a 1.6.5.1 release with proper IB support/modules is finishing testing and will be released ASAP. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.