Install step is: rpm -Uvh --nodeps e2fsprogs-devel-1.40.7.sun3-0redhat.x86_64.rpm rpm -Uvh e2fsprogs-1.40.7.sun3-0redhat.x86_64.rpm cd ../PyXML/ tar -zxvf PyXML-0.8.4.tar.gz cd PyXML-0.8.4 python setup.py build python setup.py install cd ../../Expect rpm -ivh expect-5.42.1-1.src.rpm cd ../1.6.5/ rpm -ivh kernel-lustre-smp-2.6.9-67.0.7.EL_lustre.1.6.5.x86_64.rpm rpm -ivh lustre-ldiskfs-3.0.4-2.6.9_67.0.7.EL_lustre. 1.6.5smp.x86_64.rpm rpm -ivh lustre-modules-1.6.5-2.6.9_67.0.7.EL_lustre. 1.6.5smp.x86_64.rpm when install lustre-modules, it displays warning: WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_create_cq WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_addr WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_dereg_mr WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_reject WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_disconnect WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_route WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_bind_addr WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_create_qp WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_destroy_cq WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_create_id WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_listen WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_qp WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_get_dma_mr WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_alloc_pd WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_connect WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_modify_qp WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_id WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol rdma_accept WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ lustre/ko2iblnd.ko needs unknown symbol ib_dealloc_pd Please tell me why? Thank you
Lustre doesn''t know where your ib modules symbols are. When you configured lustre (in the build sense) you pointed it to a patched kernel tree. In that directory is a Module.symvers file devoid of ib module symbols. You should also have a Module.symvers in your /usr/ src/ofa_kernel directory (assuming you built OFED as well). So... cat /usr/src/ofa_kernel/Module.symvers >> <patched_kernel_dir>/ Module.symvers and run "make install" again and it should be happy. For a 2.6.9 kernel, you probably need OFED 1.2. Charlie Taylor UF HPC Center On Jun 18, 2008, at 5:55 AM, Johnlya wrote:> Install step is: > rpm -Uvh --nodeps e2fsprogs-devel-1.40.7.sun3-0redhat.x86_64.rpm > rpm -Uvh e2fsprogs-1.40.7.sun3-0redhat.x86_64.rpm > cd ../PyXML/ > tar -zxvf PyXML-0.8.4.tar.gz > cd PyXML-0.8.4 > python setup.py build > python setup.py install > cd ../../Expect > rpm -ivh expect-5.42.1-1.src.rpm > cd ../1.6.5/ > rpm -ivh kernel-lustre-smp-2.6.9-67.0.7.EL_lustre.1.6.5.x86_64.rpm > rpm -ivh lustre-ldiskfs-3.0.4-2.6.9_67.0.7.EL_lustre. > 1.6.5smp.x86_64.rpm > rpm -ivh lustre-modules-1.6.5-2.6.9_67.0.7.EL_lustre. > 1.6.5smp.x86_64.rpm > > when install lustre-modules, it displays warning: > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_create_cq > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_addr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_dereg_mr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_reject > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_disconnect > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_route > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_bind_addr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_create_qp > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_destroy_cq > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_create_id > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_listen > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_qp > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_get_dma_mr > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_alloc_pd > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_connect > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_modify_qp > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_id > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol rdma_accept > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_dealloc_pd > > Please tell me why? > Thank you > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
On Jun 19, 2008, at 6:19 PM, Charles Taylor <tay... at hpc.ufl.edu> wrote:> Lustre doesn''t know where your ib modules symbols are. ? When you ? > configured lustre (in the build sense) you pointed it to a patched ? > kernel tree. ? In that directory is a Module.symvers file devoid of ib ? > module symbols. ?You should also have a Module.symvers in your /usr/ > src/ofa_kernel directory (assuming you built OFED as well). ? So... > > cat /usr/src/ofa_kernel/Module.symvers >> <patched_kernel_dir>/ > Module.symvers > > and run "make install" again and it should be happy. ? For a 2.6.9 ? > kernel, you probably need OFED 1.2. > > Charlie Taylor > UF HPC Center > > On Jun 18, 2008, at 5:55 AM, Johnlya wrote: > > > > > > > Install step is: > > rpm -Uvh --nodeps e2fsprogs-devel-1.40.7.sun3-0redhat.x86_64.rpm > > rpm -Uvh e2fsprogs-1.40.7.sun3-0redhat.x86_64.rpm > > cd ../PyXML/ > > tar -zxvf ?PyXML-0.8.4.tar.gz > > cd PyXML-0.8.4 > > python setup.py build > > python setup.py install > > cd ../../Expect > > rpm -ivh expect-5.42.1-1.src.rpm > > cd ../1.6.5/ > > rpm -ivh kernel-lustre-smp-2.6.9-67.0.7.EL_lustre.1.6.5.x86_64.rpm > > rpm -ivh lustre-ldiskfs-3.0.4-2.6.9_67.0.7.EL_lustre. > > 1.6.5smp.x86_64.rpm > > rpm -ivh lustre-modules-1.6.5-2.6.9_67.0.7.EL_lustre. > > 1.6.5smp.x86_64.rpm > > > when install lustre-modules, it displays warning: > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol ib_create_cq > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_addr > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol ib_dereg_mr > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_reject > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_disconnect > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_resolve_route > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_bind_addr > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_create_qp > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol ib_destroy_cq > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_create_id > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_listen > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_qp > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol ib_get_dma_mr > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol ib_alloc_pd > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_connect > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol ib_modify_qp > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_destroy_id > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol rdma_accept > > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > > lustre/ko2iblnd.ko needs unknown symbol ib_dealloc_pd > > > Please tell me why? > > Thank you > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-disc... at lists.lustre.org > >http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-disc... at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discuss- >If don''t install ib modules and don''t build OFED, Does Lustre work well. Do these warning affect Lustre working? I used Lustre 1.6.4.2 (OS kernel 2.6.9.-42 64bit) and it work well. Do I need to change it to ver 1.6.5. Thank you!
On Wed, 2008-06-18 at 02:55 -0700, Johnlya wrote:> Install step is: > rpm -Uvh --nodeps e2fsprogs-devel-1.40.7.sun3-0redhat.x86_64.rpm > rpm -Uvh e2fsprogs-1.40.7.sun3-0redhat.x86_64.rpm > cd ../PyXML/ > tar -zxvf PyXML-0.8.4.tar.gz > cd PyXML-0.8.4 > python setup.py build > python setup.py install > cd ../../Expect > rpm -ivh expect-5.42.1-1.src.rpm > cd ../1.6.5/ > rpm -ivh kernel-lustre-smp-2.6.9-67.0.7.EL_lustre.1.6.5.x86_64.rpm > rpm -ivh lustre-ldiskfs-3.0.4-2.6.9_67.0.7.EL_lustre. > 1.6.5smp.x86_64.rpm > rpm -ivh lustre-modules-1.6.5-2.6.9_67.0.7.EL_lustre. > 1.6.5smp.x86_64.rpm > > when install lustre-modules, it displays warning: > WARNING: /lib/modules/2.6.9-67.0.7.EL_lustre.1.6.5smp/kernel/net/ > lustre/ko2iblnd.ko needs unknown symbol ib_create_cqThis is an artifact of the lustre-modules RPM now having a ko2iblnd module, which needs the OFED modules, from kernel-ib against which it was (ko2iblnd) built. If you are not using IB with OFED you can safely ignore these warnings. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080619/e12b3c3a/attachment.bin
> This is an artifact of the lustre-modules RPM now having a ko2iblnd > module, which needs the OFED modules, from kernel-ib against which it > was (ko2iblnd) built. > > If you are not using IB with OFED you can safely ignore these warnings. > > b.Is there any word on where we can get the 1.6.5 kernel-ib package? I still haven''t been able to find it anywhere on the download site. --Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080619/c63d0780/attachment-0001.html
On Thu, 2008-06-19 at 12:47 -0500, Greenseid, Joseph M. wrote:> Is there any word on where we can get the 1.6.5 kernel-ib package? I > still haven''t been able to find it anywhere on the download site.1.6.5.1 which is in QA will correct this. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080620/bce456ec/attachment.bin
On Fri, 2008-06-20 at 9:51 PM, "Brian J. Murrell" <Brian.Murr... at Sun.COM> wrote:> On Thu, 2008-06-19 at 12:47 -0500, Greenseid, Joseph M. wrote: > > Is there any word on where we can get the 1.6.5 kernel-ib package? ?I > > still haven''t been able to find it anywhere on the download site. > > 1.6.5.1 which is in QA will correct this. > > b. > > ?signature.asc > 1K download > > _______________________________________________ > Lustre-discuss mailing list > Lustre-disc... at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discussMDS space is reduced after formating (mkfs.lustre --fsname=test --mdt --mgs --reformat --failnode=MDS_SLAVER /dev/sdd1). MDS disk space is 2 T. And formating is taked 1.5 hour. When I start the MDS, OSS and CLIENT. It works well. I use command "lfs df -h " to check disk space and find MDS space reduced. MDS space is 1.7T.