murray smigel
2008-May-23 18:27 UTC
[Lustre-discuss] problems getting o2ib device recognized
Hi, I am running Centos5 with 2.6.18-8.1.14.el5_lustre.1.6.4.1 as the OS. I built the custom kernel against the patched source and built lustre-1.6.1 (and 1.6.3) against the source tree as: ./configure --with-linux=/usr/src/linux-2.6.18-8.1.14.el5_lustre.1.6.4.1 --with-o2ib=/public/ofed/SRC_1.3/ofa_kernel-1.3/ The OFED source is stock 1.3. The hardware is x86-64. ---------------------------------------------------------------------------- Problems in the build: needed to edit lnet/klnds/o2iblnd/o2iblnd.h to add: #define DECLARE_PCI_UNMAP_ADDR(ADDR_NAME) dma_addr_t ADDR_NAME; needed to edit lnet/klnds/o2iblnd/o2iblnd.c to add 0 as last argument: cq = ib_create_cq(cmid->device, kiblnd_cq_completion, kiblnd_cq_event, conn, IBLND_CQ_ENTRIES(), 0); // added by me. in ldiskfs/ldiskfs/linux-stage/fs/ext3/iopen.c I had to comment out call to d_move_locked For some unknown reason I have to modprobe -f ko2iblnd since there is some sort of version conflict, but it does go in: [root at lusty lustre-1.6.4.1]# lsmod | grep ko2iblnd ko2iblnd 137504 0 rdma_cm 67348 1 ko2iblnd ib_core 97664 6 ko2iblnd,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad lnet 252840 4 ko2iblnd,ptlrpc,obdclass,ksocklnd libcfs 189656 13 ko2iblnd,mds,fsfilt_ldiskfs,mgs,mgc,lov,lquota,mdc,ptlrpc,obdclass,lvfs, ksocklnd,lnet -------------------------------------------------------------------------- After these changes the lustre modules built and installed. I edit /etc/modprobe.conf to have options lnet ''networks="tcp(eth0,eth1),o2ib(ib0)"'' I can modprobe in lnet, but I only see the tcp device [root at lusty lustre-1.6.4.1]# lctl list_nids 192.168.1.94 at tcp The lustre setup works over tcp, but that is not really what I need! Any help appreciated, thanks, murray