Hi, I am trying to run the lustre lite fs on SLES 9 and running into some problems. I hope someone has succeeded in running it on SLES 9 or SLES8 and can help me. SLES 9 comes with lustre pre-installed. Sorta I guess. I first tried setting up ll as described in the howto on lustre.org, meaning a setup with all elements (client,OST,MDS) on one machine. But I ran into the following problem after making the xml file with lmc : jo@linux:~/Desktop/MyDocuments/lustre> lconf --reformat --gdb local.xml loading module: fsfilt_ext3 srcdir None devdir lvfs ! modprobe (error 1):> FATAL: Could not open''/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko'': No such file or directory jo@linux:~/Desktop/MyDocuments/lustre> and indeed module fsfilt_ext3 was not on my system. So I installed the package km_lustre-lite-1.2.1.13.4-1.8 from the cd''s, which amongst others installs the source code for fsfilt_ext3 in /usr/src/kernel-module/lustre After recompiling my kernel and compiling the modules in /usr/src/kernel-modules/lustre as described by the README''s, I run into another problem : when I run # make install KERNEL_SOURCE-DIR=/usr/src/linux in /usr/src/kernel-modules/lustre to finally install the modules ... module fsfilt_ext3 does NOT end up in /lib/modules/... anywhere. So I added the module MANUALLY to /lib/modules/2.6.5-override-default/kernel/fs/ext3 ... after which a modprobe indeed finds the module, but the insert fails like this : # modprobe fsfilt_ext3 FATAL: Error inserting fsfilt_ext3 (/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko): Unknown symbol in module, or unknown parameter (see dmesg) # dmesg ... fsfilt_ext3: unsupported module, tainting kernel. fsfilt_ext3: Unknown symbol ext3_journal_abort_handle fsfilt_ext3: Unknown symbol ext3_xattr_get fsfilt_ext3: Unknown symbol ext3_force_commit fsfilt_ext3: Unknown symbol ext3_prep_san_write fsfilt_ext3: Unknown symbol ext3_map_inode_page fsfilt_ext3: Unknown symbol ext3_xattr_set_handle fsfilt_ext3: Unknown symbol ext3_bread Now what is weird is that those symbols ARE in /proc/kallsyms ... lot''s of questions ... suggestions anyone? Probably I am doing something fundamentally wrong? Thanks Jo NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be
Ok ok , so the fact that a symbol does appear in /proc/kallsyms does not mean that it is exported. A ''t'' in type = not exported. So ... probably this means that the kernel that comes with SLES 9 does not have all necessary patches for lustre. Will post some answers myself if I find them. Jo On Tue, 10 Aug 2004 16:03:26 +0200 jo@neolabs.be wrote:> Hi again, > > I compiled and installed the modules again with just "make modules" > and "make install" and now the fsfilt_exts is indeed installed in > /lib/modules/2.6.5-7.97-default/kernel/fs/lustre ... > > BUT : if I run the lconf command or just modprobe, I still get the > error messages about missing symbols - although at least some symbols do > seem to be exported by the kernel : > > linux:/lib/modules # modprobe fsfilt_ext3 > FATAL: Error inserting fsfilt_ext3 > (/lib/modules/2.6.5-7.97-default/kernel/fs/lustre/fsfilt_ext3.ko): > Unknown symbol in module, or unknown parameter (see dmesg) > linux:/lib/modules # dmesg > ... > fsfilt_ext3: unsupported module, tainting kernel. > fsfilt_ext3: Unknown symbol ext3_journal_abort_handle > fsfilt_ext3: Unknown symbol ext3_xattr_get > fsfilt_ext3: Unknown symbol ext3_force_commit > fsfilt_ext3: Unknown symbol ext3_prep_san_write > fsfilt_ext3: Unknown symbol ext3_map_inode_page > fsfilt_ext3: Unknown symbol ext3_xattr_set_handle > fsfilt_ext3: Unknown symbol ext3_bread > linux:/lib/modules # cat /proc/kallsyms | grep ext3_prep_san_write > linux:/lib/modules # cat /proc/kallsyms | grep ext3_map_inode_page > linux:/lib/modules # cat /proc/kallsyms | grep ext3_bread > d1978060 t ext3_bread [ext3] > linux:/lib/modules # cat /proc/kallsyms | grep ext3_journal_abort_handle > d197fdd0 t ext3_journal_abort_handle [ext3] > linux:/lib/modules # > > Thanks for any info :-) > > Jo > > > > > > > > > On Tue, 10 Aug 2004 15:26:07 +0200 jo@neolabs.be wrote: > > > Hi, > > > > I am trying to run the lustre lite fs on SLES 9 and running into some > > problems. I hope someone has succeeded in running it on SLES 9 or SLES8 > > and can help me. SLES 9 comes with lustre pre-installed. Sorta > > I guess. > > > > I first tried setting up ll as described in the howto on lustre.org, > > meaning a setup with all elements (client,OST,MDS) on one machine. But I > > ran into the following problem after making the xml file with lmc : > > > > jo@linux:~/Desktop/MyDocuments/lustre> lconf --reformat --gdb local.xml > > loading module: fsfilt_ext3 srcdir None devdir lvfs > > ! modprobe (error 1): > > > FATAL: Could not open > > ''/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko'': No > > such file or directory > > jo@linux:~/Desktop/MyDocuments/lustre> > > > > and indeed module fsfilt_ext3 was not on my system. So I installed the > > package km_lustre-lite-1.2.1.13.4-1.8 from the cd''s, which amongst > others > > installs the source code for fsfilt_ext3 > > in /usr/src/kernel-module/lustre > > > > After recompiling my kernel and compiling the modules in > > /usr/src/kernel-modules/lustre as described by the README''s, I run > > into another problem : when I run > > > > # make install KERNEL_SOURCE-DIR=/usr/src/linux > > > > in /usr/src/kernel-modules/lustre to finally install the modules ... > > module fsfilt_ext3 does NOT end up in /lib/modules/... anywhere. So > > I added the module MANUALLY to > > /lib/modules/2.6.5-override-default/kernel/fs/ext3 ... after which > > a modprobe indeed finds the module, but the insert fails like this : > > > > # modprobe fsfilt_ext3 > > FATAL: Error inserting fsfilt_ext3 > > (/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko): > > Unknown symbol in module, or unknown parameter (see dmesg) > > # dmesg > > ... > > fsfilt_ext3: unsupported module, tainting kernel. > > fsfilt_ext3: Unknown symbol ext3_journal_abort_handle > > fsfilt_ext3: Unknown symbol ext3_xattr_get > > fsfilt_ext3: Unknown symbol ext3_force_commit > > fsfilt_ext3: Unknown symbol ext3_prep_san_write > > fsfilt_ext3: Unknown symbol ext3_map_inode_page > > fsfilt_ext3: Unknown symbol ext3_xattr_set_handle > > fsfilt_ext3: Unknown symbol ext3_bread > > > > > > Now what is weird is that those symbols ARE in /proc/kallsyms > > > > ... lot''s of questions ... suggestions anyone? Probably I am doing > > something fundamentally wrong? > > > > Thanks > > > > Jo > > > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.clusterfs.com > https://lists.clusterfs.com/mailman/listinfo/lustre-discussNEOlabs - http://www.neolabs.be - mailto:info@neolabs.be
Hi Jo-- SuSE didn''t want our ext3 patches in their core ext3 module, so we opted to create a separate "ldiskfs" module just for Lustre. If you''re building your own Lustre tree, be sure to pass "--enable-ldiskfs" to configure. When you make your Lustre config file with lmc, instead of using "--fstype ext3" when you add the MDSs and OSTs, use "--fstype ldiskfs". Thanks for letting us know that our howto is out of date! We''ll update it now that SLES 9 is in the hands of users. Hope that helps-- -Phil On Wed, 2004-08-11 at 03:41, jo@neolabs.be wrote:> Ok ok , > > so the fact that a symbol does appear in /proc/kallsyms does not > mean that it is exported. A ''t'' in type = not exported. > > So ... probably this means that the kernel that comes with SLES 9 > does not have all necessary patches for lustre. Will post some > answers myself if I find them. > > Jo > > > > > > > > > On Tue, 10 Aug 2004 16:03:26 +0200 jo@neolabs.be wrote: > > > Hi again, > > > > I compiled and installed the modules again with just "make modules" > > and "make install" and now the fsfilt_exts is indeed installed in > > /lib/modules/2.6.5-7.97-default/kernel/fs/lustre ... > > > > BUT : if I run the lconf command or just modprobe, I still get the > > error messages about missing symbols - although at least some symbols do > > seem to be exported by the kernel : > > > > linux:/lib/modules # modprobe fsfilt_ext3 > > FATAL: Error inserting fsfilt_ext3 > > (/lib/modules/2.6.5-7.97-default/kernel/fs/lustre/fsfilt_ext3.ko): > > Unknown symbol in module, or unknown parameter (see dmesg) > > linux:/lib/modules # dmesg > > ... > > fsfilt_ext3: unsupported module, tainting kernel. > > fsfilt_ext3: Unknown symbol ext3_journal_abort_handle > > fsfilt_ext3: Unknown symbol ext3_xattr_get > > fsfilt_ext3: Unknown symbol ext3_force_commit > > fsfilt_ext3: Unknown symbol ext3_prep_san_write > > fsfilt_ext3: Unknown symbol ext3_map_inode_page > > fsfilt_ext3: Unknown symbol ext3_xattr_set_handle > > fsfilt_ext3: Unknown symbol ext3_bread > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_prep_san_write > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_map_inode_page > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_bread > > d1978060 t ext3_bread [ext3] > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_journal_abort_handle > > d197fdd0 t ext3_journal_abort_handle [ext3] > > linux:/lib/modules # > > > > Thanks for any info :-) > > > > Jo > > > > > > > > > > > > > > > > > > On Tue, 10 Aug 2004 15:26:07 +0200 jo@neolabs.be wrote: > > > > > Hi, > > > > > > I am trying to run the lustre lite fs on SLES 9 and running into some > > > problems. I hope someone has succeeded in running it on SLES 9 or SLES8 > > > and can help me. SLES 9 comes with lustre pre-installed. Sorta > > > I guess. > > > > > > I first tried setting up ll as described in the howto on lustre.org, > > > meaning a setup with all elements (client,OST,MDS) on one machine. But I > > > ran into the following problem after making the xml file with lmc : > > > > > > jo@linux:~/Desktop/MyDocuments/lustre> lconf --reformat --gdb local.xml > > > loading module: fsfilt_ext3 srcdir None devdir lvfs > > > ! modprobe (error 1): > > > > FATAL: Could not open > > > ''/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko'': No > > > such file or directory > > > jo@linux:~/Desktop/MyDocuments/lustre> > > > > > > and indeed module fsfilt_ext3 was not on my system. So I installed the > > > package km_lustre-lite-1.2.1.13.4-1.8 from the cd''s, which amongst > > others > > > installs the source code for fsfilt_ext3 > > > in /usr/src/kernel-module/lustre > > > > > > After recompiling my kernel and compiling the modules in > > > /usr/src/kernel-modules/lustre as described by the README''s, I run > > > into another problem : when I run > > > > > > # make install KERNEL_SOURCE-DIR=/usr/src/linux > > > > > > in /usr/src/kernel-modules/lustre to finally install the modules ... > > > module fsfilt_ext3 does NOT end up in /lib/modules/... anywhere. So > > > I added the module MANUALLY to > > > /lib/modules/2.6.5-override-default/kernel/fs/ext3 ... after which > > > a modprobe indeed finds the module, but the insert fails like this : > > > > > > # modprobe fsfilt_ext3 > > > FATAL: Error inserting fsfilt_ext3 > > > (/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko): > > > Unknown symbol in module, or unknown parameter (see dmesg) > > > # dmesg > > > ... > > > fsfilt_ext3: unsupported module, tainting kernel. > > > fsfilt_ext3: Unknown symbol ext3_journal_abort_handle > > > fsfilt_ext3: Unknown symbol ext3_xattr_get > > > fsfilt_ext3: Unknown symbol ext3_force_commit > > > fsfilt_ext3: Unknown symbol ext3_prep_san_write > > > fsfilt_ext3: Unknown symbol ext3_map_inode_page > > > fsfilt_ext3: Unknown symbol ext3_xattr_set_handle > > > fsfilt_ext3: Unknown symbol ext3_bread > > > > > > > > > Now what is weird is that those symbols ARE in /proc/kallsyms > > > > > > ... lot''s of questions ... suggestions anyone? Probably I am doing > > > something fundamentally wrong? > > > > > > Thanks > > > > > > Jo > > > > > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > > > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss@lists.clusterfs.com > > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.clusterfs.com > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss
Hi again, I compiled and installed the modules again with just "make modules" and "make install" and now the fsfilt_exts is indeed installed in /lib/modules/2.6.5-7.97-default/kernel/fs/lustre ... BUT : if I run the lconf command or just modprobe, I still get the error messages about missing symbols - although at least some symbols do seem to be exported by the kernel : linux:/lib/modules # modprobe fsfilt_ext3 FATAL: Error inserting fsfilt_ext3 (/lib/modules/2.6.5-7.97-default/kernel/fs/lustre/fsfilt_ext3.ko): Unknown symbol in module, or unknown parameter (see dmesg) linux:/lib/modules # dmesg ... fsfilt_ext3: unsupported module, tainting kernel. fsfilt_ext3: Unknown symbol ext3_journal_abort_handle fsfilt_ext3: Unknown symbol ext3_xattr_get fsfilt_ext3: Unknown symbol ext3_force_commit fsfilt_ext3: Unknown symbol ext3_prep_san_write fsfilt_ext3: Unknown symbol ext3_map_inode_page fsfilt_ext3: Unknown symbol ext3_xattr_set_handle fsfilt_ext3: Unknown symbol ext3_bread linux:/lib/modules # cat /proc/kallsyms | grep ext3_prep_san_write linux:/lib/modules # cat /proc/kallsyms | grep ext3_map_inode_page linux:/lib/modules # cat /proc/kallsyms | grep ext3_bread d1978060 t ext3_bread [ext3] linux:/lib/modules # cat /proc/kallsyms | grep ext3_journal_abort_handle d197fdd0 t ext3_journal_abort_handle [ext3] linux:/lib/modules # Thanks for any info :-) Jo On Tue, 10 Aug 2004 15:26:07 +0200 jo@neolabs.be wrote:> Hi, > > I am trying to run the lustre lite fs on SLES 9 and running into some > problems. I hope someone has succeeded in running it on SLES 9 or SLES8 > and can help me. SLES 9 comes with lustre pre-installed. Sorta > I guess. > > I first tried setting up ll as described in the howto on lustre.org, > meaning a setup with all elements (client,OST,MDS) on one machine. But I > ran into the following problem after making the xml file with lmc : > > jo@linux:~/Desktop/MyDocuments/lustre> lconf --reformat --gdb local.xml > loading module: fsfilt_ext3 srcdir None devdir lvfs > ! modprobe (error 1): > > FATAL: Could not open > ''/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko'': No > such file or directory > jo@linux:~/Desktop/MyDocuments/lustre> > > and indeed module fsfilt_ext3 was not on my system. So I installed the > package km_lustre-lite-1.2.1.13.4-1.8 from the cd''s, which amongst others > installs the source code for fsfilt_ext3 > in /usr/src/kernel-module/lustre > > After recompiling my kernel and compiling the modules in > /usr/src/kernel-modules/lustre as described by the README''s, I run > into another problem : when I run > > # make install KERNEL_SOURCE-DIR=/usr/src/linux > > in /usr/src/kernel-modules/lustre to finally install the modules ... > module fsfilt_ext3 does NOT end up in /lib/modules/... anywhere. So > I added the module MANUALLY to > /lib/modules/2.6.5-override-default/kernel/fs/ext3 ... after which > a modprobe indeed finds the module, but the insert fails like this : > > # modprobe fsfilt_ext3 > FATAL: Error inserting fsfilt_ext3 > (/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko): > Unknown symbol in module, or unknown parameter (see dmesg) > # dmesg > ... > fsfilt_ext3: unsupported module, tainting kernel. > fsfilt_ext3: Unknown symbol ext3_journal_abort_handle > fsfilt_ext3: Unknown symbol ext3_xattr_get > fsfilt_ext3: Unknown symbol ext3_force_commit > fsfilt_ext3: Unknown symbol ext3_prep_san_write > fsfilt_ext3: Unknown symbol ext3_map_inode_page > fsfilt_ext3: Unknown symbol ext3_xattr_set_handle > fsfilt_ext3: Unknown symbol ext3_bread > > > Now what is weird is that those symbols ARE in /proc/kallsyms > > ... lot''s of questions ... suggestions anyone? Probably I am doing > something fundamentally wrong? > > Thanks > > Jo > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.beNEOlabs - http://www.neolabs.be - mailto:info@neolabs.be
got the kernel patched and fsfilt_ext3 gets loaded correctly. Below is the lcm stuff that creates my local.xml file. I have understood that a lov is needed even for a setup with all on a single node so that is what I am trying to do : linux:/home/jo/Desktop/MyDocuments/lustre # more run.sh lmc -o local.xml --add node --node localhost lmc -m local.xml --add net --node localhost --nid localhost --nettype tcp lmc -m local.xml --format --add mds --node localhost --mds mds1 --fstype ext3 --dev /tmp/mds1 --size lmc -m local.xml --add lov --lov lov1 --mds mds1 --stripe_sz 65536 --stripe_cnt 0 --stripe_pattern 0 lmc -m local.xml --add ost --node localhost --lov lov1 --ost ost1 --fstype ext3 --dev /tmp/ost1 --si lmc -m local.xml --add ost --node localhost --lov lov1 --ost ost2 --fstype ext3 --dev /tmp/ost2 --si lmc -m local.xml --add mtpt --node localhost --path /mnt/lustre --mds mds1 --lov lov1 However when I run : # lconf -v --reformat --gdb ./local.xml --node localhost I get the following error : ... The GDB module script is in /tmp/ogdb-linux Service: network NET_localhost_tcp NET_localhost_tcp_UUID Service: ldlm ldlm ldlm_UUID Service: osd OSD_ost2_localhost OSD_ost2_localhost_UUID Service: osd OSD_ost1_localhost OSD_ost1_localhost_UUID Service: mdsdev MDD_mds1_localhost MDD_mds1_localhost_UUID MDSDEV: mds1 mds1_UUID /tmp/mds1 ext3 yes + losetup /dev/loop0 + losetup /dev/loop1 + losetup /dev/loop2 WARNING file: /tmp/mds1 already mapped to /dev/loop2 + /usr/sbin/lctl attach mds mds1 mds1_UUID quit + /usr/sbin/lctl cfg_device mds1 setup /dev/loop2 ext3 <<<<<<< THIS DOES NOT WORK IF I DO IT INTERACTIVELY !!! quit + /usr/sbin/lctl ignore_errors cfg_device $mds1 cleanup detach quit ! /usr/sbin/lctl (22): error: setup: Invalid argument linux:/home/jo/Desktop/MyDocuments/lustre # dmesg gives : ... EXT3-fs: mounted filesystem with ordered data mode. Lustre: 4968:(/usr/src/kernel-modules/lustre-lite/sources/obdfilter/filter.c:367:f ilter_init_server_data()) ost1: initializing new last_rcvd Lustre: 4968:(/usr/src/kernel-modules/lustre-lite/sources/obdfilter/filter.c:425:f ilter_init_server_data()) ost1: recovery support OFF EXT3-fs: Unrecognized mount option "iopen_nopriv" or missing value LustreError: 5051:(/usr/src/kernel-modules/lustre-lite/sources/mds/handler.c:1412:mds_s etup()) do_kern_mount failed: rc = -22 LustreError: 5052:(/usr/src/kernel-modules/lustre-lite/sources/obdclass/obd_config.c:28 6:class_cleanup()) Device 3 not setup eth0: link up, 10Mbps, half-duplex, lpa 0x0000 eth0: no IPv6 routers present EXT3-fs: Unrecognized mount option "iopen_nopriv" or missing value LustreError: 5247:(/usr/src/kernel-modules/lustre-lite/sources/mds/handler.c:1412:mds_s etup()) do_kern_mount failed: rc = -22 LustreError: 5248:(/usr/src/kernel-modules/lustre-lite/sources/obdclass/obd_config.c:28 6:class_cleanup()) Device 3 not setup linux:/home/jo/Desktop/MyDocuments/lustre # Giving up for today - any pointers are more than welcome. CU Jo On Wed, 11 Aug 2004 9:41:19 +0200 jo@neolabs.be wrote:> Ok ok , > > so the fact that a symbol does appear in /proc/kallsyms does not > mean that it is exported. A ''t'' in type = not exported. > > So ... probably this means that the kernel that comes with SLES 9 > does not have all necessary patches for lustre. Will post some > answers myself if I find them. > > Jo > > > > > > > > > On Tue, 10 Aug 2004 16:03:26 +0200 jo@neolabs.be wrote: > > > Hi again, > > > > I compiled and installed the modules again with just "make modules" > > and "make install" and now the fsfilt_exts is indeed installed in > > /lib/modules/2.6.5-7.97-default/kernel/fs/lustre ... > > > > BUT : if I run the lconf command or just modprobe, I still get the > > error messages about missing symbols - although at least some symbols do > > seem to be exported by the kernel : > > > > linux:/lib/modules # modprobe fsfilt_ext3 > > FATAL: Error inserting fsfilt_ext3 > > (/lib/modules/2.6.5-7.97-default/kernel/fs/lustre/fsfilt_ext3.ko): > > Unknown symbol in module, or unknown parameter (see dmesg) > > linux:/lib/modules # dmesg > > ... > > fsfilt_ext3: unsupported module, tainting kernel. > > fsfilt_ext3: Unknown symbol ext3_journal_abort_handle > > fsfilt_ext3: Unknown symbol ext3_xattr_get > > fsfilt_ext3: Unknown symbol ext3_force_commit > > fsfilt_ext3: Unknown symbol ext3_prep_san_write > > fsfilt_ext3: Unknown symbol ext3_map_inode_page > > fsfilt_ext3: Unknown symbol ext3_xattr_set_handle > > fsfilt_ext3: Unknown symbol ext3_bread > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_prep_san_write > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_map_inode_page > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_bread > > d1978060 t ext3_bread [ext3] > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_journal_abort_handle > > d197fdd0 t ext3_journal_abort_handle [ext3] > > linux:/lib/modules # > > > > Thanks for any info :-) > > > > Jo > > > > > > > > > > > > > > > > > > On Tue, 10 Aug 2004 15:26:07 +0200 jo@neolabs.be wrote: > > > > > Hi, > > > > > > I am trying to run the lustre lite fs on SLES 9 and running into some > > > problems. I hope someone has succeeded in running it on SLES 9 or > SLES8 > > > and can help me. SLES 9 comes with lustre pre-installed. Sorta > > > I guess. > > > > > > I first tried setting up ll as described in the howto on lustre.org, > > > meaning a setup with all elements (client,OST,MDS) on one > machine. But I > > > ran into the following problem after making the xml file with lmc : > > > > > > jo@linux:~/Desktop/MyDocuments/lustre> lconf --reformat --gdb > local.xml > > > loading module: fsfilt_ext3 srcdir None devdir lvfs > > > ! modprobe (error 1): > > > > FATAL: Could not open > > > > ''/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko'': No > > > such file or directory > > > jo@linux:~/Desktop/MyDocuments/lustre> > > > > > > and indeed module fsfilt_ext3 was not on my system. So I installed the > > > package km_lustre-lite-1.2.1.13.4-1.8 from the cd''s, which amongst > > others > > > installs the source code for fsfilt_ext3 > > > in /usr/src/kernel-module/lustre > > > > > > After recompiling my kernel and compiling the modules in > > > /usr/src/kernel-modules/lustre as described by the README''s, I run > > > into another problem : when I run > > > > > > # make install KERNEL_SOURCE-DIR=/usr/src/linux > > > > > > in /usr/src/kernel-modules/lustre to finally install the modules ... > > > module fsfilt_ext3 does NOT end up in /lib/modules/... anywhere. So > > > I added the module MANUALLY to > > > /lib/modules/2.6.5-override-default/kernel/fs/ext3 ... after which > > > a modprobe indeed finds the module, but the insert fails like this : > > > > > > # modprobe fsfilt_ext3 > > > FATAL: Error inserting fsfilt_ext3 > > > (/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko): > > > Unknown symbol in module, or unknown parameter (see dmesg) > > > # dmesg > > > ... > > > fsfilt_ext3: unsupported module, tainting kernel. > > > fsfilt_ext3: Unknown symbol ext3_journal_abort_handle > > > fsfilt_ext3: Unknown symbol ext3_xattr_get > > > fsfilt_ext3: Unknown symbol ext3_force_commit > > > fsfilt_ext3: Unknown symbol ext3_prep_san_write > > > fsfilt_ext3: Unknown symbol ext3_map_inode_page > > > fsfilt_ext3: Unknown symbol ext3_xattr_set_handle > > > fsfilt_ext3: Unknown symbol ext3_bread > > > > > > > > > Now what is weird is that those symbols ARE in /proc/kallsyms > > > > > > ... lot''s of questions ... suggestions anyone? Probably I am doing > > > something fundamentally wrong? > > > > > > Thanks > > > > > > Jo > > > > > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > > > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss@lists.clusterfs.com > > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.beNEOlabs - http://www.neolabs.be - mailto:info@neolabs.be
On 2004-08-11T09:41:19, jo@neolabs.be said:> So ... probably this means that the kernel that comes with SLES 9 > does not have all necessary patches for lustre. Will post some > answers myself if I find them.What you should do instead is use the ldiskfs (shipped with the kernel) instead of the ext3 one as the backing store. Sincerely, Lars Marowsky-Brée <lmb@suse.de> -- High Availability & Clustering \ Philosophy proclaiming reason to be SUSE Labs, Research and Development | the supreme human virtue is falling SUSE LINUX AG - A Novell company \ prey to self-adulation.
Martin,
I was having lots of problems with the e1000 driver and recv.
I fixed things by adding these options to the driver:
options e1000 RxDescriptors=4096,4096,4096
InterruptThrottleRate=80000,80000,80000
I also enabled NAPI. So if your switch is really the problem then this
won''t help.
Originally we thought our switch was bad too but this was not the case.
Hope this helps..
paul
Martin Vogt wrote:
> jo@neolabs.be wrote:
>
>> My problem was the following :
>>
>> I blindly followed the instructions in the howto - and as a result
>> I tried to set up everything with ''ext3''. But Suse
don''t want to put
>> changes in ext3, so the lustre guys made a new "filesystem"
for Suse :
>> ldiskfs. This is problably just ext3 + the patches.
>>
>> So I just replaced all occurences of ''ext3'' by
''ldiksfs'' while
>> setting up the fs and all went well.
>>
>>
>>
>
> Hello,
>
> I made it with ldiskfs:
>
>
> for example:
> #- add OSTs
> ${LMC} --add ost --node lustre2 --lov media-lov --ost media-ost1
> --dev /dev/sda
> 4 --fstype ldiskfs --mkfsoptions "-T largefile4"
>
> But I had lockups from client/server machines. It was a bit better
> when using
> an nosmp kernel and using current e1000 drivers. I even check newer
> 3w-xxxx
> drivers, but in most cases the system crashed (silently, no errors)
> I''m currently thinking that the real problem in my setup maybe the
> switch, which
> drops Rx packets.
>
> regards,
>
> Martin
>
>
>
>
>>
>>
>> On Fri, 20 Aug 2004 17:11:55 +0200 Martin Vogt wrote:
>>
>>
>>
>>>
>>>
>>>> / Hi,
>>>>
>>>
>>> />/ />/ I am trying to run the lustre lite fs on SLES 9 and
running
>>> into some
>>> />/ problems. I hope someone has succeeded in running it on SLES
9 /
>>>
>>> Hello,
>>>
>>>
>>> I tried it too with SLES9, but I had no luck.
>>>
>>> I''m currently trying to build a stable lustre system, but
currently
>>> with no success. Its very fast, but at least on
>>> my setup, not for very long :-)
>>>
>>> regards,
>>>
>>> Martin
>>>
>>>
>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss@lists.clusterfs.com
>>> https://lists.clusterfs.com/mailman/listinfo/lustre-discuss
>>>
>>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@lists.clusterfs.com
> https://lists.clusterfs.com/mailman/listinfo/lustre-discuss
PAulN wrote:> Martin, > I was having lots of problems with the e1000 driver and recv. > I fixed things by adding these options to the driver: > options e1000 RxDescriptors=4096,4096,4096 > InterruptThrottleRate=80000,80000,80000 > > I also enabled NAPI. So if your switch is really the problem then > this won''t help. > Originally we thought our switch was bad too but this was not the case. > Hope this helps.. > paul >Hello, my attempt was the following: >options e1000 FlowControl=3,3 RxIntDelay=32,32,32 This made it better, but did not solved it completely. On thing: There are different chip versions for the e1000, which have different default configs. (according to the intel www) Martin
I think the RxDescriptors flag is the important one. By default the driver only allows for 256.. if too many packets come in before they can be processed then unprocessed data is overwritten. You should give it a try :) p Martin Vogt wrote:> PAulN wrote: > >> Martin, >> I was having lots of problems with the e1000 driver and recv. >> I fixed things by adding these options to the driver: >> options e1000 RxDescriptors=4096,4096,4096 >> InterruptThrottleRate=80000,80000,80000 >> >> I also enabled NAPI. So if your switch is really the problem then >> this won''t help. >> Originally we thought our switch was bad too but this was not the case. >> Hope this helps.. >> paul >> > > Hello, > > my attempt was the following: > > >options e1000 FlowControl=3,3 RxIntDelay=32,32,32 > > This made it better, but did not solved it completely. > On thing: There are different chip versions for the e1000, which have > different default configs. (according to the intel www) > > Martin >
Hi guys, thank you all!!! Am going to try that today. So if I understand it correctly ldiskfs = ext3 + the patches, right? Makes sense. Well ... at least I got to have some fun yesterday applying the patches from the km-lustre... rpm to the core ext3 module. Got pretty "far" as fsfilt_ext3 was finally loadable. If you still feel like it, maybe one of you can take a look at my last mail yesterday to see if "what finally went wrong" was due to my modding the kernel code or rather due to a typo somewhere in my lustre setup. Error messages are below. I have the feeling maybe you know what is the cause, since "/usr/sbin/lctl (22): error: setup: Invalid argument " seems to be a regular error message. Only if you immediately see the cause :-) - don''t waste more than 30 secs. Last mail is pasted below. PS what does "make prepare / make prepare-all" do ??? Have not found this in the new kernel howto or on google yet. Still must dig a bit deeper in kernel package docs but if there is a simple explanation : shoot. I know it prepares the kernel tree for extra module compilation (as without it - the compile of the lustre modules did not work *grin*). Is this 2.6 specific? Just point me to a doc if you know one and I''ll be fine. Sorry for my many questions! Thanks again. Jo BELOW IS LAST ATTEMPT WITH MANUALLY PATCHED CORE EXT3 : got the kernel patched and fsfilt_ext3 gets loaded correctly. Below is the lcm stuff that creates my local.xml file. I have understood that a lov is needed even for a setup with all on a single node so that is what I am trying to do : linux:/home/jo/Desktop/MyDocuments/lustre # more run.sh lmc -o local.xml --add node --node localhost lmc -m local.xml --add net --node localhost --nid localhost --nettype tcp lmc -m local.xml --format --add mds --node localhost --mds mds1 --fstype ext3 --dev /tmp/mds1 --size lmc -m local.xml --add lov --lov lov1 --mds mds1 --stripe_sz 65536 --stripe_cnt 0 --stripe_pattern 0 lmc -m local.xml --add ost --node localhost --lov lov1 --ost ost1 --fstype ext3 --dev /tmp/ost1 --si lmc -m local.xml --add ost --node localhost --lov lov1 --ost ost2 --fstype ext3 --dev /tmp/ost2 --si lmc -m local.xml --add mtpt --node localhost --path /mnt/lustre --mds mds1 --lov lov1 However when I run : # lconf -v --reformat --gdb ./local.xml --node localhost I get the following error : ... The GDB module script is in /tmp/ogdb-linux Service: network NET_localhost_tcp NET_localhost_tcp_UUID Service: ldlm ldlm ldlm_UUID Service: osd OSD_ost2_localhost OSD_ost2_localhost_UUID Service: osd OSD_ost1_localhost OSD_ost1_localhost_UUID Service: mdsdev MDD_mds1_localhost MDD_mds1_localhost_UUID MDSDEV: mds1 mds1_UUID /tmp/mds1 ext3 yes + losetup /dev/loop0 + losetup /dev/loop1 + losetup /dev/loop2 WARNING file: /tmp/mds1 already mapped to /dev/loop2 + /usr/sbin/lctl attach mds mds1 mds1_UUID quit + /usr/sbin/lctl cfg_device mds1 setup /dev/loop2 ext3 <<<<<<< THIS DOES NOT WORK IF I DO IT INTERACTIVELY !!! quit + /usr/sbin/lctl ignore_errors cfg_device $mds1 cleanup detach quit ! /usr/sbin/lctl (22): error: setup: Invalid argument linux:/home/jo/Desktop/MyDocuments/lustre # dmesg gives : ... EXT3-fs: mounted filesystem with ordered data mode. Lustre: 4968:(/usr/src/kernel-modules/lustre-lite/sources/obdfilter/filter.c:367:f ilter_init_server_data()) ost1: initializing new last_rcvd Lustre: 4968:(/usr/src/kernel-modules/lustre-lite/sources/obdfilter/filter.c:425:f ilter_init_server_data()) ost1: recovery support OFF EXT3-fs: Unrecognized mount option "iopen_nopriv" or missing value LustreError: 5051:(/usr/src/kernel-modules/lustre-lite/sources/mds/handler.c:1412:mds_s etup()) do_kern_mount failed: rc = -22 LustreError: 5052:(/usr/src/kernel-modules/lustre-lite/sources/obdclass/obd_config.c:28 6:class_cleanup()) Device 3 not setup eth0: link up, 10Mbps, half-duplex, lpa 0x0000 eth0: no IPv6 routers present EXT3-fs: Unrecognized mount option "iopen_nopriv" or missing value LustreError: 5247:(/usr/src/kernel-modules/lustre-lite/sources/mds/handler.c:1412:mds_s etup()) do_kern_mount failed: rc = -22 LustreError: 5248:(/usr/src/kernel-modules/lustre-lite/sources/obdclass/obd_config.c:28 6:class_cleanup()) Device 3 not setup linux:/home/jo/Desktop/MyDocuments/lustre # On Wed, 11 Aug 2004 19:18:08 -0400 Phil Schwan wrote:> Hi Jo-- > > SuSE didn''t want our ext3 patches in their core ext3 module, so we opted > to create a separate "ldiskfs" module just for Lustre. > > If you''re building your own Lustre tree, be sure to pass > "--enable-ldiskfs" to configure. > > When you make your Lustre config file with lmc, instead of using > "--fstype ext3" when you add the MDSs and OSTs, use "--fstype ldiskfs". > > Thanks for letting us know that our howto is out of date! We''ll update > it now that SLES 9 is in the hands of users. > > Hope that helps-- > > -Phil > > On Wed, 2004-08-11 at 03:41, jo@neolabs.be wrote: > > Ok ok , > > > > so the fact that a symbol does appear in /proc/kallsyms does not > > mean that it is exported. A ''t'' in type = not exported. > > > > So ... probably this means that the kernel that comes with SLES 9 > > does not have all necessary patches for lustre. Will post some > > answers myself if I find them. > > > > Jo > > > > > > > > > > > > > > > > > > On Tue, 10 Aug 2004 16:03:26 +0200 jo@neolabs.be wrote: > > > > > Hi again, > > > > > > I compiled and installed the modules again with just "make modules" > > > and "make install" and now the fsfilt_exts is indeed installed in > > > /lib/modules/2.6.5-7.97-default/kernel/fs/lustre ... > > > > > > BUT : if I run the lconf command or just modprobe, I still get the > > > error messages about missing symbols - although at least some > symbols do > > > seem to be exported by the kernel : > > > > > > linux:/lib/modules # modprobe fsfilt_ext3 > > > FATAL: Error inserting fsfilt_ext3 > > > (/lib/modules/2.6.5-7.97-default/kernel/fs/lustre/fsfilt_ext3.ko): > > > Unknown symbol in module, or unknown parameter (see dmesg) > > > linux:/lib/modules # dmesg > > > ... > > > fsfilt_ext3: unsupported module, tainting kernel. > > > fsfilt_ext3: Unknown symbol ext3_journal_abort_handle > > > fsfilt_ext3: Unknown symbol ext3_xattr_get > > > fsfilt_ext3: Unknown symbol ext3_force_commit > > > fsfilt_ext3: Unknown symbol ext3_prep_san_write > > > fsfilt_ext3: Unknown symbol ext3_map_inode_page > > > fsfilt_ext3: Unknown symbol ext3_xattr_set_handle > > > fsfilt_ext3: Unknown symbol ext3_bread > > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_prep_san_write > > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_map_inode_page > > > linux:/lib/modules # cat /proc/kallsyms | grep ext3_bread > > > d1978060 t ext3_bread [ext3] > > > linux:/lib/modules # cat /proc/kallsyms | grep > ext3_journal_abort_handle > > > d197fdd0 t ext3_journal_abort_handle [ext3] > > > linux:/lib/modules # > > > > > > Thanks for any info :-) > > > > > > Jo > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 10 Aug 2004 15:26:07 +0200 jo@neolabs.be wrote: > > > > > > > Hi, > > > > > > > > I am trying to run the lustre lite fs on SLES 9 and running > into some > > > > problems. I hope someone has succeeded in running it on SLES 9 > or SLES8 > > > > and can help me. SLES 9 comes with lustre pre-installed. Sorta > > > > I guess. > > > > > > > > I first tried setting up ll as described in the howto on lustre.org, > > > > meaning a setup with all elements (client,OST,MDS) on one > machine. But I > > > > ran into the following problem after making the xml file with lmc : > > > > > > > > jo@linux:~/Desktop/MyDocuments/lustre> lconf --reformat --gdb > local.xml > > > > loading module: fsfilt_ext3 srcdir None devdir lvfs > > > > ! modprobe (error 1): > > > > > FATAL: Could not open > > > > > ''/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko'': No > > > > such file or directory > > > > jo@linux:~/Desktop/MyDocuments/lustre> > > > > > > > > and indeed module fsfilt_ext3 was not on my system. So I > installed the > > > > package km_lustre-lite-1.2.1.13.4-1.8 from the cd''s, which amongst > > > others > > > > installs the source code for fsfilt_ext3 > > > > in /usr/src/kernel-module/lustre > > > > > > > > After recompiling my kernel and compiling the modules in > > > > /usr/src/kernel-modules/lustre as described by the README''s, I run > > > > into another problem : when I run > > > > > > > > # make install KERNEL_SOURCE-DIR=/usr/src/linux > > > > > > > > in /usr/src/kernel-modules/lustre to finally install the modules ... > > > > module fsfilt_ext3 does NOT end up in /lib/modules/... anywhere. So > > > > I added the module MANUALLY to > > > > /lib/modules/2.6.5-override-default/kernel/fs/ext3 ... after which > > > > a modprobe indeed finds the module, but the insert fails like this : > > > > > > > > # modprobe fsfilt_ext3 > > > > FATAL: Error inserting fsfilt_ext3 > > > > (/lib/modules/2.6.5-override-default/kernel/fs/ext3/fsfilt_ext3.ko): > > > > Unknown symbol in module, or unknown parameter (see dmesg) > > > > # dmesg > > > > ... > > > > fsfilt_ext3: unsupported module, tainting kernel. > > > > fsfilt_ext3: Unknown symbol ext3_journal_abort_handle > > > > fsfilt_ext3: Unknown symbol ext3_xattr_get > > > > fsfilt_ext3: Unknown symbol ext3_force_commit > > > > fsfilt_ext3: Unknown symbol ext3_prep_san_write > > > > fsfilt_ext3: Unknown symbol ext3_map_inode_page > > > > fsfilt_ext3: Unknown symbol ext3_xattr_set_handle > > > > fsfilt_ext3: Unknown symbol ext3_bread > > > > > > > > > > > > Now what is weird is that those symbols ARE in /proc/kallsyms > > > > > > > > ... lot''s of questions ... suggestions anyone? Probably I am doing > > > > something fundamentally wrong? > > > > > > > > Thanks > > > > > > > > Jo > > > > > > > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > > > > > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > > > _______________________________________________ > > > Lustre-discuss mailing list > > > Lustre-discuss@lists.clusterfs.com > > > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss > > > > NEOlabs - http://www.neolabs.be - mailto:info@neolabs.be > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss@lists.clusterfs.com > > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.clusterfs.com > https://lists.clusterfs.com/mailman/listinfo/lustre-discussNEOlabs - http://www.neolabs.be - mailto:info@neolabs.be
>/ Hi,/>/ />/ I am trying to run the lustre lite fs on SLES 9 and running into some />/ problems. I hope someone has succeeded in running it on SLES 9 / Hello, I tried it too with SLES9, but I had no luck. I''m currently trying to build a stable lustre system, but currently with no success. Its very fast, but at least on my setup, not for very long :-) regards, Martin
jo@neolabs.be wrote:>My problem was the following : > >I blindly followed the instructions in the howto - and as a result >I tried to set up everything with ''ext3''. But Suse don''t want to put >changes in ext3, so the lustre guys made a new "filesystem" for Suse : >ldiskfs. This is problably just ext3 + the patches. > >So I just replaced all occurences of ''ext3'' by ''ldiksfs'' while setting up >the fs and all went well. > > >Hello, I made it with ldiskfs: for example: #- add OSTs ${LMC} --add ost --node lustre2 --lov media-lov --ost media-ost1 --dev /dev/sda 4 --fstype ldiskfs --mkfsoptions "-T largefile4" But I had lockups from client/server machines. It was a bit better when using an nosmp kernel and using current e1000 drivers. I even check newer 3w-xxxx drivers, but in most cases the system crashed (silently, no errors) I''m currently thinking that the real problem in my setup maybe the switch, which drops Rx packets. regards, Martin> > >On Fri, 20 Aug 2004 17:11:55 +0200 Martin Vogt wrote: > > > >> >> >>>/ Hi, >>> >>> >>/>/ >>/>/ I am trying to run the lustre lite fs on SLES 9 and running into some >>/>/ problems. I hope someone has succeeded in running it on SLES 9 / >> >>Hello, >> >> >>I tried it too with SLES9, but I had no luck. >> >>I''m currently trying to build a stable lustre system, but >>currently with no success. Its very fast, but at least on >>my setup, not for very long :-) >> >>regards, >> >>Martin >> >> >> >> >>_______________________________________________ >>Lustre-discuss mailing list >>Lustre-discuss@lists.clusterfs.com >>https://lists.clusterfs.com/mailman/listinfo/lustre-discuss >> >>