leen smit
2010-Jun-16 14:04 UTC
[Lustre-discuss] Can''t use any lfs commands on clients...
Dear All,
I have just successfully setup a Lustre enviroment. Every thing seems
just A-OK so far. But...
After completing some performance test with iozone and now wanting to
see the performance difference with striping enabled I noticed that I
cant use any lfs commands on the client side.
Digging further, I noticed lctl is not showing any nids, on the client
I went through the logs on the OSS''es, nothing noteworthy in there, so
the only loggin I have is coming from the client, and is not very
helpful to me so I''m hoping some one here can point at some flaw - or
at
least provide pointers on what to check...I guessing that the fact im
not seeing any nids is already a bad start, but I don''t know how I can
add one (the nid is listed on the mgs/oss though..)
Any help would be appreciated!
Leen
Outputs (if more info is needed I''ll be glad to provide it!)
Client:~# mount
..
192.168.21.32 at tcp0:192.168.21.33 at tcp:/datafs on /data type lustre
(rw,_netdev,noatime)
Client:~# lctl device_list
0 UP mgc MGC192.168.21.32 at tcp 2367fdc6-ef75-f1fc-b1cb-611c46a0ced3 5
1 UP lov datafs-clilov-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 4
2 UP mdc datafs-MDT0000-mdc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
3 UP osc datafs-OST0000-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
4 UP osc datafs-OST0001-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
5 UP osc datafs-OST0002-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
6 UP osc datafs-OST0003-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
7 UP osc datafs-OST0004-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
8 UP osc datafs-OST0005-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
9 UP osc datafs-OST0006-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
10 UP osc datafs-OST0007-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
11 UP osc datafs-OST0008-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
12 UP osc datafs-OST0009-osc-ffff810037802400
951384b1-8456-c2f3-575e-4926f70919c9 5
Client:~# lctl list_nids
IOC_LIBCFS_GET_NI error 22: Invalid argument
Client:~# lctl list_nids
IOC_LIBCFS_GET_NI error 22: Invalid argument
prd-as-022:~# lfs
lfs > osts
error: can''t get lov name: /data: Invalid argument (22)
lfs > df
UUID 1K-blocks Used Available Use% Mounted on
error: llapi_obd_statfs(): Invalid argument (-22)
lfs > getstripe /data
error: can''t get lov name: /data: Invalid argument (22)
lfs >
syslog excerpt:
Jun 16 15:26:49 prd-as-022 kernel: Lustre: MGC192.168.21.32 at tcp:
Reactivating import
Jun 16 15:26:49 prd-as-022 kernel: Lustre: Client datafs-client has started
Jun 16 15:26:57 prd-as-022 kernel: ioctl32(lfs:4645): Unknown cmd fd(4)
cmd(c00466a4){t:''f'';sz:4} arg(ffb0a34c) on /data
Jun 16 15:27:54 prd-as-022 kernel: ioctl32(lfs:4646): Unknown cmd fd(4)
cmd(c00466a4){t:''f'';sz:4} arg(ff90294c) on /data
Jun 16 15:28:44 prd-as-022 kernel: ioctl32(lfs:4648): Unknown cmd fd(4)
cmd(c00466a4){t:''f'';sz:4} arg(ffd925dc) on /data
Jun 16 15:29:00 prd-as-022 kernel: ioctl32(lfs:4649): Unknown cmd fd(4)
cmd(c00466a4){t:''f'';sz:4} arg(fff32f2c) on /data
Jun 16 15:29:38 prd-as-022 kernel: ioctl32(lfs:4649): Unknown cmd fd(3)
cmd(c0046683){t:''f'';sz:4} arg(fff37190) on /data
Jun 16 15:29:46 prd-as-022 kernel: ioctl32(lfs:4649): Unknown cmd fd(3)
cmd(c0046683){t:''f'';sz:4} arg(fff37190) on /data
Jun 16 15:31:07 prd-as-022 kernel: ioctl32(lfs:4650): Unknown cmd fd(4)
cmd(c0046683){t:''f'';sz:4} arg(ffcb6750) on /data
Jun 16 15:48:17 prd-as-022 kernel: ioctl32(lctl:4672): Unknown cmd fd(3)
cmd(c0046532){t:''e'';sz:4} arg(fffd23ec) on /dev/lnet
Jun 16 15:51:48 prd-as-022 kernel: ioctl32(lfs:4674): Unknown cmd fd(4)
cmd(c0046683){t:''f'';sz:4} arg(ffc556f0) on /data
Jun 16 15:51:52 prd-as-022 kernel: ioctl32(lfs:4674): Unknown cmd fd(4)
cmd(c00466a4){t:''f'';sz:4} arg(ffc5144c) on /data
Jun 16 15:52:04 prd-as-022 kernel: ioctl32(lfs:4674): Unknown cmd fd(3)
cmd(c0046683){t:''f'';sz:4} arg(ffc556b0) on /data
OST''s created with
fs-ost-002:~# mkfs.lustre --ost --mgsnode=192.168.21.32 at tcp
--mgsnode=192.168.21.33 at tcp --failnode=192.168.21.34 at tcp
--fsname=datafs /dev/VG1/ost1
Permanent disk data:
Target: datafs-OSTffff
Index: unassigned
Lustre FS: datafs
Mount type: ldiskfs
Flags: 0x72
(OST needs_index first_time update )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=192.168.21.32 at tcp mgsnode=192.168.21.33 at tcp
failover.node=192.168.21.34 at tcp
checking for existing Lustre data: not found
device size = 10240MB
2 6 22
WARNING: The e2fsprogs package currently installed on your system does
not support "mmp" feature.
Please install the latest version of e2fsprogs from
http://www.clusterfs.com/downloads/public/Lustre/Tools/e2fsprogs/
to enable this feature.
Feature will not be enabled until e2fsprogs is updated and ''tune2fs -O
mmp %{device}'' is run.
formatting backing filesystem ldiskfs on /dev/VG1/ost1
target name datafs-OSTffff
4k blocks 0
options -J size=400 -i 16384 -I 256 -q -O
dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L datafs-OSTffff -J size=400 -i 16384
-I 256 -q -O dir_index,uninit_groups -F /dev/VG1/ost1
Writing CONFIGS/mountdata
On OSS:
fs-ost-001:~# lctl list_nids
192.168.21.34 at tcp
fs-mgs-001:~# lctl list_nids
192.168.21.32 at tcp
Andreas Dilger
2010-Jun-16 20:08 UTC
[Lustre-discuss] Can''t use any lfs commands on clients...
On 2010-06-16, at 08:04, leen smit wrote:> After completing some performance test with iozone and now wanting to > see the performance difference with striping enabled I noticed that I > cant use any lfs commands on the client side.> Jun 16 15:26:57 prd-as-022 kernel: ioctl32(lfs:4645): Unknown cmd fd(4) > cmd(c00466a4){t:''f'';sz:4} arg(ffb0a34c) on /dataDo you have 32-bit userspace running on a 64-bit kernel? We have a problem with the IOC numbers not being correctly defined and so the userspace tools need to match the kernel. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.