DT Piotr Wadas
2009-Dec-31 21:08 UTC
[Lustre-discuss] client and ost on the same machine, quota
Hello,
Using lustre 1.8.1.1 build from source, with lustre-client-source.
I''ve read warnings in lustre manual about running client and ost on
the same maching, well, seems fine until I''ll keep lustre client
read-only
from time-to-time, just to access some particular data in special
ocasions.
Anyway it''s not the point - something I''like to ask is how is
quota
and some lctl subcommands related to such situation - I noticed
the following, no matter whether local lustre client is read-only, rw
or whatever, also makes no difference whether I use 1.8.1/1.8.1.1
server/client builds ( I tested various combinations with various
versions on server and client, this is experimental environment).
aleft:~# lctl lustre_build_version
error: lustre_build_version: Invalid argument
^^^^^^^^^^^^^^^^^^^^^^^^^
lctl version:
1.8.1.1-20091009163626-PRISTINE-2.6.27.29-0.1_lustre.1.8.1.1-default
This is what appears on messages log:
[ 4259.594339] ioctl32(lctl:24118): Unknown cmd fd(3)
cmd(c0046690){t:''f'';sz:4} arg(ff8cdef0) on /dev/obd
While using the same command on external lustre client, I got
client:~# lctl lustre_build_version
Lustre version:
1.8.1-20090725062217-PRISTINE-.lib.modules.2.6.27.23.source-2.6.27.23
lctl version: UNKNOWN-20090725042217-PRISTINE--UNKNOWN-KERNEL
client:~#
The same I got when I try to create quota files with
lfs quotacheck -ug /mnt/lfs00
command - I''m able to do this from remote client on the same fs,
but unable with local client (rw mounted).
[ 1700.798389] ioctl32(lctl:23023): Unknown cmd fd(3)
cmd(c0046690){t:''f'';sz:4} arg(ffc2fe70) on /dev/obd
[ 2842.829452] ioctl32(lfs:23409): Unknown cmd fd(3)
cmd(c00466a2){t:''f'';sz:4} arg(ffc0a740) on /mnt/lfs00
[ 4075.661809] ioctl32(lctl:23729): Unknown cmd fd(3)
cmd(c004653b){t:''e'';sz:4} arg(fffc9930) on /dev/lnet
[ 4106.129849] ioctl32(lctl:23729): Unknown cmd fd(3)
cmd(c0046532){t:''e'';sz:4} arg(fffc9930) on /dev/lnet
[ 4119.216135] ioctl32(lctl:23735): Unknown cmd fd(3)
cmd(c0046532){t:''e'';sz:4} arg(fff0eee0) on /dev/lnet
[ 4259.594339] ioctl32(lctl:24118): Unknown cmd fd(3)
cmd(c0046690){t:''f'';sz:4} arg(ff8cdef0) on /dev/obd
[ 5414.655030] ioctl32(lctl:24228): Unknown cmd fd(3)
cmd(c0046690){t:''f'';sz:4} arg(ff819780) on /dev/obd
[ 5425.431048] ioctl32(lctl:24231): Unknown cmd fd(3)
cmd(c0046690){t:''f'';sz:4} arg(fff5f2b0) on /dev/obd
My /etc/modprobe.d/lustre.conf is
aleft:~# cat /etc/modprobe.d/lustre.conf
options lnet networks=tcp0(eth3,lo)
aleft:~#
Except this issue with some strange behaviour accessing lustre
fs from local client, everything seems to work fine.
Regards, happy new year''s eve!
DT
--
http://dtpw.pl/buell [ 25th anniversary of Buell - American Motorcycles ]
Linux aleft 2.6.27.29-0.1_lustre.1.8.1.1-default #1 SMP
drbd 8.3.6-(api:88/proto:86-91)
pacemaker 1.0.6-cebe2b6ff49b36b29a3bd7ada1c4701c7470febe
DT Piotr Wadas
2010-Jan-02 11:50 UTC
[Lustre-discuss] client and ost on the same machine, quota
On Thu, 31 Dec 2009, DT Piotr Wadas wrote:> The same I got when I try to create quota files with > lfs quotacheck -ug /mnt/lfs00 > command - I''m able to do this from remote client on the same fs, > but unable with local client (rw mounted). > > [ 1700.798389] ioctl32(lctl:23023): Unknown cmd fd(3) > cmd(c0046690){t:''f'';sz:4} arg(ffc2fe70) on /dev/obd > [ 2842.829452] ioctl32(lfs:23409): Unknown cmd fd(3) > cmd(c00466a2){t:''f'';sz:4} arg(ffc0a740) on /mnt/lfs00 > [ 4075.661809] ioctl32(lctl:23729): Unknown cmd fd(3) > cmd(c004653b){t:''e'';sz:4} arg(fffc9930) on /dev/lnet > [ 4106.129849] ioctl32(lctl:23729): Unknown cmd fd(3) > cmd(c0046532){t:''e'';sz:4} arg(fffc9930) on /dev/lnet > [ 4119.216135] ioctl32(lctl:23735): Unknown cmd fd(3) > cmd(c0046532){t:''e'';sz:4} arg(fff0eee0) on /dev/lnet > [ 4259.594339] ioctl32(lctl:24118): Unknown cmd fd(3) > cmd(c0046690){t:''f'';sz:4} arg(ff8cdef0) on /dev/obd > [ 5414.655030] ioctl32(lctl:24228): Unknown cmd fd(3) > cmd(c0046690){t:''f'';sz:4} arg(ff819780) on /dev/obd > [ 5425.431048] ioctl32(lctl:24231): Unknown cmd fd(3) > cmd(c0046690){t:''f'';sz:4} arg(fff5f2b0) on /dev/obd > > My /etc/modprobe.d/lustre.conf is > > aleft:~# cat /etc/modprobe.d/lustre.conf > options lnet networks=tcp0(eth3,lo) > aleft:~# > > Except this issue with some strange behaviour accessing lustre > fs from local client, everything seems to work fine.Regarding Andreas Dilger''s suggestion about using 32bit tools on 64bit kernel - tools were build and make-installed on the same machine, actually after booting to 64bit.. I did an strace on lctl lustre_build_version command, here''s significant part (I guess) read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@\n\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0644, st_size=9736, ...}) = 0 mmap2(NULL, 12408, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xf7d9d000 mmap2(0xf7d9f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0xf7d9f000 close(3) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xf7d9c000 set_thread_area({entry_number:-1 -> 12, base_addr:0xf7d9c8d0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 open("/dev/urandom", O_RDONLY) = 3 read(3, "\"c\6\340", 4) = 4 close(3) = 0 mprotect(0xf7d9f000, 4096, PROT_READ) = 0 mprotect(0xf7efe000, 8192, PROT_READ) = 0 mprotect(0x809e000, 4096, PROT_READ) = 0 mprotect(0xf7f98000, 4096, PROT_READ) = 0 munmap(0xf7f70000, 34125) = 0 shmget(IPC_PRIVATE, 16468, 0600) = 229376 shmat(229376, 0, 0) = 0xf7f74000 shmctl(229376, IPC_64|IPC_RMID, 0) = 0 brk(0) = 0x9ca9000 brk(0x9cca000) = 0x9cca000 open("/dev/obd", O_RDWR) = 3 ioctl(3, 0xc0046690, 0xffc6b58c) = -1 EINVAL (Invalid argument) write(2, "error: lustre_build_version: Inv"..., 46error: lustre_build_version: Invalid argument ) = 46 I tried various kernel sources available on sun download site, still the same. DT