Ettore Enrico Delfino Ligorio
2009-May-20 14:42 UTC
[Lustre-discuss] Lustre + XEN in Debian
Hi,
Please, i need help.
Anyone had success using Lustre (from debian package) with xen in
debian (lenny or unstable)? What i must to make this work?
It didn''t worked for me.
I have done the following steps:
puma2:~# apt-get install liblustre linux-patch-lustre lustre-dev
lustre-source lustre-tests lustre-utils ldiskfsck lfsck
puma2:~# module-assistant build lustre
puma2:~# module-assistant install lustre
puma2:~# cat /etc/rc.local |grep lustre
modprobe lustre
puma2:~# reboot
puma2:~# mkfs.lustre --mgs --mdt --fsname puma2 /dev/sda5
Permanent disk data:
Target: puma2-MDTffff
Index: unassigned
Lustre FS: puma2
Mount type: ldiskfs
Flags: 0x75
(MDT MGS needs_index first_time update )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: mdt.group_upcall=/usr/sbin/l_getgroups
checking for existing Lustre data: not found
device size = 262774MB
2 6 26
formatting backing filesystem ldiskfs on /dev/sda5
target name puma2-MDTffff
4k blocks 0
options -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L puma2-MDTffff -J size=400 -i 4096
-I 512 -q -O dir_index,uninit_groups -F /dev/sda5
mkfs.lustre: Unable to mount /dev/sda5: No such device
Is the ldiskfs module available?
mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 19 (No such device)
puma2:~# mkdir /mdt
puma2:~# mount -t lustre /dev/sda5 /mdt/
mount.lustre: mount /dev/sda5 at /mdt failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems
Note ''alias lustre llite'' should be removed from modprobe.conf
puma2:~# fdisk -l /dev/sda
Disk /dev/sda: 299.4 GB, 299439751168 bytes
255 heads, 63 sectors/track, 36404 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00059841
Device Boot Start End Blocks Id System
/dev/sda1 * 1 79 634536 83 Linux
/dev/sda2 80 294 1726987+ 82 Linux swap / Solaris
/dev/sda3 295 2905 20972857+ 83 Linux
/dev/sda4 2906 36404 269080717+ 5 Extended
/dev/sda5 2906 36404 269080686 83 Linux
puma2:~# uname -a
Linux puma2 2.6.26-2-xen-amd64 #1 SMP Wed May 13 18:43:45 UTC 2009
x86_64 GNU/Linux
This is description of booted kernel in /boot/grub/menu.lst
title Xen 3.2-1-amd64 / Debian GNU/Linux, kernel 2.6.26-2-xen-amd64
root (hd0,0)
kernel /xen-3.2-1-amd64.gz
module /vmlinuz-2.6.26-2-xen-amd64 root=/dev/sda3 ro console=tty0
module /initrd.img-2.6.26-2-xen-amd64
puma2:~# lsmod |grep lustre
lustre 446816 0
lov 274056 1 lustre
mdc 102840 1 lustre
osc 149512 1 lustre
ptlrpc 716936 5 lustre,lov,mdc,lquota,osc
obdclass 530776 6 lustre,lov,mdc,lquota,osc,ptlrpc
lnet 233256 4 lustre,ptlrpc,ksocklnd,obdclass
lvfs 71112 7 lustre,lov,mdc,lquota,osc,ptlrpc,obdclass
libcfs 120312 10
lustre,lov,mdc,lquota,osc,ptlrpc,ksocklnd,obdclass,lnet,lvfs
puma2:~# lsmod |grep ldisk
puma2:~#
>From another machine i try, with same configuration:
puma30:~# mkdir /ost
puma30:~# mkfs.lustre --ost --fsname puma2 --mgsnode=puma2 at tcp0 /dev/sda5
Permanent disk data:
Target: puma2-OSTffff
Index: unassigned
Lustre FS: puma2
Mount type: ldiskfs
Flags: 0x72
(OST needs_index first_time update )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=10.0.2.2 at tcp
checking for existing Lustre data: not found
device size = 262774MB
2 6 26
formatting backing filesystem ldiskfs on /dev/sda5
target name puma2-OSTffff
4k blocks 0
options -J size=400 -i 16384 -I 256 -q -O dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L puma2-OSTffff -J size=400 -i 16384
-I 256 -q -O dir_index,uninit_groups -F /dev/sda5
mkfs.lustre: Unable to mount /dev/sda5: No such device
Is the ldiskfs module available?
mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 19 (No such device)
puma30:~# mount -t lustre puma2 at tcp0:/puma2 /ost
mount.lustre: mount puma2 at tcp0:/puma2 at /ost failed: Cannot send
after transport endpoint shutdown
puma30:~#
Thanks in advance.
--
Ettore Enrico Delfino Ligorio
ettoredelfinoligorio at gmail.com
55-11-9145-6151
Hello! On May 20, 2009, at 10:42 AM, Ettore Enrico Delfino Ligorio wrote:> Anyone had success using Lustre (from debian package) with xen in > debian (lenny or unstable)? What i must to make this work? >Show us what is in the dmesg after the failed attempt. Do you happen to have SELinux enabled? (if you do, you have to disable it on the Lustre server nodes, SELinux would prevent the mds/ost mounting because they do not advertise xattr support). Bye, Oleg
Ettore Enrico Delfino Ligorio
2009-May-21 04:14 UTC
[Lustre-discuss] Lustre + XEN in Debian
Hi,
Thank you for the reply.
There is no firewall and no selinux.
Is there any ldiskfs module for debian xen compatible kernel?
Below are the requested output:
(These machines ar using Debian stable (Lenny))
puma2:~# /sbin/iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
puma2:~#
puma2:~# dmesg |tail -n 20
[ 81.829983] NET: Registered protocol family 10
[ 81.829983] lo: Disabled Privacy Extensions
[ 81.829983] ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 84.839690] bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex
[ 84.841266] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[ 85.603393] bnx2: eth0 NIC Copper Link is Down
[ 88.880264] bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
[ 92.755753] eth0: no IPv6 routers present
[ 94.016709] suspend: event channel 44
[ 95.150592] eth1: no IPv6 routers present
[ 96.144998] Bridge firewalling registered
[ 96.776796] Lustre: Added LNI 10.0.2.2 at tcp [8/256]
[ 96.776953] Lustre: Accept secure, port 988
[ 96.881533] Lustre: Lustre Client File System; info at clusterfs.com
[140919.261749] LustreError:
20910:0:(obd_mount.c:1241:server_kernel_mount()) premount
/dev/sda5:0x0 ldiskfs failed: -19, ldiskfs2 failed: -19. Is the
ldiskfs module available?
[140919.261749] LustreError:
20910:0:(obd_mount.c:1560:server_fill_super()) Unable to mount device
/dev/sda5: -19
[140919.261749] LustreError:
20910:0:(obd_mount.c:1951:lustre_fill_super()) Unable to mount (-19)
[190541.769779] LustreError:
27018:0:(obd_mount.c:1241:server_kernel_mount()) premount
/dev/sda5:0x0 ldiskfs failed: -19, ldiskfs2 failed: -19. Is the
ldiskfs module available?
[190541.769779] LustreError:
27018:0:(obd_mount.c:1560:server_fill_super()) Unable to mount device
/dev/sda5: -19
[190541.769779] LustreError:
27018:0:(obd_mount.c:1951:lustre_fill_super()) Unable to mount (-19)
puma2:~#
puma30:~# dmesg |tail -n 10
[140767.809478] LustreError: 15c-8: MGC10.0.2.2 at tcp: The configuration
from log ''puma2-client'' failed (-108). This may be the result
of
communication errors between this node and the MGS, a bad
configuration, or other errors. See the syslog for more information.
[140767.809636] LustreError:
21147:0:(llite_lib.c:1061:ll_fill_super()) Unable to process log: -108
[140767.809955] Lustre: client ffff8803e7693c00 umount complete
[140767.809998] LustreError:
21147:0:(obd_mount.c:1951:lustre_fill_super()) Unable to mount (-108)
[189735.081445] Lustre: Request x16 sent from MGC10.0.2.2 at tcp to NID
10.0.2.2 at tcp 5s ago has timed out (limit 5s).
[189735.081531] LustreError:
27174:0:(client.c:716:ptlrpc_import_delay_req()) @@@ IMP_INVALID
req at ffff8803e5e5f600 x18/t0 o501->MGS at MGC10.0.2.2@tcp_0:26/25 lens
136/248 e 0 to 100 dl 0 ref 1 fl Rpc:/0/0 rc 0/0
[189735.081669] LustreError: 15c-8: MGC10.0.2.2 at tcp: The configuration
from log ''puma2-client'' failed (-108). This may be the result
of
communication errors between this node and the MGS, a bad
configuration, or other errors. See the syslog for more information.
[189735.081794] LustreError:
27174:0:(llite_lib.c:1061:ll_fill_super()) Unable to process log: -108
[189735.082065] Lustre: client ffff8803e7694800 umount complete
[189735.082106] LustreError:
27174:0:(obd_mount.c:1951:lustre_fill_super()) Unable to mount (-108)
puma30:~#
puma2:~# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:22:19:55:28:59
inet addr:10.0.2.2 Bcast:10.0.7.255 Mask:255.255.248.0
inet6 addr: fe80::222:19ff:fe55:2859/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:3570658 errors:0 dropped:0 overruns:0 frame:0
TX packets:128400 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:229098082 (218.4 MiB) TX bytes:8463859 (8.0 MiB)
Interrupt:16 Memory:f8000000-f8012100
puma30:~# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:22:19:58:65:20
inet addr:10.0.2.30 Bcast:10.0.7.255 Mask:255.255.248.0
inet6 addr: fe80::222:19ff:fe58:6520/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:3572109 errors:0 dropped:0 overruns:0 frame:0
TX packets:128662 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:229242725 (218.6 MiB) TX bytes:8539988 (8.1 MiB)
Interrupt:16 Memory:f8000000-f8012100
puma30:~#
(This machine below is running Debian Unstable)
puma58:~# mkfs.lustre --mgs --mdt --fsname puma58 /dev/sda5
Permanent disk data:
Target: puma58-MDTffff
Index: unassigned
Lustre FS: puma58
Mount type: ldiskfs
Flags: 0x75
(MDT MGS needs_index first_time update )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: mdt.group_upcall=/usr/sbin/l_getgroups
checking for existing Lustre data: not found
device size = 264437MB
2 6 26
formatting backing filesystem ldiskfs on /dev/sda5
target name puma58-MDTffff
4k blocks 0
options -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L puma58-MDTffff -J size=400 -i 4096
-I 512 -q -O dir_index,uninit_groups -F /dev/sda5
mkfs.lustre: Unable to mount /dev/sda5: No such device
Is the ldiskfs module available?
mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 19 (No such device)
puma58:~# dmesg |tail -n 25
[ 81.643875] bnx2: eth1 NIC Copper Link is Down
[ 84.827523] bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex
[ 84.948040] eth1: no IPv6 routers present
[ 87.247905] eth0: no IPv6 routers present
[ 91.968742] Bridge firewalling registered
[ 91.965490] tmpbridge: Dropping NETIF_F_UFO since no NETIF_F_HW_CSUM feature.
[ 102.362278] no ownder
[ 102.362383] map irq failed
[ 102.530116] ADDRCONF(NETDEV_UP): peth1: link is not ready
[ 105.012554] bnx2: peth1 NIC Copper Link is Up, 1000 Mbps full duplex
[ 105.013251] ADDRCONF(NETDEV_CHANGE): peth1: link becomes ready
[ 105.574356] device peth1 entered promiscuous mode
[ 105.606375] eth1: port 1(peth1) entering learning state
[ 105.606615] eth1: topology change detected, propagating
[ 105.607468] eth1: port 1(peth1) entering forwarding state
[ 115.864955] peth1: no IPv6 routers present
[ 116.344377] eth1: no IPv6 routers present
[ 138.949463] suspend: event channel 46
[ 142.314790] Lustre: OBD class driver, http://www.lustre.org/
[ 142.314900] Lustre: Lustre Version: 1.6.7
[ 142.315815] Lustre: Build Version:
1.6.7-19691231210000-PRISTINE-.lib.modules.2.6.26-2-xen-amd64.build-2.6.26-2-xen-amd64
[ 142.446536] Lustre: Added LNI 10.0.8.58 at tcp [8/256]
[ 142.446536] Lustre: Accept secure, port 988
[ 142.446536] LustreError:
3721:0:(router_proc.c:1020:lnet_proc_init()) couldn''t create proc
entry sys/lnet/stats
[ 142.680752] Lustre: Lustre Client File System; http://www.lustre.org/
puma58:~#
puma58:~# cat /var/log/messages |tail -n 35
May 20 22:43:21 puma58 kernel: [ 36.549284] EXT3 FS on sda3, internal journal
May 20 22:43:21 puma58 kernel: [ 37.473899] loop: module loaded
May 20 22:43:21 puma58 kernel: [ 37.921432] kjournald starting.
Commit interval 5 seconds
May 20 22:43:21 puma58 kernel: [ 37.925504] EXT3 FS on sda1, internal journal
May 20 22:43:21 puma58 kernel: [ 37.925920] EXT3-fs: mounted
filesystem with ordered data mode.
May 20 22:43:21 puma58 kernel: [ 38.442582] no ownder
May 20 22:43:21 puma58 kernel: [ 38.442682] map irq failed
May 20 22:43:21 puma58 kernel: [ 39.028686] no ownder
May 20 22:43:21 puma58 kernel: [ 39.028785] map irq failed
May 20 22:43:21 puma58 kernel: [ 41.689422] bnx2: eth0 NIC Copper
Link is Up, 1000 Mbps full duplex
May 20 22:43:21 puma58 kernel: [ 42.340784] bnx2: eth1 NIC Copper
Link is Up, 1000 Mbps full duplex
May 20 22:43:21 puma58 kernel: [ 73.968015] NET: Registered protocol family 10
May 20 22:43:21 puma58 kernel: [ 73.968694] lo: Disabled Privacy Extensions
May 20 22:43:21 puma58 kernel: [ 73.969094] ADDRCONF(NETDEV_UP):
eth0: link is not ready
May 20 22:43:21 puma58 kernel: [ 76.917142] bnx2: eth0 NIC Copper
Link is Up, 1000 Mbps full duplex
May 20 22:43:21 puma58 kernel: [ 76.917743] ADDRCONF(NETDEV_CHANGE):
eth0: link becomes ready
May 20 22:43:21 puma58 kernel: [ 84.827523] bnx2: eth1 NIC Copper
Link is Up, 1000 Mbps full duplex
May 20 22:43:21 puma58 rsyslogd: [origin software="rsyslogd"
swVersion="3.22.0" x-pid="3147"
x-info="http://www.rsyslog.com"]
restart
May 20 22:43:28 puma58 kernel: [ 91.968742] Bridge firewalling registered
May 20 22:43:38 puma58 kernel: [ 102.362278] no ownder
May 20 22:43:38 puma58 kernel: [ 102.362383] map irq failed
May 20 22:43:38 puma58 kernel: [ 102.530116] ADDRCONF(NETDEV_UP):
peth1: link is not ready
May 20 22:43:41 puma58 kernel: [ 105.012554] bnx2: peth1 NIC Copper
Link is Up, 1000 Mbps full duplex
May 20 22:43:41 puma58 kernel: [ 105.013251] ADDRCONF(NETDEV_CHANGE):
peth1: link becomes ready
May 20 22:43:41 puma58 kernel: [ 105.574356] device peth1 entered
promiscuous mode
May 20 22:43:42 puma58 kernel: [ 105.606375] eth1: port 1(peth1)
entering learning state
May 20 22:43:42 puma58 kernel: [ 105.606615] eth1: topology change
detected, propagating
May 20 22:43:42 puma58 kernel: [ 105.607468] eth1: port 1(peth1)
entering forwarding state
May 20 22:44:15 puma58 kernel: [ 138.949463] suspend: event channel 46
May 20 22:44:18 puma58 kernel: [ 142.314790] Lustre: OBD class
driver, http://www.lustre.org/
May 20 22:44:18 puma58 kernel: [ 142.314900] Lustre: Lustre
Version: 1.6.7
May 20 22:44:18 puma58 kernel: [ 142.315815] Lustre: Build
Version:
1.6.7-19691231210000-PRISTINE-.lib.modules.2.6.26-2-xen-amd64.build-2.6.26-2-xen-amd64
May 20 22:44:18 puma58 kernel: [ 142.446536] Lustre: Added LNI
10.0.2.58 at tcp [8/256]
May 20 22:44:18 puma58 kernel: [ 142.446536] Lustre: Accept secure, port 988
May 20 22:44:18 puma58 kernel: [ 142.680752] Lustre: Lustre Client
File System; http://www.lustre.org/
Thanks.
--
Ettore Enrico Delfino Ligorio
ettoredelfinoligorio at gmail.com
55-11-9145-6151
On Wed, May 20, 2009 at 5:02 PM, Oleg Drokin <Oleg.Drokin at sun.com>
wrote:> Hello!
>
> On May 20, 2009, at 10:42 AM, Ettore Enrico Delfino Ligorio wrote:
>>
>> Anyone had success using Lustre (from debian package) with xen in
>> debian (lenny or unstable)? What i must to make this work?
>>
>
> Show us what is in the dmesg after the failed attempt.
> Do you happen to have SELinux enabled? (if you do, you have to disable
> it on the Lustre server nodes, SELinux would prevent the mds/ost mounting
> because they do not advertise xattr support).
>
> Bye,
> ? ?Oleg
>