While performing an LFSCK after upgrade lustre-1.6.6, the MDS claims
to have 512 OSTs:
MDS: num_osts = 512
I am in the middle of an update, in a currently asymmetric heartbeat
config, with a passive MDS and OSS on lustre-1.6.5.1, and the
currently active MDS and OSS (where num_osts comes out wrong) on
lustre-1.6.6.
The rest of the e2fsck output looks perfectly normal and in line with
previous runs. I run two lustre file systems, /home and /sandbox off
the same MGS, and e2fsck for BOTH file systems reports 512 OSTs.
Running e2fsck on the 1.6.5.1 servers gives the same blown up OST list.
Does this mean my MGS is trashed?
In context:
=
=
=
=
=======================================================================# e2fsck
-n -v --mdsdb /tmp/mdsdb-sandbox /dev/dm-2; date
e2fsck 1.40.11.sun1 (17-June-2008)
device /dev/dm-2 mounted by lustre per /proc/fs/lustre/mds/sandbox-
MDT0000/mntdev
Warning! /dev/dm-2 is mounted.
Warning: skipping journal recovery because doing a read-only
filesystem check.
sandbox-MDT0000 has been mounted 47 times without being checked, check
forced.
Pass 1: Checking inodes, blocks, and sizes
MDS: ost_idx 0 max_id 1192499
MDS: ost_idx 1 max_id 0
MDS: ost_idx 2 max_id 0
MDS: ost_idx 3 max_id 0
MDS: ost_idx 4 max_id 0
...
MDS: ost_idx 509 max_id 0
MDS: ost_idx 510 max_id 0
MDS: ost_idx 511 max_id 0
MDS: got 4096 bytes = 512 entries in lov_objids
MDS: max_files = 36420
MDS: num_osts = 512
mds info db file written
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Pass 6: Acquiring information for lfsck
MDS: max_files = 36420
MDS: num_osts = 512
MDS: ''sandbox-MDT0000_UUID'' mdt idx 0: compat 0x4 rocomp 0x1
incomp 0x4
36420 inodes used (0.05%)
3 non-contiguous inodes (0.0%)
# of inodes with ind/dind/tind blocks: 1/0/0
9283672 blocks used (12.67%)
0 bad blocks
1 large file
32592 regular files
3818 directories
0 character device files
0 block device files
0 fifos
1 link
1 symbolic link (1 fast symbolic link)
0 sockets
--------
36412 files
=
=
=
=
=======================================================================
I did an LFSCK just prior to the upgrade, which showed num_osts = 1.
This is on RHEL-5.2 with the latest e2fsprogs:
# e2fsck -V
e2fsck 1.40.11.sun1 (17-June-2008)
Using EXT2FS Library version 1.40.11.sun1, 17-June-2008
I run two MDS and OSS each in heartbeat; I encountered an LBUG during
a heartbeat hiccup (output appended).
Kernels for the updated and active MDS and OSS, and some clients are:
2.6.18-92.1.10.el5_lustre.1.6.6smp
the passive MDS and OSS, and remaining clients are:
2.6.18-53.1.14.el5_lustre.1.6.5.1smp
Regards, Michael
Jan 2 15:41:20 mds02 kernel: LustreError: 14500:0:(mgs_handler.c:
194:mgs_setup()) ASSERTION(!lvfs_check_rdonly(lvfs_sbdev(mnt-
>mnt_sb))) failed
Jan 2 15:41:20 mds02 kernel: LustreError: 14500:0:(tracefile.c:
450:libcfs_assertion_failed()) LBUG
Jan 2 15:41:20 mds02 kernel: Lustre: 14500:0:(linux-debug.c:
185:libcfs_debug_dumpstack()) showing stack for process 14500
Jan 2 15:41:20 mds02 kernel: mount.lustre R running task 0
14500 14499 (NOTLB)
Jan 2 15:41:20 mds02 kernel: 0000000000000040 0000000000000020
ffff8102105a5568 ffffffff88966410
Jan 2 15:41:20 mds02 kernel: ffffffff888d3b80 ffff810200b242c0
0000000000000000 0000000000000000
Jan 2 15:41:20 mds02 kernel: 0000000000000000 0000000000000000
ffff8102105a5598 ffffffff80143a09
Jan 2 15:41:20 mds02 kernel: Call Trace:
Jan 2 15:41:20 mds02 kernel: [<ffffffff8006b499>] dump_trace
+0x211/0x23a
Jan 2 15:41:20 mds02 kernel:
[<ffffffff88966410>] :ptlrpc:lprocfs_rd_pool_state+0x0/0x200
Jan 2 15:41:20 mds02 kernel:
[<ffffffff888d3b80>] :obdclass:lprocfs_wr_atomic+0x0/0x60
Jan 2 15:41:20 mds02 kernel: [<ffffffff8006b4f6>] show_trace+0x34/0x47
Jan 2 15:41:20 mds02 kernel: [<ffffffff8006b5fb>] _show_stack+0xdb/
0xea
Jan 2 15:41:20 mds02 kernel:
[<ffffffff8882ac2a>] :libcfs:lbug_with_loc+0x7a/0xc0
Jan 2 15:41:20 mds02 kernel:
[<ffffffff88832874>] :libcfs:libcfs_assertion_failed+0x54/0x60
Jan 2 15:41:20 mds02 kernel: [<ffffffff88beab81>] :mgs:mgs_setup
+0x301/0x800
Jan 2 15:41:20 mds02 kernel:
[<ffffffff888dff12>] :obdclass:class_setup+0x942/0xc70
Jan 2 15:41:20 mds02 kernel:
[<ffffffff888e263d>] :obdclass:class_process_config+0x14bd/0x19e0
Jan 2 15:41:20 mds02 kernel: [<ffffffff888ec524>] :obdclass:do_lcfg
+0x924/0xb20
Jan 2 15:41:20 mds02 kernel:
[<ffffffff888ee190>] :obdclass:lustre_start_simple+0x130/0x1d0
Jan 2 15:41:20 mds02 kernel:
[<ffffffff888f1bb3>] :obdclass:server_start_mgs+0x223/0x320
Jan 2 15:41:20 mds02 OpenSM[14448]: Entering MASTER state
Jan 2 15:41:20 mds02 kernel:
[<ffffffff888f3552>] :obdclass:server_fill_super+0x18a2/0x1fb0
Jan 2 15:41:21 mds02 kernel:
[<ffffffff886d5ac7>] :sunrpc:rpc_call_sync+0x9e/0xa8
Jan 2 15:41:21 mds02 kernel: [<ffffffff800963e7>] recalc_sigpending
+0xe/0x25
Jan 2 15:41:21 mds02 attrd: [14276]: info: main: Starting mainloop...
Jan 2 15:41:21 mds02 kernel: [<ffffffff80009516>] __d_lookup+0xb0/0xff
Jan 2 15:41:21 mds02 OpenSM[14448]: SUBNET UP
Jan 2 15:41:21 mds02 crmd: [14277]: notice: populate_cib_nodes: Node:
mds02 (uuid: f03a4080-8623-4049-a725-723ff0995fe2)
Jan 2 15:41:21 mds02 kernel: [<ffffffff8882b178>] :libcfs:cfs_alloc
+0x28/0x60
Jan 2 15:41:21 mds02 crmd: [14277]: notice: populate_cib_nodes: Node:
mds01 (uuid: a25627aa-9f96-486f-8f17-4ef11b62dc69)
Jan 2 15:41:21 mds02 kernel:
[<ffffffff888e7693>] :obdclass:lustre_init_lsi+0x263/0x4c0
Jan 2 15:41:21 mds02 kernel:
[<ffffffff888f51b3>] :obdclass:lustre_fill_super+0x1553/0x16d0
Jan 2 15:41:21 mds02 kernel: [<ffffffff800e3aab>] get_filesystem
+0x12/0x3b
Jan 2 15:41:21 mds02 kernel: [<ffffffff800dc008>] set_anon_super
+0x0/0xab
Jan 2 15:41:21 mds02 kernel:
[<ffffffff888f3c60>] :obdclass:lustre_fill_super+0x0/0x16d0
Jan 2 15:41:21 mds02 kernel: [<ffffffff800dc273>] get_sb_nodev+0x4f/
0x97
Jan 2 15:41:21 mds02 kernel: [<ffffffff800dbb5a>] vfs_kern_mount
+0x93/0x11a
Jan 2 15:41:21 mds02 kernel: [<ffffffff800dbc23>] do_kern_mount
+0x36/0x4d
Jan 2 15:41:21 mds02 kernel: [<ffffffff800e53ca>] do_mount+0x68c/0x6fc
Jan 2 15:41:21 mds02 kernel: [<ffffffff80008b39>] __handle_mm_fault
+0x4ea/0xe17
Jan 2 15:41:21 mds02 kernel: [<ffffffff80021f02>] __up_read+0x19/0x7f
Jan 2 15:41:21 mds02 kernel: [<ffffffff80066858>] do_page_fault
+0x4fe/0x830
Jan 2 15:41:21 mds02 kernel: [<ffffffff80008b39>] __handle_mm_fault
+0x4ea/0xe17
Jan 2 15:41:21 mds02 kernel: [<ffffffff800c6029>] zone_statistics
+0x3e/0x6d
Jan 2 15:41:21 mds02 kernel: [<ffffffff8000f04d>] __alloc_pages+0x5c/
0x2c5
Jan 2 15:41:21 mds02 kernel: [<ffffffff8004bb8b>] sys_mount+0x8a/0xcd
Jan 2 15:41:21 mds02 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Jan 2 15:41:21 mds02 kernel:
Jan 2 15:41:21 mds02 kernel: LustreError: dumping log to /tmp/lustre-
log.1230932480.14500
I wrote: On Jan 2, 2009, at 19:24 , Michael Sternberg wrote:> While performing an LFSCK after upgrade lustre-1.6.6, the MDS claims > to have 512 OSTs [while there is only 1]: > > MDS: num_osts = 512Well, I found an interesting section "How to fix bad LAST_ID on an OST" in the operations manual, Appendix D, which pointed more or less in the right direction. Briefly, /lov_objid within the MDS ldiskfs is constructed from /O/0/ LAST_ID in all the OSTs'' ldiskfs. In my case, with only one OST, the first and only entry agrees between these two files, and the rest of lov_objid was padded with NULs to 4KB (block size?). I do not understand what caused this, but, given prior e2fsck output indicating length 8, I decided to simply chop off the NULs, which indeed made e2fsck happy. lfsck on a client found a handful of empty orphans, but a backup on the client side still matches the contents of the rest of the fs (phew!). Regards, Michael
On Jan 04, 2009 14:25 -0600, Michael Sternberg wrote:> On Jan 2, 2009, at 19:24 , Michael Sternberg wrote: > > While performing an LFSCK after upgrade lustre-1.6.6, the MDS claims > > to have 512 OSTs [while there is only 1]: > > > > MDS: num_osts = 512 > > > Well, I found an interesting section "How to fix bad LAST_ID on an > OST" in the operations manual, Appendix D, which pointed more or less > in the right direction. > > Briefly, /lov_objid within the MDS ldiskfs is constructed from /O/0/ > LAST_ID in all the OSTs'' ldiskfs. In my case, with only one OST, the > first and only entry agrees between these two files, and the rest of > lov_objid was padded with NULs to 4KB (block size?). I do not > understand what caused this, but, given prior e2fsck output indicating > length 8, I decided to simply chop off the NULs, which indeed made > e2fsck happy. lfsck on a client found a handful of empty orphans, but > a backup on the client side still matches the contents of the rest of > the fs (phew!).As a further note - lfsck is not really a required operation. Lustre will normally handle the problems that lfsck would fix, with the exception of corruption that results in two inodes referencing the same object (which can also be fixed by hand if necessary). Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.