Lustre discuss - Jun 2008 - LBUG when mounting MDT

Hello,

I see the attached LBUG when trying to mount the MDT.
From what I understand this actually means that some llog
delivers bad data. A filesystem check revealed many errors
on the MDT, and the same LBUG when trying to mount.

Is there anything I could still try to do? Remove or rewrite
configuration, maybe extract with debugfs what is still visible,
reformat?

Regards,
Erich


Jun 13 15:44:36 dmds2 kernel: LustreError:
18697:0:(mds_log.c:207:mds_llog_init()) lov_llog_init err -2
Jun 13 15:44:36 dmds2 kernel: LustreError:
18697:0:(llog_obd.c:392:llog_cat_initialize()) rc: -2
Jun 13 15:44:36 dmds2 kernel: LustreError: 18697:0:(llog_obd.c:134:llog_setup())
ASSERTION(ctxt->loc_obd == obd) failed
Jun 13 15:44:36 dmds2 kernel: LustreError:
18697:0:(tracefile.c:431:libcfs_assertion_failed()) LBUG
Jun 13 15:44:36 dmds2 kernel: Lustre:
18697:0:(linux-debug.c:168:libcfs_debug_dumpstack()) showing stack for process
18697
Jun 13 15:44:36 dmds2 kernel: mount.lustre  R  running task       0 18697  18696
(NOTLB)
Jun 13 15:44:36 dmds2 kernel:  ffff810001004400 00000006801595ba
ffff810001004d58 ffff810212a565a8
Jun 13 15:44:36 dmds2 kernel:  ffff81022f94d820 ffffffff80087c53
ffff810001004400 ffff81022f94d820
Jun 13 15:44:36 dmds2 kernel:  ffff810212a565c8 ffffffff80087c8f
0000020994a64e83 0000000000000006
Jun 13 15:44:36 dmds2 kernel: Call Trace:
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80087c53>] enqueue_task+0x41/0x56
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80087c8f>]
__activate_task+0x27/0x39
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80044b12>]
try_to_wake_up+0x407/0x418
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8009b3c6>]
autoremove_wake_function+0x9/0x2e
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80087c53>] enqueue_task+0x41/0x56
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80087c8f>]
__activate_task+0x27/0x39
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80087c53>] enqueue_task+0x41/0x56
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80087c8f>]
__activate_task+0x27/0x39
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80044b12>]
try_to_wake_up+0x407/0x418
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80087c53>] enqueue_task+0x41/0x56
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80087c8f>]
__activate_task+0x27/0x39
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80044b12>]
try_to_wake_up+0x407/0x418
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8009b3c6>]
autoremove_wake_function+0x9/0x2e
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800868b0>]
__wake_up_common+0x3e/0x68
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8002de82>] __wake_up+0x38/0x4f
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8008d439>] vprintk+0x29e/0x2ea
Jun 13 15:44:36 dmds2 last message repeated 2 times
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8008d4d7>] printk+0x52/0xbd
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8008d4d7>] printk+0x52/0xbd
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800a395c>]
kallsyms_expand_symbol+0x42/0x63
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800a3fdb>]
kallsyms_lookup+0xc2/0x17b
Jun 13 15:44:36 dmds2 last message repeated 3 times
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800695c7>]
printk_address+0x9f/0xab
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8008d4d7>] printk+0x52/0xbd
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800a04ed>]
module_text_address+0x33/0x3b
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8009999d>]
kernel_text_address+0x1a/0x26
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800692ad>] dump_trace+0x211/0x23a
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80019de8>] vsnprintf+0x559/0x59e
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8006930a>] show_trace+0x34/0x47
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8006940f>] _show_stack+0xdb/0xea
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff88438a9a>]
:libcfs:lbug_with_loc+0x7a/0xc0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884405b4>]
:libcfs:libcfs_assertion_failed+0x54/0x60
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884cbf55>]
:obdclass:llog_setup+0x2d5/0x860
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff889e8c9e>]
:osc:osc_llog_init+0x15e/0x3d0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884ca8d5>]
:obdclass:__llog_ctxt_put+0x25/0xe0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884c9929>]
:obdclass:obd_llog_init+0x179/0x210
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff886772ca>]
:lov:lov_llog_init+0x2ca/0x400
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884c9929>]
:obdclass:obd_llog_init+0x179/0x210
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8002c751>]
mntput_no_expire+0x19/0x89
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff889681ad>]
:mds:mds_llog_init+0x1ad/0x270
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80001c20>] _stext+0xc20/0x1000
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884c9929>]
:obdclass:obd_llog_init+0x179/0x210
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800c76d5>]
__vmalloc_area_node+0x12b/0x153
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884c9d75>]
:obdclass:llog_cat_initialize+0x3b5/0x670
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8868bd57>]
:lov:lov_get_info+0xa57/0xb20
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff886a3853>]
:lov:qos_add_tgt+0x643/0x6f0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8897162a>]
:mds:mds_lov_update_desc+0xc3a/0xe20
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff889744fc>]
:mds:mds_notify+0x36c/0x690
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff886806f7>]
:lov:lov_notify+0xa97/0xfd0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff886817df>]
:lov:lov_add_target+0xbaf/0xd20
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8868b047>]
:lov:lov_process_config+0x2f7/0x5b0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884e8a19>]
:obdclass:class_add_conn+0x6a9/0x7c0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884ed6f0>]
:obdclass:class_process_config+0x1870/0x19e0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884f06cb>]
:obdclass:class_config_llog_handler+0x155b/0x19b0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80026fe7>] do_filp_open+0x2a/0x38
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884c6b96>]
:obdclass:llog_lvfs_next_block+0x1a6/0x710
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884c10f8>]
:obdclass:llog_process+0xa08/0xe20
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff88438fe8>]
:libcfs:cfs_alloc+0x28/0x60
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884ef170>]
:obdclass:class_config_llog_handler+0x0/0x19b0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884eefff>]
:obdclass:class_config_parse_llog+0x43f/0x5b0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8000cebe>] dput+0x34/0x10a
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff88468042>]
:lvfs:lustre_rename+0x482/0x530
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884befba>]
:obdclass:llog_close+0x1aa/0x230
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8890fe06>]
:mgc:mgc_process_log+0x20f6/0x2640
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff88910b80>]
:mgc:mgc_blocking_ast+0x0/0x460
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8855ee60>]
:ptlrpc:ldlm_completion_ast+0x0/0x6a0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8890d85c>]
:mgc:config_log_find+0x19c/0x340
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff88913fb5>]
:mgc:mgc_process_config+0xdf5/0x1270
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884f45e4>]
:obdclass:lustre_process_log+0xb44/0xf00
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884f4a70>]
:obdclass:server_find_mount+0x80/0x190
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884fa7f6>]
:obdclass:server_start_targets+0xb36/0x17e0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884fec77>]
:obdclass:server_fill_super+0x18c7/0x1ee0
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800c345d>]
zone_statistics+0x3e/0x6d
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8000a779>]
get_page_from_freelist+0x35d/0x3cf
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80021c6a>] __up_read+0x19/0x7f
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8000948b>] __d_lookup+0xb0/0xff
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff88438fe8>]
:libcfs:cfs_alloc+0x28/0x60
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884f28af>]
:obdclass:lustre_init_lsi+0x29f/0x660
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884ff290>]
:obdclass:lustre_fill_super+0x0/0x1b10
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff88500c23>]
:obdclass:lustre_fill_super+0x1993/0x1b10
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800d92bc>]
set_anon_super+0x3c/0xab
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800d9280>]
set_anon_super+0x0/0xab
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff884ff290>]
:obdclass:lustre_fill_super+0x0/0x1b10
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800d94eb>] get_sb_nodev+0x4f/0x97
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800d8dd2>]
vfs_kern_mount+0x93/0x11a
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800d8e9b>]
do_kern_mount+0x36/0x4d
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800e25e6>] do_mount+0x68c/0x6fc
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80008acf>]
__handle_mm_fault+0x4e1/0xded
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80021c6a>] __up_read+0x19/0x7f
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80064a15>]
do_page_fault+0x4eb/0x81d
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff800c345d>]
zone_statistics+0x3e/0x6d
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8000ee85>]
__alloc_pages+0x5c/0x2ab
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff80049e9c>] sys_mount+0x8a/0xcd
Jun 13 15:44:36 dmds2 kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
Jun 13 15:44:36 dmds2 kernel: 
Jun 13 15:44:36 dmds2 kernel: LustreError: dumping log to
/tmp/lustre-log.1213364676.18697

On Wed, Jun 18, 2008 at 03:31:10PM +0200, Erich Focht
wrote:
> Is there anything I could still try to do? Remove or rewrite
> configuration, maybe extract with debugfs what is still visible,
> reformat?
> 
> Jun 13 15:44:36 dmds2 kernel: LustreError:
18697:0:(mds_log.c:207:mds_llog_init()) lov_llog_init err -2
> Jun 13 15:44:36 dmds2 kernel: LustreError:
18697:0:(llog_obd.c:392:llog_cat_initialize()) rc: -2
> Jun 13 15:44:36 dmds2 kernel: LustreError:
18697:0:(llog_obd.c:134:llog_setup()) ASSERTION(ctxt->loc_obd == obd) failed
See bugzilla ticket 16002. The quick solution is to mount the MDT and
to remove the CATALOGS file.

Johann

On Mittwoch 18 Juni 2008, Johann Lombardi wrote:
> See bugzilla ticket 16002. The quick solution is to mount the MDT and
> to remove the CATALOGS file.

Johann, thanks very much! I need to increase my bugzilla searching
capabilities...

Best regards,
Erich