thr3ads.net - Lustre discuss - [Lustre-discuss] lustre errors [Mar 2007]

If this information is useful, please help other people find it:
Share via:

konrad.gutkowski@ffs.pl

2007-Mar-21 16:15 UTC

[Lustre-discuss] lustre errors

Hi,

I have recently upgraded my lustre server and clients from 1.6 b5 to b7, 
since then im getting those in dmesg:

-------- SERVER:
[179654.788677] LustreError: 3472:0:(lustre_dlm.h:604:lock_bitlock()) 
ASSERTION(lock->l_pidb == 0) failed
[179654.788715] LustreError: 
3472:0:(tracefile.c:433:libcfs_assertion_failed()) LBUG
[179654.788743] Lustre: 3472:0:(linux-debug.c:166:libcfs_debug_dumpstack()) 
showing stack for process 3472
[179654.788748] ll_mdt_04     R running     0  3472      1          3473 
3471 (L-TLB)

--------- CLIENT:
[97141.689073] LustreError: 
5573:0:(client.c:942:ptlrpc_expire_one_request()) @@@ timeout (sent at 
1174512276, 100s ago) req@ffff810031eae400 x7183964/t0 
o101->beta7-MDT0000_UUID@192.168.0.4@tcp:12 lens 232/240 ref 1 fl Rpc:/0/0 
rc 0/0
[97141.689227] LustreError: beta7-MDT0000-mdc-ffff81003e657c00: Connection 
to service beta7-MDT0000 via nid 192.168.0.4@tcp was lost; in progress 
operations using this service will wait for recovery to complete.
[97141.689695] ----------- [cut here ] --------- [please bite 
here ] ---------
[97141.689744] Kernel BUG at fs/locks.c:2025
(etc...)

server is 2.6.12 client 2.6.18.7

TiA
Konrad Gutkowski 
-------------- next part --------------
[97141.689073] LustreError: 5573:0:(client.c:942:ptlrpc_expire_one_request())
@@@ timeout (sent at 1174512276, 100s ago) req@ffff810031eae400 x7183964/t0
o101->beta7-MDT0000_UUID@192.168.0.4@tcp:12 lens 232/240 ref 1 fl Rpc:/0/0 rc
0/0
[97141.689227] LustreError: beta7-MDT0000-mdc-ffff81003e657c00: Connection to
service beta7-MDT0000 via nid 192.168.0.4@tcp was lost; in progress operations
using this service will wait for recovery to complete.
[97141.689695] ----------- [cut here ] --------- [please bite here ] ---------
[97141.689744] Kernel BUG at fs/locks.c:2025
[97141.689788] invalid opcode: 0000 [1] 
[97141.689830] CPU 0 
[97141.689869] Modules linked in: osc mgc lustre lov lquota mdc ksocklnd ptlrpc
obdclass lvfs lnet libcfs
[97141.689964] Pid: 5573[#3171], comm: couriertcpd Not tainted 2.6.18.7 #1
[97141.690011] RIP: 0010:[<ffffffff80227a7a>]  [<ffffffff80227a7a>]
locks_remove_flock+0x12a/0x1a0
[97141.690099] RSP: 0018:ffff8100353bfcf8  EFLAGS: 00010246
[97141.690145] RAX: ffff810039bc7298 RBX: ffff8100353ac260 RCX: 0000000000000002
[97141.690191] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff81003ffef5c0
[97141.690240] RBP: ffff81003f3a0680 R08: 0000000000000000 R09: ffff810031eae400
[97141.690286] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8100353ac150
[97141.690335] R13: ffff8100353bfda8 R14: ffff81003f4fa880 R15: 0000000000000000
[97141.690383] FS:  00002b21ccbc6d90(0000) GS:ffffffff80577000(0000)
knlGS:0000000000000000
[97141.690466] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[97141.690511] CR2: 000000000040e000 CR3: 000000003e670000 CR4: 00000000000006e0
[97141.690561] Process couriertcpd (pid: 5573[#3171], threadinfo
ffff8100353be000, task ffff810039abc790)
[97141.690643] Stack:  0000000000000000 0000000000000000 0000000000000000
0000000000000000
[97141.690727]  0000000000000000 0000000000000000 00000000000015c5
0000000000000000
[97141.690810]  0000000000000000 ffff81003f3a0680 0000000000000202
0000000000000000
[97141.690858] Call Trace:
[97141.690944]  [<ffffffff80212518>] __fput+0x98/0x1e0
[97141.690990]  [<ffffffff8022416e>] filp_close+0x6e/0x90
[97141.691038]  [<ffffffff80239c5b>] put_files_struct+0x6b/0xe0
[97141.691084]  [<ffffffff80215d3d>] do_exit+0x23d/0x890
[97141.691132]  [<ffffffff80249819>] do_group_exit+0x99/0xa0
[97141.691178]  [<ffffffff8025ea12>] system_call+0x7e/0x83
[97141.691224] 
[97141.691261] 
[97141.691262] Code: 0f 0b 68 29 8d 48 80 c2 e9 07 48 89 c3 48 8b 03 eb bc 66 66
[97141.691395] RIP  [<ffffffff80227a7a>] locks_remove_flock+0x12a/0x1a0
[97141.691445]  RSP <ffff8100353bfcf8>
[97141.691807]  <1>Fixing recursive fault but reboot is needed!
[97141.692930] Lustre: beta7-MDT0000-mdc-ffff81003e657c00: Connection restored
to service beta7-MDT0000 using nid 192.168.0.4@tcp.
-------------- next part --------------
ET upcall /usr/lib/lustre/lnet_upcall
LBUG,/root/SERVER-02-2007/lustre-1.5.97/lnet/libcfs/tracefile.c,libcfs_assertion_failed,433
[12732.936216] Lustre: 3469:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for
pid 3470: it was inactive for 100000ms
[12732.936230] Lustre: 3469:0:(linux-debug.c:166:libcfs_debug_dumpstack())
showing stack for process 3470
[12732.936235] ll_mdt_02     D C0490348     0  3470      1          3471  3469
(L-TLB)
[12732.936245] f5257ae0 00000046 c17e9a00 c0490348 00000b7d c0102dfb f60df0c0
00000001
[12732.936252]        c17e9a00 00000a02 7fc768f1 00000b7d c17e9a00 f60df0c0
f60df1e4 f5256000
[12732.936259]        f8a626bc f8a614dd 000001b1 f8a53d51 f8a626bc f8a614dd
000001b1 f8a626eb
[12732.936267] Call Trace:
[12732.936274]  [<c0102dfb>] show_stack+0x82/0x97
[12732.936287]  [<f8a53d51>] lbug_with_loc+0x9b/0xae [libcfs]
[12732.936317]  [<f8a5d0f8>] trace_assertion_failed+0x0/0xa8 [libcfs]
[12732.936333]  [<f8fc105e>] lock_res_and_lock+0x5e/0x63 [ptlrpc]
[12732.936406]  [<f8fc7840>] ldlm_run_bl_ast_work+0xd2/0x465 [ptlrpc]
[12732.936431]  [<f8fc1b4e>] ldlm_lock_remove_from_lru+0x48/0x1dd [ptlrpc]
[12732.936456]  [<f8aed6f7>] mds_getattr_lock+0x3c0/0x119c [mds]
[12732.936505]  [<f8fc2003>] ldlm_lock_destroy_internal+0x320/0x5c4
[ptlrpc]
[12732.936531]  [<f8ffd012>] ldlm_process_flock_lock+0x0/0x2db5 [ptlrpc]
[12732.936562]  [<f8fc74c6>] ldlm_reprocess_queue+0xbd/0x365 [ptlrpc]
[12732.936589]  [<f8fc2530>] ldlm_lock_destroy_nolock+0x8d/0x1e7 [ptlrpc]
[12732.936616]  [<f8ffd900>] ldlm_process_flock_lock+0x8ee/0x2db5 [ptlrpc]
[12732.936645]  [<f8fc336f>] __ldlm_handle2lock+0xb2/0x6c0 [ptlrpc]
[12732.936673]  [<f88d1152>] lnet_ni_send+0x46/0xd3 [lnet]
[12732.936709]  [<f88d22de>] lnet_send+0x2e6/0xdb2 [lnet]
[12732.936726]  [<f901a9e9>] lustre_pack_reply_v2+0x1a2/0x77f [ptlrpc]
[12732.936758]  [<f8fc6b0f>] ldlm_lock_enqueue+0x10e/0xa08 [ptlrpc]
[12732.936786]  [<f8ff2ba5>] ldlm_handle_enqueue+0xd41/0x23cd [ptlrpc]
[12732.936816]  [<f8ff04e8>] ldlm_server_blocking_ast+0x0/0xc45 [ptlrpc]
[12732.936843]  [<f8ff112d>] ldlm_server_completion_ast+0x0/0x74b [ptlrpc]
[12732.936873]  [<f8af783e>] mds_handle+0x358f/0x59e0 [mds]
[12732.936892]  [<f8ff112d>] ldlm_server_completion_ast+0x0/0x74b [ptlrpc]
[12732.936920]  [<f8ff04e8>] ldlm_server_blocking_ast+0x0/0xc45 [ptlrpc]
[12732.936947]  [<f8ff57f2>] ldlm_handle_bl_callback+0x141/0x54f [ptlrpc]
[12732.936976]  [<f901fbc6>] lustre_msg_get_conn_cnt+0xb9/0x122 [ptlrpc]
[12732.937006]  [<c0106482>] do_gettimeofday+0x18/0xb0
[12732.937016]  [<f902986f>] ptlrpc_server_handle_request+0xc17/0x16b8
[ptlrpc]
[12732.937045]  [<f8fc1472>] ldlm_lock_put+0x23b/0x820 [ptlrpc]
[12732.937070]  [<c0106482>] do_gettimeofday+0x18/0xb0
[12732.937076]  [<f8a60751>] lcw_update_time+0x15/0x172 [libcfs]
[12732.937094]  [<c0112b17>] recalc_task_prio+0x8e/0x155
[12732.937104]  [<f8a60948>] lc_watchdog_touch+0x9a/0x254 [libcfs]
[12732.937119]  [<f902b45a>] ptlrpc_main+0x74a/0xd92 [ptlrpc]
[12732.937151]  [<c01135f2>] default_wake_function+0x0/0x12
[12732.937158]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[12732.937208]  [<c0102786>] ret_from_fork+0x6/0x14
[12732.937213]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[12732.937241]  [<f902ad10>] ptlrpc_main+0x0/0xd92 [ptlrpc]
[12732.937268]  [<c0100cf9>] kernel_thread_helper+0x5/0xb
[12732.937354] LustreError: dumping log to /tmp/lustre-log.1174345303.3470
[12732.969904] Lustre: 3472:0:(ldlm_lib.c:497:target_handle_reconnect())
beta7-MDT0000: ced5bd80-9f94-3866-95e3-f8aabe447db5 reconnecting
[81283.864527] LustreError: 3475:0:(lustre_dlm.h:604:lock_bitlock())
ASSERTION(lock->l_pidb == 0) failed
[81283.864563] LustreError: 3475:0:(tracefile.c:433:libcfs_assertion_failed())
LBUG
[81283.864590] Lustre: 3475:0:(linux-debug.c:166:libcfs_debug_dumpstack())
showing stack for process 3475
[81283.864596] ll_mdt_07     R running     0  3475      1          3476  3474
(L-TLB)
[81283.864604] 00000009 00000000 c0495480 f63e37e0 00000246 00000000 f63e37e0
f8a60948
[81283.864611]        00000000 00000000 00000009 ffffffff ffffffff 46002233
00048f0b 00000246
[81283.864618]        00000000 f469ff00 f5100960 f902b45a f5100960 f469ff00
ffffffff 76000018
[81283.864625] Call Trace:
[81283.864633]  [<f8a60948>] lc_watchdog_touch+0x9a/0x254 [libcfs]
[81283.864665]  [<f902b45a>] ptlrpc_main+0x74a/0xd92 [ptlrpc]
[81283.864748]  [<c01135f2>] default_wake_function+0x0/0x12
[81283.864761]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[81283.864788]  [<c0102786>] ret_from_fork+0x6/0x14
[81283.864795]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[81283.864822]  [<f902ad10>] ptlrpc_main+0x0/0xd92 [ptlrpc]
[81283.864848]  [<c0100cf9>] kernel_thread_helper+0x5/0xb
[81283.865111] LustreError: dumping log to /tmp/lustre-log.1174413875.3475
[81283.987766] Lustre: 3475:0:(linux-debug.c:98:libcfs_run_upcall()) Invoked
LNET upcall /usr/lib/lustre/lnet_upcall
LBUG,/root/SERVER-02-2007/lustre-1.5.97/lnet/libcfs/tracefile.c,libcfs_assertion_failed,433
[81309.798573] LustreError:
3444:0:(ldlm_lockd.c:594:ldlm_server_completion_ast()) ### enqueue wait took
4245463432us from 1174345231 ns: mds-beta7-MDT0000_UUID lock:
ef2b2940/0x9e7724618e0c0135 lrc: 2/0,0 mode: PW/PW res: 1590113/3964360148 rrc:
21 type: FLK pid: 5216 [0->9223372036854775807] flags: 0 remote:
0x542c414bd7799ac5 expref: 20034 pid: 3473
[81309.799473] LustreError:
3446:0:(ldlm_lockd.c:594:ldlm_server_completion_ast()) ### enqueue wait took
4231904405us from 1174345244 ns: mds-beta7-MDT0000_UUID lock:
eef4da60/0x9e7724618e0ea1dd lrc: 2/0,0 mode: PW/PW res: 1590113/3964360148 rrc:
20 type: FLK pid: 5228 [0->9223372036854775807] flags: 0 remote:
0x542c414bd77b3e2b expref: 20035 pid: 3469
[81309.799489] LustreError:
3446:0:(ldlm_lockd.c:594:ldlm_server_completion_ast()) Skipped 3 previous
similar messages
[81383.803398] Lustre: 0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid
3475: it was inactive for 100000ms
[81383.803410] Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing
stack for process 3475
[81383.803415] ll_mdt_07     D C048FEA0     0  3475      1          3476  3474
(L-TLB)
[81383.803425] f4b05ae0 00000046 f4601100 c048fea0 f4601100 c0102dfb f4601100
f4b05ea4
[81383.803432]        f4601100 00000a8d 6737a0fb 000049ed c03c4b80 f4601100
f4601224 f4b04000
[81383.803440]        f8a626bc f8a614dd 000001b1 f8a53d51 f8a626bc f8a614dd
000001b1 f8a626eb
[81383.803447] Call Trace:
[81383.803454]  [<c0102dfb>] show_stack+0x82/0x97
[81383.803467]  [<f8a53d51>] lbug_with_loc+0x9b/0xae [libcfs]
[81383.803497]  [<f8a5d0f8>] trace_assertion_failed+0x0/0xa8 [libcfs]
[81383.803512]  [<f8fc105e>] lock_res_and_lock+0x5e/0x63 [ptlrpc]
[81383.803585]  [<f8fc7840>] ldlm_run_bl_ast_work+0xd2/0x465 [ptlrpc]
[81383.803611]  [<f8fc1b4e>] ldlm_lock_remove_from_lru+0x48/0x1dd [ptlrpc]
[81383.803637]  [<f8fc2003>] ldlm_lock_destroy_internal+0x320/0x5c4
[ptlrpc]
[81383.803661]  [<f8ffd012>] ldlm_process_flock_lock+0x0/0x2db5 [ptlrpc]
[81383.803691]  [<f8fc74c6>] ldlm_reprocess_queue+0xbd/0x365 [ptlrpc]
[81383.803716]  [<f8fc2530>] ldlm_lock_destroy_nolock+0x8d/0x1e7 [ptlrpc]
[81383.803742]  [<f8ffd900>] ldlm_process_flock_lock+0x8ee/0x2db5 [ptlrpc]
[81383.803769]  [<f88d22de>] lnet_send+0x2e6/0xdb2 [lnet]
[81383.803806]  [<f88d1152>] lnet_ni_send+0x46/0xd3 [lnet]
[81383.803821]  [<f88d22de>] lnet_send+0x2e6/0xdb2 [lnet]
[81383.803838]  [<f901a9e9>] lustre_pack_reply_v2+0x1a2/0x77f [ptlrpc]
[81383.803871]  [<f8fc6b0f>] ldlm_lock_enqueue+0x10e/0xa08 [ptlrpc]
[81383.803899]  [<f8ff2ba5>] ldlm_handle_enqueue+0xd41/0x23cd [ptlrpc]
[81383.803928]  [<f8ff04e8>] ldlm_server_blocking_ast+0x0/0xc45 [ptlrpc]
[81383.803956]  [<f8ff112d>] ldlm_server_completion_ast+0x0/0x74b [ptlrpc]
[81383.803985]  [<f8af783e>] mds_handle+0x358f/0x59e0 [mds]
[81383.804034]  [<f8ff112d>] ldlm_server_completion_ast+0x0/0x74b [ptlrpc]
[81383.804062]  [<f8ff04e8>] ldlm_server_blocking_ast+0x0/0xc45 [ptlrpc]
[81383.804089]  [<f8ff57f2>] ldlm_handle_bl_callback+0x141/0x54f [ptlrpc]
[81383.804118]  [<f901fbc6>] lustre_msg_get_conn_cnt+0xb9/0x122 [ptlrpc]
[81383.804148]  [<c0106482>] do_gettimeofday+0x18/0xb0
[81383.804158]  [<f902986f>] ptlrpc_server_handle_request+0xc17/0x16b8
[ptlrpc]
[81383.804187]  [<f8fc1472>] ldlm_lock_put+0x23b/0x820 [ptlrpc]
[81383.804212]  [<c0106482>] do_gettimeofday+0x18/0xb0
[81383.804218]  [<f8a60751>] lcw_update_time+0x15/0x172 [libcfs]
[81383.804236]  [<c0112b17>] recalc_task_prio+0x8e/0x155
[81383.804247]  [<f8a60948>] lc_watchdog_touch+0x9a/0x254 [libcfs]
[81383.804261]  [<f902b45a>] ptlrpc_main+0x74a/0xd92 [ptlrpc]
[81383.804294]  [<c01135f2>] default_wake_function+0x0/0x12
[81383.804301]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[81383.804329]  [<c0102786>] ret_from_fork+0x6/0x14
[81383.804334]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[81383.804363]  [<f902ad10>] ptlrpc_main+0x0/0xd92 [ptlrpc]
[81383.804412]  [<c0100cf9>] kernel_thread_helper+0x5/0xb
[81383.804426] LustreError: dumping log to /tmp/lustre-log.1174413975.3475
[81383.837034] Lustre: 3474:0:(ldlm_lib.c:497:target_handle_reconnect())
beta7-MDT0000: ced5bd80-9f94-3866-95e3-f8aabe447db5 reconnecting
[82590.759600] Lustre: 3:0:(linux-debug.c:98:libcfs_run_upcall()) Invoked LNET
upcall /usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,192.168.0.3@tcp,down,1174415099
[82603.751711] Lustre: 3:0:(linux-debug.c:98:libcfs_run_upcall()) Invoked LNET
upcall /usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,192.168.0.2@tcp,down,1174415150
[82663.353933] Lustre: MGS: haven''t heard from client
edc5a483-55ab-86da-985f-20a8e7183ce9 (at 192.168.0.3@tcp) in 249 seconds. I
think it''s dead, and I am evicting it.
[82686.142643] LustreError: 3471:0:(client.c:942:ptlrpc_expire_one_request())
@@@ timeout (sent at 1174415272, 6s ago) req@f4444400 x6735250/t0
o104->????@NET_0x20000c0a80002_UUID:15 lens 232/128 ref 1 fl Rpc:/0/0 rc 0/0
[82686.142675] LustreError: A client on nid 192.168.0.2@tcp was evicted from
service beta7-MDT0000.
[82686.142695] LustreError: 3471:0:(ldlm_lockd.c:428:ldlm_failed_ast()) ###
blocking AST failed (-110): evicting client
ced5bd80-9f94-3866-95e3-f8aabe447db5@NET_0x20000c0a80002_UUID NID
192.168.0.2@tcp (192.168.0.2@tcp) ns: mds-beta7-MDT0000_UUID lock:
da413ee0/0x9e772461934c7d36 lrc: 2/0,0 mode: CR/CR res: 1343531/1158387142 bits
0x3 rrc: 4 type: IBT flags: 4000020 remote: 0x542c414bda623394 expref: 19983 pid
3472
[82802.664158] Lustre: MGS: haven''t heard from client
58cd2e28-192f-3219-8ea8-dd556a10e063 (at 192.168.0.2@tcp) in 244 seconds. I
think it''s dead, and I am evicting it.
[82802.664168] Lustre: Skipped 1 previous similar message
[82827.649009] Lustre: beta7-OST0000: haven''t heard from client
ced5bd80-9f94-3866-95e3-f8aabe447db5 (at 192.168.0.2@tcp) in 269 seconds. I
think it''s dead, and I am evicting it.
[82902.603536] Lustre: MGS: haven''t heard from client
1a12d39e-c8c7-6be9-543f-b97d7468e9e8 (at 192.168.0.3@tcp) in 164 seconds. I
think it''s dead, and I am evicting it.
[83000.989734] Lustre: MGS: haven''t heard from client
1864ae37-f467-df9a-42fd-ac37d7da75f8 (at 192.168.0.3@tcp) in 227 seconds. I
think it''s dead, and I am evicting it.
[179654.788677] LustreError: 3472:0:(lustre_dlm.h:604:lock_bitlock())
ASSERTION(lock->l_pidb == 0) failed
[179654.788715] LustreError: 3472:0:(tracefile.c:433:libcfs_assertion_failed())
LBUG
[179654.788743] Lustre: 3472:0:(linux-debug.c:166:libcfs_debug_dumpstack())
showing stack for process 3472
[179654.788748] ll_mdt_04     R running     0  3472      1          3473  3471
(L-TLB)
[179654.788757] 00000020 00000000 c0495480 f63e3960 00000246 00000000 f63e3960
f8a60948
[179654.788764]        00000000 00000000 00000020 ffffffff ffffffff 4601a294
0007394d 00000246
[179654.788771]        00000000 f469f140 f5100960 f902b45a f5100960 f469f140
ffffffff 00000000
[179654.788779] Call Trace:
[179654.788787]  [<f8a60948>] lc_watchdog_touch+0x9a/0x254 [libcfs]
[179654.788819]  [<f902b45a>] ptlrpc_main+0x74a/0xd92 [ptlrpc]
[179654.788902]  [<c01135f2>] default_wake_function+0x0/0x12
[179654.788914]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[179654.788941]  [<c0102786>] ret_from_fork+0x6/0x14
[179654.788948]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[179654.788975]  [<f902ad10>] ptlrpc_main+0x0/0xd92 [ptlrpc]
[179654.789002]  [<c0100cf9>] kernel_thread_helper+0x5/0xb
[179654.789051] LustreError: dumping log to /tmp/lustre-log.1174512276.3472
[179654.998183] Lustre: 3472:0:(linux-debug.c:98:libcfs_run_upcall()) Invoked
LNET upcall /usr/lib/lustre/lnet_upcall
LBUG,/root/SERVER-02-2007/lustre-1.5.97/lnet/libcfs/tracefile.c,libcfs_assertion_failed,433
[179754.728032] Lustre: 0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid
3472: it was inactive for 100000ms
[179754.728045] Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing
stack for process 3472
[179754.728050] ll_mdt_04     D C0490348     0  3472      1          3473  3471
(L-TLB)
[179754.728060] f4cbbae0 00000046 c17e9a00 c0490348 0000a365 c0102dfb f60dd0e0
00000001
[179754.728067]        c17e9a00 0000096e 30597367 0000a365 c17e9a00 f60dd0e0
f60dd204 f4cba000
[179754.728075]        f8a626bc f8a614dd 000001b1 f8a53d51 f8a626bc f8a614dd
000001b1 f8a626eb
[179754.728083] Call Trace:
[179754.728091]  [<c0102dfb>] show_stack+0x82/0x97
[179754.728104]  [<f8a53d51>] lbug_with_loc+0x9b/0xae [libcfs]
[179754.728134]  [<f8a5d0f8>] trace_assertion_failed+0x0/0xa8 [libcfs]
[179754.728150]  [<f8fc105e>] lock_res_and_lock+0x5e/0x63 [ptlrpc]
[179754.728223]  [<f8fc7840>] ldlm_run_bl_ast_work+0xd2/0x465 [ptlrpc]
[179754.728249]  [<f8fc1b4e>] ldlm_lock_remove_from_lru+0x48/0x1dd
[ptlrpc]
[179754.728275]  [<f8fc2003>] ldlm_lock_destroy_internal+0x320/0x5c4
[ptlrpc]
[179754.728300]  [<f8ffd012>] ldlm_process_flock_lock+0x0/0x2db5 [ptlrpc]
[179754.728329]  [<f8fc74c6>] ldlm_reprocess_queue+0xbd/0x365 [ptlrpc]
[179754.728354]  [<f8fc2530>] ldlm_lock_destroy_nolock+0x8d/0x1e7 [ptlrpc]
[179754.728380]  [<f8ffd900>] ldlm_process_flock_lock+0x8ee/0x2db5
[ptlrpc]
[179754.728407]  [<f88d22de>] lnet_send+0x2e6/0xdb2 [lnet]
[179754.728446]  [<f88d6fda>] LNetPut+0x3fb/0xd04 [lnet]
[179754.728464]  [<f901a9e9>] lustre_pack_reply_v2+0x1a2/0x77f [ptlrpc]
[179754.728497]  [<f8fc6b0f>] ldlm_lock_enqueue+0x10e/0xa08 [ptlrpc]
[179754.728525]  [<f8ff2ba5>] ldlm_handle_enqueue+0xd41/0x23cd [ptlrpc]
[179754.728554]  [<f8ff04e8>] ldlm_server_blocking_ast+0x0/0xc45 [ptlrpc]
[179754.728582]  [<f8ff112d>] ldlm_server_completion_ast+0x0/0x74b
[ptlrpc]
[179754.728611]  [<f8af783e>] mds_handle+0x358f/0x59e0 [mds]
[179754.728662]  [<f8ff112d>] ldlm_server_completion_ast+0x0/0x74b
[ptlrpc]
[179754.728690]  [<f8ff04e8>] ldlm_server_blocking_ast+0x0/0xc45 [ptlrpc]
[179754.728719]  [<f901fbc6>] lustre_msg_get_conn_cnt+0xb9/0x122 [ptlrpc]
[179754.728749]  [<c0132c00>] handle_IRQ_event+0x39/0x6d
[179754.728760]  [<c0106482>] do_gettimeofday+0x18/0xb0
[179754.728769]  [<f902986f>] ptlrpc_server_handle_request+0xc17/0x16b8
[ptlrpc]
[179754.728798]  [<c0106482>] do_gettimeofday+0x18/0xb0
[179754.728804]  [<f8a60751>] lcw_update_time+0x15/0x172 [libcfs]
[179754.728824]  [<f8a60948>] lc_watchdog_touch+0x9a/0x254 [libcfs]
[179754.728838]  [<f902b45a>] ptlrpc_main+0x74a/0xd92 [ptlrpc]
[179754.728871]  [<c01135f2>] default_wake_function+0x0/0x12
[179754.728882]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[179754.728910]  [<c0102786>] ret_from_fork+0x6/0x14
[179754.728915]  [<f902acff>] ptlrpc_retry_rqbds+0x0/0x11 [ptlrpc]
[179754.728943]  [<f902ad10>] ptlrpc_main+0x0/0xd92 [ptlrpc]
[179754.728971]  [<c0100cf9>] kernel_thread_helper+0x5/0xb
[179754.728987] LustreError: dumping log to /tmp/lustre-log.1174512376.3472
[179754.761665] Lustre: 3471:0:(ldlm_lib.c:497:target_handle_reconnect())
beta7-MDT0000: de506883-763c-ada5-83f2-446a09fc507e reconnecting

Oleg Drokin

2007-Mar-26 05:10 UTC

head link

[Lustre-discuss] lustre errors

Hello!

On Thu, Mar 22, 2007 at 12:14:20AM +0100, konrad.gutkowski@ffs.pl wrote:
> I have recently upgraded my lustre server and clients from 1.6 b5 to b7, 
> since then im getting those in dmesg:
flock and posix locks are still somewhat flawed, but we are working on it
currently.
You might find a patch from bug 5145 helpful in your situation, or might be not.
In any case if you can somehow reduce the scope of the problem to some simple
testcases, that would be really great.

Thanks.

Bye,
    Oleg

Konrad Gutkowski

2007-Mar-26 06:35 UTC

head link

[Lustre-discuss] lustre errors

Hi,

I''ve fixed my problem by commenting out ll_file_noflock from 
ll_file_operations on client side,
I think it completely restored the b5 behavior,
as i dont need cluster-wide locks only node wide it should work just fine 
for now.

As for a test case....
Appliaction causing the bug was courier-imap daemon running inside my mail 
handling VServer guest,
evry time I killed courier-imap daemon it hanged and in the same moment mdt 
on the server was dead,
server is in a semi-production state so i would really like it to be 
running.

Konrad Gutkowski

----- Original Message ----- 
From: "Oleg Drokin" <green@clusterfs.com>
To: <konrad.gutkowski@ffs.pl>
Cc: <lustre-discuss@clusterfs.com>
Sent: Monday, March 26, 2007 2:09 PM
Subject: Re: [Lustre-discuss] lustre errors

> Hello!
>
> On Thu, Mar 22, 2007 at 12:14:20AM +0100, konrad.gutkowski@ffs.pl wrote:
>
>> I have recently upgraded my lustre server and clients from 1.6 b5 to
b7,
>> since then im getting those in dmesg:
>
> flock and posix locks are still somewhat flawed, but we are working on it
> currently.
> You might find a patch from bug 5145 helpful in your situation, or might 
> be not.
> In any case if you can somehow reduce the scope of the problem to some 
> simple
> testcases, that would be really great.
>
> Thanks.
>
> Bye,
>    Oleg
>

Oleg Drokin

2007-Mar-26 06:45 UTC

head link

[Lustre-discuss] lustre errors

Hello!

On Mon, Mar 26, 2007 at 03:35:02PM +0200, Konrad Gutkowski
wrote:> I''ve fixed my problem by commenting out ll_file_noflock from 
> ll_file_operations on client side,
> I think it completely restored the b5 behavior,
> as i dont need cluster-wide locks only node wide it should work just fine 
> for now.
Ah, I see.
Patch in bug 11563 introduces new mount option "localflock" which is
the same
as noflock in previous versions. It only allows for local noe flock
consistency, not cross-node consistency.
> As for a test case....
> Appliaction causing the bug was courier-imap daemon running inside my mail 
> handling VServer guest,
> evry time I killed courier-imap daemon it hanged and in the same moment mdt
> on the server was dead,
> server is in a semi-production state so i would really like it to be 
> running.
I see.
Thanks.

Bye,
    Oleg

Bernd Schubert

2007-Apr-03 08:05 UTC

head link

[Lustre-discuss] Lustre Errors

Hi,

we just got rpc errors and ptlrpcd stayed at 100% cpu, anyone having an idea 
what might have caused that or where I should look at?


Thanks,
Bernd


[  241.261217] Lustre: Lustre Lite Client File System; info@clusterfs.com
[ 2668.962381] LustreError: 3211:0:(events.c:129:client_bulk_callback()) event 
type 0, status -5, desc ffff810075c0c000
[ 2668.973017] LustreError: 3212:0:(events.c:53:request_out_callback()) @@@ 
type 4, status -103  req@ffff81007b007b50 x187993/t0 
o4->ost1_UUID@beo-101_UUID:28 lens 328/288 ref 3 fl Rpc:/0/0 rc 0/0
[ 2668.990284] LustreError: 3209:0:(events.c:53:request_out_callback()) @@@ 
type 4, status -5  req@ffff81007b0062b0 x187987/t9602764 
o4->ost1_UUID@beo-101_UUID:28 lens 328/288 ref 3 fl Bulk:R/4/0 rc 0/0
[ 2669.008051] LustreError: 3211:0:(events.c:129:client_bulk_callback()) event 
type 0, status -103, desc ffff81006d282000
[ 2669.018824] LustreError: 3212:0:(events.c:53:request_out_callback()) @@@ 
type 4, status -103  req@ffff81007d1040d8 x187992/t0 
o4->ost1_UUID@beo-101_UUID:28 lens 328/288 ref 3 fl Rpc:/0/0 rc 0/0
[ 2669.036120] LustreError: 3208:0:(events.c:53:request_out_callback()) @@@ 
type 4, status -5  req@ffff8100325a34c8 x187988/t9602766 
o4->ost1_UUID@beo-101_UUID:28 lens 328/288 ref 4 fl Bulk:R/4/0 rc 0/0
[ 2669.053955] LustreError: 3211:0:(events.c:129:client_bulk_callback()) event 
type 0, status -5, desc ffff8100d9fbc000
[ 2669.064591] LustreError: 3210:0:(events.c:129:client_bulk_callback()) event 
type 0, status -5, desc ffff8100d9f80000
[ 2669.075208] LustreError: 3211:0:(events.c:53:request_out_callback()) @@@ 
type 4, status -5  req@ffff81007d104508 x187989/t9602767 
o4->ost1_UUID@beo-101_UUID:28 lens 328/288 ref 4 fl Bulk:R/4/0 rc 0/0
[ 2669.075348] LustreError: 3260:0:(client.c:894:ptlrpc_check_set()) @@@ bulk 
transfer failed  req@ffff810077cb6488 x187986/t9602763 
o4->ost1_UUID@beo-101_UUID:28 lens 328/288 ref 2 fl Bulk:R/4/0 rc 0/0
[ 2669.075355] LustreError: 3260:0:(client.c:895:ptlrpc_check_set()) LBUG
[ 2669.075360] Lustre: 3260:0:(linux-debug.c:166:libcfs_debug_dumpstack()) 
showing stack for process 3260
[ 2669.075363] ptlrpcd       R  running task       0  3260      1          
3261  3231 (L-TLB)
[ 2669.075369]  0000000000000000 0000000000000000 ffffffff8812ea1e 
0000000000000000
[ 2669.075374]  ffff810075c0c008 ffff810077cb6488 ffff810077f07dc0 
ffffffff881f89c9
[ 2669.075379]  ffff810077f07d50 ffffffff8046e2dd ffff8100326be080 
ffff81011d498000
[ 2669.075383] Call Trace:
[ 2669.075439]  [<ffffffff881f89c9>] :ptlrpc:ptlrpc_check_set+0x804/0xa27
[ 2669.075446]  [<ffffffff8046e2dd>] _spin_lock_irqsave+0xe/0x15
[ 2669.075488]  [<ffffffff8821b1c7>] :ptlrpc:ptlrpcd_check+0x16a/0x273
[ 2669.075493]  [<ffffffff8046cb16>] schedule_timeout+0x98/0xb4
[ 2669.075499]  [<ffffffff802323e9>] process_timeout+0x0/0xb
[ 2669.075536]  [<ffffffff8821b5c5>] :ptlrpc:ptlrpcd+0x2f5/0x42b
[ 2669.075541]  [<ffffffff80234243>] do_notify_parent+0x19a/0x1a7
[ 2669.075547]  [<ffffffff8022387f>] default_wake_function+0x0/0xf
[ 2669.075585]  [<ffffffff881f749a>] :ptlrpc:ptlrpc_expired_set+0x0/0x177
[ 2669.075622]  [<ffffffff881f749a>] :ptlrpc:ptlrpc_expired_set+0x0/0x177
[ 2669.075628]  [<ffffffff8022387f>] default_wake_function+0x0/0xf
[ 2669.075632]  [<ffffffff8046e3d0>] _spin_unlock_irq+0x9/0xc
[ 2669.075638]  [<ffffffff8020a3e8>] child_rip+0xa/0x12
[ 2669.075678]  [<ffffffff8821b2d0>] :ptlrpc:ptlrpcd+0x0/0x42b
[ 2669.075682]  [<ffffffff8020a3de>] child_rip+0x0/0x12
[ 2669.075684]
[ 2669.249120] LustreError: 3211:0:(events.c:53:request_out_callback()) 
Skipped 1 previous similar message
[ 2669.262065] LustreError: 3208:0:(events.c:129:client_bulk_callback()) event 
type 0, status -5, desc ffff8100a46dc000
[ 2669.269022] LustreError: 14:0:(linux-debug.c:91:libcfs_run_upcall()) 
Error -2 invoking LNET upcall /usr/lib/lustre/lnet_upcall 
ROUTER_NOTIFY,192.168.41.101@o2ib,down,1175601855; 
check /proc/sys/lnet/upcall
[ 2669.290995] LustreError: 3211:0:(events.c:53:request_out_callback()) @@@ 
type 4, status -5  req@ffff81007d10a4c8 x187990/t0 
o4->ost1_UUID@beo-101_UUID:28 lens 328/288 ref 3 fl Rpc:/0/0 rc 0/0
[ 2669.308211] LustreError: dumping log to /tmp/lustre-log.1175601855.3260
[ 2669.314858] LustreError: 3260:0:(linux-debug.c:91:libcfs_run_upcall()) 
Error -2 invoking LNET upcall /usr/lib/lustre/lnet_upcall 
LBUG,/data/home/bschubert/tmpa/Lustre/lustre-1.4.9/lustre-1.4.9/lustre/ptlrpc/client.c,ptlrpc_check_set,895;
check /proc/sys/lnet/upcall



-- 
Bernd Schubert
Q-Leap Networks GmbH

Herb Wartens

2007-Apr-03 10:59 UTC

head link

[Lustre-discuss] Lustre Errors

Bernd,
Do you access to the cfs bugzilla database?  I believe that this bug is:
https://bugzilla.clusterfs.com/show_bug.cgi?id=11544
Currently there is no approved fix for the bug, but there is a patch that
gets some extra debug information to help run down exactly what is going on.
Hope this helps.

-Herb

Bernd Schubert

2007-Apr-04 01:35 UTC

head link

[Lustre-discuss] Lustre Errors

On Tuesday 03 April 2007 18:58:49 Herb Wartens wrote:> Bernd,
> Do you access to the cfs bugzilla database?  I believe that this bug is:
Thanks, I promise to first look into bugzilla next time I find a bug ;)
> https://bugzilla.clusterfs.com/show_bug.cgi?id=11544
> Currently there is no approved fix for the bug, but there is a patch that
> gets some extra debug information to help run down exactly what is going
> on. Hope this helps.
Ok, going to apply the patch now to see what happens.



-- 
Bernd Schubert
Q-Leap Networks GmbH

Lustre discuss - Mar 2007 - lustre errors

[Lustre-discuss] lustre errors

[Lustre-discuss] lustre errors

[Lustre-discuss] lustre errors

[Lustre-discuss] lustre errors

[Lustre-discuss] Lustre Errors

[Lustre-discuss] Lustre Errors

[Lustre-discuss] Lustre Errors