Lustre devel - May 2006 - Re: Lustre-1.6beta on 2.6.15/16?

I was able to use the http://mrvn.homeip.net/lustre/ patches to patch 
lustre and a 2.6.15.6 kernel, which went amazingly well (the few patch 
rejects against lustre were virtually all things that had already been 
adjusted, so those patches were unneeded).

Mountconf was very cool.  Note that the mkfs.lustre command doesn''t
seem
to deal with "--mgsnode=node1:node2" properly, although
"--mgsnode=node1
--mgsnode=node2" works fine.

Unfortunately, when actually writing to the filesystem, it turned out to 
be rather unstable (an mdt_thread would freak out and die), and, trying to 
export via NFS wouldn''t work (rpc.nfsd would freak).

Dang, so (seemingly) close...

Thanks,

Brent

PS Attached is the result of "dmesg |cut -c 16-500 |ksymoops" on the 
MDT/MGS node, if anyone has any insights.

On Tue, 2 May 2006, Brent A Nelson wrote:
> Has anyone succeeded in patching 2.6.15/16 kernels for Lustre-1.6beta2? Or,
> is anyone presently working on it?
>
> Thanks,
>
> Brent Nelson
> Director of Computing
> Dept. of Physics
> University of Florida
>
-------------- next part --------------
ksymoops 2.4.11 on i686 2.6.15.6-ubuntu1.060503.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.6.15.6-ubuntu1.060503/ (default)
     -m /boot/System.map-2.6.15.6-ubuntu1.060503 (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I''ll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Error (regular_file): read_ksyms stat /proc/ksyms failed
No modules in ksyms, skipping objects
No ksyms, skipping lsmod
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad8c973>] ldlm_cli_enqueue+0xb13/0x6620 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<faa9052c>] mds_fid2dentry+0x36c/0xca0 [mds]
 [<faa8ff10>] mds_fid2locked_dentry+0x1b0/0x460 [mds]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fab01546>] mds_open+0x6c6/0x69d0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<fad500f2>] l_unlock+0x32/0x160 [ptlrpc]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<faa9b7e0>] mds_reint+0x140/0xb10 [mds]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_27     S C180FE20     0  4442      1          4443  4441 (L-TLB)
Using defaults from ksymoops -t elf32-i386 -a i386
f71ed55c 00000001 c180f460 c180fe20 d01adacf d01adfff fa59b318 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000531 fa86ae5f 00000286 c180f460 00000001 00000949 50fadc41 00004193 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<fadfe837>] reply_in_callback+0x337/0xc50 [ptlrpc]
 [<c011c8cb>] activate_task+0x9b/0xe0
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<fa6f3c52>] lnet_finalize+0x142/0x250 [lnet]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_15     R running     0  4430      1          4431  4429 (L-TLB)
ll_mdt_31     S C1807E20     0  4446      1          4447  4445 (L-TLB)
f720555c 00000000 c1807460 c1807e20 d02ccc9b d02ccfff fa59c608 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000365 fa86ae5f 00000286 c1807460 00000000 00000a84 51b61d39 000041e1 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<fadfe837>] reply_in_callback+0x337/0xc50 [ptlrpc]
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<fa6f3c52>] lnet_finalize+0x142/0x250 [lnet]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_13     S C1807E20     0  4428      1          4429  4427 (L-TLB)
f719955c 00000000 c1807460 c1807e20 d032beba d032bfff fa59cf80 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000146 fa86ae5f 00000286 c1807460 00000000 00000869 95a67d91 000041ff 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<fadfe837>] reply_in_callback+0x337/0xc50 [ptlrpc]
 [<c011c8cb>] activate_task+0x9b/0xe0
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<fa6f3c52>] lnet_finalize+0x142/0x250 [lnet]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_26     R running     0  4441      1          4442  4440 (L-TLB)
ll_mdt_27     S F6D1F6E0     0  4442      1          4443  4441 (L-TLB)
f71ed618 f7af4a90 c180f460 f6d1f6e0 fad505c3 f6d1f704 0000115a 00000000 
       d0527570 d0527560 f71ed5c0 f71ed5c0 00000000 f7555a90 d0527540 fad5c551 
       f6d1f6e0 00000000 00000000 c180f460 00000001 00006991 cafdc163 00004240 
Call Trace:
 [<fad505c3>] l_check_no_ns_lock+0x13/0x238 [ptlrpc]
 [<fad5c551>] ldlm_run_ast_work+0x51/0x500 [ptlrpc]
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fad853db>] ldlm_completion_ast+0xd8b/0x3d20 [ptlrpc]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fad83330>] ldlm_expired_completion_wait+0x0/0x1320 [ptlrpc]
 [<fad83320>] interrupted_completion_wait+0x0/0x10 [ptlrpc]
 [<fad8a558>] ldlm_cli_enqueue_local+0xb38/0x1ae0 [ptlrpc]
 [<c0189c04>] dput+0x54/0x200
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad8c973>] ldlm_cli_enqueue+0xb13/0x6620 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<faa9052c>] mds_fid2dentry+0x36c/0xca0 [mds]
 [<faa8ff10>] mds_fid2locked_dentry+0x1b0/0x460 [mds]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fab01546>] mds_open+0x6c6/0x69d0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<fad500f2>] l_unlock+0x32/0x160 [ptlrpc]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<faa9b7e0>] mds_reint+0x140/0xb10 [mds]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
Warning (Oops_read): Code line not seen, dumping what data is available

Proc;  ll_mdt_27
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; fadfe837 <pg0+3a9ae837/3fbae400>
Trace; c011c8cb <activate_task+9b/e0>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; fa6f3c52 <pg0+3a2a3c52/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_15
>>EIP; 00000000 Before first symbol
Proc;  ll_mdt_31
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; fadfe837 <pg0+3a9ae837/3fbae400>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; fa6f3c52 <pg0+3a2a3c52/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_13
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; fadfe837 <pg0+3a9ae837/3fbae400>
Trace; c011c8cb <activate_task+9b/e0>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; fa6f3c52 <pg0+3a2a3c52/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_26
>>EIP; 00000000 Before first symbol
Proc;  ll_mdt_27
>>EIP; f6d1f6e0 <pg0+368cf6e0/3fbae400>   <====
Trace; fad505c3 <pg0+3a9005c3/3fbae400>
Trace; fad5c551 <pg0+3a90c551/3fbae400>
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fad853db <pg0+3a9353db/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fad83330 <pg0+3a933330/3fbae400>
Trace; fad83320 <pg0+3a933320/3fbae400>
Trace; fad8a558 <pg0+3a93a558/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad8c973 <pg0+3a93c973/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; faa9052c <pg0+3a64052c/3fbae400>
Trace; faa8ff10 <pg0+3a63ff10/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fab01546 <pg0+3a6b1546/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; fad500f2 <pg0+3a9000f2/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; faa9b7e0 <pg0+3a64b7e0/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>

ll_mdt_24     S C1807E20     0  4439      1          4440  4438 (L-TLB)
f71d755c 00000000 c1807460 c1807e20 cfcb3bbc cfcb3fff fa59b318 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000444 fa86ae5f 00000286 c1807460 00000000 00000a0d df90beca 0000426b 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c01243d2>] __call_console_drivers+0x62/0x70
 [<c011f120>] default_wake_function+0x0/0x20
 [<c0124883>] vprintk+0x1a3/0x2c0
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c011c8cb>] activate_task+0x9b/0xe0
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_00     R running     0  4415      1          4416  4409 (L-TLB)
ll_mdt_11     S C180FE20     0  4426      1          4427  4425 (L-TLB)
f718d55c 00000001 c180f460 c180fe20 cfd93161 cfd93fff fa59c608 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000e9f fa86ae5f 00000286 c180f460 00000001 000007c6 e3a22bdd 000042b9 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<fadfe837>] reply_in_callback+0x337/0xc50 [ptlrpc]
 [<c0114e4f>] smp_send_reschedule+0x2f/0x70
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<fa6f3c52>] lnet_finalize+0x142/0x250 [lnet]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<c011f181>] __wake_up_common+0x41/0x70
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_18     S C1807E20     0  4433      1          4434  4432 (L-TLB)
f71b355c 00000000 c1807460 c1807e20 cfe76664 cfe76fff fa59cf80 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       0000099c fa86ae5f 00000286 c1807460 00000000 000008f5 2a6ee3d9 000042d8 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c01559ea>] cache_alloc_refill+0x5a/0x210
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
Warning (Oops_read): Code line not seen, dumping what data is available

Proc;  ll_mdt_24
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c01243d2 <__call_console_drivers+62/70>
Trace; c011f120 <default_wake_function+0/20>
Trace; c0124883 <vprintk+1a3/2c0>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c011c8cb <activate_task+9b/e0>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_00
>>EIP; 00000000 Before first symbol
Proc;  ll_mdt_11
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; fadfe837 <pg0+3a9ae837/3fbae400>
Trace; c0114e4f <smp_send_reschedule+2f/70>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; fa6f3c52 <pg0+3a2a3c52/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; c011f181 <__wake_up_common+41/70>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_18
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c01559ea <cache_alloc_refill+5a/210>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>

ll_mdt_29     S C180FE20     0  4444      1          4445  4443 (L-TLB)
f71f955c 00000001 c180f460 c180fe20 cff42e3c cff42fff fa59a9a0 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       000001c4 fa86ae5f 00000286 c180f460 00000001 000007df 65150a88 00004312 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c014ff04>] bad_range+0x34/0x50
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<c011f181>] __wake_up_common+0x41/0x70
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_06     S C180FE20     0  4421      1          4422  4420 (L-TLB)
f716b618 00000001 c180f460 c180fe20 fad505c3 f6d1f704 00001145 00000000 
       d7c1caf0 d7c1cae0 f716b5c0 f716b5c0 d4ec67c0 d7c1caf0 d7c1cac0 fad5c551 
       f6d1f6e0 d4ec67c0 f716b604 c180f460 00000001 00006aa4 36c6ae2b 00004318 
Call Trace:
 [<fad505c3>] l_check_no_ns_lock+0x13/0x238 [ptlrpc]
 [<fad5c551>] ldlm_run_ast_work+0x51/0x500 [ptlrpc]
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fad853db>] ldlm_completion_ast+0xd8b/0x3d20 [ptlrpc]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fad83330>] ldlm_expired_completion_wait+0x0/0x1320 [ptlrpc]
 [<fad83320>] interrupted_completion_wait+0x0/0x10 [ptlrpc]
 [<fad8a558>] ldlm_cli_enqueue_local+0xb38/0x1ae0 [ptlrpc]
 [<c0189c04>] dput+0x54/0x200
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad8c973>] ldlm_cli_enqueue+0xb13/0x6620 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<faa9052c>] mds_fid2dentry+0x36c/0xca0 [mds]
 [<faa8ff10>] mds_fid2locked_dentry+0x1b0/0x460 [mds]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fab01546>] mds_open+0x6c6/0x69d0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<fad500f2>] l_unlock+0x32/0x160 [ptlrpc]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<c0150d20>] get_page_from_freelist+0x70/0xe0
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
Warning (Oops_read): Code line not seen, dumping what data is available

Proc;  ll_mdt_29
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c014ff04 <bad_range+34/50>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; c011f181 <__wake_up_common+41/70>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_06
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; fad505c3 <pg0+3a9005c3/3fbae400>
Trace; fad5c551 <pg0+3a90c551/3fbae400>
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fad853db <pg0+3a9353db/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fad83330 <pg0+3a933330/3fbae400>
Trace; fad83320 <pg0+3a933320/3fbae400>
Trace; fad8a558 <pg0+3a93a558/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad8c973 <pg0+3a93c973/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; faa9052c <pg0+3a64052c/3fbae400>
Trace; faa8ff10 <pg0+3a63ff10/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fab01546 <pg0+3a6b1546/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; fad500f2 <pg0+3a9000f2/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; c0150d20 <get_page_from_freelist+70/e0>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>

ll_mdt_14     S C1807E20     0  4429      1          4430  4428 (L-TLB)
f719b55c 00000000 c1807460 c1807e20 cf8745ab cf874fff fa59b318 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000a55 fa86ae5f 00000286 c1807460 00000000 000009dd 4bd38984 00004343 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<f8989c4c>] e1000_intr+0x7c/0x130 [e1000]
 [<c0236771>] add_interrupt_randomness+0x31/0x40
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_16     R running     0  4431      1          4432  4430 (L-TLB)
ll_mdt_19     S C1807E20     0  4434      1          4435  4433 (L-TLB)
f71bd55c 00000000 c1807460 c1807e20 cf978162 cf978fff fa59c608 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000e9e fa86ae5f 00000286 c1807460 00000000 0000095a 4bd7b13d 00004391 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c01243d2>] __call_console_drivers+0x62/0x70
 [<c011f120>] default_wake_function+0x0/0x20
 [<c0124883>] vprintk+0x1a3/0x2c0
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_15     S C180FE20     0  4430      1          4431  4429 (L-TLB)
f71a555c 00000001 c180f460 c180fe20 cf9da170 cf9dafff fa59cf80 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000e90 fa86ae5f 00000286 c180f460 00000001 0000086d 907f350b 000043af 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<c0150d20>] get_page_from_freelist+0x70/0xe0
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_26     S C180FE20     0  4441      1          4442  4440 (L-TLB)
f71e355c 00000001 c180f460 c180fe20 cf8df93c cf8dffff fa59a9a0 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       000006c4 fa86ae5f 00000286 c180f460 00000001 00000863 f7b8413a 000043ea 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0114e4f>] smp_send_reschedule+0x2f/0x70
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<c0150d20>] get_page_from_freelist+0x70/0xe0
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_04     S C1807E20     0  4419      1          4420  4418 (L-TLB)
f715f618 00000000 c1807460 c1807e20 fad505c3 f6d1f704 00001143 00000000 
       d74c1eb0 d74c1ea0 f715f5c0 f715f5c0 d1edd4e0 d74c1eb0 d74c1e80 fad5c551 
       f6d1f6e0 d1edd4e0 f715f604 c1807460 00000000 000073be c9ba4418 000043f0 
Call Trace:
 [<fad505c3>] l_check_no_ns_lock+0x13/0x238 [ptlrpc]
 [<fad5c551>] ldlm_run_ast_work+0x51/0x500 [ptlrpc]
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fad853db>] ldlm_completion_ast+0xd8b/0x3d20 [ptlrpc]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fad83330>] ldlm_expired_completion_wait+0x0/0x1320 [ptlrpc]
 [<fad83320>] interrupted_completion_wait+0x0/0x10 [ptlrpc]
 [<fad8a558>] ldlm_cli_enqueue_local+0xb38/0x1ae0 [ptlrpc]
 [<c0189c04>] dput+0x54/0x200
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad8c973>] ldlm_cli_enqueue+0xb13/0x6620 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<faa9052c>] mds_fid2dentry+0x36c/0xca0 [mds]
 [<faa8ff10>] mds_fid2locked_dentry+0x1b0/0x460 [mds]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fab01546>] mds_open+0x6c6/0x69d0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<c0129d31>] __do_softirq+0x81/0x100
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<fad500f2>] l_unlock+0x32/0x160 [ptlrpc]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<faa9b7e0>] mds_reint+0x140/0xb10 [mds]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
Warning (Oops_read): Code line not seen, dumping what data is available

Proc;  ll_mdt_14
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; f8989c4c <pg0+38539c4c/3fbae400>
Trace; c0236771 <add_interrupt_randomness+31/40>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_16
>>EIP; 00000000 Before first symbol
Proc;  ll_mdt_19
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c01243d2 <__call_console_drivers+62/70>
Trace; c011f120 <default_wake_function+0/20>
Trace; c0124883 <vprintk+1a3/2c0>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_15
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; c0150d20 <get_page_from_freelist+70/e0>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_26
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0114e4f <smp_send_reschedule+2f/70>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; c0150d20 <get_page_from_freelist+70/e0>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_04
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; fad505c3 <pg0+3a9005c3/3fbae400>
Trace; fad5c551 <pg0+3a90c551/3fbae400>
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fad853db <pg0+3a9353db/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fad83330 <pg0+3a933330/3fbae400>
Trace; fad83320 <pg0+3a933320/3fbae400>
Trace; fad8a558 <pg0+3a93a558/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad8c973 <pg0+3a93c973/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; faa9052c <pg0+3a64052c/3fbae400>
Trace; faa8ff10 <pg0+3a63ff10/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fab01546 <pg0+3a6b1546/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; c0129d31 <__do_softirq+81/100>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; fad500f2 <pg0+3a9000f2/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; faa9b7e0 <pg0+3a64b7e0/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>

ll_mdt_11     S C180FE20     0  4426      1          4427  4425 (L-TLB)
f718d55c 00000001 c180f460 c180fe20 cfb3a829 cfb3afff fa59b318 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       000007d7 fa86ae5f 00000286 c180f460 00000001 000007cb b38eede6 0000441a 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_16     S C180FE20     0  4431      1          4432  4430 (L-TLB)
f71a755c 00000001 c180f460 c180fe20 cf4875ab cf487fff fa59bc90 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000a55 fa86ae5f 00000286 c180f460 00000001 00000818 9bbb22e7 00004444 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c01243d2>] __call_console_drivers+0x62/0x70
 [<c011f120>] default_wake_function+0x0/0x20
 [<c0124883>] vprintk+0x1a3/0x2c0
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0114e4f>] smp_send_reschedule+0x2f/0x70
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_30     S C1807E20     0  4445      1          4446  4444 (L-TLB)
f71fb55c 00000000 c1807460 c1807e20 cf52e986 cf52efff fa59c608 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       0000067a fa86ae5f 00000286 c1807460 00000000 0000096d b393233d 00004468 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_03     S C1807E20     0  4418      1          4419  4417 (L-TLB)
f715d55c 00000000 c1807460 c1807e20 cf58f5ac cf58ffff fa59cf80 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000a54 fa86ae5f 00000286 c1807460 00000000 000009a0 fa5fe0c0 00004486 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<fadfe837>] reply_in_callback+0x337/0xc50 [ptlrpc]
 [<c011c8cb>] activate_task+0x9b/0xe0
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c014ff04>] bad_range+0x34/0x50
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_25     S C180FE20     0  4440      1          4441  4439 (L-TLB)
f71e155c 00000001 c180f460 c180fe20 cf5f7e5d cf5f7fff fa59a9a0 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       000001a3 fa86ae5f 00000286 c180f460 00000001 000007c9 3505fb14 000044c1 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_28     S 000044C7     0  4443      1          4444  4442 (L-TLB)
f71ef618 f755c550 c180f4a4 000044c7 fad505c3 f6d1f704 0000115b 00000000 
       d74c1eb0 d74c1ea0 06f4861a 000044c7 00000002 f755c550 d74c1e80 fad5c551 
       f6d1f6e0 00000000 00000000 c180f460 00000001 0004cb6f 06f4861a 000044c7 
Call Trace:
 [<fad505c3>] l_check_no_ns_lock+0x13/0x238 [ptlrpc]
 [<fad5c551>] ldlm_run_ast_work+0x51/0x500 [ptlrpc]
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fad853db>] ldlm_completion_ast+0xd8b/0x3d20 [ptlrpc]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fad83330>] ldlm_expired_completion_wait+0x0/0x1320 [ptlrpc]
 [<fad83320>] interrupted_completion_wait+0x0/0x10 [ptlrpc]
 [<fad8a558>] ldlm_cli_enqueue_local+0xb38/0x1ae0 [ptlrpc]
 [<c0189c04>] dput+0x54/0x200
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad8c973>] ldlm_cli_enqueue+0xb13/0x6620 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<faa9052c>] mds_fid2dentry+0x36c/0xca0 [mds]
 [<faa8ff10>] mds_fid2locked_dentry+0x1b0/0x460 [mds]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fab01546>] mds_open+0x6c6/0x69d0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<fad500f2>] l_unlock+0x32/0x160 [ptlrpc]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<faa9b7e0>] mds_reint+0x140/0xb10 [mds]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
Warning (Oops_read): Code line not seen, dumping what data is available

Proc;  ll_mdt_11
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_16
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c01243d2 <__call_console_drivers+62/70>
Trace; c011f120 <default_wake_function+0/20>
Trace; c0124883 <vprintk+1a3/2c0>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0114e4f <smp_send_reschedule+2f/70>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_30
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_03
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; fadfe837 <pg0+3a9ae837/3fbae400>
Trace; c011c8cb <activate_task+9b/e0>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c014ff04 <bad_range+34/50>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_25
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_28
>>EIP; 000044c7 Before first symbol   <====
Trace; fad505c3 <pg0+3a9005c3/3fbae400>
Trace; fad5c551 <pg0+3a90c551/3fbae400>
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fad853db <pg0+3a9353db/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fad83330 <pg0+3a933330/3fbae400>
Trace; fad83320 <pg0+3a933320/3fbae400>
Trace; fad8a558 <pg0+3a93a558/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad8c973 <pg0+3a93c973/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; faa9052c <pg0+3a64052c/3fbae400>
Trace; faa8ff10 <pg0+3a63ff10/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fab01546 <pg0+3a6b1546/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; fad500f2 <pg0+3a9000f2/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; faa9b7e0 <pg0+3a64b7e0/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>

ll_mdt_16     S C180FE20     0  4431      1          4432  4430 (L-TLB)
f71a755c 00000001 c180f460 c180fe20 cf69fe8e cf69ffff fa59b318 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000172 fa86ae5f 00000286 c180f460 00000001 00000873 f119bddd 000044f0 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c01243d2>] __call_console_drivers+0x62/0x70
 [<c011f120>] default_wake_function+0x0/0x20
 [<c0124883>] vprintk+0x1a3/0x2c0
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c011c8cb>] activate_task+0x9b/0xe0
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_21     S C1807E20     0  4436      1          4437  4435 (L-TLB)
f71c955c 00000000 c1807460 c1807e20 cf7c68e3 cf7c6fff fa59bc90 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       0000071d fa86ae5f 00000286 c1807460 00000000 00000a15 d945f2a6 0000451a 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c068e8>] cfs_alloc+0x58/0xa0 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0286100>] kfree_skbmem+0x10/0x90
 [<c028c45e>] net_tx_action+0x4e/0x140
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0103d1e>] common_interrupt+0x1a/0x20
 [<fa6f3c52>] lnet_finalize+0x142/0x250 [lnet]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_17     S C1807E20     0  4432      1          4433  4431 (L-TLB)
f71b155c 00000000 c1807460 c1807e20 cf0350b6 cf035fff fa59c608 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000f4a fa86ae5f 00000286 c1807460 00000000 00000b20 efafd8a8 0000453e 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c011e1f1>] find_busiest_group+0x121/0x400
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c02ff00c>] schedule+0x6cc/0xd90
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_25     S C180FE20     0  4440      1          4441  4439 (L-TLB)
f71e155c 00000001 c180f460 c180fe20 cf7cc6c4 cf7ccfff fa59cf80 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       0000093c fa86ae5f 00000286 c180f460 00000001 000006f7 341a366b 0000455d 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>]<4>Lustre:
4424:0:(ldlm_lib.c:484:target_handle_reconnect()) lustre-MDT0000:
1051db53-1e27-65cf-afa9-573d254fdebb reconnecting
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0114e4f>] smp_send_reschedule+0x2f/0x70
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_02     S C1807E20     0  4417      1          4418  4416 (L-TLB)
f715355c 00000000 c1807460 c1807e20 cf1629d3 cf162fff fa59a9a0 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       0000062d fa86ae5f 00000286 c1807460 00000000 00000af3 9a5f541d 00004598 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_00     S 0000459E     0  4415      1          4416  4409 (L-TLB)
f714b618 f7cff570 c180791c 0000459e fad505c3 f6d1f704 0000113f 00000000 
       d0527930 d0527920 6c91af65 0000459e 00000002 f7cff570 d0527900 fad5c551 
       f6d1f6e0 00000000 00000000 c1807460 00000000 0005825e 6c91af65 0000459e 
Call Trace:
 [<fad505c3>] l_check_no_ns_lock+0x13/0x238 [ptlrpc]
 [<fad5c551>] ldlm_run_ast_work+0x51/0x500 [ptlrpc]
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fad853db>] ldlm_completion_ast+0xd8b/0x3d20 [ptlrpc]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fad83330>] ldlm_expired_completion_wait+0x0/0x1320 [ptlrpc]
 [<fad83320>] interrupted_completion_wait+0x0/0x10 [ptlrpc]
 [<fad8a558>] ldlm_cli_enqueue_local+0xb38/0x1ae0 [ptlrpc]
 [<c0189c04>] dput+0x54/0x200
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad8c973>] ldlm_cli_enqueue+0xb13/0x6620 [ptlrpc]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<faa9052c>] mds_fid2dentry+0x36c/0xca0 [mds]
 [<c0150de7>] __alloc_pages+0x57/0x320
 [<f8c068e8>] cfs_alloc+0x58/0xa0 [libcfs]
 [<faa8ff10>] mds_fid2locked_dentry+0x1b0/0x460 [mds]
 [<fad88370>] ldlm_blocking_ast+0x0/0x16a0 [ptlrpc]
 [<fad84650>] ldlm_completion_ast+0x0/0x3d20 [ptlrpc]
 [<fab01546>] mds_open+0x6c6/0x69d0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<fad500f2>] l_unlock+0x32/0x160 [ptlrpc]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<faa9b7e0>] mds_reint+0x140/0xb10 [mds]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
Warning (Oops_read): Code line not seen, dumping what data is available

Proc;  ll_mdt_16
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c01243d2 <__call_console_drivers+62/70>
Trace; c011f120 <default_wake_function+0/20>
Trace; c0124883 <vprintk+1a3/2c0>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c011c8cb <activate_task+9b/e0>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_21
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c068e8 <pg0+387b68e8/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0286100 <kfree_skbmem+10/90>
Trace; c028c45e <net_tx_action+4e/140>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0103d1e <common_interrupt+1a/20>
Trace; fa6f3c52 <pg0+3a2a3c52/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_17
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c011e1f1 <find_busiest_group+121/400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c02ff00c <schedule+6cc/d90>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_25
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0114e4f <smp_send_reschedule+2f/70>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_02
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_00
>>EIP; 0000459e Before first symbol   <====
Trace; fad505c3 <pg0+3a9005c3/3fbae400>
Trace; fad5c551 <pg0+3a90c551/3fbae400>
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fad853db <pg0+3a9353db/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fad83330 <pg0+3a933330/3fbae400>
Trace; fad83320 <pg0+3a933320/3fbae400>
Trace; fad8a558 <pg0+3a93a558/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad8c973 <pg0+3a93c973/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; faa9052c <pg0+3a64052c/3fbae400>
Trace; c0150de7 <__alloc_pages+57/320>
Trace; f8c068e8 <pg0+387b68e8/3fbae400>
Trace; faa8ff10 <pg0+3a63ff10/3fbae400>
Trace; fad88370 <pg0+3a938370/3fbae400>
Trace; fad84650 <pg0+3a934650/3fbae400>
Trace; fab01546 <pg0+3a6b1546/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; fad500f2 <pg0+3a9000f2/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; faa9b7e0 <pg0+3a64b7e0/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>

ll_mdt_04     S C180FE20     0  4419      1          4420  4418 (L-TLB)
f715f55c 00000001 c180f460 c180fe20 cf2d23e7 cf2d2fff fa59b318 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000c19 fa86ae5f 00000286 c180f460 00000001 000007c5 81d4b901 000045c9 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c011c8cb>] activate_task+0x9b/0xe0
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_12     S C1807E20     0  4427      1          4428  4426 (L-TLB)
f718f55c 00000000 c1807460 c1807e20 cf35c664 cf35cfff fa59bc90 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       0000099c fa86ae5f 00000286 c1807460 00000000 00000a4f 6a010251 000045f3 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0286100>] kfree_skbmem+0x10/0x90
 [<c028c45e>] net_tx_action+0x4e/0x140
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<c0150d20>] get_page_from_freelist+0x70/0xe0
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_26     R running     0  4441      1          4442  4440 (L-TLB)
ll_mdt_20     S C1807E20     0  4435      1          4436  4434 (L-TLB)
f71bf55c 00000000 c1807460 c1807e20 cec00664 cec00fff fa59cf80 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       0000099c fa86ae5f 00000286 c1807460 00000000 000009c4 c3e13ca4 00004635 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c068e8>] cfs_alloc+0x58/0xa0 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<f8989d75>] e1000_clean_tx_irq+0x75/0x290 [e1000]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_17     S C1807E20     0  4432      1          4433  4431 (L-TLB)
f71b155c 00000000 c1807460 c1807e20 cecf1ddb cecf1fff fa59a9a0 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       00000225 fa86ae5f 00000286 c1807460 00000000 000008be 2bd1524e 00004671 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c068e8>] cfs_alloc+0x58/0xa0 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
Warning (Oops_read): Code line not seen, dumping what data is available

Proc;  ll_mdt_04
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c011c8cb <activate_task+9b/e0>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_12
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0286100 <kfree_skbmem+10/90>
Trace; c028c45e <net_tx_action+4e/140>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; c0150d20 <get_page_from_freelist+70/e0>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_26
>>EIP; 00000000 Before first symbol
Proc;  ll_mdt_20
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c068e8 <pg0+387b68e8/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; f8989d75 <pg0+38539d75/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_17
>>EIP; c1807e20 <pg0+13b7e20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c068e8 <pg0+387b68e8/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>

ll_mdt_26     R running     0  4441      1          4442  4440 (L-TLB)
ll_mdt_23     S C180FE20     0  4438      1          4439  4437 (L-TLB)
f71d555c 00000001 c180f460 c180fe20 ced84604 ced84fff fa59bc90 00000000 
       00000010 00000008 00000007 00000001 ffffffff ffffffff 00000008 00000048 
       000009fc fa86ae5f 00000286 c180f460 00000001 0000085b d0113e05 000046ca 
Call Trace:
 [<c02ffeb1>] schedule_timeout+0x51/0xa0
 [<c012ebe0>] process_timeout+0x0/0x10
 [<c013b3fd>] add_wait_queue+0x1d/0x50
 [<fa868460>] osc_create+0x850/0x2bb0 [osc]
 [<c01243d2>] __call_console_drivers+0x62/0x70
 [<c011f120>] default_wake_function+0x0/0x20
 [<c0124883>] vprintk+0x1a3/0x2c0
 [<fa7fa544>] qos_remedy_create+0x6d4/0xdd0 [lov]
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fa7e62cc>] create_done+0xec/0x2a20 [lov]
 [<fa7e8d19>] lov_fini_create_set+0x119/0x350 [lov]
 [<fa7aeb88>] lov_create+0x5e8/0x11b0 [lov]
 [<f88d5cbb>] do_get_write_access+0x39b/0x560 [jbd]
 [<faaf7a4e>] mds_create_objects+0x421e/0x9d10 [mds]
 [<faa474c0>] ldiskfs_xattr_get+0xa0/0xb0 [ldiskfs]
 [<fa89a9a0>] fsfilt_ldiskfs_get_md+0x50/0x1d0 [fsfilt_ldiskfs]
 [<faaff096>] mds_finish_open+0x596/0xee0 [mds]
 [<fab02dfc>] mds_open+0x1f7c/0x69d0 [mds]
 [<c011c8cb>] activate_task+0x9b/0xe0
 [<c011d384>] try_to_wake_up+0x334/0x420
 [<c0134a4e>] groups_alloc+0x3e/0xb0
 [<f8b8a97e>] entry_set_group_info+0xfe/0x7a0 [lvfs]
 [<c0189c04>] dput+0x54/0x200
 [<f8b852b3>] push_ctxt+0x143/0x2b0 [lvfs]
 [<c018f14d>] mntput_no_expire+0x1d/0xc0
 [<faadfbcc>] mds_reint_rec+0x10c/0x3e6 [mds]
 [<fadfb860>] lustre_swab_mds_rec_create+0x0/0xa0 [ptlrpc]
 [<faa9b77d>] mds_reint+0xdd/0xb10 [mds]
 [<faab6999>] mds_intent_policy+0xcf9/0x1cc0 [mds]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<fad5b9b2>] ldlm_lock_enqueue+0x192/0x9b0 [ptlrpc]
 [<fadad976>] ldlm_handle_enqueue+0x1556/0x68f0 [ptlrpc]
 [<fadf3535>] ptlrpc_send_reply+0x185/0x670 [ptlrpc]
 [<faaa5d04>] mds_handle+0x6764/0xb500 [mds]
 [<fadaa060>] ldlm_server_completion_ast+0x0/0x1680 [ptlrpc]
 [<fada8820>] ldlm_server_blocking_ast+0x0/0x1820 [ptlrpc]
 [<c01f5053>] vsnprintf+0x2f3/0x550
 [<f8c1312c>] libcfs_debug_msg+0x30c/0x360 [libcfs]
 [<fae05a07>] ptlrpc_server_handle_request+0x797/0x2210 [ptlrpc]
 [<c012e054>] lock_timer_base+0x24/0x50
 [<c012e120>] __mod_timer+0xa0/0xd0
 [<f8c16bf1>] lc_watchdog_touch+0xa1/0x250 [libcfs]
 [<fae087cd>] ptlrpc_main+0x7dd/0xfd0 [ptlrpc]
 [<c011f120>] default_wake_function+0x0/0x20
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<c01031e2>] ret_from_fork+0x6/0x14
 [<fae07fd0>] ptlrpc_retry_rqbds+0x0/0x20 [ptlrpc]
 [<fae07ff0>] ptlrpc_main+0x0/0xfd0 [ptlrpc]
 [<c01014d5>] kernel_thread_helper+0x5/0x10
ll_mdt_12     R running     0  4427      1          4428  4426 (L-TLB)
ll_mdt_16     R running     0  4431      1          4432  4430 (L-TLB)
Warning (Oops_read): Code line not seen, dumping what data is available

Proc;  ll_mdt_26
>>EIP; 00000000 Before first symbol
Proc;  ll_mdt_23
>>EIP; c180fe20 <pg0+13bfe20/3fbae400>   <====
Trace; c02ffeb1 <schedule_timeout+51/a0>
Trace; c012ebe0 <process_timeout+0/10>
Trace; c013b3fd <add_wait_queue+1d/50>
Trace; fa868460 <pg0+3a418460/3fbae400>
Trace; c01243d2 <__call_console_drivers+62/70>
Trace; c011f120 <default_wake_function+0/20>
Trace; c0124883 <vprintk+1a3/2c0>
Trace; fa7fa544 <pg0+3a3aa544/3fbae400>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fa7e62cc <pg0+3a3962cc/3fbae400>
Trace; fa7e8d19 <pg0+3a398d19/3fbae400>
Trace; fa7aeb88 <pg0+3a35eb88/3fbae400>
Trace; f88d5cbb <pg0+38485cbb/3fbae400>
Trace; faaf7a4e <pg0+3a6a7a4e/3fbae400>
Trace; faa474c0 <pg0+3a5f74c0/3fbae400>
Trace; fa89a9a0 <pg0+3a44a9a0/3fbae400>
Trace; faaff096 <pg0+3a6af096/3fbae400>
Trace; fab02dfc <pg0+3a6b2dfc/3fbae400>
Trace; c011c8cb <activate_task+9b/e0>
Trace; c011d384 <try_to_wake_up+334/420>
Trace; c0134a4e <groups_alloc+3e/b0>
Trace; f8b8a97e <pg0+3873a97e/3fbae400>
Trace; c0189c04 <dput+54/200>
Trace; f8b852b3 <pg0+387352b3/3fbae400>
Trace; c018f14d <mntput_no_expire+1d/c0>
Trace; faadfbcc <pg0+3a68fbcc/3fbae400>
Trace; fadfb860 <pg0+3a9ab860/3fbae400>
Trace; faa9b77d <pg0+3a64b77d/3fbae400>
Trace; faab6999 <pg0+3a666999/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; fad5b9b2 <pg0+3a90b9b2/3fbae400>
Trace; fadad976 <pg0+3a95d976/3fbae400>
Trace; fadf3535 <pg0+3a9a3535/3fbae400>
Trace; faaa5d04 <pg0+3a655d04/3fbae400>
Trace; fadaa060 <pg0+3a95a060/3fbae400>
Trace; fada8820 <pg0+3a958820/3fbae400>
Trace; c01f5053 <vsnprintf+2f3/550>
Trace; f8c1312c <pg0+387c312c/3fbae400>
Trace; fae05a07 <pg0+3a9b5a07/3fbae400>
Trace; c012e054 <lock_timer_base+24/50>
Trace; c012e120 <__mod_timer+a0/d0>
Trace; f8c16bf1 <pg0+387c6bf1/3fbae400>
Trace; fae087cd <pg0+3a9b87cd/3fbae400>
Trace; c011f120 <default_wake_function+0/20>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; c01031e2 <ret_from_fork+6/14>
Trace; fae07fd0 <pg0+3a9b7fd0/3fbae400>
Trace; fae07ff0 <pg0+3a9b7ff0/3fbae400>
Trace; c01014d5 <kernel_thread_helper+5/10>
Proc;  ll_mdt_12
>>EIP; 00000000 Before first symbol
Proc;  ll_mdt_16
>>EIP; 00000000 Before first symbol

9 warnings and 1 error issued.  Results may not be reliable.

Brent A Nelson <brent@phys.ufl.edu> writes:
> I was able to use the http://mrvn.homeip.net/lustre/ patches to patch
> lustre and a 2.6.15.6 kernel, which went amazingly well (the few patch
> rejects against lustre were virtually all things that had already been
> adjusted, so those patches were unneeded).
>
> Mountconf was very cool.  Note that the mkfs.lustre command
doesn''t
> seem to deal with "--mgsnode=node1:node2" properly, although
> "--mgsnode=node1 --mgsnode=node2" works fine.
>
> Unfortunately, when actually writing to the filesystem, it turned out
> to be rather unstable (an mdt_thread would freak out and die), and,
> trying to export via NFS wouldn''t work (rpc.nfsd would freak).
>
> Dang, so (seemingly) close...
>
> Thanks,
>
> Brent
Must be something new in the lustre beta. :( We did quite some testing
with the released one without problems. No nfs reexport though.

I guess I can put off testing it for a while till someone finds the
bug. One less thing on my ToDo.

MfG
        Goswin

There isn''t any problem running with a kernel HZ of 250 instead of 100,
is
there? That is one thing I tweaked in my kernel config (and it seems 
rather related to schedule_timeout mentioned in my oops data for the 
mdt_threads)...

Also, I wanted to mention that I was exceptionally daring, and tried the 
tcp-zero-copy patch that was posted not long ago for 2.6.12 on this 
kernel (2.6.15.6), but it oopsed on boot (mentioning sk_release_data).

Thanks,

Brent

On Thu, 4 May 2006, Brent A Nelson wrote:
> I was able to use the http://mrvn.homeip.net/lustre/ patches to patch
lustre
> and a 2.6.15.6 kernel, which went amazingly well (the few patch rejects 
> against lustre were virtually all things that had already been adjusted, so
> those patches were unneeded).
>
> Mountconf was very cool.  Note that the mkfs.lustre command
doesn''t seem to
> deal with "--mgsnode=node1:node2" properly, although
"--mgsnode=node1
> --mgsnode=node2" works fine.
>
> Unfortunately, when actually writing to the filesystem, it turned out to be
> rather unstable (an mdt_thread would freak out and die), and, trying to 
> export via NFS wouldn''t work (rpc.nfsd would freak).
>
> Dang, so (seemingly) close...
>
> Thanks,
>
> Brent
>
> PS Attached is the result of "dmesg |cut -c 16-500 |ksymoops" on
the MDT/MGS
> node, if anyone has any insights.
>
> On Tue, 2 May 2006, Brent A Nelson wrote:
>
>> Has anyone succeeded in patching 2.6.15/16 kernels for Lustre-1.6beta2?
Or,
>> is anyone presently working on it?
>> 
>> Thanks,
>> 
>> Brent Nelson
>> Director of Computing
>> Dept. of Physics
>> University of Florida
>

Yep, I''ve also been using the patchless beta, but I''d like to
be ready for
the 1.6 release, which should be out by the time I deploy my setup.

I haven''t done heavy testing, but you might be interested to know that 
I''ve been running patchless SERVER (2.6.15 kernel is claimed to be 
required, as is one tiny patch to the kernel) with no real problems (one 
interesting looking error popped up once, but it didn''t seem to hurt 
anything and may have been due to me mixing a 1.4.6 patchless MDS with 
some 1.4.6.1 patched servers or possibly a bad ethernet driver).  I 
haven''t tinkered with it, so I don''t know if it works, but I
even enabled
quota support.

With the patchless setup, it was surprisingly easy to setup NFS exporting 
with an LVS director.  Very cool, indeed.

Thanks,

Brent

PS One gotcha.  Lustre (either patched or patchless) doesn''t seem to
like
gcc-4 (at least not the gcc-4.0.3 on my Ubuntu Dapper setup); OSS servers 
work fine with it, but the MDS dies immediately with an OOPS from an 
mdt-thread.  I''m not sure about the client.

On Tue, 2 May 2006, EKC wrote:
> Hi,
>
> I''ve got the Lustre patchless beta release running as a Lustre
client
> on linux-2.6.17-rc1 and it works like a charm. I''m running the
> production Lustre 1.4.6 release on linux-2.6.12 for my Lustre OSS/OST
> and MDS nodes. I''ve got some production code that requires the
latest
> linux kernel release and the Lustre patchless beta code has played
> along really well. Although Lustre patchless is _not_ production code,
> I haven''t run into any major bugs yet.
>
> EKC
>
> On 5/2/06, Brent A Nelson <brent@phys.ufl.edu> wrote:
>> Has anyone succeeded in patching 2.6.15/16 kernels for Lustre-1.6beta2?
>> Or, is anyone presently working on it?
>> 
>> Thanks,
>> 
>> Brent Nelson
>> Director of Computing
>> Dept. of Physics
>> University of Florida
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss@clusterfs.com
>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>> 
>> 
>

Hi 

We haven''t had much time to run a patchless server ourselves.  Let me
just draw your attention to two gotchas:

1. we have seen occasional stack overflow - this basically crashes your
server and we still need to ensure this cannot happen.

2. on servers there is an "invisible" patch required for block
devices,
without having this patch you cannot really use failover (unless you
make special arrangements). The patch is present to safely perform
failover from a running server to another and disables writes to a block
device.  We are working on a device mapper which would do the same -
with that, you can run the server over the mapped block device and
achieve the same.

It is our aim, of course, to move in the direction of patchless
everything!

- Peter -

 
> -----Original Message-----
> From: lustre-discuss-bounces@clusterfs.com 
> [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of 
> Brent A Nelson
> Sent: Tuesday, May 02, 2006 5:13 PM
> To: EKC
> Cc: lustre-discuss@clusterfs.com
> Subject: Re: [Lustre-discuss] Lustre-1.6beta on 2.6.15/16?
> 
> Yep, I''ve also been using the patchless beta, but I''d
like to
> be ready for the 1.6 release, which should be out by the time 
> I deploy my setup.
> 
> I haven''t done heavy testing, but you might be interested to 
> know that I''ve been running patchless SERVER (2.6.15 kernel 
> is claimed to be required, as is one tiny patch to the 
> kernel) with no real problems (one interesting looking error 
> popped up once, but it didn''t seem to hurt anything and may 
> have been due to me mixing a 1.4.6 patchless MDS with some 
> 1.4.6.1 patched servers or possibly a bad ethernet driver).  
> I haven''t tinkered with it, so I don''t know if it works,
but
> I even enabled quota support.
> 
> With the patchless setup, it was surprisingly easy to setup 
> NFS exporting with an LVS director.  Very cool, indeed.
> 
> Thanks,
> 
> Brent
> 
> PS One gotcha.  Lustre (either patched or patchless) doesn''t 
> seem to like
> gcc-4 (at least not the gcc-4.0.3 on my Ubuntu Dapper setup); 
> OSS servers work fine with it, but the MDS dies immediately 
> with an OOPS from an mdt-thread.  I''m not sure about the client.
> 
> On Tue, 2 May 2006, EKC wrote:
> 
> > Hi,
> >
> > I''ve got the Lustre patchless beta release running as a 
> Lustre client 
> > on linux-2.6.17-rc1 and it works like a charm. I''m running
the
> > production Lustre 1.4.6 release on linux-2.6.12 for my 
> Lustre OSS/OST 
> > and MDS nodes. I''ve got some production code that requires 
> the latest 
> > linux kernel release and the Lustre patchless beta code has played 
> > along really well. Although Lustre patchless is _not_ 
> production code, 
> > I haven''t run into any major bugs yet.
> >
> > EKC
> >
> > On 5/2/06, Brent A Nelson <brent@phys.ufl.edu> wrote:
> >> Has anyone succeeded in patching 2.6.15/16 kernels for 
> Lustre-1.6beta2?
> >> Or, is anyone presently working on it?
> >> 
> >> Thanks,
> >> 
> >> Brent Nelson
> >> Director of Computing
> >> Dept. of Physics
> >> University of Florida
> >> _______________________________________________
> >> Lustre-discuss mailing list
> >> Lustre-discuss@clusterfs.com
> >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> >> 
> >> 
> >
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> 
>

Has anyone succeeded in patching 2.6.15/16 kernels for Lustre-1.6beta2? 
Or, is anyone presently working on it?

Thanks,

Brent Nelson
Director of Computing
Dept. of Physics
University of Florida