Ocfs2 users - Dec 2011 - ocfs2 - Kernel panic on many write/read from both servers

I do for all night tests with write/read files from ocfs2 on both servers 
something like this:
On MAIL1 server:
#!/bin/bash
while true
do
rm -rf /mnt/EMC/MAIL1
mkdir /mnt/EMC/MAIL1
cp -r /usr /mnt/EMC/MAIL1
rm -rf /mnt/EMC/MAIL1
done;
On MAIL2 server:
#!/bin/bash
while true
do
rm -rf /mnt/EMC/MAIL2
mkdir /mnt/EMC/MAIL2
cp -r /usr /mnt/EMC/MAIL2
rm -rf /mnt/EMC/MAIL2
done;

Today i check logs and see:
o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
(rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
(touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status = -17
(touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
(touch,26137,0):ocfs2_create:631 ERROR: status = -17
(rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b

I am not a expert, but you have a problem in you EMC system (multipath
system) or drivers.

Did you test this before put in production? or test this NAS with
other filesystem, xfs for example?.

As I read I see" that hung_task_timeout_secs" some task wait more that
120 seg may be a problem of you EMC/Fiber/cable problem...

2011/12/4 Marek Kr?likowski <admin at
wset.edu.pl>:
> I do for all night tests with write/read files from ocfs2 on both servers
> something like this:
> On MAIL1 server:
> #!/bin/bash
> while true
> do
> rm -rf /mnt/EMC/MAIL1
> mkdir /mnt/EMC/MAIL1
> cp -r /usr /mnt/EMC/MAIL1
> rm -rf /mnt/EMC/MAIL1
> done;
> On MAIL2 server:
> #!/bin/bash
> while true
> do
> rm -rf /mnt/EMC/MAIL2
> mkdir /mnt/EMC/MAIL2
> cp -r /usr /mnt/EMC/MAIL2
> rm -rf /mnt/EMC/MAIL2
> done;
>
> Today i check logs and see:
> o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
> o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
> (rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
> (touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status = -17
> (touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
> (touch,26137,0):ocfs2_create:631 ERROR: status = -17
> (rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users

hey m8
Like i say i am not expert too but when i use ext3 and write/read working 
with np.


-----Oryginalna wiadomo??----- 
From: Eduardo Diaz - Gmail
Sent: Tuesday, December 06, 2011 3:06 AM
To: Marek Kr?likowski
Cc: ocfs2-users at oss.oracle.com
Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read from both 
servers

I am not a expert, but you have a problem in you EMC system (multipath
system) or drivers.

Did you test this before put in production? or test this NAS with
other filesystem, xfs for example?.

As I read I see" that hung_task_timeout_secs" some task wait more that
120 seg may be a problem of you EMC/Fiber/cable problem...

2011/12/4 Marek Kr?likowski <admin at
wset.edu.pl>:
> I do for all night tests with write/read files from ocfs2 on both servers
> something like this:
> On MAIL1 server:
> #!/bin/bash
> while true
> do
> rm -rf /mnt/EMC/MAIL1
> mkdir /mnt/EMC/MAIL1
> cp -r /usr /mnt/EMC/MAIL1
> rm -rf /mnt/EMC/MAIL1
> done;
> On MAIL2 server:
> #!/bin/bash
> while true
> do
> rm -rf /mnt/EMC/MAIL2
> mkdir /mnt/EMC/MAIL2
> cp -r /usr /mnt/EMC/MAIL2
> rm -rf /mnt/EMC/MAIL2
> done;
>
> Today i check logs and see:
> o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
> o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
> (rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
> (touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status = -17
> (touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
> (touch,26137,0):ocfs2_create:631 ERROR: status = -17
> (rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users

Try to use other filesystem for example xfs and make a full test.

Try create a one node cluster and filesystem and make tests..
(fileindex can be the problem)

make a fsck to the filesystem

Try upgrade ocfs2 to last version and use the max features, only has
two nodes?..

I will do. make a backup, create a new filesystem with all features
that you need and make mkfs. the cluster only with the number of the
nodes that you will use.

Restore de data.

Make extensive test for a week before put in production :-)..

On Tue, Dec 6, 2011 at 2:04 PM, Marek Krolikowski <admin at wset.edu.pl>
wrote:
> hey m8
> Like i say i am not expert too but when i use ext3 and write/read working
> with np.
>
>
> -----Oryginalna wiadomo??----- From: Eduardo Diaz - Gmail
> Sent: Tuesday, December 06, 2011 3:06 AM
> To: Marek Kr?likowski
> Cc: ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read from
both
> servers
>
>
> I am not a expert, but you have a problem in you EMC system (multipath
> system) or drivers.
>
> Did you test this before put in production? or test this NAS with
> other filesystem, xfs for example?.
>
> As I read I see" that hung_task_timeout_secs" some task wait more
that
> 120 seg may be a problem of you EMC/Fiber/cable problem...
>
> 2011/12/4 Marek Kr?likowski <admin at wset.edu.pl>:
>>
>> I do for all night tests with write/read files from ocfs2 on both
servers
>> something like this:
>> On MAIL1 server:
>> #!/bin/bash
>> while true
>> do
>> rm -rf /mnt/EMC/MAIL1
>> mkdir /mnt/EMC/MAIL1
>> cp -r /usr /mnt/EMC/MAIL1
>> rm -rf /mnt/EMC/MAIL1
>> done;
>> On MAIL2 server:
>> #!/bin/bash
>> while true
>> do
>> rm -rf /mnt/EMC/MAIL2
>> mkdir /mnt/EMC/MAIL2
>> cp -r /usr /mnt/EMC/MAIL2
>> rm -rf /mnt/EMC/MAIL2
>> done;
>>
>> Today i check logs and see:
>> o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
>> o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
>> (rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
>> (touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status = -17
>> (touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
>> (touch,26137,0):ocfs2_create:631 ERROR: status = -17
>> (rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>
>>
>> _______________________________________________
>> Ocfs2-users mailing list
>> Ocfs2-users at oss.oracle.com
>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>
>

Hello
I use: sys-fs/ocfs2-tools-1.6.4 and create file system with all features:
mkfs.ocfs2 -N 2 -L MAIL --fs-feature-level=max-features /dev/dm-0
and after this got kernel panic :(


-----Oryginalna wiadomo??----- 
From: Eduardo Diaz - Gmail
Sent: Wednesday, December 07, 2011 11:08 AM
To: Marek Krolikowski
Cc: ocfs2-users at oss.oracle.com
Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read from both
servers

Try to use other filesystem for example xfs and make a full test.

Try create a one node cluster and filesystem and make tests..
(fileindex can be the problem)

make a fsck to the filesystem

Try upgrade ocfs2 to last version and use the max features, only has
two nodes?..

I will do. make a backup, create a new filesystem with all features
that you need and make mkfs. the cluster only with the number of the
nodes that you will use.

Restore de data.

Make extensive test for a week before put in production :-)..

On Tue, Dec 6, 2011 at 2:04 PM, Marek Krolikowski <admin at wset.edu.pl>
wrote:
> hey m8
> Like i say i am not expert too but when i use ext3 and write/read working
> with np.
>
>
> -----Oryginalna wiadomo??----- From: Eduardo Diaz - Gmail
> Sent: Tuesday, December 06, 2011 3:06 AM
> To: Marek Kr?likowski
> Cc: ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read from
> both
> servers
>
>
> I am not a expert, but you have a problem in you EMC system (multipath
> system) or drivers.
>
> Did you test this before put in production? or test this NAS with
> other filesystem, xfs for example?.
>
> As I read I see" that hung_task_timeout_secs" some task wait more
that
> 120 seg may be a problem of you EMC/Fiber/cable problem...
>
> 2011/12/4 Marek Kr?likowski <admin at wset.edu.pl>:
>>
>> I do for all night tests with write/read files from ocfs2 on both
servers
>> something like this:
>> On MAIL1 server:
>> #!/bin/bash
>> while true
>> do
>> rm -rf /mnt/EMC/MAIL1
>> mkdir /mnt/EMC/MAIL1
>> cp -r /usr /mnt/EMC/MAIL1
>> rm -rf /mnt/EMC/MAIL1
>> done;
>> On MAIL2 server:
>> #!/bin/bash
>> while true
>> do
>> rm -rf /mnt/EMC/MAIL2
>> mkdir /mnt/EMC/MAIL2
>> cp -r /usr /mnt/EMC/MAIL2
>> rm -rf /mnt/EMC/MAIL2
>> done;
>>
>> Today i check logs and see:
>> o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
>> o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
>> (rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
>> (touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status = -17
>> (touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
>> (touch,26137,0):ocfs2_create:631 ERROR: status = -17
>> (rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>> [<ffffffff81145900>] ? evict+0x70/0x140
>> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>> INFO: task rm:5192 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>> Call Trace:
>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
>> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>
>>
>> _______________________________________________
>> Ocfs2-users mailing list
>> Ocfs2-users at oss.oracle.com
>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>
>

You can make a fsck.ocfs2 to see if the filesystem are broken?

On Wed, Dec 7, 2011 at 11:30 AM, Marek Krolikowski <admin at wset.edu.pl>
wrote:
> Hello
> I use: sys-fs/ocfs2-tools-1.6.4 and create file system with all features:
> mkfs.ocfs2 -N 2 -L MAIL --fs-feature-level=max-features /dev/dm-0
> and after this got kernel panic :(
>
>
>
> -----Oryginalna wiadomo??----- From: Eduardo Diaz - Gmail
> Sent: Wednesday, December 07, 2011 11:08 AM
> To: Marek Krolikowski
>
> Cc: ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read from
both
> servers
>
> Try to use other filesystem for example xfs and make a full test.
>
> Try create a one node cluster and filesystem and make tests..
> (fileindex can be the problem)
>
> make a fsck to the filesystem
>
> Try upgrade ocfs2 to last version and use the max features, only has
> two nodes?..
>
> I will do. make a backup, create a new filesystem with all features
> that you need and make mkfs. the cluster only with the number of the
> nodes that you will use.
>
> Restore de data.
>
> Make extensive test for a week before put in production :-)..
>
> On Tue, Dec 6, 2011 at 2:04 PM, Marek Krolikowski <admin at
wset.edu.pl> wrote:
>>
>> hey m8
>> Like i say i am not expert too but when i use ext3 and write/read
working
>> with np.
>>
>>
>> -----Oryginalna wiadomo??----- From: Eduardo Diaz - Gmail
>> Sent: Tuesday, December 06, 2011 3:06 AM
>> To: Marek Kr?likowski
>> Cc: ocfs2-users at oss.oracle.com
>> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read from
>> both
>> servers
>>
>>
>> I am not a expert, but you have a problem in you EMC system (multipath
>> system) or drivers.
>>
>> Did you test this before put in production? or test this NAS with
>> other filesystem, xfs for example?.
>>
>> As I read I see" that hung_task_timeout_secs" some task wait
more that
>> 120 seg may be a problem of you EMC/Fiber/cable problem...
>>
>> 2011/12/4 Marek Kr?likowski <admin at wset.edu.pl>:
>>>
>>>
>>> I do for all night tests with write/read files from ocfs2 on both
servers
>>> something like this:
>>> On MAIL1 server:
>>> #!/bin/bash
>>> while true
>>> do
>>> rm -rf /mnt/EMC/MAIL1
>>> mkdir /mnt/EMC/MAIL1
>>> cp -r /usr /mnt/EMC/MAIL1
>>> rm -rf /mnt/EMC/MAIL1
>>> done;
>>> On MAIL2 server:
>>> #!/bin/bash
>>> while true
>>> do
>>> rm -rf /mnt/EMC/MAIL2
>>> mkdir /mnt/EMC/MAIL2
>>> cp -r /usr /mnt/EMC/MAIL2
>>> rm -rf /mnt/EMC/MAIL2
>>> done;
>>>
>>> Today i check logs and see:
>>> o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
>>> o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
>>> (rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
>>> (touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status = -17
>>> (touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
>>> (touch,26137,0):ocfs2_create:631 ERROR: status = -17
>>> (rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2 0x00000000
>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
>>> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0
[ocfs2]
>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338 0x00000000
>>> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
>>> Call Trace:
>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
[ocfs2]
>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>
>>>
>>> _______________________________________________
>>> Ocfs2-users mailing list
>>> Ocfs2-users at oss.oracle.com
>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>
>>
>>
>

I make the same test that you...

and.. error too :-(

75120.532071] INFO: task o2quot/0:3714 blocked for more than 120 seconds.
[75120.532091] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[75120.532108] o2quot/0      D f5939dc8     0  3714      2 0x00000000
[75120.532113]  f6bc4c80 00000046 c1441e20 f5939dc8 f6bc4e3c c1441e20
c1441e20 f9bbdbf2
[75120.532121]  f6bc4e3c c3a08e20 00000000 c33d1f43 0000442f f9a7c716
f6bc4c80 f6bc4c80
[75120.532128]  eedd5b7c f6bc4e3c f6bd6414 f6bd643c 00000004 00000000
011cd9ce f6bd6630
[75120.532136] Call Trace:
[75120.532154]  [<f9bbdbf2>] ? ocfs2_metadata_cache_io_unlock+0x11/0x12
[ocfs2]
[75120.532163]  [<f9a7c716>] ? start_this_handle+0x2fb/0x37e [jbd2]
[75120.532171]  [<c127f4d7>] ? __mutex_lock_common+0xe8/0x13b
[75120.532176]  [<c127f539>] ? __mutex_lock_slowpath+0xf/0x11
[75120.532180]  [<c127f5ca>] ? mutex_lock+0x17/0x24
[75120.532184]  [<c127f5ca>] ? mutex_lock+0x17/0x24
[75120.532198]  [<f9bc3e91>] ? ocfs2_sync_dquot_helper+0x166/0x2c5 [ocfs2]
[75120.532204]  [<c10ec21d>] ? dquot_scan_active+0x63/0xab
[75120.532217]  [<f9bc3d2b>] ? ocfs2_sync_dquot_helper+0x0/0x2c5 [ocfs2]
[75120.532231]  [<f9bc325b>] ? qsync_work_fn+0x23/0x3b [ocfs2]
[75120.532236]  [<c1047917>] ? worker_thread+0x141/0x1bd
[75120.532249]  [<f9bc3238>] ? qsync_work_fn+0x0/0x3b [ocfs2]
[75120.532254]  [<c104a65a>] ? autoremove_wake_function+0x0/0x2d
[75120.532258]  [<c10477d6>] ? worker_thread+0x0/0x1bd
[75120.532262]  [<c104a428>] ? kthread+0x61/0x66
[75120.532266]  [<c104a3c7>] ? kthread+0x0/0x66
[75120.532271]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
[75120.532276] INFO: task jbd2/drbd0-21:3723 blocked for more than 120 seconds.

On Wed, Dec 7, 2011 at 11:42 AM, Eduardo Diaz - Gmail
<ediazrod at gmail.com> wrote:
> You can make a fsck.ocfs2 to see if the filesystem are broken?
>
> On Wed, Dec 7, 2011 at 11:30 AM, Marek Krolikowski <admin at
wset.edu.pl> wrote:
>> Hello
>> I use: sys-fs/ocfs2-tools-1.6.4 and create file system with all
features:
>> mkfs.ocfs2 -N 2 -L MAIL --fs-feature-level=max-features /dev/dm-0
>> and after this got kernel panic :(
>>
>>
>>
>> -----Oryginalna wiadomo??----- From: Eduardo Diaz - Gmail
>> Sent: Wednesday, December 07, 2011 11:08 AM
>> To: Marek Krolikowski
>>
>> Cc: ocfs2-users at oss.oracle.com
>> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read from
both
>> servers
>>
>> Try to use other filesystem for example xfs and make a full test.
>>
>> Try create a one node cluster and filesystem and make tests..
>> (fileindex can be the problem)
>>
>> make a fsck to the filesystem
>>
>> Try upgrade ocfs2 to last version and use the max features, only has
>> two nodes?..
>>
>> I will do. make a backup, create a new filesystem with all features
>> that you need and make mkfs. the cluster only with the number of the
>> nodes that you will use.
>>
>> Restore de data.
>>
>> Make extensive test for a week before put in production :-)..
>>
>> On Tue, Dec 6, 2011 at 2:04 PM, Marek Krolikowski <admin at
wset.edu.pl> wrote:
>>>
>>> hey m8
>>> Like i say i am not expert too but when i use ext3 and write/read
working
>>> with np.
>>>
>>>
>>> -----Oryginalna wiadomo??----- From: Eduardo Diaz - Gmail
>>> Sent: Tuesday, December 06, 2011 3:06 AM
>>> To: Marek Kr?likowski
>>> Cc: ocfs2-users at oss.oracle.com
>>> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read
from
>>> both
>>> servers
>>>
>>>
>>> I am not a expert, but you have a problem in you EMC system
(multipath
>>> system) or drivers.
>>>
>>> Did you test this before put in production? or test this NAS with
>>> other filesystem, xfs for example?.
>>>
>>> As I read I see" that hung_task_timeout_secs" some task
wait more that
>>> 120 seg may be a problem of you EMC/Fiber/cable problem...
>>>
>>> 2011/12/4 Marek Kr?likowski <admin at wset.edu.pl>:
>>>>
>>>>
>>>> I do for all night tests with write/read files from ocfs2 on
both servers
>>>> something like this:
>>>> On MAIL1 server:
>>>> #!/bin/bash
>>>> while true
>>>> do
>>>> rm -rf /mnt/EMC/MAIL1
>>>> mkdir /mnt/EMC/MAIL1
>>>> cp -r /usr /mnt/EMC/MAIL1
>>>> rm -rf /mnt/EMC/MAIL1
>>>> done;
>>>> On MAIL2 server:
>>>> #!/bin/bash
>>>> while true
>>>> do
>>>> rm -rf /mnt/EMC/MAIL2
>>>> mkdir /mnt/EMC/MAIL2
>>>> cp -r /usr /mnt/EMC/MAIL2
>>>> rm -rf /mnt/EMC/MAIL2
>>>> done;
>>>>
>>>> Today i check logs and see:
>>>> o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
>>>> o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
>>>> (rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
>>>> (touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status =
-17
>>>> (touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
>>>> (touch,26137,0):ocfs2_create:631 ERROR: status = -17
>>>> (rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
>>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>> Call Trace:
>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>>
>>>>
>>>> _______________________________________________
>>>> Ocfs2-users mailing list
>>>> Ocfs2-users at oss.oracle.com
>>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>>
>>>
>>>
>>

I make more test I think my problem are for network interface not,
from ocfs2.. I will change the network interfaces for other..

Dec  7 11:56:41 servidoradantra2 kernel: [77153.937731] r8169
0000:05:01.0: eth3: link down
Dec  7 11:56:47 servidoradantra2 kernel: [77160.049405] r8169
0000:05:01.0: eth3: link up

Dec  7 11:54:48 servidoradantra2 kernel: [77040.644084] o2quot/0
D 00000000     0  2721      2 0x00000000
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644089]  f67dd940
00000046 00000400 00000000 c0426bc0 c1441e20 c1441e20 f68df800
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644096]  f67ddafc
c3a08e20 00000000 c10d5782 c0426e80 f6dfa600 ed599b40 f68df800
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644103]  00000000
f67ddafc f54a22d8 00000005 00000000 f54a2308 0c000000 f54a227c
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644111] Call Trace:
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644118]  [<c10d5782>]
? __find_get_block+0x163/0x16d
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644125]  [<c127f0e9>]
? schedule_timeout+0x20/0xb0
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644141]  [<fcfe7bce>]
? ocfs2_metadata_cache_unlock+0x11/0x12 [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644155]  [<fcfe7d08>]
? ocfs2_buffer_cached+0xe2/0x137 [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644171]  [<fb3de16e>]
? o2cb_dlm_lock+0x40/0x57 [ocfs2_stack_o2cb]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644176]  [<c127eff2>]
? wait_for_common+0xa4/0x100
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644182]  [<c10335ff>]
? default_wake_function+0x0/0x8
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644200]  [<fcfaf3e0>]
? __ocfs2_cluster_lock+0x7eb/0x808 [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644211]  [<fcfb30d6>]
? ocfs2_inode_lock_full_nested+0x156/0xa85 [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644220]  [<fcfed2d7>]
? ocfs2_lock_global_qf+0x1d/0x6d [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644229]  [<fcfed2d7>]
? ocfs2_lock_global_qf+0x1d/0x6d [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644238]  [<fcfeddca>]
? ocfs2_sync_dquot_helper+0x9f/0x2c5 [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644241]  [<c10ec21d>]
? dquot_scan_active+0x63/0xab
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644250]  [<fcfedd2b>]
? ocfs2_sync_dquot_helper+0x0/0x2c5 [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644259]  [<fcfed25b>]
? qsync_work_fn+0x23/0x3b [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644262]  [<c1047917>]
? worker_thread+0x141/0x1bd
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644271]  [<fcfed238>]
? qsync_work_fn+0x0/0x3b [ocfs2]
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644274]  [<c104a65a>]
? autoremove_wake_function+0x0/0x2d
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644277]  [<c10477d6>]
? worker_thread+0x0/0x1bd
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644280]  [<c104a428>]
? kthread+0x61/0x66
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644282]  [<c104a3c7>]
? kthread+0x0/0x66
Dec  7 11:54:48 servidoradantra2 kernel: [77040.644286]  [<c1008d87>]
? kernel_thread_helper+0x7/0x10
Dec  7 11:56:41 servidoradantra2 kernel: [77153.937731] r8169
0000:05:01.0: eth3: link down
Dec  7 11:56:47 servidoradantra2 kernel: [77160.049405] r8169
0000:05:01.0: eth3: link up
Dec  7 11:56:48 servidoradantra2 kernel: [77160.644100] o2quot/0
D 00000000     0  2721      2 0x00000000
Dec  7 11:56:48 servidoradantra2 kernel: [77160.644106]  f67dd940
00000046 00000400 00000000 c0426bc0 c1441e20 c1441e20 f68df800
Dec  7 11:56:48 servidoradantra2 kernel: [77160.644114]  f67ddafc
c3a08e20 00000000 c10d5782 c0426e80 f6dfa600 ed599b40 f68df800
Dec  7 11:56:48 servidoradantra2 kernel: [77160.644121]  00000000
f67ddafc f54a22d8 00000005 00000000 f54a2308 0c000000 f54a227c
Dec  7 11:56:48 servidoradantra2 kernel: [77160.644128] Call Trace:


On Wed, Dec 7, 2011 at 12:02 PM, Eduardo Diaz - Gmail
<ediazrod at gmail.com> wrote:
> I make the same test that you...
>
> and.. error too :-(
>
> 75120.532071] INFO: task o2quot/0:3714 blocked for more than 120 seconds.
> [75120.532091] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [75120.532108] o2quot/0 ? ? ?D f5939dc8 ? ? 0 ?3714 ? ? ?2 0x00000000
> [75120.532113] ?f6bc4c80 00000046 c1441e20 f5939dc8 f6bc4e3c c1441e20
> c1441e20 f9bbdbf2
> [75120.532121] ?f6bc4e3c c3a08e20 00000000 c33d1f43 0000442f f9a7c716
> f6bc4c80 f6bc4c80
> [75120.532128] ?eedd5b7c f6bc4e3c f6bd6414 f6bd643c 00000004 00000000
> 011cd9ce f6bd6630
> [75120.532136] Call Trace:
> [75120.532154] ?[<f9bbdbf2>] ?
ocfs2_metadata_cache_io_unlock+0x11/0x12 [ocfs2]
> [75120.532163] ?[<f9a7c716>] ? start_this_handle+0x2fb/0x37e [jbd2]
> [75120.532171] ?[<c127f4d7>] ? __mutex_lock_common+0xe8/0x13b
> [75120.532176] ?[<c127f539>] ? __mutex_lock_slowpath+0xf/0x11
> [75120.532180] ?[<c127f5ca>] ? mutex_lock+0x17/0x24
> [75120.532184] ?[<c127f5ca>] ? mutex_lock+0x17/0x24
> [75120.532198] ?[<f9bc3e91>] ? ocfs2_sync_dquot_helper+0x166/0x2c5
[ocfs2]
> [75120.532204] ?[<c10ec21d>] ? dquot_scan_active+0x63/0xab
> [75120.532217] ?[<f9bc3d2b>] ? ocfs2_sync_dquot_helper+0x0/0x2c5
[ocfs2]
> [75120.532231] ?[<f9bc325b>] ? qsync_work_fn+0x23/0x3b [ocfs2]
> [75120.532236] ?[<c1047917>] ? worker_thread+0x141/0x1bd
> [75120.532249] ?[<f9bc3238>] ? qsync_work_fn+0x0/0x3b [ocfs2]
> [75120.532254] ?[<c104a65a>] ? autoremove_wake_function+0x0/0x2d
> [75120.532258] ?[<c10477d6>] ? worker_thread+0x0/0x1bd
> [75120.532262] ?[<c104a428>] ? kthread+0x61/0x66
> [75120.532266] ?[<c104a3c7>] ? kthread+0x0/0x66
> [75120.532271] ?[<c1008d87>] ? kernel_thread_helper+0x7/0x10
> [75120.532276] INFO: task jbd2/drbd0-21:3723 blocked for more than 120
seconds.
>
> On Wed, Dec 7, 2011 at 11:42 AM, Eduardo Diaz - Gmail
> <ediazrod at gmail.com> wrote:
>> You can make a fsck.ocfs2 to see if the filesystem are broken?
>>
>> On Wed, Dec 7, 2011 at 11:30 AM, Marek Krolikowski <admin at
wset.edu.pl> wrote:
>>> Hello
>>> I use: sys-fs/ocfs2-tools-1.6.4 and create file system with all
features:
>>> mkfs.ocfs2 -N 2 -L MAIL --fs-feature-level=max-features /dev/dm-0
>>> and after this got kernel panic :(
>>>
>>>
>>>
>>> -----Oryginalna wiadomo??----- From: Eduardo Diaz - Gmail
>>> Sent: Wednesday, December 07, 2011 11:08 AM
>>> To: Marek Krolikowski
>>>
>>> Cc: ocfs2-users at oss.oracle.com
>>> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read
from both
>>> servers
>>>
>>> Try to use other filesystem for example xfs and make a full test.
>>>
>>> Try create a one node cluster and filesystem and make tests..
>>> (fileindex can be the problem)
>>>
>>> make a fsck to the filesystem
>>>
>>> Try upgrade ocfs2 to last version and use the max features, only
has
>>> two nodes?..
>>>
>>> I will do. make a backup, create a new filesystem with all features
>>> that you need and make mkfs. the cluster only with the number of
the
>>> nodes that you will use.
>>>
>>> Restore de data.
>>>
>>> Make extensive test for a week before put in production :-)..
>>>
>>> On Tue, Dec 6, 2011 at 2:04 PM, Marek Krolikowski <admin at
wset.edu.pl> wrote:
>>>>
>>>> hey m8
>>>> Like i say i am not expert too but when i use ext3 and
write/read working
>>>> with np.
>>>>
>>>>
>>>> -----Oryginalna wiadomo??----- From: Eduardo Diaz - Gmail
>>>> Sent: Tuesday, December 06, 2011 3:06 AM
>>>> To: Marek Kr?likowski
>>>> Cc: ocfs2-users at oss.oracle.com
>>>> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many
write/read from
>>>> both
>>>> servers
>>>>
>>>>
>>>> I am not a expert, but you have a problem in you EMC system
(multipath
>>>> system) or drivers.
>>>>
>>>> Did you test this before put in production? or test this NAS
with
>>>> other filesystem, xfs for example?.
>>>>
>>>> As I read I see" that hung_task_timeout_secs" some
task wait more that
>>>> 120 seg may be a problem of you EMC/Fiber/cable problem...
>>>>
>>>> 2011/12/4 Marek Kr?likowski <admin at wset.edu.pl>:
>>>>>
>>>>>
>>>>> I do for all night tests with write/read files from ocfs2
on both servers
>>>>> something like this:
>>>>> On MAIL1 server:
>>>>> #!/bin/bash
>>>>> while true
>>>>> do
>>>>> rm -rf /mnt/EMC/MAIL1
>>>>> mkdir /mnt/EMC/MAIL1
>>>>> cp -r /usr /mnt/EMC/MAIL1
>>>>> rm -rf /mnt/EMC/MAIL1
>>>>> done;
>>>>> On MAIL2 server:
>>>>> #!/bin/bash
>>>>> while true
>>>>> do
>>>>> rm -rf /mnt/EMC/MAIL2
>>>>> mkdir /mnt/EMC/MAIL2
>>>>> cp -r /usr /mnt/EMC/MAIL2
>>>>> rm -rf /mnt/EMC/MAIL2
>>>>> done;
>>>>>
>>>>> Today i check logs and see:
>>>>> o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
>>>>> o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0
1
>>>>> (rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
>>>>> (touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR:
status = -17
>>>>> (touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
>>>>> (touch,26137,0):ocfs2_create:631 ERROR: status = -17
>>>>> (rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
>>>>> INFO: task kworker/u:2:20246 blocked for more than 120
seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff81481b71>] ?
__mutex_lock_slowpath+0xd1/0x140
>>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90
[ocfs2]
>>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>>> [<ffffffff811653d8>] ?
fsnotify_find_inode_mark+0x28/0x40
>>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>>> INFO: task kworker/u:2:20246 blocked for more than 120
seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff81481b71>] ?
__mutex_lock_slowpath+0xd1/0x140
>>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90
[ocfs2]
>>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>>> [<ffffffff811653d8>] ?
fsnotify_find_inode_mark+0x28/0x40
>>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>>> INFO: task kworker/u:2:20246 blocked for more than 120
seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff81481b71>] ?
__mutex_lock_slowpath+0xd1/0x140
>>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90
[ocfs2]
>>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>>> [<ffffffff811653d8>] ?
fsnotify_find_inode_mark+0x28/0x40
>>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>>> INFO: task kworker/u:2:20246 blocked for more than 120
seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff81481b71>] ?
__mutex_lock_slowpath+0xd1/0x140
>>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90
[ocfs2]
>>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>>> [<ffffffff811653d8>] ?
fsnotify_find_inode_mark+0x28/0x40
>>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>>> INFO: task kworker/u:2:20246 blocked for more than 120
seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> kworker/u:2 ? ? D ffff88107f4525c0 ? ? 0 20246 ? ? ?2
0x00000000
>>>>> ffff880b730b57d0 0000000000000046 ffff8810201297d0
00000000000125c0
>>>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff81481b71>] ?
__mutex_lock_slowpath+0xd1/0x140
>>>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
>>>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690
[ocfs2]
>>>>> [<ffffffffa0935cfb>] ?
ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
>>>>> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
[ocfs2]
>>>>> [<ffffffffa0919a10>] ?
ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
>>>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50
[ocfs2]
>>>>> [<ffffffff81145900>] ? evict+0x70/0x140
>>>>> [<ffffffffa0919322>] ?
__ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
>>>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90
[ocfs2]
>>>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
>>>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff8106f120>] ?
manage_workers.clone.21+0x120/0x120
>>>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
>>>>> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
>>>>> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
>>>>> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
>>>>> INFO: task rm:5192 blocked for more than 120 seconds.
>>>>> "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> rm ? ? ? ? ? ? ?D ffff88107f2725c0 ? ? 0 ?5192 ?16338
0x00000000
>>>>> ffff881014ccb040 0000000000000082 ffff8810206b8040
00000000000125c0
>>>>> ffff8804d7697fd8 00000000000125c0 00000000000125c0
00000000000125c0
>>>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
00000000000125c0
>>>>> Call Trace:
>>>>> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
>>>>> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190
[ocfs2_dlm]
>>>>> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
>>>>> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
>>>>> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
>>>>> [<ffffffffa0928a38>] ?
__ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
>>>>> [<ffffffffa0928fcc>] ?
ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
>>>>> [<ffffffffa0943c1b>] ?
ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
>>>>> [<ffffffffa09454ba>] ?
ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
>>>>> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90
[ocfs2]
>>>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
>>>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
>>>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
>>>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
>>>>> [<ffffffff811653d8>] ?
fsnotify_find_inode_mark+0x28/0x40
>>>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
>>>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
>>>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ocfs2-users mailing list
>>>>> Ocfs2-users at oss.oracle.com
>>>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>>>
>>>>
>>>>
>>>

Anyone can help wme with this?



-----Oryginalna wiadomo??----- 
From: Marek Kr?likowski
Sent: Sunday, December 04, 2011 11:15 AM
To: ocfs2-users at oss.oracle.com
Subject: ocfs2 - Kernel panic on many write/read from both servers

I do for all night tests with write/read files from ocfs2 on both servers
something like this:
On MAIL1 server:
#!/bin/bash
while true
do
rm -rf /mnt/EMC/MAIL1
mkdir /mnt/EMC/MAIL1
cp -r /usr /mnt/EMC/MAIL1
rm -rf /mnt/EMC/MAIL1
done;
On MAIL2 server:
#!/bin/bash
while true
do
rm -rf /mnt/EMC/MAIL2
mkdir /mnt/EMC/MAIL2
cp -r /usr /mnt/EMC/MAIL2
rm -rf /mnt/EMC/MAIL2
done;

Today i check logs and see:
o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
(rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
(touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status = -17
(touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
(touch,26137,0):ocfs2_create:631 ERROR: status = -17
(rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
Call Trace:
[<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
[<ffffffff814818d3>] ? mutex_lock+0x23/0x40
[<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
[<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370 [ocfs2]
[<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
[<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
[<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
[<ffffffff81145900>] ? evict+0x70/0x140
[<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
[<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
[<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
[<ffffffff8106f279>] ? worker_thread+0x159/0x330
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
[<ffffffff81073fa6>] ? kthread+0x96/0xa0
[<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff8148bb20>] ? gs_change+0x13/0x13
INFO: task rm:5192 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm              D ffff88107f2725c0     0  5192  16338 0x00000000
ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
Call Trace:
[<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
[<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
[<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
[<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
[<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
[<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490 [ocfs2]
[<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
[<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
[<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
[<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
[<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
[<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
[<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
[<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
[<ffffffff81166324>] ? dnotify_flush+0x54/0x110
[<ffffffff8112b07f>] ? filp_close+0x5f/0x90
[<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b

I think you are running into a known issue. Are there lot of orphan 
files in orphan directory? I am not sure if the problem is still there, 
if not please run the same test and once you see the same symptoms, 
please run the following and provide me the output

echo "ls //orphan_dir:0000"|debugfs.ocfs2 <device>|wc
echo "ls //orphan_dir:0001"|debugfs.ocfs2 <device>|wc

On 12/15/2011 12:06 PM, Marek Kr?likowski wrote:
> Anyone can help wme with this?
>
>
>
> -----Oryginalna wiadomo??-----
> From: Marek Kr?likowski
> Sent: Sunday, December 04, 2011 11:15 AM
> To: ocfs2-users at oss.oracle.com
> Subject: ocfs2 - Kernel panic on many write/read from both servers
>
> I do for all night tests with write/read files from ocfs2 on both servers
> something like this:
> On MAIL1 server:
> #!/bin/bash
> while true
> do
> rm -rf /mnt/EMC/MAIL1
> mkdir /mnt/EMC/MAIL1
> cp -r /usr /mnt/EMC/MAIL1
> rm -rf /mnt/EMC/MAIL1
> done;
> On MAIL2 server:
> #!/bin/bash
> while true
> do
> rm -rf /mnt/EMC/MAIL2
> mkdir /mnt/EMC/MAIL2
> cp -r /usr /mnt/EMC/MAIL2
> rm -rf /mnt/EMC/MAIL2
> done;
>
> Today i check logs and see:
> o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
> o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
> (rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
> (touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status = -17
> (touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
> (touch,26137,0):ocfs2_create:631 ERROR: status = -17
> (rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> kworker/u:2     D ffff88107f4525c0     0 20246      2 0x00000000
> ffff880b730b57d0 0000000000000046 ffff8810201297d0 00000000000125c0
> ffff880f5a399fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff880f5a398000 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> Call Trace:
> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
[ocfs2]
> [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0 [ocfs2]
> [<ffffffffa0919a10>] ? ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> [<ffffffff81145900>] ? evict+0x70/0x140
> [<ffffffffa0919322>] ? __ocfs2_drop_dl_inodes.clone.2+0x32/0x60
[ocfs2]
> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> [<ffffffff81073fa6>] ? kthread+0x96/0xa0
> [<ffffffff8148bb24>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81073f10>] ? kthread_worker_fn+0x1a0/0x1a0
> [<ffffffff8148bb20>] ? gs_change+0x13/0x13
> INFO: task rm:5192 blocked for more than 120 seconds.
> "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> rm              D ffff88107f2725c0     0  5192  16338 0x00000000
> ffff881014ccb040 0000000000000082 ffff8810206b8040 00000000000125c0
> ffff8804d7697fd8 00000000000125c0 00000000000125c0 00000000000125c0
> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8 00000000000125c0
> Call Trace:
> [<ffffffff8148148d>] ? schedule_timeout+0x1ed/0x2e0
> [<ffffffffa0886162>] ? dlmconvert_master+0xe2/0x190 [ocfs2_dlm]
> [<ffffffffa08878bf>] ? dlmlock+0x7f/0xb70 [ocfs2_dlm]
> [<ffffffff81480e0a>] ? wait_for_common+0x13a/0x190
> [<ffffffff8104bc50>] ? try_to_wake_up+0x280/0x280
> [<ffffffffa0928a38>] ? __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0
[ocfs2]
> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
[ocfs2]
> [<ffffffffa0943c1b>] ? ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0
[ocfs2]
> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280 [ocfs2]
> [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users

> I think you are running into a known issue. Are there lot of orphan 
> files in orphan directory? I am not sure if the problem is still there, 
> if not please run the same test and once you see the same symptoms, 
> please run the following and provide me the output
> 
> echo "ls //orphan_dir:0000"|debugfs.ocfs2 <device>|wc
> echo "ls //orphan_dir:0001"|debugfs.ocfs2 <device>|wc
Hello
Thank You for answer - strange i don`t get email with Your answer.
This is what You want:
TEST-MAIL1# echo "ls //orphan_dir:0000"|debugfs.ocfs2 /dev/dm-0|wc
debugfs.ocfs2 1.6.4
5239722 26198604 246266859
TEST-MAIL2 ~ # echo "ls //orphan_dir:0000"|debugfs.ocfs2 /dev/dm-0|wc
debugfs.ocfs2 1.6.4
5239722 26198604 246266859
This is my testing cluster so if u need do more tests please tell me i do for
You.
Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20111220/5a508ca3/attachment.html