What could be happening is as follows:
Node RAC1 is having disk io issues and is probably no longer
seeing node RAC2 as alive which is leading RAC1 to drop
the connection with RAC2.
RAC2 inturn is seeing RAC1 disk heartbeating but also sees the net
connection drop which leads it to fence itself.
ERROR: fencing this node because it is connected to a half-quorum of 1
out of 2 nodes which doesn't include the lowest active node 1
Bottomline, address the disk io issues which you are seeing on node
RAC1.
doron tamir wrote:
>Hi
>We get a kernel panic problem by using ocfs2 on suse 9.2.
>
>I attached to log files of it :
>
>Rac2:
>=====>Sep 19 16:23:39 mdent-rac2 -- MARK --
>Sep 19 16:43:39 mdent-rac2 -- MARK --
>Sep 19 16:59:01 mdent-rac2 /USR/SBIN/CRON[18145]: (root) CMD ( rm -f
>/var/spool/cron/lastrun/cron.hourly)
>Sep 19 17:12:48 mdent-rac2 kernel: (8541,2):o2net_set_nn_state:414 no
>longer connected to node mdent-rac1 at 192.168.52.134:7777
>Sep 19 17:13:06 mdent-rac2 kernel: (8541,2):o2net_check_quorum:1468
>ERROR: fencing this node because it is connected to a half-quorum of 1
>out of 2 nodes which doesn't include the lowest active node 1
>Sep 19 17:13:06 mdent-rac2 kernel: (8541,2):o2hb_stop_all_regions:1589
>ERROR: stopping heartbeat on all active regions.
>Sep 19 17:13:06 mdent-rac2 kernel: Kernel panic: ocfs2 is very sorry to
>be fencing this system by panicing
>Sep 19 17:39:29 mdent-rac2 syslogd 1.4.1: restart.
>Sep
>
>
>
>Rac1:
>====>Sep 19 16:59:01 mdent-rac1 /USR/SBIN/CRON[19733]: (root) CMD ( rm -f
>/var/spool/cron/lastrun/cron.hourly)
>Sep 19 17:12:48 mdent-rac1 kernel: (0,2):o2net_state_change:512
>connection to node mdent-rac2 num 2 at 192.168.52.135:7777 has been idle
>for 10 seconds, shutting it down.
>Sep 19 17:12:48 mdent-rac1 kernel: (8890,2):o2net_set_nn_state:414 no
>longer connected to node mdent-rac2 at 192.168.52.135:7777
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:0:2 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 7 [RAIDarray.mpp]MdentFastT600T:0:0
>Path Failed
>Sep 19 17:13:13 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:0:0:2 Cmnd failed-retry on a new path.
>vcmnd SN 5713232 pdev H1:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:0:1 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:0:0:1 Cmnd failed-retry on a new path.
>vcmnd SN 5713238 pdev H1:C0:T0:L1 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:0:2 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:0:0:2 Cmnd failed-retry on a new path.
>vcmnd SN 5713240 pdev H1:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:0:3 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:0:0:3 Cmnd failed-retry on a new path.
>vcmnd SN 5713246 pdev H1:C0:T0:L3 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:0:2 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:0:0:2 Cmnd failed-retry on a new path.
>vcmnd SN 5713297 pdev H1:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:0:3 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:0:0:3 Cmnd failed-retry on a new path.
>vcmnd SN 5713299 pdev H1:C0:T0:L3 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:3 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 7 [RAIDarray.mpp]MdentFastT600T:0:1
>Path Failed
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:3 No new path: fall to failover
>controller case. vcmnd SN 5713234 pdev H2:C0:T0:L3 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 106 [RAIDarray.mpp]MdentFastT600T:0:3
>AVT Failover
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:3 Failed controller to 1. retry. vcmnd
>SN 5713234 pdev H2:C0:T0:L3 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:2 No new path: fall to failover
>controller case. vcmnd SN 5713236 pdev H2:C0:T0:L2 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Failed controller to 1. retry. vcmnd
>SN 5713236 pdev H2:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:2 No new path: fall to failover
>controller case. vcmnd SN 5713242 pdev H2:C0:T0:L2 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Failed controller to 1. retry. vcmnd
>SN 5713242 pdev H2:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:4 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:4 No new path: fall to failover
>controller case. vcmnd SN 5713244 pdev H2:C0:T0:L4 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:4 Failed controller to 1. retry. vcmnd
>SN 5713244 pdev H2:C0:T0:L4 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:3 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:3 No new path: fall to failover
>controller case. vcmnd SN 5713248 pdev H2:C0:T0:L3 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:3 Failed controller to 1. retry. vcmnd
>SN 5713248 pdev H2:C0:T0:L3 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:2 No new path: fall to failover
>controller case. vcmnd SN 5713232 pdev H2:C0:T0:L2 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Failed controller to 1. retry. vcmnd
>SN 5713232 pdev H2:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:1 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:1 No new path: fall to failover
>controller case. vcmnd SN 5713238 pdev H2:C0:T0:L1 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:1 Failed controller to 1. retry. vcmnd
>SN 5713238 pdev H2:C0:T0:L1 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:2 No new path: fall to failover
>controller case. vcmnd SN 5713240 pdev H2:C0:T0:L2 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Failed controller to 1. retry. vcmnd
>SN 5713240 pdev H2:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:3 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:3 No new path: fall to failover
>controller case. vcmnd SN 5713246 pdev H2:C0:T0:L3 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:3 Failed controller to 1. retry. vcmnd
>SN 5713246 pdev H2:C0:T0:L3 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:2 No new path: fall to failover
>controller case. vcmnd SN 5713297 pdev H2:C0:T0:L2 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Failed controller to 1. retry. vcmnd
>SN 5713297 pdev H2:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:3 Selection Retry count exhausted
>Sep 19 17:13:13 mdent-rac1 kernel: 496
>[RAIDarray.mpp]MdentFastT600T:0:1:3 No new path: fall to failover
>controller case. vcmnd SN 5713299 pdev H2:C0:T0:L3 0x00/0x00/0x00
>0x00010000 mpp_status:6
>Sep 19 17:13:13 mdent-rac1 kernel: 497
>[RAIDarray.mpp]MdentFastT600T:0:1:3 Failed controller to 1. retry. vcmnd
>SN 5713299 pdev H2:C0:T0:L3 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:14 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:2 Cmnd failed-retry the same path.
>vcmnd SN 5713242 pdev H2:C0:T1:L2 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:14 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713248 pdev H2:C0:T1:L3 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:14 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:2 Cmnd failed-retry the same path.
>vcmnd SN 5713240 pdev H2:C0:T1:L2 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:14 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713299 pdev H2:C0:T1:L3 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:24 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:1:0:4 Selection Retry count exhausted
>Sep 19 17:13:24 mdent-rac1 kernel: 7 [RAIDarray.mpp]MdentFastT600T:1:0
>Path Failed
>Sep 19 17:13:24 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:1:0:4 Cmnd failed-retry on a new path.
>vcmnd SN 5713244 pdev H1:C0:T1:L4 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:24 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:1:0:1 Selection Retry count exhausted
>Sep 19 17:13:24 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:1:0:1 Cmnd failed-retry on a new path.
>vcmnd SN 5713238 pdev H1:C0:T1:L1 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:24 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:1:0:3 Selection Retry count exhausted
>Sep 19 17:13:24 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:1:0:3 Cmnd failed-retry on a new path.
>vcmnd SN 5713246 pdev H1:C0:T1:L3 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:24 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:0:3 Cmnd failed-retry the same path.
>vcmnd SN 5713234 pdev H1:C0:T1:L3 0x00/0x00/0x00 0x00020000 mpp_status:2
>Sep 19 17:13:24 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:0:2 Cmnd failed-retry the same path.
>vcmnd SN 5713236 pdev H1:C0:T1:L2 0x00/0x00/0x00 0x00020000 mpp_status:2
>Sep 19 17:13:24 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:0:2 Cmnd failed-retry the same path.
>vcmnd SN 5713232 pdev H1:C0:T1:L2 0x00/0x00/0x00 0x00020000 mpp_status:2
>Sep 19 17:13:24 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:0:2 Cmnd failed-retry the same path.
>vcmnd SN 5713297 pdev H1:C0:T1:L2 0x00/0x00/0x00 0x00020000 mpp_status:2
>Sep 19 17:13:26 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:1:0:3 Selection Retry count exhausted
>Sep 19 17:13:26 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:1:0:3 Cmnd failed-retry on a new path.
>vcmnd SN 5713234 pdev H1:C0:T1:L3 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:26 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:1:0:2 Selection Retry count exhausted
>Sep 19 17:13:26 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:1:0:2 Cmnd failed-retry on a new path.
>vcmnd SN 5713236 pdev H1:C0:T1:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:26 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:1:0:2 Selection Retry count exhausted
>Sep 19 17:13:26 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:1:0:2 Cmnd failed-retry on a new path.
>vcmnd SN 5713232 pdev H1:C0:T1:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:26 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:1:0:2 Selection Retry count exhausted
>Sep 19 17:13:26 mdent-rac1 kernel: 495
>[RAIDarray.mpp]MdentFastT600T:1:0:2 Cmnd failed-retry on a new path.
>vcmnd SN 5713297 pdev H1:C0:T1:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:13:39 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:4 Cmnd failed-retry the same path.
>vcmnd SN 5713244 pdev H2:C0:T1:L4 0x00/0x00/0x00 0x00020000 mpp_status:2
>Sep 19 17:13:39 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:1 Cmnd failed-retry the same path.
>vcmnd SN 5713238 pdev H2:C0:T1:L1 0x00/0x00/0x00 0x00020000 mpp_status:2
>Sep 19 17:13:39 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713246 pdev H2:C0:T1:L3 0x00/0x00/0x00 0x00020000 mpp_status:2
>Sep 19 17:13:39 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:2 Cmnd failed-retry the same path.
>vcmnd SN 5713242 pdev H2:C0:T1:L2 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:39 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713248 pdev H2:C0:T1:L3 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:39 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:2 Cmnd failed-retry the same path.
>vcmnd SN 5713240 pdev H2:C0:T1:L2 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:39 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713299 pdev H2:C0:T1:L3 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:40 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713234 pdev H2:C0:T1:L3 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:40 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:2 Cmnd failed-retry the same path.
>vcmnd SN 5713236 pdev H2:C0:T1:L2 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:40 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:2 Cmnd failed-retry the same path.
>vcmnd SN 5713232 pdev H2:C0:T1:L2 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:40 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:2 Cmnd failed-retry the same path.
>vcmnd SN 5713297 pdev H2:C0:T1:L2 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:42 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:4 Cmnd failed-retry the same path.
>vcmnd SN 5713244 pdev H2:C0:T1:L4 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:42 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:1 Cmnd failed-retry the same path.
>vcmnd SN 5713238 pdev H2:C0:T1:L1 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:42 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713246 pdev H2:C0:T1:L3 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:42 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:2 Cmnd failed-retry the same path.
>vcmnd SN 5713242 pdev H2:C0:T1:L2 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:42 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713248 pdev H2:C0:T1:L3 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:42 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:2 Cmnd failed-retry the same path.
>vcmnd SN 5713240 pdev H2:C0:T1:L2 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:42 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713299 pdev H2:C0:T1:L3 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:43 mdent-rac1 kernel: 493
>[RAIDarray.mpp]MdentFastT600T:1:1:3 Cmnd failed-retry the same path.
>vcmnd SN 5713234 pdev H2:C0:T1:L3 0x02/0x04/0x01 0x08000002 mpp_status:1
>Sep 19 17:13:43 mdent-ra
>.
>.
>.
>Sep 19 17:16:46 mdent-rac1 kernel: SCSI error : <3 0 0 2> return code
>0x10000
>Sep 19 17:16:46 mdent-rac1 kernel: end_request: I/O error, dev sdc,
>sector 1983
>Sep 19 17:16:46 mdent-rac1 kernel: (5,1):o2hb_bio_end_io:166 ERROR: IO
>Error -5
>Sep 19 17:16:46 mdent-rac1 kernel: SCSI error : <3 0 0 4> return code
>0x10000
>Sep 19 17:16:46 mdent-rac1 kernel: end_request: I/O error, dev sde,
>sector 941
>Sep 19 17:16:46 mdent-rac1 kernel: (5,1):o2hb_bio_end_io:166 ERROR: IO
>Error -5
>Sep 19 17:16:46 mdent-rac1 kernel: SCSI error : <3 0 0 3> return code
>0x10000
>Sep 19 17:16:46 mdent-rac1 kernel: end_request: I/O error, dev sdd,
>sector 294404
>Sep 19 17:16:46 mdent-rac1 kernel: SCSI error : <3 0 0 1> return code
>0x10000
>Sep 19 17:16:46 mdent-rac1 kernel: end_request: I/O error, dev sdb,
>sector 2208
>Sep 19 17:16:46 mdent-rac1 kernel: (5,1):o2hb_bio_end_io:166 ERROR: IO
>Error -5
>Sep 19 17:16:47 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:3 Selection Retry count exhausted
>Sep 19 17:16:47 mdent-rac1 kernel: 492
>[RAIDarray.mpp]MdentFastT600T:0:1:3 IO FAILURE. vcmnd SN 5713234 pdev
>H2:C0:T0:L3 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:16:47 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Selection Retry count exhausted
>Sep 19 17:16:47 mdent-rac1 kernel: 492
>[RAIDarray.mpp]MdentFastT600T:0:1:2 IO FAILURE. vcmnd SN 5713240 pdev
>H2:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:16:47 mdent-rac1 kernel: 94
>[RAIDarray.mpp]MdentFastT600T:0:1:2 Selection Retry count exhausted
>Sep 19 17:16:47 mdent-rac1 kernel: 492
>[RAIDarray.mpp]MdentFastT600T:0:1:2 IO FAILURE. vcmnd SN 5713232 pdev
>H2:C0:T0:L2 0x00/0x00/0x00 0x00010000 mpp_status:6
>Sep 19 17:16:47 mdent-rac1 kernel: SCSI error : <3 0 0 3> return code
>0x10000
>Sep 19 17:16:47 mdent-rac1 kernel: end_request: I/O error, dev sdd,
>sector 1017166
>Sep 19 17:16:47 mdent-rac1 kernel: SCSI error : <3 0 0 2> return code
>0x10000
>Sep 19 17:16:47 mdent-rac1 kernel: end_request: I/O error, dev sdc,
>sector 6194335
>Sep 19 17:16:47 mdent-rac1 kernel: SCSI
>
>
>
>Thanks for any help
>doron
>
>
>