Ocfs2 users - Apr 2011 - Problem mounting volume on second node after uncontrolled reboot.

Hi, 

I have the following setup: 

2 linux nodes connected to shared SAN, both nodes have shared OCFS2 volume as
/dev/sdb1
Both servers are running CentOS 5.5

Node 1 has volume mounted RW.
Node 2 has volume mounted RO.

After an unscheduled reboot of node 2 it is no longer able to mount filesystem: 

# /etc/init.d/ocfs2 start
Starting Oracle Cluster File System (OCFS2) 
mount.ocfs2: Device or resource busy while mounting /dev/sdb1 on /data/. 
Check 'dmesg' for more information on this error.
                                                          [FAILED]

dmesg has no information of why it failed.

I've used strace on mount command and believe this is the relevant problem: 

mount("/dev/sdb1", "/tmp/sadgfdgerwterfdsvcew/",
"ocfs2", 0, "heartbeat=local") = -1 EBUSY (Device or
resource busy)

Note that I've tried mounting it on "/tmp/sadgfdgerwterfdsvcew/"
which is a random directory created just before mounting to rule out the
possibility of a busy mount point.

Controlled reboot of node 2 did not resolve problem.

I believe this show that only node 1 has filesystem mounted: 
# echo "slotmap" | debugfs.ocfs2 -n /dev/sdb1
	Slot#   Node#
	    0       1

All commands shown here is done on node 2. 
I'm also able to read disk with other tools on server.

Please advise on how to proceed with debugging and resolving this issue.

Regards, 
Asle Ness

Could be that the device has been claimed by some other entity. Like lvm,
multipath, etc.

On Apr 13, 2011, at 4:08 AM, Asle N?ss <Asle.Naess at telio.no> wrote:
> Hi, 
> 
> I have the following setup: 
> 
> 2 linux nodes connected to shared SAN, both nodes have shared OCFS2 volume
as /dev/sdb1
> Both servers are running CentOS 5.5
> 
> Node 1 has volume mounted RW.
> Node 2 has volume mounted RO.
> 
> After an unscheduled reboot of node 2 it is no longer able to mount
filesystem:
> 
> # /etc/init.d/ocfs2 start
> Starting Oracle Cluster File System (OCFS2) 
> mount.ocfs2: Device or resource busy while mounting /dev/sdb1 on /data/. 
> Check 'dmesg' for more information on this error.
>                                                          [FAILED]
> 
> dmesg has no information of why it failed.
> 
> I've used strace on mount command and believe this is the relevant
problem:
> 
> mount("/dev/sdb1", "/tmp/sadgfdgerwterfdsvcew/",
"ocfs2", 0, "heartbeat=local") = -1 EBUSY (Device or
resource busy)
> 
> Note that I've tried mounting it on
"/tmp/sadgfdgerwterfdsvcew/" which is a random directory created just
before mounting to rule out the possibility of a busy mount point.
> 
> Controlled reboot of node 2 did not resolve problem.
> 
> I believe this show that only node 1 has filesystem mounted: 
> # echo "slotmap" | debugfs.ocfs2 -n /dev/sdb1
>    Slot#   Node#
>        0       1
> 
> All commands shown here is done on node 2. 
> I'm also able to read disk with other tools on server.
> 
> Please advise on how to proceed with debugging and resolving this issue.
> 
> Regards, 
> Asle Ness
> 
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users

On Wed, Apr 13, 2011 at 06:32:12AM -0700, Sunil Mushran
wrote:
> Could be that the device has been claimed by some other entity. Like lvm,
multipath, etc.
	Yeah, the EBUSY would make that seem likely.  Is there anything
in /sys/block/sdb/sdb1/holders/?

Joel

-- 

"You must remember this:
 A kiss is just a kiss,
 A sigh is just a sigh.
 The fundamental rules apply
 As time goes by."

			http://www.jlbec.org/
			jlbec at evilplan.org