Ocfs2 users - Jun 2011 - cannot write to filesystem, permission denied?

Hello,

We have installed Fedora 15 to get the latest ocfs2 release nicely
packaged in a 2.6.38 kernel and ocfs2-tools 1.6.3.
Setup went fine, mkfs.ocfs2 went fine, a cluster was created and the
local node added:

# o2cb_ctl -C -i -n mycluster -t cluster
Cluster mycluster created
# o2cb_ctl -C -i -n lab -t node -a number=1 -a
ip_address=192.168.111.5 -a ip_port=7778 -a cluster=mycluster


O2CB is configured and starts up as it should and reports that
mycluster is active, the O2CB heartbeat is active and the filesystem
mounts OK.

But: writes to the filesystem all fail with 'permission denied'.
Example:

# date > /o2/file.txt
-bash: /o2/file.txt: Permission denied
# mkdir /o2/dir
mkdir: cannot create directory `/o2/dir': Permission denied
# id
uid=0(root) gid=0(root)
groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
# mount | grep o2
/dev/mapper/vg01-lv_mycluster on /o2 type ocfs2
(rw,relatime,_netdev,heartbeat=local,nointr,data=ordered,errors=remount-ro,coherency=full,user_xattr,acl)

According to strace, the permission denied is from

mkdir("/o2/dir", 0777)                  = -1 EACCES (Permission
denied)



Do you know what is the cause of this apparent "read-only" mode?
(although it is reported as a mounted rw)

Any pointers for further troubleshooting is greatly appreciated!

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sven Karlsson [07.06.2011 14:01]:
> Hello,
Hi Sven,
> O2CB is configured and starts up as it should and reports that
> mycluster is active, the O2CB heartbeat is active and the filesystem
> mounts OK.
> 
> But: writes to the filesystem all fail with 'permission denied'.
Example:
> # mount | grep o2
> /dev/mapper/vg01-lv_mycluster on /o2 type ocfs2
>
(rw,relatime,_netdev,heartbeat=local,nointr,data=ordered,errors=remount-ro,coherency=full,user_xattr,acl)
> Do you know what is the cause of this apparent "read-only" mode?
> (although it is reported as a mounted rw)
> 
> Any pointers for further troubleshooting is greatly appreciated!
Unfortunately, mount shows the filesystem as rw when even when it was
remounted ro because of errors (mount option: errors=remount-ro). Maybe
dmesg can tell you if there were errors and that the drive is in ro
access now.

I suggest that you unmount the filesystem on all nodes and do a

fsck.ocfs2 -f dev/mapper/vg01-lv_mycluster

on one node. Since the fs is new and empty, you may even want to add the
- -y option ;-) (see man fsck.ocfs2).

HTH
Werner
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iEYEARECAAYFAk3uGboACgkQk33Krq8b42P3mQCeK6/UexEic7Q0lw2G2Wxcr3e8
Jc4AnA3YsaBS8TFEO4dfODJuUcitwgop
=cAzC
-----END PGP SIGNATURE-----

On Tue, Jun 7, 2011 at 2:29 PM, Werner Flamme <werner.flamme at ufz.de>
wrote:
> Sven Karlsson [07.06.2011 14:01]:
> Unfortunately, mount shows the filesystem as rw when even when it was
> remounted ro because of errors (mount option: errors=remount-ro). Maybe
> dmesg can tell you if there were errors and that the drive is in ro
> access now.
No such message in dmesg. Perhaps this info is available elsewhere?

> I suggest that you unmount the filesystem on all nodes and do a
It's only one node at the moment so no problem there. mounted.ocfs2
also reports that its only accessed at the local node.
> fsck.ocfs2 -f dev/mapper/vg01-lv_mycluster
Now this is strange:

# fsck.ocfs2 -f /dev/vg01/lv_mycluster
fsck.ocfs2 1.6.3
fsck.ocfs2: Could not create domain while initializing the DLM
#

However, if the filesystem is mounted, it is checked OK
with fsck.ocfs2 -nf ?...

Maybe there is some locking problem? I also tried mounting the
filesystem with heartbeat=none, mount=local, but these options seem
to be for older ocfs2 versions.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sven Karlsson [07.06.2011 14:47]:
> On Tue, Jun 7, 2011 at 2:29 PM, Werner Flamme <werner.flamme at
ufz.de> wrote:
>> Sven Karlsson [07.06.2011 14:01]:
>> Unfortunately, mount shows the filesystem as rw when even when it was
>> remounted ro because of errors (mount option: errors=remount-ro). Maybe
>> dmesg can tell you if there were errors and that the drive is in ro
>> access now.
> 
> No such message in dmesg. Perhaps this info is available elsewhere?
> 
> 
>> I suggest that you unmount the filesystem on all nodes and do a
> 
> It's only one node at the moment so no problem there. mounted.ocfs2
> also reports that its only accessed at the local node.
> 
>> fsck.ocfs2 -f dev/mapper/vg01-lv_mycluster
> 
> Now this is strange:
> 
> # fsck.ocfs2 -f /dev/vg01/lv_mycluster
> fsck.ocfs2 1.6.3
> fsck.ocfs2: Could not create domain while initializing the DLM
> #
> 
> However, if the filesystem is mounted, it is checked OK
> with fsck.ocfs2 -nf  ...
> 
> Maybe there is some locking problem? I also tried mounting the
> filesystem with heartbeat=none, mount=local, but these options seems
> to be for older ocfs2 versions.
Sven,

as to dmesg: I know these infos only from dmesg. Sometimes I am
surprised how many infos I am missing when I only look at
/var/log/messages instead... I found all important ocfs2 messages in
dmesg. May be other on your box, I have SLES only.

Did you do the "o2cb configure" thing before mounting the filesystem?
The "Could not create domain" sounds to me like a bit of
misconfiguration. And the output of "/etc/init.d/o2cb status" should
look like (depending on the values you gave at configure time):

Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 61
  Network idle timeout: 60000
  Network keepalive delay: 10000
  Network reconnect delay: 10000
Checking O2CB heartbeat: Active

Maybe you have to "/etc/init.d/o2cb offline" and reconfigure? That
helped for me (in 1 case out of my 15 nodes). After reconfiguration, you
may mount again and see the results via dmesg (I do). This may be
necessary when you hat o2cb online first and created the cluster
afterwards. I found it necessary to first create the cluster config,
then "o2cb configure", and then mounting.

Can you see your cluster in ocfs2console? Is your
/etc/ocfs2/cluster.conf correct? My man page for o2cb_ctl says "Users
are not advised to use this program directly but instead use the O2CB
init service and/or ocfs2console." I learned to obey such hints ;-)

Regards,
Werner
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iEYEARECAAYFAk3uIkoACgkQk33Krq8b42NQBgCeK7TGnqvHoJ8loFlqcyhbIKiX
XnkAniEQ7UU14vAmfw1RD20ew7W9Wglf
=xZTa
-----END PGP SIGNATURE-----

On Tue, Jun 7, 2011 at 3:06 PM, Werner Flamme <werner.flamme at ufz.de>
wrote:
> Sven Karlsson [07.06.2011 14:47]:
>> Now this is strange:
>>
>> # fsck.ocfs2 -f /dev/vg01/lv_mycluster
>> fsck.ocfs2 1.6.3
>> fsck.ocfs2: Could not create domain while initializing the DLM
>> #
>>
>> However, if the filesystem is mounted, it is checked OK
>> with fsck.ocfs2 -nf ?...
>>
>> Maybe there is some locking problem?
Turns out it all was a SELinux problem. SELinux was enabled by
default. By editing /etc/selinux/config and setting SELINUX=disabled
and subsequent reboot solved both the file/directory creation problem,
and the fsck as above also works.

On 06/07/2011 05:01 AM, Sven Karlsson wrote:
> Hello,
>
> We have installed Fedora 15 to get the latest ocfs2 release nicely
> packaged in a 2.6.38 kernel and ocfs2-tools 1.6.3.
> Setup went fine, mkfs.ocfs2 went fine, a cluster was created and the
> local node added:
>
> # o2cb_ctl -C -i -n mycluster -t cluster
> Cluster mycluster created
> # o2cb_ctl -C -i -n lab -t node -a number=1 -a
> ip_address=192.168.111.5 -a ip_port=7778 -a cluster=mycluster
>
>
> O2CB is configured and starts up as it should and reports that
> mycluster is active, the O2CB heartbeat is active and the filesystem
> mounts OK.
>
> But: writes to the filesystem all fail with 'permission denied'.
Example:
>
> # date>  /o2/file.txt
> -bash: /o2/file.txt: Permission denied
> # mkdir /o2/dir
> mkdir: cannot create directory `/o2/dir': Permission denied
> # id
> uid=0(root) gid=0(root)
> groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> # mount | grep o2
> /dev/mapper/vg01-lv_mycluster on /o2 type ocfs2
>
(rw,relatime,_netdev,heartbeat=local,nointr,data=ordered,errors=remount-ro,coherency=full,user_xattr,acl)
>
> According to strace, the permission denied is from
>
> mkdir("/o2/dir", 0777)                  = -1 EACCES (Permission
denied)
>
>
>
> Do you know what is the cause of this apparent "read-only" mode?
> (although it is reported as a mounted rw)
>
> Any pointers for further troubleshooting is greatly appreciated!
You have acl enabled. Maybe that is what is triggering EACCES.