Gluster users - Jan 2015 - Changes in SELinux handling in 3.6+

Hi all

I run a small HPC with a single storage node (Scientific Linux 6, using Gluster
3.5.2 RPMs). SELinux is set to Enforcing. Home directories are shared to a
handful of compute nodes where SELinux is also set to Enforcing.

The system is quite happy provided you specify the correct SELinux context when
mounting /home (system_u:object_r:user_home_t:s0). With 3.5 this done with

$ mount -t glusterfs storage:/home /home -o
context="system_u:object_r:user_home_t:s0"


I'm going to be adding a second storage node and will be setting up
replication between the two nodes. While I'm doing that, I might as well
upgrade to 3.6+.

During testing I found that version 3.6.1 of mount.glusterfs does not support
the 'context' mount option. Is the removal of this functionality
intentional? There's unfortunately very little documentation available on
SELinux support in Gluster. Version 3.6.1 does have the 'selinux' mount
option, but it doesn't seem to do anything.

It should also be noted that a 3.5.3 client mounting a 3.6.1 server works as
expected, a 3.6.1 client never has the correct SELinux tags. The issue seems to
be limited to the mount.glusterfs utility.


Below I'll paste the output of my testing. 'storage0' runs 3.5.3 and
works as expected, 'storage1' runs 3.6.1 and doesn't honour SELinux
tags.

Any help will be appreciated.

ciao
Charl


=== Gluster 3.5.3 START ==
[root at storage0 /]$ yum install glusterfs-{server,api,libs}-3.5.3 xfsprogs
[root at storage0 /]$ mkfs.xfs -i size=512 /dev/sdb
[root at storage0 /]$ mkdir /brick1
[root at storage0 /]$ mount /dev/sdb /brick1
[root at storage0 /]$ mkdir /brick1/home
[root at storage0 /]$ ls -lsaZ /home
total 8
drwxr-xr-x. root root system_u:object_r:home_root_t:s0 .
dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..

[root at storage0 /]$ chcon system_u:object_r:home_root_t:s0 /brick1/home
[root at storage0 /]$ ls -lsaZ /brick1/home
total 0
drwxr-xr-x. root root system_u:object_r:home_root_t:s0 .
drwxr-xr-x. root root system_u:object_r:file_t:s0      ..

[root at storage0 /]$ service glusterd start
Starting glusterd:                                         [  OK  ]

[root at storage0 /]$ gluster volume create home storage0:/brick1/home
volume create: home: success: please start the volume to access data

[root at storage0 /]$ gluster volume start home
volume start: home: success

[root at storage0 /]$ mount -t glusterfs storage0:/home home
[root at storage0 /]$ ls -lsaZ /home
total 4
drwxr-xr-x. root root system_u:object_r:fusefs_t:s0    .
dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..

[testuser at launch ~]$ ssh testuser at storage0
Password:
Could not chdir to home directory /home/testuser: No such file or directory
[testuser at storage0 /]$


[root at storage0 /]$ umount home
[root at storage0 /]$ mount -t glusterfs storage0:/home home -o
context="system_u:object_r:user_home_t:s0"

[testuser at launch ~]$ ssh testuser at storage0
Password:
Creating home directory for testuser.
Last login: Tue Jan  6 10:40:29 2015 from 192.168.2.3
[testuser at storage0 ~]$


[root at storage0 /]$ ls -lsaZ /home
total 4
drwxr-xr-x. root     root  system_u:object_r:user_home_t:s0 .
dr-xr-xr-x. root     root  system_u:object_r:root_t:s0      ..
drwxr-xr-x. testuser users system_u:object_r:user_home_t:s0 testuser

[root at storage0 /]$ rpm -qa | grep gluster
glusterfs-libs-3.5.3-1.el6.x86_64
glusterfs-api-3.5.3-1.el6.x86_64
glusterfs-cli-3.5.3-1.el6.x86_64
glusterfs-server-3.5.3-1.el6.x86_64
glusterfs-3.5.3-1.el6.x86_64
glusterfs-fuse-3.5.3-1.el6.x86_64

=== Gluster 3.5.3 END ==
=== Gluster 3.6.1 START ==
[root at storage1 /]$ yum install glusterfs-{server,api,libs}-3.6.1 xfsprogs
[root at storage1 /]$ mkfs.xfs -i size=512 /dev/sdb
[root at storage1 /]$ mkdir /brick1
[root at storage1 /]$ mount /dev/sdb /brick1
[root at storage1 /]$ mkdir /brick1/home
[root at storage1 /]$ ls -lsaZ /home
total 8
drwxr-xr-x. root root system_u:object_r:home_root_t:s0 .
dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..

[root at storage1 /]$ chcon system_u:object_r:home_root_t:s0 /brick1/home
[root at storage1 /]$ ls -lsaZ /brick1/home
total 0
drwxr-xr-x. root root system_u:object_r:home_root_t:s0 .
drwxr-xr-x. root root system_u:object_r:file_t:s0      ..

[root at storage1 /]$ service glusterd start
Starting glusterd:                                         [  OK  ]

[root at storage1 /]$ gluster volume create home storage1:/brick1/home
volume create: home: success: please start the volume to access data

[root at storage1 /]$ gluster volume start home
volume start: home: success

[root at storage1 /]$ mount -t glusterfs storage1:/home home
[root at storage1 /]$ ls -lsaZ /home
total 4
drwxr-xr-x. root root system_u:object_r:fusefs_t:s0    .
dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..

[testuser at launch ~]$ ssh testuser at storage1
Password:
Could not chdir to home directory /home/testuser: No such file or directory
[testuser at storage1 /]$


[root at storage1 /]$ umount home
[root at storage1 /]$ mount -t glusterfs storage1:/home home -o
context="system_u:object_r:user_home_t:s0"
Invalid option: context

[root at storage1 /]$ mount -t glusterfs storage1:/home home -o selinux
[root at storage1 /]$ ls -lsaZ /home
total 4
drwxr-xr-x. root root system_u:object_r:fusefs_t:s0    .
dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..

[root at storage1 /]$ rpm -qa | grep gluster
glusterfs-libs-3.6.1-1.el6.x86_64
glusterfs-api-3.6.1-1.el6.x86_64
glusterfs-cli-3.6.1-1.el6.x86_64
glusterfs-server-3.6.1-1.el6.x86_64
glusterfs-3.6.1-1.el6.x86_64
glusterfs-fuse-3.6.1-1.el6.x86_64

=== Gluster 3.5.3 END ==

--
Charl M?ller | System Administrator: HPC and Linux systems | University of
Stellenbosch | t: +27 21 808 9490
https://blogs.sun.ac.za/it | http://www.sun.ac.za |@ITStellenUni
https://www.google.co.za/maps/place/Information+Technology/@-33.929269,18.865966


The integrity and confidentiality of this email is governed by these terms /
Hierdie terme bepaal die integriteit en vertroulikheid van hierdie epos.
http://www.sun.ac.za/emaildisclaimer
The integrity and confidentiality of this email is governed by these terms /
Hierdie terme bepaal die integriteit en vertroulikheid van hierdie epos.
http://www.sun.ac.za/emaildisclaimer

Hi Charl,

Sorry for the very late response. Thanks for mentioning clearly on what was the
issue. As you have mentioned, from gluster-3.6+ versions, though selinux option
is there, you are thrown an error, "Invalid option: context". It
happens when you try to set context while mounting, which was done
unintentionally. We have filed bug against mainline[1](for which the patch/fix
is merged in master) and have backported the same to 3.7[2] and 3.6[3] as well.
We are planning to get it fixed with the next minor updates in the coming
releases. With the fix, you will able to set selinux context while mounting.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1287763

[2] https://bugzilla.redhat.com/show_bug.cgi?id=1287877

[3] https://bugzilla.redhat.com/show_bug.cgi?id=1287878

Thank you :-)

--
Regards,
Manikandan Selvaganesh.
> Hi all
> 
> I run a small HPC with a single storage node (Scientific Linux 6, using
Gluster 3.5.2 RPMs). SELinux is set to Enforcing. Home directories are shared to
a handful of compute nodes where SELinux is also set to Enforcing.
> 
> The system is quite happy provided you specify the correct SELinux context
when mounting /home (system_u:object_r:user_home_t:s0). With 3.5 this done with
> 
> $ mount -t glusterfs storage:/home /home -o
context="system_u:object_r:user_home_t:s0"
> 
> 
> I'm going to be adding a second storage node and will be setting up
replication between the two nodes. While I'm doing that, I might as well
upgrade to 3.6+.
> 
> During testing I found that version 3.6.1 of mount.glusterfs does not
support the 'context' mount option. Is the removal of this functionality
intentional? There's unfortunately very little documentation available on
SELinux support in Gluster. Version 3.6.1 does have the 'selinux' mount
option, but it doesn't seem to do anything.
> 
> It should also be noted that a 3.5.3 client mounting a 3.6.1 server works
as expected, a 3.6.1 client never has the correct SELinux tags. The issue seems
to be limited to the mount.glusterfs utility.
> 
> 
> Below I'll paste the output of my testing. 'storage0' runs
3.5.3 and works as expected, 'storage1' runs 3.6.1 and doesn't
honour SELinux tags.
> 
> Any help will be appreciated.
> 
> ciao
> Charl
> 
> 
> === Gluster 3.5.3 START ==> 
> [root at storage0 /]$ yum install glusterfs-{server,api,libs}-3.5.3
xfsprogs
> [root at storage0 /]$ mkfs.xfs -i size=512 /dev/sdb
> [root at storage0 /]$ mkdir /brick1
> [root at storage0 /]$ mount /dev/sdb /brick1
> [root at storage0 /]$ mkdir /brick1/home
> [root at storage0 /]$ ls -lsaZ /home
> total 8
> drwxr-xr-x. root root system_u:object_r:home_root_t:s0 .
> dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..
> 
> [root at storage0 /]$ chcon system_u:object_r:home_root_t:s0 /brick1/home
> [root at storage0 /]$ ls -lsaZ /brick1/home
> total 0
> drwxr-xr-x. root root system_u:object_r:home_root_t:s0 .
> drwxr-xr-x. root root system_u:object_r:file_t:s0      ..
> 
> [root at storage0 /]$ service glusterd start
> Starting glusterd:                                         [  OK  ]
> 
> [root at storage0 /]$ gluster volume create home storage0:/brick1/home
> volume create: home: success: please start the volume to access data
> 
> [root at storage0 /]$ gluster volume start home
> volume start: home: success
> 
> [root at storage0 /]$ mount -t glusterfs storage0:/home home
> [root at storage0 /]$ ls -lsaZ /home
> total 4
> drwxr-xr-x. root root system_u:object_r:fusefs_t:s0    .
> dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..
> 
> [testuser at launch ~]$ ssh testuser at storage0
> Password:
> Could not chdir to home directory /home/testuser: No such file or directory
> [testuser at storage0 /]$
> 
> 
> [root at storage0 /]$ umount home
> [root at storage0 /]$ mount -t glusterfs storage0:/home home -o
context="system_u:object_r:user_home_t:s0"
> 
> [testuser at launch ~]$ ssh testuser at storage0
> Password:
> Creating home directory for testuser.
> Last login: Tue Jan  6 10:40:29 2015 from 192.168.2.3
> [testuser at storage0 ~]$
> 
> 
> [root at storage0 /]$ ls -lsaZ /home
> total 4
> drwxr-xr-x. root     root  system_u:object_r:user_home_t:s0 .
> dr-xr-xr-x. root     root  system_u:object_r:root_t:s0      ..
> drwxr-xr-x. testuser users system_u:object_r:user_home_t:s0 testuser
> 
> [root at storage0 /]$ rpm -qa | grep gluster
> glusterfs-libs-3.5.3-1.el6.x86_64
> glusterfs-api-3.5.3-1.el6.x86_64
> glusterfs-cli-3.5.3-1.el6.x86_64
> glusterfs-server-3.5.3-1.el6.x86_64
> glusterfs-3.5.3-1.el6.x86_64
> glusterfs-fuse-3.5.3-1.el6.x86_64
> 
> === Gluster 3.5.3 END ==> 
> === Gluster 3.6.1 START ==> 
> [root at storage1 /]$ yum install glusterfs-{server,api,libs}-3.6.1
xfsprogs
> [root at storage1 /]$ mkfs.xfs -i size=512 /dev/sdb
> [root at storage1 /]$ mkdir /brick1
> [root at storage1 /]$ mount /dev/sdb /brick1
> [root at storage1 /]$ mkdir /brick1/home
> [root at storage1 /]$ ls -lsaZ /home
> total 8
> drwxr-xr-x. root root system_u:object_r:home_root_t:s0 .
> dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..
> 
> [root at storage1 /]$ chcon system_u:object_r:home_root_t:s0 /brick1/home
> [root at storage1 /]$ ls -lsaZ /brick1/home
> total 0
> drwxr-xr-x. root root system_u:object_r:home_root_t:s0 .
> drwxr-xr-x. root root system_u:object_r:file_t:s0      ..
> 
> [root at storage1 /]$ service glusterd start
> Starting glusterd:                                         [  OK  ]
> 
> [root at storage1 /]$ gluster volume create home storage1:/brick1/home
> volume create: home: success: please start the volume to access data
> 
> [root at storage1 /]$ gluster volume start home
> volume start: home: success
> 
> [root at storage1 /]$ mount -t glusterfs storage1:/home home
> [root at storage1 /]$ ls -lsaZ /home
> total 4
> drwxr-xr-x. root root system_u:object_r:fusefs_t:s0    .
> dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..
> 
> [testuser at launch ~]$ ssh testuser at storage1
> Password:
> Could not chdir to home directory /home/testuser: No such file or directory
> [testuser at storage1 /]$
> 
> 
> [root at storage1 /]$ umount home
> [root at storage1 /]$ mount -t glusterfs storage1:/home home -o
context="system_u:object_r:user_home_t:s0"
> Invalid option: context
> 
> [root at storage1 /]$ mount -t glusterfs storage1:/home home -o selinux
> [root at storage1 /]$ ls -lsaZ /home
> total 4
> drwxr-xr-x. root root system_u:object_r:fusefs_t:s0    .
> dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..
> 
> [root at storage1 /]$ rpm -qa | grep gluster
> glusterfs-libs-3.6.1-1.el6.x86_64
> glusterfs-api-3.6.1-1.el6.x86_64
> glusterfs-cli-3.6.1-1.el6.x86_64
> glusterfs-server-3.6.1-1.el6.x86_64
> glusterfs-3.6.1-1.el6.x86_64
> glusterfs-fuse-3.6.1-1.el6.x86_64
> 
> === Gluster 3.5.3 END ==> 
> 
> --
> Charl M?ller | System Administrator: HPC and Linux systems | University of
Stellenbosch | t: +27 21 808 9490
> https://blogs.sun.ac.za/it | http://www.sun.ac.za |@ITStellenUni
>
https://www.google.co.za/maps/place/Information+Technology/@-33.929269,18.865966

The integrity and confidentiality of this email is governed by these terms /
Hierdie terme bepaal die integriteit en vertroulikheid van hierdie epos.
http://www.sun.ac.za/emaildisclaimer
The integrity and confidentiality of this email is governed by these terms /
Hierdie terme bepaal die integriteit en vertroulikheid van hierdie epos.
http://www.sun.ac.za/emaildisclaimer
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users