Gluster users - Oct 2011 - Permission denied after upgrade and rebalancing

Hi,

when our two-node distributed backup storage cluster, running on
GlusterFS 3.0.2 (Ubuntu Lucid packages), started running out of space,
I added a third node and tried rebalancing them. The scale-n-defrag
script didn't work, though. I always got a "stale NFS handle".

So I decided to upgrade and installed the 3.2.4 packages from
semiosis' Launchpad PPA, followed the upgrade guide for 3.0 -> 3.2 and
then started "gluster volume rebalance". After two days, the three
nodes are now about equally full, so the rebalancing worked.

Unfortunately, it seems that it also broke the cluster filesystem. We
get a lot of "permission denied" errors although file system
permissions are correct. The log contains a lot of these entries:

[2011-10-17 10:11:48.979801] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
operation failed: Permission denied
[2011-10-17 10:11:48.979892] I [dht-common.c:478:dht_revalidate_cbk]
0-glfs-dht: subvolume glfs-client-2 for /bacula/mail02 returned -1
(Permission denied)
[2011-10-17 10:11:48.980216] I [dht-layout.c:192:dht_layout_search]
0-glfs-dht: no subvolume for hash (value) = 1486987642
[2011-10-17 10:11:48.981101] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
operation failed: Permission denied
[2011-10-17 10:11:48.981581] I [dht-layout.c:192:dht_layout_search]
0-glfs-dht: no subvolume for hash (value) = 1486987642
[2011-10-17 10:11:48.982468] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
operation failed: Permission denied

How can I repair this?

Best regards,
 Jochen

Hi Jochen,

Can you try running 'gluster volume rebalance <VOLNAME> fix-layout
start'
again?

Seems that some of the layout is not valid. Also, make sure all your bricks
are up and running.

Also, are you using ACLs? can the accessing of directories/files work as
'root' user?

Regards,
Amar

On Mon, Oct 17, 2011 at 6:29 PM, Jochen Lillich <jochen at freistil.it>
wrote:
> Hi,
>
> when our two-node distributed backup storage cluster, running on
> GlusterFS 3.0.2 (Ubuntu Lucid packages), started running out of space,
> I added a third node and tried rebalancing them. The scale-n-defrag
> script didn't work, though. I always got a "stale NFS
handle".
>
> So I decided to upgrade and installed the 3.2.4 packages from
> semiosis' Launchpad PPA, followed the upgrade guide for 3.0 -> 3.2
and
> then started "gluster volume rebalance". After two days, the
three
> nodes are now about equally full, so the rebalancing worked.
>
> Unfortunately, it seems that it also broke the cluster filesystem. We
> get a lot of "permission denied" errors although file system
> permissions are correct. The log contains a lot of these entries:
>
> [2011-10-17 10:11:48.979801] I
> [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
> operation failed: Permission denied
> [2011-10-17 10:11:48.979892] I [dht-common.c:478:dht_revalidate_cbk]
> 0-glfs-dht: subvolume glfs-client-2 for /bacula/mail02 returned -1
> (Permission denied)
> [2011-10-17 10:11:48.980216] I [dht-layout.c:192:dht_layout_search]
> 0-glfs-dht: no subvolume for hash (value) = 1486987642
> [2011-10-17 10:11:48.981101] I
> [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
> operation failed: Permission denied
> [2011-10-17 10:11:48.981581] I [dht-layout.c:192:dht_layout_search]
> 0-glfs-dht: no subvolume for hash (value) = 1486987642
> [2011-10-17 10:11:48.982468] I
> [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
> operation failed: Permission denied
>
> How can I repair this?
>
> Best regards,
>  Jochen
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111017/975e2564/attachment.html>