Harry Mangalam
2011-Oct-28 21:00 UTC
[Gluster-users] some nodes mount unwritable /subvolumes/ of glusterfs (3.3b1)
Just set up and mounted a glusterfs consisting of 6 nodes:
$ gluster volume info
Volume Name: g6
Type: Distribute
Status: Started
Number of Bricks: 6
Transport-type: tcp
Bricks:
Brick1: pbs1:/data2
Brick2: pbs2:/data2
Brick3: pbs3:/data2
Brick4: pbs3:/data
Brick5: dabrick:/data2
Brick6: hef:/data2
Options Reconfigured:
auth.allow: 128.*
It's mounted on 9 similar (tho not identical) ubuntu 10.04.3 nodes
(64b Kubuntu Dektop) in a cluster (claw[1-9]), which are running a
self-compiled and installed version of 3.3b1. The gluster server
bricks are also running 10.04.3 (64b server).
However, the glusterfs mounts differentially on the nodes:
root at bduc-login:~
576 $ for ITER in $(seq 1 9); do echo -n "claw${ITER}:"; ssh
claw${ITER} 'df -h |egrep "gluz|pbs"' ; done;
claw1:pbs3:/g6 21T 114G 21T 1% /mnt/gluz
claw2:pbs3.calit2.uci.edu:/g6
21T 114G 21T 1% /mnt/gluz
claw3:pbs3.calit2.uci.edu:/g6
21T 114G 21T 1% /mnt/gluz
claw4:pbs3.calit2.uci.edu:/g6
21T 114G 21T 1% /mnt/gluz
claw5:pbs3.calit2.uci.edu:/g6
5.5T 2.7G 5.5T 1% /mnt/gluz <---
claw6:pbs3.calit2.uci.edu:/g6
21T 112G 21T 1% /mnt/gluz
claw7:pbs3.calit2.uci.edu:/g6
21T 114G 21T 1% /mnt/gluz
claw8:pbs3.calit2.uci.edu:/g6
5.5T 2.7G 5.5T 1% /mnt/gluz <---
claw9:pbs3.calit2.uci.edu:/g6
21T 112G 21T 1% /mnt/gluz
Obviously claw5 and claw8 are ... different. It looks like they're
mounting a component of the glusterfs (from the numbers, it's the
contribution of "Brick5: dabrick:/data2"), but not the whole thing.
The glusterfs was mounted on all of them with a script:
mount -t glusterfs pbs3.calit2.uci.edu:/g6 /mnt/gluz
Ive unmounted and remounted the glusterfs on claw8 with no difference.
I've also unmounted all instances of the glusterfs, then gluster-
stopped and restarted the volume on the gluster server node and then
remounted the fs as well. No diff.
When mounted, claw[58] can see some of the files not all of them (the
same ones that are on the 'dabrick' volume, as you might guess) but
cannot write to the mounted filesystem, even tho the user has
permissions to write.
...?!
The nodes that are having the problem have never had a gluster fs
mounted on them before.
--
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[ZOT 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
--
This signature has been OCCUPIED!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111028/d589fa81/attachment.html>
Harry Mangalam
2011-Oct-28 21:33 UTC
[Gluster-users] SOLVED: some nodes mount unwritable /subvolumes/ of glusterfs (3.3b1)
Answering my own question but it reveals an oddity: The problem was that the 2 nodes (claw[58]) that showed the mount of the subvolume did not have the /etc/hosts entries that the rest of the nodes did (I know, a hanging offense to use /etc/hosts to do DNS bookeeping and this is one reason why). But it seems like an bug if mounts appear to succeed instead of fail with the appropriate error (which did show up in the log when I went looking for it). Partial successes should be prefaced with some kind of error or explanation; just sayin'. Instead, the erroneous mount responded the same way as a successful mount and showed up as a successful mount with a 'df'. A user would only know that it was a failure when he tried to write or compared the df value to a successful mount. So, onwards! Harry On Friday 28 October 2011 14:00:54 Harry Mangalam wrote:> Just set up and mounted a glusterfs consisting of 6 nodes: > > $ gluster volume info > > Volume Name: g6 > Type: Distribute > Status: Started > Number of Bricks: 6 > Transport-type: tcp > Bricks: > Brick1: pbs1:/data2 > Brick2: pbs2:/data2 > Brick3: pbs3:/data2 > Brick4: pbs3:/data > Brick5: dabrick:/data2 > Brick6: hef:/data2 > Options Reconfigured: > auth.allow: 128.* > > > It's mounted on 9 similar (tho not identical) ubuntu 10.04.3 nodes > (64b Kubuntu Dektop) in a cluster (claw[1-9]), which are running a > self-compiled and installed version of 3.3b1. The gluster server > bricks are also running 10.04.3 (64b server). > > However, the glusterfs mounts differentially on the nodes: > > root at bduc-login:~ > 576 $ for ITER in $(seq 1 9); do echo -n "claw${ITER}:"; ssh > claw${ITER} 'df -h |egrep "gluz|pbs"' ; done; > > claw1:pbs3:/g6 21T 114G 21T 1% /mnt/gluz > claw2:pbs3.calit2.uci.edu:/g6 > 21T 114G 21T 1% /mnt/gluz > claw3:pbs3.calit2.uci.edu:/g6 > 21T 114G 21T 1% /mnt/gluz > claw4:pbs3.calit2.uci.edu:/g6 > 21T 114G 21T 1% /mnt/gluz > claw5:pbs3.calit2.uci.edu:/g6 > 5.5T 2.7G 5.5T 1% /mnt/gluz <--- > claw6:pbs3.calit2.uci.edu:/g6 > 21T 112G 21T 1% /mnt/gluz > claw7:pbs3.calit2.uci.edu:/g6 > 21T 114G 21T 1% /mnt/gluz > claw8:pbs3.calit2.uci.edu:/g6 > 5.5T 2.7G 5.5T 1% /mnt/gluz <--- > claw9:pbs3.calit2.uci.edu:/g6 > 21T 112G 21T 1% /mnt/gluz > > Obviously claw5 and claw8 are ... different. It looks like they're > mounting a component of the glusterfs (from the numbers, it's the > contribution of "Brick5: dabrick:/data2"), but not the whole thing. > The glusterfs was mounted on all of them with a script: > > mount -t glusterfs pbs3.calit2.uci.edu:/g6 /mnt/gluz > > Ive unmounted and remounted the glusterfs on claw8 with no > difference. I've also unmounted all instances of the glusterfs, > then gluster- stopped and restarted the volume on the gluster > server node and then remounted the fs as well. No diff. > > When mounted, claw[58] can see some of the files not all of them > (the same ones that are on the 'dabrick' volume, as you might > guess) but cannot write to the mounted filesystem, even tho the > user has permissions to write. > > ...?! > > The nodes that are having the problem have never had a gluster fs > mounted on them before.-- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [ZOT 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) -- This signature has been OCCUPIED! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111028/bd3081c0/attachment.html>