Gluster users - Feb 2009 - starting 4th node in 4 node dht cluster fails

Using dht translator to cluster together 4 nodes each with 4 disks.
Starting glusterfs on the 4th causes "Structure needs cleaning" when
ls-ing the mount point on any of them. It's fine with 3 nodes started.
Using fuse-2.7.4
GlusterFS 2.0.0rc1
Linux 2.6.18-53.el5 kernel

Errors from the log:


2009-02-06 15:23:51 E [dht-layout.c:460:dht_layout_normalize] dist1: found
anomalies in /. holes=1 overlaps=3
2009-02-06 15:23:51 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing
assignment on /
2009-02-06 15:23:51 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the
directory is not a virgin
2009-02-06 15:23:51 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge:
revalidate of / failed (Structure needs cleaning)
2009-02-06 15:23:51 E [dht-layout.c:460:dht_layout_normalize] dist1: found
anomalies in /. holes=1 overlaps=3
2009-02-06 15:23:51 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing
assignment on /
2009-02-06 15:23:51 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the
directory is not a virgin
2009-02-06 15:23:51 E [fuse-bridge.c:404:fuse_entry_cbk] glusterfs-fuse: 2:
LOOKUP() / => -1 (Structure needs cleaning)

Config for one of the machines:

volume posix-d1
 type storage/posix
 option directory /mnt/chard1/export
end-volume

volume locks1
  type features/locks
  subvolumes posix-d1 
end-volume


volume posix-d2
 type storage/posix
 option directory /mnt/chard2/export
end-volume


volume locks2
  type features/locks
  subvolumes posix-d2 
end-volume


volume posix-d3
 type storage/posix
 option directory /mnt/chard3/export
end-volume

volume locks3
  type features/locks
  subvolumes posix-d3 
end-volume


volume posix-d4
 type storage/posix
 option directory /mnt/chard4/export
end-volume

volume locks4
  type features/locks
  subvolumes posix-d4 
end-volume

volume home-ns
 type storage/posix
 option directory /var/local/glusterfs/namespace1
end-volume

volume home
 type cluster/distribute
 subvolumes locks1 locks2 locks3 locks4
end-volume

volume server
 type protocol/server
 option transport-type tcp
 subvolumes home
 option auth.addr.home.allow *
end-volume


volume zwei
      type protocol/client
      option transport-type tcp
      option remote-host zwei
      option remote-subvolume home
end-volume

volume char
      type protocol/client
      option transport-type tcp
      option remote-host char
      option remote-subvolume home
end-volume

volume pente
      type protocol/client
      option transport-type tcp
      option remote-host pente
      option remote-subvolume home
end-volume

volume tres
      type protocol/client
      option transport-type tcp
      option remote-host tres
      option remote-subvolume home
end-volume

volume dist1
    type cluster/distribute
    subvolumes pente char tres zwei
end-volume
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090206/a4bc3660/attachment.html>

Hi Craig,
 As you are using 'distribute' (client side) over 'distribute'
(server
side), this will not be working right now. To get it working right now, you
can have 4 export volumes from each server exported, and in client have 4x4
client protocol volumes, which you can aggregate with a single
'cluster/distribute' (which will have 16 subvolumes).

To get the below mentioned configuration working as is, you need to wait for
a week more IMO.

Regards,
Amar

2009/2/6 Craig Flockhart <craigflockhart at yahoo.com>
> Using dht translator to cluster together 4 nodes each with 4 disks.
> Starting glusterfs on the 4th causes "Structure needs cleaning"
when ls-ing
> the mount point on any of them. It's fine with 3 nodes started.
> Using fuse-2.7.4
> GlusterFS 2.0.0rc1
> Linux 2.6.18-53.el5 kernel
>
> Errors from the log:
>
>
> 2009-02-06 15:23:51 E [dht-layout.c:460:dht_layout_normalize] dist1: found
> anomalies in /. holes=1 overlaps=3
> 2009-02-06 15:23:51 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing
> assignment on /
> 2009-02-06 15:23:51 E [dht-selfheal.c:422:dht_selfheal_directory] dist1:
> the directory is not a virgin
> 2009-02-06 15:23:51 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge:
> revalidate of / failed (Structure needs cleaning)
> 2009-02-06 15:23:51 E [dht-layout.c:460:dht_layout_normalize] dist1: found
> anomalies in /. holes=1 overlaps=3
> 2009-02-06 15:23:51 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing
> assignment on /
> 2009-02-06 15:23:51 E [dht-selfheal.c:422:dht_selfheal_directory] dist1:
> the directory is not a virgin
> 2009-02-06 15:23:51 E [fuse-bridge.c:404:fuse_entry_cbk] glusterfs-fuse: 2:
> LOOKUP() / => -1 (Structure needs cleaning)
>
> Config for one of the machines:
>
> volume posix-d1
>  type storage/posix
>  option directory /mnt/chard1/export
> end-volume
>
> volume locks1
>   type features/locks
>   subvolumes posix-d1
> end-volume
>
>
> volume posix-d2
>  type storage/posix
>  option directory /mnt/chard2/export
> end-volume
>
>
> volume locks2
>   type features/locks
>   subvolumes posix-d2
> end-volume
>
>
> volume posix-d3
>  type storage/posix
>  option directory /mnt/chard3/export
> end-volume
>
> volume locks3
>   type features/locks
>   subvolumes posix-d3
> end-volume
>
>
> volume posix-d4
>  type storage/posix
>  option directory /mnt/chard4/export
> end-volume
>
> volume locks4
>   type features/locks
>   subvolumes posix-d4
> end-volume
>
> volume home-ns
>  type storage/posix
>  option directory /var/local/glusterfs/namespace1
> end-volume
>
> volume home
>  type cluster/distribute
>  subvolumes locks1 locks2 locks3 locks4
> end-volume
>
> volume server
>  type protocol/server
>  option transport-type tcp
>  subvolumes home
>  option auth.addr.home.allow *
> end-volume
>
>
> volume zwei
>       type protocol/client
>       option transport-type tcp
>       option remote-host zwei
>       option remote-subvolume home
> end-volume
>
> volume char
>       type protocol/client
>       option transport-type tcp
>       option remote-host char
>       option remote-subvolume home
> end-volume
>
> volume pente
>       type protocol/client
>       option transport-type tcp
>       option remote-host pente
>       option remote-subvolume home
> end-volume
>
> volume tres
>       type protocol/client
>       option transport-type tcp
>       option remote-host tres
>       option remote-subvolume home
> end-volume
>
> volume dist1
>     type cluster/distribute
>     subvolumes pente char tres zwei
> end-volume
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
>

-- 
Amar Tumballi
Gluster/GlusterFS Hacker
[bulde on #gluster/irc.gnu.org]
http://www.zresearch.com - Commoditizing Super Storage!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090206/ef563d87/attachment.html>

> I tried this but still get the same result (3 nodes ok, 4th node breaks it
- doesn't matter which the 4th node is). "getfattr -d" returned
nothing on the mount directory btw.
I had this same sort of problem.  Do a man on getfattr and you will find
that the -m option is needed since it will only return extended attributes
that match the pattern "^user\\." by default.  If you use:

    getfattr -d -m '\.' /path-to-backend

And you will get something like

getfattr: Removing leading '/' from absolute path names
# file: local/scratch2
trusted.glusterfs.afr.entry-pending=0sAAAAAAAAAAAtrusted.glusterfs.dht=0sAAAAAQAAAACf///7v///+Q=trusted.glusterfs.test="working\000"

After using a setfattr -x trusted.glusterfs.dht on the backend mount points
everything worked again.

I just recently installed 2.0.0rc1 and set up the nufa translator on
top of the afr translator after trying out nufa alone across my cluster.
Unfortunately I made a couple of small mistakes in the configuration and
got the "Structure needs cleaning" issue.  So I kept banging my head
until
I found this solution.

-Tim



-- 
=================================================================| J. Timothy
Kwiatkowski, PhD   jtk at ou.edu      (405) 325-6913   |
| SCIF (Shell Crustal Imaging Facility, Geology and Geophysics)  |
| SEC E1468                                   (Mailing SEC 810)  |
| University of Oklahoma                                         |
==================================================================