Gluster users - Oct 2008 - Glusterfs 13.12 client crashes; how to proceed?

Friends, 

Hi!  I am so besieged with gluster-setup problems, I am not sure where
to begin.  I've compiled and glusterfs-1.3.12
(glusterfs--mainline--2.5--patch-797) on 3 seperate computers with the
intention of building an HA Cluster as in Daniel Maher's howto:
http://gluster.org/docs/index.php/High-availability_storage_using_server-side_AFR
glusterfsd -f /etc/glusterfs/glusterfs-server.vol -L DEBUG
-l /var/log/gluster Two of the servers have essentially the same server
config, as shown in the howto.  They only differ in that they refer to
each other.  

The client config is also pulled directly from the howto.  I have
reason to believe the configs are working (fixed the typos already) but
cannot be sure.  The servers are both started with the line:

glusterfsd -f /etc/glusterfs/glusterfs-server.vol -L DEBUG
  -l /var/log/glusterfsd 

The logs mention a rather worrying: 

  2008-10-01 22:15:56 D [xlator.c:145:xlator_set_type] xlator:
  dlsym(notify)
  on /usr/local/lib/glusterfs/1.3.12/xlator/features/posix-locks.so:
  undefined symbol: notify -- neglecting

other than that, things seem to  go well until I get the other server
fired up, they talk fora a while, and then one serve2008-10-01 22:16:19

D [ip.c:120:gf_auth] wwwfs-ds-locks: allowed = "172.17.1.*", received
ip addr = "172.17.1.101" 2008-10-01 22:16:19 D
[server-protocol.c:5674:mop_setvolume] server: accepted client from
172.17.1.101:1023 2008-10-01 22:16:19 D
[server-protocol.c:5717:mop_setvolume] server: creating inode table
with lru_limit=1024, xlator=wwwfs-ds-locks 2008-10-01 22:16:19 D
[inode.c:1163:inode_table_new] wwwfs-ds-locks: creating new inode table
with lru_limit=1024, sizeof(inode_t)=96 2008-10-01 22:16:19 D
[inode.c:577:__create_inode] wwwfs-ds-locks/inode: create inode(1)
2008-10-01 22:16:19 D [inode.c:367:__active_inode]
wwwfs-ds-locks/inode: activating inode(1), lru=0/1024 r ends with: 
==
The other  says: 

2008-10-01 22:16:54 D [tcp-client.c:161:tcp_connect] wwwfs-cedar-ds:
connect on 7 in progress (non-blocking)
2008-10-01 22:16:54 D [tcp-client.c:205:tcp_connect] wwwfs-cedar-ds:
connection on 7 success
2008-10-01 22:16:54 D [client-protocol.c:5362:notify] wwwfs-cedar-ds:
got GF_EVENT_CHILD_UP
2008-10-01 22:16:54 D
[client-protocol.c:5103:client_protocol_handshake_reply]
wwwfs-cedar-ds: reply frame has callid: 424242
2008-10-01 22:16:54 D
[client-protocol.c:5137:client_protocol_handshake_reply]
wwwfs-cedar-ds: SETVOLUME on remote-host succeeded
2008-10-01 22:16:54 D [afr.c:6001:notify] wwwfs-ds-afr:
GF_EVENT_CHILD_UP from wwwfs-cedar-ds
2008-10-01 22:16:54 D [afr.c:5933:afr_check_xattr_cbk] wwwfs-ds-afr:
'wwwfs-cedar-ds' supports Extended attribute
2008-10-01 22:17:49 D
[client-protocol.c:4763:client_protocol_reconnect] wwwfs-cedar-ds:
breaking reconnect chain
/var/log/glusterfsd lines 138-149/


The client really bombs: 

 glusterfs -f /etc/glusterfs/glusterfs-client.vol -L DEBUG
-l /var/log/glusterfs /var/www/

It too says: 

2008-10-01 21:29:46 D [xlator.c:145:xlator_set_type] xlator:
dlsym(notify)
on /usr/local/lib/glusterfs/1.3.12/xlator/performance/write-behind.so:
undefined symbol: notify -- neglecting

As well as 

2008-10-01 22:34:08 E
[client-protocol.c:5133:client_protocol_handshake_reply] www-fs:
SETVOLUME on remote-host failed: ret=-1 error=remote-subvolume
"www-fs"
is not found

Finally, it gives up:

2008-10-01 22:37:59 D [tcp.c:87:tcp_disconnect] www-fs: connection
disconnected

I am at something of a loss as to how to proceed.  Is there a better
version to be running?  Did I miss something building/installing?  Is
the configuration botched?  I would very much appreciate it if someone
could help.  

SERVER 1, cedar

# dataspace local, on oak.  
volume wwwfs-ds
  type storage/posix
  option directory /var/wwwfs-ds
end-volume

# posix locks
volume wwwfs-ds-locks
  type features/posix-locks
  subvolumes wwwfs-ds
end-volume

# dataspace on storage2
volume wwwfs-oak-ds
  type protocol/client
  option transport-type tcp/client
  option remote-host  oak.storage.farrellit.net
  option remote-subvolume wwwfs-ds-locks
  option transport-timeout 10          
end-volume

# automatic file replication translator for dataspace
volume wwwfs-ds-afr
  type cluster/afr
  subvolumes wwwfs-ds-locks wwwfs-oak-ds         

# the actual exported volume
volume wwwfs
  type performance/io-threads
  option thread-count 8
  option cache-size 64MB
  subvolumes wwwfs-ds-afr
end-volume

# finally, the server declaration
volume server
  type protocol/server
  option transport-type tcp/server
  subvolumes wwwfs
  # storage network access only
  option auth.ip.wwwfs-ds-locks.allow 172.17.1.*,127.0.0.1
  option auth.ip.wwwfs.allow 172.17.1.*
end-volume

#####################

server 2, oak

# dataspace local, on oak.  
volume wwwfs-ds
  type storage/posix
  option directory /var/wwwfs-ds
end-volume

# posix locks
volume wwwfs-ds-locks
  type features/posix-locks
  subvolumes wwwfs-ds
end-volume

# dataspace on storage2
volume wwwfs-cedar-ds
  type protocol/client
  option transport-type tcp/client
  option remote-host cedar.storage.farrellit.net
  option remote-subvolume wwwfs-ds-locks
  option transport-timeout 10        
be set rel atively low
end-volume

# automatic file replication translator for dataspace
volume wwwfs-ds-afr
  type cluster/afr
  subvolumes wwwfs-ds-locks wwwfs-cedar-ds         # local and remote
dataspaces end-volume

# the actual exported volume
volume wwwfs
  type performance/io-threads
  option thread-count 8
  option cache-size 64MB
  subvolumes wwwfs-ds-afr
end-volume

# finally, the server declaration
volume server
  type protocol/server
  option transport-type tcp/server
  subvolumes wwwfs
  # storage network access only
  option auth.ip.wwwfs-ds-locks.allow 172.17.1.*,127.0.0.1
  option auth.ip.wwwfs.allow 172.17.1.*
end-volume


Logs available upon request.  Please, guide me!  

Thank you!  

Dan Farrell

At 08:39 PM 10/1/2008, Dan Farrell wrote:
>Friends,
>
>[...]
>
>other than that, things seem to  go well until I get the other server
>fired up, they talk fora a while, and then one serve2008-10-01 22:16:19
I''ve run into some serious problems with fuse 2.7.4  so first look
there.
it seems gluster and fuse 2.7.4 dont currently get along well.

2.7.3 is better, and the gluster modified 2.7.3 is better yet.

my configuration is similar to yours although i dont have a 
client,  the servers which AFR eachother are their own clients.  it 
works mostly ok, and it seems, aside form a few gluster bugs that are 
already known with AFR and I think are currently holding up 1.4 
release candidate, most of my issues have seemed to be fuse related.

Good luck.

At 08:39 PM 10/1/2008, Dan Farrell wrote:
>Friends,
>
>[...]
>
>other than that, things seem to  go well until I get the other server
>fired up, they talk fora a while, and then one serve2008-10-01 22:16:19
I''ve run into some serious problems with fuse 2.7.4  so first look
there.
it seems gluster and fuse 2.7.4 dont currently get along well.

2.7.3 is better, and the gluster modified 2.7.3 is better yet.

my configuration is similar to yours although i dont have a 
client,  the servers which AFR eachother are their own clients.  it 
works mostly ok, and it seems, aside form a few gluster bugs that are 
already known with AFR and I think are currently holding up 1.4 
release candidate, most of my issues have seemed to be fuse related.

Good luck.

At 08:39 PM 10/1/2008, Dan Farrell wrote:
>Friends,
>
>[...]
>
>other than that, things seem to  go well until I get the other server
>fired up, they talk fora a while, and then one serve2008-10-01 22:16:19
I've run into some serious problems with fuse 2.7.4  so first look there.
it seems gluster and fuse 2.7.4 dont currently get along well.

2.7.3 is better, and the gluster modified 2.7.3 is better yet.

my configuration is similar to yours although i dont have a 
client,  the servers which AFR eachother are their own clients.  it 
works mostly ok, and it seems, aside form a few gluster bugs that are 
already known with AFR and I think are currently holding up 1.4 
release candidate, most of my issues have seemed to be fuse related.

Good luck.

At 08:39 PM 10/1/2008, Dan Farrell wrote:
>Friends,
>
>[...]
>
>other than that, things seem to  go well until I get the other server
>fired up, they talk fora a while, and then one serve2008-10-01 22:16:19
I''ve run into some serious problems with fuse 2.7.4  so first look
there.
it seems gluster and fuse 2.7.4 dont currently get along well.

2.7.3 is better, and the gluster modified 2.7.3 is better yet.

my configuration is similar to yours although i dont have a 
client,  the servers which AFR eachother are their own clients.  it 
works mostly ok, and it seems, aside form a few gluster bugs that are 
already known with AFR and I think are currently holding up 1.4 
release candidate, most of my issues have seemed to be fuse related.

Good luck.