Mark Sullivan
2011-Nov-14 20:11 UTC
[Gluster-users] Cannot create a simple 2-brick volume.
Gluster Community,
I'm having a terrible time just trying to get started with gluster.
I'm running Centos 5.7 on a few nodes, and have installed gluster 3.2.4 and
its prereqs from RPMs.
Yet I'm finding it impossible to create a simple 2-brick distributed volume.
I keep seeing this error a lot:
reading from socket failed. Error (Transport endpoint is not connected)
referring to both the localhost and peers. There is no iptables running
on any of these machines, and all machines can ssh to each other
and report that their peers are connected.
I've googled this and other errors I've seen, and many results point
into this site, but none of the suggestions I've read have helped me.
The glusterfsd's are running. The peers are connected. I've done
multiple reboots and restarts of daemons. This is a fresh install.
Details are listed below.
Can someone please help me out?
Thanks!
-Mark Sullivan
Diviner Lunar Radiometer Experiment
===========================================================================================================================================================================================================================================================================================================================
On gluster03, creating a volume "glue" which is comprised of
gluster03:/g1 and gluster04:/g1
gluster volume create glue transport tcp gluster03:/g1 gluster04:/g1
gluster volume set glue auth.allow 10.*
gluster volume start glue
The "etc*" log files show this:
[2011-11-13 16:10:22.429786] I
[glusterd-handler.c:900:glusterd_handle_create_volume] 0-glusterd:
Received create volume req
[2011-11-13 16:10:22.430303] I [glusterd-utils.c:243:glusterd_lock]
0-glusterd: Cluster lock held by fb1f46cf-a03a-4fcd-b103-735040af3ced
[2011-11-13 16:10:22.430330] I
[glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired
local lock
[2011-11-13 16:10:22.430777] I
[glusterd-rpc-ops.c:752:glusterd3_1_cluster_lock_cbk] 0-glusterd:
Received ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:22.431182] I
[glusterd-op-sm.c:6543:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op
req to 1 peers
[2011-11-13 16:10:22.431814] I
[glusterd-rpc-ops.c:1050:glusterd3_1_stage_op_cbk] 0-glusterd: Received
ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:22.470773] I
[glusterd-op-sm.c:6660:glusterd_op_ac_send_commit_op] 0-glusterd: Sent
op req to 1 peers
[2011-11-13 16:10:22.489143] I
[glusterd-rpc-ops.c:1236:glusterd3_1_commit_op_cbk] 0-glusterd: Received
ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:22.489566] I
[glusterd-rpc-ops.c:811:glusterd3_1_cluster_unlock_cbk] 0-glusterd:
Received ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:22.489604] I
[glusterd-op-sm.c:7077:glusterd_op_txn_complete] 0-glusterd: Cleared
local lock
[2011-11-13 16:10:22.492971] W
[socket.c:1494:__socket_proto_state_machine] 0-socket.management:
reading from socket failed. Error (Transport endpoint is not connected),
peer (127.0.0.1:1023)
[2011-11-13 16:10:22.611682] I [glusterd-utils.c:243:glusterd_lock]
0-glusterd: Cluster lock held by fb1f46cf-a03a-4fcd-b103-735040af3ced
[2011-11-13 16:10:22.611709] I
[glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired
local lock
[2011-11-13 16:10:22.612096] I
[glusterd-rpc-ops.c:752:glusterd3_1_cluster_lock_cbk] 0-glusterd:
Received ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:22.896543] I
[glusterd-op-sm.c:6543:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op
req to 1 peers
[2011-11-13 16:10:23.55185] I
[glusterd-rpc-ops.c:1050:glusterd3_1_stage_op_cbk] 0-glusterd: Received
ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:23.64798] I
[glusterd-op-sm.c:6660:glusterd_op_ac_send_commit_op] 0-glusterd: Sent
op req to 1 peers
[2011-11-13 16:10:23.74209] I
[glusterd-rpc-ops.c:1236:glusterd3_1_commit_op_cbk] 0-glusterd: Received
ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:23.74527] I
[glusterd-rpc-ops.c:811:glusterd3_1_cluster_unlock_cbk] 0-glusterd:
Received ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:23.74558] I
[glusterd-op-sm.c:7077:glusterd_op_txn_complete] 0-glusterd: Cleared
local lock
[2011-11-13 16:10:23.79190] W
[socket.c:1494:__socket_proto_state_machine] 0-socket.management:
reading from socket failed. Error (Transport endpoint is not connected),
peer (127.0.0.1:1020)
[2011-11-13 16:10:23.198846] I
[glusterd-handler.c:1078:glusterd_handle_cli_start_volume] 0-glusterd:
Received start vol reqfor volume glue
[2011-11-13 16:10:23.198913] I [glusterd-utils.c:243:glusterd_lock]
0-glusterd: Cluster lock held by fb1f46cf-a03a-4fcd-b103-735040af3ced
[2011-11-13 16:10:23.198938] I
[glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired
local lock
[2011-11-13 16:10:23.199364] I
[glusterd-rpc-ops.c:752:glusterd3_1_cluster_lock_cbk] 0-glusterd:
Received ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:23.199819] I
[glusterd-op-sm.c:6543:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op
req to 1 peers
[2011-11-13 16:10:23.200396] I
[glusterd-rpc-ops.c:1050:glusterd3_1_stage_op_cbk] 0-glusterd: Received
ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:23.724138] I
[glusterd-utils.c:1095:glusterd_volume_start_glusterfs] 0-: About to
start glusterfs for brick gluster03:/g1
[2011-11-13 16:10:23.989454] I
[glusterd-op-sm.c:6660:glusterd_op_ac_send_commit_op] 0-glusterd: Sent
op req to 1 peers
[2011-11-13 16:10:24.7044] I [glusterd-pmap.c:237:pmap_registry_bind]
0-pmap: adding brick /g1 on port 24009
[2011-11-13 16:10:24.39658] W
[socket.c:1494:__socket_proto_state_machine] 0-socket.management:
reading from socket failed. Error (Transport endpoint is not connected),
peer (127.0.0.1:1017)
[2011-11-13 16:10:24.816411] I
[glusterd-rpc-ops.c:1236:glusterd3_1_commit_op_cbk] 0-glusterd: Received
ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:24.816940] I
[glusterd-rpc-ops.c:811:glusterd3_1_cluster_unlock_cbk] 0-glusterd:
Received ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
[2011-11-13 16:10:24.816993] I
[glusterd-op-sm.c:7077:glusterd_op_txn_complete] 0-glusterd: Cleared
local lock
[2011-11-13 16:10:24.818726] W
[socket.c:1494:__socket_proto_state_machine] 0-socket.management:
reading from socket failed. Error (Transport endpoint is not connected),
peer (127.0.0.1:1019)
[2011-11-13 16:10:24.859565] W
[socket.c:1494:__socket_proto_state_machine] 0-socket.management:
reading from socket failed. Error (Transport endpoint is not connected),
peer (10.1.1.24:1019)
=========================================================================================================
My volume info looks okay, I guess...
gluster volume info
Volume Name: glue
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: gluster03:/g1
Brick2: gluster04:/g1
Options Reconfigured:
auth.allow: 10.*
When I mount the volume "glue" on gluster03 using "mount -t nfs
gluster03:/glue /mnt", the nfs.log shows:
[2011-11-13 16:18:06.83447] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glue-client-0: remote
operation failed: Invalid argument
[2011-11-13 16:18:06.83507] I [dht-common.c:478:dht_revalidate_cbk]
0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument)
[2011-11-13 16:18:06.84676] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glue-client-0: remote
operation failed: Invalid argument
[2011-11-13 16:18:06.84704] I [dht-common.c:478:dht_revalidate_cbk]
0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument)
[2011-11-13 16:18:06.85687] W [rpc-common.c:64:xdr_to_generic]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
[0x2ae52ccad6fd]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
[0x2ae52ccad502]
(-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91)
[0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed
[2011-11-13 16:18:06.85723] E [client3_1-fops.c:398:client3_1_stat_cbk]
0-glue-client-0: error
[2011-11-13 16:18:06.85748] I [client3_1-fops.c:411:client3_1_stat_cbk]
0-glue-client-0: remote operation failed: Invalid argument
[2011-11-13 16:18:06.86273] W [rpc-common.c:64:xdr_to_generic]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
[0x2ae52ccad6fd]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
[0x2ae52ccad502]
(-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91)
[0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed
[2011-11-13 16:18:06.86301] E [client3_1-fops.c:398:client3_1_stat_cbk]
0-glue-client-0: error
[2011-11-13 16:18:06.86324] I [client3_1-fops.c:411:client3_1_stat_cbk]
0-glue-client-0: remote operation failed: Invalid argument
=========================================================================================================
When I do "touch /mnt/new", I get "No such file or
directory", and
nfs.log shows:
[2011-11-13 16:18:06.83447] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glue-client-0: remote
operation failed: Invalid argument
[2011-11-13 16:18:06.83507] I [dht-common.c:478:dht_revalidate_cbk]
0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument)
[2011-11-13 16:18:06.84676] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glue-client-0: remote
operation failed: Invalid argument
[2011-11-13 16:18:06.84704] I [dht-common.c:478:dht_revalidate_cbk]
0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument)
[2011-11-13 16:18:06.85687] W [rpc-common.c:64:xdr_to_generic]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
[0x2ae52ccad6fd]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
[0x2ae52ccad502]
(-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91)
[0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed
[2011-11-13 16:18:06.85723] E [client3_1-fops.c:398:client3_1_stat_cbk]
0-glue-client-0: error
[2011-11-13 16:18:06.85748] I [client3_1-fops.c:411:client3_1_stat_cbk]
0-glue-client-0: remote operation failed: Invalid argument
[2011-11-13 16:18:06.86273] W [rpc-common.c:64:xdr_to_generic]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
[0x2ae52ccad6fd]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
[0x2ae52ccad502]
(-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91)
[0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed
[2011-11-13 16:18:06.86301] E [client3_1-fops.c:398:client3_1_stat_cbk]
0-glue-client-0: error
[2011-11-13 16:18:06.86324] I [client3_1-fops.c:411:client3_1_stat_cbk]
0-glue-client-0: remote operation failed: Invalid argument
[2011-11-13 16:19:48.424842] I [dht-layout.c:192:dht_layout_search]
0-glue-dht: no subvolume for hash (value) = 1407928635
[2011-11-13 16:19:48.425129] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glue-client-0: remote
operation failed: Invalid argument
[2011-11-13 16:19:48.425751] I [dht-layout.c:192:dht_layout_search]
0-glue-dht: no subvolume for hash (value) = 1407928635
[2011-11-13 16:19:48.425991] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glue-client-0: remote
operation failed: Invalid argument
[2011-11-13 16:19:48.449516] I [dht-layout.c:192:dht_layout_search]
0-glue-dht: no subvolume for hash (value) = 1407928635
[2011-11-13 16:19:48.449662] E [fd.c:465:fd_unref]
(-->/opt/glusterfs/3.2.4/lib64/libglusterfs.so.0(default_create_cbk+0xb4)
[0x2ae52ca65cc4]
(-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/debug/io-stats.so(io_stats_create_cbk+0x20c)
[0x2aaaab76263c]
(-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/nfs/server.so(nfs_fop_create_cbk+0x73)
[0x2aaaab988a13]))) 0-fd: fd is NULL
[2011-11-13 16:19:48.449859] W [rpc-common.c:64:xdr_to_generic]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
[0x2ae52ccad6fd]
(-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
[0x2ae52ccad502]
(-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_statfs_cbk+0x7e)
[0x2aaaaaac806e]))) 0-xdr: XDR decoding failed
[2011-11-13 16:19:48.449888] E
[client3_1-fops.c:624:client3_1_statfs_cbk] 0-glue-client-0: error
[2011-11-13 16:19:48.449912] I
[client3_1-fops.c:637:client3_1_statfs_cbk] 0-glue-client-0: remote
operation failed: Invalid argument
[2011-11-13 16:19:48.450030] I [dht-layout.c:192:dht_layout_search]
0-glue-dht: no subvolume for hash (value) = 1407928635
[2011-11-13 16:19:48.450260] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glue-client-0: remote
operation failed: Invalid argument
=========================================================================================================
And from the brick log g1.log, in case this helps:
[2011-11-13 21:46:05.929654] I [glusterfsd.c:1493:main]
0-/opt/glusterfs/3.2.4/sbin/glusterfsd: Started Running
/opt/glusterfs/3.2.4/sbin/glusterfsd version 3.2.4
[2011-11-13 21:46:05.946509] W [socket.c:419:__socket_keepalive]
0-socket: failed to set keep idle on socket 8
[2011-11-13 21:46:05.946618] W
[socket.c:1846:socket_server_event_handler] 0-socket.glusterfsd: Failed
to set keep-alive: Operation not supported
[2011-11-13 21:46:06.72770] W [graph.c:291:gf_add_cmdline_options]
0-glue-server: adding option 'listen-port' for volume
'glue-server' with
value '24010'
[2011-11-13 21:46:06.73873] W
[rpc-transport.c:447:validate_volume_options] 0-tcp.glue-server: option
'listen-port' is deprecated, preferred is
'transport.socket.listen-port', continuing with correction
[2011-11-13 21:46:06.74204] W [posix.c:4686:init] 0-glue-posix: Posix
access control list is not supported.
Given volfile:
+------------------------------------------------------------------------------+
1: volume glue-posix
2: type storage/posix
3: option directory /g1
4: end-volume
5:
6: volume glue-access-control
7: type features/access-control
8: subvolumes glue-posix
9: end-volume
10:
11: volume glue-locks
12: type features/locks
13: subvolumes glue-access-control
14: end-volume
15:
16: volume glue-io-threads
17: type performance/io-threads
18: subvolumes glue-locks
19: end-volume
20:
21: volume glue-marker
22: type features/marker
23: option volume-uuid 2b567c80-ab30-44b2-9b17-e67e6e679096
24: option timestamp-file /etc/glusterd/vols/glue/marker.tstamp
25: option xtime off
26: option quota off
27: subvolumes glue-io-threads
28: end-volume
29:
30: volume /g1
31: type debug/io-stats
32: option latency-measurement off
33: option count-fop-hits off
34: subvolumes glue-marker
35: end-volume
36:
37: volume glue-server
38: type protocol/server
39: option transport-type tcp
40: option auth.addr./g1.allow 10.*
41: subvolumes /g1
42: end-volume
+------------------------------------------------------------------------------+
[2011-11-13 21:46:09.133670] E [authenticate.c:227:gf_authenticate]
0-auth: no authentication module is interested in accepting
remote-client (null)
[2011-11-13 21:46:09.133729] E [server-handshake.c:553:server_setvolume]
0-glue-server: Cannot authenticate client from 127.0.0.1:1023 3.2.4
[2011-11-13 21:46:09.389447] I [server-handshake.c:542:server_setvolume]
0-glue-server: accepted client from 10.1.1.24:1022 (version: 3.2.4)
Mohit Anchlia
2011-Nov-14 20:19 UTC
[Gluster-users] Cannot create a simple 2-brick volume.
[2011-11-13 21:46:09.133729] E [server-handshake.c:553:server_setvolume] 0-glue-server: Cannot authenticate client from 127.0.0.1:1023 3.2.4 Can you remove the access 10.* and see if it works first? On Mon, Nov 14, 2011 at 12:11 PM, Mark Sullivan <marks at mars.ucla.edu> wrote:> Gluster Community, > > I'm having a terrible time just trying to get started with gluster. > I'm running Centos 5.7 on a few nodes, and have installed gluster 3.2.4 and > its prereqs from RPMs. > Yet I'm finding it impossible to create a simple 2-brick distributed volume. > I keep seeing this error a lot: > > ? reading from socket failed. Error (Transport endpoint is not connected) > > referring to both the localhost and peers. ? There is no iptables running > on any of these machines, and all machines can ssh to each other > and report that their peers are connected. > > I've googled this and other errors I've seen, and many results point > into this site, but none of the suggestions I've read have helped me. > The glusterfsd's are running. ?The peers are connected. ?I've done > multiple reboots and restarts of daemons. ? This is a fresh install. > > Details are listed below. > Can someone please help me out? > Thanks! > -Mark Sullivan > Diviner Lunar Radiometer Experiment > > =========================================================================================================> =========================================================================================================> =========================================================================================================> > On gluster03, creating a volume "glue" which is comprised of > gluster03:/g1 and gluster04:/g1 > > gluster volume create glue transport tcp gluster03:/g1 gluster04:/g1 > gluster volume set glue auth.allow 10.* > gluster volume start glue > > The "etc*" log files show this: > > [2011-11-13 16:10:22.429786] I > [glusterd-handler.c:900:glusterd_handle_create_volume] 0-glusterd: Received > create volume req > [2011-11-13 16:10:22.430303] I [glusterd-utils.c:243:glusterd_lock] > 0-glusterd: Cluster lock held by fb1f46cf-a03a-4fcd-b103-735040af3ced > [2011-11-13 16:10:22.430330] I > [glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired local > lock > [2011-11-13 16:10:22.430777] I > [glusterd-rpc-ops.c:752:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received > ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:22.431182] I > [glusterd-op-sm.c:6543:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req > to 1 peers > [2011-11-13 16:10:22.431814] I > [glusterd-rpc-ops.c:1050:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC > from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:22.470773] I > [glusterd-op-sm.c:6660:glusterd_op_ac_send_commit_op] 0-glusterd: Sent op > req to 1 peers > [2011-11-13 16:10:22.489143] I > [glusterd-rpc-ops.c:1236:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC > from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:22.489566] I > [glusterd-rpc-ops.c:811:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received > ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:22.489604] I > [glusterd-op-sm.c:7077:glusterd_op_txn_complete] 0-glusterd: Cleared local > lock > [2011-11-13 16:10:22.492971] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (127.0.0.1:1023) > [2011-11-13 16:10:22.611682] I [glusterd-utils.c:243:glusterd_lock] > 0-glusterd: Cluster lock held by fb1f46cf-a03a-4fcd-b103-735040af3ced > [2011-11-13 16:10:22.611709] I > [glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired local > lock > [2011-11-13 16:10:22.612096] I > [glusterd-rpc-ops.c:752:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received > ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:22.896543] I > [glusterd-op-sm.c:6543:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req > to 1 peers > [2011-11-13 16:10:23.55185] I > [glusterd-rpc-ops.c:1050:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC > from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:23.64798] I > [glusterd-op-sm.c:6660:glusterd_op_ac_send_commit_op] 0-glusterd: Sent op > req to 1 peers > [2011-11-13 16:10:23.74209] I > [glusterd-rpc-ops.c:1236:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC > from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:23.74527] I > [glusterd-rpc-ops.c:811:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received > ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:23.74558] I > [glusterd-op-sm.c:7077:glusterd_op_txn_complete] 0-glusterd: Cleared local > lock > [2011-11-13 16:10:23.79190] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (127.0.0.1:1020) > [2011-11-13 16:10:23.198846] I > [glusterd-handler.c:1078:glusterd_handle_cli_start_volume] 0-glusterd: > Received start vol reqfor volume glue > [2011-11-13 16:10:23.198913] I [glusterd-utils.c:243:glusterd_lock] > 0-glusterd: Cluster lock held by fb1f46cf-a03a-4fcd-b103-735040af3ced > [2011-11-13 16:10:23.198938] I > [glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired local > lock > [2011-11-13 16:10:23.199364] I > [glusterd-rpc-ops.c:752:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received > ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:23.199819] I > [glusterd-op-sm.c:6543:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req > to 1 peers > [2011-11-13 16:10:23.200396] I > [glusterd-rpc-ops.c:1050:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC > from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:23.724138] I > [glusterd-utils.c:1095:glusterd_volume_start_glusterfs] 0-: About to start > glusterfs for brick gluster03:/g1 > [2011-11-13 16:10:23.989454] I > [glusterd-op-sm.c:6660:glusterd_op_ac_send_commit_op] 0-glusterd: Sent op > req to 1 peers > [2011-11-13 16:10:24.7044] I [glusterd-pmap.c:237:pmap_registry_bind] > 0-pmap: adding brick /g1 on port 24009 > [2011-11-13 16:10:24.39658] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (127.0.0.1:1017) > [2011-11-13 16:10:24.816411] I > [glusterd-rpc-ops.c:1236:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC > from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:24.816940] I > [glusterd-rpc-ops.c:811:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received > ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d > [2011-11-13 16:10:24.816993] I > [glusterd-op-sm.c:7077:glusterd_op_txn_complete] 0-glusterd: Cleared local > lock > [2011-11-13 16:10:24.818726] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (127.0.0.1:1019) > [2011-11-13 16:10:24.859565] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (10.1.1.24:1019) > > =========================================================================================================> > My volume info looks okay, I guess... > > gluster volume info > > Volume Name: glue > Type: Distribute > Status: Started > Number of Bricks: 2 > Transport-type: tcp > Bricks: > Brick1: gluster03:/g1 > Brick2: gluster04:/g1 > Options Reconfigured: > auth.allow: 10.* > > When I mount the volume "glue" on gluster03 using "mount -t nfs > gluster03:/glue /mnt", the nfs.log shows: > > [2011-11-13 16:18:06.83447] I [client3_1-fops.c:2228:client3_1_lookup_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:18:06.83507] I [dht-common.c:478:dht_revalidate_cbk] > 0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument) > [2011-11-13 16:18:06.84676] I [client3_1-fops.c:2228:client3_1_lookup_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:18:06.84704] I [dht-common.c:478:dht_revalidate_cbk] > 0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument) > [2011-11-13 16:18:06.85687] W [rpc-common.c:64:xdr_to_generic] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d) > [0x2ae52ccad6fd] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) > [0x2ae52ccad502] > (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91) > [0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed > [2011-11-13 16:18:06.85723] E [client3_1-fops.c:398:client3_1_stat_cbk] > 0-glue-client-0: error > [2011-11-13 16:18:06.85748] I [client3_1-fops.c:411:client3_1_stat_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:18:06.86273] W [rpc-common.c:64:xdr_to_generic] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d) > [0x2ae52ccad6fd] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) > [0x2ae52ccad502] > (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91) > [0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed > [2011-11-13 16:18:06.86301] E [client3_1-fops.c:398:client3_1_stat_cbk] > 0-glue-client-0: error > [2011-11-13 16:18:06.86324] I [client3_1-fops.c:411:client3_1_stat_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > > =========================================================================================================> > When I do "touch /mnt/new", I get "No such file or directory", and nfs.log > shows: > > [2011-11-13 16:18:06.83447] I [client3_1-fops.c:2228:client3_1_lookup_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:18:06.83507] I [dht-common.c:478:dht_revalidate_cbk] > 0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument) > [2011-11-13 16:18:06.84676] I [client3_1-fops.c:2228:client3_1_lookup_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:18:06.84704] I [dht-common.c:478:dht_revalidate_cbk] > 0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument) > [2011-11-13 16:18:06.85687] W [rpc-common.c:64:xdr_to_generic] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d) > [0x2ae52ccad6fd] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) > [0x2ae52ccad502] > (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91) > [0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed > [2011-11-13 16:18:06.85723] E [client3_1-fops.c:398:client3_1_stat_cbk] > 0-glue-client-0: error > [2011-11-13 16:18:06.85748] I [client3_1-fops.c:411:client3_1_stat_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:18:06.86273] W [rpc-common.c:64:xdr_to_generic] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d) > [0x2ae52ccad6fd] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) > [0x2ae52ccad502] > (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91) > [0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed > [2011-11-13 16:18:06.86301] E [client3_1-fops.c:398:client3_1_stat_cbk] > 0-glue-client-0: error > [2011-11-13 16:18:06.86324] I [client3_1-fops.c:411:client3_1_stat_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:19:48.424842] I [dht-layout.c:192:dht_layout_search] > 0-glue-dht: no subvolume for hash (value) = 1407928635 > [2011-11-13 16:19:48.425129] I [client3_1-fops.c:2228:client3_1_lookup_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:19:48.425751] I [dht-layout.c:192:dht_layout_search] > 0-glue-dht: no subvolume for hash (value) = 1407928635 > [2011-11-13 16:19:48.425991] I [client3_1-fops.c:2228:client3_1_lookup_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:19:48.449516] I [dht-layout.c:192:dht_layout_search] > 0-glue-dht: no subvolume for hash (value) = 1407928635 > [2011-11-13 16:19:48.449662] E [fd.c:465:fd_unref] > (-->/opt/glusterfs/3.2.4/lib64/libglusterfs.so.0(default_create_cbk+0xb4) > [0x2ae52ca65cc4] > (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/debug/io-stats.so(io_stats_create_cbk+0x20c) > [0x2aaaab76263c] > (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/nfs/server.so(nfs_fop_create_cbk+0x73) > [0x2aaaab988a13]))) 0-fd: fd is NULL > [2011-11-13 16:19:48.449859] W [rpc-common.c:64:xdr_to_generic] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d) > [0x2ae52ccad6fd] > (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) > [0x2ae52ccad502] > (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_statfs_cbk+0x7e) > [0x2aaaaaac806e]))) 0-xdr: XDR decoding failed > [2011-11-13 16:19:48.449888] E [client3_1-fops.c:624:client3_1_statfs_cbk] > 0-glue-client-0: error > [2011-11-13 16:19:48.449912] I [client3_1-fops.c:637:client3_1_statfs_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > [2011-11-13 16:19:48.450030] I [dht-layout.c:192:dht_layout_search] > 0-glue-dht: no subvolume for hash (value) = 1407928635 > [2011-11-13 16:19:48.450260] I [client3_1-fops.c:2228:client3_1_lookup_cbk] > 0-glue-client-0: remote operation failed: Invalid argument > > =========================================================================================================> > And from the brick log g1.log, in case this helps: > > [2011-11-13 21:46:05.929654] I [glusterfsd.c:1493:main] > 0-/opt/glusterfs/3.2.4/sbin/glusterfsd: Started Running > /opt/glusterfs/3.2.4/sbin/glusterfsd version 3.2.4 > [2011-11-13 21:46:05.946509] W [socket.c:419:__socket_keepalive] 0-socket: > failed to set keep idle on socket 8 > [2011-11-13 21:46:05.946618] W [socket.c:1846:socket_server_event_handler] > 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported > [2011-11-13 21:46:06.72770] W [graph.c:291:gf_add_cmdline_options] > 0-glue-server: adding option 'listen-port' for volume 'glue-server' with > value '24010' > [2011-11-13 21:46:06.73873] W [rpc-transport.c:447:validate_volume_options] > 0-tcp.glue-server: option 'listen-port' is deprecated, preferred is > 'transport.socket.listen-port', continuing with correction > [2011-11-13 21:46:06.74204] W [posix.c:4686:init] 0-glue-posix: Posix access > control list is not supported. > Given volfile: > +------------------------------------------------------------------------------+ > ?1: volume glue-posix > ?2: ? ? type storage/posix > ?3: ? ? option directory /g1 > ?4: end-volume > ?5: > ?6: volume glue-access-control > ?7: ? ? type features/access-control > ?8: ? ? subvolumes glue-posix > ?9: end-volume > 10: > 11: volume glue-locks > 12: ? ? type features/locks > 13: ? ? subvolumes glue-access-control > 14: end-volume > 15: > 16: volume glue-io-threads > 17: ? ? type performance/io-threads > 18: ? ? subvolumes glue-locks > 19: end-volume > 20: > 21: volume glue-marker > 22: ? ? type features/marker > 23: ? ? option volume-uuid 2b567c80-ab30-44b2-9b17-e67e6e679096 > 24: ? ? option timestamp-file /etc/glusterd/vols/glue/marker.tstamp > 25: ? ? option xtime off > 26: ? ? option quota off > 27: ? ? subvolumes glue-io-threads > 28: end-volume > 29: > 30: volume /g1 > 31: ? ? type debug/io-stats > 32: ? ? option latency-measurement off > 33: ? ? option count-fop-hits off > 34: ? ? subvolumes glue-marker > 35: end-volume > 36: > 37: volume glue-server > 38: ? ? type protocol/server > 39: ? ? option transport-type tcp > 40: ? ? option auth.addr./g1.allow 10.* > 41: ? ? subvolumes /g1 > 42: end-volume > > +------------------------------------------------------------------------------+ > [2011-11-13 21:46:09.133670] E [authenticate.c:227:gf_authenticate] 0-auth: > no authentication module is interested in accepting remote-client (null) > [2011-11-13 21:46:09.133729] E [server-handshake.c:553:server_setvolume] > 0-glue-server: Cannot authenticate client from 127.0.0.1:1023 3.2.4 > [2011-11-13 21:46:09.389447] I [server-handshake.c:542:server_setvolume] > 0-glue-server: accepted client from 10.1.1.24:1022 (version: 3.2.4) > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >