Pat Haley
2014-Mar-18 16:07 UTC
[Gluster-users] Unable to access directories in wounded gluster setup
Hi,
We are employing gluster to merge the disks from
3 servers into a common name-space. We just had
a power outage, and when the power came back on,
one of the servers had it's power supply damaged.
While working on this problem, we tried to get the
gluster area (/gdata) up with the the 2 working
servers, so at least the files on them would be
available while we repair the other server. This
hasn't worked as expected. From past experience
we expected to be able to see all the subdirectories
with some files simply absent. What we see instead
is that 4 of the top-level directories are unaccessible
from client:
ls /gdata
ls: cannot access /gdata/test: Invalid argument
ls: cannot access /gdata/projects: Invalid argument
ls: cannot access /gdata/harvard-data2: Invalid argument
ls: cannot access /gdata/temp_home: Invalid argument
drwxr-xr-x 6 root root 152 Dec 5 12:27 harvard
drwxr-xr-x 4 root root 74 Dec 5 12:27 harvard-archive
?????????? ? ? ? ? ? harvard-data2
drwxr-xr-x 104 pierrel 8310 16K Dec 5 12:27 harvard-data3
drwx------ 2 root root 12 Dec 5 12:27 lost+found
?????????? ? ? ? ? ? projects
drwxrwxr-x 8 root software 4.1K Dec 5 12:27 software
drwxr-xr-x 5 root root 4.1K Dec 5 12:27 src
?????????? ? ? ? ? ? temp_home
?????????? ? ? ? ? ? test
We have tried
- testing the communications between the bricks and
between the client & bricks (comms fine)
- restarting the gluster daemons on the bricks
- unmounting and remounting /gdata on the client
- mounting the gluster area directly on one of the
servers using mount -t glusterfs mseas-data:/gdata /gdata
All give the same result. The status reported by gluster
looks fine (given that 1 brick is down). The reports are
included below. The log files from the mounting/ls tests are
included after that.
What should we do/look at next to solve/debug this problem?
Thanks.
Pat
---------------------------------------------------------------------
[root at mseas-data glusterfs]# gluster --version
glusterfs 3.3.1 built on Oct 11 2012 22:01:05
---------------------------------------------------------------------
[root at mseas-data bricks]# gluster volume info
Volume Name: gdata
Type: Distribute
Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: gluster-0-0:/mseas-data-0-0
Brick2: gluster-0-1:/mseas-data-0-1
Brick3: gluster-data:/data
---------------------------------------------------------------------
[root at mseas-data bricks]# gluster volume status
Status of volume: gdata
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick gluster-0-0:/mseas-data-0-0 24010 Y
11346
Brick gluster-data:/data 24010 Y
28791
NFS Server on localhost 38467 Y
28797
NFS Server on gluster-0-0 38467 Y
11352
---------------------------------------------------------------------
[root at mseas-data bricks]# gluster peer status
Number of Peers: 2
Hostname: gluster-0-1
Uuid: 978e0f76-6474-4203-8617-ed5ad7d29239
State: Peer in Cluster (Disconnected)
Hostname: gluster-0-0
Uuid: 3f73f5cc-39d8-4d9a-b442-033cb074b247
State: Peer in Cluster (Connected)
---------------------------------------------------------------------
============================================================Log files from test
mount from client:
--------------------------------------
gluster-data: /var/log/glusterfs/bricks/data.log
-------------------------------------------------
[2014-03-18 10:43:57.149035] I [server.c:703:server_rpc_notify]
0-gdata-server: disconnecting connectionfrom
compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0
[2014-03-18 10:43:57.149073] I
[server-helpers.c:741:server_connection_put] 0-gdata-server: Shutting
down connection
compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0
[2014-03-18 10:43:57.149127] I
[server-helpers.c:629:server_connection_destroy] 0-gdata-server:
destroyed connection of
compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0
[2014-03-18 10:44:10.149539] I [server-handshake.c:571:server_setvolume]
0-gdata-server: accepted client from
compute-3-0.local-15405-2014/03/18-10:44:06:129552-gdata-client-2-0
(version: 3.3.1)
gluster-0-0: /var/log/glusterfs/bricks/data.log
-----------------------------------------------
[2014-03-18 10:43:57.141122] I [server.c:703:server_rpc_notify]
0-gdata-server: disconnecting connectionfrom
compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-0-0
[2014-03-18 10:43:57.141209] I
[server-helpers.c:741:server_connection_put] 0-gdata-server: Shutting
down connection
compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-0-0
[2014-03-18 10:43:57.141299] I
[server-helpers.c:629:server_connection_destroy] 0-gdata-server:
destroyed connection of
compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-0-0
[2014-03-18 10:44:10.143643] I [server-handshake.c:571:server_setvolume]
0-gdata-server: accepted client from
compute-3-0.local-15405-2014/03/18-10:44:06:129552-gdata-client-0-0
(version: 3.3.1)
client: /var/log/glusterfs/gdata.log
-------------------------------------
[2014-03-18 10:43:57.139136] I [fuse-bridge.c:4091:fuse_thread_proc]
0-fuse: unmounting /gdata
[2014-03-18 10:43:57.139423] W [glusterfsd.c:831:cleanup_and_exit]
(-->/lib64/libc.so.6(clone+0x6d) [0x3e6a8e5ccd]
(-->/lib64/libpthread.so.0() [0x3e6b0077f1]
(-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d4d]))) 0-:
received signum (15), shutting down
[2014-03-18 10:43:57.139443] I [fuse-bridge.c:4648:fini] 0-fuse:
Unmounting '/gdata'.
[2014-03-18 10:44:06.132964] I [glusterfsd.c:1666:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
[2014-03-18 10:44:06.139237] I [io-cache.c:1549:check_cache_size_ok]
0-gdata-quick-read: Max cache size is 66901987328
[2014-03-18 10:44:06.139290] I [io-cache.c:1549:check_cache_size_ok]
0-gdata-io-cache: Max cache size is 66901987328
[2014-03-18 10:44:06.141718] I [client.c:2142:notify] 0-gdata-client-0:
parent translators are ready, attempting connect on transport
[2014-03-18 10:44:06.144062] I [client.c:2142:notify] 0-gdata-client-1:
parent translators are ready, attempting connect on transport
[2014-03-18 10:44:06.146039] I [client.c:2142:notify] 0-gdata-client-2:
parent translators are ready, attempting connect on transport
Given volfile:
+------------------------------------------------------------------------------+
1: volume gdata-client-0
2: type protocol/client
3: option remote-host gluster-0-0
4: option remote-subvolume /mseas-data-0-0
5: option transport-type tcp
6: end-volume
7:
8: volume gdata-client-1
9: type protocol/client
10: option remote-host gluster-0-1
11: option remote-subvolume /mseas-data-0-1
12: option transport-type tcp
13: end-volume
14:
15: volume gdata-client-2
16: type protocol/client
17: option remote-host gluster-data
18: option remote-subvolume /data
19: option transport-type tcp
20: end-volume
21:
22: volume gdata-dht
23: type cluster/distribute
24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2
25: end-volume
26:
27: volume gdata-write-behind
28: type performance/write-behind
29: subvolumes gdata-dht
30: end-volume
31:
32: volume gdata-read-ahead
33: type performance/read-ahead
34: subvolumes gdata-write-behind
35: end-volume
36:
37: volume gdata-io-cache
38: type performance/io-cache
39: subvolumes gdata-read-ahead
40: end-volume
41:
42: volume gdata-quick-read
43: type performance/quick-read
44: subvolumes gdata-io-cache
45: end-volume
46:
47: volume gdata-md-cache
48: type performance/md-cache
49: subvolumes gdata-quick-read
50: end-volume
51:
52: volume gdata
53: type debug/io-stats
54: option latency-measurement off
55: option count-fop-hits off
56: subvolumes gdata-md-cache
57: end-volume
+------------------------------------------------------------------------------+
[2014-03-18 10:44:06.148454] I [rpc-clnt.c:1657:rpc_clnt_reconfig]
0-gdata-client-2: changing port to 24010 (from 0)
[2014-03-18 10:44:06.148529] I [rpc-clnt.c:1657:rpc_clnt_reconfig]
0-gdata-client-0: changing port to 24010 (from 0)
[2014-03-18 10:44:09.146764] E [socket.c:1715:socket_connect_finish]
0-gdata-client-1: connection to failed (No route to host)
[2014-03-18 10:44:10.141499] I
[client-handshake.c:1636:select_server_supported_programs]
0-gdata-client-2: Using Program GlusterFS 3.3.1, Num (1298437), Version
(330)
[2014-03-18 10:44:10.141748] I
[client-handshake.c:1433:client_setvolume_cbk] 0-gdata-client-2:
Connected to 10.1.1.2:24010, attached to remote volume '/data'.
[2014-03-18 10:44:10.141759] I
[client-handshake.c:1445:client_setvolume_cbk] 0-gdata-client-2: Server
and Client lk-version numbers are not same, reopening the fds
[2014-03-18 10:44:10.141866] I
[client-handshake.c:453:client_set_lk_version_cbk] 0-gdata-client-2:
Server lk version = 1
[2014-03-18 10:44:10.143526] I
[client-handshake.c:1636:select_server_supported_programs]
0-gdata-client-0: Using Program GlusterFS 3.3.1, Num (1298437), Version
(330)
[2014-03-18 10:44:10.143800] I
[client-handshake.c:1433:client_setvolume_cbk] 0-gdata-client-0:
Connected to 10.1.1.10:24010, attached to remote volume
'/mseas-data-0-0'.
[2014-03-18 10:44:10.143810] I
[client-handshake.c:1445:client_setvolume_cbk] 0-gdata-client-0: Server
and Client lk-version numbers are not same, reopening the fds
[2014-03-18 10:44:10.147967] I [fuse-bridge.c:4191:fuse_graph_setup]
0-fuse: switched to graph 0
[2014-03-18 10:44:10.148147] I
[client-handshake.c:453:client_set_lk_version_cbk] 0-gdata-client-0:
Server lk version = 1
[2014-03-18 10:44:10.148251] I [fuse-bridge.c:3376:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13
kernel 7.13
[2014-03-18 10:44:10.148993] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /. holes=1 overlaps=0
[2014-03-18 10:44:10.149012] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.335068] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 1072520554
[2014-03-18 10:48:34.335110] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /test
[2014-03-18 10:48:34.335131] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 7: LOOKUP() /test => -1 (Invalid argument)
[2014-03-18 10:48:34.336648] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 1092090275
[2014-03-18 10:48:34.336660] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /projects
[2014-03-18 10:48:34.336670] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 16: LOOKUP() /projects => -1 (Invalid argument)
[2014-03-18 10:48:34.337271] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /src. holes=1 overlaps=0
[2014-03-18 10:48:34.337310] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.337400] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 485417449
[2014-03-18 10:48:34.337414] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /harvard-data2
[2014-03-18 10:48:34.337424] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 18: LOOKUP() /harvard-data2 => -1 (Invalid argument)
[2014-03-18 10:48:34.337988] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /harvard. holes=1 overlaps=0
[2014-03-18 10:48:34.338003] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.338564] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /harvard-archive. holes=1 overlaps=0
[2014-03-18 10:48:34.338576] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.339173] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /harvard-data3. holes=1 overlaps=0
[2014-03-18 10:48:34.339184] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.339740] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2014-03-18 10:48:34.339751] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.340290] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /software. holes=1 overlaps=0
[2014-03-18 10:48:34.340302] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.340872] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /tmp. holes=1 overlaps=0
[2014-03-18 10:48:34.340883] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.341463] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /z.rdiff-backup-data. holes=1 overlaps=0
[2014-03-18 10:48:34.341474] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.341561] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 124395703
[2014-03-18 10:48:34.341576] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /temp_home
[2014-03-18 10:48:34.341586] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 26: LOOKUP() /temp_home => -1 (Invalid argument)
[2014-03-18 10:48:34.342137] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /data. holes=1 overlaps=0
[2014-03-18 10:48:34.342152] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 10:48:34.342737] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /bibliography. holes=1 overlaps=0
[2014-03-18 10:48:34.342748] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[root at compute-3-0 glusterfs]#
============================================================Log file from test
mount on gluster-data:
-----------------------------------------
/var/log/glusterfs/gdata.log
-----------------------------
[2014-03-18 11:29:23.549718] I [glusterfsd.c:1666:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
[2014-03-18 11:29:23.751872] I [io-cache.c:1549:check_cache_size_ok]
0-gdata-quick-read: Max cache size is 8374652928
[2014-03-18 11:29:23.767321] I [io-cache.c:1549:check_cache_size_ok]
0-gdata-io-cache: Max cache size is 8374652928
[2014-03-18 11:29:23.770541] I [client.c:2142:notify] 0-gdata-client-0:
parent translators are ready, attempting connect on transport
[2014-03-18 11:29:23.773247] I [client.c:2142:notify] 0-gdata-client-1:
parent translators are ready, attempting connect on transport
[2014-03-18 11:29:23.775734] I [client.c:2142:notify] 0-gdata-client-2:
parent translators are ready, attempting connect on transport
Given volfile:
+------------------------------------------------------------------------------+
1: volume gdata-client-0
2: type protocol/client
3: option remote-host gluster-0-0
4: option remote-subvolume /mseas-data-0-0
5: option transport-type tcp
6: end-volume
7:
8: volume gdata-client-1
9: type protocol/client
10: option remote-host gluster-0-1
11: option remote-subvolume /mseas-data-0-1
12: option transport-type tcp
13: end-volume
14:
15: volume gdata-client-2
16: type protocol/client
17: option remote-host gluster-data
18: option remote-subvolume /data
19: option transport-type tcp
20: end-volume
21:
22: volume gdata-dht
23: type cluster/distribute
24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2
25: end-volume
26:
27: volume gdata-write-behind
28: type performance/write-behind
29: subvolumes gdata-dht
30: end-volume
31:
32: volume gdata-read-ahead
33: type performance/read-ahead
34: subvolumes gdata-write-behind
35: end-volume
36:
37: volume gdata-io-cache
38: type performance/io-cache
39: subvolumes gdata-read-ahead
40: end-volume
41:
42: volume gdata-quick-read
43: type performance/quick-read
44: subvolumes gdata-io-cache
45: end-volume
46:
47: volume gdata-md-cache
48: type performance/md-cache
49: subvolumes gdata-quick-read
50: end-volume
51:
52: volume gdata
53: type debug/io-stats
54: option latency-measurement off
55: option count-fop-hits off
56: subvolumes gdata-md-cache
57: end-volume
+------------------------------------------------------------------------------+
[2014-03-18 11:29:23.778543] I [rpc-clnt.c:1657:rpc_clnt_reconfig]
0-gdata-client-2: changing port to 24010 (from 0)
[2014-03-18 11:29:23.778755] I [rpc-clnt.c:1657:rpc_clnt_reconfig]
0-gdata-client-0: changing port to 24010 (from 0)
[2014-03-18 11:29:23.981081] E [socket.c:1715:socket_connect_finish]
0-gdata-client-1: connection to failed (No route to host)
[2014-03-18 11:29:27.736927] I
[client-handshake.c:1636:select_server_supported_programs]
0-gdata-client-2: Using Program GlusterFS 3.3.1, Num (1298437), Version
(330)
[2014-03-18 11:29:27.737135] I
[client-handshake.c:1433:client_setvolume_cbk] 0-gdata-client-2:
Connected to 10.1.1.2:24010, attached to remote volume '/data'.
[2014-03-18 11:29:27.737152] I
[client-handshake.c:1445:client_setvolume_cbk] 0-gdata-client-2: Server
and Client lk-version numbers are not same, reopening the fds
[2014-03-18 11:29:27.737242] I
[client-handshake.c:453:client_set_lk_version_cbk] 0-gdata-client-2:
Server lk version = 1
[2014-03-18 11:29:27.739760] I
[client-handshake.c:1636:select_server_supported_programs]
0-gdata-client-0: Using Program GlusterFS 3.3.1, Num (1298437), Version
(330)
[2014-03-18 11:29:27.740067] I
[client-handshake.c:1433:client_setvolume_cbk] 0-gdata-client-0:
Connected to 10.1.1.10:24010, attached to remote volume
'/mseas-data-0-0'.
[2014-03-18 11:29:27.740081] I
[client-handshake.c:1445:client_setvolume_cbk] 0-gdata-client-0: Server
and Client lk-version numbers are not same, reopening the fds
[2014-03-18 11:29:27.747148] I [fuse-bridge.c:4191:fuse_graph_setup]
0-fuse: switched to graph 0
[2014-03-18 11:29:27.747322] I
[client-handshake.c:453:client_set_lk_version_cbk] 0-gdata-client-0:
Server lk version = 1
[2014-03-18 11:29:27.747458] I [fuse-bridge.c:3376:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13
kernel 7.8
[2014-03-18 11:29:27.748446] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /. holes=1 overlaps=0
[2014-03-18 11:29:27.748467] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.543344] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 1072520554
[2014-03-18 11:30:27.543374] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /test
[2014-03-18 11:30:27.543394] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 7: LOOKUP() /test => -1 (Invalid argument)
[2014-03-18 11:30:27.544817] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 1092090275
[2014-03-18 11:30:27.544832] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /projects
[2014-03-18 11:30:27.544846] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 12: LOOKUP() /projects => -1 (Invalid argument)
[2014-03-18 11:30:27.545336] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /src. holes=1 overlaps=0
[2014-03-18 11:30:27.545356] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.545476] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 485417449
[2014-03-18 11:30:27.545491] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /harvard-data2
[2014-03-18 11:30:27.545505] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 14: LOOKUP() /harvard-data2 => -1 (Invalid argument)
[2014-03-18 11:30:27.545995] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /harvard. holes=1 overlaps=0
[2014-03-18 11:30:27.546010] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.546600] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /harvard-archive. holes=1 overlaps=0
[2014-03-18 11:30:27.546616] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.547206] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /harvard-data3. holes=1 overlaps=0
[2014-03-18 11:30:27.547222] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.547901] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2014-03-18 11:30:27.547917] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.548423] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /software. holes=1 overlaps=0
[2014-03-18 11:30:27.548439] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.548981] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /tmp. holes=1 overlaps=0
[2014-03-18 11:30:27.548997] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.549500] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /z.rdiff-backup-data. holes=1 overlaps=0
[2014-03-18 11:30:27.549515] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.549629] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 124395703
[2014-03-18 11:30:27.549644] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /temp_home
[2014-03-18 11:30:27.549658] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 22: LOOKUP() /temp_home => -1 (Invalid argument)
[2014-03-18 11:30:27.550134] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /data. holes=1 overlaps=0
[2014-03-18 11:30:27.550149] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:30:27.550793] I [dht-layout.c:593:dht_layout_normalize]
0-gdata-dht: found anomalies in /bibliography. holes=1 overlaps=0
[2014-03-18 11:30:27.550830] W
[dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes
down -- not fixing
[2014-03-18 11:32:00.358667] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 1072520554
[2014-03-18 11:32:00.358700] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /test
[2014-03-18 11:32:00.358717] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 35: LOOKUP() /test => -1 (Invalid argument)
[2014-03-18 11:32:00.359934] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 1092090275
[2014-03-18 11:32:00.359949] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /projects
[2014-03-18 11:32:00.359963] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 44: LOOKUP() /projects => -1 (Invalid argument)
[2014-03-18 11:32:00.360343] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 485417449
[2014-03-18 11:32:00.360358] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /harvard-data2
[2014-03-18 11:32:00.360372] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 47: LOOKUP() /harvard-data2 => -1 (Invalid argument)
[2014-03-18 11:32:00.362629] W [dht-layout.c:186:dht_layout_search]
0-gdata-dht: no subvolume for hash (value) = 124395703
[2014-03-18 11:32:00.362645] E [dht-common.c:1372:dht_lookup]
0-gdata-dht: Failed to get hashed subvol for /temp_home
[2014-03-18 11:32:00.362660] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 62: LOOKUP() /temp_home => -1 (Invalid argument)
==========================================================================
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Pat Haley
2014-Mar-18 16:58 UTC
[Gluster-users] Unable to access directories in wounded gluster setup
Hi Again, I was again going over my notes from the other times I've been in trouble. In particular, back in November I received the following "reset" procedure: ---------- > If gluster-data is pingable from the other bricks, you could try > detaching and retttaching it from gluster-0-0 or 0-1. > 1) On gluster-0-0: > `gluster peer detach gluster-data`, if that fails, > `gluster peer detach gluster-data force` > 2) On gluster-data: > `rm -rf /var/lib/glusterd` > `service glusterd restart` > 3) Again on gluster-0-0: > 'gluster peer probe gluster-data' ---------- Would such a "reset" be safe to try when only 2 out of 3 bricks are up? Would it be likely to help? Thanks. Pat> > > Hi, > > We are employing gluster to merge the disks from > 3 servers into a common name-space. We just had > a power outage, and when the power came back on, > one of the servers had it's power supply damaged. > While working on this problem, we tried to get the > gluster area (/gdata) up with the the 2 working > servers, so at least the files on them would be > available while we repair the other server. This > hasn't worked as expected. From past experience > we expected to be able to see all the subdirectories > with some files simply absent. What we see instead > is that 4 of the top-level directories are unaccessible > > from client: > ls /gdata > ls: cannot access /gdata/test: Invalid argument > ls: cannot access /gdata/projects: Invalid argument > ls: cannot access /gdata/harvard-data2: Invalid argument > ls: cannot access /gdata/temp_home: Invalid argument > drwxr-xr-x 6 root root 152 Dec 5 12:27 harvard > drwxr-xr-x 4 root root 74 Dec 5 12:27 harvard-archive > ?????????? ? ? ? ? ? harvard-data2 > drwxr-xr-x 104 pierrel 8310 16K Dec 5 12:27 harvard-data3 > drwx------ 2 root root 12 Dec 5 12:27 lost+found > ?????????? ? ? ? ? ? projects > drwxrwxr-x 8 root software 4.1K Dec 5 12:27 software > drwxr-xr-x 5 root root 4.1K Dec 5 12:27 src > ?????????? ? ? ? ? ? temp_home > ?????????? ? ? ? ? ? test > > We have tried > - testing the communications between the bricks and > between the client & bricks (comms fine) > - restarting the gluster daemons on the bricks > - unmounting and remounting /gdata on the client > - mounting the gluster area directly on one of the > servers using mount -t glusterfs mseas-data:/gdata /gdata > > All give the same result. The status reported by gluster > looks fine (given that 1 brick is down). The reports are > included below. The log files from the mounting/ls tests are > included after that. > > What should we do/look at next to solve/debug this problem? > > Thanks. > > Pat > > --------------------------------------------------------------------- > [root at mseas-data glusterfs]# gluster --version > glusterfs 3.3.1 built on Oct 11 2012 22:01:05 > --------------------------------------------------------------------- > [root at mseas-data bricks]# gluster volume info > > Volume Name: gdata > Type: Distribute > Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d > Status: Started > Number of Bricks: 3 > Transport-type: tcp > Bricks: > Brick1: gluster-0-0:/mseas-data-0-0 > Brick2: gluster-0-1:/mseas-data-0-1 > Brick3: gluster-data:/data > > --------------------------------------------------------------------- > > [root at mseas-data bricks]# gluster volume status > Status of volume: gdata > Gluster process Port Online Pid > ------------------------------------------------------------------------------ > > Brick gluster-0-0:/mseas-data-0-0 24010 Y 11346 > Brick gluster-data:/data 24010 Y 28791 > NFS Server on localhost 38467 Y 28797 > NFS Server on gluster-0-0 38467 Y 11352 > > --------------------------------------------------------------------- > > [root at mseas-data bricks]# gluster peer status > Number of Peers: 2 > > Hostname: gluster-0-1 > Uuid: 978e0f76-6474-4203-8617-ed5ad7d29239 > State: Peer in Cluster (Disconnected) > > Hostname: gluster-0-0 > Uuid: 3f73f5cc-39d8-4d9a-b442-033cb074b247 > State: Peer in Cluster (Connected) > --------------------------------------------------------------------- > > > ============================================================> Log files from test mount from client: > -------------------------------------- > > gluster-data: /var/log/glusterfs/bricks/data.log > ------------------------------------------------- > [2014-03-18 10:43:57.149035] I [server.c:703:server_rpc_notify] > 0-gdata-server: disconnecting connectionfrom > compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0 > [2014-03-18 10:43:57.149073] I > [server-helpers.c:741:server_connection_put] 0-gdata-server: Shutting > down connection > compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0 > [2014-03-18 10:43:57.149127] I > [server-helpers.c:629:server_connection_destroy] 0-gdata-server: > destroyed connection of > compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-2-0 > [2014-03-18 10:44:10.149539] I [server-handshake.c:571:server_setvolume] > 0-gdata-server: accepted client from > compute-3-0.local-15405-2014/03/18-10:44:06:129552-gdata-client-2-0 > (version: 3.3.1) > > gluster-0-0: /var/log/glusterfs/bricks/data.log > ----------------------------------------------- > [2014-03-18 10:43:57.141122] I [server.c:703:server_rpc_notify] > 0-gdata-server: disconnecting connectionfrom > compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-0-0 > [2014-03-18 10:43:57.141209] I > [server-helpers.c:741:server_connection_put] 0-gdata-server: Shutting > down connection > compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-0-0 > [2014-03-18 10:43:57.141299] I > [server-helpers.c:629:server_connection_destroy] 0-gdata-server: > destroyed connection of > compute-3-0.local-15332-2014/03/18-10:10:31:456263-gdata-client-0-0 > [2014-03-18 10:44:10.143643] I [server-handshake.c:571:server_setvolume] > 0-gdata-server: accepted client from > compute-3-0.local-15405-2014/03/18-10:44:06:129552-gdata-client-0-0 > (version: 3.3.1) > > client: /var/log/glusterfs/gdata.log > ------------------------------------- > [2014-03-18 10:43:57.139136] I [fuse-bridge.c:4091:fuse_thread_proc] > 0-fuse: unmounting /gdata > [2014-03-18 10:43:57.139423] W [glusterfsd.c:831:cleanup_and_exit] > (-->/lib64/libc.so.6(clone+0x6d) [0x3e6a8e5ccd] > (-->/lib64/libpthread.so.0() [0x3e6b0077f1] > (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d4d]))) 0-: > received signum (15), shutting down > [2014-03-18 10:43:57.139443] I [fuse-bridge.c:4648:fini] 0-fuse: > Unmounting '/gdata'. > [2014-03-18 10:44:06.132964] I [glusterfsd.c:1666:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1 > [2014-03-18 10:44:06.139237] I [io-cache.c:1549:check_cache_size_ok] > 0-gdata-quick-read: Max cache size is 66901987328 > [2014-03-18 10:44:06.139290] I [io-cache.c:1549:check_cache_size_ok] > 0-gdata-io-cache: Max cache size is 66901987328 > [2014-03-18 10:44:06.141718] I [client.c:2142:notify] 0-gdata-client-0: > parent translators are ready, attempting connect on transport > [2014-03-18 10:44:06.144062] I [client.c:2142:notify] 0-gdata-client-1: > parent translators are ready, attempting connect on transport > [2014-03-18 10:44:06.146039] I [client.c:2142:notify] 0-gdata-client-2: > parent translators are ready, attempting connect on transport > Given volfile: > +------------------------------------------------------------------------------+ > > 1: volume gdata-client-0 > 2: type protocol/client > 3: option remote-host gluster-0-0 > 4: option remote-subvolume /mseas-data-0-0 > 5: option transport-type tcp > 6: end-volume > 7: > 8: volume gdata-client-1 > 9: type protocol/client > 10: option remote-host gluster-0-1 > 11: option remote-subvolume /mseas-data-0-1 > 12: option transport-type tcp > 13: end-volume > 14: > 15: volume gdata-client-2 > 16: type protocol/client > 17: option remote-host gluster-data > 18: option remote-subvolume /data > 19: option transport-type tcp > 20: end-volume > 21: > 22: volume gdata-dht > 23: type cluster/distribute > 24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2 > 25: end-volume > 26: > 27: volume gdata-write-behind > 28: type performance/write-behind > 29: subvolumes gdata-dht > 30: end-volume > 31: > 32: volume gdata-read-ahead > 33: type performance/read-ahead > 34: subvolumes gdata-write-behind > 35: end-volume > 36: > 37: volume gdata-io-cache > 38: type performance/io-cache > 39: subvolumes gdata-read-ahead > 40: end-volume > 41: > 42: volume gdata-quick-read > 43: type performance/quick-read > 44: subvolumes gdata-io-cache > 45: end-volume > 46: > 47: volume gdata-md-cache > 48: type performance/md-cache > 49: subvolumes gdata-quick-read > 50: end-volume > 51: > 52: volume gdata > 53: type debug/io-stats > 54: option latency-measurement off > 55: option count-fop-hits off > 56: subvolumes gdata-md-cache > 57: end-volume > > +------------------------------------------------------------------------------+ > > [2014-03-18 10:44:06.148454] I [rpc-clnt.c:1657:rpc_clnt_reconfig] > 0-gdata-client-2: changing port to 24010 (from 0) > [2014-03-18 10:44:06.148529] I [rpc-clnt.c:1657:rpc_clnt_reconfig] > 0-gdata-client-0: changing port to 24010 (from 0) > [2014-03-18 10:44:09.146764] E [socket.c:1715:socket_connect_finish] > 0-gdata-client-1: connection to failed (No route to host) > [2014-03-18 10:44:10.141499] I > [client-handshake.c:1636:select_server_supported_programs] > 0-gdata-client-2: Using Program GlusterFS 3.3.1, Num (1298437), Version > (330) > [2014-03-18 10:44:10.141748] I > [client-handshake.c:1433:client_setvolume_cbk] 0-gdata-client-2: > Connected to 10.1.1.2:24010, attached to remote volume '/data'. > [2014-03-18 10:44:10.141759] I > [client-handshake.c:1445:client_setvolume_cbk] 0-gdata-client-2: Server > and Client lk-version numbers are not same, reopening the fds > [2014-03-18 10:44:10.141866] I > [client-handshake.c:453:client_set_lk_version_cbk] 0-gdata-client-2: > Server lk version = 1 > [2014-03-18 10:44:10.143526] I > [client-handshake.c:1636:select_server_supported_programs] > 0-gdata-client-0: Using Program GlusterFS 3.3.1, Num (1298437), Version > (330) > [2014-03-18 10:44:10.143800] I > [client-handshake.c:1433:client_setvolume_cbk] 0-gdata-client-0: > Connected to 10.1.1.10:24010, attached to remote volume '/mseas-data-0-0'. > [2014-03-18 10:44:10.143810] I > [client-handshake.c:1445:client_setvolume_cbk] 0-gdata-client-0: Server > and Client lk-version numbers are not same, reopening the fds > [2014-03-18 10:44:10.147967] I [fuse-bridge.c:4191:fuse_graph_setup] > 0-fuse: switched to graph 0 > [2014-03-18 10:44:10.148147] I > [client-handshake.c:453:client_set_lk_version_cbk] 0-gdata-client-0: > Server lk version = 1 > [2014-03-18 10:44:10.148251] I [fuse-bridge.c:3376:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 > kernel 7.13 > [2014-03-18 10:44:10.148993] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /. holes=1 overlaps=0 > [2014-03-18 10:44:10.149012] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.335068] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 1072520554 > [2014-03-18 10:48:34.335110] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /test > [2014-03-18 10:48:34.335131] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 7: LOOKUP() /test => -1 (Invalid argument) > [2014-03-18 10:48:34.336648] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 1092090275 > [2014-03-18 10:48:34.336660] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /projects > [2014-03-18 10:48:34.336670] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 16: LOOKUP() /projects => -1 (Invalid argument) > [2014-03-18 10:48:34.337271] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /src. holes=1 overlaps=0 > [2014-03-18 10:48:34.337310] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.337400] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 485417449 > [2014-03-18 10:48:34.337414] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /harvard-data2 > [2014-03-18 10:48:34.337424] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 18: LOOKUP() /harvard-data2 => -1 (Invalid argument) > [2014-03-18 10:48:34.337988] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /harvard. holes=1 overlaps=0 > [2014-03-18 10:48:34.338003] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.338564] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /harvard-archive. holes=1 overlaps=0 > [2014-03-18 10:48:34.338576] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.339173] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /harvard-data3. holes=1 overlaps=0 > [2014-03-18 10:48:34.339184] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.339740] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /lost+found. holes=1 overlaps=0 > [2014-03-18 10:48:34.339751] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.340290] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /software. holes=1 overlaps=0 > [2014-03-18 10:48:34.340302] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.340872] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /tmp. holes=1 overlaps=0 > [2014-03-18 10:48:34.340883] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.341463] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /z.rdiff-backup-data. holes=1 overlaps=0 > [2014-03-18 10:48:34.341474] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.341561] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 124395703 > [2014-03-18 10:48:34.341576] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /temp_home > [2014-03-18 10:48:34.341586] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 26: LOOKUP() /temp_home => -1 (Invalid argument) > [2014-03-18 10:48:34.342137] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /data. holes=1 overlaps=0 > [2014-03-18 10:48:34.342152] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 10:48:34.342737] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /bibliography. holes=1 overlaps=0 > [2014-03-18 10:48:34.342748] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [root at compute-3-0 glusterfs]# > > ============================================================> Log file from test mount on gluster-data: > ----------------------------------------- > > /var/log/glusterfs/gdata.log > ----------------------------- > [2014-03-18 11:29:23.549718] I [glusterfsd.c:1666:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1 > [2014-03-18 11:29:23.751872] I [io-cache.c:1549:check_cache_size_ok] > 0-gdata-quick-read: Max cache size is 8374652928 > [2014-03-18 11:29:23.767321] I [io-cache.c:1549:check_cache_size_ok] > 0-gdata-io-cache: Max cache size is 8374652928 > [2014-03-18 11:29:23.770541] I [client.c:2142:notify] 0-gdata-client-0: > parent translators are ready, attempting connect on transport > [2014-03-18 11:29:23.773247] I [client.c:2142:notify] 0-gdata-client-1: > parent translators are ready, attempting connect on transport > [2014-03-18 11:29:23.775734] I [client.c:2142:notify] 0-gdata-client-2: > parent translators are ready, attempting connect on transport > Given volfile: > +------------------------------------------------------------------------------+ > > 1: volume gdata-client-0 > 2: type protocol/client > 3: option remote-host gluster-0-0 > 4: option remote-subvolume /mseas-data-0-0 > 5: option transport-type tcp > 6: end-volume > 7: > 8: volume gdata-client-1 > 9: type protocol/client > 10: option remote-host gluster-0-1 > 11: option remote-subvolume /mseas-data-0-1 > 12: option transport-type tcp > 13: end-volume > 14: > 15: volume gdata-client-2 > 16: type protocol/client > 17: option remote-host gluster-data > 18: option remote-subvolume /data > 19: option transport-type tcp > 20: end-volume > 21: > 22: volume gdata-dht > 23: type cluster/distribute > 24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2 > 25: end-volume > 26: > 27: volume gdata-write-behind > 28: type performance/write-behind > 29: subvolumes gdata-dht > 30: end-volume > 31: > 32: volume gdata-read-ahead > 33: type performance/read-ahead > 34: subvolumes gdata-write-behind > 35: end-volume > 36: > 37: volume gdata-io-cache > 38: type performance/io-cache > 39: subvolumes gdata-read-ahead > 40: end-volume > 41: > 42: volume gdata-quick-read > 43: type performance/quick-read > 44: subvolumes gdata-io-cache > 45: end-volume > 46: > 47: volume gdata-md-cache > 48: type performance/md-cache > 49: subvolumes gdata-quick-read > 50: end-volume > 51: > 52: volume gdata > 53: type debug/io-stats > 54: option latency-measurement off > 55: option count-fop-hits off > 56: subvolumes gdata-md-cache > 57: end-volume > > +------------------------------------------------------------------------------+ > > [2014-03-18 11:29:23.778543] I [rpc-clnt.c:1657:rpc_clnt_reconfig] > 0-gdata-client-2: changing port to 24010 (from 0) > [2014-03-18 11:29:23.778755] I [rpc-clnt.c:1657:rpc_clnt_reconfig] > 0-gdata-client-0: changing port to 24010 (from 0) > [2014-03-18 11:29:23.981081] E [socket.c:1715:socket_connect_finish] > 0-gdata-client-1: connection to failed (No route to host) > [2014-03-18 11:29:27.736927] I > [client-handshake.c:1636:select_server_supported_programs] > 0-gdata-client-2: Using Program GlusterFS 3.3.1, Num (1298437), Version > (330) > [2014-03-18 11:29:27.737135] I > [client-handshake.c:1433:client_setvolume_cbk] 0-gdata-client-2: > Connected to 10.1.1.2:24010, attached to remote volume '/data'. > [2014-03-18 11:29:27.737152] I > [client-handshake.c:1445:client_setvolume_cbk] 0-gdata-client-2: Server > and Client lk-version numbers are not same, reopening the fds > [2014-03-18 11:29:27.737242] I > [client-handshake.c:453:client_set_lk_version_cbk] 0-gdata-client-2: > Server lk version = 1 > [2014-03-18 11:29:27.739760] I > [client-handshake.c:1636:select_server_supported_programs] > 0-gdata-client-0: Using Program GlusterFS 3.3.1, Num (1298437), Version > (330) > [2014-03-18 11:29:27.740067] I > [client-handshake.c:1433:client_setvolume_cbk] 0-gdata-client-0: > Connected to 10.1.1.10:24010, attached to remote volume '/mseas-data-0-0'. > [2014-03-18 11:29:27.740081] I > [client-handshake.c:1445:client_setvolume_cbk] 0-gdata-client-0: Server > and Client lk-version numbers are not same, reopening the fds > [2014-03-18 11:29:27.747148] I [fuse-bridge.c:4191:fuse_graph_setup] > 0-fuse: switched to graph 0 > [2014-03-18 11:29:27.747322] I > [client-handshake.c:453:client_set_lk_version_cbk] 0-gdata-client-0: > Server lk version = 1 > [2014-03-18 11:29:27.747458] I [fuse-bridge.c:3376:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 > kernel 7.8 > [2014-03-18 11:29:27.748446] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /. holes=1 overlaps=0 > [2014-03-18 11:29:27.748467] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.543344] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 1072520554 > [2014-03-18 11:30:27.543374] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /test > [2014-03-18 11:30:27.543394] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 7: LOOKUP() /test => -1 (Invalid argument) > [2014-03-18 11:30:27.544817] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 1092090275 > [2014-03-18 11:30:27.544832] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /projects > [2014-03-18 11:30:27.544846] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 12: LOOKUP() /projects => -1 (Invalid argument) > [2014-03-18 11:30:27.545336] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /src. holes=1 overlaps=0 > [2014-03-18 11:30:27.545356] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.545476] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 485417449 > [2014-03-18 11:30:27.545491] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /harvard-data2 > [2014-03-18 11:30:27.545505] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 14: LOOKUP() /harvard-data2 => -1 (Invalid argument) > [2014-03-18 11:30:27.545995] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /harvard. holes=1 overlaps=0 > [2014-03-18 11:30:27.546010] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.546600] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /harvard-archive. holes=1 overlaps=0 > [2014-03-18 11:30:27.546616] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.547206] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /harvard-data3. holes=1 overlaps=0 > [2014-03-18 11:30:27.547222] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.547901] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /lost+found. holes=1 overlaps=0 > [2014-03-18 11:30:27.547917] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.548423] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /software. holes=1 overlaps=0 > [2014-03-18 11:30:27.548439] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.548981] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /tmp. holes=1 overlaps=0 > [2014-03-18 11:30:27.548997] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.549500] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /z.rdiff-backup-data. holes=1 overlaps=0 > [2014-03-18 11:30:27.549515] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.549629] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 124395703 > [2014-03-18 11:30:27.549644] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /temp_home > [2014-03-18 11:30:27.549658] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 22: LOOKUP() /temp_home => -1 (Invalid argument) > [2014-03-18 11:30:27.550134] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /data. holes=1 overlaps=0 > [2014-03-18 11:30:27.550149] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:30:27.550793] I [dht-layout.c:593:dht_layout_normalize] > 0-gdata-dht: found anomalies in /bibliography. holes=1 overlaps=0 > [2014-03-18 11:30:27.550830] W > [dht-selfheal.c:875:dht_selfheal_directory] 0-gdata-dht: 1 subvolumes > down -- not fixing > [2014-03-18 11:32:00.358667] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 1072520554 > [2014-03-18 11:32:00.358700] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /test > [2014-03-18 11:32:00.358717] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 35: LOOKUP() /test => -1 (Invalid argument) > [2014-03-18 11:32:00.359934] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 1092090275 > [2014-03-18 11:32:00.359949] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /projects > [2014-03-18 11:32:00.359963] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 44: LOOKUP() /projects => -1 (Invalid argument) > [2014-03-18 11:32:00.360343] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 485417449 > [2014-03-18 11:32:00.360358] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /harvard-data2 > [2014-03-18 11:32:00.360372] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 47: LOOKUP() /harvard-data2 => -1 (Invalid argument) > [2014-03-18 11:32:00.362629] W [dht-layout.c:186:dht_layout_search] > 0-gdata-dht: no subvolume for hash (value) = 124395703 > [2014-03-18 11:32:00.362645] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for /temp_home > [2014-03-18 11:32:00.362660] W [fuse-bridge.c:292:fuse_entry_cbk] > 0-glusterfs-fuse: 62: LOOKUP() /temp_home => -1 (Invalid argument) > > ==========================================================================> > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > Pat Haley Email: phaley at mit.edu > Center for Ocean Engineering Phone: (617) 253-6824 > Dept. of Mechanical Engineering Fax: (617) 253-8125 > MIT, Room 5-213 http://web.mit.edu/phaley/www/ > 77 Massachusetts Avenue > Cambridge, MA 02139-4301 > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley at mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301