TomK
2018-Apr-09 06:02 UTC
[Gluster-users] volume start: gv01: failed: Quorum not met. Volume operation not allowed.
Hey All, In a two node glusterfs setup, with one node down, can't use the second node to mount the volume. I understand this is expected behaviour? Anyway to allow the secondary node to function then replicate what changed to the first (primary) when it's back online? Or should I just go for a third node to allow for this? Also, how safe is it to set the following to none? cluster.quorum-type: auto cluster.server-quorum-type: server [root at nfs01 /]# gluster volume start gv01 volume start: gv01: failed: Quorum not met. Volume operation not allowed. [root at nfs01 /]# [root at nfs01 /]# gluster volume status Status of volume: gv01 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick nfs01:/bricks/0/gv01 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 25561 Task Status of Volume gv01 ------------------------------------------------------------------------------ There are no active volume tasks [root at nfs01 /]# [root at nfs01 /]# gluster volume info Volume Name: gv01 Type: Replicate Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: nfs01:/bricks/0/gv01 Brick2: nfs02:/bricks/0/gv01 Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off nfs.trusted-sync: on performance.cache-size: 1GB performance.io-thread-count: 16 performance.write-behind-window-size: 8MB performance.readdir-ahead: on client.event-threads: 8 server.event-threads: 8 cluster.quorum-type: auto cluster.server-quorum-type: server [root at nfs01 /]# ==> n.log <=[2018-04-09 05:08:13.704156] I [MSGID: 100030] [glusterfsd.c:2556:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.13.2 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=nfs01 --volfile-id=/gv01 /n) [2018-04-09 05:08:13.711255] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2018-04-09 05:08:13.728297] W [socket.c:3216:socket_connect] 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" [2018-04-09 05:08:13.729025] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2018-04-09 05:08:13.737757] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2018-04-09 05:08:13.738114] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3 [2018-04-09 05:08:13.738203] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4 [2018-04-09 05:08:13.738324] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 5 [2018-04-09 05:08:13.738330] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 6 [2018-04-09 05:08:13.738655] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 7 [2018-04-09 05:08:13.738742] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 8 [2018-04-09 05:08:13.739460] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-gv01-readdir-ahead: option 'parallel-readdir' is not recognized [2018-04-09 05:08:13.739787] I [MSGID: 114020] [client.c:2360:notify] 0-gv01-client-0: parent translators are ready, attempting connect on transport [2018-04-09 05:08:13.747040] W [socket.c:3216:socket_connect] 0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" [2018-04-09 05:08:13.747372] I [MSGID: 114020] [client.c:2360:notify] 0-gv01-client-1: parent translators are ready, attempting connect on transport [2018-04-09 05:08:13.747883] E [MSGID: 114058] [client-handshake.c:1571:client_query_portmap_cbk] 0-gv01-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2018-04-09 05:08:13.748026] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-gv01-client-0: disconnected from gv01-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2018-04-09 05:08:13.748070] W [MSGID: 108001] [afr-common.c:5391:afr_notify] 0-gv01-replicate-0: Client-quorum is not met [2018-04-09 05:08:13.754493] W [socket.c:3216:socket_connect] 0-gv01-client-1: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" Final graph: +------------------------------------------------------------------------------+ 1: volume gv01-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host nfs01 5: option remote-subvolume /bricks/0/gv01 6: option transport-type socket 7: option transport.address-family inet 8: option username 916ccf06-dc1d-467f-bc3d-f00a7449618f 9: option password a44739e0-9587-411f-8e6a-9a6a4e46156c 10: option event-threads 8 11: option transport.tcp-user-timeout 0 12: option transport.socket.keepalive-time 20 13: option transport.socket.keepalive-interval 2 14: option transport.socket.keepalive-count 9 15: option send-gids true 16: end-volume 17: 18: volume gv01-client-1 19: type protocol/client 20: option ping-timeout 42 21: option remote-host nfs02 22: option remote-subvolume /bricks/0/gv01 23: option transport-type socket 24: option transport.address-family inet 25: option username 916ccf06-dc1d-467f-bc3d-f00a7449618f 26: option password a44739e0-9587-411f-8e6a-9a6a4e46156c 27: option event-threads 8 28: option transport.tcp-user-timeout 0 29: option transport.socket.keepalive-time 20 30: option transport.socket.keepalive-interval 2 31: option transport.socket.keepalive-count 9 32: option send-gids true 33: end-volume 34: 35: volume gv01-replicate-0 36: type cluster/replicate 37: option afr-pending-xattr gv01-client-0,gv01-client-1 38: option quorum-type auto 39: option use-compound-fops off 40: subvolumes gv01-client-0 gv01-client-1 41: end-volume 42: 43: volume gv01-dht 44: type cluster/distribute 45: option lock-migration off 46: subvolumes gv01-replicate-0 47: end-volume 48: 49: volume gv01-write-behind 50: type performance/write-behind 51: option cache-size 8MB 52: subvolumes gv01-dht 53: end-volume 54: 55: volume gv01-read-ahead 56: type performance/read-ahead 57: subvolumes gv01-write-behind 58: end-volume 59: 60: volume gv01-readdir-ahead 61: type performance/readdir-ahead 62: option parallel-readdir off 63: option rda-request-size 131072 64: option rda-cache-limit 10MB 65: subvolumes gv01-read-ahead 66: end-volume 67: 68: volume gv01-io-cache 69: type performance/io-cache 70: option cache-size 1GB 71: subvolumes gv01-readdir-ahead 72: end-volume 73: 74: volume gv01-quick-read 75: type performance/quick-read 76: option cache-size 1GB 77: subvolumes gv01-io-cache 78: end-volume 79: 80: volume gv01-open-behind 81: type performance/open-behind 82: subvolumes gv01-quick-read 83: end-volume 84: 85: volume gv01-md-cache 86: type performance/md-cache 87: subvolumes gv01-open-behind 88: end-volume 89: 90: volume gv01 91: type debug/io-stats 92: option log-level INFO 93: option latency-measurement off 94: option count-fop-hits off 95: subvolumes gv01-md-cache 96: end-volume 97: 98: volume meta-autoload 99: type meta 100: subvolumes gv01 101: end-volume 102: +------------------------------------------------------------------------------+ [2018-04-09 05:08:13.922631] E [socket.c:2374:socket_connect_finish] 0-gv01-client-1: connection to 192.168.0.119:24007 failed (No route to host); disconnecting socket [2018-04-09 05:08:13.922690] E [MSGID: 108006] [afr-common.c:5164:__afr_handle_child_down_event] 0-gv01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2018-04-09 05:08:13.926201] I [fuse-bridge.c:4205:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2018-04-09 05:08:13.926245] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse: switched to graph 0 [2018-04-09 05:08:13.926518] I [MSGID: 108006] [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes up [2018-04-09 05:08:13.926671] E [MSGID: 101046] [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null [2018-04-09 05:08:13.926762] E [fuse-bridge.c:4271:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected) [2018-04-09 05:08:13.927207] I [MSGID: 108006] [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes up [2018-04-09 05:08:13.927262] E [MSGID: 101046] [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null [2018-04-09 05:08:13.927301] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected) [2018-04-09 05:08:13.927339] E [fuse-bridge.c:900:fuse_getattr_resume] 0-glusterfs-fuse: 2: GETATTR 1 (00000000-0000-0000-0000-000000000001) resolution failed [2018-04-09 05:08:13.931497] I [MSGID: 108006] [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes up [2018-04-09 05:08:13.931558] E [MSGID: 101046] [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null [2018-04-09 05:08:13.931599] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected) [2018-04-09 05:08:13.931623] E [fuse-bridge.c:900:fuse_getattr_resume] 0-glusterfs-fuse: 3: GETATTR 1 (00000000-0000-0000-0000-000000000001) resolution failed [2018-04-09 05:08:13.937258] I [fuse-bridge.c:5093:fuse_thread_proc] 0-fuse: initating unmount of /n [2018-04-09 05:08:13.938043] W [glusterfsd.c:1393:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fb80b05ae25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560b52471675] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560b5247149b] ) 0-: received signum (15), shutting down [2018-04-09 05:08:13.938086] I [fuse-bridge.c:5855:fini] 0-fuse: Unmounting '/n'. [2018-04-09 05:08:13.938106] I [fuse-bridge.c:5860:fini] 0-fuse: Closing fuse connection to '/n'. ==> glusterd.log <=[2018-04-09 05:08:15.118078] W [socket.c:3216:socket_connect] 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" ==> glustershd.log <=[2018-04-09 05:08:15.282192] W [socket.c:3216:socket_connect] 0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" [2018-04-09 05:08:15.289508] W [socket.c:3216:socket_connect] 0-gv01-client-1: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" -- Cheers, Tom K. ------------------------------------------------------------------------------------- Living on earth is expensive, but it includes a free trip around the sun.
Alex K
2018-Apr-09 06:45 UTC
[Gluster-users] volume start: gv01: failed: Quorum not met. Volume operation not allowed.
Hi, You need 3 nodes at least to have quorum enabled. In 2 node setup you need to disable quorum so as to be able to still use the volume when one of the nodes go down. On Mon, Apr 9, 2018, 09:02 TomK <tomkcpr at mdevsys.com> wrote:> Hey All, > > In a two node glusterfs setup, with one node down, can't use the second > node to mount the volume. I understand this is expected behaviour? > Anyway to allow the secondary node to function then replicate what > changed to the first (primary) when it's back online? Or should I just > go for a third node to allow for this? > > Also, how safe is it to set the following to none? > > cluster.quorum-type: auto > cluster.server-quorum-type: server > > > [root at nfs01 /]# gluster volume start gv01 > volume start: gv01: failed: Quorum not met. Volume operation not allowed. > [root at nfs01 /]# > > > [root at nfs01 /]# gluster volume status > Status of volume: gv01 > Gluster process TCP Port RDMA Port Online > Pid > > ------------------------------------------------------------------------------ > Brick nfs01:/bricks/0/gv01 N/A N/A N > N/A > Self-heal Daemon on localhost N/A N/A Y > 25561 > > Task Status of Volume gv01 > > ------------------------------------------------------------------------------ > There are no active volume tasks > > [root at nfs01 /]# > > > [root at nfs01 /]# gluster volume info > > Volume Name: gv01 > Type: Replicate > Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: nfs01:/bricks/0/gv01 > Brick2: nfs02:/bricks/0/gv01 > Options Reconfigured: > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: off > nfs.trusted-sync: on > performance.cache-size: 1GB > performance.io-thread-count: 16 > performance.write-behind-window-size: 8MB > performance.readdir-ahead: on > client.event-threads: 8 > server.event-threads: 8 > cluster.quorum-type: auto > cluster.server-quorum-type: server > [root at nfs01 /]# > > > > > ==> n.log <=> [2018-04-09 05:08:13.704156] I [MSGID: 100030] [glusterfsd.c:2556:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version > 3.13.2 (args: /usr/sbin/glusterfs --process-name fuse > --volfile-server=nfs01 --volfile-id=/gv01 /n) > [2018-04-09 05:08:13.711255] W [MSGID: 101002] > [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is > deprecated, preferred is 'transport.address-family', continuing with > correction > [2018-04-09 05:08:13.728297] W [socket.c:3216:socket_connect] > 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" > [2018-04-09 05:08:13.729025] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > [2018-04-09 05:08:13.737757] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 2 > [2018-04-09 05:08:13.738114] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 3 > [2018-04-09 05:08:13.738203] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 4 > [2018-04-09 05:08:13.738324] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 5 > [2018-04-09 05:08:13.738330] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 6 > [2018-04-09 05:08:13.738655] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 7 > [2018-04-09 05:08:13.738742] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 8 > [2018-04-09 05:08:13.739460] W [MSGID: 101174] > [graph.c:363:_log_if_unknown_option] 0-gv01-readdir-ahead: option > 'parallel-readdir' is not recognized > [2018-04-09 05:08:13.739787] I [MSGID: 114020] [client.c:2360:notify] > 0-gv01-client-0: parent translators are ready, attempting connect on > transport > [2018-04-09 05:08:13.747040] W [socket.c:3216:socket_connect] > 0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > [2018-04-09 05:08:13.747372] I [MSGID: 114020] [client.c:2360:notify] > 0-gv01-client-1: parent translators are ready, attempting connect on > transport > [2018-04-09 05:08:13.747883] E [MSGID: 114058] > [client-handshake.c:1571:client_query_portmap_cbk] 0-gv01-client-0: > failed to get the port number for remote subvolume. Please run 'gluster > volume status' on server to see if brick process is running. > [2018-04-09 05:08:13.748026] I [MSGID: 114018] > [client.c:2285:client_rpc_notify] 0-gv01-client-0: disconnected from > gv01-client-0. Client process will keep trying to connect to glusterd > until brick's port is available > [2018-04-09 05:08:13.748070] W [MSGID: 108001] > [afr-common.c:5391:afr_notify] 0-gv01-replicate-0: Client-quorum is not met > [2018-04-09 05:08:13.754493] W [socket.c:3216:socket_connect] > 0-gv01-client-1: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > Final graph: > > +------------------------------------------------------------------------------+ > 1: volume gv01-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host nfs01 > 5: option remote-subvolume /bricks/0/gv01 > 6: option transport-type socket > 7: option transport.address-family inet > 8: option username 916ccf06-dc1d-467f-bc3d-f00a7449618f > 9: option password a44739e0-9587-411f-8e6a-9a6a4e46156c > 10: option event-threads 8 > 11: option transport.tcp-user-timeout 0 > 12: option transport.socket.keepalive-time 20 > 13: option transport.socket.keepalive-interval 2 > 14: option transport.socket.keepalive-count 9 > 15: option send-gids true > 16: end-volume > 17: > 18: volume gv01-client-1 > 19: type protocol/client > 20: option ping-timeout 42 > 21: option remote-host nfs02 > 22: option remote-subvolume /bricks/0/gv01 > 23: option transport-type socket > 24: option transport.address-family inet > 25: option username 916ccf06-dc1d-467f-bc3d-f00a7449618f > 26: option password a44739e0-9587-411f-8e6a-9a6a4e46156c > 27: option event-threads 8 > 28: option transport.tcp-user-timeout 0 > 29: option transport.socket.keepalive-time 20 > 30: option transport.socket.keepalive-interval 2 > 31: option transport.socket.keepalive-count 9 > 32: option send-gids true > 33: end-volume > 34: > 35: volume gv01-replicate-0 > 36: type cluster/replicate > 37: option afr-pending-xattr gv01-client-0,gv01-client-1 > 38: option quorum-type auto > 39: option use-compound-fops off > 40: subvolumes gv01-client-0 gv01-client-1 > 41: end-volume > 42: > 43: volume gv01-dht > 44: type cluster/distribute > 45: option lock-migration off > 46: subvolumes gv01-replicate-0 > 47: end-volume > 48: > 49: volume gv01-write-behind > 50: type performance/write-behind > 51: option cache-size 8MB > 52: subvolumes gv01-dht > 53: end-volume > 54: > 55: volume gv01-read-ahead > 56: type performance/read-ahead > 57: subvolumes gv01-write-behind > 58: end-volume > 59: > 60: volume gv01-readdir-ahead > 61: type performance/readdir-ahead > 62: option parallel-readdir off > 63: option rda-request-size 131072 > 64: option rda-cache-limit 10MB > 65: subvolumes gv01-read-ahead > 66: end-volume > 67: > 68: volume gv01-io-cache > 69: type performance/io-cache > 70: option cache-size 1GB > 71: subvolumes gv01-readdir-ahead > 72: end-volume > 73: > 74: volume gv01-quick-read > 75: type performance/quick-read > 76: option cache-size 1GB > 77: subvolumes gv01-io-cache > 78: end-volume > 79: > 80: volume gv01-open-behind > 81: type performance/open-behind > 82: subvolumes gv01-quick-read > 83: end-volume > 84: > 85: volume gv01-md-cache > 86: type performance/md-cache > 87: subvolumes gv01-open-behind > 88: end-volume > 89: > 90: volume gv01 > 91: type debug/io-stats > 92: option log-level INFO > 93: option latency-measurement off > 94: option count-fop-hits off > 95: subvolumes gv01-md-cache > 96: end-volume > 97: > 98: volume meta-autoload > 99: type meta > 100: subvolumes gv01 > 101: end-volume > 102: > > +------------------------------------------------------------------------------+ > [2018-04-09 05:08:13.922631] E [socket.c:2374:socket_connect_finish] > 0-gv01-client-1: connection to 192.168.0.119:24007 failed (No route to > host); disconnecting socket > [2018-04-09 05:08:13.922690] E [MSGID: 108006] > [afr-common.c:5164:__afr_handle_child_down_event] 0-gv01-replicate-0: > All subvolumes are down. Going offline until atleast one of them comes > back up. > [2018-04-09 05:08:13.926201] I [fuse-bridge.c:4205:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 > kernel 7.22 > [2018-04-09 05:08:13.926245] I [fuse-bridge.c:4835:fuse_graph_sync] > 0-fuse: switched to graph 0 > [2018-04-09 05:08:13.926518] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes up > [2018-04-09 05:08:13.926671] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null > [2018-04-09 05:08:13.926762] E [fuse-bridge.c:4271:fuse_first_lookup] > 0-fuse: first lookup on root failed (Transport endpoint is not connected) > [2018-04-09 05:08:13.927207] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes up > [2018-04-09 05:08:13.927262] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null > [2018-04-09 05:08:13.927301] W > [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: > 00000000-0000-0000-0000-000000000001: failed to resolve (Transport > endpoint is not connected) > [2018-04-09 05:08:13.927339] E [fuse-bridge.c:900:fuse_getattr_resume] > 0-glusterfs-fuse: 2: GETATTR 1 (00000000-0000-0000-0000-000000000001) > resolution failed > [2018-04-09 05:08:13.931497] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes up > [2018-04-09 05:08:13.931558] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null > [2018-04-09 05:08:13.931599] W > [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: > 00000000-0000-0000-0000-000000000001: failed to resolve (Transport > endpoint is not connected) > [2018-04-09 05:08:13.931623] E [fuse-bridge.c:900:fuse_getattr_resume] > 0-glusterfs-fuse: 3: GETATTR 1 (00000000-0000-0000-0000-000000000001) > resolution failed > [2018-04-09 05:08:13.937258] I [fuse-bridge.c:5093:fuse_thread_proc] > 0-fuse: initating unmount of /n > [2018-04-09 05:08:13.938043] W [glusterfsd.c:1393:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7e25) [0x7fb80b05ae25] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560b52471675] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560b5247149b] ) 0-: > received signum (15), shutting down > [2018-04-09 05:08:13.938086] I [fuse-bridge.c:5855:fini] 0-fuse: > Unmounting '/n'. > [2018-04-09 05:08:13.938106] I [fuse-bridge.c:5860:fini] 0-fuse: Closing > fuse connection to '/n'. > > ==> glusterd.log <=> [2018-04-09 05:08:15.118078] W [socket.c:3216:socket_connect] > 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" > > ==> glustershd.log <=> [2018-04-09 05:08:15.282192] W [socket.c:3216:socket_connect] > 0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > [2018-04-09 05:08:15.289508] W [socket.c:3216:socket_connect] > 0-gv01-client-1: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > > > > > > > > -- > Cheers, > Tom K. > > ------------------------------------------------------------------------------------- > > Living on earth is expensive, but it includes a free trip around the sun. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180409/9138de1c/attachment.html>
TomK
2018-Apr-11 01:35 UTC
[Gluster-users] volume start: gv01: failed: Quorum not met. Volume operation not allowed.
On 4/9/2018 2:45 AM, Alex K wrote: Hey Alex, With two nodes, the setup works but both sides go down when one node is missing. Still I set the below two params to none and that solved my issue: cluster.quorum-type: none cluster.server-quorum-type: none Thank you for that. Cheers, Tom> Hi, > > You need 3 nodes at least to have quorum enabled. In 2 node setup you > need to disable quorum so as to be able to still use the volume when one > of the nodes go down. > > On Mon, Apr 9, 2018, 09:02 TomK <tomkcpr at mdevsys.com > <mailto:tomkcpr at mdevsys.com>> wrote: > > Hey All, > > In a two node glusterfs setup, with one node down, can't use the second > node to mount the volume.? I understand this is expected behaviour? > Anyway to allow the secondary node to function then replicate what > changed to the first (primary) when it's back online?? Or should I just > go for a third node to allow for this? > > Also, how safe is it to set the following to none? > > cluster.quorum-type: auto > cluster.server-quorum-type: server > > > [root at nfs01 /]# gluster volume start gv01 > volume start: gv01: failed: Quorum not met. Volume operation not > allowed. > [root at nfs01 /]# > > > [root at nfs01 /]# gluster volume status > Status of volume: gv01 > Gluster process? ? ? ? ? ? ? ? ? ? ? ? ? ? ?TCP Port? RDMA Port > Online? Pid > ------------------------------------------------------------------------------ > Brick nfs01:/bricks/0/gv01? ? ? ? ? ? ? ? ? N/A? ? ? ?N/A? ? ? ? N > ? ? ?N/A > Self-heal Daemon on localhost? ? ? ? ? ? ? ?N/A? ? ? ?N/A? ? ? ? Y > 25561 > > Task Status of Volume gv01 > ------------------------------------------------------------------------------ > There are no active volume tasks > > [root at nfs01 /]# > > > [root at nfs01 /]# gluster volume info > > Volume Name: gv01 > Type: Replicate > Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: nfs01:/bricks/0/gv01 > Brick2: nfs02:/bricks/0/gv01 > Options Reconfigured: > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: off > nfs.trusted-sync: on > performance.cache-size: 1GB > performance.io-thread-count: 16 > performance.write-behind-window-size: 8MB > performance.readdir-ahead: on > client.event-threads: 8 > server.event-threads: 8 > cluster.quorum-type: auto > cluster.server-quorum-type: server > [root at nfs01 /]# > > > > > ==> n.log <=> [2018-04-09 05:08:13.704156] I [MSGID: 100030] [glusterfsd.c:2556:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version > 3.13.2 (args: /usr/sbin/glusterfs --process-name fuse > --volfile-server=nfs01 --volfile-id=/gv01 /n) > [2018-04-09 05:08:13.711255] W [MSGID: 101002] > [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is > deprecated, preferred is 'transport.address-family', continuing with > correction > [2018-04-09 05:08:13.728297] W [socket.c:3216:socket_connect] > 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > [2018-04-09 05:08:13.729025] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > [2018-04-09 05:08:13.737757] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 2 > [2018-04-09 05:08:13.738114] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 3 > [2018-04-09 05:08:13.738203] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 4 > [2018-04-09 05:08:13.738324] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 5 > [2018-04-09 05:08:13.738330] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 6 > [2018-04-09 05:08:13.738655] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 7 > [2018-04-09 05:08:13.738742] I [MSGID: 101190] > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 8 > [2018-04-09 05:08:13.739460] W [MSGID: 101174] > [graph.c:363:_log_if_unknown_option] 0-gv01-readdir-ahead: option > 'parallel-readdir' is not recognized > [2018-04-09 05:08:13.739787] I [MSGID: 114020] [client.c:2360:notify] > 0-gv01-client-0: parent translators are ready, attempting connect on > transport > [2018-04-09 05:08:13.747040] W [socket.c:3216:socket_connect] > 0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > [2018-04-09 05:08:13.747372] I [MSGID: 114020] [client.c:2360:notify] > 0-gv01-client-1: parent translators are ready, attempting connect on > transport > [2018-04-09 05:08:13.747883] E [MSGID: 114058] > [client-handshake.c:1571:client_query_portmap_cbk] 0-gv01-client-0: > failed to get the port number for remote subvolume. Please run 'gluster > volume status' on server to see if brick process is running. > [2018-04-09 05:08:13.748026] I [MSGID: 114018] > [client.c:2285:client_rpc_notify] 0-gv01-client-0: disconnected from > gv01-client-0. Client process will keep trying to connect to glusterd > until brick's port is available > [2018-04-09 05:08:13.748070] W [MSGID: 108001] > [afr-common.c:5391:afr_notify] 0-gv01-replicate-0: Client-quorum is > not met > [2018-04-09 05:08:13.754493] W [socket.c:3216:socket_connect] > 0-gv01-client-1: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > Final graph: > +------------------------------------------------------------------------------+ > ? ?1: volume gv01-client-0 > ? ?2:? ? ?type protocol/client > ? ?3:? ? ?option ping-timeout 42 > ? ?4:? ? ?option remote-host nfs01 > ? ?5:? ? ?option remote-subvolume /bricks/0/gv01 > ? ?6:? ? ?option transport-type socket > ? ?7:? ? ?option transport.address-family inet > ? ?8:? ? ?option username 916ccf06-dc1d-467f-bc3d-f00a7449618f > ? ?9:? ? ?option password a44739e0-9587-411f-8e6a-9a6a4e46156c > ? 10:? ? ?option event-threads 8 > ? 11:? ? ?option transport.tcp-user-timeout 0 > ? 12:? ? ?option transport.socket.keepalive-time 20 > ? 13:? ? ?option transport.socket.keepalive-interval 2 > ? 14:? ? ?option transport.socket.keepalive-count 9 > ? 15:? ? ?option send-gids true > ? 16: end-volume > ? 17: > ? 18: volume gv01-client-1 > ? 19:? ? ?type protocol/client > ? 20:? ? ?option ping-timeout 42 > ? 21:? ? ?option remote-host nfs02 > ? 22:? ? ?option remote-subvolume /bricks/0/gv01 > ? 23:? ? ?option transport-type socket > ? 24:? ? ?option transport.address-family inet > ? 25:? ? ?option username 916ccf06-dc1d-467f-bc3d-f00a7449618f > ? 26:? ? ?option password a44739e0-9587-411f-8e6a-9a6a4e46156c > ? 27:? ? ?option event-threads 8 > ? 28:? ? ?option transport.tcp-user-timeout 0 > ? 29:? ? ?option transport.socket.keepalive-time 20 > ? 30:? ? ?option transport.socket.keepalive-interval 2 > ? 31:? ? ?option transport.socket.keepalive-count 9 > ? 32:? ? ?option send-gids true > ? 33: end-volume > ? 34: > ? 35: volume gv01-replicate-0 > ? 36:? ? ?type cluster/replicate > ? 37:? ? ?option afr-pending-xattr gv01-client-0,gv01-client-1 > ? 38:? ? ?option quorum-type auto > ? 39:? ? ?option use-compound-fops off > ? 40:? ? ?subvolumes gv01-client-0 gv01-client-1 > ? 41: end-volume > ? 42: > ? 43: volume gv01-dht > ? 44:? ? ?type cluster/distribute > ? 45:? ? ?option lock-migration off > ? 46:? ? ?subvolumes gv01-replicate-0 > ? 47: end-volume > ? 48: > ? 49: volume gv01-write-behind > ? 50:? ? ?type performance/write-behind > ? 51:? ? ?option cache-size 8MB > ? 52:? ? ?subvolumes gv01-dht > ? 53: end-volume > ? 54: > ? 55: volume gv01-read-ahead > ? 56:? ? ?type performance/read-ahead > ? 57:? ? ?subvolumes gv01-write-behind > ? 58: end-volume > ? 59: > ? 60: volume gv01-readdir-ahead > ? 61:? ? ?type performance/readdir-ahead > ? 62:? ? ?option parallel-readdir off > ? 63:? ? ?option rda-request-size 131072 > ? 64:? ? ?option rda-cache-limit 10MB > ? 65:? ? ?subvolumes gv01-read-ahead > ? 66: end-volume > ? 67: > ? 68: volume gv01-io-cache > ? 69:? ? ?type performance/io-cache > ? 70:? ? ?option cache-size 1GB > ? 71:? ? ?subvolumes gv01-readdir-ahead > ? 72: end-volume > ? 73: > ? 74: volume gv01-quick-read > ? 75:? ? ?type performance/quick-read > ? 76:? ? ?option cache-size 1GB > ? 77:? ? ?subvolumes gv01-io-cache > ? 78: end-volume > ? 79: > ? 80: volume gv01-open-behind > ? 81:? ? ?type performance/open-behind > ? 82:? ? ?subvolumes gv01-quick-read > ? 83: end-volume > ? 84: > ? 85: volume gv01-md-cache > ? 86:? ? ?type performance/md-cache > ? 87:? ? ?subvolumes gv01-open-behind > ? 88: end-volume > ? 89: > ? 90: volume gv01 > ? 91:? ? ?type debug/io-stats > ? 92:? ? ?option log-level INFO > ? 93:? ? ?option latency-measurement off > ? 94:? ? ?option count-fop-hits off > ? 95:? ? ?subvolumes gv01-md-cache > ? 96: end-volume > ? 97: > ? 98: volume meta-autoload > ? 99:? ? ?type meta > 100:? ? ?subvolumes gv01 > 101: end-volume > 102: > +------------------------------------------------------------------------------+ > [2018-04-09 05:08:13.922631] E [socket.c:2374:socket_connect_finish] > 0-gv01-client-1: connection to 192.168.0.119:24007 > <http://192.168.0.119:24007> failed (No route to > host); disconnecting socket > [2018-04-09 05:08:13.922690] E [MSGID: 108006] > [afr-common.c:5164:__afr_handle_child_down_event] 0-gv01-replicate-0: > All subvolumes are down. Going offline until atleast one of them comes > back up. > [2018-04-09 05:08:13.926201] I [fuse-bridge.c:4205:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 > kernel 7.22 > [2018-04-09 05:08:13.926245] I [fuse-bridge.c:4835:fuse_graph_sync] > 0-fuse: switched to graph 0 > [2018-04-09 05:08:13.926518] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes up > [2018-04-09 05:08:13.926671] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null > [2018-04-09 05:08:13.926762] E [fuse-bridge.c:4271:fuse_first_lookup] > 0-fuse: first lookup on root failed (Transport endpoint is not > connected) > [2018-04-09 05:08:13.927207] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes up > [2018-04-09 05:08:13.927262] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null > [2018-04-09 05:08:13.927301] W > [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: > 00000000-0000-0000-0000-000000000001: failed to resolve (Transport > endpoint is not connected) > [2018-04-09 05:08:13.927339] E [fuse-bridge.c:900:fuse_getattr_resume] > 0-glusterfs-fuse: 2: GETATTR 1 (00000000-0000-0000-0000-000000000001) > resolution failed > [2018-04-09 05:08:13.931497] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes up > [2018-04-09 05:08:13.931558] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null > [2018-04-09 05:08:13.931599] W > [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: > 00000000-0000-0000-0000-000000000001: failed to resolve (Transport > endpoint is not connected) > [2018-04-09 05:08:13.931623] E [fuse-bridge.c:900:fuse_getattr_resume] > 0-glusterfs-fuse: 3: GETATTR 1 (00000000-0000-0000-0000-000000000001) > resolution failed > [2018-04-09 05:08:13.937258] I [fuse-bridge.c:5093:fuse_thread_proc] > 0-fuse: initating unmount of /n > [2018-04-09 05:08:13.938043] W [glusterfsd.c:1393:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7e25) [0x7fb80b05ae25] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560b52471675] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560b5247149b] ) 0-: > received signum (15), shutting down > [2018-04-09 05:08:13.938086] I [fuse-bridge.c:5855:fini] 0-fuse: > Unmounting '/n'. > [2018-04-09 05:08:13.938106] I [fuse-bridge.c:5860:fini] 0-fuse: Closing > fuse connection to '/n'. > > ==> glusterd.log <=> [2018-04-09 05:08:15.118078] W [socket.c:3216:socket_connect] > 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > > ==> glustershd.log <=> [2018-04-09 05:08:15.282192] W [socket.c:3216:socket_connect] > 0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > [2018-04-09 05:08:15.289508] W [socket.c:3216:socket_connect] > 0-gv01-client-1: Error disabling sockopt IPV6_V6ONLY: "Protocol not > available" > > > > > > > > -- > Cheers, > Tom K. > ------------------------------------------------------------------------------------- > > Living on earth is expensive, but it includes a free trip around the > sun. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://lists.gluster.org/mailman/listinfo/gluster-users >-- Cheers, Tom K. ------------------------------------------------------------------------------------- Living on earth is expensive, but it includes a free trip around the sun.
Reasonably Related Threads
- volume start: gv01: failed: Quorum not met. Volume operation not allowed.
- volume start: gv01: failed: Quorum not met. Volume operation not allowed.
- volume start: gv01: failed: Quorum not met. Volume operation not allowed.
- volume start: gv01: failed: Quorum not met. Volume operation not allowed.
- [SOLVED] [Nfs-ganesha-support] volume start: gv01: failed: Quorum not met. Volume operation not allowed.