On March 12, 2020 6:03:48 AM GMT+02:00, John Burt <johnburt.jab at
gmail.com> wrote:>I'm struggling to keep bricks online, where they keep crashing almost
>immediately after forcing the volume online. Any suggestions for
>troubleshooting?
>
>Here is an excerpt from the logs from one of the bricks going offline.
>
>[2020-03-12 03:46:03.589778] W [socket.c:774:__socket_rwv]
>0-tcp.gvolume0-server: readv on 192.168.1.204:49005 failed (No data
>available)
>[2020-03-12 03:46:03.589863] I [MSGID: 115036]
>[server.c:501:server_rpc_notify] 0-gvolume0-server: disconnecting
>connection from
>CTX_ID:a9d855a5-5db9-466f-9b59-7486e5e95671-GRAPH_ID:0-PID:1398-HOST:odroid4-PC_NAME:gvolume0-client-5-RECON_NO:-1
>[2020-03-12 03:46:03.590182] I [MSGID: 101055]
>[client_t.c:436:gf_client_unref] 0-gvolume0-server: Shutting down
>connection
>CTX_ID:a9d855a5-5db9-466f-9b59-7486e5e95671-GRAPH_ID:0-PID:1398-HOST:odroid4-PC_NAME:gvolume0-client-5-RECON_NO:-1
>[2020-03-12 03:46:04.713141] W [socket.c:774:__socket_rwv]
>0-tcp.gvolume0-server: readv on 192.168.1.219:49016 failed (No data
>available)
>[2020-03-12 03:46:04.713208] I [MSGID: 115036]
>[server.c:501:server_rpc_notify] 0-gvolume0-server: disconnecting
>connection from
>CTX_ID:589c452c-81fb-42b4-9e1a-67bea2930457-GRAPH_ID:0-PID:1314-HOST:odroid19-PC_NAME:gvolume0-client-5-RECON_NO:-1
>[2020-03-12 03:46:04.713490] I [MSGID: 101055]
>[client_t.c:436:gf_client_unref] 0-gvolume0-server: Shutting down
>connection
>CTX_ID:589c452c-81fb-42b4-9e1a-67bea2930457-GRAPH_ID:0-PID:1314-HOST:odroid19-PC_NAME:gvolume0-client-5-RECON_NO:-1
>[2020-03-12 03:46:05.624023] W [MSGID: 100032]
>[glusterfsd.c:1596:cleanup_and_exit] 0-: received signum (15), shutting
>down
>[2020-03-12 03:47:18.126938] I [MSGID: 100030] [glusterfsd.c:2867:main]
>0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version
>7.2
>(args: /usr/sbin/glusterfsd -s odroid20 --volfile-id
>gvolume0.odroid20.mnt-brick2-gvolume0 -p
>/var/run/gluster/vols/gvolume0/odroid20-mnt-brick2-gvolume0.pid -S
>/var/run/gluster/59a03312d88125a3.socket --brick-name
>/mnt/brick2/gvolume0
>-l /var/log/glusterfs/bricks/mnt-brick2-gvolume0.log --xlator-option
>*-posix.glusterd-uuid=07fa2e6a-43ea-4483-a4a9-f2321ae5ff86
>--process-name
>brick --brick-port 49154 --xlator-option
>gvolume0-server.transport.rdma.listen-port=49155 --xlator-option
>gvolume0-server.listen-port=49154
>--volfile-server-transport=socket,rdma)
>[2020-03-12 03:47:18.128162] I [glusterfsd.c:2594:daemonize]
>0-glusterfs:
>Pid of current running process is 13222
>[2020-03-12 03:47:18.136097] I [socket.c:957:__socket_server_bind]
>0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9
>[2020-03-12 03:47:18.138848] I [MSGID: 101190]
>[event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-03-12 03:47:18.138959] I [MSGID: 101190]
>[event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-03-12 03:47:18.149567] I
>[rpcsvc.c:2690:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
>Configured
>rpc.outstanding-rpc-limit with value 64
>[2020-03-12 03:47:18.151945] W [MSGID: 103071]
>[rdma.c:4472:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
>channel creation failed [No such device]
>[2020-03-12 03:47:18.151986] W [MSGID: 103055] [rdma.c:4782:init]
>0-rdma.gvolume0-server: Failed to initialize IB Device
>[2020-03-12 03:47:18.152013] W [rpc-transport.c:366:rpc_transport_load]
>0-rpc-transport: 'rdma' initialization failed
>[2020-03-12 03:47:18.152303] W [rpcsvc.c:1981:rpcsvc_create_listener]
>0-rpc-service: cannot create listener, initing the transport failed
>[2020-03-12 03:47:18.152328] E [MSGID: 115045]
>[server.c:1199:server_init]
>0-gvolume0-server: creation of 1 listeners failed, continuing with
>succeeded transport
>[2020-03-12 03:47:18.156858] I [socket.c:957:__socket_server_bind]
>0-socket.gvolume0-changelog: closing (AF_UNIX) reuse check socket 13
>[2020-03-12 03:47:18.157403] I [trash.c:2450:init] 0-gvolume0-trash: no
>option specified for 'eliminate', using NULL
>Final graph:
>+------------------------------------------------------------------------------+
> 1: volume gvolume0-posix
> 2: type storage/posix
> 3: option glusterd-uuid 07fa2e6a-43ea-4483-a4a9-f2321ae5ff86
> 4: option directory /mnt/brick2/gvolume0
> 5: option volume-id 5ffb0119-6f15-4b7c-9856-4747ba31b358
> 6: option fips-mode-rchecksum on
> 7: option shared-brick-count 1
> 8: option reserve-size 0
> 9: end-volume
> 10:
> 11: volume gvolume0-arbiter
> 12: type features/arbiter
> 13: subvolumes gvolume0-posix
> 14: end-volume
> 15:
> 16: volume gvolume0-trash
> 17: type features/trash
> 18: option trash-dir .trashcan
> 19: option brick-path /mnt/brick2/gvolume0
> 20: option trash-internal-op off
> 21: subvolumes gvolume0-arbiter
> 22: end-volume
> 23:
> 24: volume gvolume0-changelog
> 25: type features/changelog
> 26: option changelog-brick /mnt/brick2/gvolume0
>27: option changelog-dir /mnt/brick2/gvolume0/.glusterfs/changelogs
> 28: option changelog-barrier-timeout 120
> 29: subvolumes gvolume0-trash
> 30: end-volume
> 31:
> 32: volume gvolume0-bitrot-stub
> 33: type features/bitrot-stub
> 34: option export /mnt/brick2/gvolume0
> 35: option bitrot disable
> 36: subvolumes gvolume0-changelog
> 37: end-volume
> 38:
> 39: volume gvolume0-access-control
> 40: type features/access-control
> 41: subvolumes gvolume0-bitrot-stub
> 42: end-volume
> 43:
> 44: volume gvolume0-locks
> 45: type features/locks
> 46: option enforce-mandatory-lock off
> 47: subvolumes gvolume0-access-control
> 48: end-volume
> 49:
> 50: volume gvolume0-worm
> 51: type features/worm
> 52: option worm off
> 53: option worm-file-level off
> 54: option worm-files-deletable on
> 55: subvolumes gvolume0-locks
> 56: end-volume
> 57:
> 58: volume gvolume0-read-only
> 59: type features/read-only
> 60: option read-only off
> 61: subvolumes gvolume0-worm
> 62: end-volume
> 63:
> 64: volume gvolume0-leases
> 65: type features/leases
> 66: option leases off
> 67: subvolumes gvolume0-read-only
> 68: end-volume
> 69:
> 70: volume gvolume0-upcall
> 71: type features/upcall
> 72: option cache-invalidation off
> 73: subvolumes gvolume0-leases
> 74: end-volume
> 75:
> 76: volume gvolume0-io-threads
> 77: type performance/io-threads
> 78: subvolumes gvolume0-upcall
> 79: end-volume
> 80:
> 81: volume gvolume0-selinux
> 82: type features/selinux
> 83: option selinux on
> 84: subvolumes gvolume0-io-threads
> 85: end-volume
> 86:
> 87: volume gvolume0-marker
> 88: type features/marker
> 89: option volume-uuid 5ffb0119-6f15-4b7c-9856-4747ba31b358
>90: option timestamp-file
>/var/lib/glusterd/vols/gvolume0/marker.tstamp
> 91: option quota-version 0
> 92: option xtime off
> 93: option gsync-force-xtime off
> 94: option quota off
> 95: option inode-quota off
> 96: subvolumes gvolume0-selinux
> 97: end-volume
> 98:
> 99: volume gvolume0-barrier
>100: type features/barrier
>101: option barrier disable
>102: option barrier-timeout 120
>103: subvolumes gvolume0-marker
>104: end-volume
>105:
>106: volume gvolume0-index
>107: type features/index
>108: option index-base /mnt/brick2/gvolume0/.glusterfs/indices
>109: option xattrop-dirty-watchlist trusted.afr.dirty
>110: option xattrop-pending-watchlist trusted.afr.gvolume0-
>111: subvolumes gvolume0-barrier
>112: end-volume
>113:
>114: volume gvolume0-quota
>115: type features/quota
>116: option volume-uuid gvolume0
>117: option server-quota off
>118: option deem-statfs off
>119: subvolumes gvolume0-index
>120: end-volume
>121:
>122: volume /mnt/brick2/gvolume0
>123: type debug/io-stats
>124: option auth.addr./mnt/brick2/gvolume0.allow *
>125: option auth-path /mnt/brick2/gvolume0
>126: option
>auth.login.6b0eba26-2c74-4542-a792-7c07595e1c53.password
>9938f115-0d9e-4d7c-bc2b-8627dac85e42
>127: option auth.login./mnt/brick2/gvolume0.allow
>6b0eba26-2c74-4542-a792-7c07595e1c53
>128: option unique-id /mnt/brick2/gvolume0
>129: option log-level INFO
>130: option threads 16
>131: option latency-measurement off
>132: option count-fop-hits off
>133: option global-threading off
>134: subvolumes gvolume0-quota
>135: end-volume
>136:
>137: volume gvolume0-server
>138: type protocol/server
>139: option transport.socket.listen-port 49154
>140: option rpc-auth.auth-glusterfs on
>141: option rpc-auth.auth-unix on
>142: option rpc-auth.auth-null on
>143: option rpc-auth-allow-insecure on
>144: option transport.rdma.listen-port 49155
>145: option transport-type rdma
>146: option auth.login./mnt/brick2/gvolume0.allow
>6b0eba26-2c74-4542-a792-7c07595e1c53
>147: option
>auth.login.6b0eba26-2c74-4542-a792-7c07595e1c53.password
>9938f115-0d9e-4d7c-bc2b-8627dac85e42
>148: option auth-path /mnt/brick2/gvolume0
>149: option auth.addr./mnt/brick2/gvolume0.allow *
>150: option transport.socket.keepalive 1
>151: option transport.socket.ssl-enabled off
>152: option transport.socket.keepalive-time 20
>153: option transport.socket.keepalive-interval 2
>154: option transport.socket.keepalive-count 9
>155: option transport.listen-backlog 1024
>156: subvolumes /mnt/brick2/gvolume0
>157: end-volume
>158:
>+------------------------------------------------------------------------------+
>[2020-03-12 03:47:19.622547] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.219"
>[2020-03-12 03:47:19.622761] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.622837] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:9ba4a0b3-15ef-46da-a2c8-94e6425fb2fe-GRAPH_ID:0-PID:1543-HOST:odroid19-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.627203] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.203"
>[2020-03-12 03:47:19.627482] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.627879] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:94ee8e47-2d8b-4661-94ff-ec339c928883-GRAPH_ID:0-PID:2859-HOST:odroid3-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.638411] I [rpcsvc.c:866:rpcsvc_handle_rpc_call]
>0-rpc-service: spawned a request handler thread for queue 0
>[2020-03-12 03:47:19.639320] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.217"
>[2020-03-12 03:47:19.639372] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.639413] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:5ef50bb9-ca91-4834-bd0f-f1d1a4ebfe82-GRAPH_ID:0-PID:2860-HOST:odroid17-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.640263] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.211"
>[2020-03-12 03:47:19.640314] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.640352] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:305eee4b-944a-4929-b9d1-0d983cbe5e5c-GRAPH_ID:0-PID:2953-HOST:odroid11-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.651496] I [rpcsvc.c:866:rpcsvc_handle_rpc_call]
>0-rpc-service: spawned a request handler thread for queue 1
>[2020-03-12 03:47:19.652290] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.205"
>[2020-03-12 03:47:19.652348] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.652389] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:41955f0b-756a-4b23-abb5-5915065c7988-GRAPH_ID:0-PID:1841-HOST:odroid5-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.653788] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.215"
>[2020-03-12 03:47:19.653955] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.654030] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:955893d7-2363-4b8c-9859-cd025f9c5a9d-GRAPH_ID:0-PID:2877-HOST:odroid15-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.655833] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.202"
>[2020-03-12 03:47:19.655917] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.655975] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:5d06cc3b-20b3-4aa1-a9e4-9f2a8ad2ec60-GRAPH_ID:0-PID:2873-HOST:odroid2-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.656185] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.212"
>[2020-03-12 03:47:19.656253] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.656308] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:9733b48d-6568-4ac5-a7c4-7ee6dcbe5eb9-GRAPH_ID:0-PID:1736-HOST:odroid12-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.657924] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.209"
>[2020-03-12 03:47:19.658030] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.658090] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:d1b92e61-b6b9-4256-aec9-9e74928cfa6d-GRAPH_ID:0-PID:2882-HOST:odroid9-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.659290] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.214"
>[2020-03-12 03:47:19.659499] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.659683] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:89dc1052-5b31-4e14-a9b8-4289887a5539-GRAPH_ID:0-PID:2748-HOST:odroid14-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.660733] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.206"
>[2020-03-12 03:47:19.660798] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.660837] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:b60d4094-86c8-48e9-b227-9d0ff2145e08-GRAPH_ID:0-PID:2858-HOST:odroid6-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.669176] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.213"
>[2020-03-12 03:47:19.669282] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.669499] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:1b1e0280-9b53-4244-ba2b-c37f934126b9-GRAPH_ID:0-PID:1687-HOST:odroid13-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.674585] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.218"
>[2020-03-12 03:47:19.674671] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.674758] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:5fc6cfcf-6e58-48eb-894f-c16d4377af29-GRAPH_ID:0-PID:2888-HOST:odroid18-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.678824] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.201"
>[2020-03-12 03:47:19.678881] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.678918] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:ccc64172-a27b-45d4-9e61-a982e0c6129a-GRAPH_ID:0-PID:3038-HOST:odroid1-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.679921] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.207"
>[2020-03-12 03:47:19.680014] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.680088] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:b2aece5f-c214-4afe-9f35-debc378b148b-GRAPH_ID:0-PID:2880-HOST:odroid7-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.684127] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.208"
>[2020-03-12 03:47:19.684190] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.684232] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:d9be2285-e369-41cb-9333-8d8d8d631fe6-GRAPH_ID:0-PID:2870-HOST:odroid8-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.696075] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.216"
>[2020-03-12 03:47:19.696248] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.696321] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:885af054-aa6d-405a-b8e9-31a7a807fed7-GRAPH_ID:0-PID:2870-HOST:odroid16-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.698740] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.210"
>[2020-03-12 03:47:19.698809] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.698851] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:61a16350-2549-45cb-ac0a-717ea3c12415-GRAPH_ID:0-PID:2297-HOST:odroid10-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:19.770633] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.220"
>[2020-03-12 03:47:19.770679] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:19.770709] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:6b9faf45-b335-4054-a990-ca279994b816-GRAPH_ID:0-PID:13383-HOST:odroid20-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:20.237138] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.2"
>[2020-03-12 03:47:20.237195] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:dd5899a9-a5eb-4efc-a283-5dbc56e11f60-GRAPH_ID:0-PID:3417-HOST:localhost.localdomain-PC_NAME:gvolume0-client-5-RECON_NO:-4
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:20.239653] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.2"
>[2020-03-12 03:47:20.239704] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:dd5899a9-a5eb-4efc-a283-5dbc56e11f60-GRAPH_ID:2-PID:3417-HOST:localhost.localdomain-PC_NAME:gvolume0-client-5-RECON_NO:-2
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>[2020-03-12 03:47:21.135088] I [addr.c:54:compare_addr_and_update]
>0-/mnt/brick2/gvolume0: allowed = "*", received addr =
"192.168.1.204"
>[2020-03-12 03:47:21.135135] I [login.c:110:gf_auth] 0-auth/login:
>allowed
>user names: 6b0eba26-2c74-4542-a792-7c07595e1c53
>[2020-03-12 03:47:21.135162] I [MSGID: 115029]
>[server-handshake.c:549:server_setvolume] 0-gvolume0-server: accepted
>client from
>CTX_ID:0bb3b680-0594-45e2-894c-92761aa51ab7-GRAPH_ID:0-PID:1955-HOST:odroid4-PC_NAME:gvolume0-client-5-RECON_NO:-0
>(version: 7.2) with subvol /mnt/brick2/gvolume0
>pending frames:
>frame : type(1) op(SETXATTR)
>patchset: git://git.gluster.org/glusterfs.git
>signal received: 6
>time of crash:
>2020-03-12 03:48:01
>configuration details:
>argp 1
>backtrace 1
>dlfcn 1
>libpthread 1
>llistxattr 1
>setfsid 1
>spinlock 1
>epoll.h 1
>xattr.h 1
>st_atim.tv_nsec 1
>package-string: glusterfs 7.2
>
>
>Thanks,
>--
>
>John
Hi John,
My experience always tell me to upgrade to the latest version before starting
banging my head in the wall.
Can you update gluater to 7.3.
What is your op-version ?
Maybe more detailed logs are needed in order to find the reason.
If you increase to trace/debug level, do not keep that log level for too long,
as it takes a lot of space.
Best Regards,
Strahil Nikolov