jrwillen at aol.com
2020-Sep-30 21:25 UTC
[Gluster-users] Help: Client application cannot open existing gluster file and write over it
Problem:? Although there are indications of trouble in the log?files, gluster appears normal on boot.? Clients can copy files?to the distributed file system, but disconnect when an application?tries to open an existing file and write over it. I hope this is a newbie installation error, but I'm tired of pounding?my head against the wall.? I started with a much more complex?application that had file-access problems and came up with this simple?example.? Thanks in advance for any guidance about how to address this?problem. Here is a short list of my observations, followed by?more?details?for?each one: 1) At boot, lots of complaints in the file system log2) At boot, glusterd log warns of errors through RPC layer and peers?being disconnected3) Client A can copy a directory to the gluster file system4) Client A can write a gluster file from an application5) Client B verifies the write6) Client A tries to open the file and overwrite, but crashes7) Client A is now disconnected from gluster; sees core dump in root8) After rebooting client A, file system log provides some details?of the crash.9) glusterd log doesn't provide any additional details10) Gluster was installed via apt, not compiled from source11) Running Raspbian GNU/Linux 10 (buster).? 32 RPI-4 clients.? Testing with 2?RPI-3b+ servers.?12) Clients see the 4 Tb distributed storage under /gfdata13) At boot, clusterd isn't always ready when the clients get to the?point of mounting disks, so I forcibly mount them with sudo mount -a14) The gluster volume looks sensible at boot time15) The pool looks okay too =================================Details================================1) At boot, lots of complaints in the file system log:pi at pi00:~ $ sudo cat /var/log/glusterfs/gfdata.log[2020-09-30 11:57:58.596738] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing[2020-09-30 11:57:59.049660] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing[2020-09-30 11:57:59.051730] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuingThe message "I [MSGID: 100011] [glusterfsd.c:1570:reincarnate] 0-glusterfsd: Fetching the volume file from server..." repeated 2 times between [2020-09-30 11:57:57.312930] and [2020-09-30 11:57:57.685434][2020-09-30 12:08:07.427162] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:08:07.427319] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:08:07.429373] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:08:17.161867] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:08:17.161877] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:08:17.163889] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:10:54.285700] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:10:54.285853] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:10:54.287936] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:10:54.701882] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in (null) (gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088). Holes=1 overlaps=0[2020-09-30 12:10:54.702033] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in (null) (gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6). Holes=1 overlaps=0[2020-09-30 12:10:54.702176] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in (null) (gfid = e81dc1be-b7f7-4baf-8049-1122c5844019). Holes=1 overlaps=0[2020-09-30 12:10:55.081494] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /examples (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0[2020-09-30 12:10:55.106417] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0[2020-09-30 12:10:55.167731] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test3 (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0[2020-09-30 12:11:06.349404] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:11:06.361029] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test3: Disk layout missing, gfid = e81dc1be-b7f7-4baf-8049-1122c5844019[2020-09-30 12:11:06.361248] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test3, gfid = e81dc1be-b7f7-4baf-8049-1122c5844019[2020-09-30 12:11:06.364042] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test3 (gfid = e81dc1be-b7f7-4baf-8049-1122c5844019). Holes=1 overlaps=0[2020-09-30 12:11:06.963973] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:06.964619] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.014836] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.014970] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.188570] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.189452] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.192062] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.192346] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.222268] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.222880] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.225861] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.227039] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.227411] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.227699] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.228019] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.231825] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-sizeThe message "I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001" repeated 2 times between [2020-09-30 12:10:54.285853] and [2020-09-30 12:11:07.410603][2020-09-30 12:11:07.412422] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0The message "I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001" repeated 3 times between [2020-09-30 12:10:54.285700] and [2020-09-30 12:11:10.533318][2020-09-30 12:11:10.533328] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:11:10.551494] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:11:10.574455] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /examples: Disk layout missing, gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088[2020-09-30 12:11:10.574620] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /examples, gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088[2020-09-30 12:11:10.577251] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /examples (gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088). Holes=1 overlaps=0[2020-09-30 12:11:10.592811] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test: Disk layout missing, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6[2020-09-30 12:11:10.592927] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6[2020-09-30 12:11:10.595673] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test (gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6). Holes=1 overlaps=0[2020-09-30 12:14:35.163945] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:14:35.164099] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:14:35.166219] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:14:35.178003] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test: Disk layout missing, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6[2020-09-30 12:14:35.178111] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6[2020-09-30 12:14:35.180824] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test (gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6). Holes=1 overlaps=0[2020-09-30 12:14:35.694281] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.694455] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.695229] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.698612] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.698766] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.699413] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.184500] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.191364] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.192896] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.193355] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.193607] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.195633] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:17:39.677016] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:17:39.677160] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:17:39.679343] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 2) On boot, glusterd log warns of errors through RPC layer and peers?being disconnectedpi at pi00:~ $ sudo cat /var/log/glusterfs/glusterd.log[2020-09-30 11:57:57.678768] E [MSGID: 106167] [glusterd-handshake.c:1996:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later[2020-09-30 11:57:57.678999] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(2)), xid = 0x5, unique = 0, sent = 2020-09-29 21:30:38.833820, timeout = 600 for 192.168.1.79:24007[2020-09-30 11:57:57.679015] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8004fcb7-2a9f-481a-8e2d-79409ea39375[2020-09-30 11:57:58.276231] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:57:59.341029] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Gluster MGMT Handshake), op(MGMT-VERS-ACK(2)), xid = 0x4, unique = 0, sent = 2020-09-29 21:30:47.473226, timeout = 600 for 192.168.1.35:24007[2020-09-30 11:57:59.341224] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 83800431-f92d-4c2b-a402-160bdd4a4935[2020-09-30 11:57:59.341328] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:57:59.379583] E [MSGID: 106167] [glusterd-handshake.c:1885:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Error through RPC layer, retry again later[2020-09-30 11:57:59.379769] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(2)), xid = 0x5, unique = 0, sent = 2020-09-29 21:30:45.579517, timeout = 600 for 192.168.1.19:24007[2020-09-30 11:57:59.379886] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 03d20361-606d-4040-8b70-76473821706a[2020-09-30 11:57:59.379945] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (5)[2020-09-30 11:57:59.380004] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.19:24007[2020-09-30 11:57:59.380154] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.19> (<95e08fe4-534e-4388-94f9-c4db97b51795>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:57:59.380374] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:57:59.380419] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:00.137761] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a[2020-09-30 11:58:00.379959] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x7, unique = 0, sent = 2020-09-29 21:30:36.588912, timeout = 600 for 192.168.1.61:24007[2020-09-30 11:58:04.189199] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.189357] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.189444] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:36.557644, timeout = 600 for 192.168.1.61:24007[2020-09-30 11:58:04.189506] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.189574] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.189686] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:42.471874, timeout = 600 for 192.168.1.13:24007[2020-09-30 11:58:04.189744] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.189863] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.189956] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:48.137287, timeout = 600 for 192.168.1.52:24007[2020-09-30 11:58:04.190015] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.190083] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.190213] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x7, unique = 0, sent = 2020-09-29 21:30:33.681570, timeout = 600 for 192.168.1.48:24007[2020-09-30 11:58:04.190271] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.190339] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.190881] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:47.454357, timeout = 600 for 192.168.1.15:24007[2020-09-30 11:58:04.190946] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.191018] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.191150] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(2)), xid = 0x5, unique = 0, sent = 2020-09-29 21:30:49.384128, timeout = 600 for 192.168.1.69:24007[2020-09-30 11:58:04.191239] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:49.399827, timeout = 600 for 192.168.1.20:24007[2020-09-30 11:58:04.191295] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.191362] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.191614] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.192070] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.20:24007[2020-09-30 11:58:04.192233] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.20> (<ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.192628] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.192718] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.193072] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.27> (<1e90f804-99fa-43b8-8df3-07d5da49c0da>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.193768] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.194105] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.194398] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (7)[2020-09-30 11:58:04.194964] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.48:24007[2020-09-30 11:58:04.195226] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.48> (<40e7d4fe-702d-4b5d-93f0-cf2ff4880f27>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.195604] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.195686] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.196108] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:04.223503] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.223944] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.79 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.267435] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.268218] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (5)[2020-09-30 11:58:04.268377] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.79:24007[2020-09-30 11:58:04.268513] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.79> (<1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.268914] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.268994] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.269128] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.269189] E [rpc-clnt.c:346:saved_frames_unwind] (null) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2020-09-30 11:58:04.246959 (xid=0x6)[2020-09-30 11:58:04.269251] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.269321] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.269608] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.269754] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.52:24007[2020-09-30 11:58:04.269920] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.52> (<6beeff80-a573-4571-b83e-2c14e3add80a>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.270294] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.270376] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.271032] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:04.292492] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.292896] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.53 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.329487] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9[2020-09-30 11:58:04.346693] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.347000] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.13 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.420122] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:04.455643] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.455779] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.15:24007[2020-09-30 11:58:04.455902] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.15> (<8b612b17-7f2b-47b5-a27a-c1e87b845241>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.456220] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.456293] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.456643] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:04.472699] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.473023] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.70 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.528589] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.70> (<8f357d63-f96f-4a09-b3a7-b128e4780419>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.528934] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.528993] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.529256] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.35> (<99f71a38-0935-46a3-853e-ab09a1a4903a>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.529477] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.529642] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.530071] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.531818] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.61:24007[2020-09-30 11:58:04.531933] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.61> (<75306501-6238-4f4e-b562-74c6b10a614a>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.532228] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.532293] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.532875] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a[2020-09-30 11:58:04.552980] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.553292] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.65 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.599958] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (5)[2020-09-30 11:58:04.600235] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.69:24007[2020-09-30 11:58:04.601589] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.69> (<d98ebd55-514c-4c00-ab5b-7d06ba7b402d>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.601987] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.602067] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.602464] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:04.669632] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c, host: 192.168.1.53, port: 0[2020-09-30 11:58:04.728815] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.728949] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.13:24007[2020-09-30 11:58:04.729039] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.13> (<4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.729308] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.729357] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.729445] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.729481] E [rpc-clnt.c:346:saved_frames_unwind] (null) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2020-09-30 11:58:04.402849 (xid=0x7)[2020-09-30 11:58:04.729519] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.729562] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.729844] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 6ee2eb12-592f-486c-83dd-b9508066fe51, host: 192.168.1.47, port: 0[2020-09-30 11:58:04.754469] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 6ee2eb12-592f-486c-83dd-b9508066fe51[2020-09-30 11:58:04.959493] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:05.586162] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:05.869649] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:06.643933] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:06.975254] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:14.481239] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: b93ce138-cce4-4a95-994a-319a0387efb3[2020-09-30 11:58:14.544391] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:14.544614] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.68 (0), ret: 0, op_ret: 0[2020-09-30 11:58:14.614575] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8004fcb7-2a9f-481a-8e2d-79409ea39375[2020-09-30 11:58:14.614718] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 6ee2eb12-592f-486c-83dd-b9508066fe51[2020-09-30 11:58:14.614809] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:14.663601] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9[2020-09-30 11:58:14.903834] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:15.739950] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a[2020-09-30 11:58:16.030240] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:16.564846] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d[2020-09-30 11:58:16.565899] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a[2020-09-30 11:58:16.566003] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:16.566134] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:16.777107] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:17.278005] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:17.278249] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.69 (0), ret: 0, op_ret: 0[2020-09-30 11:58:18.515292] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: b93ce138-cce4-4a95-994a-319a0387efb3[2020-09-30 11:58:18.666299] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:18.949557] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:18.956347] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:18.956620] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.19 (0), ret: 0, op_ret: 0[2020-09-30 11:58:19.011957] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: b93ce138-cce4-4a95-994a-319a0387efb3[2020-09-30 11:58:19.014177] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:19.014935] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:19.132556] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:19.464173] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:19.516527] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:19.543397] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:19.549901] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:19.550132] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.61 (0), ret: 0, op_ret: 0[2020-09-30 11:58:19.576012] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9, host: 192.168.1.13, port: 0[2020-09-30 11:58:19.623547] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 75306501-6238-4f4e-b562-74c6b10a614a, host: 192.168.1.61, port: 0[2020-09-30 11:58:19.659552] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241, host: 192.168.1.15, port: 0[2020-09-30 11:58:19.669103] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9[2020-09-30 11:58:19.759223] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:21.934219] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:22.112225] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:22.516139] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd, host: 192.168.1.20, port: 0[2020-09-30 11:58:22.532749] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:22.729363] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:43.939426] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd[2020-09-30 11:58:43.986065] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:44.452170] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 1e90f804-99fa-43b8-8df3-07d5da49c0da, host: 192.168.1.27, port: 0[2020-09-30 11:58:44.463225] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 6beeff80-a573-4571-b83e-2c14e3add80a, host: 192.168.1.52, port: 0[2020-09-30 11:58:44.474079] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1e90f804-99fa-43b8-8df3-07d5da49c0da[2020-09-30 11:58:44.539423] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:45.327726] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 6beeff80-a573-4571-b83e-2c14e3add80a[2020-09-30 11:58:45.569036] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:45.810147] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:45.810337] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a, host: 192.168.1.35, port: 0[2020-09-30 11:58:45.820901] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:45.821070] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:45.821073] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:45.830232] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9[2020-09-30 11:58:45.830418] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:45.830615] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d[2020-09-30 11:58:45.907496] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:46.092804] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:46.093769] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:46.101145] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:46.101300] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.15 (0), ret: 0, op_ret: 0[2020-09-30 11:58:46.120581] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:46.188794] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:46.404964] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd[2020-09-30 11:58:46.405549] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 6beeff80-a573-4571-b83e-2c14e3add80a[2020-09-30 11:58:46.405686] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1e90f804-99fa-43b8-8df3-07d5da49c0da[2020-09-30 11:58:46.406132] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:46.406493] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:46.415743] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:46.492945] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:46.681793] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:46.682456] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:46.690024] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:46.690306] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.70 (0), ret: 0, op_ret: 0[2020-09-30 11:58:46.747693] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a[2020-09-30 11:58:46.747909] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a[2020-09-30 11:58:46.756824] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:46.757107] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.35 (0), ret: 0, op_ret: 0[2020-09-30 11:58:46.778487] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:46.849136] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:47.063765] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795, host: 192.168.1.19, port: 0[2020-09-30 11:58:47.073740] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 40e7d4fe-702d-4b5d-93f0-cf2ff4880f27, host: 192.168.1.48, port: 0[2020-09-30 11:58:47.084813] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419, host: 192.168.1.70, port: 0[2020-09-30 11:58:47.095474] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:47.183420] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:47.396429] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d, host: 192.168.1.69, port: 0[2020-09-30 11:58:47.407953] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a[2020-09-30 11:58:47.439796] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:47.674369] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 40e7d4fe-702d-4b5d-93f0-cf2ff4880f27[2020-09-30 11:58:47.780995] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:48.012688] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:48.076704] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:48.267824] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d[2020-09-30 11:58:48.343564] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:48.548045] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e, host: 192.168.1.79, port: 0[2020-09-30 11:58:48.559020] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a[2020-09-30 11:58:48.559193] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:48.559320] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 40e7d4fe-702d-4b5d-93f0-cf2ff4880f27[2020-09-30 11:58:48.559441] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:48.559670] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:48.631854] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:48.879914] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d[2020-09-30 11:58:48.880121] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:48.912996] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:48.939447] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:48.939731] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.53 (0), ret: 0, op_ret: 0[2020-09-30 11:58:49.049668] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:49.263217] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:49.728404] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:50.950398] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:50.951296] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:50.969707] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:50.985716] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:50.986057] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.79 (0), ret: 0, op_ret: 0[2020-09-30 11:58:51.017655] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:51.233916] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:51.724591] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 12:09:07.963638] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2020-09-30 12:09:07.964701] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2020-09-30 12:09:37.014817] I [MSGID: 106487] [glusterd-handler.c:1497:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 3) The bit about peers disconnecting from glusterd seems to be a red?herring:pi at pi00:~ $ ssh 192.168.1.15 'df 'Filesystem? ? ? ? ? 1K-blocks? ? ?Used? Available Use% Mounted on/dev/root? ? ? ? ? ? 30491968? 2517040? ?26703808? ?9% /devtmpfs? ? ? ? ? ? ? ?829976? ? ? ? 0? ? ?829976? ?0% /devtmpfs? ? ? ? ? ? ? ? ? 962072? ? ? ? 0? ? ?962072? ?0% /dev/shmtmpfs? ? ? ? ? ? ? ? ? 962072? ? ?9564? ? ?952508? ?1% /runtmpfs? ? ? ? ? ? ? ? ? ? 5120? ? ? ? 4? ? ? ?5116? ?1% /run/locktmpfs? ? ? ? ? ? ? ? ? 962072? ? ? ? 0? ? ?962072? ?0% /sys/fs/cgroup/dev/mmcblk0p1? ? ? ? ?258096? ? 54605? ? ?203491? 22% /boot192.168.1.41:/data 3907024444 39370916 3867653528? ?2% /gfdatatmpfs? ? ? ? ? ? ? ? ? 192412? ? ? ? 0? ? ?192412? ?0% /run/user/1000 3) Client A can copy a directory to the gluster file systempi at pi00:~ $ ls test3a.out? filetest.f95? filetest.f95~pi at pi00:~ $ cat test3/filetest.f95program mainimplicit noneinteger:: i, n, nmaxreal:: x(1024)!!? Try banging on a gluster file.!nmax = 1do n=1, nmax? do i=1, 1024? ? x(i) = float(n)*float(i)? ? enddo? open(10, file='testfile', form='unformatted')? write(10) x? close(10)? write(6,*) 'n = ', n? enddostopendpi at pi00:~ $ cp -r test3 /gfdata 4) Client A can write a gluster file from an applicationpi at pi00:~ $ cd /gfdata/test3pi at pi00:/gfdata/test3 $ dateWed 30 Sep 07:31:07 CDT 2020pi at pi00:/gfdata/test3 $ ./a.out?n =? ? ? ? ? ? 1 5) Client B verifies the writepi at pi10:~ $ ls -altr /gfdata/test3total 22-rwxrwxrwx 1 root root 8684 Sep 30 07:29 a.out-rwxrwxrwx 1 root root? 291 Sep 30 07:29 filetest.f95-rwxrwxrwx 1 root root? 265 Sep 30 07:29 filetest.f95~-rwxrwxrwx 1 root root 4104 Sep 30 07:31 testfiledrwxrwxrwx 1 root root 4096 Sep 30 07:31 ..drwxrwxrwx 1 root root 4096 Sep 30 07:31 .pi at pi10:~ $ od -f /gfdata/test3/testfile0000000? ? ? ? 5.74e-42? ? ? ? ? ? ? ?1? ? ? ? ? ? ? ?2? ? ? ? ? ? ? ?30000020? ? ? ? ? ? ? ?4? ? ? ? ? ? ? ?5? ? ? ? ? ? ? ?6? ? ? ? ? ? ? ?7...0007760? ? ? ? ? ? 1020? ? ? ? ? ? 1021? ? ? ? ? ? 1022? ? ? ? ? ? 10230010000? ? ? ? ? ? 1024? ? ? ? 5.74e-420010010pi at pi10:~ $ 6) Client A tries to open the file and write again, but crashespi at pi00:/gfdata/test3 $ dateWed 30 Sep 07:31:46 CDT 2020pi at pi00:/gfdata/test3 $ ./a.outAt line 14 of file filetest.f95 (unit = 10, file = 'testfile')Fortran runtime error: Software caused connection abort Error termination. Backtrace: 7) Client A is now disconnected from gluster.? Sees core dump in rootpi at pi00:/gfdata/test3 $ ls /ls: cannot access '/gfdata': Transport endpoint is not connectedbin? ?core? etc? ? ?home? lost+found? mnt? proc? run? ?srv? sys? usrboot? dev? ?gfdata? lib? ?media? ? ? ?opt? root? sbin? ssh? tmp? varpi at pi00:/gfdata/test3 $ dfdf: /gfdata: Transport endpoint is not connectedFilesystem? ? ?1K-blocks? ? Used Available Use% Mounted on/dev/root? ? ? ?30491968 3982908? 25237940? 14% /devtmpfs? ? ? ? ? 829976? ? ? ?0? ? 829976? ?0% /devtmpfs? ? ? ? ? ? ?962072? ? ? ?0? ? 962072? ?0% /dev/shmtmpfs? ? ? ? ? ? ?962072? ? 9580? ? 952492? ?1% /runtmpfs? ? ? ? ? ? ? ?5120? ? ? ?4? ? ? 5116? ?1% /run/locktmpfs? ? ? ? ? ? ?962072? ? ? ?0? ? 962072? ?0% /sys/fs/cgroup/dev/mmcblk0p1? ? 258096? ?54612? ? 203484? 22% /boottmpfs? ? ? ? ? ? ?192412? ? ? ?0? ? 192412? ?0% /run/user/1000 8)? After reboot, file system log provides some details of the crashpi at pi00:~ $ sudo cat /var/log/glusterfs/gfdata.log...[2020-09-30 12:29:44.427052] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:29:44.427212] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:29:44.429319] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:29:44.472259] E [MSGID: 114031] [client-rpc-fops_v2.c:216:client4_0_mkdir_cbk] 0-data-client-1: remote operation failed. Path: /test3 [No data available][2020-09-30 12:29:44.480524] W [MSGID: 114031] [client-rpc-fops_v2.c:1985:client4_0_setattr_cbk] 0-data-client-1: remote operation failed [Stale file handle]The message "I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0" repeated 2 times between [2020-09-30 12:29:44.429319] and [2020-09-30 12:31:02.580946][2020-09-30 12:31:02.592738] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test3: Disk layout missing, gfid = 9873ee11-e2e6-4ee8-9322-3401863dfa8f[2020-09-30 12:31:02.592966] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test3, gfid = 9873ee11-e2e6-4ee8-9322-3401863dfa8f[2020-09-30 12:31:02.595817] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test3 (gfid = 9873ee11-e2e6-4ee8-9322-3401863dfa8f). Holes=1 overlaps=0[2020-09-30 12:31:14.008297] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:14.009026] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-sizeThe message "I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001" repeated 2 times between [2020-09-30 12:29:44.427052] and [2020-09-30 12:31:02.578949]The message "I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001" repeated 2 times between [2020-09-30 12:29:44.427212] and [2020-09-30 12:31:02.578960][2020-09-30 12:31:48.616479] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:48.616780] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:48.627752] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:48.628042] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:48.635271] E [stripe.c:2949:stripe_ftruncate] 0-data-stripe-0: no stripe countpending frames:frame : type(1) op(FTRUNCATE)frame : type(1) op(FTRUNCATE)frame : type(1) op(OPEN)frame : type(0) op(0)patchset: git://git.gluster.org/glusterfs.gitsignal received: 6time of crash:2020-09-30 12:31:48configuration details:argp 1backtrace 1dlfcn 1libpthread 1llistxattr 1setfsid 1spinlock 1epoll.h 1xattr.h 1st_atim.tv_nsec 1package-string: glusterfs 5.5---------[2020-09-30 12:42:21.300134] I [MSGID: 100030] [glusterfsd.c:2715:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.5 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=192.168.1.41 --volfile-id=/data /gfdata)[2020-09-30 12:42:21.398194] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1[2020-09-30 12:42:21.694558] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2[2020-09-30 12:42:21.704186] I [MSGID: 114020] [client.c:2358:notify] 0-data-client-0: parent translators are ready, attempting connect on transport[2020-09-30 12:42:21.708976] I [MSGID: 114020] [client.c:2358:notify] 0-data-client-1: parent translators are ready, attempting connect on transportFinal graph:+------------------------------------------------------------------------------+? 1: volume data-client-0? 2:? ? ?type protocol/client? 3:? ? ?option ping-timeout 42? 4:? ? ?option remote-host 192.168.1.41? 5:? ? ?option remote-subvolume /mnt/ArmorA60/data? 6:? ? ?option transport-type socket? 7:? ? ?option transport.address-family inet? 8:? ? ?option transport.tcp-user-timeout 0? 9:? ? ?option transport.socket.keepalive-time 20?10:? ? ?option transport.socket.keepalive-interval 2?11:? ? ?option transport.socket.keepalive-count 9?12:? ? ?option send-gids true?13: end-volume?14:?15: volume data-client-1?16:? ? ?type protocol/client?17:? ? ?option ping-timeout 42?18:? ? ?option remote-host 192.168.1.20?19:? ? ?option remote-subvolume /mnt/ArmorA60/data?20:? ? ?option transport-type socket?21:? ? ?option transport.address-family inet?22:? ? ?option transport.tcp-user-timeout 0?23:? ? ?option transport.socket.keepalive-time 20?24:? ? ?option transport.socket.keepalive-interval 2?25:? ? ?option transport.socket.keepalive-count 9?26:? ? ?option send-gids true?27: end-volume?28:?29: volume data-stripe-0?30:? ? ?type cluster/stripe?31:? ? ?subvolumes data-client-0 data-client-1?32: end-volume?33:?34: volume data-dht?35:? ? ?type cluster/distribute?36:? ? ?option lock-migration off?37:? ? ?option force-migration off?38:? ? ?subvolumes data-stripe-0?39: end-volume?40:?41: volume data-write-behind?42:? ? ?type performance/write-behind?43:? ? ?subvolumes data-dht?44: end-volume?45:?46: volume data-read-ahead?47:? ? ?type performance/read-ahead?48:? ? ?subvolumes data-write-behind?49: end-volume?50:?51: volume data-readdir-ahead?52:? ? ?type performance/readdir-ahead?53:? ? ?option parallel-readdir off?54:? ? ?option rda-request-size 131072?55:? ? ?option rda-cache-limit 10MB?56:? ? ?subvolumes data-read-ahead?57: end-volume?58:?59: volume data-io-cache?60:? ? ?type performance/io-cache?61:? ? ?subvolumes data-readdir-ahead?62: end-volume?63:?64: volume data-quick-read?65:? ? ?type performance/quick-read?66:? ? ?subvolumes data-io-cache?67: end-volume?68:?69: volume data-open-behind?70:? ? ?type performance/open-behind?71:? ? ?subvolumes data-quick-read?72: end-volume?73:?74: volume data-md-cache?75:? ? ?type performance/md-cache?76:? ? ?subvolumes data-open-behind?77: end-volume?78:?79: volume data-io-threads?80:? ? ?type performance/io-threads?81:? ? ?subvolumes data-md-cache?82: end-volume?83:?84: volume data?85:? ? ?type debug/io-stats?86:? ? ?option log-level INFO?87:? ? ?option latency-measurement off?88:? ? ?option count-fop-hits off?89:? ? ?subvolumes data-io-threads?90: end-volume?91:?92: volume meta-autoload?93:? ? ?type meta?94:? ? ?subvolumes data?95: end-volume?96:+------------------------------------------------------------------------------+[2020-09-30 12:42:21.718355] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-data-client-1: changing port to 49152 (from 0)[2020-09-30 12:42:21.718726] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-data-client-0: changing port to 49152 (from 0)[2020-09-30 12:42:21.725344] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-data-client-1: Connected to data-client-1, attached to remote volume '/mnt/ArmorA60/data'.[2020-09-30 12:42:21.731976] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-data-client-0: Connected to data-client-0, attached to remote volume '/mnt/ArmorA60/data'.[2020-09-30 12:42:21.747464] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.31[2020-09-30 12:42:21.747593] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0[2020-09-30 12:42:21.760574] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:42:21.776484] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:42:21.776598] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:42:21.778843] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 9) glusterd log doesn't provide any additional details...[2020-09-30 12:09:07.963638] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2020-09-30 12:09:07.964701] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2020-09-30 12:09:37.014817] I [MSGID: 106487] [glusterd-handler.c:1497:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2020-09-30 12:41:50.915448] W [MSGID: 100032] [glusterfsd.c:1500:cleanup_and_exit] 0-: received signum (15), shutting down[2020-09-30 12:42:10.042885] I [MSGID: 100030] [glusterfsd.c:2715:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 5.5 (args: /usr/sbin/glusterd -p /run/glusterd.pid --log-level INFO)[2020-09-30 12:42:10.247943] I [MSGID: 106478] [glusterd.c:1435:init] 0-management: Maximum allowed open file descriptors set to 65536[2020-09-30 12:42:10.248270] I [MSGID: 106479] [glusterd.c:1491:init] 0-management: Using /var/lib/glusterd as working directory[2020-09-30 12:42:10.248356] I [MSGID: 106479] [glusterd.c:1497:init] 0-management: Using /var/run/gluster as pid file working directory[2020-09-30 12:42:10.313231] W [MSGID: 103071] [rdma.c:4475:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device][2020-09-30 12:42:10.313405] W [MSGID: 103055] [rdma.c:4774:init] 0-rdma.management: Failed to initialize IB Device[2020-09-30 12:42:10.313500] W [rpc-transport.c:339:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed[2020-09-30 12:42:10.314354] W [rpcsvc.c:1789:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed[2020-09-30 12:42:10.314450] E [MSGID: 106244] [glusterd.c:1798:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport[2020-09-30 12:42:20.719035] I [MSGID: 106513] [glusterd-store.c:2282:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 50400[2020-09-30 12:42:20.734156] I [MSGID: 106544] [glusterd.c:152:glusterd_uuid_init] 0-management: retrieved UUID: 315aa9d5-76a8-4221-9187-4727b283406a[2020-09-30 12:42:20.783134] I [MSGID: 106498] [glusterd-handler.c:3647:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0The message "I [MSGID: 106498] [glusterd-handler.c:3647:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0" repeated 32 times between [2020-09-30 12:42:20.783134] and [2020-09-30 12:42:20.817711][2020-09-30 12:42:20.817930] W [MSGID: 106061] [glusterd-handler.c:3453:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout 10) Gluster was installed via apt, not compiled from source 11) Running Raspbian GNU/Linux 10 (buster).? 32 RPI-4 clients, 2RPI-3b+ servers.? I'll upgrade to 4 RPI-4 servers when I'm confidentgluster is working correctly. 12) Clients see the 4 Tb distributed storage under /gfdatapi at pi00:~ $ ls /bin? ?dev? gfdata? lib? ? ? ? ?media? opt? ?root? sbin? ssh? tmp? varboot? etc? home? ? lost+found? mnt? ? proc? run? ?srv? ?sys? usrpi at pi00:~ $ dfFilesystem? ? ? ? ? 1K-blocks? ? ?Used? Available Use% Mounted on/dev/root? ? ? ? ? ? 30491968? 3982860? ?25237988? 14% /devtmpfs? ? ? ? ? ? ? ?829976? ? ? ? 0? ? ?829976? ?0% /devtmpfs? ? ? ? ? ? ? ? ? 962072? ? ? ? 0? ? ?962072? ?0% /dev/shmtmpfs? ? ? ? ? ? ? ? ? 962072? ? ?9580? ? ?952492? ?1% /runtmpfs? ? ? ? ? ? ? ? ? ? 5120? ? ? ? 4? ? ? ?5116? ?1% /run/locktmpfs? ? ? ? ? ? ? ? ? 962072? ? ? ? 0? ? ?962072? ?0% /sys/fs/cgroup/dev/mmcblk0p1? ? ? ? ?258096? ? 54612? ? ?203484? 22% /boot192.168.1.41:/data 3907024444 39371016 3867653428? ?2% /gfdatatmpfs? ? ? ? ? ? ? ? ? 192412? ? ? ? 0? ? ?192412? ?0% /run/user/1000pi at pi00:~ $ 13) At boot, clusterd isn't always ready when the clients get to the?point of mounting disks, so I forcibly mount them with sudo mount -a pi at pi00:~/gluster-build $ head check1echo pi00 ; ls /gfdata/testssh pi at 192.168.1.79 'echo pi01 ; ls /gfdata/test'ssh pi at 192.168.1.69 'echo pi02 ; ls /gfdata/test'ssh pi at 192.168.1.70 'echo pi03 ; ls /gfdata/test'ssh pi at 192.168.1.65 'echo pi04 ; ls /gfdata/test'ssh pi at 192.168.1.66 'echo pi05 ; ls /gfdata/test'ssh pi at 192.168.1.64 'echo pi06 ; ls /gfdata/test'... pi at pi00:~/gluster-build $ ./check1pi00a.out? checkout.txt? reset_aout? test_mkdir? testmpi.f? testmpi.f90...pi10ls: cannot access '/gfdata/test': No such file or directorypi11ls: cannot access '/gfdata/test': No such file or directorypi12... 14) The gluster volume looks sensiblepi at pi00:~ $ sudo gluster volume info Volume Name: dataType: StripeVolume ID: ee3fc5a6-edae-4896-b3af-a7bf2a2d33c1Status: StartedSnapshot Count: 0Number of Bricks: 1 x 2 = 2Transport-type: tcpBricks:Brick1: 192.168.1.41:/mnt/ArmorA60/dataBrick2: 192.168.1.20:/mnt/ArmorA60/dataOptions Reconfigured:nfs.disable: ontransport.address-family: inet 15) The pool looks okay toopi at pi00:~ $ sudo gluster pool listUUID? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Hostname? ? ? ? State9709d800-4920-4e1b-b531-b952b2520323? ? 192.168.1.14? ? Connected75306501-6238-4f4e-b562-74c6b10a614a? ? 192.168.1.61? ? Connected4c1e76fd-0458-46b4-bd4a-cbb8dd9c508b? ? 192.168.1.66? ? Connectede29d6f3a-8a32-4d20-9388-e2392054557b? ? 192.168.1.34? ? Connected03d20361-606d-4040-8b70-76473821706a? ? 192.168.1.43? ? Connecteddc400830-fef2-4039-85d3-92a3085d528e? ? 192.168.1.41? ? Connectedab97b2b5-2e15-4358-86d0-62324b93007a? ? 192.168.1.65? ? Connectedc56df059-0f40-4138-91be-e410a9b776db? ? 192.168.1.32? ? Connected8f357d63-f96f-4a09-b3a7-b128e4780419? ? 192.168.1.70? ? Connected1e90f804-99fa-43b8-8df3-07d5da49c0da? ? 192.168.1.27? ? Connected8004fcb7-2a9f-481a-8e2d-79409ea39375? ? 192.168.1.33? ? Connected4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9? ? 192.168.1.13? ? Connectedb3d4b8f9-9a56-4bed-b10b-a9bb895ab746? ? 192.168.1.54? ? Connected6beeff80-a573-4571-b83e-2c14e3add80a? ? 192.168.1.52? ? Connecteda40cc7c3-ee91-4ea9-9059-a89a3bd4fad2? ? 192.168.1.55? ? Connectedb93ce138-cce4-4a95-994a-319a0387efb3? ? 192.168.1.68? ? Connected83800431-f92d-4c2b-a402-160bdd4a4935? ? 192.168.1.16? ? Connected8b612b17-7f2b-47b5-a27a-c1e87b845241? ? 192.168.1.15? ? Connectede71a2ffe-0463-492c-a367-e6dc7a95845c? ? 192.168.1.81? ? Connected603f4f57-e270-4a7d-a29f-6e2376aee8a1? ? 192.168.1.29? ? Connected95e08fe4-534e-4388-94f9-c4db97b51795? ? 192.168.1.19? ? Connectedd98ebd55-514c-4c00-ab5b-7d06ba7b402d? ? 192.168.1.69? ? Connected0fac620b-761c-49c9-a019-9c996dd5c4a8? ? 192.168.1.64? ? Connected6ee2eb12-592f-486c-83dd-b9508066fe51? ? 192.168.1.47? ? Connectede239fc30-5f0b-447a-9742-fd5512fc7eac? ? 192.168.1.60? ? Connected1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e? ? 192.168.1.79? ? Connected99f71a38-0935-46a3-853e-ab09a1a4903a? ? 192.168.1.35? ? Connected096a50a2-2d86-450f-9f7d-929e91444e2e? ? 192.168.1.28? ? Connectedae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd? ? 192.168.1.20? ? Connected3d64bd36-4078-49af-b251-07d9cbfddf9c? ? 192.168.1.53? ? Connecteddaf3673f-91e6-4b11-b119-8a3be918cca7? ? 192.168.1.44? ? Connected92728258-7422-41f3-b095-b733f858d49c? ? 192.168.1.23? ? Connected40e7d4fe-702d-4b5d-93f0-cf2ff4880f27? ? 192.168.1.48? ? Connected315aa9d5-76a8-4221-9187-4727b283406a? ? localhost? ? ? ?Connected -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200930/e0b2e424/attachment.html>